How Does Fidelity of Implementation Matter? Using Multilevel Models to Detect Relationships Between Participant Outcomes and the Delivery and Receipt of Treatment

Abstract

Multilevel modeling techniques facilitated examination of relationships between fidelity indicators and outcomes associated with a summer literacy intervention. Three-level growth models were specified to capture the extent to which students experienced instruction and to demonstrate the ways in which dosage–response relationships manifest in program evaluation contexts. The observation that outcome-related deviations from program protocol occurred both at the provider and at the recipient levels suggests that evaluators will often need to conceptualize, measure, and model “treatment fidelity” as a multilevel, multidimensional construct.

Keywords

treatment fidelity summer school summer learning early childhood literacy

Get full access to this article

View all access options for this article.

References

Alexander

K. L.

Entwisle

D. R.

Olson

L. S.

(2001). Schools, achievement, and inequality: A seasonal perspective. Educational Evaluation and Policy Analysis, 23, 171–191.

Bellg

A. J.

Borrelli

Resnick

Hecht

Minicucci

D. S.

Ory

Czajkowski

(2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations for the NIH Behavior Change Consortium. Health Psychology, 23, 443–451.

Berman

McLaughlin

M. W.

(1976). Implementation of educational innovation. Educational Forum, 40, 345–370.

Bloom

H. S.

Hill

C. J.

Riccio

J. A.

(2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22, 551–575.

Borman

G. D.

Dowling

N. M.

(2006). The longitudinal achievement effects of multiyear summer school: Evidence from the Teach Baltimore randomized field trial. Educational Evaluation and Policy Analysis, 28, 25–48.

Borrelli

Sepinwall

Ernst

Bellg

A. J.

Czajkowski

Breger

Orwig

(2005). A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology, 73, 852–860.

Century

Rudnick

Freeman

(2010). A framework for measuring fidelity of implementation. A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31, 199–218.

Chen

H. T.

(1990). Theory-driven evaluations. Newbury Park, CA: Sage.

Children’s Educational Services. (1987). Test of Oral Reading Fluency (TORF). Eden Prairie, MN: Author.

10.

Cooper

Charlton

Valentine

J. C.

Muhlenbruck

(2000). Making the most of summer school: A meta-analytic and narrative review. Monographs of the Society for Research in Child Development, 65, Serial No. 260.

11.

Cooper

Nye

Charlton

Lindsay

Greathouse

(1996). The effects of summer vacation on achievement test scores: A narrative and meta-analytic review. Review of Educational Research, 66, 227–268.

12.

Dane

A. V.

Schneider

B. H.

(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.

13.

Dedrick

R. F.

Ferron

J. M

Hess

M. R.

Hogarty

K. Y.

Kromrey

J. D.

Lang

T. R.

Lee

R. S.

(2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79, 69–102.

14.

Donaldson

S. I.

(2007). Program Theory-Driven Evaluation Science: Strategies and Applications. Mahwah, NJ: Lawrence Erlbaum.

15.

Durlack

J. A.

DuPre

E. P.

(2008). Implementation matters: A review on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.

16.

Dusenbury

Brannigan

Falco

Hansen

W. B.

(2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256.

17.

Entwisle

D. R.

Alexander

K. L.

Olson

L. S.

(1997). Children, schools, and inequality. Boulder, CO: Westview Press.

18.

Fullan

Pomfret

(1977). Research on curriculum and instruction. Review of Educational Research, 47, 335–397.

19.

Greenwood

C. R.

(2009). Treatment integrity: Revisiting some big ideas. School Psychology Review 38, 547–553.

20.

Gresham

F. M.

(2009). Evolution of the treatment integrity concept: Current status and future directions. School Psychology Review, 38, 533–540.

21.

Hagermoser Sanetti

L. M.

Kratochwill

T. R.

(2008). Treatment integrity in behavioral consultation: Measurement, promotion, and outcomes. International Journal of Behavioral Consultation and Therapy, 4, 95–114.

22.

Hagermoser Sanetti

L. M.

Kratochwill

T. R.

(2009). Toward developing a science of treatment integrity: Introduction to the special series, School Psychology Review, 38, 445–459.

23.

Hall

G. E.

Loucks

S. F.

(1977). A developmental model for determining whether the treatment is actually implemented. American Educational Research Journal, 14, 263–276.

24.

Hall

G. E.

Loucks

S. F.

Rutherford

W. L.

Newlove

B. W.

(1975). Levels of use of the innovation: A framework for analyzing innovation adoption. Journal of Teacher Education, 26, 52–56.

25.

Heyns

(1978). Summer learning and the effects of schooling. New York, NY: Academic Press.

26.

Heyns

(1987). Schooling and cognitive development: Is there a season for learning? Child Development, 58, 1151–1160.

27.

Hulleman

C. S.

Cordray

D. S.

(2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110.

28.

Institute for Education Sciences. (2009). Request for Applications: Education Research Grants. Retrieved from http://ies.ed.gov/funding/pdf/2009–84305A.pdf

29.

(2002). Estimation of intervention effects with noncompliance: Alternative model specifications. Journal of Educational and Behavioral Statistics, 27, 385–409.

30.

Muthén

B. O.

(2001). Modeling of intervention effects with noncompliance: A latent variable approach for randomized trials. In Marcoulides

G. A.

Schumacker

R. E.

(Eds.), New developments and techniques in structural equation modeling (pp. 57–87). Mahwah, NJ: Lawrence Erlbaum.

31.

Jones

H. A.

Clark

A. T.

Power

T. J.

(2008). Expanding the concept of intervention integrity: A multidimensional model of participant engagement. Balance, 23, 4–5.

32.

McGrew

J. H.

Bond

G. R.

Dietzen

Salyers

(1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62, 670–678.

33.

McKenna

S. A.

Rosenfield

Gravois

T. A.

(2009). Evaluating the validity of instructional consultation: Level of Implementation Scale—Revised. School Psychology Review, 38, 496–509.

34.

Moncher

F. J.

Prinz

R. J.

(1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11, 247–266.

35.

Mowbray

C. T.

Bybee

Holter

Lewandowski

(2006). Validation of a fidelity rating instrument for consumer-operated services. American Journal of Evaluation, 27, 9–27.

36.

Mowbray

C. T.

Holter

M. C.

Teague

G. B.

Bybee

(2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340.

37.

National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction (NIH Pub. No. 00-4754). Rockville, MD: National Institutes of Health.

38.

National Research Council Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials, Mathematical Sciences Education Board, Center for Education, Division of Behavioral and Social Sciences and Education. (2004). On evaluating curricular effectiveness: Judging the quality of K-12 mathematics evaluations. Washington, DC: National Academies Press.

39.

Noell

G. H.

(2008). Research examining the relationships among consultation process, treatment integrity, and outcomes. In Erchul

W. P.

Sheridan

S. M.

(Eds.), Handbook of research in school consultation: Empirical foundations for the field (pp. 315–334). Mahwah, NJ: Lawrence Erlbaum.

40.

O’Donnell

C. L.

(2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84.

41.

Peterson

Homer

A. L.

Wonderlich

S. A.

(1982). The integrity of independent variables in behavior analysis. Journal of Applied Behavior Analysis, 15, 477–492.

42.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.

43.

Raudenbush

S. W.

Bryk

A. S.

Cheong

Y. F.

Congdon

R. T.

(2004). HLM 6: Hierarchical linear and nonlinear modeling. Chicago, IL: Scientific Software International.

44.

Raudenbush

S. W.

Xiaofeng

(2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199–213.

45.

Raudenbush

S. W.

Xiaofeng

(2001). Effects of study duration, frequency of observation, and sample size on power in studies of group differences in polynomial change. Psychological Methods, 6, 387–401.

46.

Scheirer

M. A.

(1987). Program theory and program implementation: Implications for evaluators. In Bickman

(Ed.), Using program theory in evaluation (New Directions for Program Evaluation, No. 33, pp. 59–76). San Francisco, CA: Jossey-Bass.

47.

Schulte

A. C.

Parker

Easton

(2009). Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38, 460–475.

48.

Sechrest

West

S. G.

Phillips

M. A.

Redner

Yeaton

(1979). Some neglected problems in evaluation research: Strength and integrity of treatments. In Sechrest

West

S. G.

Phillips

M. A.

Redner

Yeaton

(Eds.), Evaluation studies review annual (Vol. 4, pp. 15–35). Thousand Oaks, CA: Sage.

49.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

50.

Tindal

Marston

Deno

S. L.

(1983). The reliability of direct and repeated measurement (Research Rep. 109). Minneapolis: University of Minnesota Institute for Research on Learning Disabilities.

51.

Yeaton

W. H.

Sechrest

(1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49(2), 156–167.

52.

Zvoch

(2009). Treatment fidelity in multisite evaluation: A multilevel longitudinal examination of provider adherence status and change. American Journal of Evaluation, 30, 44–61.

53.

Zvoch

Letourneau

L. E.

Parker

R. P.

(2007). A multilevel multisite outcomes-by-implementation evaluation of an early childhood literacy model. American Journal of Evaluation, 28, 132–150.

54.

Zvoch

Stevens

J. J.

(2011). Summer school and summer learning: An examination of the short and longer term changes in student literacy. Early Education and Development, 22, 649–675.