Why Does Construct Validity Matter in Measuring Implementation Fidelity? A Methodological Case Study

Abstract

In reading intervention research, implementation fidelity is assumed to be positively related to student outcomes, but the methods used to measure fidelity are often treated as an afterthought. Fidelity has been conceptualized and measured in many different ways, suggesting a lack of construct validity. One aspect of construct validity is the fidelity index of a measure. This methodological case study examined how different decisions in fidelity indices influence relative rank ordering of individuals on the construct of interest and influence our perception of the relation between the construct and intervention outcomes. Data for this study came from a large state-funded project to implement multi-tiered systems of support for early reading instruction. Analyses were conducted to determine whether the different fidelity indices are stable in relative rank ordering participants and if fidelity indices of dosage and adherence data influence researcher decisions on model building within a multilevel modeling framework. Results indicated that the fidelity indices resulted in different relations to outcomes with the most commonly used fidelity indices for both dosage and adherence being the worst performing. The choice of index to use should receive considerable thought during the design phase of an intervention study.

Keywords

implementation fidelity construct validity adherence exposure intervention research

Get full access to this article

View all access options for this article.

References

Abry

Hulleman

C. S.

Rimm-Kaufman

S. E.

(2015). Using indices of fidelity to intervention core components to identify program active ingredients. American Journal of Evaluation, 36(3), 320–338. https://doi.org/10.1177/1098214014557009

Bailet

L. L.

Repper

K. K.

Piasta

S. B.

Murphy

S. P.

(2009). Emergent literacy intervention for prekindergarteners at risk for reading failure. Journal of Learning Disabilities, 42(4), 336–355. https://doi.org/10.1177/0022219409335218

Benjamini

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.

Benner

G. J.

Nelson

J. R.

Stage

S. A.

Ralston

N. C.

(2011). The influence of fidelity of implementation on the reading outcomes of middle school students experiencing reading difficulties. Remedial and Special Education, 32(1), 79–88. https://doi.org/10.1177/0741932510361265

Brand

Henley

A. J.

DiGennaro Reed

F. D.

Gray

Crabbs

(2019). A review of published studies involving parametric manipulations of treatment integrity. Journal of Behavioral Education, 28(1), 1–26. https://doi.org/10.1007/s10864-018-09311-8

Buckley

Moore

Boardman

A. G.

Arya

D. J.

Maul

(2017). Validating a fidelity scale to understand intervention effects in classroom-based studies. American Educational Research Journal, 54(6), 1378–1413. https://doi.org/10.3102/0002831217726522

Cantrell

S. C.

Almasi

J. F.

Carter

J. C.

Rintamaa

(2013). Reading intervention in middle and high schools: Implementation fidelity, teacher efficacy, and student achievement. Reading Psychology, 34(1), 26–58. https://doi.org/10.1080/02702711.2011.577695

Century

Cassata

(2016). Implementation Research: Finding common ground on What, How, Why, Where, and Who. Review of Research in Education, 40(1), 169–215. https://doi.org/10.3102/0091732X16665332

Coffey

D. J.

(2013). Helping struggling adolescent readers: Is implementation of different components of scholastic’s read 180 associated with differences in student achievement gains?. University of Delaware.

10.

Connor

C. M.

Piasta

S. B.

Fishman

Glasney

Schatschneider

Crowe

Underwood

Morrison

F. J.

(2009). Individualizing student instruction precisely: Effects of child x instruction interaction on first graders’ literacy development. Child Development, 80(2), 77–100.

11.

Coyne

M. D.

Oldham

Dougherty

S. M.

Leonard

Koriakin

Gage

N. A.

Burns

Gillis

(2018). Evaluating the effects of supplemental reading intervention within an MTSS or RTI reading reform initiative using a regression discontinuity design. Exceptional Children, 84(4), 350–367.

12.

Coyne

M. D.

Oldham

Leonard

Burns

Gage

(2016). Delving into the details: Implementing multitiered K-3 reading supports in high-priority schools. New Directions for Child and Adolescent Development, 154, 67–85.

13.

Dane

A. V.

Schneider

B. H.

(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45. https://doi.org/10.1016/S0272-7358(97)00043-3

14.

Darrow

C. L.

(2013). The effectiveness and precision of intervention fidelity measures in preschool intervention research. Early Education and Development, 24(8), 1137–1160. https://doi.org/10.1080/10409289.2013.765786

15.

Dewey

E. N.

Powell-Smith

K. A.

Good

R. H.

Kaminski

R. A.

(2015). AcadienceTM reading technical adequacy brief. Dynamic Measurement Group, Inc.

16.

Fien

Smith

J. L. M.

Smolkowski

Baker

S. K.

Nelson

N. J.

Chaparro

(2015). An examination of the efficacy of a multitiered intervention on early reading outcomes for first grade students at risk for reading difficulties. Journal of Learning Disabilities, 48(6), 602–621.

17.

Fogarty

Oslund

Simmons

D. C.

David

Simmons

Anderson

Clemens

Roberts

(2014). Examining the effectiveness of a multi component reading comprehension intervention in Middle Schools: A focus on treatment fidelity. Educational Psychology Review, 26, 425–449. https://doi.org/10.1007/s10648-014-9270-6

18.

Fuller

J. L.

Fienup

D. M.

(2018). A preliminary analysis of mastery criterion level: Effects on response maintenance. Behavior Analysis in Practice, 11(1), 1–8.

19.

García

Weiss

(2018). Student absenteeism: Who misses school and how missing school matters for performance. Economic Policy Institute. https://www.epi.org/152438

20.

Gersten

Fuchs

L. S.

Compton

Coyne

M. D.

Greenwood

Innocenti

M. S.

(2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71(2), 149–164.

21.

Good

R. H.

Kaminski

R. A.

(Eds.) (2002). Dynamic indicators of basic early literacy skills (6th ed). Institute for the Development of Educational Achievement.

22.

Gresham

F. M.

MacMillan

D. L.

Beebe-Frankenberger

M. E.

Bocian

K. M.

(2000). Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented?. Learning Disabilities Research & Practice, 15(4), 198–205.

23.

Hill

H. C.

Erickson

(2019). Using implementation fidelity to aid in interpreting program impacts: A brief review. Educational Researcher, 48(9), 590–598. https://doi.org/10.3102/0013189X19891436

24.

Horner

R. H.

Todd

A. W.

Lewis-Palmer

Irvin

L. K.

Sugai

Boland

J. B.

(2004). The School-Wide Evaluation Tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6(1), 3–12. https://doi.org/10.1177/10983007040060010201

25.

Mathes

P. G.

(2005). Proactive early interventions in reading. SRA/McGraw-Hill.

26.

Mendive

Weiland

Yoshikawa

Snow

C. E.

(2016). Opening the black box: Intervention fidelity in a randomized trial of a preschool teacher professional development program. Journal of Educational Psychology, 108(1), 130–145. https://doi.org/10.1037/edu0000047

27.

Messer

Nash

(2018). An evaluation of the effectiveness of a computer-assisted reading intervention. Journal of Research in Reading, 41(1), 140–158. https://doi.org/10.1111/1467-9817.12107

28.

Mowbray

C. T.

Holter

M. C.

Teague

G. B.

Bybee

(2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3), 315–340.

29.

Nagelkerke

N. J. D.

(1991). Miscellanea A note on a general definition of the coefficient of determination. Biometrika, 78(3), 691–692.

30.

National Institute of Child Health and Human Development (NICHD). (2000). Report of the National Reading Panel: Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups (NIH 00-4754). U. S. Government Printing Office.

31.

Noell

G. H.

(2008). Research examining the relationships among consultation process, treatment integrity, and outcomes. In Erchul

W. P.

Sheridan

S. M.

(Eds.), Handbook of research in school consultation (pp. 323–341). Taylor & Francis.

32.

Noell

G. H.

Gansle

K. A.

(2014). The use of performance feedback to improve intervention implementation in schools. In Sanetti

L. M. H.

Kratochwill

T. R.

(Eds.), Treatment integrity: A foundation for evidence-based practice in applied psychology (pp. 161–184). American Psychological Association.

33.

O’Donnell

C. L.

(2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78(1), 33–84. https://doi.org/10.3102/0034654307313793

34.

Parson

(2012). Using student performance during a reading intervention to predict student outcomes and performance on accountability measures of reading [Doctoral dissertation]. University of Minnessota

35.

Paulson

R. I.

Post

R. L.

Herinckx

H. A.

Risser

(2002). Beyond components: Using fidelity scales to measure and assure choice in program implementation and quality assurance. Community Mental Health Journal, 38(2), 119–128. https://doi.org/10.1023/A:1014591020400

36.

Piasta

S. B.

Justice

L. M.

McGinty

Mashburn

Slocum

(2015). A comprehensive examination of preschool teachers’ implementation fidelity when using a supplemental language and literacy curriculum. Child & Youth Care Forum, 44(5), 731–755. https://doi.org/10.1007/s10566-015-9305-2

37.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). SAGE.

38.

Ready

D. D.

(2010). Socioeconomic disadvantage, school attendance, and early cognitive development: The differential effects of school exposure. Sociology of Education, 83(4), 271–286. https://doi.org/10.1177/0038040710383520

39.

Roberts

Lewis

Fall

Vaughn

(2016). Implementation fidelity: Examples from the reading for understanding initiative. In Roberts

Vaughn

Beretvas

S. N.

Wong

(Eds.), Treatment fidelity in studies of educational intervention (pp. 69–87). Routledge

40.

Sanetti

L. M. H.

Kratochwill

T. R.

(2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445–459.

41.

Schulte

A. C.

Easton

J. E.

Parker

(2009). Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38(4),460–475.

42.

Shadish

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

43.

Sutherland

K. S.

Conroy

M. A.

McLeod

B. D.

Algina

Kunemund

R. L.

(2018). Factors Associated with Teacher Delivery of a Classroom-Based Tier 2 Prevention Program. Prevention Science, 19(2), 186–196. https://doi.org/10.1007/s11121-017-0832-y

44.

van Dijk

Lane

H. B.

Gage

N. A

. (2019). The relation between implementation fidelity and students’ reading outcomes: A systematic review of the literature [Preprint]. EdArXiv. https://doi.org/10.35542/osf.io/vhrp5

45.

Warren

S. F.

Fey

M. E.

Yoder

P. J.

(2007). Differential treatment intensity research: A missing link to creating optimally effective communication interventions. Mental Retardation and Developmental Disabilities Research Reviews, 13(1), 70–77. https://doi.org/10.1002/mrdd.20139

46.

Woodcock

R. W.

(1998). Woodcock reading mastery tests: Normative update: WRMT-R/NU: Examiner’s manual: Forms G and H. NCS Pearson.

47.

Zvoch

(2016). Intent-to-treat and treatment take-up effects of summer school on the literacy outcomes of struggling early readers. In Roberts

Vaughn

Beretvas

S. N.

Wong

(Eds.), Treatment fidelity in studies of educational intervention (pp. 80–99). Routledge.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.32 MB