In reading intervention research, implementation fidelity is assumed to be positively related to student outcomes, but the methods used to measure fidelity are often treated as an afterthought. Fidelity has been conceptualized and measured in many different ways, suggesting a lack of construct validity. One aspect of construct validity is the fidelity index of a measure. This methodological case study examined how different decisions in fidelity indices influence relative rank ordering of individuals on the construct of interest and influence our perception of the relation between the construct and intervention outcomes. Data for this study came from a large state-funded project to implement multi-tiered systems of support for early reading instruction. Analyses were conducted to determine whether the different fidelity indices are stable in relative rank ordering participants and if fidelity indices of dosage and adherence data influence researcher decisions on model building within a multilevel modeling framework. Results indicated that the fidelity indices resulted in different relations to outcomes with the most commonly used fidelity indices for both dosage and adherence being the worst performing. The choice of index to use should receive considerable thought during the design phase of an intervention study.
AbryT.HullemanC. S.Rimm-KaufmanS. E. (2015). Using indices of fidelity to intervention core components to identify program active ingredients. American Journal of Evaluation, 36(3), 320–338. https://doi.org/10.1177/1098214014557009
2.
BailetL. L.RepperK. K.PiastaS. B.MurphyS. P. (2009). Emergent literacy intervention for prekindergarteners at risk for reading failure. Journal of Learning Disabilities, 42(4), 336–355. https://doi.org/10.1177/0022219409335218
3.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.
4.
BennerG. J.NelsonJ. R.StageS. A.RalstonN. C. (2011). The influence of fidelity of implementation on the reading outcomes of middle school students experiencing reading difficulties. Remedial and Special Education, 32(1), 79–88. https://doi.org/10.1177/0741932510361265
5.
BrandD.HenleyA. J.DiGennaro ReedF. D.GrayE.CrabbsB. (2019). A review of published studies involving parametric manipulations of treatment integrity. Journal of Behavioral Education, 28(1), 1–26. https://doi.org/10.1007/s10864-018-09311-8
6.
BuckleyP.MooreB.BoardmanA. G.AryaD. J.MaulA. (2017). Validating a fidelity scale to understand intervention effects in classroom-based studies. American Educational Research Journal, 54(6), 1378–1413. https://doi.org/10.3102/0002831217726522
7.
CantrellS. C.AlmasiJ. F.CarterJ. C.RintamaaM. (2013). Reading intervention in middle and high schools: Implementation fidelity, teacher efficacy, and student achievement. Reading Psychology, 34(1), 26–58. https://doi.org/10.1080/02702711.2011.577695
8.
CenturyJ.CassataA. (2016). Implementation Research: Finding common ground on What, How, Why, Where, and Who. Review of Research in Education, 40(1), 169–215. https://doi.org/10.3102/0091732X16665332
9.
CoffeyD. J. (2013). Helping struggling adolescent readers: Is implementation of different components of scholastic’s read 180 associated with differences in student achievement gains?. University of Delaware.
10.
ConnorC. M.PiastaS. B.FishmanB.GlasneyS.SchatschneiderC.CroweE.UnderwoodP.MorrisonF. J. (2009). Individualizing student instruction precisely: Effects of child x instruction interaction on first graders’ literacy development. Child Development, 80(2), 77–100.
11.
CoyneM. D.OldhamA.DoughertyS. M.LeonardK.KoriakinT.GageN. A.BurnsD.GillisM. (2018). Evaluating the effects of supplemental reading intervention within an MTSS or RTI reading reform initiative using a regression discontinuity design. Exceptional Children, 84(4), 350–367.
12.
CoyneM. D.OldhamA.LeonardK.BurnsD.GageN. (2016). Delving into the details: Implementing multitiered K-3 reading supports in high-priority schools. New Directions for Child and Adolescent Development, 154, 67–85.
13.
DaneA. V.SchneiderB. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control?Clinical Psychology Review, 18(1), 23–45. https://doi.org/10.1016/S0272-7358(97)00043-3
14.
DarrowC. L. (2013). The effectiveness and precision of intervention fidelity measures in preschool intervention research. Early Education and Development, 24(8), 1137–1160. https://doi.org/10.1080/10409289.2013.765786
15.
DeweyE. N.Powell-SmithK. A.GoodR. H.KaminskiR. A. (2015). AcadienceTM reading technical adequacy brief. Dynamic Measurement Group, Inc.
16.
FienH.SmithJ. L. M.SmolkowskiK.BakerS. K.NelsonN. J.ChaparroE. (2015). An examination of the efficacy of a multitiered intervention on early reading outcomes for first grade students at risk for reading difficulties. Journal of Learning Disabilities, 48(6), 602–621.
17.
FogartyM.OslundE.SimmonsD. C.DavidJ.SimmonsL.AndersonL.ClemensN.RobertsG. (2014). Examining the effectiveness of a multi component reading comprehension intervention in Middle Schools: A focus on treatment fidelity. Educational Psychology Review, 26, 425–449. https://doi.org/10.1007/s10648-014-9270-6
18.
FullerJ. L.FienupD. M. (2018). A preliminary analysis of mastery criterion level: Effects on response maintenance. Behavior Analysis in Practice, 11(1), 1–8.
19.
GarcíaE.WeissE. (2018). Student absenteeism: Who misses school and how missing school matters for performance. Economic Policy Institute. https://www.epi.org/152438
20.
GerstenR.FuchsL. S.ComptonD.CoyneM. D.GreenwoodC.InnocentiM. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71(2), 149–164.
21.
GoodR. H.KaminskiR. A. (Eds.) (2002). Dynamic indicators of basic early literacy skills (6th ed). Institute for the Development of Educational Achievement.
22.
GreshamF. M.MacMillanD. L.Beebe-FrankenbergerM. E.BocianK. M. (2000). Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented?. Learning Disabilities Research & Practice, 15(4), 198–205.
23.
HillH. C.EricksonA. (2019). Using implementation fidelity to aid in interpreting program impacts: A brief review. Educational Researcher, 48(9), 590–598. https://doi.org/10.3102/0013189X19891436
24.
HornerR. H.ToddA. W.Lewis-PalmerT.IrvinL. K.SugaiG.BolandJ. B. (2004). The School-Wide Evaluation Tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6(1), 3–12. https://doi.org/10.1177/10983007040060010201
25.
MathesP. G. (2005). Proactive early interventions in reading. SRA/McGraw-Hill.
26.
MendiveS.WeilandC.YoshikawaH.SnowC. E. (2016). Opening the black box: Intervention fidelity in a randomized trial of a preschool teacher professional development program. Journal of Educational Psychology, 108(1), 130–145. https://doi.org/10.1037/edu0000047
27.
MesserD.NashG. (2018). An evaluation of the effectiveness of a computer-assisted reading intervention. Journal of Research in Reading, 41(1), 140–158. https://doi.org/10.1111/1467-9817.12107
28.
MowbrayC. T.HolterM. C.TeagueG. B.BybeeD. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3), 315–340.
29.
NagelkerkeN. J. D. (1991). Miscellanea A note on a general definition of the coefficient of determination. Biometrika, 78(3), 691–692.
30.
National Institute of Child Health and Human Development (NICHD). (2000). Report of the National Reading Panel: Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups (NIH 00-4754). U. S. Government Printing Office.
31.
NoellG. H. (2008). Research examining the relationships among consultation process, treatment integrity, and outcomes. In ErchulW. P.SheridanS. M. (Eds.), Handbook of research in school consultation (pp. 323–341). Taylor & Francis.
32.
NoellG. H.GansleK. A. (2014). The use of performance feedback to improve intervention implementation in schools. In SanettiL. M. H.KratochwillT. R. (Eds.), Treatment integrity: A foundation for evidence-based practice in applied psychology (pp. 161–184). American Psychological Association.
33.
O’DonnellC. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78(1), 33–84. https://doi.org/10.3102/0034654307313793
34.
ParsonL. (2012). Using student performance during a reading intervention to predict student outcomes and performance on accountability measures of reading [Doctoral dissertation]. University of Minnessota
35.
PaulsonR. I.PostR. L.HerinckxH. A.RisserP. (2002). Beyond components: Using fidelity scales to measure and assure choice in program implementation and quality assurance. Community Mental Health Journal, 38(2), 119–128. https://doi.org/10.1023/A:1014591020400
36.
PiastaS. B.JusticeL. M.McGintyA.MashburnA.SlocumL. (2015). A comprehensive examination of preschool teachers’ implementation fidelity when using a supplemental language and literacy curriculum. Child & Youth Care Forum, 44(5), 731–755. https://doi.org/10.1007/s10566-015-9305-2
37.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). SAGE.
38.
ReadyD. D. (2010). Socioeconomic disadvantage, school attendance, and early cognitive development: The differential effects of school exposure. Sociology of Education, 83(4), 271–286. https://doi.org/10.1177/0038040710383520
39.
RobertsG.LewisN.FallA.VaughnS. (2016). Implementation fidelity: Examples from the reading for understanding initiative. In RobertsG.VaughnS.BeretvasS. N.WongV. (Eds.), Treatment fidelity in studies of educational intervention (pp. 69–87). Routledge
40.
SanettiL. M. H.KratochwillT. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445–459.
41.
SchulteA. C.EastonJ. E.ParkerJ. (2009). Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38(4),460–475.
42.
ShadishW.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
43.
SutherlandK. S.ConroyM. A.McLeodB. D.AlginaJ.KunemundR. L. (2018). Factors Associated with Teacher Delivery of a Classroom-Based Tier 2 Prevention Program. Prevention Science, 19(2), 186–196. https://doi.org/10.1007/s11121-017-0832-y
44.
van DijkW.LaneH. B.GageN. A. (2019). The relation between implementation fidelity and students’ reading outcomes: A systematic review of the literature [Preprint]. EdArXiv. https://doi.org/10.35542/osf.io/vhrp5
45.
WarrenS. F.FeyM. E.YoderP. J. (2007). Differential treatment intensity research: A missing link to creating optimally effective communication interventions. Mental Retardation and Developmental Disabilities Research Reviews, 13(1), 70–77. https://doi.org/10.1002/mrdd.20139
46.
WoodcockR. W. (1998). Woodcock reading mastery tests: Normative update: WRMT-R/NU: Examiner’s manual: Forms G and H. NCS Pearson.
47.
ZvochK. (2016). Intent-to-treat and treatment take-up effects of summer school on the literacy outcomes of struggling early readers. In RobertsG.VaughnS.BeretvasS. N.WongV. (Eds.), Treatment fidelity in studies of educational intervention (pp. 80–99). Routledge.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
