K–12 intervention studies often include fidelity of implementation (FOI) as a mediating variable, though most do not report the validity of fidelity measures. This article discusses the critical need for validated FOI scales. To illustrate our point, we describe the development and validation of the Implementation Validity Checklist (IVC-R), an observation tool for measuring FOI of a research-based instructional reading approach, Collaborative Strategic Reading. Following Kane (2006), Wilson (2004), and the guidelines of the Standards for Educational and Psychological Testing (Standards: AERA, APA, & NCME, 2014), findings suggest the IVC-R is a valid instrument for measuring fidelity to CSR. We hope this process will provide an informative model for the validation of FOI observation tools in future classroom-based efficacy studies.
AbbottM. L.FoutsJ. T. (2003). Constructivist teaching and student achievement: The results of a school-level classroom observation study in Washington. Lynnwood, WA: Washington School Research Center. Retrieved from: http://eric.ed.gov/?id=ED481694
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington DC: American Educational Research Association.
3.
BelskyJ.Booth-LaForceC. L.BradleyR.BrownellC. A.BurchinalM.CampbellS. B.WeinraubM. (2005). A day in third grade: A large-scale study of classroom quality and teacher and student behavior. Elementary School Journal, 105(3), 305–323. doi:10.1086/428746.
4.
BoardmanA. G.BuckleyP.VaughnS.RobertsG.ScornavaccoK.KlingnerJ. (2016a). The relationship between implementation of Collaborative Strategic Reading and student outcomes for adolescents with disabilities. Journal of Learning Disabilities, 49(6), 644–657. doi:10.1177/0022219416640784.
5.
BoardmanA. G.KlingnerJ. K.BuckleyP.AnnammaS.LasserC. J. (2015). Collaborative Strategic Reading in content classes: Results from year 1 of a randomized control trial. Reading and Writing: An Interdisciplinary Journal, 28(9), 1257–1283. doi:10.1007/s11145-015-9570-3.
6.
BoardmanA. G.VaughnS.BuckleyP.ReutebuchC.RobertsG.KlingnerJ. (2016b). Efficacy of Collaborative Strategic Reading with upper elementary school students: Results of a randomized control trial. Exceptional Children, 82(4), 409–427. doi:10.1177/0014402915625067.
7.
BollenK.LennoxR. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110, 305–314. doi:10.1037/0033-2909.110.2.305.
8.
BradshawC. P.MitchellM. M.LeafP. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavioral Interventions, 12, 161–179. doi:10.1177/1098300709334798.
9.
CappellaE.KimH.Y.NealJ. W.JacksonD. R. (2013). Classroom peer relationships and behavioral engagement in elementary school: The role of social network equity. American Journal of Community Psychology, 52, 367–379. doi:10.1007/s10464-013-9603-5.
10.
CarrollC.PattersonM.WoodS.BoothA.RiskJ.BalainS. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40. doi:10.1186/1748-5908-2–40
11.
CavanaghR.F.KoehlerM.J. (2013) A turn toward specifying validity criteria in the measurement of Technological Pedagogical Content Knowledge (TPACK). Journal of Research on Technology in Education, 46(2), 129–148. doi: 10.1080/15391523.2013.10782616.
12.
ChildsK. E.KincaidD.GeorgeH. P.GageN. A. (2016). The relationship between school-wide implementation of positive behavior intervention and supports and student discipline outcomes. Journal of Positive Behavior Interventions, 18(2), 89–99. doi:10.1177/1098300715590398.
13.
CohenR.KincaidD.ChildsK. E. (2007). Measuring School-wide Positive Behavior Support implementation: Development and validation of the Benchmarks of Quality. Journal of Positive Behavior Interventions, 9(4), 203–213. doi:10.1177/10983007070090040301.
14.
CordrayD. S.PionG. M. (2006). Treatment strength and integrity: Models and methods. In BootzinR. R.McKnightP. E. (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 103–124). Washington, DC: American Psychological Association.
15.
CrockerL. M.AlginaJ. (1986). Introduction to classical and modern test theory. New York, NY: Holt, Rinehart, and Winston.
16.
DaneA. V.SchneiderB. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control?Clinical Psychology Review, 18, 23–45. doi:10.1016/S0272-7358(97)00043-3.
17.
DevineS. L.RappJ. T.TestaJ. R.HenricksonM. L.SchnerchG. (2011). Detecting changes in simulated events using partial-interval recording and momentary time sampling III: Evaluating sensitivity as a function of session length. Behavioral Interventions, 26, 103–124. doi:10.1002/bin.328.
18.
DrakeR. E.ResnickS. G. (1998). Models of community care for severe mental illness: A review of research on case management. Schizophrenia Bulletin, 24, 37–43. doi:https://doi.org/10.1093/oxfordjournals.schbul.a033314.
19.
EdwardsJ. R.BagozziR. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5, 155–174. doi:10.1037/1082-989X.5.2.155.
20.
FishM. C.DaneE. (2000). The classroom systems observation scale: Development of an instrument to assess classrooms using a systems perspective. Learning Environments Research, 3, 67–92. doi:10.1023/A:1009979122896.
21.
FlavellJ. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34, 906–911. doi:10.1037/0003-066X.34.10.906.
22.
FlowerA.McKennaJ.MuethingC. S.BryantD. P.BryantB. R. (2014). Effects of the good behavior game on classwide off-task behavior in a high school basic algebra resource classroom. Behavior Modification, 38(1), 45–68. doi:10.1177/0145445513507574.
23.
FreemanJ.SimonsenB.McCoachD. B.SugaiG.LombardiA.HornerR. (2016). Relationship between School-Wide Positive Behavior Interventions and Supports and academic, attendance, and behavior outcomes in high schools. Journal of Positive Behavior Interventions, 18(1), 41–51. doi:10.1177/1098300715580992.
24.
GaretM. S.PorterA. C.DesimoneL.BirmanB. F.YoonK. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915–945. doi:10.3102/00028312038004915.
25.
GeorgeH. P.ChildsK. E. (2012). Evaluating implementation of schoolwide behavior support: Are we doing it well?Preventing School Failure, 56, 197–206. doi:10.1080/1045988X.2011.645909.
26.
GreenhalghT.RobertG.MacfarlaneF.BateP.KyriakidouO. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Quarterly, 82(4), 581–629. doi:10.1111/j.0887-378X.2004.00325.x.
27.
GutiérrezK. D.PenuelW. R. (2014). Relevance to practice as a criterion for rigor. Educational Researcher, 43, 19–23. doi:10.3102/0013189X13520289.
28.
GuoY.ConnorC. M.YangYRoehrigA. D.MorrisonF. J. (2012). The effects of teacher qualification, teacher self-efficacy, and classroom practices on fifth graders’ literacy outcomes, The Elementary School Journal, 113(1), 3–24. doi:10.1086/665816.
29.
HamreB. K.PiantaR. C.MashburnA. J.DownerJ. T. (2007). Building a science of classrooms: Application of the CLASS framework in over 4,000 US early childhood and elementary classrooms. New York, NY: Foundation for Child Development. Retrieved from http://fcd-us.org/sites/default/files/BuildingAScienceOfClassroomsPiantaHamre.pdf.
30.
HeckS.SteigelbauerS. M.HallG. E.LoucksS. F. (1981). Measuring innovation configurations: Procedures and applications. Austin, TX. Research and Development Center for Teacher Education, University of Texas.
31.
HillH. C.CharalambousC. Y.KraftM. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41, 56–64. doi:10.3102/0013189X12437203.
32.
HillH. C.GrossmanP. (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83(2), 371–384. doi:10.17763/haer.83.2.d11511403715u376.
33.
HornerR. H.ToddA. W.Lewis-PalmerT.IrvinL. K.SugaiG.BolandJ. B. (2004). The School-wide Evaluation Tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6(1), 3–12. doi:10.1177/10983007040060010201.
34.
HornerR. H.SugaiG.AndersonC. M. (2010). Examining the evidence base for school-wide positive behavior support. Focus on Exceptionality, 42(8), 1–14.
35.
HullemanC. S.CordrayD. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research and Educational Effectiveness, 2, 88–110. doi:10.1080/19345740802539325.
36.
JohnsonD. W.JohnsonR. T. (1999). Making cooperative learning work. Theory into Practice, 38(2), 67–73. doi:10.1080/00405849909543834.
37.
KaganS. (1986). Cooperative learning and sociocultural factors in schooling. In Bilingual Education Office, California State Department of Education (Ed.), Beyond language: Social and cultural factors in schooling language minority students. Los Angeles, CA: Evaluation, Dissemination and Assessment Center, California State University.
38.
KaneM. T. (2006). Validation. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 17–64). Santa Barbara, CA: Greenwood Publishing Group.
39.
KazdinA. E.KendallP. C. (1998). Current progress and future plans for developing effective treatments: Comment and perspectives. Journal of Clinical Child Psychology, 60, 733–747. doi:10.1207/s15374424jccp2702_8.
40.
KimA. H.VaughnS.KlingnerJ. K.WoodruffA. L.KleinC.KouzekananiK. (2006). Improving the reading comprehension of middle school students with disabilities through Computer-Assisted Collaborative Strategic Reading (CACSR). Remedial and Special Education, 27, 235–248. doi:10.1177/07419325060270040401.
41.
KlingnerJ. K.VaughnS.ArguellesM. E.HughesM. T.LeftwichS.A. (2004). Collaborative Strategic Reading: “Real world” lessons from classroom teachers. Remedial and Special Education, 25, 291–302. doi:10.1177/07419325040250050301.
42.
KlingnerJ. K.VaughnS.BoardmanA. G.SwansonE. (2012). Now we get it! Boosting comprehension with Collaborative Strategic Reading. San Francisco, CA: Jossey Bass.
43.
KlingnerJ. K.VaughnS.SchummJ. S. (1998). Collaborative Strategic Reading during social studies in heterogeneous fourth-grade classrooms. Elementary School Journal, 99, 3–22. doi:http://dx.doi.org/10.1086/461914.
44.
LawrenceJ. F.SnowC. (2010). Oral discourse and reading. In KamilM.PearsonP. D.MojeE. B.AfflerbachP. (Eds.), Handbook of reading research (Vol. 4). Mahwah, NJ: Erlbaum.
45.
LewisT. J.SugaiG. (1999). Effective behavior support: A systems approach to proactive schoolwide management. Focus on Exceptional Children, 31(6), 1.
McKennelA. C. (1974). Surveying attitude structures. Amsterdam, the Netherlands: Elsevier
48.
MessickS. (1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In WainerH.BraunH. (Eds.), Test validity (pp. 33–48). Hillsdale, NJ: Erlbaum.
49.
MowbrayC.HolterM. C.TeagueG. B.BybeeD. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340. doi:10.1177/109821400302400303.
50.
MunterC.WilhelmA. G.CobbP.CordrayD. S. (2014). Assessing fidelity of implementation of an unprescribed, diagnostic mathematics intervention, Journal of Research on Educational Effectiveness, 7(1), 83–113. doi:10.1080/19345747.2013.809177.
51.
NelsonM. C.CordrayD. S.HullemanC. S.DarrowC. L.SommerE. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. The Journal of Behavioral Health Services & Research, 39(4), 374–396. doi:10.1007/s11414-012-9295-x.
O’DonnellC. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relation to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84. doi:10.3102/0034654307313793.
54.
PalincsarA. S.BrownA. L. (1984). The reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 1, 117–175. doi:10.1207/s1532690xci0102_1.
55.
PiantaR. C.HamreB. K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38, 109–119. doi:10.3102/0013189X09332374.
RaudenbushS. W.BrykT.CongdonR. (2010). HLM 7 hierarchical linear and nonlinear modeling. [Computer software]. Skokie, IL: Scientific Software International.
58.
RaudenbushS. W.BrykT (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
59.
RogersE. (2003). Diffusion of innovations. New York, NY: Free Press.
60.
RosenbaumP. R. (1995). Observational Studies. New York, NY: Springer-Verlag.
61.
SawadaD.PiburnM. D.JudsonE.TurleyJ.FalconerK.BenfordR.BloomI. (2002). Measuring reform practices in science and mathematics classrooms: The Reformed Teaching Observation Protocol. Measuring Reform Practices, 10(6), 245–253. doi:10.1111/j.1949-8594.2002.tb17883.x.
62.
ScammaccaN.RobertsG.VaughnS.EdmondsM.WexlerJ.ReutebuchC. K.TorgesenJ. K. (2007). Reading interventions for adolescent struggling readers: A meta-analysis with implications for practice. Portsmouth, NH: RMC Research Corporation, Center on Instruction. Retrieved from http://files.eric.ed.gov/fulltext/ED521837.pdf.
63.
SchechterS.BlairJ.HeyJ. V. (1996). Conducting cognitive interviews to test self-administered and telephone surveys: Which methods should we use? In 1996 Proceedings of the Section on Survey Research Methods (pp. 10–17). Alexandria, VA: American Statistical Association.
64.
SchoenwaldS. K.GarlandA. F.ChapmanJ. E.FrazierS. L.SheidowA. J.Southam-GerowM. (2011). Toward the effect and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1): 32–43. doi:10.1007/s10488-010-0321-0.
65.
ShulmanL. (1990). Foreword. In M.Ben-Peretz, The teacher–curriculum encounter: Freeing teachers from the tyranny of texts. Albany, NY: State University of New York Press.
66.
TigerJ. H.MillerS. J.MeversJ. L.MintzJ.ScheithauerM. C.AlvarezJ. (2013). On the representativeness of behavior samples in classrooms. Journal of Applied Behavior Analysis, 46, 424–435. doi:10.1002/jaba.39
67.
VaughnS.KlingnerJ.SwansonE. A.BoardmanA. G.RobertsG.MohammedS.Stillman-SpisakS. J. (2011). Efficacy of collaborative strategic reading with middle school students. American Educational Research Journal, 48, 938–964. doi:10.3102/0002831211410305.
68.
VaughnS.RobertsG.KlingnerJ.SwansonE.BoardmanA.Stillman-SpisakS.J.MohammedS.LerouxA. (2013). Collaborative strategic reading: Findings from experienced implementers. Journal of Research on Educational Effectiveness, 6(2), 137–163. doi:10.1080/19345747.2012.741661.
69.
WanzekJ.VaughnS.KentS.SwansonE. A.RobertsGHaynesM.. . . SolisM. (2014). The effects of team-based learning on social studies knowledge acquisition in high school. Journal of Research on Educational Effectiveness, 7, 183–204. doi:doi:10.1080/19345747.2013.836765.
70.
WilsonM. (2004). Constructing measures: An item response modeling approach. New York, NY: Routledge.
