The relation between fidelity of implementation and student outcomes in a computer-based middle school mathematics curriculum was measured empirically. Participants included 485 students and 23 teachers from 11 public middle schools across seven states. Implementation fidelity was defined using two constructs: fidelity to structure and fidelity to process. Because of the nested nature of the data, we used a two-level hierarchical linear model for analysis. Four variables, all categorized as fidelity to structure variables, proved significant—total time in intervention (p <.001), concentration of time in intervention (p = .03), direct observation of intervention fidelity (p = .04), and pretest score (p <.001). Fidelity to process was found to be nonsignificant. The importance of measuring the relation between implementation fidelity and student outcomes is discussed as well as implications for researchers and teachers.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2.
BakerE. (2001). Technology: How do we know it works? In HeinekeW.BlasiL. (Eds.), Methods of evaluating educational technology (pp. 77-84). Greenwich, CT: Information Age.
3.
BeckerH. J. (1994). How exemplary computer-using teachers differ from other teachers: Implications for realizing the potential of computers in schools. Journal of Research on Computing in Education, 26, 291–321.
4.
CampbellD. T.StanleyJ. C. (1963). Experimental and quasi-experimental designs for research. Dallas, TX: Houghton Mifflin.
5.
CavanaughC. L.KimA.WanzekJ.VaughnS. (2004). Kindergarten reading interventions for at-risk students: Twenty years of research. Learning Disabilities: A Contemporary Journal, 2, 9–21.
6.
ChristT. J. (2007). Experimental control and threats to internal validity of concurrent and nonconcurrent multiple baseline designs. Psychology in the Schools, 44, 451–459.
7.
CohenP. A.KulikJ. A.KulikC. C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19, 237–248.
8.
CookS. B.ScruggsT. E.MastropieriM. A.CastoG. C. (1985). Handicapped students as tutors. Journal of Special Education, 19, 483–492.
9.
CrawfordL. (2008, March). Effects of an online supplementary mathematics curriculum for English language learners. Paper presented at the annual meeting of the American Educational Research Association, New York, NY.
10.
Digital Directions International. (n.d.). HELP Math: Summary of research findings (Technical report). Carbondale, CO: Author. Retrieved from http://www.helpprogram.net.
11.
ElbaumB.VaughnS. (2001). School-based interventions to enhance the self-concept of student with learning disabilities: A meta-analysis. Elementary School Journal, 101, 303–329.
12.
FletcherJ. M.VaughnS. (2009). Response to intervention: Preventing and remediating academic difficulties. Child Development Perspectives, 3, 30–37.
13.
GallM. D.GallJ. P.BorgW. R. (2007). Educational research: An introduction (8th ed.). Boston, MA: Allyn & Bacon.
14.
GerstenR.BakerS.LloydJ. W. (2000). Designing high-quality research in special education: Group experimental design. Journal of Special Education, 34, 2–18.
15.
GerstenR.EdyburnD. (2007). Defining quality indicators for group design in special education technology research. Journal of Special Education Technology, 22, 3–18.
16.
GerstenR.FuchsL. S.ComptonD.CoyneM.GreenwoodC.InnocentiM. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149–164.
17.
GreshamF. M. (1989). Assessment of treatment integrity in school consultation and prereferral intervention. School Psychology Review, 18, 37–50.
18.
GreshamF. M.GansleK.NoellG. H. (1993). Treatment integrity in applied behavior analysis with children. Journal of Applied Behavior Analysis, 26, 257–263.
19.
GreshamF. M.MacMillanD. L.Beebe-FrankenbergerM. E.BocianK. M. (2000). Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented?Learning Disabilities Research & Practice, 15, 198–205.
HofmannD. A.GavinM. B. (1998). Centering decisions in hierarchical linear models: Implications for research in organizations. Journal of Management, 24, 623–641.
22.
HohmannA. A.ShearM. K. (2002). Community-based intervention research: Coping with the “noise” of real life in study design. American Journal of Psychiatry, 159, 201–207.
23.
HoxJ. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum.
24.
HullemanC. S.CordrayD. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110.
25.
MadduxC. D.JohnsonL.HarlowS. (1993). The state of the art in computer education: Issues for discussion with teachers-in-training. Journal of Technology and Teacher Education, 1, 219–228.
26.
MeldeC.EsbensenF.TusinskiK. (2006). Addressing program fidelity using onsite observations and program provider descriptions of program delivery. Evaluation Review, 30, 714–740.
27.
MellardD. F.JohnsonE. (2008). RTI: A practitioner’s guide to implementing response to intervention. Thousand Oaks, CA: Corwin Press.
28.
MillsS.RaganT. (2000). A tool for analyzing implementation fidelity of an integrated learning system. Educational Technology Research and Development, 48, 21–41.
29.
MislevyR. J. (2007). Validity by design. Educational Researcher, 36, 463–469.
30.
MowbrayC. T.HolterM. C.TeagueG. B.BybeeD. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340.
31.
National Research Council of the National Academies: Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials. (2004). On evaluating curricular effectiveness: Judging the quality of K–12 mathematics evaluations. Washington, DC: Author.
32.
NoellG. H.GreshamF. M.GansleK. A. (2002). Does treatment integrity matter? A preliminary investigation of instructional implementation and mathematics performance. Journal of Behavioral Education, 11, 51–67.
33.
O’DonnellC. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relation to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84.
34.
Pearson Education. (2008–2009). SuccessMaker 2.0 math reference guide. Upper Saddle River, NJ: Author.
35.
PierangeloR.GiulianiG. (2008). Frequently asked questions about response to intervention. Thousand Oaks, CA: Corwin Press.
36.
PowerT. J.Blom-HoffmanJ.ClarkeA. T.Riley-TillmanT. C.KelleherC.ManzP. H. (2005). Reconceptualizing intervention integrity: A partnership-based framework for linking research with practice. Psychology in the Schools, 42, 495–507.
37.
RohrbeckC. A.Ginsburg-BlockM. D.FantuzzoJ. W.MillerT. R. (2003). Peer-assisted learning interventions with elementary school students: A meta-analytic review. Journal of Educational Psychology, 95, 240–257.
38.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth.
39.
SlavinR. E. (2008). Perspectives on evidence-based research in education: What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(5), 5–14.
40.
SongerN. B.GotwalsA. W. (2005, April). Fidelity of implementation in three sequential curricular units. In LynchS. (Chair) & O’DonnellC. L., “Fidelity of implementation” in implementation and scale-up research designs: Applications from four studies of innovative science curriculum materials and diverse populations. Symposium conducted at the annual meeting of the American Educational Research Association, Montreal, Canada.
41.
TuckerA. R.BlytheB. (2008). Attention to treatment fidelity in social work outcomes: A review of the literature from the 1990s. Social Work Research, 32, 185–190.
42.
WestonT. (2004). Formative evaluation for implementation: Evaluating educational technology applications and lessons. American Journal of Evaluation, 25, 51–64.
43.
YsseldykeJ.BoltD. M. (2007). Effect of technology-enhanced continuous progress monitoring on math achievement. School Psychology Review, 36, 453–467.
44.
YsseldykeJ.SpicuzzaR.KosciolekS.TeelucksinghE.BoysC.LemkuilA. (2003). Using curriculum-based instructional management system to enhance math achievement in urban schools. Journal of Education for Students Placed at Risk, 8, 247–265.
45.
ZvochK. (2009). Treatment fidelity in multisite evaluation. American Journal of Evaluation, 30, 44–61.
