Young children with and at risk for emotional/behavioral disorders (EBD) present challenges for early childhood teachers. Evidence-based programs designed to address these young children’s behavior problems exist, but there are a number of barriers to implementing these programs in early childhood settings. Advancing the science of treatment integrity measurement can assist researchers and consumers interested in implementing evidence-based programs in early childhood classrooms. To provide guidance for researchers interested in assessing the integrity of implementation efforts, we describe a conceptual model of implementation of evidence-based programs designed to prevent EBD when applied in early childhood settings. Next, we describe steps that can be used to develop treatment integrity measures. Last, we discuss factors to consider when developing treatment integrity measures with specific emphasis on psychometrically strong measures that have maximum utility for implementation research in early childhood classrooms.
AaronsG. A.HurlburtM.HorwitzS. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4-23. doi:10.1007/s10488-010-0327-7
2.
BakerC. N.KupersmidtJ. B.Voegler-LeeM. E.ArnoldD.WilloughbyM. T. (2010). Predicting teacher participation in a classroom-based, integrated preventive intervention for preschoolers. Early Childhood Research Quarterly, 25, 270-283. doi:10.1016/j.ecresq.2009.09.005
3.
BarberJ. P.SharplessB. A.KlostermannS.McCarthyK. (2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology: Science and Practice, 38, 493-500. doi:10.1037/0735-7028.38.5.493
4.
BarnettW. S.JungK.YaroszD. J.ThomasJ.HornbeckA.StechukR.BurnsS. (2008). Educational effects of the Tools of the Mind curriculum: A randomized trial. Early Childhood Research Quarterly, 23, 233-313. doi:10.1016/j.ecresq.2008.03.001
5.
BeckerK. D.DomitrovichC. E. (2011). The conceptualization, integration, and support of evidence-based interventions in the schools. School Psychology Review, 40, 582-589.
6.
BerlinL. J.Brooks-GunnJ.McCartonC.McCormickM. C. (1998). The effectiveness of early intervention: Examining risk factors and pathways to enhance development. Preventive Medicine, 27, 238-245.
7.
BeutlerL. E.BakerM. (1998). The movement toward empirical validation: At what level should we analyze, and who are the consumers? In DobsonK. S.CraigK. D. (Eds.), Empirically supported therapies (pp. 43-65). Thousand Oaks, CA: SAGE.
8.
BiermanK. L.NixR. L.GreenbergM. T.BlairC.DomitrovichC. E. (2008). Executive functions and school readiness intervention: Impact, moderation, and mediation in the Head Start REDI program. Development and Psychopathology, 20, 821-843. doi:10.1017/S0954579408000394
9.
BurchinalM.HowesC.KontosS. (2002). Structural predictors of child care quality in child care homes. Early Childhood Research Quarterly, 17, 87-105. doi:http://dx.doi.org/10.1016/S0885-2006(02)00132-1
10.
CarrollK. M.NichC.SiftyR. L.NuroK. F.FrankforterT. L.BallS. A.RounsavilleB. J. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research. Drug and Alcohol Dependence, 57, 225-238.
11.
CarrollK. M.NuroK. F. (2002). One size cannot fit all: A stage model for psychotherapy manual development. Clinical Psychology: Science and Practice, 9, 396-406.
12.
ChorpitaB. F.DaleidenE. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77, 566-579. doi:10.1037/a0014565
13.
ChuB. C.ChoudhuryM. S.ShorttA. L.PincusD. B.CreedT. A.KendallP. C. (2004). Alliance, technology, and outcome in the treatment of anxious youth. Cognitive and Behavioral Practice, 11, 44-55. doi:http://dx.doi.org/10.1016/S1077-7229(04)80006-3
14.
ChuB. C.KendallP. C. (2004). Positive association of child involvement and treatment outcome within a manual-based cognitive-behavioral treatment for children with anxiety. Journal of Consulting and Clinical Psychology, 72, 821-829.
15.
ConroyM. A.SutherlandK. S.VoA. K.CarrS. E.OgstonP. (2013). Early childhood teachers’ use of effective instructional practices and the collateral effects on young children’s behavior. Journal of Positive Behavior Interventions. Advance online publication. doi:10.1177/1098300713478666
16.
DaneA. V.SchneiderB. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control?Clinical Psychology Review, 18, 23-45.
17.
DomitrovichC. E.CortesR. C.GreenbergM. T. (2007). Improving young children’s social and emotional competence: A randomized trial of the preschool “PATHS” curriculum. The Journal of Primary Prevention, 28, 67-91.
18.
DomitrovichC. E.GestS. D.JonesD.GillS.DeRousieR. M. S. (2010). Implementation quality: Lessons learned in the context of the Head Start REDI trial. Early Childhood Research Quarterly, 25, 284-298.
19.
DomitrovichC. E.MooreJ. E.GreenbergM. T. (2012). Maximizing the effectiveness of social-emotional interventions for young children through high-quality implementation of evidence-based interventions. In KellyB.PerkinsD. F. (Eds.), Handbook of implementation science for psychology in education (pp. 207-229). Cambridge, UK: Cambridge University Press.
20.
DonabedianA. (1988). The quality of care: How can it be assessed?Journal of the American Medical Association, 260, 1743-1748.
21.
DossB. D. (2004). Changing the way we study change in psychotherapy. Clinical Psychology: Science and Practice, 11, 368-386. doi:10.1093/clipsy.bph094
22.
DriscollK. C.PiantaR. C. (2010). Banking time in Head Start: Early efficacy of an intervention designed to promote supportive teacher-child relationships. Early Education and Development, 21, 38-64. doi:10.1080/10409280802657449
23.
DurlakJ. A. (2010). The importance of doing well in whatever you do: A commentary on the special section, “Implementation research in early childhood education.”Early Childhood Research Quarterly, 25, 348-357. doi:10.1016/j.ecresq.2010.03.003
24.
EcclesM. P.MittmanB. S. (2006). Welcome to implementation science. Implementation Science, 1, Article 1.
25.
EmbryD. D.BiglanA. (2008). Evidence-based kernels: Fundamental units of behavioral influence. Clinical Child and Family Psychology Review, 11, 75-113. doi:10.1007/s10567-008-0036-x
26.
FixsenD. L.NaoomS. F.BlaséK. A.FriedmanR. M.WallaceF. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).
27.
GarlandA. F.BickmanL.ChorpitaB. F. (2010). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health, 37, 15-26. doi:10.1007/s10488-010-0279-y
28.
GoldfriedM. R.PadawerW. (1982). Current status and future directions in psychotherapy. In GoldfriedM. R. (Ed.), Converging themes in psychotherapy: Trends in psychodynamic, humanistic, and behavioral practice (pp. 3-49). New York, NY: Springer.
29.
HamreB. K.JusticeL.PiantaR. C.KildayC.SweenyB.DownerJ.LeachA. (2010). Implementation fidelity of the My Teaching Partner literacy and language activities: Associations with preschoolers’ language and literacy growth. Early Childhood Research Quarterly, 25, 329-347. doi:10.1016/j.ecresq.2009.07.002
30.
HamreB. K.PiantaR. C.BurchinalM.FieldS.LoCasale-CrouchJ.DownerJ. T.Scott-LittleC. (2012). A course on effective teacher-child interactions: Effects on teacher beliefs, knowledge, and observed practice. American Educational Research Journal, 49, 88-123. doi:10.3102/0002831211434596
31.
HanS. S.CatronT.WeissB.MarcielK. K. (2005). A teacher-consultation approach to social skills training for pre-kindergarten children: Treatment model and short-term outcome effects. Journal of Abnormal Child Psychology, 33, 681-693. doi:10.1007/s10802-005-7647-1
32.
HanS. S.WeissB. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665-679. doi:10.1007/s10802-005-7646-2
33.
HarnB.ParisiD.StoolmillerM. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools?Exceptional Children, 79, 181-193.
34.
HillC. E.O’GradyK. E.ElkinI. (1992). Applying the Collaborative Study Psychotherapy Rating Scale to rate therapist adherence in cognitive-behavior therapy, interpersonal therapy, and clinical management. Journal of Consulting and Clinical Psychology, 60, 73-79.
35.
HogueA.DauberS.ChinchillaP.FriedA.HendersonC.InclanJ.LiddleH. A. (2008). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35, 137-147.
36.
HogueA.DauberS.HendersonC. E. (2013). Therapist self-report of evidence-based practices in usual care for adolescent problems: Factor and construct validity. Administration and Policy in Mental Health and Mental Health Services Research. Advance online publication. doi:10.1007/s10488-012-0442-8
37.
HogueA.HendersonC. E.DauberS.BarajasP. C.FriedA.LiddleH. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544-555. doi:10.1037/0022-006X.76.4.544
38.
HogueA.LiddleH. A.RoweC. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy: Theory, Research, Practice, Training, 33, 332-345.
JoyceB.ShowersB. (2002). Student achievement through staff development (3rd ed.). Alexandria, VA: Association for Supervision and Curriculum Development.
41.
KarverM. S.HandelsmanJ. B.FieldsS.BickmanL. (2006). Meta-analysis of therapeutic relationship variables in youth and family therapy: The evidence for different relationship variables in the child and adolescent treatment outcome literature. Clinical Psychology Review, 26, 50-65.
42.
KnoxL. M.AspyC. B. (2011). Quality improvement as a tool for translating evidence based interventions into practice: What the youth violence prevention community can learn from healthcare. American Journal of Community Psychology, 48, 56-64. doi:10.1007/s10464-010-9406-x
43.
KratochwillT. R.HoagwoodK. E.KazakA. E.WeiszJ. R.HoodK.VargasL. A.BanezG. A. (2012). Practice-based evidence for children and adolescents: Advancing the research agenda in schools. School Psychology Review, 41, 215-235.
44.
McLeodB. D. (2011). The relation of the alliance with outcomes in youth psychotherapy: A meta-analysis. Clinical Psychology Review, 31, 603-616. doi:10.1016/j.cpr.2011.02.003
45.
McLeodB. D.IslamN. Y.WheatE. (2013). Designing, conducting, and evaluating therapy process research. In ComerJ.KendallP. (Eds.), The Oxford handbook of research strategies for clinical psychology (pp. 142-164). New York, NY: Oxford University Press.
46.
McLeodB. D.Southam-GerowM. A.TullyC. B.RodriguezA.SmithM. M. (2013). Making a case for treatment integrity as a psychological treatment quality indicator. Clinical Psychology: Science and Practice, 20, 14-32. doi:10.1111/cpsp.12020
47.
McLeodB. D.Southam-GerowM. A.WeiszJ. R. (2009). Conceptual and methodological issues in treatment integrity measurement. School Psychology Review, 38, 541-546.
48.
McLeodB. D.WeiszJ. R. (2010). The Therapy Process Observational Coding System for Child Psychotherapy Strategies Scale. Journal of Clinical Child and Adolescent Psychology, 39, 436-443. doi:10.1080/15374411003691750
49.
MendelP.MeredithL. S.SchoenbaumM.SherbourneC. D.WellsK. B. (2008). Interventions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research. Administration and Policy in Mental Health and Mental Health Services Research, 35, 21-37. doi:10.1007/s10488-007-0144-9
50.
NelsonJ. R.StageS.Duppong-HurleyD.SynhorstL.EpsteinM. H. (2007). Risk factors predictive of the problem behavior of children at risk for emotional and behavioral disorders. Exceptional Children, 73, 367-379.
51.
OdomS. L.FlemingK.DiamondK.LieberJ.HansonM.ButeraG.MarquisJ. (2010). Examining different forms of implementation and in early childhood curriculum research. Early Child Research Quarterly, 25, 314-328.
52.
PerepletchikovaF.KazdinA. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology Science and Practice, 12, 365-383. doi:10.1093/clipsy/bpi045
53.
PiantaR. C.Rimm-KaufmanS. (2006). The social ecology of the transition to school: Classrooms, families, and children. In McCartneyK.PhillipsD. (Eds.), Blackwell handbooks of developmental psychology (pp. 490-507). Malden, MA: Blackwell Publishing.
54.
PiantaR. C.StuhlmanM. W. (2004). Conceptualizing risk in relational terms: Associations among the quality of child-adult relationships prior to school entry and children’s developmental outcomes in first grade. Educational and Child Psychology, 21, 32-45.
55.
QiC. H.KaiserA. P. (2003). Behavior problems of preschool children from low-income families: Review of the literature. Topics in Early Childhood Special Education, 23, 188-216.
56.
QuesenberryA.HemmeterM. L.OstroskyM. (2011). Addressing challenging behavior in Head Start: A closer look at program policies and procedures. Topics in Early Childhood Special Education, 30, 209-220. doi:10.1177/0271121410371985
57.
RaverC. C.JonesS. M.Li-GriningC.ZhaiF.MetzgerM. W.SolomonB. (2009). Targeting children’s behavior problems in preschool classrooms: A cluster-randomized controlled trial. Journal of Consulting and Clinical Psychology, 77, 302-316. doi:10.1037/a0015302
58.
SanettiL. M.DiGennaro ReedF. D. (2012). Barriers to implementing treatment integrity procedures in school psychology research: Survey of treatment outcome researchers. Assessment for Effective Intervention, 38, 1-8. doi:10.1177/1534508411432466
59.
SanettiL. M.DobeyL. M.GritterK. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999 to 2009. Journal of Positive Behavior Interventions, 14, 29-46. doi:10.1177/1098300711405853
60.
SanettiL. M.GritterK. L.DobeyL. M. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review, 40, 72-84.
61.
SanettiL. M.KratochwillT. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445-459.
62.
SchoenwaldS. K.GarlandA. F.ChapmanJ. E.FrazierS. L.SheidowA. J.Southam-GerowM. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32-43. doi:10.1007/s10488-010-0321-0
63.
SchoenwaldS. K.HoagwoodK. (2001). Effectiveness, transportability, and dissemination of interventions: What matters when?Psychiatric Services, 52, 1190-1197.
64.
Southam-GerowM. A.McLeodB. D. (2013). Advances in applying treatment integrity research for dissemination and implementation science: Introduction to special issue. Clinical Psychology: Science and Practice, 20, 1-13. doi:10.1111/cpsp.12019
65.
Southam-GerowM. A.WeiszJ. R.ChuB. C.McLeodB. D.GordisE. B.Connor-SmithJ. K. (2010). Does cognitive behavioral therapy for youth anxiety outperform usual care in community clinics? An initial effectiveness test. Journal of the American Academy of Child and Adolescent Psychiatry, 49, 1043-1052. doi:10.1016/j.jaac.2010.06.009
66.
SutherlandK. S.Lewis-PalmerT.StichterJ.MorganP. (2008). Examining the influence of teacher behavior and classroom context on the behavioral and academic outcomes for students with emotional or behavioral disorders. Journal of Special Education, 41, 223-233. doi:10.1177/0022466907310372
67.
SutherlandK. S.McLeodB. D.ConroyM. A.AbramsL.SmithM. M. (2013). Preliminary psychometric properties of the BEST in CLASS Adherence and Competence Scale. Journal of Emotional and Behavioral Disorders. Advance online publication. doi:10.1177/1063426613493669
68.
SwansonE.WanzekJ.HaringC.CiulloS.McCulleyL. (2011). Intervention fidelity in special and general education research journals. The Journal of Special Education, 45, 1-11. doi:10.1177/0022466911419516
69.
VoA. K.SutherlandK. S.ConroyM. A. (2012). Best in class: A classroom-based model for ameliorating problem behavior in early childhood settings. Psychology in the Schools, 49, 402-415. doi:10.1002/pits.21609
70.
WaltzJ.AddisM. E.KoernerK.JacobsonN. E. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620-630.
71.
Webster-StrattonC.ReidM. J.HammondM. (2001). Preventing conduct problems, promoting social competence: A parent and teacher training partnership in Head Start. Journal of Clinical Child Psychology, 30, 283-302.
72.
Webster-StrattonC.ReidM. J.HammondM. (2004). Treating children with early-conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of Clinical Child and Adolescent Psychology, 33, 105-124. doi:10.1207/S15374424JCCP3301_11
73.
Webster-StrattonC.ReidM. J.StoolmillerM. (2008). Preventing conduct problems and improving school readiness: Evaluation of the Incredible Years Teacher and Child Training Programs in high-risk schools. Journal of Child Psychology and Psychiatry, 49, 471-488. doi:10.1111/j.1469-7610.2007.01861.x
74.
WeilandC.UlvestadK.SachsJ.YoshikawaH. (2013). Associations between classroom quality and children’s vocabulary and executive function skills in an urban public prekindergarten program. Early Childhood Research Quarterly, 28, 199-209. doi:http://dx.doi.org.proxy.library.vcu.edu/10.1016/j.ecresq.2012.12.002
75.
WoleryM. (2011). Intervention research: The importance of fidelity measurement. Topics in Early Childhood Education, 31, 155-157. doi:10.1177/0271121411408621
76.
YoderP.SymonsF. (2010). Observational measurement of behavior. New York, NY: Springer.
