The measurement of treatment integrity is critical to evaluate the efficacy and effectiveness of evidence-based programs (EBPs) designed to improve the developmental outcomes of young children at risk of emotional/behavioral disorders. Unfortunately, the science of treatment integrity measurement lags behind the development and evaluation of EBP for young, high-risk children. This article describes the development and preliminary psychometric properties of the BEST in CLASS Adherence and Competence Scale (BiCACS), designed to measure the adherence and competence of delivery of the BEST in CLASS prevention program. Independent observers coded videotaped (n = 116) and live (n = 289) observations of teachers delivering the BEST in CLASS program. The BiCACS showed good interrater reliability and analyses provided some support for the validity of the measure. Implications for future research and integrity measurement work are discussed.
BarberJ. P.MercerD.KrakauerI.CalvoN. (1996). Development of adherence/competence rating scale for individual drug counseling. Drug and Alcohol Dependence, 43, 125–132. doi:10.1016/S0376-8716(96)01305-1
2.
BarberJ. P.SharplessB. A.KlostermannS.McCarthyK. S. (2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology: Research and Practice, 38, 493–500. doi:10.1037/0735-7028.38.5.493
3.
BiermanK. L.DomitrovichC. E.NixR. L.GestS. G.WelshJ. A.GreenbergM. T.GillS. (2008). Promoting academic and social-emotional school readiness: The head start REDI program. Child Development, 79, 1802–1817. doi:0009-3920/2008/7906-0019
4.
BirchS. H.LaddG. W. (1997). The teacher–child relationship and children’s early school adjustment. Journal of School Psychology, 35, 61–79. doi:10.1016/S0022-4405(96)00029-5
5.
BirchS. H.LaddG. W. (1998). Children’s interpersonal behaviors and the teacher–child relationship. Developmental Psychology, 34, 934–946. doi:10.1037/0012-1649.34.5.934
6.
CarrollK. M.NichC.SiftyR. L.NuroK. F.FrankfurterT. L.BallS. A.RounsavilleB. J. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research. Drug and Alcohol Dependence, 57, 225–238. doi:10.1016/S0376-8716(99)00049-6
7.
CarrollK. M.NuroK. F. (2002). One size cannot fit all: A stage model for psychotherapy manual development. Clinical Psychology: Science and Practice, 9, 396–406. doi:10.1093/clipsy/9.4.396
8.
CateD.DiefendorfM.McCulloughK.PetersM. L.WhaleyK. (Eds.). (2010). Quality indicators of inclusive early childhood programs/practices: A compilation of selected resources. Chapel Hill: The University of North Carolina, FPG Child Development Institute, National Early Childhood Technical Assistance Center.
9.
CicchettiD. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. doi:10.1037/1040-3590.6.4.284.
10.
ConroyM. A.SutherlandK. S.VoA. K.CarrS. E.OgstonP. (2013). Early childhood teachers’ use of effective instructional practices and the collateral effects on young children’s behavior. Journal of Positive Behavior Interventions. Advance online publication. doi:10.1177/1098300713478666
11.
DomitrovichC. E.CortesR. C.GreenbergM. T. (2007). Improving young children’s social and emotional competence: A randomized trial of the preschool “PATHS” curriculum. The Journal of Primary Prevention, 28, 67–91. doi:10.1007/s10935-007-0081-0
12.
DomitrovichC. E.MooreJ. E.GreenbergM. T. (2012). Maximizing the effectiveness of social-emotional interventions for young children through high-quality implementation of evidence-based interventions. In KellyB.PerkinsD. F. (Eds.), Handbook of implementation science for psychology in education (pp. 207–229). Cambridge, UK: Cambridge University Press.
13.
DriscollK. C.PiantaR. C. (2010). Banking time in head start: Early efficacy of an intervention designed to promote supportive teacher-child relationships. Early Education and Development, 21, 38–64. doi:10.1080/10409280802657449
14.
DurlakJ. A. (2010). The importance of doing well in whatever you do: A commentary on the special section, “Implementation research in early childhood education.”Early Childhood Research Quarterly, 25, 348–357. doi:10.1016/j.ecresq.2010.03.003
15.
Hagermoser SanettiL. M.DobeyL. M.GritterK. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999 to 2009. Journal of Positive Behavior Interventions, 14, 29–46. doi:10.1177/1098300711405853
16.
Hagermoser SanettiL. M.GritterK. L.DobeyL. M. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review, 40, 72–84.
17.
Hagermoser SanettiL. M.KratochwillT. R. (2009). Treatment integrity assessment in the schools: An evaluation of the treatment integrity planning protocol. School Psychology Quarterly, 24, 24–35. doi:10.1037/a0015431
18.
HamreB. K.PiantaR. C. (2001). Early teacher–child relationship and the trajectory of children’s school outcome through eighth grade. Child Development, 72, 625–638. doi:10.1111/1467-8624.00301
19.
HarnB.ParisiD.StoolmillerM. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools?Exceptional Children, 79, 181–193.
20.
HemmeterM. L.CorsoR.CheathamG. (2006, February). Issues in addressing challenging behaviors in young children: A national survey of early childhood educators. Paper presented at the Conference on Research Innovations in Early Intervention, San Diego, CA.
21.
HogueA.HendersonC. E.DauberS.BarajasP. C.FriedA.LiddleH. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555. doi:10.1037/0022-006X.60.1.73
22.
HogueA.LiddleH. A.RoweC. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy, 33, 332–345. doi:10.1037/0033-3204.33.2.332
23.
HowesC.HamiltonC. E. (1992). Children’s relationships with caregivers. Child Development, 63, 859–866. doi:10.2307/1131238
24.
HowesC.RitchieS. (1999). Attachment organizations in children with difficult life circumstances. Development and Psychopathology, 11, 251–268. doi:10.1017/S0954579499002047
25.
KennedyC. H. (2005). Single case designs for educational research. Boston, MA: Allyn & Bacon.
26.
LambertM.HillC. (1994). Assessing psychotherapy outcomes and processes. In BerginA.GarfieldS. (Eds.), Handbook of psychotherapy and behavior change (4th ed., pp. 72–113). New York, NY: John Wiley & Sons.
27.
MargolinG.OliverP.GordisE.O’HearnH.MedinaA.GhoshC.MorlandL. (1998). The nuts and bolts of behavioral observation of marital and family interaction. Clinical Child & Family Psychology Review, 1, 195–213. doi:10.1023/A:1022608117322
28.
McLeodB. D.IslamN. Y.WheatE. (2013). Designing, conducting, and evaluating therapy process research. In ComerJ.KendallP. (Eds.), The Oxford handbook of research strategies for clinical psychology (pp. 142–164). New York, NY: Oxford University Press.
29.
McLeodB. D.Southam-GerowM. A.BairC. E.RodriguezA.SmithM. M. (2013). Making a case for treatment integrity as a psychological treatment quality indicator. Clinical Psychology: Science and Practice, 20, 14–32. doi:10-1111/cpsp.12020
30.
McLeodB. D.WeiszJ. R. (2010). The therapy process observational coding system for child psychotherapy strategies scale. Journal of Clinical Child & Adolescent Psychology, 39, 436–443. doi:10.1080/15374411003691750
PiantaR. C.HamreB. (2001). Students, teachers, and relationship support (STARS). Lutz, FL: Psychological Assessment Resources.
33.
PiantaR. C.La ParoK. M.PayneC.CoxM. J.BradleyR. (2002). The relation of kindergarten classroom environment to teacher, family, and school characteristics and child outcomes. Elementary School Journal, 102, 225–238. doi:10.1086/499701
34.
QuesenberryA. C.HemmeterM. L.OstroskyM. M. (2011). Addressing challenging behaviors in head start: A closer look at program policies and procedures. Topics in Early Childhood Special Education, 30, 209–220. doi:10.1177/0271121410371985
35.
SchulteA. C.EastonJ. E.ParkerJ. (2009). Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38, 460–475.
36.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. doi:10.1037/0033-2909.86.2.420
37.
Southam-GerowM. A.McLeodB. D. (2013). Advances in applying treatment integrity research for dissemination and implementation science: Introduction to special issue. Clinical Psychology: Science and Practice, 20, 1–13. doi:10-1111/cpsp.12019
38.
StartupM.ShapiroD. A. (1993). Therapist treatment fidelity in prescriptive vs. exploratory psychotherapy. British Journal of Clinical Psychology, 32, 443–456.
39.
SutherlandK. S.ConroyM.AbramsL.VoA. (2010). Improving interactions between teachers and young children with problem behavior: A strengths-based approach. Exceptionality, 18, 70–81. doi:10.1080/09362831003673101
40.
SutherlandK. S.McLeodB. D. (2010). Treatment adherence and competence measure for classroom based interventions. Richmond: Virginia Commonwealth University.
41.
ThompsonT.FelceD.SymonsF. J (Eds.). (2000). Behavioral observation: Technology and applications in developmental disabilities. Baltimore, MD: Brookes Publishing.
42.
VoA. K.SutherlandK. S.ConroyM. A. (2012). Best in class: A classroom-based model for ameliorating problem behavior in early childhood settings. Psychology in the Schools, 49(5), 402–415.doi:10.1002/pits.21609
43.
WalkerH.SeversonH.FeilE. (1995). Early screening project: A proven child find process, examiner’s manual. Longmont, CO: Sopris West Publishing.
44.
WaltzJ.AddisM. E.KoernerK.JacobsonN. E. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620–630. doi:10.1037/0022-006X.61.4.620
45.
WebbC. A.DeRubeisR. J.BarberJ. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211. doi:10.1037/a0018912
46.
Webster-StrattonC.ReidJ. M.HammondM. (2004). Treating children with early-onset conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of Clinical Child & Adolescent Psychology, 33, 105–124. doi:10.1207/S15374424JCCP3301_11
47.
WoleryM. (2011). Intervention research: The importance of fidelity measurement. Topics in Early Childhood Special Education, 31, 155–157. doi:10.1177/0271121411408621
