Traditionally, treatment adherence has been measured in outcome studies by highly trained, independent raters who code audio- or video-taped treatment sessions to understand therapists’ adherence to preestablished treatment session components. Unfortunately, this method of assessing treatment adherence is time-, labor-, and cost-intensive, and does not translate well to community settings. This study compared expert-coded behavior observations, therapist and caregiver report of therapists’ adherence to nine teaching technique items assessed in treatment sessions using Alternatives for Families: A Cognitive Behavioral Therapy (AF-CBT) to determine whether other raters (outside of traditional expert coders) could effectively and accurately measure therapist adherence. A total of 533 therapy sessions were coded across experts (n = 2), therapists (n = 20), and caregivers (n = 42). Outcomes indicated strikingly different ratings across all reporters suggesting that therapist and caregiver reports may be supplement to, but not substitute for, observer ratings.
AaronsG. A.SommerfeldD. H.HechtD. B.SilovskyJ. F.ChaffinM. J. (2009). The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting & Clinical Psychology, 77, 270–280.
2.
Baker-EriczenM. J.HurlburtM. S.Brookman-FrazeeL.JenkinsM. M.HoughR. L. (2010). Comparing child, parent, and family characteristics in usual care and empirically supported treatment research samples for children with disruptive behavior disorders. Journal of Emotional and Behavioral Disorders, 18, 82–99.
3.
BaldwinS. A.ImelZ. E. (2013). Therapist effects: Findings and methods. In LambertM. J. (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (5th ed., pp. 258–297). Hoboken, NJ: Wiley.
4.
BaumelA. (2018). The treatment of childhood disruptive behavior disorders in ambulatory mental health centers. Journal of Child and Family Studies, 27, 1110–1114.
5.
BearmanS. K.WeiszJ. R.ChorpitaB. F.HoagwoodK.WardA.UguetoA. M.BernsteinA. (2013). More practice, less preach? The role of supervision processes and therapist characteristics in EBP implementation. Administration and Policy in Mental Health and Mental Health Services Research, 40, 518–529.
6.
BeidasR. S.KendallP. C. (2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice, 17, 1–30.
7.
Bennett-LevyJ.McManusF.WestlingB. E.FennellM. (2009). Acquiring and refining CBT skills and competencies: Which training methods are perceived to be most effective?Behavioural and Cognitive Psychotherapy, 37, 571–583.
8.
BrauhardtA.de ZwaanM.HerpertzS.ZipfelS.SvaldiJ.FriederichH.HilbertA. (2014). Therapist adherence in individual cognitive-behavioral therapy for binge-eating disorder: Assessment, course, and predictors. Behaviour Research and Therapy, 61, 5–60.
9.
CarlsonR. G.WeismandeMamaniA. G. (2010). Client characteristics and therapist competence and adherence to family therapy for schizophrenia. Revista Interamericana de Psicologia, 44, 342–351.
10.
CarrollK. M.NichC.RounsavilleB. J. (1998). Utility of therapist session checklists to monitor delivery of coping skills treatment for cocaine abusers. Psychotherapy Research, 8, 307–320.
11.
ChapmanJ. E.McCartM. R.LetourneauE. J.SheidowA. J. (2013). Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol. Journal of Consulting & Clinical Psychology, 81, 674–680.
12.
Child Sexual Abuse Task Force and Research & Practice Core, National Child Traumatic Stress Network. (2004). How to implement trauma-focused cognitive behavioral therapy. Durham, NC: National Center for Child Traumatic Stress.
13.
FosterE. M.KelschC. C.KamradtB.SosnaT.YangZ. (2001). Expenditures and sustainability in systems of care. Journal of Emotional and Behavioral Disorders, 9, 53–70.
14.
FriedbergR. D.McClureJ. M. (2002). Clinical practice of cognitive therapy with children and adolescents: The nuts and bolts. New York, NY: Guilford.
15.
GarbaczL. L.BrownD. M.SpeeG. A.PoloA. J.BuddK. S. (2014). Establishing treatment fidelity in evidence-based parent training programs for externalizing disorders in children and adolescents. Clinical Child and Family Psychology Review, 17, 230–247.
16.
GarlandA. F.BickmanL.ChorpitaB. F. (2010). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health, 37, 15–26.
17.
GarlandA. F.Brookman-FrazeeL.HurlburtM. S.AccursoE. C.ZoffnessR. J.Haine-SchlagelR.GangerW. (2010). Mental health care for children with disruptive behavior problems: A view inside therapists’ offices. Psychiatric Services, 61, 788–795.
18.
GlasgowR. E.VinsonC.ChambersD.KhouryM. J.KaplanR. M.HunterC. (2012). National Institutes of Health approaches to dissemination and implementation science: Current and future directions. American Journal of Public Health, 102, 1274–1281.
19.
GottfredsonD. C.CookT. D.GardnerF. E. M.Gorman-SmithD.HoweG. W.SandlerI. N.. . . ZafftK. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893–926.
20.
GuydishJ.CampbellB. K.ManuelJ. K.DelucchiK. L.LeT.PeavyK. M.McCartyD. (2014). Does treatment fidelity predict client outcomes in 12-Step Facilitation for stimulant abuse?Drug and Alcohol Dependence, 134, 330–336.
21.
HenggelerS. W.MeltonG. B.BrondinoM. J.SchererD. G.HanleyJ. H. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting & Clinical Psychology, 65, 821–833.
22.
HenggelerS. W.PickrelS. G.BrondinoM. J. (1999). Multisystemic treatment of substance-abusing and dependent delinquents: Outcomes, treatment fidelity, and transportability. Mental Health Services Research, 1, 171–184.
23.
HerschellA. D. (2010). Fidelity in the field: Developing infrastructure and fine-tuning measurement. Clinical Psychologist, 17, 253–257.
24.
HerschellA. D.KolkoD. J.BaumannB. L.BrownE. J. (2012). Application of alternatives for families: A cognitive-behavioral therapy to school settings. Journal of Applied School Psychology, 28, 270–293.
25.
HofmannS. G.AsnaaniA.VonkI. J. J.SawyerA. T.FangA. (2012). The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognitive Therapy & Research, 36, 427–440.
26.
HogueA.DauberS.HendersonC. E.LiddleH. A. (2014). Reliability of therapist self-report on treatment targets and focus in family-based intervention. Administration & Policy in Mental Health & Mental Health Services Research, 41, 697-705.
27.
HogueA.DauberS.LichvarE.BobekM.HendersonC. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration & Policy in Mental Health & Mental Health Services Research, 42, 229–243.
28.
HogueA.HendersonC. E.DauberS.BarajasP. C.FriedA.LiddleH. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555. doi:10.1037/0022-006X.76.4.544
29.
HurleyK. D.LambertM. C.GrossT. J.ThompsonR. W.FarmerE. M. Z. (2017). The role of therapeutic alliance and fidelity in predicting youth outcomes during therapeutic residential care. Journal of Emotional and Behavioral Disorders, 25, 37–45.
30.
ImelZ. E.BaldwinS. A.BaerJ. S.HartzlerB.DunnC.RosengrenD. B.AtkinsD. C. (2014). Evaluating therapist adherence in motivational interviewing by comparing performance with standardized and real patients. Journal of Consulting & Clinical Psychology, 82, 472–481.
31.
KaminskiJ. W.ValleL. A.FileneJ. H.BoyleC. L. (2008). A meta-analytic review of components associated with parent training program effectiveness. Journal of Abnormal Child Psychology, 36, 567–589.
32.
KolkoD. J. (1996). Individual cognitive-behavioral treatment and family therapy for physically abused children and their offending parents: A comparison of clinical outcomes. Child Maltreatment, 1, 322–342.
33.
KolkoD. J.BaumannB. L.HerschellA. D.
HartJ. A.HoldenE. A.WisniewskiS. R. (2012). Implementation of AF-CBT by community practitioners serving child welfare and mental health: A randomized trial. Child Maltreatment, 17(1), 32–46.
34.
KolkoD. J.IselinA. R.GullyK. J. (2011). Evaluation of the sustainability and clinical outcome of Alternatives for Families: A Cognitive-Behavioral Therapy (AF-CBT) in a child protection center. Child Abuse & Neglect, 35, 105–116.
35.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.
36.
LeichsenringF.SalzerS.HilsenrothM. J.LeibingE.LewekeF.RabungS. (2011). Treatment integrity: An unresolved issue in psychotherapy research. Current Psychiatry Reviews, 7, 313–321.
37.
LiberJ. M.McLoedB. D.Van WidenfeltB. M.GoedhartA. W.van der LeedenA. J. M.UtensE. M. W. J.TreffersP. D. A. (2010). Examining the relation between the therapeutic alliance, treatment adherence, and outcome of cognitive behavioral therapy for children with anxiety disorders. Behavior Therapy, 41, 172–186.
38.
MartinoS.BallS. A.NichC.FrankforterT. L.CarrollK. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19, 181–193.
39.
McLeodB. D.SmithM. M.Southam-GerowM. A.WeiszJ. R.KendallP. C. (2015). Measuring treatment differentiation for implementation research: The Therapy Process Observational Coding System for Child Psychotherapy Revised Strategies Scale. Psychological Assessment, 27, 314–325.
40.
McLeodB. D.Southam-GerowM. A.Jensen-DossA.HogueA.KendallP. C.WeiszJ. R. (2017). Benchmarking treatment adherence and therapist competence in individual cognitive-behavioral treatment for youth anxiety disorders. Journal of Clinical Child & Adolescent Psychology, 53, 1–13. doi:10.1080/15374416.2017.1381914
41.
McLeodB. D.WeiszJ. (2005). The Therapy Process Observational Coding System-Alliance Scale: Measure characteristics and prediction of outcome in usual clinical practice. Journal of Consulting & Clinical Psychology, 73, 323–333.
42.
McLeodB. D.WeiszJ. R. (2010). The Therapy Process Observational Coding System for Child Psychotherapy-Strategies Scale. Journal of Clinical Child & Adolescent Psychology, 39, 436–443.
43.
MorganN. R.DavisK. D.RichardsonC.PerkinsD. F. (2018). Common components analysis: An adapted approach for evaluating programs. Evaluation and Program Planning, 67, 1–9.
44.
National Institute of Health. (2011). Optimizing fidelity of empirically-supported behavioral treatments for mental disorders (R21/R33) (National Institutes of Health Funding Opportunities). Retrieved from https://grants.nih.gov/grants/guide/rfa-files/rfa-mh-12-050.html
O’DonohueW. T.FisherJ. E. (2009). General principles and empirically supported techniques of cognitive behavior therapy. Hoboken, NJ: John Wiley.
47.
PerepletchikovaF.HiltL. M.CherejiE.KazdinA. E. (2009). Barriers to implementing treatment integrity procedures: Survey of treatment outcome researchers. Journal of Consulting & Clinical Psychology, 77, 212–218.
48.
PerepletchikovaF.TreatT. A.KazdinA. E. (2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting & Clinical Psychology, 75, 829–841.
49.
PrinceM. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93, 223–231.
50.
ProctorE.SilmereH.RaghavanR.HovmandP.AaronsG.BungerA.. . . HensleyM. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration & Policy in Mental Health & Mental Health Services Research, 38, 65–76.
51.
RealK.PooleM. S. (2005). Innovation implementation: Conceptualization and measurement in organizational research. Research in Organizational Change and Development, 15, 63–134.
52.
Rotheram-BorusM. J.SwendemanD.FlanneryD.RiceE.AdamsonD. M.IngramB. (2009). Common factors in effective HIV prevention programs. AIDS Behavior, 13, 399–408.
53.
RyanS. R.CunninghamP. B.FosterS. L.BrennanP. A.BrockR. L.WhitmoreE. (2013). Predictors of therapist adherence and emotional bond in multisystemic therapy: Testing ethnicity as a moderator. Journal of Child & Family Studies, 22, 122–136.
54.
SchoenwaldS. K.ChapmanJ. E.SheidowA. J.CarterR. E. (2009). Long-term youth criminal outcomes in MST transport: The impact of therapist adherence and organizational climate and structure. Journal of Clinical Child & Adolescent Psychology, 38, 91–105.
55.
SchoenwaldS. K.GarlandA. F. (2013). A review of treatment adherence measurement methods. Psychological Assessment, 25, 146–156.
56.
SchoenwaldS. K.GarlandA. F.ChapmanJ. E.FrazierS. L.SheidowA. J.Southam GerowM. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43.
57.
SchoenwaldS. K.HenggelerS. W.BrondinoM. J.RowlandM. D. (2000). Multisystemic therapy: Monitoring treatment fidelity. Family Process, 39, 83–103.
58.
SeligmanM. E. P. (1995). The effectiveness of psychotherapy. American Psychologist, 50, 965–974.
59.
SextonT.TurnerC. W. (2010). The effectiveness of functional family therapy for youth with behavioral problems in a community practice setting. Journal of Family Psychology, 24, 339–348.
60.
ShafranR.ClarkD. M.FairburnC. G.ArntzA.BarlowD. H.EhlersA.. . . WilsonG. T. (2009). Mind the gap: Improving the dissemination of CBT. Behaviour Research and Therapy, 47, 902–909.
61.
Southam-GerowM. A.McLeodB. D.ArnoldC. A.RodriguezA.CoxJ. R.ReiseS. P.. . . KendallP. C. (2016). Initial development of a treatment adherence measure for cognitive-behavioral therapy for child anxiety. Psychological Assessment, 28, 70–80. doi:10.1037/pas0000141
62.
SutherlandK. S.McLeodB. D.ConroyM. A.AbramsL. M.SmithM. M. (2014). Preliminary psychometric properties of the BEST in CLASS adherence and competence scale. Journal of Emotional and Behavioral Disorders, 22, 249–259.
63.
TaylorW. D.Asgary-EdenV.LeeC. M.LaRocheK. J. (2015). Service providers’ adherence to an evidence-based parenting program: What are they missing and why?Journal of Child and Family Studies, 24, 50–56.
64.
TschuschkeV.CrameriA.KoehlerM.BerglarJ.MuthK.StaczanP.. . . Koemeda-LutzM. (2015). The role of therapists’ treatment adherence, professional experience, therapeutic alliance, and clients’ severity of psychological problems: Prediction of treatment outcome in eight different psychotherapy approaches. Preliminary results of a naturalistic study. Psychotherapy Research, 25, 420–434.
65.
WebbC. A.DeRubeisR. J.BarberJ. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting & Clinical Psychology, 78, 200–211.
66.
WebbC. A.DeRubeisR. J.DimidjianS.HollonS. D.AmsterdamJ. D.SheltonR. C. (2012). Predictors of patient cognitive therapy skills and symptom change in two randomized clinical trials: The role of therapist adherence and the therapeutic alliance. Journal of Consulting and Clinical Psychology, 80, 373–381.
67.
WeckF.HillingC.Schermelleh-EngelK.RudariV.StangierU. (2011). Reliability of adherence and competence assessment in cognitive behavioral therapy: Influence of clinical experience. The Journal of Nervous & Mental Disease, 199, 276–279.
68.
WeersingR. V.WeiszJ. R.DonenbergG. R. (2002). Development of the Therapy Procedures Checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology, 31, 168–180.
69.
WeiszJ. R.KuppensS.EckshtainD.UguetoA. M.HawleyK. M.Jensen-DossA. (2013). Performance of evidence-based youth psychotherapies compared with usual clinical care: A multilevel meta-analysis. Journal of the American Medical Association Psychiatry, 70, 750–761.
