Provider adherence to system of care principles in service interactions with families is an important indicator of behavioral health service quality for youth; however, valid and pragmatic measures suitable for monitoring this quality indicator at population scale have not been developed. This article reports on two studies that developed and evaluated such a measure. In Study 1, an iterative, family-partnered process resulted in generation of 18 items that demonstrated unidimensionality and strong reliability among caregivers of youth participating in behavioral health services (N = 141). In Study 2, data from a second, statewide, stratified random sample of caregivers (N = 351) confirmed the items’ unidimensionality, discriminant validity, and criterion-related validity. Higher scores on the System of Care Adherence Scale were associated with lower risk of youth psychiatric hospitalization, greater perceived improvement in youth functioning, and greater increases in caregivers’ self-efficacy to access services. Item response theory analyses indicated the items were strongly related to adherence; however, most were optimal for differentiating between low to moderate levels of adherence. The System of Care Adherence Scale is a psychometrically sound measure suitable for population surveillance of the extent to which families experience system of care principles in their interactions with providers.
AchenbachT. M.RuffleT. M. (2000). The Child Behavior Checklist and related forms for assessing behavioral/emotional problems and competencies. Pediatrics in Review, 21(8), 265–271.
AndersonR. L.LyonsJ. S.GilesD. M.PriceJ. A.EstleG. (2003). Reliability of the Child and Adolescent Needs and Strengths-Mental Health (CANS-MH) Scale. Journal of Child and Family Studies, 12(3), 279–289.
4.
BakerF. (2001). The basics of item response theory. ERIC Clearinghouse on Assessment and Evaluation, University of Maryland.
5.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, 57(1), 289–300.
6.
BickmanL. (1996). A continuum of care: More is not always better. American Psychologist, 51(7), 689–701.
7.
BockR. D.AitkinM. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443–459.
8.
BrannanA. M.SonnichsenS. E.HeflingerC. A. (1996). Measuring satisfaction with children’s mental health services: Validity and reliability of the satisfaction scales. Evaluation and Program Planning, 19(2), 131–141.
9.
BrunsE. J.BurchardJ. D.SuterJ. C.Leverentz-BradyK.ForceM. M. (2004). Assessing fidelity to a community-based treatment for youth: The Wraparound Fidelity Index. Journal of Emotional and Behavioral Disorders, 12(2), 79–89.
10.
BryantF. B.YarnoldP. R. (1995). Principal-components analysis and exploratory and confirmatory factor analysis. In GrimmL. G.YarnoldP. R. (Eds.), Reading and understanding multivariate statistics (pp. 99–136). American Psychological Association.
11.
ChalmersR. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29.
12.
DeVellisR. F. (2016). Scale development: Theory and applications (Vol. 26). SAGE.
13.
DillmanD. A.SmythJ. D.ChristianL. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. John Wiley & Sons.
14.
EdelenM. O.ReeveB. B. (2007). Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Quality of Life Research, 16(1), 5–18.
15.
EndersC. K. (2010). Applied missing data analysis. Guilford Press.
16.
GarlandA. F.Haine-SchlagelR.Brookman-FrazeeL.Baker-EriczenM.TraskE.Fawley-KingK. (2013). Improving community-based mental health care for children: Translating knowledge into action. Administration and Policy in Mental Health and Mental Health Services Research, 40(1), 6–22.
17.
GreenbaumP. E.WangW.BoothroydR.KutashK.FriedmanR. M. (2011). Multilevel confirmatory factor analysis of the systems of care implementation survey (SOCIS). Journal of Behavioral Health Services & Research, 38(3), 303–326.
18.
HambletonR. K.SwaminathanH.RogersH. J. (1991). Fundamentals of item response theory. SAGE.
19.
HaytonJ. C.AllenD. G.ScarpelloV. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191–205.
20.
HenggelerS. W.MeltonG. B.BrondinoM. J.SchererD. G.HanleyJ. J. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821–833.
21.
HernandezM.GomezA.LipienL.GreenbaumP. E.ArmstrongK. H.GonzalezP. (2001). Use of the system-of-care practice review in the national evaluation: Evaluating the fidelity of practice to system-of-care principles. Journal of Emotional and Behavioral Disorders, 9(1), 43–52.
22.
HornJ. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185.
23.
KahnJ. H. (2006). Factor analysis in counseling psychology research, training, and practice: Principles, advances, and applications. The Counseling Psychologist, 34(5), 684–718.
24.
KangT.CohenA. S.SungH. J. (2009). Model selection indices for polytomous items. Applied Psychological Measurement, 33(7), 499–518.
25.
KlineR. B. (2015). Principles and practice of structural equation modeling. Guilford Press.
26.
KnitzerJ. (1982). Unclaimed children: The failure of public responsibility to children and adolescents in need of mental health services. Children’s Defense Fund.
27.
KutashK.GreenbaumP. E.WangW.BoothroydR. A.FriedmanR. M. (2011). Levels of system of care implementation: A national benchmark study. Journal of Behavioral Health Services & Research, 38(3), 342–357.
28.
LangeA. M. C.van der RijkenR. E. A.DelsingM. J. M. H.BusschbachJ. J. V.ScholteR. H. J. (2019). Development of therapist adherence in relation to treatment outcomes of adolescents with behavioral problems. Journal of Clinical Child and Adolescent Psychology, 48(Suppl. 1), S337–S346.
29.
LiC. H. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949.
30.
MerikangasK. R.HeJ. P.BrodyD.FisherP. W.BourdonK.KoretzD. S. (2010). Prevalence and treatment of mental disorders among US children in the 2001–2004 NHANES. Pediatrics, 125, 75–81.
31.
MuthénL. K.MuthénB. O. (2017). Mplus user’s guide (Version 8).
32.
NguyenT. H.HanH. R.KimM. T.ChanK. S. (2014). An introduction to item response theory for patient-reported outcome measurement. The Patient-Patient-Centered Outcomes Research, 7(1), 23–35.
33.
OglesB. M.CarlsonB.HatfieldD.KarpenkoV. (2008). Models of case mix adjustment for Ohio mental health consumer outcomes among children and adolescents. Administration and Policy in Mental Health and Mental Health Services Research, 35(4), 295–304.
34.
OrlandoM.ThissenD. (2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24(1), 50–64.
35.
OstiniR.NeringM. L. (2006). Polytomous item response theory models (No. 144). SAGE.
36.
PatelM. M.BrownJ. D.CroakeS.LewisR.LiuJ.PattonL.PotterD. E. B.ScholleS. H. (2015). The current state of behavioral health quality measures: Where are the gaps?Psychiatric Services, 66(8), 865–871.
37.
PerouR.BitskoR. H.BlumbergS. J.PastorP.GhandourR. M.GfroererJ. C.HuangL. N. (2013). Mental health surveillance among children—United States, 2005–2011. MMWR Supplements, 62(2), 1–35.
38.
PiresS. A. (2002). Building systems of care: A primer. National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development Center.
39.
SchoenwaldS. K.GarlandA. F.ChapmanJ. E.FrazierS. L.SheidowA. J.Southam-GerowM. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health, 38, 32–43.
40.
SchreiberJ. B.NoraA.StageF. K.BarlowE. A.KingJ. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99(6), 323–338.
41.
StanickC. F.HalkoH. M.DorseyC. N.WeinerB. J.PowellB. J.PalinkasL. A.LewisC. C. (2018). Operationalizing the “pragmatic” measures construct using a stakeholder feedback and a multi-method approach. BMC Health Services Research, 18(1), Article 882.
42.
StephensR. L.HoldenE. W.HernandezM. (2004). System-of-care practice review scores as predictors of behavioral symptomatology and functional impairment. Journal of Child and Family Studies, 13(2), 179–191.
43.
StroulB. A.BlauG.FriedmanR. M. (2010). Updating the system of care concept and philosophy. Georgetown University Center for Child and Human Development, National Technical Assistance Center for Children’s Mental Health.
44.
StroulB. A.FriedmanR. M. (1986). A system of care for severely emotionally disturbed youth. Georgetown University.
45.
TeichJ. L.MutterR.GibbonsB.MillerK.LynchS.AliM. M. (2018). Use of inpatient behavioral health services by children and adolescents with private insurance coverage. Psychiatric Services, 69(9), 1036–1039.
46.
U.S. Department of Health and Human Services. (1999). Mental health: A report of the Surgeon General. U.S. Department of Health and Human Services, U.S. Public Health Service.
47.
VinsonN. B.BrannanA. M.BaughmanL. N.WilceM.GawronT. (2001). The system-of-care model: Implementation in twenty-seven communities. Journal of Emotional and Behavioral Disorders, 9(1), 30–42.
48.
WilliamsN. J.ScottL.AaronsG. A. (2018). Prevalence of serious emotional disturbance among US children: A meta-analysis. Psychiatric Services, 69(1), 32–40.
