Integrated service systems of care for children with severe emotional disturbance based on the systems-of-care framework have historically been viewed and studied as programmatic and clinical interventions. Consequently, researchers have most commonly applied program evaluation methods to study their effectiveness.These methods rely largely on quasi-experimental designs with repeated outcome measures. Systems of care can also be understood, however, as a set of systems-level policies and principles designed to alter the delivery of services provided to youth and their families. From this perspective, researchers can apply analytical and methodological tools traditionally used in health policy analysis to study the multiple levels of systems-of-care impacts. This article examines how these and other methods for policy analysis can be utilized to measure the impact of the systems-of-care approach.Where possible, results from existing systems-of-care evaluation studies illustrate how these analyses and methods can be practically employed.
Get full access to this article
View all access options for this article.
References
1.
Aday, L.A., & Andersen, R. (1981). Equity of access to medical care: A conceptual and empirical overview. Medical Care, 19, 4-27.
2.
Aday, L.A., Begley, C.E., Lairson, D.R., & Slater, C.H. (1998). Evaluating the healthcare system: Effectiveness, efficiency and equity. Chicago: Health Administration Press.
3.
Angold, A., Costello, E.J., Bums, B.J., Erkanli, A., & Farmer, E.M.Z. (2000). Effectiveness of nonresidential specialty mental health services for children and adolescents in the "real world." Journal of the American Academy of Child andAdolescentPsychiatry, 39, 154-160.
4.
Bickman, L. (1996a). The evaluation of a children's mental health managed care demonstration. The Journal of Mental Health Administration , 23, 7-15.
5.
Bickman, L. (1996b). Reinterpreting the Fort Bragg evaluation findings: The message does not change. The Journal of Mental Health Administration, 23, 137-145.
6.
Bickman, L., Noser, K., & Summerfelt, W.T. (1999). Long-term effects of a system of care on children and adolescents. The Journal of Behavioral Health Services & Research, 26, 185-202.
7.
Brannan, A.M., Baughman, L.N., Reed, E.D., & Katz-Leavy, J. (2002). System-of-care assessment: Cross-site comparison of findings. Children's Services: Social Policy, Research, and Practice, 5(1), 37-56.
8.
Brook, R., & Lohr, K. (1985). Efficacy, effectiveness, variations, and quality: Boundary-crossing research. Medical Care, 23, 710-722.
9.
Bruns, E.J., & Burchard, J.D. (2000). Impact of respite care services for families with children experiencing emotional and behavioral problems. Children's Services: Social Policy, Research, and Practice, 3(1), 39-61.
10.
Bruns, E.J., Burchard, J.D., & Yoe, J.T. (1995). Evaluating the Vermont system of care: Outcomes associated with community-based wraparound services. Journal of Child and Family Studies, 4, 321-339.
11.
Bums, B.J., Hoagwood, K., & Maultsby, L.T. (1998). Improving outcomes for children and adolescents with serious emotional and behavioral disorders: Current and future directions. In M. H. Epstein , K. Kutash, & A. J. Duchnowski (Eds.), Outcomes for children and youth with emotional and behavioral disorders and theirfamilies: Programs and evaluation best practices (pp. 686-707). Austin, TX: PRO-ED.
12.
Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally .
13.
Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton-Mifflin.
14.
Donabedian, A. (1982). Explorations in quality assessment and monitoring: Vol. 2. The criteria and standards of quality. Chicago : Health Administration Press.
15.
Drummond, M.F.B., O'Brien, G.L., Stoddart, G.L., & Torrance, G.W. (1997). Methods for the economic evaluation of health care programmes. New York: Oxford University Press.
16.
Etzioni, A. (1967). Mixed scanning: A third approach to decision making . Public Administration Review, 27, 385-392.
17.
Evans, R. G., Barer, M. L., & Marmor, T. R. (Eds.). (1994). Why are some people healthy and others not? The determinants of health of populations. New York : Aldine De Gruyter.
18.
Farmer, E.M.Z. (2000). Issues confronting effective services in systems of care. Children and Youth Services Review, 22, 627-650.
19.
Friedman, R.M. (1999). A conceptual framework for developing and implementing effective policy in children's mental health. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute , Research and Training Center for Children's Mental Health.
20.
Glisson, C., & Hemmelgam, A. (1998). The effects of organizational climate and inter-organizational coordination on the quality and outcomes of children's service systems. Child Abuse and Neglect, 22, 401-421.
21.
Hayes, M.T. (1992). Incrementalism and public policy. New York: Longman.
22.
Henggeler, S.W., & Borduin, C.M. (1990). Family therapy and beyond: A multisystemic approach to treating the behavior problems of children and adolescents. Pacific Grove, CA: Brooks/Cole.
23.
Henggeler, S.W., Melton, G.B., & Smith, L.A. (1992). Multisystemic treatment of serious juvenile offenders: An effective alternative to incarceration. Journal of Consulting and Clinical Psychology, 60, 953-961.
24.
Hoagwood, K., Hibbs, E., Brent, D., & Jensen, P. (1995). Efficacy and effectiveness studies of child and adolescent psychotherapy. Journal of Consulting and Clinical Psychology, 63, 683-687.
25.
Hoagwood, K., Jensen, P.S., Petti, T., & Burns, B.J. (1996). Outcomes of mental health care for children and adolescents: I. A comprehensive conceptual model. Journal of the American Academy of Child and Adolescent Psychiatry, 35, 1064-1077.
26.
Hogan, C.D. (1999). Vermont communities count: Using results to strengthen services for families and children. Baltimore: Annie E. Casey Foundation.
27.
Holden, E.W., & Brannan, A.M. (2002). Background for the special issue. Children's Services: Social Policy, Research, and Practice, 5(1), 1-2.
28.
Holden, E.W., De Carolis, G., & Huff, B. (2002). Policy implications of the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program. Children's Services: Social Policy, Research, and Practice, 5(1), 57-66.
29.
Kagan, S.L., & Neville, P.R. (1993). Integrating services for children and families: Understanding the past to shape the future. New Haven, CT: Yale University Press.
30.
Knitzer, J. (1982). Unclaimed children. Washington, DC: Children's Defense Fund.
31.
Kutash, K., & Rivera, V.R. (1996). What works in children's mental health services? Uncovering answers to critical questions. Baltimore: Brookes.
32.
Lasswell, H.D. (1951). The policy orientation. In D. Lerneer & H. D. Lasswell (Eds.), The policy sciences: Recent developments in scope and methods (pp. 3-15). Stanford, CA: Stanford University Press.
33.
Manteuffel, B., Stephens, R.L., & Santiago, R. (2002). Overview of the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program and summary of current findings. Children's Services: Social Policy, Research, and Practice, 5(1), 3-20.
34.
Milio, N. (1983). Primary care and the public's mental health. Lexington, MA: Lexington Books.
35.
National Advisory Mental Health Council. ( 1999). Bridging science and service: A report by the National Advisory Mental Health Council's Clinical Treatment and Services Research Workgroup. Bethesda, MD: National Institute of Mental Health.
36.
Pandiani, J.A., Schacht, L.M., & Banks, S.M. (1996). A longitudinal evaluation of a system of care for children and adolescents with severe emotional disturbance. Evaluation and Program Planning, 19, 365-376.
37.
Quinn, K.P., & Epstein, M.H. (1998). Characteristics of children, youth, and families served by local interagency systems of care. In M. H. Epstein, K. Kutash, & A. Duchnowski (Eds.), Outcomes for children and youth with emotional and behavioral disorders and their families: Programs and evaluation best practices (pp. 81-114). Austin, TX: PRO-ED.
38.
Rosenblatt, A. (1998). Assessing the child and family outcomes of systems of care for youth with severe emotional disturbance. In M. H. Epstein, K. Kutash , & A. Duchnowski (Eds.), Outcomes for children and youth with emotional and behavioral disorders and their families: Programs and evaluation best practices (pp. 329-362). Austin, TX: PRO-ED.
39.
Rosenblatt, A., & Attkisson, C.C. (1997). Integrating systems of care in California for youth with severe emotional disturbance: IV. Educational attendance and achievement. Journal of Child and Family Studies , 6, 113-129.
40.
Rosenblatt, A., Attkisson, C.C., & Fernandez, A.J. (1992). Integrating systems of care in California for youth with severe emotional disturbance: II. Initial group home expenditure and utilization findings from the California AB377 Evaluation Project. Journal of Child and Family Studies, 1, 263-286.
41.
Schoenwald, S.K., Borduin, C.M., & Henggeler, S.W. (1998). Multisystemic therapy: Changing the natural and service ecologies of adolescents and families. In M. H. Epstein, K. Kutash, & A. Duchnowski (Eds.), Outcomes for children and youth with emotional and behavioral disorders and their families: Programs and evaluation best practices (pp. 485-511). Austin, TX: PRO-ED.
42.
Sechrest, L. B., Perrin E. , & Bunker, J. (Eds.). ( 1990). Research methodology: Strengthening causal interpretations of nonexperimental data, PHS90-3545. Rockville, MD: Agency for Health Care Policy and Research.
43.
Stroul, B.A. (1993). Systems of care for children and adolescents with severe emotional disturbances: What are the results?Washington, DC: Georgetown University Child Development Center .
44.
Stroul, B.A., & Friedman, R.M. (1986). A system of care for seriously emotionally disturbed children and youth. Washington, DC : Georgetown University Child Development Center.
45.
Stroul, B.A., & Friedman, R.M. (1996). The system of care concept and philosophy. In B. A. Stroul, (Ed.), Children's mental health: Creating systems of care in a changing society (pp. 3-21). Baltimore: Brookes .
46.
Stroul, B.A., Pires, S.A., Armstrong, M.I., & Zaro, S. (2002). The impact of managed care on systems of care that serve children with serious emotional disturbances and their families. Children's Services: Social Policy, Research, and Practice , 5(1), 21-36.
47.
Trochim, W.M.K. (1984). Research design for program evaluation: The regression discontinuity approach. Beverly Hills, CA: Sage.
48.
U.S. Department of Health and Human Services. ( 1999). Mental health: A report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.
49.
Weisz, J.R., Weiss, B., Alicke, M.D., & Klotz, M.L. (1987). Effectiveness of psychotherapy with children and adolescents: A meta-analysis for clinicians. Journal of Consulting and Clinical Psychology, 55, 542-549.
50.
Weisz, J.R., Weiss, B., Han, S.S., Granger, D.A., & Morton, T. (1995). Effects of psychotherapy with children and adolescents revisited: A metananalysis of treatment outcome studies. Psychological Bulletin,117, 450-468.
51.
Wennberg, J.E. (1990). Small area analysis and the medical care outcome problem. In L. Sechrest, E. Perrin, & J. Bunker (Eds.), Research methodology: Strengthening causal interpretations of nonexperimental data, PHS90-3454. Rockville, MD: Agency for Health Care Policy and Research.
52.
Woodbridge, M.W., & Huang, L.N. (2000). Using evaluation data to manage, improve, market, and sustain children's services. Systems of care: Promising practices in children's mental health (2000 Series, Vol. 2). Washington, DC: Center for Effective Collaboration and Practice, American Institutes for Research .
53.
Yoe, J.T., Santarcangelo, S., Atkins, M., & Burchard, J.D. (1996). Wraparound care in Vermont: Program development, implementation, and evaluation of a statewide system of individualized services. Journal of Child and Family Studies , 5, 23-37.
