Findings from an evaluation of a model system for delivering mental health services to youth were reassessed to determine the robustness of key findings to the use of methodologies unavailable to the original analysts. These analyses address a key concern about earlier findings—that the quasi-experimental design involved the comparison of two noncomparable groups. The authors employed propensity score methodology to reconsider between-group baseline differences in observed characteristics of participating families. The authors also considered the possible effect of unobserved between-group differences. The data support previous studies that show few differences in outcomes, but the findings are sensitive to unobserved heterogeneity.
Aakvik, A.2001. Bounding a matching estimator: The case of a Norwegian training program. Oxford Bulletin of Economics and Statistics63:115-43.
2.
Achenbach, T.M. (1991). Manual for the child behavior checklist and 1991 profile. Burlington, VA: University of Vermont.
3.
Austin, P.C., P. Grootendorst, and G.M. Anderson.2007. A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: A Monte Carlo study. Statistics in Medicine26:734-53.
4.
Austin, P.C., P. Grootendorst, S.L.T. Normand, and G. M. Anderson. 2007. Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: A Monte Carlo study. Statistics in Medicine26:754-68.
5.
Bickman, L.1997. Resolving issues raised by the Fort Bragg evaluation: New directions for mental health services research. American Psychologist52:562-5.
Bickman, L., P.R. Guthrie, E.M. Foster, E.W. Lambert, T.W. Summerfelt, C. Breda, and C.A. Heflinger.1995. Evaluating managed mental health services: The Fort Bragg experiment, New York. NY: Plenum Press.
8.
Bickman, L., E.W. Lambert, A.R. Andrade, and R.V. Penaloza.2000. The Fort Bragg continuum of care for children and adolescents: mental health outcomes over 5 years. Journal of Consulting and Clinical Psychology68:710-6.
9.
Bickman, L., K. Noser, and W.T. Summerfelt .1999. Long-term effects of a system of care on children and adolescents[a]. Journal of Behavioral Health Services and Research26:18.
10.
Bickman, L., W.T. Summerfelt, and E.M. Foster.1996. Research on systems of care: Implications of the Fort Bragg evaluation. Children's mental health: Creating systems of care in a changing society, ed. B. A. Stroul, Baltimore, MD: Brookes Publishing Company.
11.
Brookhart, M.A., S. Schneeweiss, K.J. Rothman, R.J. Glynn, J. Avorn, and T. Stürmer.2006. Variable selection for propensity score models. American Journal of Epidemiology163:1149-56.
12.
Campbell, D.T., and J.C. Stanley.1963. Experimental and quasi-experimental designs for research on teaching. Handbook of research on teaching, ed. N. L. Gage, Chicago, IL: Rand McNally.
13.
---. 1966. Experimental and quasi-experimental designs for research , Chicago, IL: Rand McNally.
14.
Cook, T.D., and D.T. Campbell.1979. Quasi-experimentation: Design & analysis issues for field settings, Chicago, IL: Rand McNally College Publishing Company.
15.
Cook, T.D., W.R. Shadish, and V.C. Wong. Forthcoming. Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management .
16.
Cornfield, J., W. Haenszel, E.C. Hammond, A.M. Lilienfeld, M.B. Shimkin, and E.L. Wynder.1959. Smoking and lung cancer: Recent evidence and a discussion of some questions. Journal of the National Cancer Institute22:173-203.
17.
Foster, E.M., and L. Bickman.1996. An evaluator's guide to detecting attrition problems. Evaluation Review20:695-723.
18.
---. 2000. Refining the costs analyses of the Fort Bragg evaluation: The impact of cost offset and cost shifting. Mental Health Services Research2:13-25.
19.
Foster, E.M., T.W. Summerfelt, and R. Saunders.1996. The costs of a continuum of care: The lessons of the Fort Bragg demonstration. Journal of Mental Health Administration23:92-106.
20.
Gangl, M.2004. RBOUNDS: Stata module to perform Rosenbaum sensitivity analysis for average treatment effects on the treated. Stata Version7.
21.
Gu, X.S., and P.R. Rosenbaum.1993. Comparison of multivariate matching methods: Structures, distances, and algorithms. Journal of Computational and Graphical Statistics2:405-20.
22.
Hahn, J., P. Todd, and W. Klaauw.2001. Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica69:201-09.
23.
Hansen, B.B.2006. Full matching in an observational study of coaching for the SAT. Journal of the American Statistical Association99:609-18.
24.
Hansen, B.B., and S.O. Klopfer.2006. Optimal full matching and related designs via network flows. Journal of Computational and Graphical Statistics15:609-27.
25.
Heckman, J.J., H. Ichimura, and P.E. Todd.1997. Matching as an econometric evaluation estimator: Evidence from evaluating a job training program. Review of Economic Studies64:605-54.
26.
---. 1998. Matching as an econometric evaluation estimator. Review of Economic Studies65:261-94.
27.
Heckman, J.J., and E.J. Vytlacil.2007. Econometric evaluation of social programs, part I: Causal models, structural models and econometric policy evaluation . Handbook of Econometrics6:4779-874.
28.
Hodges, K., and J. Gust.1995. Measures of impairment for children and adolescents. The Journal of Mental Health Administration22:403-13.
29.
Hodges, K., and M.M. Wong.1997. Use of the child and adolescent functional assessment scale to predict service utilization and cost. Journal of Mental Health Administration24:278-90.
30.
Holland, P.W.1986. Statistics and causal inference. Journal of the American Statistical Association81:945-60.
31.
Lahiri, K., and J.G. Song.2000. The effect of smoking on health using a sequential self-selection model. Health Economics9:491-511.
32.
Lee, M.-J.2005. Micro-econometrics for policy, program, and treatment effects. Advanced texts in econometrics, Oxford, NY: Oxford University Press.
33.
Leuven, E., and B. Sianesi.2003. PSMATCH2: Stata module to perform full Mahalanobis and propensity score matching, common support graphing, and covariate imbalance testing. Statistical Software Components S 432001.
34.
Mordock, J.B.1997. The Fort Bragg continuum of care demonstration project: The population served was unique and the outcomes are questionable. Child Psychiatry and Human Development27:241-54.
35.
Morgan, S.L., and C. Winship.2007. Counterfactuals and causal inference: Methods and principles for social research, New York, NY : Cambridge University Press.
36.
Pires, S.A.1997. Lessons learned from the Fort Bragg demonstration: An overview. Lessons learned from the Fort Bragg demonstration, ed. S. A. Pires, Tampa, FL: The Louis de la Parte Florida Mental Health Institute.
37.
Reich, S.M., and L. Bickman.2005. Quasi-experimental design. Outcomes for children and youth with emotional and behavioral disorders and their families: Programs and evaluations, best practices, eds. M. H. Epstein, K. Kutash, and A. J. Duchnowski, Austin, TX: PRO-ED.
38.
Rosenbaum, P.R.1988. Sensitivity analysis for matching with multiple controls . Biometrika75:577.
39.
---. 2002a. Does a does-response relationship reduce sensitivity to hidden bias? Biostatistics4:1-10.
40.
---. 2002b. Observational studies. 2nd ed. Springer series in statistics. New York, NY : Springer.
41.
---. 2004. Design sensitivity in observational studies. Biometrika91:153-64.
42.
Rosenbaum, P.R., and D.B. Rubin.1983. The Central role of the propensity score in observational studies for causal effects. Biometrika70:41-55.
43.
---. 1985. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician39:33-8.
44.
Rubin, D.B.2005. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association100:322-31.
45.
StataCorp.2005. Stata statistical software: Release 9, College Station, TX: StataCorp LP.
46.
U.S. Department of Health and Human Services.1999. Mental health: A report of the surgeon general, Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.
47.
Verbeek, M., and T. Nijman.1992. Testing for selectivity bias in panel data models. International Economic Review33:681-703.
48.
Winship, C., and S.L. Morgan.1999. The estimation of causal effects from observational data. Annual Review of Sociology25:659.
