This article describes the core features of outcome research and then explores issues confronting researchers who engage in outcome studies. Using an intervention research perspective, descriptive and explanatory methods are distinguished. Emphasis is placed on the counterfactual causal perspective, designing programs that fit culture and context, and developing nuanced explanations for program outcomes. Five emerging challenges are discussed: (a) adapting interventions to the contexts and cultures in which programs are to be implemented, (b) avoiding potentially false attributions of program failure due to differential implementation, (c) making causal inferences from observational data with propensity score analysis (PSA), (d) examining person-centered outcomes in program evaluation, and (e) adjusting for rater effects in longitudinal research.
AagaardJ.Müller-NielsenJ. (2011). Clinical outcome of assertive community treatment (ACT) in a rural area in Denmark: A case-control study with a 2-year follow-up. Nordic Journal of Psychiatry, 1–7. Advanced online publication. doi:10.3109/08039488.2010.544405
2.
AsparouhovT.MuthénB. O. (2008). Multilevel mixture models. In HancockG. R.SamuelsenK. M. (Eds.), Advances in latent variable mixture models (pp. 27–51). Charlotte, NC: Information Age.
3.
BauerD. J.CurranP. J. (2003). Distributional assumptions of growth mixture models: Implications for over-extraction of latent trajectory classes. Psychological Methods, 8, 338–363. doi:10.1037/1082-989X.8.3.338.
4.
BauerD. J.CurranP. J. (2004). The integration of continuous and discrete latent variable models: Potential problems and promising opportunities. Psychological Methods, 1, 3–29. doi:10.1037/1082-989X.9.1.3.
5.
BearmanS. K.UguetoA.AlleyneA.WeiszJ. R. (2010). Adapting cognitive-behavioral therapy for depression to fit diverse youths and contexts. In WeiszJ. R.KazdinA. E. (Eds.), Evidence-based psychotherapies for children and adolescents (pp. 466–481). New York, NY: Guilford Press.
6.
BerkR. A. (2004). Regression analysis: A constructive critique. Thousand Oaks, CA: SAGE.
7.
BothaU. A.KoenL.JoskaJ. A.HeringL. M.OosthuizenP. P. (2010). Assessing the efficacy of a modified assertive community-based treatment programme in a developing country. BMC Psychiatry, 10, 73–81. doi:10.1186/1471-244X-10-73.
8.
BrookhartM. A.SchneeweissS.RothmanK. J.GlynnR. J.AvornJ.StürmerT. (2006). Variable selection for propensity score models. American Journal of Epidemiology, 163, 1149–1156. doi:10.1093/aje/kwj149.
9.
CarrollC.PattersonM.WoodS.BoothA.RickJ.BalainS. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40.doi:10.1186/1748-5908-2-40.
10.
CarrollK. M.NuroK. F. (2002). One size cannot fit all: A stage model for psychotherapy manual development. Clinical Psychology: Science and Practice, 9, 396–406. doi:10.1093/clipsy/9.4.396.
11.
ColeS. R.HernánM. A. (2008). Constructing inverse probability weights for marginal structural models. American Journal of Epidemiology, 168, 656–664. doi:10.1093/aje/kwn164.
12.
CollinsL. M.GrahamJ. W.RousculpS. S.FidlerP. L.PanJ.HansenW. B. (1994). Latent transition analysis and how it can help address prevention research questions. In CollinsL. M.SeitzL. A. (Eds.), Advances in data analysis for prevention intervention research (pp. 81–111). Rockville, MD: National Institutes of Health. Retrieved fromhttp://archives.drugabuse.gov/pdf/monographs/142.pdf
13.
CollinsL. M.WugalterS. E. (1992). Latent class models for stage-sequential dynamic latent variables. Multivariate Behavioral Research, 27, 131–157. doi:10.1207/s15327906mbr2701_8.
14.
CoombesL.AllenD.MarshM.FoxcroftD. (2009). The Strengthening Families Programme (SFP) 10-14 and substance misuse in Barnsley: The perspectives of facilitators and families. Child Abuse Review, 18, 41–59. doi:10.1002/car.1055.
15.
CronbachL. J.GleserG. G.NandaH.RajaratnamN. (1972). The dependability of behavioral measurement: Theory of generalizability of scores and profiles. New York, NY: John Wiley.
16.
CurtisN. M.RonanK. R.BourduinC. M. (2004). Multisystemic treatment: A meta-analysis of outcome studies. Journal of Family Psychology, 18, 411–419. doi:10.1037/0893-3200.18.3.411.
17.
CurtisN. M.RonanK. R.HeiblumN.CrellinK. (2009). Dissemination and effectiveness of multisystemic treatment in New Zealand: A benchmarking study. Journal of Family Psychology, 23, 119–129. doi:10.1037/a0014974.
18.
D’AgostinoR. B.,Jr. (1998). Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine, 17, 2265–2281. doi:10.1002/(SICI)1097-0258(19981015)17:19<2265::AID-SIM918>3.0.CO;2-B.
19.
DrakeC. (1993). Effects of misspecification of the propensity score on estimators of treatment effect. Biometrics, 49, 1231–1236. doi:10.2307/2532266.
20.
FixsenD. L.BlaséK. ADudaM. A.NaoomS. F.Van DykeM. (2010). Implementation of evidence-based treatments for children and adolescents. In WeiszJ. R.KazdinA. E. (Eds.), Evidence-based psychotherapies for children and adolescents (pp. 435–450). New York, NY: Guilford Press.
21.
FoxcroftD. R.IrelandD. J.Lister-SharpD.LoweG.BreenR. (2003). Longer-term primary prevention for alcohol misuse in young people: A systematic review. Addiction, 98, 397–411.
22.
FraserM. W.GalinskyM. J.SmokowskiP. R.DayS. H.TerzianM. A.RoseR. A.GuoS. (2005). Social information-processing skills training to promote social competence and prevent aggressive behavior in third grade. Journal of Consulting and Clinical Psychology, 73, 1045–1055. doi:10.1037/0022-006X.73.6.1045.
FraserM. W.RichmanJ. M.GalinskyM. J. (1999). Risk, protection, and resilience: Toward a conceptual framework for social work practice. Social Work Research, 23, 131–143. doi:10.1177/104973159900900211.
25.
FraserM. W.RichmanJ. M.GalinskyM. J.DayS. H. (2009). Intervention research: Developing social programs. New York, NY: Oxford University Press.
26.
FreedmanD. A.BerkR. A. (2008). Weighting regressions by propensity scores. Evaluation Review, 32, 392–409. doi:10.1177/0193841X08317586.
27.
FreyK. S.NolanS. B.Van Schoiack EdstromL.HirschsteinM. K. (2005). Effects of a school-based, social-emotional competence program: Linking children’s goals, attributions, and behaviour. Journal of Applied Developmental Psychology, 26, 171–200. doi:10.1016/j.appdev.2004.12.002.
28.
FriedmanJ.HastieT.TibshiraniR. (2000). Additive logistic regression: A statistical view of boosting. Annals of Statistics, 28, 337–374. doi:10.1214/aos/1016218223.
29.
GibsonW. A. (1959). Three multivariate models: Factor analysis, latent structure analysis, and latent profile analysis. Psychometrika, 24, 229–252. doi:10.1007/BF02289845.
30.
GlynnR. J.SchneeweissS.StürmerT. (2006). Indications for propensity scores and review of their use in pharmacoepidemiology. Basic & Clinical Pharmacology & Toxicology, 98, 253–259. doi:10.1111/j.1742-7843.2006.pto_293.x.
GreeneW. H. (2003). Econometric analysis. 5th edUpper Saddle River, NJ: Prentice Hall.
33.
GreenlandS.RobinsJ. M. (1986). Identifiability, exchangeability, and epidemiological confounding. International Journal of Epidemiology, 15, 413–419. doi:10.1093/ije/15.3.413.
34.
GreenlandS.RobinsJ. M. (2009). Identifiability, exchangeability and confounding revisited. Epidemiologic Perspectives & Innovations, 6, 4.doi:10.1186/1742-5573-6-4.
35.
GreenwaldP.CullenJ. W. (1985). The new emphasis in cancer control. Journal of the National Cancer Institute, 74, 543–551.
36.
GuoS.FraserM. W. (2010). Propensity score analysis: Statistical methods and applications. Thousand Oaks, CA: SAGE.
37.
HarderV. S.StuartE. A.AnthonyJ. C. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15, 234–249. doi:10.1037/a0019623.
38.
HenggelerS. W.SchaefferC. M. (2010). Treating serious emotional and behavioral problems using multisystemic therapy. Australian and New Zealand Journal of Family Therapy, 31, 149–164.
39.
HolsenI.SmithB. H.FreyK. S. (2008). Outcomes of the social competence program Second Step in Norwegian elementary schools. School Psychology International, 29, 71–88. doi:10.1177/0143034307088504.
40.
HueyS. J.PoloA. J. (2010). Assessing the effects of evidence-based psychotherapies with ethnic minority youths. In WeiszJ. R.KazdinA. E. (Eds.), Evidence-based psychotherapies for children and adolescents (pp. 451–465). New York, NY: Guilford Press.
41.
ImbensG. W.WooldridgeJ. M. (2009). Recent developments in the econometrics of program evaluation. Journal of Economic Literature, 47, 5–86. doi:10.1257/jel.47.1.5.
42.
Institute of Education Sciences. (2010). Efficacy of school-wide programs to promote social and character development and reduce problem behavior in elementary school children. Retrieved fromhttp://ies.ed.gov/ncer/pubs/20112001/index.asp
43.
ItoJ.OshimaI.NishioM.KunoE. (2009). Initiative to build a community-based mental health system including assertive community treatment for people with severe mental illness in Japan. American Journal of Psychiatric Rehabilitation, 12, 247–260. doi: 10.1080/15487760903066404.
44.
KennedyP. (2003). A guide to econometrics (5th ed.).
Cambridge, MA: MIT Press.
45.
KillaspyH.BebbingtonP.BlizardR.JohnsonS.NolanF.PillingS.KingM. (2006). The REACT Study: Randomised evaluation of assertive community treatment in North London. British Medical Journal, 332, 815–819. doi:10.1136/bmj.38773.518322.7C.
46.
KillaspyH.KingettS.BebbingtonP.BlizardR.JohnsonS.NolanF.KingM. (2009). Randomised evaluation of assertive community treatment: 3-year outcomes. British Journal of Psychiatry, 195, 81–82. doi:10.1192/bjp.bp.108.059303.
47.
KlineR. B. (2005). Principles and practices of structural equation modeling (2nd ed.).
New York, NY: Guilford Press.
48.
KumpferK. L.AlvaradoR. (2003). Family strengthening approaches for the prevention of youth problem behaviors. American Psychologist, 58, 457–465. doi:10.1037/0003-066X.58.6-7.457.
49.
KumpferK. L.PinyuchonM.Teixeira de MeloA.WhitesideH. O. (2008). Cultural adaptation process for international dissemination of the Strengthening Families Program. Evaluation and the Health Professions, 31, 226–239. doi: 10.1177/0163278708315926.
50.
KumpferK. L.WhitesideH. O.GreeneJ. A.AllenK. C. (2010). Effectiveness outcomes of four age versions of the Strengthening Families Program in statewide field sites. Group Dynamics: Theory, Research, and Practice, 14, 211–229. doi:10.1037/a0020602.
51.
LambertM.BockT.SchottleD.GolksD.MeisterK.RietschelL, … Schimmelmann, B. G. (2010). Assertive community treatment as part of integrated care versus standard care: A 12-month trial in patients with first- and multiple-episode schizophrenia spectrum disorders treated with quetiapine immediate release. Journal of Clinical Psychiatry, 71, 1313–1323. doi:10.4088/JCP.09m05113yel.
52.
LarsenT.SamdalO. (2007). Implementing Second Step: Balancing fidelity and program adaptation. Journal of Educational and Psychological Consultation, 17, 1–29. doi: 10.1207/s1532768Xjepc1701_1.
53.
LeshiedA.CunninghamA. (2002). Seeking effective interventions for serious young offenders: Interim results of a four-year randomized study of multisystemic therapy in Ontario, Canada. London and Ontario, CA: Centre for Children & Families in the Justice System. Retrieved fromhttp://www.lfcc.on.ca/seeking.html
54.
LindleyD. V.SmithA. F. M. (1972). Bayes estimates for the linear model. Journal of the Royal Statistical Society, Series B, 34, 1–41.
55.
LoY.MendallN. R.RubinD. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88, 767–778. doi:10.1093/biomet/88.3.767.
56.
LuellenJ. K.ShadishW. R.ClarkM. H. (2005). Propensity scores: An introduction and experimental test. Evaluation Review, 29, 530–558. doi:10.1177/0193841X05275596.
57.
MacKinnonD. P. (2008). Introduction to statistical mediation analysis. New York, NY: Lawrence Erlbaum.
58.
MagnussonD. (1998). The logical implications of a person-oriented approach. In CarinsR. B.BergmanL. R.KaganJ. (Eds.), Methods and models for studying the individual (pp. 33–64). Thousand Oaks, CA: SAGE.
59.
MaldonadoG. (2009). Update: Greenland and Robins (1986). Identifiability, exchangeability and epidemiological confounding. Epidemiologic Perspectives & Innovations, 6, 3.doi:10.1186/1742-5573-6-3.
60.
MarshallM.LockwoodA. (1998). Assertive community treatment for people with severe mental disorders. Cochrane Database of Systematic Reviews, 1998, (2):CD001089.doi:10.1002/14651858.CD001089.
61.
McCaffreyD. F.RidgewayG.MorralA. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9, 403–425. doi:10.1037/1082-989X.9.4.403.
62.
McCutcheonA. L. (2002). Basic concepts and procedures in single and multiple group latent class analysis. In HagenaarsJ. A.McCutcheonA. L. (Eds.), Applied latent class analysis (pp. 56–88). New York, NY: Cambridge University Press.
63.
MihalicS. (2004). The importance of implementation fidelity. Report on Emotional and Behavioral Disorders in Youth, 4, 83–90.
64.
MitchellC. M.PlunkettM. (2000). The latent structure of substance use among American Indian adolescents: An example using categorical variables. American Journal of Community Psychology, 28, 105–125. doi:10.1023/A:1005146530634.
65.
MuthénB. O. (2001). Latent variable mixture modeling. In MarcoulidesG. A.SchumackerR. E. (Eds.), New developments and techniques in structural equation modeling (pp. 1–33). Mahwah, NJ: Lawrence Erlbaum.
66.
MuthénB. O. (2003). Statistical and substantive checking in growth mixture modeling: Comment on Bauer and Curran (2003). Psychological Methods, 8, 369–377. doi:10.1037/1082-989X.8.3.369.
67.
MuthénB. O.MuthénL. (2000). Integrating person-centered and variable-centered analysis: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical and Experimental Research, 24, 882–891.
68.
MuthénL. K.MuthénB. O. (2010). Mplus user’s guide (6th ed.).
Los Angeles, CA: Author.
NylundK. A.MuthénB. O.BellmoreA.GrahamS. (2006). Stability and instability of peer victimization during middle school: Using latent transition analysis with covariates, distal outcomes, and modeling extensions. Retrieved fromhttp://www.statmodel.com/download/LTA_DP_FINAL.pdf
71.
OdgenT.Halliday-BoykinsC. A. (2004). Multisystemic treatment of antisocial adolescents in Norway: Replication of clinical outcomes outside of the US. Child and Adolescent Mental Health, 9, 77–83. doi:10.1111/j.1475-3588.2004.00085.x.
72.
PerkinsS. M.TuW.UnderhillM. G.ZhouX. H.MurrayM. D. (2000). The use of propensity scores in pharmacoepidemiologic research. Pharmacoepidemiology & Drug Safety, 9, 93–101. doi:10.1002/(SICI)1099-1557(200003/04)9:2<93::AID-PDS474>3.0.CO;2-I.
73.
ProctorE.SilmereH.RaghavanR.HovmandP.AaronsG.BungerA.HensleyM. (2011). Administration and Policy in Mental Health, 38, 65–76. doi:10.1007/s10488-010-0319-7.
74.
RaudenbushS. W. (1993). A crossed random effects model for unbalanced data with applications in cross-sectional and longitudinal research. Journal of Educational Statistics, 18, 321–349.
75.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.).
Thousand Oaks, CA: SAGE.
76.
ReboussinB. A.ReboussinD. M.LiangK. Y.AnthonyJ. C. (1998). Latent transition modeling of the progression of health-risk behavior. Multivariate Behavioral Research, 33, 457–478. doi:10.1207/s15327906mbr3304_2.
77.
RobinsJ. M.HernánM. A.BrumbackB. (2000). Marginal structural models and causal inference in epidemiology. Epidemiology, 11, 550–560. doi:10.1097/00001648-200009000-00011.
78.
RosenbaumP. R. (2010). Design of observational studies. New York, NY: Springer.
79.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55. doi:10.1093/biomet/70.1.41.
80.
RosenbaumP. R.RubinD. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516–524. doi:10.2307/2288398.
81.
RothmanJ. (1974). Planning and organizing for social change: Action principles from social science research. New York, NY: Columbia University Press.
82.
In RothmanJ.ThomasE. J. (Eds.), (1994). Intervention research: Design and development for human services. New York, NY: Haworth Press.
83.
RubinD. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology, 2, 169–188. doi:10.1023/A:1020363010465.
84.
RubinD. B. (2004). On principles for modeling propensity scores in medical research. Pharmacoepidemiology & Drug Safety, 13, 855–857. doi:10.1002/pds.968.
85.
RubinD. B. (2007). The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine, 26, 20–36. doi:10.1002/sim.2739.
86.
RubinD. B.ThomasN. (1996). Matching using estimated propensity scores: Relating theory to practice. Biometrics, 52, 249–264. doi:10.2307/2533160.
87.
SchickA.CierpkaM. (2005). Faustlos: Evaluation of a curriculum to prevent violence in elementary schools. Applied and Preventive Psychology, 11, 157–165. doi:10.1016/j.appsy.2005.05.001.
88.
SchoenwaldS. K.GarlandA. F.ChapmanJ. E.FrazierS. L.SheidowA. J.Southam-GerowM. A. (2010). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health, 38, 32–43. doi:10.1007/s10488-010-0321-0.
89.
SchoenwaldS. K.HeiblumN.SaldanaL.HenggelerS. W. (2008). The international implementation of multisystemic therapy. Evaluation and the Health Profession, 31, 211–225. doi:10.1177/0163278708315925.
90.
SchuermanJ. R.RzepnickiT. L.LittellJ. H. (1995). Putting families first: An experiment in family preservation. New York, NY: Aldine de Gruyter.
91.
SetoguchiS.SchneeweissS.BrookhartM. A.GlynnR. J.CookE. F. (2008). Evaluating uses of data mining techniques in propensity score estimation: A simulation study. Pharmacoepidemiology & Drug Safety, 17, 546–555. doi:10.1002/pds.1555.
92.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York, NY: Houghton Mifflin.
93.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Newbury Park, CA: SAGE.
94.
SkärstrandE.LarssonJ.AndréassonS. (2008). Cultural adaptation of the Strengthening Families Programme to a Swedish setting. Health Education, 108, 287–300. doi:10.1108/09654280810884179.
95.
SkärstrandE.SundellK.AndréassonS. (2010). Evaluation of a Swedish version of the Strengthening Families Programme: Three-year outcomes of a cluster randomized trial. In SkärstrandE. (Ed.), Prevention of alcohol and drug problems among adolescents: Evaluating a Swedish version of the Strengthening Families Program. Stockholm: Division of Social Medicine, Department of Public Health Sciences, Karolinska Institutet.
96.
SteinerP. M.CookT. D.ShadishW. R.ClarkM. H. (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15, 250–267. doi:10.1037/a0018719.
97.
StoneR. A.ObroskyD. S.SingerD. E.KapoorW. N.FineM. J. (1995). Propensity score adjustment for pretreatment differences between hospitalized and ambulatory patients with community-acquired pneumonia. Pneumonia Patient Outcomes Research Team (PORT) Investigators. Medical Care, 33(Suppl. 4):AS56–66.
98.
StuartE. A. (2008). Developing practical recommendations for the use of propensity scores: Discussion of “A critical appraisal of propensity score matching in the medical literature between 1996 and 2003 by Peter Austin, Statistics in Medicine.”. Statistics in Medicine, 27, 2062–2065. doi:10.1002/sim.3207.
99.
StürmerT.JoshiM.GlynnR. J.AvornJ.RothmanK. J.SchneeweissS. (2006). A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. Journal of Clinical Epidemiology, 59, 437–447. doi:10.1016/j.jclinepi.2005.07.004.
100.
SundellK.HanssonK.LöfholmC. A.OlssonT.GustleL.KadesjöC (2008). The transportability of multisystemic therapy to Sweden: Short-term results from a randomized trial of conduct-disordered youths. Journal of Family Psychology, 3, 550–560. doi:10.1037/a0012790.
101.
SytemaS.WunderinkL.BloemersW.RoordaL.WiersmaD. (2007). Assertive community treatment in the Netherlands: A randomized controlled trial. Acta Psychiatrica Scandinavica, 116, 105–112. doi:10.1111/j.1600-0447.2007.01021.x.
102.
ThomasE. J. (1978). Generating innovation in social work: The paradigm of developmental research. Journal of Social Service Research, 2, 95–115. doi:10.1300/J079v02n01_08.
103.
VelicerW. F.MartinR. A.CollinsL. M. (1996). Latent transition analysis for longitudinal data. Addiction, 91, 197–209. doi:10.1080/09652149638926.
104.
WeissbergR. P.KumpferK. L.SeligmanM. E. P. (2003). Prevention that works for children and youth. American Psychologist, 58, 425–432.
105.
WeitzenS.LapaneK. L.ToledanoA. Y.HumeA. L.MorV. (2004). Principles for modeling propensity scores in medical research: A systematic literature review. Pharmacoepidemiology & Drug Safety, 13, 841–853. doi:10.1002/pds.969.
