Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of “evidence-based” psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.
AhmadN.BoutronI.DechartresA.DurieuxP.RavaudP. (2010). Applicability and generalisability of the results of systematic reviews to public health practice and policy: A systematic review. Trials, 11(1), 20. https://doi.org/10.1186/1745-6215-11-20
2.
AvellarS. A.ThomasJ.KleinmanR.Sama-MillerE.WoodruffS. E.CoughlinR.WestbrookT. R. (2017). External validity: The next step for systematic reviews?Evaluation Review, 41(4), 283–325. https://doi.org/10.1177/0193841X16665199
3.
AxfordN.MorpethL.BjornstadG.HobbsT.BerryV. (2022). “What works” registries of interventions to improve child and youth psychosocial outcomes: A critical appraisal. Children and Youth Services Review, 137, 106469. https://doi.org/10.1016/j.childyouth.2022.106469
4.
BeckerB. J. (2017). Improving the design and use of meta analyses in evidence-based practice. In SampsonJ. P.Bullock-YowellE.DozierV. C.OsbornD. S.LenzJ. G. (Eds.), Integrating theory, research, and practice in vocational psychology: Current status and future directions (pp. 95–107), Florida State University Libraries. https://doi.org/10.17125/svp2016.ch10
5.
BellS. H.OlsenR. B.OrrL. L.StuartE. A. (2016). Estimates of external validity bias whem impact evaluations select sites nonrandomly. Educational Evaluation and Policy Analysis, 38(2), 318–335. https://doi.org/10.3102/0162373715617549
6.
BurchettH. E. D.BlanchardL.KnealeD.ThomasJ. (2018). Assessing the applicability of public health intervention evaluations from one setting to another: A methodological study of the usability and usefulness of assessment tools and frameworks. Health Research Policy and Systems, 16(1), 88. https://doi.org/10.1186/s12961-018-0364-3
7.
BurchettH. E. D.KnealeD.BlanchardL.ThomasJ. (2020). When assessing generalisability, focusing on differences in population or setting alone is insufficient. Trials, 21(1), 286. https://doi.org/10.1186/s13063-020-4178-6
8.
CahanA.CahanS.CiminoJ. J. (2017). Computer-aided assessment of the generalizability of clinical trial results. International Journal of Medical Informatics, 99, 60–66. https://doi.org/10.1016/j.ijmedinf.2016.12.008
9.
CampbellD. T. (1986). Relabeling internal and external validity for applied social scientists. New Directions for Program Evaluation, 1986(31), 67–77. https://doi.org/10.1002/ev.1434
10.
CartwrightN. (2012). Presidential address: Will this policy work for you? Predicting effectiveness better: How philosophy helps. Philosophy of Science, 79(5), 973–989. https://doi.org/10.1086/668041
11.
CartwrightN. (2020). Middle-range theory: Without it what could anyone do?THEORIA. An International Journal for Theory, History and Foundations of Science, 35(3), 269–323. https://doi.org/10.1387/theoria.21479
12.
CartwrightN.StegengaJ. (2011). A theory of evidence for evidence-based policy. In DawidP.TwiningW.VasilakiM. (Eds.), Evidence, inference and enquiry, Oxford University Press.
13.
CookT. D. (1990). The generalization of causal connections: Multiple theories in search of clear practice. In SechrestL.PerrinE.BunkerJ. (Eds.), Research methodology: Strengthening causal interpretations of nonexperimental data (DHHS Publication No. PHS 90-3454 (pp. 9–31): Department of Health and Human Services.
14.
DahabrehI. J.PetitoL. C.RobertsonS. E.HernánM. A.SteingrimssonJ. A. (2020). Toward causally interpretable meta-analysis: Transporting inferences from multiple randomized trials to a new target population. Epidemiology, 31(3), 334–344. https://doi.org/10.1097/EDE.0000000000001177
15.
DahabrehI. J.RobertsonS. E.PetitoL. C.HernánM. A.SteingrimssonJ. A. (2023). Efficient and robust methods for causally interpretable meta‐analysis: Transporting inferences from multiple randomized trials to a target population. Biometrics, 79(2), 1057–1072. https://doi.org/10.1111/biom.13716
16.
DwanK.AltmanD. G.ClarkeM.GambleC.HigginsJ. P. T.SterneJ. A. C.WilliamsonP. R.KirkhamJ. J. (2014). Evidence for the selective reporting of analyses and discrepancies in clinical trials: A systematic review of cohort studies of clinical trials. PLoS Medicine, 11(6), Article e1001666. https://doi.org/10.1371/journal.pmed.1001666
17.
DwanK.GambleC.WilliamsonP. R.KirkhamJ. J.Reporting Bias GroupReporting Bias Group. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS One, 8(7), Article e66844. https://doi.org/10.1371/journal.pone.0066844
18.
DyrvigA.-K.KidholmK.GerkeO.VondelingH. (2014). Checklists for external validity: A systematic review. Journal of Evaluation in Clinical Practice, 20(6), 857–864. https://doi.org/10.1111/jep.12166
19.
EvansR. E.CraigP.HoddinottP.LittlecottH.MooreL.MurphyS.O'CathainA.PfadenhauerL.RehfuessE.SegrottJ.MooreG. (2019). When and how do ‘effective’ interventions need to be adapted and/or re-evaluated in new contexts? The need for guidance. Journal of Epidemiology & Community Health, 73(6), 481–482. https://doi.org/10.1136/jech-2018-210840
FlayB. R.BiglanA.BoruchR. F.CastroF. G.GottfredsonD.KellamS.MościckiE. K.SchinkeS.ValentineJ. C.JiP. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science: The Official Journal of the Society for Prevention Research, 6(3), 151–175. https://doi.org/10.1007/s11121-005-5553-y
22.
FonagyP.ButlerS.CottrellD.ScottS.PillingS.EislerI.FuggleP.KraamA.ByfordS.WasonJ.EllisonR.SimesE.GanguliP.AllisonE.GoodyerI. M. (2018). Multisystemic therapy versus management as usual in the treatment of adolescent antisocial behaviour (START): A pragmatic, randomised controlled, superiority trial. The Lancet Psychiatry, 5(2), 119–133. https://doi.org/10.1016/S2215-0366(18)30001-4
23.
FonagyP.ButlerS.CottrellD.ScottS.PillingS.EislerI.FuggleP.KraamA.ByfordS.WasonJ.SmithJ. A.AnokhinaA.EllisonR.SimesE.GanguliP.AllisonE.GoodyerI. M. (2020). Multisystemic therapy versus management as usual in the treatment of adolescent antisocial behaviour (START): 5-year follow-up of a pragmatic, randomised, controlled, superiority trial. The Lancet Psychiatry, 7(5), 420–430. https://doi.org/10.1016/S2215-0366(20)30131-0
24.
GottfredsonD. C.CookT. D.GardnerF. E. M.Gorman-SmithD.HoweG. W.SandlerI. N.ZafftK. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science: The Official Journal of the Society for Prevention Research, 16(7), 893–926. https://doi.org/10.1007/s11121-015-0555-x
25.
GrimshawJ. M.EcclesM. P.LavisJ. N.HillS. J.SquiresJ. E. (2012). Knowledge translation of research findings. Implementation Science: IS, 7(1), 50. https://doi.org/10.1186/1748-5908-7-50
26.
HeZ.TangX.YangX.GuoY.GeorgeT. J.CharnessN.Quan HemK. B.HoganW.BianJ. (2020). Clinical trial generalizability assessment in the big data era: A review. Clinical and Translational Science, 13(4), 675–684. https://doi.org/10.1111/cts.12764
27.
HendersonC. R. (1975). Best linear unbiased estimation and prediction under a selection model. Biometrics, 31(2), 423–447. https://doi.org/10.2307/2529430
28.
HenggelerS. W. (2011). Efficacy studies to large-scale transport: The development and validation of multisystemic therapy programs. Annual Review of Clinical Psychology, 7, 351–381. https://doi.org/10.1146/annurev-clinpsy-032210-104615
29.
HenggelerS. W.SheidowA. J. (2012). Empirically supported family-based treatments for conduct disorder and delinquency in adolescents. Journal of Marital and Family Therapy, 38(1), 30–58. https://doi.org/10.1111/j.1752-0606.2011.00244.x
30.
HenrichJ.HeineS. J.NorenzayanA. (2010). The weirdest people in the world?The Behavioral and brain sciences, 33(2–3), 61–135. https://doi.org/10.1017/S0140525X0999152X
31.
HuebschmannA. G.LeavittI. M.GlasgowR. E. (2019). Making health research matter: A call to increase attention to external validity. Annual Review of Public Health, 40(1), 45–63. https://doi.org/10.1146/annurev-publhealth-040218-043945
32.
IntHoutJ.IoannidisJ. P. A.RoversM. M.GoemanJ. J. (2016). Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open, 6(7), Article e010247. https://doi.org/10.1136/bmjopen-2015-010247
33.
JungA.BalzerJ.BraunT.LuedtkeK. (2022). Identification of tools used to assess the external validity of randomized controlled trials in reviews: A systematic review of measurement properties. BMC Medical Research Methodology, 22(1), 100. https://doi.org/10.1186/s12874-022-01561-5
34.
KaleA.LeeS.GoanT.TiptonE.HullmanJ. (2023). MetaExplorer: Facilitating reasoning with epistemic uncertainty in meta-analysis. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, IEEE. https://doi.org/10.1145/3544548.3580869
35.
KazdinA. E. (1998). Psychosocial treatments for conduct disorder in children. In NathanP. E.GormanJ. M. (Eds.), A guide to treatments that work (pp. 65–89), Oxford University Press.
36.
KazdinA. E. (2015). Psychosocial treatments for conduct disorder in children and adolescents. In NathanP. E.GormanJ. M. (Eds.), A guide to treatments that work (4th ed., pp. 141–173), Oxford University Press.
37.
KazdinA. E.WeiszJ. R. (1998). Identifying and developing empirically supported child and adolescent treatments. Journal of Consulting and Clinical Psychology, 66(1), 19–36. https://doi.org/10.1037/0022-006X.66.1.19
38.
Kennedy-MartinT.CurtisS.FariesD.RobinsonS.JohnstonJ. (2015). A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials, 16(1), 495. https://doi.org/10.1186/s13063-015-1023-4
39.
LangeA. M.DelsingM. J.ScholteR. H.van der RijkenR. E. (2018). Factorial structure of the Therapist Adherence Measure-Revised (TAM-R) within multisystemic therapy. European Journal of Psychological Assessment, APA. https://doi.org/10.1027/1015-5759/a000505
40.
LittellJ. H.CorcoranJ.PillaiV. (2008). Systematic reviews and meta-analysis, Oxford University Press.
41.
LittellJ. H.PigottT. D.NilsenK. H.GreenS. J.MontgomeryO. L. K. (2021). Multisystemic Therapy® for social, emotional, and behavioural problems in youth age 10 to 17: An updated systematic review and meta-analysis. Campbell Systematic Reviews, 17(4), Article e1158. https://doi.org/10.1002/cl2.1158
42.
LittellJ. H.PigottT. D.NilsenK. H.RobertsJ.LabrumT. K. (2023). Functional family therapy for families of youth (age 11–18) with behaviour problems: A systematic review and meta-analysis. Campbell Systematic Reviews, 19(3), Article e1324. https://doi.org/10.1002/cl2.1324
43.
MackieJ. L. (1965). Causes and conditions. American Philosophical Quarterly, 2(4), 245–264.
44.
MalmivaaraA. (2020). Generalizability of findings from systematic reviews and meta-analyses in the leading general medical journals. Journal of Rehabilitation Medicine, 52(3), jrm00031. https://doi.org/10.2340/16501977-2659
McLeodB. D.WeiszJ. R. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72(2), 235–251. https://doi.org/10.1037/0022-006X.72.2.235
47.
McMahonS.FleuryJ. (2012). External valdity of physical activity interventions for community-dwelling older adults with fall risk: A quantitative systematic literature review. Journal of Advanced Nursing, 68(10), 2140–2154. https://doi.org/10.1111/j.1365-2648.2012.05974.x
48.
PigottT. D.ValentineJ. C.PolaninJ. R.WilliamsR. T.CanadaD. D. (2013). Outcome-reporting bias in education research. Educational Researcher, 42(8), 424–432. https://doi.org/10.3102/0013189X13507104
49.
PustejovskyJ. E.TiptonE. (2022). Meta‐analysis with robust variance estimation: Expanding the range of working models. Prevention Science: The Official Journal of the Society for Prevention Research, 23(3), 425–438. https://doi.org/10.1007/s11121-021-01246-3
50.
RobbinsM. S.AlexanderJ. F.TurnerC. W.HollimonA. (2016). Evolution of functional family therapy as an evidence-based practice for adolescents with disruptive behavior problems. Family Process, 55(3), 543–557. https://doi.org/10.1111/famp.12230
51.
RobinsonG. K. (1991). That BLUP is a good thing: The estimation of random effects. Statistical Science, 6(1), 15–32). https://doi.org/10.1214/ss/1177011926
52.
RossiP. H.LipseyM. W.HenryG. T. (2019). Evaluation: A systematic approach (8th ed.), Sage Publications.
53.
RothsteinHannahSAlex J.BorensteinM. (2005). Publication bias in meta-analysis: Prevention, assessment, and adjustments. Chichester, UK: John Wiley & Sons.
54.
SchoenwaldS. K. (2008). Toward evidence-based transport of evidence-based treatments: MST as an example. Journal of Child & Adolescent Substance Abuse, 17(3), 69–91. https://doi.org/10.1080/15470650802071671
55.
SchünemannH. J.HigginsJ. P. T.VistG. E.GlasziouP.AklE. A.SkoetzN.GuyattG. H. (2023). Chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence. In Higginsj. P. T.ThomasJ.ChandlerJ.CumpstonM.LiT.PageM. J.WelchV. A. (Eds.), Cochrane handbook for systematic reviews of interventions version 6.4. Cochrane. https://www.training.cochrane.org/handbook
56.
SextonT.TurnerC. W. (2010). The effectiveness of functional family therapy for youth with behavioral problems in a community practice setting. Journal of Family Psychology: JFP: journal of the Division of Family Psychology of the American Psychological Association (Division 43), 24(3), 339–348. https://doi.org/10.1037/a0019406
57.
ShackelfordG. E.MartinP. A.HoodA. S. C.ChristieA. P.KulinskayaE.SutherlandW. J. (2021). Dynamic meta-analysis: A method of using global evidence for local decision making. BMC Biology, 19(1), 33. https://doi.org/10.1186/s12915-021-00974-w
58.
ShadishW. R. (1995). The logic of generalization: Five principles common to experiments and ethnographies. American Journal of Community Psychology, 23(3), 419–428. https://doi.org/10.1007/BF02506951
59.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for general causal inference, Houghton Mifflin.
60.
SloughT.TysonS. A. (2023). External validity and meta-analysis. American Journal of Political Science, 67(2), 440–455. DOI:10.1111/ajps.12742
61.
SongF.ParekhS.HooperL.LokeY. K.RyderJ.SuttonA. J.HingC.KwokC. S.PangC.HarveyI. (2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14(8), 1–193. https://doi.org/10.3310/hta14080
62.
SongF.Parekh-BhurkeS.HooperL.LokeY.RyderJ.SuttonA.HingC.HarveyI. (2009). Extent of publication bias in different categories of research cohorts: A meta-analysis of empirical studies. BMC Medical Research Methodology, 9(1), 79. https://doi.org/10.1186/1471-2288-9-79
63.
ThomsonH. J.ThomasS. (2012). External validity in healthy public policy: Application of the RE-AIM tool to the field of housing improvement. BMC Public Health, 12(1), 633. https://doi.org/10.1186/1471-2458-12-633
64.
TiptonE.HallbergK.HedgesL. V.ChanW. (2017). Implications of small samples for generalization: Adjustments and rules of thumb. Evaluation Review, 41(5), 472–505. https://doi.org/10.1177/0193841X16655665
65.
TiptonE.OlsenR. B. (2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 47(8), 516–524. https://doi.org/10.3102/0013189X18781522
66.
TolinD. F.McKayD.FormanE. M.KlonskyE. D.ThombsB. D. (2015). Empirically supported treatment: Recommendations for a new model. Clinical Psychology: Science and Practice, 22(4), 317–338. https://doi.org/10.1111/cpsp.12122
67.
ValentineJ. C.BiglanA.BoruchR. F.CastroF. G.CollinsL. M.FlayB. R.KellamS.MościckiE. K.SchinkeS. P. (2011). Replication in prevention science. Prevention Science: The Official Journal of the Society for Prevention Research, 12(2), 103–117. https://doi.org/10.1007/s11121-011-0217-6
68.
WeiseA.BüchterR.PieperD.MathesT. (2020). Assessing context suitability (generalizability, external validity, applicability or transferability) of findings in evidence syntheses in healthcare—An integrative review of methodological guidance. Research Synthesis Methods, 11(6), 760–779. https://doi.org/10.1002/jrsm.1453
69.
WholeyJ. S. (1987). Evaluability assessment: Developing program theory. New Directions for Program Evaluation, 1987(33), 77–92. https://doi.org/10.1002/ev.1447
70.
WilliamsM. J. (2019). External validity and policy adaptation: From impact evaluation to policy design. The World Bank Research Observer, 35(2), 158–191. https://doi.org/10.1093/wbro/lky010
