Multisite mediation studies represent an analytic cornerstone in the development and assessment of theories across a broad array of disciplines because they can critically map out development processes and create opportunities to learn how context constrains or facilitates mediating mechanisms. Despite the widespread use of multisite mediation analyses, a notable missing component is methods for predicting statistical power and sample sizes that position studies to detect such effects. The literature has emphasized the importance of complementing main effect power analyses with mediation effect power analyses (even if considered a secondary objective) because they frequently suggest different designs, sample sizes, and resource allocations. Evaluation of different designs considering competing estimands during planning stages helps to balance competing priorities and strengthen the overall quality of evidence generated from studies. In this study, we develop formulas to predict power and guide the design of multisite studies probing mediation under a linear structural formulation.
ArnoldM. E. (2006). Developing evaluation capacity in extension 4-H field faculty: A framework for success. The American Journal of Evaluation, 27(2), 257–269. https://doi.org/10.1177/1098214006287989
2.
AvinC.ShpitserI.PearlJ. (2005). Identifiability of Path-Specific Effects.
BaiF.XieY.KelceyB.AtanekaA.McLeanL.PhelpsG. (2024). Design parameter values for planning mediation studies with teacher and student mathematics outcomes. Journal of Research on Educational Effectiveness, 18(3), 704–738. https://doi.org/10.1080/19345747.2024.2349670
5.
BauerD. J.PreacherK. J.GilK. M. (2006). Conceptualizing and testing random indirect effects and moderated mediation in multilevel models: New procedures and recommendations. Psychological Methods, 11(2), 142–163. https://doi.org/10.1037/1082-989X.11.2.142
6.
BaumeisterR. F.BushmanB. J.TiceD. M. (2023). Multi-site replications in social psychology: Reflections, implications, and future directions. The Spanish Journal of Psychology, 26, Article e3. https://doi.org/10.1017/SJP.2023.6
7.
BaumeisterR. F.TiceD. M.BushmanB. J. (2022). A review of multisite replication projects in social psychology: Is it viable to sustain any confidence in social psychology’s knowledge base?Perspectives on Psychological Science, 18(4), 912–935. https://doi.org/10.1177/17456916221121815. (Original work published 2023)
8.
BollenK. A.StineR. (1990). Direct and indirect effects: Classical and bootstrap estimates of variability. Sociological Methodology, 20, 115–140. https://doi.org/10.2307/271084
9.
CarlisleJ. F.KelceyB.BerebitskyD. (2013). Teachers’ support of students’ vocabulary learning during literacy instruction in high poverty elementary schools. American Educational Research Journal, 50(6), 1360–1391. https://doi.org/10.3102/0002831213492844
10.
CoxK.KelceyB. (2019a). Optimal design of cluster- and multisite-randomized studies using fallible outcome measures. Evaluation Review, 43(3-4), 189–225. https://doi.org/10.1177/0193841X19870878
11.
CoxK.KelceyB. (2019b). Optimal sample allocation in group-randomized mediation studies with a group-level mediator. The Journal of Experimental Education, 87(4), 616–640. https://doi.org/10.1080/00220973.2018.1496060
12.
CoxK.KelceyB.LuceH. (2024). Power to detect moderated effects in studies with three-level partially nested data. The Journal of Experimental Education, 92(1), 130–153. https://doi.org/10.1080/00220973.2022.2130130
13.
DongN.KelceyB.SpybrookJ. (2021). Design considerations in multisite randomized trials probing moderated treatment effects. Journal of Educational and Behavioral Statistics, 46(5), 527–559. https://doi.org/10.3102/1076998620961492
14.
DongN.KelceyB.SpybrookJ. (2023). Experimental design and power for moderation in multisite cluster randomized trials. The Journal of Experimental Education, 92(4), 741–757. https://doi.org/10.1080/00220973.2023.2226934
15.
DongN.KelceyB.SpybrookJ.XieY.PhamD.QiuP.SuiN. (2024). A practical guide to power analyses of moderation effects in multisite individual and cluster randomized trials. The Journal of Experimental Education, 1–20. https://doi.org/10.1080/00220973.2024.2338521
16.
DonovanD. M.BogenschutzM. P.PerlH.ForcehimesA.AdinoffB.MandlerR.OdenN.WalkerR. (2012). Study design to examine the potential role of assessment reactivity in the screening, motivational assessment, referral, and treatment in emergency departments (SMART-ED) protocol. Addiction Science & Clinical Practice, 7(1), 16–11. https://doi.org/10.1186/1940-0640-7-16
17.
DonovanD. M.DaleyD. C.BrighamG. S.HodgkinsC. C.PerlH. I.FloydA. S. (2011). How practice and science are balanced and blended in the NIDA clinical trials network: The bidirectional process in the development of the STAGE-12 protocol as an example. The American Journal of Drug and Alcohol Abuse, 37(5), 408–416. https://doi.org/10.3109/00952990.2011.596970
18.
FairchildA. J.McDanielH. L. (2017). Best (but oft-forgotten) practices: Mediation analysis1,2. The American Journal of Clinical Nutrition, 105(6), 1259–1271. https://doi.org/10.3945/ajcn.117.152546
19.
Language and Reading Research Consortium (LARRC), FarquharsonK.MurphyK. (2016). Ten steps to conducting a large, multi-site, longitudinal investigation of language and reading in young children. Frontiers in Psychology, 7, 419–419. https://doi.org/10.3389/fpsyg.2016.00419
20.
FussellH. E.KunkelL. E.McCartyD.LewyC. S. (2011). Standardized patient walkthroughs in the national drug abuse treatment clinical trials network: Common challenges to protocol implementation. The American Journal of Drug and Alcohol Abuse, 37(5), 434–439. https://doi.org/10.3109/00952990.2011.596969
21.
GardnerF.HutchingsJ.BywaterT.WhitakerC. (2010). Who benefits and how does it work? Moderators and mediators of outcome in an effectiveness trial of a parenting intervention. Journal of Clinical Child and Adolescent Psychology, 39(4), 568–580. https://doi.org/10.1080/15374416.2010.486315
GreenfieldS. F.RosaC.PutninsS. I.GreenC. A.BrooksA. J.CalsynD. A.CohenL. R.EricksonS.GordonS. M.HaynesL.KilleenT.MieleG.TrossS.WinhusenT. (2011). Gender research in the National Institute on Drug Abuse National Treatment Clinical Trials Network: A summary of findings. The American Journal of Drug and Alcohol Abuse, 37(5), 301–312. https://doi.org/10.3109/00952990.2011.596875
24.
HongG. (2015). Causality in a social world: moderation, meditation and spill-over (1st ed.). Wiley.
25.
HongG.RaudenbushS. W. (2006). Evaluating kindergarten retention policy: A case study of causal inference for multilevel observational data. Journal of the American Statistical Association, 101(475), 901–910. https://doi.org/10.1198/016214506000000447
JacobR.HillH.CoreyD. (2017). The impact of a professional development program on Teachers’ mathematical knowledge for teaching, instruction, and student achievement. Journal of Research on Educational Effectiveness, 10(2), 379–407. https://doi.org/10.1080/19345747.2016.1273411
29.
KanM. L.RamirezD. D. (2020). Preventing subsequent teenage pregnancy: A multisite analysis of goal orientation and social supports. Children & Schools, 42(4), 225–235. https://doi.org/10.1093/cs/cdaa023
30.
KelceyB.BaiF.XieY. (2020). Statistical power in partially nested designs probing multilevel mediation. Psychotherapy Research, 30(8), 1061–1074. https://doi.org/10.1080/10503307.2020.1717012
31.
KelceyB.DongN.SpybrookJ.CoxK. (2017a). Statistical power for causally-defined indirect effects in group-randomized trials with individual-level mediators. Journal of Educational and Behavioral Statistics, 42(5), 499–530. https://doi.org/10.3102/1076998617695506
32.
KelceyB.DongN.SpybrookJ.ShenZ. (2017b). Experimental power for indirect effects in group-randomized studies with group-level mediators. Multivariate Behavioral Research, 52(6), 699–719. https://doi.org/10.1080/00273171.2017.1356212
33.
KelceyB.HillH. C.ChinM. J. (2019a). Teacher mathematical knowledge, instructional quality, and student outcomes: A multilevel quantile mediation analysis. School Effectiveness and School Improvement, 30(4), 398–431. https://doi.org/10.1080/09243453.2019.1570944
34.
KelceyB.PhelpsG. (2013a). Considerations for designing group randomized trials of professional development with teacher knowledge outcomes. Educational Evaluation and Policy Analysis, 35(3), 370–390. https://doi.org/10.3102/0162373713482766. (Original work published 2013)
35.
KelceyB.PhelpsG. (2013b). Strategies for improving power in school-randomized studies of professional development. Evaluation Review, 37(6), 520–554. https://doi.org/10.1177/0193841X14528906
36.
KelceyB.ShenZ. (2020). Strategies for efficient experimental design in studies probing 2-1-1 mediation. The Journal of Experimental Education, 88(2), 311–334. https://doi.org/10.1080/00220973.2018.1533796
KelceyB.SpybrookJ.DongN.BaiF. (2020). Cross-level mediation in school-randomized studies of teacher development: Experimental design and power. Journal of Research on Educational Effectiveness, 13(3), 459–487. https://doi.org/10.1080/19345747.2020.1726540
39.
KelceyB.SpybrookJ.PhelpsG.JonesN.ZhangJ. (2017c). Designing large-scale multisite and cluster-randomized studies of professional development. The Journal of Experimental Education, 85(3), 389–410. https://doi.org/10.1080/00220973.2016.1220911
40.
KelceyB.XieY.SpybrookJ.DongN. (2021). Power and sample size determination for multilevel mediation in three-level cluster-randomized trials. Multivariate Behavioral Research, 56(3), 496–513. https://doi.org/10.1080/00273171.2020.1738910
LévesqueA.CampbellA. N. C.PavlicovaM.HuM.-C.WalkerR.McClureE. A.GhitzaU. E.BaileyG.StitzerM.NunesE. V. (2017). Coping strategies as a mediator of internet-delivered psychosocial treatment: Secondary analysis from a NIDA CTN multisite effectiveness trial. Addictive Behaviors, 65, 74–80. https://doi.org/10.1016/j.addbeh.2016.09.012
43.
LiuX. (2003). Statistical power and Optimum sample allocation ratio for treatment and control having unequal costs per unit of randomization. Journal of Educational and Behavioral Statistics, 28(3), 231–248. https://doi.org/10.3102/10769986028003231. (Original work published 2003)
44.
MacKinnonD. P.LockwoodC.HoffmanJ. (1998). A new method to test for mediation. In Annual meeting of the society for prevention research, Park City, UT (pp. 268–274).
45.
MacKinnonD. P.LockwoodC. M.HoffmanJ. M.WestS. G.SheetsV. (2002). A comparison of methods to test mediation and other intervening Variable effects. Psychological Methods, 7(1), 83–104. https://doi.org/10.1037/1082-989X.7.1.83
46.
MacKinnonD. P.LockwoodC. M.WilliamsJ. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39(1), 99–128. https://doi.org/10.1207/s15327906mbr3901_4
47.
MoerbeekM.BreukelenG. J. P. V.BergerM. P. F.LeeuwJ. d.MeijerE. (2008). Optimal designs for multilevel studies. In Handbook of multilevel analysis (pp. 177–205). Springer New York. https://doi.org/10.1007/978-0-387-73186-5_4
48.
MossB. G.KelceyB.ShowersN. (2014). Does classroom composition matter? College classrooms as moderators of developmental education effectiveness. Community College Review, 42(3), 201–220. https://doi.org/10.1177/0091552114529153. Original work published in 2014
49.
MuthénB. (2011). Applications of causally defined direct and indirect effects in mediation analysis using SEM in Mplus.
50.
National Institute of Child Health and Human Development (NICHD). (2023). Multisite Clinical Research: Leveraging Network Infrastructure to Advance Research for Women, Children, Pregnant and Lactating Individuals, and Persons with Disabilities (U01 Clinical Trial Optional), PAR-23-037. Available at https://grants.nih.gov/grants/guide/pa-files/PAR-23-037.html.
PhillipsG.LindemanP.AdamesC. N.BettinE.BaystonC.StonehouseP.KernD.JohnsonA. K.BrownC. H.GreeneG. J. (2019). Empowerment evaluation: A case study of citywide implementation within an HIV prevention context. The American Journal of Evaluation, 40(3), 318–334. https://doi.org/10.1177/1098214018796991
54.
PituchK. A.StapletonL. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41(4), 630–670. https://doi.org/10.1177/0049124112460380. (Original work published 2012)
PreacherK. J.SeligJ. P. (2010). Monte Carlo method for assessing multilevel mediation: An interactive tool for creating confidence intervals for indirect effects in 1-1-1 multilevel models [Computer software]. Available from http://quantpsy.org/.
57.
PreacherK. J.ZyphurM. J.ZhangZ. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209–233. https://doi.org/10.1037/a0020141
58.
QinX. (2024). Sample size and power calculations for causal mediation analysis: A tutorial and shiny app. Behavior Research Methods, 56(3), 1738–1769. https://doi.org/10.3758/s13428-023-02118-0
59.
QinX.DeutschJ.HongG. (2021). Unpacking complex mediation mechanisms and their heterogeneity between sites in a job corps evaluation. Journal of Policy Analysis and Management, 40(1), 158–190. https://doi.org/10.1002/pam.22268
60.
QinX.HongG. (2017). A weighting method for assessing between-site heterogeneity in causal mediation mechanism. Journal of Educational and Behavioral Statistics, 42(3), 308–340. https://doi.org/10.3102/1076998617694879
61.
QinX.HongG.DeutschJ.BeinE. (2019). Multisite causal mediation analysis in the presence of complex sample and survey designs and non-random non-response. Journal of the Royal Statistical Society. Series A, Statistics in Society, 182(4), 1343–1370. https://doi.org/10.1111/rssa.12446
62.
RaudenbushS. W.BloomH. S. (2015). Learning about and from a distribution of program impacts using multisite trials. The American Journal of Evaluation, 36(4), 475–499. https://doi.org/10.1177/1098214015600515
63.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods. Sage Publications.
64.
RaudenbushS. W.LiuX. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5(2), 199–213. https://doi.org/10.1037/1082-989x.5.2.199
65.
RogD. J. (2015). Designing, managing, and analyzing multisite evaluations. In NewcomerK. E.HatryH. P.WholeyJ. S. (Eds.), Handbook of practical program evaluation (pp. 225–258). John Wiley & Sons, Inc. https://doi.org/10.1002/9781119171386.ch10
66.
SchimmelpfennigR.SpicerR.WhiteC.GervaisW. M.NorenzayanA.HeineS.MuthukrishnaM. (2023). A problem in theory and more: Measuring the moderating role of culture in Many Labs 2. https://doi.org/10.31234/osf.io/hmnrx
67.
ShenZ.KelceyB. (2020). Optimal sample allocation under unequal costs in cluster-randomized trials. Journal of Educational and Behavioral Statistics, 45(4), 446–474. https://doi.org/10.3102/1076998620912418
68.
ShenZ.KelceyB. (2022a). Optimal sample allocation for three-level multisite cluster-randomized trials. Journal of Research on Educational Effectiveness, 15(1), 130–150. https://doi.org/10.1080/19345747.2021.1953200
69.
ShenZ.KelceyB. (2022b). Optimal sample allocation in multisite randomized trials. The Journal of Experimental Education, 90(3), 693–711. https://doi.org/10.1080/00220973.2020.1830361
70.
ShirleyM. R.BonamerJ.ClarkeC.HassS.IvoryC.KittoS.PriceD.WeissM.YakushevaO.ChappellK.JohantgenM. (Eds.). (2021). Multi-site Research Playbook: A practical guide to support multi-site research studies for greater impact. American Nurses Credentialing Center's Research Council, American Nurses Credentialing Center. Available at https://www.nursingworld.org/ancc/nursing-research/multi-site-research-playbook/.
71.
SimM.KimS.SuhY. (2022). Sample size requirements for simple and complex mediation models. Educational and Psychological Measurement, 82(1), 76–106. https://doi.org/10.1177/00131644211003261
72.
SkinnsL.SorsbyA.RiceL. (2020). “Treat them as a human being”: Dignity in police detention and its implications for ‘good’ police custody. British Journal of Criminology, 60(6), 1667–1688. https://doi.org/10.1093/bjc/azaa051
73.
SobelM. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290–312. https://doi.org/10.2307/270723
74.
SobelM. E. (1986). Some new results on indirect effects and their standard errors in covariance structure models. Sociological Methodology, 16, 159–186. https://doi.org/10.2307/270922
75.
SpybrookJ.WestineC. D.TaylorJ. A. (2016). Design parameters for impact research in science education: A multistate analysis. AERA Open, 2(1), 1–15.https://doi.org/10.1177/2332858415625975
76.
SpybrookJ.ZhangQ.KelceyB.DongN. (2020). Learning from cluster randomized trials in education: An assessment of the capacity of studies to determine what works, for whom, and under what conditions. Educational Evaluation and Policy Analysis, 42(3), 354–374. https://doi.org/10.3102/0162373720929018
77.
SpybrookJ. K. (2007). Examining the experimental designs and statistical power of group randomized trials funded by the Institute of Education Sciences (Doctoral dissertation, University of Michigan).
78.
TaiB.SparenborgS.LiuD.StrausM. (2011). The national drug abuse treatment clinical trials network: Forging a partnership between research knowledge and community practice. Substance Abuse and Rehabilitation, 2(default), 21–28. https://doi.org/10.2147/SAR.S16756
79.
TofighiD.WestS. G.MacKinnonD. P. (2013). Multilevel mediation analysis: The effects of omitted variables in the 1-1-1 model. British Journal of Mathematical & Statistical Psychology, 66(2), 290–307. https://doi.org/10.1111/j.2044-8317.2012.02051.x
80.
TrossS.CampbellA. N. C.CalsynD. A.MetschL. R.SorensenJ. L.ShoptawS.HaynesL.WoodyG. E.MalowR. M.BrownL. S.FeasterD. J.BoothR. E.MandlerR. N.MassonC.HolmesB. W.ColfaxG.BrooksA. J.HienD. A.SchackmanB. R. … Clinical Trials Network HIV Special Interest Group. (2011). NIDA's clinical trials network: An opportunity for HIV research in community substance abuse treatment programs. The American Journal of Drug and Alcohol Abuse, 37(5), 283–293. https://doi.org/10.3109/00952990.2011.596977
VanderWeeleT. J. (2010). Bias formulas for sensitivity analysis for direct and indirect effects. Epidemiology (Cambridge, Mass.), 21(4), 540–551. https://doi.org/10.1097/EDE.0b013e3181df191c
83.
VanderweeleT. J.VansteelandtS. (2009). Conceptual issues concerning mediation, interventions and composition. Statistics and Its Interface, 2(4), 457–468. https://doi.org/10.4310/SII.2009.v2.n4.a7
84.
WakimP. G.RosaC.KothariP.MichelM. E. (2011). Relation of study design to recruitment and retention in CTN trials. The American Journal of Drug and Alcohol Abuse, 37(5), 426–433. https://doi.org/10.3109/00952990.2011.596972
85.
WaldA. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society, 54, 426–482. https://doi.org/10.1090/S0002-9947-1943-0012401-3
86.
WestineC. D. (2016). Finding efficiency in the design of large multisite evaluations: Estimating variances for science achievement studies. The American Journal of Evaluation, 37(3), 311–325. https://doi.org/10.1177/1098214015624014
87.
WestineC. D.UnluF.TaylorJ.SpybrookJ.ZhangQ.AndersonB. (2020). Design parameter values for impact evaluations of science and mathematics interventions involving teacher outcomes. Journal of Research on Educational Effectiveness, 13(4), 816–839. https://doi.org/10.1080/19345747.2020.1821849
88.
WilfleyD. E.SaelensB. E.SteinR. I.BestJ. R.KolkoR. P.SchechtmanK. B.EpsteinL. H. (2017). Dose, content, and mediators of family-based treatment for childhood obesity: A multisite randomized clinical trial. JAMA Pediatrics, 171(12), 1151–1159. https://doi.org/10.1001/jamapediatrics.2017.2960
89.
WilsonD. B.OlaghereA.KimbrellC. S. (2018). Effectiveness of restorative justice principles in juvenile justice: A meta-analysis. Inter-university Consortium for Political and Social Research, Available at https://www.ojp.gov/pdffiles1/ojjdp/grants/250872.pdf
90.
XuZ.GaoF.FaA.QuW.ZhangZ. (2024). Statistical power analysis and sample size planning for moderated mediation models. Behavior Research Methods, 56(6), 6130–6149. https://doi.org/10.3758/s13428-024-02342-2
91.
YaremychH. E.PreacherK. J.HedekerD. (2023). Centering categorical predictors in multilevel models: Best practices and interpretation. Psychological Methods, 28(3), 613–630. https://doi.org/10.1037/met0000434
92.
YoshidaD. T.SendjayaS.HirstG.CooperB. (2014). Does servant leadership foster creativity and innovation? A multi-level mediation study of identification and prototypicality. Journal of Business Research, 67(7), 1395–1404. https://doi.org/10.1016/j.jbusres.2013.08.013
93.
ZhangZ.ZyphurM. J.PreacherK. J. (2008). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12(4), 695–719. https://doi.org/10.1177/1094428108327450. (Original work published 2009)
94.
ZhouX.ShuL.XuZ.PadrónY. (2023). The effect of professional development on in-service STEM teachers’ self-efficacy: A meta-analysis of experimental studies. International Journal of STEM Education, 10(1), 37–20. https://doi.org/10.1186/s40594-023-00422-x
95.
ZvochK.LetourneauL. E.ParkerR. P. (2007). A multilevel multisite outcomes-by-implementation evaluation of an early childhood literacy model. The American Journal of Evaluation, 28(2), 132–150. https://doi.org/10.1177/1098214007301138
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
