Analysis of the differential treatment effects across targeted subgroups and contexts is a critical objective in many evaluations because it delineates for whom and under what conditions particular programs, therapies or treatments are effective. Unfortunately, it is unclear how to plan efficient and effective evaluations that include these moderated effects when the design includes partial nesting (i.e., disparate grouping structures across treatment conditions). In this study, we develop statistical power formulas to identify requisite sample sizes and guide the planning of evaluations probing moderation under two-level partially nested designs. The results suggest that the power to detect moderation effects in partially nested designs is substantially influenced by sample size, moderation effect size, and moderator variance structure (i.e., varies within groups only or within and between groups). We implement the power formulas in the R-Shiny application PowerUpRShiny and demonstrate their use to plan evaluations.
AguinisH.BeatyJ.BoikR.PierceC. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90, 94–107. https://doi.org/10.1037/0021-9010.90.1.94
2.
AshcraftM.KrauseJ. (2007). Working memory, math performance, and math anxiety. Psychonomic Bulletin & Review, 14(2), 243–248.
3.
AshcraftM.MooreA. (2009). Mathematics anxiety and the affective drop in performance. Journal of Psychoeducational Assessment, 27(3), 197–205.
BauerD. J.SterbaS. K.HallforsD. D. (2008). Evaluating group-based interventions when control participants are ungrouped. Multivariate Behavioral Research, 43, 210–236.
7.
BloomH. S. (2005). Randomizing groups to evaluate place-based programs. In BloomH. S. (Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 115–172). Russell Sage Foundation.
8.
BloomH. S.Richburg-HayesL.BlackA. R. (2007). Using covariates to improve precision: Empirical guidelines for studies that randomize schools to measure the impacts of educational interventions. Educational Evaluation and Policy Analysis, 29, 30–59.
9.
BulusM.DongN.KelceyB.SpybrookJ. (2019). PowerUpR: Power Analysis Tools for Multilevel Randomized Experiments. R package version 1.0.4. https://CRAN.R-project.org/package=PowerUpR
10.
CandlishJ.TeareM.DimairoM.FlightL.MandefieldL.WaltersS. (2018). Appropriate statistical methods for analysing partially nested randomised controlled trials with continuous outcomes: A simulation study. BMC Medical Research Methodology, 18, 105.
11.
DishionT. J.PoulinF.BurrastonB. (2001). Peer group dynamics associated with iatrogenic effects in group interventions with high-risk young adolescents. New Directions for Child and Adolescent Development, (91), 79–92.
12.
DongN.KelceyB.SpybrookJ. (2018). Power analyses for moderator effects in three-level cluster randomized trials. The Journal of Experimental Education, 86(3), 489–514.
13.
DongN.KelceyB.SpybrookJ.MaynardR. A. (2016). PowerUp!-Moderator: A tool for calculating statistical power and minimum detectable effect size differences of the moderator effects in cluster randomized trials[Software]. http://www.causalevaluation.org/
14.
HallP.ZehrC.PaulitzkiJ.RhodesR.HallP. (2014). Implementation intentions for physical activity behavior in older adult women: An examination of executive function as a moderator of treatment effects. Annals of Behavioral Medicine: A Publication of the Society of Behavioral Medicine, 48(1), 130–136.
15.
HedgesL.CitkowiczM. (2015). Estimating effect size when there is clustering in one treatment group. Behavior Research Methods, 47, 1295–1308.
16.
HedgesL.HedbergE. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87.
17.
Institute of Education Sciences. (2016). Request for application: Statistical and research methodology in education. U.S. Department of Education. http://ies.ed.gov/funding/pdf/2017_84305D.pdf
18.
JaciwA.LinL.MaB.KelceyB.SpybrookJ. (2016). An empirical study of design parameters for assessing differential impacts for students in group randomized trials. Evaluation Review, 40(5), 410–443.
19.
KelceyB.DongN.SpybrookJ.CoxK. (2017). Statistical power for causally-defined indirect effects in group-randomized trials with individual-level mediators. Journal of Educational and Behavioral Statistics, 42, 499–530.
20.
KelceyB.ShenZ.SpybrookJ. (2016). Intraclass correlation coefficients for designing cluster-randomized trials in Sub-Saharan Africa education. Evaluation Review, 40(6), 500–525.
21.
KennyD.CalsynR.MorseG.KlinkenbergW.WinterJ.TrustyM. (2004). Evaluation of treatment programs for persons with severe mental illness: Moderator and mediator effects. Evaluation Review, 28(4), 294–324.
22.
KorendijkE.MaasC.HoxJ.MoerbeekM. (2012). The robustness of the parameter and standard error estimates in trials with partially nested data: A simulation study. In KorendijkE. (Ed.), Robustness and optimal design issues for cluster randomized trials (pp. 59–94) [Dissertation]. Utrecht University.
23.
LachowiczM. J.SterbaS. K.PreacherK. J. (2015). Investigating multilevel mediation with fully or partially nested data. Group Processes & Intergroup Relations, 18, 274–289.
24.
LeeK. J.ThompsonS. G. (2005). The use of random effects models to allow for clustering in individually randomized trials. Clinical Trials, 2, 163–173.
25.
LohrS.SchochetP. Z.SandersE. (2014). Partially nested randomized controlled trials in education research: A guide to design and analysis (NCER 2014-2000). National Center for Education Research, Institute of Education Sciences, U.S. Department of Education. http://ies.ed.gov/
26.
LowrieH.HarrisD.LoganT.HegartyM. (2019). The impact of a spatial intervention program on students’ spatial reasoning and mathematics performance. The Journal of Experimental Education, 1–19.
27.
LüdtkeT.WestermannS.PultL.SchneiderB.PfuhlG.MoritzS. (2018). Evaluation of a brief unguided psychological online intervention for depression: A controlled trial including exploratory moderator analyses. Internet Interventions, 13, 73–81.
28.
MackinnonD.SoydanH.SundellK. (2011). Integrating mediators and moderators in research design. Research on Social Work Practice, 21(6), 675–681.
29.
Mayer-DavisE.D’AntonioA.SmithS.KirknerG.MartinS.Parra-MedinaD.SchultzR. (2004). Pounds off with empowerment (POWER): A clinical trial of weight management strategies for Black and White adults with diabetes who live in medically underserved rural communities. The American Journal of Public Health, 94(10), 1736–17342.
30.
MoerbeekM.WongW. (2008). Sample size formulae for trials com- paring group and individual treatments in a multilevel model. Statistics in Medicine, 27, 2850–2864.
31.
MorrellC.WaltersS.DixonS.CollinsK.BreretonL.PetersJ.BrookerC. (1998). Cost effectiveness of community leg ulcer clinics: Randomised controlled trial. British Medical Association, 316(7143), 1487–1491.
32.
RaudenbushS. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185.
33.
RaudenbushS. W.MartinezA.SpybrookJ. (2007). Strategies for improving precision in group randomized experiments. Educational Evaluation and Policy Analysis, 29, 5–29.
34.
RiveraR.SantosD.BrändleG.CárdabaM. (2016). Design effectiveness analysis of a media literacy intervention to reduce violent video games consumption among adolescents: The relevance of lifestyles segmentation. Evaluation Review, 40, 142–161.
35.
RobertsC.RobertsS. A. (2005). Design and analysis of clinical trials with clustering effects due to treatment. Clinical Trials, 2, 152–162.
36.
RobertsJ.WilliamsK.CarterM.EvansD.ParmenterT.SiloveN.ClarkT.WarrenA. (2011). A randomized controlled trial of two early intervention programs for young children with autism: Centre-based with parent program and home-based. Research in Autism Spectrum Disorders, 5, 1553–1566.
ScheierM.CarverC. (1992). Effects of optimism on psychological and physical well-being: Theoretical overview and empirical update. Cognitive Therapy and Research, 16(2), 201–228.
39.
SchochetP. Z. (2011). Do typical RCTs of education interventions have sufficient statistical power for linking impacts on teacher practice and student achievement outcomes?Journal of Educational and Behavioral Statistics, 36, 441–471.
40.
SchweigJ.PaneJ. (2016). Intention-to-treat analysis in partially nested randomized controlled trials with real-world complexity. International Journal of Research & Method in Education: Special Issue (PART 1): Is the Educational “What Works” Agenda Working? Critical Methodological Developments, 39(3), 268–286.
41.
ShackletonN.HaleD.BonellC.VinerR. (2016). Intraclass correlation values for adolescent health outcomes in secondary schools in 21 European countries. SSM—Population Health, 2(C), 217–225.
42.
SmithN.BrandonP.LawtonB.Krohn-ChingV. (2010). Evaluation exemplar: Exemplary aspects of a small group-randomized local educational program evaluation. American Journal of Evaluation, 31(2), 254–265.
SpybrookJ.KelceyB.DongN. (2016). Power for detecting treatment by moderator effects in two- and three-level cluster randomized trials. Journal of Educational and Behavioral Statistics, 41(6), 605–627.
Stone-RomeroE. F.LiakhovitskiD. (2002). Strategies for detecting moderator variables: A review of conceptual and empirical issues. Research in Personnel and Human Resources Management, 21, 333–372.
47.
Van NguyenH.HuangH.WongM.YangY.HuangT.TengC. (2018). Moderator roles of optimism and weight control on the impact of playing exergames on happiness: The perspective of social cognitive theory using a randomized controlled trial. Games for Health Journal, 7(4), 246–252.
48.
ZvochK.LetourneauL.ParkerR. (2007). A multilevel multisite outcomes-by-implementation evaluation of an early childhood literacy model. American Journal of Evaluation, 28(2), 132–150.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
