Random encouragement designs evaluate treatments that aim to increase participation in a program or activity. These randomized controlled trials (RCTs) can also assess the mediated effects of participation itself on longer term outcomes using a complier average causal effect (CACE) estimation framework. This article considers power analysis methods for such CACE analyses for a range of RCT designs, including nonclustered, clustered, and random block designs. The focus is on behavioral encouragements to promote action, such as text messaging, that are increasingly being tested in education trials. We derive asymptotic distributions of the CACE estimators using generalized estimating equations theory, which underlie the power formulas. We incorporate noncompliance from both the actual receipt of the encouragement and participation itself. An illustrative power analysis provides sample size guidance using an available Shiny R dashboard.
AbadieA. (2002). Bootstrap tests for distributional treatment effects in instrumental variable models. Journal of the American Statistical Association, 97(457), 284–292.
2.
AngristJ.ImbensG.RubinD. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–472.
3.
AngristJ. D.ImbensG. W. (1995). Two-stage least squares estimation of average causal effects in models with variable treatment intensity. Journal of the American Statistical Association, 90(430), 431–442. https://doi.org/10.1080/01621459.1995.10476535
4.
BallS.BogatzG. A. (1970). The first year of Sesame Street: An evaluation. Educational Testing Service.
5.
BallS.BogatzG. A. (1971). The second year of Sesame Street: An evaluation. Educational Testing Service.
6.
BansakK. (2020). A generalized approach to power analysis for local average treatment effects. Statistical Science, 35(2), 254–271.
7.
BergmanP.ChanE. W. (2021). Leveraging parents through low-cost technology: The impact of high-frequency information on student achievement. Journal of Human Resources, 56(1), 125–158.
8.
BirdK. A.CastlemanB. L.DenningJ. T.GoodmanJ.LambertonC.RosingerK. O. (2021). Nudging at scale: Experimental evidence from FAFSA completion campaigns. Journal of Economic Behavior and Organization, 183, 105–128.
9.
BlandholC.BonneyJ.MogstadM.TorgovitskyA. (2022). When is TSLS actually LATE?Working Paper 29709. National Bureau of Economic Research, Cambridge, MA.
10.
BloomH. S. (1995). Minimum detectable effects: A simple way to report the statistical power of experimental designs. Evaluation Review, 19(5), 547–556.
11.
BrionM. J. A.ShakhbazovK.VisscherP. M. (2013). Calculating statistical power in Mendelian randomization studies. International Journal of Epidemiology, 42(5), 1497–1501.
12.
CameronA. C.MillerD. L. (2015). A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50(2), 317–372.
13.
CastlemanB. L.HaskinsR.AkersB.BaronJ.DynarskiS.FarranD.FeldmanA.JonesD.KeyesB.MaynardR.SirinidesP.ZinmanJ. (2017). Behavioral science and policy: Implications for education. Behavioral Science and Policy, 3(1), 43–50.
14.
CochranW. G. (1977). Sampling techniques (3rd ed.). John Wiley.
15.
DonnerA.KlarN. (2000). Design and analysis of cluster randomization trials in health research. Arnold.
16.
DufloE.GlennersterR.KremerM. (2007). Using randomization in development economics research: A toolkit. Technical Report. Centre for Economic Policy Research, London.
17.
DziakJ. J.DierkerL. C.AbarB. (2020). The interpretation of statistical power after the data have been gathered. Current Psychology, 39(3), 870–877.
18.
FreemanG.CowlingB. J.SchoolingC. M. (2013). Power and sample size calculations for Mendelian randomization studies using one genetic instrument. International Journal of Epidemiology, 42(4), 1157–1163.
19.
HansenB.LeeS. (2019). Asymptotic theory for clustered samples. Journal of Econometrics, 210(2), 268–290.
20.
HedbergE.SantanaR.HedgesL. (2004). The variance structure of academic achievement in America. Working Paper, University of Chicago.
21.
HillJ.BloomH.BlackR.LipseyM. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177.
22.
HoenigJ. M.HeiseyD. M. (2001). The abuse of power: The pervasive fallacy of power calculations in data analysis. The American Statistician, 55(1), 19–24.
23.
HummelD.MaedcheA. (2019). How effective is nudging? A quantitative review on the effect sizes and limits of empirical nudging studies. Journal of Behavioral and Experimental Economics, 80, 47–58.
24.
ImaiK.TingleyD.YamamotoT. (2013). Experimental designs for identifying causal mechanisms. Journal of the Royal Statistical Society Series A, 176(1), 5–32.
25.
ImbensG.AngristJ. (1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467–476.
26.
ImbensG.RubinD. (2015). Causal inference for statistics in social, and biomedical sciences: An introduction. Cambridge University Press.
27.
JennisonC.TurnbullB. W. (2000). Group sequential methods with applications to clinical trials. Chapman & Hall/CRC.
28.
KishL. (1995). Survey sampling. John Wiley.
29.
LaiT. L.LavoriP. W.ShihM. C. (2012). Adaptive trial designs. Annual Review of Pharmacology Toxicology, 52, 101–110.
30.
LiangK.ZegerS. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73(1), 13–22.
31.
LipseyM. W.YunP. K.HebertM. A.Steinka-FryK.ColeM. W.RobertsM.AnthonyK. S.BusickM. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. NCSER 2013-3000, Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education Research.
32.
MertensS.HerberzM.HahnelU. J. J.BroschT. (2022). The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains. Proceedings of the National Academy of Sciences USA, 119(1), e2107346118.
33.
MurrayD. (1998). Design and analysis of group-randomized trials. Oxford University Press.
34.
OttoboniK. N.PoulosJ. V. (2020). Estimating population average treatment effects from experiments with noncompliance. Journal of Causal Inference, 8(1), 108–130.
35.
RubinD. B. (1986). Which ifs have causal answers? Discussion of Holland’s “Statistics and causal inference.”Journal of the American Statistical Association, 81(396), 961–962.
36.
SchochetP. Z. (2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1), 62–87.
37.
SchochetP. Z. (2011). Do typical RCTs of education interventions have sufficient statistical power for linking impacts on teacher and student outcomes?Journal of Educational and Behavioral Statistics, 36(4), 441–471.
38.
SchochetP. Z. (2013). Statistical power for education RCTs with binary outcomes. Journal of Research on Educational Effectiveness, 6(3), 263–294.
39.
SchochetP. Z. (2016). Design-based causal inference for RCTs: Statistical theory for the RCT-YES software. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Research.
40.
SchochetP. Z.ChiangH. (2011). Estimation and identification of the complier average causal effect parameter in education RCTs. Journal of Educational and Behavioral Statistics, 36(3), 307–345.
41.
SchochetP. Z.PashleyN. E.MiratrixL. W.KautzT. (2021). Design-based ratio estimators and central limit theorems for clustered, blocked RCTs. Journal of the American Statistical Association, 117(540), 2135–2146. https://doi.org/10.1080/01621459.2021.1906685
42.
SobelM. (2008). Identification of causal parameters in randomized studies with mediating variables. Journal of Educational and Behavioral Statistics, 33(2), 230–251.
43.
SommerA.ZegerS. (1991). On estimating efficacy in clinical trials. Statistics in Medicine, 10(1), 45–52.
44.
StockJ.YogoM. (2005). Testing for weak instruments in linear IV regression. In AndrewsD. W. K. (Ed.), Identification and inference for econometric models: Essays in honor of Thomas Rothenberg (pp. 80–108). Cambridge University Press.
45.
ThalerR. H.SunsteinC. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.
46.
WangX.JiangY.ZhangN. R.SmallD. S. (2018). Sensitivity analysis and power for instrumental variable studies. Biometrics, 74(4), 1150–1160.
47.
What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance (NCEE). https://ies.ed.gov/ncee/wwc/Handbooks
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
