Sequential, multiple assignment randomized trials (SMARTs), which assist in the optimization of adaptive interventions, are growing in popularity in education and behavioral sciences. This is unsurprising, as adaptive interventions reflect the sequential, tailored nature of learning in a classroom or school. Nonetheless, as is true elsewhere in education research, observed effect sizes in education-based SMARTs are frequently small. As a consequence, statistical efficiency is of paramount importance in their analysis. The contributions of this manuscript are twofold. First, we provide an overview of adaptive interventions and SMART designs for researchers in education science. Second, we propose four techniques that have the potential to improve statistical efficiency in the analysis of SMARTs. We demonstrate the benefits of these techniques in SMART settings both through the analysis of a SMART designed to optimize an adaptive intervention for increasing cognitive behavioral therapy delivery in school settings and through a comprehensive simulation study. Each of the proposed techniques is easily implementable, either with over-the-counter statistical software or through R code provided in Supplemental Material.
AlmirallD.GriffinB. A.McCaffreyD. F.RamchandR.YuenR. A.MurphyS. A. (2014). Time-varying effect moderation using the structural nested mean model: Estimation using inverse-weighted regression with residuals. Statistics in Medicine, 33(20), 3466–3487.
2.
AlmirallD.KasariC.McCaffreyD. F.Nahum-ShaniI. (2018a). Developing optimized adaptive interventions in education. Journal of Research on Educational Effectiveness, 11(1), 27–34.
3.
AlmirallD.Nahum-ShaniI.WangL.KasariC. (2018b). Experimental designs for research on adaptive interventions: Singly and sequentially randomized trials. In CollinsL.KuglerK. (Eds.), Optimization of Behavioral, Biobehavioral, and Biomedical Interventions (pp. 89–120). Statistics for Social and Behavioral Sciences.
4.
ArendaleD. R. (1994). Understanding the supplemental instruction model. Josey-Bass.
5.
BallingerG. A. (2004). Using generalized estimating equations for longitudinal data analysis. Organizational Research Methods, 7(2), 127–150.
6.
BergdahlN. (2022). Adaptive professional development during the pandemic. Designs for Learning, 14(1), 1–13.
7.
BloomH. S.Richburg-HayesL.BlackA. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59.
8.
BrumbackB. A. (2009). A note on using the estimated versus the known propensity score to estimate the average treatment effect. Statistics & Probability Letters, 79(4), 537–542.
9.
CharvatJ. (2012). Research on the relationship between mental health and academic achievement. National Association of School Psychologists.
10.
ColeS. R.HernánM. A. (2008). Constructing inverse probability weights for marginal structural models. American Journal of Epidemiology, 168(6), 656–664.
11.
ColeS. R.PlattR. W.SchistermanE. F.ChuH.WestreichD.RichardsonD.PooleC. (2010). Illustrating bias due to conditioning on a collider. International Journal of Epidemiology, 39(2), 417–420.
12.
CollinsL. M.MurphyS. A.StrecherV. (2007). The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent ehealth interventions. American Journal of Preventive Medicine, 32(5), S112–S118.
13.
ElwertF.WinshipC. (2014). Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology, 40, 31.
14.
FleuryV. P.TowsonJ. A. (2021). Early lessons learned in designing an adaptive shared reading intervention for preschoolers with autism. Exceptional Children, 88(1), 45–64.
15.
FuchsL. S.FuchsD.ZumetaR. O. (2008). Response to intervention. In GrigorenkoE. L. (Ed.), Educating individuals with disabilities: IDEIA 2004 and beyond (p. 115). Springer
16.
HeppenJ. B.KurkiA.BrownS. (2020). Can texting parents improve attendance in elementary school? A test of an adaptive messaging strategy. Appendix. NCEE 2020-006a. National Center for Education Evaluation and Regional Assistance.
17.
HernánM. A.BrumbackB. A.RobinsJ. M. (2002). Estimating the causal effect of zidovudine on CD4 count with a marginal structural model for repeated measures. Statistics in Medicine, 21(12), 1689–1709.
18.
HernánM. A.Hernández-DazS.RobinsJ. M. (2004). A structural approach to selection bias. Epidemiology, 15, 615–625.
19.
HiranoK.ImbensG. W.RidderG. (2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71(4), 1161–1189.
20.
HollandP. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.
21.
KilbourneA. M.SmithS. N.ChoiS. Y.KoschmannE.LiebrechtC.RuschA.AbelsonJ. L.EisenbergD.HimleJ. A.FitzgeraldK.AlmirallD. (2018). Adaptive school-based implementation of CBT (ASIC): Clustered-SMART for building an optimized adaptive implementation intervention to improve uptake of mental health interventions in schools. Implementation Science, 13(1), 1–15.
22.
KimJ. S.AsherC. A.BurkhauserM.MesiteL.LeyvaD. (2019). Using a sequential multiple assignment randomized trial (SMART) to develop an adaptive K–2 literacy intervention with personalized print texts and app-based digital activities. AERA Open, 5(3), 1–18.
23.
KraftM. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253.
LiangK.-Y.ZegerS. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73(1), 13–22.
26.
LuX.Nahum-ShaniI.KasariC.LynchK. G.OslinD. W.PelhamW. E.FabianoG.AlmirallD. (2016). Comparing dynamic treatment regimes using repeated-measures outcomes: Modeling considerations in SMART studies. Statistics in Medicine, 35(10), 1595–1615.
27.
MartiniR.HiltR.MarxL.ChenvenM.NaylorM.SarvetB.PtakowskiK. (2012). Best principles for integration of child psychiatry into the pediatric health home. American Academy of Child & Adolescent Psychiatry.
28.
MaybinJ.MercerN.StiererB. (1992). Scaffolding learning in the classroom. Thinking Voices: The Work of the National Oracy Project, 186, 195.
29.
MurphyS. A. (2005). An experimental design for the development of adaptive treatment strategies. Statistics in Medicine, 24, 1455–1481.
30.
MurphyS. A.BinghamD. (2009). Screening experiments for developing dynamic treatment regimes. Journal of the American Statistical Association, 104(485), 391–408.
31.
MurphyS. A.LynchK. G.OslinD.McKayJ. R.TenHaveT. (2007). Developing adaptive treatment strategies in substance abuse research. Drug and Alcohol Dependence, 88, S24–S30.
32.
MurphyS. A.van der LaanM. J.RobinsJ. M.GroupC. P. P. R. (2001). Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96(456), 1410–1423.
33.
Nahum-ShaniI.QianM.AlmirallD.PelhamW. E.GnagyB.FabianoG. A.WaxmonskyJ. G.YuJ.MurphyS. A. (2012). Experimental design and primary data analysis methods for comparing adaptive interventions. Psychological Methods, 17(4), 457.
34.
NeCampT.KilbourneA.AlmirallD. (2017). Comparing cluster-level dynamic treatment regimens using sequential, multiple assignment, randomized trials: Regression estimation and sample size considerations. Statistical Methods in Medical Research, 26(4), 1572–1589.
35.
OettingA. I.LevyJ. A. (2007). Statistical methodology for a SMART design in the development of adaptive treatment strategies. In ShroutP.KeyesK.OrnsteinK. (Eds.), Causality and psychopathology (pp. 179–205). Oxford University Press.
36.
OrellanaL.RotnitzkyA.RobinsJ. M. (2010). Dynamic regime marginal structural mean models for estimation of optimal dynamic treatment regimes, Part I: Main content. The International Journal of Biostatistics, 6(2), 1–47.
37.
Pelham JrW. E.FabianoG. A.WaxmonskyJ. G.GreinerA. R.GnagyE. M.Pelham IIIW. E.CoxeS.VerleyJ.BhatiaI.HartK.KarchK.KonijnendijkE.TrescoK.Nahum-ShaniI.MurphyS. A. (2016). Treatment sequencing for childhood adhd: A multiple-randomization study of adaptive medication and behavioral interventions. Journal of Clinical Child & Adolescent Psychology, 45(4), 396–415.
38.
RaudenbushS. W. (2001). Comparing personal trajectories and drawing causal inferences from longitudinal data. Annual Review of Psychology, 52, 501.
39.
RaudenbushS. W. (2008). Advancing educational policy by advancing research on instruction. American Educational Research Journal, 45(1), 206–230.
40.
RaudenbushS. W.HernandezM.Goldin-MeadowS.CarrazzaC.FoleyA.LeslieD.SorkinJ. E.LevineS. C. (2020). Longitudinally adaptive assessment and instruction increase numerical skills of preschool children. Proceedings of the National Academy of Sciences, 117(45), 27945–27953.
41.
RobertsG.ClemensN.DoablerC. T.VaughnS.AlmirallD.Nahum-ShaniI. (2021). Multitiered systems of support, adaptive interventions, and SMART designs. Exceptional Children, 88(1), 8–25.
42.
RobinsJ. (1986). A new approach to causal inference in mortality studies with a sustained exposure period—Application to control of the healthy worker survivor effect. Mathematical Modelling, 7(9–12), 1393–1512.
43.
RobinsJ. M. (1994). Correcting for non-compliance in randomized trials using structural nested mean models. Communications in Statistics-Theory and Methods, 23(8), 2379–2412.
44.
RobinsJ. M.RotnitzkyA. (1995). Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429), 122–129.
45.
RowanB.HansenB. B.WhiteM.LycurgusT.ScottL. J. (2019). A summary of the BURST [R]: Reading efficacy trial. Institute for Social Research.
46.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688.
SeewaldN. J.KidwellK. M.Nahum-ShaniI.WuT.McKayJ. R.AlmirallD. (2020). Sample size considerations for comparing dynamic treatment regimens in a sequential multiple-assignment randomized trial with a continuous longitudinal outcome. Statistical Methods in Medical Research, 29(7), 1891–1912.
49.
TchetgenE. J. T.GlymourM. M.WeuveJ.RobinsJ. (2012). Specifying the correlation structure in inverse-probability-weighting estimation for repeated measures. Epidemiology, 23(4), 644–646.
50.
VansteelandtS. (2007). On confounding, prediction and efficiency in the analysis of longitudinal and cross-sectional clustered data. Scandinavian Journal of Statistics, 34(3), 478–498.
51.
WilliamsonE. J.ForbesA.WhiteI. R. (2014). Variance reduction in randomised trials by inverse probability weighting using the propensity score. Statistics in Medicine, 33(5), 721–737.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
