There is wide agreement on the importance of implementation data from randomized effectiveness studies in behavioral science; however, there are few methods available to incorporate these data into causal models, especially when they are multivariate or longitudinal, and interest is in low-dimensional summaries. We introduce a framework for studying how treatment effects vary between subjects who implement an intervention differently, combining principal stratification with latent variable measurement models; since principal strata are latent in both treatment arms, we call it “fully latent principal stratification” (FLPS). We describe FLPS models including item-response-theory measurement, show that they are feasible in a simulation study, and illustrate them in an analysis of hint usage from a randomized study of computerized mathematics tutors.
AngristJ. D.ImbensG. W.RubinD. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455.
2.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397–424). Addison-Wesley.
3.
CarrollC.PattersonM.WoodS.BoothA.RickJ.BalainS. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 1–9.
4.
CommentL.MealliF.HaneuseS.ZiglerC. (2019). Survivor average causal effects for continuous time: A principal stratification approach to causal inference with semicompeting risks. arXiv preprint arXiv:1902.09304.
5.
De BoeckP.WilsonM. (2013). Explanatory item response models: A generalized linear and nonlinear approach. Springer Science & Business Media.
6.
Decker-WoodrowL. E.MasonC. A.LeeJ.-E.ChanJ. Y.-C.SalesA.LiuA.TuS. (2023). The impacts of three educational technologies on algebraic understanding in the context of covid-19. AERA Open, 9, 23328584231165919.
7.
DingP.LuJ. (2017). Principal stratification analysis using principal scores. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79(3), 757–777.
8.
FellerA.GrindalT.MiratrixL.PageL. C. (2016). Compared to what? Variation in the impacts of early childhood education by alternative care type. The Annals of Applied Statistics, 10(3), 1245–1285.
9.
FellerA.MealliF.MiratrixL. (2017). Principal score methods: Assumptions, extensions, and practical considerations. Journal of Educational and Behavioral Statistics, 42(6), 726–758.
10.
FrangakisC. E.RubinD. B. (2002). Principal stratification in causal inference. Biometrics, 58 (1), 21–29.
11.
FyfeE. R.de LeeuwJ. R.CarvalhoP. F.GoldstoneR. L.ShermanJ.AdmiraalD.AlfordL. K.BonnerA.BrassilC. E.BrooksC. A.CarbonettoT.ChangS. H.CruzL.Czymoniewicz-KlippelM.DanielF.DriessenM.HabashyN.Hanson-BradleyC. L.HirtE. R.… MotzB. A. (2021). Manyclasses 1: Assessing the generalizable effect of immediate feedback versus delayed feedback across many college classes. Advances in Methods and Practices in Psychological Science, 4(3), 25152459211027575.
12.
GallopR.SmallD. S.LinJ. Y.ElliottM. R.JoffeM.Ten HaveT. R. (2009). Mediation analysis with principal stratification. Statistics in Medicine, 28(7), 1108–1130.
13.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472.
14.
GoodsonB.PriceC.DarrowC. (2014). Measuring fidelity: The present and future. In MyersB.BrandW. C. (Eds.), Implementation fidelity in education research (pp. 190–207). Routledge.
15.
GriffinB. A.McCafferyD. F.MorralA. R. (2008). An application of principal stratification to control for institutionalization at follow-up in studies of substance abuse treatment programs. The Annals of Applied Statistics, 2(3), 1034.
16.
HoN.FellerA.GreifE.MiratrixL.PillaiN. (2022). Weak separation in mixture models and implications for principal stratification. In International conference on artificial intelligence and statistics (pp. 5416–5458). PMLR.
17.
HoffmanM. D.GelmanA. (2014). The no-u-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593–1623.
ImbensG. W.RubinD. B. (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge University Press.
20.
JiangZ.DingP. (2021). Identification of causal effects within principal strata using auxiliary variables. Statistical Science, 36(4), 493–508.
21.
JinH.RubinD. B. (2008). Principal stratification for causal inference with extended partial compliance. Journal of the American Statistical Association, 103(481), 101–111.
22.
KeoghR. H.ShawP. A.GustafsonP.CarrollR. J.DeffnerV.DoddK. W.KüchenhoffH.ToozeJ. A.WallaceM. P.KipnisV.FreedmanL. S. (2020). Stratos guidance document on measurement error and misclassification of variables in observational epidemiology: Part 1—Basic theory and simple methods of adjustment. Statistics in Medicine, 39(16), 2197–2231.
23.
KlineR. B. (2023). Principles and practice of structural equation modeling. Guilford Publications.
24.
LiS.ZhuY.EllisR. (2016). The effects of the timing of corrective feedback on the acquisition of a new linguistic structure. The Modern Language Journal, 100(1), 276–295.
25.
LiY.TaylorJ. M.ElliottM. R. (2010, 06). A Bayesian approach to surrogacy assessment using principal stratification in clinical trials. Biometrics, 66(2), 523–531.
26.
LyuT.BornkampB.Mueller-VeltenG.SchmidliH. (2023). Bayesian inference for a principal stratum estimand on recurrent events truncated by death. Biometrics, 79(4), 3792–3802.
27.
MastersG. N. (2016). Partial credit model. In van der LindenW. J. (Ed.), Handbook of item response theory, volume one (pp. 137–154). Chapman and Hall/CRC.
28.
MatteiA.LiF.MealliF. (2013). Exploiting multiple outcomes in Bayesian principal stratification analysis with application to the evaluation of a job training program. The Annals of Applied Statistics, 7(4), 2336–2360.
29.
MealliF.MatteiA. (2012). A refreshing account of principal stratification. The International Journal of Biostatistics, 8(1). Article 8.
30.
MurakiE. (1997). A generalized partial credit model. In van der LindenW. J.HambletonR. K. (Eds.), Handbook of modern item response theory (pp. 153–164). Springer.
31.
MuthénB. O. (2002). Beyond sem: General latent variable modeling. Behaviormetrika, 29(1), 81–117.
NeymanJ. (1923). Sur les applications de la théorie des probabilités aux experiences agricoles: Essai des principes [On the application of probability theory to agricultural experiments: Essay on principles]. Roczniki Nauk Rolniczych, 10, 1–51.
34.
PageL. C. (2012). Principal stratification as a framework for investigating mediational processes in experimental settings. Journal of Research on Educational Effectiveness, 5(3), 215–244.
35.
PageL. C.FellerA.GrindalT.MiratrixL.SomersM.-A. (2015). Principal stratification: A tool for understanding variation in program effects across endogenous subgroups. American Journal of Evaluation, 36(4), 514–531.
36.
RaschG. (1960). Studies in mathematical psychology: I. probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.
37.
R Core Team. (2016). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. https://www.R-project.org/
38.
RichardsonS.GilksW. R. (1993). A Bayesian approach to measurement error problems in epidemiology using conditional independence models. American Journal of Epidemiology, 138(6), 430–442.
39.
RoyJ.HoganJ. W.MarcusB. H. (2007). Principal stratification with predictors of compliance for randomized trials with 2 active treatments. Biostatistics, 9(2), 277–289.
40.
RubinD. B. (1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, 6, 34–58.
41.
SalesA. C.PaneJ. F. (2019). The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable. The Annals of Applied Statistics, 13(1), 420–443.
42.
SalesA. C.PaneJ. F. (2021). Student log-data from a randomized evaluation of educational technology: A causal case study. Journal of Research on Educational Effectiveness, 14(1), 241–269.
43.
SalesA. C.VanacoreK. P.KangH.-A.WhittakerT. A. (2024, July). Problem-solving behavior and EdTech effectiveness: A model for exploratory causal analysis. In PaaßenB.EppC. D. (Eds.), Proceedings of the 17th international conference on educational data mining (pp. 385–395). International Educational Data Mining Society.
44.
SalesA. C.VanacoreK. P.OttmarE. R. (2022). Geepers: Principal stratification using principal scores and stacked estimating equations. arXiv preprint arXiv:2212.10406. https://doi.org/10.48550/arXiv.2212.10406
45.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34(4), 100.
46.
SchoenwaldS. K.GarlandA. F.ChapmanJ. E.FrazierS. L.SheidowA. J.Southam-GerowM. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43.
47.
ShawP. A.GustafsonP.CarrollR. J.DeffnerV.DoddK. W.KeoghR. H.KipnisV.ToozeJ. A.WallaceM. P.KüchenhoffH.FreedmanL. S. (2020). Stratos guidance document on measurement error and misclassification of variables in observational epidemiology: Part 2—More complex methods of adjustment and advanced topics. Statistics in Medicine, 39(16), 2232–2263.
48.
SmithH.OttmarE.NgoV.ClosserA. H.ChanJ. Y.-C.VanacoreK.SalesA. C. (2024). Adding to the debate: Does immediate or delayed feedback make a difference? (Paper in Preparation).
49.
Stan Development Team. (2016a). RStan: The R interface to Stan (R package version 2.14.1). http://mc-stan.org/.
50.
Stan Development Team. (2016b). RStan: The R interface to Stan (R package version 2.17.3). https://mc-stan.org/
51.
StekhovenD. J.BühlmannP. (2012). Missforest—Non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1), 112–118.
52.
VanacoreK.GurungA.SalesA.HeffernanN. T. (2024). The effect of assistance on gamers: Assessing the impact of on-demand hints & feedback availability on learning for students who game the system. In Proceedings of the 14th learning analytics and knowledge conference. Kyoto, Japan, pp. 462–472. Association for Computing Machinery.
53.
VanacoreK.OttmarE.LiuA.SalesA. C. (2023). Remote monitoring of implementation fidelity using log-file data from multiple online learning platforms. PsyArXiv. psyarxiv.com/7ru2x
54.
VanacoreK.SalesA.LiuA.OttmarE. (2023). Benefit of gamification for persistent learners: Propensity to replay problems moderates algebra-game effectiveness. In L@S ’23: Proceedings of the Tenth ACM Conference on Learning @ Scale. Copenhagen, Denmark, pp. 164–173. Association for Computing Machinery.
55.
VanderWeeleT. J.HernanM. A. (2013). Causal inference under multiple versions of treatment. Journal of Causal Inference, 1(1), 1–20.
56.
VanderWeeleT. J.LiY. (2019). Simple sensitivity analysis for differential measurement error. American Journal of Epidemiology, 188(10), 1823–1829.
57.
WrightB. D. (1977). Solving measurement problems with the Rasch Model. Journal of Educational Measurement, 14, 97–116.
58.
ZhangJ. L.RubinD. B.MealliF. (2009). Likelihood-based analysis of causal effects of job-training programs using principal stratification. Journal of the American Statistical Association, 104(485), 166–176.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
