We consider estimating the effect of a treatment on a given outcome measured on subjects tested both before and after treatment assignment in observational studies. A vast literature compares the competing approaches of modelling the post-test score conditionally on the pre-test score versus modelling the difference, namely, the gain score. Our contribution lies in analyzing the merits and drawbacks of two approaches in a multilevel setting. This is relevant in many fields, such as education, where students are nested within schools. The multilevel structure raises peculiar issues related to contextual effects and the distinction between individual-level and cluster-level treatments. We compare the two approaches through a simulation study. For individual-level treatments, our findings align with existing literature. However, for cluster-level treatments, the scenario is more complex, as the cluster mean of the pre-test score plays a key role. Its reliability crucially depends on the cluster size, leading to potentially unsatisfactory estimators with small clusters.
AhernJ. (2018). Start with the “C-word,” follow the roadmap for causal inference. American Journal of Public Health, 108(5), 621. https://doi.org/10.2105/AJPH.2018.304358
2.
AllisonP. D. (1990). Change scores as dependent variables in regression analysis. Sociological Methodology, 20, 93–114. https://doi.org/10.2307/271083.
3.
ArpinoB.AassveA. (2013). Estimating the causal effect of fertility on economic well-being: Data requirements, identifying assumptions and estimation methods. Empirical Economics, 44(1), 355–385. https://doi.org/10.1007/s00181-010-0356-9
4.
ArpinoB.MatteiA. (2016). Assessing the causal effects of financial aids to firms in tuscany allowing for interference. Annals of Applied Statistics, 10(3), 1170–1194. https://doi.org/10.1214/15-aoas902
5.
ArpinoB.MealliF. (2011). The specification of the propensity score in multilevel observational studies. Computational Statistics & Data Analysis, 55(4), 1770–1780. https://doi.org/10.1016/j.csda.2010.11.008
6.
BaiocchiM.ChengJ.SmallD. S. (2014). Instrumental variable methods for causal inference. Statistics in Medicine, 33(13), 2297–2340. https://doi.org/10.1002/sim.6128
7.
BereiterC. (1963). Some persisting dilemmas in the measurement of change. In HarrisC. W. (Ed.), Problems in measuring change (pp. 3–20). University of Wisconsin Press.
8.
CallawayB.Goodman-BaconA.Sant’AnnaP. H. (2021). Difference-in-differences with a continuous treatment.arXiv preprint arXiv:2107.02637.
9.
CardoneM.FalzettiP.SaccoC. (2019). INVALSI data for school system improvement: The value added. Working papers INVALSI, n. 5694, Invalsi.
10.
CarrilloP. E.OnofaM.PonceJ. (2011). Information technology and student achievement: Evidence from a randomized experiment in Ecuador. Research Department Publications 4698, Inter-American Development Bank, Research Department.
11.
CroonM. A.van VeldhovenM. J. P. M. (2007). Predicting group-level outcome variables from variables measured at the individual level: A latent variable multilevel model. Psychological Methods, 12(1), 45–57. https://doi.org/10.1037/1082-989X.12.1.45
12.
DingP.LiF. (2019). A bracketing relationship between difference-in-differences and lagged-dependent-variable adjustment. Political Analysis, 27(4), 605–615. https://doi.org/10.1017/pan.2019.25
13.
ForastiereL.AiroldiE. M.MealliF. (2021). Identification and estimation of treatment and interference effects in observational studies on networks. Journal of the American Statistical Association, 116(534), 901–918. https://doi.org/10.1080/01621459.2020.1768100
14.
GrilliL.RampichiniC. (2011). The role of sample cluster means in multilevel models: A view on endogeneity and measurement error issues. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 7(4), 121–133. https://doi.org/10.1027/1614-2241/a000030
15.
HaY.ParkH. J. (2017). Can after-school programs and private tutoring help improve students’ achievement? Revisiting the effects in Korean secondary schools. Asia Pacific Education Review, 18(1), 65–79. https://doi.org/10.1007/s12564-016-9451-8
16.
HongG.RaudenbushS. W. (2006). Evaluating kindergarten retention policy: A case study of causal inference for multilevel observational data. Journal of the American Statistical Association, 101(475), 901–910. https://doi.org/10.1198/016214506000000447
ImbensG. W.RubinD. W. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press.
19.
KimY.SteinerP. M. (2021a). Gain scores revisited: A graphical models perspective. Sociological Methods & Research, 50(3), 1353–1375. https://doi.org/10.1177/0049124119826155
20.
KimY.SteinerP. M. (2021b). Causal graphical views of fixed effects and random effects models. British Journal of Mathematical and Statistical Psychology, 74(2), 165–183. https://doi.org/10.1111/bmsp.12217
21.
KlarN.DarlingtonG. (2004). Methods for modelling change in cluster randomization trial. Statistics in Medicine, 23(15), 2341–2357. https://doi.org/10.1002/sim.1858
22.
KöhlerC.HartigJ.SchmidC. (2020). Deciding between the covariance analytical approach and the change-score approach in two wave panel data. Multivariate Behavioral Research, 56(3), 447–458. https://doi.org/10.1080/00273171.2020.1726723
23.
LechnerM. (2011). The estimation of causal effects by difference-in-difference methods. Foundations and Trends in Econometrics, 4(3), 165–224. https://doi.org/10.1561/0800000014
24.
LordF. M. (1963). Elementary models for measuring change. In HarrisC. W. (Ed.), Problems in measuring change. University of Wisconsin Press.
25.
LordF. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bullettin, 68(5), 304–305. https://doi.org/10.1037/h0025105
26.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Addison-Wesley.
MarshH. W.LüdtkeO.RobitzschA.TrautweinU.AsparouhovT.MuthenB.NagengastB. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44(6), 764–802. https://doi.org/10.1080/00273170903333665
29.
RaschG. (1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.
30.
RaudenbushS. W. (2005). Learning from attempts to improve schooling: The contribution of methological diversity. Educational Researcher, 34(5), 25–31. https://doi.org/10.3102/0013189x034005025
31.
RaudenbushS. W.SchwartzD. (2020). Randomized experiments in education, with implications for multilevel causal inference. Annual Review of Statistics and Its Application, 7(1), 177–208. https://doi.org/10.1146/annurev-statistics-031219-041205
32.
RosenbaumP. R. (2002). Observational studies (2nd ed.). Springer-Verlag.
33.
SAS Institute Inc. (2018). SAS/STAT 15.1 User‘s guide. SAS Institute Inc.
34.
SnijdersT.BoskerR. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Sage Publications Ltd.
35.
SteinerP. M.KimY. (2016). The mechanics of omitted variable bias: Bias amplification and cancellation of offsetting biases. Journal of Causal Inference, 4(2), 20160009. https://doi.org/10.1515/jci-2016-0009
36.
StiglerS. M. (1997). Regression towards the mean, historically considered. Statistical Methods in Medical Research, 6(2), 103–114. https://doi.org/10.1177/096228029700600202
37.
ThoemmesF. J.WestS. G. (2011). The use of propensity scores for nonrandomized designs with clustered data. Multivariate Behavioral Research, 46(3), 514–543. https://doi.org/10.1080/00273171.2011.569395
38.
van BreukelenG. J. P. (2013). ANCOVA versus CHANGE from baseline in nonrandomized studies: The difference. Multivariate Behavioral Research, 48(6), 895–922. https://doi.org/10.1080/00273171.2013.831743
39.
WingC.SimonK.Bello-GomezR. A. (2018). Designing difference in difference studies: Best practices for public health policy research. Annual Review of Public Health, 39, 453–469. https://doi.org/10.1146/annurev-publhealth-040617-013507
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
