Evaluating blocked randomized experiments from a potential outcomes perspective has two primary branches of work. The first focuses on larger blocks, with multiple treatment and control units in each block. The second focuses on matched pairs, with a single treatment and control unit in each block. These literatures not only provide different estimators for the standard errors of the estimated average impact, but they are also built on different sets of assumptions. Neither literature handles cases with blocks of varying size that contain singleton treatment or control units, a case which can occur in a variety of contexts, such as with different forms of matching or poststratification. In this article, we reconcile the literatures by carefully examining the performance of variance estimators under several different frameworks. We then use these insights to derive novel variance estimators for experiments containing blocks of different sizes.
AbadieA.ImbensG. W. (2008). Estimation of the conditional variance in paired experiments. Annales d’Économie et de Statistique, 91/92, 175–187.
2.
AronowP. M.GreenD. P.LeeD. K. (2014). Sharp bounds on the variance in randomized experiments. The Annals of Statistics, 42(3), 850–871.
3.
Centers for Disease Control and Prevention (CDC). National Center for Health Statistics (NCHS). (2013–2014). National health and nutrition examination survey data. U.S. Department of Health and Human Services, CDC.
4.
CochranW. G. (1953). Matching in analytical studies. American Journal of Public Health and the Nations Health, 43(6_Pt_1), 684–691.
5.
CochranW. G. (1977). Sampling techniques. Wiley series in probability and mathematical statistics-applied (3rd ed.). John Wiley & Sons.
6.
CochranW. G.CoxG. M. (1950). Experimental designs. John Wiley & Sons.
7.
DingP.LiX.MiratrixL. W. (2017). Bridging finite and super population causal inference. Journal of Causal Inference, 5(2), 1–8. https://doi.org/10.1515/jci-2016-0027
8.
FisherR. A. (1926). The arrangement of field experiments. Journal of Ministry of Agriculture, 33, 503–513.
9.
FogartyC. B. (2018). On mitigating the analytical limitations of finely stratified experiments. Journal of the Royal Statistical Society. Series B, 80(5), 1035–1056.
10.
GerberA. S.GreenD. P. (2012). Field experiments: Design, analysis and interpretation. Norton.
11.
HansenB. B. (2004). Full matching in an observational study of coaching for the SAT. Journal of the American Statistical Association, 99(467), 609–618.
12.
HansenB. B.KlopferS. O. (2006). Optimal full matching and related designs via network flows. Journal of the American Statistical Association, 15(3), 609–627.
13.
HinkleyD. V. (1977). Jackknifing in unbalanced situations. Technometrics, 19(3), 285–292.
14.
IacusS. M.KingG.PorroG. (2012). Causal inference without balance checking: Coarsened exact matching. Political Analysis, 20(1), 1–24.
ImaiK. (2008). Variance identification and efficiency analysis in randomized experiments under the matched-pair design. Statistics in Medicine, 27(24), 4857–4873.
17.
ImaiK.KingG.StuartE. A. (2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A, 171(2), 481–502.
18.
ImbensG. W. (2011). Experimental design for unit and cluster randomid trials. Conference International Initiative for Impact Evaluation, Cuernavaca.
19.
ImbensG. W.RubinD. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press.
MiratrixL. W.SekhonJ. S.TheodoridisA. G.CamposL. F. (2018). Worth weighting? How to think about and use weights in survey experiments. Political Analysis, 26(3), 275–291.
25.
MiratrixL. W.SekhonJ. S.YuB. (2013). Adjusting treatment effect estimates by post-stratification in randomized experiments. Journal of the Royal Statistical Society. Series B, 75(2), 369–396.
26.
MiratrixL.W.WeissM.HendersonB. (2020). An applied researcher’s guide to estimating effects from multisite individually randomized trials: Estimands, estimators, and estimates [Unpublished manuscript].
27.
MukerjeeR.DasguptaT.RubinD. B. (2018). Using standard tools from finite population sampling to improve causal inference for complex experiments. Journal of the American Statistical Association, 113(522), 868–881.
28.
PashleyN. E.MiratrixL. W. (2020). Block what you can, except when you shouldn’t [Unpublished manuscript].
29.
RaudenbushS. W.SchwartzD. (2020). Randomized experiments in education, with implications for multilevel causal inference. Annual Review of Statistics and Its Application, 7, 177–208.
30.
R Core Team. (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
31.
RosenbaumP. R. (1991). A characterization of optimal designs for observational studies. Journal of the Royal Statistical Society. Series B, 53(3), 597–610.
32.
RosenbaumP. R. (2010). Design of observational studies. Springer series in statistics. Springer.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.
35.
RubinD. B. (1980). Randomization analysis of experimental data: The Fisher randomization test comment. Journal of the American Statistical Association, 75(371), 591–593.
36.
SärndalC.-E.SwenssonB.WretmanJ. (2003). Model assisted survey sampling. Springer.
37.
SävjeF. (2015). The performance and efficiency of threshold blocking. arXiv preprint arXiv:1506.02824. https://arxiv.org/abs/1506.02824
38.
SchochetP. Z. (2016). Statistical theory for the RCT-YES software: Design-based causal inference for RCTs (2nd ed.) (Technical Report (NCEE 2015-4011)). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development.
39.
ScosyrevE. (2014). Causal inference in block-randomized experiments: Analysis based on Neyman’s stochastic causal model [Unpublished manuscript].
40.
Splawa-NeymanJ.DabrowskaD. M.SpeedT. (1923). On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statistical Science, 5(4), 465–472. (original work published 1990)
41.
StataCorp. (2017). Stata Statistical Software: Release 15. StataCorp.
42.
WuC. F. J.HamadaM. S. (2000). Experiments: Planning, analysis, and parameter design optimization. John Wiley & Sons.
43.
ZhaoQ.SmallD. S.RosenbaumP. R. (2018). Cross-screening in observational studies that test many hypotheses. Journal of the American Statistical Association, 113(523), 1070–1084.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
