Several branches of the potential outcome causal inference literature have discussed the merits of blocking versus complete randomization. Some have concluded it can never hurt the precision of estimates, and some have concluded it can hurt. In this article, we reconcile these apparently conflicting views, give a more thorough discussion of what guarantees no harm, and discuss how other aspects of a blocked design can cost, all in terms of estimator precision. We discuss how the different findings are due to different sampling models and assumptions of how the blocks were formed. We also connect these ideas to common misconceptions; for instance, we show that analyzing a blocked experiment as if it were completely randomized, a seemingly conservative method, can actually backfire in some cases. Overall, we find that blocking can have a price but that this price is usually small and the potential for gain can be large. It is hard to go too far wrong with blocking.
BoxG. E.HunterJ. S.HunterW. G. (2005). Statistics for experimenters: Design, innovation, and discovery, 2nd edition. In BaldingD. J.BloomfieldP.CressieN. A. C.FisherN. I.JohnstoneI. M.KadaneJ. B.RyanL. M.ScottD. W.SmithA. F. M.TeugelsJ. L. (eds.) Wiley series in probability and statistics (p. 93). John Wiley.
2.
BoxG. E.HunterW. G.HunterJ. S. (1978). Statistics for experimenters: An introduction to design, data analysis, and model building. Wiley series in probability and mathematical statistics: Applied probability and statistics. John Wiley.
3.
BransonZ.DasguptaT.RubinD. B. (2016). Improving covariate balance in 2k factorial designs via rerandomization with an application to a New York City Department of Education high school study. The Annals of Applied Statistics, 10(4), 1958–1976.
4.
HellerJ. I.DaehlerK. R.WongN.ShinoharaM.MiratrixL. W. (2012). Differential effects of three professional development models on teacher knowledge and student achievement in elementary science. Journal of Research in Science Teaching, 49(3), 333–362.
5.
HigginsM. J.SävjeF.SekhonJ. S. (2015). Blocking estimators and inference under the Neyman-Rubin model. arXiv preprint arXiv:1510.01103. https://arxiv.org/abs/1510.01103
6.
HigginsM. J.SävjeF.SekhonJ. S. (2016). Improving massive experiments with threshold blocking. Proceedings of the National Academy of Sciences, 113(27), 7369–7376.
7.
ImaiK. (2008). Variance identification and efficiency analysis in randomized experiments under the matched-pair design. Statistics in Medicine, 27(24), 4857–4873.
8.
ImaiK.KingG.StuartE. A. (2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society. Series A, 171(2), 481–502.
9.
ImbensG. W. (2011). Experimental design for unit and cluster randomized trials[Conference session]. Conference International Initiative for Impact Evaluation, Cuernavaca, Mexico.
10.
ImbensG. W.RubinD. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press.
11.
LiX.DingP.RubinD. B. (2018). Asymptotic theory of rerandomization in treatment–control experiments. Proceedings of the National Academy of Sciences, 115(37), 9157–9162.
12.
MiratrixL. W.SekhonJ. S.TheodoridisA. G.CamposL. F. (2018). Worth weighting? How to think about and use weights in survey experiments. Political Analysis, 26(3), 275–291.
13.
MiratrixL. W.SekhonJ. S.YuB. (2013). Adjusting treatment effect estimates by post-stratification in randomized experiments. Journal of the Royal Statistical Society. Series B, 75(2), 369–396.
14.
MorganK. L.RubinD. B. (2012). Rerandomization to improve covariate balance in experiments. The Annals of Statistics, 40(2), 1263–1282.
15.
MorganK. L.RubinD. B. (2015). Rerandomization to balance tiers of covariates. Journal of the American Statistical Association, 110(512), 1412–1421.
16.
NordinM.SchultzbergM. (2020). Properties of restricted randomization with implications for experimental design. arXiv preprint arXiv:2006.14888. https://arxiv.org/abs/2006.14888
17.
PashleyN. E.MiratrixL. W. (2021). Insights on variance estimation for blocked and matched pairs designs. Journal of Educational and Behavioral Statistics, 46(3), 271–296.
18.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.
19.
RubinD. B. (1980). Randomization analysis of experimental data: The Fisher randomization test comment. Journal of the American Statistical Association, 75(371), 591–593.
20.
SärndalC.-E.SwenssonB.WretmanJ. (2003). Model assisted survey sampling. Springer.
21.
SävjeF. (2015). The performance and efficiency of threshold blocking. arXiv preprint arXiv:1506.02824. https://arxiv.org/abs/1506.02824
SnedecorG.CochranW. (1989). Statistical methods (8th ed.). Iowa State University Press.
24.
Splawa-NeymanJ.DabrowskaD. M.SpeedT. (1990). On the application of probability theory to agricultural experiments: Essay on principles: Section 9. Statistical Science, 5(4), 465–472. (Original work published 1923)
