Policymakers have grown increasingly interested in how experimental results may generalize to a larger population. However, recently developed propensity score–based methods are limited by small sample sizes, where the experimental study is generalized to a population that is at least 20 times larger. This is particularly problematic for methods such as subclassification by propensity score, where limited sample sizes lead to sparse strata. This article explores the potential of small area estimation methods to improve the precision of estimators in sparse strata using population data as a source of auxiliary information to borrow strength. Results from simulation studies identify the conditions under which small area estimators outperform conventional estimators and the limitations of this application to causal generalization studies.
BatteseG. E.HarterR. M.FullerW. A. (1988). An error-components model for prediction of county crop areas using survey and satellite data. Journal of the American Statistical Association, 83, 28–36.
2.
BellS. H.OlsenR. B.OrrL. L.StuartE. A. (2016). Estimates of external validity bias when impact evaluations select sites nonrandomly. Educational Evaluation and Policy Analysis. Retrieved fromhttps://doi.org/10.3102/0162373715617549
3.
CochranW. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24, 295–313.
4.
ColeS. R.StuartE. A. (2010). Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. American Journal of Epidemiology, 172, 107–115.
5.
CornfieldJ.TukeyJ. W. (1956). Average values of mean squares in factorials. The Annals of Mathematical Statistics, 27, 907–949.
6.
DattaG. S.LahiriP. (2000). A unified measure of uncertainty of estimated best linear unbiased predictors in small area estimation problems. Statistica Sinica, 10, 613–627.
7.
GelmanA. (2006). Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian Analysis, 1, 515–534.
8.
GreenburgD.SchroderM. (2004). The digest of social experiments (3rd ed.). Washington, DC: The Urban Institute Press.
HorvitzD. G.ThompsonD. J. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47, 663–685.
11.
ImaiK.KingG.StuartE. A. (2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A (Statistics in Society), 171, 481–502.
12.
KangJ. D.SchaferJ. L. (2007). Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22, 523–539.
13.
KonstantopoulosS.MillerS. R.van der PloegA. (2013). The impact of Indiana’s system of interim assessments on mathematics and reading achievement. Educational Evaluation and Policy Analysis, 35, 481–499.
14.
KruschkeJ. (2014). Doing Bayesian data analysis: A tutorial with R, JAGS, and BUGS. Burlington, MA: Academic Press.
15.
KruskalW.MostellerF. (1980). Representative sampling, IV: The history of the concept in statistics, 1895–1939. International Statistical Review/Revue Internationale de Statistique, 48, 169–195.
16.
LeeB. K.LesslerJ.StuartE. A. (2011). Weight trimming and propensity score weighting. PLoS One, 6, e18174.
17.
MolinaI.NandramB.RaoJ. N. K. (2014). Small area estimation of general parameters with application to poverty indicators: A hierarchical Bayes approach. The Annals of Applied Statistics, 8, 852–885.
18.
OlsenR. B.OrrL. L.BellS. H.StuartE. A. (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32, 107–121.
19.
O’MuircheartaighC.HedgesL. V. (2014). Generalizing from unrepresentative experiments: a stratified propensity score approach. Journal of the Royal Statistical Society: Series C (Applied Statistics), 63, 195–210.
20.
PfeffermannD. (2013). New important developments in small area estimation. Statistical Science, 28, 40–68.
21.
PurcellN. J.KishL. (1979). A biometrics invited paper. Estimation for small domains. Biometrics, 35, 365–384.
22.
R Core Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved fromhttp://www.R-project.org/
23.
RaoJ. N.MolinaI. (2015). Small area estimation. New York, NY: John Wiley.
24.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (vol. 1). Thousand Oaks, CA: Sage.
25.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
26.
RosenbaumP. R.RubinD. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516–524.
27.
RoyallR. M. (1976). The linear least-squares prediction approach to two-stage sampling. Journal of the American Statistical Association, 71, 657–664.
28.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.
29.
RubinD. B. (1977). Assignment to treatment group on the basis of a covariate. Journal of Educational and Behavioral Statistics, 2, 1–26.
30.
RubinD. B. (1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, 6, 34–58.
31.
RubinD. B. (1980). Randomization analysis of experimental data: The Fisher randomization test comment. Journal of the American Statistical Association, 75, 591–593.
32.
RubinD. B. (1986). Statistics and causal inference: Comment: Which ifs have causal answers. Journal of the American Statistical Association, 81, 961–962.
33.
RubinD. B. (1990). Formal mode of statistical inference for causal effects. Journal of Statistical Planning and Inference, 25, 279–292.
34.
StuartE. A.BellS. H.EbnesajjadC.OlsenR. B.OrrL. L. (2016). Characteristics of school districts that participate in rigorous national educational evaluations. Journal of Research on Educational Effectiveness, 10, 168–206.
35.
StuartE. A.ColeS. R.BradshawC. P.LeafP. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174, 369–386.
36.
TiptonE. (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38, 239–266.
37.
TiptonE. (2014). How generalizable is your experiment? Comparing a sample and population through a generalizability index. Journal of Educational and Behavioral Statistics, 39, 478–501.
38.
TiptonE.HallbergK.HedgesL. V.ChanW. (2016). Implications of small samples for generalization adjustments and rules of thumb. Evaluation Review. doi:0193841X16655665
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
