Although a large-scale experiment can provide an estimate of the average causal impact for a program, the sample of sites included in the experiment is often not drawn randomly from the inference population of interest. In this article, we provide a generalizability index that can be used to assess the degree of similarity between the sample of units in an experiment and one or more inference populations on a set of selected covariates. The index takes values between 0 and 1 and indicates both when a sample is like a miniature of the population and how well reweighting methods may perform when differences exist. Results of simulation studies are provided that develop rules of thumb for interpretation as well as an example.
AherneF. J.ThackerN. A.RockettP. I. (1998). The Bhattacharyya metric as an absolute similarity measure for frequency coded data. Kybernetika, 34, 363–368.
2.
BareinboimE.PearlJ. (2013). A general algorithm for deciding transportability of experimental results. Journal of Causal Inference, 1, 107–134.
3.
BhattacharyyaA. (1943). On a measure of divergence between two statistical populations defined by their probability distributions. Bulletin of the Calcutta Mathematical Society, 35, 99–109.
4.
BhattacharyyaA. (1946). On a measure of divergence between two multinomial populations. Sankhya: The Indian Journal of Statistics, 7, 401–406.
5.
BelitserS. V.MartensE. P.PestmanW. R.GroenwoldR. H. H.de BoerA.KlungelO. H. (2011). Measuring balance and model selection in propensity score methods. Pharmacoepidemiology and Drug Safety, 20, 1115–1129.
6.
BormanG. D.SlavinR. E.CheungA.ChamberlainA. M.MaddenN. A.ChambersB. (2005). Success for all: First-year results from the national randomized field trial. Educational Evaluation and Policy Analysis, 27, 1–22.
7.
ChattopadhyayA.ChattopadhyayA. K.RaoC. B. (2009). Bhattacharyya’s distance measure as a precursor of genetic distance measures. Journal of Biosciences, 29, 135–138.
8.
ChernoffH. (1952) A measure of asymptotic efficiency of tests for hypothesis based on a sum of observations. Mathematical Statistics, 23, 493–507.
9.
ComaniciuD.RameshV.MeerP. (2000). Real-time tracking of non-rigid objects using mean shift. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2, 142–149.
10.
CornfieldJ.TukeyJ. W. (1956). Average values of mean squares in factorials. The Annals of Mathematical Statistics, 27, 907–949.
11.
FranklinJ. M.RassenJ. A.AckermannD.BartelsD. B.SchneeweissS. (2014). Metrics for covariate balance in cohort studies of causal effects. Statistics in Medicine, 33, 1685–1699.
12.
GreenburgD.SchroderM. (2004). The digest of social experiments (3rd ed.). Washington, DC: The Urban Institute Press.
13.
GrovesR. M.FowlerF. J.CouperM. P.LepkowskiJ. M.SingerE. (2009). Survey methodology. New York, NY: John Wiley.
14.
HedgesL. V.O’MuircheartaighC. A. (2011). Improving generalizations from designed experiments (Working Paper). Evanston, IL: Northwestern University.
15.
HellingerE. (1909). Neue Begrundung der Theorie quadratischer Formen von unendlichvielen Veranderlichen. J Reine Angew Math, 36, 210–271.
16.
HoD. E.ImaiK.KingG.StuartE. A. (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15, 199–236.
17.
JensenJ. L. W. V. (1906). Sur les fonctions convexes et les inegalites entre les valeurs moyennes. Acta Mathematica, 30, 175–193.
18.
KhalidM. S.IlyasM. U.SarfarazM. S.AjazM. A. (2006). Bhattacharyya coefficient in correlation of gray-scale objects. Journal of Multimedia, 1, 56–61.
19.
KitagawaE. M. (1964). Standardized comparisons in population research. Demography, 1, 296–315.
20.
MatusitaK. (1967). On the notion of affinity of several distributions and some of its applications. Annals of the Institute of Statistical Mathematics, 19, 181–192.
21.
NummiaroK.Koller-MeierE.Van GoolL. (2003). An adaptive color-based particle filter. Image and Vision Computing, 21, 99–110.
22.
OlsenR. B.OrrL. L.BellS. H.StuartE. A. (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32, 107–121.
23.
O’MuircheartaighC.HedgesL. V. (2014). Generalizing from unrepresentative experiments: A stratified propensity score approach. Journal of the Royal Statistical Society: Series C (Applied Statistics), 63, 195–2010.
24.
R Development Core Team. (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. Retrieved fromhttp://www.R-project.org/
25.
RaoC. R. (1949). On the distance between two populations. Sankhyā: The Indian Journal of Statistics, 9, 246–248.
26.
RaoC. R. (1982). Diversity and dissimilarity coefficient: A unified approach. Theoretical Population Biology, 21, 24–43.
27.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Volume 1 of Advanced Quantitative Techniques in the Social Sciences. Newbury Park, CA: Sage.
28.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
29.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.
30.
RubinD. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services & Outcomes Research Methodology, 2, 169–188.
31.
RubinD. B.ThomasN. (1996). Matching using estimated propensity scores: Relating theory to practice. Biometrics, 52, 249–264.
32.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin.
33.
ShenH.NelsonG.NelsonD. E.KennedyS.SpillerD. G.GriffithsT.…KellD. B. (2006). Automated tracking of gene expression in individual cells and cell compartments. Journal of The Royal Society Interface, 3, 787–794.
34.
SilvermanB. W. (1986). Density estimation for statistics and data analysis (Vol. 26). Boca Raton, FL: CRC Press.
35.
StuartE. (2010) Matching methods for causal inference: A review and a look forward. Statistical Science, 25, 1–21.
36.
StuartE. A.ColeS. R.BradshawC. P.LeafP. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society, Series A, 174, 369–386.
37.
StuartE. A.OlsenR. B.BellS. H.OrrL. L. (2012). Estimates of external validity bias when impact evaluations select sites purposively. Proceedings from the Society for Research on Annual Effectiveness Spring Meeting, Washington, DC.
38.
TiptonE. (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38, 239–266.
39.
TiptonE. (2014). Stratified sampling using cluster analysis: A balanced-sampling strategy for improved generalizations from experiments. Evaluation Review, 37, 109–139.
40.
TiptonE.HedgesL. V.Vaden-KiernanM.BormanG.SullivanK.CaverlyS. (2014). Designing a sample selection plan to improve generalizations from two scale-up experiments. Journal of Research on Educational Effectiveness, 7, 114–135.
41.
ZhangS.WangT. (2009). A new distance-based approach for phylogenetic analysis of protein sequences. International Journal of Biology and Biomedical Engineering, 3, 35–42.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
