Directed acyclic graphs (DAGs) are now a popular tool to inform causal inferences. We discuss how DAGs can also be used to encode theoretical assumptions about nonprobability samples and survey nonresponse and to determine whether population quantities including conditional distributions and regressions can be identified. We describe sources of bias and assumptions for eliminating it in various selection scenarios. We then introduce and analyze graphical representations of multiple selection stages in the data collection process, and highlight the strong assumptions implicit in using only design weights. Furthermore, we show that the common practice of selecting adjustment variables based on correlations with sample selection and outcome variables of interest is ill-justified and that nonresponse weighting when the interest is in causal inference may come at severe costs. Finally, we identify further areas for survey methodology research that can benefit from advances in causal graph theory.
AbadieAlbertoAtheySusanImbensGuido W.WooldridgeJeffrey M.. 2020. “Sampling-Based Versus Design-Based Uncertainty in Regression Analysis.” Econometrica88(1):265-96.
2.
AnsolabehereStephenHershEitan. 2012. “Validation: What Big Data Reveal About Survey Misreporting and the Real Electorate.” Political Analysis20(4):437-59.
3.
BareinboimEliasTianJinPearlJudea. 2014. “Recovering from Selection Bias in Causal and Statistical Inference.” in Twenty-Eighth AAAI Conference on Artificial Intelligence.
4.
BareinboimEliasPearlJudea. 2016. “Causal Inference and the Data-fusion Problem.” Proceedings of the National Academy of Sciences113(27):7345-52.
5.
BlumbergStephen J.LukeJulian V.. 2007. “Coverage Bias in Traditional Telephone Surveys of Low-Income and Young Adults.” Public Opinion Quarterly71(5):734-49.
6.
CornesseCarinaBlomAnnelies G.DutwinDavidKrosnickJon A.De LeeuwEdith D.LegleyeStéphanePasekJoshPennayDarrenPhillipsBenjaminSakshaugJoseph W., et al. 2020. “A Review of Conceptual Approaches and Empirical Evidence on Probability and Nonprobability Sample Survey Research.” Journal of Survey Statistics and Methodology8(1):4-36.
7.
DanielRhian M.KenwardMichael G.CousensSimon N.De StavolaBianca L.. 2012. “Using Causal Diagrams to Guide Analysis in Missing Data Problems.” Statistical Methods in Medical Research21(3):243-56.
8.
DasMitaliNeweyWhitney K.VellaFrancis. 2003. “Nonparametric Estimation of Sample Selection Models.” The Review of Economic Studies70(1):33-58.
9.
DidelezVanessaKreinerSvendKeidingNiels. 2010. “Graphical Models for Inference Under Outcome-Dependent Sampling.” Statistical Science25(3):368-87.
10.
ElliottMichael R.ValliantRichard. 2017. “Inference for Nonprobability Samples.” Statistical Science32(2):249-64.
11.
ElwertFelixWinshipChristopher. 2014. “Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable.” Annual Review of Sociology40:31-53.
12.
GelmanAndrew. 2007. “Struggles with Survey Weighting and Regression Modeling.” Statistical Science22(2):153-64.
13.
GelmanAndrewGoelSharadRiversDouglasRothschildDavid, et al. 2016. “The Mythical Swing Voter.” Quarterly Journal of Political Science11(1):103-30.
14.
GriffithGarethMorrisTim T.TudballMattHerbertAnnieMancanoGiuliaPikeLindseySharpGemma C.PalmerTom M.SmithGeorge DaveyTillingKateZuccoloLuisaDaviesNeil M.HemaniGibran. 2020. “Collider Bias Undermines our Understanding of COVID-19 Disease Risk and Severity.”Nature Communications11(1):5749.
15.
GrovesRobert M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly70(5):646-75.
16.
GrovesRobert M.SingerEleanorCorningAmy. 2000. “Leverage-Saliency Theory of Survey Participation: Description and an Illustration.” The Public Opinion Quarterly64(3):299-308.
17.
GrovesRobert M.PeytchevaEmilia. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” Public Opinion Quarterly72(2):167-89.
GrovesRobert M.LybergLars. 2010. “Total Survey Error: Past, Present, and Future.” Public Opinion Quarterly74(5):849-79.
20.
GrovesRobert M.PresserStanleyDipkoSarah. 2004. “The Role of Topic Interest in Survey Participation Decisions.” Public Opinion Quarterly68(1):2-31.
21.
HeckmanJames J. 1979. “Sample Selection Bias as a Specification Error.” Econometrica47(1):153-61.
22.
KennedyCourtneyBlumenthalMarkClementScottClintonJoshua D.DurandClaireFranklinCharlesMcGeeneyKyleyMiringoffLeeOlsonKristenRiversDouglas, et al. 2018. “An Evaluation of the 2016 Election Polls in the United States.” Public Opinion Quarterly82(1):1-33.
23.
KennedyLaurenGelmanAndrew. 2021. “Know Your Population and Know Your Model: Using Model-based Regression and Poststratification to Generalize Findings Beyond the Observed Sample.” Psychological Methods26(5):547-58.
24.
KohlerUlrichKreuterFraukeStuartElizabeth A.. 2019. “Nonprobability Sampling and Causal Analysis.” Annual Review of Statistics and its Application6:149-72.
KreuterFraukeOlsonKristen. 2013. “Paradata for Nonresponse Error Investigation,” pp. 11-42 in Improving Surveys with Paradata: Analytic Uses of Process Information, edited by Kreuter Frauke. Hobken, NJ: John Wiley & Sons.
27.
KreuterFraukeOlsonKristenWagnerJamesYanTingEzzati-RiceTrena M.Casas-CorderoCarolinaLemayMichaelPeytchevAndyGrovesRobert M.RaghunathanTrivellore E.. 2010. “Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Non-response: Examples From Multiple Surveys.” Journal of the Royal Statistical Society: Series A (Statistics in Society)173(2):389-407.
28.
LahtinenHannuMartikainenPekkaMattilaMikkoWassHannaRapeliLauri. 2019. “Do Surveys Overestimate or Underestimate Socioeconomic Differences in Voter Turnout? Evidence from Administrative Registers.” Public Opinion Quarterly83(2):363-85.
29.
LiXinranDingPeng. 2017. “General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference.” Journal of the American Statistical Association112(520):1759-69.
30.
LittleRoderick J. A. 1986. “Survey Nonresponse Adjustments for Estimates of Means.” International Statistical Review/Revue Internationale de Statistique54(2):139-57.
31.
LittleRoderick J. 2022. “A Note About the Definition of Response Propensity for Survey Nonresponse.” Journal of Survey Statistics and Methodology10(4):1098-106.
32.
LittleRoderick J. A.RubinDonald B.. 2019. Statistical Analysis with Missing Data. Hoboken, NJ: John Wiley & Sons.
33.
LohrSharon L. 2019. Sampling: Design and Analysis. Boca Raton, FL: Chapman and Hall/CRC.
34.
ManskiCharles F. 2009. Identification for Prediction and Decision. Cambridge, Massachusetts and London, England: Harvard University Press.
35.
MercerAndrew W.KreuterFraukeKeeterScottStuartElizabeth A.. 2017. “Theory and Practice in Nonprobability Surveys: Parallels Between Causal Inference and Survey Inference.” Public Opinion Quarterly81(S1):250-71.
36.
MohanKarthikaThoemmesFelixPearlJudea. 2018. “Estimation with Incomplete Data: The Linear Case.” Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence Main track: 5082-88.
37.
MohanKarthikaPearlJudea. 2021. “Graphical Models for Processing Missing Data.” Journal of the American Statistical Association116(534):1023-37.
38.
Moreno-BetancurMargaritaLeeKatherine J.LeacyFinbarr P.WhiteIan R.SimpsonJulie A.CarlinJohn B.. 2018. “Canonical Causal Diagrams to Guide the Treatment of Missing Data in Epidemiologic Studies.” American Journal of Epidemiology187(12):2705-15.
39.
MorganStephen L.WinshipChristopher. 2015. Counterfactuals and Causal Inference. Cambridge, MA and London, England: Cambridge University Press.
40.
OlsonKristen. 2019. “Comments on “How Errors Cumulate: Two Examples” by Roger Tourangeau.” Journal of Survey Statistics and Methodology8(3):442-52.
41.
PearlJudea. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Francisco, CA: Morgan Kaufmann.
42.
PearlJudea. 2009. Causality. Cambridge, MA and London, England: Cambridge University Press.
43.
PearlJudea. 2011. “Invited Commentary: Understanding Bias Amplification.” American Journal of Epidemiology174(11):1223-7.
44.
PearlJudeaGlymourMadelynJewellNicholas P. 2016. Causal Inference in Statistics: A Primer. Hoboken, NJ: John Wiley & Sons.
45.
PeytchevAndy. 2013. “Consequences of Survey Nonresponse.” The ANNALS of the American Academy of Political and Social Science645(1):88-111.
46.
PeytchevAndyCarley-BaxterLisa R.BlackMichele C.. 2011. “Multiple Sources of Nonobservation Error in Telephone Surveys: Coverage and Nonresponse.” Sociological Methods & Research40(1):138-68.
47.
PeytchevAndyPresserStanleyZhangMengmeng. 2018. “Improving Traditional Nonresponse Bias Adjustments: Combining Statistical Properties with Social Theory.” Journal of Survey Statistics and Methodology6(4):491-515.
48.
PfeffermannDannySverchkovMichail. 2009. “Inference under informative sampling.” Pp. 455–87 in Handbook of statistics. Vol. 29, Elsevier.
49.
RubinDonald B. 1976. “Inference and Missing Data.” Biometrika63(3):581-92.
50.
SakshaugJoseph W.AntoniManfred. 2019. “Evaluating the Utility of Indirectly Linked Federal Administrative Records for Nonresponse Bias Adjustment.” Journal of Survey Statistics and Methodology7(2):227-49.
51.
SchaferJoseph L.GrahamJohn W.. 2002. “Missing Data: Our View of the State of the Art.” Psychological Methods7(2):147.
52.
Shirani-MehrHoushmandRothschildDavidGoelSharadGelmanAndrew. 2018. “Disentangling Bias and Variance in Election Polls.” Journal of the American Statistical Association113(522):607-14.
53.
SmithLouisa H.VanderWeeleTyler J.. 2019. “Bounding Bias Due to Selection.” Epidemiology (Cambridge, Mass.)30(4):509-16.
54.
SmithTom W.DavernMichaelFreeseJeremyMorganStephen L.. 2019. General Social Surveys, 1972–2018: Cumulative Codebook. Chicago: NORC.
55.
SteinerPeter M.KimYongnam. 2016. “The Mechanics of Omitted Variable Bias: Bias Amplification and Cancellation of Offsetting Biases.” Journal of Causal Inference4(2):20160009.
56.
StoopIneke A. L. 2005. The Hunt for the Last Respondent: Nonresponse in Sample Surveys. The Hague: Sociaal en Cultureel Planbu.
57.
ThoemmesFelixMohanKarthika. 2015. “Graphical Representation of Missing Data Problems.” Structural Equation Modeling: A Multidisciplinary Journal22(4):631-42.
58.
TourangeauRoger. 2019. “How Errors Cumulate: Two Examples.” Journal of Survey Statistics and Methodology8(3):413-32.
59.
ValliantRichard. 2020. “Comparing Alternatives for Estimation From Nonprobability Samples.” Journal of Survey Statistics and Methodology8(2):231-63.
60.
ValliantRichardDorfmanAlan H.RoyallRichard M.. 2000. Finite Population Sampling and Inference. A Prediction Approach. Hoboken, NJ: John Wiley & Sons.
61.
van der VeldWilliamSarisWillem E.. 2004. “Separation of Error, Method Effects, Instability, and Attitude Strength.” Pp. 37–59 in Studies in Public Opinion: Attitudes, Nonattitudes, Measurement Error, and Change, edited by Willem E. Saris and Paul M. Sniderman. Princeton, NJ: Princeton University Press.
62.
VanderWeeleTyler J.HernánMiguel A.. 2012. “Results on Differential and Dependent Measurement Error of the Exposure and the Outcome Using Signed Directed Acyclic Graphs.” American Journal of Epidemiology175(12):1303-10.
63.
WhiteHalbertLuXun. 2011. “Causal Diagrams for Treatment Effect Estimation with Application to Efficient Covariate Selection.” Review of Economics and Statistics93(4):1453-9.
64.
WingCoady. 2019. “What Can Instrumental Variables Tell Us About Nonresponse In Household Surveys and Political Polls?” Political Analysis27(3):320-38.
65.
WooldridgeJeffrey M. 2007. “Inverse Probability Weighted Estimation for General Missing Data Problems.” Journal of econometrics141(2):1281-301.
66.
ZhangLi-Chun. 2000. “Post-Stratification and Calibration—A Synthesis.” The American Statistician54(3):178-84.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
