This study evaluated three types of bias—total, measurement, and selection bias (SB)—in three sequential mixed-mode designs of the Dutch Crime Victimization Survey: telephone, mail, and web, where nonrespondents were followed up face-to-face (F2F). In the absence of true scores, all biases were estimated as mode effects against two different types of benchmarks. In the single-mode benchmark (SMB), effects were evaluated against a F2F reference survey. In an alternative analysis, a “hybrid-mode benchmark” (HMB) was used, where effects were evaluated against a mix of the measurements of a web survey and the SB of a F2F survey. A special reinterview design made available additional auxiliary data exploited in estimation for a range of survey variables. Depending on the SMB and HMB perspectives, a telephone, mail, or web design with a F2F follow-up (SMB) or a design involving only mail and/or web but not a F2F follow-up (HMB) is recommended based on the empirical findings.
BethlehemJelkeBiffignandiSilvia. 2011. Handbook of Web Surveys. Hoboken, NJ: John Wiley & Sons, Inc.
2.
BethlehemJelkeCobbenFannieSchoutenBarry. 2011. Handbook of Nonresponse in Household Surveys. Hoboken, NJ: Wiley.
3.
BiemerPaul P.1988. “Measuring Data Quality.” Pp. 341–75 in Telephone Survey Methodology, edited by GrovesRobert M.BiemerPaul P.LybergLarsMasseyJames T.NichollsWilliamIIWaksbergJoseph. New York: Wiley.
4.
BiemerPaul P.2010. “Total Survey Error: Design, Implementation, and Evaluation.” Public Opinion Quarterly74:817–48.
5.
BiemerPaul P.LybergLars E.. 2003. Introduction to Survey Quality. Hoboken, NJ: Wiley.
6.
CouperMick P.2000. “Review: Web Surveys: A Review of Issues and Approaches.” Public Opinion Quarterly64:464–94.
7.
CouperMick P.TraugottMichael W.LamiasMark J.. 2001. “Web Survey Design and Administration.” Public Opinion Quarterly65:230–53.
8.
De LeeuwEdith. 1992. Data Quality in Mail, Telephone, and Face to Face Surveys. Amsterdam, the Netherlands: TT-Publicaties.
9.
De LeeuwEdith. 2005. “To Mix or Not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics21:233–55.
10.
De LeeuwEdith. 2008. “Choosing the Method of Data Collection.” Pp. 113–35 in International Handbook of Survey Methodology, edited by LeeuwEdith DeHoxJoop J.DillmanDon A.. New York: Taylor & Francis.
11.
De LeeuwEdithDillmanDon A.HoxJoop J.. 2008. “Mixed Mode Surveys: When and Why.” Pp. 299–316 in International Handbook of Survey Methodology, edited by LeeuwEdith DeDillmanDon A.HoxJoop J.. New York: Lawrence Erlbaum.
12.
DillmanDon A.SmythJolene D.ChristianLeah Melani. 2009. Internet, Mail, and Mixed-mode Surveys. The Tailored Design Method. Hoboken, NJ: Wiles & Sons.
13.
ElliottMichael R.LittleRoderick J. A.LewitzkySteve. 2000. “Subsampling Callbacks to Improve Survey Efficiency.” Journal of the American Statistical Association95:730–38.
14.
FowlerFloyd JacksonGallagherP. M.StringfellowV. L.ZaslavskyA. M.ThompsonJ. W.ClearyP. D.. 2002. “Using Telephone Interviews to Reduce Nonresponse Bias to Mail Surveys of Health Plan Members.” Medical Care40:190–200.
15.
FrickerScottGalesicMirtaTourangeauRogerYanTing. 2005. “An Experimental Comparison of Web and Telephone Surveys.” Public Opinion Quarterly69:370–92.
16.
GrovesRobert M.2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly70:646–75.
GrovesRobert M.LybergLars. 2010. “Total Survey Error: Past, Present, and Future.” Public Opinion Quarterly74:849–79.
19.
HansenMorris H.HurwitzWilliam N.. 1946. “The Problem of Non-response in Sample Surveys.” Journal of the American Statistical Association41:517–29.
20.
ImbensGuido W.2004. “Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review.” Review of Economics and Statistics86:4–29.
21.
KeeterScottMillerCarolynKohutAndrewGrovesRobert M.PresserStanley. 2000. “Consequences of Reducing Nonresponse in a National Telephone Survey.” Public Opinion Quarterly64:125–48.
22.
KerstenHubert M. P.BethlehemJelke. 1984. “Exploring and Reducing the Nonresponse Bias by Asking the Basic Question.” Statistical Journal of the United Nations Economic Commission for Europe2:369–80.
23.
KirchnerA.FeldererB.. (2013). “The Effect of Survey Mode on Nonresponse Bias and Measurement Error: A Validation Approach.” Paper presented at the 68th Annual Conference of the American Association for Public Opinion Research, May 16–19, 2013, Boston.
24.
KlauschThomasHoxJoop J.SchoutenBarry. 2013. “Measurement Effects of Survey Mode on the Equivalence of Attitudinal Rating Scale Questions.” Sociological Methods & Research42:227–63.
25.
KlauschT.HoxJ.SchoutenB.. 2015. “Selection Error in Single- and Mixed Mode Surveys of the Dutch General Population.” Journal of the Royal Statistical Society: Series A (Statistics in Society). http://doi.org/10.1111/rssa.12102.
KörmendiE.1988. “The Quality of Income Information in Telephone and Face to Face Surveys.” Pp. 341–75 in Telephone Survey Methodology, edited by GrovesRobert M.BiemerPaul P.LybergLarsMasseyJames T.NichollsWilliamIIWaksbergJoseph. New York: Wiley.
28.
KreuterFraukeMüllerGerritTrappmannMark. 2010. “Nonresponse and Measurement Error in Employment Research: Making Use of Administrative Data.” Public Opinion Quarterly74:880–906.
KreuterFraukePresserStanleyTourangeauRoger. 2008. “Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity.” Public Opinion Quarterly72:847–65.
31.
KrosnickJon A.1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology5:213–36.
32.
LinkMichael W.MokdadAli H.. 2006. “Can Web and Mail Survey Modes Improve Participation in an RDD-based National Health Surveillance?” Journal of Official Statistics22:293–312.
33.
LittleRoderick J. A.RubinDonald B.. 2002. Statistical Analysis with Missing Data. 2nd ed. Hoboken, NJ: Wiley.
34.
LumleyThomas S.2010. Complex Surveys: A Guide to Analysis Using R. Hoboken, NJ: Wiley.
35.
LynnPeter. 2013. “Alternative Sequential Mixed-mode Designs: Effects on Attrition Rates, Attrition Bias, and Costs.” Journal of Survey Statistics and Methodology1:183–205.
36.
MillarMorgan M.DillmanDon A.. 2011. “Improving Response to Web and Mixed-mode Surveys.” Public Opinion Quarterly75:249–69.
37.
MillerPeter V.GrovesRobert M.. 1985. “Matching Survey Responses to Official Records: An Exploration of Validity in Victimization Reporting.” Public Opinion Quarterly49:366–80.
38.
MorganStephen L.WinshipChirstopher. 2007. Counterfactuals and Causal Inference. Cambridge, UK: Cambridge University Press.
39.
OlsonKristen. 2006. “Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias.” Public Opinion Quarterly70:737–58.
40.
PearlJudea. 2009. Causality. Models, Reasoning, and Inference. 2nd ed. New York: Cambridge University Press.
41.
PeytchevAndy. 2012. “Multiple Imputation for Unit Nonresponse and Measurement Error.” Public Opinion Quarterly76:214–37.
42.
RaghunathanTrivellore E.LepkowskiJamesHoewykJohn vanSolenbergerPeter. 2001. “A Multivariate Technique for Multiply Imputing Missing Values Using a Sequence of Regression Models.” Survey Methodology27:85–95.
43.
RevillaMelanie. 2010. “Quality in Unimode and Mixed-mode Designs: A Multitrait-multimethod Approach.” Survey Research Methods4:151–64.
44.
RubinDonald B.1978. “Bayesian Inference for Causal Effects: The Role of Randomization.” The Annals of Statistics6:34–58.
45.
RubinDonald B.1987. Multiple Imputation for Nonresponse in Surveys. New York: John Wiley & Sons.
46.
RubinDonald B.2003. “Nested Multiple Imputation of NMES via Partially Incompatible MCMC.” Statistica Neerlandica57:3–18.
47.
RubinDonald B.2005. “Causal Inference Using Potential Outcomes.” Journal of the American Statistical Association100:322–31.
48.
SakshaugJoseph W.YanTingTourangeauRoger. 2010. “Nonresponse Error, Measurement Error, and Mode of Data Collection: Tradeoffs in a Multi-mode Survey of Sensitive and Non-sensitive Items.” Public Opinion Quarterly74:907–33.
49.
SarisWillem E.GallhoferIrmtraud. 2007. “Estimation of the Effects of Measurement Characteristics on the Quality of Survey Questions.” Survey Research Methods1:29–43.
SchaferJoseph L.GrahamJohn W.. 2002. “Missing Data: Our View of the State of the Art.” Psychological Methods7:147–77.
52.
SchaferJoseph L.KangJoseph. 2008. “Average Causal Effects from Nonrandomized Studies: A Practical Guide and Simulated Example.” Psychological Methods13:279–313.
53.
SchoutenBarryCobbenFannieBethlehemJelke. 2009. “Indicators for the Representativeness of Survey Response.” Survey Methodology35:101–13.
54.
SchoutenBarryBrakelJan van denBuelensBartLaanJan van derKlauschThomas. 2013. “Disentangling Mode-specific Selection and Measurement Bias in Social Surveys.” Social Science Research42:1555–70.
55.
TourangeauRogerGrovesRobert M.RedlineCleo D.. 2010. “Sensitive Topics and Reluctant Respondents: Demonstrating a Link between Nonresponse Bias and Measurement Error.” Public Opinion Quarterly74:413–32.
56.
TourangeauRogerRipsLance J.RasinskiKenneth. 2000. The Psychology of Survey Response. Cambridge, UK: Cambridge University Press.
57.
TourangeauRogerYanTing. 2007. “Sensitive Questions in Surveys.” Psychological Bulletin133:859–83.
58.
Van BuurenStef. 2012. Flexible Imputation of Missing Data. Boca Raton, FL: CRC Press.
59.
Van BuurenStefBrandJ. P. L.Groothuis-OudshoornC. G. M.RubinDon B.. 2006. “Fully Conditional Specification in Multivariate Imputation.” Journal of Statistical Computation and Simulation76:1049–64.
60.
Van BuurenStefGroothuis-OudshoornKarin. 2011. “Mice: Multivariate Imputation by Chained Equations in R.” Journal of Statistical Software45:1–67.
61.
VannieuwenhuyzeJorre. 2014. “On the Relative Advantage of Mixed-mode Versus Single-mode Surveys.” Survey Research Methods8:31–42.
62.
VannieuwenhuyzeJorreLoosveldtGeert. 2013. “Evaluating Relative Mode Effects in Mixed-mode Surveys: Three Methods to Disentangle Selection and Measurement Effects.” Sociological Methods & Research42:82–104.
63.
VannieuwenhuyzeJorreLoosveldtGeertMolenberghsGeert. 2010. “A Method for Evaluating Mode Effects in Mixed-mode Surveys.” Public Opinion Quarterly74:1027–45.
64.
VannieuwenhuyzeJorreLoosveldtGeertMolenberghsGeert. 2014. “Evaluating Mode Effects in Mixed-mode Survey Data Using Covariate Adjustment Models.” Journal of Official Statistics30:1–21.
65.
VoogtRobert J. J.SarisWillem E.. 2005. “Mixed Mode Designs: Finding the Balance between Nonresponse Bias and Mode Effects.” Journal of Official Statistics21:367–87.
66.
WagnerJames. 2012. “A Comparison of Alternative Indicators for the Risk of Nonresponse Bias.” Public Opinion Quarterly76:555–75.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
