This study leverages a randomized experimental design of a mixed-mode mail- and web-based survey to examine mode effects separately from sample selectivity issues. Using data from the Cognitive Economics Study, which contains some sensitive financial questions, we analyze two sets of questions: fixed-choice questions posed nearly identically across mode, and dollar-value questions that exploit features available only on web mode. Focusing on differences in item nonresponse and response distributions, our results indicate that, in contrast to mail mode, web mode surveys display lower item nonresponse for all questions. While respondents appear to prefer providing financial information in ranges, use of reminder screens on the web version yields greater use of exact values without large sacrifices in item response. Still, response distributions for all questions are similar across mode, suggesting that data on sensitive financial questions collected from the two modes can be pooled.
TourangeauRSmithTW. Asking sensitive questions: The impact of data collection mode, question format, and question context. Public Opinion Quarterly1996; 60(2): 275-304.
3.
KreuterFPresserSTourangeauR. Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly2008; 72: 847-865. doi: 10.1093/poq/nfn063.
4.
TourangeauRRipsLJRasinskiK. The psychology of survey response. Cambridge: Cambridge University Press, 2000.
5.
PreisendörferPWolterF. Who is telling the truth? A Validation Study on Determinants of Response Behavior in Surveys. Public Opinion Quarterly2014; 78: 126-146. doi: 10.1093/poq/nft079.
6.
MavletovaACouperMP. Sensitive topics in PC web and mobile web surveys: Is there a difference? Survey Research Methods2013; 7(3): 191-205. doi: 10.18148/srm/2013.v7i3.5458.
7.
Booth-KewleySLarsonGEMiyoshiDK. Social desirability effects on computerized and paper-and-pencil questionnaires. Computers in Human Behavior2007; 23: 463-477. doi: 10.1016/j.chb.2004.10.020.
8.
SmythJDDillmanDAChristianLM. Context effects in Internet surveys: New issues and evidence. The Handbook of Internet Psychology, Chapter 27. Oxford: Oxford University Press, 2007. doi: 10.1093/oxfordhb/9780199561803.001.000.
9.
CouperMP. The future of modes of data collection. Public Opinion Quarterly2011; 75(5): 889-908. doi: 10.1093/poq/nfr046.
10.
RichmanWLKieslerSWeisbandSDrasgowF. A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology1999; 84(5): 754-775. doi: 10.1037/0021-9010.84.5.754.
11.
de LeeuwEHoxJScherpenzeelA. Mode effect or question wording? Measurement error in mixed mode surveys. Proceedings of the Survey Research Methods Section of the American Statistical Association2011; 5959-5967.
12.
YeCFultonJTourangeauR. More positive or more extreme? A meta-analysis of mode differences in response choice. Public Opinion Quarterly2011; 75(2): 349-365. doi: 10.1093/poq/nfr009.
13.
RevillaMASarisWE. A comparison of the quality of questions in a face-to-face and a web survey. International Journal of Public Opinion Research2012; 25(2): 242-253. doi: 10.1093/ijpor/eds007.
14.
Cognitive Economics Study. CogEcon Public Core Release (Version 1.0) public use dataset. Produced and distributed by the University of Michigan with funding from the National Institute on Aging (grant number NIA P01 AG026571). Ann Arbor, 2011. http://ebp-projects.isr.umich.edu/CogEcon/data.html.
15.
Cognition and Aging in the USA Study. Waves 1 and 2 datasets. Produced by the University of Michigan for the University of Southern California with funding from the National Institute on Aging (grant number NIA R 37 AG007137, PI: McArdle). Ann Arbor, 2011. http://cogusc.usc.edu/Pages/Data/Data.html.
16.
Health and Retirement Study. Sample Sizes and Response Rates. Ann Arbor, 2011. http://hrsonline.isr.umich.edu/sitedocs/sampleresponse.pdf.
17.
HsuJWFisherGGWillisRJ. Internet access and cognitive ability: An analysis of the selectivity of internet interviews in the cognitive economics survey. Essays on Aging and Human Capital, Hsu JW Ph.D. dissertation, University of Michigan, 2011.
18.
van GelderMMBretveldRWRoeleveldN. Web-based questionnaires: The future in epidemiology? American Journal of Epidemiology2010; 172(11): 1292-1298. doi: doi:10.1093/aje/kwq291.
19.
PerssonMSolevidM. Measuring political participation – testing social desirability bias in a web-survey experiment. International Journal of Public Opinion Research2013; 26(1): 98-112. doi: 10.1093/ijpor/edt002.
20.
HolbrookALKrosnickJA. Social desirability bias in voter turnout reports: Tests using the item count technique. Public Opinion Quarterly2010; 74(1): 37-67. doi: 10.1093/poq/nfp065.
21.
MorrisonMBrownTC. Testing the effectiveness of certainty scales, cheap talk, and dissonance-minimization in reducing hypothetical bias in contingent valuation studies. Environmental and Resource Economics2009; 44(3): 307-326. doi: 10.1007/s10640-009-9287-3.
22.
DuffBHanmerMJParkWWhiteLK. Good excuses: Understanding who votes with an improved turnout question. Public Opinion Quarterly2007; 71(1): 67-90. doi: 10.1093/poq/nfl045.
23.
de RadaVDDomínguez-ÁlvarezJA. Response quality of self-administered questionnaires: A comparison between paper and web questionnaires. Social Science Computer Review2014; 32: 256-269. doi: 10.1177/0894439313508516.
24.
RevillaM. Impact of the mode of data collection on the quality of answers to survey questions depending on respondent characteristics. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique2012; 116: 44-60. doi: 10.1177/0759106312456510.
