Web surveys are being increasingly incorporated into national survey data collection programs in the United States because of their cost/time-efficiencies. Yet, response rates and data quality issues in web surveys remain important challenges. As a basic study designed to better understand data quality in a mixed mode national survey, this article investigates the degree to which web versus mail survey modes affect unit and item responses. Findings indicate that the web survey mode produces a lower unit response rate compared to the mail mode. However, the web mode elicits higher data quality in terms of item responses to both closed- and open-ended questions. These mode effects on data quality remain after sociodemographic variables are held constant. Given the increasing integration of web survey questionnaires into mixed mode studies, additional research is necessary to understand and document the processes that underlie mode differences when responding to self-administered surveys.
AklE. A.MarounN.KlockeR. A.MontoriV.SchunemannH. J. (2005). Electronic mail was not better than postal mail for surveying residents and faculty. Journal of Clinical Epidemiology, 58, 425–429.
2.
American Association for Public Opinion Research. (2011). Standard definitions: Final disposition case codes and outcome rates for surveys. (7th ed.). AAPOR.
3.
BachmannD.ElfrinkJ.VazzanaG. (1996). Tracking the progress of e-mail versus snail-mail. Marketing Research, 8, 31–35.
4.
BandillarW.BosnjakM.AltdorferP. (2003). Survey administration effects? A comparison of web-based and traditional written self-administrated surveys using the ISSP environment module. Social Science Computer Review, 21, 235–243.
5.
BechM.KristensenM. B. (2009). Differential response rates in postal and web-based surveys among older respondents. Survey Research Methods, 3, 1–6.
6.
BeebeT. J.LockeG. R. III, Barnes, S. A.DavernM. E.AndersonK. J. (2007). Mixing web and mail methods in a survey of physicians. Health Services Research, 42, 1219–1234.
7.
BörkanB. (2010). The mode effect in mixed-mode surveys mail and web surveys. Social Science Computer Review, 28, 371–380.
8.
ChristianL. M.DillmanD. A. (2004). The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly, 68, 57–80.
9.
ConverseP. D.WolfeE. W.HuangX.OswaldF. L. (2008). Response rates for mixed-mode surveys using mail and e-mail/web. American Journal of Evaluation, 29, 99–107.
10.
CouperM. P. (2000). Web Surveys: A review of issues and approaches. Public Opinion Quarterly, 64 (4), 464–494.
11.
CouperM. P. (2008). Designing effective web surveys. New York, NY: Cambridge University Press.
12.
CouperM. P.BlairJ.TriplettT. (1999). A comparison of mail and e-mail for a survey of employees in U.S. statistics agencies. Journal of Official Statistics, 15, 39–56.
13.
CouperM. P.MillerP. V. (2008). Web survey method: Introduction. Public Opinion Quarterly, 72, 831–835.
14.
DasM.EsterP.KaczmirekL. (2011). Social and behavioral research and the internet: Advances in applied methods and research strategies. New York, NY: Routledge.
15.
De LeeuwE. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21, 233–255.
16.
DenscombeM. (2006). Web-based questionnaires and the mode effect: An evaluation based on completion rates and data contents of near-identical questionnaires delivered in different modes. Social Science Computer Review, 24, 246–254.
17.
DenscombeM. (2008). The length of responses to open-ended questions: A comparison of online and paper questionnaires in terms of a mode effect. Social Science Computer Review, 26, 359–368.
18.
DenscombeM. (2009). Item non-response rates: A comparison of online and paper questionnaires. International Journal of Social Research Methodology, 12, 281–291.
19.
DillmanD. A.SmythJ. D.ChristianM. E. (2009a). Internet, mail and mixed-mode surveys: The tailored design method. Hoboken, NJ: Wiley.
20.
DillmanD. A.PhelpsG.TortoraR.SwiftK.KohrellJ.BerckJ.MesserB. (2009b). Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the internet. Social Science Research, 38, 1–18.
21.
GreenlawC.Brown-WeltyS. (2008). A comparison of web-based and paper-based survey methods: Testing assumptions of survey mode and response cost. Evaluation Review, 33, 464–480.
22.
GrondinC.SunL. (2009). 2006 Census Internet Mode Effect Study. In In 2008 Proceedings of the section on survey research methods (pp. 1255–1262). Alexandria, VA: American Statistical Association.
23.
HeerweghD.LoosveldtG. (2008). Face-to-face versus web surveying in a high internet coverage population: Differences in response quality. Public Opinion Quarterly, 72, 836–846.
24.
KaplowitzM. D.HadlockT. D.LevineR. (2004). A comparison of web and mail survey response rates. Public Opinion Quarterly, 68, 94–101.
25.
KnappH.KirkS. A. (2003). Using pencil and paper, internet and touch-tone phones for self-administered surveys: Does methodology matter?Computers in Human Behavior, 19, 117–134.
26.
KwakN.RadlerB. (2002). A comparison between mail and web surveys: Response pattern, respondent profile, and data quality. Journal of Official Statistics, 8, 257–273.
27.
LinkM. W.MokdadA. H. (2005). Effects of survey mode on self-reports of adult alcohol consumption: Comparison of web, mail, and telephone approaches. Journal of Studies on Alcohol, 66, 239–245.
28.
LinkM. W.MokdadA. H. (2006). Can web and mail survey modes improve participation in an RDD-based national health surveillance?Journal of Official Statistics, 22, 293–312.
29.
LorencB. (2010). Item nonresponse analysis for a mixed-mode survey, in CarlsonM.NyquistH.VillaniM. (eds), Official Statistics – Methodology and Applications in Honour of Daniel Thorburn (pp. 117–136). Stockholm, Sweden: Stockholm University. Retrieved fromhttp://officialstatistics.wordpress.com
30.
ManfredaL.BosnjakK. M.BerzelakJ.HaasI.VehovarV. (2008). Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research, 50, 79–104.
31.
McCabeS. E. (2004). Comparison of web and mail surveys in collecting illicit drug use data: A randomized experiment. Journal of Drug Education, 34, 61–72.
32.
McCabeS. E.BoydC. J.CouperM. P.CrawfordS.d’ArcyH. (2002). Mode effects for collecting alcohol and other drug use data: Web and U.S. mail. Journal of Studies on Alcohol, 63, 755–761.
33.
McCabeS. E.DiezA.BoydC. J.NelsonT. F.WeitzmanE. R. (2006). Comparing web and mail responses in a mixed mode survey in college alcohol use research. Addictive Behaviors, 31, 1619–1627.
34.
MillerT. I.KobayashiM. M.CaldwellE.ThurstonS.CollettB. (2002). Citizen surveys on the web: General population surveys of community opinion. Social Science Computer Review, 20, 124–136.
35.
RaoK.KaminskaO.MccutcheonA. (2010). Recruiting probability samples of a multi-mode research panel with internet and mail components. Public Opinion Quarterly, 74, 68–84.
36.
RosterC. A.RogersR. D.AlbaurnG.KleinJ. D. (2004). A comparison of response characteristics from web and telephone surveys. International Journal of Market Research, 46, 359–373.
37.
SchaeferD.DillmanD. (1998). Development of a standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62, 378–397.
38.
SchonlauM.AschB.DuC. (2003). Web surveys as part of a mixed-mode strategy for populations that cannot be contacted by e-mail. Social Science Computer Review, 21, 218–222.
39.
ShihT.FanX. (2008). Comparing response rates from web and mail surveys: A meta-analysis. Field Methods, 20, 249–271.
40.
SmythJ. D.DillmanD. A.ChristianL. M.McbrideM. (2009). Open-ended questions in web surveys: Can increasing the size of answer boxes and providing extra verbal instructions improve response quality?Public Opinion Quarterly, 73, 325–337.
41.
WolfeE. W.ConverseP. D.OswaldF. L. (2008). Item-level non-response rates in an attitudinal survey of teachers delivered via mail and web. Journal of Computer-Mediated Communication, 14, 35–66.
