Self-administered surveys may be administered with a single mode or mixed data collection modes. How mixing modes of data collection affects survey costs is not well understood. We examine whether cost structures differ for mail-only versus web+mail mixed-mode surveys, what design features are associated with costs, and whether survey costs are associated with response rates. Using administrative survey cost data from two academic survey centers, we find that survey costs per sampled unit and per complete vary substantially across individual surveys. The average cost per sampled unit is surprisingly similar across mail-only and web+mail surveys. How the budget is allocated across printing, postage, incentive, and staff time varies across these designs: printing and postage costs are higher in mail-only surveys, and more of the budget is allocated to incentive costs and project management costs in web+mail surveys. Furthermore, higher cost surveys are associated with higher response rates, particularly for incentive costs.
American Association for Public Opinion Research. 2023. “Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.” American Association for Public Opinion Research.
2.
BiemerPaul P.MurphyJoeZimmerStephanieBerryChipDengGraceLewisKatie. 2018. “Using Bonus Monetary Incentives to Encourage Web Response in Mixed-Mode Household Surveys.” Journal of Survey Statistics and Methodology6(2):240‐61.
3.
BrickJ. MichaelTourangeauRoger. 2017. “Responsive Survey Designs for Reducing Nonresponse Bias.” Journal of Official Statistics33(3):735‐52.
4.
DaikelerJessicaBošnjakMichaelManfredaKatja Lozar. 2020. “Web Versus Other Survey Modes: An Updated and Extended Meta-Analysis Comparing Response Rates.” Journal of Survey Statistics and Methodology8(3):513‐39.
5.
de LeeuwEdith. 2005. “To Mix or Not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics21(2):233‐55.
6.
DelnevoCristine D.SinghBinu. 2021. “The Effect of a Web-Push Survey on Physician Survey Responses Rates: A Randomized Experiment.” Survey Practice14:1. no pp.
7.
DillmanDon A.SmythJolene D.ChristianLeah Melani. 2014. Internet, Phone, Mail, and Mixed Mode Surveys: The Tailored Design Method. Hoboken: John Wiley & Sons.
8.
DutwinDavidBuskirkTrent D.. 2020. “Telephone Sample Surveys: dearly Beloved or Nearly Departed? Trends in Survey Errors in the Era of Declining Response Rates.” Journal of Survey Statistics and Methodology9(3):353‐80.
9.
DykemaJenniferStevensonJohnAssadNadiaKnissChadTaylorCatherine A.. 2021. “Effects of Sequential Prepaid Incentives on Response Rates, Data Quality, Sample Representativeness, and Costs in a Mail Survey of Physicians.” Evaluation & the Health Professions44:235‐44.
10.
DykemaJenniferStevensonJohnKleinLisaKimYujinDayBrendan. 2013. “Effects of E-Mailed Versus Mailed Invitations and Incentives on Response Rates, Data Quality, and Costs in a Web Survey of University Faculty.” Social Science Computer Review31(3):359‐70.
11.
GelmanAndrewStevensMattChanValerie. 2003. “Regression Modeling and Meta-Analysis for Decision Making.” Journal of Business & Economic Statistics21(2):213‐25.
12.
GreenlawCoreyBrown-WeltySharon. 2009. “A Comparison of Web-Based and Paper-Based Survey Methods: Testing Assumptions of Survey Mode and Response Cost.” Evaluation Review33(5):464‐80.
13.
GriffisStanley E.GoldsbyThomas J.CooperMartha. 2003. “Web-Based and Mail Surveys: A Comparison of Response, Data, and Cost.” Journal of Business Logistics24(2):237‐58.
14.
GrovesRobert M. 1989. Survey Errors and Survey Costs. New York: John Wiley & Sons, Inc.
15.
GrovesRobert M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly70(5):646‐75.
16.
GrovesRobert M.PeytchevaEmilia. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” Public Opinion Quarterly72(2):167‐89.
17.
HanDaifengMontaquilaJill M.Michael BrickJ.. 2013. “An Evaluation of Incentive Experiments in a Two-Phase Address-Based Sample Mail Survey.” Survey Research Methods7(3):207‐18.
18.
HarterRachelBattagliaMichael P.BuskirkTrent D.DillmanDon A.EnglishNedFahimiMansourFrankelMartin R.KennelTimothyMcMichaelJoseph P.McPheeCameronMontaquilaJill M.YanceyTraceyZukerbergAndrew. 2016. “Task Force Report on Address-Based Sampling.” American Association for Public Opinion Research. Available from https://aapor.org/wp-content/uploads/2022/11/AAPOR_Report_1_7_16_CLEAN-COPY-FINAL-2.pdf
19.
JudkinsDavidWaksbergJosephNorthrupDoris. 1990. “Cost Functions for NHIS and Implications for Survey Design.” Proceedings of the American Statistical Association, Survey Research Methods Section: 34‐43. Available from http://www.asasrms.org/Proceedings/papers/1990_006.pdf.
20.
KaplowitzMichael D.HadlockTimothy D.LevineRalph. 2004. “A Comparison of Web and Mail Survey Response Rates.” Public Opinion Quarterly68(1):94‐101.
21.
LinkMichael W.BattagliaMichael P.FrankelMartin R.OsbornLarryMokdadAli H.. 2008. “A Comparison of Address-Based Sampling (ABS) Versus Random-Digit Dialing (RDD) for General Population Surveys.” Public Opinion Quarterly72(1):6‐27.
22.
LongJ. ScottFreeseJeremy. 2014. Regression Models for Categorical Dependent Variables Using Stata. 3rd ed. College Station: Stata Press.
23.
LuitenAnnemiekeHoxJoopde LeeuwEdith. 2020. “Survey Nonresponse Trends and Fieldwork Effort in the 21st Century: Results of an International Study Across Countries and Surveys.” Journal of Official Statistics36(3):469‐87.
24.
McMasterHope SeibLeardMannCynthia A.SpeigleStevenDillmanDon A and Team Millennium Cohort Family Study. 2017. “An Experimental Comparison of Web-Push vs. Paper-Only Survey Procedures for Conducting an In-Depth Health Survey of Military Spouses.” BMC Medical Research Methodology17(1):73.
25.
OlsonKristen. 2021. “Unpacking the Black Box of Survey Costs.” Research in Social and Administrative Pharmacy17(7):1342‐46.
26.
OlsonKristenSmythJolene DHorwitzRachelKeeterScottLesserVirginiaMarkenStephanieMathiowetzNancy AMcCarthyJaki SO’BrienEileenOpsomerJean DSteigerDarbySterrettDavidSuJenniferSuzer-GurtekinZ TubaTurakhiaChintanWagnerJames. 2021. “Transitions From Telephone Surveys to Self-Administered and Mixed-Mode Surveys: Aapor Task Force Report.” Journal of Survey Statistics and Methodology9(3):381‐411.
27.
OlsonKristenWagnerJamesAndersonRaeda. 2021. “Survey Costs: Where are We and What is the Way Forward?” Journal of Survey Statistics and Methodology9(5):921‐42.
28.
OlsonKristenStevensonJohnAssadNadiaWitt-SwansonLindseyJonesCameron P.E.GanshertAmandaDykemaJennifer. 2024. Data and Stata Code for “Examining Variation in Survey Costs across Surveys”. doi: https://doi.org/10.17605/OSF.IO/VR43W.
29.
PresserStanleyMcCullochSusan. 2011. “The Growth of Survey Research in the United States: Government-Sponsored Surveys, 1984–2004.” Social Science Research40(4):1019‐24.
30.
RobertsCarolineVandenplasCaroline. 2017. “Estimating Components of Mean Squared Error to Evaluate the Benefits of Mixing Data Collection Modes.” Journal of Official Statistics33(2):303.
31.
SinclairMarthaO’TooleJoanneMalawaraarachchiManoriLederKarin. 2012. “Comparison of Response Rates and Cost-Effectiveness for a Community-Based Survey: Postal, Internet and Telephone Modes With Generic or Personalised Recruitment Approaches.” BMC Medical Research Methodology12(1):132.
32.
SingerEleanorYeCong. 2013. “The Use and Effects of Incentives in Surveys.” The Annals of the American Academy of Political and Social Science645(1):112‐41.
33.
TourangeauRogerConradFrederick G.CouperMick P.. 2013. The Science of Web Surveys. New York: Oxford University Press.
34.
WagnerJames. 2019. “Estimation of Survey Cost Parameters Using Paradata.” Survey Practice12:1. no pp.
35.
WagnerJamesGuyerHeidiEvanchekChrissy. 2021. “Using Time Series Models to Understand Survey Costs.” Journal of Survey Statistics and Methodology9(5): 943-60.
36.
ZuidgeestMarloesHendriksMichelleKoopmanLauraSpreeuwenbergPeterRademakersJany. 2011. “A Comparison of a Postal Survey and Mixed-Mode Survey Using a Questionnaire on Patients’ Experiences With Breast Care.” Journal of Medical Internet Research13(3):e68.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
