Responsive survey design is a technique aimed at improving the efficiency or quality of surveys by using incoming data from the field to make design changes. The technique was pioneered on large national surveys, but the tools can also be applied on the smaller-scale surveys most commonly used by sociologists. We demonstrate responsive survey design in a small-scale, list-based sample survey of students on the topic of sexual misconduct. We investigate the impact of individual incentive levels and a two-phase responsive design with changes to mode of contact as approaches for limiting the potential of nonresponse bias in data from such surveys. Our analyses demonstrate that a two-phase design introducing telephone and face-to-face reminders to complete the survey can produce stronger change in response rates and characteristics of those who respond than higher incentive levels. These findings offer tools for sociologists designing smaller-scale surveys of special populations or sensitive topics.
AxinnWilliam G.LinkCynthia F.GrovesRobert M.. 2011. “Responsive Survey Design, Demographic Data Collection, and Models of Demographic Behavior.” Demography48(3):1127–49. doi: 10.1007/s13524-011-0044-1.
3.
BandillaWolfgangCouperMick P.KaczmirekLars. 2014. “The Effectiveness of Mailed Invitations for Web Surveys and the Representativeness of Mixed-mode versus Internet-only Samples.” Survey Practice7(4):1–9.
4.
BerzofskyMarcus E.LangtonLynnKrebsChristopherLindquistChristinePlantyMichael. 2019. “Methods for Improving Representativeness in a Web Survey on Sexual Assault Among College Students.” Journal of Interpersonal Violence34(23–24):4838–59. doi: 10.1177/0886260519871526.
5.
BiemerPaul P.MurphyJoeZimmerStephanieBerryChipDengGraceLewisKatie. 2018. “Using Bonus Monetary Incentives to Encourage Web Response in Mixed-mode Household Surveys.” Journal of Survey Statistics and Methodology6(2):240–61. doi: 10.1093/jssam/smx015.
6.
BrickJ. MichaelWilliamsDouglas. 2013. “Explaining Rising Nonresponse Rates in Cross-Sectional Surveys.” The ANNALS of the American Academy of Political and Social Science645(1):36–59. doi: 10.1177/0002716212456834.
CohenEmma D.2018. “Gendered Styles of Student-faculty Interaction among College Students.” Social Science Research75:1170–29. doi: 10.1016/j.ssresearch.2018.06.004.
9.
CouperMick P.2008. Designing Effective Web Surveys. New York: Cambridge University Press.
10.
CouperMick P.GremelGarretAxinnWilliamGuyerHeidiWagnerJamesWestBrady T.. 2018. “New Options for National Population Surveys: The Implications of Internet and Smartphone Coverage.” Social Science Research73:221–35. doi: 10.1016/j.ssresearch.2018.03.008.
11.
CurtinRichardPresserStanleySingerEleanor. 2005. “Changes in Telephone Survey Nonresponse over the Past Quarter Century.” Public Opinion Quarterly69(1):87–98. doi: 10.1093/poq/nfi002.
DuganJohn P.TurmanNatasha T.TorrezMark Anthony. 2015. “When Recreation Is More Than Just Sport: Advancing the Leadership Development of Students in Intramurals and Club Sports.” Recreational Sports Journal39(1):37–48. doi: 10.1123/rsj.2015-0008.
FanWeimiaoYanZheng. 2010. “Factors Affecting Response Rates of the Web Survey: A Systematic Review.” Computers in Human Behavior26(2):132–9. doi: 10.1016/j.chb.2009.10.015.
16.
FinamoreJohnReistBenjaminCoffeyStephanie. 2013. “2013 National Survey of College Graduates: A Practice-based Investigation of Adaptive Design.” Presented at the2013 American Association for Public Opinion Research (AAPOR) Annual Conference, May 16–19, Boston, MA.
17.
GrovesRobert M.CialdiniRobert B.CouperMick P.. 1992. “Understanding the Decision to Participate in a Survey.” Public Opinion Quarterly56(4):475–95. doi: 10.1086/269338.
18.
GrovesRobert M.CouperMick P.. 1998. Nonresponse in Household Interview Surveys. New York: John Wiley & Sons.
19.
GrovesRobert M.CouperMick P.PresserStanleySingerEleanorTourangeauRogerAcostaGiorgina PianiNelsonLindsay. 2006. “Experiments in Producing Nonresponse Error.” Public Opinion Quarterly70(5):720–36.
20.
GrovesRobert M.FowlerFloyd J.JrCouperMick P.LepkowskiJames M.SingerEleanorTourangeauRoger. 2004. Survey Methodology. New York: John Wiley & Sons.
21.
GrovesRobert M.HeeringaSteven G.. 2006. “Responsive Design for Household Surveys: Tools for Actively Controlling Survey Nonresponse and Costs.” Journal of the Royal Statistical Society: Series A169(3):439–57.
22.
GrovesRobert M.LybergLars. 2010. “Total Survey Error: Past, Present, and Future.” Public Opinion Quarterly74(5):849–79.
23.
GrovesRobert M.PeytchevaEmilia. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” Public Opinion Quarterly72(2):167–89. doi: 10.1093/poq/nfn011.
24.
GrovesRobert M.PresserStanleyDipkoSarah. 2004. “The Role of Topic Interest in Survey Participation Decisions.” Public Opinion Quarterly68(1):2–31.
25.
GrovesRobert M.SingerEleanorCorningAmy. 2000. “Leverage-saliency Theory of Survey Participation: Description and an Illustration.” Public Opinion Quarterly64(3):299–308. doi: 10.1086/317990.
26.
GuoGuangLiYiOwenCraigWangHongyuDuncanGreg J.. 2015. “A Natural Experiment of Peer Influences on Youth Alcohol Use.” Social Science Research52:193–207. doi: 10.1016/j.ssresearch.2015.01.002.
27.
HansenMorris H.HurwitzWilliam N.. 1946. “The Problem of Non-response in Sample Surveys.” Journal of the American Statistical Association41(236):517–29. doi: 10.1080/01621459.1946.10501894.
28.
JansMattRomanAnthony. 2007. “National Response Rates for Surveys of College Students: Institutional, Regional, and Design Factors.” Presented at the annual meeting of the American Association for Public Opinion Research, May 17-20, Anaheim, CA.
29.
KrebsChristopherLindquistChristineBerzofskyMarcusShook-SaBonniePetersonKimberlyPlantyMichaelLangtonLynnStroopJessica. 2016. Campus Climate Survey Validation Study: Final Technical Report. Washington, DC: Bureau of Justice Statistics, Office of Justice Programs. Retrieved March 23, 2021 (https://www.ojp.gov/pdffiles1/bjs/grants/249545.pdf).
30.
KreuterFraukePresserStanleyTourangeauRoger. 2008. “Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity.” Public Opinion Quarterly72(5):847–65.
31.
LundquistPeterSärndalCarl-Erik. 2013. “Aspects of Responsive Design with Applications to the Swedish Living Conditions Survey.” Journal of Official Statistics29(4):557–82. doi: 10.2478/jos-2013-0040.
32.
ManfredaKatja LozarBosnjakMichaelBerzelakJernejHaasIrisVehovarVasjaBerzelakN.. 2008. “Web Surveys versus Other Survey Modes: A Meta-Analysis Comparing Response Rates.” Journal of the Market Research Society50(1):79–104.
33.
McClintockElizabeth A.2010. “When Does Race Matter? Race, Sex, and Dating at an Elite University.” Journal of Marriage and Family72(1):45–72. doi: 10.1111/j.1741-3737.2009.00683.x.
34.
MercerAndrewCaporasoAndrewCantorDavidTownsendReanne. 2015. “How Much Gets You How Much? Monetary Incentives and Response Rates in Household Surveys.” Public Opinion Quarterly79(1):105–29. doi: 10.1093/poq/nfu059.
35.
MillarMorgan M.DillmanDon A.. 2011. “Improving Response to Web and Mixed-Mode Surveys.” Public Opinion Quarterly75(2):249–69. doi: 10.1093/poq/nfr003.
36.
National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Panel on Measuring Rape and Sexual Assault in Bureau of Justice Statistics Household Surveys, edited by KruttschnittC.KalsbeekW. D.HouseC. C.. Washington, DC: Committee on National Statistics, Division of Behavioral and Social Sciences and Education, The National Academies Press.
37.
PeytchevAndy. 2013. “Consequences of Survey Nonresponse.” The ANNALS of the American Academy of Political and Social Science645(1):88–111. doi: 10.1177/0002716212461748.
38.
PeytchevAndyBaxterRodneyCarley-BaxterLisa. 2009. “Not All Survey Effort is Equal: Reduction of Nonresponse Bias and Nonresponse Error.” Public Opinion Quarterly73(4):785–806. doi: 10.1093/poq/nfp037.
39.
PeytchevAndyPeytchevEmiliaGrovesRobert M. 2010. “Measurement Error, Unit Nonresponse, and Self-Reports of Abortion Experience.” The Public Opinion Quarterly74(2):319–27.
40.
RooseHenkLievensJohnWaegeHans. 2007. “The Joint Effect of Topic Interest and Follow-up Procedures on the Response in a Mail Questionnaire: An Empirical Test of the Leverage-saliency Theory in Audience Research.” Sociological Methods & Research35(3):410–28. doi: 10.1177/0049124106290447.
41.
RyuEricaCouperMick P.MaransRobert W.. 2006. “Survey Incentives: Cash vs. In-kind; Face-to-face vs. Mail; Response Rate vs. Nonresponse Error.” International Journal of Public Opinion Research18(1):89–106. doi: 10.1093/ijpor/edh089.
42.
SarrafShimon A.ColeJames S.. 2014. “Survey Lottery Incentives and Institutional Response Rates: An Exploratory Analysis.” Paper presented at theAssociation for Institutional Research Annual Forum, May, Orlando, FL.
43.
SchoutenBarryCobbenFannieLundquistPeterWagnerJames. 2016. “Does More Balanced Survey Response Imply Less Non-response Bias?” Journal of the Royal Statistical Society: Series A (Statistics in Society)179(3):727–48. doi: 10.1111/rssa.12152.
44.
ShihTse-HuaFanXitao. 2008. “Comparing Response Rates from Web and Mail Surveys: A Meta-Analysis.” Field Methods20(3):249–71.
45.
SingerEleanorCouperMick P.. 2008. “Do Incentives Exert Undue Influence on Survey Participation? Experimental Evidence.” Journal of Empirical Research on Human Research Ethics3(3):49–56.doi: 10.1525/jer.2008.3.3.49.
46.
SingerEleanorYeCong. 2013. “The Use and Effects of Incentives in Surveys.” The ANNALS of the American Academy of Political and Social Science645(1):112–41. doi: 10.1177/0002716212458082.
47.
SmithSandra S.MooreMignon R.. 2000. “Intraracial Diversity and Relations among African-Americans: Closeness among Black Students at a Predominantly White University.” American Journal of Sociology106(1):1–39.
48.
TourangeauRogerConradFrederick G.CouperMick P.. 2013. The Science of Web Surveys. New York: Oxford University Press.
49.
TourangeauRogerYanTing. 2007. “Sensitive Questions in Surveys.” Psychological Bulletin133(5):859–83.
50.
UeckerJeremy E.2015. “Social Context and Sexual Intercourse among First-year Students at Selective Colleges and Universities in the United States.” Social Science Research52: 59–71. doi: 10.1016/j.ssresearch.2015.01.005.
51.
WagnerJamesWestBrady T.KirgisNicoleLepkowskiJames M.AxinnWilliam G.Kruger NdiayeShonda. 2012. “Use of Paradata in a Responsive Design Framework to Manage a Field Data Collection.” Journal of Official Statistics28(4):477–499.
52.
White House Council on Women and Girls. 2014. Rape and Sexual Assault: A Renewed Call to Action. Washington, DC: The White House.
53.
WilliamsDouglasBrickJ. Michael. 2018. “Trends in U.S. Face-to-Face Household Survey Nonresponse and Level of Effort.” Journal of Survey Statistics and Methodology6(2):186–211. doi: 10.1093/jssam/smx019.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
