A User-Driven Method for Using Research Products to Empirically Assess Item Importance in National Surveys

Abstract

Large-scale, nationally representative surveys serve many vital functions, but these surveys can be long and burdensome for respondents. Cutting survey length can help to reduce respondent burden and may improve data quality but removing items from these surveys is not a trivial matter. We propose a method to empirically assess item importance and associated burden in national surveys and guide this decision-making process using different research products produced from such surveys. This method is demonstrated using the Survey of Doctorate Recipients (SDR), a biennial survey administered to individuals with a science, engineering, and health doctorate. We used three main sources of information on the SDR variables: a bibliography of documents using the SDR data as a measure of item use and importance, SDR data table download statistics from the Scientists and Engineers Statistical Data System as an additional measure of item use, and web timing paradata and break-off rates as a measure of burden. Putting this information together, we identified 35 unused items (17% of the survey) and found that the most burdensome items are highly important. We conclude with general recommendations for those hoping to employ similar methodologies in the future.

Keywords

Respondent burden public use data survey length

References

Andreadis, I., and E., Kartsounidou. 2020. “The impact of splitting a long online questionnaire on data quality.” Survey Research Methods 14 (1): 31–42. DOI: https://doi.org/10.18148/srm/2020.v14i1.7294.

Bradburn, N. 1978. “Respondent Burden.” In Proceedings of the Survey Research Methods Section of the American Statistical Association, American Statistical Association Alexandria, VA. Available at: http://www.asasrms.org/Proceedings/papers/1978_007.pdf (accessed December 2020).

Coleman-Jensen, Al., M.P. Rabbitt, P. Matthew, C.G. Gregory, and A. Singh. 2020. “Household Food Security in the United States in 2019”. ERR-275, U.S. Department of Agriculture, Economic Research Service. Available at: https://www.ers.usda.gov/publications/pub-details/?pubid=99281 (accessed December 2020).

Corley, E.A., and M. Sabharwal. 2007. “Foreign-born academic scientists and engineers: producing more and getting less than their US-born peers?” Research in Higher Education 48 (8): 909–940. DOI: https://doi.org/10.1007/s11162-007-9055-6.

De Heer, W., and E. de Leeuw. 2002. “Trends in household survey nonresponse: A longitudinal and international comparison.” Survey nonresponse 41: 41–54. New York: Wiley.

Deutskens, E., K. de Ruyter, M. Wetzels, and P. Oosterveld. 2004. “Response rate and response quality of internet-based surveys: An experimental study.” Marketing Letters 15(1): 21–36. Available at: https://www.jstor.org/stable/40216512 (accessed December 2020).

Foundations for Evidence-Based Policymaking Act of 2018. Available at: https://www.congress.gov/bill/115th-congress/house-bill/4174 (accessed January 2021).

Galesic, M., and M. Bosnjak. 2009. “Effects of questionnaire length on participation and indicators of response quality in a web survey.” Public Opinion Quarterly 73(2): 349–360. DOI: https://doi.org/10.1093/poq/nfp031.

Luiten, A., J. Hox, and E. de Leeuw. 2020. “Survey Nonresponse Trends and Fieldwork Effort in the 21st Century: Results of an International Study across Countries and Surveys”, Journal of Official Statistics 36(3): 469–487. DOI: https://doi.org/10.2478/jos-2020-0025.

10.

Moguérou, P. 2002. “Job satisfaction among US Ph. D. graduates: the effects of gender and employment sector.” Labor and Demography EconWPA. Available at: https://econpapers.repec.org/paper/wpawuwpla/0204002.htm (accessed December 2020).

11.

NCSES (Center for Science and Engineering Statistics), a. “Survey of Doctorate Recipients.” Available at: https://www.nsf.gov/statistics/srvydoctoratework/ (accessed December 2020).

12.

NCSES (Center for Science and Engineering Statistics), b. “Scientists and Engineers Statistical Data System Data Tool.” Available at: https://ncsesdata.nsf.gov/sestat/ (accessed December 2020).

13.

Peytchev, A. 2009. “Survey break-off.” Public Opinion Quarterly 73(1): 74–97. DOI: https://doi.org/10.1093/poq/nfp014.

14.

Peytchev, A., and E. Peytcheva. 2017. “Reduction of measurement error due to survey length: Evaluation of the split questionnaire design approach.” Survey Research Methods 11(4): 361–368. DOI: 10.18148/srm/2017.v11i4.7145.

15.

Peytchev, A., E. Peytcheva, J.G. Conzelmann, A. Wilson, and J. Wine. 2020. “Modular survey design: Experimental manipulation of survey length and monetary incentive structure.” Journal of Survey Statistics and Methodology 8(2): 370–384. DOI: 10.1093/jssam/smz006.

16.

Schoenborn, C.A., P.F. Adams, P M. Barnes, J.L. Vickerie, and J S. Schiller. 2004. “Health behaviors of adults; United States, 1999–2001.” Available at: https://stacks.cdc.gov/view/cdc/7007 (accessed December 2020).

17.

The Paperwork Reduction Act. 1995. 44 U.S.C. chap. 1 § 101; 44 U.S.C. chap. 35, subchap. I § 3501 et seq. Available at: https://uscode.house.gov/view.xhtml?path=/prelim@title44/chapter35&edition=prelim (accessed December 2020).

18.

Toepoel, V., and P. Lugtig. 2018 “Modularization in an Era of Mobile Web: Investigating the Effects of Cutting a Survey Into Smaller Pieces on Data Quality.” Social Science Computer Review 40 (1). DOI: https://doi.org/10.1177/0894439318784882.

19.

U.S. Bureau of Labor Statistics. 2021. The employment situation – December 2020. Available at: https://www.bls.gov/news.release/pdf/empsit.pdf (accessed Jan 2021).

20.

West, B.T., D. Ghimire, and W. Axinn. 2015. “Evaluating a Modular Design Approach to Collecting Survey Data Using Text Messages.” Survey Research Methods 9 (2): 111–123. DOI: https://doi.org/10.18148/srm/2015.v9i2.6135.

21.

Williams, D, and J.M. Brick. 2018. “Trends in US face-to-face household survey nonresponse and level of effort.” Journal of Survey Statistics and Methodology 6 (2) 2: 186–211. DOI: https://doi.org/10.1093/jssam/smx019.