Testing a Planned Missing Design to Reduce Respondent Burden in Web and SMS Administrations of the CAHPS Clinician and Group Survey (CG-CAHPS)

Abstract

We test a planned missing design to reduce respondent burden in Web and SMS administrations of the CAHPS Clinician and Group Survey (CG-CAHPS), a survey of patient experiences widely used by health care providers. Members of an online nonprobability panel were randomly assigned to one of three invitation and data collection mode protocols: email invitation to a Web survey, SMS invitation to a Web survey, or SMS invitation to an SMS survey. Within these three mode protocols, respondents were randomly assigned to a planned missing design, which shortened the survey by about 40%, or to a control group that received the survey in its entirety. We compare survey duration, breakoff and completion rates, and five key patient experience measures across conditions to assess the effect of the planned missing design across the three modes. We found that a planned missing design worked well with our Web survey, reducing survey duration and breakoff without changing estimates relative to the full-survey control condition. However, mixed findings in the SMS survey suggest that even shortened, 15-item surveys may be too long to substantially reduce respondent burden. We conclude with recommendations for future research.

Keywords

Online data collection text messaging health

References

Ahmad, F., A.K. Jhajj, D.E. Stewart, M. Burghardt, and A.S. Bierman. 2014. “Single Item Measures of Self-Rated Mental Health: A Scoping Review.” BMC Health Services Research 14: 398. DOI: https://doi.org/10.1186/1472-6963-14-398.

AHRQ, Agency for Healthcare Research and Quality. n.d. “How to Report Results of the CAHPS Clinician & Group Survey.” Available at: https://www.ahrq.gov/sites/default/-files/wysiwyg/cahps/surveys-guidance/cg/cgkit/HowtoReportResultsofCGCAHPS080610FINAL.pdf (accessed September 2021).

Alemi, F., and H.M. Jasper. 2014. “An Alternative to Satisfaction Surveys: Let the Patients Talk.” Quality Management in Health Care 23(1): 10–19. DOI: https://doi.org/10.1097/QMH.0000000000000014.

Antoun, C., and A. Cernat. 2020. “Factors Affecting Completion Times: A Comparative Analysis of Smartphone and PC Web Surveys.” Social Science Computer Review 38(4): 477–489. DOI: https://doi.org/10.1177/0894439318823703.

Ayers, R., S. Brothers, and W. Jansen. 2014. “Guidelines on Mobile Device Forensics.” National Institute of Standards and Technology, Special Publication 800-101r1. Washington, DC: US Department of Commerce. DOI: http://dx.doi.org/10.6028/NIST.Sp.800-101r1.

Blumberg, S.J., and J.V. Luke. 2020. “Wireless substitution: Early release of estimates from the National Health Interview Survey, July-December 2019.” National Center for Health Statistics. Available at: https://www.cdc.gov/nchs/nhis.htm. (accessed September 2021).

Bosnjak, M., W. Neibarth, M.P. Couper, W. Bandilla, and L. Kaczmirek. 2008. “Prenotification in Web-Based Access Panel Surveys.” Social Science Computer Review 26(2): 213–223. DOI: https://doi.org/10.1177/0894439307305895.

Bradburn, N. 1978. “Respondent Burden.” In Proceedings of the Survey Research Methods Section of the American Statistical Association: 35–40. Alexandria, VA: American Statistical Association. Available at: http://www.asasrms.org/Proceedings/y1978f.html (accessed September 2021).

Brenner, P.S., and J.D. DeLamater. 2013. “Paradata Correlates of Data Quality in an SMS Time Use Study–Evidence from a Validation Study.” Electronic International Journal of Time Use Research 10(1): 38–54. DOI: dx.doi.org/10.13085/eIJTUR.10.1.38-54.

10.

Conrad, FG., M.P. Couper, R. Tourangeau, and C. Zhang. 2017. “Reducing Speeding in Web Surveys by Providing Immediate Feedback.” Survey Research Methods 11(1): 45–61. DOI: https://doi.org/10.18148/srm/2017.v11i1.6304.

11.

Crawford, S.D., M.P. Couper, and M.J. Lamias. 2001. “Web Surveys: Perception of Burden.” Social Science Computer Review 19(2): 146–162. DOI: https://doi.org/10.1177/089443930101900202.

12.

De Leeuw, E.D. 2018. “Mixed-Mode: Past, Present, and Future.” Survey Research Methods 12(2): 75–89. DOI: doi: 10.18148/srm/2018.v12i2.7402.

13.

DeSalvo, K.B., V.S. Fan, M.B. McDonell, and S.D. Fihn. 2004. “Predicting Mortality and Healthcare Utilization with a Single Question.” Health Services Research 40(4): 1234–1246. DOI: https://doi.org/10.1111/j.1475-6773.2005.00404.x.

14.

Dillman, D.A., J.D. Smyth, and L.M. Christian. 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. New York: Wiley.

15.

Dyer, N., J.S. Sorra, S.A. Smith, P. Cleary, and R. Hays. 2012. “Psychometric Properties of the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group Adult Visit Survey.” Medical Care 50: S28. DOI: https://doi.org/10.1097/MLR.0b013e31826cbc0d.

16.

Elliott M.N., M.K. Beckett, W.G. Lehrman, P. Cleary, C.W. Cohea, L.A. Giordano, E.H. Goldstein, and C.L. Damberg. 2016. “Understanding The Role Played By Medicare’s Patient Experience Points System In Hospital Reimbursement.” Health Affairs 35(9): 1673–1680. DOI: https://doi.org/10.1377/hlthaff.2015.0691.

17.

Elliott, M.N., A.M. Zaslavsky, E. Goldstein, W. Lehrman, K. Hambarsoomians, M.K. Beckett, and L. Giordano. 2009. Effects of survey mode, patient mix, and nonresponse on CAHPSw hospital survey scores. Health Services Research, 44 (2p1): 501–518. DOI: https://doi.org/10.1111/j.1475-6773.2008.00914.x.

18.

Enders, C.K. 2010. Applied Missing Data Analysis. New York: Guilford.

19.

Galesic, M., and M. Bosnjak. 2009. “Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey.” Public Opinion Quarterly 73(2): 349–360. DOI: https://doi.org/10.1093/poq/nfp031.

20.

Garrido, T., B. Raymond, and B. Wheatley. 2016. “Lessons From More than a Decade In patient Portals.” Health Affairs Blog. DOI: https://doi.org/10.1377/hblog20160407.054362.

21.

Graham, J.W., B.J. Taylor, A.E. Olchowski, and P.E. Cumsille. 2006. “Planned Missing Data Designs in Psychological Research.” Psychological Methods 11(4): 323–343. DOI: https://doi.org/10.1037/1082-989X.11.4.323.

22.

Hillygus, D.S., N. Jackson, and M. Young. 2014. “Professional respondents in non-probability online panels.” In Online panel research: A data quality perspective, edited by M. Callegaro, R. Baker, J. Bethlehem, A.S. Góritz, J.A. Krosnick and P.J. Lavrakas: 219–237. New York: Wiley.

23.

Holt, J.M. 2019. “Patient Experience in Primary Care: A Systematic Review of CGCAHPS Surveys.” Journal of Patient Experience 6(2): 93–102. DOI: https://doi.org/10.1177/2374373518793143.

24.

Johnson, D.R., V. Roth, and R. Young. 2013. “Planned Missing Data Designs in Health Surveys.” In Proceedings of the Tenth Conference on Health Survey Research Methods, November 2013: 43–53. Hyattsville, MD: Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics. Available at: https://www.cdc.gov/nchs/data/hsrmc/hsrmc_10th_proceedings_2011.pdf (accessed February 2020).

25.

Keller, S., A.J. O’Malley, R.D. Hays, R.A. Matthew, A.M. Zaslavsky, K.A. Hepner, and P.D. Cleary. 2005. “Methods Used to Streamline the CAHPSw Hospital Survey.” Health Services Research, 40: 2057–2077. DOI: https://doi.org/10.1111/j.1475-6773.2005.00478.x.

26.

Kim, Y., J. Dykema, J. Stevenson, P. Black, and D.P. Moberg. 2019. “Straightlining: Overview of Measurement, Comparison of Indicators, and Effects in Mail–Web Mixed-Mode Surveys.” Social Science Computer Review 37(2): 214–233. DOI: https://doi.org/10.1177/0894439317752406.

27.

Lau, C.Q., H. Sanders, and A. Lombaard. 2019. “Questionnaire Design in Short Message Service (SMS) Surveys.” Field Methods 31(3): 214–229. DOI: https://doi.org/10.1177/1525822X19862506.

28.

Lee, S.S.S., X. Xin, W.P. Lee, E.J. Sim, B. Tan, M.P.G. Bien, A.S.T. Lau, and J. Thumboo. 2013. “The Feasibility of Using SMS as a Health Survey Tool: An Exploratory Study in Patients with Rheumatoid Arthritis.” International Journal of Medical Informatics 82(5): 427–434. DOI: https://doi.org/10.1016/j.ijmedinf.2012.12.003.

29.

Mavletova, A. and M.P. Couper. 2014. “Mobile Web Survey Design: Scrolling versus Paging, SMS versus E-mail Invitations.” Journal of Survey Statistics and Methodology 2: 498–518. DOI: https://doi.org/10.1093/jssam/smu015.

30.

Mavletova, A., and M.P. Couper. 2015. “A Meta-Analysis of Breakoff Rates in Mobile Web Surveys.” In Mobile Research Methods: Opportunities and Challenges of Mobile Research Methodologies, edited by D. Toninelli, R. Pinter, and P. de Pedraza: 81–98. London: Ubiquity Press.

31.

Mavletova, A., M.P. Couper, and D. Lebedev. 2018. “Grid and Item-by-Item Formats in PC and Mobile Web Surveys.” Social Science Computer Review 36(6):647-668. DOI: https://doi.org/10.1177/0894439317735307.

32.

Morgan, S.L., and E.S. Taylor Poppe. 2015. “A Design and a Model For Investigating the Heterogeneity of Context Effects in Public Opinion Surveys.” Sociological Methodology 45(1): 184–222. DOI: https://doi.org/10.1177/0081175015570095.

33.

Ng, J.H., E. Henry, T. Oberlander, P. Shi, S.H. Scholle. 2016. “Shortening a Patient Experiences Survey for Medical Homes.” Healthcare 4:1. DOI: https://doi.org/10.3390/healthcare4010001.

34.

Paddison, C.A.M., G.A. Abel, M.O. Roland, M.N. Elliott, G. Lyratzopoulos, and J.L. Campbell. 2013. “Drivers of Overall Satisfaction with Primary Care: Evidence from the English General Practice Patient Survey.” Health Expectations 18: 1081-1092. DOI: https://doi.org/10.1111/hex.12081.

35.

Pew Research Center. 2019. Mobile Technology and Home Broadband. Available at: https://www.pewresearch.org/internet/wp-content/uploads/sites/9/2019/06/PI_2019.06.13_Mobile-Technology-and-Home-Broadband_FINAL2.pdf. (accessed September 2021).

36.

Peytchev, A. 2009. “Survey Breakoff.” Public Opinion Quarterly 73(1): 74–97. DOI: https://doi.org/10.1093/poq/nfp014.

37.

Peytchev, A. and E. Peytcheva. 2017. “Reduction of Measurement Error due to Survey Length: Evaluation of the Split Questionnaire Design Approach.” Survey Research Methods 11(4): 361–368. DOI: https://doi.org/10.18148/srm/2017.v11i4.7145.

38.

Revilla, M. and C. Ochoa. 2017. “Ideal and Maximum Length for a Web Survey.” International Journal of Market Research 59(5): 557–565. DOI: https://doi.org/10.2501/IJMR-2017-039.

39.

Rhemtulla, M., V. Savalei, and T.D. Little. 2016. “On the Asymptotic Relative Efficiency of Planned Missingness Designs.” Psychometrika 81(1): 60–89. DOI: https://doi.org/10.1007/s11336-014-9422-0.

40.

Schober, M.F., F.G. Conrad, C. Antoun, P. Ehlen, S. Fail, A.L. Hupp, M. Johnston, L. Vickers, H.Y. Yan, and C. Zhang. 2015. “Precision and Disclosure in Text and Voice Interviews on Smartphones.” PLoSONE 10. DOI: https://doi.org/10.1371/journal.pone.0128337.

41.

Schonlau, M. and M. Couper. 2017. “Options for Conducting Web Surveys.” Statistical Science 32(2): 279–292. DOI: https://doi.org/10.1214/16-STS597.

42.

Sharp, L.M., and J. Frankel. 1983. “Respondent Burden: A Test of Some Common Assumptions.” Public Opinion Quarterly 47: 36–53. DOI: https://doi.org/10.1086/268765.

43.

Spranca, M., D.E. Kanouse, M. Elliott, P.F. Short, D.O. Farley, and R.D. Hays. 2000. “Do Consumer Reports of Health Plan Quality Affect Health Plan Selection?” Health Services Review 35(5): 933–947. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1089177. (accessed September 2021).

44.

Steinbrecher, M, J. Roßmann, and J.E. Blumenstiel. 2015. “Why Do Respondents Break Off Web Surveys and Does it Matter? Results from Four Follow-up Surveys.” International Journal of Public Opinion Research 27(2): 289–302. DOI: https://doi.org/10.1093/ijpor/edu025.

45.

Stucky, B.D., R.D. Hays, M.O. Edelen, J. Gurvey, and J.A. Brown. 2016. “Possibilities for Shortening the CAHPS Clinician and Group Survey.” Medical Care 54(1): 32–37. DOI: https://doi.org/10.1097/MLR.0000000000000452.

46.

Swain, M.S. 2015. “The Effects of a Planned Missingness Design on Examinee Motivation and Psychometric Quality.” PhD Dissertation, James Madison University. Available at: https://www.proquest.com/docview/1685390254 (accessed September 2021).

47.

Toepoel, V., and P. Lugtig. 2018. “Modularization in an Era of Mobile Web: Investigating the Effects of Cutting a Survey Into Smaller Pieces on Data Quality.” Social Science Computer Review: 1–15. DOI: https://doi.org/10.1177/0894439318784882.

48.

Toepoel, V., M. Das, and A. van Soest. 2008. “Effects of Design in Web Surveys: Comparing Trained and Fresh Respondents.” Public Opinion Quarterly 72(5): 985–1007. DOI: https://doi.org/10.1093/poq/nfn060.

49.

Villar, A., M. Callegaro, and Y. Yang. 2013. “Where am I? A meta-analysis of experiments on the effects of progress indicators for web surveys.” Social Science Computer Review 31(6): 744–762. DOI: https://doi.org/10.1177/0894439313497468.

50.

West, B.T., D. Ghimire, and W.G. Axinn. 2015. “Evaluating a Modular Design Approach to Collecting Survey Data Using Text Messages.” Survey Research Methods 9:111–123. DOI: https://doi.org/10.18148/srm/2015.v9i2.6135.

51.

Yan, T., S. Fricker, and S. Tsai. 2020. “Response Burden: What Is It and What Predicts It?” In Advances in Questionnaire Design, Development, Evaluation, and Testing, edited by P.C. Beatty, D. Collins, L. Kaye, J.L. Padilla, G. B. Willis, and A. Wilmot: 193–212. New York: Wiley.

52.

Zhang, C., and F.G. Conrad. 2018. “Intervening to Reduce Satisficing Behaviors in Web Surveys: Evidence From Two Experiments on How it Works.” Social Science Computer Review 36(1): 57–81. DOI: https://doi.org/10.1177/0894439316683923