We assess the association between survey layout and response rates (RRs) in the 2017 Medicare Advantage Consumer Assessment of Healthcare Providers and Systems mail survey. Among 438 Medicare Advantage plans surveyed by six vendors, there was latitude in survey layout, and plans could add up to 12 supplemental items. Regression models predicted survey response from survey characteristics (page count, number of supplemental items, and survey attractiveness), and beneficiary sociodemographics. Beneficiary-age-by-survey-characteristic interactions assessed whether survey characteristics were more strongly related to RRs among older beneficiaries. We found that surveys with more supplemental items and less attractive layouts had lower adjusted odds of response. RRs were more sensitive to format among older beneficiaries. The difference in adjusted RRs for the most favorable versus the least favorable survey design was 14.5%. For a 65-year-old, this difference was 13.6%; for an 80-year-old, it was 21.0%. These findings suggest that even within a relatively standardized survey, formatting can substantially influence RRs.
BeckettM. K.ElliottM. N.GaillotS.HaasA.DemboskyJ. W.GiordanoL. A.BrownJ. (2016). Establishing limits for supplemental items on a standardized national survey. Public Opinion Quarterly, 80, 964-976. doi:10.1093/poq/nfw028
4.
BeebeT. J.DavernM. E.McAlpineD. D.CallK. T.RockwoodT. H. (2005). Increasing response rates in a survey of Medicaid enrollees: The effect of a prepaid monetary incentive and mixed modes (mail and telephone). Medical Care, 43, 411-414.
5.
BenkíJ.BroomeJ.ConradF.GrovesR. M.KreuterF. (2011). Effects of speech rate, pitch, and pausing on survey participation decisions. Paper presented at the American Association for Public Opinion Research Annual Meeting, Phoenix, AZ. Retrieved from https://pdfs.semanticscholar.org/8e2d/7c8e52ceb43dde2a95dc96aa8e99e0495333.pdf
6.
CouperM. P.SingerE.ConradF. G.GrovesR. M. (2008). Risk of disclosure, perceptions of risk, and concerns about privacy and confidentiality as factors in survey participation. Journal of Official Statistics, 24, 255-275.
7.
de RadaV. D. (2005). Influence of questionnaire design on response to mail surveys. International Journal of Social Research Methodology, 8, 61-78. doi:10.1080/1364557021000025991
8.
DillmanD. A.SinclairM. D.ClarkJ. R. (1993). Effects of questionnaire length, respondent-friendly design, and a difficult question on response rates for occupant-addressed census mail surveys. Public Opinion Quarterly, 57, 289-304.
9.
DillmanD. A.SmythJ. D.ChristianL. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Hoboken, NJ: John Wiley.
10.
EdwardsP.RobertsI.ClarkeM.DiGuiseppiC.PratapS.WentzR.KwanI. (2002). Increasing response rates to postal questionnaires: Systematic review. British Medical Journal, 324, 1183. doi:10.1136/bmj.324.7347.1183
11.
ElliottM. N.BeckettM. K.LehrmanW. G.ClearyP. D.CoheaC.GiordanoL. A.GoldsteinE.DambergC., (2016). Understanding the role played by medicare’s patient experience points system in hospital reimbursement. Health Affairs, 35, 1673-1680.
12.
ElliottM. N.BrownJ. A.LehrmanW. G.BeckettM. K.HambarsoomianK.GiordanoL. A.GoldsteinE. H. (2013). A randomized experiment investigating the suitability of speech-enabled IVR and web modes for publicly reported surveys of patients’ experience of hospital care. Medical Care Research and Review, 70, 165-184. doi:10.1177/1077558712464907
13.
FowlerF. J.Jr.GallagherP. M.StringfellowV. L.ZaslavskyA. M.ThompsonJ. W.ClearyP. D. (2002). Using telephone interviews to reduce nonresponse bias to mail surveys of health plan members. Medical Care, 40, 190-200.
14.
FoxR. J.CraskM. R.KimJ. (1988). Mail survey response rate: A meta-analysis of selected techniques for inducing response. Public Opinion Quarterly, 52, 467-491.
15.
GraubardB. I.KornE. L. (1999). Predictive margins with survey data. Biometrics, 55, 652-659.
16.
GrovesR. M.CialdiniR. B.CouperM. P. (1992). Understanding the decision to participate in a survey. Public Opinion Quarterly, 56, 475-495.
17.
GrovesR. M.O’HareB. C.Gould-SmithD.BenkíJ.MaherP. (2007). Telephone interviewer voice characteristics and the survey participation decision. In LepkowskiJ. M.TuckerC.BrickJ. M.de LeeuwE. D.JapecL.LavrakasP. J.SangsterR. L. (Eds.), Advances in telephone survey methodology (pp. 385-400). New York, NY: Wiley.
18.
GrovesR. M.PeytchevaE. (2009). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72, 167-189.
HunsakerA.HargittaiE. (2018). A review of Internet use among older adults. New Media & Society, 20, 3937-3954. doi:10.1177/1461444818787348
23.
JansenJ. H. (1985). Effect of questionnaire layout and size and issue-involvement on response rates in mail surveys. Perceptual and Motor Skills, 61, 139-142. doi:10.2466/pms.1985.61.1.139
24.
KanukL.BerensonC. (1975). Mail surveys and response rates: A literature review. Journal of Marketing Research, 12, 440-453. doi:10.2307/3151093
25.
KleinD. J.ElliottM. N.HavilandA. M.SalibaD.BurkhartQ.EdwardsC.ZaslavskyA. M. (2011). Understanding nonresponse to the 2007 Medicare CAHPS survey. The Gerontologist, 51, 843-855. doi:10.1093/geront/gnr046
26.
KramerJ. M.SchwartzA. (2017). Reducing barriers to patient-reported outcome measures for people with cognitive impairments. Archives of Physical Medicine and Rehabilitation, 98, 1705-1715. doi:10.1016/j.apmr.2017.03.011
RolandM.ElliottM.LyratzopoulosG.BarbiereJ.ParkerR. A.SmithP.. . . CampbellJ. (2009). Reliability of patient responses in pay for performance schemes: Analysis of national General Practitioner Patient Survey data in England. British Medical Journal, 339, b3851. doi:10.1136/bmj.b3851
29.
SaundersC. L.ElliottM. N.LyratzopoulosG.AbelG. A. (2016). Do differential response rates to patient surveys between organizations lead to unfair performance comparisons? Evidence from the English Cancer Patient Experience Survey. Medical Care, 54, 45-54. doi:10.1097/mlr.0000000000000457
30.
SchnaierJ. A.SweenyS. F.WilliamsV. S.KosiakB.LubalinJ. S.HaysR. D.Harris-KojetinL. D. (1999). Special issues addressed in the CAHPS survey of Medicare managed care beneficiaries: Consumer Assessment of Health Plans Study. Medical Care, 37(3 Suppl.), Ms69-Ms78.
31.
ShihT.-H.XitaoF. (2008). Comparing response rates from web and mail surveys: A meta-analysis. Field Methods, 20, 249-271. doi:10.1177/1525822X08317085
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
