Sage Journals: Discover world-class research

Abstract

For survey researchers, physicians in the United States are a difficult-to-reach subgroup. The purpose of this study is to quantify the effect of email reminders on web-based survey response rates targeting physicians. We conducted a retrospective analysis of 11 American Board of Internal Medicine surveys from 2017 to 2019. We compute aggregate response rates for the periods between weekly email contacts across the 11 surveys, while controlling for survey time to complete, physician age, gender, region, board certification status, and initial exam performance. The overall predicted response rate after six weekly email contacts was 23.7%, 95% CI: (17.1%, 33.0%). Across the 11 surveys, we found response rate for the first period to be 8.9%, 95% CI: (6.5%, 12.2%). We observed a 50% decrease in response from the first to the second period, which had a 4.4%, 95% CI: (3.2%, 6.2%), response rate. The third and fourth response periods yielded similar response rates of 3.0%, 95% CI: (2.3%, 3.9%) and 3.3%, 95%CI: (2.4%, 4.6%), respectively. The fifth and sixth response periods yielded similar response rates of 2.2%, 95%CI: (1.5%, 3.3%) and 1.9%, 95% CI: (1.3%, 2.7%), respectively. The results were further stratified into different levels of participant survey interest, and are helpful for cost and sample size considerations when designing a physician survey.

Keywords

survey research methodology response rate physician survey online survey email reminder sample size

Get full access to this article

View all access options for this article.

References

ABIM. (2021a). About ABIM—Mission. https://www.abim.org/about/mission.aspx

ABIM. (2021b). MOC requirements: Interventional cardiology. http://www.abim.org/maintenance-of-certification/moc-requirements/interventional-cardiology.aspx

ABIM. (2021c). American Society of Clinical Oncology collaborative maintenance pathway. https://www.abim.org/maintenance-of-certification/assessment-information/assessment-options/collaborative-maintenance-pathways/asco.aspx

Albaum

Smith

S. M.

(2012). Why people agree to participate in surveys. In Gideon

(Ed.), Handbook of survey methodology for the social sciences (pp. 179–193). Springer.

Alchemer. (2021). Alchemer Landing Page. https://www.alchemer.com/

Allison

P. D

. (2010). Survival analysis using SAS: A practical guide. SAS Institute.

Allison

Greene

(2012). Do we really need zero-inflated models. Statistical Horizons.

Beebe

T. J.

Jacobson

R. M.

Jenkins

S. M.

Lackore

K. A.

Rutten

L. J. F.

(2018). Testing the impact of mixed-mode designs (mail and web) and multiple contact attempts within mode (mail or web) on clinician survey response. Health Services Research, 53, 3070–3083.

Beebe

T. J.

Locke

G. R.

III Barnes

S. A.

Davern

M. E.

Anderson

K. J.

(2007). Mixing web and mail methods in a survey of physicians. Health Services Research, 42(3p1), 1219–1234.

10.

Bjertnaes

O. A.

Garratt

Botten

(2008). Nonresponse bias and cost-effectiveness in a Norwegian survey of family physicians. Evaluation & the Health Professions, 31(1), 65–80.

11.

Blair

Brick

P. D.

Brick

J. M.

(2013). Survey topic saliency: An examination of potential effects and remedies [PowerPoint slides]. AAPOR. http://www.aapor.org/AAPOR_Main/media/AnnualMeetingProceedings/2013/Session_B-1-3-Brick.pdf

12.

Bolt

E. E.

van der Heide

Onwuteaka-Philipsen

B. D.

(2014). Reducing questionnaire length did not improve physician response rate: A randomized trial. Journal of Clinical Epidemiology, 67(4), 477–481.

13.

Boyle

Berman

Dayton

Iachan

Jans

Locke

ZuWallack

(2019). Topic salience and propensity to respond to surveys: Findings from a national mobile panel [PowerPoint slides]. U.S. Census Bureau. https://www.census.gov/fedcasic/fc2019/ppt/7ABoyleTopicSalience.pdf

14.

Brtnikova

Crane

L. A.

Allison

M. A.

Hurley

L. P.

Beaty

B. L.

Kempe

(2018). A method for achieving high response rates in national surveys of US primary care physicians. PloS One, 13(8), e0202755.

15.

Callegaro

Manfreda

K. L.

Vehovar

(2015). Web survey methodology. Sage.

16.

Cho

Y. I.

Johnson

T. P.

VanGeest

J. B.

(2013). Enhancing surveys of health care professionals: A meta-analysis of techniques to improve response. Evaluation & the Health Professions, 36(3), 382–407.

17.

Cook

D. A.

Wittich

C. M.

Daniels

W. L.

West

C. P.

Harris

A. M.

Beebe

T. J.

(2016). Incentive and reminder strategies to improve response rate for internet-based physician surveys: A randomized experiment. Journal of Medical Internet Research, 18(9), e244. https://doi.org/10.2196/jmir.6318

18.

Cook

J. V.

Dickinson

H. O.

Eccles

M. P.

(2009). Response rates in postal surveys of healthcare professionals between 1996 and 2005: An observational study. BMC Health Services Research, 9(1), 160.

19.

Crouch

Robinson

Pitts

(2011). A comparison of general practitioner response rates to electronic and postal surveys in the setting of the National STI Prevention Program. Australian and New Zealand Journal of Public Health, 35(2), 187–189. https://doi.org/10.1111/j.1753-6405.2011.00687.x

20.

Cull

W. L.

O’connor

K. G.

Sharp

Tang

S. f. S.

(2005). Response rates and response bias for 50 surveys of pediatricians. Health Services Research, 40(1), 213–226.

21.

Dai

Rocke

(2006). Hierarchical logistic regression modeling with SAS GLIMMIX. In Proceedings of the thirty-first annual SAS users group international conference. SAS Institute Inc. https://www.lexjansen.com/wuss/2006/analytics/ANL-Dai.pdf

22.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. John Wiley & Sons.

23.

Draugalis

J. R.

Coons

S. J.

Plaza

C. M.

(2008). Best practices for survey research reports: A synopsis for authors and reviewers. American Journal of Pharmaceutical Education, 72(1), 11–11. https://doi.org/10.5688/aj720111

24.

Dykema

Jones

N. R.

Piché

Stevenson

(2013). Surveying clinicians by web: Current issues in design and administration. Evaluation & the Health Professions, 36(3), 352–381.

25.

Edwards

P. J.

Roberts

Clarke

M. J.

DiGuiseppi

Wentz

Kwan

Cooper

Felix

LM.

Pratap

(2009). Methods to increase response to postal and electronic questionnaires. Cochrane Database of Systematic Reviews, (3), MR000008. https://doi.org/10.1002/14651858.MR000008.pub4

26.

Ene

Leighton

E. A.

Blue

G. L.

Bell

B. A.

(2015, April). Multilevel models for categorical data using SAS® PROC GLIMMIX: The basics. In SAS global forum (no. 3430-2015). https://support.sas.com/resources/papers/proceedings15/3430-2015.pdf

27.

Flanigan

T. S.

McFarlane

Cook

(2008). Conducting survey research among physicians and other medical professionals: A review of current literature. In Paper presented at the proceedings of the survey research methods section. American Statistical Association. http://www.asasrms.org/Proceedings/y2008/Files/flanigan.pdf

28.

Freed

G. L.

(2019). Bias in physician workforce research. Pediatrics, 144(3), e20190469.

29.

Group, U. S. C. (2019). How do I run a random effect zero-inflated Poisson model using NLMIXED? https://stats.idre.ucla.edu/sas/faq/how-do-i-run-a-random-effect-zero-inflated-poisson-model-using-nlmixed/

30.

Groves

R. M.

Heeringa

S. G.

(2006). Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(3), 439–457.

31.

Groves

R. M.

Peytcheva

(2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189.

32.

Harrison

Henderson

Alderdice

Quigley

M. A.

(2019). Methods to increase response rates to a population-based maternity survey: A comparison of two pilot studies. BMC Medical Research Methodology, 19(1), 1–8.

33.

Heberlein

T. A.

Baumgartner

(1978). Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review, 43(4), 447–462.

34.

Holford

T. R.

(1980). The analysis of rates and of survivorship using log-linear models. Biometrics, 36(2), 299–305. https://doi.org/10.2307/2529982

35.

Jafari

Ansari-Pour

(2019). Why, when and how to adjust your P values? Cell Journal, 20(4), 604–607. https://doi.org/10.22074/cellj.2019.5992

36.

Jepson

Asch

D. A.

Hershey

J. C.

Ubel

P. A.

(2005). In a mailed physician survey, questionnaire length had a threshold effect on response rate. Journal of Clinical Epidemiology, 58(1), 103–105.

37.

Johnson

T. P.

Wislar

J. S.

(2012). Response rates and nonresponse errors in surveys. JAMA, 307(17), 1805–1806.

38.

Kaplowitz

M. D.

Lupi

Couper

M. P.

Thorp

(2012). The effect of invitation design on web survey response rates. Social Science Computer Review, 30(3), 339–349.

39.

Klabunde

C. N.

Willis

G. B.

Casalino

L. P.

(2013). Facilitators and barriers to survey participation by physicians: A call to action for researchers. Evaluation & the Health Professions, 36(3), 279–295.

40.

Klabunde

C. N.

Willis

G. B.

McLeod

C. C.

Dillman

D. A.

Johnson

T. P.

Greene

S. M.

Brown

M. L.

(2012). Improving the quality of surveys of physicians and medical groups: A research agenda. Evaluation & the Health Professions, 35(4), 477–506.

41.

Laird

Olivier

(1981). Covariance analysis of censored survival data using log-linear analysis techniques. Journal of the American Statistical Association, 76(374), 231–240. https://doi.org/10.1080/01621459.1981.10477634

42.

Lee

J.-H.

Han

Fulp

Giuliano

(2012). Analysis of overdispersed count data: Application to the human papillomavirus infection in men (HIM) study. Epidemiology & Infection, 140(6), 1087–1094.

43.

Redden

D. T.

(2015). Small sample performance of bias-corrected sandwich estimators for cluster-randomized trials with binary outcomes. Statistics in Medicine, 34(2), 281–296.

44.

Linzer

Poplau

Babbott

Collins

Guzman-Corrales

Menk

Murphy

M. L.

Ovington

(2016). Worklife and wellness in academic general internal medicine: Results from a national survey. Journal of General Internal Medicine, 31(9), 1004–1010.

45.

McFarlane

Olmsted

M. G.

Murphy

Hill

C. A.

(2007). Nonresponse bias in a mail survey of physicians. Evaluation & the Health Professions, 30(2), 170–185.

46.

McLeod

C. C.

Klabunde

C. N.

Willis

G. B.

Stark

(2013). Health care provider surveys in the United States, 2000–2010: A review. Evaluation & the Health Professions, 36(1), 106–126.

47.

Menon

Muraleedharan

(2020). Internet-based surveys: Relevance, methodological considerations and troubleshooting strategies. General Psychiatry, 33(5), e100264.

48.

Morel

J. G.

Bokossa

Neerchal

N. K.

(2003). Small sample correction for the variance of GEE estimators. Biometrical Journal: Journal of Mathematical Methods in Biosciences, 45(4), 395–409.

49.

Muñoz-Leiva

Sánchez-Fernández

Montoro-Ríos

Ibáñez-Zapata

J. Á.

(2010). Improving the response rate and quality in Web-based surveys through the personalization and frequency of reminder mailings. Quality & Quantity, 44(5), 1037–1052.

50.

Parsons

J. A.

Warnecke

R. B.

Czaja

R. F.

Barnsley

Kaluzny

(1994). Factors associated with response rates in a national survey of primary care physicians. Evaluation Review, 18(6), 756–766.

51.

PASS 15 Power Analysis & Sample Size Software. (2017). NCSS, LLC. ncss.com/software/pass

52.

Payne

E. H.

Hardin

J. W.

Egede

L. E.

Ramakrishnan

Selassie

Gebregziabher

(2017). Approaches for dealing with various sources of overdispersion in modeling count data: Scale adjustment versus modeling. Statistical Methods in Medical Research, 26(4), 1802–1823.

53.

Pit

S. W.

Pyakurel

(2014). The effectiveness of recruitment strategies on general practitioner’s survey response rates–a systematic review. BMC Medical Research Methodology, 14(1), 76.

54.

Rao

S. K.

Kimball

A. B.

Lehrhoff

S. R.

Hidrue

M. K.

Colton

D. G.

Ferris

T. G.

Torchiana

D. F.

(2017). The impact of administrative burden on academic physicians: Results of a hospital-wide physician survey. Academic Medicine, 92(2), 237–243.

55.

Robertson

S. L.

Robinson

M. D.

Reid

(2017). Electronic health record effects on work-life balance and burnout within the I3 population collaborative. Journal of Graduate Medical Education, 9(4), 479–484.

56.

Rodríguez

(Producer). (2017). The piece-wise exponential model. http://data.princeton.edu/wws509/notes/c7s4.html

57.

Safir

Black

Steinbach

(2001). Using paradata to examine the effects of interviewer characteristics on survey response and data quality. In Proceedings of the annual meeting of the American statistical association (p. 58). https://www.researchgate.net/publication/237672405_USING_PARADATA_TO_EXAMINE_THE_EFFECTS_OF_INTERVIEWER_CHARACTERISTICS_ON_SURVEY_RESPONSE_AND_DATA_QUALITY

58.

SAS Institute, I. (n d). SAS/STAT(R) 9.22 user’s guide example 38.14 generalized poisson mixed model for over dispersed count data. https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_glimmix_sect028.htm

59.

Sauermann

Roach

(2013). Increasing web survey response rates in innovation research: An experimental study of static and dynamic contact design features. Research Policy, 42(1), 273–286.

60.

Scott

Jeon

S. H.

Joyce

C. M.

Humphreys

J. S.

Kalb

Witt

Leahy

(2011). A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors. BMC Medical Research Methodology, 11(1), 1–12.

61.

Short

C. E.

Rebar

A. L.

Vandelanotte

(2015). Do personalised e-mail invitations increase the response rates of breast cancer survivors invited to participate in a web-based behaviour change intervention? A quasi-randomised 2-arm controlled trial. BMC Medical Research Methodology, 15(1), 1–4.

62.

Shinohara

Aoki

Tsujimoto

Suganuma

A. M.

Furukawa

T. A.

(2018). Effect of recruitment methods on response rate in a web-based study for primary care physicians: Factorial randomized controlled trial. Journal of Medical Internet Research, 20(2), e28.

63.

Stedman

R. C.

Connelly

N. A.

Heberlein

T. A.

Decker

D. J.

Allred

S. B.

(2019). The end of the (research) world as we know it? Understanding and coping with declining response rates to mail surveys. Society & Natural Resources, 32(10), 1139–1154.

64.

Stokes

M. E.

Davis

C. S.

Koch

G. G.

(2012). Categorical data analysis using SAS. SAS Institute Inc.

65.

Sudman

(1985). Mail surveys of reluctant professionals. Evaluation Review, 9(3), 349–360.

66.

Taylor

Scott

(2019). Do physicians prefer to complete online or mail surveys? Findings from a national longitudinal survey. Evaluation & the Health Professions, 42(1), 41–70.

67.

Van Mol

(2017). Improving web survey efficiency: The impact of an extra reminder and reminder content on web survey response. International Journal of Social Research Methodology, 20(4), 317–327. https://doi.org/10.1080/13645579.2016.1185255

68.

Vandergrift

J. L.

Gray

B. M.

Barnhart

B. J.

Lynn

L. A.

Lipner

R. S.

(2020). Opportunities for maintenance of certification to better reflect scope of practice among medical oncologists. JCO Oncology Practice, 16(8), e641–e648.

69.

VanGeest

J. B.

Beebe

T. J.

Johnson

T. P.

(2015). Surveys of physicians. Handbook of Health Survey Methods, 515–543.

70.

VanGeest

J. B.

Johnson

T. P.

(2013). Surveying clinicians: An introduction to the special issue. Evaluation & the Health Professions, 36(3), 275–278.

71.

VanGeest

J. B.

Johnson

T. P.

Welch

V. L.

(2007). Methodologies for improving response rates in surveys of physicians: A systematic review. Evaluation & the Health Professions, 30(4), 303–321.

72.

Voronca

D. C.

Egede

L. E.

Gebregziabher

Johnson

(2014). Analysis of zero inflated longitudinal data using PROC NLMIXED [Unpublished manuscript]. North Carolina State University.

73.

Wood

K. E.

Krasowski

M. D.

(2020). Academic e-mail overload and the burden of “academic spam”. Academic Pathology, 7. https://doi.org/10.1177/2374289519898858

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB

Remind Me Again: Physician Response to Web Surveys: The Effect of Email Reminders Across 11 Opinion Survey Efforts at the American Board of Internal Medicine from 2017 to 2019

Abstract

Keywords

Get full access to this article

References

Supplementary Material