Enhancing Participation in Probability-Based Online Panels: Two Incentive Experiments and Their Effects on Response and Panel Recruitment

Abstract

This article investigates how mail-based online panel recruitment can be facilitated through incentives. The analysis relies on two incentive experiments and their effects on panel recruitment, and the intermediate participation in the recruitment survey. The experiments were implemented in the context of the German Emigration and Remigration Panel Study and encompass two samples of randomly sampled persons. Tested incentives include a conditional lottery, conditional monetary incentives, and the combination of unconditional money-in-hand with conditional monetary incentives. For an encompassing evaluation of the link between incentives and panel recruitment, the article further assesses the incentives’ implications for demographic composition and panel recruitment unit costs. Multivariate analysis indicates that low combined incentives (€5/€5) or, where unconditional disbursement is unfeasible, high conditional incentives (€20) are most effective in enhancing panel participation. In terms of demographic bias, low combined incentives (€5/€5) and €10 conditional incentives are the favored options. The budget options from the perspective of panel recruitment include the lottery and the €10 conditional incentive which break-even at net sample sizes of 1000.

Keywords

Probability-based panel recruitment incentive experiment combined incentives mobile populations prepaid cash money-in-hand generous lottery recruitment costs demographic bias

Get full access to this article

View all access options for this article.

References

AAPOR (2016). Standard definitions. Final dispositions of case codes and outcome rates for surveys (9th ed.). Washington: American Association for Public Opinion Research.

Becker

Mehlkop

(2011). Effects of prepaid monetary incentives on mail survey response rates and on self-reporting about delinquency - Empirical findings. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 111(1), 5–25.

Becker

Möser

Glauser

(2019). Cash vs. vouchers vs. gifts in web surveys of a mature panel study - Main effects in a long-term incentives experiment across three panel waves. Social Science Research, 81, 221–234.

Beckert

Lutter

(2012). Why the poor play the lottery: Sociological approaches to explaining class-based lottery play. Sociology, 47(6), 1152–1170. https://doi.org/10.1177/0038038512457854.

Best

Wolf

(2015). Logistic regression. In Best

Wolf

(Eds.), The SAGE handbook of regression analysis and causal inference. SAGE Publications.

Blom

A. G.

Gathmann

Krieger

(2015). Setting up an online panel representative of the general population: The German internet panel. Field Methods, 27(2), 391–408. https://doi.org/10.1177/1525822X15574494.

Bosnjak

Batinic

(2002). Understanding the willingness to participate in online-surveys. In Batinic

Reips

U.-D.

Bosnjak

Werner

(Eds.), Online social sciences (pp. 81–92). Hofgrefe & Huber.

Bosnjak

Das

Lynn

(2015). Methods for probability-based online and mixed-mode panels: Selected recent trends and future perspectives. Social Science Computer Review, 34(1), 3–7. https://doi.org/10.1177/0894439315579246.

Bosnjak

Tuten

T. L.

(2003). Prepaid and promised incentives in web surveys: An experiment. Social Science Computer Review, 21(2), 208–217. https://doi.org/10.1177/0894439303021002006.

10.

Callegaro

Baker

Bethlehem

J. G.

Göritz

A. S.

Krosnick

J. A.

Lavrakas

P. J.

(2014). Online panel research: History, concepts, applications and a look at the future. In Callegaro

Baker

Bethlehem

J. G.

Göritz

Krosnick

J. A.

Lavrakas

P. J.

(Eds.), Wiley series in survey methodology. Online panel research: A data quality perspective (pp. 1–22). John Wiley & Sons Inc.

11.

Callegaro

DiSogra

(2008). Computing response metrics for online panels. Public Opinion Quarterly, 72(5), 1008–1032. https://doi.org/10.1093/poq/nfn065.

12.

Christmann

Gummer

Hähnel

Wolf

(2019, July). Does the mode matter? An experimental comparison of survey responses between face-to-face and mixed-mode surveys. ESRA, Zagreb.

13.

Conlisk

(1993). The utitlity of gambling. Journal of Risk and Uncertainty, 6, 255–275.

14.

Cornesse

Felderer

Fikel

Krieger

Blom

A. G.

(2021). Recruiting a probability-based online panel via postal mail: Experimental evidence. Social Science Computer Review, 14(2), 1-26. https://doi.org/10.1177/08944393211006059.

15.

Coryn

C. L. S.

Becho

L. W.

Westine

C. D.

Mateu

P. F.

Abu-Obaid

R. N.

Hobson

K. A.

Ramlow

(2020). Material incentives and other potential factors associated with response rates to internet surveys of American evaluation association members: Findings from a randomized experiment. American Journal of Evaluation, 41(2), 277–296. https://doi.org/10.1177/1098214018818371.

16.

Décieux

J. P.

Mergener

(2021). German labor emigration in times of technological change: Occupational characteristics and geographical patterns. Sustainability, 13(3), 1219. https://doi.org/10.3390/su13031219.

17.

Dillman

D. A.

(2017). The promise and challenge of pushing respondents to the Web in mixed-mode surveys. Survey Methodology, 43(1), 3–30.

18.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. Wiley & Sons.

19.

DiSogra

Callegaro

(2015). Metrics and design tool for building and evaluating probability-based online panels. Social Science Computer Review, 34(1), 26–40. https://doi.org/10.1177/0894439315573925.

20.

Edwards

Cooper

Roberts

Frost

(2005). Meta-analysis of randomised trials of monetary incentives and response to mailed questionnaires. Journal of Epidemiology and Community Health, 59(11), 987–999. https://doi.org/10.1136/jech.2005.034397

21.

Emerson

R. M.

(1976). Social exchange theory. Annual Review of Sociology, 2, 335–362.

22.

Ette

Décieux

J. P.

Erlinghagen

Genoni

Guedes Auditor

Knirsch

Witte

(2020). German emigration and remigration panel study: Methodology and data manual of the baseline survey (Wave 1). BiB Daten- und Methodenbericht: 1/2020. Bundesinstitut für Bevölkerungsforschung.

23.

Ette

Witte

(2021). Brain drain or brain circulation? Economic and non-economic factors driving the international migration of German citizens. In Erlinghagen

Ette

Schneider

N. F.

Witte

(Eds.), The global lives of German migrants. Consequences of international migration across the life course. IMISCOE research series. Springer.

24.

Fan

Yan

(2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26(2), 132–139. https://doi.org/10.1016/j.chb.2009.10.015.

25.

Feliciano

. (2020). Immigrant selectivity effects on health, labor market, and educational outcomes. Annu. Rev. Sociol, 46(1), 315–334. https://doi.org/10.1146/annurev-soc-121919-054639.

26.

Fishburn

P. C.

(1981). Subjective expected utility: A review of normative theories. Theory and Decision, 13(2), 139–199. https://doi.org/10.1007/BF00134215.

27.

Gajic

Cameron

Hurley

(2012). The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community survey. The European Journal of Health Economics, 13(6), 789–799. https://doi.org/10.1007/s10198-011-0332-0.

28.

Genoni

Décieux

J. P.

Ette

Witte

(2021). Setting up probability-based online panels of migrants through push-to-web recruitment: Methodological challenges and lessons learned from the German Emigration and Remigration Panel Study (GERPS). In Erlinghagen

Ette

Schneider

N. F.

Witte

(Eds.), The global lives of German migrants. Consequences of international migration across the life course. IMISCOE research series. Springer.

29.

Göritz

A. S.

(2008). The long-trm effect of material incentives on participation in online panels. Field Methods, 20(3), 211–225. https://doi.org/10.1177/1525822X08317069.

30.

Göritz

A. S.

Luthe

S. C.

(2013). Effects of lotteries on response behavior in online panels. Field Methods, 25(3), 219–237. https://doi.org/10.1177/1525822X12472876.

31.

Gouldner

A. W.

(1960). The norm of reciprocity: A preliminary statement. American Sociological Review, 25(2), 161–178. https://doi.org/10.2307/2092623.

32.

Groves

R. M.

Singer

Corning

(2000). Leverage-saliency theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64(3), 299–308.

33.

Guo

Kopec

J. A.

Cibere

L. C.

Goldsmith

C. H.

(2016). Population survey features and response rates: A randomized experiment. American Journal of Public Health, 106(8), 1422–1426. https://doi.org/10.2105/AJPH.2016.303198.

34.

Jann

(2014). Plotting regression coefficients and other estimates. The Stata Journal, 14(4), 708–737. https://doi.org/10.1177/1536867X1401400402.

35.

Kaczmirek

Phillips

Pennay

Lavrakas

Neiger

(2019). Building a probability-based online panel: Life in Australia™. Kaczmirek

Phillips

Pennay

Lavrakas

Neiger

(Eds.) (CSRM & SRC Methods Paper No. 2). Canberra.

36.

Kolm

S.-C.

(2006). Introduction to the economics of altruism, giving and reciprocity. In Kolm

S.-C.

Mercier Ythier

(Eds.), Handbooks in economics: Vol. 23. Handbook of the economics of giving, altruism and reciprocity, Vol. 1 (1st ed.). Elsevier.

37.

Kretschmer

Müller

(2017). The wave 6 NEPS adult study incentive experiment. Methods, Data, Analyses, 11(1), 7–28. https://doi.org/10.12758/mda.2016.014.

38.

Krieger

(2018). A penny for your thoughts: The use of cash incentives in face-to-face surveys [Dissertation, Universität Mannheim, Mannheim].

39.

Lengacher

J. E.

Sullivan

C. M.

Couper

M. P.

Groves

R. M.

(1995). Once reluctant, always reluctant? Effects of differential incentives on later survey participation in a longitudinal study. In Proceedings of the Survey Research Methods Section, 1029–1034.

40.

Lohr

S. L.

(2010). Sampling: Design and analysis. Cengage.

41.

Lynn

(2017). From standardised to targeted survey procedures for tackling non-response and attrition. Survey Research Methods, 11(1), 93–103. https://doi.org/10.18148/srm/2017.v11i1.6734.

42.

Lynn

(2020). Evaluating push-to-web methodology for mixed-mode surveys using address-based samples. Survey Research Methods, 14(1), 19-30. https://doi.org/10.18148/srm/2020.v14i1.7591.

43.

Martinsson

Riedel

(2015). Postal recruitment to a probability based web panel: Long term consequences for response rates, representativeness and costs. LORE working paper. (1).

44.

McCaffery

E. J.

(1994). Why people play lotteries and why it matters. Wisconsin Law Review, 71, 71–122.

45.

Mercer

Caporaso

Cantor

Townsend

(2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79(1), 105–129. https://doi.org/10.1093/poq/nfu059.

46.

Messer

B. L.

Dillman

D. A.

(2011). Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opinion Quarterly, 75(3), 429–457. https://doi.org/10.1093/poq/nfr021.

47.

Mize

(2019). Best practices for estimating, interpreting, and presenting nonlinear interaction effects. Sociological Science, 6(4), 81–117. https://doi.org/10.15195/v6.a4.

48.

Okasha

(2007). Rational choice, risk aversion, and evolution. Journal of Philosophy, 104(5), 217–235. https://doi.org/10.5840/jphil2007104523.

49.

Oscarsson

Arkhede

(2020). Effects of conditional incentives on response rate, non-response bias and measurement error in a high response-rate context. International Journal of Public Opinion Research, 32(2), 354–368. https://doi.org/10.1093/ijpor/edz015.

50.

Pandita

. (2017). Internet a change agent: An overview of internet penetration and growth across the world. International Journal of Information Dissemination and Technology, 7(2), 83. https://doi.org/10.5958/2249-5576.2017.00001.2.

51.

Pedersen

M. J.

Nielsen

C. V.

(2016). Improving survey response rates in online panels. Social Science Computer Review, 34(2), 229–243. https://doi.org/10.1177/0894439314563916.

52.

Pforr

Blohm

Blom

A. G.

Erdel

Felderer

Fräßdorf

Rammstedt

(2015). Are incentive effects on response rates and nonresponse bias in large-scale, face-to-face surveys generalizable to Germany? Evidence from ten experiments. Public Opinion Quarterly, 79(3), 740–768. https://doi.org/10.1093/poq/nfv014.

53.

Raymer

Rogers

(2007). Using age and spatial flow structures in the indirect estimation of migration streams. Demography, 44(2), 199–223. https://doi.org/10.1353/dem.2007.0016.

54.

Ryu

Couper

M. P.

Marans

R. W.

(2005). Survey incentives: Cash vs. In-Kind; face-to-face vs. mail; Response rate vs. Nonresponse error. International Journal of Public Opinion Research, 18(1), 89–106. https://doi.org/10.1093/ijpor/edh089.

55.

Sakshaug

Hülle

Schmucker

Liebig

(2020). Panel survey recruitment with or without interviewers? Implications for nonresponse, panel consent, and total recruitment bias. Journal of Survey Statistics and Methodology, 8(3), 540–565. https://doi.org/10.1093/jssam/smz012.

56.

Scherpenzeel

Toepoel

(2012). Recruiting a probability sample for an online panel. Public Opinion Quarterly, 76(3), 470–490. https://doi.org/10.1093/poq/nfs037.

57.

Schoeni

R. F.

Stafford

Mcgonagle

K. A.

Andreski

(2013). Response rates in national panel surveys. The ANNALS of the American Academy of Political and Social Science, 645(1), 60–87. https://doi.org/10.1177/0002716212456363.

58.

Schröder

Saßenroth

Körtner

Kroh

Schupp

(2013). Experimental evidence of the effect of monetary incentives on cross-sectional and longitudinal response: Experiences from the socio-economic panel (SOEP) (SOEP papers on multidisciplinary panel data research No. 603). Berlin.

59.

Schumann

Lück

Naderi

Bujard

Schneider

N. F.

(2019, July). Which incentive works best? Findings from a push-to-web experiment in the GGP context in Germany. ESRA 2019. Zagreb.

60.

Simon

H. A.

(1990). Bounded rationality. In Eatwell

Milgate

Newman

(Eds.), Utility and probability (pp. 15–18). United Kingdom: Palgrave Macmillan. https://doi.org/10.1007/978-1-349-20568-4_5.

61.

Singer

(2011). Toward a benefit-cost theory or survey participation: Evidence, further tests, and implications. Journal of Official Statistics, 27(2), 379–392.

62.

Singer

(2013). The use and effects of incentives in surveys. The ANNALS of the American Academy of Political and Social Science, 645(1), 112–141. https://doi.org/10.1177/0002716212458082.

63.

Suzer-Gurtekin

Z. T.

Elkasabi

Liu

Lepkowski

J. M.

Curtin

McBee

(2016). Effect of a pre-paid incentive on response rates to an Address-Based Sampling (ABS) Web-Mail survey. Survey Practice, 9(4), 1–7. https://doi.org/10.29115/SP-2016-0025.

64.

Tillé

(2006). Sampling algorithms. Springer.

65.

Witte

Pollak

Ette

(2021). Social origins of German emigrants: Maintaining social status by international mobility? In Erlinghagen

Ette

Schneider

N. F.

Witte

(Eds.), The Global Lives of German Migrants. Consequences of International Migration Across the Life Course. IMISCOE Research Series. Springer.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.86 MB