Politician responses to material incentives for participation in surveys: Experimental evidence from South Africa

Abstract

Political scientists routinely focus their investigations on the attitudes and behaviours of politicians and increasingly study them through systematic surveys. As with all surveys, researchers aim to maximize response rates, but we know little about the incentives that motivate politicians to participate in survey research. In this paper, we consider whether financial incentives in the form of small cash payments or participation in lotteries for slightly larger rewards increase participation. Based on a randomized control trial (RCT) conducted as part of an effort to survey local councillors in South Africa, we find that, if anything, material incentives depress response rates overall – weighing against their use in future surveys. However, we also find some heterogeneity in estimated effects by political party, which may indicate that the intentional provision of material incentives can affect sample compositions in ways that may be beneficial for some studies. While our findings in this regard are only suggestive, they should inform future research and research methods.

Introduction

Social scientists routinely seek to study the views and opinions of political, social and business elites, as their attitudes and behaviours tend to have an outsized influence on key outcomes of interest (Kertzer and Renshon, 2022; Rodriguez-Teruel and Daloz, 2017). However, conducting surveys of such individuals is often challenging. The relevant populations of interest tend to be small and restricted (Kertzer and Renshon, 2022; Vis and Stolwijk, 2020), they typically have busy schedules, and they are often shielded from researcher access by gatekeepers, all of which lead to low response rates (Druckman and Lupia, 2012; Hafner-Burton et al., 2013; Hoffman-Lange, 2008).

Although the difficulties of conducting elite surveys are well recognized (see, e.g. Hoffman-Lange, 2007), relatively little research has been completed looking at how to overcome these access challenges, and the impact that particular approaches might have on the quality of data obtained (Bakkalabioglu, 2020; Kertzer and Renshon, 2022). While significant time and journal space has been dedicated to discussing the use of monetary incentives in non-elite surveys, ¹ for example, few scholars have rigorously tested the impact of such approaches in elite-focused research (Kertzer and Renshon, 2022).

In recent years, a few key contributions have provided rigorous analysis and guidance on the use of elite incentives. For example, Safarpour et al. (2022) show that monetary incentives increase survey response rates among US-based NGO workers, while Heinzel et al. (2024) find that the promise to enter participants into a lottery declines participation among World Bank Staffers. However, we note that these studies have focused on bureaucratic elites and not on politicians, who are our subject of focus. In fact, the prevailing view in the literature is that monetary incentives are often simply ‘inappropriate’ for political elites (Dietrich et al., 2021a, 2021b; Linde and Vis, 2017), and researchers should instead do little more than offer to share research findings in order to encourage participation (Maestas et al., 2003; Vis and Stolwijk, 2020). ²

From our own work in sub-Saharan Africa and elsewhere, however, we know that promises to share research findings are often insufficient to motivate acceptable response rates among politicians. Moreover, given the frequently asserted assumption that elites and politicians in particular tend to be ‘rent-seeking’ (e.g. Krueger, 1974; Laband and Sophocleus, 2019), it certainly could follow that politicians would be differently motivated by material incentives in a research transaction than are bureaucratically focused workers. Additionally, while the provision of small material incentives may be less common in politician-focused research than it is in mass surveys, we know that researchers often do provide small material incentives to induce their participation, without any assurance of their effectiveness (Kertzer and Renshon, 2022).

Such practice leaves open the central questions of if and how such inducements affect response rates in this unique population? In addition – and just as importantly – while elite samples are already biased in terms of who is willing to comply, does the provision of incentives introduce new sources of bias that compound (or mitigate) such problems?

In this project, we seek to provide a systematic answer to these questions by adding a methodological study to a survey of local politicians in South Africa. This study will make three primary contributions to the literature:

1. It will investigate a number of important drivers of response rates, taking seriously the reality that many researchers do offer inducements when conducting elite surveys.

2. It will examine the impact of commonly used inducements on the quality of data obtained.

3. It will propose a set of practice recommendations for elite survey implementation, based on our findings and the goals of driving high response rates in an economical manner with respect to money and effort.

We note that within the broad category of ‘elites’, many gradations of status and power – including scope of influence (local vs national) – are likely to affect behaviour and engagement with respect to standardized surveys (Kertzer and Renshon, 2022). In this study, we consider lower-level, local political elites (elected councillors), who hold some status and a steady income within their communities – but these are clearly distinct from ‘ultra-elites’ (national government officials, high-level executives, etc.), whose incentive structures and gate-keeping infrastructure are likely very different. Our findings should therefore be interpreted with this constraint in mind.

Detailing a series of pre-registered analyses, we find that on average, conditional financial incentives – whether a small payment in the form of a gift card immediately upon completion, or entry into a lottery for a larger reward – did not cause councillors to respond at higher rates. In fact, we find substantially lower rates of starting and similar or lower rates of completing the survey in response to both cash and lottery incentives relative to control.

Nonetheless, the implications of our study are not decidedly dispositive for the use of material incentives in future surveys. We find substantial heterogeneity in treatment effects by political party – notably that councillors from the ruling party were somewhat positively motivated by the incentives and that councillors from the leading opposition party were substantially repelled by them. While not pre-registered, those findings invite further examination given their implications for attaining a representative sample and learning about subjects of interest. We consider candidate explanations for this heterogeneity.

Hypotheses

The motivating questions for our study were relatively intuitive: Does the addition of material/reward-based (extrinsic) incentives lead to higher politician response rates on surveys? If so, what are the most effective and cost-efficient incentives to provide? We advance two primary hypotheses in this regard: ³

1. Councillors are more likely to be responsive when they receive some sort of material inducement than when they just receive basic information on the (intrinsic) value of the survey.

2. Councillors will be more likely to be responsive when they receive entry into a lottery that offers a small chance of winning a large reward, versus the promise of a guaranteed small payment upon completion.

Our first hypothesis emanates from the broader literature on citizen level surveys (Dillman, 2014; Maestas et al., 2003; Singer and Ye, 2013). This literature suggests that in general the provision of material incentives increases citizen-level response rates (Dietrich et al., 2021b; though see Godwin, 1979), although there is more mixed evidence surrounding the relative benefits of prepaid versus conditional incentives, as well as direct payments versus entry into a lottery (Church, 1993).

With respect to the second, studies have shown that the relative effectiveness of lotteries as an inducement is likely to depend on the characteristics of the population being surveyed (Preece et al., 2010). We theorize that appetite for risk is an important such characteristic, with individuals with a higher risk appetite more likely to be motivated by the potential for winning a larger amount – even with a high probability of receiving no reward – at least relative to the offer of a guaranteed smaller post-completion payment. In turn, politicians tend to be more risk-loving than ordinary citizens (Hafner-Burton et al., 2013; Heß et al., 2018), and we therefore hypothesize that they are more likely to be motivated by the lottery condition than the small payment one.

Research design

To test our hypotheses, we randomized incentives offered to respondents in the context of a substantive study of local councillors in South Africa being fielded by the authors. Councillors are generally busy individuals with wide-ranging duties, including a mix of constituency service, participation in local council meetings, and party-based obligations, making the challenge of survey responsiveness highly salient. In the country’s mixed electoral system in local government, approximately half of all councillors are elected as ward councillors (albeit with strong party brands and involvement), while the other half are indirectly elected via the allocation of seat shares based on a proportional representation vote and party-based rank ordering of individuals to fill those seats.

South African local councillors hold status within their communities, and frequently face fierce competition for their positions, but again, they are not ‘ultra-elites'. Councillors earn approximately $20–$30,000/year depending on experience, rank, and location/cost-of-living. For comparison, the average annual salary in South Africa in September 2022 was $16,394 (Quarterly Employment Statistics, 2022).

Based on lists published by the Electoral Commission of South Africa, we first identified a target list of politicians across 47 urban and near-urban municipalities across South Africa (N = 3089). This survey sought to understand local councillor perspectives on representation and accountability in a young democracy.

Second, we randomized in equal parts, and at the individual level, three distinct approaches for inviting participation. All three invitations included a request to participate in an academic survey and a statement about the value of the research, with a focus on the importance of gaining a better understanding of how local democracy is functioning, and the resources and support it needs to flourish. In the control condition, there were no additional motivations. In the first treatment condition (T1 – cash), subjects were also promised a ZAR 100 (approx. US$6) gift card if they completed the survey; and in the second treatment condition (T2 – lottery), they were promised entry into a lottery to win one of two ZAR 5000 (approx. US$300) gift cards. In both cases, the provision of material incentives was clearly separated from the survey itself in order to preserve anonymity. Upon completion of the survey, respondents were taken to a separate page to complete their details for receipt of the monetary incentives or entry into the lottery, and they were reminded at the beginning of the survey and in all communications that the survey was anonymous and their responses would not be linked back to them in any way.

We used a block randomization method to avoid spatial-, councillor position-, and enumerator-level confounding. Each of the country’s eight Metropolitan Municipalities (the largest municipalities) is its own district, and the other local councils were grouped into their respective districts (see Appendix for details). Within each block, we followed the approach discussed in Bruhn and McKenzie (2009) and conducted multiple draws of assignment to treatment (100), choosing the one that gave the best balance in terms of pre-treatment variables. Note that the only information available to us about the councillors pre-treatment was: (1) the council they were in; (2) whether they were a Ward (directly elected) or Proportional Representation (party list) councillor; (3) their political party, or in the case of some ward councillors, their status as Independents; (4) through matching with previous years’ lists, whether they had served as a councillor prior to the 2021 elections; and (5) if so, whether they had been contacted for and/or responded to prior waves of this survey. We provide additional details, including of balance across treatment arms, in our appendix.

We also randomized assignment to research assistants (RA’s) – all students based at the University of Cape Town, who underwent several days of training prior to the start of the survey, and varied in terms of gender, ethnicity, and home language. Each RA was assigned approximately 100 councillors. We identified councillor email addresses and phone numbers from publicly available data (e.g. lists published on council websites, party websites, and the website of the South African Parliamentary Monitoring Group). Following a consistent set of procedures (see the Appendix), RA’s made initial contact with their assigned councillors via email where possible, and via phone where they were unable to locate an email address. Following initial contact, RA’s continued to reach out to local councillors by email and telephone (call, sms, and WhatsApp), alternating contact approaches where possible, until either the councillor took the survey, they obtained a firm refusal, or they had completed five (5) contact attempts. Councillors were offered the option to complete the survey themselves using a unique link, or to have it enumerated to them over the phone or in a video call.

Based on our pre-analysis plan, we estimated that with a sample size of 3,089, and a control group response rate of 15%, we were sufficiently powered at the 0.80 level to detect an effect size of 4 percentage points in a one-sided test. However, we note that we were not able to identify the contact information of 710 councillors, and a further 55 reported that they were no longer a councillor, and we excluded those councillors from the analyses, such that our actual study population was N = 2324. ⁴

We recorded the following three indicators of our outcome measure:

1. Whether or not the survey was started (opened) [0–1]

2. Item response rate [percentage of questions answered in the survey ranging from 0 to 100%]

3. Whether or not the survey was completed, that is, completed the final question of the survey [0–1] ⁵

We received 723 responses to our survey, but after removals owing to missing names, we recorded a total of 678 (29%) started; and of those, 569 (24.5%) completed surveys.

Findings

We estimated effect sizes with Ordinary Least Squares (OLS) regression for all outcomes with the following model

$Outcom{e}_i=\alpha +{\beta }_1{L}_i+{\beta }_2{S}_i+C_1{Z}_{1i}+C_2{Z}_{2i}+\cdots +C_{25}{Z}_{25i}+X_{\gamma }^{\prime }+{ϵ}_i$

(1)

We included parameters for both whether or not the inducement was a lottery (L _i ) or a guaranteed small cash payment (S _i ), and in turn, we report in Figure 1 estimates for β₁ and β₂; X _i is a vector of the individual-level covariates we balanced on in randomization (the party of the councillor and whether or not the councillor is a ward councillor), as well as whether the councillor was a councillor during previous survey rounds, and whether the councillor responded to previous survey rounds. We also include fixed effects for the vector of municipal councils, Z _i . Note that in the appendix (Table 19), we also report estimates of Logit models for the binary outcomes, and – following Mutz et al. (2019) – estimates of models without covariate adjustment, and we find no substantive differences in our results.

Figure 1.

OLS estimates of the effects of material incentives on survey participation among South African local government councillors (2022) markers indicate estimate effect of cash or lottery incentive invitations on one of three outcomes (legend). Quantities are effect sizes in terms of change in rates of participation (percentage points) or change in percentage of questions completed; thin (thick) lines indicate 95 (90) percent confidence intervals.

In Figure 1, we report the main findings from our experimental study, which depicts the effect sizes of our two experimental treatments.

In the Appendix, we also report results for pooled material incentive (cash or lottery), but as these are essentially the average of the two, the findings conveyed here are substantively equivalent. Note that while the lottery did lead to higher participation rates relative to the cash/gift card incentive (as predicted), in all cases, estimated differences in the effect of the material incentives were small and not statistically significant. Our findings are robust to a range of alternative specifications.

As depicted in Figure 1, the very clear finding is that when considering the full population of councillors, neither cash nor lottery incentive invitations had the intended effect of increasing the likelihood of councillors’ willingness to complete, let alone to start, the survey. On the contrary: among those councillors who received any invitation at all, in the (unadjusted) full sample, the rate of starting the survey was 31.6 percent for the control group, 29.7 percent for the lottery incentive group, and 26.3 percent for the cash incentive group. With covariate adjustment, we calculate a 5.3% decrease for the cash incentive, and 2.4% point decrease for the lottery, but only the former estimate is significant at conventional levels.

When we consider the effects of material incentives not simply on whether individuals started, but whether they actually completed the survey – either from the perspective of the percent of questions completed, or a binary variable for finishing the survey – we still recover negative estimates, but they are less pronounced. As shown in Figure 1, the effect of cash incentives was slightly negative, and the effect of the lottery was essentially zero. Thus, among those subjects who started the survey, those who had originally been offered the cash or lottery incentives were more likely to actually complete the survey, relative to those not originally offered such an incentive. Note, however, that we cannot infer from this that material incentives will ‘work’ if offered after people start the survey, because in our design, the decision to start the study was made post-treatment.

Given our motivation, and prior findings with general populations, the overall negative effect of material incentives on councillor participation in the survey was surprising. What might account for such patterns? One key insight was gleaned from some of our research assistants who reported that some councillors said that they viewed receiving material incentives for participation in the survey as ‘unethical’. Although we do not believe that the small incentives offered in fact violated any actual ethical obligations on the part of councillors, ⁶ we recognize that in South Africa – as in many countries – concerns about corruption and bribery are highly salient (Butler and Pereira, 2018; Gottschalk, 2022; Madonsela, 2019; Vos, 2023). The councillors’ stated reactions highlight the possibility that prevailing understanding of norms contra-indicated the offering of such incentives and/or that the act of refusing to accept such rewards could be used as a relatively low-cost signal of ‘political virtue’.

Additional research would be necessary to more credibly test whether material incentives for politicians might have a depressing effect owing to concerns about being seen as unethical. Indeed, our study was not designed to explore the mechanisms behind why an extrinsic reward might be a disincentive for some. Nonetheless, we attempt a plausibility probe of the claim, by using party membership as a proxy for concern with corrupt practices. Specifically, the ruling African National Congress (ANC) party (in power at the national level and dominant at other levels of power since 1994) has been associated with widespread corruption scandals since at least the late 1990s, whereas the leading opposition party, the Democratic Alliance (DA), has made as a central platform its rejection of corruption (Lannegren and Ito, 2017). If the offer of a material incentive to complete a survey is perceived to be akin to a bribe, we ought to see different responses across parties in line with their ‘appetite’ for corruption.

To test this, we investigate the effects of treatment separately for councillors from the ANC, the DA, the third largest party (Economic Freedom Fighters or EFF), and for the remaining small parties and independent councillors (pooled). We present the results of those analyses in Figure 2.

Figure 2.

OLS estimates of the effects of material incentives on survey participation among South African local government councillors (2022), by party markers indicate estimate effect of cash or lottery incentive invitations on one of three outcomes (legend). Quantities are effect sizes in terms of change in rates of participation (percentage points) or change in percentage of questions completed; thin (thick) lines indicate 95 (90) percent confidence intervals.

We note that these analyses were not pre-registered, but they are illuminating. Most strikingly, we find that material incentives had strong negative effects on participation among the broadly centrist Democratic Alliance (DA) councillors, resulting in overall lower rates of completion. For ANC councillors, none of the effects of cash incentives were statistically significant, but lottery incentives had a positive impact on completion. For the far-left Economic Freedom Fighter (EFF) party, and for small party (including independent) councillors, all estimates were statistically insignificant (and imprecisely estimated owing to relatively small sample sizes). Yet, in neither sub-sample did point estimates suggest positive effects of either material incentive. We find similar results when reporting the interaction of party dummies and a dummy variable for any material incentive, estimating a strong negative effect of those incentives only for DA councillors (Appendix Table 18).

While these findings are in line with the above-stated hypothesis concerning differing party-based approaches to taking a stand against corruption and/or to the value of transactional exchange, due to the highly limited pre-treatment data on councillors in our study (and, indeed, the rationale for the survey itself), we cannot conclude this with certainty. It is possible other mechanisms may be driving the results. For example, one possibility is that differential familiarity with what constitutes an ethical breach across parties may have facilitated differential levels of incentive acceptance. More research is, therefore, needed to identify the mechanisms behind the heterogeneous effects.

Implications and conclusions

Our findings have potentially important implications for researchers searching for evidence-based recommendations concerning how to maximize response rates on politician surveys. For those interested only in the overall survey completion rate, the lessons from this study are clear: financial incentives had no positive effect and are not indicated. On the other hand, important differences in the effects across parties suggest that researchers should consider how varied motivations for participation may affect the types of subjects likely to participate. Limiting the presumed motivation for participation to intrinsic rewards may bias the sample away from the potentially quite interesting subset of subjects who respond positively to extrinsic rewards. In our study, among those who received no financial incentives (control), ANC councillors turned in completed surveys at less than half the rate of councillors from all other parties (15.6 percent vs 32.6 percent). With material incentives, that gap shrunk substantially (19.8 percent for the ANC vs 26.8 percent for other parties). In other words, and quite plausibly, incentives have different effects for different types of politicians. For some purposes, that more balanced response rate might be desirable, especially if it elicits additional responses from incentive-driven councillors – a potentially important characteristic that is generally unknowable prior to survey enumeration. Thus, for some studies, it may be useful to do as we did – randomize assignment to such incentives, blocking on pre-treatment predictors of likely responsiveness. This would help promote the attainment of a more representative sample overall even if not a necessarily larger sample. In turn, treatment assignment should be (carefully) considered in analyses of the resulting survey data, recognizing that any subsequent effects on responses or on the selection of councillor types responding to the survey is conditional on agreeing to participate in the survey post-treatment.

Additional studies in other contexts, at varied levels of elite seniority, and with varying levels and types of incentives would shed further light on the findings presented here. For example, researchers could provide unconditional material rewards to elite subjects, and/or they could offer donations to a worthy cause, as well as possibly recognition of their participation. In the meantime, researchers conducting studies of politicians could simultaneously improve the representativeness of their samples and contribute to a larger body of methodological knowledge by randomizing the offering of material and non-material incentives for participation. As always, they should do so while paying close attention to context, especially the ethics of such experimentation.

Supplemental Material

Supplemental Material - Politician responses to material incentives for participation in surveys: Experimental evidence from South Africa

Supplemental Material for Politician responses to material incentives for participation in surveys: Experimental evidence from South Africa by Sarah Lockwood and Evan Lieberman in Journal of Research & Politics.