Introduction to Symposium: Experiments with Politicians: Ethics,Power,and the Boundaries of Political Science

Abstract

This introduction sets out the context for this symposium, which is the discontent expressed by UK MPs and the Speaker of the House of Commons in March 2021 about a research project using e-mails from fictitious constituents to audit the responsiveness of legislators to constituent emails. The article reviews the research literature on experiments on politicians and summarises the debate in the academy about the ethical conduct of these randomised controlled trials. Contributors to the symposium defend and challenge approaches to carrying out these elite experiments, whether using fictitious identities, confederates, and/or partnerships with politicians, as well as refine a cost-benefit approach to the design of these studies.

Studying politics is about power. To research power, it is natural that political scientists should wish to conduct research on the very people who exercise it, the politicians themselves. As a result, politicians frequently appear as research subjects in classic studies like Fenno’s (1978) Home Style and Searing’s (1994) Westminster World. Politicians have also been participants in survey experiments (Naurin and Öhberg, 2021), and in observational survey research (Norris and Lovenduski, 1995). While many standard considerations for survey and interview methods apply to politicians, researchers are also aware that it can be hard to secure responses from and interviews with these busy people, and that politicians are more likely to dominate interviews rather than answer questions passively (Goldstein, 2002).

The rediscovery of randomised experiments in political science in the late 1990s transformed the study of mass and elite political behaviour, introducing a new method to study a range of research questions (Druckman and Green, 2021). While citizens and voters were the subjects of the first wave of field experiments, researchers quickly realised that elites could also be studied experimentally with reasonable levels of statistical precision (Grose, 2014). Successful audit studies have been conducted with legislators, which permit researchers to measure responsiveness, accountability, and discriminatory behaviours, many done in the US (Broockman, 2013; Butler and Broockman, 2011; Butler and Nickerson, 2011; Kalla et al., 2018; Nyhan and Reifler, 2015), as well as in other contexts, such as the UK House of Commons (Habel and Birch, 2019), British local government (Crawfurd and Ramli, 2021), Denmark (Dinesen et al., 2021), New Zealand (Golder et al., 2019), Canada (Dhima, 2020), the European Parliament (de Vries et al., 2016), and the Global South (for example, Grossman and Michelitch, 2018; McClendon, 2016). Recently, comparative elite experiments implement the same research design across countries (for example, Magni and de Leon, 2021).

All those who participate in research should expect fair treatment and respect for their autonomy. At the same time, there is the legitimate need for academics to be able to ask a full range of research questions. Achieving this balance is a core aim of research ethics. However, politicians are different from other research participants by virtue of the power they exercise. Politicians may use this power to prevent studies on themselves from happening, or at least to deter them. If politicians are unlikely to approve of studies conducted without their consent and using deception, should researchers still be permitted to carry them out? One answer could be yes, in specific circumstances, which is why the American Political Science Association (2020) recognises that ‘when conducting research with powerful parties, including some public officials, other actors, institutions, and corporations, covert or deceptive research with more than minimal harm may sometimes be ethically permissible’. But some academics worry about carrying out research with deception and without consent (Desposato, 2015), favouring instead collaboration between researchers and politicians (Butler, 2019).

Many of these issues came into the fore over the research project led by Rosie Campbell at King’s College London (KCL), which in February 2021 tried a well-known approach to measuring the responsiveness of elected representatives to e-mails from ‘fictitious constituents’ by randomly varying the senders’ perceived identities. As required by KCL’s Research Ethics Committee, project members debriefed the research subjects, which created furore among MPs and a Twitter/media storm. The Speaker of the House of Commons complained to the Principal and President of KCL, and also wrote to the Chief Executive of the Economic and Social Research Council (ESRC), which funded the study.

This symposium is designed to encourage debate on these matters, such as the power of politicians to influence the agenda for research, types of deception seen as necessary in elite experiments, the practice of debriefing, compensation for perceived harms, and alternative approaches to working with politicians. The contributors advance different perspectives so readers can make up their own minds about the issues raised and the solutions researchers and politicians could adopt. This symposium contributes to the lively debate about ethics in field experimentation (see Phillips, 2021 for a more general overview, and Nathan and White, 2020 for the related case of street-level bureaucrats).

Rosie Campbell and her co-researcher Diane Bolet (2021) kick off the symposium by explaining what happened. They outline that the project had the highest aims, asked an important question for the workings of democracy, followed best practice over its implementation, and respected ethics procedures. The ethical approval is not in question, even though the KCL ethics committee went beyond other universities in requiring a debriefing (contrasting, for instance, with Habel and Birch, 2019). While debriefing subjects is a standard method used to reduce the costs of deception, informing politicians in this case did not assure them of the study’s ethics.

The contribution by Pedersen et al. (2021) outlines the wider comparative project in which the Campbell/Bolet study is part. They demonstrate the serious comparative research questions asked and range of reputable funders involved. Their account also reveals that the requirement to debrief was not just an artefact of KCL’s Research Ethics Committee, but was required in Denmark and prompted the researchers there to withdraw their study. The contribution reveals the diversity of practices for ethical review, which vary across as well as within countries and indicate the lack of consensus about how to deal with elite experiments. The article also highlights that the decisions involved in designing and conducting the study involved real ethical trade-offs and assessment of costs and benefits. The authors discuss some alternative design choices they could have made, for instance the use of confederates, and explain why they opted against using them.

Bischof et al. (2021) discuss the advantages and limitations of using confederates and eliminating identity deception, in detail. Confederates are MPs’ constituents who are recruited to the research project and express genuine opinions to politicians (see also Butler et al., 2012; Breunig et al., 2020; de Vries et al., 2016). The authors argue that identity deception is not a strictly necessary design element of audit studies. Moreover, identity deception can increase measurement bias as politicians and their staff can check whether enquiries originate from constituents with known addresses. Some politicians did exactly that in the Campbell/Bolet study. Precautions are likely to further increase in the wake of the murder of the MP David Amess in October 2021, as staff will be more cautious about hearing from people who are not constituents. As Bischof et al. (2021) acknowledge, two other types of deception, motivation and activity deception, cannot be eliminated without increasing the probability of inducing the type of bias that audit experiments are meant to counter in the first place. One practical limitation of this method is that large numbers of confederates within each constituency are needed, which means that confederate studies are more costly to implement and difficult to scale up. Rather than as a problem, some contributors to this symposium see increasing the costs of audit studies as an important means of improving ethical standards (Bischof et al. 2021). Butler and Desposato (2021) propose offering politicians donations, either in money or time, to a pre-registered charity to offset some of the costs or harms caused by audit studies. Time spent responding to e-mails was mentioned by MPs after the Campbell/Bolet study became public. Donations to charity might also signal the higher social purpose of the research project, which could make the use of deception more palatable to politicians.

Both Desposato (2021) and Crabtree and Dhima (2021) advocate the use of cost-benefit analysis. Desposato (2021) conceptualises the types of public harms that audit studies can cause: aggregate and line-cutting harms, which go beyond the minimal individual harms considered in Pedersen et al. (2021). He proposes that researchers minimise those harms by thinking carefully about the total number of hours legislators spend answering enquiries. Crabtree and Dhima (2021) mostly focus on the benefits of audit studies. Audit experiments provide larger benefits to learning than alternative means of gaining an answer to questions about discrimination because the estimator is unbiased. Crabtree and Dhima’s (2021) also make sensible recommendations to journal editors, such as the introduction of registered reports that allow for feedback from reviewers before studies are fielded. Overall, cost-benefit analysis is a useful framework in which to think about the design of studies and can help researchers make more effective and ethical design choices, discounting some design elements and studies that are high cost and low benefit. Whether cost-benefit analysis alone is enough to address the moral outrage that some politicians express at audit experiments is another issue.

Finally, there is a more collaborative approach to working with legislators, which appears in contributions by Zelizer (2021), Loewen and Rubenson (2021), and Cowley (2021), the last author writing from outside experimental political science. Cowley (2021) reminds experimentalists that there are many existing research contacts with MPs and he highlights the general virtue of good relationships between researchers and politicians. His practical tips can apply to both experimental and non-experimental political scientists. Zelizer (2021) argues that instead of research in legislatures being an intrusion, a new research programme could offer clear benefits to both the scientific understanding of how legislatures and legislators work, and be a genuine aid to politicians. Loewen and Rubenson (2021) advocate a transparent approach to elite experiments, which can contribute three types of value: self-knowledge, knowledge about others, and consumption value. They argue that Value-added and Transparent (VAT) Experiments should involve politicians at the design stage of research. The draw-back, as Pedersen et al. (2021) argue, is that briefing individual politicians at the design stage might rule out some research questions, such as those asked in Campbell and Bolet (2021). As an alternative to individual consent, a body representing politicians’ interests, such as in the UK, for instance the Speaker of the House or a joint committee of MPs and academics, could review studies in advance. However, there is also a danger of giving powerful actors a de facto veto over research designed to highlight problematic practices and patterns of discrimination. Citizens have rightly been critical of instances when politicians use their power to judge their own behaviour (Clarke et al., 2018). Zelizer’s (2021) contribution can also be read as a call to move from the first generation of audit experiments to ‘evidence-based legislating’ and ‘interventionist research in legislatures’. While these second-generation studies follow a collaborative approach, audits may still be needed to highlight where representative institutions and processes do not function well. Whatever the way forward, there needs to be continued debate on experimental ethics and the value of independence from power in the academy.