Value-added and Transparent Experiments

Abstract

Experimental research by political scientists on elites has grown dramatically in recent years. Experimenting on and with elites raises important questions, both practical and ethical. Elites are busy people, doing important work under public scrutiny. Therefore, any experiments that use up political elites’ time, risk impairing their ability to do their jobs as well as possible, or put at risk the larger research community’s access to elites should be avoided. Nevertheless, despite these risks and challenges, we argue experimenting with elites has enough benefits both to the research community and to elites themselves, that it should still be done. The relevant question then becomes how should we think about doing experiments with political elites? We propose a framework of value-added and transparent experiments. Our framework is guided by the following two simple rules: Elite subjects should individually benefit from the process of doing the experiment. It should add value to their role as representatives. Second, the identity of the researchers and purposes of the experiment should be transparent. As we argue, these two combined features can still accommodate a large range of experiments, can creatively spark researchers to think up new designs and can protect access to elites for future research. We review two such examples at the end of this essay.

Introduction

Audit experiments are widely used in political science (Butler and Crabtree, 2021; Grose, 2021). These experiments typically take the form of an experimental subject interacting with another individual or organization, except that the second actor is a confederate of the experiment (see Butler et al., 2012), and may not be a real person at all (e.g. Butler and Broockman, 2011; Pfaff et al., 2021). For example, the experiment may take the form of seemingly real constituents contacting a politician for assistance (Butler and Broockman, 2011; Loewen and MacKenzie, 2019), a parent contacting a school for information about registration (Pfaff et al., 2021), or a potential car buyer negotiating with a salesperson (Ayres and Siegelman, 1995). In a stylized form, an audit experiment involves a researcher examining how a subject behaves when they believe they are interacting with another, real person.

Audit experiments have seen a substantial growth in political science, especially in the study of political elites. ¹ This is for a good reason. First, it is difficult to learn about the behaviour of elites through either surveys or deep ethnographic or participant observation investigations. Any observational study of elites may involve problems both of survey or investigator demand and challenges of obtaining a representative sample (but see Sheffer et al., 2018 and Walgrave and Joly, 2018). By contrast, the nature of audit experiments means that even non-participation can be taken as an observation, and the one-sided blindness of the studies limits experimenter demand effects. Second, audit experiments are desirable because they allow us to study subjects for whom there is no student sample or general population substitute. As Druckman and Lupia (2012: 1178) observe, ‘typical experimental subjects often lack the experience needed to act “as if” they were professional legislators; yet, legislators themselves are often reluctant to participate in experiments as subjects’. Audit experiments present a ready, often low cost and seemingly easy-to-execute solution to these problems. They can and have made major contributions to our knowledge of politics.

Despite their appeal however, there are at least three reasons why we should be reluctant to engage in audit experiments – especially those conducted among elected officials (see also Desposato, 2018, 2021). First, we risk doing substantial harm to other researchers who also wish to access politicians for their studies. Modern democratic governance relies on a very small number of elected officials as a share of any population. A badly executed audit experiment has the potential to sour politicians on their willingness to engage with other academics, especially those doing non-audit experiments, as they will be more easily identified. Moreover, politicians often oversee research funding agencies, and so the more capricious among them may react to a badly executed experiment by reducing social science funding for all academics (Mervis, 2013). The small number of politicians thus makes audit experiments riskier than those done among much larger student samples, or the general population. An argument might be mounted that the experiments are worth the risk because they will expose corruption or malfeasance by some politicians. But this ignores an important problem of how audit experiments are reported. A research ethics board is unlikely to allow a study to identify individually malfeasant politicians. Instead, it will allow aggregate reporting. This does potential harm to non-malfeasant research subjects, who are now tarred as being ‘malfeasant on average’. Second, and perhaps more importantly, audit experiments risk doing real harm to non-academics. If an audit experiment is caught out, it risks casting doubt on every request for help to a politician, thus reducing the effort politicians will put into helping and communicating with constituents. Any reduction in the aid or information that politicians give to real citizens is a real and potentially substantial harm. Finally, audit experiments take up time from elected officials or civil servants. Time that presumably could be spent (and otherwise would be spent) doing the real work of being an elected representative or a bureaucrat.

These three observations can be generalized; the potential harms of audit experiments may outweigh the benefits of any individual experiment. The effects are not limited to just politicians, of course. For example, the school principal whose tolerance is tested (Pfaff et al., 2021) may not be able to legislate against such experiments or cut funding to universities, but he may still be more reluctant or suspicious of other inquiries in the future. Similarly, a car dealer may be more suspicious of minority customers (Ayres and Siegelman, 1995). While we do not wish to overstate the case here, we do note that the essays in this collection were motivated by just such an example of an otherwise important audit study raising the suspicion and ire of elected officials.

Value-Added and Transparent Experiments

What then should we do in place of audit experiments? We propose a framework of value-added and transparent (VAT) experiments. These are guided by two simple rules: Elite subjects should individually benefit from the process of doing the experiment. It should add value to their role as representatives. Second, the researchers and purposes of the experiment should be transparent. As we argue, these two combined features can still accommodate a large range of experiments, can creatively spark researchers to think up new designs and can protect access to elites for future research. This framework is motivated by the recent debate and controversies around political science audit experiments, but it is important to note the VAT framework is not limited to audit experiments per se but can be applied to any experiment involving political agents and elites.

We begin with the principle of ‘valued-added’. Our core claim is that participants in research studies should be compensated for their participation. Of course, this is more or less the norm with studies run online or in a laboratory. A corollary to this is that there should be no one who is not compensated for the costs of an experiment. Outside of audit experiments, there are two basic approaches to compensation. First, many subject participants are often compensated, whether through direct payment, prize money, course credit or survey incentives. And even when it is not compensated materially, it is voluntary. So we might assume from this that experimental subjects at least get some reward from participating, or else they would not. Second, and especially in the context of field experiments, a treatment manipulates the allocation of some resource or information that the subject would otherwise not receive. Provided that treatment does not cause undue harm, the subject (and those around them) is no worse off. In contrast to this, when elites are made to be subjects in audit experiments, they are not compensated for their time, which is often in very short supply (Sheffer, 2018). On the contrary, they are participating at the time cost of helping other people in the public. So, often their participation comes not only at a personal cost, but at some greater cost to others.

Our recommendation is not that we design studies in which politicians are compensated monetarily. This will be infeasible in many circumstances, a potential threat to inference and in some jurisdictions may be illegal. But we should recognize that we can design experiments that politicians can value participating in. We see, in general terms, three types of value that a politician may derive from an experiment itself. First, self knowledge. Elites may learn about themselves in the experiment in a way that helps them do their job better, or know themselves better. By reflecting on how they make decisions, who they choose to represent, what information they attend to, elites may gain new insights into how they can be better representatives. Second, knowledge about others. Elites may learn new information, about their constituents or an issue, in a fashion that will help them do their job better (see, for example, Butler and Nickerson, 2011). Third, consumption value. Elites may, like others, simply enjoy participating in an experiment.

As we see it, this generates a simple rule by which a researcher can judge whether their experiment is in fact a value-added experiment: Would the subject benefit from the experiment if it ends with their participation. If the data were never analysed, and the researchers never shared their results, would the subject still have received some benefit? Of course, some of this value-added element can occur even when the subject does not realize that they are in the experiment, as in most audit experiments. Crucially, however, some of this – maybe even a lot of this – will require them knowing they are in the experiment, and are engaging in sufficient self-reflection during the experiment to see the benefit of what they are doing.

There is a related consideration, that the knowledge generated by an experiment may have such social value that it outweighs the costs of individual participation. Subjects may participate in an experiment for that reason. However, they cannot choose to make that calculation in the context of an audit experiment, where they are not aware that they are engaged in that creation and transfer of value. Critically, this requires that they know they are in the experiment – that the basis for the experiment is transparent.

The second criterion we propose is that experiments be transparent. There are two main elements to this proposed transparency. First, that it be transparent who the principals are leading the experiment. We think there are two main reasons for this. One, consent arguably requires subjects knowing with whom they are engaging. This is not possible if the researchers are unknown to the subject. Two, we think researchers should take responsibility for their experiments and that they should view themselves as being in a relationship with elite subjects (Loewen et al., 2010). If experiments are viewed as one-shot affairs – and deeply one-sided ones in the case of audit experiments – researchers will have a little incentive to consider how much value they are providing – individually or socially – to the elite they are engaging. Moreover, by revealing who they are, experimenters are effectively making themselves more accountable to their subjects. At a minimum, this should help limit the fall out of bad experiments to other researchers.

The second element of transparency is that the goals of the experiment should be evident. Subjects should know, in terms at least as specific as those revealed to other experimental subjects, what is the purpose of the experiment they are engaging in. They should likewise be able to get a debrief as comprehensive as any other subject. Importantly, this may well help in unlocking for them all of the values of the experiment articulated earlier. Related to this, scholars can involve elites much earlier in the research process – for some types of experiments even as early at the design stage. This serves to provide more transparency but crucially it can also lead to better research designs, more buy in, more benefit for the elite partners and, likely, more future access. Wantchekon (2003), López-Moctezuma et al. (2020), Loewen and Rubenson (2011) and Dewan et al. (2014) all provide examples of experiments organized in cooperation with elites.

There are of course important potential objections to these conditions. We would highlight two, recognizing that they represent important trade-offs from audit experiments as presently executed. First, by making clear the objectives of an experiment and the researchers conducting the experiment, there may be various types of experimenter demands effects induced from subjects. As subjects first learn they are in a study, and then learn more about it, this may well cause them to consider how they are expected to act. There are a number of other well-known threats to both internal and construct validity that can be similarly induced once information about an experiment is known (Cook et al., 2002). Of course, once politicians learn about their participation in an audit experiment, they may update not only how they interact with researchers in the future, but also real people with real needs in the here and now. Second, transparency may well lead to differential selection into the experiment, threatening the generalizability of the results. Of course, there are potential mitigation strategies for all of these threats. In the end, researchers should consider whether these threats to validity are so great that they outweigh the consequences of conducting experiments with deception.

Our argument is not that audit experiments should never be conducted. They can, under some conditions, be justified. What we do hope to have convincingly argued is that there is a higher threshold for VAT experiments, and this threshold is worth meeting. We believe that academics are capable of weighing the costs and benefits of various research designs, and if they are not already, should be required to do so by their institutional research ethics boards.

To help demonstrate elite experiments which meet these conditions, we (rather unabashedly) share two different experiments which we think meet these standards, completed jointly with other coauthors. The goal here is to look at these experiments – and this is a sort of post hoc challenge for us – and identify whether they were transparent and what the value added to the politician was.

What are examples (or potential examples) of value-added and transparent experiments? One example is McAndrews et al. (2020). In this article, we were interested in whether politicians who are unelected – in this case, Canadian senators – are motivated to represent constituents who share their ascriptive characteristics. By examining the behaviour of unelected representatives, the notion is that any correlation between the concentration of some group and the election of members of that group would be broken, so any extra effort put to representing that group by a politician would suggest some intrinsic motivation to represent those groups to which they may belong.

McAndrews et al. explore this by sharing with the senators real public opinion data (Loewen et al., 2018) on several issues. The key design feature of this experiment is that when they were presented with data, the politicians were asked to choose a breakdown of the data. It could be viewed by region, gender, age group, ethnicity and so on. By examining whether politicians chose breakdowns that were more relevant to their own identities, the researchers could back out whether politicians had some intrinsic motivation.

Why does this qualify as a VAT experiment? And how might it have been done instead as an audit? First, the experiment offered clear added value by sharing with politicians data that were relevant to their own work. By understanding various breakdowns of opinions and preferences, senators could arguably better represent interests in their work. Transparency came from an existing relationship with senators. These data were shared as a part of a larger project – the Local Parliament Project – in which the researchers regularly and transparently shared public opinion data with Canadian politicians. Senators were offered background information on the project in their invitation letter and on the project’s website. Accordingly, politicians could ascertain that the data being shared came from a legitimate source. And if they had any objections or concerns about the study, they could directly contact the investigators (or their research ethics board).

In another experiment (Loewen et al., 2021), we were interested in studying whether elected representatives seek out more information on an issue when they are further offside the average opinion in their constituency on that issue. We partnered with a leading Canadian school of public policy, and jointly invited Canadian members of parliament (MPs) and their staff to a webinar on the issue at hand – oil pipelines – where a variety of viewpoints were presented. A random subset of MPs also received information ahead of the webinar about the distribution of opinion of their constituents on this issue.

This is an example of a value-added transparent experiment. If they wanted, politicians could learn about the issue by attending the expert webinar and/or reading the transcript and report from the experts. This added value to them and was to their benefit. While it clearly cost time, it was time spent doing real work, engaging with real experts and learning about a salient issue. There was also transparency since MPs and their staff knew the webinar was hosted by a university unit led by one of the researchers.

Conclusion

Audit experiments with elected and unelected political elites have proliferated in political science in recent years. One central critique of these studies is that the researchers who conduct them have not internalized the costs in terms of time and effort on the part of their research subjects – in this case, elites who perhaps instead should be spending their time working in public interest. While a legitimate and important criticism that we take seriously, we also argue there are ways to carry out experiments with elites that take advantage of the opportunities for learning by studying elected representatives, their staff or civil servants in a way that avoids some of the main pitfalls of audit studies. In this short article, we have outlined a model of value-added and transparent experiments. The experiment should add value to elite subjects in that they should individually benefit from the process of doing the study. We also argue for more transparency. The identity of the researchers and purposes of the experiment should be known.

Audit experiments have grown in popularity partly as a result of the ease and low cost with which they can be deployed to study novel and exciting research questions. But a more deliberate approach that takes seriously the costs to our elite subjects and indeed provides them with value can still, we argue, accommodate a large range of experiments, can creatively spark researchers to think up new designs and can protect elite access for future research.