Public Impacts from Elite Audit Experiments: Aggregate and Response Delay Harms

Abstract

What are the potential harms from elite audit experiments? Such studies seem genuinely harmless and all in a day’s work for elites. In addition, such studies may provide valuable information about elites’ performance and service to their publics. However, there are a number of potential harms of such studies which are generally not captured by the standard human subjects framework. In this essay, I consider the various harms that can result from excessive experimentation on elites. I identify several previously ignored public harms, including aggregate and response delay harms. I offer several potential strategies to assuage and avoid these harms.

Introduction

Audit studies of elites are a useful research tool for evaluating the performance of public institutions and powerful actors. Scholars have used these types of studies to examine race and representation, the influence of money in politics, cross-ideological responsiveness, transparency, and many other topics. ¹ Audit studies of elites are frequently used to examine the performance of elected officials, but have also been used to study bureacracies, including school districts and local housing authorities. The utility of the design is reflected in its widespread adoption, with audit studies performed by political scientists in many countries and contexts.

Audit studies rely on deception and a lack of informed consent in order to measure real-world behavior and responsiveness. If members of the parliament (MPs), for example, knew they were being studied, they would probably be careful to respond to researchers’ inquiries cautiously and equally, regardless of race, gender, or partisanship. When they are unaware of the research, then any bias, misrepresentation, or discrimination can be observed and measured.

These studies offer potential benefits with usually small costs to subjects. The typical treatment is often just an email to the public officials, usually asking them to provide some service that is part of their job responsibility. In the US, members of Congress are estimated to receive an average of nearly 400,000 constituent communications annually(Grose, 2016). In such an environment, the marginal cost of one additional request would seem to be quite small. Yet, when subjects detect audit studies, they generally respond with anger and complaints. But as public officials, does their anger indicate meaningful harm that deserves our consideration?

In this essay, I examine the potential harms of audit studies of elites. I identify potential subject, public, and disciplinary harms. I argue that although the individual subject harms are usually small and justifiable, other types of harm can be severe and should be considered by researchers. These new types of harm include disciplinary harm which affects all research as well as public harm which involve the exploitation of public resources for research purposes. In particular, I identify the following two types of public harms: aggregate harm and response delay harm. I propose a framework for assessing the harms of such studies and offer recommendations to minimize such harms.

To be clear, in the discussion that follows, I focus on audit studies of the following type. Researchers pretend to be some interested party and contact the subjects with a question or request. Subjects are all public officials, either elected leaders or powerful bureaucrats. ² Researchers then measure whether and how public officials respond. I am assuming that data are anonymized and individual subjects are not identified. ³ I am not considering audit studies that involve private citizens or audit studies that could pose severe risks of physical harm to subjects. ⁴

Harms from Audit Studies

Audit studies may involve a variety of harms. Typically, the individual harms to subjects are modest, perhaps involving a small waste of time by subjects or some risk of embarrassment. The generally small nature of these harms as well as the public responsibilities of the subjects usually justifies the conduct of audit studies without any informed consent. However, these studies also involve public harms that can be quite substantial. These public harms are often overlooked by scholars and by ethical committees as they are unobserved and dispersed. I will argue below that these harms are usually the most consequential harms of audit studies and deserve careful consideration. There are also potential disciplinary harms when studying powerful political actors, as excessive or offensive experimentation may ultimately lead to action by governments against research generally.

Subject Harms

Subject harms are those suffered by the subject. These might be individuals, like an MP, a member of their staff, or a bureaucrat. These could also be institutions, like an agency, a business, or a team. In Western democracies, these are usually limited to the harms of deception and wasted time. When public figures discover an audit study, they generally express anger and frustration. However, many of these individual subject harms may be considered justifiable given subjects’ choices to occupy positions of power and authority, and given that research on power, justice, and representation are in the public interest.

The Harm of Deception and Nonconsenting Subjects

Audit studies lack informed consent and are deceptive in that researchers pretend to be someone they are not—they lie about their identity and intentions. ⁵ In many cases, this is “deception with implied benefit” (Desposato, 2016a) where the subject believes that their sincere interaction will yield some benefit. ⁶ The subject (MP) believes they are doing their job and advancing their career by responding (or not) to an audit study, but they are in fact just contributing to the scholar’s research. Regardless of the magnitude of the intervention, one may argue that deception and research without informed consent are directly harmful to subjects as they reduce subject autonomy and violate respect for persons. This lack of consent conflicts directly with foundational norms of research with human subjects that give primacy to the voluntary participation of subjects. ⁷

Subject Harm: Time Wasted

A second subject harm is that of time lost. In some cases, dealing with an audit study’s request could be as easy as sending a standardized reply, and the time lost could be trivially small. However, when subjects have limited staff and budgets and receive few inquiries, responding to some requests could be laborious. In one study of US city governments, researchers sent inquiries about the possibility of a foreign firm relocating to their jurisdiction. ⁸ Staff in the City of Los Angeles might have had a relocation packet already prepared and responded quickly with minimal effort or time wasted. But staff in the City of Eureka ⁹ might have invested hours of effort into crafting a response to such an inquiry. One problematic feature of audit studies is that the harms imposed by the research are never directly observed. Scholars may estimate the effort required to respond to an inquiry, but the surreptitious nature of the research means that actual harms are never known and also remain hidden.

When Are Individual Subject Harms Justifiable?

When subjects are private individuals with no public life, these harms may not be justifiable. ¹⁰ However, for individuals who have chosen a public life and exercise substantial authority and who control access to public resources, harms of deception and some reasonable degree of wasted time are often considered acceptable. In fact, some would argue that political scientists have a responsibility to conduct research on the performance of powerful actors and institutions.

For example, the American Political Science Association (APSA Ad Hoc Committee on Human Subjects, 2020) recently adopted a set of Principles to guide research involving Human Subjects. They examined harms to public officials from audit studies and other types of research. APSA concluded that in some cases, it would be appropriate to conduct research without consent that might cause more than minimal harm to public officials. Indeed, APSA acknowledged that such research might even be a duty of scholars: “Assessing the performance of public officials and other powerful actors and their role in political systems is seen by many to be both a specific responsibility of political science and a public service” (APSA Ad Hoc Committee on Human Subjects, 2020: 3).

At the same time, APSA’s new Principles also encourage scholars to minimize harms and consider broader impacts. The Principles require that scholars minimize harm to subjects, seek consent whenever possible, and justify choices to cause harm, employ deception, or fail to seek consent from subjects. In addition, APSA requires that scholars also minimize harm to “other parties affected by the research” (APSA Ad Hoc Committee on Human Subjects, 2020: 4). Other party harms would include the public harms of audit studies discussed below as well as disciplinary harms caused by backlash against research generally. ¹¹

Another challenge is that of defining who should be considered public officials, and thus, be appropriate targets for audit studies. As others have noted, some leaders and bureaucrats might deserve additional protections and might not be appropriate targets for audit studies (Desposato, 2016a; Malesky, 2016). For example, village chiefs and volunteer firefighters, given their limited public authority, would deserve more protections than department heads or provincial governors.

Public Harms

A second category of harms are those inflicted on members of the public. Generally, for individual members of the public, these are unperceived—both atomized and anonymized. As a result, members of the public generally do not complain about these costs, because they are unaware of them. Nevertheless, they are likely the more serious and consequential of the harms, and the ones that deserve more attention. In each case, the potential very large scale of some audit studies creates and aggravates the potential harms.

Aggregate Harm

The most common public harm from audit studies is that of aggregate harm: the total loss of public service due to an audit study, combining all subjects in the study. Consider an audit study that involves sending a letter to elected officials. Responding to the letter takes each elected official $t_i$ minutes. The harm to the public from subject $i$ ’s participation in the study is thus $h_i=t_i\times c_i$ , where $c_i$ is the cost per minute of official $i$ ’s time. We may define the aggregate harm as,

$$H_A=\sum _{i=1}^N{h}_i$$

For example, if a single intern answers researchers’ correspondence as part of a study, perhaps her time costs US$15/hour and the response takes 15 minutes. Then the cost to the public is US$3.75—this is publicly-funded time, paid for by taxes, that is being captured by the researcher for her own purposes. This cost would seem fairly low and an acceptable price for an important study on racial discrimination and representation.

However, suppose the same study involves 10,000 subjects, ¹² the correspondence takes 30 minutes, and each public official’s time is worth US$50/hour. ¹³ Now the aggregate cost is,

$$H_A=10,000\times 50\times \frac{1}{2}=US\$250,000$$

and the total amount of time spent by unwitting subjects is 10,000 × 0.5 = 5000 hours. This is equivalent to over 2.5 years of full-time labor in the US. ¹⁴

In this example, an audit study captures a substantial amount of publicly funded time paid for by taxpayers. It may indeed be a good use of public resources that contributes to an important research finding. But this use of US$250,000 of public labor was accomplished through deception and fraud—effectively a form of theft from the public. While this effort may support research on an important topic, the public did not allocate these funds through a transparent grant application process. Instead, these resources have been effectively stolen to support research. ¹⁵

Because existing ethics committees focus on harms administered to individual subjects, these broader atomized costs are usually ignored. But they can be substantial and represent a misuse of public resources. From a cost-benefit perspective, the aggregate harm of an audit study may be no different than siphoning off electricity from a public utility or cheating on one’s taxes.

An ethics committee would never allow a research project that deceived individuals into dedicating a year of their life, uncompensated, in support of a political science experiment. Yet, because human subjects protections norms are oriented around individual harm, these aggregate costs are generally ignored. Especially with an increasing number of very large audit studies being conducted, these aggregate harms must be considered.

Response Delay Harms

A second type of harm affects the subset of constituents who are seeking assistance or information from an official. In an experiment targeting MPs, complaints from MPs’ offices noted their busy schedules and heavy constituency service burdens during the COVID-19 pandemic(BBC, 2021). These complaints suggest examination of response delay harms: the harms to constituents when their requests are delayed by audit studies. ¹⁶ Suppose that an intervention results in a 15-minute delay to other constituents’ inquiries being addressed. These delays have potential individual and broader harms. For most individual constituents, a small delay is not likely to cause more than minimal harm. But if thousands of constituents’ requests for help were delayed, some of them might suffer more than minimal harm because resolution of an issue did not come quickly enough. ¹⁷

Even when no individual constituent suffers a significant harm due to response delays, the total response delay harm may be excessive. Suppose that each MP $i$ of $n$ MPs in the study has a backlog of $B_i$ constituent communications or requests. The research study will result in a $t$ minute delay in response to each communication. Each constituent in the queue experiences this delay. If there are 500 MPs ( $n=500$ ), each with an identical backlog of 1000 constituents ( $B_i=1000\forall i$ ), and dealing with the audit study’s request takes 15 minutes ( $t=15$ ), then the total response delay harm ( $R$ ) for constituents can be calculated as:

$$R=\sum _{i=1}^n{B}_i\times T=500\times 1000\times 0.25=125,000\mathrm{hours}$$

In other words, an audit study of 500 legislators that takes their staff 15 minutes could increase total constituent waiting time by 125,000 hours, or above 14 years. ¹⁸ Costs could be even higher with different (although unrealistic) assumptions. Assuming a constant backlog, an audit study could increase the wait time for every subsequent inquiry, for years into the future. Such a framework would lead to potentially infinite response delay harms.

Qualifications and Assumptions

Any calculation of aggregate harm and response delay harm requires some assumptions about public employees and their use of time. Calculation of such harms involves a counterfactual framework: how would the subject have used the time spent responding to an audit study, in the absence of the study? Had they used that time to check social media or chat at the water cooler, then these public costs would have been 0. If subjects were to extend their workday to deal with an audit study’s email, then the public cost would also be effectively 0. In these two cases, the harm would only be to the subject—private harm in time lost, without any public harm. Similarly, if subjects were to triage their work obligations and put an audit study email aside until requests addressing basic needs were met, the public costs would have been similarly small. Finally, response delay harms are largest when there is a large backlog, but can be reduced dramatically if backlogs are often cleared completely or if delays are not calculated assuming continuous costs. ¹⁹

Notably, we generally have no idea what the true cost of responding to an audit study is, because the surreptitious nature of such studies means scholars do not observe or measure these costs. Subjects are usually not debriefed and do not provide any feedback on the experiment, unless, as in recent controversies, they detect the research. Given the large scale of many field experiments and the potentially large public harms, aggregate and response delay harms should be considered in ethical calculations.

Public Permission

Since concern is with public harms, it would seem that audit studies should be asking the public for permission to conduct such studies. There is evidence of some support for these studies. For example, in the few comments after Professor Campbell’s note regarding her MP study(Campbell, 2021), readers noted that “I was genuinely interested to see the findings” and “I think this would be a valuable piece of research.”

More generally, in previous work, I conducted a survey of a sample of US residents and asked them to judge a hypothetical audit study. I randomly varied several features of the study to see how they viewed these designs. I will share the following three findings, and refer the interested reader to Desposato (2018):

1. Respondents had lower ethical evaluations of larger studies, suggesting that excessive use of public time is not favored by the public.

2. Respondents had lower ethical evaluations of studies that required more time of subjects and were more of a burden on them.

3. Respondents had higher ethical evaluations of research that seemed to have a clear social value (investigating discrimination by officials) but a lower ethical evaluation of basic research with less-clear social value (researching the communication style of officials).

This survey (although only conducted in the US) suggests that the public is aware of the costs of large and burdensome audit studies, as well as the potential benefits of research in the public interest. To justify audit studies using public resources, additional research on public opinion on these issues may illuminate the path that political scientists should take. Widespread support for such studies might be viewed as a form of societal consent. ²⁰ Widespread opposition could indicate, however, that researchers should avoid audit studies. Research on the public’s view of such studies could also reveal which features of audit studies are most and least tolerated, aiding in the design of appropriate experiments in the future.

Scholars could also explore alternative forms of societal consent for audit studies. Such alternative forms could include public announcements with opt-out provisions, superset consent, and many others. Any of these would help provide assurance that scholars are not misusing public resources. ²¹

Disciplinary Harms

Besides subject and public harms, these types of studies may also cause disciplinary harms. These are potential harms to science created by alienating powerful actors. Public officials control funding for research, for higher education, and also control regulations of such research. Political science in the US has faced a number of challenges from elected officials, including a temporary elimination of research funding for political science through the National Science Foundation and a change in human subjects rules to restrict the ability to conduct audit studies of public officials. ²²

Research that angers powerful actors may be necessary—but runs the risk of harming the entire discipline. Given the potential consequences, it would seem appropriate for scholars conducting audit studies to give consideration to these risks, and weigh them against the potential benefits of the research.

Discussion

Audit studies of elites are an important tool and studies of elites and powerful institutions are a responsibility of political scientists. However, scholars can easily overreach and cause excessive harm to individuals, to the public, and to the research enterprise.

Avoiding these harms requires accounting for harms that are not part of the usual human subjects research checklist—the aggregate harms to the public, the response delay harms to constituents, and the broader disciplinary harms. Both aggregate and response delay harms have at least two parameters that scholars can manipulate to reduce these harms; the amount of time needed to respond to an inquiry and the number of subjects contacted. The implication is that audit studies should use minimal interventions that are generally easy for subjects to address and that scholars should not treat more subjects than is necessary.

A first step toward minimizing the burden of audit experiments is to measure these costs. Such an accounting is extremely rare, although scholars could easily obtain an estimate by contacting a small number of current or former public officials and simply asking them about the burden of a potential experiment. In addition, many political scientists have worked in politics, have handled constituent requests, and should be able to provide reasonable and meaningful estimates.

One potential downside of minimizing the burden of audit studies is that trivially easy requests might not capture quantities of interest. For example, it may be that MPs only discriminate against some types of constituents when facing difficult requests, not when facing easy ones. Yet, this is an empirical question that scholars can investigate.

Scholars can also minimize public harms by avoiding excessively large studies(Desposato, 2016b, 2018; Louwerse et al., in press). Scholars should conduct a power analysis and manipulate the minimal number of subjects. The discipline right now seems to privilege n-maximization, with some audit studies contacting tens of thousands of subjects. Studies of that size are not needed, unless scholars are compensating for something, perhaps very, very small effect sizes that might in fact be too small to be of substantive concern. One institutional change that might deter unnecessary n-maximization is a norm of compensation to the public harmed by such studies, as discussed elsewhere in this volume. A related change that might discourage excess would be a norm that scholars include in published work a estimate of the public costs of their research in both time and money.23

Beyond harm minimization, much additional investigation of the ethics of audit studies is needed. One issue involves how to weigh harm when subjects are not public officials. For example, research on businesses presumably has some impact on owners or shareholders. It may also reduce performance metrics for the employees with whom the researcher is interacting. Another issue that scholars should consider involves the risks to themselves and their institutions when conducting especially aggressive studies. Some audit studies might be considered a form of fraud and lead to litigation. This may seem unlikely, but an audit study of New York businesses was detected by the subjects and led to a lawsuit against Columbia University (Crohan, 2014; Gajda, 2010; Kifner, 2001).

I offer two final suggestions. First, do good. Members of the public are more supportive of scholars using taxpayer funded labor to study questions of pressing social concern than obscure points of basic science. Second, our discipline needs more empirical research on audit studies: their actual burden on subjects and the public, the number and magnitude of studies being conducted, and the public’s view of this work.

Audit studies are a useful and powerful tool, and political scientists have a responsibility to study the performance of powerful public actors. However, scholars should be cautious to avoid wasting public resources and harming constituents in need of assistance.