Abstract
With the advent of remote-video technology and recent pushes to include video feeds in U.S. Supreme Court hearings, many are concerned about the effect that video and streaming might have on the behavior of U.S. judges and court participants. Previous research has shown that judges react to public sentiment, and anecdotal evidence suggests that introducing video recording might induce greater levels of “performative judging,” where behavior changes when there are cameras in the courtroom. We use AI-powered diarization to analyze the transcripts of oral arguments, leveraging the quasi-random adoption of video feeds in the U.S. Ninth Circuit Court. We find suggestive evidence that judges speak more and are more likely to interrupt attorneys when proceedings are video recorded rather than only audio recorded. However, these results are mixed and model dependent. When our estimation strategies account for interactive effects or include only judges that spoke in both audio-only and video hearings, we observe a camera effect for some outcomes of interest. By contrast, non-interactive models that include all judges result in null findings. These mixed findings provide novel but preliminary evidence exploring the performative judging hypothesis and suggest that courts and policy makers should consider the potential effects on judges’ behavior when deciding whether to introduce cameras into the courtroom.
1. Introduction
Cameras are ubiquitous in modern society. From security cameras and dashboard cams to webcams, bodycams, and cell phones, we are constantly surrounded by devices that record video. We have also become increasingly accustomed to the acts of recording and being recorded. When we enter the public sphere, we know (expect, even) that at any moment, someone might be capturing our image. And when we return to our homes at night, we divert ourselves by browsing platforms that host videos and images of others.
But if we have become accustomed to and accepting of recording devices in most places, there is one public space that has largely evaded the camera’s reach: the federal courtroom. Though federal judicial proceedings are generally open to in-person, public observation, most courts restrict photography, video recording, and streaming. And though some commentators believe that video broadcasts could make trials more open, fair, and accessible (McElroy, 2012), others (including many judges and lawyers) have long feared that introducing cameras into the courtroom will corrupt the judicial process by transforming proceedings into “performances” (Estes v. Texas, 1965; Marder, 2021).
Over the last 50 years, academics, legislators, and the Judicial Conference of the United States have explored these conflicting perspectives by researching the risks and benefits of cameras in the courtroom (Cohn & Dow, 1998; Johnson & Krafka, 1994) and by experimenting with new rules regarding courtroom recording (“History of cameras,” n.d.). These efforts have shifted the no-camera norm somewhat: Cameras are now allowed in most state judicial systems and during certain (mostly ceremonial) federal proceedings. But for the most part, the no-cameras-in-federal-courts rule is firm. 1 And so, the debate over courtroom cameras continues. Media organizations and attorneys frequently petition federal courts for permission to record, livestream, and broadcast proceedings. And ever since the COVID-19 pandemic, which prompted temporary rule changes that allowed broadcasts and live streaming of Zoom proceedings, scholars and advocates have begun more seriously contemplating the benefits and risks of “virtual courts” (Rossner & Tait, 2021). In short, despite society’s penchant for recording generally, the idea of cameras in U.S. federal courts remains both unsettled and unsettling.
Because cameras in court remain controversial, there is no shortage of academic research on the topic. Thus far, though, nearly all that research has been either theoretical (discussing risks and benefits in abstract, general terms), survey-based (asking judges, jurors, and advocates to report their experience with cameras in court), or focused on public attitudes (assessing how cameras might change the public’s perception of court proceedings). 2 To our knowledge, there has been no empirical research to confirm the accuracy of the effects (or lack thereof) that judges, advocates, and jurors have reported in survey research. We have also identified no empirical research testing whether or how cameras might affect federal judges’ behavior and performance in court. 3 Consequently, while there is ample abstract discussion about the potential effects of courtroom cameras, we ultimately know very little about how video recordings or publicly accessible video feeds might influence the actual behavior of judges, attorneys, or juries (Moore et al., 2021).
In this paper, we begin to fill these gaps by utilizing a novel dataset to analyze whether and how federal appellate judges change their behavior in oral arguments when there are cameras in their courtrooms. We take advantage of a natural experiment in the U.S. Court of Appeals for the Ninth Circuit, which implemented a short-term, quasi-random camera rollout and then a sudden, full-circuit adoption of video cameras and live feeds. We use this progression to study how judges react when they are on camera compared to when they are not. Using a deep learning diarization methodology called Reference-Dependent Speaker Verification (RDSV, Tumminia et al., 2021), we automatically identify which parts of a court recording are judge speech and which judge is speaking, which allows us to calculate the proportion of each court proceeding taken up by the judges, how frequently the judges speak, how long they speak for, and how often they interrupt lawyers (as proxied by the length of individual lawyer utterances).
Our trendline data from the period when the Ninth Circuit transitioned from audio-only to video recording suggests that in cases with video recording, judges tend to speak longer and take up a larger share of proceedings compared to the attorneys. We also see that judges in camera cases interrupt attorney arguments more frequently, as proxied by attorney speech, which decreases in both overall length and average utterance length in camera cases. For both judicial and attorney outcomes, we see either a reversal in trend lines or a stark discontinuity—and sometimes both—at the same time the Ninth Circuit began introducing video cameras.
However, our identification strategy is inherently limited, and when we subject the data to more sophisticated estimation techniques, we find that the results are model dependent: Our specifications that include all judges and control judge characteristics result in null findings. This is possibly due to inherent imbalances in the some of the measurable characteristics of the comparison groups, such as the gender composition of the judicial panels. However, when we limit our dataset to only judges who participated in both audio-only and video hearings, we find significant effects consistent with our observational findings even when controlling for our full set of covariates. And when our models include interactions between the existence of cameras and those same judicial characteristics, we find that cameras have greater effects on certain types of judges: White and male judges interrupt more often but speak for shorter durations when they are on camera, and non-white judges speak longer and interrupt less.
These findings have important implications. First, our research provides preliminary (albeit mixed) support for the oft-cited theory that cameras change judicial behavior. Prior to this study, there was little empirical backing for the hypothesis that cameras cause judges to behave performatively. 4 Our study does not definitively prove or disprove that hypothesis, but it does provide novel empirical data that suggests, at the very least, that there is more to investigate. Our study is thus a helpful springboard for courts, policymakers, and future researchers who want to think more carefully about the behavioral effects of cameras in federal court.
Second, although this study does not measure the effect of judicial performance on case outcomes, we find evidence suggesting that the effect of cameras may interact with the racial/ethnic and gender composition of the attorneys and the panel of judges in the case. This speaks to recent concerns about the role that gender (and to a lesser extent, race and ethnicity) plays in the back-and-forth nature of oral arguments in U.S. federal courts (Cai et al., 2024; Jacobi & Schweers, 2017) and may provide even more reason to be cautious—or more deliberate—about introducing cameras in those proceedings. 5
Finally, this study demonstrates the potential for machine learning tools such as Reference-Dependent Speaker Verification to shed light on the content, interactions, and nature of oral arguments in the U.S. circuit courts. While some (including ourselves) have used such methods to explore the more commonly-studied U.S. Supreme Court (Cai et al., 2024; Tumminia et al., 2021), future researchers—or even the court system itself—could use these same methods to explore more courts (that feature a more diverse set of speakers) over a broader time period, especially if future developments in AI-based audio processing allow for more cost-effective and precise diarization.
This paper proceeds in five parts. In Section 2, we provide a brief history of the rules and policies governing cameras in federal court. In Section 3, we review the existing academic literature (theoretical and empirical) on cameras in courtrooms and situate our intervention. In Section 4, we describe the process of creating our dataset, including the speaker diarization procedure and important limitations. In Section 5, we outline our empirical strategy and report our findings. In Section 6, we discuss the implications of our results and consider how our findings might inspire future researchers and affect courtroom policy.
2. A Brief History of Cameras in Court
2.1. Generally
The U.S. has a longstanding aversion to cameras in federal court. Though all state courts now allow cameras to some extent (Herman, 2002), federal courts have consistently resisted recording technology. The Judicial Conference of the United States first codified this opposition in 1946, when it implemented a rule prohibiting “the taking of photographs . . . during judicial proceedings or the broadcasting of judicial proceedings from the courtroom” (Fed. R. Crim. Pro. 53, 1946). In 1972, the Judicial Conference “doubled down” on this prohibition by adding language forbidding “broadcasting, televising, recording, or . . . photographs” to the Code of Conduct for United States Judges (“History of cameras,” n.d.; Singer, 2015, p. 81).
In the 1990s, the Judicial Conference began to reassess its recording policies. It adopted a policy allowing judges in federal trial courts to authorize recording in a few limited settings. (“History of cameras,” n.d.). 6 It also launched a pilot program that introduced electronic recordings in six federal district courts and two federal appellate courts (Johnson & Krafka, 1994). When that three-year pilot concluded, the Conference declined to allow contemporaneous recording activity in either federal trial or federal appellate courts, noting that the pilot had found an intimidating effect of cameras on some witnesses and jurors (“History of cameras,” n.d.). 7
In 1996, the Judicial Conference authorized federal appellate courts to make their own decisions about whether to permit photographs, video, and other recordings during oral argument (Judicial Conference of the United States, 1996). At the same time, though, the Conference “strongly urge[d] each circuit judicial council to adopt . . . an order reflecting the Conference’s [previous] decision not to permit [cameras] in U.S. district courts” (Judicial Conference of the United States, 1996, p. 17). In response to this policy change, all federal circuit courts began to livestream and record audio of oral arguments for public distribution (“History of cameras,” n.d.). The Second and Ninth Circuits, and later the Third and Seventh, also changed their rules to permit certain camera coverage (mostly by media) during oral argument (“History of cameras,” n.d.; U.S. Court of Appeals for the Seventh Circuit, 2015). 8
Since 1996, the Judicial Conference has continued to reassess its policies on cameras in federal court (“History of cameras,” n.d.). But despite ongoing study and experimentation, its official policies remain largely unchanged. As of September 2023, the Guide to Judicial Policy prohibits recording in federal district courts in all but a few circumstances (Administrative Office of the U.S. Courts, 2023, § 420(a)). 9 Courts of appeals, however, may “determine whether appellate proceedings before [them] will be broadcast” (Administrative Office of the U.S. Courts, 2023, § 420(a)).
Current Recording Policies in U.S. Federal Appellate Courts
2.2. In the Ninth Circuit
The Ninth Circuit, which is the subject of this study, introduced cameras incrementally. In 1991, the Circuit began granting media permission to video record “a few cases a year.” (U.S. Court of Appeals for the Ninth Circuit, 2017, p. 37). 14 In 2010, it launched a more targeted initiative to “expand [] the use of cameras in the courtroom” (U.S. Court of Appeals for the Ninth Circuit, 2010, p. 20). As part of this initiative, the Circuit recorded “important” en banc proceedings, which it posted on the Court’s website (U.S. Court of Appeals for the Ninth Circuit, 2010, p. 2). The court also began livestreaming video of en banc proceedings and other notable cases “to other federal courthouses in the Ninth Circuit and elsewhere,” (U.S. Court of Appeals for the Ninth Circuit, 2010, p. 2). 15 The Circuit offered these livestreams in part to “improve public understanding of judicial processes and enhance confidence in the rule of law” (U.S. Court of Appeals for the Ninth Circuit, 2010, p. 20). It also hoped that expanding access to live proceedings would provide “an enriching educational experience” for law students (U.S. Court of Appeals for the Ninth Circuit, 2010, p. 2). 16
Over the next few years, the Ninth Circuit tried to “open the court’s doors . . . wider” by extending video streaming to more courtrooms and more proceedings (Public Information Office, 2013). In December 2013, it became the first federal appellate court to provide live video streaming of its quarterly en banc proceedings (U.S. Court of Appeals for the Ninth Circuit, 2013, p. 23). In 2014, the Circuit began a quasi-random transition towards live streaming all oral arguments: It gradually began installing camera equipment in its courtrooms and began video recording cases as those courtrooms came “online” (Email from Ninth Circuit administrator, Aug. 27, 2024).
17
By the end of 2014, it had equipped ten courtrooms (at least one in each of its four courthouses) for video streaming (U.S. Court of Appeals for the Ninth Circuit, 2014, p. 20). And by 2015, every oral argument in the Ninth Circuit—en banc and three-panel—featured both video and audio recordings (U.S. Court of Appeals for the Ninth Circuit, 2020, p. 3; U.S. Court of Appeals for the Ninth Circuit, 2020, p. 1) (see Figure 1 for an example of a video recording).
18
Still of Ninth Circuit oral argument livestream
As the Ninth Circuit explained in its 2014 annual report, “video streaming is . . . challenging technically, involving not just cameras in the courtroom but use of complex video production systems behind the scenes” (U.S. Court of Appeals for the Ninth Circuit, 2014, p. 20). To produce the livestream, a “small group of dedicated court staff” installs two cameras in each courtroom—“one focused on the bench, the other on the attorney lectern” (U.S. Court of Appeals for the Ninth Circuit, 2014, p. 20). These cameras generate two feeds which are “combined with a third image that consists of text identifying the case, including the date and location of the proceeding and composition of the panel” (U.S. Court of Appeals for the Ninth Circuit, 2014, p. 20). The combined image is fed to two video streaming units—a primary unit and a backup (U.S. Court of Appeals for the Ninth Circuit, 2014, p. 20). That unit then sends data to YouTube.com, where the feeds are distributed publicly 19 (U.S. Court of Appeals for the Ninth Circuit, 2014, p. 20). Between 2010 and August 2024, the Ninth Circuit posted more than 22,000 videos, which were viewed 7,260,541 times. (Email from Ninth Circuit administrator, Aug. 27, 2024). In 2023 alone, it provided 752.3 hr of live streams and published 2,146 videos. This content was viewed 789,986 times—a total of 124,073 hr of watch time (Email from Ninth Circuit administrator, Aug. 27, 2024).
3. Cameras and Their Effects
Because cameras in court have long been a topic of interest and debate, there is a robust academic literature that explores the effects, costs, and benefits of recording and/or broadcasting judicial proceedings. In this section, we briefly summarize the existing research on cameras in court, giving special attention to existing empirical studies of cameras and their effects.
3.1. Theoretical Arguments
Much of the existing literature on cameras in courtrooms is theoretical rather than empirical. This literature explores a variety of arguments for and against cameras, but several central themes and hypotheses emerge.
3.1.1. Cameras (Do/Do Not) Alter Courtroom Behavior
One hypothesis that emerges from the existing literature is what we call the “performative judging hypothesis”: Judges and lawyers might “change [their behavior] if they know they are being filmed” (Marder, 2021, p. 20). Particularly during trials, cameras might encourage courtroom participants to “play to the television audience” (Abrams & Kaminer, 1995, p. 37). They might make jurors “self-conscious and uneasy” and “impair [] . . . the quality of testimony” (Estes v. Texas, 1965, pp. 545, 547). Cameras also make judges more recognizable and less “obscure” (Marder, 2012, p. 1521). Among other things, public recognition could introduce new threats to judges’ safety, which might cause them to behave differently both in public and in the courtroom. In short, cameras might “destroy [the courtroom] dynamic” by altering courtroom behavior (McElroy, 2012, p. 1838, n. 7).
For the most part, the scholars who express these concerns have trial courts in mind. But if trial participants could be tempted to “ham it up for the camera,” appellate judges and attorneys might do the same (Kozinski & Johnson, 2010, pp. 1110–1111). Indeed, there are many Supreme Court Justices who do not want cameras in the high court for precisely this reason: Justice Alito, for example, has expressed worry that “lawyers would find it irresistible to try and put in a little sound bite,” and Justice Kagan has said that cameras might cause the Justices to “filter [them]selves in ways that would be unfortunate” (C-SPAN, 2019, p. 5:38–5:54; 9:38–9:43). Though there might be more opportunities to grandstand during a televised trial, some believe that the impulse to perform, self-censor, or otherwise alter behavior would exist in appellate proceedings, as well. In either context, Justice Kagan’s concern rings true: “When the observer comes in, the observed thing changes” (C-SPAN, 2019, p. 9:05–9:11).
3.1.2. Cameras (Do/Do Not) Promote Transparency
Another common hypothesis—which we label the “transparency hypothesis”—predicts that cameras might promote transparency and enhance democracy by providing accessible information about legal proceedings. Proponents of courtroom cameras often suggest that “broadcasting [courts’] work would . . . result in more informed public perception” (McElroy, 2012, pp. 1844–45). They also claim that “televised coverage of trials exposes greater numbers of our citizens to our justice system” (Tuma, 2001, p. 420) and might therefore “increase interest on the part of average Americans about what goes on” in court (McElroy, 2012, pp. 1844–45). Proponents similarly argue that “our system of justice will be aided by the availability of more accurate information” (D’Alemberte, 1980, p. 39) and that cameras can facilitate “the public’s interest in accessing its government, . . . understanding what the judicial branch does, and . . . forming its own opinions . . . from what it sees” (McElroy, 2012, p. 1899). Further, court proceedings are public events, and the “government has an obligation to make its public events as public as possible” (McElroy, 2012, p. 1844, n. 39).
Critics, by contrast, argue that cameras do not provide any additional transparency or educational value. Cameras are already allowed in many state court proceedings, and federal judges—especially U.S. Supreme Court Justices—write books and participate in interviews to educate the public about the court system (Marder, 2012). Because of this, some think it is “unclear how cameras in federal courtrooms will add to viewers’ education” (Marder, 2012, p. 1499). Others worry that recordings and broadcasts will inevitably provide an incomplete judicial education because the types of proceedings that are recordable—trials and oral arguments—are in fact “atypical events” that risk being “abstracted from the rest of the system” (Harris, 1993, p. 788). Finally, some argue that broadcasting and recording will give “a distorted image” of the court system (Harris, 1993, p. 788). The rare viewer who has the time and interest to watch an entire proceeding will inevitably gravitate toward the most sensational and high-profile cases (Abrams & Kaminer, 1995); meanwhile, everyone else will rely on news coverage, which “contain [s] a larger than normal dose of weighty, topical issues, involve celebrities, lascivious detail, or grotesque or macabre trivia” (Harris, 1993, p. 788). In short, even if broadcasting or televising judicial proceedings might provide some educational value, many worry that education would “certainly fall short compared to the kind of education acquired by a member of the public who is present in the courtroom and who observes the trial” (Marder, 2012, p. 1497).
3.1.3. Cameras (Do/Do Not) Affect the Decorum and Solemnity of Judicial Proceedings
A third common hypothesis—the “decorum hypothesis”—predicts that cameras might alter the dignity and solemnity of judicial proceedings, especially in appellate courts. Many who oppose cameras in court do so because they worry that cameras will “undermine the integrity and proper role of oral argument” (Marder, 2021, p. 20). More specifically, some worry that if judicial proceedings are recorded or broadcast, viewers are “likely to look for entertainment,” rather than substance, “in a televised oral argument” (Marder, 2021, p. 20). Indeed, in 1937, the American Bar Association banned cameras and broadcasting in criminal trials precisely because it worried that such media would “detract from the essential dignity of the proceedings, degrade the court [,] and create misconceptions with respect thereto in the mind of the public” (American Bar Association, 1952). In theory, the same degradation might occur in appellate courts as well.
Proponents of courtroom cameras reject these arguments. They note that attorney grandstanding is common even in non-recorded or untelevised cases (Abrams & Kaminer, 1995). They likewise argue that cameras might actually preserve decorum by allowing the press to obtain desired content without mobbing courtroom steps or noisily crowding in courtroom hallways (D’Alemberte, 1980). Finally, proponents of courtroom cameras note that cameras have been permitted in other reverent, dignified proceedings—religious ceremonies, weddings, and so on—“without detracting from the solemnity” (D’Alemberte, 1980, p. 9). Thus, they posit, there is little reason to worry about cameras’ effects on courtroom decorum.
3.2. Empirical Studies
In addition to the theoretical discussions described above, some researchers have used empirical research methods to gauge the effects of cameras and live feeds on U.S. courtroom proceedings. However, much of this empirical research has been methodologically limited or focused on cameras in district courts. Further, most existing studies have explored how video technology facilitates communication in the courtroom, not how it might affect the behavior of judges and participants.
One of the earliest and most thorough empirical studies of cameras in court began in 1990, when the Judicial Conference of the United States approved the pilot program we describe in Section 2, above. In 1993, the Federal Judicial Center (“FJC”) evaluated the pilot program by surveying and interviewing judges (both participating and nonparticipating), jurors, court personnel, and media representatives (Johnson & Krafka, 1994). The surveyed judges and attorneys reported that cameras had few or no effects on their own behavior or on courtroom decorum. 20 They also reported that cameras offered moderate or no educational benefits for the public. This study provided some initial support for the thesis that cameras do not affect courtroom behavior or create drama. It also indicated that cameras might not have the educational benefits that some credit them with.
In 2011, the Judicial Conference authorized a second pilot program that allowed fourteen federal district courts to record and post videos of civil proceedings (Singer, 2015, p. 79). At the end of the program, the FJC again surveyed participating judges, attorneys, and court staff about their experiences (Johnson et al., 2016). And again, the survey results revealed that the majority of participating judges perceived “little or no” effect on witnesses, judges, or jurors (Johnson et al., 2016, p. viii). Judges were also “evenly split on the extent to which video recording causes attorneys to be more theatrical” (Johnson et al., 2016, p. viii). Participants did, however, feel that “video recording, to a moderate or a great extent, educates the public about courtroom proceedings, educates the public about the legal issues in court cases, and increases public access to the federal courts” (Johnson et al., 2016, p. ix). The 2011 pilot thus offered new support for the transparency hypothesis—the idea that cameras might enhance access to and knowledge of courtroom proceedings.
Though the studies of the 1990 and 2011 pilots were both detailed and thorough, they also had obvious methodological weaknesses. As the FJC itself noted, participation in the 2011 pilot project was entirely voluntary, which meant the judges and attorneys who responded were not representative of all U.S. district courts (Johnson et al., 2016; Marder, 2012). This voluntary structure may have also skewed the results: “[B]ecause recordings were made only with consent of the participants,” the FJC noted of the 2011 pilot, “we might expect that the views of participating judges and attorneys would, on average, be more favorable than the views of judges and attorneys who would not agree to video recording” (Johnson et al., 2016, p. 3). Finally, and maybe most significantly, neither the 1990 or 2011 pilots could measure “the actual effects of video recording,” because neither study included any comparison between recorded and non-recorded proceedings (Johnson et al., 2016, p. 3). Instead, the studies revealed only “perceived effects of video recording, as reported by judges in the pilot courts and attorneys who experienced recording under the pilot program” (Johnson et al., 2016, p. 3; Marder, 2012).
More recent academic studies have responded to these shortcomings by adopting different research methods. Some have opted to do qualitative work—comparing, for example, recording rules and norms in the U.S. and other countries (Youm, 2012). Others have used experimental methods to isolate and measure, in either lab or survey settings, the ways cameras might affect perceptions of a court’s legitimacy (Black et al., 2023a, 2023b) or how it might influence court participants, such as witnesses and juries (Borgida et al., 1990).
Others have focused more squarely on the role that cameras play in facilitating communication between participants in legal proceedings, as opposed to the effect that video broadcasting or recording has on courtroom behavior. A series of recent articles have investigated whether immigration removal proceedings that are conducted remotely (i.e., with the respondent “appearing” in court via a closed video feed) have disparate outcomes relative to in-person hearings, finding that video respondents are substantially less likely to engage in the proceedings and are ultimately more likely to be removed (Eagly, 2015; Thorley & Mitts, 2019; Walsh & Walsh, 2008). Similarly, the Administrative Conference of the United States conducted empirical case studies to explore the use of videoconferencing to facilitate adjudicatory proceedings (Olorunnipa, 2011). A more recent experimental study found that remote participation in criminal proceedings had no impact on the likelihood of guilty verdicts, but the study used “mock juries” instead of looking at the outcomes of real cases (Rossner & Tait, 2021).
Though these and other studies provide useful insights about the effects of cameras in U.S. courtrooms, they also leave many important questions unaddressed. To begin, the existing literature reveals little about the validity of the performative judging hypothesis—that is, it does not explain whether cameras actually affect the behavior of judges or attorneys. Though some studies suggest that cameras have little or no effect on court participants, those studies are voluntary (participants opted in) and rely on self-reported data from judges and attorneys, which limits their explanatory power (Johnson et al., 2016). And the few studies that use experimental methods have either studied non-U.S. contexts (Liu & Tang, 2025) or considered whether cameras affect external perceptions of legitimacy, not whether cameras cause courtroom participants to perform differently than they otherwise would (Black et al., 2023a, 2023b).
Notably, a few studies have tested whether cameras affect behavior in the legislative context, and all have found that the introduction of C-SPAN led to longer legislative sessions (Mixon et al., 2007), an increase in performative legislative speeches (Caspi & Stiglitz, 2023) and Senate filibusters (Crain & Goff, 1988; Mixon et al., 2003), longer legislative sessions (Mixon et al., 2007), more “performative herding” (i.e., behavior designed to “demonstrate aligned type to partisan constituencies”) (Caspi & Stiglitz, 2023, p. 3), and an increase in emotional appeals and emotive rhetoric (Gennaro & Ash, 2023). These findings provide reason to believe that judges, too, might change their behavior in response to cameras. But we have not identified any study that tests or measures these possible effects on judicial behavior. Thus, the performative judging hypothesis remains largely untested and unproven.
Second, and more significantly, the existing literature does not point to any clear conclusion. Though there has been abundant research on cameras and courtrooms, there is no academic consensus as to whether cameras are good or bad, helpful or harmful. Indeed, some studies conclude that cameras have positive effects, others identify negative effects, and still others think cameras do not influence courtroom proceedings at all. Thus, despite the active and ongoing efforts of many skilled scholars, the “conundrum of cameras in the courtroom” remains unsolved (Marder, 2012).
4. The Dataset
Our dataset comes primarily from AI-driven identification of judicial speech in a subset of oral arguments recorded in the Ninth Circuit between 2012 and 2018. 21 We supplement the data derived from the Ninth Circuit recordings with hand-coded demographic information of the judges and attorneys who participated in the oral arguments for each case. The final dataset includes 156 cases (86 audio and 70 video) featuring 17 judges and over 17,000 distinct utterances. In this section, we outline these diarization and hand-coding processes, describe the collection of attorney- and judge-level demographic information, present basic descriptive statistics of the resulting dataset, and highlight potential dataset limitations.
4.1. Speaker Diarization
Although audio recordings (and sometimes transcripts) of U.S. circuit court oral arguments are made available as public record, these recordings are almost never diarized—that is, they do not include annotations or coding that identify the speaker or speakers at a given moment in the proceeding. 22 Even with the advent and partial use of video recordings in circuit courts, a viewer is only able to identify the speaker of a given utterance or set of arguments by watching the video and referencing the court docket, so it is difficult and labor-intensive to connect words to specific speakers. 23 And while recent years have seen an explosion in high-quality transcription methods in computer science and artificial intelligence, diarization remains an extremely challenging problem. 24
To identify speakers at scale (i.e., for more than a handful of cases or small chunks of speech), we relied on a computer-assisted diarization process. In a series of previous works, we developed a diarization workflow called Reference-Dependent Speaker Verification (RDSV) and tested it on U.S. Supreme Court oral arguments with relatively strong success (Tumminia et al., 2021). While we avoid detailing the more technical aspects of this process here, a general understanding of this RDSV pipeline—including the creation of the training sets, its limitations, and how it performs on the Ninth Circuit Court recordings we rely on in this paper—is important for understanding the implications of our empirical analysis of the effect of cameras on judicial behavior in Section 5, below.
Our RDSV process is a variation of the more commonly used Text-Independent-Speaker- Verification, and both methods follow the same basic process. First, a “training” or “reference set” of speaker representations is created using known and verified samples of judicial speech for a given judge. These samples are hand-coded to ensure accurate identification on the front end. They are then run through a pre-trained, open-source AI model created by Resemble.AI called Resemblyzer, which identifies and then encodes the unique attributes of a given speaker to create a reference audio library. 25 Second, a “development” set is used to train the AI model to derive, or diarize, individual “utterances” (i.e., chunks of speech that are attributable to a given individual) that are coded to one of the judges identified in the training set or non-judge speakers (e.g., attorneys and court personnel) in “unseen” or “new” cases. Finally, the accuracy of the trained RDSV is evaluated by diarizing a “test” set that was also hand-coded, which allows for direct comparison of the frequency, length, and speaker identified for each utterance.
In this paper, we trained the model and produced the data using audio samples from online, publicly-available repositories of oral arguments from the U.S. Court of Appeals for the Ninth Circuit. 26 To capture the period in which this court began experimenting with video recordings, we downloaded the audio recordings for all Ninth Circuit cases from late 2012 to late 2019. 27 However, our ultimate dataset does not include every oral argument from those periods. Because the diarization process described above is resource intensive—both in terms of human labor (the initial stage of hand coding sample speech) and computational resources (the training and implementation of the AI model)—we only included cases paneled by the 17 judges who provided the most optimal training data. 28 Consequently, our final dataset includes only 156 cases from the Ninth Circuit. We discuss the potential limitations of this case-selection strategy in Section 6.4, below.
For each of the cases included in our dataset, the RDSV divided the entire transcript into individual utterances that were attributed to one of four “speakers”: Judge 1 (the first of the three judges on the panel who spoke), Judge 2, Judge 3, and non-judge individuals (i.e., attorneys). 29 All non-judge speakers were bundled together because the RDSV did not have a training set of audio files for the various attorneys who participated in the oral arguments. Each of those utterances also have an associated speaking duration (in milliseconds), ordered mapping within the hearing, and transcribed text. This allows us to identify the key outcomes of interest in this study, including the overall length of hearings, the total speaking duration for judges relative to attorneys, and the average length of judge utterances.
We are also able to calculate the total number of times attorneys speak over the course of a case and the average length of those individual utterances, which we use as proxies for judicial interruption. 30 While we recognize that these datapoints (frequency and duration of attorney speech) are indeed proxies, we believe that our use of them measures as indicators for interruption relies on reasonable assumptions. In oral arguments, an attorney’s time at the podium is only interrupted by judges. 31 Because of this, judges are the primary (if not sole) drivers of both how long in total lawyers speak (via the grant of rebuttals) and how segmented the lawyer’s arguments are. Absent questions and comments from judges, non-judge speakers would have only a handful of speaking turns, and those turns would be longer. While judge interruptions are not inherently aggressive and may not necessarily cut a lawyer off mid-sentence, they are at least indicative of a judge who is more involved and more willing to speak up.
4.2. Attorney and Judge Demographic Data
Although our RDSV provided the core data necessary to measure the effect of cameras on judicial behavior, it did not provide any information about the characteristics of the attorneys or judges that participated in the oral arguments. Consequently, we hand-coded each of the cases in our dataset to identify basic demographic data for attorneys and judges. We use this data to measure discrepancies between the treatment and control groups, model around and control for heterogeneity, and measure potential interactive effects between judicial demographics and the introduction of cameras in the courtroom.
Relative to the diarization process, this hand-coding was simple. For the attorney information in each case, we first identified the transition points between the appellant’s arguments, the appellee’s arguments, and the rebuttals (where applicable) by manually searching the transcripts for linguistic clues (“your time has expired”, “may it please the court”, “on behalf of”, etc.) at the timestamps when those transitions were most likely to occur based on the diarized text. All non-judge utterances that occurred from the start of a hearing to the end of the appellant’s arguments or from the end of the appellee’s argument to the end of the first rebuttal were coded as appellant utterances, and all other utterances were coded as appellee utterances.
We then identified the names of the attorneys that were participating for each side by first referencing the lead counsel information in the relevant case metadata collected by either the Westlaw or Lexis databases. When this information was not available, we returned to the argument transcripts produced by our RDSV where, in all but eight cases (both attorneys in three cases and only the appellee’s attorney in two cases), the attorneys were introduced or introduced themselves by name. 32 Once we had a working name, we used internet searches (Google) and professional websites (Linkedin or websites for private firms or government agencies) to make visual inferences regarding the attorney’s gender and race/ethnicity in cases where the pronouns and descriptives used in professional bios were not clear. 33 In some cases—generally when the attorney worked for a small firm or the government—photos and bios were not available. In those cases, we inferred gender based on the attorney’s name, if possible, but we made no inferences regarding race/ethnicity.
Ultimately, this hand-coding process provided a near comprehensive supplementary dataset of attorney demographic information. We identified all but 8 attorneys by name (both attorneys in three cases and only the appellee’s attorney in two cases—just under 1.5% of all the attorneys in our dataset), inferred gender for all but 11 (∼2%), and inferred race/ethnicity for all but 106 (∼19%).
The demographic data for the judges in our sample was much easier to collect. The Federal Judicial Center’s Biographical Directory of Article III Federal Judges provided nearly all the information required in our study, including gender, race/ethnicity, party of appointing president, and the ABA ratings (“qualified” or “well-qualified”) of the judges when they were nominated (Federal Judicial Center, 2023).
4.3. Dataset Descriptives
Diarization Dataset Descriptives
Notes. Three additional judges are part of our larger dataset (i.e., they sat on panels in at least one case) but did not speak so are excluded from our diarization data.
Utterances per case
Case Distributions Across Time
4.4. Potential Dataset Limitations
While we discuss the various limitations in our causal framework in Section 5.1 below, it is useful at this juncture to emphasize the potential limitations inherent in the general enterprise of diarizing speech in appellate oral arguments. First, our dataset is necessarily limited in time and in scope. For reasons we mentioned previously and will revisit later, the adoption of video recording in the Ninth Circuit presents a unique time period (2012–2018) that lends itself well to causal inference regarding the effects of cameras. However, the cases that came through the court during this period may not reflect the type of cases that are currently in that circuit or the broader set of cases that are in other federal circuits, let alone the broader U.S. federal and state courts. Consequently, our conclusions may not be generalizable to those other venues.
Beyond general external validity concerns, our dataset might also be subject to selection bias induced by the resource limitations of the diarization process. Recall that our dataset excludes cases that involved any judges that were not selected to be a part of our hand-coded RDSV training sets. While the selection process, which identified judges that were featured in the most cases, is ostensibly unbiased on its face (especially under an assumption of random judicial assignment to panels), 35 there may be unobserved correlations between the types of judges and types of cases that were included in the dataset and our key outcomes of interest (amount of judge speech, number of utterances, interruption tendencies). Although non-random panel assignment is not critical to our causal model (a slight variation of an event study), the selection of cases used in our diarization might result in causal estimates that are biased. However, because this data is unobservable, the existence or nature of that bias is unknowable.
We also acknowledge the potential for error in the audio processing pipeline itself. Our pipeline consists of three audio processing steps: segmentation, diarization, and transcription. 36 Segmentation is the process of splitting an audio file into chunks where each chunk is a separate utterance; diarization is the process of assigning a specific speaker to an utterance; and transcription is converting that utterance from an audio file to text. Each of these steps is subject to error. Our RDSV pipeline has an empirical diarization error rate (DER) of 13.8%, indicating that we correctly identify the speaker for 86.2% of all utterances. While it is harder to quantify the segmentation error rate, our research assistants manually reviewed many of our machine-coded cases and indicated segmentation errors in fewer than 1% of the utterances. As long as this error is uncorrelated with whether a case was video recorded, it will likely introduce random noise and can only attenuate our empirical results.
Finally, we reemphasize the limitations in our hand-coded data for the non-judge utterances. First, recall that the diarization process can only actively identify the identities of judges because they are the only individuals in the oral argument audio files for which we have sufficient examples. Consequently, all non-judge speech is broadly defined and attributed to attorney speech, even though a small portion of it is actually the speech of court personnel, such as bailiffs. Second, our hand-coded process relied heavily on context-based inferences to code attorney gender (names, photos, and pronouns in online bios) and race/ethnicity (photos and, much less frequently, information in online bios), which may not have been fully accurate. 37
5. Empirical Analysis
5.1. Treatment and Causal Framework
Our causal model leverages the quasi-experimental process through which the Ninth Circuit Court began using cameras and video feeds. In this natural, quasi-experiment, the treatment or stimulus is the introduction of cameras that judges knew were recording video that would be made available to the public. The performative judging hypothesis relies on the assumption that court participants know they are being recorded and change their behavior accordingly. Fortunately, we have ample evidence that Ninth Circuit judges were aware that they were being recorded and understood that their proceedings would be available to audiences beyond the Courtroom. For many years, the Ninth Circuit’s Annual Reports boasted of the circuit’s move to video recording, which suggests that circuit employees viewed the change as cause for pride (U.S. Court of Appeals for the Ninth Circuit, 2010, pp. 2, 20; U.S. Court of Appeals for the Ninth Circuit, 2013, p. 23; U.S. Court of Appeals for the Ninth Circuit, 2014, p. 20). The Circuit also introduced cameras in part because it wanted arguments to be viewed by remote audiences. In fact, during the early years of the camera rollout, the Ninth Circuit’s annual report stated that it hoped that video recordings and streaming would “improve public understanding of judicial processes,” “enhance confidence in the rule of law” (U.S. Court of Appeals for the Ninth Circuit, 2010, p. 20), and provide “an enriching educational experience” for law students (U.S. Court of Appeals for the Ninth Circuit, 2010, p. 2).
Whether we are making inferences from the visual representations of the data (Section 5.2.1), controlling for important attorney- and judge-level characteristics with regressions (Section 5.2.2), examining interaction effects (Section 5.2.3), or analyzing subsets of the data (Section 5.2.4), the credibility of our causal claims ultimately rests on to whether the cases that have video recordings have the same balance of “pre-treatment” (i.e., attributes that could not have been affected by the treatment) characteristics as the audio cases. The ideal scenario would be that the court randomly assigned some cases to video and others to audio as part of a formal experiment, but we know that was not the case for our data. 38 The next-best scenario is that the allocation of cases into the treatment and control groups was “as-if random,” in that the selection process was unrelated to and uncorrelated with those pre-treatment characteristics.
Fortuitously, the timing and manner in which the Ninth Circuit introduced cameras and live feeds for standard three-judge-panel oral arguments is ostensibly one such allocation process. As discussed in Section 2 above, the court began livestreaming en banc proceedings and select other notable cases in 2010 (U.S. Court of Appeals for the Ninth Circuit, 2010, p. 2). According to a Ninth Circuit administrator who spoke to us about the court’s use of video, judges were allowed to opt out of video recording during this time (Email from Ninth Circuit administrator, Aug. 27, 2024).
During 2014, the court’s audiovisual team re-tooled all courtrooms to accommodate live video recordings of oral arguments (U.S. Court of Appeals for the Ninth Circuit, 2014, p. 20). The court “started streaming and posting from [courtrooms] . . . as [they] were completed and brought online.” 39 (Email from Ninth Circuit administrator, Aug. 27, 2024). According to our Ninth Circuit contact, this process was “sort of random before January 2015, as we were constantly tinkering with our video and sound equipment . . . [and] there were not any case type restrictions.” Note, however, that judges were able to opt out (as opposed to opting in) of this 2014 rollout. (Email from Ninth Circuit administrator, Aug. 27, 2024). As reflected in our dataset (see Table 3), fifteen of the cases that are included in our cameras treatment group were part of that late-2014 rollout. Twenty-eight cases during that period only included audio. By 2015, all oral arguments featured video cameras, providing a sharp transition for all remaining courtrooms.
Consequently, our causal framework is a variation of an event study model, with the addition of the short, quasi-random roll-out period that featured both audio and video cases. This gives us additional statistical strength. Instead of mapping the treatment onto specific temporal pre- and post-treatment periods, as is the case in a pure event study, the pre- and post-treatment periods have a small overlap. This natural, quasi-experiment assumes that the timing of this change was not correlated with any of the pre-treatment covariates we alluded to earlier. While we have no indication that the introduction of video coincided with any contemporaneous change in courtroom policy or the makeup of the types of cases that were heard by the Ninth Circuit, the critical causal assumption with event studies is that the shock did not coincide with non-random time trends such that, over the course of the period studied, the circumstances, actors, or subjects of the observations did not experience any variation. In other words, we assume that the only important difference between the cases heard before late-2014 and those heard afterwards was the inclusion of video cameras.
Balance Across Treatment Groups
Notes. Values in this table represent proportions of cases that feature the respective characteristic.
ap-values were derived using standard two-sample t-tests.
We first note that attorney characteristics do not differ in any substantial way across treatment groups. Though there is a marginally significant imbalance in the proportion of appellee lawyers that are women. Recent studies suggest that attorneys with certain demographic characteristics are more likely to argue certain types of cases (ABA Commission on Women in the Profession, 2021). The balance in attorney characteristics thus reassures us that there also is likely no imbalance in underlying case type. This is consistent with the assumption underlying our event study framework.
However, once we test the composition of the panels of judges that sat on the audio cases relative to the video cases, we see a few significant imbalances beyond what we would expect to see due to random variation. Video cases were much more likely to have panels with a higher composition of non-white judges (an increase of 20% points), a higher composition of Republican judges (20 pp), and a lower composition of judges that are ABA “Qualified” (10.5 pp) as opposed to “Well Qualified.” Interestingly, although our analyses in Parts V.B.iii and V.B.iv (below) indicate a strong correlation between the gender composition of judicial panels and many of our outcomes of interest, judge gender is quite well balanced across our audio-only and camera groups in our full dataset. Because some of our models use time-fixed effects, we also conducted the same series of balance tests on the 2014-only set of cases. We find similar imbalances in the judge panel characteristics (see Appendix: Table A3). This time, however, we see statistically significant imbalances (albeit not particularly large, substantively) in judge panel gender composition. Given the small-n nature of this 2014-only sample—43 cases total and only 15 cases in the video group—this added imbalance is not particularly surprising.
The results of these balance tests raise a number of implications that are important to our causal framework and will affect the approach we take in our analysis. At the very least, they show that we will need to control for these imbalances by including the relevant covariates in our regression estimation models. These imbalances may, however, indicate systematic differences between the control and treatment groups, such that simply controlling for the observable variables that we have in our dataset may not be enough to meet the assumptions required under our causal framework. Without a full set of possibly relevant covariates, we cannot determine whether that problematic conclusion is true, although we discuss that possibility further in our analyses below, with particular focus on the role that judge gender may play in our findings. We also recognize this limitation in our analysis and framing of our results.
5.2. Results
In this section, we present our results. We start in Section 5.2.1 with a presentation of the visual trends reflected by a raw comparison of pre- and post-camera periods. These trends are all consistent with the hypothesis that judicial speech and judicial interruptions increase with the introduction of cameras. In Section 5.2.2, we present the results from our regression adjustments, which are more mixed. Our regressions that account for time-fixed effects and attorney characteristics are fully consistent with the performative judging hypothesis. But when we introduce controls for judge panel composition, our results are generally indistinguishable from the null. This is possibly because of the imbalances in treatment groups we identified above. However, in Section 5.2.3, we introduce interaction terms to identify variation in treatment effects among different types of judges, revealing some significant interactive effects between the treatment and the race/ethnicity of the attorneys and judges. Finally, in Section 5.2.4, we present regression results from a subset of data that only includes judges who sat in both audio and video panels during the transition period and find significant effects for some of the outcomes that were null for the full dataset.
5.2.1. Raw Results and Visual Trends
The figures below depict different measures of judicial behavior across the time period we studied. In each figure, the colored lines are linear regression lines that reflect the over-time trends for the indicated outcomes. The solid-colored line represents cases that featured audio recording only, and the dotted colored line represents video-recorded cases. Vertical dotted lines indicate when the Ninth Circuit introduced video recordings, with the first line marking the initial, quasi-random rollout and the second line marking the point where all oral arguments became video recorded and streamed. Where a figure includes both judge and non-judge speakers, the regression lines are colored blue and red, respectively. The gray band around each of the lines represent standard 95% confidence intervals, which provides a rough approximation of reasonable error rates.
In Figure 3, we see preliminary evidence that judges spent more time speaking in video cases than in cases with only audio recordings. However, much of the effect appears to have been driven by time trends, as opposed to an immediate “performance” effect. The “jump” from audio to video near the end of 2014 is only moderate, and the time-trends appear to be driven largely by an increasing number of cases in which judges had noticeably long average utterances (nearly 2 min in some instances). Average judge utterance length
We begin to see clearer indications of a potential performance effect when looking at the total amount of time judges and lawyers spent speaking over the course of an entire case. The left panel in Figure 4 presents outcomes where each of the three judges’ total speech is measured individually (i.e., the values are averages of the speaking judges who sat on the panel). The right figure combines all judge speech into one panel-level measure. Consistent with the performative judging hypothesis, judges in the Ninth Circuit immediately begin speaking more with the advent of video hearings, as shown by the vertical gap, or discontinuity, between the solid blue line (indicating judge speech in audio recordings) and the dashed blue line (indicating judge speech in video recordings). We also see a converse effect on attorney speech, which is consistent with our ex ante expectations: Time in oral arguments is a limited resource, so the more the judges speak, the less the attorneys are able to. Total length of speech (minutes)
Our causal framework assumes that trends would have proceeded in the same direction after 2014 but for the introduction of video recording and streaming. However, it is possible that post-2014 cases were simply longer. While a change in case length could explain why we see an increase in overall judge speech, it would not explain the sharp discontinuity. Figure 5 explores the possibility that something other than the introduction of videos drove the increase in judicial speech. Specifically, Figure 5 summarizes Figure 4 and measures the proportion of the court proceeding taken up by judge speech as compared to non-judge speech. It shows two effects. First, judge speech switched from less than 40% to more than 50% of proceeding time immediately after the implementation of video recording, a 13% increase. Second, there is a slight change in the time trend. Prior to 2015, the proportion of proceeding time taken up with judge speech had been steadily decreasing. The switch to video recording appears to have increased the proportion of judge speech but also slowed the rate of decrease: The solid line is steeper than the dotted line. Judge speech/total speech
Figure 6 provides additional context for what we observed above. In this figure, the key outcomes of interest are the average length of speech utterances by non-judges (left) and the total amount of speech by non-judges (right). As we explained previously, we argue these metrics are reliable proxies for judicial “interruption.” In particular, we assume that, in the complete absence of judicial interruptions, a given non-judge speaker would only have one, long utterance (or a few more if they are allowed a rebuttal). Consequently, if the average length of utterances decreases but the total amount of non-judge speech does not decrease (or decreases by less), that provides suggestive evidence that judges are interrupting more rather than simply speaking more and longer. Non-judge utterance length
The important thing to observe in this figure is that with the introduction of video, the average length of non-judge utterances dropped by almost 50%, from about 30 s per utterance to about 20 s. On the other hand, the total length of non-judge speech decreased only slightly, from about 21 min (1,250 seconds) to about 17 min (1,000 s), a decrease of about 25%. In sum, Figure 6 demonstrates that when cameras were present, lawyers spoke less in total and their speech came in shorter segments, patterns that indicate more frequent interruption by judges.
5.2.2. Regression Adjustment
Regression Models on Judge Speech
Notes. Average judge utterance length is measured at the utterance level in seconds. Total length of judge speech is measured at the case level in minutes. Total judge speech (proportion) is measured at the case level as the proportion of judge speech relative to the total length of oral arguments (the sum of judge and non-judge speech). Average non-judge utterance is measured at the utterance level in seconds. Appellant and appellee controls reflect whether the first (if multiple) speaking attorney for a given side was non-white/white or female/male. Judge ratio controls are calculated by the proportion of judges on the panel in an oral argument who match that respective characteristic.
*p < .1, **p < .05, ***p < .01. Standard errors in parentheses. Case-level clustered-standard errors (CSE) are only available for average judge utterance length, which is measured at the case level.
+ Because time-fixed effects are at the year level and there is only one year (2014) that includes both the control group (audio-only cases) and treatment group (video cases), the calculation of the effect of ”Video/Streaming” is limited to only the 2014 year. Standard errors and p-values, however, incorporate the pre- and post-2014 variation in observed and unobserved factors.
The next two models add attorney-level controls that account for the gender and race/ethnicity for the first speaking attorneys representing the appellant and appellee (M2a). These models also add judge-panel composition controls that account for the proportion of the panel that is made up of women, non-White judges, Republicans, and judges who were rated “qualified” by the ABA (as opposed to “well qualified”) (M2b). The first four columns present our regression results for the average length of individual judge utterances, measured in seconds (a parallel Table to Figure 3). The next four present the results for the length of overall judge speech (parallel to Figure 4). The final four present the results for judge speech as a proportion of all speech in the oral argument (Figure 5).
As Figure 3 suggested, our first regression model (Column 1, M0) shows that the average judge utterance was 2.73 s longer in video cases than in audio-only cases. Once we introduce time-fixed effects (Column 2, M1), the estimated treatment effect of introducing cameras and live feeds on utterance length increases to 4.93 s, and the estimate rises again to 5.67 s with even stronger statistical significance with the introduction of our attorney-level covariate controls (Column 3, M2a). However, when we include the controls that account for the composition of the panel, the estimate becomes statistically indistinguishable from zero and changes sign (Column 4, M2b). The fact that the introduction of these judge-level controls influences the outcome of our model is not especially unexpected, given the significant differences we found in our balance tests between panel race/ethnicity, appointing party, and ABA designation in the treatment groups in Table 4 (Section 2.1).
Recall that Figure 4 showed that, as a group, judges spoke longer compared to non-judges with the introduction of video recording, but that the gray confidence interval bands overlapped significantly. As such, it is not surprising that our basic bivariate regression model estimating the effect of cameras on total length of judge speech (Column 5, M0) yields treatment effect estimates that are both non-significant. However, when we elaborate our model to include time-fixed effects (Column 6, M1) and attorney-level controls (Column 7, M2a), the additional statistical precision these models provide reveal statistically significant effects. Judges in the Ninth Circuit spent roughly 8 more min (7.42 in M1 and 8.01 in M2a) talking in camera cases than they did in cases prior to the introduction of video recording, a result that is statistically robust. However, once again, when we introduce the judge-level controls (Column 8, M2b), our estimated effects almost disappear. Based on the size of the coefficients, much of the explanatory power for the relationship seems to be coming from an increase in the proportion of women on these panels, which is interesting given that this factor was not significantly imbalanced across our treatment categories (see Table 4). We explore the potential implications of this significant correlation in our alternative models in Sections 5.2.3 and 5.2.4 and our discussion of implications in Section 6.
Interestingly, we see fairly significant correlations between attorney-level characteristics and the time judges spend talking in cases. On average, judges in the Ninth Circuit spoke nearly 4 min more in cases where the appellee attorney was white (compared to non-white attorneys) and 4 min less in cases where the appellee attorney is male. We see similar correlations with the race/ethnicity and gender of appellant attorneys, although the coefficients are less significant.
The third set of four models presents the results for the proportion of total speech by judges in a given case. Because this measure is a variation of total judge speech, the results are similar. In our models that have time-fixed effects (Column 10, M1) and include attorney controls (Column 11, M2a), judges in the Ninth Circuit take substantially more of the time speaking in oral arguments when the cameras are on—nearly 25% points more—and tend to speak more when the attorneys are white and female. However, once the judge panel controls are introduced, the estimated effect shrinks by over 50% and is no longer distinguishable from zero.
Regression Models on Attorney Speech (Interruption)
Notes. Average non-judge utterance is measured at the utterance level in seconds. Total length of non-judge speech is measured at the case level in minutes. Appellant and appellee controls reflect whether the first (if multiple) speaking attorney for a given side was non-white/white or female/male. Judge ratio controls are calculated by the proportion of judges on the panel in an oral argument who match that respective characteristic.
*p < .1, **p < .05, ***p < .01. Standard errors in parentheses. Case-level clustered-standard errors (CSE) are only available for average non-judge utterance length, which is measured at the case level.
+ Because time-fixed effects are at the year level and there is only one year (2014) that includes both the control group (audio-only cases) and treatment group (video cases), the calculation of the effect of ”Video/Streaming” is limited to only the 2014 year. Standard errors and p-values, however, incorporate the pre- and post-2014 variation in observed and unobserved factors.
Starting with the average length of non-judge utterances (Columns 1–4), we see large and statistically significant effects from cameras and live streaming. In our model that accounts for time-fixed effects and attorney characteristics, attorney utterances in camera cases are 11.15 s shorter on average than utterances in audio cases. Given that the mean length of non-judge utterances across the entire dataset is just over 21 seconds (see Table 2 in Section 4.3, above), this represents a substantial drop in the amount of time attorneys are allowed to speak before having to field a question or comment from the judges. Consistent with the relationship we saw between gender and judge speech above (where judges speak less when attorneys are men), we see that male attorneys are generally allowed to speak for a bit longer (roughly 3 s) than their female counterparts, with the relationship with male appellant attorneys being statistically significant. We also see a fairly substantial drop of over 8 min in the total amount of time that attorneys are allowed to speak in total in camera cases under our models that account for time-fixed effects and attorney controls (Columns 6 and 7).
However, once again, the introduction of the judge-level controls takes much of the explanatory power from the introduction of cameras that we saw in the previous models for both of our attorney outcome measures. While the size of the estimates is still relatively large—a nearly 5-s decrease in average non-judge utterance length and a 3.3-min drop in overall speech—the gender composition of the judge panels is correlated with much of the decrease during the camera period that we observed in Figure 6.
While unlikely, it is also possible that an individual judge’s behavior is largely independent of his or her panel colleagues, in which case measuring outcomes at the judge-case level might provide clearer insights into the relationship between cameras and our outcomes of interest. As a final variation, we analyzed the relationship between the introduction of video and judicial behavior on an alternative dataset where outcomes were calculated at the judge-case level. Doing this also allowed us to include judge-level fixed effects. We present the results of this judge-level variation on the judge-specific outcomes (those included in Table 5) in the Appendix (see the “M2a+” and “M2b+” columns in Table A4). While the scale of the estimates changed (not surprising given that this approach inherently looks at times for individual judges as opposed to the composite of all judges in a panel), the overall conclusions remained the same as our previous estimates. Judges spoke more and for longer durations, but some of our results are weaker or fail to achieve statistically significance once we account for panel demographics.
Taken together, these results present a mixed picture. Under our models that do not account for judge panel composition, the presence of recording, streaming cameras during oral arguments appears to affect judicial behavior. This, in turn, affects how much and for how long attorneys are allowed to speak. However, we identified imbalances across our treatment groups regarding the composition of judge panels (see Table 4). When we introduce those factors into our estimation models, the estimated effect of cameras on judge and attorney behavior shrinks and in some cases disappears entirely.
5.2.3. Treatment Interactions
The previous section shows results generally consistent with our hypothesis that cameras might cause judges to be more performative—by speaking longer and interrupting attorneys more—during oral argument. However, our model specifications with a full set of attorney- and judge-characteristic controls typically show statistically non-significant effects overall. In this section, we examine the possibility that those overall effects are masking important variation in behavior among different types of judges. In Table 7, we add interaction terms for attorney race and gender as well as for judge-panel compositions of race, gender, partisanship, and ABA rating. Each column shows these results for our five outcomes of interest under this model (M3).
Column 1 shows the results for the average length of judges’ utterances, controlling for all the same variables as model M2b in Table 5, 41 as well as the new interaction terms. The coefficient on the treatment variable indicates that when all attorneys and judges are white and male and all judges are Democratic appointees and well qualified under the ABA, 42 the presence of cameras reduces the average judge utterance length by 15 s. However, when the judges are all non-white, the presence of cameras increases the length of the average judge utterance by 12 s (the sum of the treatment coefficient and the interaction of treatment and “Non-White Panel Ratio”). Similarly, the baseline negative effect of cameras on the length of judge utterances reduces to zero when both attorneys are female. In other words, cameras seem to prompt judges to speak longer when the attorneys are female.
Column 4 shows similar results for the interactive effects of cameras on how long attorneys are able to speak. Attorneys’ utterances are substantially (over half a minute) shorter—indicating more frequent judge interruptions—when those judges are white and male, reflected by positive interaction effects on the ratio of non-white and female judges. As with the interactive effects on judicial speech length, we see that the average attorney utterance actually increases (by roughly 12 s) when an attorney argues in front of panels of non-white judges.
In regard to both average length of judicial speech and propensity for interruption, there appears to be substantial variation in the effect that the introduction of cameras has across judicial demographic categories. This may explain why our non-interactive regression models produced such strong correlations between gender composition of the judicial panels and many of our outcomes of interest. 43
To further explore this, and for ease of interpretation, we ran a series of reduced interactive models on each of our outcomes of interest where each model only interacts the treatment with one covariate at a time (see Appendix: Tables A7a, A7b, and A7c). The results of these models were, for the most part, consistent with our findings from the “saturated” models in Table 7 that interacted the treatment with all attorney and judge covariates. For example, the bottom panel of Table A7b shows first the saturated interaction model, and then a series of single interaction models predicting the average non-judge utterance length. Recall that this model tests for judicial interruptions: A shorter average attorney utterance indicates more interruptions by the judges. Overall, we see that the presence of video has a strongly negative significant effect, indicating that judges on video interrupt attorneys more. Column 6 in that panel shows the interactive effect between video and the proportion of female judges on the panel. While the main effect of video is a decrease in average non-judge utterances of about 10 s, when the judge panel is all female, the average attorney utterance is actually 2.5 s longer under video recording.
As with our non-interactive models, we also estimate interactive effects using the same model as we used in Table 7 (“M3”) but on a dataset calculated at the judge-case level that includes judge-level fixed effects (see the “M3+” columns of Table A4 in the Appendix). Aside from a few interactions that become significant, the results of this approach do not substantively differ from the results discussed above.
5.2.4. Subgroup Analysis
For our final regression specifications, we limit our analysis to a smaller set of judges who appear in cases throughout time period in which the court transitioned to cameras. In other words, this subgroup only includes judges who spanned at least part of the audio-only and part of the treatment periods. By including only judges with both audio-only and camera cases in our dataset, we can more precisely explore how the shift from audio to audio and video affected individual judges. We do this to account for the imbalances in judge characteristics we identified in Tables 4 and A3 (Appendix), which may reflect confounding factors that were introduced through the passage of time. However, this move necessarily reduces our statistical power: Only nine judges meet these inclusion criteria, so we can only analyze the cases where only those judges were empaneled.
Interactive Regression Models on Judge and Attorney Speech (Interruption)
Notes. Average judge utterance length is measured at the utterance level in seconds. Total length of judge speech is measured at the case level in minutes. Total judge speech (proportion) is measured at the case level as the proportion of judge speech relative to the total length of oral arguments (the sum of judge and non-judge speech). Average non-judge utterance is measured at the utterance level in seconds. Total length of non-judge speech is measured at the case level in minutes.
Attorney-level and judge-panel controls were included in the model but excluded from the table for clarity. The appellant and appellee controls reflect whether the first (if multiple) speaking attorney for a given side was white/non-white or male/female. Judge ratio controls are calculated by the proportion of judges on the panel in an oral argument who match that respective characteristic.
*p < .1, **p < .05, ***p < .01. Standard errors in parentheses. Case-level clustered-standard errors (CSE) are only available for average judge and non-judge utterance lengths, which are measured at the case level.
+ Because time-fixed effects are at the year level and there is only one year (2014) that includes both the control group (audio-only cases) and treatment group (video cases), the calculation of the effect of ”Video/Streaming” is limited to only the 2014 year. Standard errors and p-values, however, incorporate the pre- and post-2014 variation in observed and unobserved factors.
Our results using these subsets are generally consistent with our previous observations, with some exceptions. Unlike our standard model, there is a statistically significant increase in average judge utterance length even when controlling for all attorney- and judge-level characteristics and including fixed effects (Column 1, M2b), but when we introduce interaction terms (Column 2, M3), the direction of that effect becomes negative and indistinguishable from zero. However, we see large and statistically significant interactions with appellee gender and the proportion of non-white judges on an oral argument panel. We also see a significant reduction in the average non-judge utterance length using our interaction model (Column 8, M3), along with a large and significant interaction between cameras and the proportion of qualified (as opposed to well-qualified) judges on a panel. The total length of non-judge speech increases with marginal statistical significance in our non-interactive model that includes all controls and fixed effects.
6. Conclusions and Implications
In this study we leverage a natural experiment, the introduction of cameras and video feeds in U.S. Ninth Circuit oral arguments, to examine the oft cited but previously untested hypothesis that judicial behavior changes when the judges know they are being filmed. Examining visual trends alone (see Figures 3–6), we see that judges begin to speak longer utterances and consume a larger share of proceedings compared to attorneys at precisely the same time that the Ninth Circuit began implementing video cameras. These observations are consistent with the performative judging hypothesis.
Regression Models on Pre-Post Judge Subsets
Notes. Average judge utterance length is measured at the utterance level in seconds. Total length of judge speech is measured at the case level in minutes. Total judge speech (proportion) is measured at the case level as the proportion of judge speech relative to the total length of oral arguments (the sum of judge and non-judge speech). Average non-judge utterance is measured at the utterance level in seconds. Total length of non-judge speech is measured at the case level in minutes.
Attorney-level and judge-panel controls were included in the model but excluded from the table for clarity. The appellant and appellee controls reflect whether the first (if multiple) speaking attorney for a given side was white/non-white or male/female. Judge ratio controls are calculated by the proportion of judges on the panel in an oral argument who match that respective characteristic.
! The interactions between the treatment (video/streaming) and some covariates were excluded from some models (those with case-level outcomes) due to collinearity resulting from the smaller, 2014-only sample size.
*p < .1, **p < .05, ***p < .01. Standard errors in parentheses. Case-level clustered-standard errors (CSE) are only available for average judge and non-judge utterance lengths, which are measured at the case level. + Because time-fixed effects are at the year level and there is only one year (2014) that includes both the control group (audio-only cases) and treatment group (video cases), the calculation of the effect of ”Video/Streaming” is limited to only the 2014 year. Standard errors and p-values, however, incorporate the pre- and post-2014 variation in observed and unobserved factors.
Of course, our mixed results could reflect trends that are either partially or fully independent of the introduction of cameras. For example, it could be that the significant results we identify result from underlying time-based trends towards more active, performative appeals hearings. It is also possible (though we think unlikely) that cases in the Ninth Circuit happened to become more complex during the period we studied, and that it was this increased complexity—rather than the presence of cameras—that prompted judges to become more active and involved. More realistically, shifts in the personal characteristics of Ninth Circuit judges or attorneys might also explain the trends we observed if, for example, a particular style or personality type became dominant during the period we studied. For example, many of our preferred models show a statistically and substantively significant correlation between judge gender and our outcomes of interest, which, when accounted for, renders the treatment effect in some models null. It may be that male judges simply speak and interrupt more, 45 and the introduction of cameras in the Ninth Circuit happened to coincide with a shift towards more male-heavy judicial panels.
Our findings might also reflect broader cultural trends. As society becomes increasingly dominated by Tik Tok and Instagram, it could be that all of us, not just judges, are becoming more performative. These explanations are consistent with the fact that some of our significant results disappear when we include controls in our models (particularly judicial-panel characteristics). But these alternate possibilities do not explain why some models that do control for these characteristics nonetheless find some significant results. Alternate accounts also do not explain the reversal in time trends and the discontinuities that we identify in the raw data.
It is also possible that the underlying selection process we used in our diarization optimization methods created treatment and control groups that contain non-random variation in pre-treatment characteristics, which ultimately drive the results of our more sophisticated models. But that would require the assumption that an ostensibly unbiased process—the optimization method—yielded unobserved heterogeneity sufficient to create the stark trends we observe in the raw results and the significant estimates of some models.
Ultimately, we believe that our results tentatively support the performative judging hypothesis. This conclusion has important implications. If cameras do, in fact, change judicial behavior, then there is a real possibility that cameras could shift the focus of judicial proceedings from substance to spectacle. This, in turn, could affect case outcomes and undermine public perceptions of the judiciary (Marder, 2021, p. 20). In short, our tentative results suggest that the concerns prior scholars have raised are not completely unfounded. Though we cannot definitively say whether or how cameras affect judicial behavior, our study provides empirical evidence that courts and policymakers ought to be cautious before jumping on the camera bandwagon, and that the benefits to transparency and public access that cameras provide need to be weighed against the risks that judges will shift their behaviors in harmful ways.
Our results also illustrate the power of new, AI-driven methodologies like Reference-Dependent Speaker Verification. If used more broadly, RDSV and other deep learning processes could enhance our understanding of what happens in judicial proceedings. Future researchers should thus embrace these methodologies to enhance their study of court rulings, opinions, and proceedings. Court administrators and judges could likewise use these methodologies to learn about their own systems and behaviors. The fast-paced development of these methods might even allow courts to diarize a much broader and deeper set of cases and, in doing so, produce some of the “sunshine” and transparency courtroom cameras are meant to achieve.
Finally, and most importantly, our study reiterates the need for careful research about the effects of cameras in court. To our knowledge, this is the first empirical study to explore how cameras affect judicial behavior in federal appellate courts. But as we have discussed above, our study is limited, and our results do not definitively prove or disprove the performative judging hypothesis. Indeed, we echo the conclusions of previous researchers who have concluded that we need “more research . . . before Congress and the Court introduce cameras into the Courtroom” (Black et a., 2023b, p. 8). We urge future scholars to take up that work.
Supplemental Material
Supplemental Material - Performative Judging? Measuring the Effect of Video Recording on Judicial Behavior in Circuit Court Oral Arguments
Supplemental Material for Performative Judging? Measuring the Effect of Video Recording on Judicial Behavior in Circuit Court Oral Arguments by Aaron R. Kaufman, Dane Thorley, and Lucy Williams in Journal of Law & Empirical Analysis
Footnotes
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
