Grading Machines: Can AI Exam-Grading Replace Law Professors?

Abstract

In the past few years, large language models (LLMs) have achieved significant technical advances, enabling legal-advocacy organizations to adopt them as complements to—or substitutes for—lawyers and other human experts. The role of LLMs in legal education, however, is underexplored. While several studies have examined LLMs’ performance in taking law school exams, finding mixed results, there have been no published studies systematically analyzing LLMs’ competence at one of law professors’ chief responsibilities: grading law school exams. This paper presents results of an analysis of how LLMs perform in evaluating student responses to legal analysis questions of the kind typically contained in law school exams. The data come from exams in four subjects administered at top-30 U.S. law schools. Unlike some projects in computer or data science, our goal is not to design a new LLM that minimizes error or that maximizes agreement with human graders. Rather, we seek to determine whether existing models—which can be straightforwardly applied by most professors and students—are already suitable for law exam evaluation. We find that, when provided with a detailed rubric, the LLM grades correlate with the human grader at Pearson correlation coefficients of up to 0.93. Our findings suggest that, even if they do not fully replace humans in the near future, LLMs could soon be put to valuable tasks by law school professors, such as reviewing and validating professor grading, providing substantive feedback on ungraded midterms, and providing students feedback on self-administered practice exams.

1. Introduction

Led by OpenAI’s Generative Pre-trained Transformer (GPT) model family, large language models (LLMs) have achieved significant technical progress over the past several years. These advances have inspired legal technology firms to explore how LLMs might assist lawyers in tasks as varied as document review, contract and motion drafting, and brief writing and editing. In many cases, LLMs are being used to complement the work of lawyers, and—at least for some discrete tasks—they are partly or fully substituting for them. But given the speed of the technology’s evolution, it is still unclear if, when, and to what extent LLMs will become a genuinely suitable substitute for human legal analysis.

The questions typically contained in law school exams offer a valuable way to assess this technology’s capacity for legal analysis. These exams test foundational doctrinal knowledge, demand creative and multifaceted problem solving, and, importantly, come with a built-in comparison group of human test-takers and graders. For these reasons, several studies have examined LLMs’ performance in taking law school exams, finding mixed (but increasingly impressive) results (e.g., Fan et al., 2025). Yet no published studies have systematically analyzed LLMs’ competence in the closely related but conceptually distinct task of grading law school exam answers, a core responsibility for legal educators.

This paper analyzes how commercial LLMs, primarily OpenAI’s GPT-5, perform in evaluating law school “issue spotters,” a common exam format that requires students to identify legally relevant issues in a fact pattern and analyze them under the relevant legal rules. The data consist of four final exams, each of which was recently administered at a top-30 U.S. law school. The exams span three standard first-year courses (civil procedure, contracts, and torts) and one upper-level course (corporate law), and include both the actual student answers submitted and the grades assigned by faculty. We find that, without any guiding instructions beyond a basic prompt and the min–max range for scores, the LLM-produced grades correlate with the human-assigned (professor) grades at Pearson correlation coefficients of up to 0.80 (ranging from 0.66 to 0.80 across the four exams analyzed). When provided with a detailed rubric that the professor used to grade the exam, that figure reaches 0.93 (ranging from 0.78 to 0.93).

We acknowledge that professional regulations, ethical concerns, and inertia may prevent law schools from completely replacing human graders with LLMs in the short term. In some ways, the ethical issues surrounding machine grading echo longstanding debates about automation in other fields, such as autonomous vehicles. A primary concern is that machine graders will make errors that humans would not, raising fairness concerns—even if the aggregate number and magnitude of errors made by LLMs are comparable to or lower than those made by humans.

Many of these concerns give insufficient weight to the substantial and well-documented limitations of human grading. Human graders are prone to inconsistency, especially under fatigue. They may introduce unconscious biases based on perceived student identity, even under blind grading conditions. In contrast, AI grading has the potential to reduce these sources of unintended variation and to offer more consistent, unbiased assessments. We believe that machine grading should be evaluated against realistic institutional benchmarks that acknowledge the fallibility of human grading, rather than an idealized standard of human perfection.

Even if AI grading does not supplant human grading anytime soon, our findings suggest that LLMs could prove valuable for several related tasks in law and legal education. For example, LLMs could be deployed to review professor grading for errors or bias and to provide students with rapid, reasonably reliable practice-exam feedback tailored to the grading tendencies of their professors. Outside of the law school setting, LLMs could serve parallel functions by helping senior lawyers evaluate the work of junior lawyers and by giving junior lawyers constructive feedback on their own performance. Beyond these human-development applications, reliable AI grading of legal work product could facilitate more consistent benchmarks for assessing AI’s legal capabilities and perhaps even operate as one component of a fully automated AI lawyering system.

The rest of this Article proceeds as follows. Section 2 gives a brief history of automated essay scoring and reviews how LLMs have begun to change legal practice and pedagogy over the last few years. It then considers ethical and political issues surrounding machine evaluation of exams. Section 3 describes the study’s design and methods for measuring how LLM exam evaluation compares with that of human graders. Section 4 provides the empirical results. Section 5 discusses the implications of the findings for the future of exam grading and the assessment of legal work product more generally. Section 6 concludes.

2. Background and Motivation

2.1. Early Automated-Essay-Scoring Efforts

Automated Essay Scoring (AES) originated in the mid-1960s with the work of Ellis Batten Page, who argued that computers could be trained to score essays as reliably as human graders. In 1968, Page demonstrated this claim with Project Essay Grade, one of the first computerized essay-grading systems. Though early versions were not cost-effective, the proliferation of personal computers in the 1990s renewed the practical viability of AES (Page, 2003).

In 2012, the Hewlett Foundation sponsored the Automated Student Assessment Prize (ASAP). Several teams showed that AES systems could match human rater reliability across multiple essay prompts. Although these initial claims of equivalence were controversial due to methodological flaws, the competition nonetheless spurred interest in modern AES technologies. A year later, EdX, a nonprofit founded by Harvard and the Massachusetts Institute of Technology, introduced an AI software package for grading student essays and made it freely available, with the hope that it would free professors up for other tasks without sacrificing grading validity. EdX president Anant Agarwal claimed that the software’s grading quality was “similar to the variation you find from instructor to instructor” (Markoff, 2013). But critics such as MIT’s Les Perelman believed that AES had serious limitations, arguing that its supporters had not validly tested the package’s grading performance against human instructors. Perelman and other educators claimed that computers not only cannot “read,” but also cannot assess important aspects of writing such as reasoning, evidence, or clarity (Markoff, 2013).

2.2. Language Models

Recent advances in LLMs have begun to address these limitations. LLMs emerged from the deep-learning transformer architecture developed by a team of Google scientists in 2017 (Vaswani et al., 2017). That breakthrough laid the foundation for a new generation of LLMs such as OpenAI’s GPT and comparable models developed by Anthropic, Google, Meta, and others. These models are trained on large text corpora and generate language by predicting the next token in a sequence based on the preceding text. They have been shown to follow detailed instructions, respond to complex and structured prompts, maintain context across long inputs, and perform multi-step reasoning. Increasingly, LLMs are embedded within frameworks that structure prompts, constrain outputs, and aggregate or validate responses over multiple passes.

Such models can process text and generate new content with a fluency that often resembles human communication. Modern LLMs appear to have learned not only statistical patterns of language, but also factual information and conceptual relationships embedded within their training data. As a result, they can perform tasks—ranging from answering complex questions to summarizing dense materials—that were once thought to require human judgment. While their apparent ability to understand language remains a product of statistical pattern recognition, the scope and sophistication of these AI models blur the line between automated processing and tasks traditionally entrusted to human experts. This raises a natural question: if these systems can generate useful analyses of texts, might they also be capable of evaluating text quality against pre-specified criteria, such as the standards governing legal work product?

2.3. AI in Legal Analysis, Pedagogy, and Practice

To our knowledge, LLMs have thus far not been systematically tested on law exam grading, but they have been applied to many other aspects of legal practice and pedagogy. ¹ Over just the past few years, researchers have begun to explore how generative language models might aid or replace lawyers in an increasingly diverse set of tasks such as statutory analysis (Blair-Stanek et al., 2023; Engel & McAdams, 2024), cryptosecurities legal analysis (Trozze et al., 2024), contract review (Martin et al., 2024, finding that several LLMs outperformed lawyers in contract-review accuracy), tax problems (Nay et al., 2024), legal-text annotation (Gray et al., 2023; Savelka & Ashley, 2023), the extraction of legally relevant features from corporate governance documents (Frankenreiter, 2025; Frankenreiter & Talley, 2026), bail decisions (Kleinberg et al., 2018), and memo drafting and editing (Simon, 2023, finding that LLMs made significant errors in drafting and editing legal memos). ² During this same time period, researchers have developed several legal-benchmarking tools to systematically evaluate progress in the legal reasoning capabilities of LLMs (Fan et al., 2025; Guha et al., 2023; Posner & Saran, 2025).

In light of LLMs’ performance on legal tasks to date—their successes as much as their failures—commentators have emphasized the need for attorneys to employ these tools ethically and responsibly (Murray, 2023; Pierce & Goutos, 2023). Towards this end, some have called for new regulations governing lawyers (Cyran, 2024), while others have suggested incorporating new rules on the ethical use of LLMs into the rules of professional responsibility (Avery et al., 2023).

LLMs are also changing legal education (Bliss, 2024). Some studies have measured the ability of generative language models to produce answers to law school exams (Blair-Stanek et al., 2023, 2024; Choi et al., 2022; Hargreaves, 2023). One study found that, compared with students using traditional resources, students with access to GPT-4 showed significantly stronger performance on multiple-choice questions but no better performance on complex essay questions (Choi & Schwarcz, 2025). ³ Others have tested LLMs’ performance on the U.S. Uniform Bar Examination (Katz et al., 2024; Martínez, 2024), finding that GPT-4 scored above every jurisdiction’s passing threshold. Particularly germane to the question of LLM use in the context of “replacing” (or augmenting) law professors, a recent study evaluated commercial LLMs’ ability to answer student questions in the context of law school office hours (Ouellette et al., 2026).

2.4. Exam Evaluation and Grading

Finally, researchers have recently begun to explore LLMs’ ability to evaluate and grade test answers across a range of subject matters—including the physical sciences (Kortemeyer, 2023), mathematics (Yang & Zhu, 2022), and collaborative Java programming (Tomić et al., 2022)—as well as across different response formats, such as freeform responses (Mitros et al., 2013). Yet no study to our knowledge has attempted to gauge how well generative language models can grade law school issue spotter exams, which require students to identify and analyze legal issues presented in lengthy hypothetical fact patterns.

Evaluating answers to such legal analysis problems poses challenges that most other test formats do not. First, student answers are in a form known in the testing literature as constructed response, as opposed to so-called selected-response answers, such as multiple choice and matching. Constructed response answers are significantly more difficult to validate (Hussein et al., 2019). Moreover, most answers to legal analysis questions are extended-response items, the most complex and demanding type of constructed response, for both the test-taker and the grader (Nitko, 1996). ⁴ Because the answers are provided in prose rather than discrete options or code, a sophisticated text analysis is required to evaluate them. ⁵ Second, and relatedly, there are usually many ways to capably answer legal analysis questions, some of which the grader may not even anticipate. Therefore, algorithmic answer keys are unlikely to produce scores of acceptable accuracy.

For these reasons, AES has historically performed poorly in evaluating the sort of complex reasoning contained in law school exam answers. However, with recent advances in LLM technology, these systems now represent a potentially promising tool to assist or replace human graders in evaluating such exams.

With this background in mind, this Article compares exam-grading outcomes by a state-of-the-art, generally accessible LLM, with those of the professors who authored and officially graded those same exams. For reasons we elaborate below, including the innate fallibility of human grading (see Sections 4.3 and 5.2), we believe that perfectly replicating human performance should not be the sole objective of a machine-grading model. However, showing that an LLM can approximate the grading outcomes of human experts would at least partly address ethical and institutional concerns about the practice, possibly paving the way for their adoption in law schools and other legal-training environments.

3. Research Design: LLM Performance in Grading Exams

The rest of this Article empirically investigates the ability of LLMs to evaluate answers to legal analysis questions. In this section, we begin this assessment by outlining our research goals and research design choices, identifying the dataset of law school exams we used, and describing the specific LLM grading methods we employed.

3.1. Research Goals and Design

Our overarching goal in designing this study was to determine whether LLMs can be used to make automatic legal analysis scoring both effective and accessible to instructors. Rather than attempting to identify or develop a new model that matches human grading as closely as possible, we therefore sought to evaluate the baseline ability of current LLMs to validly grade exams across a variety of methods that instructors could easily implement using information they already have. The grading procedures we evaluate are those accessible to most law school instructors, without extensive prompt engineering. ⁶

Accordingly, all prompting approaches we examine rely on a common, reusable structure which is applied across exam questions and legal subject matter with only limited changes across settings. In particular, the model is always provided with the exam question, a student’s answer, the maximum point value for the question, and, under some approaches, the grading rubric supplied by the instructor. ⁷ In one approach we tested, we additionally varied the instructions governing score assignment to more closely mirror the structure of the grading rubric. However, as we discuss below, this approach yielded little or no marginal improvement relative to its added complexity. Other than in this approach, the wording of the prompts and the instructions governing score assignment are held constant across all four exams in our sample. While different courses and doctrinal areas inevitably require different substantive inputs, the stability of the prompt structure across courses suggests that the observed performance is not driven by idiosyncratic prompt tuning, but rather by the model’s ability to apply instructor-supplied grading criteria to new legal contexts.

This approach has important limitations if the goal of automated exam grading is to provide an accurate signal of the student’s latent (unobserved) mastery of the subject matter (Chilton et al., 2024). In the terminology of the applied econometrics literature on prediction and machine learning (Athey & Imbens, 2019; Kleinberg et al., 2015; Mullainathan & Spiess, 2017), this objective can be viewed as a prediction or signal-extraction problem: the automated grader seeks to recover a latent trait (a student’s true mastery) from the observable text of their answer. Employers and others rely on this signal when evaluating law students. As the quality of the signal deteriorates, grades become less informative, leading employers who rely on them to make worse hiring choices.

This view of grading poses a fundamental challenge for evaluating the effectiveness of automated legal analysis scoring, as it complicates the task of establishing a baseline for the evaluation. Because humans are also imperfect graders (as we discuss further in Section 5.2 below), we cannot conclude that human evaluators’ decisions perfectly reflect students’ true abilities. Thus, discrepancies between LLM-generated and human-assigned grades do not imply that the machine’s grades are invalid; mistakes by human graders could also be the source of any discrepancies. Of course, both graders could also be in error, meaning that observed agreement between the two types of graders does not imply validity. More generally, patterns of agreement or disagreement alone do not allow us to determine which of any two given evaluators—such as human and machine—has produced the more valid measure of a student’s ability.

With that understanding, this study treats human evaluations not as the target for researchers and data scientists, but as the comparator for present purposes. We do so for three reasons. First, as long as humans remain primarily responsible for assigning formal grades, human practice will remain the effective standard for people and institutions. Indeed, despite human grading’s imperfections, legal employers seem to put great weight on human-produced law school grades (valuing them over, say, objective LSAT scores). And students seeking feedback on practice exams are likely to be most interested in scores that match their own human instructor’s grading, even more so than in a “true” measure of the student’s ability.

Second, when evaluating proposals to formally replace human grading with machine grading, human performance necessarily serves as the baseline. It is the existing practice and therefore the natural point of comparison for educational decision-makers. Consider a hypothetical AI grader that always perfectly replicates the grading outcomes of a panel of expert professors who teach the course and write the exam. We cannot know with certainty that this AI grader is producing scores that perfectly measure students’ legal analysis skills. Nonetheless, most would agree that such an AI grader would be an ethically, politically, and institutionally suitable—if not superior—substitute for a typical single human grader. Conversely, a model that produced wildly different scores from humans would face strong institutional resistance to implementation, even if there were strong a priori arguments that it better measured student skill. Thus, although correlations between human and machine grading do not by themselves establish how accurately an AI system measures underlying legal competence, they do speak directly to whether an AI grader could function as a substitute for human graders within existing ethical, political, and institutional constraints.

Finally, we use human grading as a benchmark simply because no method currently exists for reliably determining the ”true” skill embodied within a law school exam answer. In theory, a longitudinal study of students-turned-lawyers could examine how well different grading techniques predict students’ later ability to conduct high-level legal analysis. Such a study, however, would take years, and would still face confounders and measurement challenges.

For these reasons, for employers, administrators, regulatory institutions like the American Bar Association, and other stakeholders, a model that approximates human-grading outcomes will be treated as more legitimate than one that deviates from those outcomes. We return to this issue in Section 4.3, where we empirically compare agreement between LLM- and human-generated grades, on one hand, to agreement between human-generated grades produced for the same exam at different periods, on the other.

3.2. Dataset

We began by assembling a dataset comprising final-exam questions and anonymized student answers from several core law school courses, including three traditional first-year courses (civil procedure, contracts, and torts) and one upper-level course (corporate law). ⁸ Each exam was recently authored, administered, and graded by one of the authors, each at a different U.S. law school ranked among the country’s top 30 (according to the U.S. News & World Reports, 2025 Law School Rankings). The four exams differ by subject matter and length, but each exam included at least two legal analysis questions in the format familiar to U.S. law students and instructors (see, e.g., note 5). Each instructor relied on a grading rubric in evaluating their exams, the complexity of which varied among the instructors. ⁹ A redacted example of one of the rubrics is included in Appendix A.

3.3. LLM Grading Method

We initially experimented with various prompting approaches to instruct LLMs to evaluate the exam answers. They ranged from simple prompts asking the model to assign points based on its understanding of the law, to more complex prompts that incorporated instructor-supplied grading rubrics with detailed scoring instructions. We also explored approaches in which the LLM conducted pairwise comparisons of exam answers, with human researchers subsequently deriving grades using a scaling model based on those comparisons.

We ultimately tested four approaches to automated legal analysis scoring, which are described below. The full prompts are included in Appendix B. All exams contained multiple independent questions based on distinct factual scenarios; we generally obtained scores for the answers to each question and calculated total scores for each student’s exam by summing those question scores. Some approaches also involved scoring individual elements of an answer (“issues,” in law school parlance) and then summing those element scores to calculate a score for the question.

3.3.1. Open

Under this prompting approach, the model receives a simple prompt directing it to assign a numerical score for the answers to each question based on the text of the question, the student’s answer, and the maximum score for the question. This approach relies solely on the internal capacity of the LLM to interpret and apply legal concepts in grading. It has the benefit of simplicity and ease of adoption. However, since it draws its information entirely from sources external to the course, it is the approach we tested that is most likely to produce scores that deviate from those assigned by the professor.

3.3.2. Rubric

This prompting approach provides the model with the same information as the Open approach and likewise asks the model to produce a score for the answers to each question, but also incorporates into the prompt the rubric information that the professor used in grading the exams. Although the rubrics used for each of the exams vary, they all outline the elements that answers should address for each question, along with maximum points allocated to each element. The prompt asks the model to consider those elements and points for each element in determining the score for each question, though it does not require the model to include those in its output. This approach provides the advantage of nudging the model to consider the information that the professor values, but it is more human-labor-intensive than the Open approach, as it requires the instructor to create a rubric for each exam question.

3.3.3. Bespoke

This approach follows the Rubric approach but includes some exam-specific language and instructions requiring the model to output disaggregated scores for each rubric element. Question scores are obtained by summing the scores that the model assigns to the answer for each rubric element. This approach amplifies both the advantages and disadvantages of the Rubric approach, as it is more guided but also more labor-intensive, as parts of the code must be customized for each different rubric.

3.3.4. Pairwise

This approach is similar to the Rubric approach in that the model is provided with the exam question, student answers, and the rubric. However, rather than asking the model to return a score for each exam, each query presents two students’ answers, and the model is asked to conduct pairwise comparisons between the two and determine which of them is better. ¹⁰ After obtaining pairwise comparisons for all answer pairs, we compute scores for each answer using the Bradley–Terry scaling algorithm (e.g., Hunter, 2004), and then rescale those scores to match the range of scores assigned by the instructor. This approach shares most of the advantages and disadvantages of the Rubric approach, but it is more complex, substantially more computationally intensive, and therefore much more expensive to implement. On the other hand, as we discuss in Section 4.2.2 below, the pairwise approach might produce more accurate grades.

3.4. Implementation Details

All reported evaluation scores were obtained using OpenAI’s GPT-5 model. ¹¹ Queries were submitted through the OpenAI Batch API via the openai package in Python. ¹² This setup allowed us to process thousands of prompts efficiently and at scale while maintaining a consistent prompt structure and data output. ¹³ The implementation of our methods involved three main stages.

3.4.1. Data Preparation

Using custom Python scripts, we converted the PDF files of each student’s exam, as generated by the ExamSoft software, into text and then split that text into separate files corresponding to each student’s answer to each question. We also created separate text files containing the text and rubric for each question, as well as a system prompt template and various user prompt templates in the form of text files. Each of the four approaches outlined above had a different template. For the Open, Rubric, and Pairwise approaches, templates were used across exams; for the Bespoke approach, the template was further edited for each exam question to match the items in the corresponding rubric. The system prompt template; the Open, Rubric, and Pairwise user prompt templates; and the Bespoke prompt template for Exam 1 are included in Appendix B.

3.4.2. Prompt Generation and Submission

We used Python to generate individual prompts for the analysis of each question, inserting the relevant student answer (or answers, for the Pairwise approach), question text, and rubric (for the Open, Bespoke, and Pairwise approaches) into the appropriate template, and saving the resulting prompt as a text file. ¹⁴ We then assembled groups of prompts and their corresponding schemas into a JSON Line file and submitted them to the OpenAI Batch API v1 as completion-style requests using the official OpenAI Python client library, requesting JSON-formatted output. ¹⁵

3.4.3. Response processing

Once each batch job was complete, we downloaded the results from the batch job and split them into separate JSON files with the scores for each prompt. We used Python to parse the JSON files and extract the scores into CSV files for analysis. For the Open and Rubric approaches, this process yielded question-level scores directly; for the Bespoke approach, we obtained such scores by aggregating element-level scores. For the Pairwise approach, we calculated question-level scores using the Bradley–Terry scaling algorithm described above.

3.5. Data and Analysis

Using the procedure outlined above, we produced scores for each student’s answer to each question under each of the four approaches examined in this study. We also constructed exam-level scores by summing question-level scores across all of the questions in each exam. ¹⁶ Our sample includes a total of N = 205 students, unevenly distributed across the four exams. ¹⁷ At the exam level, this yielded 820 LLM-generated scores (205 per approach); at the question level, 4,012 scores (1,003 per approach).

The scores derived from each of the four methods were then compared with the scores that the respective human grader assigned. Our primary measures of comparison are Pearson correlation coefficients (r) and Spearman rank correlation coefficients (ρ), both calculated between the human-graded and machine-graded scores for each student.

These two measures capture different aspects of agreement and deviation between human- and machine-assigned grades. Which of the two is more appropriate might depend in part on prevailing grading practices. The Pearson correlation coefficient captures the relative distance between raw exam scores, making it well suited to grading practices that place significant emphasis on raw scores for allocating exams to different grades (e.g., where instructors derive final grades by rescaling raw scores to letter grades such as A, A-, or B+, or to corresponding numerical grades such as 4.0, 3.7, or 3.3). By contrast, where instructors rely primarily on ranking exams and assigning grades based solely on rank (for instance, through curve bins with predetermined numbers of students assigned to each grade category), information about the distances between scores becomes less relevant than the rank order of student scores. Spearman’s rank correlation is better suited in this context, as it focuses on the relative ordering of students and is unaffected by the distribution of the scores assigned to them.

4. Results

4.1. Main Results

Figure 1 provides a high-level overview of the Pearson and Spearman correlations by exam and prompt approach. Across all exams, our most straightforward approach, Open, achieves high-to-very-high correlation levels: the Pearson correlation between LLM- and instructor-assigned grades is 0.66 or higher for all exams, with the most highly correlated exam reaching 0.80 (μ = 0.71). The corresponding Spearman correlations are slightly lower, ranging from 0.59 to 0.77 (μ = 0.67). These results suggest that LLMs, through their training alone and without further instruction, are able to capture meaningful differences in exam quality.

Figure 1.

Heatmaps of LLM performance across exams. The left heatmap shows Pearson correlations between LLM- and instructor-assigned grades for each exam and grading approach, the right heatmap shows the corresponding Spearman correlations.

Across all four exams, grading accuracy improves meaningfully under the Rubric approach, in which an instructor-created rubric is added to the information provided to the LLM in the Open approach. Pearson correlations for the Rubric approach range from 0.78 to 0.93 (μ = 0.87), and Spearman correlations range from 0.74 to 0.90 (μ = 0.85). Interestingly, there appears to be some variation in how much the switch from Open to Rubric improves performance. For example, LLM grading of Exam 4 performed roughly similar to Exam 3 under the Open approach, but improved considerably more under the Rubric approach. This difference may reflect variation in the amount of additional information contained in the rubrics themselves, or conversely, variations in the model’s ability to evaluate answers to certain exams without such information. We briefly explore the role that rubric detail might play in model performance in Section 4.4 below.

Finally, the Bespoke and Pairwise approaches both offer some potential improvement over Open, but do not appear to consistently improve performance in comparison to Rubric. We revisit the results from both of these approaches in more detail in Section 4.2 below.

For our preferred approach, Rubric, we reran the full analysis ten times to assess the stability of results across repeated model calls. Both individual grades and the resulting correlations vary only modestly across runs. ¹⁸ Given the modest variation across runs, we focus on the results from the main run in the remainder of the paper.

Figures 2 and 3 display the main results in more detail. Figure 2 contains scatterplots showing how LLM-generated, exam-level grades (on the y-axis) compare to human-assigned ones (on the x-axis), both scaled to 100. The gray dashed lines in each chart indicate a hypothetical perfect agreement: If all points fell on this line, the model would have replicated the human-assigned grades perfectly. Points that appear above the dashed line represent exams that received a higher relative grade from the LLM than from the human grader, whereas the opposite is true for points that fall below the dashed line. The solid colored line in each plot displays a linear regression line fitted to the observations.

Figure 2.

Comparison of performance across grading approaches. Each scatterplot plots LLM-assigned grades against instructor-assigned grades for one exam, with the dashed line indicating perfect agreement. The solid line denotes a linear regression line fitted to the observations. Numbers in the upper-left corner of each panel report the Pearson correlation (r), Spearman correlation (ρ), and number of observations (N).

Figure 3.

Correlations between LLM- and instructor-assigned grades at the question and exam levels. Barplots show Pearson correlations for each individual question score and total exam scores for each exam. Colors indicate the prompting approach.

Notably, for three of the four exams, the model appears to systematically over- or undershoot the human-assigned grades. This pattern emerges consistently under the Open, Rubric, and Bespoke approaches, although it appears to play out differently in Pairwise. ¹⁹ The main performance metrics we rely on do not account for this over- and undershooting, since most law schools grade on a curve; what determines the assigned grade is not absolute score, but relative score. Accordingly, when interpreting the scatterplots, the more important question is whether the points are tightly clustered around the regression line, not whether they are clustered around the dashed line indicating perfect agreement.

The relative ordering of correlations in Figure 1 is also evident in the scatterplots: the points are more tightly clustered around the regression line under the Rubric approach than under the Open approach, indicating that the model generates a closer approximation of human grading in a relative sense (even if absolute scores are systematically a bit higher or lower). Results from Bespoke appear broadly similar to those of Rubric, not only in terms of correlations but also in the overall grade distribution. By contrast, while Pairwise performs comparably to Rubric and Bespoke in terms of correlations, its grade distribution differs noticeably.

Figure 3 breaks down the exams into individual question-level scores, displaying the correlations for each question within each exam, along with the Pearson correlation for the total score for each exam. The different bar colors represent the four prompting approaches. Across all exam questions, correlations are uniformly positive, indicating that the observed alignment between LLM- and human-assigned grades is not driven by a small subset of questions. As with the overall exam scores, the Rubric approach substantially outperforms Open for almost all exam questions. The only exceptions are a few questions in Exam 3, where Open achieves comparably strong results to Rubric. These questions appear to be outliers, as the performance of all models on these questions is substantially lower than for other exam questions. Consistent with the results for total scores, Bespoke and Pairwise rarely yield meaningful improvements over Rubric at the individual question level.

Within individual exams, the correlations for total scores are generally similar to or higher than those for the questions on which LLM grading performs best. This pattern suggests that there is noise in either human or LLM grading (or both) that is canceled out when a larger number of exam questions are aggregated into a total score.

4.2. The More Resource-Intensive Approaches

The results in the previous section show that the Rubric approach outperforms Open by a substantial margin across both exams and questions. We now turn to consider in more detail whether the LLM’s performance can be improved by using either of the more labor-intensive approaches, Bespoke or Pairwise.

4.2.1. The Bespoke Approach

Although Rubric provides the model with explicit guidance on how to value the different components of a response, it ultimately relies on the model to adhere to those instructions by calculating final question scores based on the partial scores it assigns to the sub-topics specified in the rubric. The Bespoke approach differs in that it requires the model to output scores at a more granular level, which we then aggregate to generate a question-level score. This approach essentially mirrors that of an instructor who uses a spreadsheet to record the number of points a student earns for each element of her answer, summing the points to produce a total score for each question.

A priori, we expected the Bespoke approach to perform better than the Rubric approach, as it compels the model to follow the instructor’s rubric more closely. By contrast, the Rubric approach relies on the model’s own reasoning to interpret and apply the rubric’s instructions, without any assurance that it will do so faithfully.

The Bespoke approach entails additional (and potentially significant) costs for the instructor. Specifically, it requires more effort in prompt creation than Open or Rubric. The same prompt template can be used for all exams under the latter two approaches. However, because the number of topics and the points assigned to each topic typically vary across exam questions, the prompts and code for the Bespoke approach must be customized for each exam.

The third column of plots in Figure 2 above shows how grades obtained under Bespoke compare to instructor-assigned grades. Pearson correlations coefficients range from 0.80 to 0.92 (μ = 0.87). Where Bespoke yields better correlations than Rubric (Exams 1 and 3) the improvements are small (0.01 and 0.02), and, in the other two exams, Bespoke performs worse than Rubric. Figure 4 illustrates the grades obtained using the Bespoke approach (y-axis) against those obtained using the Rubric approach (x-axis). These plots confirm that the grades generated using each of these approaches are closely aligned, with Pearson correlations from 0.93 to 0.99.

Figure 4.

Comparison of performance between bespoke and rubric. Each scatterplot plots LLM-assigned grades using Bespoke against LLM-assigned grades using Rubric for one exam, with the dashed line indicating perfect agreement. Numbers in the upper-left corner of each panel report the Pearson correlation (r), Spearman correlation (ρ), and number of observations (N).

These results suggest that GPT-5 is able to implicitly implement the element-level scoring contained in a rubric without the additional assistance of a prompt that explicitly forces the model to output scores for each element. A more practical conclusion is that the more labor-intensive prompts and code required for the Bespoke approach are likely unnecessary for obtaining reliable scores from a state-of-the-art LLM.

4.2.2. The Pairwise Approach

Our second attempt to improve LLM grading relies on pairwise comparisons instead of grading on point scales. We use pairwise comparisons because, in other contexts, studies have found that they often produce more reliable ratings than numerical or absolute scales (such as Likert ratings). In areas ranging from coding of latent features of political texts (Carlson & Montgomery, 2017) to computerized tomography (CT) image-quality evaluation (Hoeijmakers et al., 2024), pairwise comparison tends to yield higher reliability and finer-grained distinctions than traditional scales. The likely reason for this advantage is that forcing direct comparative judgments reduces scale-interpretation variance and central-tendency bias. In addition, comparing items promotes effective discrimination, even when absolute standards are ambiguous.

However, pairwise comparison methods are analytically intensive, as the number of comparisons required grows quadratically with the number of total answers to grade. For instance, a class with 67 student exams, like Exam 3, requires 2,211 pairwise comparisons per question. ²⁰ This likely places the approach out of reach for users who intend to implement LLM grading manually via a chat interface, and it renders the approach more expensive by an order of magnitude for those who are able to implement it using an API. Another limitation of pairwise comparisons is their inability to deliver absolute scores anchored in a predefined scale (Hoeijmakers et al., 2024), although the output scores can simply be rescaled to the desired scale.

The fourth column of plots in Figure 2 shows how grades obtained under Pairwise compare to instructor-assigned grades. Like Bespoke, Pairwise offers at best marginal improvements over Rubric. The Pearson correlations coefficients range from 0.83 to 0.91 (μ = 0.86). However, its grade distribution differs markedly. Figure 5 plots the Pairwise total scores for each exam (y-axis) against the total scores obtained using Rubric (x-axis).

Figure 5.

Comparison of performance between Pairwise and Rubric. Each scatterplot plots LLM-assigned grades using Pairwise against LLM-assigned grades using Rubric for one exam, with the dashed line indicating perfect agreement. Numbers in the upper-left corner of each panel report the Pearson correlation (r), Spearman correlation (ρ), and number of observations (N).

Figure 5 further illustrates how grades generated under Pairwise diverge from those produced under Rubric. Although the Pairwise grades are less closely aligned with Rubric than Bespoke grades are, they remain more closely aligned with Rubric than with instructor-assigned grades. Taken together, these patterns suggest that all three approaches—Rubric, Bespoke, and Pairwise—are responding to similar features of the student answers, including components that are weighted differently or not fully reflected in human grading.

Overall, these results indicate that our more labor- and resource-intensive attempts to improve performance relative to Rubric did not significantly improve the grading performance of the LLM. In other words, the relatively low-effort and easy-to-implement Rubric approach already yields performance that appears difficult to meaningfully improve upon using simple prompting methods.

4.3. The Benchmark Question

Our results indicate substantial convergence between the LLM-generated exam scores and those assigned by the human instructor, with our preferred approach yielding Pearson correlation coefficients between 0.78 and 0.93. In other words, the LLM scores correlate at high-to-very-high levels with human scores, but not perfectly. What implications do these results have for the ability of LLMs to substitute for instructors in grading student exams?

An important consideration in attempting to answer this question is the appropriate benchmark against which to evaluate the accuracy of LLM-generated scores. As alluded to above, human grades are themselves imperfect signals of answer quality. One way to approximate the degree of noise in human-assigned scores is to examine variation in the grades assigned by a human grader to the same exam at different times, i.e., intra-coder variability. A natural benchmark for evaluating LLM grading performance is the range of disagreement in a human grader’s own intra-coder variability. ²¹

Measuring intra-coder variation involves substantial challenges. ²² In most cases, exams are graded only once. Regrading an exam is complicated by at least two possible measurement concerns. First, the instructor may remember their initial assessment. In that case, the measured intra-coder variation would be biased toward greater agreement. Second, a regrading exercise inherently occurs under lower stakes than the initial grading, possibly biasing the intra-coder variation toward disagreement. ²³

Despite these challenges, we attempted to obtain insights into this benchmark, that is, how the correlation between LLM-assigned and instructor-assigned scores, on one hand, compares to intra-coder variation, on the other. To do so, one of us regraded a portion of an exam included in this study (Exam 1, Question 2). Figure 6 presents these grades and compares them both to the original instructor-assigned grades and to the LLM-generated grades obtained under the Rubric approach. The two sets of instructor-assigned grades are more closely aligned with each other than either is with the LLM-generated grades, suggesting that, at least in this case, the divergence between LLM and instructor grading exceeds the level of intra-coder variation. We note, however, that the correlation between LLM-generated scores and human-generated scores in some other exams and questions—particularly those in Exam 2—surpasses the observed intra-coder variation for Exam 1, Question 2. ²⁴ This exercise suggests that LLM-human variation and intra-human variation are not significantly different.

Figure 6.

Comparison of grades obtained from two rounds of grading the second question of Exam 2 by the same instructor. The scatterplots plot LLM-assigned grades using Rubric against the instructor grade (col. 1); LLM-assigned grades using Rubric against the instructor regrade (col. 2); and the instructor regrade against the initial instructor grade (col. 3), in each case for Question 2 in Exam 1. The dashed line indicates perfect agreement. Numbers in the upper-left corner of each panel report the Pearson correlation (r), Spearman correlation (ρ), and number of observations (N).

4.4. Differences in Exam Structure and Rubric Detail

Law professors frequently use grading rubrics as an aid for grading essay questions. Rubrics help instructors grade essays consistently by identifying the issues that students should spot, applicable legal rules, and the elements of relevant legal analysis. While LLMs do not suffer from fatigue and distraction as instructors do, a grading rubric may serve as a kind of prompt engineering that increases the likelihood that the LLMs will provide credit for the proper responses. We thus began our research with the hypothesis that rubrics would improve the correlation between LLM grades and instructor grades.

Our results provide support for this hypothesis. Although our sample size of four exams is necessarily limited, greater rubric specificity meaningfully enhances the model’s ability to replicate human scores. The rubrics for these exams vary substantially in their level of detail, providing a useful basis for examining this relationship. For example, Exam 2 employed a highly granular rubric assigning binary (0/1) scores to each element, while Exam 3 relied solely on broad, undifferentiated question-level guidance. Exams 1 and 4 employed rubrics with a middle range of detail. Exam 2 produced the strongest correlation between human and LLM-assigned scores (0.93), whereas Exam 3 yielded the weakest (0.78). The rubrics for Exams 1 and 4 produced intermediate correlations (0.89 and 0.88). These patterns suggest that the precision and structure of grading rubrics may be an important determinant of how effectively LLMs are able to emulate human grading.

However, given our limited data, this remains a hypothesis and warrants further investigation. In future work, researchers could evaluate the impact of rubric design by varying the degree of detail of a rubric while holding the exam itself constant. We caution, however, that while increasing a rubric’s detail will likely improve the correlation between human graders and LLMs, more detailed rubrics will not necessarily lead to better grading in an objective sense. Highly specific rubrics may prevent the grader, whether human or LLM, from rewarding answers that are unusually creative and so are not anticipated in the rubric design.

5. Discussion and Implications

This study’s results indicate that currently available LLMs can replicate human grading of law school exams with a high degree of accuracy. When the prompt architecture included a detailed rubric that a human grader used in their process, OpenAI’s GPT-5 LLM achieved a Pearson correlation of up to 0.93. This is especially impressive given that even a human who regrades their own exam is unlikely to achieve a significantly higher correlation, as discussed above. In this Section, we discuss the generalizability of these findings, whether LLMs are poised to actually replace (or at least substitute for) human graders, and where future research might focus.

5.1. Scope and Generalizability

Our analysis is based on a limited number of exams, which naturally raises questions about the scope of our findings. As a starting point, we note that the courses included in the study—civil procedure, contracts, torts, and corporate law—share a common assessment format that is central to legal education: the issue-spotter exam. Such questions require students to identify legally relevant issues embedded in a factual scenario, apply doctrinal rules, and exercise judgment under uncertainty. This structure is characteristic of a wide range of law school courses beyond those we study, including criminal law, constitutional law, administrative law, and many upper-level doctrinal electives.

At the same time, we recognize that variation in the substantive content of law school courses may affect the ability of general-purpose LLMs to accurately evaluate exams in those courses. For example, it is possible that many AI models were trained on more text related to “core” law school classes (e.g., first-year courses and courses like corporate law or evidence) than to less common or more specialized courses (e.g., employment law or corporate tax). Similarly, some subject areas evolve more rapidly than others, which means that current LLMs, trained largely on older materials, may be comparatively more out of date in these domains than in areas governed by more settled law. Additionally, our findings may not extend as readily to courses with other assessment formats, such as policy-focused seminars, drafting-heavy courses, or clinics. We therefore view our results as evidence that LLM grading of issue spotter questions is feasible as a core component of law school assessment, and we encourage further research on the extent to which LLMs are compatible with other exam formats.

Another important question is what the observed correlations between LLM and instructor raw scores imply for correlations in final student grades, which are the outcomes that ultimately matter to students and institutions. Because grading regimes vary substantially across law schools, there is no single answer to this question. One dimension of complexity is whether grading serves the purpose of ranking students (an ordinal scheme), revealing their abilities against a benchmark (a cardinal scheme), or both. For example, employers might care about the relative ranking of law students, while also insisting that job applicants exceed an absolute skill threshold. Law schools typically accommodate these preferences by grading students on a curve and identifying those below the relevant skill threshold, often by reserving grades such as “F” to signal unacceptable performance or by denying the degree altogether. But they also may allow individual professors to depart from a curve when a class contains an unusually strong or weak cohort of students. Schools also differ along other dimensions in how instructors assign final grades–using different curves, different levels of granularity, different levels of compression, and so on.

These differences will inevitably affect how closely a given level of score agreement translates into agreement in final grades. Moreover, they could interact with the possibility that an LLM might grade more accurately at different levels of the score distribution. ²⁵ Against this background, Appendix C provides a rough illustration of how the observed score-level agreement would translate into letter grades under several alternative, stylized grading regimes. This analysis shows that even highly correlated scores can produce different grade outcomes depending on institutional grading rules. At the same time, it shows that for exams exhibiting the strongest score correlations, grade disagreement is limited under grading regimes commonly used in law schools. Taken together, these results suggest that the patterns documented in the main analysis are not only quantitatively strong, but also relevant for practical grading outcomes.

A final question, given the size of our sample, is whether the observed results might be driven by idiosyncratic features of the particular exams we study. The consistency of these patterns across the exams included in our analysis suggests that they are not purely exam-specific. This inference is further supported by the fact that different instructors wrote the exams, and, with one exception, they were designed and graded without the present study in mind. At the same time, the number of exams in this study does not permit a fully satisfactory resolution of this question using standard statistical techniques.

To provide additional evidence on this point, we examine whether the observed associations are robust at the level of individual exam questions. This analysis implicitly treats individual questions within an exam as independent units of observation—a simplifying assumption that is not literally true, but which allows us to assess whether the observed relationships are consistently present across different questions. Figure 3 above provides evidence on this point by reporting question-by-question correlations between human and LLM-generated scores for each prompting approach. As the figure shows, correlations are positive for every individual exam question across all four grading methods, indicating that the positive association documented in the aggregate results is not confined to a narrow subset of questions. In addition, Appendix D reports and discusses regression analyses that treat exam questions as independent clusters of observations. While these analyses are necessarily limited in what they can show, they provide further support for the conclusion that automated legal analysis grading is capable of extracting relevant evaluative information from law school issue-spotter exams more generally.

5.2. Machine Grading as a Substitute for Human Grading

Despite the impressive accuracy shown by LLMs in grading exams, instructors and legal education programs hoping to substitute human grading with machine grading are likely to face both ethical questions and institutional hurdles. For instance, even if machine-grading correlates highly with human grading, machines may make mistakes that human graders would not, resulting in some students receiving different (though not necessarily ‘wrong’) grades. Such mistakes may or may not be random; they might, for example, reflect biases in the data on which they were trained. For instance, although we found no evidence of this in our study, it is possible that LLMs could systematically penalize exams that use particular words or sentence structures associated with certain types of students or responses, even when those features are legally irrelevant and might not affect a human grader’s judgment (Barocas & Selbst, 2016).

More practically, students and prospective employers may place less trust in machine-generated grades than in human grades, even when the results are close to identical, reflecting the well-documented tendency of humans to undervalue output they believe to be generated by an LLM rather than a human (Harasta et al., 2024). In addition, some may argue that instructors are hired and paid in part to evaluate students, and thus it is ethically inappropriate for them to delegate that responsibility to machines. ²⁶ Those who advance this critique might emphasize that law students are typically required to complete examinations without AI assistance and, at least for the foreseeable future, will present arguments to human judges rather than automated systems.

Finally, there might be a tendency to demand perfect agreement between human and machine, based on the assumption that any deviation between the two necessarily reflects errors by the machine. As discussed in Sections 3.1 and 4.3 above, however, this assumption is unfounded. Given the well-documented limitations of human graders, it is plausible that at least some of the disagreement we observe is caused by human error. Thus, even as grading models continue to improve, perfect replication of human performance should not be the goal.

Indeed, human evaluation generally suffers from several limitations that may not affect machine grading. One is inconsistency: evaluators often assign different scores to similar answers, even when they use detailed rubrics (Liew & Tan, 2024; Yang et al., 2020). Scoring inconsistencies may be especially prevalent when grading is spread over long periods or conducted under varying conditions that lead to grader fatigue (Kumar & Boulanger, 2020). These conditions are hardly uncommon in the law school context, as law school essay grading is both labor-intensive and time-consuming. Grading fatigue can likely be mitigated through breaks, but not eliminated altogether. Law school instructors often grade a substantial number of exams, each consisting of many pages, a task that can take multiple days. In fact, this problem was evident within the data we used for this experiment. Upon a manual review of the individual exams with the largest disparities between the assigned and AI grades, we identified some instances in which a human grader appeared to have made straightforward grading errors. In several cases, the human grader appears to have awarded more points than was justified under their own rubric. Given that error and inconsistency are inherent shortcomings of all human grading, such grading is a highly imperfect proxy for evaluating the true capacity of LLM grading.

Moreover, even when grading under ideal conditions, well-intentioned and expert human graders bring prior experiences, expectations, and unconscious biases to their evaluations of student exams. Essay exam scores thus reflect not only the strength of the substantive arguments in the exam answer, but also the identity of the graders and their beliefs about the student who produced them. For example, experimental evidence shows that evaluators are prone to “halo bias,” where positive impressions in one domain—such as classroom performance or even grammar and style within the exam answer—result in higher scores, although they are ostensibly irrelevant under the formal grading criteria (Malouff et al., 2014). Other studies of admissions statements showed score disparities based on race, gender, and linguistic background (Breland et al., 1999), which were not fully explained by differences in writing proficiency.

Although blind-grading practices mitigate some of these biases, they do not eliminate them. One study of English as a Foreign Language (EFL) assessments found that even with blind grading and instructor training, human raters remained inconsistent, and their judgments varied systematically depending on the perceived linguistic and cultural identity of the student. In particular, the raters penalized deviations from native language use more harshly in students they presumed to be less proficient, even when the substantive content of the essays was otherwise strong (Schaefer, 2008).

Automated legal analysis scoring may be less susceptible to some of the kinds of errors and biases that affect human graders. This is almost certainly true for fatigue-related concerns: machines do not tire. ²⁷ It is also possible—though less certain—that LLMs could produce evaluations that are less biased and more consistent. Indeed, as others have argued, reducing human grading biases “such as rater fatigue, rater’s expertise, severity/leniency, scale shrinkage, stereotyping, Halo effect, rater drift, perception difference, and inconsistency” was one of the “key anticipated benefits” of using LLMs to grade essay-based exams (Kumar & Boulanger, 2020, at 2). Empirical evidence suggests that AES partially achieved these benefits, demonstrating greater consistency in some evaluation tasks than human scorers, including the reduction of interpersonal assessment bias (Attali & Burstein, 2006). Although our empirical design does not establish whether automated legal analysis scoring can achieve comparable reductions in human biases, the existing track record offers at least some reason for optimism. ²⁸

In weighing the relative strengths of LLM and human grading, another question is whether errors by LLMs and humans should be valued equally or whether mistakes made by AIs ought to be treated as more (or less) consequential. This debate parallels discussions about autonomous vehicles, where many argue that accidents caused by AI systems should carry greater weight than comparable accidents caused by human drivers (e.g., Krügel & Uhl, 2024; Geistfeld, 2017, at 1691-94). In this domain, some critics of autonomous vehicles point to mistakes made by those vehicles as a reason to prohibit them, while overlooking the proportionally larger number of equally catastrophic mistakes made by human drivers.

The debate over automated legal analysis scoring might follow a similar path. Key issues in evaluating this question—though ones we cannot resolve here—include difficult ethical considerations. For instance, if machines produce the same aggregate level of mistakes as humans, does it matter whether they make different mistakes (e.g., assigning student A a lower grade and student B a higher grade, when a human grader would have erred in the opposite direction)? Similarly, if either the human or LLM tends to produce a large number of small errors, how should that be compared to an LLM or human counterpart that tends to produce a small number of larger errors?

5.3. Alternative Uses of Automated Legal Analysis Scoring

Although our results speak most directly to the prospect of replacing human grading of law school exams with LLM grading, they also suggest alternative applications for automated legal analysis scoring, three of which we consider here. One possible application is that automated legal analysis scoring could supplement human grading. For example, divergences between LLM and human grading could trigger additional review by the human grader to reduce the incidence of errors. Of course, instructors should remain attentive to the potential bias introduced by reviewing only those student answers for which human and LLM grades diverge substantially. Even so, we expect that instructor–LLM grading discrepancies will often provide a valuable heuristic for identifying possible human grading error. ²⁹

A second application is to use automated legal analysis scoring to provide law students with individualized feedback on preliminary work product or mock exams. Evidence suggests that feedback can improve performance throughout law school, particularly for students who enter law school with lower academic indicators (Schwarcz & Farganis, 2017). But a well-known feature of legal education is that students, especially in their first year, often receive limited feedback on their exam writing. Offering written feedback on midterm exams or practice legal analysis questions is time-consuming and costly for instructors, and thus occurs less often than it should.

Providing feedback in law school courses is not merely desirable for pedagogical reasons—it has also become a matter of compliance with American Bar Association accreditation standards. In early 2025, the ABA adopted revisions requiring that “all courses in the first one-third of the credit hours earned by students in the JD program include at least one formative assessment that allows students to evaluate their performance relative to the learning outcomes in the course.” (American Bar, 2025). Many law professors have expressed concern that the mandate will impose substantial burdens on already-overextended faculty (Silverstein, 2024).

Our results suggest that automated legal analysis scoring could allow students to obtain reasonably accurate assessments of ungraded midterm exams or mock exam answers, and perhaps even meet new ABA requirements for formative assessments in the process. Our results also hint that LLM feedback could extend beyond overall quality assessments, offering insights into the strengths and weaknesses of student work, and suggesting concrete methods for improvement. The key question here—which we do not test in this paper but hope to examine in future research—is whether LLMs can generate accurate scores for individual elements within grading rubrics (as opposed to question-level scores), and do so reliably enough that those scores on individual elements can meaningfully guide students. An additional possibility, which likewise remains untested in this study, is that LLM systems could deliver helpful written feedback, explaining why particular scores were assigned and how students might improve their performance.

These potential methods for leveraging LLMs to improve law student writing could also extend beyond formal legal education into legal practice. As in law school, many junior lawyers receive limited feedback on their work from senior colleagues. Even when senior lawyers revise or rewrite junior lawyers’ work, the latter are often left uncertain about the quality of their initial draft and forced to guess which edits reflect substantive deficiencies versus the senior lawyer’s stylistic preferences. AI-generated feedback could both accelerate junior lawyers’ learning and free senior lawyers’ time. Although formal rubrics for evaluating the work of junior lawyers are not yet common, it is not difficult to envision law firms developing high-level criteria for certain types of tasks that could serve as the basis for such assessments.

A third implication of our results concerns the use of LLM-based grading strategies in developing LLM benchmarks. As noted earlier, as LLM systems become increasingly capable of performing discrete legal tasks, it is critical to measure performance reliably across tools and over time. Although a number of legal benchmarks exist, a persistent challenge has been how to score LLM outputs in response to legal questions. Some benchmarks attempt to avoid this problem by relying only on objective questions with unambiguous answers. But this approach necessarily narrows the scope of evaluation, as many important legal tasks do not have clear right and wrong answers (Fei et al., 2023). An alternative approach that has gained traction is to use LLMs themselves to score outputs (Fan et al., 2025; ValsAI, 2025). However, because there is limited evidence on the reliability of such assessments, this strategy has been criticized. Our results provide evidence that AI grading techniques may in fact be workable, particularly when LLMs are supplied with structured rubrics. This suggests that building robust legal LLM benchmarks may require not only curating a diverse set of legal analysis questions but also developing rubrics that can guide LLM systems in evaluating responses.

5.4. Future Research

Our research focused on demonstrating the practical feasibility of automated grading of law school exams and evaluating LLM performance relative to existing human grading. Several important issues remain for future work. Most notably, this study did not seek to optimize the degree to which AI-generated scores align with those of a particular human instructor, nor to determine how close automated grading can come to improved or less noisy reference benchmarks. Both questions represent important directions for further research.

With respect to optimizing alignment between human and LLM grading, this research offers several preliminary insights. It suggests that closer alignment is facilitated by the use of grading rubrics that are relatively specific and rely on a larger number of comparatively objective elements. We further expect that closer matching between human and LLM scoring can be achieved through few-shot prompting strategies that provide the model with multiple example scores generated by the human instructor. In practice, we anticipate that this approach will be both more effective and more feasible than alternatives such as fine-tuning foundation models.

Measuring the accuracy of LLM grading relative to a more objective benchmark is more challenging, in part because assessments of law school exam answers, like many forms of legal work product, necessarily involve a combination of both objective criteria and subjective judgment, and because no directly observable, error-free benchmark for student ability is available. In light of this, an important direction for future research is the construction of improved reference benchmarks against which LLM grading accuracy can be evaluated. One promising avenue would be to construct reference scores for individual exam answers based on independent evaluations by multiple instructors. The resulting average scores would likely provide a more reliable and less idiosyncratic measure of answer quality than any single instructor’s assessment. These consensus-based scores could then be used to improve AI grading through prompting strategies, fine-tuning, reinforcement learning, or related methods.

Another important direction for future research concerns optimizing AI grading to provide meaningful formative feedback to law students and junior lawyers. One concrete approach would be to test and refine the accuracy, or instructor alignment, of LLM-generated scores on individual rubric components rather than focusing solely on overall scores. Although we did not examine this question here, to the extent that AI systems can assign accurate or normatively appropriate scores to discrete rubric elements, they could play a substantial role in promoting learning. It is widely recognized that well-designed rubrics, when accurately applied, function as an important form of formative feedback by enabling students to identify specific aspects of their work that require improvement as well as areas in which they already meet or exceed expectations.

Beyond the scoring of exams, LLM grading can likely support formative feedback through developing and evaluating methods for providing qualitative feedback and concrete suggestions for improvement (e.g., Ouellette et al., 2026). A central challenge in this area lies in assessing the quality of LLM-generated qualitative feedback. One possible approach would mirror the methodology used here by comparing LLM-generated feedback with human-generated feedback and measuring their degree of similarity. That strategy, however, is ill-suited to this context, because there are many distinct ways to provide helpful qualitative feedback, making close matching an unreliable proxy for quality. An alternative approach would be to ask third parties, such as students, subject-matter experts, or even other AI systems, to blindly evaluate the helpfulness of qualitative feedback generated by humans and by LLMs. Implementing such a design, however, would require carefully masking the source of the feedback, which may itself be difficult. A more ambitious strategy would involve a randomized controlled experiment in which some students receive human-generated qualitative feedback while others receive LLM-generated feedback, followed by a comparison of learning outcomes across groups. Such a design would likely require sufficient sample sizes to detect potentially modest treatment effects in the presence of substantial outcome noise, and it would also raise nontrivial ethical concerns.

6. Conclusion

Our analysis suggests that LLMs have the capacity to roughly approximate the grading of a law professor, with the best results coming from prompts that incorporate detailed rubrics. With LLMs’ rapid improvement in text analysis tasks over just the last few years, we can probably expect continued improvement in the near future. Future analysis—including on different exams on different subjects—is likely to produce even greater agreement between humans and machines. Routine updates to this study will be necessary as LLMs evolve.

Even as LLMs increasingly approximate human graders’ performance, we acknowledge that logistical, institutional, and political challenges exist to replacing law professor grading with machine grading. For instance, some law schools have longstanding rules requiring that professors (rather than, say, teaching assistants) personally grade exams and assign grades. It is unclear if machine-graded exams would comply with such rules. But even if they would or if the rules are changed, other political forces and path dependence may prevent law schools from replacing human graders with LLMs for the foreseeable future.

Even so, our findings suggest that automated legal analysis scoring could be used effectively for other valuable grading tasks in the very near future. For instance, professors could use LLMs as a supplement to their own grading, reviewing their own evaluations to detect any errors and bias. In addition, students might use LLMs and instructor-supplied rubrics to receive feedback on their practice exams, thereby potentially meeting otherwise burdensome ABA requirements to provide formative assessments in all first-year law school classes.

Supplemental Material

Supplemental Material - Grading Machines: Can AI Exam-Grading Replace Law Professors?

Supplemental Material for Grading Machines: Can AI Exam-Grading Replace Law Professors? by Kevin L. Cope, Scott Hirst, Daniel Schwarcz, Jens Frankenreiter, Eric A. Posner, Dane Thorley in Journal of Law & Empirical Analysis.