Abstract
Expert judgement is inherently noisy, but the consequences of this variability are often underestimated. Here, I examine the U.K. Research Excellence Framework (REF) as a case study illustrating how judgement noise can interact with structural features of a multi-stage evaluation system. Simulations show that noise in both internal departmental reviews and REF panel reviews, in conjunction with the bounded REF rating scale, produces systematic underestimates of the highest-performing departments. Panel noise, rather than internal noise, had a disproportionate impact on department rankings, with even moderate noise producing severe distortions. The case of REF demonstrates that noise embedded in sequential evaluation, for example, in hiring and grant review decisions, does not “cancel out,” but instead can propagate in potentially surprising ways. At a policy level, these results underscore the need for a noise audit of REF panel ratings.
Introduction
Human judgement is the ultimate basis of institutional decision-making. A central concern of psychological science has been the reliability and validity of human judgement. Multiple systematic biases in decision-making (e.g., Tversky & Kahneman, 1973) have been identified, but less attention has been paid to the role of decision noise, or the unwanted variability of judgements (Kahneman et al., 2021). Whereas bias distorts judgement in a particular direction, noise produces inconsistency. One consequence is that two equally qualified and competent judges can give different evaluations of the same case. While noise, compared to bias, might seem relatively benign, tolerable, or even self-correcting with multiple observations, here I will use the U.K.’s Research Excellence Framework (REF) as a case study of how different sources of noise within institutional decision-making structures can distort real-world outcomes. My aim is to better understand REF as a measurement instrument, and in doing so, better understand how noise in judgement can shape outcomes in a complex, multi-stage evaluation system.
What Is REF?
The REF is the U.K.’s system for assessing the quality of research in its higher education institutions (HEIs). The REF operates on a cycle of about 7 years, and each department or “submitting unit” is assessed on its research outputs, impact case studies, and research environment during the cycle. Here, we focus on the evaluation of research outputs, which in the sciences primarily consist of peer-reviewed journal articles. Assessing the quality of outputs is a massive undertaking. In the most recent REF 2021 exercise, 1,878 departments across 34 research disciplines (or “units of assessment” [UOAs]) submitted for evaluation 185,594 outputs generated by 76,132 full-time equivalent (FTE) staff (Research Excellence Framework [REF], 2022a).
An explicit objective of the REF is to help determine how to allocate over £1 billion in annual quality-related research funding (REF, 2022d, 2025). Strong REF performance is therefore critical for the financial stability, research capability, and recruitment success of U.K. HEIs. The REF is also meant to provide accountability for public research investment and give insight into the “health” of research in U.K. HEIs. These important objectives require a transparent and reliable assessment process.
Human Judgement Is the Basis for REF Ratings
Fundamental to the REF is that outputs are rated by expert human reviewers. There is some controversy over this approach, as bibliometric approaches are inexpensive, transparent, and have been validated against human review (Baccini et al., 2020; Traag & Waltman, 2019). However, expert review, unlike bibliometrics, offers a direct and ungameable assessment of research quality: high quality research is research that experts judge to be high quality. Wilsdon (2015) led a review investigating metrics and expert judgement for research assessment in the United Kingdom, and warned against the dangers of purely metric approaches, instead arguing that the proper place of metrics was to support, not replace, judgement. Although expert judgement is therefore critical for proper assessment, the key problem, beyond the expense involved, is that human judgement is inherently noisy (Banks, 1970; Green & Swets, 1966; Hilbert, 2012), and evaluation systems must take this into account (Kahneman et al., 2021).
A noisy process is one that does not produce exactly the same results when repeated, for uncertain reasons. In the case of expert review of a scientific paper, we can identify multiple forms of noise. For example, a referee might produce different judgements of the same work on different occasions, perhaps because they’ve just eaten (Danziger et al., 2011). Kahneman et al. (2021) refer to this as “occasion noise,” which reflects the fact that to some uncertain extent, we don’t understand and aren’t able to control all the factors that might influence human judgement, like mood, fatigue, life events, and so on (Dror & Rosenthal, 2008; Mayer et al., 1992; Schwarz & Clore, 1983). Different referees reviewing the same paper might give different judgements because they have different standards, for example as to what qualifies as “internationally excellent.” In contrast, referees might agree in their judgement of a trait (e.g., experimental rigour), but differ in their weighting of that trait, and therefore in their final judgement of the paper. These are examples of what Kahneman et al. (2021) call “level noise” and “pattern noise,” respectively. When it is assessed, noise in all its forms is found to effect professional judgement in all sorts of domains (Allison et al, 2014; Cole et al., 1981; Dror & Cole, 2010; Pier et al., 2018; Simonsohn, 2007).
Unlike bias, which consistently skews outcomes in one direction, noise produces unpredictable deviations. Bias, in the statistical and everyday use of the word, is certainly a concern for the process of research review (Jappelli et al., 2017), and further, as Kahneman et al. (2021) note, bias in judgement is probably a better-known problem than noise. However, bias and noise are orthogonal components to error. Each can be reduced or amplified independently of the other, and reducing either by the same amount has the same effect on decision accuracy. Therefore, it is important to consider noise when thinking about how systems that rely on human judgement operate. Furthermore, while at first glance unbiased noise might seem unlikely to produce systematic errors, as we will see, the structure of the REF and its bounded ratings mean this view is mistaken.
The REF Consists of Two Sequential Reviews: “Internal” and “Panel”
Within the REF, the group responsible for assessing a discipline, or UOA, was tasked to use its collective judgement to grade submitted outputs on a scale of 0* to 4* for quality (REF, 2022a). I’ll call this process “panel review” and note that it could take various forms across UOAs. For example, in Psychology, Neuroscience, and Psychiatry (UOA 4), panel reviews were typically formed from the agreement of two experts (REF, 2022b). The labour and time required mean the panel can review only a fraction of all available research outputs. Departments must therefore create a submission representing their best work. In practice, departments first conduct an “internal review” of their full corpus of outputs to select the strongest possible submission. Figure 1 illustrates how each submitted output is reviewed twice: first by internal review of the department, and then by the REF panel review. Note this description simplifies the actual submission process in 2021, as in practice, there were factors besides quality to consider, for example, rules about the number of outputs that can or must be associated with each member of staff (REF, 2022e).
The “two-review” REF process.
This “two-review” process leads, inevitably, to disappointment for most departments, due to the well-known statistical phenomenon of regression to the mean. Regression to the mean occurs whenever there is less than perfect correlation between two measurements: extreme measurements on one set tend to be less extreme on the second set. In the REF context, the “regression” occurs because the highest scores from the internal review are likely to reflect both a genuine signal of good quality and a noise component that pushes the score higher rather than lower. These extreme scores form the submission. On the subsequent panel review, the genuine signal is the same, but the noise is less likely to produce the same extreme effect. The result is that high scores become less extreme on a second measure.
Here, I investigate the effects of decision noise within the structure of the two-review process. Regression to the mean, created by the combination of noisy human judgement and the sequential REF reviews, demonstrates that a department and the panel will have different views of the department’s quality. This observation motivates the simulation work presented here, investigating the separate influence of internal and panel review noise on the REF process.
The simulations address important questions about the REF I have not seen evaluated elsewhere: How accurately does the “two-review” REF process reflect the absolute ratings and relative rankings of departments? How is error affected by internal and panel noise, and do these different sources of noise have different effects? The simulations also demonstrate how unbiased noise can produce biased outcomes when the influence of noise is not carefully considered.
To foreshadow the two main conclusions: (a) unsystematic noise in the REF process produces systematic bias in department ratings, such that the best departments are downgraded the most; and (b) panel noise has a particular impact on relative department rankings. The results demonstrate that the U.K. research base requires a noise audit on the REF panel review.
Study 1. Noise Leads to Systematic Underestimation of Department Quality
The first simulation looks at how unbiased noise in the internal and panel review affects the rating of departments. All code and data are available on OSF https://osf.io/fwbqr/.
Methods
The simulations embed a simple rating process, described below, within the “two-review” structure of the REF (Figure 1). That structure defines the algorithm of the simulations. The input data is supplied with a second component, a fictitious UOA. The simulations are run repeatedly to generate a picture of the accuracy of REF on absolute and relative ratings within the fictitious UOA.
The Simulated Rating Process
Simulated ratings implement two core assumptions: first, that outputs have a true numeric quality; and second, that expert review is noisy. Each simulated output has a genuine quality from 0 to 4 inclusive, representing the 0* to 4* REF scale. I assume this score to represent a continuous latent variable of quality, which is censored left and right. Genuine quality is not directly observable but estimated through review. The judged quality of an output, or its rating, is simply the sum of its genuine quality and an unbiased error term. The error reflects judgement noise and is drawn from a normal distribution with mean 0, and a standard deviation reflecting the noisiness of the review. The simulations use this simple formula while independently varying the level of noise in the internal and panel review.
Importantly, the ratings formula abstracts over differences in review practice. REF panel reviews are typically formed from the agreement of two experts, but further reviewers can be brought in. Internal review processes will vary widely. One department might allocate each paper to multiple, independent external and internal reviewers, and then use a process of Bayesian aggregation to determine its internal ratings. A different department might simply ask authors for their own estimate of quality and leave it at that. Many departments will use a variety of review methods attempting to maximise return on review investment. For purposes here, different review practices are not modelled directly but are assumed to affect the noise term: best practice would reduce noise, poor practice increase noise.
Figure 2 shows how different levels of noise affect the ratings of outputs with genuine 3* and 4* quality in these simulations. It is already evident that unbiased noise, in combination with assumptions about output quality, is producing systematic distortions. Noise added to measurements on an unbounded scale, like temperature, doesn’t change the mean value of measurements, only their variability. But in the REF, output quality is censored, or capped, at 0 and 4. For outputs with a genuine quality of 4, noise can only reduce and never increase the judged rating. This is true to a lesser extent for genuine quality 3. The first concern is therefore that increasing noise not only increases the variability of ratings, but also lowers the mean ratings of high-quality outputs. We will see the extent of this effect in the simulations.
The effect of adding judgement noise with mean 0 (unbiased) and the indicated standard deviation to outputs with a genuine quality of 3 or 4. Noise increases variability, but given that ratings are capped at 4, noise also reduces the mean rating at these qualities.
Finally, a second and possibly insidious concern is just how reasonable the distributions of noisy judgements in Figure 2 appear. We will see that these levels of noise produce important effects in the REF simulations, yet it would be difficult or impossible to determine from the distributions alone any cause for concern.
The Fictitious UOA
A fictitious UOA was created to provide the input data for the simulations, based on the observed output ratings for UOA 4 (Psychology, Psychiatry, and Neuroscience) in the 2021 REF (2022c). Each of 93 fictitious departments was assigned frequencies of different output qualities to generate its corpus. For example, a specific department might have 20% of its outputs with genuine 4* quality; 30% 3*; 40% 2*; and 10% 1* and below. To further the plausibility and validity of the simulations, the frequencies of quality for the 93 departments were based on the observed 2021 REF ratings for UOA 4. The observed REF ratings are based on the department submissions, and these submissions are meant to reflect the best of each department. For example, a department whose observed REF results were exclusively in the 3* and 4* range still produced 2* papers, they were just not submitted. The observed REF results for a department therefore overestimate the quality of the underlying full corpus. To compensate, in our fictitious departments the rates of 3* and 4* outputs in the corpus were reduced relative to the 2021 REF (by 50% and 75%, respectively), and the rate increased for 2* (doubled) and below (a low minimum). Using the compensated corpus distributions and “lower” internal and panel noise, the correlation of grade point average (GPA) for departments and observed REF results was r(92) = .9. The aim was not to simulate real departments, but to create an input data source with a range of departments that capture a range of plausible, genuine qualities
A Single Simulation
The simulations examined the effects of four parameters, each with two levels (“lower” and “higher”): internal noise (0.5 or 1.25), external noise (0.5 or 1.25), FTE (25 or 125), and productivity (5 or 10). A single simulation run was structured in line with the process illustrated in Figure 1.
Values for the four simulation parameters were fixed for the run and used for all departments.
A corpus of outputs was generated for each department. The genuine quality of outputs was determined by the rates of quality assigned to each department within the fictitious UOA (Figure 1A). The size of each corpus was the product of the FTE and productivity parameters.
Each corpus was passed through the “internal review” stage (Figure 1B). For each output in the corpus, the genuine quality was added to noise drawn from a normal distribution with mean 0 and standard deviation equal to the internal noise parameter. Ratings were constrained between 0 and 4 inclusive.
A submission was created for each department, made from the outputs with the highest internal ratings (Figure 1C). In line with REF 2021 rules, the number of outputs in the submission was 2.5 × FTE. Outputs were not “assigned” to individual staff as in the 2021 rules.
The department submissions were then passed through “panel review” (Figure 1D). The panel review was identical to the internal review (step 3), except that the panel noise parameter was used instead of internal noise.
The average quality of the submission, usually referred to as GPA, was calculated for each department, in two ways. The observed GPA for a submission was the mean of its panel ratings. This reflects both noise from the internal review, which determined the submission, and from the panel review, which produced the final ratings. The genuine GPA was what a department would receive in a world free of noise: the average of the genuinely best outputs in the department corpus (the number of outputs equal to the required submission size). Error for a department was its observed GPA minus its genuine GPA, meaning that underestimates by the panel show as negatively-signed errors, and overestimates as positively-signed.
Each combination of parameters was run 100 times, for a total of 148,800 simulated submissions (100 repetitions × 24 conditions × 93 departments).
Results and Discussion: Noise on the Bounded REF Scale Penalises the Best
Figure 3 shows the effects of internal and panel noise, averaging over FTE and productivity factors. Overall, error is negative, meaning that observed GPA underestimates genuine GPA. Furthermore, the magnitude of error increases with the genuine quality of departments: the better the department, the greater the underestimate. As highlighted in discussion of Figure 2, this systematic error is inherent to the REF cap on output quality, and particularly the effects of the cap on panel review. For an output of genuine 4* quality, panel noise can only move the rating down. The more genuine 4* outputs a department has (and to a lesser extent 3*), the greater this ceiling effect will be, and the greater the underestimate of genuine quality. The bias against quality is structural and will apply to any department, with the magnitude increasing with the proportion of high-quality (3*/4*) outputs. The result demonstrates how bias can emerge within a system where there is no bias in judgement.
Error (observed–genuine GPA) shows how the REF process underestimates quality as a function of department genuine quality. This error increases in magnitude with both internal and panel noise.
Linear mixed-effects regression confirmed the pattern in Figure 3: the magnitude of error (i.e., the underestimation of quality) became more pronounced with higher true quality (β = −.66), greater internal noise (β = −0.26), and greater panel noise (β = −.15), all p < .00001. FTE and productivity were included to support the generation of a realistic departmental corpora and showed minimal practical impact on error. Although statistically significant (FTE: β = .000055; productivity: β = .017), their maximum effects across the observed ranges were just .0055 and .087, respectively.
Implications of a Bias Against Quality
One might reasonably claim that problems resulting from noise in the internal review belong to the departments doing those reviews. But noise in the panel review is a property of the REF itself. Because panel noise lowers the assessed quality of the U.K.’s best research and best departments, it works against an accurate assessment of two of the REF’s stated objectives: providing a public accounting for research investment, and monitoring the health of U.K. research. The good news is that the reality of U.K. research excellence will be better than what is suggested by its official, dedicated, and expensive assessment. The bad news is that without knowing the level of panel noise, no one can say by how much.
Perhaps most significantly, the absence of a clear understanding of how noise is distorting research quality ratings has the potential to undermine the final stated objective of REF, the allocation of block-based QR funding across institutions. To the extent that funding is allocated on absolute quality thresholds (e.g., the percentage of 4* outputs), then we have a problem as REF underestimates absolute quality. To the extent funding is allocated on relative quality (e.g., through an attempt to balance funding for a similar proportion of the best departments across UOAs), then a bias against quality might seem less of a concern, something that would only compress department differences at the top end of the UOA, without affecting their ranking. However, we should look more closely before concluding relative ranking is not a problem.
Study 2. Panel Noise Has Disproportionate Impact on Department Rankings
Here I examine how the relative rankings of departments are affected by internal and panel noise. The same simulation data from study 1 were used to determine the relative ranking of departments. The rank of each department within the fictitious UOA was determined separately for each simulation. Figure 4 shows distributions of department rankings for different levels of internal and panel noise, averaging over the effects of productivity and FTE. Each row represents a single department and shows the distribution of rankings that department received across simulations. The departments are ordered along the y-axis by their genuine quality.
Each row illustrates the distribution of a department’s rankings across the 100 simulations. The y-axis is ordered by the department’s ranking based on genuine quality in all simulations. The red line indicates where the department would rank in the absence of noise. Panel noise has a bigger effect on ranking variability than internal noise. For clarity, only the first 45 of the 93 departments of the fictitious UOA are shown.
The striking aspect of Figure 4 is the asymmetric effect of noise. Panel noise (the difference between the two rows of Figure 4) has a visibly larger effect on ranking variability than internal noise (the difference between the two columns of Figure 4). The asymmetric impact illustrates that the two-stage REF process is not a linear system. That is, the impact of internal and panel noise on ranking is not simply their sum. Bayesian regression predicting the standard deviation of department rankings as a function of internal noise, panel noise, and their interaction confirmed that panel noise had a disproportionate influence compared to internal noise, roughly five times larger (Figure 5). Explanatory power of the model as measured by Bayesian R² (Gelman et al., 2019) was .95 (95% CI [0.947, 0.952]).
Estimates of factors predicting the standard deviation of rankings illustrated in Figure 4, verifying the asymmetrical impact of panel noise compared to internal noise.
Multiple factors contribute to the asymmetry of noise effects. First, asymmetries will relate to the different distributions on which the two review stages are operating. The internal review looks at the department’s full corpus, that is, the entirety of outputs generated by the department. The full corpus has relatively large variation in genuine quality, from a department’s very best to its worst. In contrast, the submission is created from the internal review and reflects (noisily) the department’s best outputs. The submission, which is reviewed by the panel, is therefore much more restricted in its range of quality, generally reflecting the best, not the worst, of the department. If all is working as intended, internal noise is therefore added to the relatively high variance of the full corpus (from which the submission is drawn), while panel noise is added to the relatively low variance of the submission itself. Adding noise to a low variance distribution will distort the data more than the same degree of a noise added to a high variance distribution. That is, panel noise operating on the submission (with its relatively low variance in genuine quality), will constitute a greater proportion of the total variance in ratings than equal internal noise operating on the corpus (with its higher variance in genuine quality).
Another asymmetry is that noise within the submission is restricted by the dependency between genuine quality and internal noise. For example, in our “higher noise” simulations, the internal noise component for the full corpus is drawn from a normal distribution with mean 0 and SD 1.25. However, if we isolate the internal noise term for the submission rather than the corpus, it has a positive mean, inflating rather than deflating scores, and SD less than 1.25, as it reflects only generally positive scores (in fact, in this condition, the mean for internal noise on the submission was 1.1 with SD 0.86). In contrast, for the panel review of the submission, there is no dependency between genuine scores and panel noise, and the panel noise component for the submission approximates mean zero and SD 1.25. This difference in noise distribution is another way to describe the basis for regression to the mean. But it shows how, even when internal and panel noise are drawn from the same distribution, their effects are not the same.
Finally, the two-review process creates structural bases for the asymmetry in noise effects. Noise in the internal review affects which outputs are submitted, and therefore how closely the submission reflects the outputs of the highest genuine quality. The impact of internal noise depends in part on the distribution of quality within the corpus. For an extreme example, consider a hypothetical case where the entire corpus of a department had the same genuine quality – all 3*. Noise in the internal review would have no effect on the genuine quality of the submission, it would still be all 3*. Similarly, to the extent that the corpus has an excess of high-quality outputs, above what the submission requires, then internal noise may have relatively minor effects, for example, one 3* paper may be submitted in place of a similar other. As the variation of genuine quality within the corpus increases, the possibility that internal noise will disrupt the best submission likewise increases. Noise in the panel review, however, directly affects variability in observed GPA, and will always affect department rankings.
Whatever the causes, the variations in ranking seen in Figure 4 show that rankings are unstable, especially with higher panel noise. Distributions of rankings are wide and shallow, and it is easy to find egregious levels of uncertainty. For example, with higher panel and lower internal noise, the department with the genuine ranking of 26 is likely to be ranked somewhere between the top 5 and the bottom half of the UOA (Figure 4, lower left panel).
General Discussion: The Need to Assess Noise Within the REF Panel Review
The simulations lead to two main findings. First, the REF process systematically underestimates genuine department quality, with underestimation increasing for higher-performing departments. Second, panel noise exerts a much stronger influence on department rankings than internal noise. While internal noise may reasonably be considered a departmental concern, noise in panel reviews represents a systemic issue for the REF as a whole.
The effects of panel noise on rankings are important, even though the REF itself does not publish or endorse league tables. First, REF rankings need to be understood because they are routinely produced and circulated by leading sector outlets (e.g., Times Higher Education, 2022) and are readily available to stakeholders. For example, university leadership may feel it is reasonable to look to national ranking data to establish institutional priorities. Second, a plausible reaction to the results of Simulation 1, showing the REF is biased against its best departments, might be to seek comfort in the possibility that the REF preserves the relative standing of its departments. The results of Simulation 2 show this is an unsafe position. Third, REF guidance does indeed emphasise quality profiles on the grounds they arguably capture “pockets of excellence” (REF, 2022e). However, surely it is not sound reasoning to emphasise profiles simply because current panel-noise levels are unknown and may make rankings practically unstable; the remedy is to measure that uncertainty.
If estimates of panel noise were available, some distortions could be addressed. For example, corrections for regression to the mean could be undertaken if the correlation between internal and panel ratings were known. But panel noise is an unknown. It may be that panel noise is relatively low and all is well. Or it may be that panel noise is high enough to produce effects like those in Figure 4. Policy makers, funders, the departments and universities themselves, and the public using the results currently have literally no way of knowing. The inescapable conclusion is that the REF panel review requires a noise audit.
Kahneman et al. (2021) define a noise audit as a systematic measurement of the disagreement among professionals judging the same cases. They are clear that there is no shame in revealing disagreement, which is inherent in human judgement, including domains with objective standards, like fingerprint analysis (Dror & Rosenthal, 2008). The purpose of the audit is to measure and understand the degree and effects of noise. Kahneman et al. (2021) note that the most useful data for a noise audit are cases where multiple experts make judgements to the same cases. The independent ratings of two expert reviewers for each output in the REF is an example of just such data. This data allows for the analysis of consistency in judgements, correlations between experts, and identification of different types of noise: “level noise” arises from differences in judgements between experts, and “pattern noise” reflects variability in how individual cases are evaluated. A third category of “occasion noise” is not assessable, as it refers to inconsistencies in how the same experts judge the same cases on different occasions.
However, for good reasons, the REF will not make public the ratings given to specific outputs, nor the judgements of individual panel members. This kind of data is highly sensitive and could break careers or damage the integrity of the process. For example, it might only take a couple of low REF scores, publicly declared, to destroy an early researcher’s career. Or a panel reviewer might find it difficult to give honest evaluations of their colleagues’ papers, knowing their judgements would be public knowledge.
Fortunately, a useful noise audit does not require this level of identification or granularity. Imagine if, as part of an accountability process, the REF published a spreadsheet with 2 columns and 185,000 rows (i.e., 1 for each reviewed output), showing the ratings of 2 unknown reviewers to each of the 185,000 unspecified submitted outputs. From this bare minimum, an overall correlation between reviews could be calculated, and the overall noise component of judgements. One could safely go further, tagging reviews by UOA, and compute reviewer correlation and the degree of level noise within each UOA. There is guidance on best practices for data anonymisation, both from academic (e.g., Elliot et al., 2016; Esayas, 2015; Mackey, 2020) and governmental sources (e.g., Information Commissioner’s Office, 2012). From safely anonymised data, the research community could begin to understand the reliability of REF ratings and rankings within and across UOAs.
In the absence of any data from the REF, we cannot assess the level of noise in the panel review. However, previous analyses of inter-rater reliability in scientific review are not comforting. Meta-analysis of peer-review from Bornmann (2010), covering a total of 19,443 manuscripts, found only low levels of inter-rater reliability. From a comprehensive analysis of grant reviews for the Australian Research Council, Marsh et al. (2008) estimated that 6 reviews per proposal were needed for adequate reliability. Of course, the REF is a somewhat different undertaking than grant and manuscript review, and there may be mitigating factors: the expectations, demands, and experience of reviewers for the REF panel are high. But again, we simply don’t know.
If, after conducting the audit, noise was found to be uncomfortably high, corrective measures could be considered. Simulation and analysis could investigate whether the REF process might be improved by redistributing the expert panel workforce. For example, one way to reduce panel noise is to increase the number of independent reviewers. But to do so while maintaining panel reviewer workload, the number of submitted outputs would need to be decreased. Would lower noise on fewer submissions produce more reliable and useful ratings? Perhaps or perhaps not, but the larger point is that although noise is a problem that can be addressed, the solutions depend on understanding the current noise in the system.
At present, these prospects are for nothing as no noise audit is possible. The REF destroys all data relating to reviews, as confirmed by a Freedom of Information request I filed (REF Information Rights Team, personal communication, 2023). As I’ve tried to illustrate, this policy is overly conservative in terms of protecting reviewers and researchers yet is fatal to any prospect of a noise audit and understanding whether the current process is fit for purpose. The U.K. framework for research assessment has multiple important obligations. It is obliged to protect its reviewers and its researchers, but it is also obliged to demonstrate in a transparent way that its review processes are trustworthy and meet their objectives. A reasonably constructed noise audit can achieve both obligations and would represent a valuable addition to future REF exercises.
Looking ahead to REF 2029, the case for a noise audit strengthens. Current guidance suggests outputs will remain the largest single component for outcomes, while committing to transparency in criteria, procedures, and published outcomes (REF, 2025). These priorities create a practical route to implement a safely anonymised noise audit. This would permit sector-wide reliability estimates without compromising confidentiality, align with the transparency agenda, and help institutions calibrate their internal processes. Appropriately anonymised review data should be both possible and valuable under the new framework.
I have focused on the case study of the U.K. REF, but the approach here is relevant to other multi-stage evaluation systems, where an initial screening is made prior to a final quality evaluation. For example, in so-called triage models, a small subset of panellists conduct detailed evaluation and then present to a larger committee for full decision. A related but different structure applies to many hiring decisions, in which an initial panel screens applicants to produce a shortlist, which is then reviewed in detail. And of course, the REF is an important model for other nations, like Australia and China, developing their systems of research assessment. The simulation approach developed here can be adapted to evaluate and experiment with formalised decision frameworks, such as linear sequential unmasking-expanded (Dror & Kukucka, 2021), which seeks to reduce noise and bias in forensic decision-making by sequencing information release, prioritising evidence before contextual information. The arising issue in all these cases is that noise in these systems is not merely be a nuisance, or something that might “cancel out.” It is a potential source of structural distortion, and addressing such distortions requires noise to be measured and understood at multiple levels of evaluation.
Footnotes
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.