Tens of thousands of scientists contribute to peer review as journal editors and reviewers of the millions of manuscripts submitted every year. How do they decide what is quality work? What values do they apply in evaluating which science merits publication and which does not? How do they respond to dissensus and uncertainty? Who has the greatest influence over the final outcome? This study combines close reading with large language models to analyze 80,000 reviews of 28,000 accepted and rejected manuscripts in engineering and the life sciences. By following reviewers’ value judgments and editorial decisions, we come to a different view of how epistemic cultures are practiced in journal science. Instead of a consensual dialogue revealing salient norms, we find reviewers differently weigh (“commensurate”) their judgments to attribute value to works. Their pluralistic viewpoints elevate uncertainty about the work, and editors respond by aligning with the most negative of reviewers. Surprisingly, we observe engineers and life scientists find the same epistemic criteria are salient, valued, and influential, with novelty and accuracy being primary. These results underscore how contingency and uncertainty are structural features of STEM peer review and essential to its effectiveness and legitimacy.
BakanicVonMcPhailClarkSimonRita J.1987. “The Manuscript Review and Decision-Making Process.” American Sociological Review52(5):631–42 (https://doi.org/10.2307/2095599).
2.
BakanicVonMcPhailClarkSimonRita J.1989. “Mixed Messages: Referees’ Comments on the Manuscripts They Review.” The Sociological Quarterly30(4):639–54 (https://doi.org/10.1111/j.1533-8525.1989.tb01540.x).
3.
BarlösiusEva. 2019. “Concepts of Originality in the Natural Science, Medical, and Engineering Disciplines: An Analysis of Research Proposals.” Science, Technology, & Human Values44(6):915–37 (https://doi.org/10.1177/0162243918808370).
4.
BeckerHoward S.2008. Art Worlds: Updated and Expanded. Berkeley: University of California Press.
5.
BordageGeorges. 2001. “Reasons Reviewers Reject and Accept Manuscripts: The Strengths and Weaknesses in Medical Education Reports.” Academic Medicine76(9):889–96.
6.
BornmannLutzDanielHans-Dieter. 2010. “The Manuscript Reviewing Process: Empirical Research on Review Requests, Review Sequences, and Decision Rules in Peer Review.” Library & Information Science Research32(1):5–12.
7.
BornmannLutzMutzRüdigerDanielHans-Dieter. 2010. “A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants.” PLoS ONE5(12):e14331 (https://doi.org/10.1371/journal.pone.0014331).
8.
BourdieuPierre. 1974. “Cultural Reproduction and Social Reproduction.” Pp. 56–68 in Knowledge, Education and Cultural Change: Papers in the Sociology of Education, edited by BrownR.London, UK: Tavistock Publications.
9.
BourdieuPierre. 1988. Homo Academicus. Stanford, CA: Stanford University Press.
CarsonRichard T.Graff ZivinJoshuaLouviereJordan J.SadoffSallyShraderJeffrey G.2022. “The Risk of Caution: Evidence from an Experiment.” Management Science68(12):9042–60 (https://doi.org/10.1287/mnsc.2021.4292).
12.
CetinaKarin Knorr. 1999. Epistemic Cultures: How the Sciences Make Knowledge. Cambridge, MA: Harvard University Press.
ChengMengjieScott SmithDanielRenXiangCaoHanchengSmithSanneMcFarlandDaniel A.2023. “How New Ideas Diffuse in Science.” American Sociological Review88(3):522–61 (https://doi.org/10.1177/00031224231166955).
16.
ChubinDaryl E.HackettEdward J.1990. Peerless Science: Peer Review and U.S. Science Policy. Albany: State University of New York Press
17.
CicchettiDomenic V.1991. “The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation.” Behavioral and Brain Sciences14(1):119–35 (https://doi.org/10.1017/S0140525X00065675).
CraneDiana. 1967. “The Gatekeepers of Science: Some Factors Affecting the Selection of Articles for Scientific Journals.” The American Sociologist2(4):195–201.
20.
DahlanderLinusThomasArneWallinMartin W.ÅngströmRebecka C.2023. “Blinded by the Person? Experimental Evidence from Idea Evaluation.” Strategic Management Journal44(10):2443–59 (https://doi.org/10.1002/smj.3501).
21.
DemarestBradfordFreemanGuoSugimotoCassidy R.2014. “The Reviewer in the Mirror: Examining Gendered and Ethnicized Notions of Reciprocity in Peer Review.” Scientometrics101(1):717–35 (https://doi.org/10.1007/s11192-014-1354-z).
22.
DevlinJacobChangMing-WeiLeeKentonToutanovaKristina. 2019. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” Pp. 4171–86 in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1 (Long and Short Papers), edited by BursteinJ.DoranC.SolorioT.Minneapolis, MN: Association for Computational Linguistics.
23.
DruckmanJames N.2022. “Threats to Science: Politicization, Misinformation, and Inequalities.” ANNALS of the American Academy of Political and Social Science700(1):8–24 (https://doi.org/10.1177/00027162221095431).
24.
EaglyAlice H.KarauSteven J.2002. “Role Congruity Theory of Prejudice toward Female Leaders.” Psychological Review109(3):573–98 (https://doi.org/10.1037/0033-295X.109.3.573).
25.
EaglyAlice H.MakhijaniMona G.KlonskyBruce G.1992. “Gender and the Evaluation of Leaders: A Meta-Analysis.” Psychological Bulletin111(1):3–22 (https://doi.org/10.1037/0033-2909.111.1.3).
FlaminioSquazzoniGiangiacomoBravoMikeFarjamAnaMarusicBaharMehmaniMichaelWillisAliaksandrBirukouPierpaoloDondioFranciscoGrimaldo. 2022. “Peer Review and Gender Bias: A Study on 145 Scholarly Journals.” Science Advances7(2):eabd0299 (https://doi.org/10.1126/sciadv.abd0299).
28.
FleckLudwik. 1979. Genesis and Development of a Scientific Fact. Chicago: The University of Chicago Press.
29.
FoxCharles W.MeyerJenniferAiméEmilie. 2023. “Double-Blind Peer Review Affects Reviewer Ratings and Editor Decisions at an Ecology Journal.” Functional Ecology37(5):1144–57 (https://doi.org/10.1111/1365-2435.14259).
30.
GalloStephen A.SullivanJoanne H.GlissonScott R.2016. “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE11(10):e0165147.
31.
GlontiKetevanCauchiDanielCoboErikBoutronIsabelleMoherDavidHrenDarko. 2019. “A Scoping Review on the Roles and Tasks of Peer Reviewers in the Manuscript Review Process in Biomedical Journals.” BMC Medicine17(1):118 (https://doi.org/10.1186/s12916-019-1347-0).
32.
GroverAdityaLeskovecJure. 2016. “Node2vec: Scalable Feature Learning for Networks.” Pp. 855–64 in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (https://doi.org/10.1145/2939672.2939754).
33.
GuetzkowJoshuaLamontMichèleMallardGrégoire. 2004. “What Is Originality in the Humanities and the Social Sciences?”American Sociological Review69(2):190–212 (https://doi.org/10.1177/000312240406900203).
HugSven E.OchsnerMichael. 2022. “Do Peers Share the Same Criteria for Assessing Grant Applications?”Research Evaluation31(1):104–17 (https://doi.org/10.1093/reseval/rvab034).
39.
IezzoniLisa I.2018. “Explicit Disability Bias in Peer Review.” Medical Care56(4):277–78.
40.
JayasingheUpali W.MarshHerbert W.BondNigel. 2003. “A Multilevel Cross-Classified Modelling Approach to Peer Review of Grant Proposals: The Effects of Assessor and Researcher Attributes on Assessor Ratings.” Journal of the Royal Statistical Society: Series A (Statistics in Society)166(3):279–300 (https://doi.org/10.1111/1467-985X.00278).
41.
JohnsonCathrynDowdTimothy J.RidgewayCecilia L.2006. “Legitimacy as a Social Process.” Annual Review of Sociology32:53–78.
42.
KaganJerome. 2009. The Three Cultures: Natural Sciences, Social Sciences, and the Humanities in the 21st Century. Cambridge, UK: Cambridge University Press.
43.
KennardNeha NayakO’GormanTimSharmaAkshayBagchiChhandakClintonMatthewYelugamPranay KumarDasRajarshiZamaniHamedMcCallumAndrew. 2021. “DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions” (https://doi.org/10.48550/arxiv.2110.08520).
44.
Kern-GoldbergerAdina R.JamesRichardBerghellaVincenzoMillerEmily S.2022. “The Impact of Double-Blind Peer Review on Gender Bias in Scientific Publishing: A Systematic Review.” American Journal of Obstetrics & Gynecology227(1):43–50.e4 (https://doi.org/10.1016/j.ajog.2022.01.030).
45.
KriegerJoshuaMyersKyleSternAriel Dora. 2023. “How Important Is Editorial Gatekeeping? Evidence from Top Biomedical Journals.” Harvard Business School Entrepreneurial Management, Working Paper No. 22-011 (https://dx.doi.org/10.2139/ssrn.3882949).
46.
KuhnThomas S.1977. “Objectivity, Value Judgement, and Theory Choice.” Pp. 320–39 in The Essential Tension: Selected Studies in Scientific Tradition and Change. Chicago: University of Chicago Press.
47.
KuhnThomas S.1996. The Structure of Scientific Revolutions. Chicago: The University of Chicago Press.
48.
LamontMichèle. 2015. How Professors Think: Inside the Curious World of Academic Judgment. Cambridge, MA: Harvard University Press.
LamontMichèleHuutoniemiKatri. 2011. “Comparing Customary Rules of Fairness: Evaluative Practices in Various Types of Peer Review Panels.” Pp. 209–32 in Social Knowledge in the Making, edited by CamicC.GrossN.LamontM.Chicago: The University of Chicago Press.
51.
LamontMichèleMallardGrégoire. 2005. “Peer Evaluation in the Social Sciences and the Humanities Compared: The United States, the United Kingdom, and France.” Report prepared for the Social Sciences and Humanities Research Council of Canada.
52.
LaneJacqueline N.TeplitskiyMishaGrayGaryRanuHardeepMeniettiMichaelGuinanEva C.LakhaniKarim R.2022. “Conservatism Gets Funded? A Field Experiment on the Role of Negative Information in Novel Project Evaluation.” Management Science68(6):4478–95 (https://doi.org/10.1287/mnsc.2021.4107).
53.
LangfeldtLiv. 2004. “Expert Panels Evaluating Research: Decision-Making and Sources of Bias.” Research Evaluation13(1):51–62.
54.
LatourBruno. 1987. Science in Action: How to Follow Scientists and Engineers through Society. Cambridge, MA: Harvard University Press.
55.
LaudanLarry. 1984. Science and Values: The Aims of Science and Their Role in Scientific Debate. Berkeley: University of California Press.
56.
LeeCarole J.2012. “A Kuhnian Critique of Psychometric Research on Peer Review.” Philosophy of Science79(5):859–70.
57.
LeeCarole J.2015. “Commensuration Bias in Peer Review.” Philosophy of Science82(5):1272–83 (https://doi.org/10.1086/683652).
58.
LeeCarole J.SugimotoCassidy R.ZhangGuoCroninBlaise. 2013. “Bias in Peer Review.” Journal of the American Society for Information Science and Technology64(1):2–17 (https://doi.org/10.1002/asi.22784).
59.
LinJialiangSongJiaxinZhouZhangpingChenYidongShi.Xiaodong2023. “Automated Scholarly Paper Review: Concepts, Technologies, and Challenges.” Information Fusion98:101830 (https://doi.org/10.1016/j.inffus.2023.101830).
60.
LiuYinhanOttMyleGoyalNamanDuJingfeiJoshiMandarChenDanqiLevyOmerLewisMikeZettlemoyerLukeStoyanov.Veselin2019. “RoBERTa: A Robustly Optimized BERT Pretraining Approach” (https://doi.org/10.48550/arXiv.1907.11692).
61.
LonginoHelen. 1983. “Beyond ‘Bad Science’: Skeptical Reflections on the Value-Freedom of Scientific Inquiry.” Science, Technology, & Human Values8(1):7–17.
62.
LuukkonenTerttu. 2012. “Conservatism and Risk-Taking in Peer Review: Emerging ERC Practices.” Research Evaluation21(1):48–60 (https://doi.org/10.1093/reseval/rvs001).
63.
MachamerPeterOsbeckLisa. 2004. “The Social in the Epistemic.” Pp. 78–89 in Science, Values, and Objectivity, edited by MachamerP.WoltersG.Pittsburgh, PA: University of Pittsburgh Press.
64.
MallardGrégoireLamontMichèleGuetzkowJoshua. 2009. “Fairness as Appropriateness: Negotiating Epistemological Differences in Peer Review.” Science, Technology, & Human Values34(5):573–606 (https://doi.org/10.1177/0162243908329381).
65.
MarshHerbert W.BallSamuel. 1989. “The Peer Review Process Used to Evaluate Manuscripts Submitted to Academic Journals: Interjudgmental Reliability.” Journal of Experimental Education57(2):151–69.
66.
McMullinErnan.1982. “Values in Science.” PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association1982:3–28 (http://www.jstor.org/stable/192409).
67.
McPhersonMillerSmith-LovinLynnCookJames M.2001. “Birds of a Feather: Homophily in Social Networks.” Annual Review of Sociology27(1):415–44.
68.
MertonRobert. 1973. “The Normative Structure of Science.” Pp. 267–78 in The Sociology of Science: Theoretical and Empirical Investigations. Chicago: The University of Chicago Press.
MillerC. Chet. 2006. “From the Editors: Peer Review in the Organizational and Management Sciences: Prevalence and Effects of Reviewer Hostility, Bias, and Dissensus.” Academy of Management Journal49(3):425–31.
71.
National Research Council. 1978. Peer Review in the National Science Foundation: Phase One of a Study. Washington, DC: The National Academies Press (https://doi.org/10.17226/20041).
72.
NielsenMathias WullumFriis BakerChristineBradyEmerPetersenMichael BangJensAndersenPeter. 2021. “Meta-Research: Weak Evidence of Country- and Institution-Related Status Bias in the Peer Review of Abstracts.” eLife10:e64561 (https://doi.org/10.7554/eLife.64561).
73.
OreskesNaomi. 2019. Why Trust Science?Princeton, NJ: Princeton University Press.
74.
RoumbanisLambros. 2022. “Disagreement and Agonistic Chance in Peer Review.” Science, Technology, & Human Values47(6):1302–33 (https://doi.org/10.1177/01622439211026016).
ScottWilliam A.1974. “Interreferee Agreement on Some Characteristics of Manuscripts Submitted to the Journal of Personality and Social Psychology.” American Psychologist29(9):698–702 (https://doi.org/10.1037/h0037631).
ShapinSteven. 1992. “Discipline and Bounding: The History and Sociology of Science as Seen through the Externalism-Internalism Debate.” History of Science30(4):333–69 (https://doi.org/10.1177/007327539203000401).
SilerKyleLeeKirbyBeroLisa. 2015. “Measuring the Effectiveness of Scientific Gatekeeping.” Proceedings of the National Academy of Sciences112(2):360–65 (https://doi.org/10.1073/pnas.1418218112).
81.
SizoAmandaLinoAdrianoReisLuis PauloRochaÁlvaro. 2019. “An Overview of Assessing the Quality of Peer Review Reports of Scientific Articles.” International Journal of Information Management46:286–93 (https://doi.org/https://doi.org/10.1016/j.ijinfomgt.2018.07.002).
SnowCharles Percy. [1959] 2012. The Two Cultures. Cambridge, UK: Cambridge University Press.
84.
SquazzoniFlaminioBravoGiangiacomoFarjamMikeMarusicAnaMehmaniBaharWillisMichaelBirukouAliaksandrDondioPierpaoloGrimaldoFrancisco. 2021. “Peer Review and Gender Bias: A Study on 145 Scholarly Journals.” Science Advances7(2) (https://doi.org/10.1126/sciadv.abd0299).
85.
SteelDaniel. 2010. “Epistemic Values and the Argument from Inductive Risk.” Philosophy of Science77(1):14–34 (https://doi.org/10.1086/650206).
86.
TeplitskiyMishaAcunaDanielElamrani-RaoultAïdaKördingKonradEvansJames. 2018. “The Sociology of Scientific Validity: How Professional Networks Shape Judgement in Peer Review.” Research Policy47(9):1825–41 (https://doi.org/https://doi.org/10.1016/j.respol.2018.06.014).
87.
TeramaEmmaSmallmanMelanieLockSimon J.JohnsonCharlotteAustwickMartin Zaltz. 2017. “Correction: Beyond Academia – Interrogating Research Impact in the Research Excellence Framework.” PLOS ONE12(2):e0172817 (https://doi.org/10.1371/journal.pone.0172817).
88.
ToulminStephen E.2003. The Uses of Argument. Cambridge, UK: Cambridge University Press.
89.
TravisG. D. L.CollinsH. M.1991. “New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System.” Science, Technology & Human Values16(3):322–41 (https://doi.org/10.1177/016224399101600303).
90.
WenneråsChristineWoldAgnes. 2010. “Nepotism and Sexism in Peer-Review.” Pp. 64–70 in Women, Science, and Technology, edited by WyerM.BarbercheckM.CookmeyerD.OzturkH.Wayne.M.New York: Routledge.
91.
WolframDietmarWangPeilingHembreeAdamParkHyoungjoo. 2020. “Open Peer Review: Promoting Transparency in Open Science.” Scientometrics125(2):1033–51 (https://doi.org/10.1007/s11192-020-03488-4).
92.
YuanWeizheLiuPengfeiNeubigGraham. 2022. “Can We Automate Scientific Reviewing?” Journal of Artificial Intelligence Research75:171–212 (https://doi.org/10.1613/jair.1.12862).
93.
ZhuangZhenzhenChenJiandongXuHongfengJiangYuwenLinJialiang. 2025. “Large Language Models for Automated Scholarly Paper Review: A Survey.” Information Fusion124:103332 (https://doi.org/10.1016/j.inffus.2025.103332).
94.
ZuckermanHarrietMertonRobert K.1971. “Patterns of Evaluation in Science: Institutionalisation, Structure and Functions of the Referee System.” Minerva9(1):66–100 (https://doi.org/10.1007/BF01553188).
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
