Abstract
Peer review has never been a uniform practice but is now more diverse than ever. Despite a vast literature, little is known of how different disciplines organize peer review. This paper draws on ninety-five qualitative interviews with editors and publishers and several hundred written reports to analyze the organization of peer review in pure mathematics. This article focuses on the practice of “quick opinions” at top journals in mathematics: asking (senior) experts about a paper’s importance, and only after positive evaluation sending the paper for a full review (which most importantly means checking the paper’s correctness). Quick opinions constitute a form of “importance only” peer review and are thus the opposite of the “soundness only” approach at mega-journals such as PLOS ONE. Quick opinions emerged in response to increasing submissions and the fact that checking correctness in mathematics is particularly time-consuming. Quick opinions are informal and are often only addressed to editors. They trade on, indeed reinforce, a journal hierarchy, where journal names are often used as a “members’ measurement system” to characterize importance. Finally, quick opinions highlight that a key function of the peer-reviewed journal today, apart from validation and filtration, is “designation”—giving authors items on their CV.
Introduction
Scientific articles and journals are an indispensable part of science, providing a platform on which scientists may register discoveries and communicate their findings (Fyfe et al, 2020). Since the eighteenth century, they have also become a key mechanism through which scientists demonstrate their expertise and success (Clark 2006). A crucial aspect of journal publication is peer review, which is often taken as “the warp that holds the complex fabric of science together” (Jukes 1977, 203).
Peer review comes in many different forms (Chubin and Hackett 1990; Horbach and Halffman 2020, 140) and has evolved in disorderly ways (Burnham 1990). The emphasis on external reviewers only emerged in the twentieth century (Csiszar 2018; Lalli 2016). Electronic publishing has offered possibilities for new formats of journals and peer review (Johnson et al. 2018, 7). Since the 1990s, some scientists have distributed their papers prior to peer review on preprint servers such as arXiv (Bohlin 2004; Delfanti 2016). In the mid-2000s, open-access journals such as PLOS ONE introduced a new form of peer review, “soundness only,” for which reviewers explicitly refrain from judging the significance of the results and only assess the validity of the research (Eve et al. 2021, 13-14). Such journals defer the more “subjective” judgment of importance to the scientific community after publication. Other journals, particularly in medicine, developed various forms of “open peer review” (Ross-Hellauer 2017), where reviewers’ names and reports are published as part of the article. A recent innovation, “overlay journals,” no longer publish papers but only certify papers that have already been published on preprint servers.
Despite a vast literature on peer review (Bornmann 2011), very few studies have explored how peer review is organized in different disciplines. A notable exception are two studies of peer review in sociology (Hirschauer 2010; Merriman 2021). This paper investigates the organization of peer review in pure mathematics on the basis of qualitative interviews with ninety-five editors and publishers of mathematics journals, as well as several hundred written referee reports.
Adopting an ethnomethodological perspective (Lynch 1993), the aim of this paper is to explore how peer review is actually done in practice, focusing on how editors—and reviewers—decide which papers to accept and which to reject. This paper explores this question in the context of pure mathematics, which is a “perspicuous setting” (Garfinkel 2002, 135) through which to investigate peer review. It is often treated as an epistemic exception, because it relies on deductive proof rather than empirical evidence. Mathematicians have been among the earliest adopters of preprint servers, which is how most of them learn about new results in their field (Crowley et al. 2011, 1128). In addition, many mathematicians are critical of profit-oriented commercial publishers in academic publishing and have organized a boycott of Elsevier (Harvie et al. 2013). Some have experimented with new types of online journal (Electronic Journal of Combinatorics), open-access journals (Forum of Mathematics), and overlay journals (Discrete Analysis). Yet mathematicians have not done away with peer review, which they refer to as “refereeing.” In line with common practice in the natural sciences, peer review is typically “single-blind,” but not “double-blind,” in other words, referees know the names of the authors of the paper under review.
For any discipline, journal peer review is widely understood to involve two key judgments: “whether the claims of the papers are sufficiently well substantiated, and whether it possesses a sufficient degree of scientific interest to be worth publishing” (Polanyi 1951, 53). The first judgment concerns the soundness or correctness of the methods used; the second asks about the importance or significance of the results. While some “mega-journals” such as PLOS ONE are now asking their referees to assess only the soundness of a draft, here I discuss the opposite practice of “quick opinions” in mathematics: asking about the significance of the results before sending the paper to a referee to check whether it is correct.
Studying Mathematical Peer Review
The main issue in conducting research on peer review is access (Chubin and Hackett 1990, 96), both to people (who want to remain anonymous) and to documents (which have been obtained with guarantees of confidentiality).
For this project, I conducted qualitative interviews with editors of journals in pure mathematics. I focused on editors for several reasons. First, editors have a broad understanding of how peer review is organized at a particular journal. Second, editors could tell me about the choice of referees. Third, editors know the evaluation criteria. Finally, every editor has also acted as a referee and has submitted manuscripts to journals as an author, so is able to talk about the peer-review process from the perspectives of editor, referee, and author.
The aim was to talk to editors who had a lot of experience, that is, had been editor-in-chief or managing editors or many years of experience as an editor. Practical considerations also mattered: I arranged interviews around a number of visits to cities with renowned mathematical centers in the United States and Europe and tried to contact as many editors in those cities. I also contacted editors in other places to ask whether they would be happy to be interviewed remotely.
Overall, I conducted ninety-five interviews, both in-person (forty-nine) and via telephone/video-conferencing (forty-six). Of my interviewees, eighty-seven were mathematicians who had worked as editors (often of more than one journal), while eight worked for journal publishers. The majority of the editors worked in North America (thirty-nine) and Europe (thirty-seven), some were based in Asia (nine) and two were working Africa (one) and Australia (one). Most editors interviewed in for this research (sixty-five of eighty-seven) were senior, that is, they were or had been editor-in-chief or managing editor of one or more journals.
Many of my interviewees were editors of more than one journal, and different interviewees were sometimes editors of the same journal. The 87 editors talked about 111 journals that they had been affiliated with (a list which may contain the same journal more than once, if different editors talked about the same journal). The journals were almost exclusively in pure mathematics, but three interviewees talked about journals covering applied mathematics too.
Perhaps the most significant gap with respect to my sampling of journals concerns the “prestige” of the journals discussed, which did not cover the whole spectrum of journals, and were concentrated toward the “top” end. I realized this half-way through the interviews, but it turned out to be much easier to get responses from editors at “good” journals. Using the Scimago Journal Ranking for Mathematics (which in 2016 contained 1,284 journals), thirty-two journals were in the top 10 percent, thirty-eight journals in the next 15 percent. So, 70 of the 111 journals talked about (63 percent) were in the first quartile. The rest of the journals were distributed as follows: eighteen in the second quartile, eleven in the third quartile, eight in the fourth quartile, as well as four journals that were new or not listed.
After each interview, I asked my interviewees whether they would be willing to share any materials (emails, quick opinions, and technical reports) to which they had access. In total, I collected 120 quick opinions, 100 technical reports, and 50 emails (e.g., correspondence asking for quick opinions or accepting/rejecting a paper).
Very early in my interviews, I learned about the phenomenon of quick opinions. As a result, I added questions to my interview guide that further explored the emergence of quick opinions and tried to elicit what editors thought about this practice. For the analysis, I adopted a top-down coding approach, where I worked through all of my interviews to identify all possible aspects related to the phenomenon of quick opinions, which included their emergence and functions, their authorship and recipience, and interviewees’ general views about them. I also explored places where interviewees talked about the overall journal landscape and expressed their views about peer review in general. I then grouped these topics into several themes, which structure the analysis section below.
I have anonymized the identities of my interviewees and distinguish between them through my internal numbering in square brackets, e.g., [12]. I have also anonymized the content of emails and reports. This project received ethics approval from the Survey and Behavioural Research Ethics Committee of The Chinese University of Hong Kong (reference: CUHK 14600218).
Empirical Findings
The Emergence of Quick Opinions
Peer review involves two key judgments. First, checking soundness, that is, making sure that the research has been conducted correctly and that conclusions are empirically supported. Second, judging the significance, that is, assessing how important, new, and interesting the findings are (Horbach and Halffman 2018, 11; Merriman 2021, 343). Mathematicians refer to the soundness assessment as checking the results’ correctness: “There are two judgements that are always made in deciding whether to publish a math paper—the correctness and the significance” [34].
Interviewees described a time of transition in the early 2000s, when “top” journals in mathematics split the two judgments in peer review into separate steps, that is, introduced a “two-stage reviewing” process. Editors first ask experts about a papers’ significance and invite someone else to check the correctness only after receiving an initial positive evaluation: These days, I believe that the journals tend to split the refereeing work into two parts. First asking for general advice from usually a well renowned mathematician who has a lot of experience. Then, depending on the general opinion, to ask for a more precise and more detailed report from an expert, a younger one, maybe less busy. [49] The editors of <journal> would greatly appreciate it if you could give us your quick opinion on this paper.…I am not asking for a full referee report, but only for your judgement of the importance of the results in the paper, so we can decide whether to get a full referee report. I’d be extremely grateful if you could give me an executive report. That is, an opinion (and a few lines of justification) on whether,
The main reason for the emergence of quick opinions was a dramatic growth of papers in mathematics, which resulted in papers being rejected after several months and even years. Such rejections were not due to problems with the results’ correctness, but with their significance: “most papers which are rejected are rejected not because of mistakes but because they’re not interesting enough” [24].
Quick opinions are also a result of the view among mathematicians that checking correctness in pure mathematics, a deductive discipline, is harder and more challenging than checking soundness in the empirical sciences (Greiffenhagen 2023). Reading a mathematics paper is a labor-intensive process (Barany and MacKenzie 2014). Therefore, technical refereeing in mathematics is regarded as much slower than in other disciplines. My interviewees reported that refereeing a paper often takes them several days or even weeks of actual work, which is much longer than in other disciplines (Weller 2001, 157). As one of my interviewees put it: Reviewing a mathematical paper is a lot more work perhaps than reviewing a scientific paper. There is a sense in which, if a referee is checking the mathematical details in earnest, that’s like repeating the experiments that are done in experimental sciences. [50]
Quick opinions are solicited for serious papers that are being considered at top journals, that is, “high-rejection” journals (Zuckerman and Merton 1971, 78). Such rejection rates seem to have increased recently: “In the past, a journal would be rejecting 70 percent and taking 30 percent. Now they’re rejecting 90 percent. The number of papers is just much greater, the competition is much greater” [32].
Quick opinions involve experts outside the editorial board. These are the same experts that in previously would have agreed to write a full report, but who stopped doing so, since they were asked for full reports too frequently: In the old days, I think people weren’t doing that, then we would just send it to a referee and expect a referee report.…you would bother somebody with one paper every six months. But the number of papers is so large, the number of experts that you want to hear from are still the same guys, that you just can’t expect people to referee long papers. [32]
Editors do not just split the refereeing process into two parts—“quick opinions” and “technical/full reports”—they also assign these to different categories of mathematicians: “Typically you ask a big shot to comment on the importance of the paper and you ask a lesser known but very competent mathematician to check all the details” [11]. In other words, editors ask senior mathematicians for quick opinions, and more junior ones for technical reports. The reason for this is that senior mathematicians are seen as too busy to do the time-consuming work of technical refereeing (“they’re unlikely to have time to read the paper in depth” [36]). In contrast, junior mathematicians are perceived as having more time and also as benefiting from working through a paper. Some disciplines have the problem of “ghost refereeing,” where a senior person is asked to be the reviewer, which is actually written by a postdoc or doctoral student. By contrast, in mathematics, it is openly acknowledged that “unknown” younger mathematicians may do the technical refereeing and indeed may do a better job than more senior ones.
Quick opinions can also be seen as a result of the ever-increasing specialization of mathematics (Hagstrom 1965, 228). The institutionalization of peer review in the twentieth century was partly due to increased specialization (Burnham 1990, 1323; Lalli 2016, 168). Quick opinions are also a result of increased specialization. It is not just that handling editors may not have the necessary expertise to check the correctness of a paper but may not feel confident in judging how important a paper is. Handling editors, especially at generalist journals, may know that a paper is a serious piece of mathematics but may be unsure how novel it is. The specialization of mathematics means that even if a paper is within an editor’s expertise, they may still not feel confident enough to make that judgment: Ninety-five percent of the time, even though I’m a topologist and this is in some neighborhood of topology, it’s not something that I’m really expert at. So, I solicit one or more quick opinions. Again, this might be something that I’m not expert in, but I know who the experts are. [72] Then the next step is to ask for an overview from established people who will suggest other names. Then you often get to some people from the younger generation, who most of the time you don’t know at all because if they are not in your topic they are not famous, they are maybe recent PhDs or postdocs or young assistant professors. [69] I think you have missed out on one of the functions of quick opinions, which is to provide suggestions for suitable referees. You generally ask for quick opinions from people you think would know the area better than you do, and their ideas for referees are usually better than yours. [5]
Finally, what about the chance of acceptance if a paper “survives” the quick opinion stage and is sent out for technical refereeing? While I was not able to obtain any statistical information, I gathered some general impressions from editors. It seems there are two types of journals. For one group, if the paper receives a positive quick opinion, then the editors “kind of pre-accept it” [21]. In other words, “we have an a priori understanding that the paper will be accepted if it is correct” [68]. Editors spoke of an 80-90 percent chance of acceptance in such journals. Yet in another group of journals, the chance of acceptance after the quick opinion stage is still less than half. At these journals, the technical referee checks the results’ correctness and judges the significance of the paper.
The Quickness and Informality of Quick Opinions
Editors expect mathematicians to deal with their requests quickly. In contrast to referees, who are supposed to check correctness, it is assumed that quick-opinion givers spend very little time coming up with a judgment: “I expect those persons to look at the paper for one to five minutes and send me back their opinions” [11]. Of course, finding the right expert is crucial: “If you are an expert in an area, looking for half an hour at a paper, you really get a good idea of the significance of the results” [43].
Editors think that referees may already know of the paper, having either seen it presented at a conference or seen a version on arXiv. In the written quick opinions that I was able to gather, I found evidence of this: “I was already familiar with the material before it was submitted,” “I saw the <author’s> paper before and discussed it with <colleague>,” or “I already invited <author> to do a talk about this paper.”
Rather than working through the whole paper to understand every step and checking the correctness of the proofs, scholars invited to give a quick opinion are only supposed “to read through the introduction and skim through the rest of the paper” [84]. One interviewee said that referees “just sniff the paper” [32], reflecting a remark by Halmos (1985, 119) that papers should “smell good” to experts.
Quick opinions are both quickly produced and “written in a very colloquial or informal style” [65]. Indeed, quick opinions are sometimes called an “informal opinion” [24]. This informality is exhibited in various ways, for example, a “very, very frank” [81] tone. The expert may write things such as “I’ve seen nothing like this before! It seems most surprising” or: <Theorem A> was definitely something I did not know about <Topic X> and I wasn’t able to reprove it using tools familiar to me, so I think the authors have really found something new.
Quick opinions are “not always completely anonymous” [68] and may acknowledge contact between referee and author, for example: “I discussed with the author many times about this paper.” Quick opinions also display a familiarity between the referee and editor. For example, a quick-opinion giver wrote “I have heard this kid lecture several times on the paper.” In referring to the author as “this kid,” the quick-opinion giver puts both the editor and referee in the same category (“senior colleagues”), passing judgment about a junior colleague. Quick-opinion givers often mention other people, who they expect the editor knows: I am not an expert on this material, but <Z> is. I remember <A> telling me many years ago about <Topic>. He thought it was very important and came up with an alternate approach. This paper is a generalization of that result and is consequently of great importance.
Quick opinions are similar to the discussions that mathematicians might have during afternoon tea in departments all over the world, and which are “informal times when students and faculty gather to discuss and share ideas and problems” (Walker et al. 2008, 130). In these situations, mathematicians are more open than they would be in a formal, written referee report. Indeed, one editor stated that an advantage of working in a big department was that you can ask colleagues directly about their views on a paper, adding: “There are things people prefer to tell you and not to write” [73]. Such views are precisely what quick opinions are trying to capture.
Owing to their informality, quick opinions are addressed to the editor: “I’m [as an editor] writing to the opinion person and he or she is writing back to me, it’s a letter to me” [72]. As another interviewee put it: “The whole point about quick opinions is that the person who is asking is so massively overwhelmed they don’t have time to formulate every sentence perfectly” [26]. Thus, quick opinions resemble the 1930s referee reports at Physical Review discussed by Lalli (2016, 158), which were characterized by a familiarity between editor and referee and consisted of direct judgments and jokes. They also parallel the “semi-private advice to editors” by sociology journal editorial board members (Merriman 2021, 351) and the editorial votes within an editorial board described by Hirschauer (2010, 77).
At many “top” journals, the decision to accept or reject a manuscript is made by an editorial board, rather than by the handling editor. This can take many forms. Some editorial boards have regular in-person or online meetings where a handling editors present articles that they recommend for acceptance to the editorial board; other boards require the editorial board to vote on recommended articles (using software such as EditFlow). At such journals, quick opinions are important “data points” [84] for editors on the board when making their decision. The content of the documents and the names of the quick-opinion givers are made available: “It is important to know who gives the quick overall opinions” [64].
Interestingly, there is a deliberate ambiguity about who read quick opinions, since they may not just be seen by the handling editor, but by the editorial board. Some interviewees thought this was widely known: “I’ve always assumed anything I’ve sent to a journal, the entire editorial board has access to” [84]. Others thought that this was less clear: “I don’t think we ever tell them that. I don’t know if some of them would be upset” [83]. At least one interviewee was not aware of this and told me that while he did not disagree with this practice, editors should be “more transparent” [81]. However, as handling editors typically have the authority to reject a paper, quick opinions seen by the entire editorial board are typically positive. Only handling editors see the negative quick opinions.
Peer-review reports are often used to formulate “grounds for rejection” (Hirschauer 2010, 87) and to transfer responsibility from editors to anonymous referees (Lalli 2016, 157). Alternatively, they may contain “suggestions for revision” (Hirschauer 2010, 87). This does not seem to be the case for quick opinions. Some editors reproduce sections of the quick opinions when writing back to authors, but only after they have been “suitably edited” [36]. Several editors said there was very little in the quick opinions they could relay back to authors: But also the vast majority of the rejections I cannot tell them why. I could say, “Asked experts informally. The conclusion was that this is interesting, but some of the consensus was this is not sufficiently a landmark result.” There is no more reason than that. Sometimes people respond saying, “how can you reject it without a referee report?” They don’t understand the system, there is nothing for me to give. [26]
Trading on, and Reinforcing, a Shared Hierarchy of Journals
Quick opinions make judgments about a paper’s significance, but how is this formulated? Simply saying “it’s significant” is not enough. The most frequent practice is to name a journal to indicate a paper’s “level.” All academics are familiar with such practices; one may speak of “a Nature paper” or “an ST&HV paper.” In other words, academics frequently name a journal to indicate an assessment of a paper that is published there. Such assessments trade on a hierarchy of journals, the “pecking order of journals” (House of Commons Science and Technology Committee 2011, Volume 2, Ev w99) that exits in every discipline.
Such hierarchies vary from discipline to discipline. For example, Glenn (1971, 302) noted that “there is not a highly integrated system of prestige in sociology,” where consensus only existed on the most prestigious outlets. For mathematics, the hierarchy seems to be more shared. One interviewee argued that “for pure math the global understanding is quite well synchronised” [36], and another interviewee believed that there was “an extraordinary degree in mathematics of consensus regarding the standards” [67]. Finally, another interviewee argued that “there is a more or less consensus about how the journals are situated” [64].
What is most remarkable about mathematics is how open mathematicians are about acknowledging the journal hierarchy. One interviewee started the interview with the following comment: Let me start by saying that there are over 400 mathematics journals.…They fall into different classes. At the very top are the so-called super prestige journals.…They are very, very selective and basically their policy is that they won’t print anything unless it solves a major unsolved problem. Then there are the quite good journals that are just one level down below from that. Then there are still pretty good journals just one slight notch below that, and then there are also specialty journals. [42] I looked at this paper. It looks My feeling is that a journal like <Journal-B> is I think that it is interesting, but My best guess is that it is
Quick-opinion givers use the names of different journals to characterize a paper’s importance and “level.”
Perhaps the clearest evidence of how open mathematicians are about a status hierarchy of journals can be seen in the fact that editors—in their emails asking about quick opinions—use similar formulations to characterize their journal ( I am writing to ask for your quick opinion about the paper <Title> by <Author> that has been submitted to <Journal>, where the acceptance standards (in terms of novelty/significance of the results and/or methods) <Journal> aims to be a top journal publishing high-quality papers Consequently, we can accept this paper only if it is a very substantial and important contribution to the field (
The final characterization, “fit for initial consideration, maybe not acceptance,” is perhaps the most remarkable. This characterization assumes that the recipient is aware of the status of the journal that is used for comparison (in this case: Inventiones) but acknowledges that the journal asking for the quick opinion is slightly lower than that.
Sometimes, an expert may not be familiar with the level of the journal that is asking for a quick opinion, but in those cases, they may use other journals to characterize the paper under consideration: “I don’t myself have a sense of the strength of <Journal-A>, but I would reject it at e.g. <Journal-B>, or <Journal-C>, probably even <Journal-D>.”
It can occasionally also happen that a referee exhibits, in the eyes of the editor, the wrong understanding of the level of a journal.
While my interviewees were open about a hierarchy of journals and papers, they also insisted that the former is not based on the journal impact factor (JIF). As one interviewee stated, “But the very good journals, they have been known. They are known. You don’t need impact factors to know which they are” [1]. Indeed, the interviewed mathematicians were very critical about JIFs as quality measure (Adler et al. 2009).
Mathematicians thus are not “metrics obsessed” (Wilsdon et al. 2015), but they may be “status obsessed.” Mathematicians clearly enjoy rankings, whether it is the ranking of journals, the ranking of papers, the ranking of important problems (such as the Millennium Prize Problems of the Clay Mathematics Institute), or even the ranking of people. In my interviews, I was told of a chair of a mathematics department asking foreign guest speakers before their lecture, “Who are the best three mathematicians in your country?” ([66], see also Halmos 1985, 304-5). Further, when mathematicians from different subfields talk to each other, they may ask of each other: “What are the top problems in your field?” Mathematicians, as Harris (2015, 11) has observed, “seem peculiarly obsessed with ordered lists.”
Regardless of whether the journal hierarchy is really shared in mathematics, it is clear that mathematicians proceed as if the hierarchy is shared. Indeed, authors, editors, and reviewers reinforce this hierarchy. Authors give journals importance by submitting their best paper to them, “We have a certain feeling that some journals are important…or at least we give them more importance. They attract important papers” [65]. Similarly, experts in their quick opinions, give journals importance by using their names to characterize the level of a paper.
None of my interviewees really challenged this journal hierarchy. Of course, I spoke to editors at “top” journals, who were successful, senior mathematicians and therefore not representative of the whole community. The only criticism that I encountered was not with respect to the existence of a hierarchy per se, but a “bias” in terms of mathematical subject areas featured in these top journals. In other words, “the top journals are usually controlled by a certain group of mathematicians” [28], and thus “papers that are easy to be published in these top journals are actually those papers solving the problems proposed by this top group of mathematicians” [28].
Treating the Level of a Paper as Fixed
Quick opinions ask about the significance of a paper before the paper is sent out for detailed refereeing. The paper’s introduction is important because it is often the only part of the paper that quick-opinion givers read. Indeed, it could be argued that it is the only part of the paper that can be read (rather than having to be “worked through”), since it is written in prose and is “about” the paper. An introduction should state the main results of the paper and the key ideas or methods used to obtain the results (Barany and MacKenzie 2014, 121). Papers will be revised because of the technical report, which often points to typos, requests for clarification, and suggestions for editing. Yet, it seems that mathematicians do not think that the significance of the paper is changed through in-depth correctness reports. Quick opinions ask how significant a paper is, not how significant a paper could be after revisions. Such a stance is reflected in the language used by quick-opinion givers. The main theorem and the method of proof are talked about in the present tense: “The results of this paper
Treating the level of a paper as fixed may account for another practice in mathematics. Namely, accepting a paper for publication in a second journal based on the reports solicited for a first journal. This can happen in two ways. First, an editor may edit multiple journals, and in their rejection letter to an author, they might suggest that the paper could be accepted at another journal “as is:” If something is pretty good, but then gets rejected, then I might say, “Why don’t you submit it to <Journal-B>,” I can, like, accept it, immediately, based on those reports.…Cos I’m an editor of both journals. [21] Especially if the recommendations were all reasonably positive but we didn’t accept it. I say: “Well, wherever you submit next, have that editor contact me and I can pass on to them the identities of the referees and they can more or less use the same reports.” [14]
In sum, the peer-review process in mathematics does not seem to place much importance on a paper per se. Instead, the goal seems to be mainly about determining “how a paper sits” [15] in the journal hierarchy.
Views about Quick Opinions and Journals
Most interviewees saw the emergence of quick opinions as a “a very good thing” [84] or at least as “totally unavoidable” [47] in light of increased submission rates in “top” journals: “If you need a detailed report on everything that comes in, you simply can’t do it” [47]. In general, the interviewees were happy with “this level of informality” and thought the process was “quite fair” [43].
Quick opinions were seen mainly as a service to authors to reject papers quickly. Ironically, this practice has had an unintended side effect in the form of increased submission rates at top journals. Whereas previously submitting a paper to a top journal meant that it was under review for a long time, after the introduction of quick opinions, it now would be—if unsuccessful—be rejected quickly, thus allowing the author to submit their work for publication elsewhere. As one interviewee states, “It’s kind of harmless to try. So, there is over-submission, there is less self-selection” [26]. Such an attitude is reflected in the number of quick opinions that senior mathematicians are asked to give every year. Indeed, one interviewee received three requests in his inbox during our one-hour interview.
Some editors thought that there was a danger of “more superficiality” [84], since “the vast majority of the papers are getting rejected quickly without a referee’s report (we’re talking about the really top journals)” [26]. In other words, some interviewees were worried that with very technical work, it is impossible to judge the importance of a paper only by glancing at it: “if you just take a superficial look, you disregard things that…could be the most important part and are somehow hidden in some details” [49].
Only one interviewee wondered whether there might also be an “increase[d]…level of bias” [19]. The interviewee listed various elements that may lend more weight to a quick opinion, “is it coming from a prestigious affiliation? Is it coming from an author who has a long track record of papers in top journals? Is it coming from an author who’s senior and male and in a developed country? This sort of thing” [19]. However, I also heard an anecdote from an editor where a quick-opinion giver was very critical of some famous mathematicians: “He said: ‘good paper—but, geez, these authors should find a more interesting application for their method. This is not such an interesting result’” [14]. So, fame does not guarantee a positive quick opinion.
Toward the end of the interviews, I asked interviewees how they viewed the functions of journals today. Many immediately replied that as researchers, they no longer read journals but instead access relevant papers through the arXiv preprint server (Crowley et al. 2011, 1128). Yet, some interviewees mentioned that journals are a way “to help other people figure out what is worth reading” [25]. Thus, mathematicians not only rely on the arXiv, but they also consult journals to keep up to date with “important” results and developments. In fact, a publication in a “top” journal is an “advertisement to the larger mathematical community…that this is something that you might be interested in paying attention to” [13]. Therefore, the journal hierarchy serves as a “filter” (Bohlin 2004, 372; Clarke 2010; Hirschauer 2010, 72), whereas the arXiv can be “an undifferentiated melange of work” (Krantz 2011, 893).
The most frequent answer to the question about the function of journals today was about the importance for people’s careers. Many of my interviewees emphasized this: “The sense I get is that by far the primary reason people submit things to journals is because they want to get promoted” [36], and “so, the main purpose of these publications is really to put the name of the journal in the CV” [16]. In other words, the interviewees prioritized the “designation” function (Clarke 2010), which treats a publication in a good journal as “academic capital” (Fochler 2016, 933). In that sense, “the primary function of a journal is to rate a paper” [81].
In sum, interviewees viewed publication in a journal as a “stamp of approval” [15]. On the one hand, publication certifies that the paper has been checked. On the other, publication “grades” the paper via the level of the journal. The main role of journals today is to do exactly these two things: a) this has been verified or checked by experts and b) this group of people who are leaders of the field think…that this paper is so good that it belongs in such and such niche or category. So I don’t think that journals will die. [40] I mean you don’t go to war with your colleagues for a paper in <Journal>, but you do go to war for a paper in Inventiones. It can change a career, and so the pressure is enormous. [11]
Discussion
Importance Only
Since their emergence around twenty years ago, quick opinions are now standard practice at all “top” mathematical journals. Quick opinions involve judging “importance only” of papers, which is almost the exact opposite of how peer review is organized at mega-journals such as PLOS ONE, which asses “soundness only.” Such journals promise to publish all papers that are judged to be technically sound, while evaluation of the paper’s importance is left to be judged by its impact after publication (Eve et al. 2021, 13-14; Horbach and Halffman 2018, 6-7). After all, many academics see judging a paper’s soundness as more objective than judging its significance (Oliver 2016, 349; Teplitskiy et al. 2018, 1828).
Mathematicians do value checking the correctness of a paper and they do think that it is a (relatively) objective judgment. Yet, they also consider it to be incredibly difficult and time-consuming. Mathematicians therefore “front-load” the more subjective judgment of importance—and only ask referees to check the correctness if the paper has a relatively high chance of acceptance.
Judging a paper’s importance before sending it out for review is not unique to mathematics. Indeed, it is something that is done at top journals in other fields (e.g., Nature or Science), which consult members of the editorial board (or other formal advisors) to decide whether a paper is worth sending out for peer review. However, quick opinions in mathematics involve experts outside the editorial board. Quick opinions are part of the official peer-review process, which can be seen in the fact that in the editorial management system developed by mathematicians (EditFlow), editors have two choices when contacting an outside expert: “they have an automated system…you can ask for a quick opinion or for a detailed review” [5].
Merriman (2021, 350) describes how ad hoc (outside) reviewing was introduced in sociology as an alternative to reviewing by editorial boards in order to deal with the increase of submissions and the resultant pressure on editorial board members. Outside reviewing meant that more people could now be involved in the peer-review process. Quick opinions are a similar organizational adaption, but they change what the outside reviewers are asked to do. Quick opinions are a way to “tap into” the knowledge of what are considered the “top” people in the field who may be willing to offer their “opinion,” but not do the time-consuming work of checking. As one of my senior interviewees openly admitted: “I don’t referee for anybody on the grounds that I’m too busy, but I’ll give a quick opinion” [31].
In their quick opinions, senior researchers may also propose junior researchers who can complete the subsequent technical reviewing. Quick opinions serve to recruit more researchers into the review process.
In sum, quick opinions are a means of managing the time for the whole mathematical community. They relieve senior experts from having to write full reviews, while still engaging them in the peer-review process, for example, by helping to identify junior experts who can undertake the detailed refereeing. Yet these “additional” junior researchers are not involved in making the crucial decision about which articles get accepted at top journals. Instead, they do the most time-consuming work of actually reviewing the papers. Junior researchers are not usually in a position to challenge this organization of peer review and editorial work.
Journal Names as a Measurement System
How do you talk about a paper’s importance? Many journals have “rating schemes” as part of their peer-review management system, either as a numerical scale (one to five) or as a series of descriptive terms. Reviewers are supposed to evaluate the paper using this scheme. However, this introduces a problem: what “very good” means can differ widely between experts.
If people are in frequent contact with each other, they may be able to synchronize their understanding of such evaluations. For example, Hirschauer (2010) describes a voting system within a sociology journal’s editorial board. But how can such a system be put in place when experts are spread across institutions and countries? In their quick opinions, mathematicians often use a journal’s name as a way to indicate their judgment about a paper’s importance (“probably at the level of <Journal-B> rather than <Journal>”). Such statements rely on a hierarchy of journals (Bohlin 2004, 372; Clarke 2010). While mathematicians acknowledge that this hierarchy is not perfectly shared, they see it as “quite well synchronized” [35]. By using different journal names as comparison, both quick-opinion givers and editors treat the hierarchy as shared, indeed, they reinforce this hierarchy (as do authors by submitting their “best” papers to what they consider the “best” journals).
Using journal names to characterize a paper’s importance is perhaps surprising in light of the fact that mathematicians are very critical of the JIF. A widely cited report by the International Mathematical Union (Adler et al. 2009) contained many criticisms of citation-based statistics. It argued that the JIF was particularly unsuitable for mathematics, which is a low-citation discipline and where articles are not cited immediately. The report also contained a detailed critique of what it described as the “journal-as-proxy” method, that is, using the JIF to assess a paper’s importance. Such a method misuses a journal-level metric as an article-level one (see also Eve et al 2021, 11; Osterloh and Frey 2020, 2). Despite widespread criticism of the journal-as-proxy method, this practice remains widespread because it is quick and convenient. It avoids engaging with a paper’s content and also eliminates delays in observing a paper’s impact (e.g., in terms of citations gathered after publication). As Biagioli (2018, 259) so nicely puts it, the journal-as-proxy method means that today “a publication is born evaluated.”
This may be the consequence of quick opinions: if an author publishes a paper in a prestigious journal on the basis of positive quick opinions, their university may use the journal’s name to evaluate the paper (and the author) when considering whether to award tenure or a promotion. This is not what the quick-opinion givers are doing when using journal names to talk about the paper’s importance in their comments to editors. Unlike recruitment panels, quick-opinion givers do engage with the scientific content of the paper (albeit only quickly). The journal names are simply used as a vocabulary to talk about the importance of the paper, that is, as a “members’ measurement system” (Sacks 1988). There may not be a fully shared understanding among mathematicians about what it means for a paper to be “at the level of Annals, Duke, or IMRN,” but it seems to be more useful and accurate than saying a paper is “excellent” or “very good.” Instead of using journal names as a proxy measure after an article’s publication, quick-opinion givers use journal names before publication, using the names as part of a shared language used by experts in the field.
An Evaluative Model of Peer Review (for a “Frame-less” Discipline)
As documented by Merriman (2021), a “developmental model” of peer review was adopted in many sociology journals and is indeed now the norm in many social sciences (Kaltenbrunner et al. 2022): reviewers not only evaluate a manuscript but also provide “a detailed roadmap for revision” (Oliver 2016, 347). What is changed through this process is typically the “frame” of the empirical results and analysis (Teplitskiy 2016)—and thereby also the (possible) significance. Peer review is part of the construction—not just evaluation—of quality (Teplitskiy 2016, 268). In contrast, in mathematics, by asking about a paper’s importance at the beginning of the peer-review process, quick opinions do not construct quality—they only evaluate it. Just as mega-journals such as PLOS ONE have adopted an evaluative model by assessing soundness, quick opinions do the same for importance. Quick opinions ask where a paper sits in the hierarchy of journals, not where it could sit.
Peer review clearly improves papers and my interviewees valued this aspect. Apart from pointing out typographical mistakes and mathematical errors, reviewers may suggest improvements for the overall structure of the paper. However, in mathematics, there seems to be no equivalent to the “frame” in the social sciences. Math papers typically have a main result and main method for obtaining that result. These are described in the introduction: “In the introduction you see the background, the landscape, the motivation. You see the theorem, the main result itself. And you see, usually, a sketch of the proof, or at least the strategy” [67]. The main theorem and proof strategy are of course mentioned in the introduction, but it does not make sense to say that they are “framed.”
In the social sciences, authors know that even a positive review process may involve several rounds of revision so some may submit papers “in a state that still admits of ready improvement” (Merriman 2021, 361). Indeed, Oliver (2016, 348) reports that young scholars in sociology may submit papers “too rough to get published to top journals with the goal of ‘getting good reviews.’” In my interviews, I did not hear of similar practices in mathematics. Indeed, it is difficult to imagine what such practices would involve: how could the main theorem be improved through the peer-review process in mathematics?
In mathematics, peer review changes a paper’s exposition, but it does not seem to improve the significance. Put differently, papers in the social sciences can change dramatically through the peer-review process, while papers in mathematics seem to change very little. This may explain why preprint servers are more widespread in mathematics: the published paper will be quite similar to the preprint version, which is why both authors and readers are happy working with preprints. This fact also accounts for the practice of accepting narrowly rejected papers “as is” at another journal, whereas in the social sciences, a “lower” journal may well ask the paper to be “re-framed’. Finally, this finding explains why it was easy for mathematicians to adopt an “evaluative model” of peer review through the practices of quick opinions.
“Disseminate, Filter, then Publish” Model of Scholarly Communication
Mathematics and theoretical physics are disciplines that have been using preprint repositories for several decades (Bohlin 2004; Larivière et al. 2014; Delfanti 2016). Preprint servers are how new papers are “disseminated” (Clarke 2010). Similarly, arXiv is how authors can “register” (Clarke 2010) their discoveries and thereby deal with any potential priority disputes. Indeed, a prominent example can be found in Perelman’s proof of the Poincaré conjecture, which was only disseminated via the arXiv and not submitted to a traditional journal. However, this was still seen as sufficient for Perelman to considered for the Fields Medal and the Clay Millennium Prize in 2006 (Jackson 2006). Yet, putting papers on arXiv is not considered sufficient, as one interviewee noted: “Online publishing, we have that. People post papers on the arXiv, that’s fine, but that’s not the final step, that’s not the end step” [15].
Mathematicians still value the checking of correctness that is done, if imperfectly, as part of peer review: “the refereeing process isn’t a complete validation of the correctness of the paper, but it’s a whole lot better than just having a preprint on say the arXiv. Orders of magnitude better” [84]. In other words, mathematicians treasure the “validation” (Clarke 2010) aspect of peer review. Mathematicians also think that a paper published in a “top” journal works like as an “advertisement” ([13]; see also Harnad 1998, 285) of what the community treats as an important development in the field. In that sense, peer review also provides “filtration” (Clarke 2010) of the many results published every year. Still, the most important function of peer-reviewed journals is for people’s careers, what Clarke (2010) calls “designation.” In that sense, journals are still the main method through which researchers obtain “academic credit,” especially outside their own narrow specialty, such as with university administrators or funders (Delfanti 2016, 631).
Mathematicians have split the five functions of journals described by Clarke (2010) over two different media: preprint repositories, which are primarily used for dissemination and registration, and traditional journals, which are used for validation, filtration, and—most importantly—designation (see also Bohlin 2004, 372; Horbach and Halffman 2018, 11).
Disciplines in which megajournals are popular and have moved from the traditional model of “filter, then publish,” to “publish, then filter” (Shirky 2008, 98), where the filtering happens after publication through the attention an article receives. Mathematicians, in contrast, have adopted a “dissemination, filter, then publish” model. Disseminating a paper through preprint repositories is often the first step, but rarely the final one. Journals are “places where their preprints go to rest”—as an anonymous peer reviewer of this paper said. Preprint servers have not led to “the demise of peer-reviewed journal articles” (Larivière et al. 2014, 1168).
A clear expression of this is in the emergence of a new type of journal in mathematics: arXiv “overlay journals,” such as Discrete Analysis. These journals no longer “publish” papers, instead evaluating freely available papers through a rigorous peer-review process to determine the paper’s significance (Horbach & Halffman 2018, 11). A “publication” at such journals thus consists mainly in listing the article on their website, with a link to arXiv, and writing an “editorial introduction” to the article. These overlay journals may have an impact on the economy of academic publishing by providing a much cheaper model, which is something that mathematicians have long advocated for (Harvie et al. 2013). Overlay journals do not really challenge the functions performed by journals.
The Surprising Resilience of Journals
It is perhaps surprising that the internet (and the possibilities for online dissemination) has not disrupted scholarly publishing more than it has (Clarke 2010). Of course, there have been innovations, such as preprint repositories, “soundness only” peer review, or even open peer review. However, in many disciplines, the system has remained relatively stable. As one interviewee put it: “When arXiv was starting to develop, there was a very strong feeling that at some point the whole system would change completely.…Actually it did not” [73].
Perhaps the key reason is that publications, especially in top outlets, have actually increased in their importance for scientists’ careers. Consequently, changing the system of scholarly publication would not only change scientific communication but also the whole social system of science (Csiszar 2018, 282). Given the spread of a “ranking bureaucracy” (Osterloh and Frey 2020, 5), the journal-as-proxy method of evaluation is difficult to replace. In many universities, annual research performance is assessed by counting the number of articles in “top” journals, which is quick and convenient, and means that assessors from a different discipline (or even no academic training, like university human resource staff) do not have to engage with the content of papers in other disciplines. While it is easy to agree with what is problematic about the current system, it is difficult to imagine how an alternative system could be put into place, since the initial effort of activating the new system would be too high (Tregoning 2018).
The current “metric tide” (Wilsdon et al. 2015) is driven by an attempt to replace (supposedly) subjective expertise with more (supposedly) objective measures: “reliance on numbers and quantitative manipulation minimises the need for intimate knowledge and personal trust” (Porter 1995, ix). In practice, the subjective judgment of experts is simply displaced to a different point in time and place as illustrated by the practice of soliciting quick opinions in mathematics journals studied in this paper. Rather than asking for subjective judgments from experts at the end of the process (e.g., by asking people to read someone’s papers submitted for a tenure review), the expert judgment is simply shifted to earlier in the process: to the views of a few experts at the beginning of the peer-review process.
Footnotes
Acknowledgments
I am extremely grateful to all my interviewees for taking the time to share their views and experiences about peer review in mathematics with me. Many thanks to the editors and three anonymous reviewers from this journal for their generous feedback and help in revising this article. I would also like to thank Michael Barany, David Edmonds, Ilana Gershon, Murray Goulden, Julian Hamann, CF Helgesson, Michael Lynch, Roman Matvienko, Niki Vermeulen, and Steve Woolgar for their time and comments on earlier versions of this article. Special thanks to Stefan Hirschauer whose insights were extremely helpful in the initial stages of the paper.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work described in this paper was substantially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. CUHK14600218).