Abstract
Postpublication critique, such as letters to the editor, can contribute to the validity and trustworthiness of scientific research. We conducted a cross-sectional analysis of the policy and practice of postpublication critique in (a) randomly selected (N = 100) and (b) prominent (N = 100) psychology journals. In 2023, an explicit submission option for postpublication critique was available at 23% (95% confidence interval [CI] = [16%, 32%]) of randomly sampled psychology journals and 38% of the most prominent psychology journals. Journals sometimes imposed limits on the length and time allowed to submit critiques. We manually inspected two random samples of empirical articles published in 2020 (articles per sample: N = 101), estimating the prevalence of postpublication critique to be 0% (95% CI = [0%, 3.7%]) in psychology journals generally and 1% (95% CI = [0.2%, 5.4%]) in the most prominent psychology journals. The policy and practice of postpublication critique is seriously neglected in psychology journals.
Keywords
Critical discourse can contribute to the validity and trustworthiness of scientific research (Merton, 1973; Vazire & Holcombe, 2022). Criticism can highlight errors, limitations, and alternative interpretations, helping research consumers to calibrate their beliefs in scientific claims. However, the proliferation of poor-quality research in psychology (and other fields) indicates that the dominant mode of critical discourse, conventional peer review, is not sufficient to maintain high standards in the scientific literature (Hardwicke et al., 2020; Nosek et al., 2022).
Scientific journals can support the ongoing critical scrutiny of research by offering avenues for postpublication critique, such as letters to the editor (Altman, 2002). Journals’ obligation to facilitate postpublication critique is explicitly codified in the principles of the Committee on Publication Ethics (COPE), an organization that most prominent journals are members of (COPE, 2021; Hardwicke et al., 2022). However, prior research has indicated that many (37%) of the most prominent academic journals across a broad range of scientific disciplines do not explicitly allow submissions of postpublication critique (Hardwicke et al., 2022). Moreover, journals that do allow postpublication critiques in principle rarely publish them in practice and often impose restrictive limits on length and time to submit. In the present research, we closely followed the methods used in Hardwicke et al. (2022) but extended this prior work by providing a focused evaluation of how psychology journals in particular handle postpublication critique (Hardwicke et al., 2022, examined 15 journals in a combined psychiatry and psychology category and 315 journals from other disciplines).
For the purposes of the present study, we operationally defined postpublication critique as any journal-based avenue for sharing peer-initiated critical discourse related to specific research articles previously published in the same journal (for a complete operational definition, see Supplementary Information B in the Supplemental Material available online). A prototypical example of postpublication critique is a letter to the editor, although there are other variations, such as commentary articles and website comments.
Our study was conducted in two parts. In the first part, we aimed to describe policies related to postpublication critique at (a) all psychology journals publishing empirical research (via a random sample of 100 journals) and (b) the 100 most prominent 1 psychology journals publishing empirical research. Specifically, we identified whether options for submitting critiques (e.g., letters to the editor) are available and whether limits are imposed in terms of length, time to submit since publication of the target article, number of references, and whether critiques were sent for independent peer review. In the second part of our study, we aimed to estimate the prevalence of postpublication critiques published in (a) all psychology journals (via a random sample of 100 empirical psychology articles) and (b) the 100 most prominent psychology journals (via a random sample of 100 empirical articles published in these journals).
Part 1: Postpublication-Critique Policies
Method
The methods and analysis plan were preregistered (https://osf.io/nqj7b). All deviations from the preregistered protocol were considered minor and are detailed in Supplementary Information A in the Supplemental Material and, where relevant, mentioned below.
Design
This investigation had a cross-sectional design. All measured variables are described in detail in Supplementary Table C1 in the Supplemental Material. In brief, we extracted and described journal policies related to postpublication critique (according to our operational definition, Supplementary Information B in the Supplemental Material) provided on journal websites in 2023. We recorded the name and description of any options for postpublication critique offered by each journal (e.g., article types such as “letters” or “commentaries” or web comments) or any explicit statement that the journal does not accept postpublication critique; limits imposed on postpublication critique in terms of length (e.g., number of words), time to submit (e.g., weeks since publication of the target article), and/or number of references; and whether it was stated that postpublication critiques would be sent for independent external peer review (i.e., reviews solicited from individuals who were not members of the editorial team or authors of the target article). We recorded qualitative and quantitative limits; for example, a length limit might be stated as “500 words” or “short.” We also identified whether journals were members of COPE, an organization that outlines several core publishing practices, including the importance of facilitating postpublication critique (COPE, 2021).
Sample
Target populations
The target populations of the two samples of journals were (a) all psychology journals publishing empirical research in 2023 and (b) prominent journals in psychology publishing empirical research in 2023. We were interested only in legitimate (nonpredatory) peer-reviewed journals.
Sample sizes
The sample consisted of 100 randomly selected journals and 100 prominent journals. Justification for these sample sizes was based on a precision analysis (details provided in Supplementary Information E in the Supplemental Material). Eleven journals appeared in both samples and were not replaced to maintain the random-sampling scheme.
Preparing samples (inclusion/exclusion criteria)
To reduce the likelihood of predatory journals being selected, we included only journals that were in the Web of Science Core Collection, meaning they met 24 basic quality criteria, such as having a peer-review policy and publishing scholarly content (for details, see Clarivate, 2023). As of March 15, 2023, there were 867 unique journals included in this database that were classified as belonging to a subject category related to “psychology.” Of these 867 journals, 115 were classified by Web of Science as non-English language and excluded, leaving 752 journals remaining.
For the random sample of psychology journals, a list of all included 752 psychology journals was randomly shuffled using the R function sample, and we selected the first 600 on the shuffled list (to allow for exclusions). For the prominent-journals sample, we sorted journals in order of their 2021 Clarivate Journal Impact Factors and selected the first 600 journals (to allow for exclusions). The list of these journals is available at https://osf.io/awpxq. For both samples, journals were then evaluated sequentially from the top of their respective list until the target of 100 eligible journals in each sample was reached. If a journal needed to be excluded after manual inspection, it was replaced with the next available journal in its list, thus maintaining the respective sampling schemes.
We excluded an additional six journals from the randomly selected journal list and 15 from the prominent list because they did not publish empirical research. This determination involved some degree of subjective judgment and was based primarily on each journal’s scope as defined on its website. If the journal’s description of its scope was ambiguous, we searched the journal’s archive for examples of empirical articles to inform our determination. This represented a slight deviation from the preregistered protocol (for details, see Supplementary Information A in the Supplemental Material).
Procedure
Piloting and training
The data-extraction procedures were piloted with 10 journals that were not included in the sample. All investigators were trained and familiarized with the data-extraction process and practiced with positive and negative training examples.
Collating journal policy information
One investigator (A. Whamond or N. Moodie) identified, downloaded, and archived the “scope” and “article types” sections (or equivalent) provided on each journal’s website (https://osf.io/z4h72/). This information was used to determine whether the journal published empirical articles and if so, whether postpublication-critique article types were accepted by the journal. If these sections could not be found, a second investigator (T. E. Hardwicke) double-checked.
Data extraction
The policy information (outlined in Supplementary Information C in the Supplemental Material using a Google Form https://osf.io/ndt4j) was extracted and classified for each journal by two investigators. Primary coding was conducted by A. Whamond (n = 183) and N. Moodie (n = 37). Secondary coding was conducted by R. T. Thibault (n = 82), S. Vazire (n = 80), and T. E. Hardwicke (n = 41). Disagreements between primary and secondary coders, usually arising because of unclear descriptions of article types, were resolved by T. E. Hardwicke. This was necessary for 40 journals (for details, see Supplementary Information H in the Supplemental Material).
For coding whether journals provided an option for postpublication critique, we first examined the journal website’s description of the journal’s article types (typically found in journals’ submission guidelines or instructions to authors). If a journal’s description of its article types was not provided or unclear, we searched the journal’s archive to see if any examples of articles meeting our operational definition of postpublication critique (Supplementary Information B in the Supplemental Material) had been published within the last 5 years (i.e., since the beginning of 2019). Note that if a journal had an article type that sounded like it might be used for postpublication critique (e.g., “Commentary”), that was not sufficient for a positive classification; the description of the article type or examples of that article type also had to meet our operational definition of postpublication critique (see Supplementary Information B in the Supplemental Material).
Data harmonization
Journals used various naming conventions for similar types of postpublication critique and various units (e.g., characters, words, or pages for length limits) to specify limits. To address this, we harmonized the various names used for critique formats to four types (letters [to the editor], commentaries, web comments, and other) and transformed length-limit units to words and time-to-submit-limit units to weeks (for details, see Supplementary Information G in the Supplemental Material).
Results
Journal characteristics
The median 2021 Journal Impact Factor was 2.48 (interquartile range = 1.83–3.81) for the randomly sampled journals (25 of the randomly selected journals did not have Journal Impact Factors) and 6.92 (interquartile range = 5.60–8.69) for the prominent journals. Sixty-five of the 100 randomly sampled journals and 84 of the 100 prominent journals were members of COPE. For journal characteristics stratified by psychology subfield, see Supplementary Information I in the Supplemental Material.
How many journals offer postpublication critique?
For a full list of journals and their postpublication-critique options and limits, see Supplementary Information J (in the Supplemental Material; raw data are available at https://osf.io/z4h72/). Among the 100 randomly selected journals, 23 (23%, 95% confidence interval [CI] = [16%, 32%]) explicitly stated that they allowed submissions of postpublication critique. Of these 23 journals, two had two types of critique formats, and the other 21 each had one critique format. One journal (Clinical Psychology: Science and Practice, also included in the prominent-journals sample) stated explicitly that it did not accept postpublication critique. Twelve journals offered commentaries, 12 journals offered letters, zero journals offered web comments, and one journal (Cortex) offered a format called “verification reports,” which we classified as “other.” Of the 23 journals that explicitly accepted postpublication critique, 15 (65%) were COPE members. Of the 77 journals that did not explicitly accept postpublication critique, 50 (65%) were COPE members.
Among the 100 prominent journals, 38 (38%) explicitly stated that they allowed submissions of postpublication critique. 2 Of these 38 journals, three had two types of critique format, and the other 35 journals had one critique format. Two journals (Clinical Psychology: Science and Practice and Psychological Bulletin) stated explicitly that they did not accept postpublication critiques. No reason was provided. Twenty-five journals offered commentaries, 16 journals offered letters, and zero journals offered web comments. Of the 38 journals that explicitly accepted postpublication critique, 33 (87%) were COPE members. Of the 62 journals that did not have a policy related to postpublication critique, 51 (82%) were COPE members.
What limits did journals place on postpublication critique?
Table 1 (randomly selected journals) and Table 2 (prominent journals) show the limits journals imposed on postpublication critique in terms of length, time to submit, and number of references. Tables 1 and 2 also show whether a peer-review policy for critiques was stated. Length limits were stated for most critique formats, especially those in prominent journals. Length limits were usually stated quantitatively (e.g., “1,000 words”). A few formats had qualitative limits (e.g., “brief”). Letters had tighter length limits than commentaries, on average. The most restrictive length limit was 400 words at the Journal of Adolescent Health. Time-to-submit limits were not stated for most critique formats; they were somewhat more common in prominent journals. Time-to-submit limits, when present, were often stated qualitatively (e.g., “brief”) and sometimes quantitatively (e.g., “3 months”). The most restrictive time-to-submit limit was 4 weeks at the Journal of Applied Psychology. For most critique formats, journals did not state whether they would be sent for independent peer review. For a full list of stated limits at each journal, see Supplementary Information J in the Supplemental Material.
Postpublication-Critique Types Identified in Randomly Selected Psychology Journals and Their Length, Time-to-Submit, and Reference Limits
Note: This table also indicates whether the journal states that critiques will be sent for independent external peer review. Note that ns refer to number of critique formats, not number of journals (some journals had more than one format of postpublication critique). All results are rounded to the nearest integer. IQR = interquartile range.
Postpublication-Critique Types Identified in Prominent Psychology Journals and Their Length, Time-to-Submit, and Reference Limits
Note: Table 2 also indicates whether the journal states that the critiques will be sent for independent external peer review. Note that ns refer to number of critique formats, not number of journals (some journals had more than one format of postpublication critique). All results are rounded to the nearest integer. IQR = interquartile range.
Part 2: Postpublication-Critique Prevalence
Method
The methods and analysis plan used in this study were preregistered (https://osf.io/nqj7b). All deviations from the preregistered protocol were considered minor and are detailed in Supplementary Information A in the Supplemental Material and, where relevant, mentioned below.
Design
This investigation had a cross-sectional design. All measured variables are described in detail in Supplementary Information C2 in the Supplemental Material. In brief, we estimated the prevalence of postpublication critique among empirical articles published in 2020 using two methods: (a) We identified whether sampled articles contained a link to a postpublication critique of the article (“linked method”), and (b) we identified whether sampled articles were themselves examples of postpublication critique (“instance method”). 3 These two methods of estimating prevalence have complementary strengths and weaknesses; for more details, see Supplementary Information D in the Supplemental Material.
Sample
Target populations
The target populations were (a) all empirical articles published in 2020 in all psychology journals and (b) all empirical articles published in 2020 in the 100 most prominent psychology journals identified in Part 1. We were interested only in articles published in legitimate (nonpredatory), peer-reviewed journals.
Sample sizes
We examined 101 randomly selected empirical articles published in psychology journals and 101 randomly selected empirical articles published in prominent psychology journals (our target sample size was 100 articles in each sample, but deviating from our preregistration, we accidently examined one additional article in each sample; for details, see Supplementary Information A in the Supplemental Material). For justification for these sample sizes, see Supplementary Information E in the Supplemental Material.
Preparing samples (inclusion/exclusion criteria)
For the random sample of articles published in psychology journals, we used the Web of Science Core Collection to download bibliographic records classified with a Web of Science document type of “article,” “discussion,” “editorial material,” or “letter” published in 2020 and classified as belonging to a psychology-research area. The search returned 61,735 records (for exact search string, see Supplementary Information F in the Supplemental Material).
For the random sample of articles published in the most prominent psychology journals, we used the Web of Science Core Collection to download bibliographic records classified with a Web of Science document type of “article,” “discussion,” “editorial material,” or “letter” published in 2020 in the 100 most prominent psychology journals (identified in Part 1). The search returned 12,255 records (for exact search string, see Supplementary Information F in the Supplemental Material).
The two lists of sampled articles were each randomly shuffled using the R function sample, and the top 600 entries in each shuffled list were selected (to allow for exclusions). Articles were then evaluated sequentially until the target of 100 eligible articles in each sample was reached. Articles were excluded because they (a) could not be accessed (n = 9), (b) were non-English language (n = 4), (c) indicated they had been retracted (n = 2), (d) were nonempirical (n = 55), or (e) were conference abstracts or “supplement” articles rather than complete articles (n = 9). Assessing whether an article is empirical involved some subjective judgment, but we used the following guidelines: Empirical research must include some analysis of quantitative or qualitative data, and the following do not count as empirical research: news, book reviews, editorials, previews, opinion pieces, simulations, proofs, theoretical papers, and reviews. We included primary research, secondary data analysis, and meta-analyses. If an article needed to be excluded after manual inspection, it was replaced with the next available article in its list, thus maintaining the random-sampling scheme.
Note that the above applies to the linked method for calculating prevalence; for the instance method for calculating prevalence estimates, articles that were themselves postpublication critique (e.g., “letters”) were included, but they did not count toward the 100 empirical articles sample-size target.
Procedure
Piloting and training
The data-extraction procedures were piloted with 10 articles that were not included in the sample. All investigators were trained and familiarized with the data-extraction process and practiced with positive and negative training examples.
Data extraction
To code whether sampled articles were linked to postpublication critique (linked method), two investigators independently viewed each article on the journal’s website, assessed if the article met the eligibility criteria, and if it did, thoroughly checked the page for any linked postpublication critique.
To code whether the sampled article itself was an example of postpublication critique (instance method), the same two coders also read the title and, if necessary, the abstract of the sampled article.
Primary coding was conducted by A. Whamond (n = 246) and N. Moodie (n = 35). Secondary coding was conducted by B. Clarke (n = 122), S. Schiavone (n = 110), and T. E. Hardwicke (n = 49). Data extraction/classification disagreements were resolved by T. E. Hardwicke, which was necessary for 15 articles (for details, see Supplementary Information H in the Supplemental Material).
Results
Among the 101 empirical articles randomly sampled from all psychology journals, we did not find any examples of postpublication critiques based on the linked method or the instance method, resulting in a prevalence estimate of 0% (95% CI = [0%, 3.7%]) for both methods.
Among the 101 empirical articles randomly sampled from prominent psychology journals, we found only one postpublication critique using the linked method (i.e., one sampled article was associated with a linked postpublication critique), yielding a prevalence estimate of 1% (95% CI = [0.2%, 5.4%]). Using the instance method, we found no postpublication critiques (0%, 95% CI = [0%, 3.7%]; i.e., none of the sampled articles were themselves instances of postpublication critique).
General Discussion
Postpublication critique is often viewed as an important mechanism of scientific self-correction (Brewin, 2023; Merton, 1973; Vazire & Holcombe, 2022). However, our study found that across the field of psychology, only a minority of journals (23%, 95% CI = [16%, 32%]) had an explicit submission format for postpublication critique, and actual examples of postpublication critique were extremely rare. Prominent psychology journals were more likely to have an explicit format for postpublication critique according to their policies (38%), but the majority still did not. No journals in either sample supported web comments. Journals that did allow submission of postpublication critique sometimes imposed restrictions on length and time to submit. Occasionally, these limits were quite restrictive; the strictest length limit was 400 words, and the strictest time-to-submit limit was 4 weeks after the target article was published. In practice, postpublication critiques were rare; only a single linked example was found within the articles sampled from prominent journals. Overall, our findings suggest that the policy and practice of postpublication critique is seriously neglected in psychology journals.
Previous research by Hardwicke et al. (2022) found that 67% of prominent psychology and psychiatry journals accepted postpublication-critique submissions, markedly higher than our present results. In addition, this study reported a higher estimate for the prevalence of published postpublication critiques (17%). However, these findings are not directly comparable with the present study. First, the Hardwicke et al. sample included both psychology and psychiatry journals, and the latter were more likely to accept postpublication critique (four out of nine psychology journals vs. five out of six psychiatry journals accepted critique). Psychiatry, as a medical specialty, may operate more in alignment with other medical journals, which are generally more likely to facilitate postpublication critique relative to other disciplines (Hardwicke et al., 2022). Second, the psychology journals included in Hardwicke et al. were the top nine as ranked by Impact Factor, whereas the current study included a much wider distribution of Journal Impact Factors. Journals with higher Impact Factors tend to receive more attention and therefore are probably more likely to attract postpublication critiques; this could also explain the differences found between randomly selected versus prominent-journal policies in the present study. Third, the prevalence estimate for published postpublication critiques came from examining only articles published in journals in which postpublication-critique submission policies had been identified. By contrast, in the present study, we estimated prevalence for all empirical psychology articles regardless of journal policy. Thus, it is unsurprising that the current study yielded lower estimates.
Should all journals offer an explicit avenue for postpublication critique? It is unclear if publishers or journal editors have principled reasons for not explicitly offering postpublication critique. Postpublication critique is recognized as an important mechanism of scientific self-correction (Brewin, 2023; Merton, 1973; Vazire & Holcombe, 2022), and facilitating critique is often considered to be a journal’s responsibility (e.g., according to COPE). In both the randomly selected and prominent samples, most (65% and 82%, respectively) of the journals that did not offer postpublication critique were members of COPE. However, COPE guidelines allow for flexible interpretation by stating that “journals must allow for post-publication discussion on their site, through letters to the editor or on an external moderated website [italics added]” (COPE, 2021). Technically, a journal does not need to do anything to comply with this recommendation because it can be left to independent sites, such as PubPeer, to shoulder the responsibility of facilitating critical discussion. These guidelines could be strengthened to emphasize journal accountability.
It could be argued that postpublication critique can take place in other contexts, such as social media or dedicated platforms, such as PubPeer (Bastian, 2014); however, journals are in a privileged position to encourage, curate, and publish critiques in the scientific record, which improves permanence and discoverability (i.e., by linking to critiques on the same webpage as the target article). Critiques posted informally on external platforms may be overlooked by journals and readers (Allison et al., 2016). There is also a risk that third-party sites will become defunct—as in the case of PubMed Commons, which was shut down in 2018 (Insights, 2018). Some prominent psychologists have even argued that critiques should appear only in venues subject to editorial oversight (e.g., Fiske, 2016).
Are journal limits on postpublication critique justifiable? Length limits may promote concision; however, strict limits (e.g., a few hundred words) are likely to seriously reduce the scope and substance of critiques (Altman, 2005). There is no obvious justification for imposing time-to-submit limits. Such limits imply that there is effectively a “statute of limitations” on scientific critique once an article has been published for a given time (often no more than a few months). Only a few policies stated that critiques would be sent for independent external peer review. It was unclear whether journals routinely send critiques to the original authors for review, but anecdotally, we know this can occur. Independent peer review can provide more impartial assessment of critiques and bolster their legitimacy (Schriger & Altman, 2010). Without independent review, the decision about whether to publish a critique may be based solely on the perspectives of individuals with a conflict of interest, that is, the journal editors and the original authors.
Although our data indicate that postpublication critique is neglected in psychology journals, they do not provide insight into why this is the case. Potential reasons could be cultural (e.g., disciplines have different attitudes toward scientific criticism and how to handle it), pragmatic (e.g., some disciplines have a greater need for critique than others because of lower quality standards), bureaucratic (e.g., availability of resources to support postpublication critique), historic (e.g., particular individuals or events that have highlighted the value of postpublication critique), or reputational (e.g., a perception that postpublication critique could damage journal reputation). Other factors that may deter submission of postpublication critique include fear of retaliation, such as public defamation and legal threats from individuals whose research they questioned (Besançon et al., 2022).
Our research has several limitations. First, our operational definition of postpublication critique was intentionally focused on journal accountability, and therefore, some types of critical discourse (e.g., externally hosted critiques or criticisms embedded in research or review articles that do not target a specific article) were beyond the scope of this study. Our definition also excluded errata, corrections, and retractions, some of which may have been motivated by postpublication critiques that were ultimately not published. Second, we relied on policy information explicitly published on journal websites; it is possible that some journals accept postpublication critique even though they do not have an explicit policy saying so on their website. Third, it is possible that we missed some examples of postpublication critique because they were not clearly linked to the target article; we did our best to minimize this possibility by having two coders independently assess the presence of critique for each article. Fourth, our study focused on constraints imposed by journals on postpublication critique; however, there are likely many contributing factors to the low prevalence of critique, such as authors’ ability and willingness to write and submit critiques and the extent to which critiques are recognized in hiring, promotion, and funding decisions.
Conclusion
Since 2011, psychology has been navigating a period of considerable epistemological turmoil, with serious concerns raised about research quality (Nosek et al., 2022). Postpublication critique is a valuable mechanism for identifying problems in published research and highlighting them to readers (Hardwicke et al., 2022; Merton, 1973; Vazire & Holcombe, 2022). In the present study, we show that postpublication critique is seriously neglected in policy and practice at psychology journals. Most journals do not have explicit formats for the submission of postpublication critiques, and critiques are very rarely published. In Box 1, we provide some tentative policy suggestions that may help to improve this situation.
Recommendations for Journal Policies That May Facilitate Postpublication Critique (Adapted From Hardwicke et al., 2022)
Supplemental Material
sj-docx-1-amp-10.1177_25152459251366102 – Supplemental material for How Do Psychology Journals Handle Postpublication Critique? A Cross-Sectional Study of Policy and Practice
Supplemental material, sj-docx-1-amp-10.1177_25152459251366102 for How Do Psychology Journals Handle Postpublication Critique? A Cross-Sectional Study of Policy and Practice by Annie Whamond, Simine Vazire, Beth Clarke, Nicholas Moodie, Sarah Schiavone, Robert T. Thibault and Tom E. Hardwicke in Advances in Methods and Practices in Psychological Science
Footnotes
Acknowledgements
A. Whamond thanks her father, Robert Whamond, for providing technical assistance and support with software issues that arose during this project.
Transparency
Action Editor: Katie Corker
Editor: David A. Sbarra
Author Contributions
ORCID iDs
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
