Abstract
Peer review plays a critical role in much of the best scientific publishing. The best peer reviewers are both subject matter experts and masters of the rules and rhythms of scientific writing. True subject matter expertise can take years to develop, as can true mastery over scientific writing, but competent reviewers can emerge with more modest capabilities in both areas with appropriate training. It is common, however, for reviewer skills to be learned informally over time rather than through a formal training curriculum.
Journals rely on peer reviewers, but often with little knowledge of the capabilities of reviewers and almost always with no real control over their process. Although the value of peer review and editing ability may be recognized, there is no mandate on when or how the relevant skills should be acquired. Individuals actively engaged in research are more likely to refine their skills as they practice scientific writing, but those with a nonresearch focus or those who do not lead in scientific writing efforts may never get the grounding that would best serve them in this area.
One of the economical ways to help reviewers to develop their reviewing skills is to share the reviews and comments of all reviewers and editors who look at a given manuscript. The motivated reviewer will look to see if their recommendations matched the editorial decision and then look through the comments of others to try to understand any disagreements and/or to try to refine their insight and evaluation skills.
The sharing of review content can provide substantial learning potential with little extra work for journal editors or administrative staff, but it is a passive effort, relying on reviewers to self-assess their performance and hone their skills. Direct engagement with reviewers is often limited to egregious issues, possibly associated with decisions to remove individuals from reviewer rosters.
Regular, direct engagement with reviewers can amplify the lessons learned from shared reviews. The quality of an individual’s contribution can be frankly assessed, both to directly inform and to promote greater attention to the package. Candid comments can reassure or sting, with a common goal of motivating reviewers to strengthen their reviewing efforts.
Wilderness & Environmental Medicine implemented a reviewer-in-training (RIT) program in 2018 with the goal of complementing reviewer training workshops held at society conferences. The program was voluntary, with participants being promised a direct evaluation of their reviews by the editor-in-chief, and sometimes by associate or section editors. Comments were always intended to be concise to minimize the time required to produce them, with a focus on perceived shortcomings that could foreseeably be overcome. RIT reviews follow the standard 2 reviews of a manuscript, ultimately providing more feedback to authors.
Participation in the RIT program has increased steadily, from 24 individuals in 2018 to 77 in 2021. The greatest amount of feedback is usually provided in the first reviews of a manuscript, with subsequent comments variable as commensurate with the review content. In retrospect, a notable deficiency of the initial program was the undefined endpoint for participation. It was expected that individuals would remove themselves as they wished, and there was a vague plan that when an individual’s reviews were largely seen to be without important fault (a clearly subjective state), they would be “graduated.” These endpoints were not well communicated, though, and the reality was that determination of a call for graduation was rarely made, given the variable nature of every review and a reticence to arbitrarily push anyone out.
The problem with the lack of clarity regarding endpoints came out in 2021 when a participant expressed marked dissatisfaction regarding the value of the feedback. The participant’s choice to stay in the program to that point was probably due to an expectation that “graduation” was the only successful way to move forward. This prompted review and revision of the program description, with a new simple guideline stating that participation is voluntary and should only be continued as long as the feedback is thought to be beneficial. The goal is to help reviewers to develop their skill, but there is no passing metric, so control over the RIT designation is in the hands of the participant.
The response to the RIT program has been overwhelmingly positive, allowing individuals to gain confidence through the extra support. RIT reviews receive closer editorial scrutiny to avoid minefields, and authors may be a bit more forgiving of imperfections in reviewer tone or focus when seeing the trainee designation.
The RIT program will continue, along with scientific writing and reviewing workshops, as an investment in our community of reviewers and authors. Feedback will continue to be concise, with an intent to highlight areas where reviewer focus can enhance the quality of reviews. The goal is to have each review be as clear, informative, and respectful as feasible, to help authors bring out the best in their academic writing. We encourage other journals to consider similar programs, perhaps learning from our experience.