Sage Journals: Discover world-class research

Abstract

Psychological scientists are increasingly using preregistration as a tool to increase the credibility of research findings. Many of the benefits of preregistration rest on the assumption that preregistered plans are followed perfectly. However, research suggests that this is the exception rather than the norm, and there are many reasons why researchers may deviate from their preregistered plans. Preregistration can still be a valuable tool, even in the presence of deviations, as long as those deviations are well documented and transparently reported. Unfortunately, most preregistration deviations in psychology go unreported or are reported in unsystematic ways. In the current article, we offer a solution to this problem by providing a framework for transparent and standardized reporting of preregistration deviations, which was developed by drawing on our own experiences with preregistration, existing unpublished templates, feedback from colleagues and reviewers, and the results of a survey of 34 psychology-journal editors. This framework provides a clear template for what to do when things do not go as planned. We conclude by encouraging researchers to adopt this framework in their own preregistered research and by suggesting that journals implement structural policies around the transparent reporting of preregistration deviations.

Keywords

preregistration deviation open science transparency

Over the past decade, new research practices have been proposed to increase the replicability and credibility of psychological research (Vazire, 2018), including preregistration (Moore, 2016; Simmons et al., 2011; van’t Veer & Giner-Sorolla, 2016). Preregistration involves specifying research questions, hypotheses, methods, analytic approaches, and/or inferential criteria before collecting or analyzing data (Nosek et al., 2018; van’t Veer & Giner-Sorolla, 2016; Weston et al., 2019). Before 2011, preregistration was extremely rare in psychological science (Simmons et al., 2021), and although preregistration is far from common practice in the field (Hardwicke et al., 2022), usage is increasing (Nosek & Lindsay, 2018). When preregistered plans are followed perfectly, preregistration offers several benefits that decrease risk of bias and the likelihood of false-positive results and increase the credibility of the scientific literature (Nosek et al., 2018). However, preregistrations are rarely followed perfectly (Claesen et al., 2021; Heirene et al., 2021; Ofosu & Posner, 2021), and there are many legitimate reasons that a researcher may deviate from their preregistered plans. We define a deviation as any discrepancy between what the authors said they would do in the preregistration and what the authors actually did in the final article. Although preregistration allows readers and reviewers to distinguish between what was planned and unplanned, this benefit can be realized only if deviations are transparently reported.

In this article, we begin by explaining why a lack of transparency in reporting preregistration deviations is a problem and propose a standardized template to report deviations as one potential solution to this problem. Then, we describe our process for developing a standardized preregistration-deviations template. As part of this process, we conducted a survey of 34 psychology-journal editors to understand current editorial practices and their perceptions of preregistration deviations and to gather feedback on an initial draft of our template. After reporting the results of this survey, we present our template and make specific recommendations for how researchers and journals can adopt this framework to systematize the reporting of preregistration deviations. Finally, we conclude by addressing other questions that readers might have about preregistration deviations, such as how to prevent deviations, when to deviate, when it may be better to update an existing preregistration or create a new preregistration, and what to do when there are numerous deviations. The ultimate goal of the framework presented in this article is to help researchers fine-tune their preregistration skills to reduce deviations while also providing a simple and transparent way to report deviations when they do occur.

The Problem and Opportunity

Despite most scientists’ best efforts and intentions, it is sometimes necessary or preferable to deviate from preregistered plans, whether because of new innovations, data-collection errors, responding to reviewer requests, or simply correcting meaningful typos. We are not the first to document this observation or the first to make calls for preregistration deviations to be transparently reported (e.g., Brodeur et al., 2022; Campbell et al., 2023; Claesen et al., 2021; Nosek et al., 2018, 2019; Simmons et al., 2021). However, currently, there is no standardized way to report deviations from preregistrations or to evaluate the impact of those deviations on study results. Researchers tend to adopt different methods for reporting deviations, which we believe hinders transparency because there is no systematic way for reviewers, editors, and/or readers to identify and evaluate deviations. To illustrate, we share some approaches that we have used in our own research, such as uploading separate preregistration-deviation lists to OSF (e.g., Atherton et al., 2022, 2023), weaving deviations throughout the article (e.g., Willroth et al., 2021), creating tables that describe deviations by research question (e.g., Willroth et al., 2020; Willroth, Hill, et al., 2023), and reporting deviations in footnotes (e.g., Willroth, Luo, et al., 2023). In addition, many researchers do not report deviations at all (e.g., see Brodeur et al., 2022; Claesen et al., 2021). In fact, of 27 preregistered studies published in Psychological Science, 93% included at least one deviation from the preregistration (Claesen et al., 2021). Of the 25 studies that included a deviation, 89% did not report all deviations, and 36% did not report any deviations. When asked, authors reported that they did not disclose deviations from the preregistration because the deviations were “suggestions from reviewers, honest mistakes, such as typos in the preregistration plan, or that they simply considered certain details as too irrelevant to report either in the preregistration plan or in the paper” (Claesen et al., 2021). Therein lies both a problem and a unique opportunity in the current system.

The lack of standardized reporting of preregistration deviations is a problem for authors, reviewers, editors, and scientific progress. Some of the main reasons authors report being reluctant to adopt preregistration include beliefs that deviations are simply not allowed, misperceptions that deviating from the preregistration defeats its purpose, or a lack of resources and clear guidelines to support preregistration (Simmons et al., 2021; Washburn et al., 2018). Likewise, many authors may not know how to weigh the costs and benefits of deviating or how to communicate those costs and benefits to others. As a result, we argue that a standardized approach would make open-science practices easier to adopt among preregistration veterans and newcomers alike because it normalizes the occurrence and reporting of deviations and makes those deviations transparent for readers.

The lack of standardized reporting of preregistration deviations also poses notable problems for peer reviewers and editors who are already overburdened by the system. Presently, it is up to reviewers and editors to review any preregistration(s) at will and then decipher whether and in what ways the manuscript differs from the preregistration. This is inefficient and costly for reviewers and editors, who often do not have hours to spend identifying the differences across multiple documents. Standardized reporting of preregistration deviations would help reviewers and editors evaluate work more easily. Finally, a lack of standardized reporting is a problem for scientific progress in the field. Preregistrations are more valuable when they are specified in detail and followed perfectly and/or when deviations are transparently reported. If readers are not able to easily discern adherence or nonadherence to the preregistration, this system may erode trust among members of the field and undermine the credibility of psychological science. In sum, developing a tool for standardized reporting of preregistration deviations has the potential to make open science easier to adopt, implement, and evaluate.

Developing a Tool for Standardized Reporting of Preregistration Deviations

The motivation for developing a standardized mechanism of reporting preregistration deviations stemmed from our personal experiences with preregistration. We found that it was difficult to identify the most transparent way to report deviations in our own work and to identify what, where, and when preregistration deviations occurred in the articles for which we served as peer reviewers. Thus, to develop a tool for standardized reporting of preregistration deviations, we drew on our own experiences with preregistration and existing unpublished templates (Brodeur et al., 2022; Campbell et al., 2023; Claesen et al., 2021; Nosek et al., 2018, 2019; Simmons et al., 2021), that led us to create a “Preregistration Deviations Table” template. On the basis of reviewer feedback on this article, we surveyed editors of psychology journals to understand current editorial perceptions and practices concerning preregistration deviations and to gather feedback on an initial draft of our template (the initial draft is available on our OSF page at https://osf.io/yhzdc). We then refined our template based on reviewer, colleague, and participant feedback. In the following sections, we describe the method and results of our editorial survey and present the final template for reporting preregistration deviations.

Method

This study was approved by the Washington University in St. Louis Institutional Review Board (project title: “Preregistration Survey”; IRB 20230922). We conducted a preregistered survey of head editors and action editors from 16 psychology journals. We report how we determined our sample size, all data exclusions, and all measures in the study. The preregistration, survey items, data, and analytic code are available on OSF (https://osf.io/et6km/). We have no deviations from the preregistration to report. The survey asked editors about their perceptions and editorial practices concerning preregistration deviations, including prevalence of deviations, formal evaluation mechanisms, and justifiability of deviations, and their own experiences with preregistration and demographic characteristics. The items and corresponding response options are described before each set of results in the Results section below.

Our primary sampling goals were to (a) obtain data from editors and associate editors at general psychology journals and subdiscipline-specific journals and (b) obtain data from journals that vary in the extent to which they adopt Transparency and Openness Promotion (TOP) guidelines/preregistration. To achieve these goals, we identified six primary subfields of psychology (cognitive psychology and neuroscience, clinical, developmental, health, industrial-organizational, and social-personality). For general psychology, we chose two journals that were associated with the Association for Psychological Science and two journals that have high TOP scores based on the following website: https://topfactor.org/journals?disciplines=Psychology. This led us to target editors and associate editors at Advances in Methods and Practices in Psychological Science, Psychological Science, Meta-Psychology, and Collabra. For each subdiscipline, we identified two journals that (a) have the highest impact American Psychological Association affiliation for the discipline and (b) was an Association for Psychological Science journal (if applicable) or the journal associated with the main professional society within that subdiscipline. This led us to the following list: Journal of Experimental Psychology: Learning Memory and Cognition; The Journal of Neuroscience; Journal of Psychopathology and Clinical Science; Clinical Psychological Science; Health Psychology; Psychosomatic Medicine; Journal of Applied Psychology; Industrial and Organizational Psychology: Perspectives on Science and Practice; Journal of Personality and Social Psychology; and Personality and Social Psychology Bulletin.

We recruited participants via direct emails with requests for survey participation. In total, we emailed 170 potential participants. The survey was completed in Qualtrics. Participation was completely voluntary, and no compensation was provided. We preregistered that we would keep data collection open for 1 week after the initial emails were sent, and if the total N was fewer than 100 survey responses, we would send a reminder email and extend data collection for 3 additional days. After sending a reminder email and waiting 3 days, we closed data collection. Forty-four editors consented to participate in the study. Based on preregistered exclusion criteria, four participants were excluded for completing less than 10% of the survey, and six additional participants were excluded for spending less than 2 min completing the survey. The final analytic sample included 34 editors.

Results

Participants

The sample included 32% cognitive psychologists/neuroscientists, 18% clinical psychologists, 15% developmental psychologists, 6% health psychologists, 3% industrial-organizational psychologists, 41% social-personality psychologists, 9% who identified with a different psychological subdiscipline such as methodology/ metapsychology, and 4% who did not report their subdiscipline. Participants were allowed to select multiple subfields. Three percent of participants were postdoctoral researchers, 3% were assistant professors, 26% were associate professors, 53% were full professors, 3% held a position other than those listed here, and 12% did not report their academic position. Six percent of participants had been a journal editor for less than 1 year, 21% had been a journal editor for 1 to 2 years, 35% had been a journal editor for 2 to 5 years, 18% had been a journal editor for 5 to 10 years, 9% had been a journal editor for 10+ years, and 12% did not report the length of their editorship. In their role as editor, participants reported that, on average, 44% (SD = 22%, range = 0%–90%) of the manuscripts that they handled in the past 2 years were preregistered. In addition, participants reported that they preregistered, on average, 60% (SD = 31%, range = 0%–100%) of their own research.

Editorial perceptions

On average, editors estimated that 82% (SD = 22%, range = 24%–100%) of preregistered studies in psychology contain at least one deviation from the preregistered plan and that 55% (SD = 25%, range = 9%–100%) of preregistered studies in psychology report at least one deviation from the preregistered plan. For comparison, Claesen and colleagues (2021) identified at least one preregistration deviation in 93% of Psychological Science studies with preregistration badges; 64% of those studies reported at least one deviation.

We instructed editors to “imagine that the authors of a preregistered study deviated from their preregistered plan in one or more ways. The authors report the deviation(s) along with an explanation for why they deviated in the manuscript.” On a scale from 1 (significant negative impact) to 5 (significant positive impact), editors indicated that reported deviations would not affect their perception or slightly positively affect their perception of the manuscript, on average (M = 3.3, SD = 1.3, range = 1–5). Thirty-eight percent of editors noted that “it depends,” listing reasons such as the magnitude of the deviation, the plausibility that the deviation was strategic, and whether the authors appropriately calibrated their conclusions. On a scale from 1 (not at all) to 3 (a lot), editors indicated that the reported deviations would somewhat affect their editorial decisions, on average (M = 1.9, SD = 0.5, range = 1–3). Fifty-nine percent of editors noted that “it depends,” listing reasons such as the significance of the deviation, whether the deviation was an improvement or deterioration of the plan, and whether the deviation was transparently disclosed and justifiable.

Next, we asked editors to consider the same scenario except the authors did not report the deviation(s) and that instead, the deviation(s) was identified during the peer-review process. On a scale from 1 (significant negative impact) to 5 (significant positive impact), editors indicated that the reported deviations would significantly negatively affect their perception of the manuscript, on average (M = 1.3, SD = 0.5, range = 1–3). Six percent of editors noted that “it depends,” listing reasons such as the severity, scale, and nature of the deviations. On a scale from 1 (not at all) to 3 (a lot), editors indicated that the reported deviations would affect their editorial decisions somewhat to a lot, on average (M = 2.6, SD = 0.5, range = 2–3). Twelve percent of editors noted that “it depends,” listing reasons such as the size of the deviation, the effect of the deviation on the results, whether the deviation seemed likely to be motivated by bias, and how likely it is that authors consciously decided to not be transparent about the deviation.

We then asked participants to rank-order several factors that may affect their editorial decision on a manuscript that contains preregistration deviations. For a plot of the mean ranked values, see Figure 1. Editors were also asked to rate the extent to which several specific types of deviations and reasons for deviating were justifiable on a scale from 1 (never justifiable) to 4 (always justifiable). The means and standard deviations for each preregistration type and reason are plotted in Figures 2 and 3. In addition, participants were given the option to select “N/A: I would not consider this a deviation.” The percentages of participants that selected this option are shown in Figures 2 and 3.

Fig. 1.

Mean rankings of the extent to which each factor would affect editorial decisions on a manuscript containing preregistration deviations. Error bars depict 1 SD. Values closer to 1 indicate higher perceptions of impact on editorial decision.

Fig. 2.

Mean justifiability ratings for different types of preregistration deviations. Error bars depict 1 SD.

Fig. 3.

Mean justifiability ratings for different reasons for preregistration deviations. Error bars depict 1 SD.

Editorial practices

In their role as editors, participants indicated that they personally evaluated the extent to which authors followed their preregistered plan 65% of the time, on average (SD = 34%, range = 9%–100%). Likewise, in their role as reviewers, participants indicated that they personally evaluated the extent to which authors followed their preregistered plan 65% of the time, on average (SD = 33%, range = 9%–100%).

Next, we asked editors whether their journal had any formal mechanisms for evaluating the extent to which authors followed their preregistered plan. Twenty-four percent reported that there are instructions on how to report deviations in the author guidelines. Three percent reported that authors are prompted to report preregistration deviations in the submission portal. Nine percent reported that reviewers are instructed to check for preregistration deviations. In addition, 6% of participants reported other formal mechanisms, such as including statistical reproducibility check for undisclosed deviations, as a norm during the initial manuscript-triage process. Thirty-two percent reported that they did not know if there were any formal mechanisms at their journal, and 35% reported that there are no formal mechanisms at their journal.

At the end of the survey, we shared an initial draft of our preregistration-deviations template with participants and asked for their feedback. All anonymous open-ended responses containing feedback are included on the OSF project page. In response to participant feedback, we modified the original template in the following ways: (a) added a section for authors to describe “unregistered steps” to outline aspects of the preregistration that were ambiguous and necessitated a decision that was not specified in advance; (b) added a sentence to the table note indicating that if more than one type or reason applies to the same deviation, then authors can replace the drop-down menu with a narrative account; and (c) revised some minor wording choices (e.g., difference between typo vs. oversight vs. error). Finally, we asked whether they would support a policy at their journal that would require authors to transparently report any deviations from their preregistered plans using this template. On a scale from 1 (I would definitely NOT support this policy) to 5 (I would definitely support this policy), editors reported, on average, that they would probably support such a policy (M = 4.1, SD = 1.0, range = 2–5).

A Template for Reporting Preregistration Deviations

To ensure that deviations are reported in a transparent and standardized way, we created a Preregistration Deviations Table that authors can complete and include in their manuscripts. For a completed example based on deviations from our own preregistrations, see Table 1. A blank version of the table is available in Table S1 in the Supplemental Material available online and as a downloadable Google doc at https://docs.google.com/document/d/1m7k53z38w18AJe56ucftunnHuFM7wDlMFjpoGenwN6k/edit?usp=sharing. The table template specifies each deviation in a separate row, providing a straightforward way for reviewers, editors, and readers to quantify the number of deviations. Then, authors select from drop-down menus to aid readers in understanding how the deviation could be characterized in terms of type, reason, and timing. Finally, there are columns for authors to include the original text of the preregistration, a description of the deviation, and a reader-impact statement. In addition to deviations, we also included a place in the table to report unregistered steps (or unspecified decision points) that may result from a vague or underspecified preregistration.

Table 1.

Example Preregistration Deviations Table Template

Deviations
No.	Details		Original wording	Deviation description	Reader impact
1	TypeReasonTiming	VariablesMiscommunicationAfter data access	“Before conducting the models, we will center the data at age 26 (the age of the youngest participant at the first assessment) and divide by 10 to scale the coefficients to be equal to change over one decade and facilitate interpretation.”	There was a miscommunication about centering age when creating the data analysis plan, and the preregistration initially misstated how age was going to be specified. As stated in the preregistration, age was centered by subtracting age scores by 26 (the youngest participant in the sample at the first assessment) to scale the latent growth curves by the earliest point in development. However, age centering was not divided by 10.	This deviation has a small impact on readers’ interpretations of the findings. This deviation was due to a miscommunication when writing the preregistration, which was corrected when preparing the data before analysis. The results were not already known, so the risk of bias is low. In addition, dividing age by 10 is a linear transformation of the variable, so the direction, magnitude, and statistical significance of the effects would not be different regardless of whether we divide by 10 or not. It changes only the units that the resulting coefficients are expressed in.
Unregistered steps
No.	Details		Original wording	Unregistered step description	Reader impact
1	TypeTiming	VariablesAfter results known	“Rurality: Will be recoded into three categories: RUCCs 1-3, RUCCs 4-6, RUCCs 7-9. For 1993, we will recode ‘0s’ → ‘1s’ to be on a comparable scale to the 2003 and 2013 RUCCs.”“We will use a series of seven random-intercept multilevel models to examine how rurality predicts each Big Five trait and each well-being outcome in separate models. Rurality will be entered as a level 1 predictor in each model and allowed to vary across timepoints (i.e., if individuals move counties between timepoints). In a second series of models, we will add socio-demographic covariates. In a third set of models, we will add socio-demographic covariates as well as social network covariates.”	In the original preregistration, we did not specify whether we were going to look at our three ordered RUCC categories as a continuous or categorical variable, which affects the types of results that lmer outputs for the multilevel models. Because of a miscommunication, the first author ran the models as continuous variables first, before realizing that the original intention was to run them as factor/categorical variables. We report the categorical results in the text, and we note in Footnote 5 that: “We also repeated all analyses while specifying rurality as a continuous variable rather than a categorical variable, and all of the results mirror the patterns of significant and nonsignificant findings we observed when examining rurality as a categorical variable (with three groups).”	Overall, this unregistered step may affect readers’ interpretation of the findings because the ambiguity in how to treat the variable in R affects how the results are presented, and how the results were presented changed after coauthors saw the results. Ultimately, we present the findings in both ways, when RUCC is treated as a categorical and continuous variable. The direction, magnitude, and statistical significance of the effects do not change irrespective of whether we treat RUCC as a categorical or continuous variable. However, the categorical reporting of RUCC is slightly more nuanced because it allows readers to see whether any nonlinear effects were evident. Thus, the ambiguity in the preregistration likely has little impact on the current results and paper. However, it is important for future researchers to consider this issue when studying RUCC in the future. And we now understand how to include this analytic approach in future preregistrations.

Note: Choose one characteristic for each drop-down menu. If more than one type or reason applies to the same deviation, then replace the drop-down menu with details. Provide one deviation per row and add or delete rows as needed.

To ensure that all deviations are identified and reported, researchers should document deviations throughout the research process and carefully review their preregistration at the end of the research process. Researchers should disclose all deviations and unregistered steps in the Preregistration Deviation Table so that reviewers and readers do not need to compare documents or search for this information. When page and table limits permit, we encourage researchers to include the Preregistration Deviations Table in the main body of their manuscript or appendix. When it is not possible to include the Preregistration Deviation Table in the main manuscript (e.g., because of journal restrictions), researchers should provide the table in supplemental online material and reference it in the main text of the manuscript with a summary of deviations.

In addition to transparently reporting preregistration deviations, whenever possible, researchers should also include the results of the preregistered analyses alongside the deviated analyses, for example. In some cases, it may be appropriate to include these results in the supplemental online material and to summarize them in text. This will help researchers and readers evaluate how the deviations may have affected conclusions. In the section that follows, we describe each of these aspects of reporting in more detail and provide examples.

Type of deviation

There are a number of different types of deviations that could occur throughout the research process, including changes to the study design, inclusion/exclusion criteria, research questions, hypotheses, sample size, data preparation, variable operationalizations or computations, analytic approach, covariates, or inferential criteria. Table 2 provides examples of each of these types of deviations.

Table 2.

Types of Deviations

Type	Example(s)
Study design	• Change to wording of measure/manipulation• Added new measurement occasions, control sample, or data set
Inclusion/exclusion criteria	• Modified criteria for passing a manipulation check• Modified criteria for what is considered an outlier• Adapted age restrictions, handedness, demographic qualifiers, etc.
Research question(s)	• Added or removed research question(s)• Modified original research question(s)
Hypotheses	• Added new hypotheses• Did not test original hypotheses
Sample size	• Collected fewer/more participants than planned
Data preparation	• Change to the data preprocessing procedure
Variable operationalization or computation	• Changed what was considered the primary operationalization of independent variable or dependent variable• Combined or removed indicators• Used latent variables instead of observed variables
Analytic approach	• Conducted a different type of analysis• Used a different software or package• Used a different estimator• Chose a different method for handling missing data (e.g., imputation vs. maximum likelihood)
Covariates	• Included new covariate(s)• Did not include planned covariate(s)• Used a different operationalization of covariate (e.g., education vs. income)
Inferential criteria	• Changed alpha level or sidedness (e.g., one- vs. two-sided test)• Changed p value correction method (e.g., false-discovery rate vs. Bonferroni)• Changed confidence interval reporting

Reason for deviation

Deviations from preregistrations can occur because of typos or oversights in the preregistration, data-collection or documentation errors, the preregistered plan was not possible or was inappropriate because of characteristics of the data, the researchers or the field gained new knowledge, new suggestions by peer reviewers/editors, or miscommunications among the coauthor team at the time of preregistration. Table 3 provides examples of each of these reasons for deviations.

Table 3.

Reasons for Deviations

Reason	Example(s)
Typo/oversight	• Mistyped a word in the preregistration, which changed the meaning of sentence• Did not consider a logical or logistical problem in one’s preregistered approach
Data-collection or documentation error	• One or more variables was not collected• Item wording, response options, or manipulations differed from the preregistered plan
Preregistered plan not possible or inappropriate because of data	• Cell sizes are too small to analyze a categorical variable as planned• Extreme restriction of range for a continuous variable• Model assumptions are violated
Researchers gained new knowledge	• The researchers learned that their analytic approach was not appropriate for their research question or their data• The researchers learned about best practices they were not previously aware of• The researchers discovered an existing analytic approach they were not previously aware of• A new analytic approach was developed that improves on limitations of the preregistered approach• Researchers learned that a given measure does not adequately assess their construct of interest or has very poor psychometric properties
Suggested by reviewer/editor	• A reviewer or editor identifies a problem or limitation in the preregistered approach• A reviewer or editor suggests an alternative approach based on their preferences or perceptions of field norms• A reviewer or editor suggests an additional supplemental or sensitivity analysis
Miscommunication among coauthor team	• One or more coauthors did not read or agree on an important decision point before preregistering• One or more coauthors was added to the project after preregistering and offers new expertise or conflicting opinions about the best analytic approach

Timing of deviation

The timing of when the deviation occurred can affect the diagnosticity and credibility of the presented results (e.g., Hardwicke & Wagenmakers, 2023; Nosek et al., 2018). Deviations can occur at any point in the research process, including before data were collected, during data collection, after data collection but before data were accessed, after data were accessed but before results of preregistered approach were known, and after results of preregistered approach were known. It is important to transparently report the timing of when the deviation was recognized or occurred because data-dependent deviations increase risk of bias (Hardwicke & Wagenmakers, 2023).

Reader impact

The extent to which deviations from the preregistration affected study results (e.g., magnitude and direction of effect sizes) and whether and how they may have affected the diagnosticity of study results (e.g., control over researcher degrees of freedom and false positives) may affect how readers interpret the findings of preregistered research that includes deviations. For example, when numerous substantive deviations occur, readers may choose to evaluate the evidence in a similar manner to how they would evaluate exploratory or unregistered research.

Unregistered steps

In addition to deviations, there may be instances in which the preregistration did not consider a particular step/decision point or lacked sufficient detail (e.g., indicated parent education level would be used but did not specify how mother and father education level would be combined). These unregistered steps should be reported in a separate section after deviations in the Preregistration Deviations Table.

Other Considerations

What is considered a deviation?

The line between what is considered deviation is debatable. Is a typo in the preregistration that is corrected in the final manuscript considered a deviation? Is lack of specificity in the preregistration that requires the researcher to make an unregistered analytic decision considered a deviation? As Figures 2 and 3 show, even psychology-journal editors have different opinions about what counts as a deviation. Given the inherent subjectivity in this determination, we recommend authors err on the side of caution and transparently report anything that could be perceived as a deviation from the original plan. For example, a typo may not necessarily be a deviation in the same way that changes to study protocol or variable operationalizations are, but typos that change the meaning of a statement are still worth reporting so that readers can be assured that these oversights in the preregistration were identified by the study team. As others have discussed (e.g., Nosek et al., 2018), although lack of specificity resulting in an unregistered analytic decision may not be a deviation per se, it should still be transparently reported. There are certainly instances in which researchers could not possibly anticipate all forking paths that could occur in the research process, and in these cases, we encourage researchers to make their preregistrations and backup plans as detailed as they possibly can to avoid unregistered steps. For example, if researchers did not specify how they were going to treat missing item-level data when computing a sum score, then this should be transparently reported as unregistered steps because it introduces researcher degrees of freedom that occurred during the analytic process and could have been specified in advance. Or lack of specificity might occur when a researcher does not create three different backup plans for the problems that might occur when using structural equation modeling. If variables are not distributed appropriately, latent variable models do not fit, and model assumptions are violated, a researcher may need to change course in a way that was not anticipated. Future research should investigate which areas of preregistration researchers have the most trouble specifying details about and develop tools to facilitate specificity in preregistrations, beyond templates. As a result, we find it important to highlight that there may be clear “deviations” (e.g., changed analytic model), whereas others may be unregistered steps (e.g., lack of specificity in preregistration), but both are important to report in a standardized way for readers to evaluate. In addition to deviations and unregistered steps, researchers may choose to include additional research questions or analyses beyond those that they preregistered. These should be clearly labeled as unregistered when they are first presented in text.

Preventing deviations and unregistered steps

Although deviations are sometimes necessary and unregistered steps may happen, it is important to acknowledge that there are several steps researchers can take to reduce the need for deviations and unregistered steps in the first place. First, a certain degree of prep work may be necessary to write a preregistration that is unlikely to be deviated from. For example, researchers should review the relevant literature and familiarize themselves with the details of their planned analytic approach before preregistering. Note that this prep work will save time during the analysis and writing phases, so it should not add to the total amount of time needed to complete a project. Second, preregistrations should be sufficiently detailed to reduce the likelihood that researchers will face analytic choices that were not addressed in the preregistration (Claesen et al., 2021). This can be accomplished by using detailed preregistration templates and by supplementing those templates with knowledge of one’s specific research area and typical research process. Third, researchers may consider documenting and registering standard operating procedures for their lab or for a given data set (Lin & Green, 2016; Nosek et al., 2018; Tackett et al., 2020). This can prevent unregistered steps that arise from failures to document analytic steps that may be considered routine to some researchers, such as treatment of duplicate respondents in survey research (e.g., Claesen et al., 2021). Fourth, researchers should consider potential forking paths in which analytic decisions depend on currently unknown features of the data and preregister decision trees that outline how those analytic decisions will be made (Nosek et al., 2018). For example, use of measures may depend on their reliabilities, or the selection of statistical tests may depend on whether model assumptions are met or violated. Fifth, researchers can consider drafting the analytic code in advance and including it as a supplemental file with the preregistration because doing so can make some unregistered steps of the preregistration more apparent (e.g., Graham et al., 2022; Willroth et al., 2022). In addition, when skill sets allow, it may be helpful to use the drafted analytic code to conduct simulations because simulations can aid in visualizing potential data patterns that affect preregistered plans. Finally, collaborators should read and provide feedback on preregistrations the same way that they would for a final manuscript. Because researchers differ in their familiarity with and use of preregistration, it may be beneficial for research teams to discuss the importance of preregistration and their plans for how to handle preregistration deviations before finalizing the time-stamped document.

In addition to considering strategies that researchers can engage in to prevent deviations and unregistered steps, it may also be useful to consider contextual factors that may affect the likelihood of deviations. For example, Claesen et al. (2021) identified a handful of studies in which preregistrations were followed exactly (Study 4, Hawkins et al., 2015; Pilot + Study 1, Pittarello et al., 2015). What makes these preregistrations different from preregistrations in which deviations occurred? The most defining features of the aforementioned examples may be that they are extensions of prior studies and/or the investigations were fairly simple in study design and analytic approach (e.g., analysis of variance). This points to the possibility that study or analytic complexity may be one reason for preregistration deviations. Future research should empirically examine how studies with no deviations differ from studies with deviations and to innovate solutions so that simple and complex projects alike can adhere to preregistrations with ease.

When should researchers deviate?

As can be seen from editors’ perceptions of the justifiability of different types of (and reasons for) deviations in Figures 2 and 3, the question of when to deviate is not straightforward to answer. In this case, it is useful to consider a spectrum of deviations, from strictly necessary deviations to completely arbitrary deviations, and the timing of the deviation. Risk of bias increases depending on when the deviation occurs (e.g., after the researcher has already seen the data; Hardwicke & Wagenmakers, 2023). At one extreme, some deviations may be necessary to carry out any analyses at all. For example, a researcher may plan to use a specific variable and later find out that the variable was not collected because of an error in data collection or data documentation. Other deviations may not be strictly necessary but may unequivocally improve the quality of the research. For example, researchers may learn that their planned analysis does not match their research question, or the data may violate important model assumptions. In both cases, deviations are likely to be justifiable and should be reported transparently. Moreover, depending on the type and timing of the deviation, there may be ways for researchers to safeguard against risk of bias from deviations. For example, researchers could consider having an independent statistician read their preregistration and provide feedback on the analytic plan to reduce risk of bias (e.g., Hardwicke & Wagenmakers, 2023). Next, one can consider deviations that are not strictly necessary and that do not unequivocally improve the quality of the research but may be preferred by some. For example, a peer reviewer may recommend that a researcher use the reviewer’s preferred statistical approach or variable operationalization. Often, this situation can be handled by following the preregistered plan in primary analyses and then including an additional supplemental or sensitivity analysis that follows the alternative plan. Of course, this may not be possible in all cases (e.g., smaller sample size than expected). But regardless of whether the alternative choice is testable, it is necessary to provide a narrative account of the meaning of the deviation (and overall risk of bias) in the reader-impact column of the Preregistration Deviation Table. Finally, some deviations are arbitrary, increase risk of bias, and are likely difficult to justify. For example, changing the alpha level or sidedness of a preregistered analysis (e.g., from a two-tailed to a one-tailed test). In instances such as this, deviations should be avoided. In sum, researchers can take several steps to reduce the occurrence of deviations, but deviations may still be necessary, and unregistered steps may need to be reported. When deviations are not strictly necessary, researchers should consider whether the deviation unequivocally improves the quality of the research. If either criterion is met, the deviation is likely justifiable and should be reported transparently. If neither criterion is met, it may be better to stick to the preregistered plan and include the alternative analysis as supplemental or sensitivity analyses, if applicable. In all cases, unregistered steps, no matter how trivial they seem, should be reported, too.

When to deviate versus update versus create a new preregistration?

Some preregistration repositories provide the option to “update” a preregistration. For example, at the time of this writing, OSF allows researchers to create a time-stamped update to their preregistration, including a description of the change, the rationale for the change, and the impact of the change on the study. The OSF support page instructs researchers to use this feature for “events outside your control” and “unexpected anomalies.” This feature may also be useful to correct accidental omissions from the preregistration before data collection or data analysis begin. Once results are known, preregistrations should not be updated. Instead, researchers should report their deviations or unregistered steps using the Preregistrations Deviations Table in the resulting article. This ensures that results-dependent changes to the preregistered plan are transparently documented in the final publication. In the event of major changes to a preregistration, researchers may also consider registering a supplemental registration or coregistration that outlines the planned deviations, including when during the research process these changes were implemented (Benning et al., 2019; Kirtley et al., 2021). Finally, consider a scenario in which a reviewer suggests that the researcher adds another study to their manuscript. Because the researcher has not conducted the study yet, it would be appropriate to create a new preregistration for the new study and to document that the latter study was preregistered after results from the first study were known.

When are there too many deviations?

There is no clear-cut answer to this question, but a good rule of thumb is to overreport rather than underreport. The utility of a preregistration is transparency for readers, but risk of bias occurs when there are deviations from the preregistration (Hardwicke & Wagenmakers, 2023), particularly when deviations occur after looking at the data. We encourage researchers to use the Preregistration Deviations Table to report all deviations no matter the number. We also encourage researchers to always disclose if there was a preregistration, even if they feel that there were too many deviations for the research to be considered preregistered. Although it might be misleading to apply for a preregistration badge when the number of deviations and data-dependent decisions renders risk of bias high, it would also be misleading to not report that there was a preregistration in the first place. At the end of the day, preregistration is a skill, and preregistrations are likely to be messy while that skill is being honed (Kirtley et al., 2021). We hope that researchers will also be able to learn from filling out the Preregistration Deviations Table such that they will recognize where they need to be more specific in future preregistrations or how they can anticipate analytic violations more directly. Preregistration is still a valuable tool for distinguishing between planned and unplanned steps in the research process, even when many deviations are made. As Nosek and colleagues (2019) put it: “Having some plans is better than having no plans, and sharing those plans in advance is better than not sharing them” (p. 817).

Concluding Remarks

To maximize the benefits of preregistration, researchers should take steps to ensure that they follow their preregistered plan closely and to reduce the need to deviate from that plan. However, even the best laid plans do not work out sometimes. Preregistration can still be a valuable tool for increasing the credibility of scientific findings so long as preregistration deviations are transparently reported. In the current article, we offer recommendations to help researchers determine when a deviation from preregistered plans is necessary or potentially justifiable and provide a framework to standardize the transparent reporting of preregistration deviations and unregistered steps. We encourage researchers to adopt this framework in their own preregistered research. To support transparent reporting of preregistration deviations, we also urge reviewers and editors not to penalize authors simply for reporting preregistration deviations. As the results of our editorial survey showed, editors already have neutral to slightly positive perceptions of disclosed deviations on average compared with significantly negative perceptions of undisclosed deviation on average. Likewise, among seven different factors, including the number of deviations and extent to which the deviation affected substantive conclusion, editors ranked “the extent to which authors were transparent” as the top factor for influencing their editorial decisions on average. Thus, reporting all deviations is beneficial to authors, editors, readers, and the field alike. Finally, we call on journals to implement structural-level policies that encourage transparent reporting of preregistration deviations. The adoption of this framework will provide researchers with a clear template for what to do when things do not go as planned, alleviate burden on reviewers and editors, and increase the transparency and credibility of preregistered research.

Supplemental Material

sj-docx-1-amp-10.1177_25152459231213802 – Supplemental material for Best Laid Plans: A Guide to Reporting Preregistration Deviations

Supplemental material, sj-docx-1-amp-10.1177_25152459231213802 for Best Laid Plans: A Guide to Reporting Preregistration Deviations by Emily C. Willroth and Olivia E. Atherton in Advances in Methods and Practices in Psychological Science

Footnotes

Transparency

Action Editor: David A. Sbarra

Editor: David A. Sbarra

Author Contributions

Both authors contributed equally and share first authorship. The order in which the authors are presented was decided by a coin flip.

Emily C. Willroth: Conceptualization; Data curation; Formal analysis; Investigation; Visualization; Writing – original draft; Writing – review & editing.

Olivia E. Atherton: Conceptualization; Data curation; Formal analysis; Investigation; Visualization; Writing – original draft; Writing – review & editing.

ORCID iD

Olivia E. Atherton

Supplemental Material

Additional supporting information can be found at

References

Atherton

O. E.

Sutin

A. R.

Terracciano

Robins

R. W.

(2022). Stability and change in the Big Five personality traits: Findings from a longitudinal study of Mexican-origin adults. Journal of Personality and Social Psychology, 122(2), 337–350.

Atherton

O. E.

Willroth

E. C.

Luo

Graham

E. K.

Mroczek

D. K.

Lewis-Thames

M. W.

(2023). Rural-urban differences in personality traits and well-being in adulthood. Journal of Personality. https://doi.org/10.1111/jopy.12818

Benning

S. D.

Bachrach

R. L.

Smith

E. A.

Freeman

A. J.

Wright

A. G. C.

(2019). The registration continuum in clinical science: A guide toward transparent practices. Journal of Abnormal Psychology, 128(6), 528–540.

Brodeur

Cook

N. M.

Hartley

J. S.

Heyes

(2022). Do pre-registration and pre-analysis plans reduce p-hacking and publication bias? (GLO Discussion Paper, No. 1147). Global Labor Organization.

Campbell

Harris

Flake

J. K.

Fried

E. I.

Beck

E. D.

Kline Struhl

Etz

Lindsay

D. S.

Feldman

van’t Veer

Vazire

(2023). Reporting deviations from pre-registration. OSF. https://osf.io/xv5rp/

Claesen

Gomes

Tuerlinckx

Vanpaemel

(2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), Article 211037. https://doi.org/10.1098/rsos.211037

Graham

E. K.

Willroth

E. C.

Weston

S. J.

Muniz-Terrera

Clouston

S. A.

Hofer

S. M.

Mroczek

D. K.

Piccinin

A. M.

(2022). Coordinated data analysis: Knowledge accumulation in lifespan developmental psychology. Psychology and Aging, 37(1), 125–135. https://doi.org/10.1037/pag0000612

Hardwicke

T. E.

Thibault

R. T.

Kosie

J. E.

Wallach

J. D.

Kidwell

M. C.

Ioannidis

J. P.

(2022). Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014–2017). Perspectives on Psychological Science, 17(1), 239–251.

Hardwicke

T. E.

Wagenmakers

E. J.

(2023). Reducing bias, increasing transparency and calibrating confidence with preregistration. Nature Human Behaviour, 7(1), 15–26.

10.

Hawkins

C. B.

Fitzgerald

C. E.

Nosek

B. A.

(2015). In search of an association between conception risk and prejudice. Psychological Science, 26(2), 249–252.

11.

Heirene

LaPlante

Louderback

E. R.

Keen

Bakker

Serafimovska

Gainsbury

S. M.

(2021). Preregistration specificity & adherence: A review of preregistered gambling studies & cross-disciplinary comparison. PsyArXiv. https://doi.org/10.31234/osf.io/nj4es

12.

Kirtley

O. J.

Lafit

Achterhof

Hiekkaranta

A. P.

Myin-Germeys

(2021). Making the black box transparent: A template and tutorial for registration of studies using experience-sampling methods. Advances in Methods and Practices in Psychological Science, 4(1). https://doi.org/10.1177/2515245920924686

13.

Lin

Green

D. P.

(2016). Standard operating procedures: A safety net for pre-analysis plans. Political Science & Politics, 49, 495–500.

14.

Moore

D. A.

(2016). Preregister if you want to. American Psychologist, 71(3), 238–239.

15.

Nosek

B. A.

Beck

E. D.

Campbell

Flake

J. K.

Hardwicke

T. E.

Mellor

D. T.

van ’t Veer

A. E.

Vazire

(2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23(10), 815–818. https://doi.org/10.1016/j.tics.2019.07.009

16.

Nosek

B. A.

Ebersole

C. R.

DeHaven

A. C.

Mellor

D. T.

(2018). The preregistration revolution. Proceedings of the National Academy of Sciences, USA, 115(11), 2600–2606.

17.

Nosek

B. A.

Lindsay

D. S.

(2018). Preregistration becoming the norm in psychological science. APS Observer, 31, 19–21.

18.

Ofosu

G. K.

Posner

D. N.

(2021). Pre-analysis plans: An early stocktaking. Perspectives on Politics, 21, 174–190.

19.

Pittarello

Leib

Gordon-Hecker

Shalvi

(2015). Justifications shape ethical blind spots. Psychological Science, 26(6), 794–804.

20.

Simmons

J. P.

Nelson

L. N.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. https://doi.org/10.1177/0956797611417632

21.

Simmons

J. P.

Nelson

L. N.

Simonsohn

(2021). Pre-registration: Why and how. Journal of Consumer Psychology, 31(1), 151–162.

22.

Tackett

J. L.

Brandes

C. M.

Dworak

E. M.

Shields

A. N.

(2020). Bringing the (pre)registration revolution to graduate training. Canadian Psychology/Psychologie Canadienne, 61(4), 299–309.

23.

van’t Veer

Giner-Sorolla

. (2016). Pre-registration in social psychology: A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.

24.

Vazire

(2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13(4), 411–417.

25.

Washburn

A. N.

Hanson

B. E.

Motyl

Skitka

L. J.

Yantis

Wong

K. M.

Sun

Prims

J. P.

Mueller

A. B.

Melton

Z. J.

Carsel

T. S.

(2018). Why do some psychology researchers resist adopting proposed reforms to research practices? A description of researchers’ rationales. Advances in Methods and Practices in Psychological Science, 1(2), 166–173. https://doi.org/10.1177/2515245918757427

26.

Weston

S. J.

Ritchie

S. J.

Rohrer

J. M.

Przybylski

A. K.

(2019). Recommendations for increasing the transparency of analysis of preexisting data sets. Advances in Methods and Practices in Psychological Science, 2(3), 214–227.

27.

Willroth

E. C.

Graham

E. K.

Mroczek

D. K.

(2022). Challenges and opportunities in preregistration of coordinated data analysis: A tutorial and template. Psychology and Aging, 37(1), 136–140.

28.

Willroth

E. C.

Hill

P. L.

Pfund

G. N.

Mroczek

D. K.

(2023). Perceptions of a good life: Associations with culture, age, wellbeing, and health. European Journal of Personality. Advance online publication. https://doi.org/10.1177/08902070231186344

29.

Willroth

E. C.

John

O. P.

Biesanz

J. C.

Mauss

I. B.

(2020). Understanding short-term variability in life satisfaction: The Individual Differences in Evaluating Life Satisfaction (IDELS) model. Journal of Personality and Social Psychology, 119(1), 229–248.

30.

Willroth

E. C.

Luo

Atherton

O. E.

Weston

S. J.

Drewelies

Batterham

Condon

D. M.

Gerstorf

Huisman

Spiro

Mroczek

D. K.

Graham

E. K.

(2023). Personality traits and healthcare use: A coordinated analysis of 15 international samples. Journal of Personality and Social Psychology, 125(3), 629–648. https://doi.org/10.1037/pspp0000465

31.

Willroth

E. C.

Smith

A. M.

Shallcross

A. J.

Graham

E. K.

Mroczek

D. K.

Ford

B. Q.

(2021). The health behavior model of personality in the context of a public health crisis. Psychosomatic Medicine, 83(4), 363–367.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB