Reflections on the Registered Report Process for “Taking It to the Next Level: A Field Experiment to Improve Instructor-Student Relationships”

On the whole, we enjoyed and benefited from the Registered Report experience. We noted a couple of areas where the process differed—generally in a good way—from a more traditional journal submission.

First, the review process was particularly interesting. Contrasting this experience to the more common peer-review after submitting the complete paper, we as authors found that reviewers were much more focused on the merits of the study design. While this might be unsurprising given the structure of a registered report, it resulted in reviewers being much more humble in their recommendations. That is, because they did not have the benefit of knowing the results as they were evaluating the study, they had much fewer critiques about what they thought the researchers should have done. It was a very refreshing peer-review experience!

Second, the initial submission involves a lot more upfront work, but it resulted in less work later on in the process. It definitely streamlined the review process, because reviewers can provide helpful insights into analyses and ask for them upfront, resulting in less back-and-forth once the data analyses have been completed.

Finally, one thing that we believe the Registered Report format will have to grapple with is how to handle exploratory analyses. We very explicitly planned to include exploratory results because we knew we had a novel dataset that could provide insights into unanswered questions, but that is rare and unlikely to be the norm. On one hand, authors may feel handcuffed to stick to the data, and the field would miss out on important ramifications from the study that may not have been a priori expected. On the other hand, if there is no proper oversight throughout the review process, authors may take liberties in the reporting of their exploratory analyses. For instance, given that the bulk of the review is done upfront and focused on the preregistered hypotheses, it may open to the door to making claims regarding exploratory analyses that are not properly reviewed.

Because education research is complex and spans disciplines, we imagine that best practices and norms will continuously be updated. We are hopeful that educational science will improve as scholars continue to submit Registered Reports and innovate upon the process.