Reflections on the Registered Report Process for “Smart but Evil? Student-Teachers’ Perception of Educational Researchers’ Epistemic Trustworthiness”

All in all, writing the registered report was quite fun, especially during the Phase I revisions, in which it really was about working together with the reviewers in order to enhance our study design. It was a good feeling to contribute to the open science movement by “early adopting” the registered report format, too.

The registered report format was perfect for our study since it consisted of a replication of an effect we had already found once. Formulating very specific confirmatory hypotheses was therefore rather straightforward—an endeavor that would have been much harder when developing completely new materials, methods, or designs.

However, we noticed that registered reports require a considerably longer methods section. In fact, it is vital that the reviewers gain an in-depth understanding of the reasons for the chosen design and statistical methods in order to be able to identify potential problems or areas of improvement before the study is carried out. While this was sometimes a bit challenging, we, in hindsight, perceive it as a further advantage of the format: Writing an extensive methods section and trying to convince the reviewers ensures that everything is meticulously thought through before data collection. However, in our case, we should have communicated this focus on methods more clearly to the reviewers.

Furthermore, we think pre-registering analysis plans fosters every researcher’s methodological skills. For example, we never thought about power analysis for confirmatory factor analysis (CFA) models before conducting our first preregistered study (which was another paper), but now do so every time.

Finally, writing the discussion section after Phase I acceptance was a new experience to us: Without fearing that the paper might be rejected, we felt that the discussion became more honest and vivid as we could, for example, give sharper opinions.