Abstract
Keywords
Some education researchers may feel like registered reports (RRs) are a big change, but after our experience, it will be hard for any of us to go back to the old way of doing research. The benefits are too extensive to ignore. Collaborating with reviewers on the best way to analyze data beforehand saves time, in comparison to going back and forth over multiple rounds of postanalysis revisioning, while also preventing any of us (authors or reviewers) from being influenced by the data. It also makes for a much more collaborative and friendly process—especially when compared with traditional peer review.
In essence, RRs apply the dissertation process to a broader range of research. In both of these, a proposal is submitted to a group (committees vs. reviewers) who all agree ahead of time on how the process should proceed. Phase I RR acceptance is based on the quality and rigor of the question being asked and the method proposed to answer it. Final acceptance is not determined by how interesting the results are, but on whether researchers did what they planned. Life happens, so changes may be required, but RRs assure that those changes are framed accurately and are fully disclosed. In the dissertation process, the candidate knows what he or she needs to do to get out the door. In RRs, it is the same for authors. The pathway to completion (i.e., publication) is much more concrete.
RRs help make sure that a good story does not get in the way of the facts. In our study, when the results for one set of models were less dramatic than expected, we still reported the original model and did not spend time running lots of alternative models trying to explain why this was the case. We just reported the results as planned, because that is what everyone (editors and reviewers) had agreed on before we examined the data.
Going from the old way of doing research, in which reviews are received after all the work has been done, to RRs is like upgrading from a typewriter to a computer. It is all benefits with no real costs (for most projects). We see relatively minor downsides with rather large potential benefits. RRs may not be appropriate for every study (e.g., if data collection must start by a specific date), but every author should at least consider the RR approach before following the more traditional path. Producing more trustworthy, replicable results while also reducing the overall time and energy to reach publication is a win both for individual authors and for the field. In our next registered report (already under way), we are working on dealing with length. Because of the level of detail needed to fully explain our analysis plan, in our first RR our Phase I submission alone was already over the journal’s total page limit for completed manuscripts.
One interesting—and unexpected—exchange occurred during our Phase I and Phase II 2 review process. One reviewer believed our particular type of study was not appropriate for an RR because the outcome could have been expected. Their comments, on both phases of review, spurred a great deal of discussion among our research team and with the editors. At the end of the process, our team concluded, as did the journal editors, that preregistration and peer review prior to data analysis was appropriate for all types of research. Again, perhaps not in all contexts, but the basic principles of RRs are applicable to most forms of empirical research.
In the end, the RR process made the time to publication much shorter! The back and forth with reviewers happened early on and did not require multiple rounds of reanalysis (which would have increased our Type I error rate, in addition to adding time to publication). We saved substantial time on analyses by not having to redo 10 different analyses to address reviewer criticisms. We also did not have to worry about the file drawer problem of our research not seeing the light of day if the results had not been statistically significant. In our experience, the registered reports process helped us demonstrate efficiency, credibility, and transparency.
Footnotes
ORCID iDs
Authors
SCOTT J. PETERS is a professor of assessment and research methodology at the University of Wisconsin–Whitewater. His research work focuses on educational assessment and data use, gifted and talented student identification, equity within advanced educational opportunities, and educational policy.
KAREN E. RAMBO-HERNANDEZ is an associate professor in the College of Education and Human Development at Texas A&M University. Her research interests include novel applications of multilevel modeling and growth modeling, the assessment of educational interventions to improve STEM education, and access for all students—particularly high achieving and underrepresented students—to high-quality education.
MATTHEW C. MAKEL is the director of research and evaluation for Duke University’s Talent Identification Program. His research focuses on academic talent development and research methods.
MICHAEL S. MATTHEWS is professor and director of the academically and intellectually gifted graduate programs at the University of North Carolina at Charlotte and coeditor of Gifted Child Quarterly. His research interests focus on assessment and identification of gifted children, education policy, parenting (including homeschooling) of gifted learners, and gifted and academically advanced learners from diverse backgrounds, particularly English learners.
JONATHAN A. PLUCKER is the Julian C. Stanley Professor of Talent Development at Johns Hopkins University. His work focuses on creativity and intelligence, education policy, and talent development.