Reflections on the Registered Report Process for “Save Your Strokes: Handwriting Ineffective in Chinese Second Language Instruction”

AERA Open’s Registered Reports Special Topic process was an excellent opportunity to conduct a replication of work that had already been conducted as a pilot study. Replication is critical because it minimizes Type I error and improves confidence in any conclusions that are drawn. The rigor of the preregistration process allowed us to run our experiment and conduct our analyses without bias, helping us to link our new work with our pilot work in more robust ways.

The structure of preregistration was also helpful in a preparatory sense. The bulk of our proposal was prepared, and we could submit an initial version of our introductory, methods, and planned analyses based on our pilot study. We were then able to use suggested revisions to strengthen our work as we aggregated and processed our dataset and followed through with our analytic goals, a benefit that traditional journal publications lack.

“In-principle acceptance” was a helpful strategy, forcing us to stick to planned analyses rather than venture too far into exploratory analyses. Rerunning the design of our previous work allowed use to reexamine our methods and rethink the achievements of our preset goals.

The Registered Reports Special Topic process also offers an excellent opportunity for academic communication. It removes enormous amounts of confusion and uncertainty in imagining the shape of data by making datasets available, open, and transparent. This promotes replication work and digging deeper. We hope to see that our data are reused and that our design is replicated, and this process has made us hopeful of interacting with other researchers pursuing the same goals.

Ours is, of course, still a journal paper. There are issues that plague most academics when it comes to preparing and submitting journal papers: deadlines are harder to adhere to, length is more of an issue, keeping focus and shape amid such a large document. However, the Registered Reports Special Topic process incorporated multiple revisions and submissions thanks to reviewers who understood the editors’ goals for in-principle acceptance—patience was excellent among all parties.

These factors helped our team craft a paper that is stronger than anything we could have made without the registered reporting process. We thank the reviewers and editors alike for their service and for the opportunity to participate in this Special Topics process.