This editorial is companion to the first registered report published in the Canadian Journal of School Psychology entitled, “Scope of School Psychological Practice in Germany: Part 1” by Alexa von Hagen et al. This manuscript outlines the importance of evidence-based practices in school psychology, identifies weaknesses in the foundation of current research practices, and discusses registered reports as a tool to enhance research rigor by mitigating biases such as p-hacking and publication bias. Registered reports have gained traction despite initial reservations from researchers due to perceived constraints and barriers; this approach to publication of scholarly articles can lead to a positive shift in the relationship between authors and editors during the publication process, fostering collaboration, transparency, and credibility in the research practice and ultimately leading to improved evidence-based practice. Initial experiences from the first registered report in the field of school psychology are examined, noting advantages, disadvantages, and future recommendations. The adoption of registered reports signifies a cultural shift toward more robust, transparent, and credible research practices in school psychology, leading to true evidence-based practice with increased likelihood of implementation.
The Canadian Journal of School Psychology is pleased to publish the first registered report in school psychology entitled, “Scope of School Psychological Practice in Germany: Part 1” by von Hagen et al. (2024). Since announcing the infrastructure, development, consideration, and encouragement of this form of publication in 2019 (Shaw et al., 2019), numerous workshops, conversations, and contact with researchers have taken place. For many scholars, the idea of changing their process of publishing research findings and the notion that the registered report process is too time consuming were impediments to shifting approaches (Chambers et al., 2014; Roberson et al., 2020). This editorial describes some of lessons learned, value of registered reports, and goals for future use of registered reports. The author of the first article to follow the registered report approach with CJSP reports their experiences, ideas, and concerns with this publication process.
Background
The sixth revision of the Canadian Psychological Association’s Accreditation Standards for Doctoral and Residency Programs in Professional Psychology (2022) states that the training and practice of psychology commit to the scientific application of psychological knowledge to enhance human development and well-being. All aspects of professional psychology in the training of psychologists revolve around evidence-based and competency-based training and practice. “Clinical practice is enhanced and empowered by amalgamating research findings and clinician experience with clients’ needs, goals, priorities, expectations, values and wisdom (p. 4).” Accrediting bodies such as the American Psychological Association (2022) and the National Association of School Psychologists (2020) have placed similar emphases on the importance of research-guided clinical practice. The profession of school psychology requires a foundation of relevant, replicable, credible, and accessible scientific research (Shaw & Pecsi, 2021). There is reason to believe that currently this foundation is weak and requires important reconsideration and reinforcement (Malott, 2018; Shernoff et al., 2017).
A common activity in the practice of school psychology is to search for scientific support for interventions or assessment processes before implementation (Canivez, 2019). Among the most trustworthy sources of scientific support are refereed scholarly journals or sources such as the What Works Clearinghouse (https://ies.ed.gov/ncee/WWC), that aggregate information from refereed scholarly journals. This traditional form of scientific literature involves peer review that vouches for the value and quality of the information provided. Assumed is that multiple published research studies supporting a clinical practice create cumulative evidence that allows psychologists to infer whether an intervention or practice is truly effective versus a single study that may merely show statistically significant improvements due to error (Gehlbach & Robinson, 2021; Soderberg et al., 2021). Conversely, a single positive outcome that is in error is assumed to be corrected through reproductions and replications because science is self-correcting in concept (Vazire & Holcombe, 2020). However, most psychologists who engage in this activity find that the scientific support for the intervention or assessment is nearly always present; almost every intervention or practice evaluated and published shows significant positive effects, and even rare replications tend to support previous studies (Shaw, 2021). Is this because researchers are so good at developing interventions and assessments that they are almost always effective, or because there is a bias (i.e., non-random influences) in the system (Scheel et al., 2021)? If there is a bias, then the usefulness of cumulative science through replication, which effectively corrects many forms of error, is unlikely to bring psychologists closer to true evidence-based practices in real word applications. And if there is bias, then how can the nature of scientific support through refereed scholarly journals be corrected and regain status as the foundation of evidence-based practice?
Registered Reports: A Partial Solution
To better increase the quality of research evidence, registered reports have been proposed (Chambers & Tzavella, 2020). Registered report is a publishing format that requires peer review prior to data collection. Research protocols undergo peer review much like any other manuscript. The evaluation of the research protocol (known as “Stage 1 review”) is based on theoretical and clinical importance and rationale of the study, and quality of research design and data analysis plans. Stage 1 protocols that are accepted for publication are provisionally accepted for publication (known as an “in-principal acceptance”). If the authors follow the registered methods and interpretation is based on answering the registered research questions and hypotheses in the Stage 1 protocol, then the manuscript will be accepted and published no matter if the findings are statistically significant or not (Grand et al., 2018; Reich, 2021). The logic of a registered report is that the introduction, research questions and hypotheses, research methods, and data analysis plan are submitted for peer review before data are collected (Shaw et al., 2019). The primary goals of preregistration are to avoid fishing expeditions (i.e., p-hacking) that are major sources of error and problems with reproducibility of published research (Lakens, 2019). When a study is preregistered it is not possible to collect a large amount of data, examine relationships or differences that are significant at p < .05, and then create hypotheses after the study data are gathered (i.e., Hypothesis After Results are Known [Harking]) (Tackett et al., 2020). Developing a sampling plan also addresses the issue of statistical power so that data collection ends when the power metric proposed in the preregistration is achieved, rather than when statistical significance is achieved (Nosek et al., 2018). Nor can sampling be altered to bias the data to yield statistically significant results (i.e., p-hacking). Likewise, multiple statistical tests, altering how outliers are addressed, or using different data exclusionary criteria are part of a transparent and reviewed data management and analysis strategy and cannot be gamed until the desired statistically significant result is found (Nosek & Lindsay, 2018). Preregistration ensures results of a study are predicted and are not posthoc explanations that likely capitalize on error (Hardwicke & Ioannidis, 2018). For example, registered reports are published whether the results demonstrate effectiveness, or the interventions fail to show effective outcomes. Both are equally valuable. The reduction of publication bias is a major advantage of using registered reports as the method of publication (Bishop, 2019).
A secondary value of registered reports as conducted in the Canadian Journal of School Psychology is that there is a change in the relationship between author and action editor (Shaw et al., 2019). Once an in-principal acceptance has been made, there is a focus from both parties on making the research and manuscript as high quality as possible. Authors do not need to worry about or consider whether the data collected will yield statistically significant results; their only focus is collecting high quality data consistent with the protocol in described in Stage 1 (Lakens, 2024). This minimizes file drawer issues where authors never submit non-significant results for fear that the study will not be published (McClain et al., 2021). Authors check in with the editor with frequent progress reports. Timelines are negotiated. Discussion of any required changes in the research procedures, sampling, measures, or data analysis are considered (Lakens, 2024). Rather than a somewhat adversarial relationship, the registered report process encourages discussion, improvement, and adherence to a peer-reviewed and planned research protocol. The assumption is that collaborative and transparent research leads to high quality research, in contrast to traditional publication models in which researchers are trying to guess what the editor and reviewers value (Pennington & Heim, 2022).
Initial Experiences from CJSP’s First Registered Report
Advantages
My experience publishing my first registered report in the Canadian Journal of School Psychology confirms the advantages commonly pointed out in the literature and described in this editorial in relation to minimizing questionable research practices (e.g., D’Intino et al., 2018). However, what truly stood out to me was the profoundly positive and collaborative peer review experience. It felt like the action editor was a valuable member of our research team, responding to our frequent updates on the progress of the project with encouraging feedback and valuable ideas on how to handle difficulties, report deviations from our initial plans in a transparent way and improve the final manuscript. For early-career researchers, this invaluable and enriching experience undoubtedly helps alleviate some of the common challenges inherent in the current, often arduous research culture.
Disadvantages
Many researchers note the perception of longer review times compared to traditional reports as an important disadvantage of registered reports. Our experience seems to align, with our registered report taking from first submission in 2020 to final publication in 2024. However, it is crucial to clarify that this delay was not due to the peer review process, which I found surprisingly quick compared to past experiences. Instead, it was primarily influenced by typical early-career life-circumstances (i.e., maternity leave), resource constraints and demands to be working on several projects at the same time. Despite these delays, the Canadian Journal of School Psychology was remarkably accommodating, providing timely responses throughout.
Recommendations
One idea that could potentially improve the process is finding a way to share the collaborative interactions during the peer review process of registered reports with the readers of the final report. This could be a chance to enhance the value of these exchanges beyond just the action editor and authors. This information could greatly enrich the understanding of readers, offering insights into the research journey, the challenges faced during project implementation, and collaborative strategies employed to address them. Incorporating these details alongside the final report could potentially enhance transparency and foster a culture of open collaboration and continuous improvement within the research community. Some journals are already sharing peer review comments of traditional reports together with the final report, suggesting that a similar approach could be feasible for registered reports as well.
Conclusions
The Canadian Journal of School Psychology is one of the few clinically oriented scholarly journals to embrace and support registered reports. For school psychology, research conducted in classrooms, communities, or therapy settings reflects the complexity and context of interventions and their implementation; yet careful control of variables is often difficult or impossible to control. Pre-registration allows clinically oriented research to be predictive and to integrate the complex context into the first phase of the project. Therefore, clinical research is not a weakness or source of error, but allows logically predictive inferences to be made. The value of pre-registering research is that questions, hypotheses, methods, and data analysis plans improve the likelihood that published research represent true findings in specific contexts (Tackett et al., 2020). Under traditional publication models, it is impossible to know whether results were reproducible and credible, or simply an organized interpretation of random events that are probable in the relative chaos of classroom or other clinical contexts (e.g., Lindsay, 2020; Open Science Collaboration, 2015). Capitalization on random events can result in interventions and ideas that may do harm to the children with whom we work with or waste resources through the implementation of ineffective interventions (Mulder et al., 2018). Peer-reviewed scholarly research journals that publish important intervention research are considered the gold standard by which modern professional practice is guided (Huber, 2007). With the use of registered reports, there is an increase in the value, trustworthiness, and credibility of research over that of traditional publication models.
For scholars, registered reports represent a productive model of publication that is efficient, collaborative, and rigorous. This approach to scholarship is part of a culture change in the science of school psychology (Braden, 2024). It is time to consider new criteria of what represents publishable research (Fanelli, 2010). Statistical significance, novelty, excitement, large effect sizes, and counterintuitive results have always been large factors in research being accepted for publication in peer-reviewed journals (Fanelli, 2012). Yet, these factors have not been productive for the implementation of robust interventions in school psychology (Berliner, 2016). Openness, trustworthiness, transparency, theoretical, robust to modification, flexible, collaborative, culturally appropriate, credible, and useful are the new key constructs defining quality clinical research, which lead to a true evidence-based practice with an increased likelihood of implementation (Lewis, 2021). This first effort serves as a proof of concept in registered reports. We hope that this format will continue and expand to include many types of research in clinically oriented professional journals. The Canadian Journal of School Psychology is pleased to be among the leaders in this effort.
