Registered Reports in School Psychology Research: Initial Experiences,Analyses,and Future

Abstract

This editorial is companion to the first registered report published in the Canadian Journal of School Psychology entitled, “Scope of School Psychological Practice in Germany: Part 1” by Alexa von Hagen et al. This manuscript outlines the importance of evidence-based practices in school psychology, identifies weaknesses in the foundation of current research practices, and discusses registered reports as a tool to enhance research rigor by mitigating biases such as p-hacking and publication bias. Registered reports have gained traction despite initial reservations from researchers due to perceived constraints and barriers; this approach to publication of scholarly articles can lead to a positive shift in the relationship between authors and editors during the publication process, fostering collaboration, transparency, and credibility in the research practice and ultimately leading to improved evidence-based practice. Initial experiences from the first registered report in the field of school psychology are examined, noting advantages, disadvantages, and future recommendations. The adoption of registered reports signifies a cultural shift toward more robust, transparent, and credible research practices in school psychology, leading to true evidence-based practice with increased likelihood of implementation.

Keywords

evidence-based practices research reform registered reports publishing research

The Canadian Journal of School Psychology is pleased to publish the first registered report in school psychology entitled, “Scope of School Psychological Practice in Germany: Part 1” by von Hagen et al. (2024). Since announcing the infrastructure, development, consideration, and encouragement of this form of publication in 2019 (Shaw et al., 2019), numerous workshops, conversations, and contact with researchers have taken place. For many scholars, the idea of changing their process of publishing research findings and the notion that the registered report process is too time consuming were impediments to shifting approaches (Chambers et al., 2014; Roberson et al., 2020). This editorial describes some of lessons learned, value of registered reports, and goals for future use of registered reports. The author of the first article to follow the registered report approach with CJSP reports their experiences, ideas, and concerns with this publication process.

Background

The sixth revision of the Canadian Psychological Association’s Accreditation Standards for Doctoral and Residency Programs in Professional Psychology (2022) states that the training and practice of psychology commit to the scientific application of psychological knowledge to enhance human development and well-being. All aspects of professional psychology in the training of psychologists revolve around evidence-based and competency-based training and practice. “Clinical practice is enhanced and empowered by amalgamating research findings and clinician experience with clients’ needs, goals, priorities, expectations, values and wisdom (p. 4).” Accrediting bodies such as the American Psychological Association (2022) and the National Association of School Psychologists (2020) have placed similar emphases on the importance of research-guided clinical practice. The profession of school psychology requires a foundation of relevant, replicable, credible, and accessible scientific research (Shaw & Pecsi, 2021). There is reason to believe that currently this foundation is weak and requires important reconsideration and reinforcement (Malott, 2018; Shernoff et al., 2017).

A common activity in the practice of school psychology is to search for scientific support for interventions or assessment processes before implementation (Canivez, 2019). Among the most trustworthy sources of scientific support are refereed scholarly journals or sources such as the What Works Clearinghouse (https://ies.ed.gov/ncee/WWC), that aggregate information from refereed scholarly journals. This traditional form of scientific literature involves peer review that vouches for the value and quality of the information provided. Assumed is that multiple published research studies supporting a clinical practice create cumulative evidence that allows psychologists to infer whether an intervention or practice is truly effective versus a single study that may merely show statistically significant improvements due to error (Gehlbach & Robinson, 2021; Soderberg et al., 2021). Conversely, a single positive outcome that is in error is assumed to be corrected through reproductions and replications because science is self-correcting in concept (Vazire & Holcombe, 2020). However, most psychologists who engage in this activity find that the scientific support for the intervention or assessment is nearly always present; almost every intervention or practice evaluated and published shows significant positive effects, and even rare replications tend to support previous studies (Shaw, 2021). Is this because researchers are so good at developing interventions and assessments that they are almost always effective, or because there is a bias (i.e., non-random influences) in the system (Scheel et al., 2021)? If there is a bias, then the usefulness of cumulative science through replication, which effectively corrects many forms of error, is unlikely to bring psychologists closer to true evidence-based practices in real word applications. And if there is bias, then how can the nature of scientific support through refereed scholarly journals be corrected and regain status as the foundation of evidence-based practice?

Registered Reports: A Partial Solution

To better increase the quality of research evidence, registered reports have been proposed (Chambers & Tzavella, 2020). Registered report is a publishing format that requires peer review prior to data collection. Research protocols undergo peer review much like any other manuscript. The evaluation of the research protocol (known as “Stage 1 review”) is based on theoretical and clinical importance and rationale of the study, and quality of research design and data analysis plans. Stage 1 protocols that are accepted for publication are provisionally accepted for publication (known as an “in-principal acceptance”). If the authors follow the registered methods and interpretation is based on answering the registered research questions and hypotheses in the Stage 1 protocol, then the manuscript will be accepted and published no matter if the findings are statistically significant or not (Grand et al., 2018; Reich, 2021). The logic of a registered report is that the introduction, research questions and hypotheses, research methods, and data analysis plan are submitted for peer review before data are collected (Shaw et al., 2019). The primary goals of preregistration are to avoid fishing expeditions (i.e., p-hacking) that are major sources of error and problems with reproducibility of published research (Lakens, 2019). When a study is preregistered it is not possible to collect a large amount of data, examine relationships or differences that are significant at p < .05, and then create hypotheses after the study data are gathered (i.e., Hypothesis After Results are Known [Harking]) (Tackett et al., 2020). Developing a sampling plan also addresses the issue of statistical power so that data collection ends when the power metric proposed in the preregistration is achieved, rather than when statistical significance is achieved (Nosek et al., 2018). Nor can sampling be altered to bias the data to yield statistically significant results (i.e., p-hacking). Likewise, multiple statistical tests, altering how outliers are addressed, or using different data exclusionary criteria are part of a transparent and reviewed data management and analysis strategy and cannot be gamed until the desired statistically significant result is found (Nosek & Lindsay, 2018). Preregistration ensures results of a study are predicted and are not posthoc explanations that likely capitalize on error (Hardwicke & Ioannidis, 2018). For example, registered reports are published whether the results demonstrate effectiveness, or the interventions fail to show effective outcomes. Both are equally valuable. The reduction of publication bias is a major advantage of using registered reports as the method of publication (Bishop, 2019).

A secondary value of registered reports as conducted in the Canadian Journal of School Psychology is that there is a change in the relationship between author and action editor (Shaw et al., 2019). Once an in-principal acceptance has been made, there is a focus from both parties on making the research and manuscript as high quality as possible. Authors do not need to worry about or consider whether the data collected will yield statistically significant results; their only focus is collecting high quality data consistent with the protocol in described in Stage 1 (Lakens, 2024). This minimizes file drawer issues where authors never submit non-significant results for fear that the study will not be published (McClain et al., 2021). Authors check in with the editor with frequent progress reports. Timelines are negotiated. Discussion of any required changes in the research procedures, sampling, measures, or data analysis are considered (Lakens, 2024). Rather than a somewhat adversarial relationship, the registered report process encourages discussion, improvement, and adherence to a peer-reviewed and planned research protocol. The assumption is that collaborative and transparent research leads to high quality research, in contrast to traditional publication models in which researchers are trying to guess what the editor and reviewers value (Pennington & Heim, 2022).

Initial Experiences from CJSP’s First Registered Report

Advantages

My experience publishing my first registered report in the Canadian Journal of School Psychology confirms the advantages commonly pointed out in the literature and described in this editorial in relation to minimizing questionable research practices (e.g., D’Intino et al., 2018). However, what truly stood out to me was the profoundly positive and collaborative peer review experience. It felt like the action editor was a valuable member of our research team, responding to our frequent updates on the progress of the project with encouraging feedback and valuable ideas on how to handle difficulties, report deviations from our initial plans in a transparent way and improve the final manuscript. For early-career researchers, this invaluable and enriching experience undoubtedly helps alleviate some of the common challenges inherent in the current, often arduous research culture.

Disadvantages

Many researchers note the perception of longer review times compared to traditional reports as an important disadvantage of registered reports. Our experience seems to align, with our registered report taking from first submission in 2020 to final publication in 2024. However, it is crucial to clarify that this delay was not due to the peer review process, which I found surprisingly quick compared to past experiences. Instead, it was primarily influenced by typical early-career life-circumstances (i.e., maternity leave), resource constraints and demands to be working on several projects at the same time. Despite these delays, the Canadian Journal of School Psychology was remarkably accommodating, providing timely responses throughout.

Recommendations

One idea that could potentially improve the process is finding a way to share the collaborative interactions during the peer review process of registered reports with the readers of the final report. This could be a chance to enhance the value of these exchanges beyond just the action editor and authors. This information could greatly enrich the understanding of readers, offering insights into the research journey, the challenges faced during project implementation, and collaborative strategies employed to address them. Incorporating these details alongside the final report could potentially enhance transparency and foster a culture of open collaboration and continuous improvement within the research community. Some journals are already sharing peer review comments of traditional reports together with the final report, suggesting that a similar approach could be feasible for registered reports as well.

Conclusions

The Canadian Journal of School Psychology is one of the few clinically oriented scholarly journals to embrace and support registered reports. For school psychology, research conducted in classrooms, communities, or therapy settings reflects the complexity and context of interventions and their implementation; yet careful control of variables is often difficult or impossible to control. Pre-registration allows clinically oriented research to be predictive and to integrate the complex context into the first phase of the project. Therefore, clinical research is not a weakness or source of error, but allows logically predictive inferences to be made. The value of pre-registering research is that questions, hypotheses, methods, and data analysis plans improve the likelihood that published research represent true findings in specific contexts (Tackett et al., 2020). Under traditional publication models, it is impossible to know whether results were reproducible and credible, or simply an organized interpretation of random events that are probable in the relative chaos of classroom or other clinical contexts (e.g., Lindsay, 2020; Open Science Collaboration, 2015). Capitalization on random events can result in interventions and ideas that may do harm to the children with whom we work with or waste resources through the implementation of ineffective interventions (Mulder et al., 2018). Peer-reviewed scholarly research journals that publish important intervention research are considered the gold standard by which modern professional practice is guided (Huber, 2007). With the use of registered reports, there is an increase in the value, trustworthiness, and credibility of research over that of traditional publication models.

For scholars, registered reports represent a productive model of publication that is efficient, collaborative, and rigorous. This approach to scholarship is part of a culture change in the science of school psychology (Braden, 2024). It is time to consider new criteria of what represents publishable research (Fanelli, 2010). Statistical significance, novelty, excitement, large effect sizes, and counterintuitive results have always been large factors in research being accepted for publication in peer-reviewed journals (Fanelli, 2012). Yet, these factors have not been productive for the implementation of robust interventions in school psychology (Berliner, 2016). Openness, trustworthiness, transparency, theoretical, robust to modification, flexible, collaborative, culturally appropriate, credible, and useful are the new key constructs defining quality clinical research, which lead to a true evidence-based practice with an increased likelihood of implementation (Lewis, 2021). This first effort serves as a proof of concept in registered reports. We hope that this format will continue and expand to include many types of research in clinically oriented professional journals. The Canadian Journal of School Psychology is pleased to be among the leaders in this effort.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Steven R. Shaw

Alexa von Hagen

Author Biographies

Steven R. Shaw is chair of the Department of Educational and Counselling Psychology at McGill University.

Sierra Pecsi is a doctoral student in School and Applied Child Psychology at McGill University.

Erika Infantino is a doctoral student in School and Applied Child Psychology at McGill University.

Yeon Hee Kang is a master’s student in educational psychology at McGill University.

Neha Verma is a doctoral student in School and Applied Child Psychology at McGill University.

Alexa von Hagen is a postdoctoral researcher at the Competence Centre School Psychology Hesse at the Goethe-University Frankfurt.

References

American Psychological Association. (2022). Standards of accreditation for health service psychology and accreditation operating procedures. https://irp.cdn-website.com/a14f9462/files/uploaded/standards-of-accreditation.pdf

Berliner

D. C.

(2016). Comment: Educational research: The hardest science of all. Educational Researcher, 31(1), 18–20. https://doi.org/10.3102/0013189X031008018

Bishop

(2019). Rein in the four horsemen of irreproducibility. Nature, 568(7753), 435–435. https://doi.org/10.1038/d41586-019-01307-2

Braden

J. P.

(2024). Registered reports and replications: An ongoing Journal of School Psychology initiative. Journal of School Psychology, 103, 101294.

Canadian Psychological Association. (2022). accreditation standards for doctoral and residency programs in professional psychology. https://cpa.ca/docs/File/Accreditation/Proposed%20Revisions%206th%20Revision%20CPA%20Accred%20Stds.pdf

Canivez

G. L.

(2019). Evidence-based assessment for school psychology: Research, training, and clinical practice. Contemporary School Psychology, 23(2), 194–200. https://doi.org/10.1007/s40688-019-00238-z

Chambers

C. D.

Feredoes

Muthukumaraswamy

S. D.

Etchells

(2014). Instead of “playing the game” it is time to change the rules: Registered reports at AIMS neuroscience and beyond. AIMS Neuroscience, 1(1), 4–17. https://doi.org/10.3934/Neuroscience.2014.1.4

Chambers

C. D.

Tzavella

(2020). Registered reports: Past, present and future [Preprint]. MetaArXiv. https://doi.org/10.31222/osf.io/43298

D’Intino

Lysenko

Shaw

S. R.

(2018). Questionable research practices and evidence-based practices in school psychology. Communique, 47(4), 8–11.

10.

Fanelli

(2010). “Positive” results increase down the hierarchy of the sciences. PLoS One, 5(4), e10068. https://doi.org/10.1371/journal.pone.0010068

11.

Fanelli

(2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. https://doi.org/10.1007/s11192-011-0494-7

12.

Gehlbach

Robinson

C. D.

(2021). From old school to open science: The implications of new research norms for educational psychology and beyond. Educational Psychologist, 56(2), 79–89. https://www.tandfonline.com/doi/pdf/10.1080/00461520.2021.1898961

13.

Grand

J. A.

Rogelberg

S. G.

Banks

G. C.

Landis

R. S.

Tonidandel

(2018). From outcome to process focus: Fostering a more robust psychological science through registered reports and results-blind reviewing. Perspectives on Psychological Science, 13(4), 448–456. https://doi.org/10.1177/1745691618767883

14.

Hardwicke

T. E.

Ioannidis

J. P. A.

(2018). Mapping the universe of registered reports. Nature Human Behaviour, 2(11), 793–796. https://doi.org/10.1038/s41562-018-0444-y

15.

Huber

D. R.

(2007). Is the scientist-practitioner model viable for school psychology practice? American Behavioral Scientist, 50(6), 778–788. https://doi.org/10.1177/0002764206296456

16.

Lakens

(2019). The value of preregistration for psychological science: A conceptual analysis [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/jbh4w

17.

Lakens

(2024). When and how to deviate from a preregistration. Collabra: Psychology, 10(1): 117094. https://doi.org/10.1525/collabra.117094

18.

Lewis

N. A.

Jr. (2021). What counts as good science? How the battle for methodological legitimacy affects public psychology. American Psychologist, 76, 1323–1333. https://doi.org/10.1037/amp0000870

19.

Lindsay

D. S.

(2020). Seven steps toward transparency and replicability in psychological science. Canadian Psychology/Psychologie Canadienne, No Pagination Specified-No Pagination Specified. https://doi.org/10.1037/cap0000222

20.

Malott

R. W.

(2018). A science-based practitioner model. Education and Treatment of Children, 41(3), 371–384. https://doi.org/10.1353/etc.2018.0020

21.

McClain

M. B.

Callan

G. L.

Harris

Floyd

R. G.

Haverkamp

C. R.

Golson

M. E.

Longhurst

D. N.

Benallie

K. J.

(2021). Methods for addressing publication bias in school psychology journals: A descriptive review of meta-analyses from 1980 to 2019. Journal of School Psychology, 84, 74–94. https://doi.org/10.1016/j.jsp.2020.11.002

22.

Mulder

Singh

A. B.

Hamilton

Das

Outhred

Morris

Bassett

Baune

B. T.

Berk

Boyce

Lyndon

Parker

Malhi

G. S.

(2018). The limitations of using randomised controlled trials as a basis for developing treatment guidelines. Evidence-Based Mental Health, 21(1), 4–6. https://doi.org/10.1136/eb-2017-102701

23.

National Association of School Psychologists. (2020). Standards for graduate preparation of school psychologists. https://www.nasponline.org/standards-and-certification/nasp-2020-professional-standards-adopted

24.

Nosek

B. A.

Ebersole

C. R.

DeHaven

A. C.

Mellor

D. T.

(2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

25.

Nosek

B. A.

Lindsay

D. S.

(2018). Preregistration becoming the norm in psychological science. APS Observer, 31.

26.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

27.

Pennington

C. R.

Heim

(2022). Reshaping the publication process: Addiction research & Theory joins Peer community in registered reports. Addiction Research & Theory, 30(1), 1–4. https://doi.org/10.1080/16066359.2021.1931142

28.

Reich

(2021). Preregistration and registered reports. Educational Psychologist, 56(2), 101–109.

29.

Roberson

A. J.

Farmer

R. L.

Shaw

Upton

Zaheer

(2020, December 11). Opening doors to open science and scholarship for school psychology research, training, and practice. https://doi.org/10.31234/osf.io/su53p

30.

Scheel

A. M.

Schijen

M. R. M. J.

Lakens

(2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2), 25152459211007468. https://doi.org/10.1177/25152459211007467

31.

Shaw

S. R.

(2021). Implementing evidence-based practices in school psychology: excavation by de-implementing the disproved. Canadian Journal of School Psychology, 36(2), 91– 97. https://doi.org/10.1177/08295735211000513

32.

Shaw

S. R. S.

D’Intino

Lysenko

(2019). Registered reports, replication, and the Canadian Journal of School Psychology: Improving the evidence in evidence-based school psychology. Canadian Journal of School Psychology, 34(3), 175–187. https://doi.org/10.1177/0829573519843027

33.

Shaw

S. R.

Pecsi

(2021). When is the evidence sufficiently supportive of real-world application? Evidence-based practices, open science, clinical readiness level. Psychology in the Schools, 58(10), 1891–1901. https://doi.org/10.1002/pits.22537

34.

Shernoff

E. S.

Bearman

S. K.

Kratochwill

T. R.

(2017). Training the next generation of school psychologists to deliver evidence-based mental health practices: Current challenges and future directions. School Psychology Review, 46(2), 219–232. https://doi.org/10.17105/SPR-2015-0118.V46-2

35.

Soderberg

C. K.

Errington

T. M.

Schiavone

S. R.

Bottesini

Thorn

F. S.

Vazire

. . . & Nosek

B. A.

(2021). Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour, 5(8), 990-997.

36.

Tackett

J. L.

Brandes

C. M.

Dworak

E. M.

Shields

A. N.

(2020). Bringing the (pre)registration revolution to graduate training. Canadian Psychology/Psychologie Canadienne, No Pagination Specified-No Pagination Specified. https://doi.org/10.1037/cap0000221

37.

Vazire

Holcombe

A. O.

(2020). Where are the self-correcting mechanisms in science? PsyArXiv. https://doi.org/10.31234/osf.io/kgqzt

38.

von Hagen

Müller

Vannini

Rublevskis

Schaaf

Jeck

Müller-Staske

Bachmann

Sedlak

Wegerer

Büttner

(2024). Scope of school psychological practice in Germany: Part 1. Canadian Journal of School Psychology, 08295735231226195. https://doi.org/10.1177/08295735231226195