In the present study, completed as a Registered Report, we assessed the level of training on issues related to open science and the replication crisis in clinical-psychology PhD programs in the United States. Syllabi were surveyed and coded from research-methods courses in these programs for topics related to open science and the replication crisis. In addition, a survey of graduate students was conducted evaluating their exposure to and opinions of open science and the replication crisis. We found that overall, open-science and replication-crisis topics are not widely covered in these programs, although there is evidence of students’ interest in learning about these topics.
BakkerM.van DijkA.WichertsJ. M. (2012). The rules of the game we call psychological science. Perspectives on Psychological Science, 7, 543–554. https://doi.org/10.1177/17456916112459060
2.
BanksG. C.RogelbergS. G.WoznyjH. M.LandisR. S.RuppD. E. (2016). Editorial: Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business Psychology, 31, 323–338. https://doi.org/10.1007/s10869-016-9456-7
3.
BemD. J. (2011). Feeling the future: Experimental evidence for abnormal retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407–425. https://doi.org/10.1037/a0021524
BenningS. D.BachrachR. L.SmithE. A.FreemanA. J.WrightA. G. (2019). The registration continuum in clinical science: A guide toward transparent practices. Journal of Abnormal Psychology, 128(6), 528–540. https://doi.org/10.1037/abn0000451
BorderR.JohnsonE. C.EvansL. M.SmolenA.BerleyN.SullivanP. F.KellerM. C. (2019). No support for historical candidate gene or candidate gene-by-interaction hypotheses for major depression across multiple large samples. The American Journal of Psychiatry, 176(5), 376–387. https://doi.org/10.1176/appi.ajp.2018.18070881
8.
BlázquezD.BotellaJ.SueroM. (2017). The debate on the ego-depletion effect: Evidence from meta-analysis with the uniform method. Frontiers in Psychology, 8, Article 197. https://doi.org/10.3389/fpsyg.2017.00197x
9.
BrandtM. J.IjzermanH.DijksterhuisA.FarachF. J.GellerJ.Giner-SorollaR.GrangeJ. A.PeruginiM.SpiesJ. R.van’t VeerA. (2014). The replication recipe: What makes for a convincing replication?Journal of Experimental Social Psychology, 50, 217–224. https://doi.org/10.1016/j.jesp.2013.10.005
10.
ChambersC. D.FeredoesE.MuthukumaraswamyS. D.EtchellsP. J. (2014). Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1, 4–17. https://doi.org/10.3934/Neuroscience2014.1.4
11.
CohenJ. (1962). The statistical power of abnormal-social psychological research: A review. The Journal of Abnormal and Social Psychology, 65, 145–153. https://doi.org/10.1037/h0045186
12.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.
13.
CoursolA.WagnerE. E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology: Research and Practice, 17, 136–137. https://doi.org/10.1037/0735-7028.17.2.136
14.
CraneN. J.PearsonZ. (2011). Can we get a pub from this? Reflections on competition and the pressure to publish while in graduate school. The Geographic Bulletin, 52, 77–80.
15.
DangJ.BarkerP.BaumertA.BentvelzenM.BerkmanE.BuchholzN.BucznyJ.ChenZ.De CristofaroV.de VriesL.DewitteS.GiacomantonioM.GongR.HomanM.ImhoffR.IsmailI.JiaL.KubiakT.LangeF.. . . ZinkernagelA. (2021). A multilab replication of the ego depletion effect. Social Psychological and Personality Science, 12, 14–24. https://doi.org/10.1177/1948550619887702
16.
FergusonC. J.HeeneM. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7, 555–561. https://doi.org/10.1177/1745691612459059
17.
GelmanA.LokenE. (2014). The statistical crisis in science. American Scientist, 102, 460–465.
18.
GervaisW. M.JewellJ. A.NajleM. B.NgB. K. L. (2015). A powerful nudge? Presenting calculable consequences of underpowered research shifts incentives towards adequately powered designs. Social Psychological and Personality Science, 6, 847–854. https://doi.org/10.1177/1948550615584199
19.
GreenwaldA. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
20.
HoltzP.MonnerjahnP. (2016). Falsificationism is not just “potential” falsifiability, but requires “actual” falsification: Social psychology, critical rationalism, and progress in science. Journal for the Theory of Social Behaviour, 47, 348–362. https://doi.org/10.1111/jtsb.12134
JohnL. K.LoewenstienG.PrelecD. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. https://doi.org/10.1177/0956797611430953
KleinR. A.RatliffK. A.VianelloM.AdamsR. B.BahnikŠ.BernsteinM. J.BocianK.BrandtM. J.BrooksB.BrumbaughC. C.CemalcilarZ.ChandlerJ.CheongW.DavisW. E.DevosT.EisnerM.FrankowskaN.FurrowD.GallianiE. M.. . . NosekB. A. (2014a). Data from investigating variation in replicability: A “many labs” replication project, Journal of Open Psychology Data, 2, Article e4. http://doi.org/10.5334/jopd.ad
25.
KleinR. A.RatliffK. A.VianelloM.AdamsR. B.BahníkŠ.BernsteinM. J.BocianK.BrandtM. J.BrooksB.BrumbaughC. C.CemalcilarZ.ChandlerJ.CheongW.DavisW. E.DevosT.EisnerM.FrankowskaN.FurrowD.GallianiE. M.. . . NosekB. A. (2014b). Investigating variation in replicability: A “many labs” project. Social Psychology, 45, 142–152. https://doi.org/10.1027/1864-9335/a000178
26.
LevensteinM. C.LyleJ. A. (2018). Data: Sharing is caring. Advances in Methods and Practices in Psychological Science, 1, 95–103. https://doi.org/10.1177/2515245918758319
27.
MurphyK. R.AguinisH. (2019). HARKing: How badly can cherry-picking and question trolling produce bias in published results. Journal of Business and Psychology, 34, 1–17. https://doi.org/10.1007/s10869-017-9524-7
28.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, Article aac4716. https://doi.org/10.1126/science.aac4716
29.
PopperK. R. (1963). Science as falsification. Conjectures and Refutations, 1(1963), 33–39.
SaccoD. F.BrownM. (2019). Assessing the efficacy of a training intervention to reduce acceptance of questionable research practices in psychology graduate students. Journal of Empirical Research on Human Research Ethics, 14, 209–218. https://doi.org/10.1177/1556264619840525
33.
ScheelA. M.SchijenM. R. M.LakensD. (2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4. https://doi.org/10.1177/25152459211007467
SchonbrodtF. D.PeruginiM. (2013). At what sample size do correlations stabilize?Journal of Research in Personality, 47, 609–612. https://doi.org/10.1016/j.jrp.2013.05.009
36.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. https://doi.org/10.1177/0956797611417632
37.
StefanA. M.SchönbrodtF. D. (2023). Big little lies: A compendium and simulation of p-hacking strategies. Royal Society Open Science, 10(2), 220346. https://doi.org/10.1098/rsos.220346
38.
StoilescuD.McDougallD. (2010). Starting to publish academic research as a doctoral student. International Journal of Doctoral Studies, 5, 79–92.
39.
SwiatkowskiW.DompnierB. (2017). Replicability crisis in social psychology: Looking at the past to find new pathways for the future. International Review of Social Psychology, 30, 111–124. https://doi.org/10.5334/irsp.66
40.
SwiftJ. K.ChristophersonC. D.BirdM. O.ZoldA.GoodeJ. (2022). Questionable research practices among faculty and students in APA-accredited clinical and counseling psychology doctoral programs. Training and Education in Professional Psychology, 16(3), 299–305. https://doi.org/10.1037/tep0000322
41.
TackettJ. L.LilienfeldS. O.PatrickC. J.JohnsonS. L.KruegerR. F.MillerJ. D.OltmannsT. F.ShroutP. E. (2017). It’s time to broaden the replicability conversation: Thoughts for and from clinical psychological science. Perspectives on Psychological Science, 12, 742–756. https://doi.org/10.1177/1745691617690042
42.
TackettJ. L.MillerJ. D. (2019). Introduction to the special section on increasing replicability, transparency, and openness in clinical psychology. Journal of Abnormal Psychology, 128, 487–492. http://dx.doi.org/10.1037/abn0000455
43.
TressoldiP. E. (2012). Replication unreliability in psychology: Elusive phenomena or “elusive” statistical power?Frontiers in Psychology, 3, Article 218. https://doi.org/10.3389/fpsyg.2012.00218
44.
van’t VeerA. E.Giner-SorollaR. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. https://doi.org/10.1016/j.jesp.2016.03.004
45.
Vicente-SaezR.Martinez-FuentesC. (2018). Open science now: A systematic literature review for an integrated definition. Journal of Business Research, 88, 428–436. https://doi.org/10.1016/j.jbusres.2017.12.043
46.
WalshC. G.XiaW.LiM.DennyJ. C.HarrisP. A.MalinB. (2018). Enabling open-science initiatives in clinical psychology and psychiatry without sacrificing patients’ privacy: Current practices and future challenges. Advances in Methods and Practices in Psychological Science, 1, 104–114. https://doi.org/10.1177/2515245917749652
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
