Is Psychological Science Self-Correcting? Citations Before and After Successful and Failed Replications

Abstract

In principle, successful replications should enhance the credibility of scientific findings, and failed replications should reduce credibility. Yet it is unknown how replication typically affects the influence of research. We analyzed the citation history of 98 articles. Each was published by a selective psychology journal in 2008 and subjected to a replication attempt published in 2015. Relative to successful replications, failed replications reduced citations of replicated studies by only 5% to 9% on average, an amount that did not differ significantly from zero. Less than 3% of articles citing the original studies cited the replication attempt. It does not appear that replication failure much reduced the influence of nonreplicated findings in psychology. To increase the influence of replications, we recommend (a) requiring authors to cite replication studies alongside the individual findings and (b) enhancing reference databases and search engines to give higher priority to replication studies.

Keywords

citation history scientific methodology motivation goals reward replicability replication reproducibility

Get full access to this article

View all access options for this article.

References

Allison

P. D.

Waterman

R. P.

(2002). Fixed–effects negative binomial regression models. Sociological Methodology, 32(1), 247–265. https://doi.org/10.1111/1467-9531.00117

Alter

Gonzalez

(2018). Responsible practices for data sharing. American Psychologist, 73(2), 146–156. https://doi.org/10.1037/amp0000258

Angrist

J. D.

Pischke

J.-S.

(2009). Mostly harmless econometrics: An empiricist’s companion (1st ed.). Princeton University Press.

Begley

C. G.

Ellis

L. M.

(2012). Raise standards for preclinical cancer research. Nature, 483(7391), 531–533. https://doi.org/10.1038/483531a

Bolland

M. J.

Grey

Avenell

(2022). Citation of retracted publications: A challenging problem. Accountability in Research, 29(1), 18–25. https://doi.org/10.1080/08989621.2021.1886933

Brooke-Hitching

(2018). The phantom atlas: The greatest myths, lies and blunders on maps. Chronicle Books.

Cameron

A. C.

Miller

D. L.

(2015). A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50(2), 317–372. https://doi.org/10.3368/jhr.50.2.317

Carney

D. R.

Cuddy

A. J. C.

Yap

A. J.

(2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21(10), 1363–1368. https://doi.org/10.1177/0956797610383437

Center for Open Science. (2021). Registered reports. https://www.cos.io/initiatives/registered-reports

10.

Cesario

Jonas

K. J.

Carney

D. R.

(2017). CRSP special issue on power poses: What was the point and what did we learn? Comprehensive Results in Social Psychology, 2(1), 1–5. https://doi.org/10.1080/23743603.2017.1309876

11.

Chawla

D. S.

(2021, February 2). New bot flags scientific studies that cite retracted papers. Nature Index. https://www.natureindex.com/news-blog/new-bot-flags-scientific-research-studies-that-cite-retracted-papers

12.

Chivers

(2019). What’s next for psychology’s embattled field of social priming. Nature, 576(7786), 200–202. https://doi.org/10.1038/d41586-019-03755-2

13.

Colbert

(Writer). (2013, April 23). Austerity’s spreadsheet error—Thomas Herndon (Season 9, Episode 90) [TV series episode]. In The Colbert Report. Comedy Central. https://www.cc.com/video/kbgnf0/the-colbert-report-austerity-s-spreadsheet-error-thomas-herndon

14.

Ding

(2019). A bracketing relationship between difference-in-differences and lagged-dependent-variable adjustment. Political Analysis, 27(4), 605–615. https://doi.org/10.1017/pan.2019.25

15.

Dreber

Pfeiffer

Almenberg

Isaksson

Wilson

Chen

Nosek

B. A.

Johannesson

(2015). Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences, USA, 112(50), 15343–15347. https://doi.org/10.1073/pnas.1516179112

16.

Fleischmann

Pons

(1989). Electrochemically induced nuclear fusion of deuterium. Journal of Electroanalytical Chemistry, 261(2A), 301–308.

17.

Google. (2021). About Google Scholar. https://scholar.google.com/intl/en/scholar/about.html

18.

Herndon

Ash

Pollin

(2014). Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38(2), 257–279. https://doi.org/10.1093/cje/bet075

19.

Herring

(2009). Does diversity pay? Race, gender, and the business case for diversity. American Sociological Review, 74(2), 208–224. https://doi.org/10.1177/000312240907400203

20.

Hilsenrath

J. E.

(2005, October 24). Novel way to assess school competition stirs academic row. Wall Street Journal. https://www.wsj.com/articles/SB113011672134577225

21.

Hoxby

C. M.

(2000). Does competition among public schools benefit students and taxpayers? The American Economic Review, 90(5), 1209–1238.

22.

Hoxby

C. M.

(2007). Does competition among public schools benefit students and taxpayers? Reply. The American Economic Review, 97(5), 2038–2055. https://doi.org/10.1257/aer.97.5.2038

23.

Iacobucci

Schneider

M. J.

Popovich

D. L.

Bakamitsos

G. A.

(2016). Mean centering helps alleviate “micro” but not “macro” multicollinearity. Behavior Research Methods, 48(4), 1308–1317. https://doi.org/10.3758/s13428-015-0624-x

24.

Ioannidis

J. P. A.

(2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA, 294(2), 218–228. https://doi.org/10.1001/jama.294.2.218

25.

Ioannidis

J. P. A.

(2008). Why most discovered true associations are inflated. Epidemiology, 19(5), 640–648.

26.

Ioannidis

J. P. A.

(2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7(6), 645–654. https://doi.org/10.1177/1745691612464056

27.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

28.

Krugman

(2013, April 19). The excel depression. The New York Times. https://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html

29.

Long

J. S.

Ervin

L. H.

(2000). Using heteroscedasticity consistent standard errors in the linear regression model. The American Statistician, 54(3), 217–224. https://doi.org/10.1080/00031305.2000.10474549

30.

Nosek

B. A.

Ebersole

C. R.

DeHaven

A. C.

Mellor

D. T.

(2018). The preregistration revolution. Proceedings of the National Academy of Sciences, USA, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

31.

Nosek

B. A.

Lakens

(2014). Registered reports. Social Psychology, 45(3), 137–141. https://doi.org/10.1027/1864-9335/a000192

32.

Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of Psychological Science. Perspectives on Psychological Science, 7(6), 657–660. https://doi.org/10.1177/1745691612462588

33.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). https://doi.org/10.1126/science.aac4716

34.

Open Science Collaboration. (2016). Reproducibility Project: Psychology. OSF. https://osf.io/ezcuj/

35.

Oransky

(2019, June 12). Want to check for retractions in your personal library—And get alerts—For free? Now you can. Retraction Watch. https://retractionwatch.com/2019/06/12/want-to-check-for-retractions-in-your-personal-library-and-get-alerts-for-free-now-you-can/

36.

Puhani

P. A.

(2012). The treatment effect, the cross difference, and the interaction term in nonlinear “difference-in-differences” models. Economics Letters, 115(1), 85–87. https://doi.org/10.1016/j.econlet.2011.11.025

37.

Ranehill

Dreber

Johannesson

Leiberg

Sul

Weber

R. A.

(2015). Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychological Science, 26(5), 653–656. https://doi.org/10.1177/0956797614553946

38.

R Core Team. (2021). R: A language and environment for statistical computing (Version 4.0.5) [Computer software]. R Foundation for Statistical Computing. http://www.R-project.org

39.

Reinhart

Rogoff

(2010). Growth in a time of debt. American Economic Review: Papers & Proceedings, 100, 573–578.

40.

Rothstein

(2007a). Rejoinder to Hoxby. https://eml.berkeley.edu/~jrothst/publications/rothstein-hoxbycomment-rejoinder.pdf

41.

Rothstein

(2007b). Does competition among public schools benefit students and taxpayers? Comment. American Economic Review, 97(5), 2026–2037. https://doi.org/10.1257/aer.97.5.2026

42.

Rovira

Codina

Guerrero-Solé

Lopezosa

(2019). Ranking by relevance and citation counts, a comparative study: Google Scholar, Microsoft Academic, WoS and Scopus. Future Internet, 11(9), 202. https://doi.org/10.3390/fi11090202

43.

Serra-Garcia

Gneezy

(2021). Nonreplicable publications are cited more than replicable ones. Science Advances, 7(21), Article eabd1705. https://doi.org/10.1126/sciadv.abd1705

44.

Simmons

J. P.

Simonsohn

(2017). Power posing: P-curving the evidence. Psychological Science, 28(5), 687–693. https://doi.org/10.1177/0956797616658563

45.

Stojmenovska

Bol

Leopold

(2017). Does diversity pay? A replication of Herring (2009). American Sociological Review, 82(4), 857–867. https://doi.org/10.1177/0003122417714422

46.

Taubes

(1993). Bad science: The short life and weird times of cold fusion (1st ed.). Random House.

47.

Turner

B. O.

Paul

E. J.

Miller

M. B.

Barbey

A. K.

(2018). Small sample sizes reduce the replicability of task-based fMRI studies. Communications Biology, 1(1), Article 62. https://doi.org/10.1038/s42003-018-0073-z

48.

van Aert

R. C. M.

van Assen

M. A. L. M.

(2018). Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication. Behavior Research Methods, 50(4), 1515–1539. https://doi.org/10.3758/s13428-017-0967-6

49.

Keirstead

Jefferis

Getzinger

Cimentada

Czapanskiy

Makowski

(2021). scholar: Analyse citation data from Google Scholar (Version 0.2.0) [Computer software]. Retrieved from http://cran.r-project.org/package=scholar

50.

Zotero. (2019). Retracted item notifications with Retraction Watch integration. https://www.zotero.org/blog/retracted-item-notifications/

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.39 MB