Forcing a Deterministic Frame on Probabilistic Phenomena: A Communication Blind Spot in Media Coverage of the “Replication Crisis”

Abstract

The current controversy surrounding research replication in biomedical and psychosocial sciences often overlooks the uncertainties surrounding both the original and replication studies. Overemphasizing single attempts as definitive replication successes or failures, as exemplified by media coverage of the landmark Reproducibility Project: Psychology, fosters misleading dichotomies and erodes public trust. To avoid such unintended consequences, science communicators should more clearly articulate statistical variation and other uncertainty sources in replication, while emphasizing the cumulative nature of science in general and replication in particular.

Keywords

bias confidence intervals hypothesis tests replication reproducibility statistical significance trust in science uncertainty

Get full access to this article

View all access options for this article.

References

Altman

(1994). The scandal of poor medical research. British Medical Journal, 308, 283–284. https://doi.org/10.1136/bmj.308.6941.1438b

Amrhein

Trafimow

Greenland

(2019). Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication. The American Statistician, 73(Suppl. 1), 262–270. https://doi.org/10.1080/00031305.2018.1543137

Bakker

van Dijk

Wicherts

J. M.

(2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543–554. https://doi.org/10.1177/1745691612459060

Bower

(2015, August 27). Psychology results evaporate upon further review. Science News. https://www.sciencenews.org/article/psychology-results-evaporate-upon-further-review

Boykoff

M. T.

(2013). Public enemy no. 1? Understanding media representations of outlier views on climate change. American Behavioral Scientist, 57(6), 796–817. https://doi.org/10.1177/0002764213476846

Boykoff

M. T.

Boykoff

J. M.

(2007). Climate change and journalistic norms: A case-study of US mass-media coverage. Geoforum, 38(6), 1190–1204. https://doi.org/10.1016/j.geoforum.2007.01.008

Button

K. S.

Ioannidis

J. P.

Mokrysz

Nosek

B. A.

Flint

Robinson

E. S.

Munafò

M. R.

(2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org/10.1038/nrn3475

Camerer

C. F.

Dreber

Holzmeister

T.-H.

Huber

Johannesson

Kirchler

Nave

Nosek

B. A.

Pfeiffer

Altmejd

Buttrick

Chan

Chen

Forsell

Gampa

Heikensten

Hummer

Imai

(2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644. https://doi.org/10.1038/s41562-018-0399-z

Collins

(1985). Changing order: Replication and induction in scientific practice. SAGE.

10.

Earp

B. D.

Everett

J. A. C.

(2015, October 25). How to fix psychology’s replication crisis. The Chronicle of Higher Education. https://www.chronicle.com/article/how-to-fix-psychologys-replication-crisis/

11.

Errington

T. M.

Mathur

Soderberg

C. K.

Denis

Perfito

Iorns

Nosek

B. A.

(2021). Investigating the replicability of preclinical cancer biology. eLife, 10, Article e71601. https://doi.org/10.7554/elife.71601

12.

Gelman

(2016). The problems with p-values are not just with p-values. The American Statistician, Supplemental Materials to ASA Statement on P-values and Statistical Significance, 70, 1–2. http://www.stat.columbia.edu/~gelman/research/published/asa_pvalues.pdf

13.

Gelman

Carlin

(2014). Beyond power calculations. Perspectives on Psychological Science, 9(6), 641–651. https://doi.org/10.1177/1745691614551642

14.

Gelman

Loken

(2014). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. https://statmodeling.stat.columbia.edu/2021/03/16/the-garden-of-forking-paths-why-multiple-comparisons-can-be-a-problem-even-when-there-is-no-fishing-expedition-or-p-hacking-and-the-research-hypothesis-was-posited-ahead-of-time-2/

15.

Gelman

Stern

(2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328–331. https://doi.org/10.1198/000313006X152649

16.

Goodman

S. N.

(1992). A comment on replication, P-values and evidence. Statistics in Medicine, 11(7), 875–879. https://doi.org/10.1002/sim.4780110705

17.

Goodman

S. N.

Fanelli

Ioannidis

J. P.

(2016). What does research reproducibility mean? Science Translational Medicine, 8(341), 341ps12. https://doi.org/10.1126/scitranslmed.aaf5027

18.

Greenland

(2017). The need for cognitive science in methodology. American Journal of Epidemiology, 186, 639–645. https://academic.oup.com/aje/article/186/6/639/3886035

19.

Greenland

(2019). Valid P-values behave exactly as they should: Some misleading criticisms of P-values and their resolution with S-values. The American Statistician, 73(Suppl. 1), 106–114. https://doi.org/10.1080/00031305.2018.1529625

20.

Greenland

Mansournia

M. A.

Joffe

(2022). To curb research misreporting, replace significance and confidence by compatibility. Preventive Medicine, 164, 107–127. https://doi.org/10.1016/j.ypmed.2022.107127

21.

Greenland

O’Rourke

(2008). Ch. 33 Meta-analysis. In Rothman

K. J.

Greenland

Lash

T. L.

(Eds.), Modern epidemiology (3rd ed.). (pp. 652–682). Lippincott-Wolters-Kluwer.

22.

Greenland

Senn

S. J.

Rothman

K. J.

Carlin

J. B.

Poole

Goodman

S. N.

Altman

D. G.

(2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. The American Statistician, 70. https://amstat.tandfonline.com/doi/suppl/10.1080/00031305.2016.1154108/suppl_file/utas_a_1154108_sm5368.pdf

23.

Jamieson

K. H.

(2018). Crisis or self-correction: Rethinking media narratives about the well-being of science. Proceedings of the National Academy of Sciences, 115(11), 2620–2627. https://doi.org/10.1073/pnas.1708276114

24.

Johnson

C. K.

(2021, December 7). Study can’t confirm lab results for many cancer experiments. The Associated Press. https://apnews.com/article/science-business-health-cancer-marcia-mcnutt-93219170405e3de753651b89d4308461

25.

Luntz

(2021, May 22). Studies that are exciting but less likely to be true are cited more often in academia. Iflscience. https://www.iflscience.com/studies-that-are-exciting-but-less-likely-to-be-true-are-cited-more-often-in-academia-59798

26.

Martin

(2015, December 11). Wee problem with credibility of studies. The Sydney Morning Herald. https://www.smh.com.au/opinion/wee-problem-with-credibility-of-studies-20151211-gll6ic.html

27.

Maxwell

S. E.

Lau

M. Y.

Howard

G. S.

(2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487–498. https://doi.org/10.1037/a0039400

28.

Mayr

(1985). How biology differs from the physical sciences. In Depew

Weber

(Eds.), Evolution at a crossroads: The new biology and the new philosophy of science (pp. 43–63). A Bradford Book.

29.

McShane

B. B.

Bradlow

E. T.

Lynch

J. G.

Meyer

R. J.

(2024). “Statistical significance” and statistical reporting: moving beyond binary. Journal of Marketing. Advance online publication. https://doi.org/10.1177/00222429231216910

30.

McShane

B. B.

Gal

Gelman

Robert

Tackett

J. L.

(2019). Abandon statistical significance. The American Statistician, 73(Suppl. 1), 235–245.

31.

Meng

(2020). Reproducibility, replicability, and reliability. Harvard Data Science Review, 2(4). https://doi.org/10.1162/99608f92.dbfce7f9

32.

Mulkay

(1984). The scientist talks back: A one-act play, with a moral, about replication in science and reflexivity in sociology. Social Studies of Science, 14(2), 265–282. https://doi.org/10.1177/030631284014002008

33.

Nelson

R. R.

(2016). The sciences are different and the differences matter. Research Policy, 45(9), 1692–1701. https://doi.org/10.1016/j.respol.2015.05.014

34.

Open Science Collaboration. (2015a). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

35.

Open Science Collaboration. (2015b, August 27). Massive collaboration testing reproducibility of psychology studies publishes findings [Press release]. https://www.cos.io/about/news/massive-collaboration-testing-reproducibility-psychology-studies-publishes-findings

36.

Peterson

Panofsky

(2023). Metascience as a scientific social movement. Minerva, 61(2), 147–174. https://doi.org/10.1007/s11024-023-09490-3

37.

Rafi

Greenland

(2020). Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise. BMC Medical Research Methodology, 20, 244. https://doi.org/10.1186/s12874-020-01105-9

38.

Sample

(2021, May 21). Research findings that are probably wrong cited far more than robust ones, study finds. The Guardian. https://www.theguardian.com/science/2021/may/21/research-findings-that-are-probably-wrong-cited-far-more-than-robust-ones-study-finds#:~:text=2%20years%20old-,Research%20findings%20that%20are%20probably%20wrong%20cited,than%20robust%20ones%2C%20study%20finds&text=Scientific%20research%20findings%20that%20are,for%20papers%20with%20grabbier%20conclusions

39.

Schafmeister

(2021). The effect of replications on citation patterns: Evidence from a large-scale reproducibility project. Psychological Science, 32(10), 1537–1548. https://doi.org/10.1177/09567976211005767

40.

Schneider

(2010). Making space for the “nuances of truth”: Communication and uncertainty at an environmental journalists’ workshop. Science Communication, 32(2), 171–201. https://doi.org/10.1177/1075547009340344

41.

Schoenfeld

J. D.

Ioannidis

J. P.

(2013). Is everything we eat associated with cancer? A systematic cookbook review. The American Journal of Clinical Nutrition, 97(1), 127–134. https://doi.org/10.3945/ajcn.112.047142

42.

Serra-Garcia

Gneezy

(2021). Nonreplicable publications are cited more than replicable ones. Science Advances, 7(21), eabd1705. https://doi.org/10.1126/sciadv.abd1705

43.

Shiffrin

R. M.

Börner

Stigler

S. M.

(2018). Scientific progress despite irreproducibility: A seeming paradox. Proceedings of the National Academy of Sciences, 115(11), 2632–2639. https://doi.org/10.1073/pnas.1711786114

44.

Sismondo

(2010). An introduction to science and technology studies. Wiley-Blackwell.

45.

Smaldino

P. E.

McElreath

(2016). The natural selection of bad science. Royal Society Open Science, 3(9), 160384. https://doi.org/10.1098/rsos.160384

46.

van Zwet

Gelman

Greenland

Imbens

Schwab

Goodman

S. N

. (2024). A new look at P values for randomized clinical trials. NEJM Evidence. https://doi.org/10.1056/evidoa2300003

47.

van Zwet

Schwab

Greenland

. (2021). Addressing exaggeration of effects from single RCTs. Significance, 18(6), 16–21. https://doi.org/10.1111/1740-9713.01587

48.

von Hippel

P. T.

(2022). Is Psychological science self-correcting? Citations before and after successful and failed replications. Perspectives on Psychological Science, 17(6), 1556–1565. https://doi.org/10.1177/17456916211072525

49.

Wasserstein

R. L.

Lazar

N. A.

(2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108

50.

Wasserstein

R. L.

Schirm

A. L.

Lazar

N. A.

(2019). Moving to a world beyond “p < 0.05.” The American Statistician, 73(Suppl. 1), 1–19. https://doi.org/10.1080/00031305.2019.1583913

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB

0.29 MB

0.34 MB

0.02 MB