Sage Journals: Discover world-class research

Abstract

Recent controversies in psychology have spurred conversations about the nature and quality of psychological research. One topic receiving substantial attention is the role of replication in psychological science. Using the complete publication history of the 100 psychology journals with the highest 5-year impact factors, the current article provides an overview of replications in psychological research since 1900. This investigation revealed that roughly 1.6% of all psychology publications used the term replication in text. A more thorough analysis of 500 randomly selected articles revealed that only 68% of articles using the term replication were actual replications, resulting in an overall replication rate of 1.07%. Contrary to previous findings in other fields, this study found that the majority of replications in psychology journals reported similar findings to their original studies (i.e., they were successful replications). However, replications were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles. Moreover, despite numerous systemic biases, the rate at which replications are being published has increased in recent decades.

Keywords

replication research methodology content analysis

Get full access to this article

View all access options for this article.

References

Allport

G. W.

Kramer

B. M.

(1946). Some roots of prejudice. Journal of Psychology, 22, 9–39.

Bem

D. J.

(2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407–425. doi:10.1037/a0021524

Bem

D. J.

Utts

Johnson

W. O.

(2011). Must psychologists change the way they analyze their data? Journal of Personality and Social Psychology, 101, 716–719. doi:10.1037/a0024777

Campbell

Stanley

(1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand-McNally.

Carpenter

(2012). Psychology’s bold initiative. Science, 335, 1558–1561.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

Collins

H. M.

(1985). Changing order. London, England: SAGE.

Crocker

Cooper

M. L.

(2011). Addressing scientific fraud. Science, 334(6060), 1182. doi:10.1126/science.1216775

Evanschitzky

Baumgarth

Hubbard

Armstrong

J. S.

(2007). Replication research’s disturbing trend. Journal of Business Research, 60, 411–415. doi:10.1016/j.jbusres.2006.12.003

10.

Fanelli

(2010). “Positive” results increase down the hierarchy of the sciences. PloS ONE, 5, doi:10.1371/journal.pone.0010068

11.

Fanelli

(2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904. doi:10.1007/s11192-011-0494-7

12.

Hubbard

Armstrong

J. S.

(1994). Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing, 11, 233–248. doi:10.1016/0167-8116(94)90003-5

13.

Hunt

(1975). Do we really need more replications? Psychological Reports, 36, 587–593.

14.

Ioannidis

J. P. A.

(2005). Why most published research findings are false. PloS MEDICINE, 2, 696–701. doi:10.1371/journal.pmed.0020124

15.

Jasny

B. R.

Chin

Chong

Vignieri

(2011). Again, and again, and again. Introduction. Science, 334, 1225.

16.

Kelly

C. W.

Chase

L. J.

Tucker

R. K.

(1979). Replication in experimental communication research: An analysis. Human Communication Research, 5, 338–342.

17.

Lykken

D. T.

(1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151–159. doi:10.1037/h0026141

18.

Lyndsay

R. M.

Ehrenberg

A. S. C.

(1993). The design of replicated studies. The American Statistician, 47, 217–228. doi:10.2307/2684982

19.

Neuliep

J. W.

Crandall

(1993). Reviewer bias against replication research. Journal of Social Behavior and Personality, 8, 21–29.

20.

Roediger

H. L.

(2012). Psychology’s woes and a partial cure: The value of replication. Academic Observer. Retrieved from http://www.psychologicalscience.org/index.php/publications/observer/2012/february-12/psychologys-woes-and-a-partial-cure-the-value-of-replication.html

21.

Rosenblith

J. F.

(1949). A replication of some roots of prejudice. Journal of Abnormal and Social Psychology, 44, 470–489.

22.

Rosnow

R. L.

Rosenthal

(1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284. doi:10.1037//0003-066x.44.10.1276

23.

Schmidt

(2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90–100. doi:10.1037/a0015108

24.

Tukey

J. W.

(1969). Analyzing data: Sanctification or detective work. American Psychologist, 24, 83–91. doi:10.1037/h0027 108

25.

Wagenmakers

E.-J.

Wetzels

Borsboom

van der Maas

H. L. J.

(2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432. doi:10.1037/a0022790

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.03 MB

0.00 MB