Surrogate Science

Abstract

The application of statistics to science is not a neutral act. Statistical tools have shaped and were also shaped by its objects. In the social sciences, statistical methods fundamentally changed research practice, making statistical inference its centerpiece. At the same time, textbook writers in the social sciences have transformed rivaling statistical systems into an apparently monolithic method that could be used mechanically. The idol of a universal method for scientific inference has been worshipped since the “inference revolution” of the 1950s. Because no such method has ever been found, surrogates have been created, most notably the quest for significant p values. This form of surrogate science fosters delusions and borderline cheating and has done much harm, creating, for one, a flood of irreproducible results. Proponents of the “Bayesian revolution” should be wary of chasing yet another chimera: an apparently universal inference procedure. A better path would be to promote both an understanding of the various devices in the “statistical toolbox” and informed judgment to select among these.

Keywords

research methods regression analysis psychometrics Bayesian methods

Get full access to this article

View all access options for this article.

References

Aitkenhead

2013, December 6. Peter Higgs: I wouldn’t be productive enough for today’s academic system. The Guardian. Retrieved from http://www.theguardian.com/science/2013/dec/06/peter-higgs-boson-academic-system

Altmann

2007. Editorial journal policies and procedures. Cognition, 102: 1-6.

American Psychological Association. 1974. Publication manual of the American Psychological Association, 2nd ed. Baltimore, MD: Garamond/Pridemark.

Arbuthnott

1710. An argument for divine providence, taken from the constant regularity observ’d in the births of both sexes. Philosophical Transactions of the Royal Society, 27: 186-190.

Bacon

1902. Novum organum. New York: P. F. Collier & Son. (Original work published 1620)

Boring

E. G.

1963. History, psychology, and sciences: Selected papers. New York: Wiley.

Bozarth

J. D.

Roberts

R. R.

Jr.

1972. Signifying significant significance. American Psychologist, 27: 774-775.

Breiman

Friedman

J. H.

Olshen

R. A.

Stone

C. J.

1993. Classification and regression trees. New York: Chapman & Hall.

Danziger

1990. Constructing the subject: Historical origins of psychological research. Cambridge, UK: Cambridge University Press.

10.

Daston

1988. Classical probability in the Enlightenment. Princeton, NJ: Princeton University Press.

11.

Daston

2005. Scientific error and the ethos of belief. Social Research, 72: 1-28.

12.

Earman

1992. Bayes or bust? A critical examination of Bayesian confirmation theory. Cambridge, MA: MIT Press.

13.

Edwards

Lindman

Savage

L. J.

1963. Bayesian statistical inference for psychological research. Psychological Review, 70: 193-242.

14.

Fienberg

S. E.

2006. When did Bayesian inference become “Bayesian”? Bayesian Analysis, 1: 1-40.

15.

Fisher

R. A.

1955. Statistical methods and scientific induction. Journal of the Royal Statistical Society, 17 (Series B): 69-77.

16.

Fisher

R. A.

1956. Statistical methods and scientific inference. Edinburgh, UK: Oliver & Boyd.

17.

Fisher

R. A.

1971. The design of experiments, reprint of the 8th ed. New York: Hafner. (Original work published 1935)

18.

Franco

Z. E.

Blau

Zimbardo

P. G.

2011. Heroism: A conceptual analysis and differentiation between heroic action and altruism. Review of General Psychology, 15: 99-113.

19.

Geman

Bienenstock

Doursat

1992. Neural networks and the bias/variance dilemma. Neural Computation, 4: 1-58.

20.

Gerrig

R. J.

Zimbardo

P. G.

2002. Psychology and life, 16th ed. Boston: Allyn & Bacon.

21.

Gigerenzer

1987. Probabilistic thinking and the fight against subjectivity. In Krüger

Gigerenzer

Morgan

(Eds.), The probabilistic revolution: Vol. 2. Ideas in the sciences, 11-33. Cambridge, MA: MIT Press.

22.

Gigerenzer

2002. Calculated risks: How to know when numbers deceive you. New York: Simon & Schuster.

23.

Gigerenzer

2004. Mindless statistics. Journal of Socio-Economics, 33: 587-606.

24.

Gigerenzer

2011. What are natural frequencies? British Medical Journal, 343: d6386.

25.

Gigerenzer

Brighton

2009. Homo heuristicus: Why biased minds make better inferences. Topics in Cognitive Science, 1: 107-143.

26.

Gigerenzer

Hell

Blank

1988. Presentation and content: The use of base rates as a continuous variable. Journal of Experimental Psychology: Human Perception and Performance, 14: 513-525.

27.

Gigerenzer

Hertwig

Pachur

(Eds.). 2011. Heuristics: The foundations of adaptive behavior. Oxford, UK: Oxford University Press.

28.

Gigerenzer

Murray

D. J.

1987. Cognition as intuitive statistics. Hillsdale, NJ: Lawrence Erlbaum.

29.

Gigerenzer

Swijtink

Porter

Daston

Beatty

Krüger

1989. The empire of chance: How probability changed science and everyday life. Cambridge, UK: Cambridge University Press.

30.

Good

I. J.

1971. 46656 varieties of Bayesianism. American Statistician, 25: 62-63.

31.

Good

P. I.

Harding

J. W.

2003. Common errors in statistics (and how to avoid them). New York: Wiley.

32.

Guilford

J. P.

1942. Fundamental statistics in psychology and education. New York: McGraw-Hill.

33.

Hacking

1987. Prussian numbers 1860-1882. In Krüger

Daston

Heidelberger

(Eds.), The probabilistic revolution: Vol. 1. Ideas in history: 377-394. Cambridge, MA: MIT Press.

34.

Haller

Krauss

2002. Misinterpretations of significance: A problem students share with their teachers? Methods of Psychological Research—Online, 7: 1-20.

35.

Hubbard

Armstrong

J. S.

1994. Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing, 11: 233-248.

36.

Information for contributors. 2005. Psychological Science, 16: i.

37.

International Committee of Medical Journal Editors. 1988. Uniform requirements for manuscripts submitted to biomedical journals. British Medical Journal, 296: 401-405.

38.

Ioannidis

J. P. A.

Greenland

Hlatky

M. A.

Khoury

M. J.

Macleod

M. R.

Moher

. . . Tibshirani

2014. Increasing value and reducing waste in research design, conduct, and analysis. The Lancet, 383: 166-175.

39.

Iverson

G. J.

Lee

M. D.

Zhang

Wagenmakers

E. J.

2009. Prep: An agony in five Fits. Journal of Mathematical Psychology, 53: 195-202.

40.

Jeffreys

1961. Theory of probability. Oxford, UK: Oxford University Press.

41.

John

Loewenstein

G. F.

Prelec

2012. Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23: 524-532.

42.

Kahneman

Tversky

1972. Subjective probability: A judgment of representativeness. Cognitive Psychology, 3: 430-454.

43.

Kass

R. E.

Raftery

A. E.

1995. Bayes factors. Journal of the American Statistical Association, 90: 773-795.

44.

Kerr

N. L.

1998. HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2: 196-217.

45.

Knight

F. H.

1921. Risk, uncertainty, and profit. New York: Houghton Mifflin.

46.

Koehler

J. J.

1992. Probabilities in the courtroom: An evaluation of the objections and policies. In Kagehiro

D. K.

Laufer

W. S.

(Eds.), Handbook of psychology and law: 167-184. New York: Springer-Verlag.

47.

Krüger

Daston

L. J.

Heidelberger

(Eds.). 1987. The probabilistic revolution: Vol. 1. Ideas in history. Cambridge, MA: MIT Press.

48.

Krüger

Gigerenzer

Morgan

M. S.

(Eds.). 1987. The probabilistic revolution: Vol. 2. Ideas in the sciences. Cambridge, MA: MIT Press.

49.

Laplace

P. S.

1774. Mémoire sur la probabilité des causes par les évènemens. Mémoires de mathématique et de physique, presentés a l’Académie Royale des Sciences, par divers savans, & lû dans ses assemblées, 6: 621-656. (English translation in Stigler, S. M. 1986. Laplace’s 1774 memoir on inverse probability. Statistical Science, 1: 359-378.)

50.

Laplace

P. S.

1951. A philosophical essay on probabilities. New York: Dover. (Original work published 1814)

51.

Leibniz

G. W.

1951. Toward a universal characteristic. In Wiener

P. P.

(Ed.), Selections: 17-25. New York: Scribner’s. (Original work published 1677)

52.

Lindley

D. V.

1975. The future of statistics: A Bayesian 21st century. Advances in Applied Probability (Supplement), 7: 106-115.

53.

Lindley

D. V.

1983. Theory and practice of Bayesian statistics. Journal of the Royal Statistical Society. Series D (The Statistician), 32: 1-11.

54.

Lindley

D. V.

1986. Comment. American Statistician, 40: 6-7.

55.

Luce

R. D.

1988. The tools-to-theory hypothesis. Review of G. Gigerenzer & D. J. Murray, “Cognition as Intuitive Statistics.” Contemporary Psychology, 33: 582-583.

56.

Madden

C. S.

Easley

R. W.

Dunn

M. G.

1995. How journal editors view replication research. Journal of Advertising, 24: 77-79.

57.

Maslow

A. H.

1966. The psychology of science. New York: Harper & Row.

58.

Melton

A. W.

1962. Editorial. Journal of Experimental Psychology, 64: 553-557.

59.

Newton

1952. Opticks. Or a treatise of the reflections, refractions, inflections, and colours of light. New York: Dover. (Original work published 1704)

60.

Neyman

1957. Inductive behavior as a basic concept of philosophy of science. International Statistical Review, 25: 7-22.

61.

Nunnally

J. C.

1975. Introduction to statistics for psychology and education. New York: McGraw-Hill.

62.

Oakes

1986. Statistical inference: A commentary for the social and behavioral sciences. Chichester, UK: Wiley.

63.

Olby

2003. Quiet debut for the double helix. Nature, 276: 565-570.

64.

Rothman

K. J.

1998. Writing for Epidemiology. Epidemiology, 9: 333-337.

65.

Savage

L. J.

1972. The foundations of statistics. New York: Dover (Original work published 1954)

66.

Simkin

M. V.

Roychowdhury

V. P.

2003. Read before you cite! Complex Systems, 14: 269-274.

67.

Simmons

J. P.

Nelson

L. D.

Simonsohn

2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 20: 1-8.

68.

Simon

H. A.

1979. Rational-decision making in business organizations (Nobel Memorial Lecture, 8 December, 1978). American Economic Review, 69: 493-513.

69.

Simon

H. A.

1992. What is an “explanation” of behavior? Psychological Science, 3: 150-161.

70.

Soros

2009. The crash of 2008 and what it means: The new paradigm for financial markets. New York: Public Affairs.

71.

Stegmüller

1973. Jenseits von Popper und Carnap [Beyond Popper and Carnap]. Berlin, Germany: Springer.

72.

Stevens

S. S.

1960. The predicament in design and significance. Contemporary Psychology, 9: 273-276.

73.

Stigler

S. M.

1983. Who discovered Bayes’s theorem? American Statistician, 37: 290-296.

74.

Stigler

S. M.

1999. Statistics on the table: The history of statistical concepts and methods. Cambridge, MA: Harvard University Press.

75.

Style guide for authors. 2011. Academy of Management Journal, 54: 1081-1084.

76.

Tressoldi

P. E.

Giofré

Sella

Cumming

2013. High impact = high statistical standards? Not necessarily so. PLOS ONE, 8: 1-7.

77.

Trotta

2007. Applications of Bayesian model selection to cosmological parameters. Monthly Notices of the Royal Astronomical Society, 378: 72-82.

78.

Tukey

J. W.

1977. Exploratory data analysis. Reading, MA: Addison-Wesley.

79.

Vul

Harris

Winkielman

Pashler

2009. Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4: 274-290.

80.

Weick

K. E.

1996. Drop your tools: An allegory for organizational studies. Administrative Science Quarterly, 41: 301-313.

81.

Ziliak

S. T.

McCloskey

D. N.

2004. Size matters: The standard error of regressions in the American Economic Review. Journal of Socio-Economics, 33: 527-546.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB