Ronald Fisher created statistical significance tests to provide an easy method anyone could perform. Their simplicity and general applicability spurred adoption, and they became universal in statistical training, and universal training made these tests universal in social science. Editors and reviewers expected to see statistical significance in every paper. But the method has serious deficiencies. Today's more advanced computational capabilities have created opportunities to address these deficiencies and to use statistical analyses that provide better information. This essay introduces four lessons we have learned during our two-decade effort to inform management scholars about limitations of statistical significance tests. First, methodological change is generational and benefits from a focus on doctoral students. Second, criticizing the status quo is not enough: introducing and teaching alternative approaches is essential. Third, in a publish-or-perish world, change initiatives must address publication. Fourth, to speed up progress, leadership by academic organizations and journal editors is essential.
AguinisH.WernerS.Lanza AbbottJ.AngertC.ParkJ. H.KohlhausenD. (2010). Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13(3), 515–539. https://doi.org/10.1177/1094428109333339
BayesT.PriceR. (1763). An essay towards solving a problem in the doctrine of chance. By the late Rev. Mr. Bayes, communicated by Mr. Price, in a letter to John Canton, A. M. F. R. S. Philosophical Transactions of the Royal Society of London, 53, 370–418. https://doi.org/10.1098/rstl.1763.0053
ConnorE. F.SimberloffD. (1983). Interspecific competition and species co-occurrence patterns on islands: Null models and the evaluation of evidence. Oikos, 41, 455–465. https://doi.org/10.2307/3544105
7.
ConnorE. F.SimberloffD. (1986). Competition, scientific method, and null models in ecology. American Scientist, 74(2), 155–162. http://www.jstor.org/stable/27854031
8.
CummingG. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge/Taylor & Francis Group.
9.
DenrellJ.FangC.LiuC. (2015). Chance explanations in the management sciences. Organization Science, 26(3), 923–940. https://doi.org/10.1287/orsc.2014.0946
10.
DuttonJ. M.StarbuckW. H. (1971). The history of simulation models. In DuttonJ. M.StarbuckW. (Eds.), Computer simulation of human behavior (pp. 9–102). Wiley.
11.
ElliottJ. W. (1973). A direct comparison of short-run GNP forecasting models. Journal of Business, 46(1), 33–60. https://doi.org/10.1086/295506
12.
FidlerF.CummingG.BurgmanM.ThomasonN. (2004). Statistical reform in medicine, psychology, and ecology. Journal of Socio-Economics, 33(5), 615–630. https://doi.org/10.1016/j.socec.2004.09.035
13.
FischerH. (2010). A history of the central limit theorem: From classical to modern probability theory. Springer.
14.
FisherR. A. (1925). Statistical methods for research workers. Oliver and Boyd.
15.
FisherR. A. (1935). The design of experiments. Oliver and Boyd.
16.
GelmanA.CarlinJ. B.SternH. S.DunsonD. B.VehtariA.RubinD. B. (2014). Bayesian data analysis (3rd ed). Chapman & Hall/CRC.
17.
GreenlandS.SennS. J.RothmanK. J. (2016). Statistical tests, p-values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31, 337–350. https://doi.org/10.1007/s10654-016-0149-3
18.
HallerH.KraussS. (2002). Misinterpretations of significance: A problem students share with their teachers?Methods of Psychology Research, 7(1), 1–20.
19.
HubbardR.ArmstrongJ. S. (2006). Why we don't really know what statistical significance means: Implications for educators. Journal of Marketing Education, 28(2), 114–120. https://doi.org/10.1177/0273475306288399
20.
MayoD. (2006). Statistics, philosophy of. In S. Sarkar & J. Pfeifer (Eds.), The philosophy of science: An encyclopedia (pp. 802–815). Routledge.
21.
McElreathR. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan (2nd ed). CRC Press.
22.
McGrayneS. B. (2011). The theory that would not die: How Bayes’ rule cracked the Enigma code, hunted down Russian submarines, and emerged triumphant from two centuries of controversy. Yale University Press.
23.
NeymanJ.PearsonE. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London A, 231(694–706), 289–337. https://doi.org/10.1098/rsta.1933.0009
24.
OakesM. W. (1986). Statistical inference: A commentary for the social and behavioural sciences. Wiley.
25.
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 943. aac4716. https://doi.org/10.1126/science.aac4716
26.
PeachJ. T.WebbJ. L. (1983). Randomly specified macroeconomic models: Some implications for model election. Journal of Economic Issues, 17(3), 697–720. https://doi.org/10.1080/00213624.1983.11504150
27.
SalsburgD. (2002). The lady tasting tea: How statistics revolutionized science in the twentieth century. Henry Holt.
SchwabA.StarbuckW. H. (2012). Using baseline models to improve theories about emerging markets. In WangC.BerghD.KetchenD. (Eds.), Research methodology in strategy and management (vol. 7, pp. 3–33). Bingley.
30.
SchwabA.StarbuckW. H. (2013). Why baseline modeling is better than null-hypothesis testing: Examples from research about international management, developing countries, and emerging markets. In DevinneyT.PedersenT.TihanyiL. (Eds.), Advances in international management, 26 (pp. 171–195). Emerald.
StangA.DeckertM.PooleC.RothmanK. J. (2016). Statistical inference in abstracts of major medical and epidemiology journals 1975–2014: A systematic review. European Journal of Epidemiology, 32, 21–29. https://doi.org/10.1007/s10654-016-0211-1
WassersteinR. L.LazarN. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108
WebsterJ.StarbuckW. H. (1988). Theory building in industrial and organizational psychology. In CooperC. L.RobertsonI. T. (Eds.), International review of industrial and organizational psychology (pp. 93–138). Wiley.
