This essay proposes ways to improve editorial evaluations of manuscripts and to make published research more reliable and trustworthy. It points to troublesome properties of current editorial practices and suggests that editorial evaluations could become more reliable by making more allowance for reviewers’ human limitations. The essay also identifies some troublesome properties of prevalent methodology, such as statistical significance tests, HARKing, and p-Hacking, and proposes editorial policies to mitigate these detrimental behaviors.
AcharyaA.VerstakA.SuzukiH.HendersonS.IakhiaevM.ChiungC.LinY.ShettyN.2014 “Rise of the rest: The growing impact of non-elite journals.” Google, Inc., October9. arXiv:1410.2217.
2.
ArmstrongJ. S.1980 “Unintelligible management research and academic prestige.” Interfaces, 10: 80–86.
3.
BailarJ. C.1991 “Reliability, fairness, objectivity and other inappropriate goals in peer review.” Behavioral and Brain Sciences, 14: 137–138.
4.
BedeianA. G.2004 “Peer review and the social construction of knowledge in the management discipline.” Academy of Management Learning and Education, 3: 198–216.
5.
BedeianA. G.2008 “Balancing authorial voice and editorial omniscience: The ‘It’s my paper and I’ll say what I want to’/’Ghostwriters in the sky’ minuet.” In BaruchY.KonradA.AguinisH.StarbuckW. H. (eds.), Opening the Black Box of Editorship: 134–142. Basingstoke, UK: Palgrave Macmillan.
6.
BedeianA. G.SturmanM. C.StreinerD. L.2009 “Decimal dust, significant digits, and the search for stars.” Organizational Research Methods, 12: 687–694.
7.
BedeianA. G.TaylorS. G.MillerA. N.2010 “Management science on the credibility bubble: Cardinal sins and various misdemeanors.” Academy of Management Learning and Education, 9: 715–725.
8.
BettisR. A.EthirajS.GambardellaA.HelfatC.MitchellW.2015 “Creating repeatable cumulative knowledge in strategic management.” Strategic Management Journal (forthcoming), published online ahead of print. DOI: 10.1002/smj.2477.
9.
BlankR. M.1991 “The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review.” American Economic Review, 81: 1041–1067.
10.
BonesA. K.2012 “We knew the future all along: Scientific hypothesizing is much more accurate than other forms of precognition—A satire in one part.” Perspectives on Psychological Science, 7: 307–309.
11.
BornmannL.MungraP.2011 “Improving peer review in scholarly journals.” European Science Editing, 37 (2): 41–43.
12.
BranchM.2014 “Malignant side-effects of null-hypothesis testing.” Theory and Psychology, 24: 256–277.
13.
BrembsB.ButtonK.MunafoM.2013 “Deep impact: Unintended consequences of journal rank.” Frontiers in Human Neuroscience, 7 (291). DOI: 10.3389/fnhum.2013.00291.
14.
CicchettiD. V.2003 “The peer review of scientific documents: Suggestions for improvements.” Presentation to the Committee on Research in Education, National Research Council, February 26.
CummingG.FidlerF.LeonardM.KalinowskiP.ChristiansenA.KleinigA.LoJ.McMenaminN.WilsonS.2007 “Statistical reform in psychology: Is anything changing?” Psychological Science, 18: 230–232.
17.
EinhornH. J.HogarthR. M.1975 “Unit weighting schemes for decision making.” Organizational Behavior and Human Performance, 13: 171–192.
18.
EllisonG.2002 “The slowdown of the economics publishing process.” Journal of Political Economy, 110: 947–993.
19.
ErevI.BarronG.2005 “On adaptation, maximization, and reinforcement learning among cognitive strategies.” Psychological Review, 112: 912–931.
20.
ErikssonK.2012 “The nonsense math effect.” Judgment and Decision Making, 7: 746–749.
21.
FidlerF.2005 “From statistical significance to effect estimation: Statistical reform in psychology, medicine and ecology.” Doctoral dissertation, University of Melbourne. Accessed at http://www.latrobe.edu.au/psy/staff/fidler.html.
22.
FidlerF.CummingG.BurgmanM.ThomasonN.2004 “Statistical reform in medicine, psychology and ecology.” Journal of Socio-Economics, 33: 615–630.
23.
ForguesB.LiarteS.2013 “Academic publishing: Past and future.” M@n@gement, 16: 739–756.
24.
FrancisG.TanzmanJ.MatthewsW. J.2014 “Excess success for psychology articles in the journalScience.” PLoS One, 9 (12): e114255.
25.
GauchH. G.Jr.2006 “Winning the accuracy game.” American Scientist, 94: 135–143.
26.
GottfredsonS. D.1978 “Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments.” American Psychologist, 33: 920–934.
27.
HargensL. L.HertingJ. R.1990 “Neglected considerations in the analysis of agreement among journal referees.” Scientometrics, 19: 91–106.
28.
HeintzelmanM.NocettiD.2009 “Where should we submit our manuscript? An analysis of journal submission strategies.” Berkeley Electronic Journal of Economic Analysis and Policy. Available at http://ssrn.com/abstract=1314850.
29.
HollenbeckJ. R.MannorM. J.2007 “Career success and weak paradigms: The role of activity, resiliency, and true scores.” Journal of Organizational Behavior, 28: 933–942.
30.
HooverK. D.1995 “In defense of data mining: Some preliminary thoughts.” In HooverK. D.SheffrinS. M. (eds.), Monetarism and the Methodology of Economics: Essays in Honour of Thomas Mayer: 242–257. Cheltenham, UK: Edward Elgar.
31.
HubbardR.2016Corrupt Research: The Case for Reconceptualizing Empirical Management and Social Science. Thousand Oaks, CA: Sage.
32.
HubbardR.LindsayR. M.2008 “Why p values are not a useful measure of evidence in statistical significance testing.” Theory and Psychology, 18: 69–88.
33.
IoannidisJ. P. A.2005 “Why most published research findings are false.” PLoS Med, 2 (8): e124. DOI: 10.1371/journal.pmed.0020124.
34.
KepesS.McDanielM. A.2013 “How trustworthy is the scientific literature in industrial and organizational psychology?” Industrial and Organizational Psychology, 6: 252–268.
35.
KieslerC. A.1991 “Confusion between reviewer reliability and wise editorial and funding decisions.” Behavioral and Brain Sciences, 14: 151–152.
36.
LabandD. N.PietteM. J.1994 “Favoritism versus search for good papers: Empirical evidence regarding the behavior of journal editors.” Journal of Political Economy, 102: 194–203.
37.
LeaheyE.2005 “Alphas and asterisks: The development of statistical significance testing standards in sociology.” Social Forces, 84: 1–24.
38.
LichtensteinS.FischhoffB.PhillipsL.1982 “Calibration probabilities: The state of the art to 1980.” In KahnemanD.SlovicP.TverskyA. (eds.), Judgment under Uncertainty: Heuristics and Biases: 306–333. Cambridge: Cambridge University Press.
39.
LovellM. C.1983 “Data mining.” Review of Economics and Statistics, 65: 1–12.
MayerT.1993Truth versus Precision in Economics, chapter 10. Cheltenham, UK: Edward Elgar.
42.
MazzolaJ. J.DeulingJ. K.2013 “Forgetting what we learned as graduate students: HARKing and selective outcome reporting in I–O journal articles.” Industrial and Organizational Psychology, 6: 279–284.
43.
McBrideM. F.FidlerF.BurgmanM. A.2012 “Evaluating the accuracy and calibration of expert predictions under uncertainty: Predicting the outcomes of ecological research.” Diversity and Distributions, 18: 782–794.
44.
McShaneB. B.GalD.2015 “Blinding us to the obvious? The effect of statistical training on the evaluation of evidence.” Management Science (forthcoming), published online ahead of print. Available at http://dx.doi.org/10.1287/mnsc.2015.2212.
MillerC. C.2006 “Peer review in the organizational and management sciences: Prevalence and effects of reviewer hostility, bias, and dissensus.” Academy of Management Journal, 49: 425–431.
47.
MillerC. C.Van de VenA.2015 “Peer review, root canals, and other amazing life events.” Academy of Management Discoveries (forthcoming), published online ahead of print. DOI: 10.5465/amd.2015.0039.
48.
NelsonR. R.WinterS. G.2002 “Evolutionary theorizing in economics.” Journal of Economic Perspectives, 16 (2): 23–46.
49.
NylennaM.RiisP.KarlssonY.1994 “Multiple blinded reviews of the same two manuscripts: Effects of referee characteristics and publication language.” JAMA, 272: 149–151.
50.
Open Science Collaboration2015 “Estimating the reproducibility of psychological science.” Science, 349 (6251). DOI: 10.1126/science.aac4716.
51.
PantP. N.StarbuckW. H.1990 “Innocents in the forest: Forecasting and research methods.” Journal of Management, 16: 433–460.
52.
PetersD. P.CeciS. J.1982 “Peer-review practices of psychological journals: The fate of published articles, submitted again.” Behavioral and Brain Sciences, 5: 187–255.
53.
RosenbergB.HougletM.1974 “Error rates in CRSP and Compustat data bases and their implications.” Journal of Finance, 29: 1303–1310.
54.
San MiguelJ. G.1977 “The reliability of R&D data in Compustat and 10-K reports.” Accounting Review, 52: 638–641.
55.
SchwabA.AbrahamsonE.StarbuckW. H.FidlerF.2011 “Researchers should make thoughtful assessments instead of null-hypothesis significance tests.” Organization Science, 22: 1105–1120.
56.
ShashokK.2008 “Content and communication: How can peer review provide helpful feedback about the writing?” BMC Medical Research Methodology, 8 (3): 1–9.
57.
ShroutP.1997 “Should significance tests be banned? Introduction to a special section exploring the pros and cons.” Psychological Science, 8: 1–2.
58.
SilerK.LeeK.BeroL.2015 “Measuring the effectiveness of scientific gatekeeping.” PNAS (Proceedings of the National Academy of Sciences), 112 (2): 360–365.
59.
SimmonsJ. P.NelsonL. D.SimonsohnU.2011 “False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.” Psychological Science, 22: 1359–1366.
60.
StarbuckW. H.2005 “How much better are the most prestigious journals? The statistics of academic publication.” Organization Science, 16: 180–200.
61.
StarbuckW. H.2006The Production of Knowledge. Oxford: Oxford University Press.
62.
StarbuckW. H.2009 “Cognitive reactions to rare events: Perceptions, uncertainty, and learning.” Organization Science, 20: 925–937.
63.
StarbuckW. H.2013 “Why and where do academics publish?” M@n@gement, 16: 707–718.
64.
StarbuckW. H.AguinisH.KonradA. M.BaruchY.2008 “Tradeoffs among editorial goals in complex publishing environments.” In BaruchY.KonradA.AguinisH.StarbuckW. H. (eds.), Opening the Black Box of Editorship: 250–270. Basingstoke, UK: Palgrave Macmillan.
65.
TardyC. M.MatsudaP. K.2009 “The construction of author voice by editorial board members.” Written Communication, 26: 32–52.
66.
WareM.2008 “Peer review: Benefits, perceptions and alternatives.” London: Publishing Research Consortium.
67.
WebbT. J.O’HaraB.FreckletonR. P.2008 “Does double-blind review benefit female authors?” Trends in Ecology and Evolution, 23: 351–353.
68.
WebsterE. J.StarbuckW. H.1988 “Theory building in industrial and organizational psychology.” In CooperC. L.RobertsonI. (eds.), International Review of Industrial and Organizational Psychology: 93–138. London: Wiley.
69.
YankauerA.1991 “How blind is blind review?” American Journal of Public Health, 81: 843–845.
70.
YoungN. S.IoannidisJ. P. A.Al-UbaydliO.2008 “Why current publication practicesmaydistort science.” PLoS Med, 5 (10): e201. DOI: 10.1371/journal.pmed.0050201.
