Questions About Questionable Research Practices in the Field of Management

Abstract

The discussion regarding questionable research practices (QRPs) in management as well as the broader natural and social sciences has increased substantially in recent years. Despite the attention, questions remain regarding research norms and the implications for both theoretical and practical advancements. The aim of the current article is to address these issues in a question-and-answer format while drawing upon both past research and the results of a series of new studies conducted using a mixed-methods design. Our goal is to encourage a systematic, collegial, and constructive dialogue regarding QRPs in management research.

Keywords

questionable research practices (QRPs)research methodology philosophy of science

Get full access to this article

View all access options for this article.

References

Aguinis

Gottfredson

R. K.

Joo

2013. Best-practice recommendations for defining, identifying, and handling outliers. Organizational Research Methods, 16: 270-301.

Alsheikh-Ali

A. A.

Qureshi

Al-Mallah

M. H.

Ioannidis

J. P.

2011. Public availability of published research data in high-impact journals. PLoS ONE, 6(9): e24357. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0024357

Anderson

M. S.

Martinson

B. C.

De Vries

2007. Normative dissonance in science: Results from a national survey of US scientists. Journal of Empirical Research on Human Research Ethics, 2: 3-14.

Anderson

M. S.

Ronning

E. A.

De Vries

Martinson

B. C.

2007. The perverse effects of competition on scientists’ work and relationships. Science and Engineering Ethics, 13: 437-461.

Andraszewicz

Scheibehenne

Rieskamp

Grasman

Verhagen

Wagenmakers

E.-J.

2014. An introduction to Bayesian hypothesis testing for management research. Journal of Management, 41: 521-543.

Bakker

van Dijk

Wicherts

J. M.

2012. The rules of the game called psychological science. Perspectives on Psychological Science, 7: 543-554.

Bakker

Wicherts

J. M.

2011. The (mis)reporting of statistical results in psychology journals. Behavior Research, 43: 666-678. doi:10.3758/s13428-011-0089-5

Bakker

Wicherts

J. M.

2014. Outlier removal and the relation with reporting errors and quality of psychological research. PLoS ONE, 9(7): e103360. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0103360

Banks

G. C.

Kepes

McDaniel

M. A.

2012. Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20: 182-196. doi:10.1111/j.1468-2389.2012.00591.x

10.

Banks

G. C.

Kepes

McDaniel

M. A.

2015. Publication bias: Understand the myths concerning threats to the advancement of science. In Lance

C. E.

Vandenberg

R. J.

(Eds.), More statistical and methodological myths and urban legends: 36-64. New York: Routledge.

11.

Banks

G. C.

McDaniel

M. A.

2011. The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4: 40-44. doi:10.1111/j.1754-9434.2010.01292.x

12.

Banks

G. C.

O’Boyle

E. H.

Jr.

2013. Why we need industrial-organizational psychology to fix industrial-organizational psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6: 291-294.

13.

Bedeian

A. G.

2003. The manuscript review process: The proper roles of authors, referees, and editors. Journal of Management Inquiry, 12: 331-338. doi:10.1177/1056492603258974

14.

Bedeian

A. G.

Taylor

S. G.

Miller

A. N.

2010. Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9: 715-725. doi:10.5465/amle.2010.56659889

15.

Bosco

F. A.

Aguinis

Field

J. G.

Pierce

C. A.

Dalton

D. R.

in press. HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology. doi:10.1111/peps.12111

16.

Cortina

J. M.

2015. A revolution with a solution. Opening plenary presented at the meeting of the Society for Industrial/Organizational Psychology, Philadelphia.

17.

De Vries

Anderson

M. S.

Martinson

B. C

. 2006. Normal misbehavior: Scientists talk about the ethics of research. Journal of Empirical Research on Human Research Ethics, 1: 43-50.

18.

Eich

2014. Business not as usual. Psychological Science, 25: 3-6.

19.

Elsevier. 2015. Publishing and research ethics: Discussing the major types of scientific misconduct and how to avoid them. https://www.publishingcampus.elsevier.com/pages/63//ethics/Publishing-ethics.html. Accessed August 20, 2015.

20.

Emerson

G. B.

Warme

W. J.

Wolf

F. M.

Heckman

J. D.

Brand

R. A.

Leopold

S. S.

2010. Testing for the presence of positive-outcome bias in peer review: A randomized controlled trial. Archives of Internal Medicine, 170: 1934-1939. doi:10.1001/archinternmed.2010.406

21.

Fanelli

2009. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5): e5738. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0005738

22.

Fanelli

2013. Redefine misconduct as distorted reporting. Nature, 494: 149.

23.

Fang

F. C.

Casadevall

2011. Retracted science and the retraction index. Infection and Immunity, 79: 3855-3859.

24.

Fang

F. C.

Casadevall

2015. Competitive science: Is competition ruining science? Infection and Immunity, 83: 1229-1233.

25.

Finkel

E. J.

Eastwick

P. W.

Reis

H. T.

2015. Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology, 108: 275-297.

26.

Francis

Tanzman

Matthews

W. J.

2014. Excess success for psychology articles in the journal Science. PLoS ONE, 9(12): e114255. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0114255

27.

Franco

Malhotra

Simonovits

2014. Publication bias in the social sciences: Unlocking the file drawer. Science, 345: 1502-1505.

28.

Hambrick

D. C.

2007. The field of management’s devotion to theory: Too much of a good thing? Academy of Management Journal, 50: 1346-1352.

29.

Harrison

J. S.

Banks

G. C.

Pollack

J. M.

O’Boyle

E. H.

Jr. Short

J. C.

in press. Publication bias in strategic management research. Journal of Management. doi:10.1177/0149206314535438

30.

Hitchcock

Sober

2004. Prediction versus accommodation and the risk of overfitting. The British Journal for the Philosophy of Science, 55: 1-34.

31.

Hotz

R. L.

2015. How many scientists does it take to write a paper? Apparently, thousands. Wall Street Journal. http://www.wsj.com/articles/how-many-scientists-does-it-take-to-write-a-paper-apparently-thousands-1439169200. Accessed November 11, 2015.

32.

John

L. K.

Loewenstein

Prelec

2012. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23: 524-532. doi:10.1177/0956797611430953

33.

Kepes

Banks

G. C.

McDaniel

M. A.

Whetzel

D. L.

2012. Publication bias in the organizational sciences. Organizational Research Methods, 15: 624-662. doi:10.1177/1094428112452760

34.

Kepes

McDaniel

M. A.

2013. How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6: 252-268.

35.

Kerr

N. L.

1998. HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2: 196-217. doi:10.1207/s15327957pspr0203_4

36.

Leek

J. T.

Peng

R. D.

2015. P values are just the tip of the iceberg. Nature, 520: 612.

37.

Leung

2011. Presenting post hoc hypotheses as a priori: Ethical and theoretical issues. Management and Organizational Review, 7: 471-479.

38.

Locke

E. A.

2007. The case for inductive theory building. Journal of Management, 33: 867-890.

39.

Martinson

B. C.

Anderson

M. S.

Crain

A. L.

De Vries

2006. Scientists’ perceptions of organizational justice and self-reported misbehaviors. Journal of Empirical Research on Human Research Ethics, 1: 51-66.

40.

Matlack

2013. Research fraud allegations trail a German B-school wunderkind. Bloomberg Businessweek. http://www.businessweek.com/articles/2013-06-24/research-fraud-allegations-trail-a-german-b-school-wunderkind. Accessed November 11, 2015.

41.

Mazzola

J. J.

Deuling

J. K.

2013. Forgetting what we learned as graduate students: HARKing and selective outcome reporting in I-O journal articles. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6: 279-284.

42.

Nosek

B. A.

Alter

Banks

G. C.

Borsboom

Bowman

S. D.

Breckler

Buck

Chambers

Chin

Christensen

Contestabile

Dafoe

Eich

Freese

Glennerster

Goroff

Green

D. P.

Heese

Humphreys

Ishiyama

Karlan

Kraut

Lupia

Marbry

Madon

Malhotra

Wilson

E. M.

McNutt

Miguel

Paluck

E. L.

Simonsohn

Soderberg

Spellman

B. A.

Tornow

Turitto

VandenBos

G. R.

Vazire

Wagenmakers

E. J.

Wilson

Yarkoni

2015. Promoting an open research culture: Author guidelines for journals to promote transparency, openness, and reproducibility. Science, 348: 1422-1425.

43.

Nosek

B. A.

Bar-Anan

2012. Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23: 217-243.

44.

Nosek

B. A.

Spies

J. R.

Motyl

2012. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7: 615-631. doi:10.1177/1745691612459058

45.

Nuijten

M. B.

Hartgerink

C. H. J.

Van Assen

M. A. L. M.

Epskamp

Wicherts

J. M.

in press. The prevalence of statistical reporting errors in psychology (1985-2013). Behavior Research Methods. doi:10.3758/s13428-015-0664-2

46.

O’Boyle

E. H.

Banks

G. C.

Carter

Walter

Yuan

2015. A 20-year review of outcome reporting bias in moderated multiple regression. Paper presented at the annual meeting of the Academy of Management, Vancouver.

47.

O’Boyle

E. H.

Jr. Banks

G. C.

Gonzalez-Mule

. in press. The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management. doi:10.1177/0149206314527133

48.

Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science, 349: 6251. doi:10.1126/science.aac4716

49.

Pigott

T. D.

Valentine

J. C.

Polanin

J. R.

Williams

R. T.

Canada

D. D.

2013. Outcome-reporting bias in education research. Educational Researcher, 42: 424-432. doi:10.3102/0013189X13507104

50.

Riley

J. W.

1958. Proceedings of the thirteenth conference on public opinion research. Public Opinion Quarterly, 22: 169-216.

51.

Schmidt

F. L.

Hunter

J. E.

2015. Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.

52.

Sijtsma

Veldkamp

C. L.

Wicherts

J. M.

in press. Improving the conduct and reporting of statistical analysis in psychology. Psychometrika. doi:10.1007/s11336-015-9444-2

53.

Simmons

J. P.

Nelson

L. D.

Simonsohn

2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22: 1359-1366. doi:10.1177/0956797611417632

54.

Wagenmakers

E.-J.

Wetzels

Borsboom

Van Der Maas

H. L.

2011. Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100: 426-432.

55.

Wagenmakers

E.-J.

Wetzels

Borsboom

Van Der Maas

H. L.

Kievit

R. A.

2012. An agenda for purely confirmatory research. Perspectives on Psychological Science, 7: 632-638.

56.

Wicherts

J. M.

Borsboom

Kats

Molenaar

2006. The poor availability of psychological research data for reanalysis. American Psychologist, 61: 726-728.

57.

Wiese

C. W.

Fullick

J. M.

2014. The fantastic four years: Recommendations for industrial–organizational programs. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7: 21-26.

58.

Wigboldus

D. H.

Dotsch

in press. Encourage playing with data and discourage questionable reporting practices. Psychometrika. doi:10.1007/s11336-015-9445-1

59.

Zyphur

M. J.

Oswald

F. L.

2015. Bayesian estimation and inference: A user’s guide. Journal of Management, 41: 390-420.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.41 MB