How replication studies can improve doctoral student education

Abstract

In addition to helping advance theory, replication studies offer rich and complementary learning experiences for doctoral students, enabling them to learn general research skills, through the process of striving to imitate good studies. In addition, students gain replication-specific methodological skills and learn about the important roles replications play for making management knowledge trustworthy. We outline best practices for enabling doctoral students and their supervisors to select studies to replicate, execute their replications, and increase the probability of successfully publishing their findings. We also discuss the crucial role of faculty mentors in supporting and guiding replication-based learning of doctoral students. Ultimately, educating doctoral students on how to execute high-quality replication studies helps to answer wider calls for more replication studies in the field of management, an important stepping stone along the journey toward open and responsible research.

Keywords

Replication Studies Doctoral Student Education

Get full access to this article

View all access options for this article.

References

Aguinis

Cascio

W. F.

Ramani

R. S.

(2017). Science’s reproducibility and replicability crisis: International business is not immune. Journal of International Business Studies, 48(6), 653–663.

Aguinis

Cronin

M. A.

(in press). It’s the theory, stupid. Organizational Psychology Review, 204138662210806. https://doi.org/10.1177/20413866221080629

Aguinis

Cummings

Ramani

R. S.

Cummings

T. G.

(2020). “An A is an A:” The new bottom line for valuing academic research. Academy of Management Perspectives, 34(1), 135–154.

Aguinis

Ramani

R. S.

Alabduljader

(2018). What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12(1), 83–110.

Amrhein

Trafimow

Greenland

(2019). Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication. The American Statistician, 73(supl1), 262–270.

Anderson

S. F.

Maxwell

S. E.

(2016). There’s more than one way to conduct a replication study: Beyond statistical significance. Psychological Methods, 21(1), 1–12.

Asendorpf

J. B.

Conner

De Fruyt

De Houwer

Denissen

J. J. A.

Fiedler

Funder

D. C.

Kliegl

Nosek

B. A.

Perugini

Roberts

B. W.

Schmitt

Van Aken

M. A. G.

Weber

Wicherts

J. M.

(2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119.

Bamberger

P. A.

(2019). On the replicability of abductive research in management and organizations: Internal replication and its alternatives. Academy of Management Discoveries, 5(2), 103–108.

Bem

D. J.

(2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425.

10.

Bergh

D. D.

Sharp

B. M.

Aguinis

(2017). Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings. Strategic Organization, 15(3), 423–436.

11.

Bettis

R. A.

Ethiraj

Gambardella

Helfat

Mitchell

(2016a). Creating repeatable cumulative knowledge in strategic management: A call for a broad and deep conversation among authors, referees, and editors. Strategic Management Journal, 37(2), 257–261.

12.

Bettis

R. A.

Helfat

C. E.

Shaver

J. M.

(2016b). The necessity, logic, and forms of replication. Strategic Management Journal, 37(11), 2193–2203.

13.

Block

J. H.

Fisch

Kanwal

Lorenzen

Schulze

(in press). Replication studies in top management journals: An empirical investigation of prevalence, types, outcomes, and impact. Management Review Quarterly. https://doi.org/10.1007/s11301-022-00269-6

14.

Bonett

D. G.

(2012). Replication-extension studies. Current Directions in Psychological Science, 21(6), 409–412.

15.

Bonett

D. G.

(2021). Design and analysis of replication studies. Organizational Research Methods, 24(3), 513–529.

16.

Bonett

D. G.

Price

R. M.

(2015). Varying coefficient meta-analysis methods for odds ratios and risk ratios. Psychological Methods, 20(3), 394–406.

17.

Child

(1972a). Organizational structure, environment, and performance. Sociology, 6(1), 1–22.

18.

Child

(1972b). Organization structure and strategies of control: A replication of the Aston study. Administrative Science Quarterly, 17(2), 163–177.

19.

Child

(1973). Strategies of control and organizational behavior. Administrative Science Quarterly, 18(1), 1–17.

20.

Child

(1997). From the Aston programme to strategic choice: A journey from concepts to theory. In Clarke

(Ed.), Advancement in Organizational Behaviour: Essays in Honour of Derek S. Pugh (pp. 45–71). Farnham, UK: Ashgate.

21.

Crawford

G. C.

Skorodziyevskiy

Frid

C. J.

Nelson

T. E.

Booyavi

Diana

Hechavarria

D. M.

Reynolds

P. D.

Teymourian

(2022). Advancing entrepreneurship theory through replication: A case study on contemporary methodological challenges, future best practices, and an entreaty for communality. Entrepreneurship Theory and Practice, 46(3), 779–799.

22.

Cucina

J. M.

McDaniel

M. A.

(2016). Pseudotheory proliferation is damaging the organizational sciences. Journal of Organizational Behavior, 37(8), 1116–1125.

23.

Cumming

Finch

(2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60(2), 170–180.

24.

Dau

L. A.

Santangelo

G. D.

van Witteloostuijn

(2022). Replications in international business. Journal of International Business Studies, 53(2), 215–230.

25.

Dreher

G. F.

Carter

N. M.

Dworkin

(2019). The pay premium for high-potential women: A constructive replication and refinement. Personnel Psychology, 72(4), 495–511.

26.

Eden

(2002). From the editors: Replication, meta-analysis, scientific progress, and AMJ's publication policy. Academy of Management Journal, 45(5), 841–846.

27.

Everett

J. A. C.

Earp

B. D.

(2015). A tragedy of the (academic) commons: Interpreting the replication crisis in psychology as a social dilemma for early-career researchers. Frontiers in Psychology, 6(1152), 1–4.

28.

Fecher

Fräßdorf

Wagner

G. G.

(2016). Perceptions and practices of replication by social and behavioral scientists: Making replications a mandatory element of curricula would be useful. DIW Berlin Discussion Paper No. 1572. Available at SSRN: http://dx.doi.org/10.2139/ssrn.2768629.

29.

Fisher

R. A.

(1966). The design of experiments. Edinburgh, UK: Oliver and Boyd.

30.

Frank

M. C.

Saxe

(2012). Teaching replication. Perspectives on Psychological Science, 7(6), 600–604.

31.

Galak

LeBoeuf

R. A.

Nelson

L. D.

Simmons

J. P.

(2012). Correcting the past: Failures to replicate psi. Journal of Personality and Social Psychology, 103(6), 933–948.

32.

Gelman

Stern

(2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328–331.

33.

Goldfarb

King

A. A.

(2016). Scientific apophenia in strategic management research: Significance tests and mistaken inference. Strategic Management Journal, 37(1), 167–176.

34.

Gronau

Q. F.

Heck

D. W.

Berkhout

S. W.

Haaf

J. M.

Wagenmakers

E. J.

(2021). A primer on Bayesian model-averaged meta-analysis. Advances in Methods and Practices in Psychological Science, 4(3), 1-19.

35.

Hambrick

D. C.

(2007). The field of management’s devotion to theory: Too much of a good thing? Academy of Management Journal, 50(6), 1346–1352.

36.

Hammond

M. M.

Schyns

Lester

G. V.

Clapp-Smith

Thomas

J. S.

(in press). The romance of leadership: Rekindling the fire through replication of Meindl and Ehrlich. The Leadership Quarterly. https://doi.org/10.1016/j.leaqua.2021.101538

37.

Hawkins

R. X.

Smith

E. N.

Arias

J. M.

Catapano

Hermann

Frank

M. C.

(2018). Improving the replicability of psychological science through pedagogy. Advances in Methods and Practices in Psychological Science, 1(1), 7–18.

38.

Hodgkinson

G. P.

Rousseau

D. M.

(2009). Bridging the rigour-relevance gap in management research: It's already happening! Journal of Management Studies, 46(3), 534–546.

39.

Hofling

C. K.

Brotzman

Dalrymple

Graves

Pierce

C. M.

(1966). An experimental study in nurse-physician relationships. Journal of Nervous and Mental Disease, 143(2), 171–180.

40.

Hubbard

(2016). Corrupt research: The case for reconceptualizing empirical management and social science. Los Angeles, CA: Sage Publications

41.

Ioannidis

J. P. A.

(2018). Why replication has more scientific value than original discovery. Behavioral and Brain Sciences, 41, E137.

42.

Janz

(2016). Bringing the gold standard into the classroom: Replication in university teaching. International Studies Perspectives, 17(4), 392–407.

43.

Kane

E. J.

(1984). Why journal editors should encourage the replication of applied econometric research. Quarterly Journal of Business and Economics, 23(1), 3–8.

44.

Kelly

C. W.

Chase

L. J.

Tucker

R. K.

(1979). Replication in experimental communication research: An analysis. Human Communication Research, 5(4), 338–342.

45.

Kirschner

Sweller

Clark

R. E.

(2006). Why unguided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psychologist, 41(2), 75–86.

46.

Köhler

Cortina

J. M.

(2021). Play it again, Sam! An analysis of constructive replication in the organizational sciences. Journal of Management, 47(2), 488–518.

47.

Kolb

(1984). Experiential learning: Experience as the source of learning and development. Upper Saddle River, NJ: Prentice Hall.

48.

König

Graf-Vlachy

Schöberl

(2021). Opportunity/threat perception and inertia in response to discontinuous change: Replicating and extending Gilbert (2005). Journal of Management, 47(3), 771–816.

49.

LeBel

E. P.

Vanpaemel

Cheung

Campbell

(2019). A brief guide to evaluate replications. Meta-Psychology, 3, 1–9. https://doi.org/10.15626/MP.2019

50.

Locke

E. A.

(2015). Theory building, replication, and behavioral priming: Where do we need to go from here? Perspectives on Psychological Science, 10(3), 408–414.

51.

Locke

E. A.

Latham

G. P.

(2020). Building a theory by induction: The example of goal setting theory. Organizational Psychology Review, 10(3-4), 223–239.

52.

Lykken

D. T.

(1968). Statistical significance in psychological research. Psychological Bulletin, 70(3), 151–159.

53.

Marwick

Wang

L. Y.

Robinson

Loiselle

(2020). How to use replication assignments for teaching integrity in empirical archaeology. Advances in Archaeological Practice, 8(1), 78–86.

54.

Miller

C. C.

Bamberger

(2016). Exploring emergent and poorly understood phenomena in the strangest of places: The footprint of discovery in replications, meta-analyses, and null findings. Academy of Management Discoveries, 2(4), 313–319.

55.

Minefee

McDonnell

M. H.

Werner

(2021). Reexamining investor reaction to covert corporate political activity: A replication and extension of Werner (2017). Strategic Management Journal, 42(6), 1139–1158.

56.

Nicholson

Cole

S. G.

Rocklin

(1985). Conformity in the Asch situation: A comparison between contemporary British and US university students. British Journal of Social Psychology, 24(1), 59–63.

57.

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

58.

Patil

S. V.

(2019). “The public doesn’t understand”: The self-reinforcing interplay of image discrepancies and political ideologies in law enforcement. Administrative Science Quarterly, 64(3), 737–769.

59.

Perrin

Spencer

(1981). Independence or conformity in the Asch experiment as a reflection of cultural and situational factors. British Journal of Social Psychology, 20(3), 205–209.

60.

Platt

(1964). Strong inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others. Science, 146(3642), 347–353.

61.

Pugh

D. S.

Hickson

D. J.

Hinings

C. R.

Turner

(1968). Dimensions of organization structure. Administrative Science Quarterly, 13(1), 65–105.

62.

Rank

S. G.

Jacobson

C. K.

(1977). Hospital nurses’ compliance with medication overdose orders: A failure to replicate. Journal of Health and Social Behavior, 18(2), 188–193.

63.

Ritchie

S. J.

Wiseman

French

C. C.

(2012). Failing the future: Three unsuccessful attempts to replicate Bem's ‘retroactive facilitation of recall’ effect. PLoS One, 7(3), e33423.

64.

Rousseau

D. M.

Manning

Denyer

(2008). Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. Academy of Management Annals, 2(1), 475–515.

65.

Sanchez-Ku

M. L.

Arthur

(2000). A dyadic protocol for training complex skills: A replication using female participants. Human Factors, 42(3), 512–520.

66.

Schwab

Abrahamson

Fiedler

Starbuck

W. H.

(2011). Researchers should make thoughtful assessments instead of null-hypothesis significance tests. Organization Science, 22(4), 1105–1120.

67.

Schwab

Starbuck

W. H.

(2017). A call for openness in research reporting: How to turn covert practices into helpful tools. Academy of Management Learning and Education, 16(1), 125–141.

68.

Shanks

D. R.

Newell

B. R.

Lee

E. H.

Balakrishnan

Ekelund

Cenac

Moore

(2013). Priming intelligent behavior: An elusive phenomenon. PLoS One, 8(4), e56515.

69.

Simons

D. J.

(2014). The value of direct replication. Perspectives on Psychological Science, 9(1), 76–80.

70.

Sodhi

M. M. S.

Tang

C. S.

(2014). Guiding the next generation of doctoral students in operations management. International Journal of Production Economics, 150, 28–36.

71.

Standing

L. G.

(2016). How to use replication team projects in a research methods course. In Altman

Stein

Westfall

(Eds.), Essays from Excellence in Teaching (Vol. 15, pp. 26–31). Washington, DC: Society for the Teaching of Psychology.

72.

Standing

L. G.

Grenier

Lane

E. A.

Roberts

M. S.

Sykes

S. J.

(2014). Using replication projects in teaching research methods. Psychology Teaching Review, 20(1), 96–104.

73.

Starbuck

W. H.

(2006). The production of knowledge: The challenge of social science research. Oxford, UK: Oxford University Press.

74.

Stojmenovska

Bol

Leopold

(2019). Teaching replication to graduate students. Teaching Sociology, 47(4), 303–313.

75.

Stroebe

Strack

(2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59–71.

76.

Toncar

M. F.

Munch

J. M.

(2010). Meaningful replication: When is a replication no longer a replication? A rejoinder to Stella and Adam (2008). Journal of Marketing Theory and Practice, 18(1), 71–80.

77.

Tsang

E. W.

Kwan

K. M.

(1999). Replication and theory development in organizational science: A critical realist perspective. Academy of Management Review, 24(4), 759–780.

78.

Tullett

A. M.

Vazire

(2018). Scientific progress is like doing a puzzle, not building a wall. Behavioral and Brain Sciences, 41, E154.

79.

Wagenmakers

E.-J.

Wetzels

Borsboom

van der Maas

H. L. J.

(2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100(3), 426–432.

80.

Wagge

J. R.

Brandt

M. J.

Lazarevic

L. B.

Legate

Christopherson

Wiggins

Grahe

J. E.

(2019). Publishing research with undergraduate students via replication work: The collaborative replications and education project. Frontiers in Psychology, 10(247), 1–4.

81.

Wismans

Kamei

Thurik

Torrès

(2021). The link between attention-deficit hyperactivity disorder symptoms and entrepreneurial orientation in Japanese business owners. Management Review Quarterly, 71(4), 857–872.

82.

Wismans

Thurik

Verheul

Torrès

Kamei

(2020). Attention-deficit hyperactivity disorder symptoms and entrepreneurial orientation: A replication note. Applied Psychology, 69(3), 1093–1112.

83.

Wolfson

M. A.

Mathieu

J. E.

(2021). Deploying human capital resources: Accentuating effects of situational alignment and social capital resources. Academy of Management Journal, 64(2), 435–457.

84.

Zwaan

R. A.

Etz

Lucas

R. E.

Donnellan

M. B.

(2018). Making replication mainstream. Behavioral and Brain Sciences, 41, E120.