Replication is an essential part of any science, confirming or adjusting our understanding of the world through repeated exploration of a phenomenon of interest. While there has been an increased interest in the role of replication studies, there also exists skepticism regarding the need for more replication. Our empirical analysis of 406 recent studies that use the term “replication” suggests that this criticism stems from a lack of appreciation of the different forms that replication can take, the prevalence (or lack thereof) of many of these forms, and the objectives that are met by one of the least common forms, constructive replication. As such, the purposes of our paper are (1) to explore the different forms that constructive replication can take and the objectives at which each can be directed, (2) to distinguish these forms from other forms of replication with which they are often confused, (3) to determine how common each form of replication is in our field, and (4) to provide concrete examples of different forms of constructiveness from published studies in order to pave the way towards more (and more useful) replications in the future.
AdamH.ShirakoA.2013. Not all anger is created equal: The impact of the expresser’s culture on the social effects of anger in negotiations. Journal of Applied Psychology, 98: 785-798.
2.
AdamsA. T.AjrouchK. J.HendersonH.HeardI.2005. Service-learning outcomes research: The role and scarcity of replication studies. Journal of Applied Sociology, 22: 55-74.
3.
AguinisH.EdwardsJ. R.BradleyK. J.2017. Improving our understanding of moderation and mediation in strategic management research. Organizational Research Methods, 20: 665-685.
4.
AguinisH.RamaniR. S.AlabduljaderN.2018. What you see is what you get? Enhancing methodological transparency in management research. The Academy of Management Annals, 12: 1-28.
5.
AndersonS. F.MaxwellS. E.2016. There’s more than one way to conduct a replication study: Beyond statistical significance. Psychological Methods, 21: 1-12.
6.
AntonakisJ.2017. Editorial: The future ofThe Leadership Quarterly. The Leadership Quarterly, 28: 1-4.
7.
AsendorpfJ. B.ConnerM.De FruytF.De HouwerJ.DenissenJ. J. A.FiedlerK.FiedlerS.FunderD. C.KlieglR.NosekB. A.PeruginiM.RobertsB. W.SchmittM.van AkenM. A. G.WeberH.WichertsJ. M.2013. Recommendations for increasing replicability in psychology. European Journal of Personality, 27: 108-119.
8.
AustinJ. L.1962. How to do things with words. Oxford, England: Oxford University Press.
9.
BaconF.1858. Novum organum. In SpeddingJ.EllisR. L.HeathD. D. (Eds.), The works of Francis Bacon: 39-248. London: Longman. (Original work published 1620)
10.
BennettA. A.GabrielA. S.CalderwoodC.DahlingJ. J.TrougakosJ. P.2016. Better together? Examining profiles of employee recovery experiences. Journal of Applied Psychology, 101: 1635-1654.
11.
BensonL.BorregoM.2015. The role of replication in engineering education research. Journal of Engineering Education, 104: 388-392.
12.
BettisR. A.HelfatC. E.ShaverJ. M.2016. The necessity, logic, and forms of replication. Strategic Management Journal, 37: 2193-2203.
13.
BonnerJ. M.GreenbaumR. L.QuadeM. J.2017. Employee unethical behavior to shame as an indicator of self-image threat and exemplification as a form of self-image protection: The exacerbating role of supervisor bottom-line mentality. Journal of Applied Psychology, 102: 1203-1221.
14.
CareyB.2016. New critique sees flaws in landmark analysis of psychology studies. New York Times, March3. Retrieved from http://nyti.ms/1OUoPmE.
Carson MarrJ.CableD. M. 2014. Do interviewers sell themselves short? The effects of selling orientation on interviewers’ judgments. Academy of Management Journal, 57: 624-651.
17.
ChanM. E.ArveyR. D.2012. Meta-analysis and the development of knowledge. Perspectives on Psychological Science, 7: 79-92.
18.
ChenM.ChenC. C.SheldonO. J.2016. Relaxing moral reasoning to win: How organizational identification relates to unethical pro-organizational behavior. Journal of Applied Psychology, 7: 1-30.
19.
ConlonE. J.ParksJ. M.1987. Information requests in the context of escalation. Journal of Applied Psychology, 72: 344-350.
20.
CortinaJ. M.1993. What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78: 98-104.
21.
DavisM. S.1971. That’s interesting! Toward a phenomenology of sociology and a sociology of phenomenology. Philosophy of Social Science, 1: 209-344.
22.
DillonR. L.TinsleyC. H.MadsenP. M.RogersE. W.2016. Organizational correctives for improving recognition of near-miss events. Journal of Management, 42: 671-697.
23.
DineenB. R.DuffyM. K.HenleC. A.KiyoungL. E. E.2017. Green by comparison: Deviant and normative transmutations of job search envy in a temporal context. Academy of Management Journal, 60: 295-320.
24.
EasleyR. W.MaddenC. S.DunnM. G.2000. Conducting marketing science: The role of replication in the research process. Journal of Business Research, 48: 83-92.
25.
EdenD.ShaniA. B.1982. Pygmalion goes to boot camp: Expectancy, leadership, and trainee performance. Journal of Applied Psychology, 67: 194-199.
26.
EdwardsJ. R.2010. Reconsidering theoretical progress in organizational and management research. Organizational Research Methods, 13: 615-619.
27.
EdwardsJ. R.BerryJ. W.KayV. S.2013. Bridging the great divide between theoretical and empirical management research. Working paper, Kenan-Flagler Business School, University of North Carolina, Chapel Hill.
28.
EdwardsJ. R.HarrisonR. V.1993. Job demands and worker health: Three-dimensional reexamination of the relationship between person-environment fit and strain. Journal of Applied Psychology, 78: 628-648.
29.
FiedlerK.2018. The creative cycle and the growth of psychological science. Perspectives on Psychological Science, 13: 433-438.
30.
FournierV.GreyC.2000. At the critical moment: Conditions and prospects for critical management studies. Human Relations, 53: 7-32.
31.
GarciaP. R. J. M.RestubogS. L. D.KiewitzC.ScottK. L.TangR. L.2014. Roots run deep: Investigating psychological mechanisms between history of family aggression and abusive supervision. Journal of Applied Psychology, 99: 883-897.
32.
GilbertD. T.KingG.PettigrewS.WilsonT. D.2016. Comment on “Estimating the reproducibility of psychological science.”Science, 351: 1037-a.
GómezO. S.JuristoN.VegasS.2014. Understanding replication of experiments in software engineering: A classification. Information and Software Technology, 56: 1033-1048.
35.
GreenbaumR. L.QuadeM. J.MawritzM. B.KimJ.CrosbyD.2014. When the customer is unethical: The explanatory role of employee emotional exhaustion onto work-family conflict, relationship conflict with coworkers, and job neglect. Journal of Applied Psychology, 99: 1188-1203.
36.
GuniaB. C.BrettJ. M.NandkeolyarA. K.KamdarD.2011. Paying a price: Culture, trust, and negotiation consequences. Journal of Applied Psychology, 96: 774-789.
37.
HanekK. J.GarciaS. M.TorA.2016. Gender and competitive preferences: The role of competition size. Journal of Applied Psychology, 101: 1122-1133.
38.
HarmanR. P.EllingtonK. J.SurfaceE. A.ThompsonL. F.2015. Exploring qualitative training reactions: Individual and contextual influences on trainee commenting. Journal of Applied Psychology, 100: 894-916.
39.
HaroldC. M.OhI.-S.HoltzB. C.HanS.GiacaloneR. A.2016. Fit and frustration as drivers of targeted counterproductive work behaviors: A multifoci perspective. Journal of Applied Psychology, 101: 1513-1535.
40.
HochwarterW. A.FerrisG. R.ZinkoR.ArnellB.JamesM.2007. Reputation as a moderator of political behavior–work outcomes relationships: A two-study investigation with convergent results. Journal of Applied Psychology, 92: 567-576.
41.
HollenbeckJ. R.DeRueD. S.MannorM.2006. Statistical power and parameter stability when subjects are few and tests are many: Comment on Peterson, Smith, Martorana, and Owens (2003). Journal of Applied Psychology, 91: 1-5.
42.
JacquartP.AntonakisJ.2015. When does charisma matter for top-level leaders? Effect of attributional ambiguity. Academy of Management Journal, 58: 1051-1074.
43.
JiangL.ProbstT. M.2017. The rich get richer and the poor get poorer: Country- and state-level income inequality moderates the job insecurity–burnout relationship. Journal of Applied Psychology, 102: 672-681.
44.
JohnsonV. E.PayneR. D.WangT.AsherA.MandalS.2017. On the reproducibility of psychological science. Journal of the American Statistical Association, 112: 1-10.
45.
KellyC. D.2006. Replicating empirical research in behavioral ecology: How and why it should be done but rarely ever is. The Quarterly Review of Biology, 81: 221-236.
46.
KellyC. W.ChaseL. J.TuckerR. K.1979. Replication in experimental communication research: An analysis. Human Communication Research, 5: 338-342.
47.
KiniasZ.SimJ.2016. Facilitating women’s success in business: Interrupting the process of stereotype threat through affirmation of personal values. Journal of Applied Psychology, 101: 1585-1597.
48.
KrummS.LievensF.HuffmeierJ.LipnevichA. A.BendelsH.HertelG.2015. How “situational” is judgment in situational judgment tests?Journal of Applied Psychology, 100: 399-416.
49.
KukenbergerM. R.MathieuJ. E.RuddyT.2015. A cross-level test of empowerment and process influences on members’ informal learning and team commitment. Journal of Management, 41: 987-1016.
50.
LeavittK.MitchellR. R.PetersonJ.2010. Theory pruning: Strategies to reduce our dense theoretical landscape. Organizational Research Methods, 13: 644-667.
51.
LockeK.Golden-BiddleK.1997. Constructing opportunities for contribution: Structuring intertextual coherence and “problematizing” in organizational studies. Academy of Management Journal, 40: 1023-1062.
52.
LongE. C.ChristianM. S.2015. Mindfulness buffers retaliatory responses to injustice: A regulatory approach. Journal of Applied Psychology, 100: 1409-1422.
53.
Lopez-KidwellV.GrosserT. J.DineenB. R.BorgattiS. P.2013. What matters when: A multistage model and empirical examination of job search effort. Academy of Management Journal, 56: 1655-1678.
54.
LoschelderD. D.TrotschelR.SwaabR. I.FrieseM.GalinskyA. D.2016. The information-anchoring model of first offers: When moving first helps versus hurts negotiators. Journal of Applied Psychology, 101: 995-1012.
55.
LüdtkeR.2008. Do that to me one more time! What kind of trial replications do we need?Complementary Therapies in Medicine, 16: 181-182.
MackeyJ. D.FriederR. E.BreesJ. R.MartinkoM. J.2017. Abusive supervision: A meta-analysis and empirical review. Journal of Management, 43: 1940-1965.
58.
MartinezL. R.SawyerK. B.ThoroughgoodC. N.RuggsE. N.SmithN. A.2017. The importance of being “me”: The relation between authentic identity expression and transgender employees’ work-related attitudes and experiences. Journal of Applied Psychology, 102: 215-226.
59.
MathieuJ. E.KukenbergerM. R.D’InnocenzoL.ReillyG.2015. Modeling reciprocal team cohesion–performance relationships, as impacted by shared leadership and members’ competence. Journal of Applied Psychology, 100: 1-22.
MertonR. K.1973. The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press. (Original work published 1942)
62.
MittelstaedtR. A.ZornT. S.1984. Econometric replication: Lessons from the experimental sciences. Quarterly Journal of Business and Economics, 23: 9-15.
63.
NelsonL. D.SimmonsJ.SimonsohnU.2018. Psychology’s renaissance. Annual Review of Psychology, 69: 511-534.
64.
NosekB. A.EbersoleC. R.DeHavenA. C.MellorD. T.2018. The preregistration revolution. Proceedings of the National Academy of Sciences, USA, 115: 2600-2606.
65.
NosekB. A.LakensD.2014. Registered reports: A method to increase the credibility of published results. Social Psychology, 45: 137-141.
66.
Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science, 349: aac4716.
67.
OwensB. P.BakerW. E.SumpterD. M.CameronK. S.2016. Relational energy at work: Implications for job engagement and job performance. Journal of Applied Psychology, 101: 35-49.
68.
PashlerH.HarrisC. R.2012. Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7: 531-536.
69.
PashlerH.WagenmakersE.-J.2012. Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?Perspectives on Psychological Science, 7: 528-530.
70.
PopperK.2005. The logic of scientific discovery. New York: Taylor & Francis e-Library.
71.
SandbergJ.AlvessonM.2011. Ways of constructing research questions: Gap-spotting or problematization?Organization, 18: 23-44.
72.
SchultzeT.PfeifferF.Schulz-HardtS.2012. Biased information processing in the escalation paradigm: Information search and information evaluation as potential mediators of escalating commitment. Journal of Applied Psychology, 97: 16-32.
73.
ShossM. K.EisenbergerR.RestubogS. L.ZagenczykT. J.2013. Blaming the organization for abusive supervision: The roles of perceived organizational support and supervisor’s organizational embodiment. Journal of Applied Psychology, 98: 158-168.
74.
SimonsD. J.2014. The value of direct replication. Perspectives on Psychological Science, 9: 76-80.
75.
SinhaR.JanardhananN. S.GreerL. L.ConlonD. E.EdwardsJ. R.2016. Skewed task conflicts in teams: What happens when a few members see more conflict than the rest?Journal of Applied Psychology, 101: 1045-1055.
76.
SliterM. T.SinclairR. R.YuanZ.MohrC. D.2014. Don’t fear the reaper: Trait death anxiety, mortality salience, and occupational health. Journal of Applied Psychology, 99: 759-769.
77.
SpicerA.AlvessonM.KärremanD.2009. Critical performativity: The unfinished business of critical management studies. Human Relations, 62: 537-560.
78.
SteelP.Kammeyer-MuellerJ.PatersonT. A.2015. Improving the meta-analytic assessment of effect size variance with an informed Bayesian prior. Journal of Management, 41: 718-743.
79.
StraussK.GriffinM. A.ParkerS. K.2012. Future work selves: How salient hoped-for identities motivate proactive career behaviors. Journal of Applied Psychology, 97: 580-598.
80.
StroebeW.StrackF.2014. The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9: 59-71.
81.
ThauS.Derfler-RozinR.PitesaM.MitchellM. S.PillutlaM. M.2015. Unethical for the sake of the group: Risk of social exclusion and pro-group unethical behavior. Journal of Applied Psychology, 100: 98-113.
82.
TsangE. W. K.KwanK.-M.1999. Replication and theory development in organizational science: A critical realist perspective. Academy of Management Review, 24: 759-780.
83.
Van IddekingeC. H.AguinisH.MackeyJ. D.DeOrtentiisP. S. 2018. A meta-analysis of the interactive, additive, and relative effects of cognitive ability and motivation on performance. Journal of Management, 44: 249-279.
84.
Van KleefG. A.van den BergH.HeerdinkM. W. 2015. The persuasive power of emotions: Effects of emotional expressions on attitude formation and change. Journal of Applied Psychology, 100: 1124-1142.
85.
VazireS.2018. Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13: 411-417.
86.
VenkataramaniV.LabiancaG.GrosserT.2013. Positive and negative workplace relationships, social satisfaction, and organizational attachment. Journal of Applied Psychology, 98: 1028-1039.
87.
VogelR. M.MitchellM. S.2017. The motivational effects of diminished self-esteem for employees who experience abusive supervision. Journal of Management, 43: 2218-2251.
88.
WagenmakersE. J.DutilhG.SarafoglouA.2018. The creativity-verification cycle in psychological science: New methods to combat old idols. Perspectives on Psychological Science, 13: 418-427.
89.
WangL.MurnighanJ. K.2017. The dynamics of punishment and trust. Journal of Applied Psychology, 102: 1385-1402.
90.
WeickK. E.1989. Theory construction as disciplined imagination. Academy of Management Review, 14: 516-531.
91.
WhettenD. A.RodgersZ. J.GreenC. D.ColesR. S.2014. What, really, constitutes a theoretical contribution?Academy of Management Proceedings, 2014(1): 17067.
WhitingS. W.MaynesT. D.PodsakoffN. P.PodsakoffP. M.2012. Effects of message, source, and context on evaluations of employee voice behavior. Journal of Applied Psychology, 97: 159-182.
94.
WoD. X. H.AmbroseM. L.SchminkeM.2015. What drives trickle-down effects? A test of multiple mediation processes. Academy of Management Journal, 58: 1848-1868.
95.
WuC.-H.GriffinM. A.2012. Longitudinal relationships between core self-evaluations and job satisfaction. Journal of Applied Psychology, 97: 331-342.
96.
WuC.-H.LiuJ.KwanH. K.LeeC.2016. Why and when workplace ostracism inhibits organizational citizenship behaviors: An organizational identification perspective. Journal of Applied Psychology, 101: 362-378.
97.
ZhuL.AquinoK.VaderaA. K.2016. What makes professors appear credible: The effect of demographic characteristics and ideological beliefs. Journal of Applied Psychology, 101: 862-880.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
