This article contributes to the practice of coding in meta-analyses by offering direction and advice for experienced and novice meta-analysts on the “how” of coding. The coding process, the invisible architecture of any meta-analysis, has received comparably little attention in methodological resources, leaving the research community with insufficient guidance on “how” it should be rigorously planned (i.e., cohere with the research objective), conducted (i.e., make reliable and valid coding decisions), and reported (i.e., in a sufficiently transparent manner for readers to comprehend the authors’ decision-making). A lack of rigor in these areas can lead to erroneous results, which is problematic for entire research communities who build their future knowledge upon meta-analyses. Along four steps, the guidelines presented here elucidate “how” the coding process can be performed in a coherent, efficient, and credible manner that enables connectivity with future research, thereby enhancing the reliability and validity of meta-analytic findings. Our recommendations also support editors and reviewers in advising authors on how to improve the rigor of their coding and ultimately establish higher quality standards in meta-analytic research.
AguinisH.DaltonD. R.BoscoF. A.PierceC. A.DaltonC. M. (2011). Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly impact. Journal of Management, 37(1), 5-38.
2.
AguinisH.RamaniR. S.AlabduljaderN. (2018). What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12(1), 83-110.
3.
AguinisH.RamaniR. S.AlabduljaderN. (2020). Best-practice recommendations for producers, evaluators, and users of methodological literature reviews. Organizational Research Methods, 1-31.
4.
AnconaD. G.CaldwellD. F. (1992). Demography and design: Predictors of new product team performance. Organization Science, 3(3), 321-341.
5.
AytugZ. G.RothsteinH. R.ZhouW.KernM. C. (2012). Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15(1), 103-133.
6.
Bhaskar-ShrinivasP. M.HarrisonD.ShafferM.LukD. M. (2005). Input-based and time-based models of international adjustment: Meta-analytical evidence and theoretical extensions. Academy of Management Journal, 48(2), 257-281.
7.
BlumeB. D.FordJ. K.BaldwinT. T.HuangJ. L. (2010). Transfer of training: A meta-analytic review. Journal of Management, 36(4), 1065-1105.
8.
BoothA.SuttonA.PapaioannouD. (2016). Systematic approaches to a successful literature review (2nd ed.). Sage Publications.
9.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta-analysis. John Wiley & Sons, Ltd.
10.
BrewertonP.MillwardL. (2001). Organisational research methods. Sage Publications.
11.
BundersonJ. S.SutcliffeK. M. (2002). Comparing alternative conceptualizations of functional diversity in management teams: Process and performance effects. Academy of Management Journal, 45(5), 875-893.
12.
CardN. A. (2015). Applied meta-analysis for social science research. Guilford Publications.
13.
CarlsonK.JiF. (2011). Citing and building on meta-analytic findings: A review and recommendations. Organizational Research Methods, 14(4), 696-717.
14.
ChamberlinM.NewtonD. W.LePineJ. A. (2017). A meta-analysis of voice and its promotive and prohibitive forms: Identification of key associations, distinctions, and future research directions. Personnel Psychology, 70(1), 11-71.
15.
ChapmanD. S.UggerslevK. L.CarrollS. A.PiasentinK. A.JonesD. A. (2005). Applicant attraction to organizations and job choice: A meta-analytic review of the correlates of recruiting outcomes. Journal of Applied Psychology, 90(5), 928-944.
16.
ChristianM. S.EdwardsB. D.BradleyJ. C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63(1), 83-117.
17.
ChristianM. S.GarzaA. S.SlaughterJ. E. (2011). Work engagement: A quantitative review and test of its relations with task and contextual performance. Personnel Psychology, 64(1), 89-136.
18.
ChurchillG. A. (1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16(1), 63-73.
19.
ChurchillG. A.PeterJ. P. (1984). Research design effects on the reliability of rating scales: A meta-analysis. Journal of Marketing Research, 21(4), 360-375.
20.
CombsJ.CrookT. R.RauchA. (2019). Meta-analytic research in management: Contemporary approaches, unresolved controversies, and rising stars. Journal of Management Studies, 56(1), 1-18.
21.
CooperH.HedgesL. V.ValentineJ. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). Russell Sage Foundation.
22.
CooperC.HedgesL. V.ValentineJ. C. (2019). The handbook of research synthesis and meta-analysis (3rd ed.). Russell Sage Foundation.
23.
CooperH. (2009). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Sage Publications.
24.
CooperH. (2017). Research synthesis and meta-analysis: A step-by-step approach (5th ed.). Sage Publications.
25.
CovinJ. G.SlevinD. P. (1989). Strategic management of small firms in hostile and benign environments. Strategic Management Journal, 10(1), 75-87.
26.
CreswellJ. W.Plano ClarkV. L. (2007). Designing and conducting mixed methods research. Sage Publications.
27.
DamanpourF. (1991). Organizational innovation: A meta-analysis of effects of determinants and moderators. Academy of Management Journal, 34(3), 555-590.
28.
DeSimoneJ.BrannickM. T.O'BoyleE. H.RyuJ. W. (2020). Recommendations for reviewing meta-analyses in organizational research. Organizational Research Methods, 24(4), 694-717.
29.
DeSimoneJ. A.KöhlerT.SchoenJ. L. (2019). If it were only that easy: The use of meta-analytic research by organizational scholars. Organizational Research Methods, 22(4), 867-891.
30.
EllisP. D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press.
31.
ErikssonP.KovalainenA. (2015). The research philosophy. In ErikssonP.KovalainenA. (Eds.), Qualitative methods in business research (2nd ed., pp. 10-24). Sage Publications.
32.
FainshmidtS.PezeshkanA.FrazierM. L.NairA.MarkowskiE. (2016). Dynamic capabilities and organizational performance: A meta-analytic evaluation and extension. Journal of Management Studies, 53(8), 1348-1380.
33.
FrazierM. L.FainshmidtS.KlingerR. L.PezeshkanA.VrachevaV. (2017). Psychological safety: A meta-analytic review and extension. Personnel Psychology, 70(1), 113-165.
34.
GaurA.KumarM. (2018). A systematic approach to conducting review studies: An assessment of content analysis in 25 years of IB research. Journal of World Business, 53(2), 280-289.
35.
GeyskensI.KrishnanR.SteenkampJ. B. E. M.CunhaP. V. (2009). A review and evaluation of meta-analysis practices in management research. Journal of Management, 35(2), 393-419.
36.
GeyskensI.SteenkampJ. B. E. M.KumarN. (2006). Make, buy, or ally: A transaction cost theory meta-analysis. Academy of Management Journal, 49(3), 519-543.
37.
GlassG. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8.
38.
GlassG. V.SmithM. L.McGawB. (1981). Meta-analysis in social research. Sage Publications.
39.
Gonzalez-MuléE.AguinisH. (2018). Advancing theory by assessing boundary conditions with metaregression: A critical review and best-practice recommendations. Journal of Management, 44(6), 2246-2273.
40.
GreenJ.TonidandelS.CortinaJ. (2016). Getting through the gate: Statistical and methodological issues raised in the reviewing process. Organizational Research Methods, 19(3), 402-432.
41.
HeugensP. P. M. A. R.LanderM. W. (2009). Structure! agency! (and other quarrels): A meta-analysis of institutional theories of organization. Academy of Management Journal, 52(1), 61-85.
42.
HieblM. R. (2021). Sample selection in systematic literature reviews of management research. Organizational Research Methods,1-33.
43.
HigginsJ. P. T.DeeksJ. J. (2008). Selecting studies and collecting data. In HigginsJ. P. T.GreenS. (Eds.), Cochrane handbook for systematic reviews of interventions (pp. 151-183). The Cochrane Collaboration.
44.
HigginsJ. P. T.GreenS. (2008). Cochrane handbook for systematic reviews of interventions. The Cochrane Collaboration.
45.
HülshegerU. R.AndersonN.SalgadoJ. F. (2009). Team-level predictors of innovation at work: A comprehensive meta-analysis spanning three decades of research. Journal of Applied Psychology, 94(5), 1128-1145.
46.
HuntM. (1997). How science takes stock: The story of meta-analysis. Russell Sage Foundation.
47.
HunterJ.SchmidtF.JacksonG. (1982). Meta-analysis: Cumulating research findings across studies. Sage Publications.
48.
HunterJ. E.SchmidtF. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Sage Publications.
49.
IlgenD. R. (1999). Teams embedded in organizations: Some implications. American Psychologist, 54(2), 129-139.
50.
JiangK.LepakD.HuJ.BaerJ. (2012). How does human resource management influence organizational outcomes? A meta-analytic investigation of mediating mechanisms. Academy of Management Journal, 55(6), 1264-1294.
51.
JoshiA.RohH. (2009). The role of context in work team diversity research: A meta-analytic review. Academy of Management Journal, 52(3), 599-627.
52.
JudgeT. A.IliesR. (2002). Relationship of personality to performance motivation: A meta-analytic review. Journal of Applied Psychology, 87(4), 797-807.
53.
KaramE. P.HuJ.DavisonR. B.JuravichM.NahrgangJ. D.HumphreyS. E.Scott DeRueD. (2019). Illuminating the “face” of justice: A meta-analytic examination of leadership and organizational justice. Journal of Management Studies, 56(1), 134-171.
54.
KepesS.McDanielM. A.BrannickM. T.BanksG. C. (2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the meta-analytic reporting standards). Journal of Business and Psychology, 28, 123-143.
55.
KerlingerF. N.LeeH. B. (2000). Foundations of behavioral research. Wadsworth Thompson Learning.
56.
KlierH.SchwensC.ZapkauF. B.DikovaD. (2017). Which resources matter how and where? A meta-analysis on firms’ foreign establishment mode choice. Journal of Management Studies, 54(3), 304-339.
57.
KnightC.PattersonM.DawsonJ. (2017). Building work engagement: A systematic review and meta-analysis investigating the effectiveness of work engagement interventions. Journal of Organizational Behavior, 38(6), 792-812.
58.
KooijD. T. A. M.De LangeA. H.JansenP. G. W.KanferR.DikkersJ. S. (2011). Age and work-related motives: Results of a meta-analysis. Journal of Organizational Behavior, 32(2), 197-225.
59.
KunischS.MenzM.BartunekJ. M.CardinalL. B.DenyerD. (2018). Feature topic at organizational research methods: How to conduct rigorous and impactful literature reviews?Organizational Research Methods, 21(3), 519-523.
60.
LarsenK. R.BongC. H. (2016). A tool for addressing construct identity in literature reviews and meta-analyses. MIS Quarterly, 40(3), 529-551.
61.
Le BretonJ. M.SenterJ. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815-852.
62.
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Sage Publications.
63.
LittellJ. H.CorcoranJ.PillaiV. (2008). Systematic reviews and meta-analysis. Oxford University Press.
64.
LoignonA. C.WoehrD. J. (2018). Social class in the organizational sciences: A conceptual integration and meta-analytic review. Journal of Management, 44(1), 61-88.
65.
LovelaceK.ShapiroD. L.WeingartL. R. (2001). Maximizing cross-functional new product teams’ innovativeness and constraint adherence: A conflict communications perspective. Academy of Management Journal, 44(4), 779-793.
66.
LumpkinG. T.DessG. G. (1996). Clarifying the entrepreneurial orientation construct and linking it to performance. Academy of Management, 21(1), 135-172.
67.
MachiL. A.McEvoyB. T. (2016). The literature review: Six steps to success. Corwin Press.
68.
MackeyJ. D.FriederR. E.BreesJ. R.MartinkoM. J. (2017). Abusive supervision: A meta-analysis and empirical review. Journal of Management, 43(6), 1940-1965.
69.
MARS (Meta-Analysis Reporting Standards); APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology: Why do we need them? What might they be?American Psychologist, 63(9), 839-851.
70.
MARS (Meta-Analysis Reporting Standards); AppelbaumM.CooperH.KlineR. B.Mayo-WilsonE.NezuA. M.RaoS. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3-25.
71.
MartinR.GuillaumeY.ThomasG.LeeA.EpitropakiO. (2016). Leader–member exchange (LMX) and performance: A meta-analytic review. Personnel Psychology, 69(1), 67-121.
72.
MathieuJ.MaynardM. T.RappT.GilsonL. (2008). Team effectiveness 1997-2007: A review of recent advancements and a glimpse into the future. Journal of Management, 34(3), 410-476.
73.
MontanoD.ReeskeA.FrankeF.HüffmeierJ. (2017). Leadership, followers’ mental health and job performance in organizations: A comprehensive meta-analysis from an occupational health perspective. Journal of Organizational Behavior, 38(3), 327-350.
74.
MOOSE (Meta-Analysis of Observational Studies in Epidemiology); StroupD. F.BerlinJ. A.MortonS. C.OlinI.WilliamsonG. D.RennieD.MoherD.BeckerB. J.SipeT. A.ThackerS. B. (2000). Meta-analysis of observational studies in epidemiology: A proposal for reporting. Meta-analysis Of observational studies in epidemiology (MOOSE) group. JAMA, 283(15), 2008-2012.
75.
MuellerV.RosenbuschN.BauschA. (2013). Success patterns of exploratory and exploitative innovation: A meta-analysis of the influence of institutional factors. Journal of Management, 39(6), 1606-1636.
76.
MurrayA. L. (2011). The validity of the meta-analytic method in addressing the issue of psi replicability. Journal of Parapsychology, 75(2), 261-277.
77.
NgT. W. H.FeldmanD. C. (2008). Long work hours: A social identity perspective on meta-analysis data. Journal of Organizational Behavior, 29(7), 853-880.
78.
O'ReillyK.PaperD.MarxS. (2012). Demystifying grounded theory for business research. Organizational Research Methods, 15(2), 247-262.
79.
OrwinR. G.VeveaJ. L. (2009). Evaluating coding decisions. In CooperH.HedgesL. V. (Eds.), The handbook of research synthesis (pp. 139-162). Russell Sage Foundation.
80.
ParkM. H. J.LimJ. W.Birnbaum-MoreP. H. (2009). The effect of multiknowledge individuals on performance in cross-functional new product development teams. Journal of Product Innovation Management, 26(1), 86-96.
81.
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses); MoherD.LiberatiA.TetzlaffJ.AltmanD. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. British Medical Journal, 339(2535), 332-336.
82.
QUOROM (Quality of Reporting of Meta-Analyses); MoherD.CookD. J.EastwoodS.OlkinI.RennieD.StroupD. (1999). Improving the quality of reporting of meta-analysis of randomized controlled trials: The QUOROM statement. The Lancet, 354(9193), 1896-1900.
83.
RosenbuschN.RauchA.BauschA. (2013). The mediating role of entrepreneurial orientation in the task environment–performance relationship: A meta-analysis. Journal of Management, 39(3), 633-659.
84.
RothsteinH. R.HopewellS. (2009). Grey literature. In CooperH. M.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (pp. 103-126). Russel Sage Foundation.
85.
RothsteinH. R.McDanielM. A. (1989). Guidelines for conducting and reporting meta-analyses. Psychological Reports, 65(3), 759-770.
86.
RousseauD. M.ManningJ.DenyerD. (2008). Evidence in management and organizational science: Assembling the field's Full weight of scientific knowledge through syntheses. Academy of Management Annals, 2(1), 475-515.
87.
SchweigerS. A.StettlerT. R.BaldaufA.ZamudioC. (2019). The complementarity of strategic orientations: A meta-analytic synthesis and theory extension. Strategic Management Journal, 40(11), 1-30.
88.
SiddawayA.WoodA.HedgesL. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70(1), 747-770.
89.
SihagV.RijsdijkS. A. (2019). Organizational controls and performance outcomes: A meta-analytic assessment and extension. Journal of Management Studies, 56(1), 91-133.
90.
SivasubramaniamN.LiebowitzS. J.LackmanC. L. (2012). Determinants of new product development team performance: A meta-analytic review. Journal of Product Innovation Management, 29(5), 803-820.
91.
SleesmanD. J.ConlonD. E.McNamaraG.MilesJ. E. (2012). Cleaning up the big muddy: A meta-analytic review of the determinants of escalation of commitment. Academy of Management Journal, 55(3), 541-562.
92.
StanleyT. D.DoucouliagosH.GilesM.HeckemeyerJ. H.JohnstonR. J.LarocheP.NelsonJ.PaldamM.PootJ.PughG.RosenbergerR. S.RostK. (2013). Meta-analyses of economics research reporting guidelines. Journal of Economic Surveys, 27(2), 390-394.
93.
SteelP.BeugelsdijkS.AguinisH. (2021). The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews. Journal of International Business Studies, 52(1), 23-44.
94.
SteinkampF. (1998). A guide to independent coding in meta-analysis. Journal of Parapsychology, 62(2), 123.
95.
ThatcherS. M. B.PatelP. C. (2012). Group faultlines: A review, integration, and guide to future research. Journal of Management, 38(4), 969-1009.
96.
ThomasD. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of Evaluation, 27(2), 237-246.
97.
Van den BroeckA.FerrisD. L.ChangC. H.& RosenC.C. (2016). A review of self-determination theory's Basic psychological needs at work. Journal of Management, 42(5), 1195-1229.
98.
Van Der VegtG. S.BundersonJ. S. (2005). Learning and performance in multidisciplinary teams: The importance of collective team identification. Academy of Management Journal, 48(3), 532-547.
99.
WanousJ. P.SullivanS. E.MalinakJ. (1989). The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74(2), 259-264.
WilsonD. B. (2009). Systematic coding. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (pp. 159-176). Russell Sage Foundation.
102.
WoodJ. (2008). Methodology for dealing with duplicate study effects in a meta-analysis. Organizational Research Methods, 11(1), 79-95.
103.
ZhaoH.WayneS. J.GlibkowskiB. C.BravoJ. (2007). The impact of psychological contract breach on work-related outcomes: A meta-analysis. Personnel Psychology, 60(3), 647-680.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
