Heterogeneity refers to the variability in effect sizes across different samples and is one of the major criteria to judge the importance and advancement of a scientific area. To determine how studies in the organizational sciences address heterogeneity, we conduct two studies. In study 1, we examine how meta-analytic studies conduct heterogeneity assessments and report and interpret the obtained results. To do so, we coded heterogeneity-related information from meta-analytic studies published in five leading journals. We found that most meta-analytic studies report several heterogeneity statistics. At the same time, however, there tends to be a lack of detail and thoroughness in the interpretation of these statistics. In study 2, we review how primary studies report heterogeneity-related results and conclusions from meta-analyses. We found that the quality of the reporting of heterogeneity-related information in primary studies tends to be poor and unrelated to the detail and thoroughness with which meta-analytic studies report and interpret the statistics. Based on our findings, we discuss implications for practice and provide recommendations for how heterogeneity assessments should be conducted and communicated in future research.
AhnS.AmesA. J.MyersN. D. (2012). A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82(4), 436-476. https://doi.org/10.3102/0034654312458162
2.
AppelbaumM.CooperH.KlineR. B.Mayo-WilsonE.NezuA. M.RaoS. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3-25. https://doi.org/10.1037/amp0000191
3.
AytugZ. G.RothsteinH. R.ZhouW.KernM. C. (2012). Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15(1), 103-133. https://doi.org/10.1177/1094428111403495
4.
BinningJ. F.BarrettG. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74(3), 478-494. https://doi.org/10.1037/0021-9010.74.3.478
5.
BolinoM. C.HsiungH.-H.HarveyJ.LePineJ. A. (2015). Well, I’m tired of tryin’!” Organizational citizenship behavior and citizenship fatigue. Journal of Applied Psychology, 100(1), 56-74. https://doi.org/10.1037/a0037583
6.
BorensteinM. (2019). Heterogeneity in meta-analysis. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (pp. 453-470). Russel Sage Foundation.
7.
BorensteinM.HedgesL. V.HigginsJ. P.RothsteinH. R. (2009). Introduction to meta-analysis. Wiley.
8.
BrannickM. T.FrenchK. A.RothsteinH. R.KiselicaA. M.ApostoloskiN. (2021). Capturing the underlying distribution in meta-analysis: Credibility and tolerance intervals. Research Synthesis Methods, 12(3), 264-290. https://doi.org/10.1002/jrsm.1479
9.
ByronK.KhazanchiS.NazarianD. (2010). The relationship between stressors and creativity: A meta-analysis examining competing theoretical models. Journal of Applied Psychology, 95(1), 201-212. https://doi.org/10.1037/a0017868
10.
CarlsonK. D.JiF. X. (2011). Citing and building on meta-analytic findings: A review and recommendations. Organizational Research Methods, 14(4), 696-717. https://doi.org/10.1177/1094428110384272
CooperH. M.HedgesL. V. (2009). Potentials and limitations. In CooperH. M.HedgesL. V.ValentineJ. C. (Eds.), Handbook of research synthesis and meta-analysis (pp. 561-571). Russell Sage Foundation.
13.
CortinaJ. M. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6(4), 415-439. https://doi.org/10.1177/1094428103257358
14.
CortinaJ. M.GoldsteinN. B.PayneS. C.DavisonH. K.GillilandS. W. (2000). The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Personnel Psychology, 53(2), 325-351. https://doi.org/10.1111/j.1744-6570.2000.tb00204.x
15.
CrawfordE. R.LePineJ. A.RichB. L. (2010). Linking job demands and resources to employee engagement and burnout: A theoretical extension and meta-analytic test. Journal of Applied Psychology, 95(5), 834-848. https://doi.org/10.1037/a0019364
DeSimoneJ. A.BrannickM. T.O’BoyleE. H.RyuJ. W. (2021). Recommendations for reviewing meta-analyses in organizational research. Organizational Research Methods, 24(4), 694-717. https://doi.org/10.1177/1094428120967089
18.
DeSimoneJ. A.KöhlerT.SchoenJ. L. (2019). If it were only that easy: The use of meta-analytic research by organizational scholars. Organizational Research Methods, 22(4), 867-891. https://doi.org/10.1177/1094428118756743
19.
DhananiL. Y.BeusJ. M.JosephD. L. (2018). Workplace discrimination: A meta-analytic extension, critique, and future research agenda. Personnel Psychology, 71(2), 147-179. https://doi.org/10.1111/peps.12254
20.
DieckmannN. F.MalleB. F.BodnerT. E. (2009). An empirical assessment of meta-analytic practice. Review of General Psychology, 13(2), 101-115. https://doi.org/10.1037/a0015107
21.
EatoughE. M.ChangC.-H.MiloslavicS. A.JohnsonR. E. (2011). Relationships of role stressors with organizational citizenship behavior: A meta-analysis. Journal of Applied Psychology, 96(3), 619-632. https://doi.org/10.1037/a0021887
22.
GeyskensI.KrishnanR.SteenkampJ.-B. E. M.CunhaP. V. (2009). A review and evaluation of meta-analysis practices in management research. Journal of Management, 35(2), 393-419. https://doi.org/10.1177/0149206308328501
23.
HalbeslebenJ. R. B. (2011). Sources of social support and burnout: A meta-analytic test of the conservation of resources model": Correction. Journal of Applied Psychology, 96(1), 182-182. https://doi.org/10.1037/a0021982
24.
HartnellC. A.OuA. Y.KinickiA. (2011). Organizational culture and organizational effectiveness: A meta-analytic investigation of the competing values framework's theoretical suppositions. Journal of Applied Psychology, 96(4), 677-694. https://doi.org/10.1037/a0021987
25.
HedgesL. V.OlkinI. (1985). Statistical methods for meta-analysis. Academic Press.
HigginsJ. P. T.ThompsonS. G.DeeksJ. J.AltmanD. G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327(7414), 557-560. https://doi.org/10.1136/bmj.327.7414.557
28.
HigginsJ. P.ThomasJ.ChandlerJ.CumpstonM.LiT.PageM. J.WelchV. A. (Eds.). (2021). Cochrane handbook for systematic reviews of interventions; Version 6.2. The Cochrane Collaboration. https://training.cochrane.org/handbook.
29.
HighhouseS.ZickarM. J.MelickS. R. (2020). Prestige and relevance of the scholarly journals: Impressions of SIOP members. Industrial and Organizational Psychology, 13(3), 273-290. https://doi.org/10.1017/iop.2020.2
30.
HooblerJ. M.MastersonC. R.NkomoS. M.MichelE. J. (2018). The business case for women leaders: Meta-analysis, research critique, and path forward. Journal of Management, 44(6), 2473-2499. https://doi.org/10.1177/0149206316628643
31.
HunterJ. E.SchmidtF. L. (1989). Methods of meta-analysis: Correcting error and bias in research findings. Sage.
32.
HunterJ. E.SchmidtF. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Sage.
33.
IoannidisJ. P. A.PatsopoulosN. A.RothsteinH. R. (2008). Reasons or excuses for avoiding meta-analysis in forest plots. BMJ, 336(7658), 1413-1415. https://doi.org/10.1136/bmj.a117
34.
KepesS.BanksG.McDanielC.WhetzelM. A.LD. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15(4), 624-662. https://doi.org/10.1177/1094428112452760
35.
KepesS.BennettA. A.McDanielM. A. (2014). Evidence-based management and the trustworthiness of our cumulative scientific knowledge: Implications for teaching, research, and practice. Academy of Management Learning & Education, 13(3), 446-466. https://doi.org/10.5465/amle.2013.0193
36.
KepesS.McDanielM. A.BrannickM. T.BanksG. C. (2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the meta-analytic reporting standards). Journal of Business and Psychology, 28(2), 123-143. https://doi.org/10.1007/s10869-013-9300-2
37.
KepesS.WangW.CortinaJ. M. (in press). Assessing publication bias: A 7-step user’s guide with best-practice recommendations. Journal of Business and Psychology. https://doi.org/10.1007/s10869-022-09840-0
38.
KisamoreJ. L.BrannickM. T. (2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11(1), 35-53. https://doi.org/10.1177/1094428106287393
39.
KooijD. T. A. M.GuestD. E.ClintonM.KnightT.JansenP. G. W.DikkersJ. S. E. (2013). How the impact of HR practices on employee well-being and performance changes with age. Human Resource Management Journal, 23(1), 18-35. https://doi.org/10.1111/1748-8583.12000
40.
KoslowskyM.SagieA. (1993). On the efficacy of credibility intervals as indicators of moderator effects in meta-analytic research. Journal of Organizational Behavior, 14(7), 695-699. https://doi.org/10.1002/job.4030140708
41.
KöhlerT.CortinaJ. M. (2021). Play it again, Sam! An analysis of constructive replication in the organizational sciences. Journal of Management, 47(2), 488-518. https://doi.org/10.1177/0149206319843985
42.
LandayK.HarmsP. D.CredéM. (2019). Shall we serve the dark lords? A meta-analytic review of psychopathy and leadership. Journal of Applied Psychology, 104(1), 183-196. https://doi.org/10.1037/apl0000357
LeungO.BartunekJ. M. (2012). Enabling evidence-based management: Bridging the gap between academics and practitioners. In The Oxford handbook of evidence-based management (pp. 165-180). Oxford University Press.
45.
MackeyJ. D.McAllisterC. P.MaherL. P.WangG. (2019). Leaders and followers behaving badly: A meta-analytic examination of curvilinear relationships between destructive leadership and followers’ workplace behaviors. Personnel Psychology, 72(1), 3-47. https://doi.org/10.1111/peps.12286
46.
MakelM. C.PluckerJ. A.HegartyB. (2012). Replications in psychology research: How often do they really occur?Perspectives on Psychological Science, 7(6), 537-542. https://doi.org/10.1177/1745691612460688
47.
Marín-MartínezF.Sánchez-MecaJ. (2010). Weighting by inverse variance or by sample size in random-effects meta-analysis. Educational and Psychological Measurement, 70(1), 56-73. https://doi.org/10.1177/0013164409344534
48.
MoherD.LiberatiA.TetzlaffJ.AltmanD. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264-269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135
OhI.-S. (2020). Beyond meta-analysis: Secondary uses of meta-analytic data. Annual Review of Organizational Psychology and Organizational Behavior, 7(1), 125-153. https://doi.org/10.1146/annurev-orgpsych-012119-045006
51.
PearlmanK.SchmidtF. L.HunterJ. E. (1980). Validity generalization results for tests used to predict job proficiency and training success in clerical occupations. Journal of Applied Psychology, 65(4), 373-406. https://doi.org/10.1037/0021-9010.65.4.373
52.
PetrouP.DemeroutiE.SchaufeliW. B. (2018). Crafting the change: The role of employee job crafting behaviors for successful organizational change. Journal of Management, 44(5), 1766-1792. https://doi.org/10.1177/0149206315624961
53.
PodsakoffP. M.MacKenzieS. B.PodsakoffN. P.BachrachD. G. (2008). Scholarly influence in the field of management: A bibliometric analysis of the determinants of university and author impact in the management literature in the past quarter century. Journal of Management, 34(4), 641-720. https://doi.org/10.1177/0149206308319533
54.
PorterC. M.WooS. E.AllenD. G.KeithM. G. (2019). How do instrumental and expressive network positions relate to turnover? A meta-analytic investigation. Journal of Applied Psychology, 104(4), 511-536. https://doi.org/10.1037/apl0000351
55.
PraetA. (2013). Family firms and the divestment decision: An agency perspective. Journal of Family Business Strategy, 4(1), 34-41. https://doi.org/10.1016/j.jfbs.2012.12.002
56.
RipleyB.VenablesB.BatesD. M.HornikK.GebhardtA.FirthD. (2023). MASS: Support functions and datasets for Venables and Ripley’s MASS (R package version 7.3-58.2).
57.
RozeboomW. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57(5), 416-428. https://doi.org/10.1037/h0042040
58.
RussoM. W. (2007). How to review a meta-analysis. Gastroenterology & Hepatolog, 3(8), 637-642.
59.
RückerG.SchwarzerG.CarpenterJ. R.SchumacherM. (2008). Undue reliance on I2 in assessing heterogeneity may mislead. BMC Medical Research Methodology, 8(1), 79. https://doi.org/10.1186/1471-2288-8-79
60.
SacramentoC. A.FayD.WestM. A. (2013). Workplace duties or opportunities? Challenge stressors, regulatory focus, and creativity. Organizational Behavior and Human Decision Processes, 121(2), 141-157. https://doi.org/10.1016/j.obhdp.2013.01.008
61.
SarwarA.MuhammadL. (2021). Impact of organizational mistreatment on employee performance in the hotel industry. International Journal of Contemporary Hospitality Management, 33(2), 513-533. http://doi.org/10.1108/IJCHM-01-2020-0051
62.
SchalkenN.RietbergenC. (2017). The reporting quality of systematic reviews and meta-analyses in industrial and organizational psychology: A systematic review. Frontiers in Psychology, 8, article 1395. https://doi.org/10.3389/fpsyg.2017.01395
SchmidtF. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1(2), 115-129. https://doi.org/10.1037/1082-989X.1.2.115
65.
SchmidtF. L.HunterJ. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62(5), 529-540. https://doi.org/10.1037/0021-9010.62.5.529
66.
SchmidtF. L.HunterJ. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274. https://doi.org/10.1037/0033-2909.124.2.262
67.
SchmidtF. L.LawK.HunterJ. E.RothsteinH. R.PearlmanK.McDanielM. (1993). Refinements in validity generalization methods: Implications for the situational specificity hypothesis. Journal of Applied Psychology, 78(1), 3-12. https://doi.org/10.1037/0021-9010.78.1.3
68.
SheaB. J.ReevesB. C.WellsG.ThukuM.HamelC.MoranJ.MoherD.TugwellP.WelchV.KristjanssonE.HenryD. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008. https://doi.org/10.1136/bmj.j4008
69.
ThatcherS. M. B.PatelP. C. (2011). Demographic faultlines: A meta-analysis of the literature. Journal of Applied Psychology, 96(6), 1119-1139. https://doi.org/10.1037/a0024167
70.
ViswesvaranC.SanchezJ. I. (1998). Moderator search in meta-analysis: A review and cautionary note on existing approaches. Educational and Psychological Measurement, 58(1), 77-87. https://doi.org/10.1177/0013164498058001007
71.
von HippelP. T. (2015). The heterogeneity statistic I2 can be biased in small meta-analyses. BMC Medical Research Methodology, 15(1), 35. https://doi.org/10.1186/s12874-015-0024-z
72.
WhitenerE. M. (1990). Confusion of confidence intervals and credibility intervals in meta-analysis. Journal of Applied Psychology, 75(3), 315-321. https://doi.org/10.1037/0021-9010.75.3.315
73.
YuJ.DownesP. E.CarterK. M.O'BoyleE. H. (2016). The problem of effect size heterogeneity in meta-analytic structural equation modeling. Journal of Applied Psychology, 101(10), 1457-1473. https://doi.org/10.1037/apl0000141
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
