Heterogeneity in Meta-Analytic Effect Sizes: An Assessment of the Current State of the Literature

Abstract

Heterogeneity refers to the variability in effect sizes across different samples and is one of the major criteria to judge the importance and advancement of a scientific area. To determine how studies in the organizational sciences address heterogeneity, we conduct two studies. In study 1, we examine how meta-analytic studies conduct heterogeneity assessments and report and interpret the obtained results. To do so, we coded heterogeneity-related information from meta-analytic studies published in five leading journals. We found that most meta-analytic studies report several heterogeneity statistics. At the same time, however, there tends to be a lack of detail and thoroughness in the interpretation of these statistics. In study 2, we review how primary studies report heterogeneity-related results and conclusions from meta-analyses. We found that the quality of the reporting of heterogeneity-related information in primary studies tends to be poor and unrelated to the detail and thoroughness with which meta-analytic studies report and interpret the statistics. Based on our findings, we discuss implications for practice and provide recommendations for how heterogeneity assessments should be conducted and communicated in future research.

Keywords

heterogeneity between-study variability meta-analysis

Get full access to this article

View all access options for this article.

References

Ahn

Ames

A. J.

Myers

N. D.

(2012). A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82(4), 436-476. https://doi.org/10.3102/0034654312458162

Appelbaum

Cooper

Kline

R. B.

Mayo-Wilson

Nezu

A. M.

Rao

S. M.

(2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3-25. https://doi.org/10.1037/amp0000191

Aytug

Z. G.

Rothstein

H. R.

Zhou

Kern

M. C.

(2012). Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15(1), 103-133. https://doi.org/10.1177/1094428111403495

Binning

J. F.

Barrett

G. V.

(1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74(3), 478-494. https://doi.org/10.1037/0021-9010.74.3.478

Bolino

M. C.

Hsiung

H.-H.

Harvey

LePine

J. A.

(2015). Well, I’m tired of tryin’!” Organizational citizenship behavior and citizenship fatigue. Journal of Applied Psychology, 100(1), 56-74. https://doi.org/10.1037/a0037583

Borenstein

(2019). Heterogeneity in meta-analysis. In Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (pp. 453-470). Russel Sage Foundation.

Borenstein

Hedges

L. V.

Higgins

J. P.

Rothstein

H. R.

(2009). Introduction to meta-analysis. Wiley.

Brannick

M. T.

French

K. A.

Rothstein

H. R.

Kiselica

A. M.

Apostoloski

(2021). Capturing the underlying distribution in meta-analysis: Credibility and tolerance intervals. Research Synthesis Methods, 12(3), 264-290. https://doi.org/10.1002/jrsm.1479

Byron

Khazanchi

Nazarian

(2010). The relationship between stressors and creativity: A meta-analysis examining competing theoretical models. Journal of Applied Psychology, 95(1), 201-212. https://doi.org/10.1037/a0017868

10.

Carlson

K. D.

F. X.

(2011). Citing and building on meta-analytic findings: A review and recommendations. Organizational Research Methods, 14(4), 696-717. https://doi.org/10.1177/1094428110384272

11.

Cochran

W. G.

(1950). The comparison of percentages in matched samples. Biometrika, 37(3-4), 256-266. https://doi.org/10.1093/biomet/37.3-4.256

12.

Cooper

H. M.

Hedges

L. V.

(2009). Potentials and limitations. In Cooper

H. M.

Hedges

L. V.

Valentine

J. C.

(Eds.), Handbook of research synthesis and meta-analysis (pp. 561-571). Russell Sage Foundation.

13.

Cortina

J. M.

(2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6(4), 415-439. https://doi.org/10.1177/1094428103257358

14.

Cortina

J. M.

Goldstein

N. B.

Payne

S. C.

Davison

H. K.

Gilliland

S. W.

(2000). The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Personnel Psychology, 53(2), 325-351. https://doi.org/10.1111/j.1744-6570.2000.tb00204.x

15.

Crawford

E. R.

LePine

J. A.

Rich

B. L.

(2010). Linking job demands and resources to employee engagement and burnout: A theoretical extension and meta-analytic test. Journal of Applied Psychology, 95(5), 834-848. https://doi.org/10.1037/a0019364

16.

DerSimonian

Laird

(1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7(3), 177-188. https://doi.org/10.1016/0197-2456(86)90046-2

17.

DeSimone

J. A.

Brannick

M. T.

O’Boyle

E. H.

Ryu

J. W.

(2021). Recommendations for reviewing meta-analyses in organizational research. Organizational Research Methods, 24(4), 694-717. https://doi.org/10.1177/1094428120967089

18.

DeSimone

J. A.

Köhler

Schoen

J. L.

(2019). If it were only that easy: The use of meta-analytic research by organizational scholars. Organizational Research Methods, 22(4), 867-891. https://doi.org/10.1177/1094428118756743

19.

Dhanani

L. Y.

Beus

J. M.

Joseph

D. L.

(2018). Workplace discrimination: A meta-analytic extension, critique, and future research agenda. Personnel Psychology, 71(2), 147-179. https://doi.org/10.1111/peps.12254

20.

Dieckmann

N. F.

Malle

B. F.

Bodner

T. E.

(2009). An empirical assessment of meta-analytic practice. Review of General Psychology, 13(2), 101-115. https://doi.org/10.1037/a0015107

21.

Eatough

E. M.

Chang

C.-H.

Miloslavic

S. A.

Johnson

R. E.

(2011). Relationships of role stressors with organizational citizenship behavior: A meta-analysis. Journal of Applied Psychology, 96(3), 619-632. https://doi.org/10.1037/a0021887

22.

Geyskens

Krishnan

Steenkamp

J.-B. E. M.

Cunha

P. V.

(2009). A review and evaluation of meta-analysis practices in management research. Journal of Management, 35(2), 393-419. https://doi.org/10.1177/0149206308328501

23.

Halbesleben

J. R. B.

(2011). Sources of social support and burnout: A meta-analytic test of the conservation of resources model": Correction. Journal of Applied Psychology, 96(1), 182-182. https://doi.org/10.1037/a0021982

24.

Hartnell

C. A.

A. Y.

Kinicki

(2011). Organizational culture and organizational effectiveness: A meta-analytic investigation of the competing values framework's theoretical suppositions. Journal of Applied Psychology, 96(4), 677-694. https://doi.org/10.1037/a0021987

25.

Hedges

L. V.

Olkin

(1985). Statistical methods for meta-analysis. Academic Press.

26.

Hedges

L. V.

Vevea

J. L.

(1998). Fixed-and random-effects models in meta-analysis. Psychological Methods, 3(4), 486-504. https://doi.org/10.1037/1082-989X.3.4.486

27.

Higgins

J. P. T.

Thompson

S. G.

Deeks

J. J.

Altman

D. G.

(2003). Measuring inconsistency in meta-analyses. BMJ, 327(7414), 557-560. https://doi.org/10.1136/bmj.327.7414.557

28.

Higgins

J. P.

Thomas

Chandler

Cumpston

Page

M. J.

Welch

V. A.

(Eds.). (2021). Cochrane handbook for systematic reviews of interventions; Version 6.2. The Cochrane Collaboration. https://training.cochrane.org/handbook .

29.

Highhouse

Zickar

M. J.

Melick

S. R.

(2020). Prestige and relevance of the scholarly journals: Impressions of SIOP members. Industrial and Organizational Psychology, 13(3), 273-290. https://doi.org/10.1017/iop.2020.2

30.

Hoobler

J. M.

Masterson

C. R.

Nkomo

S. M.

Michel

E. J.

(2018). The business case for women leaders: Meta-analysis, research critique, and path forward. Journal of Management, 44(6), 2473-2499. https://doi.org/10.1177/0149206316628643

31.

Hunter

J. E.

Schmidt

F. L.

(1989). Methods of meta-analysis: Correcting error and bias in research findings. Sage.

32.

Hunter

J. E.

Schmidt

F. L.

(2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Sage.

33.

Ioannidis

J. P. A.

Patsopoulos

N. A.

Rothstein

H. R.

(2008). Reasons or excuses for avoiding meta-analysis in forest plots. BMJ, 336(7658), 1413-1415. https://doi.org/10.1136/bmj.a117

34.

Kepes

Banks

McDaniel

Whetzel

M. A.

(2012). Publication bias in the organizational sciences. Organizational Research Methods, 15(4), 624-662. https://doi.org/10.1177/1094428112452760

35.

Kepes

Bennett

A. A.

McDaniel

M. A.

(2014). Evidence-based management and the trustworthiness of our cumulative scientific knowledge: Implications for teaching, research, and practice. Academy of Management Learning & Education, 13(3), 446-466. https://doi.org/10.5465/amle.2013.0193

36.

Kepes

McDaniel

M. A.

Brannick

M. T.

Banks

G. C.

(2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the meta-analytic reporting standards). Journal of Business and Psychology, 28(2), 123-143. https://doi.org/10.1007/s10869-013-9300-2

37.

Kepes

Wang

Cortina

J. M.

(in press). Assessing publication bias: A 7-step user’s guide with best-practice recommendations. Journal of Business and Psychology. https://doi.org/10.1007/s10869-022-09840-0

38.

Kisamore

J. L.

Brannick

M. T.

(2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11(1), 35-53. https://doi.org/10.1177/1094428106287393

39.

Kooij

D. T. A. M.

Guest

D. E.

Clinton

Knight

Jansen

P. G. W.

Dikkers

J. S. E.

(2013). How the impact of HR practices on employee well-being and performance changes with age. Human Resource Management Journal, 23(1), 18-35. https://doi.org/10.1111/1748-8583.12000

40.

Koslowsky

Sagie

(1993). On the efficacy of credibility intervals as indicators of moderator effects in meta-analytic research. Journal of Organizational Behavior, 14(7), 695-699. https://doi.org/10.1002/job.4030140708

41.

Köhler

Cortina

J. M.

(2021). Play it again, Sam! An analysis of constructive replication in the organizational sciences. Journal of Management, 47(2), 488-518. https://doi.org/10.1177/0149206319843985

42.

Landay

Harms

P. D.

Credé

(2019). Shall we serve the dark lords? A meta-analytic review of psychopathy and leadership. Journal of Applied Psychology, 104(1), 183-196. https://doi.org/10.1037/apl0000357

43.

Lee

Madhavan

(2010). Divestiture and firm performance: A meta-analysis. Journal of Management, 36(6), 1345-1371. https://doi.org/10.1177/0149206309360931

44.

Leung

Bartunek

J. M.

(2012). Enabling evidence-based management: Bridging the gap between academics and practitioners. In The Oxford handbook of evidence-based management (pp. 165-180). Oxford University Press.

45.

Mackey

J. D.

McAllister

C. P.

Maher

L. P.

Wang

(2019). Leaders and followers behaving badly: A meta-analytic examination of curvilinear relationships between destructive leadership and followers’ workplace behaviors. Personnel Psychology, 72(1), 3-47. https://doi.org/10.1111/peps.12286

46.

Makel

M. C.

Plucker

J. A.

Hegarty

(2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537-542. https://doi.org/10.1177/1745691612460688

47.

Marín-Martínez

Sánchez-Meca

(2010). Weighting by inverse variance or by sample size in random-effects meta-analysis. Educational and Psychological Measurement, 70(1), 56-73. https://doi.org/10.1177/0013164409344534

48.

Moher

Liberati

Tetzlaff

Altman

D. G.

(2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264-269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135

49.

NHLBI (2013). Study quality assessment tools. Retrieved March 2, 2023, from https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools.

50.

I.-S.

(2020). Beyond meta-analysis: Secondary uses of meta-analytic data. Annual Review of Organizational Psychology and Organizational Behavior, 7(1), 125-153. https://doi.org/10.1146/annurev-orgpsych-012119-045006

51.

Pearlman

Schmidt

F. L.

Hunter

J. E.

(1980). Validity generalization results for tests used to predict job proficiency and training success in clerical occupations. Journal of Applied Psychology, 65(4), 373-406. https://doi.org/10.1037/0021-9010.65.4.373

52.

Petrou

Demerouti

Schaufeli

W. B.

(2018). Crafting the change: The role of employee job crafting behaviors for successful organizational change. Journal of Management, 44(5), 1766-1792. https://doi.org/10.1177/0149206315624961

53.

Podsakoff

P. M.

MacKenzie

S. B.

Podsakoff

N. P.

Bachrach

D. G.

(2008). Scholarly influence in the field of management: A bibliometric analysis of the determinants of university and author impact in the management literature in the past quarter century. Journal of Management, 34(4), 641-720. https://doi.org/10.1177/0149206308319533

54.

Porter

C. M.

Woo

S. E.

Allen

D. G.

Keith

M. G.

(2019). How do instrumental and expressive network positions relate to turnover? A meta-analytic investigation. Journal of Applied Psychology, 104(4), 511-536. https://doi.org/10.1037/apl0000351

55.

Praet

(2013). Family firms and the divestment decision: An agency perspective. Journal of Family Business Strategy, 4(1), 34-41. https://doi.org/10.1016/j.jfbs.2012.12.002

56.

Ripley

Venables

Bates

D. M.

Hornik

Gebhardt

Firth

(2023). MASS: Support functions and datasets for Venables and Ripley’s MASS (R package version 7.3-58.2) .

57.

Rozeboom

W. W.

(1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57(5), 416-428. https://doi.org/10.1037/h0042040

58.

Russo

M. W.

(2007). How to review a meta-analysis. Gastroenterology & Hepatolog, 3(8), 637-642.

59.

Rücker

Schwarzer

Carpenter

J. R.

Schumacher

(2008). Undue reliance on I² in assessing heterogeneity may mislead. BMC Medical Research Methodology, 8(1), 79. https://doi.org/10.1186/1471-2288-8-79

60.

Sacramento

C. A.

Fay

West

M. A.

(2013). Workplace duties or opportunities? Challenge stressors, regulatory focus, and creativity. Organizational Behavior and Human Decision Processes, 121(2), 141-157. https://doi.org/10.1016/j.obhdp.2013.01.008

61.

Sarwar

Muhammad

(2021). Impact of organizational mistreatment on employee performance in the hotel industry. International Journal of Contemporary Hospitality Management, 33(2), 513-533. http://doi.org/10.1108/IJCHM-01-2020-0051

62.

Schalken

Rietbergen

(2017). The reporting quality of systematic reviews and meta-analyses in industrial and organizational psychology: A systematic review. Frontiers in Psychology, 8, article 1395. https://doi.org/10.3389/fpsyg.2017.01395

63.

Schmidt

Hunter

(2002). Are there benefits from NHST? American Psychologist, 57(1), 65-66. https://doi.org/10.1037/0003-066X.57.1.65

64.

Schmidt

F. L.

(1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1(2), 115-129. https://doi.org/10.1037/1082-989X.1.2.115

65.

Schmidt

F. L.

Hunter

J. E.

(1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62(5), 529-540. https://doi.org/10.1037/0021-9010.62.5.529

66.

Schmidt

F. L.

Hunter

J. E.

(1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274. https://doi.org/10.1037/0033-2909.124.2.262

67.

Schmidt

F. L.

Law

Hunter

J. E.

Rothstein

H. R.

Pearlman

McDaniel

(1993). Refinements in validity generalization methods: Implications for the situational specificity hypothesis. Journal of Applied Psychology, 78(1), 3-12. https://doi.org/10.1037/0021-9010.78.1.3

68.

Shea

B. J.

Reeves

B. C.

Wells

Thuku

Hamel

Moran

Moher

Tugwell

Welch

Kristjansson

Henry

D. A.

(2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008. https://doi.org/10.1136/bmj.j4008

69.

Thatcher

S. M. B.

Patel

P. C.

(2011). Demographic faultlines: A meta-analysis of the literature. Journal of Applied Psychology, 96(6), 1119-1139. https://doi.org/10.1037/a0024167

70.

Viswesvaran

Sanchez

J. I.

(1998). Moderator search in meta-analysis: A review and cautionary note on existing approaches. Educational and Psychological Measurement, 58(1), 77-87. https://doi.org/10.1177/0013164498058001007

71.

von Hippel

P. T.

(2015). The heterogeneity statistic I² can be biased in small meta-analyses. BMC Medical Research Methodology, 15(1), 35. https://doi.org/10.1186/s12874-015-0024-z

72.

Whitener

E. M.

(1990). Confusion of confidence intervals and credibility intervals in meta-analysis. Journal of Applied Psychology, 75(3), 315-321. https://doi.org/10.1037/0021-9010.75.3.315

73.

Downes

P. E.

Carter

K. M.

O'Boyle

E. H.

(2016). The problem of effect size heterogeneity in meta-analytic structural equation modeling. Journal of Applied Psychology, 101(10), 1457-1473. https://doi.org/10.1037/apl0000141

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.35 MB