An Assessment of the Magnitude of Effect Sizes

Abstract

This study compiles information from more than 250 meta-analyses conducted over the past 30 years to assess the magnitude of reported effect sizes in the organizational behavior (OB)/human resources (HR) literatures. Our analysis revealed an average uncorrected effect of r = .227 and an average corrected effect of ρ = .278 (SDρ = .140). Based on the distribution of effect sizes we report, Cohen’s effect size benchmarks are not appropriate for use in OB/HR research as they overestimate the actual breakpoints between small, medium, and large effects. We also assessed the average statistical power reported in meta-analytic conclusions and found substantial evidence that the majority of primary studies in the management literature are statistically underpowered. Finally, we investigated the impact of the file drawer problem in meta-analyses and our findings indicate that the file drawer problem is not a significant concern for meta-analysts. We conclude by discussing various implications of this study for OB/HR researchers.

Keywords

effect size meta-analysis statistical power file drawer problem

Get full access to this article

View all access options for this article.

References

Aguinis

Beaty

J. C.

Boik

R. J.

Pierce

C. A.

(2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90(1), 94-106.

Aguinis

Dalton

D. R.

Bosco

F. A.

Pierce

C. A.

Dalton

C. M.

(2011). Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly impact. Journal of Management, 37(1), 5-38.

Aguinis

Werner

Abbott

J. A. L.

Angert

Park

J. H.

Kohlhausen

(2010). Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13, 515-539.

Aiken

L. S.

West

S. G.

(1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.

American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.

Argyris

(1957). Personality and organization: The conflict between system and the individual. New York, NY: Harper & Row.

Asendorpf

J. B.

Conner

De Fruyt

De Houwer

Denissen

J. J. A.

Fiedler

. . . Wicherts

J. M.

(2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119.

Bacharach

S. B.

(1989). Organizational theories: Some criteria for evaluation. Academy of Management Review, 14, 496-515.

Bakker

van Dijk

Wicherts

J. M.

(2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554.

10.

Bass

B. M.

(1954). The leaderless group discussion. Psychological Bulletin, 51, 465-492.

11.

Bligh

M. C.

Kohles

J. C.

Pearce

C. L.

Justin

J. E.

Stovall

J. F.

(2007). When the romance is over: Follower perspectives of aversive leadership. Journal of Applied Psychology, 56, 528-557.

12.

Bosco

F. A.

Aguinis

Singh

Field

J. G.

Pierce

C. A.

(2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100, 431-449.

13.

Brewer

J. K.

(1972). On the power of statistical tests in the “American Educational Research Journal.” American Educational Research Journal, 9, 391-401.

14.

Campion

(1993). Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 705-718.

15.

Cashen

L. H.

Geiger

S. W.

(2004). Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7, 151-167.

16.

Chan

M. L. E.

Arvey

(2012). Meta-analysis and the development of knowledge. Perspectives on Psychological Science, 7(1), 79-92.

17.

Chase

L. J.

Chase

R. B.

(1976). A statistical power analysis of applied psychological research. Journal of Applied Psychology, 61, 234-237.

18.

Chase

L. J.

Tucker

R. K.

(1975). A power-analytic examination of contemporary communication research. Communications Monographs, 42(1), 29-41.

19.

Coe

(2002). It’s the effect size, stupid: What effect size is and why it is important. Paper presented at the annual conference of the British Educational Research Association, University of Exeter, England.

20.

Cohen

(1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153.

21.

Cohen

(1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.

22.

Cohen

(1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.

23.

Cohen

(1992). A power primer. Psychological Bulletin, 112, 155-159.

24.

Cohen

(1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.

25.

Dalton

D. R.

Aguinis

Dalton

C. M.

Bosco

F. A.

Pierce

C. A.

(2012). Revisiting the file drawer problem in meta-analysis: An assessment of published and nonpublished correlation matrices. Personnel Psychology, 65, 221-249.

26.

Dalton

D. R.

Daily

C. M.

Johnson

J. L.

Ellstrand

A. E.

(1999). Number of directors and financial performance: A meta-analysis. Academy of Management Journal, 19, 674-686.

27.

Edwards

E. R.

(2008). To prosper, organizational psychology should . . . overcome methodological barriers to progress. Journal of Organizational Behavior, 29, 469-491.

28.

Ferguson

C. J.

(2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40, 532-538.

29.

Ferguson

C. J.

Brannick

M. T.

(2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17(1), 120-128.

30.

Ferguson

C. J.

Heene

(2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7, 555-561.

31.

Ferguson

T. D.

Ketchen

D. J.

Jr. (1999). Organizational configurations and performance: The role of statistical power in extant research. Strategic Management Journal, 20, 385-395.

32.

Fisher

R. A.

(1925). Statistical methods for research workers. London, England: Oliver & Boyd.

33.

Fritz

Scherndl

Kühberger

(2012). A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough? Theory & Psychology, 23(1), 98-122.

34.

Glass

G. V.

Hakstian

A. R.

(1969). Measures of association in comparative experiments: Their development and interpretation. American Educational Research Journal, 6, 403-414.

35.

Glass

G. V.

McGaw

Smith

M. L.

(1981). Meta-analysis in social research (Vol. 56). Beverly Hills, CA: Sage.

36.

Greenwald

A. G.

(1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1-20.

37.

Greenwald

A. G.

(1993). Consequences of prejudice against the null hypothesis. In Keren

Lewis

(Eds.), A handbook for data analysis in the behavioral sciences (pp. 419-448). Hillsdale, NJ: Lawrence Erlbaum.

38.

Hall

de Antueno

Webber

(2007). Publication bias in the medical literature: A review by a Canadian Research Ethics Board. Canadian Journal of Anesthesia, 54, 380-388.

39.

Hall

S. M.

Brannick

M. T.

(2002). Comparison of two random-effects methods of meta-analysis. Journal of Applied Psychology, 87, 377-389.

40.

Howard

G. S.

Maxwell

S. E.

Fleming

K. J.

(2000). The proof of the pudding: An illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods, 5, 315-332.

41.

Hunter

J. E.

Hirsh

H. R.

(1987). Applications of meta-analysis. In Cooper

C. L.

Robertson

I. T.

(Eds.), International review of industrial and organizational psychology (pp. 321-357). New York, NY: Wiley.

42.

Hunter

J. E.

Schmidt

F. L.

(2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.

43.

Jenkins

W. O.

(1947). A review of leadership studies with particular reference to military problems. Psychological Bulletin, 44(1), 54-79.

44.

Judge

T. A.

Thoresen

C. J.

Bono

J. E.

Patton

G. K.

(2001). The job satisfaction–job performance relationship: A qualitative and quantitative review. Psychological Bulletin, 127, 376-407.

45.

Kepes

Banks

G. C.

McDaniel

Whetzel

D. L.

(2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624-662.

46.

Kirk

R. E.

(1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.

47.

Kluger

A. N.

DeNisi

(1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284.

48.

Kroll

R. M.

Chase

L. J.

(1975). Communication disorders: A power analytic assessment of recent research. Journal of Communication Disorders, 8, 237-247.

49.

Levine

T. R.

Asada

K. J.

Carpenter

(2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against nonsignificant findings. Communication Monographs, 76, 286-302.

50.

Lipsey

M. W.

(Ed.). (1998). Design sensitivity: Statistical power for applied experimental research. Thousand Oaks, CA: Sage.

51.

Lipsey

M. W.

Wilson

D. B.

(1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.

52.

Maxwell

S. E.

(2003). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163.

53.

Mazen

A. M.

Graf

L. A.

Kellogg

C. E.

Hemmasi

(1987). Statistical power in contemporary management research. Academy of Management Journal, 30, 369-380.

54.

Mazen

A. M.

Hemmasi

Lewis

M. F.

(2006). Assessment of statistical power in contemporary strategy research. Strategic Management Journal, 8, 403-410.

55.

Meindl

J. R.

Ehrlich

S. B.

Dukerich

J. M.

(1985). The romance of leadership. Administrative Science Quarterly, 30(1), 78-102.

56.

Mone

M. A.

Mueller

G. C.

Mauland

(1996). The perceptions and usage of statistical power in applied psychology and management research. Personnel Psychology, 49, 103-120.

57.

Penick

J. E.

Brewer

J. K.

(1972). The power of statistical tests in science teaching research. Journal of Research in Science Teaching, 9, 377-381.

58.

Peters

C. C.

Van Voorhis

W. R.

(1940). Statistical procedures and their mathematical bases. New York, NY: McGraw-Hill.

59.

Prentice

D. A.

Miller

D. T.

(1992). When small effects are impressive. Psychological Bulletin, 112, 160-164.

60.

Richard

F. D.

Bond

C. F.

Jr. Stokes-Zoota

J. J.

(2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331-363.

61.

Rosenthal

(Ed.). (1994). Parametric measures of effect size. New York, NY: Russell Sage Foundation.

62.

Rosenthal

Rubin

D. B.

(1979). A note on percent variance explained as a measure of the importance of effects. Journal of Applied Social Psychology, 9, 395-396.

63.

Rosenthal

Rubin

D. B.

(1982). A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology, 74, 166-169.

64.

Rossi

J. S.

(1990). Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting and Clinical psychology, 58, 646-656.

65.

Rothstein

H. R.

Bushman

B. J.

(2012). Publication bias in psychological science: Comment on Ferguson and Brannick 2012. Psychological Methods, 17, 129-136.

66.

Rothstein

H. R.

Sutton

A. J.

Borenstein

(2005). Publication bias in meta-analysis. West Sussex, England: John Wiley.

67.

Rozeboom

W. W.

(1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416-428.

68.

Sawyer

A. G.

Ball

A. D.

(1981). Statistical power and effect size in marketing research. Journal of Marketing Research, 18, 275-290.

69.

Scandura

Williams

(2000). Research methodology in management: Current practices, trends, and implications for future research. Academy of Management Journal, 43, 1248-1264.

70.

Schmidt

F. L.

(2008). Meta-Analysis: A constantly evolving research integration tool. Organizational Research Methods, 11(1), 96-113.

71.

Schulze

(2004). Meta-analysis: A comparison of approaches. Cambridge, MA: Hogrefe & Huber.

72.

Sedlmeier

Gigerenzer

(1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316.

73.

Smith

H. L.

Krueger

L. M.

(1933). A brief summary of literature on leadership. Bulletin of Scholastic Education, 9(4).

74.

Song

Parekh-Bhurke

Hooper

Loke

Ryder

Sutton

. . . Harvey

(2009). Extent of publication bias in different categories of research cohorts: A meta-analysis of empirical studies. BMC Medical Research Methodology, 9(1), 79. doi:10.1186/1471-2288-9-79

75.

Steel

(2013). Converting between meta-analytic variance estimates based on correlations and Fisher z. Paper presented at the 8th Annual Meeting of the Society for Research Synthesis Methodology, Providence, RI.

76.

Steel

P. D.

Kammeyer-Mueller

(2008). Bayesian variance estimation for meta-analysis: Quantifying our uncertainty. Organizational Research Methods, 11(1), 54-78.

77.

Steel

P. D.

Kammeyer-Mueller

Nguyen

(2012). Why meta-analysis exaggerates generalizability and underestimates situational specificity. In Morris

(Chair), Methodological advances in meta-analysis. Symposium conducted at the annual meeting of the Society for Industrial Organizational Psychology, San Diego, CA.

78.

Steiner

D. D.

Lane

I. M.

Dobbins

G. H.

Schnur

McConnell

(1991). A review of meta-analyses in organizational behavior and human resources management: An empirical assessment. Educational and Psychological Measurement, 51, 609-626.

79.

Stemig

M. S.

Sackett

P. R.

(2013). Another look into the file drawer problem in meta-analysis. Paper presented at the 28th Annual Conference of the Society for Industrial and Organizational Psychology, Houston, TX.

80.

Stogdill

R. M.

(1948). Personal factors associated with leadership: A survey of the literature. Journal of Psychology, 25(1), 35-71.

81.

Sutton

A. J.

(2009). Publication bias. In Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435-454). New York, NY: Russell Sage Foundation.

82.

Tett

R. P.

Meyer

J. P.

Roese

N. J.

(1994). Applications of meta-analysis: 1987-1992. In Cooper

C. L.

Robertson

I. T.

(Eds.), International review of industrial and organizational psychology (Vol. 9, pp. 71-112). London, England: John Wiley.

83.

Weber

Camerer

Rottenstreich

Knez

(2001). The illusion of leadership: Misattribution of cause in coordination games. Organization Science, 12, 582-598.

84.

Yates

(1951). The influence of statistical methods for research workers on the development of the science of statistics. Journal of the American Statistical Association, 46(253), 19-34.

85.

Zedeck

(2003). Editorial. Journal of Applied Psychology, 88, 3-5.