This study compiles information from more than 250 meta-analyses conducted over the past 30 years to assess the magnitude of reported effect sizes in the organizational behavior (OB)/human resources (HR) literatures. Our analysis revealed an average uncorrected effect of r = .227 and an average corrected effect of ρ = .278 (SDρ = .140). Based on the distribution of effect sizes we report, Cohen’s effect size benchmarks are not appropriate for use in OB/HR research as they overestimate the actual breakpoints between small, medium, and large effects. We also assessed the average statistical power reported in meta-analytic conclusions and found substantial evidence that the majority of primary studies in the management literature are statistically underpowered. Finally, we investigated the impact of the file drawer problem in meta-analyses and our findings indicate that the file drawer problem is not a significant concern for meta-analysts. We conclude by discussing various implications of this study for OB/HR researchers.
AguinisH.BeatyJ. C.BoikR. J.PierceC. A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90(1), 94-106.
2.
AguinisH.DaltonD. R.BoscoF. A.PierceC. A.DaltonC. M. (2011). Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly impact. Journal of Management, 37(1), 5-38.
3.
AguinisH.WernerS.AbbottJ. A. L.AngertC.ParkJ. H.KohlhausenD. (2010). Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13, 515-539.
4.
AikenL. S.WestS. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.
5.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
6.
ArgyrisC. (1957). Personality and organization: The conflict between system and the individual. New York, NY: Harper & Row.
7.
AsendorpfJ. B.ConnerM.De FruytF.De HouwerJ.DenissenJ. J. A.FiedlerK.. . . WichertsJ. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119.
8.
BacharachS. B. (1989). Organizational theories: Some criteria for evaluation. Academy of Management Review, 14, 496-515.
9.
BakkerM.van DijkA.WichertsJ. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554.
10.
BassB. M. (1954). The leaderless group discussion. Psychological Bulletin, 51, 465-492.
11.
BlighM. C.KohlesJ. C.PearceC. L.JustinJ. E.StovallJ. F. (2007). When the romance is over: Follower perspectives of aversive leadership. Journal of Applied Psychology, 56, 528-557.
12.
BoscoF. A.AguinisH.SinghK.FieldJ. G.PierceC. A. (2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100, 431-449.
13.
BrewerJ. K. (1972). On the power of statistical tests in the “American Educational Research Journal.”American Educational Research Journal, 9, 391-401.
14.
CampionM. (1993). Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 705-718.
15.
CashenL. H.GeigerS. W. (2004). Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7, 151-167.
16.
ChanM. L. E.ArveyR. (2012). Meta-analysis and the development of knowledge. Perspectives on Psychological Science, 7(1), 79-92.
17.
ChaseL. J.ChaseR. B. (1976). A statistical power analysis of applied psychological research. Journal of Applied Psychology, 61, 234-237.
18.
ChaseL. J.TuckerR. K. (1975). A power-analytic examination of contemporary communication research. Communications Monographs, 42(1), 29-41.
19.
CoeR. (2002). It’s the effect size, stupid: What effect size is and why it is important. Paper presented at the annual conference of the British Educational Research Association, University of Exeter, England.
20.
CohenJ. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153.
21.
CohenJ. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
22.
CohenJ. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
23.
CohenJ. (1992). A power primer. Psychological Bulletin, 112, 155-159.
24.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
25.
DaltonD. R.AguinisH.DaltonC. M.BoscoF. A.PierceC. A. (2012). Revisiting the file drawer problem in meta-analysis: An assessment of published and nonpublished correlation matrices. Personnel Psychology, 65, 221-249.
26.
DaltonD. R.DailyC. M.JohnsonJ. L.EllstrandA. E. (1999). Number of directors and financial performance: A meta-analysis. Academy of Management Journal, 19, 674-686.
27.
EdwardsE. R. (2008). To prosper, organizational psychology should . . . overcome methodological barriers to progress. Journal of Organizational Behavior, 29, 469-491.
28.
FergusonC. J. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40, 532-538.
29.
FergusonC. J.BrannickM. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17(1), 120-128.
30.
FergusonC. J.HeeneM. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7, 555-561.
31.
FergusonT. D.KetchenD. J.Jr. (1999). Organizational configurations and performance: The role of statistical power in extant research. Strategic Management Journal, 20, 385-395.
32.
FisherR. A. (1925). Statistical methods for research workers. London, England: Oliver & Boyd.
33.
FritzA.ScherndlT.KühbergerA. (2012). A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough?Theory & Psychology, 23(1), 98-122.
34.
GlassG. V.HakstianA. R. (1969). Measures of association in comparative experiments: Their development and interpretation. American Educational Research Journal, 6, 403-414.
35.
GlassG. V.McGawB.SmithM. L. (1981). Meta-analysis in social research (Vol. 56). Beverly Hills, CA: Sage.
36.
GreenwaldA. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1-20.
37.
GreenwaldA. G. (1993). Consequences of prejudice against the null hypothesis. In KerenG.LewisC. (Eds.), A handbook for data analysis in the behavioral sciences (pp. 419-448). Hillsdale, NJ: Lawrence Erlbaum.
38.
HallR.de AntuenoC.WebberA. (2007). Publication bias in the medical literature: A review by a Canadian Research Ethics Board. Canadian Journal of Anesthesia, 54, 380-388.
39.
HallS. M.BrannickM. T. (2002). Comparison of two random-effects methods of meta-analysis. Journal of Applied Psychology, 87, 377-389.
40.
HowardG. S.MaxwellS. E.FlemingK. J. (2000). The proof of the pudding: An illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods, 5, 315-332.
41.
HunterJ. E.HirshH. R. (1987). Applications of meta-analysis. In CooperC. L.RobertsonI. T. (Eds.), International review of industrial and organizational psychology (pp. 321-357). New York, NY: Wiley.
42.
HunterJ. E.SchmidtF. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
43.
JenkinsW. O. (1947). A review of leadership studies with particular reference to military problems. Psychological Bulletin, 44(1), 54-79.
44.
JudgeT. A.ThoresenC. J.BonoJ. E.PattonG. K. (2001). The job satisfaction–job performance relationship: A qualitative and quantitative review. Psychological Bulletin, 127, 376-407.
45.
KepesS.BanksG. C.McDanielM.WhetzelD. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624-662.
46.
KirkR. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
47.
KlugerA. N.DeNisiA. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284.
48.
KrollR. M.ChaseL. J. (1975). Communication disorders: A power analytic assessment of recent research. Journal of Communication Disorders, 8, 237-247.
49.
LevineT. R.AsadaK. J.CarpenterC. (2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against nonsignificant findings. Communication Monographs, 76, 286-302.
50.
LipseyM. W. (Ed.). (1998). Design sensitivity: Statistical power for applied experimental research. Thousand Oaks, CA: Sage.
51.
LipseyM. W.WilsonD. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.
52.
MaxwellS. E. (2003). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163.
53.
MazenA. M.GrafL. A.KelloggC. E.HemmasiM. (1987). Statistical power in contemporary management research. Academy of Management Journal, 30, 369-380.
54.
MazenA. M.HemmasiM.LewisM. F. (2006). Assessment of statistical power in contemporary strategy research. Strategic Management Journal, 8, 403-410.
55.
MeindlJ. R.EhrlichS. B.DukerichJ. M. (1985). The romance of leadership. Administrative Science Quarterly, 30(1), 78-102.
56.
MoneM. A.MuellerG. C.MaulandW. (1996). The perceptions and usage of statistical power in applied psychology and management research. Personnel Psychology, 49, 103-120.
57.
PenickJ. E.BrewerJ. K. (1972). The power of statistical tests in science teaching research. Journal of Research in Science Teaching, 9, 377-381.
58.
PetersC. C.Van VoorhisW. R. (1940). Statistical procedures and their mathematical bases. New York, NY: McGraw-Hill.
59.
PrenticeD. A.MillerD. T. (1992). When small effects are impressive. Psychological Bulletin, 112, 160-164.
60.
RichardF. D.BondC. F.Jr.Stokes-ZootaJ. J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331-363.
61.
RosenthalR. (Ed.). (1994). Parametric measures of effect size. New York, NY: Russell Sage Foundation.
62.
RosenthalR.RubinD. B. (1979). A note on percent variance explained as a measure of the importance of effects. Journal of Applied Social Psychology, 9, 395-396.
63.
RosenthalR.RubinD. B. (1982). A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology, 74, 166-169.
64.
RossiJ. S. (1990). Statistical power of psychological research: What have we gained in 20 years?Journal of Consulting and Clinical psychology, 58, 646-656.
65.
RothsteinH. R.BushmanB. J. (2012). Publication bias in psychological science: Comment on Ferguson and Brannick 2012. Psychological Methods, 17, 129-136.
66.
RothsteinH. R.SuttonA. J.BorensteinM. (2005). Publication bias in meta-analysis. West Sussex, England: John Wiley.
67.
RozeboomW. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416-428.
68.
SawyerA. G.BallA. D. (1981). Statistical power and effect size in marketing research. Journal of Marketing Research, 18, 275-290.
69.
ScanduraT.WilliamsE. (2000). Research methodology in management: Current practices, trends, and implications for future research. Academy of Management Journal, 43, 1248-1264.
70.
SchmidtF. L. (2008). Meta-Analysis: A constantly evolving research integration tool. Organizational Research Methods, 11(1), 96-113.
71.
SchulzeR. (2004). Meta-analysis: A comparison of approaches. Cambridge, MA: Hogrefe & Huber.
72.
SedlmeierP.GigerenzerG. (1989). Do studies of statistical power have an effect on the power of studies?Psychological Bulletin, 105, 309-316.
73.
SmithH. L.KruegerL. M. (1933). A brief summary of literature on leadership. Bulletin of Scholastic Education, 9(4).
74.
SongF.Parekh-BhurkeS.HooperL.LokeY.RyderJ.SuttonA.. . . HarveyI. (2009). Extent of publication bias in different categories of research cohorts: A meta-analysis of empirical studies. BMC Medical Research Methodology, 9(1), 79. doi:10.1186/1471-2288-9-79
75.
SteelP. (2013). Converting between meta-analytic variance estimates based on correlations and Fisher z. Paper presented at the 8th Annual Meeting of the Society for Research Synthesis Methodology, Providence, RI.
76.
SteelP. D.Kammeyer-MuellerJ. (2008). Bayesian variance estimation for meta-analysis: Quantifying our uncertainty. Organizational Research Methods, 11(1), 54-78.
77.
SteelP. D.Kammeyer-MuellerJ.NguyenB. (2012). Why meta-analysis exaggerates generalizability and underestimates situational specificity. In MorrisS. (Chair), Methodological advances in meta-analysis. Symposium conducted at the annual meeting of the Society for Industrial Organizational Psychology, San Diego, CA.
78.
SteinerD. D.LaneI. M.DobbinsG. H.SchnurA.McConnellS. (1991). A review of meta-analyses in organizational behavior and human resources management: An empirical assessment. Educational and Psychological Measurement, 51, 609-626.
79.
StemigM. S.SackettP. R. (2013). Another look into the file drawer problem in meta-analysis. Paper presented at the 28th Annual Conference of the Society for Industrial and Organizational Psychology, Houston, TX.
80.
StogdillR. M. (1948). Personal factors associated with leadership: A survey of the literature. Journal of Psychology, 25(1), 35-71.
81.
SuttonA. J. (2009). Publication bias. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435-454). New York, NY: Russell Sage Foundation.
82.
TettR. P.MeyerJ. P.RoeseN. J. (1994). Applications of meta-analysis:1987-1992. In CooperC. L.RobertsonI. T. (Eds.), International review of industrial and organizational psychology (Vol. 9, pp. 71-112). London, England: John Wiley.
83.
WeberR.CamererC.RottenstreichY.KnezM. (2001). The illusion of leadership: Misattribution of cause in coordination games. Organization Science, 12, 582-598.
84.
YatesF. (1951). The influence of statistical methods for research workers on the development of the science of statistics. Journal of the American Statistical Association, 46(253), 19-34.
85.
ZedeckS. (2003). Editorial. Journal of Applied Psychology, 88, 3-5.
