This paper evaluates how researchers are currently citing meta-analytic results and provides specific recommendations for interpreting the information provided by meta-analysis (MA). The past four decades have seen a proliferation of MA research across the organizational sciences and myriad improvements to how MA is conducted. MAs are cited more frequently than individual primary studies and have a substantial influence on subsequent research and theorizing. Yet the consumption of meta-analytic results in organizational scholarship remains superficial. We evaluate citation practices for four seminal MAs and find that authors predominantly interpret meta-analytic findings in the simplest way possible: as evidence of the existence of a relationship between variables. In focusing only on this basic finding, citing authors neglect the complexity and rich detail provided by MA. We offer advice for how researchers can more effectively leverage the strengths of meta-analytic findings to inform subsequent research by taking advantage of the benefits that meta-analytic methodology can provide for the explanation of organizational phenomena.
AguinisH.DaltonD. R.BoscoF. A.PierceC. A.DaltonC. M. (2011). Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly Impact. Journal of Management, 37, 5–38.
2.
AguinisH.RamaniR.AlabduljaderN. (2018). What you see is what you get: Enhancing methodological transparency in management research. Academy of Management Annals, 12(1), 1–28.
3.
AguinisH.PierceC. A.BoscoF. A.DaltonD. R.DaltonC. M. (2011). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14, 306–331.
4.
AlabedeJ. O. (2016). Effect of board diversity on corporate governance structure and operating performance: Evidence from the UK listed firms. Asian Journal of Accounting and Governance, 7, 67–80.
5.
American Psychological Association. (2008). Reporting standards for research in psychology: Why do we need them? What might they be?American Psychologist, 63, 839–851.
6.
AytugZ. G.RothsteinH. R.ZhouW.KernM. C. (2012). Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15, 103–133.
7.
BarrickM. R.MountM. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1–26.
8.
BarrickM. R.MountM. K.JudgeT. A. (2001). The FFM personality dimensions and job performance: Meta-analysis of meta-analyses. International Journal of Selection and Assessment, 9, 9–30.
9.
BarronL. G.CarrettaT. R.RoseM. R. (2016). Aptitude and trait predictors of manned and unmanned aircraft pilot job performance. Military Psychology, 28, 65–77.
10.
BlockJ. (1995). A contrarian view of the five-factor approach to personality description. Psychological Bulletin, 117, 187–215.
11.
BobkoP.Stone-RomeroE. F. (1998). Meta-analysis may be another useful research tool, but it is not a panacea. Research in Personnel and Human Resources Management, 16, 359–397.
12.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta-analysis. West Sussex, UK: Wiley.
13.
BowlingN. A.KhazonS.MeyerR. D.BurrusC. J. (2015). Situational strength as a moderator of the relationship between job satisfaction and job performance: A meta-analytic examination. Journal of Business and Psychology, 30, 89–104.
14.
CarlsonK. D.JiF. X. (2011). Citing and building on meta-analytic findings: A review and recommendations. Organizational Research Methods, 14, 696–717.
15.
ChanM. E.ArveyR. D. (2012). Meta-analysis and the development of knowledge. Perspectives on Psychological Science, 7, 79–92.
16.
CohenJ.CohenP.WestS. G.AikenL. S. (2003). Applied multiple correlation/regression analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
17.
ColodroJ.Garces-de-los-FayosE. J.Lopez-GarciaJ. J.Colodro-CondeL. (2016). An explanatory model of underwater adaptation. Anales De Psicologia, 32, 320–331.
18.
CortinaJ. M. (2003). Apples and oranges (and pears, oh my!). The search for moderators in meta-analysis. Organizational Research Methods, 6, 415–439.
19.
CortinaJ. M.DunlapW. P. (1997). On the logic and purpose of significance testing. Psychological Methods, 2, 161–172.
20.
DaltonD. R.DailyC. M.EllstrandA. E.JohnsonJ. L. (1998). Meta-analytic reviews of board composition, leadership structure, and financial performance. Strategic Management Journal, 19, 269–290.
21.
DeSimoneJ. A. (2014). When it’s incorrect to correct: A brief history and cautionary note. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7, 527–531.
22.
DickersinK. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 11–33). West Sussex, UK: Wiley.
23.
DigmanJ. M. (1990). Personality structure: Emergence of the Five-Factor Model. Annual Review of Psychology, 41, 417–440.
24.
GeyskensI.KrishnanR.SteenkampJ. B. E. M.CunhaP. V. (2009). A review and evaluation of meta-analysis practices in management research. Journal of Management, 35, 393–419.
25.
GlassG. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.
26.
GlassG. V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351–379.
27.
HaussmannN. S.McIntyreT.BumbyA. J.LoubserM. J. (2013). Referencing practices in physical geography: How well do we cite what we write?Progress in Physical Geography, 37, 543–549.
28.
HedgesL. V. (1983). A random effects model for effect sizes. Psychological Bulletin, 93, 388–395.
29.
HigginsJ. P. T.ThompsonS. S. (2002). Quantifying heterogeneity in meta-analysis. Statistics in Medicine, 21, 1539–1558.
30.
HunterJ. E.SchmidtF. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
31.
IoannidisJ. P. A. (2006). Meta-analysis in public health: Potentials and problems. Italian Journal of Public Health, 3, 9–14.
32.
JamesL. R.DemareeR. G.MulaikS. A.LaddR. T. (1992). Validity generalization in the context of situational models. Journal of Applied Psychology, 77, 3–14.
33.
JudgeT. A.ThoresenC. J.BonoJ. E.PattonG. K. (2001). The job satisfaction-job performance relationship: A qualitative and quantitative review. Psychological Bulletin, 127, 376–407.
34.
KepesS.BanksG. C.McDanielM.WhetzelD. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624–662.
35.
KingE. B.RogelbergS. G.HeblM. R.BraddyP. W.ShanockL. R.DoererS. C.McDowell-LarsenS. (2016). Waistlines and ratings of executives: Does executive status overcome obesity stigma?Human Resource Management, 55, 283–300.
36.
Kish-GephartJ. J.HarrisonD. A.TreviñoL. K. (2010). Bad apples, bad cases, and bad barrels: Meta-analytic evidence about sources of unethical decisions at work. Journal of Applied Psychology, 95, 1–31.
37.
KöhlerT.CortinaJ. M.KurtessisJ. N.GölzM. (2015). Are we correcting correctly? Interdependence of reliabilities in meta-analysis. Organizational Research Methods, 18, 355–428.
38.
LeBretonJ. M.SchoenJ. L.JamesL. R. (2017). Situational specificity, validity generalization, and the future of psychometric meta-analysis. In FarrJ. L.TippinsN. T. (Eds.), Handbook of employee selection (2nd ed., pp. 93–114). New York, NY: Routledge.
39.
LightR. J.SmithP. V. (1971). Accumulating evidence: Procedures for resolving contradictions among different research studies. Harvard Educational Review, 41, 429–471.
40.
MathewS.IbrahimS.ArchboldS. (2016). Boards attributes that increase firm risk-evidence from the UK. Corporate Governance: The International Journal of Business in Society, 16, 233–258.
41.
MattG. E. (1989). Decision rules for selecting effect sizes in meta-analysis. A review and reanalysis of psychotherapy outcome studies. Psychological Bulletin, 105, 106–115.
42.
MoherD.LiberatiA.TetzlaffJ.AltmanD. G., & the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement (Reprinted from Annals of Internal Medicine). Physical Therapy, 89, 873–880.
43.
MoherD.TetzlaffJ.TriccoA. C.SampsonM.AltmanD. G. (2007). Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine, 4(3), e78.
44.
MurphyK. R. (2017). What inferences can and cannot be made on the basis of meta-analysis?Human Resource Management Review, 27, 193–200.
45.
NickersonR. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.
46.
PlattS. (1989). Respectfully quoted: A dictionary of quotations requested from the Congressional Research Service. Washington, DC: Library of Congress.
47.
PotoskyD. (2016). A framework and typology of adjustment responses to extra-cultural disorientation experienced during intercultural assignments. Human Resource Management Review, 26, 227–241.
48.
RosenthalM.SuttonJ.AustinZ.TsuyukiR. T. (2015). Relationship between personality traits and pharmacist performance in a pharmacy practice research trial. Canadian Pharmacists Journal, 148, 209–216.
49.
RothsteinH. R.SuttonA. J.BorensteinM. (2005). Publication bias in meta-analysis. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 1–7). West Sussex, UK: Wiley.
50.
SanchezL. P. C.Guerrero-VillegasJ.GonzalezJ. M. H. (2017). The influence of organizational factors on board roles. Management Decision, 55, 842–871.
51.
SchmidtF. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173–1181.
52.
SchmidtF. (2008). Meta-analysis: A constantly evolving research integration tool. Organizational Research Methods, 11, 96–113.
53.
SchmidtF. L.HunterJ. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199–223.
54.
SchmidtF. L.HunterJ. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.
55.
SchmidtF. L.HunterJ. E.UrryV. W. (1976). Statistical power in criterion-related validation studies. Journal of Applied Psychology, 61, 473–485.
56.
SlavinR. E. (1995). Best evidence synthesis: An intelligent alternative to meta-analysis. Journal of Clinical Epidemiology, 48, 9–18.
57.
SpearmanC. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72–101.
58.
SuttonA. J. (2005). Evidence concerning the consequences of publication and related biases. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 175–192). West Sussex, UK: Wiley.
59.
TettR. P.BurnettD. D. (2003). A personality trait-based interactionist model of job performance. Journal of Applied Psychology, 88, 500–517.
60.
ThorndikeR. L. (1949). Personnel selection: Test and measurement techniques. New York, NY: Wiley.
61.
ToddP. A.YeoD. C. J.LiD.LadleR. J. (2007). Citing practices in ecology: Can we believe our own words?Oikos, 116, 1599–1601.
62.
Van IddekingeC. H.RothP. L.RaymarkP. H.Odle-DusseauH. N. (2012). The criterion-related validity of integrity tests: An updated meta-analysis. Journal of Applied Psychology, 97, 499–530.
63.
WanousJ. P.SullivanS. E.MalinikJ. (1989). The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74, 259–264.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
