This article encourages transparency in the reporting of meta-analytic procedures. Specifically, we highlight aspects of meta-analytic search, coding, data presentation, and data analysis where published meta-analyses often fall short in presenting sufficient information to allow replication. We identify opportunities where reviewers can request additional information or analyses that will enhance transparent reporting practices and facilitate the evaluation of quality in meta-analytic reporting. We focus on concerns specific to (or prevalent in) meta-analyses conducted in organizational research. In doing so, we reference a number of existing and emerging techniques, highlighting their contribution to meta-analysis while emphasizing key information reviewers may request. Our focus is primarily on meta-analyses, but secondary uses of meta-analytic data are also considered. We conclude by providing a checklist for reviewers in an effort to facilitate the review process as it pertains to the goals of transparency and replicability.
AguinisH.DaltonD. R.BoscoF. A.PierceC. A.DaltonC. M. (2011). Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly Impact. Journal of Management, 37, 5–38.
2.
AloeA. M.BeckerB. J.PigottT. D. (2010). An alternative to R2 for assessing linear models of effect size. Research Synthesis Methods, 1, 272–283.
3.
American Psychological Association. (2008). Reporting standards for research in psychology: Why do we need them? What might they be?American Psychologist, 63, 839–851.
4.
Anzures-CabreraJ.HigginsJ. P. (2010). Graphical displays for meta-analysis: An overview with suggestions for practice. Research Synthesis Methods, 1, 66–80.
5.
AytugZ. G.RothsteinH. R.ZhouW.KernM. C. (2012). Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15, 103–133.
6.
BanksG. C.FieldJ. G.OswaldF. L.O’BoyleE. H.LandisR. S.RuppD. E.RogelbergS. G. (2019). Answers to 18 questions about open science practices. Journal of Business and Psychology, 34, 257–270.
7.
BanksG. C.KepesS.McDanielM. A. (2012). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20, 182–196.
8.
BealD. J.CoreyD. M.DunlapW. P. (2002). On the bias of Huffcutt and Arthur’s (1995) procedure for identifying outliers in the meta-analysis of correlations. Journal of Applied Psychology, 87, 583–589.
9.
BonettD. G. (2008). Meta-analytic interval estimation for Pearson correlations. Psychological Methods, 13, 173–189.
10.
BorensteinM. (2019). Common mistakes in meta-analysis and how to avoid them. Biostat.
11.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta-analysis. Wiley.
12.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1, 97–111.
13.
BorensteinM.HedgesL.HigginsJ.RothsteinH. (2013). Comprehensive Meta-Analysis Version 3. Biostat.
14.
BormanG. D.GriggJ. A. (2009). Visual and narrative interpretation. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 497–519). Russell Sage Foundation.
15.
BrannickM. T.PotterS. M.BenitezB.MorrisS. B. (2019). Bias and precision of alternate estimators in meta-analysis: Benefits of blending Schmidt-Hunter and Hedges approaches. Organizational Research Methods, 22, 490–514.
16.
ByingtonE. K.FelpsW. (2017). Solutions to the credibility crisis in management science. Academy of Management Learning and Education, 16, 142–162.
17.
CafriG.KromreyJ. D.BrannickM. T. (2009). A SAS macro for statistical power calculations in meta-analysis. Behavior Research Methods, 41, 35–46.
18.
CarlsonK. D.JiF. X. (2011). Citing and building on meta-analytic findings: A review and recommendations. Organizational Research Methods, 14, 696–717.
19.
CasperW. J.VaziriH.WayneJ. H.DeHauwS.GreenhausJ. (2018). The jingle-jangle of work-nonwork balance: A comprehensive and meta-analytic review of its meaning and measurement. Journal of Applied Psychology, 103, 182–214.
20.
CheungM. W. L. (2014). Modeling dependent effect sizes with three-level meta-analysis: A structural equation modeling approach. Psychological Methods, 19, 211–229.
CheungS. F. (2000). Examining solutions to two practical issues in meta-analysis: Dependent correlations and missing data in correlation matrices. Unpublished doctoral dissertation, Chinese University of Hong Kong.
23.
ChristensenG.MiguelE. (2018). Transparency, reproducibility, and the credibility of economics research. Journal of Economic Literature, 56, 920–980.
24.
CochranW. G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101–129.
25.
CooperH. M. (2017). Research synthesis and meta-analysis (5th ed.). Sage.
26.
CooperH.HedgesL. V.ValentineJ. C. (2019). The handbook of research synthesis and meta-analysis (3rd ed.). Russell Sage Foundation.
27.
CortinaJ. M. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6, 415–439.
28.
CoyneJ. C.HagedoornM.ThomsB. (2011). Most published and unpublished dissertations should be excluded from meta-analyses: Comment on Meyer et al. Psycho-Oncology, 20, 224–225.
29.
CronbachL. J. (1947). Test “reliability”: It’s meaning and determination. Psychometrika, 12, 1–16.
30.
CudeckR. (1989). Analysis of correlation matrices using covariance structure models. Psychological Bulletin, 105, 317–327.
31.
CyertR. M.MarchJ. G. (1963). A behavioral theory of the firm. Prentice Hall.
32.
DahlkeJ. A.WiernikB. M. (2019). Psychmeta: An R package for psychometric meta-analysis. Applied Psychological Measurement, 43, 415–416.
33.
DeSimoneJ. A. (2014). When it’s incorrect to correct: A brief history and cautionary note. Industrial and Organizational Psychology, 7, 527–531.
34.
DeSimoneJ. A.KöhlerT.SchoenJ. L. (2019). If it were only that easy: The use of meta-analytic research by organizational scholars. Organizational Research Methods, 22, 867–891.
35.
DuvalS.TweedieR. (2000). Trim and Fill: A simple funnel plot based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463.
36.
EysenckH. J. (1978). An exercise in mega-silliness. American Psychologist, 35, 517.
37.
EysenckH. J. (1994). Meta-analysis and its problems. British Medical Journal, 309, 789–792.
GlickW. H. (1985). Conceptualizing and measuring organizational and psychological climate: Pitfalls in multilevel research. Academy of Management Review, 10, 601–616.
40.
Gonzalez-MuléE.AguinisH. (2018). Advancing theory by assessing boundary conditions with metaregression: A critical review and best-practice recommendations. Journal of Management, 44, 2246–2273.
41.
GootyJ.BanksG. C.LoignonA. C.TonidandelS.WilliamsC. E. (2019). Meta-analyses as a multi-level model. Organizational Research Methods. https://doi.org/10.1177/1094428119857471
42.
GreenhouseJ. B.IyengarS. (2009). Sensitivity analysis and diagnostics. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 417–433). Russell Sage Foundation.
43.
HamiltonB. H.NickersonJ. A. (2003). Correcting for endogeneity in strategic management research. Strategic Organization, 1, 51–78.
44.
HarariM. B.ParolaH. R.HartwellC.RiegelmanA. (2020). Literature searches in systematic reviews and meta-analyses: A review, evaluation, and recommendations. Journal of Vocational Behavior. https://doi.org/10.1016/j.jvb.2020.103377
45.
HedgesL. V.OlkinI. (1985). Statistical methods for meta-analysis. Academic Press.
46.
HedgesL. V.PigottT. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9, 426–445.
47.
HedgesL. V.TiptonE.JohnsonM. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.
48.
HeggestadE. D.ScheafD. J.BanksG. C.HausfeldM. M.TonidandelS.WilliamsE. B. (2019). Scale adaptation in organizational science research: A review and best-practice recommendations. Journal of Management, 45, 2596–2627.
49.
HigginsJ.LassersonT.ChandlerJ.ToveyD.ThomasJ.FlemyngE.ChurchillR. (2019). Methodological expectations of Cochrane intervention reviews (MECIR). https://community.cochrane.org/mecir-manual
50.
HigginsJ. P. T.ThomasJ. (2019). Cochrane handbook for systematic reviews of interventions (2nd ed.). Wiley-Blackwell.
51.
HigginsJ. P. T.ThompsonS. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558.
52.
HigginsJ.ThompsonS.DeeksJ.AltmanD. (2002). Statistical heterogeneity in systematic reviews of clinical trials: A critical appraisal of guidelines and practice. Journal of Health Services Research and Policy, 7, 51–61.
IoannidisJ. P. A. (2006). Meta-analysis in public health: Potentials and problems. Italian Journal of Public Health, 3, 9–14.
55.
IoannidisJ. P. A. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Quarterly, 94, 485–514.
56.
JacksonD.TurnerR. (2017). Power analysis for random-effects meta-analysis. Research Synthesis Methods, 8, 290–302.
57.
JohnO. P.SotoC. J. 2007. The importance of being valid: Reliability and the process of construct validation. In RobinsR. W.FraleyR. C.KrugerR. F. (Eds.), Handbook of research methods in personality psychology (pp. 461–494). Guilford.
58.
JohnsonB. T.HennessyE. A. (2019). Systematic reviews and meta-analyses in the health sciences: Best practices and methods for research synthesis. Social Science & Medicine, 223, 237–251.
59.
JüniP.WitschiA.BlochR.EggerM. (1999). The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medical Association, 282, 1054–1060.
60.
KepesS.BanksG. C.McDanielM. A.WhetzelD. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624–662.
61.
KepesS.McDanielM. A.BrannickM. T.BanksG. C. (2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the Meta-analytic Reporting Standards). Journal of Business and Psychology, 28, 123–143.
62.
KöhlerT.CortinaJ. M.KurtessisJ. N.GölzM. (2015). Are we correcting correctly? Interdependence of reliabilities in meta-analysis. Organizational Research Methods, 18, 355–428.
63.
LightR. J.PillemarD. B. (1984). Summing up: The science of reviewing research. Harvard University Press.
MackeyJ. D.FriederR. E.BreesJ. R.MartinkoM. J. (2017). Abusive supervision: A meta-analysis and empirical review. Journal of Management, 43, 1940–1965.
66.
MattG. E. (1989). Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies. Psychological Bulletin, 105, 106–115.
67.
Methods Group of the Campbell Collaboration. (2019). Methodological expectations of Campbell Collaboration intervention reviews: Reporting standards (Campbell Policies and Guidelines Series No. 4). https://doi.org/10.4073/cpg.2016.4
68.
MeyerJ.RowanB. (1977). Institutional organizations: Formal structure as myth and ceremony. American Journal of Sociology, 83, 340–363.
69.
MoherD.JadadA. R.NicholG.PenmanM.TugwellP.WalshS. (1995). Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Controlled Clinical Trials, 16, 62–73.
70.
MoherD.LiberatiA.TetzlaffJ.AltmanD. G.the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151, 264–270.
71.
MurphyK. R. (2017). What inferences can and cannot be made on the basis of meta-analysis?Human Resource Management Review, 27, 193–200.
72.
NosekB. A.AlterG.BanksG. C.BoorsboomD.BowmanS. D.BrecklerS. J.…YarkoniT. (2015). Promoting an open research culture. Science, 348, 1422–1425.
73.
O’BoyleE. H.JrBanksG. C.Gonzales-MuléE. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 376–399.
74.
O’BoyleE. H.RutherfordM. W.BanksG. C. (2014). Publication bias in entrepreneurship research: An examination of dominant relations to performance. Journal of Business Venturing, 29, 773–784.
75.
OhI.-S. (2020). Beyond meta-analysis: Secondary uses of meta-analytic data. Annual Review of Organizational Psychology and Organizational Behavior, 7, 125–153.
76.
OswaldF. L.McCloyR. A. (2003). Meta-analysis and the art of the average. In MurphyK. R. (Ed.), Validity generalization: A critical review (pp. 311–338). Lawrence Erlbaum.
77.
PatersonT. A.HarmsP. D.SteelP.CredéM. (2016). An assessment of the magnitude of effect sizes: Evidence from 30 years of meta-analysis in management. Journal of Leadership & Organizational Studies, 23, 66–81.
78.
PetersJ. L.SuttonA. J.JonesD. R.AbramsK. R.RushtonL. (2008). Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology, 61, 991–996.
79.
RajuN. S.BurkeM. J.NormandJ.LangloisG. M. (1991). A new meta-analytic approach. Journal of Applied Psychology, 76, 432–446.
80.
RosenthalR. (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641.
81.
RosenthalR.RubinD. B. (1982). Comparing effect sizes of independent studies. Psychological Bulletin, 92, 500–504.
82.
RosenthalR.RubinD. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99, 400–406.
83.
RothP. L.SwitzerF. S.IIIvan IddekingeC. H.OhI.-S. (2011). Toward better meta-analytic matrices: How input values can affect research conclusions in human resource management simulations. Personnel Psychology, 64, 899–935.
84.
RothsteinH. R.SuttonA. J.BorensteinM. (2005). Publication bias in meta-analysis. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 1–7). Wiley.
85.
RousseauD. M. (1985). Issues of level in organizational research: Multi-level and cross-level perspectives. Research in Organizational Behavior, 7, 1–37.
86.
RousseauD. M.ManningJ.DenyerD. (2008). Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. Academy of Management Annals, 2, 475–515.
87.
RubinD. B. (1981). Estimation in parallel randomized experiments. Journal of Educational Statistics, 6, 337–400.
88.
RussoM. W. (2007). How to review a meta-analysis. Gastroenterology & Hepatology, 3, 637–642.
89.
SchmidtF. L. (2017). Statistical and measurement pitfalls in the use of meta-regression in meta-analysis. Career Development International, 22, 469–476.
90.
SchmidtF. L.HunterJ. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Sage.
91.
SchmidtF. L.HunterJ. E.PearlmanK. (1982). Progress in validity generalization: Comments on Callender and Osburn and further developments. Journal of Applied Psychology, 67, 835–845.
92.
SchwabA.StarbuckW. H. (2017). A call for openness in research reporting: How to turn covert practices into helpful tools. Academy of Management Learning and Education, 16, 125–141.
93.
ShengZ.KongW.CortinaJ. M.HouS. (2016). Analyzing matrices of meta-analytic correlations: Current practices and recommendations. Research Synthesis Methods, 7, 187–208.
94.
SlavinR. E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational Researcher, 15, 5–11.
95.
SlavinR. E. (1995). Best evidence synthesis: An intelligent alternative to meta-analysis. Journal of Clinical Epidemiology, 48, 9–18.
96.
SteelP. D.Kammeyer-MuellerJ. D. (2002). Comparing meta-analytic moderator estimation techniques under realistic conditions. Journal of Applied Psychology, 87, 96–111.
97.
SterneJ. A. C.BeckerB. J.EggerM. (2005). The funnel plot. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 73–98). Wiley.
98.
StroupD. F.BerlinJ. A.MortonS. C.OlkinI.WilliamsonG. D.RennieD.…ThackerS. B. (2000). Meta-analysis of observational studies in epidemiology: A proposal for reporting. Journal of the American Medical Association, 283, 2008–2012.
99.
SuttonA. J. (2005). Evidence concerning the consequences of publication and related biases. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 175–192). Wiley.
100.
Tanner-SmithE. E.TiptonE. (2014). Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in Stat and SPSS. Research Synthesis Methods, 5, 13–30.
101.
TettR. P.HundleyN. A.ChristiansenN. D. (2017). Meta-analysis and the myth of generalizability. Industrial and Organizational Psychology, 10, 421–456.
102.
ThorndikeR. L. (1949). Personnel selection: Test and measurement techniques. Wiley.
103.
van IddekingeC. H.RothP. L.RaymarkP. H.Odle-DusseauH. N. (2012). The critical role of the research question, inclusion criteria, and transparency in meta-analyses of integrity test research: A reply to Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012). Journal of Applied Psychology, 97, 543–549.
104.
ViechtbauerW. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1–48.
105.
ViechtbauerW.CheungM. W.-L. (2010). Outlier and influence diagnostics for meta-analysis. Research Synthesis Methods, 1, 112–125.
106.
ViswesvaranC.OnesD. S. (1995). Theory testing: Combining psychometric meta-analysis and structural equations modeling. Personnel Psychology, 48, 865–885.
107.
ViswesvaranC.OnesD. S.SchmidtF. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557–574.
108.
WanousJ. P.SullivanS. E.MalinikJ. (1989). The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74, 259–264.
109.
WhitenerE. M. (1990). Confusion of confidence intervals and credibility intervals in meta-analysis. Journal of Applied Psychology, 75, 315–321.
110.
YuJ.DownesP. E.CarterK. M.O’BoyleE. H. (2016). The problem of effect size heterogeneity in meta-analytic structural equation modeling. Journal of Applied Psychology, 101, 1457–1473.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
