Meta-analyses are well known and widely implemented in almost every domain of research in management as well as the social, medical, and behavioral sciences. While this technique is useful for determining validity coefficients (i.e., effect sizes), meta-analyses are predicated on the assumption of independence of primary effect sizes, which might be routinely violated in the organizational sciences. Here, we discuss the implications of violating the independence assumption and demonstrate how meta-analysis could be cast as a multilevel, variance known (Vknown) model to account for such dependency in primary studies’ effect sizes. We illustrate such techniques for meta-analytic data via the HLM 7.0 software as it remains the most widely used multilevel analyses software in management. In so doing, we draw on examples in educational psychology (where such techniques were first developed), organizational sciences, and a Monte Carlo simulation (Appendix). We conclude with a discussion of implications, caveats, and future extensions. Our Appendix details features of a newly developed application that is free (based on R), user-friendly, and provides an alternative to the HLM program.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing (2nd ed.). Washington, DC: American Educational Research Association.
2.
AntonakisJ.BendahanS.JacquartP.LaliveR. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21, 1086–1120.
3.
BanksG. C.KepesS.BanksK. P. (2012). Publication bias: The antagonist of meta-analytic reviews and effective policymaking. Educational Evaluation and Policy Analysis, 34, 259–277.
4.
BeckerB. J. (2000). Multivariate meta-analysis. In TinsleyH. E. A.BrownS. (Eds.), Handbook of applied and multivariate statistics and mathematical modeling (pp. 499–525). San Diego, CA: Academic Press.
5.
BeckerB. J.KimJ. (2002, April). Effects of ignoring nonindependence of effect sizes on Hedge’s homogeneity test. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
6.
BeretevasS. N.PastorD. A. (2003). Using mixed-effects models in reliability generalization studies. Educational and Psychological Measurement, 63, 75–95.
7.
BlieseP. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In KleinK. J.KozlowskiS. W. (Eds.), Multi-level theory, research, and methods in organizations: Foundations, extensions and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass.
8.
BlieseP. D.HangesP. J. (2004). Being both too liberal and too conservative: The perils of treating grouped data as though they were independent. Organizational Research Methods, 7, 400–417.
9.
BorensteinM.HedgesL. V.HigginsJ. P.RothsteinH. R. (2009). Introduction to meta-analysis. West Sussex, UK: Wiley.
10.
BrinerR. B.RousseauD. M. (2011). Evidence-based I-O psychology: Not there yet. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 3–22.
11.
BrykA. S.RaudenbushS. W. (1992). Hierarchical linear models. Newbury Park, CA: Sage.
12.
ChangW.ChengJ.AllaireJ.SievertC.SchloerkeB.XieY.AllenJ.McPhersonJ.DipertA.BorgesB. (2022). _shiny: Web Application Framework for R_. R package version 1.7.2, https://CRAN.R-project.org/package=shiny.
13.
CheungM. W. L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19, 211–229.
14.
CheungS. F.ChanD. K. S. (2004). Dependent effect sizes in meta-analysis: Incorporating the degree of interdependence. Journal of Applied Psychology, 89, 780–791.
15.
CooperH. (1979). Statistically combining independent studies: A meta-analysis of sex differences in conformity research. Journal of Personality and Social Psychology, 37, 131–146.
16.
CooperH.ValentineJ. C.CharltonK.MelsonA. (2003). The effects of modified school calendars on student achievement and on school and community attitudes. Review of Educational Research, 73, 1–52.
17.
DansereauF.AluttoJ. A.YammarinoF. J. (1984). Theory testing in organizational behavior: The variant approach. Englewood Cliffs, NJ: Prentice-Hall.
18.
DavidsonR.MacKinnonJ. G. (1993). Estimation and inference in econometrics. New York, NY: Oxford University Press.
19.
de WitF. R.GreerL. L.JehnK. A. (2012). The paradox of intragroup conflict: A meta-analysis. Journal of Applied Psychology, 97, 360–390.
20.
DeSimoneJ. A.KöhlerT.SchoenJ. L. (2018). If it were only that easy: The use of meta-analytic research by organizational scholars. Organizational Research Methods. Advance online publication. doi:1094428118756743.
21.
DulebohnJ. H.BommerW. H.LidenR. C.BrouerR. L.FerrisG. R. (2012). A meta-analysis of antecedents and consequences of leader-member exchange integrating the past with an eye toward the future. Journal of Management, 38, 1715–1759.
22.
Fernandez-CastillaB.MaesM.DeclercqL.JamshidiL.BeretvasS. N.OnghenaP.Van den NoortgateW. (in press). A demonstration and evaluation of cross-classified random-effects models for meta-analysis. Behavior Research Methods.
23.
GlassG. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.
24.
GlassG. V.SmithM. L. (1979). Meta-analysis of research on class size and achievement. Educational Evaluation and Policy Analysis, 1, 2–16.
Gonzalez-MuléE.AguinisH. (2017). Advancing theory by assessing boundary conditions with metaregression: A critical review and best-practice recommendations. Journal of Management. Advance online publication. doi:10.1177/0149206317710723
27.
GootyJ.YammarinoF. J. (2011). Dyads in organizational research: Conceptual issues and multi-level analyses. Organizational Research Methods, 14, 456–483.
28.
GreenhouseJ. B.IyengarS. (1994). Sensitivity analysis and diagnostics. In CooperH.HedgesL. (Eds.), The handbook of research synthesis (pp. 383–398). New York, NY: Russell Sage.
29.
HedgesL. V.OlkinI. (1985). Statistical methods for meta-analysis. New York, NY: Academic Press.
30.
HigginsJ. P. T.ThompsonS. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558.
31.
HofmannD. A.GriffinM. A.GavinM. B. (2000). The application of hierarchical linear modeling to organizational research. In KleinK. J.KozlowskiS. W. J. (Eds.), Multi-level theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 467–511). San Francisco, CA: Jossey-Bass.
32.
HoxJ. J. (2010). Multi-level analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.
33.
HunterJ. E.SchmidtF. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
34.
HunterJ. E.SchmidtF. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Newbury Park: Sage.
35.
HunterJ. E.SchmidtF. L.LeH. (2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91, 594–612.
36.
JudgeT. A.CableD. M.ColbertA. E.RynesS. L. (2007). What causes a management article to be cited—Article, author, or journal?Academy of Management Journal, 50, 491–506.
37.
KennyD. A.JuddC. M. (1986). Consequences of violating the independence assumption in analysis of variance. Psychological Bulletin, 99, 422–431.
38.
KepesS.BanksG. C.McDanielM.WhetzelD. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624–662.
39.
KepesS.McDanielM. A.BrannickM. T.BanksG. C. (2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the Meta-Analytic Reporting Standards). Journal of Business and Psychology, 28, 123–143.
40.
KonstantopoulosS. (2011). Fixed-effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods, 2, 61–76.
41.
KozlowskiS. W. J.KleinK. J. (2000). A multi-level approach to theory and research in organizations: Contextual, temporal, and emergent processes. In KleinK. J.KozlowskiS. W. J. (Eds.), Multi-level theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 3–90). San Francisco, CA: Jossey-Bass.
42.
LandmanJ. T.DawesR. M. (1982). Psychotherapy outcomes: Smith and Glass’ conclusions stand up to scrutiny. American Psychologist, 37, 504–516.
43.
LeH.OhI.-S.ShafferJ.SchmidtF. L. (2007). Implications of methodological advances for the practice of personnel selection: How practitioners benefit from meta-analysis. Academy of Management Perspectives, 21, 6–15.
44.
LePineJ. A.PiccoloR. F.JacksonC. L.MathieuJ. E.SaulJ. R. (2008). A meta-analysis of teamwork processes: Tests of a multidimensional model and relationships with team effectiveness criteria. Personnel Psychology, 61, 273–307.
45.
Marín-MartínezF.Sánchez-MecaJ. (1999). Averaging dependent effect sizes in meta-analysis: A cautionary note about procedures. The Spanish Journal of Psychology, 2, 32–38.
46.
MartinR.GuillaumeY.ThomasG.LeeA.EpitropakiO. (2016). Leader-member exchange (LMX) and performance: A meta-analytic review. Personnel Psychology, 69, 67–121.
47.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS. N.Van den NoortgateW. (2013). Modeling external events in the three-level analysis of multiple-baseline across-participants designs: A simulation study. Behavior Research Methods, 45, 547–559.
48.
O’BoyleE. H.JrBanksG. C.Gonzalez-MuléE. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 376–399.
RaudenbushS. W.BrykA. S. (1985). Empirical Bayes meta-analysis. Journal of Educational Statistics, 10, 75–98.
51.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
52.
RomanoJ. L.KromreyJ. D. (2009). What are the consequences if the assumption of independent observations is violated in reliability generalization meta-analysis studies?Educational and Psychological Measurement, 69, 404–428.
53.
RosenthalR.RubinD. B. (1978). Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences, 1, 377–386.
54.
RosenthalR.RubinD. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99, 400–406.
55.
SchmidtF. L.HunterJ. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529–540.
56.
SchmidtF. L.HunterJ. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Newbury Park, CA: Sage.
57.
SnijdersT. A. B.BoskerR. J. (2012). Multi-level analysis: An introduction to basic and advanced multi-level modeling (2nd ed.). Los Angeles, CA: Sage.
58.
Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: Author.
59.
Tanner-SmithE. E.TiptonE.PolaninJ. R. (2016). Handling complex meta-analytic data structures using robust variance estimates: A tutorial in R. Journal of Developmental Life Course Criminology, 2, 85–112.
60.
TraczS. M.ElmoreP. B.PohlmannJ. T. (1992). Correlational meta-analysis: Independent and non-independent cases. Educational and Psychological Measurement, 52, 879–888.
61.
Van den BusscheE.Van den NoortgateW.ReynvoetB. (2009). Mechanisms of masked priming: A meta-analysis. Psychological Bulletin, 135, 452–477.
62.
Van den NoortgateW.López-LópezJ. A.Marín-MartínezF.Sánchez-MecaJ. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45, 576–594.
63.
Van den NoortgateW.López-LópezJ. A.Marín-MartínezF.Sánchez-MecaJ. (2015). Meta-analysis of multiple outcomes: A multi-level approach. Behavior Research Methods, 47, 1274–1294.
64.
ViechtbauerW. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
65.
WangS.-Y.ParrilaR.CuiY. (2013). Meta-analysis of social skills interventions of single-case research for individuals with autism spectrum disorders: Results from three-level HLM. Journal of Autism and Developmental Disorders, 43, 1701–1716.
66.
WoehrD. J.LoignonA. C.SchmidtP. B.LoughryM. L.OhlandM. W. (2015). Justifying aggregation with consensus-based constructs: A review and examination of cutoff values for common aggregation indices. Organizational Research Methods, 18, 704–737.
67.
YammarinoF. J. (1998). Multivariate aspects of the varient/WABA approach: A discussion and leadership illustration. The Leadership Quarterly, 9, 203–228.
68.
YammarinoF. J.DansereauF. (2008). Multi-level nature of and multi-level approaches to leadership. The Leadership Quarterly, 19, 135–141.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
