We compared unit, sample size, and inverse variance weighting procedures for estimating the overall mean and random-effects variance component (REVC) in random-effects meta-analysis under realistic conditions for both r and d. Root mean square error and average absolute error of estimation were used to compare weighting schemes via Monte-Carlo simulation. For r, unit weights worked surprisingly well, and sample size weights worked best overall. For d, unit weights worked poorly, and inverse variance weights worked best overall. Discussion focuses on the meta-analyst’s choice of weights, possible explanations for the differences across types of effect size, and implications for meta-analytic inferences in organizational research.
BondR.SmithP. B. (1996). Culture and conformity: A meta-analysis of studies using Asch’s (1952b, 1956) line judgment task. Psychological Bulletin, 119, 111–137.
2.
BonettD. G. (2008). Meta-analytic interval estimation for bivariate correlations. Psychological Methods, 13, 173–181.
3.
BonettD. G. (2009). Meta-analytic interval estimation for standardized and unstandardized mean differences. Psychological Methods, 14, 225–238.
4.
CareyK. B.Scott-SheldonL. A. J.CareyM. P.DeMartiniK. S. (2007). Individual-level interventions to reduce college student drinking: A meta-analytic review. Addictive Behaviors, 32, 2469–2494.
5.
DawesR. M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571–582.
6.
FanX.FelsovalyiA.SivoS.KeenanS. (2002). SAS for Monte Carlo studies: A guide for quantitative researchers [computer software]. Cary, NC: SAS Institute, Inc.
7.
FieldA. P. (2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed-and random-effects methods. Psychological Methods, 6, 161–180.
8.
FieldA. P. (2003). The problems in using fixed-effects models of meta-analysis on real-world data. Understanding Statistics, 2, 105–124.
9.
FieldA. P. (2005). Is the meta-analysis of correlation coefficients accurate when population correlations vary?. Psychological Methods, 10, 444–467.
10.
FleishmanA.I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521–532.
11.
GlassG. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.
12.
HafdahlA. R.WilliamsM. A. (2009). Meta-analysis of correlations revisited: Attempted replication and extension of Field’s (2001) simulation studies. Psychological Methods, 14, 24–42.
13.
HallS. M.BrannickM. T. (2002). Comparison of two random-effects methods of meta-analysis. Journal of Applied Psychology, 87, 377–389.
14.
HedgesL. V. (1983). A random effects model for effect sizes. Psychological Bulletin, 93, 388–395.
15.
HedgesL. V. (1994). Statistical considerations. In CooperH.HedgesL. V. (Eds.), The handbook of research synthesis (pp. 29–38). New York, NY: Russell Sage Foundation.
HedgesL. V.VeveaJ. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486–504.
18.
HermelinE.LievensF.RobertsonI. T. (2007). The validity of assessment centres for the prediction of supervisory performance ratings: A meta-analysis. International Journal of Selection and Assessment, 15, 405–411.
19.
HunterJ. E.SchmidtF. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
20.
HunterJ. E.SchmidtF. L. (2000). Fixed effects vs. random effects meta-analysis models: Implications for cumulative research knowledge. International Journal of Selection and Assessment, 8, 275–292.
21.
HunterJ. E.SchmidtF. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. 2nd ed.Newbury Park, CA: Sage.
22.
KisamoreJ. L.BrannickM. T. (2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11, 35–53.
23.
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
24.
Marin-MartinezF.Sanchez-MecaJ. (2010). Weighting by inverse variance or by sample size in random-effects meta-analysis. Educational and Psychological Measurement, 70, 56–73.
25.
MasonC.AllamR.BrannickM. T. (2007). How to meta-analyze coefficient of stability estimates: Some recommendations based on Monte Carlo studies. Educational and Psychological Measurement, 67, 765–783.
26.
National Research Council. (1992). Combining information: Statistical issues and opportunities for research (pp. 5–15). Washington, DC: National Academy Press.
27.
RosenthalR. (1994). Parametric measures of effect size. In CooperH.HedgesL. V. (Eds.), The handbook of research synthesis (pp. 231–244). New York, NY: Russell Sage Foundation.
28.
SchmidtF. L.HunterJ. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529–540.
29.
SchmidtF. L.HunterJ. E.RajuN. S. (1988). Validity generalization and situational specificity: A second look at the 75% rule and Fisher’s z transformation. Journal of Applied Psychology, 73, 665–672.
30.
SchulzeR. (2004). Meta-analysis: A comparison of approaches. Gottingen, Germany: Hogrefe & Huber.
31.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
32.
SilverN.DunlapW. (1987). Averaging coefficients: Should Fisher’s z transformation be used?. Journal of Applied Psychology, 72, 146–148.
33.
Vacha-HaaseT. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6–20.
