A primary goal in meta-analysis is determining the variance across a set of correlations after taking into account statistical and psychometric artifacts. If the residual variance is large, substantive moderators of the relationship likely exist; if there is little residual variance, the meta-analytic estimate of the effect size is expected to generalize across multiple settings. Surprisingly little attention has been directed toward some critical shortcomings of traditional methods for estimating residual variance. In this article, the authors argue that residual variance estimates are often based on an unrealistic model of the sampling distribution of residual variance. The authors review alternative Bayesian techniques for estimation that avoid these problems and provide simulation results demonstrating the superiority of the Bayesian approach.
Aguinis, H. (2001). Estimation of sampling variance of correlations in meta-analysis. Personnel Psychology, 54, 569-590.
2.
Alf, E.F., Jr., & Graf, R.G. (2002). A new maximum likelihood estimator of the population squared multiple correlation. Journal of Educational and Behavioral Statistics, 27, 223-235.
3.
Arthur, W., Jr., Bennett, W., Jr., Edens, P.S., & Bell, S.T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88, 234-245.
4.
Barrick, M.R., & Mount, M.K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1-26.
5.
Biggerstaff, B.J., & Tweedie, R.L. (1997). Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Statistics in Medicine, 16, 753-768.
Brannick, M.T. (2001). Implications of empirical Bayes meta-analysis for test validation. Journal of Applied Psychology, 86, 468-480.
8.
Brannick, M.T., & Hall, S.M. (2003, April). Confidence intervals for the random-effects variance component. In M. T. Brannick (Chair), Advances in meta-analysis. Symposium conducted at the meeting of the Society of Industrial Psychology, Orlando, Fla. Available at http://luna.cas.usf.edu/∼mbrannic/files/conf/CI_REVCpaper3.pdf
9.
Brockwell, S.E., & Gordon, I.R. (2001). A comparison of statistical methods for meta-analysis . Statistics in Medicine, 20, 825-840.
10.
Callender, J.C., & Osburn, H.G. (1981). Testing the constancy of validity with computer-generated sampling distributions of the multiplicative model variance estimate: Results for petroleum industry validation research. Journal of Applied Psychology, 66, 274-281.
11.
Cawley, B.D., Keeping, L.M., & Levy, P.E. (1998). Participation in the performance appraisal process and employee reactions: A meta-analytic review of field investigations. Journal of Applied Psychology, 83, 615-633.
12.
Cochran, W.G. (1954). The combination of estimates from different experiments . Biometrics, 10, 101-129.
13.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
14.
Cortina, J.M. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods , 6, 414-439.
15.
Edwards, W. (1998). Hailfinder: Tools for and experiences with Bayesian normative modeling. American Psychologist, 53, 416-428.
16.
Erez, A., Bloom, M.C., & Wells, M.T. (1996). Using random rather than fixed effects models in meta-analysis: Implications for situational specificity and validity generalization . Personnel Psychology, 49, 275-306.
17.
Fisher, R.A. (1912). On an absolute criterion for fitting frequency curves. Messenger of Mathematics, 41, 155-160.
18.
Gavaghan, D.J., Moore, R.A., & McQuay, H.J. (2000). An evaluation of homogeneity tests in meta-analyses in pain using simulations of individual patient data . Pain, 85, 415-424.
19.
Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102, 684-704.
20.
Givens, G.H., Smith, D.D., & Tweedie, R.L. (1997). Publication bias in meta-analysis: A Bayesian data-augmentation approach to account for issues exemplified in the passive smoking debate. Statistical Science, 12, 221-240.
21.
Glass, G.V. (2000). Meta-analysis at 25. Retrieved October 1, 2004, from http://glass.ed.asu.edu/gene/papers/meta25.html
22.
Gross, A.L. (2000). Bayesian interval estimation of multiple correlations with missing data: A Gibbs sampling approach. Multivariate Behavioral Research, 35(2), 201-227.
23.
Hall, S.M., & Brannick, M.T. (2002). Comparison of two random-effects method of meta-analysis. Journal of Applied Psychology, 87, 377-389.
24.
Hardy, R.J., & Thompson, S.G. (1998). Detecting and describing heterogeneity in meta-analysis. Statistics in Medicine, 17, 841-856.
25.
Hayduk, L.A., & Glaser, D.N. (2000). Jiving' the four-step, waltzing around factor analysis, and other serious fun. Structural Equation Modeling , 7, 1-35.
26.
Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
27.
Hedges, L.V., & Pigott, T.D. (2001). The power of statistical tests in meta-analysis . Psychological Methods, 6, 203-217.
28.
Hedges, L.V., & Vevea, J.L. (1998). Fixed- and random-effects models in meta-analysis . Methods, 3, 486-504.
29.
Higgins, J.P.T., & Spiegelhalter, D.J. (2002). Being skeptical about meta-analyses: A Bayesian perspective on magnesium trials in myocardial infarction . International Journal of Epidemiology, 31, 96-104.
30.
Hogan, J., & Holland, B. (2003). Using theory to evaluate personality and job-performance relations: A socio-analytic perspective. Journal of Applied Psychology, 88, 108-112.
31.
Hotelling, H. (1953). New light on the correlation coefficient and its transforms. Journal of the Royal Statistical Society, 15, 193-232.
32.
Hunter, J.E. (1980). Validity generalization for 12,000 jobs: An application of synthetic validity and validity generalization to the General Aptitude Test Battery (GATB). Washington, DC: Department of Labor, U.S. Employment Service.
33.
Hunter, J.E., & Schmidt, F.L. (2004). Methods of meta-analysis: Correcting for error and bias in research findings across studies (2nd ed.). Thousand Oaks, CA: Sage .
34.
Judge, T.A., & Ilies, R. (2002). Relationship of personality to performance motivation: A meta-analytic review. Journal of Applied Psychology, 87, 797-807.
35.
Judge, T.A., Piccolo, R.F., & Ilies, R. (2004). The forgotten ones? The validity of consideration and initiating structure in leadership research. Journal of Applied Psychology, 89, 36-51.
36.
Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness . Cognitive Psychology, 3, 430-454.
37.
Kanfer, R., Wanberg, C., & Kantrowitz, T.M. (2001). Job search and employment: A personality-motivational analysis and meta-analytic review. Journal of Applied Psychology, 86, 837-855.
38.
Kuncel, N.R., Hezlett, S.A., & Ones, D.S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin,127, 162-181.
39.
Kuncel, N.R., Hezlett, S.A., & Ones, D.S. (2004). Academic performance, career potential, creativity, and job performance: Can one construct predict them all?Journal of Personality and Social Psychology, 86, 148-161.
40.
Lee, R.T., & Ashforth, B.E. (1996). A meta-analytic examination of the correlates of the three dimensions of job burnout. Journal of Applied Psychology, 81, 123-133.
41.
Lindley, D.V. (1990). The 1988 Wald Memorial Lectures: The present position in Bayesian statistics. Statistical Science, 5, 44-65.
42.
Louis, T.A., & Zelterman, D. (1994). Bayesian approaches to research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 411-422). New York: Russell Sage.
43.
Lykken, D.T. (1968). Statistical significance in psychological research . Psychological Bulletin,70, 131-139.
44.
Meehl, P.E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103-115.
45.
Mento, A.J., Steel, R.P., & Karren, R.J. (1987). A meta-analytic study of the effects of goal setting on task performance: 1966-1984. Organizational Behavior and Human Decision Processes, 39, 52-83.
46.
Murphy, K.R. (2000). Impacts of assessment of validity generalization and situational specificity on the science and practice of personnel selection . International Journal of Selection and Assessment, 8, 194-206.
47.
Murphy, K.R., & Davidshofer, C.O. (1998). Psychological testing: Principles and application (4th ed.). Upper Saddle River, NJ: Prentice Hall.
48.
Murphy, K.R., & Newman, D.A. (2002). The past, present, and future of validity generalization . In K. R. Murphy (Ed.). Validity generalization: A critical review (pp. 403-424). Mahwah, NJ : Lawrence Erlbaum.
Orlitzky, M., Schmidt, F.L., & Rynes, S.L. (2003). Corporate social and financial performance: A meta-analysis. Organizational Studies, 24, 403-441.
51.
Overton, R.C. (1998). A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects. Psychological Methods, 3, 354-379.
52.
Peretz, C., & Steinberg, D.M. (2001). Improved non-negative estimation of variance components for exposure assessment. Journal of Exposure Analysis and Environmental Epidemiology, 11, 414-421.
53.
Petitti, D.B. (2001). Approaches to heterogeneity in meta-analysis. Statistics in Medicine, 20, 3625-3633.
54.
Raju, N.S., & Drasgow, F. (2003). Maximum likelihood estimation in validity generalization . In K. R. Murphy (Ed.), Validity generalization: A critical review (pp. 263-285). Mahwah, NJ : Lawrence Erlbaum.
55.
Raudenbush, S.W. (1994). Random effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York: Russell Sage.
Ray, W.J., & Ravizza, R. (1988). Methods towards a science of behavior and experience . Belmont, CA: Wadsworth.
58.
Rice, J.A. (1995). Mathematical statistics and data analysis (2nd ed.). Belmont, CA: Duxbury.
59.
Roth, P.L., Huffcutt, A.I., & Bobko, P. (2003). Ethnic group differences in measures of job performance: A new meta-analysis. Journal of Applied Psychology, 88, 694-706.
60.
Sackett, P.R., Harris, M.M., & Orr, J.M. (1986). On seeking moderator variables in the meta-analysis of correlational data: A Monte Carlo investigation of statistical power and resistance to Type I error. Journal of Applied Psychology , 71, 302-310.
61.
Salgado, J.F., & Moscoso, S. (2002). Comprehensive meta-analysis of the construct validity of the employment interview. European Journal of Work and Organizational Psychology, 11, 299-324.
62.
Schisterman, E.F., Moysich, K.B., England, L.J., & Rao, M. (2003). Estimation of the correlation coefficient using the Bayesian approach and its applications for epidemiologic research. BMC Medical Research Methodology, 3, 1-4.
63.
Schmidt, F., & Hunter, J. (2002). Are there benefits from NHST?American Psychologist, 57, 65-66.
64.
Schmidt, F.L., & Hunter, J.E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology , 62, 529-540.
65.
Schmidt, F.L., Hunter, J.E., Pearlman, K., & Hirsh, H.R. (1985). Forty questions about validity generalization and meta-analysis. Personnel Psychology, 38, 697-798.
66.
Shadish, W.R., & Haddock, C.K. (1994). Combining estimates of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 261-281). New York: Russell Sage.
Steel, P., & Kammeyer-Mueller, J. (2002). Comparing meta-analytic moderator search techniques under realistic conditions. Journal of Applied Psychology, 87, 96-111.
69.
Steel, P., & Kammeyer-Mueller, J. (2003). An evaluation of meta-analytic moderator indices. Unpublished manuscript, University of Calgary, Calgary, Alberta, Canada.
70.
Steiger, J.H. (2000). Point estimation, hypothesis testing, and interval estimation using RMSEA: Some comments and a reply to Hayduk and Glaser. Structural Equation Modeling, 7, 149-162.
71.
Stewart, W.H., Jr., & Roth, P.L. (2001). Risk propensity differences between entrepreneurs and managers: A meta-analytic review. Journal of Applied Psychology , 86, 145-153.
72.
Thompson, W.A., Jr. (1962). The problem of negative estimates of variance components. Annals of Mathematical Statistics , 33, 273-289.
73.
Trattner, N.H. (1982). Synthetic validity and its application to the uniform guidelines validation requirements. Personnel Psychology , 35, 383-397.
74.
Tyron, W.W. (2001). Evaluating statistical difference, equivalence, and indeterminancy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386.
75.
Weinberg, S.L., & Goldberg, K.P. (1990). Statistics for the behavioral science. New York: Press Syndicate of the University of Cambridge.
76.
Wright, P.M. (1990). Operationalization of goal difficulty as a moderator of the goal difficulty-performance relationship. Journal of Applied Psychology, 75, 227-234.
