We propose and illustrate a three-step procedure for testing
moderator variable hypotheses meta-analytically. The procedure is
based on Hedges and Olkin's (1985) meta-analytic approach, yet it
incorporates study-level corrections for methodological and statistical
artifacts that are typically advocated and used within psychometric
approaches to meta-analysis (e.g., Hunter & Schmidt, 1990). The three-
step procedure entails: (a) correcting study-level effect size estimates
for across-study variability due to methodological and statistical arti
facts, (b) testing the overall homogeneity of study-level effect size esti
mates after the artifactual sources of variance have been removed, and
(c) testing the effects of hypothesized moderator variables.
Get full access to this article
View all access options for this article.
References
1.
Agumis, H.1995. Statistical power problems with moderated multiple regression in management research. Journal of Management, 21: 1141-1158.
2.
Aguims, H., Bommer, W.H., & Pierce, C.A.1996. Improving the estimation of moderating effects by using computer-administered questionnaires. Educational and Psychological Measurement, 56: 1043-1047.
3.
Aguinis, H., Pierce, C.A.1998. Heterogeneity of error variance and the assessment of moderating effects of categorical variables: A conceptual review. Organizational Research Methods, 1: 296-314.
4.
Aguims, H., Pierce, C.A., & Quigley, B.M.1993. Conditions under which a bogus pipeline procedure enhances the validity of self-reported cigarette smoking: A meta-analytic review. Journal of Applied Social Psychology, 23: 352-373.
5.
Aguinis, H., Pierce, C.A., & Quigley, B.M.1995. Enhancing the validity of self-reported alcohol and marijuana consumption using a bogus pipeline procedure: A meta-analytic review. Basic and Applied Social Psychology, 16: 515-527.
6.
Aguinis, H., Pierce, C.A., & Stone-Romero, E.F.1994. Estimating the power to detect dichotomous moderators with moderated multiple regression. Educational and Psychological Measurement, 54: 690-692.
7.
Aguinis, H., & Stone-Romero, E.F.1997. Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82: 192-206.
8.
Aguinis, H., & Whitehead, R.1997. Sampling variance in the correlation coefficient under indirect range restriction : Implications for validity generalization. Journal of Applied Psychology, 82: 528-538.
9.
Alliger, G.M.1995. The small sample performance of four tests of the difference between pairs of meta-analytically derived effect sizes. Journal of Management, 21: 789-799.
10.
Burke, M.J.1996. An empirical examination of the effect of second-order sampling error on selection test validity coefficients. In M. J. Burke (Chair), Quantitative issues in personnel selection research and practice. Symposium conducted at the meeting of the Society for Industrial and Organizational Psychology, San Diego, CA.
11.
Callender, J.C., & Osburn, H.G.1988. Unbiased estimation of sampling variance of correlations . Journal of Applied Psychology, 73: 312-315.
12.
Cohen, J.1983. The cost of dichotomization. Applied Psychological Measurement, 7: 249-253.
13.
Cooper H. M., & Hedges, L. V. (Eds.) 1994. The handbook of research synthesis. New York: Russell Sage Foundation.
14.
Cooper, H.M., & Lemke, K.M.1991. On the role of meta-analysis in personality and social psychology. Personality and Social Psychology Bulletin, 17: 245-251.
15.
Cotton, J.L., & Tuttle, J.M.1986. Employee turnover: A meta-analysis and review with implications for research. Academy of Management Review, 11: 55-70.
16.
Dobbms, G.H., & Platz, S.J.1986. Sex differences in leadership: How real are they? Academy of Management Review, 11: 118-127.
17.
Glass, G.V., & Stanley, J.C.1970. Statistical methods in education and psychology. Englewood Cliffs, NJ:Prentice-Hall.
18.
Hall, J.A., & Rosenthal, R.1991. Testing for moderator variables in meta-analysis: Issues and methods. Communication Monographs, 58: 437-448.
19.
Hedges, L.V.1982a. Fitting categorical models to effect sizes from a series of experiments. Journal of Educational Statistics, 7: 119-137.
20.
Hedges, L.V.1982b. Fitting continuous models to effect size data. Journal of Educational Statistics, 7: 245-270.
Hunter, J.E., & Schmidt, F.L.1990. Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
23.
Iaffaldano, M.T., & Muchinsky, P.M.1985. Job satisfaction and job performance: A meta-analysis. Psychological Bulletin, 97: 251-273.
24.
James, L.R., Demaree, R.G., & Mulaik, S.A.1986. A note on validity generalization procedures. Journal of Applied Psychology, 71: 440-450.
25.
James, L.R., Demaree, R.G., Mulaik, S.A., & Ladd, R.T.1992. Validity generalization in the context of situational models. Journal of Applied Psychology, 77: 3-14.
26.
Johnson, B.T.1989. DSTAT: Software for the meta-analytic review of research literatures. Hillsdale, NJ: Erlbaum .
27.
Johnson, B.T., & Eagly, A.H. in press. Quantitative synthesis of social psychological research. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social psychology. London: Cambndge University Press.
28.
Johnson, B.T., Mullen, B., & Salas, E.1995. Comparison of three major meta-analytic approaches. Journal of Applied Psychology, 80: 94-106.
29.
Johnson, B.T., & Turco, R.M.1992. The value of goodness-of-fit indices in meta-analysis: A comment on Hall and Rosenthal. Communication Monographs , 59: 388-396.
30.
Muchinsky, P.M.1996. The correction for attenuation. Educational and Psychological Measurement, 56: 63-75.
31.
Mullen, B., Salas, E., & Miller, N.1991. Using meta-analysis to test theoretical hypotheses in social psychology. Personality and Social Psychology Bulletin , 17: 258-264.
32.
Murphy, K.R.1993. The situational specificity of validities: Correcting for statistical artifacts does not always reduce the trans-situational variability of correlation coefficients. International Journal of Selection and Assessment, 1: 158-162.
33.
Oliver, M.B., & Hyde, J.S.1993. Gender differences in sexuality: A meta-analysis. Psychological Bulletin, 114: 29-51.
34.
Pearson, K.1903. Mathematical contributions to the theory of evolution: II. On the influence of natural selection on the variability and correlation of organs. Royal Society Philosophical Transactions, 200 (Series A): 1-66.
35.
Petty, M.M., McGee, G.W., & Cavender, J.W.1984. A meta-analysis of the relationships between individual job satisfaction and individual performance. Academy of Management Review, 9: 712-721.
36.
Raju, N.S., Burke, M.J., Normand, J., & Langlois, G.M.1991. A new meta-analytic approach . Journal of Applied Psychology, 76: 432-446.
37.
Ree, M.J., Carretta, T.R., Earles, J.A., & Albert, W.1994. Sign changes when correcting for range restriction: A note on Pearson's and Lawley's selection formulas. Journal of Applied Psychology, 79: 298-301.
38.
Rosenthal, R.1991. Meta-analytic procedures for social research (rev ed.). Beverly Hills, CA: Sage.
39.
Roth, P.L.1994. Missing data: A conceptual review for applied psychologists . Personnel Psychology, 47: 537-560.
40.
Rothstem, H.R., Schmidt, F.L., Erwm, F.W., Owens, W.A., & Sparks, C.P.1990. Biographical data in employment selection: Can validities be made generalizable? Journal of Applied Psychology, 75: 175-184.
41.
Sagie, A., & Koslowsky, M.1993. Detecting moderators with meta-analysis: An evaluation and comparison of techniques. Personnel Psychology, 46: 629-640.
42.
Schmidt, F.L.1992. What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist , 47: 1173-1181.
43.
Schmidt, F.L., & Hunter, J.E.1996. Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1: 199-223.
44.
Schmidt, F.L., Law, K., Hunter, J.E., Rothstem, H.R., Pearlman, K., & McDamel, M.1993. Refinements in validity generalization methods: Implications for the situational specificity hypothesis. Journal of Applied Psychology, 78: 3-12.
45.
Spearman, C.1904. The proof and measurement of association between two things . American Journal of Psychology, 15: 72-101.
46.
Stone-Romero, E.F., Alliger, G.M., & Aguinis, H.1994. Type II error problems m the use of moderated multiple regression for the detection of moderating effects of dichotomous variables . Journal of Management, 20: 167-178.
47.
Thorndike, R.L.1949. Personnel selection. New York: Wiley.
48.
Wolf, F.M.1986. Meta-analysis: Quantitative methods for research synthesis . Newbury Park, CA: Sage.
49.
Zedeck, S.1971. Problems with the use of "moderator" variables. Psychological Bulletin, 76: 295-310.
