Choice of Software and Programs in Meta-Analysis Research: Does it Make a Difference?

Abstract

Along with the increased popularity of meta-analysis, a number of articles have questioned and highlighted the role that judgment calls play in the implementation and, ultimately, outcomes of studies that use this technique. In the absence of standardized data analyses packages, and coupled with the wide range of options available to meta-analysts, the current study investigated the effect of choice of data analysis programs on meta-analysis study outcomes. The research objective was to investigate the extent to which four commonly used Schmidt and Hunter validity generalization-based meta-analysis software programs, all based on the same conceptual and theoretical assumptions, produced identical outcomes when used to analyze the same data set. The results indicate that, although there are some differences in values obtained from the programs, these differences tended to be very small, typically occurring in the fourth and sometimes fifth decimal place, and did not influence the meta-analytic outcomes.

Get full access to this article

View all access options for this article.

References

Arthur, W., Jr. , Barrett, G. V. , & Alexander, R. A. (1991). Prediction of vehicular accident involvement: A meta-analysis. Human Performance, 4, 89-105.

Barrick, M. R. , & Mount, M. K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1-26.

Callender, J. C. , & Osbum, H. G. (1980). Development and test of a new model for validity generalization. Journal of Applied Psychology, 65, 543-558.

Ghiselli, E. E. , Campbell, J. P. , & Zedeck, S. (1981). Measurement theory for the behavioral sciences. New York: Freeman.

Huffcutt, A. I. , & Arthur, W., Jr. (1994). Hunter and Hunter (1984) revisited: Interview validity for entry level jobs. Journal of Applied Psychology, 79, 184-190.

Huffcutt, A. I. , Arthur, W., Jr. , & Bennett, W. (1993). Conducting meta-analysis using the Proc Means procedure in SAS. Educational and Psychological Measurement, 53, 119-131.

Hunter, J. E. , & Hirsh, H. R. (1987). Applications of meta-analysis. In C. L. Cooper & I. T. Robertson (Eds.), International review of industrial and organizational psychology 1987. London: Wiley.

Hunter, J. E. , & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.

Hunter, J. E. , & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.

10.

Lord, R. G. , DeVader, C. L. , & Alliger, G. M. (1986). A meta-analysis of the relation between personality traits and leadership perceptions: An application of validity generalization procedures. Journal of Applied Psychology, 71, 402-410.

11.

McDaniel, M. A. (1986a). Computer programs for calculating meta-analysis statistics. Educational and Psychological Measurement, 64, 175-177.

12.

McDaniel, M. A. (1986b). MAME: Meta-analysis made easy. Computer program and manual, Ver. 2.1. Bethesda, MD: Author.

13.

McDaniel, M. A. , Schmidt, F. L. , & Hunter, J. E. (1988). Job experience correlates of job performance. Journal of Applied Psychology, 73, 327-330.

14.

Pease, P. W. , & Switzer, F. S. III. (1988). Validity generalization and hypothetical reliability distributions: A test of the Schmidt-Hunter procedure. Journal of Applied Psychology, 73, 267-274.

15.

Rothstein, H. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322-327.

16.

Schmitt, N. , Gooding, R. Z. , Noe, R. A. , & Kirsch, M. (1984). Meta-analyses of validity studies published between 1964 and 1982 and the investigation of study characteristics. Personnel Psychology, 37, 407-422.

17.

Steiner, D. D. , Lane, I. M. , Dobbins, G. H. , Schnur, A. , & McConnell, S. (1991). A review of meta-analyses in organizational and human resources management: An empirical assessment. Educational and Psychological Measurement, 51, 609-626.

18.

Wanous, J. P. , Sullivan, S. H. , & Malinak, J. (1989). The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74, 259-264.

19.

Wiesner, W. H. , & Cronshaw, S. F. (1988). A meta-analysis investigation of the impact of interview format and degree of structure on the validity of the employment interview. Journal of Occupational Psychology, 61, 275-290.