Appraisal of the Homogeneity of Error Variance Assumption and Alternatives to Multiple Regression for Estimating Moderating Effects of Categorical Variables

Abstract

Homogeneity of within-subgroup error variance is a necessary condition for using multiple regression to estimate moderating effects of categorical variables. A 12-year review of Academy of Management Journal, Journal of Applied Psychology, and Personnel Psychology indicates that the assumption is violated in 40% to 60% of cases. The authors reanalyze published research to illustrate how violating the assumption may change substantive conclusions. To remedy this situation, they develop and present a computer program (i.e., ALTMMR) that (a) assesses whether a data set suffers from heterogeneity of error variance and (b) computes alternative inferential statistics to the traditional multiple regression F test when heterogeneity exists. ALTMMR, which can also be used as a teaching tool, was written in Java and is executable using an Internet Web browser or as a standalone application.

Get full access to this article

View all access options for this article.

References

Aguinis, H. (1995). Statistical power problems with moderated multiple regression in management research. Journal of Management, 21, 1141-1158.

Aguinis, H. , & Adams, S.K.R. (1998). Social-role versus structural models of gender and influence use in organizations: A strong inference approach. Group and Organization Management, 23, 414-446.

Aguinis, H. , Boik, R. J. , & Pierce, C. A. (1998, April). Estimating the statistical power of differential prediction analysis. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Dallas, TX.

Aguinis, H. , Bommer, W. H. , & Pierce, C. A. (1996). Improving the estimation of moderating effects by using computer-administered questionnaires. Educational and Psychological Measurement, 56, 1043-1047.

Aguinis, H. , & Pierce, C. A. (1998a). Heterogeneity of error variance and the assessment of moderating effects of categorical variables: A conceptual review. Organizational Research Methods, 1, 296-314.

Aguinis, H. , & Pierce, C. A. (1998b). Statistical power computations for detecting dichotomous moderator variables with moderated multiple regression. Educational and Psychological Measurement, 58, 668-676.

Aguinis, H. , Pierce, C. A. , & Stone-Romero, E. F. (1994). Estimating the power to detect dichotomous moderators with moderated multiple regression. Educational and Psychological Measurement, 54, 690-692.

Aguinis, H. , & Stone-Romero, E. F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192-206.

Alexander, R. A. , & DeShon, R. P. (1994). Effect of error variance heterogeneity on the power of tests for regression slope differences. Psychological Bulletin, 115, 308-314.

10.

Alexander, R. A. , & Govern, D. M. (1994). A new and simpler approximation for ANOVA under variance heterogeneity. Journal of Educational Statistics, 19, 91-101.

11.

American Psychological Association (APA) . (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

12.

American Psychological Association (APA) . (1997). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

13.

Bartlett, M. S. (1937). Properties of sufficiency and statistical tests. Proceedings of the Royal Society, A160, 268-282.

14.

Bobko, P. , & Russell, C. J. (1994). On theory, statistics, and the search for interactions in the organizational sciences. Journal of Management, 20, 193-200.

15.

Bohrnstedt, G. W. , & Marwell, G. (1978). The reliability of products of two random variables. In K. F. Schuessler (Ed.), Sociological methodology (pp. 254-273). San Francisco: Jossey-Bass.

16.

Cohen, J. , & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

17.

DeShon, R. P. , & Alexander, R. A. (1994). A generalization of James’s second-order approximation to the test for regression slope equality. Educational and Psychological Measurement, 54, 328-335.

18.

DeShon, R. P. , & Alexander, R. A. (1996). Alternative procedures for testing regression slope homogeneity when group error variances are unequal. Psychological Methods, 1, 261-277.

19.

Dretzke, B. J. , Levin, J. R. , & Serlin, R. C. (1982). Testing for regression homogeneity under variance heterogeneity. Psychological Bulletin, 91, 376-383.

20.

Easterling, D. V. , & Leventhal, H. (1989). Contribution of concrete cognition to emotion: Neutral symptoms as elicitors of worry about cancer. Journal of Applied Psychology, 74, 787-796.

21.

Games, P. A. , Winkler, H. B. , & Probert, D. A. (1972). Robust tests for homogeneity of variance. Educational and Psychological Measurement, 32, 887-909.

22.

Gartside, P. A. (1972). A study of methods for comparing several variances. Journal of the American Statistical Association, 67, 342-346.

23.

Gibbons, F. X. , Helweg-Larsen, M. , & Gerrard, M. (1995). Prevalence estimates and adolescent risk behavior: Cross-cultural differences in social influence. Journal of Applied Psychology, 80, 107-121.

24.

Gulliksen, H. , & Wilks, S. S. (1950). Regression tests for several samples. Psychometrika, 15, 91-114.

25.

Hattrup, K. , & Schmitt, N. (1990). Prediction of trades apprentices’performance on job sample criteria. Personnel Psychology, 43, 453-466.

26.

Hollenbeck, J. R. , Williams, C. R. , & Klein, H. J. (1989). An empirical examination of the construct validity of a self-report measure of goal commitment. Journal of Applied Psychology, 74, 18-23.

27.

James, G. S. (1951). The comparison of several groups of observations when the ratios of population variances are unknown. Biometrika, 38, 324-329.

28.

Latack, J. C. , Josephs, S. L. , Roach, B. L. , & Levine, M. D. (1987). Carpenter apprentices: Comparison of career transitions for men and women. Journal of Applied Psychology, 72, 393-400.

29.

Mael, F. (1995). Staying afloat: Within group swimming proficiency for Whites & Blacks. Journal of Applied Psychology, 80, 479-490.

30.

Marascuilo, L. A. (1966). Large-sample multiple comparisons. Psychological Bulletin, 65, 280-290.

31.

Melamed, S. , Ben-Avi, I. , Luz, J. , & Green, M. S. (1995). Objective and subjective work monotony: Effects on job satisfaction, psychological distress, and absenteeism in blue collar workers. Journal of Applied Psychology, 80, 29-42.

32.

Pulakos, E. D. , & Schmitt, N. (1995). Experience-based and situational interview questions: Studies of validity. Personnel Psychology, 48, 289-308.

33.

Russell, C. J. , & Bobko, P. (1992). Moderated regression analysis and Likert scales: Too coarse for comfort. Journal of Applied Psychology, 77D, 336-342.

34.

Sackett, P. R. , & Wilk, S. L. (1994). Within-group norming and other forms of source adjustment in preemployment testing. American Psychologist, 49, 929-954.

35.

Society for Industrial and Organizational Psychology . (1987). Principles for the validation and use of personnel selection proceures. (3rd ed.). College Park, MD: Author.

36.

Srinivas, S. , & Motowidlo, S. J. (1987). Effects of raters’stress on the dispersion and favorability of performance ratings. Journal of Applied Psychology, 72, 247-251.

37.

Starbuck, B. , & Mezias, J. (1996). Journal impact ratings. The Industrial-Organizational Psychologist, 33(4), 101-105.

38.

Stewart, G. L. , Carson, K. P. , & Cardy, R. L. (1996). The joint effects of conscientiousness and self-leadership training on employee self-directed behavior in a service setting. Personnel Psychology, 49, 143-164.

39.

Stone, E. F. (1988). Moderator variables in research: A review and analysis of conceptual and methodological issues. In G. R. Ferris & K. M. Rowland (Eds.), Research in personnel and human resources management (Vol. 6, pp. 191-229). Greenwich, CT: JAI.

40.

Stone-Romero, E. F. , Alliger, G. M. , & Aguinis, H. (1994). Type II error problems in the use of moderated multiple regression for the detection of moderating effects for dichotomous variables. Journal of Management, 20, 167-178.

41.

Stone-Romero, E. F. , & Anderson, L. E. (1994). Relative power of moderated multiple regression and the comparison of subgroup correlation coefficients for detecting moderating effects. Journal of Applied Psychology, 79, 354-359.

42.

Tubbs, M. E. (1993). Commitment as a moderator of the goal-performance relation: A case for clearer construct definition. Journal of Applied Psychology, 78, 86-97.

43.

Wilcox, R. R. (1988). A new alternative to the ANOVA F and new results on James’s secondorder method. British Journal of Mathematical and Statistical Psychology, 41, 109-117.