Most statistical/methodological topics seem to have certain components that are poorly understood by many researchers. The purpose of this paper was to discuss some of the problem areas in significance testing, meta-analysis, multiple regression, factor analysis, structural equations modeling, validity, and outliers. These topics were chosen because they have generated a variety of problems for researchers, and because these problems have come to my attention through reviews of journal submissions, discussions with colleagues, and my own reading of the published literature.
Get full access to this article
View all access options for this article.
References
1.
Abelson, R. P.1995. Statistics as principled argument. Hillsdale, NJ: Erlbaum.
2.
American Psychological Association, American Educational Research Associate, & National Council on Measurement in Education. 1985. Standards for Educational and Psychological Testing. Washington, DC: American Psychological Association.
3.
Anderson, J. C., & Gerbing, D. W.1984. The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49: 155-173.
4.
Bentler, P. M., & Bennett, D. G.1980. Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88: 588-606.
5.
Berry, W. D.1993. Understanding regression assumptions. Newbury Park, CA: Sage.
6.
Binning, J. F., & Barrett, G. V.1989. Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74: 478-494.
7.
Bollen, K. A.1987. Outliers and improper solutions: A confirmatory factor analysis example. Sociological Methods & Research, 15: 375-384.
8.
Carver, R. P.1978. The case against statistical significance testing. Harvard Educational Review, 48: 378-399.
9.
Cattell, R. B.1966. The scree test for the number of factors. Multivariate Behavioral Research, 1: 245-276.
10.
Cattell, R. B., & Vogelmann, L.1977. A comprehensive trial of the scree and KG criteria for determining the number of factors. Multivariate Behavioral Research, 12: 289-325.
11.
Chou, C. P., & Bentler, P. M.1995. Estimates and tests in structural equation modeling. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications. Newbury Park, CA: Sage.
12.
Cohen, J.1994. The Earth is round (p < .05). American Psychologist, 49: 997-1003.
13.
Cook, T. D., & Campbell, D. T.1979. Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand-McNally.
14.
Cortina, J. M.1993. What is coefficient alpha? An examination of theory and application. Journal of Applied Psychology, 78: 98-104.
15.
Cortina, J. M.1993. Interaction, nonlinearity, and multicollinearity: Implications for multiple regression. Journal of Management, 19: 915-922.
16.
Cortina, J. M., Chen, G., & Dunlap, W. P.2001. Testing interaction effects in LISREL: A primer for the SEMphobe. Organizational Research Methods, 4: 324-360.
17.
Cortina, J. M., & Deshon, R. P.1998. Determining relative importance of predictors with the observational design. Journal of Applied Psychology, 83: 798-804.
18.
Cortina, J. M., & Dunlap, W. P.1997. On the logic and purpose of significance testing. Psychological Methods, 2: 161-172.
19.
Cortina, J. M., & Folger, R.1998. When is it acceptable to accept the null hypothesis: No way Jose?Organizational Research Methods, 1: 334-350.
20.
Cronbach, L. J., Gleser, Nanda, G. C., & Rajaratnam, N.1972. The dependability of behavioral measurements: Theory of generalizability of scores and profiles. New York: Wiley.
21.
Cortina, J. M., & Gully, S. M.1999. So the great dragon was cast out... who deceives the whole world. Newsletter of the Research Methods Division of the Academy of Management, 14(1).
22.
Cortina, J. M., & Lev-Arey, D. 2000. Order out of CAOS: Establishing benchmark values for moderator detection in meta-analysis. In Paper presented at the 15th Annual conference of the Society for Industrial and Organizational Psychology. New Orleans.
23.
DeShon, R. P.1998. A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3: 412-423.
24.
Dillon, W. R., Kumar, A., & Mulani, N.1987. Offending estimates in covariance structure analysis: Comments on the causes of and solutions to Heywood cases. Psychological Bulletin, 101: 126-135.
25.
Dunlap, W. P., Cortina, J. M., Vaslow, J., & Burke, M.1996. Meta-analysis of experiments with correlated observations. Psychological Methods, 1: 170-177.
26.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J.1999. Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4: 272-299.
27.
Frick, R. W.1995. Accepting the null hypothesis. Memory & Cognition, 23: 132-138.
28.
Glass, G. V., McGaw, B., & Smith, M. L.1981. Meta-analysis in social research. Newbury Park, CA: Sage.
29.
Greenwald, A. G., Pratkanis, A. R., Leippe, M. R., & Baumgardner, M. H.1986. Under what conditions does theory obstruct research progress?Psychological Review, 93: 216-229.
30.
Guttman, L.1954. Some necessary conditions for common factor analysis. Psychometrika, 19: 149-162.
31.
Hedges, L. V., & Olkin, I.1985. Statistical methods for meta-analysis. New York: Academic Press.
32.
Hosmer, D. W., & Lemeshow, S.1989. Applied logistic regression. New York: Wiley.
33.
Hunter, J. E., & Schmidt, F. L.1990. Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park: Sage.
34.
Jaccard, J., & Wan, C. K.1995. Measurement error in the analysis of interaction effect between continuous predictors using multiple regression: Multiple indicator and structural equation approaches. Psychological Bulletin, 117: 348-357.
35.
Joreskog, K. G., & Yang, F., 1996. Nonlinear structural equation models: The Kenny-Judd model with interaction effects. In G. A. Marcoulides & R. E. Schumacker (Eds.). Advanced structural equation modeling techniques. Hillsdale, NJ: Erlbaum.
36.
Kaiser, H. F.1960. The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20: 141-151.
37.
Kenny, D. A.1979. Correlation and causality. New York: Wiley.
38.
Kenny, D. A., & Judd, C. M.1984. Estimating the nonlinear and interactive effects of latent variables. Psychological Bulletin, 96: 201-210.
39.
Lawshe, C. H.1985. Inferences from personnel tests and their validity. Journal of Applied Psychology, 70: 237-238.
40.
Lee, S. Y.1980. Estimation of covariance structure models with parameters subject to functional restraints. Psychometrika, 45: 309-324.
41.
Li, F., Harmer, P., Duncan, T. E., Duncan, S. C., & Boles, S.1998. Approaches to testing interaction effects using structural equation modeling methodology. Multivariate Behavioral Research, 33: 1-39.
42.
Little, R. J. A., & Rubin, D. R.1987. Statistical analysis with missing data. New York: Wiley.
43.
Marascuilo, L. A.1971. Statistical methods for behavioral science research. New York: McGraw-Hill.
44.
Mathieu, J. E., Tannenbaum, S. I., & Salas, E.1992. Influences of individual and situational characteristics on measures of training effectiveness. Academy of Management Journal, 35: 828-847.
45.
McClelland, G. H., & Judd, C. M.1993. Statistical difficulties of detecting interactions and moderator effects. Psychological Bulletin, 114: 376-390.
46.
Medsker, G. J., Williams, L. J., & Holohan, P. J.1994. A review of current practices for evaluating causal models in organizational behavior and human resource management research. Journal of Management, 20: 439-464.
47.
Meehl, P.1967. Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34: 103-115.
48.
Messick, S.1980. Test validity and the ethics of assessment. American Psychologist, 35: 1012-1027.
49.
Mulaik, S. A., & James, L. R.1995. Objectivity and reasoning in science and structural equation modeling. In R. H. Hoyle (Ed.). Structural equation modeling: Concepts, issues, and applications. Newbury Park, CA: Sage.
50.
Mulaik, S., James, L., Van Alstine, J., Bennett, N., Lind, S., & Stilwell, C.1989. An evaluation of goodness of fit indices for structural equation models. Psychological Bulletin, 105: 430-445.
51.
Murphy, K. R., & DeShon, R. P.2000. Inter-rater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53: 873-900.
52.
Oakes, M.1986. Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley.
53.
Ping, R. A.1995. A parsimonious estimating technique for interaction and quadratic latent variables. Journal of Marketing Research, 32: 336-347.
54.
Ping, R. A.1996. Latent variable interaction and quadratic effect estimation: A two-step technique using structural equation analysis. Psychological Bulletin, 199: 166-175.
55.
Rigdon, E. E., Schumacker, R. E., & Wohtke, W.1998. A comparative review of interaction and nonlinear modeling. In R. E. Schumacker & G. A. Marcoulides (Eds.). Interaction and nonlinear effects in structural equation modeling. Mahwah, NJ: Erlbaum.
56.
Rindskopf, D.1984. Structural equation models: Empirical identification, Heywood cases, and related problems. Sociological Methods Research, 13: 109-119.
57.
Rosenthal, R.1991. Meta-analytic procedures for social research. Beverly Hills, CA: Sage.
58.
Rozeboom, W. W.1960. The fallacy of the null hypothesis significance test. Psychological Bulletin, 57: 416-428.
59.
Sackett, P. R., Harris, M. M., & Orr, J. M.1987. On seeking moderator variables in the meta-analysis of correlation data: A Monte Carlo investigation of statistical power and resistance to Type I error. Journal of Applied Psychology, 71: 302-310.
60.
Schmidt, F. L.1996. Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1: 115-129.
61.
Shavelson, R. J., & Webb, N. M.1991. Generalizability theory: A primer. Newbury Park, CA: Sage.
62.
Spector, P. E., & Levine, E. L.1987. Meta-analysis for integrating study outcomes: A Monte Carlo study of its susceptibility to Type I and Type II errors. Journal of Applied Psychology, 72: 3-9.
63.
Whitener, E. M.1990. Confusion of confidence intervals and credibility intervals in Meta-Analysis. Journal of Applied Psychology, 315-321.
64.
Wilcox, R. R.1997. How many discoveries have been lost by ignoring modern statistical methods?American Psychologist, 53: 300-314.
65.
Wothke, W.1993. Nonpositive definite matrices in structural modeling. In K. A. Bollen & J. S. Long (Eds.). Structural equation models. Newbury Park, CA: Sage.
66.
Zwick, W. R., & Velicer, W. F.1982. Factors influencing four rules for determining the number of components to retain. Multivariate Behavioral Research, 17: 253-269.
67.
Zwick, W. R., & Velicer, W. F.1986. Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99: 432-442.
