The efficacy of two improvement-over-chance or I effect sizes, derived from predictive discriminant analysis (PDA) and logistic regression analysis (LRA), was investigated for two-group univariate mean comparisons. Data were generated under selected levels of population separation, variance pattern, sample size, and distribution shape. Based on the accuracy of sample estimates, both I indices are acceptable under optimal conditions except when both population separation and sample size are small. Under variance heterogeneity and normality, I derived from LRA is acceptable if n sizes are equal. When n sizes are unequal, I derived from LRA is acceptable only if variance heterogeneity is moderate and population separation is not small. Under nonnormality, I derived from LRA is acceptable regardless of the variance pattern provided n sizes are equal. Finally, for greater precision, I derived from LRA should be used under large sample sizes.
Get full access to this article
View all access options for this article.
References
1.
Alf, E. , & Abrahams, N. M. (1968). Relationship between percent of overlap and measures of correlation. Educational and Psychological Measurement, 28, 779-792.
2.
American Psychological Association . (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
3.
Bradley, J. V. (1978). Robustness?British Journal of Mathematical and Statistical Psychology, 31, 144-152.
4.
Carroll, R. M. , & Nordholm, L. A. (1975). A sampling characteristic of Kelly’s ε2 and Hays’ω2. Educational and Psychological Measurement, 35, 541-554.
5.
Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.
6.
Carver, R. P. (1993). The case against statistical significance testing, revisited. Journal of Experimental Education, 61, 287-292.
7.
Cohen, J. (1988). Statistical power analysis of the behavioral sciences (2nd ed.). New York: Academic Press.
8.
Cox, D. R. , & Snell, E. J. (1989). The analysis of binary data (2nd ed.). London: Chapman and Hall.
9.
Cumming, G. , & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-575.
10.
Dattalo, P. (1994). A comparison of discriminant analysis and logistic regression. Journal of Social Service Research, 19, 121-144.
11.
Efron, B. (1975). The efficiency of logistic regression compared to normal discriminant analysis. Journal of the American Statistical Association, 70, 892-898.
12.
Elster, R. S. , & Dunnette, M. D. (1971). The robustness of Tilton’s measure of overlap. Educational and Psychological Measurement, 31, 685-697.
13.
Falk, R. , & Greenbaum, C. W. (1995). Significance tests die hard. Theory & Psychology, 5, 75-98.
14.
Fan, X. , & Wang, L. (1999). Comparing linear discriminant function with logistic regression for the two-group classification problem. Journal of Experimental Education, 67, 265-286.
15.
Fleishman, A. (1978). A method for simulating nonnormal distributions. Psychometrika, 43, 521-531.
16.
Fowler, R. L. (1988). Estimating the standardized mean difference in intervention studies. Journal of Educational Statistics, 13, 337-350.
17.
Glass, G. V (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351-379.
18.
Glass, G. V , Peckham, P. D., & Sanders, J. R. (1972). Consequences of failure to meet assumptions underlying the fixed effects analysis of variance and covariance. Review of Educational Research, 42, 237-288.
19.
Greenwald, A. G. , Gonzalez, R., Harris, R. L., & Guthrie, D. (1996). Effect sizes and p values: What should be reported and what should be replicated. Psychophysiology, 33, 175-183.
20.
Harwell, M. R. , Rubenstein, E. N., Hays, W. S., & Olds, C. C. (1992). Summarizing Monte Carlo research in methodological research: The one- and two-factor fixed effects ANOVA cases. Journal of Educational Statistics, 17, 315-339.
21.
Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-128.
22.
Huberty, C. J (1994). Applied discriminant analysis. New York: John Wiley.
23.
Huberty, C. J , & Holmes, S. E. (1983). Two-group comparisons and univariate classification. Educational and Psychological Measurement, 43, 15-26.
24.
Huberty, C. J , & Lowman, L. L. (1998). Discriminant analysis in higher education research. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (pp. 181-234). New York: Agathon.
25.
Huberty, C. J , & Lowman, L. L. (2000). Group overlap as a basis for effect size. Educational and Psychological Measurement, 60, 543-563.
26.
Huberty, C. J , & Pike, C. J. (1999). On some history regarding statistical testing. In B. Thompson (Ed.), Advances in social science methodology (Vol. 5, pp. 1-22). Greenwich, CT: JAI.
27.
Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
28.
Levy, P. (1967). Substantive significance of significant differences between two groups. Psychological Bulletin, 67, 37-40.
29.
Lix, L. M. , & Keselman, H. J. (1998). To trim or not to trim: Test of location equality under heteroscedasticity and nonnormality. Educational and Psychological Measurement, 58, 409-429.
30.
McLachlan, G. J. (1992). Discriminant analysis and statistical pattern recognition. New York: John Wiley.
31.
Meshbane, A. , & Morris, J. D. (1996, April). Predictive discriminant analysis versus logistic regression in two-group classification problems. Paper presented at the annual meeting of the American Educational Research Association, New York.
32.
Neter, J. , Wasserman, W., & Kutner, M. H. (1989). Applied linear regression models (2nd ed.). Boston: Irwin.
33.
Olejnik, S. , & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241-286.
34.
Oshima, T. C. , & Algina, J. (1992). A SAS program for testing the hypothesis of the equal means under heteroscedasticity: James’s second-order test. Educational and Psychological Measurement, 52, 117-118.
35.
Press, S. J. , & Wilson, S. (1978). Choosing between logistic regression and discriminant analysis. Journal of the American Statistical Association, 73, 699-705.
36.
Richardson, J.T.E. (1996). Measures of effect size. Behavior Research Methods, Instruments, & Computers, 28, 12-22.
37.
SAS Institute Inc. (1990). SAS/IML software: Usage and reference, Version 6 (1st ed.). Cary, NC: Author.
38.
SAS Institute Inc. (1997). SAS/STAT software: Changes and enhancements through Release 6.12. Cary, NC: Author.
39.
Sawilowsky, S. S. , & Blair, R. C. (1992). A more realistic look at the robustness and Type II probabilities of the t test to departures of population normality. Psychological Bulletin, 111, 352-360.
40.
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.
41.
Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement, 61, 605-632.
42.
Strube, M. J. (1988). Some comments on the use of magnitude-of-effect estimates. Journal of Counseling Psychology, 35, 342-345.
43.
Thompson, B. (1999a). Statistical significant tests, effect size reporting, and the vain pursuit of pseudo-objectivity. Theory & Psychology, 9, 191-196.
44.
Thompson, B. (1999b). Why “encouraging” effect size reporting is not working: The etiology of researcher resistance to changing practices. Journal of Psychology, 133, 133-140.
45.
Tilton, J. W. (1937). The measurement of overlapping. Journal of Educational Psychology, 28, 656-662.
46.
Wilcox, R. R. (1987). New designs in analysis of variance. Annual Review of Psychology, 38, 29-60.
47.
Wilcox, R. R. (1996). Statistics for the social sciences. San Diego, CA: Academic Press.
48.
Wilkinson, L. , & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
