All effect sizes are sensitive to design flaws and the failure to meet analytic assumptions. But some effect sizes appear to be more robust to assumption violations (e.g., homogeneity of variance). The present study extended prior Monte Carlo research by exploring the robustness of group overlap I indices at the relatively small sample sizes used in some research. I effects are statistically appealing because these indices can be applied across (a) both univariate and multivariate analyses and (b) conditions of either variance homogeneity or variance heterogeneity.
Aaron, B., Kromrey, J. D., & Ferron, J. M. (1998, November). Equating r-based and d-based effect size indices: Problems with a commonly recommended formula. Paper presented at the annual meeting of the Florida Educational Research Association, Orlando, FL. (ERIC Document Reproduction Service No. ED 433 353)
2.
Abelson, R. P. (1997). A retrospective on the significance test ban of 1999 (If there were no significance tests, they would be invented). In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 117-141). Mahwah, NJ: Lawrence Erlbaum.
3.
Alf, E., & Abrahams, N. M. (1968). Relationship between percent overlap and measures of correlation. Educational and Psychological Measurement, 28, 779-792.
4.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
5.
Campbell, T. C. (2005). An introduction to clinical significance: An alternative index of intervention effect for group experimental design. Journal of Early Intervention, 27, 210-227.
6.
Carver, R. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.
7.
Cohen, J. (1977). Statistical power analysis for the behavioral sciences (Rev. ed.). New York: Academic Press.
8.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
9.
Cohen, J. (1994). The Earth is round (p < .05). American Psychologist, 49, 997-1003.
10.
Dunnette, M. D. (1966). Personnel selection and placement. Belmont, CA: Wadsworth.
11.
Efron, B. (1975). The efficiency of logistic regression compared to normal discriminant analysis. Journal of the American Statistical Association, 70, 892-898.
12.
Elster, R. S., & Dunnette, M. D. (1971). The robustness of Tilton’s measure of overlap. Educational and Psychological Measurement, 31, 685-697.
13.
Fan, X., Felsovalyi, A., Sivo, S. & Keenan, S. C. (2003). SAS(R) for Monte Carlo studies: A guide for quantitative researchers. Cary, NC: SAS Publishing.
14.
Fan X., & Wang, L. (1999). Comparing linear discriminant function with logistic regression for the two-group classification problem. Journal of Experimental Education, 67, 265-286.
15.
Fidler, F. (2002). The fifth edition of the APA Publication manual: Why its statistics recommendations are so controversial. Educational and Psychological Measurement, 62, 749-770.
16.
Friedman, H. (1968). Magnitude of experimental effect and a table for its rapid estimation. Psychological Bulletin, 70, 245-251.
17.
Glass, G. V, McGaw, B., & Smith, M. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
18.
Grissom, R. J. (1994). Probability of the superior outcome of one treatment over another. Journal of Applied Psychology, 79, 314-316.
19.
Grissom, R., & Kim, J. J. (2005). Effect sizes for research: A broad practical approach. Mahwah, NJ: Lawrence Erlbaum.
20.
Hess, B. (2001). The efficacy of two improvement-over-chance indices and the standardized mean difference as measures of effect size for two-group univariate comparisons under variance heterogeneity and nonnormality. Dissertation Abstracts International, 61, 3051.
21.
Hess, B., Olejnik, S., & Huberty, C. J. (2001). The efficacy of two Improvement-over-chance effect sizes for two-group univariate comparisons under variance heterogeneity and nonnormality. Educational and Psychological Measurement, 61, 909-936.
22.
Huberty, C. J. (1994). Applied discriminant analysis. New York: John Wiley.
23.
Huberty, C. J. (2002). A history of effect size indices. Educational and Psychological Measurement, 62, 227-240.
24.
Huberty, C. J, & Holmes, S. E. (1983). Two-group comparisons and univariate classification. Educational and Psychological Measurement, 43, 15-26.
25.
Huberty, C. J, & Lowman, L. L. (2000). Group overlap as a basis for effect size. Educational and Psychological Measurement, 60, 543-563.
26.
Huberty, C. J, & Pike, C. J. (1999). On some history regarding statistical testing. In B. Thompson (Ed.), Advances in social science methodology (Vol. 5, pp. 1-22). Greenwich, CT: JAI.
27.
Kieffer, K. M., Reese, R. J., & Thompson, B. (2001). Statistical techniques employed in AERJ and JCP articles from 1988 to 1997: A methodological review. Journal of Experimental Education, 69, 280-309.
28.
Kirk, R. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
29.
Kirk, R. E. (2003). The importance of effect magnitude. In S. F. Davis (Ed.), Handbook of research methods in experimental psychology (pp. 83-105). Oxford, UK: Blackwell.
30.
Kline, R. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
31.
Meehl, P. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting & Clinical Psychology, 4, 806-843.
32.
Rice, J. (1994). Logistic regression: An introduction. In B. Thompson (Ed.), Advances in social science methodology (Vol. 3, pp. 191-245). Greenwich, CT: JAI.
33.
Robinson, D. H., & Wainer, H. (2002). On the past and future of null hypothesis significance testing. Journal of Wildlife Management, 66, 263-271.
34.
Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231-244). New York: Russell Sage Foundation.
35.
Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
36.
Rozeboom, W. W. (1960). The fallacy of the null hypothesis significance test. Psychological Bulletin, 57, 416-428.
37.
Schmidt, F. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.
38.
Snyder, P., & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334-349.
39.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26-30.
40.
Thompson, B. (2001). Significance, effect sizes, stepwise methods, and other issues: Strong arguments move the field. Journal of Experimental Education, 70, 80-93.
41.
Thompson, B. (2002a). “Statistical,” “practical,” and “clinical”: How many kinds of significance do counselors need to consider?Journal of Counseling and Development, 80, 64-71.
42.
Thompson, B. (2002b). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31(3), 25-32.
43.
Thompson, B. (2006a). Foundations of behavioral statistics: An insight-based approach. New York: Guilford.
44.
Thompson, B. (2006b). Research synthesis: Effect sizes. In J. Green, G. Camilli, & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 583-603). Washington, DC: American Educational Research Association.
45.
Tilton, J. W. (1937). The measurement of overlapping. Journal of Educational Psychology, 28, 656-662.
46.
Tukey, J. (1991). The philosophy of multiple comparisons. Statistical Science, 6, 100-116.
47.
Vacha-Haase, T., & Thompson, B. (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473-481.
48.
Wilkinson, L., & American Psychological Association (APA) Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
