Some authors debate whether effect sizes should be reported (a) for all null hypothesis tests, even non–statistically significant ones, or (b) only after a finding is first determined to be statistically significant. The decision to report and interpret small effects may partially depend on the amount of bias in the effect size measure used. Based on the recognitions that variance-accounted-for effect statistics are positively biased and that standardized difference effect sizes such as Cohen’s d can be converted into r2 metrics and vice versa, the authors considered that d also may be biased. The authors therefore explored the amount of bias in Cohen’s d across a series of simulated study conditions. Results from their simulations indicated relatively no bias (close to zero) in Cohen’s d across all study conditions.
Get full access to this article
View all access options for this article.
References
1.
American Psychological Association . (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
2.
American Psychological Association . (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
3.
Bagozzi, R. P. , Fornell, C., & Larcker, D. F. (1981). Canonical correlation analysis as a special case of a structural relations model. Multivariate Behavioral Research, 16, 437-454.
4.
Cahan, S. (2000). Statistical significance is not a “Kosher Certificate” for observed effects: A critical analysis of the two-step approach to the evaluation of empirical results. Educational Researcher, 29(1), 31-36.
5.
Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426-443.
6.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
7.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
8.
Cohen, J. , & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
9.
Cumming, G. , & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-575.
10.
Ezekiel, M. (1930). Methods of correlational analysis. New York: Wiley.
11.
Friedman, H. (1968). Magnitude of experimental effect and a table for its rapid estimation. Psychological Bulletin, 70, 245-251.
12.
Harlow, L. L. , Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests?Mahwah, NJ: Lawrence Erlbaum.
13.
Henson, R. K. (2000). Demystifying parametric analyses: Illustrating canonical correlation as the multivariate general linear model. Multiple Linear Regression Viewpoints, 26(1), 11-19.
14.
Henson, R. K. , & Smith, A. D. (2000). State of the art in statistical significance and effect size reporting: A review of the APA Task Force report and current trends. Journal of Research and Development in Education, 33, 285-296.
15.
Huberty, C. J (1987). On statistical testing. Educational Researcher, 16(8), 4-9.
16.
Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
17.
Knapp, T. R. (1978). Canonical correlation analysis: A general parametric significance testing system. Psychological Bulletin, 85, 410-416.
18.
Knapp, T. R. , & Sawilowsky, S. S. (2001a). Constructive criticisms of methodological and editorial practices. Journal of Experimental Education, 70, 65-79.
19.
Knapp, T. R. , & Sawilowsky, S. S. (2001b). Strong arguments: Rejoinder to Thompson. Journal of Experimental Education, 70, 94-95.
20.
Larson, S. C. (1931). The shrinkage of the coefficient of multiple correlation. Journal of Educational Psychology, 22, 45-55.
21.
Levin, J. R. , & Robinson, D. H. (2000). Rejoinder: Statistical hypothesis testing, effect-size estimation, and the conclusion coherence of primary research studies. Educational Researcher, 29(1), 34-36.
22.
McLean, J. E. , & Ernest, J. M. (1998). The role of statistical significance testing in educational research. Research in the Schools, 5, 15-22.
23.
Robinson, D. H. , & Levin, J. R. (1997). Reflections on statistical and substantive significance. Educational Researcher, 26(5), 21-29.
24.
Rosnow, R. L. , & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
25.
Sawilowsky, S. S. , & Yoon, J. S. (2001, August). The trouble with trivials (p> .05). Paper presented at the 53rd session of the International Statistical Institute, Seoul, South Korea.
26.
Schmidt, F. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 1, 115-129.
27.
Shaver, J. (1985). Chance and nonsense. Phi Delta Kappan, 67, 57-60.
28.
Snyder, P. , & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 361-377.
29.
S-PLUS . (2000). S-PLUS 2000 for Windows. Cambridge, MA: MathSoft.
30.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26-30.
31.
Thompson, B. (2001). Significance, effect sizes, stepwise methods, and other issues: Strong arguments move the field. Journal of Experimental Education, 70, 80-93.
32.
Thompson, B. (2002). “Statistical,”“practical,” and “clinical”: How many kinds of significance do counselors need to consider?Journal of Counseling and Development.
33.
Vacha-Haase, T. , Nilsson, J. E., Reetz, D. R., Lance, T. S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, 413-425.
34.
Wherry, R. J. (1931). A new formula for predicting the shrinkage of the coefficient of multiple correlation. Annals of Mathematical Statistics, 2, 440-457.
35.
Wilkinson, L. , & American Psychological Association Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanation. American Psychologist, 54, 594-604. [Reprint available at http://www.apa.org/journals/amp/amp548594.html]
36.
Yin, P. , & Fan, X. (2001). Estimating R2 shrinkage in multiple regression: A comparison of different analytical methods. Journal of Experimental Education, 69, 203-224.
