The misinterpretation and overuse of significance testing in the social sciences has been widely criticized. This criticism is reviewed, along with several recommendations found in the literature, including the use of effect size measures to enhance the interpretation of significance testing. A review of typical effect size measures and their application is followed by an analysis of the extent to which effect size measures have been applied in three prominent journals on learning disabilities over a 10-year period. Specific recommendations are offered for using effect size measures to improve the quality of reporting on quantitative research in the field of learning disabilities.
Get full access to this article
View all access options for this article.
References
1.
Abelson, R.P. (1997). A retrospective on the significance test ban of 1999 (If there were no significance tests, they would be invented). In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 117—141). Mahwah, NJ: Erlbaum.
2.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
3.
Biskin, B. (1998). Comment on significance testing. Measurement and Evaluation in Counseling and Development, 31, 58—62.
4.
Boring, E.G. (1919). Mathematical vs. scientific significance. Psychological Review, 16, 335—338.
5.
Camp, C.J., & Maxwell, S.E. (1983). A comparison of various strengths of association measures commonly used in gerontology research. Journal of Gerontology, 38, 3—7.
6.
Carver, R.P. (1993). The case against statistical significance testing, revisited. Journal of Experimental Education, 61, 287—292.
7.
Cohen, J. (1965). Some statistical issues in psychological research . In B. Wolman (Ed.), Handbook of clinical psychology (pp. 95— 121). New York: McGraw-Hill.
8.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum .
9.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997—1003.
10.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory . Orlando, FL: Harcourt Brace Jovanovich .
11.
Cumming, G., & Finch, S. (2002). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions . Educational and Psychological Measurement, 61, 532—574.
12.
Dar, R., Serlin, R.C., & Omer, H. (1994). Misuse of statistical tests in three decades of psychotherapy research. Journal of Consulting and Clinical Psychology, 62, 75—82.
13.
Fowler, R.L. (1988). Estimating the standardized mean difference in intervention studies. Journal of Educational Statistics, 13, 337—350.
14.
Friedman, H. (1968). Magnitude of experimental effect and a table for its rapid estimation. Psychological Bulletin,70, 245— 251.
15.
Friedman, H. (1969). A simplified table for the estimation of magnitude of experimental effect. Psychonomic Science, 3, 359— 360.
16.
Friedman, H. (1982). Simplified determinations of statistical power, magnitude of effect, and research sample sizes. Educational and Psychological Measurement, 42, 521—527.
17.
Glass, G.V. (1976). Primary, secondary, and meta-analysis of research . Educational Researcher, 5(10), 3—8.
18.
Good, R. (1983). Comments on explained variance. Journal of Research in Science Teaching, 20, 93—94.
19.
Hedges, L.V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics , 6, 107—128.
20.
Hedges, L.V. (1982). Estimation of effect size from a series of independent experiments. Psychological Bulletin,92, 490—499.
21.
Hedges, L.V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational Statistics, 9, 61—85.
22.
Hinkle, D.E., & Oliver, J.D. (1983). How large should the sample be? A question with no simple answers? Or . . . Educational and Psychological Measurement, 43, 1051—1060.
23.
Huberty, C.J. (1994). A note on interpreting an R squared value. Teacher's Corner, 351— 356.
24.
Karabinus, R.A. (1983). Nonequivalent group designs: The use of ANCOVA, multiple regression, repeated ANOVA, and effect size. Evaluation Review, 7, 841— 850.
25.
Keppel, G. (1991). Design and analysis: A researcher's handbook. Englewood Cliffs, NJ: Prentice Hall.
26.
Kirk, R.E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746—759.
27.
Kraemer, H.C. (1985). A strategy to teach the concept and application of power statistical tests. Journal of Educational Statistics , 10, 173—195.
28.
Kromrey, J.D., & Foster-Johnson, L. (1996). Determining the efficacy of interventions: The use of effect sizes for data analysis in single-subject research. Journal of Experimental Education, 65, 73—93.
29.
Morse, D.T. (1998). Minsize: A computer program for obtaining minimum sample size as an indicator of effect size. Educational and Psychological Measurement, 58, 142—153.
30.
Murray, L.W., & Dosser, D.A.J. (1987). How significant is a significant difference? Problems with the measurement of magnitude of effect. Journal of Counseling Psychology, 34, 68—72.
31.
Olejnik, S.F. (1984). Planning educational research: Determining the necessary sample size. Journal of Experimental Education, 53, 40—48.
32.
Ottenbacher, K.J. (1989). Statistical conclusion validity of early intervention research with handicapped children. Exceptional Children, 55, 534—540.
33.
Paquin, M.J. (1983). Beyond significant yet meaningless results in psychotherapy research. Psychotherapy: Theory, Research & Practice, 20, 38—40.
34.
Plucker, J.A. (1997). Debunking the myth of the “highly significant” result: Effect sizes in gifted education research. Roeper Review, 2, 122—126.
35.
Preece, P.F.W. (1983). A measure of experimental effect size based on success rates. Educational and Psychological Measurement, 43, 763—766.
36.
Robinson, D.H., & Levin, J.R. (1997). Reflections on statistical and substantive significance, with a slice of replication. Educational Researcher, 26(5), 21—27.
37.
Rosenthal, R. (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin,86, 638—641.
38.
Rosenthal, R., & Rubin, D.B. (1982). A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology , 74, 166—169.
39.
Rosnow, R.L., & Rosenthal, R. (1988). Focused tests of significance and effect size estimation in counseling psychology. Journal of Counseling Psychology, 35, 203— 208.
40.
Schmidt, F.L., & Hunter, J.E. (1995). The impact of data-analysis methods on cumulative research knowledge: Statistical significance testing, confidence intervals, and meta-analyses. Evaluation and the Health Professions, 18, 408—427.
41.
Seifert, T.L. (1991). Determining effect sizes in various experimental designs. Educational and Psychological Measurement, 51, 341—347.
42.
Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement , 61, 605—632.
43.
Snijders, T.A.B. (1996). What to do with the upward bias in R2: A comment on Huberty. Journal of Educational and Behavioral Statistics, 21, 283—287.
44.
Snyder, P.A., & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334—349.
45.
Strube, M.J. (1988). Some comments on the use of magnitude-of-effect estimates. Journal of Counseling Psychology, 35, 342— 345.
46.
Swanson, H.L., Hoskyn, M., & Lee, C. (1999). Interventions for students with learning disabilities: A meta-analysis of treatment outcomes. New York: Guilford Press.
47.
Thomas, H. (1986). Effect size standard errors for the non-normal non-identically distributed case. Journal of Educational Statistics , 11, 293—303.
48.
Thompson, B. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361—377.
49.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26—30.
50.
Thompson, B. (1997). Editorial policies regarding statistical significance tests: Further comments. Educational Researcher, 26(5), 29—32.
51.
Thompson, B. (2002a). “Statistical,” “practical,” and “clinical”: How many kinds of significance do counselors need to consider?Journal of Counseling and Development, 80, 64—71.
52.
Thompson, B. (2002b). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31(3), 25—32.
53.
Thompson, B., & Snyder, P.A. (1997). Statistical significance testing practices in the Journal of Experimental Education. Journal of Experimental Education, 66, 75— 83.
54.
Thompson, K.N., & Schumacker, R.E. (1997). An evaluation of Rosenthal and Rubin's binomial effect size display. Journal of Educational and Behavioral Statistics, 22, 109—117.
55.
Thurber, S. (1985). Effect size estimates in chemical aversion treatments of alcoholism. Journal of Clinical Psychology, 41, 285—287.
56.
Thye, S.R. (2000). Reliability in experimental psychology. Social Forces, 78, 1277— 1309.
57.
Vacha-Haase, T., & Nilsson, J.E. (1998). Statistical significance reporting: Current trends and uses in MECD. Measurement and Evaluation in Counseling and Development, 31, 46—57.
58.
Vacha-Haase, T., & Thompson, B. (1998). Further comments on statistical significance tests . Measurement and Evaluation in Counseling and Development , 31, 63—67.
59.
VanSickle, R.L. (1983). Statistical power and effect size in social education research. Journal of Social Studies Research, 7, 1—17.
60.
Wampold, B.E., Furlong, M.J., & Atkinson, D.R. (1983). Statistical significance, power, and effect size: A response to the reexamination of reviewer bias. Journal of Counseling Psychology, 30, 459—463.
61.
West, L.J. (1990). Distinguishing between statistical and practical significance. Delta Pi Epsilon Journal, 32, 1—4.
62.
Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594—604.
63.
Woolley, T.W., & Dawson, G.O. (1983). A follow-up power analysis of the statistical tests used in the Journal of Research in Science Teaching. Journal of Research in Science Teaching, 20, 673—681.
64.
Young, M.A. (1993). Supplementing tests of statistical significance: Variation accounted for. Journal of Speech and Hearing Research , 36, 644—656.
65.
Yung, Y.-F. (1996). Comments on Huberty's test of the squared multiple correlation coefficient. Journal of Educational and Behavioral Statistics, 21, 288—298.
