Reform of statistical practice in the social and behavioral sciences requires wider use of confidence intervals (CIs), effect size measures, and meta-analysis. The authors discuss four reasons for promoting use of CIs: They (a) are readily interpretable, (b) are linked to familiar statistical significance tests, (c) can encourage meta-analytic thinking, and (d) give information about precision. The authors discuss calculation of CIs for a basic standardized effect size measure, Cohen’s δ (also known as Cohen’s d), and contrast these with the familiar CIs for original score means. CIs for δ require use of noncentral t distributions, which the authors apply also to statistical power and simple meta-analysis of standardized effect sizes. They provide the ESCI graphical software, which runs under Microsoft Excel, to illustrate the discussion. Wider use of CIs for δ and other effect size measures should help promote highly desirable reform of statistical practice in the social sciences.
Get full access to this article
View all access options for this article.
References
1.
Altman, D. (2000). Clinical trials and meta-analyses. In D. Altman, D. Machin, T. Bryant, & M. Gardner (Eds.), Statistics with confidence (2nd ed., pp. 120-138). London: BMJ.
2.
American Psychological Association . (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
3.
Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 423-437.
4.
Chambers, T. , & Lau, J. (1993). Meta-analytic stimulus for changes in clinical trials. Statistical Methods in Medical Research, 2, 161-172.
5.
Chandler, R. (1957). The statistical concepts of confidence and significance. Psychological Bulletin, 54(5), 429-430.
6.
Cohen, J. (1969). Statistical power analysis for the behavioral sciences. New York: Academic Press.
7.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
9.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
10.
Cook, T. , Cooper, H., Cordray, D., Hartmann, H., Hedges, L., Light, R., Louis, T., & Mosteller, F. (1992). Meta-analysis for explanation: A casebook. New York: Russell Sage.
11.
Cowles, M. (1989). Statistics in psychology: An historical perspective. Hillsdale, NJ: Lawrence Erlbaum.
12.
Cox, D. , & Hinkley, D. (1974). Theoretical statistics. London: Chapman & Hall.
13.
Duval, S. , & Tweedie, R. (2000). Trim and fill: A simple funnel plot based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 276-284.
14.
Fidler, F. , & Thompson, B. (2001). Computing correct confidence intervals for ANOVA fixed- and random-effects effect sizes. Educational and Psychological Measurement, 61, 575-604.
15.
Finch, S. , Cumming, G., & Thomason, N. (2001). Reporting of statistical inference in the Journal of Applied Psychology: Little evidence of reform. Educational and Psychological Measurement, 61, 181-210.
16.
Finch, S., Cumming, G., Williams, J., Palmer, L., Griffith, E., Alders, C., Anderson, J., & Goodman, O. (2001). Reform of statistical inference in psychology: The case of Memory and Cognition. Manuscript in preparation.
17.
Finch, S., Thomason, N., & Cumming, G. (2001). Past and future APA guidelines for statistical practice. Manuscript submitted for publication.
18.
Gigerenzer, G. , & Murray, D. (1987). Cognition as intuitive statistics. Hillsdale, NJ: Lawrence Erlbaum.
19.
Glass, G. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
20.
Glass, G. , McGaw, B., & Smith, M. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
21.
Grant, D. (1962). Testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychological Review, 60, 54-61.
22.
Greenland, S. (1987). Quantitative methods in the review of epidemiologic literature. Epidemiologic Reviews, 9, 1-30.
23.
Greenland, S. (1988). On sample-size and power calculations for studies using confidence intervals. American Journal of Epidemiology, 128, 231-237.
24.
Hays, W. (1988). Statistics (4th ed.). New York: Holt, Rinehart & Winston.
25.
Hogben, D. , Pinkham, R., & Wilk, M. (1961). The moments of the non-central t-distribution. Biometrika, 48, 465-468.
26.
Howell, D. (1997). Statistical methods for psychology (4th ed.). Belmont, CA: Duxbury.
27.
Hunter, J. E. (1997). Needed: A ban on the significance test. Psychological Science, 8, 3-7.
28.
Hunter, J. , & Schmidt, F. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
29.
Hyde, J. S. (2001). Reporting effect sizes: The roles of editors, textbook authors, and publication manuals. Educational and Psychological Measurement, 61, 225-228.
30.
Kendall, M. , & Stuart, A. (1961). The advanced theory of statistics (Vol. 2). London: Charles Griffin.
31.
Kirk, R. E. (2001). Promoting good statistical practices: Some suggestions. Educational and Psychological Measurement, 61, 213-218.
32.
LaForge, R. (1967). Confidence intervals or tests of significance in scientific research. Psychological Bulletin, 64, 446-447.
33.
Light, R. , Singer, J., & Willett, J. (1994). The visual presentation and interpretation of meta-analysis. In H. Cooper & L. Hedges (Eds.), The handbook of research synthesis (pp. 439-453). New York: Russell Sage.
34.
Lockhart, R. (1998). Introduction to statistics and data analysis for the behavioral sciences. New York: Freeman.
35.
Loftus, G. (1993). A picture is worth a thousand p values: On the irrelevance of hypothesis testing in the microcomputer age. Behavior Research Methods, Instruments & Computers, 25, 250-256.
36.
Meehl, P. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting & Clinical Psychology, 4, 806-843.
37.
Moore, D. , & McCabe, G. (1993). Introduction to the practice of statistics (2nd ed.). New York: Freeman.
38.
Nickerson, R. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241-301.
39.
Owen, D. (1968). A survey of properties and applications of the noncentral t-distribution. Technometrics, 10, 445-478.
40.
Pearson, E. , & Hartley, H. (1972). Biometrika tables for statisticians (Vol. 2). Cambridge, UK: Cambridge University Press.
41.
Rosenberg, M. , Adams, D., & Gurevitch, J. (2000). MetaWin: Statistical software for Meta- Analysis. Sunderland, MA: Sinauer.
42.
Rosenthal, R. (1991). Meta-analytic procedures for social research (Rev. ed.). Newbury Park, CA: Sage.
43.
Rubin, D. (1990). A new perspective. In K. Wachter & M. Straf (Eds.), The future of metaanalysis (pp. 155-165). New York: Russell Sage.
44.
Schmidt, F. (1992). What do data really mean? Research finding, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.
45.
Schmidt, F. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.
46.
Schmidt, F. , & Hunter, J. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In L. Harlow, S. Muliak, & J. Steiger (Eds.), What if there were no significance tests? (pp. 37-64). Mahwah, NJ: Lawrence Erlbaum.
47.
Smithson, M. (2000). Statistics with confidence. London: Sage.
48.
Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement, 61, 605-632.
49.
Steiger, J. , & Fouladi, T. (1997). Noncentrality interval estimation and the evaluation of statistical models. In L. Harlow, S. Muliak, & J. Steiger (Eds.), What if there were no significance tests? (pp. 221-257). Mahwah, NJ: Lawrence Erlbaum.
50.
Thomason, N. , Cumming, G., & Zangari, M. (1994). Understanding central concepts of statistics and experimental design in the social sciences. In K. Beattie, C. McNaught, & S. Wills (Eds.), Interactive multimedia in university education: Designing for change in teaching & learning (pp. 59-81. Amsterdam, the Netherlands: Elsevier.
51.
Thompson, B. (1998). In praise of brilliance: Where that praise really belongs. American Psychologist, 53, 799-800.
52.
Tversky, A. , & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110.
53.
Wilkinson, L. , & Task Force on Statistical Inference, APA Board of Scientific Affairs. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
