The Precision of Effect Size Estimation From Published Psychological Research

Abstract

Confidence interval (CI) widths were calculated for reported Cohen's d standardized effect sizes and examined in two automated surveys of published psychological literature. The first survey reviewed 1,902 articles from Psychological Science. The second survey reviewed a total of 5,169 articles from across the following four APA journals: Journal of Abnormal Psychology, Journal of Applied Psychology, Journal of Experimental Psychology: Human Perception and Performance, and Developmental Psychology. The median CI width for d was greater than 1 in both surveys. Hence, CI widths were, as Cohen (1994) speculated, embarrassingly large. Additional exploratory analyses revealed that CI widths varied across psychological research areas and that CI widths were not discernably decreasing over time. The theoretical implications of these findings are discussed along with ways of reducing the CI widths and thus improving precision of effect size estimation.

Keywords

Confidence Interval widths effect sizes measurement precision

Get full access to this article

View all access options for this article.

References

Algina

Keselman

H. J.

(2003) Approximate confidence intervals for effect sizes. Educational and Psychological Measurement 63: 537–553.

Altman, D. G., Machin, D., Bryant, T. N., & Gardner, M. J. (Eds.) (2000). Statistics with confidence: confidence intervals and statistical guidelines. (2nd ed.) London, UK: British Medical Journal.

American Psychological Association. (2001) (2001) Publication manual of the American Psychological Association, 5th ed. Washington, DC: Author.

American Psychological Association (2010) (2010) Publication manual of the American Psychological Association, 6th ed. Washington, DC: Author.

Bakker

Wicherts

J. M.

(2011) The (mis)reporting of statistical results in psychology journals. Behavior Research Methods 43: 666–678.

Beal

S. L.

(1989) Sample size determination for confidence intervals on the population mean and on the difference between two population means. Biometrics 45: 969–977.

Birnbaum

M. H.

(2004) Human research and data collection via the Internet. Annual Review of Psychology 55: 803–832.

Bradley

M. T.

Brand

(2013) Sweeping recommendations regarding effect size and sample size can miss important nuances: a comment on: a comprehensive review of reporting practices in psychological journals: are effect sizes really enough? Theory and Psychology 23: 6.

Brand

Bradley

M. T.

(2012) More voodoo correlations: when averaged based measures inflate correlations. The Journal of General Psychology 139: 260–272.

10.

Bradley

M. T.

Stoica

(2004) Diagnosing estimate distortion due to significance testing in literature on detection of deception. Perceptual & Motor Skills 98: 827–839.

11.

Brand

Bradley

M. T.

Best

Stoica

(2008) Accuracy of effect size estimates from published psychological research. Perceptual & Motor Skills 106: 645–649.

12.

Brand

Bradley

M. T.

Best

L. A.

Stoica

(2011) Multiple trials may yield exaggerated effect size estimates. The Journal of General Psychology 138: 1–11.

13.

Cohen

(1977) Statistical power analysis for the behavioural sciences, New York: Academic Press.

14.

Cohen

(1990) Things I have learned (so far). American Psychologist 45: 1304–1312.

15.

Cohen

(1994) The earth is round (p < .05). American Psychologist 49: 997–1003.

16.

Cumming

(2012) Understanding the new statistics: effect sizes, confidence intervals, and meta-analysis, New York: Routledge.

17.

Cumming

Fidler

Leonard

Kalinowski

Christiansen

Kleinig

Wilson

(2007) Statistical reform in psychology: is anything changing? Psychological Science 18: 230–232.

18.

Fiedler

(2011) Voodoo correlations are everywhere—not only in neuroscience. Perspectives on Psychological Science 6: 163–171.

19.

Francis

(2014) The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review 21: 1180–1187.

20.

Hoekstra

Morey

R. D.

Rouder

J. N.

Wagenmakers

E. J.

(2014) Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review 21: 1157–1164.

21.

Kelley

(2007) Methods for the behavioral, educational, and educational sciences: an R package. Behavior Research Methods 39: 979–984.

22.

Kelley

Preacher

K. J.

(2012) On effect size. Psychological Methods 17: 137–152.

23.

Kelley

Rausch

J. R.

(2006) Sample size planning for the standardized mean difference: accuracy in parameter estimation via narrow confidence intervals. Psychological Methods 11: 363–385.

24.

Liu

(2009) Sample size and the width of confidence interval for mean differences. British Journal of Mathematical and Statistical Psychology 62: 201–215.

25.

Liu

(2012) Implications of statistical power for confidence intervals. British Journal of Mathematical and Statistical Psychology 65: 427–437.

26.

Marszalek

J. M.

Barber

Kohlhart

Holmes

C. B.

(2011) Sample size in Psychological research over the past 30 years. Perceptual & Motor Skills 112: 331–348.

27.

Maxwell

S. E.

(2004) The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological Methods 9: 147–163.

28.

Maxwell

S. E.

Kelley

Rausch

J. R.

(2008) Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology 59: 537–563.

29.

Morris

S. B.

(2008) Estimating effect sizes from pretest–posttest–control group designs. Organizational Research Methods 11: 364–386.

30.

Pashler

Wagenmakers

E.-J.

(2012) Editors' introduction to the special section on replicability in psychological science: a crisis of confidence? Perspectives on Psychological Science 7: 528–530.

31.

Plant

R. R.

Turner

(2009) Millisecond precision psychological research in a world of commodity computers: new hardware, new problems? Behavior Research Methods 41: 598–614.

32.

R Development Core Team (2013) (2013) R: A language and environment for statistical computing, Vienna, Austria: R Foundation for Statistical ComputingRetrieved from http://www.r-project.org.

33.

Smithson

(2003) Confidence intervals: quantitative applications in the social sciences series. (No. 140), Thousand Oaks, CA: Sage.

34.

Stallings

W. M.

Gillmore

G. M.

(1971) A note on “accuracy” and “precision”. Journal of Educational Measurement 8: 127–129.

35.

Sterling

T. D.

(1959) Publication decisions and their possible effects on inferences drawn from tests of significance or vice versa. Journal of the American Statistical Association 54: 30–34.

36.

Wicherts

J. M.

Borsboom

Kats

Molenaar

(2006) The poor availability of psychological research data for reanalysis. American Psychologist 61: 726–728.

37.

Wilkinson

APA Task Force on Statistical Inference (1999) (1999) Statistical methods in psychology journals: guidelines and explanations. American Psychologist 54: 594–604.

38.

Yohai

V. J.

(1987) High breakdown-point and high efficiency robust estimates for regression. The Annals of Statistics 15: 642–656.