Sweeping recommendations regarding effect size and sample size can miss important nuances: A comment on “A comprehensive review of reporting practices in psychological journals”

Abstract

Statistical significance tests done with low statistical power levels can result in reports of exaggerated effect sizes. Funnel graphs can show these exaggerations for a given area of research by revealing a negative correlation between a study’s sample size on the y-axis and the effect size obtain by the study on the x-axis. A recent paper by Fritz, Scherndl, and Kühberger (2013) recommended that effect sizes should not be reported when there is a negative correlation between sample size and effect size for a given research area. This recommendation fails to consider magnitudes of the negative correlations and thereby misses the opportunity to mitigate effect size exaggerations and approximate more correct effect size estimates. This comment explains both the incorrectness of the recommendation and the approach and calculations necessary to correct effect size estimates.

Keywords

effect sizes eta funnel graphs power publication bias

Get full access to this article

View all access options for this article.

References

Begg

C. B.

(1994). Publication bias. In Cooper

Hedges

L. V.

(Eds.), The handbook of research synthesis (pp. 400–409). New York, NY: Russell Sage Foundation.

Bradley

M. T.

(2000, June). Quantitative review and quality diagnosis of detection of deception studies. Poster presented at the annual meeting of the Canadian Psychological Association, Ottawa, Ontario.

Bradley

M. T.

Best

(2006, June). Analyzing a meta-analysis of sex differences, human audition and laterality. Poster presented at the annual meeting of the Canadian Psychological Association. Calgary, Alberta.

Bradley

M. T.

Best

(2007, June). Revisiting a meta-analysis of visual field laterality and sex differences. Poster presented at the annual meeting of the Canadian Psychological Association. Ottawa, Ontario.

Bradley

M. T.

Cullen

M. C.

Stapleton

J. A.

(2003, June). Analyzing a meta-analysis of sex differ-ences and human brain laterality. Poster presented at the annual meeting of the Canadian Psychological Association. Hamilton, Ontario.

Bradley

M. T.

Stoica

(2004). Diagnosing estimate distortion due to significance testing in literature on the detection of deception. Perceptual and Motor Skills, 98, 827–839. doi: 10.2466/Pms.98.3.827–839.

Brand

Bradley

M. T.

Best

Stoica

(2008). Accuracy of effect size estimates from published psychological research. Perceptual and Motor Skills, 106, 645–649. doi: 10.2466/Pms.106.2.645–649.

Fritz

Scherndl

Kühberger

(2013). A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough? Theory & Psychology, 23, 98–122. doi: 10.1177/0959354312436870.

Light

R. J.

Pillemer

D. B.

(1984). Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.