A Monte-Carlo simulation was used to model the biasing of effect sizes in published studies. The findings from the simulation indicate that, when a predominant bias to publish studies with statistically significant results is coupled with inadequate statistical power, there will be an overestimation of effect sizes. The consequences such an effect size overestimation will then have on meta-analyses and power analyses are highlighted and discussed along with measures which can be taken to reduce the problem.
Get full access to this article
View all access options for this article.
References
1.
AbelsonR. P. (1997) On the surprising longevity of flogged horses: Why is there a case for the significance test. Psychological Science, 8, 12–15.
2.
BirnbaumM. H. (2004) Human research and data collection via the internet. Annual Review of Psychology, 55, 803–832.
3.
BradleyM. T.StoicaG. (2004) Diagnosing estimate distortion due to significance testing in literature on detection of deception. Perceptual and Motor Skills, 98, 827–839.
4.
BrophyA. L. (1985) Approximation of the inverse normal distribution function. Behavior Research Methods, Instruments, & Computers, 17, 415–417.
5.
CohenJ. (1962) The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 69, 145–153.
6.
CohenJ. (1977) Statistical power analysis for the behavioural sciences. New York: Academic Press.
7.
CohenJ. (1990) Things I have learned (so far). American Psychologist, 45, 1304–1312.
8.
CooperH.DeNeveK.CharltonK. (1997) Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2, 447–452.
9.
FidlerF.ThomasonN.CummingG.FinchF.LeemanJ. (2004) Editors can lead researchers to confidence intervals, but can't make them think. Psychological Science, 15, 119–126.
10.
HunterJ. E. (1997) Needed: A ban on the significance test. Psychological Science, 8, 3–7.
11.
KraemerH. C. (1991) To increase power without increasing sample size. Psychopharmacology Bulletin, 27, 217–224.
12.
KraemerH. C.GardnerC.BrooksJ. O.YesavageJ. A. (1998) Advantages of excluding underpowered studies in meta-analysis: Inclusionist versus exclusionist viewpoints. Psychological Methods, 3, 23–31.
13.
LaneD. M.DunlapW. P. (1978) Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
14.
LoftusG. R. (1996) Psychology will be a much better science when we change the way we analyze data. Current Directions in Psychological Science, 5, 161–171.
15.
MaxwellS. E. (2004) The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163.
16.
RosenthalR. (1979) The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.
17.
RossiJ. S. (1990) Statistical power of psychological research: What have we gained in 20 years?Journal of Consulting and Clinical Psychology, 58, 646–656.
18.
SchmidtF. L. (1992) What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173–1181.
19.
SedlmeierP.GigerenzerG. (1989) Do studies of statistical power have an effect on the power of studies?Psychological Bulletin, 105, 309–316.
20.
ShroutP. E. (1997) Should significance tests be banned? Introduction to a special section exploring the pros and cons. Psychological Science, 8, 1–2.
21.
SterlingT. D. (1959) Publication decisions and their possible effects on inferences drawn from tests of significance or vice versa. Journal of the American Statistical Association, 54, 30–34.
22.
SterlingT. D.RosenbaumW. L.WeinkamJ. J. (1995) Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108–112.
23.
WilkinsonL., & APA Task Force on Statistical Inference. (1999) Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.
