Journals tend to publish only statistically significant evidence, creating a scientific record that markedly overstates the size of effects. We provide a new tool that corrects for this bias without requiring access to nonsignificant results. It capitalizes on the fact that the distribution of significant p values, p-curve, is a function of the true underlying effect. Researchers armed only with sample sizes and test results of the published findings can correct for publication bias. We validate the technique with simulations and by reanalyzing data from the Many-Labs Replication project. We demonstrate that p-curve can arrive at conclusions opposite that of existing tools by reanalyzing the meta-analysis of the “choice overload” literature.
ArkesH. R.BlumerC. (1985). The psychology of sunk cost. Organizational Behavior and Human Decision Processes, 35, 124–140.
2.
AshenfelterO.HarmonC.OosterbeekH. (1999). A review of estimates of the schooling/earnings relationship, with tests for publication bias. Labour Economics, 6, 453–470.
3.
ButtonK. S.IoannidisJ. P.MokryszC.NosekB. A.FlintJ.RobinsonE. S.MunafòM. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.
4.
ChapmanG.JohnsonE. (1999). Anchoring, activation, and the construction of values. Organizational Behavior and Human Decision Processes, 79, 115–153.
5.
ChernevA.BockenholtU.GoodmanJ. (2010). Commentary on Scheibehenne, Greifeneder, and Todd choice overload: Is there anything to it?Journal of Consumer Research, 37, 426–428. doi:10.1086/655200
6.
CohenJ. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.
7.
ColeL. C. (1957). Biological clock in the unicorn. Science, 125, 874–876.
8.
CummingG. (2008). Replication and P intervals: P values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286–300.
9.
DuvalS.TweedieR. (2000a). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89–98.
10.
DuvalS.TweedieR. (2000b). Trim and fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463.
11.
FanelliD. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904.
12.
GelmanA.SternH. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60, 328–331.
13.
GerberA. S.MalhotraN. (2008). Publication bias in empirical sociological research do arbitrary significance levels distort published results?Sociological Methods & Research, 37, 3–30.
14.
HedgesL. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational and Behavioral Statistics, 9, 61–85.
15.
HedgesL. V.VeveaJ. L. (2005). Selection method approaches. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis (pp. 145–174). Chichester, England: Wiley.
16.
HigginsJ. P.ThompsonS. G.DeeksJ. J.AltmanD. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557.
17.
HungH. M. J.O’NeillR. T.BauerP.KohneK. (1997). The behavior of the P-value when the alternative hypothesis is true. Biometrics, 53, 11–22.
18.
IoannidisJ. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640–646.
19.
JohnL.LoewensteinG. F.PrelecD. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532.
20.
KleinR. A.RatliffK.VianelloM.ReginaldB.AdamsJ.BahnikS.. . . NosekB. A. (2014). Investigating variation in replicability: A “Many Labs” Replication Project. Retrieved from Open Science Framework osf.io/wx7ck
21.
LaneD. M.DunlapW. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
22.
OppenheimerD.MeyvisT.DavidenkoN. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.
23.
RosenthalR. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.
24.
RossiJ. S. (1990). Statistical power of psychological research: What have we gained in 20 years?Journal of Consulting and Clinical psychology, 58, 646–656.
25.
RothsteinH. R.SuttonA. J.BorensteinM. (2005). Publication bias in meta-analysis. West Sussex, England: Wiley.
26.
ScheibehenneB.GreifendederR.ToddP. M. (2010). Can there ever be too many options? A meta-analytic review of choice overload. Journal of Consumer Research, 37, 409–425.
27.
SedlmeierP.GigerenzerG. (1989). Do studies of statistical power have an effect on the power of studies?Psychological Bulletin, 105, 309–316.
28.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
29.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2012). A 21 word solution. Dialogue: The Official Newsletter of the Society for Personality and Social Psychology, 26(2), 4–7.
30.
SimonsohnU.NelsonL. D.SimmonsJ. P. (2014). P-curve: A key to the file drawer. Journal of Experimental Psychology: General, 143, 534–547.
31.
SterlingT. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance–or vice versa. Journal of the American Statistical Association, 54, 30–34.
32.
SterlingT. D.RosenbaumW.WeinkamJ. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108–112.
33.
ThalerR. (1980). Toward a positive theory of consumer choice. Journal of Economic Behavior & Organization, 1, 39–60.
34.
TverskyA.KahnemanD. (1981). The framing of decisions and the psychology of choice. Science, 211, 453–458.
35.
WallisW. A. (1942). Compounding probabilities from independent significance tests. Econometrica, 10, 229–248.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
