A novel meta-regression method, precision-effect test and precision-effect estimate with standard errors (PET-PEESE), predicts and explains recent high-profile failures to replicate in psychology. The central purpose of this article is to identify the limitations of PET-PEESE for application to social/personality psychology. Using typical conditions found in social/personality research, our simulations identify three areas of concern. PET-PEESE performs poorly in research areas where there are only a few studies, all studies use small samples, and where there is very high heterogeneity of results from study to study. Nonetheless, the statistical properties of conventional meta-analysis approaches are much worse than PET-PEESE under these same conditions. Our simulations suggest alterations to conventional research practice and ways to moderate PET-PEESE weaknesses.
AuJ.SheehanE.TsaiN.DuncanG. J.BuschkuehlM.JaeggiS. M. (2015). Improving fluid intelligence with training on working memory: A meta-analysis. Psychological Bulletin Review, 22, 366–377.
2.
BeggC. B.BerlinJ. A. (1988). Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society. Series A, 151, 419–463.
3.
BoggsT.LaseckiL. (2015). Reliable gains? Evidence for substantially underpowered designs in studies of working memory training transfer to fluid intelligence. Frontiers in Psychology. doi:10.3389/fpsyg.2014.01589
4.
BrunsS. B.IoannidisJ. P. A. (2016). p-Curve and p-hacking in observational research. PLoS One, 11, e0149144. doi:10.1371/journal. pone.0149144
5.
CarterE. C.KoflerL. M.ForsterD. E.McCulloughM. E. (2015). A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General, 144, 796–815.
6.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, MI: Lawrence Erlbaum.
EggerM.SmithG. D.SchneiderM.MinderC. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.
9.
FraleyR. C.VazireS. (2014). The n-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS One, 9, e109019. doi:10.1371/journal. pone.0109019
10.
GlassG. V.McGawB.SmithM. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
11.
GreeneW. E. (1990). Econometric analysis. New York, NY: Macmillan.
12.
HaggerM. S.ChatzisarantisN. L. D. (2016). A multi-lab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11, 546–573.
HigginsJ. P. T.ThompsonS. G. (2002). Quantifying heterogeneity in meta-analysis. Statistics in Medicine, 21, 1539–1558.
15.
IoannidisJ. P. A.StanleyT. D.DoucouliagosC. (H). (in press). The power of bias in economics research. The Economic Journal. Also as SWP, Economics Series 2016/2, Deakin University, Australia. Retrieved July 21, 2016, fromhttp://www.deakin.edu.au/__data/assets/pdf_file/0007/477763/2016_1.pdf.
16.
JohnsonN.KotzS. (1970). Distributions in statistics: Continuous univariate distribution. New York, NY: Wiley.
17.
MaxwellS. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163.
18.
McShaneB. B.BöckenholtU.HansenK. T. (2016). Adjusting for publication bias in meta-analysis: An evaluation of selection methods and some cautionary notes. Perspectives on Psychological Science, 11, 730–749.
19.
MorenoS. G.SuttonA. J.AdesA. E.StanleyT. D.AbramsK. R.PetersJ. L.CooperN. J. (2009). Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 9, 2. Retrieved November 23, 2016, fromhttp://www.biomedcentral.com/1471-2288/9/2
20.
MorenoS. G.SuttonA. J.ThompsonJ. R.AdesA. E.AbramsK. R.CooperN. J. (2012). A generalized weighting regression derived meta-analysis estimator robust to small-study effects and heterogeneity. Statistics in Medicine, 31, 1407–1417.
21.
MorenoS. G.SuttonA. J.TurnerE. H.AbramsK. R.CooperN. J.PalmerT. M.AdesA. E. (2009). Novel methods to deal with publication biases: Secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. British Medical Journal, 339, b298. Retrieved November 23, 2016, fromhttp://dx.doi.org/10.1136/bmj.b2981
22.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716–aac4716. doi:10.1126/science.aac4716
23.
PopperK. (1959). The logic of scientific discovery. New York, NY: Basic Books.
24.
RichardF. D.BondC. F. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331–363.
25.
RosenthalR. (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641.
26.
SchmidtF. L.HunterJ. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.
27.
SimonsohnU.NelsonL. D.SimmonsJ. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–547.
28.
StanleyT. D. (2008). Meta-regression methods for detecting and estimating empirical effect in the presence of publication selection. Oxford Bulletin of Economics and Statistics, 70, 103–127.
29.
StanleyT. D.DoucouliagosH. (2012). Meta-regression analysis in economics and business. Oxford, England: Routledge.
30.
StanleyT. D.DoucouliagosC. H. (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 5, 60–78.
31.
StanleyT. D.DoucouliagosC. H. (2015). Neither fixed nor random: Weighted least squares meta-analysis. Statistics in Medicine, 34, 2116–2127.
32.
StanleyT. D.DoucouliagosC. H. (2016). Neither fixed nor random: Weighted least squares meta-regression. Research Synthesis Methods. doi:10.1002/jrsm.1211
33.
StanleyT. D.& DoucouliagosC. (H)IoannidisJ. P. A. (2017). Finding the power to reduce publication bias. Statistics in Medicine. Advance online publication. doi:10.1002/sim.7228
34.
StanleyT. D.JarrellS. B.DoucouliagosH. (2010). Could it be better to discard 90% of the data? A statistical paradox. The American Statistician, 64, 70–77.
35.
SterlingT. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vice versa. Journal of the American Statistical Association, 54, 30–34.
36.
van AertR. C. M.WichertsJ. M.van AssenM. A. L. M. (2016). Conducting meta-analyses based on p values: Reservations and recommendations for applying p-uniform and p-curve. Perspectives on Psychological Science, 11, 713–729.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
