We assessed whether the most highly cited studies in emotion research reported larger effect sizes compared with meta-analyses and the largest studies on the same question. We screened all reports with at least 1,000 citations and identified matching meta-analyses for 40 highly cited observational studies and 25 highly cited experimental studies. Highly cited observational studies had effects greater on average by 1.42-fold (95% confidence interval [CI] = [1.09, 1.87]) compared with meta-analyses and 1.99-fold (95% CI = [1.33, 2.99]) compared with largest studies on the same questions. Highly cited experimental studies had increases of 1.29-fold (95% CI = [1.01, 1.63]) compared with meta-analyses and 2.02-fold (95% CI = [1.60, 2.57]) compared with the largest studies. There was substantial between-topics heterogeneity, more prominently for observational studies. Highly cited studies often did not have the largest weight in meta-analyses (12 of 65 topics, 18%) but were frequently the earliest ones published on the topic (31 of 65 topics, 48%). Highly cited studies may offer, on average, exaggerated estimates of effects in both observational and experimental designs.
AltmanD. G.BlandJ. M. (2003). Interaction revisited: The difference between two estimates. BMJ, 326(7382), Article 219. https://doi.org/10.1136/bmj.326.7382.219
2.
BakkerM.van DijkA.WichertsJ. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543–554. https://doi.org/10.1177/1745691612459060
3.
BorderR.JohnsonE. C.EvansL. M.SmolenA.BerleyN.SullivanP. F.KellerM. C. (2019). No support for historical candidate gene or candidate gene-by-interaction hypotheses for major depression across multiple large samples. American Journal of Psychiatry, 176(5), 376–387. https://doi.org/10.1176/appi.ajp.2018.18070881
4.
Botvinik-NezerR.HolzmeisterF.CamererC. F.DreberA.HuberJ.JohannessonM.KirchlerM.IwanirR.MumfordJ. A.AdcockR. A.AvesaniP.BaczkowskiB. M.BajracharyaA.BakstL.BallS.BarilariM.BaultN.BeatonD.BeitnerJ.. . . SchonbergT. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582(7810), 84–88. https://doi.org/10.1038/s41586-020-2314-9
5.
ButtonK. S.IoannidisJ. P. A.MokryszC.NosekB. A.FlintJ.RobinsonE. S. J.MunafòM. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org/10.1038/nrn3475
6.
CamererC. F.DreberA.ForsellE.HoT.-H.HuberJ.JohannessonM.KirchlerM.AlmenbergJ.AltmejdA.ChanT.HeikenstenE.HolzmeisterF.ImaiT.IsakssonS.NaveG.PfeifferT.RazenM.WuH. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. https://doi.org/10.1126/science.aaf0918
7.
CamererC. F.DreberA.HolzmeisterF.HoT.-H.HuberJ.JohannessonM.KirchlerM.NaveG.NosekB. A.PfeifferT.AltmejdA.ButtrickN.ChanT.ChenY.ForsellE.GampaA.HeikenstenE.HummerL.ImaiT.. . . WuH. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644. https://doi.org/10.1038/s41562-018-0399-z
8.
CaspiA.SugdenK.MoffittT. E.TaylorA.CraigI. W.HarringtonH.McClayJ.MillJ.MartinJ.BraithwaiteA.PoultonR. (2003). Influence of life stress on depression: moderation by a polymorphism in the 5-HTT gene. Science, 301(5631), 386–389, https://doi.org/10.1126/science.1083968
CristeaI. A.NaudetF. (2018). Defending psychiatry or defending the trivial effects of therapeutic interventions? A citation content analysis of an influential paper. Epidemiology and Psychiatric Sciences, 27(3), 230–239. https://doi.org/10.1017/s2045796017000750
11.
CulverhouseR. C.SacconeN. L.HortonA. C.MaY.AnsteyK. J.BanaschewskiT.BurmeisterM.Cohen-WoodsS.EtainB.FisherH. L.GoldmanN.GuillaumeS.HorwoodJ.JuhaszG.LesterK. J.MandelliL.MiddeldorpC. M.OliéE.VillafuerteS.. . . BierutL. J. (2018). Collaborative meta-analysis finds no evidence of a strong interaction between stress and 5-HTTLPR genotype contributing to the development of depression. Molecular Psychiatry, 23(1), 133–142. https://doi.org/10.1038/mp.2017.44
12.
DavidS. P.WareJ. J.ChuI. M.LoftusP. D.Fusar-PoliP.RaduaJ.MunafòM. R.IoannidisJ. P. A. (2013). Potential reporting bias in fMRI studies of the brain. PLOS ONE, 8(7), Article e70104. https://doi.org/10.1371/journal.pone.0070104
13.
DuncanL. E.OstacherM.BallonJ. (2019). How genome-wide association studies (GWAS) made traditional candidate gene studies obsolete. Neuropsychopharmacology, 44(9), 1518–1523. https://doi.org/10.1038/s41386-019-0389-5
14.
FanelliD.CostasR.IoannidisJ. P. A. (2017). Meta-assessment of bias in science. Proceedings of the National Academy of Sciences, USA, 114(14), 3714–3719. https://doi.org/10.1073/pnas.1618569114
15.
GelmanA. (2018). The failure of null hypothesis significance testing when studying incremental changes, and what to do about it. Personality and Social Psychology Bulletin, 44(1), 16–23. https://doi.org/10.1177/0146167217729162
GøtzscheP. C. (1987). Reference bias in reports of drug trials. British Medical Journal (Clinical Research Edition), 295(6599), 654–656.
18.
GreenbergS. A. (2009). How citation distortions create unfounded authority: Analysis of a citation network. BMJ, 339, Article 2680. https://doi.org/10.1136/bmj.b2680
19.
HardwickeT. E.IoannidisJ. P. A. (2018). Populating the data ark: An attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLOS ONE, 13(8), Article e0201856. https://doi.org/10.1371/journal.pone.0201856
20.
HaririA. R.DrabantE. M.MunozK. E.KolachanaB. S.MattayV. S.EganM. F.WeinbergerD. R. (2005). A susceptibility gene for affective disorders and the response of the human amygdala. Archives of General Psychiatry, 62(2), 146–152. https://doi.org/10.1001/archpsyc.62.2.146
21.
HaririA. R.MattayV. S.TessitoreA.KolachanaB.FeraF.GoldmanD.EganM. F.WeinbergerD. R. (2002). Serotonin transporter genetic variation and the response of the human amygdala. Science, 297(5580), 400–403. https://doi.org/10.1126/science.1071829
22.
IoannidisJ. P. A. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA, 294(2), 218–228. https://doi.org/10.1001/jama.294.2.218
IoannidisJ. P. A.PanagiotouO. A. (2011). Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. JAMA, 305(21), 2200–2210. https://doi.org/10.1001/jama.2011.713
25.
IoannidisJ. P. A.PatsopoulosN. A.RothsteinH. R. (2008). Reasons or excuses for avoiding meta-analysis in forest plots. BMJ, 336(7658), 1413–1415. https://doi.org/10.1136/bmj.a117
26.
IoannidisJ. P. A.TaroneR.McLaughlinJ. K. (2011). The false-positive to false-negative ratio in epidemiologic studies. Epidemiology, 22(4), 450–456. https://doi.org/10.1097/EDE.0b013e31821b506e
27.
KleinR. A.RatliffK. A.VianelloM.AdamsR. B.Jr.BahníkŠ.BernsteinM. J.BocianK.BrandtM. J.BrooksB.BrumbaughC. C.CemalcilarZ.ChandlerJ.CheongW.DavisW. E.DevosT.EisnerM.FrankowsaN.FurrowD.GallianiE. M.. . . NosekB. A. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142–152. https://doi.org/10.1027/1864-9335/a000178
28.
KleinR. A.VianelloM.HasselmanF.AdamsB. G.AdamsR. B.AlperS.AveyardM.AxtJ. R.BabalolaM. T.BahníkŠ.BatraR.BerkicsM.BernsteinM. J.BerryD. R.BialobrzeskaO.BinanE. D.BocianK.BrandtM. J.BuschingR.. . . NosekB. A. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225
29.
KlinA.JonesW.SchultzR.VolkmarF.CohenD. (2002). Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Archives of General Psychiatry, 59(9), 809–816. https://doi.org/10.1001/archpsyc.59.9.809
30.
KvarvenA.StrømlandE.JohannessonM. (2020). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour, 4(4), 423–434. https://doi.org/10.1038/s41562-019-0787-z
31.
LaiH. M.ClearyM.SitharthanT.HuntG. E. (2015). Prevalence of comorbid substance use, anxiety and mood disorders in epidemiological surveys, 1990-2014: A systematic review and meta-analysis. Drug and Alcohol Dependence, 154, 1–13. https://doi.org/10.1016/j.drugalcdep.2015.05.031
32.
LiG.AbbadeL. P. F.NwosuI.JinY.LeenusA.MaazM.WangM.BhattM.ZielinskiL.SangerN.BantotoB.LuoC.ShamsI.ShahidH.ChangY.SunG.MbuagbawL.SamaanZ.LevineM. A. H.. . . ThabaneL. (2017). A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Medical Research Methodology, 17(1), Article 181. https://doi.org/10.1186/s12874-017-0459-5
33.
Martinez-CalderonJ.Flores-CortesM.Morales-AsencioJ. M.Luque-SuarezA. (2019). Pain-related fear, pain intensity and function in individuals with chronic musculoskeletal pain: A systematic review and meta-analysis. Journal of Pain, 20(12), 1394–1415. https://doi.org/10.1016/j.jpain.2019.04.009
34.
McKinnonM. C.YucelK.NazarovA.MacQueenG. M. (2009). A meta-analysis examining clinical predictors of hippocampal volume in patients with major depressive disorder. Journal of Psychiatry & Neuroscience, 34(1), 41–54.
35.
MoffittT. E.CaspiA.HarringtonH.MilneB. J. (2002). Males on the life-course-persistent and adolescence-limited antisocial pathways: Follow-up at age 26 years. Development and Psychopathology, 14, 179–207. https://doi.org/10.1017/s0954579402001104
36.
MunafòM. R.BrownS. M.HaririA. R. (2008). Serotonin transporter (5-HTTLPR) genotype and amygdala activation: A meta-analysis. Biological Psychiatry, 63(9), 852–857. https://doi.org/10.1016/j.biopsych.2007.08.016
37.
NosekB. A.SpiesJ. R.MotylM. (2012). Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631. https://doi.org/10.1177/1745691612459058
38.
OdgersC. L.MoffittT. E.BroadbentJ. M.DicksonN.HancoxR. J.HarringtonH.PoultonR.SearsM. R.ThomsonW. M.CaspiA. (2008). Female and male antisocial trajectories: From childhood origins to adult outcomes. Development and Psychopathology, 20(2), 673–716. https://doi.org/10.1017/s0954579408000333
39.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716
40.
PageM. J.ShamseerL.AltmanD. G.TetzlaffJ.SampsonM.TriccoA. C.Catalá-LópezF.LiL.ReidE. K.Sarkis-OnofreR.MoherD. (2016). Epidemiology and reporting characteristics of systematic reviews of biomedical research: A cross-sectional study. PLOS Medicine, 13(5), Article e1002028. https://doi.org/10.1371/journal.pmed.1002028
41.
PatelC. J.BurfordB.IoannidisJ. P. A. (2015). Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. Journal of Clinical Epidemiology, 68(9), 1046–1058. https://doi.org/10.1016/j.jclinepi.2015.05.029
42.
PauleR. C.MandelJ. (1989). Consensus values, regressions, and weighting factors. Journal of Research of the National Institute of Standards and Technology, 94(3), 197–203. https://doi.org/10.6028/jres.094.020
PriceK. W.FloydR. G.FaganT. K.SmithsonK. (2011). Journal article citation classics in school psychology: Analysis of the most cited articles in five school psychology journals. Journal of School Psychology, 49(6), 649–667. https://doi.org/10.1016/j.jsp.2011.10.001
45.
R Core Team. (2020). R: A language and environment for statistical computing (Version 4.0.0) [Computer software]. R Foundation for Statistical Computing. http://www.R-project.org
46.
RegierD. A.FarmerM. E.RaeD. S.LockeB. Z.KeithS. J.JuddL. L.GoodwinF. K. (1990). Comorbidity of mental disorders with alcohol and other drug abuse. Results from the Epidemiologic Catchment Area (ECA) Study. JAMA, 264(19), 2511–2518.
47.
ReisingK.TtofiM. M.FarringtonD. P.PiqueroA. R. (2019). Depression and anxiety outcomes of offending trajectories: A systematic review of prospective longitudinal studies. Journal of Criminal Justice, 62, 3–15. https://doi.org/10.1016/j.jcrimjus.2018.05.002
48.
RichardsS. H.AndersonL.JenkinsonC. E.WhalleyB.ReesK.DaviesP.BennettP.LiuZ.WestR.ThompsonD. R.TaylorR. S. (2017). Psychological interventions for coronary heart disease. Cochrane Database of Systematic Reviews. https://doi.org/10.1002/14651858.CD002902.pub4
49.
RStudio Team. (2019). RStudio: Integrated development for R [Computer software] (Version 1.2.5033). http://www.rstudio.com/
50.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
51.
StanleyT. D.CarterE. C.DoucouliagosH. (2018). What meta-analyses reveal about the replicability of psychological research. Psychological Bulletin, 144(12), 1325–1346. https://doi.org/10.1037/bul0000169
52.
SteegenS.TuerlinckxF.GelmanA.VanpaemelW. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. https://doi.org/10.1177/1745691616658637
53.
SterneJ. A. C.SuttonA. J.IoannidisJ. P. A.TerrinN.JonesD. R.LauJ.CarpenterJ.RückerG.HarbordR. M.SchmidC. H.TetzlaffJ.DeeksJ. J.PetersJ.MacaskillP.SchwarzerG.DuvalS.AltmanD. G.MoherD.HigginsJ. P. T. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ, 343, Article d4002. https://doi.org/10.1136/bmj.d4002
54.
SwendsenJ. D.MerikangasK. R.CaninoG. J.KesslerR. C.Rubio-StipecM.AngstJ. (1998). The comorbidity of alcoholism with anxiety and depressive disorders in four geographic communities. Comprehensive Psychiatry, 39(4), 176–184. https://doi.org/10.1016/S0010-440X(98)90058-X
55.
SzucsD.IoannidisJ. P. A. (2020). Sample size evolution in neuroimaging research: An evaluation of highly-cited studies (1990-2012) and of latest practices (2017-2018) in high-impact journals. NeuroImage, 221, Article 117164. https://doi.org/10.1016/j.neuroimage.2020.117164
56.
TajikaA.OgawaY.TakeshimaN.HayasakaY.FurukawaT. A. (2015). Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up. British Journal of Psychiatry, 207(4), 357–362. https://doi.org/10.1192/bjp.bp.113.143701
57.
VeronikiA. A.JacksonD.ViechtbauerW.BenderR.BowdenJ.KnappG.KussO.HigginsJ. P.LanganD.SalantiG. (2016). Methods to estimate the between-study variance and its uncertainty in meta-analysis. Research Synthesis Methods, 7(1), 55–79. https://doi.org/10.1002/jrsm.1164
58.
ViechtbauerW. (2007). Confidence intervals for the amount of heterogeneity in meta-analysis. Statistics in Medicine, 26(1), 37–52. https://doi.org/10.1002/sim.2514
WenJ.RenY.WangL.LiY.LiuY.ZhouM.LiuP.YeL.LiY.TianW. (2008). The reporting quality of meta-analyses improves: A random sampling study. Journal of Clinical Epidemiology, 61(8), 770–775. https://doi.org/10.1016/j.jclinepi.2007.10.008
61.
Writing Committee for the ENRICHD Investigators. (2003). Effects of treating depression and low perceived social support on clinical events after myocardial infarction: The Enhancing Recovery in Coronary Heart Disease Patients (ENRICHD) randomized trial. JAMA, 289(23), 3106–3116. https://doi.org/10.1001/jama.289.23.3106
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
