Identifying brain biomarkers of disease risk is a growing priority in neuroscience. The ability to identify meaningful biomarkers is limited by measurement reliability; unreliable measures are unsuitable for predicting clinical outcomes. Measuring brain activity using task functional MRI (fMRI) is a major focus of biomarker development; however, the reliability of task fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments (N = 1,008) revealed poor overall reliability—mean intraclass correlation coefficient (ICC) = .397. Second, the test-retest reliabilities of activity in a priori regions of interest across 11 common fMRI tasks collected by the Human Connectome Project (N = 45) and the Dunedin Study (N = 20) were poor (ICCs = .067–.485). Collectively, these findings demonstrate that common task-fMRI measures are not currently suitable for brain biomarker discovery or for individual-differences research. We review how this state of affairs came to be and highlight avenues for improving task-fMRI reliability.
BarchD. M.BurgessG. C.HarmsM. P.PetersenS. E.SchlaggarB. L.CorbettaM.. . . WU-Minn HCP Consortium. (2013). Function in the human connectome: Task-fMRI and individual differences in behavior. NeuroImage, 80, 169–189.
2.
BennettC. M.MillerM. B. (2010). How reliable are the results from functional magnetic resonance imaging?Annals of the New York Academy of Sciences, 1191, 133–155.
3.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta-analysis. Chichester, England: John Wiley. doi:10.1002/9780470743386
4.
ButtonK. S.IoannidisJ. P. A.MokryszC.NosekB. A.FlintJ.RobinsonE. S. J.MunafòM. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.
ChenG.SaadZ. S.BrittonJ. C.PineD. S.CoxR. W. (2013). Linear mixed-effects modeling approach to FMRI group analysis. NeuroImage, 73, 176–190.
7.
ChenG.TaylorP. A.HallerS. P.KircanskiK.StoddardJ.PineD. S.. . . CoxR. W. (2018). Intraclass correlation: Improved modeling approaches and applications for neuroimaging. Human Brain Mapping, 39, 1187–1206.
8.
CicchettiD. V.SparrowS. A. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, 127–137.
9.
CooperS. R.JacksonJ. J.BarchD. M.BraverT. S. (2019). Neuroimaging of individual differences: A latent variable modeling perspective. Neuroscience & Biobehavioral Reviews, 98, 29–46. doi:10.1016/j.neubiorev.2018.12.022
10.
CrockerL.AlginaJ. (2006). Introduction to classical and modern test theory. Mason, OH: Wadsworth.
11.
CronbachL. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12, 671–684. doi:10.1037/h0043943
12.
DuboisJ.AdolphsR. (2016). Building a science of individual differences from fMRI. Trends in Cognitive Sciences, 20, 425–443.
13.
EdlundJ. E.NicholsA. L. (Eds.). (2019). Advanced research methods for the social and behavioral sciences. Cambridge, England: Cambridge University Press.
14.
EggerM.Davey SmithG.SchneiderM.MinderC. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.
15.
ElliottM. L.BelskyD. W.AndersonK.CorcoranD. L.GeT.KnodtA.. . . HaririA. R. (2018). A polygenic score for higher educational attainment is associated with larger brains. Cerebral Cortex, 29, 3496–3504. doi:10.1093/cercor/bhy219
16.
ElliottM. L.KnodtA. R.CookeM.KimM. J.MelzerT. R.KeenanR.. . . HaririA. R. (2019). General functional connectivity: Shared features of resting-state and task fMRI drive reliable and heritable individual differences in functional brain networks. NeuroImage, 189, 516–532.
17.
FröhnerJ. H.TeckentrupV.SmolkaM. N.KroemerN. B. (2019). Addressing the reliability fallacy in fMRI: Similar group effects may arise from unreliable individual effects. NeuroImage, 195, 174–189.
18.
FunderD. C.OzerD. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2, 156–168. doi:10.1177/2515245919847202
19.
GlasserM. F.CoalsonT. S.RobinsonE. C.HackerC. D.HarwellJ.YacoubE.. . . Van EssenD. C. (2016). A multi-modal parcellation of human cerebral cortex. Nature, 536, 171–178.
20.
GordonE. M.LaumannT. O.GilmoreA. W.NewboldD. J.GreeneD. J.BergJ. J.. . . DosenbachN. U. F. (2017). Precision functional mapping of individual human brains. Neuron, 95, 791–807.
21.
GrattonC.LaumannT. O.NielsenA. N.GreeneD. J.GordonE. M.GilmoreA. W.. . . PetersenS. E. (2018). Functional brain networks are dominated by stable group and individual factors, not cognitive or daily variation. Neuron, 98, 439–452.
22.
HajcakG.MeyerA.KotovR. (2017). Psychometrics and the neuroscience of individual differences: Internal consistency limits between-subjects effects. Journal of Abnormal Psychology, 126, 823–834.
23.
HanX.JovicichJ.SalatD.van der KouweA.QuinnB.CzannerS.. . . FischlB. (2006). Reliability of MRI-derived measurements of human cerebral cortical thickness: The effects of field strength, scanner upgrade and manufacturer. NeuroImage, 32, 180–194.
24.
HedgeC.PowellG.SumnerP. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50, 1166–1186.
25.
HertingM. M.GautamP.ChenZ.MezherA.VetterN. C. (2018). Test-retest reliability of longitudinal task-based fMRI: Implications for developmental studies. Developmental Cognitive Neuroscience, 33, 17–26.
26.
InfantolinoZ. P.LukingK. R.SauderC. L.CurtinJ. J.HajcakG. (2018). Robust is not necessarily reliable: From within-subjects fMRI contrasts to between-subjects comparisons. NeuroImage, 173, 146–152.
27.
KriegeskorteN.LindquistM. A.NicholsT. E.PoldrackR. A.VulE. (2010). Everything you never wanted to know about circular analysis, but were afraid to ask. Journal of Cerebral Blood Flow and Metabolism, 30, 1551–1557.
28.
LogothetisN. K.PaulsJ.AugathM.TrinathT.OeltermannA. (2001). Neurophysiological investigation of the basis of the fMRI signal. Nature, 412, 150–157.
29.
ManuckS. B.BrownS. M.ForbesE. E.HaririA. R. (2007). Temporal stability of individual differences in amygdala reactivity. The American Journal of Psychiatry, 164, 1613–1614.
30.
NordC. L.GrayA.CharpentierC. J.RobinsonO. J.RoiserJ. P. (2017). Unreliability of putative fMRI biomarkers during emotional face processing. NeuroImage, 156, 119–127.
31.
NormanK. A.PolynS. M.DetreG. J.HaxbyJ. V. (2006). Beyond mind-reading: Multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences, 10, 424–430.
32.
NosekB. A.EbersoleC. R.DeHavenA. C.MellorD. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, USA, 115, 2600–2606.
33.
NovickM. R. (1965). The axioms and principal results of classical test theory. ETS Research Bulletin Series, 1965(1), i–31. doi:10.1002/j.2333-8504.1965.tb00132.x
34.
NunnallyJ. C. (1959). Introduction to psychological measurement. New York, NY: McGraw-Hill.
35.
PoldrackR. A.BakerC. I.DurnezJ.GorgolewskiK. J.MatthewsP. M.MunafòM. R.. . . YarkoniT. (2017). Scanning the horizon: Towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience, 18, 115–126.
36.
PoultonR.MoffittT. E.SilvaP. A. (2015). The Dunedin Multidisciplinary Health and Development Study: Overview of the first 40 years, with an eye to the future. Social Psychiatry and Psychiatric Epidemiology, 50, 679–693.
37.
SchäferT.SchwarzM. A. (2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology, 10, Article 813. doi:10.3389/fpsyg.2019.00813
38.
SchönbrodtF. D.PeruginiM. (2013). At what sample size do correlations stabilize?Journal of Research in Personality, 47, 609–612. doi:10.1016/j.jrp.2013.05.009
39.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
40.
SwartzJ. R.KnodtA. R.RadtkeS. R.HaririA. R. (2015). A neural biomarker of psychological vulnerability to future life stress. Neuron, 85, 505–511.
41.
VanderwalT.EilbottJ.CastellanosF. X. (2018). Movies in the magnet: Naturalistic paradigms in developmental functional neuroimaging. Developmental Cognitive Neuroscience, 36, Article 100600.
42.
Van EssenD. C.SmithS. M.BarchD. M.BehrensT. E. J.YacoubE.UgurbilK., & WU-Minn HCP Consortium. (2013). The WU-Minn Human Connectome Project: An overview. NeuroImage, 80, 62–79.
43.
VulE.HarrisC.WinkielmanP.PashlerH. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4, 274–290.
44.
WooC.-W.ChangL. J.LindquistM. A.WagerT. D. (2017). Building better biomarkers: Brain models in translational neuroimaging. Nature Neuroscience, 20, 365–377.
45.
YarkoniT. (2009). Big correlations in little studies: Inflated fMRI correlations reflect low statistical power—commentary on Vul et al. (2009). Perspectives on Psychological Science, 4, 294–298.
46.
YarkoniT.WestfallJ. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12, 1100–1122.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
