What Is the Test-Retest Reliability of Common Task-Functional MRI Measures? New Empirical Evidence and a Meta-Analysis

Abstract

Identifying brain biomarkers of disease risk is a growing priority in neuroscience. The ability to identify meaningful biomarkers is limited by measurement reliability; unreliable measures are unsuitable for predicting clinical outcomes. Measuring brain activity using task functional MRI (fMRI) is a major focus of biomarker development; however, the reliability of task fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments (N = 1,008) revealed poor overall reliability—mean intraclass correlation coefficient (ICC) = .397. Second, the test-retest reliabilities of activity in a priori regions of interest across 11 common fMRI tasks collected by the Human Connectome Project (N = 45) and the Dunedin Study (N = 20) were poor (ICCs = .067–.485). Collectively, these findings demonstrate that common task-fMRI measures are not currently suitable for brain biomarker discovery or for individual-differences research. We review how this state of affairs came to be and highlight avenues for improving task-fMRI reliability.

Keywords

neuroimaging individual differences statistical analysis cognitive neuroscience

Get full access to this article

View all access options for this article.

References

Barch

D. M.

Burgess

G. C.

Harms

M. P.

Petersen

S. E.

Schlaggar

B. L.

Corbetta

. . . WU-Minn HCP Consortium. (2013). Function in the human connectome: Task-fMRI and individual differences in behavior. NeuroImage, 80, 169–189.

Bennett

C. M.

Miller

M. B.

(2010). How reliable are the results from functional magnetic resonance imaging? Annals of the New York Academy of Sciences, 1191, 133–155.

Borenstein

Hedges

L. V.

Higgins

J. P. T.

Rothstein

H. R.

(2009). Introduction to meta-analysis. Chichester, England: John Wiley. doi:10.1002/9780470743386

Button

K. S.

Ioannidis

J. P. A.

Mokrysz

Nosek

B. A.

Flint

Robinson

E. S. J.

Munafò

M. R.

(2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.

Champely

. (2018). Package ‘pwr.’ Retrieved from http://cran.r-project.org/package=pwr

Chen

Saad

Z. S.

Britton

J. C.

Pine

D. S.

Cox

R. W.

(2013). Linear mixed-effects modeling approach to FMRI group analysis. NeuroImage, 73, 176–190.

Chen

Taylor

P. A.

Haller

S. P.

Kircanski

Stoddard

Pine

D. S.

. . . Cox

R. W.

(2018). Intraclass correlation: Improved modeling approaches and applications for neuroimaging. Human Brain Mapping, 39, 1187–1206.

Cicchetti

D. V.

Sparrow

S. A.

(1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, 127–137.

Cooper

S. R.

Jackson

J. J.

Barch

D. M.

Braver

T. S.

(2019). Neuroimaging of individual differences: A latent variable modeling perspective. Neuroscience & Biobehavioral Reviews, 98, 29–46. doi:10.1016/j.neubiorev.2018.12.022

10.

Crocker

Algina

(2006). Introduction to classical and modern test theory. Mason, OH: Wadsworth.

11.

Cronbach

L. J.

(1957). The two disciplines of scientific psychology. American Psychologist, 12, 671–684. doi:10.1037/h0043943

12.

Dubois

Adolphs

(2016). Building a science of individual differences from fMRI. Trends in Cognitive Sciences, 20, 425–443.

13.

Edlund

J. E.

Nichols

A. L.

(Eds.). (2019). Advanced research methods for the social and behavioral sciences. Cambridge, England: Cambridge University Press.

14.

Egger

Davey Smith

Schneider

Minder

(1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.

15.

Elliott

M. L.

Belsky

D. W.

Anderson

Corcoran

D. L.

Knodt

. . . Hariri

A. R.

(2018). A polygenic score for higher educational attainment is associated with larger brains. Cerebral Cortex, 29, 3496–3504. doi:10.1093/cercor/bhy219

16.

Elliott

M. L.

Knodt

A. R.

Cooke

Kim

M. J.

Melzer

T. R.

Keenan

. . . Hariri

A. R.

(2019). General functional connectivity: Shared features of resting-state and task fMRI drive reliable and heritable individual differences in functional brain networks. NeuroImage, 189, 516–532.

17.

Fröhner

J. H.

Teckentrup

Smolka

M. N.

Kroemer

N. B.

(2019). Addressing the reliability fallacy in fMRI: Similar group effects may arise from unreliable individual effects. NeuroImage, 195, 174–189.

18.

Funder

D. C.

Ozer

D. J.

(2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2, 156–168. doi:10.1177/2515245919847202

19.

Glasser

M. F.

Coalson

T. S.

Robinson

E. C.

Hacker

C. D.

Harwell

Yacoub

. . . Van Essen

D. C.

(2016). A multi-modal parcellation of human cerebral cortex. Nature, 536, 171–178.

20.

Gordon

E. M.

Laumann

T. O.

Gilmore

A. W.

Newbold

D. J.

Greene

D. J.

Berg

J. J.

. . . Dosenbach

N. U. F.

(2017). Precision functional mapping of individual human brains. Neuron, 95, 791–807.

21.

Gratton

Laumann

T. O.

Nielsen

A. N.

Greene

D. J.

Gordon

E. M.

Gilmore

A. W.

. . . Petersen

S. E.

(2018). Functional brain networks are dominated by stable group and individual factors, not cognitive or daily variation. Neuron, 98, 439–452.

22.

Hajcak

Meyer

Kotov

(2017). Psychometrics and the neuroscience of individual differences: Internal consistency limits between-subjects effects. Journal of Abnormal Psychology, 126, 823–834.

23.

Han

Jovicich

Salat

van der Kouwe

Quinn

Czanner

. . . Fischl

(2006). Reliability of MRI-derived measurements of human cerebral cortical thickness: The effects of field strength, scanner upgrade and manufacturer. NeuroImage, 32, 180–194.

24.

Hedge

Powell

Sumner

(2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50, 1166–1186.

25.

Herting

M. M.

Gautam

Chen

Mezher

Vetter

N. C.

(2018). Test-retest reliability of longitudinal task-based fMRI: Implications for developmental studies. Developmental Cognitive Neuroscience, 33, 17–26.

26.

Infantolino

Z. P.

Luking

K. R.

Sauder

C. L.

Curtin

J. J.

Hajcak

(2018). Robust is not necessarily reliable: From within-subjects fMRI contrasts to between-subjects comparisons. NeuroImage, 173, 146–152.

27.

Kriegeskorte

Lindquist

M. A.

Nichols

T. E.

Poldrack

R. A.

Vul

(2010). Everything you never wanted to know about circular analysis, but were afraid to ask. Journal of Cerebral Blood Flow and Metabolism, 30, 1551–1557.

28.

Logothetis

N. K.

Pauls

Augath

Trinath

Oeltermann

(2001). Neurophysiological investigation of the basis of the fMRI signal. Nature, 412, 150–157.

29.

Manuck

S. B.

Brown

S. M.

Forbes

E. E.

Hariri

A. R.

(2007). Temporal stability of individual differences in amygdala reactivity. The American Journal of Psychiatry, 164, 1613–1614.

30.

Nord

C. L.

Gray

Charpentier

C. J.

Robinson

O. J.

Roiser

J. P.

(2017). Unreliability of putative fMRI biomarkers during emotional face processing. NeuroImage, 156, 119–127.

31.

Norman

K. A.

Polyn

S. M.

Detre

G. J.

Haxby

J. V.

(2006). Beyond mind-reading: Multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences, 10, 424–430.

32.

Nosek

B. A.

Ebersole

C. R.

DeHaven

A. C.

Mellor

D. T.

(2018). The preregistration revolution. Proceedings of the National Academy of Sciences, USA, 115, 2600–2606.

33.

Novick

M. R.

(1965). The axioms and principal results of classical test theory. ETS Research Bulletin Series, 1965(1), i–31. doi:10.1002/j.2333-8504.1965.tb00132.x

34.

Nunnally

J. C.

(1959). Introduction to psychological measurement. New York, NY: McGraw-Hill.

35.

Poldrack

R. A.

Baker

C. I.

Durnez

Gorgolewski

K. J.

Matthews

P. M.

Munafò

M. R.

. . . Yarkoni

(2017). Scanning the horizon: Towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience, 18, 115–126.

36.

Poulton

Moffitt

T. E.

Silva

P. A.

(2015). The Dunedin Multidisciplinary Health and Development Study: Overview of the first 40 years, with an eye to the future. Social Psychiatry and Psychiatric Epidemiology, 50, 679–693.

37.

Schäfer

Schwarz

M. A.

(2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology, 10, Article 813. doi:10.3389/fpsyg.2019.00813

38.

Schönbrodt

F. D.

Perugini

(2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47, 609–612. doi:10.1016/j.jrp.2013.05.009

39.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.

40.

Swartz

J. R.

Knodt

A. R.

Radtke

S. R.

Hariri

A. R.

(2015). A neural biomarker of psychological vulnerability to future life stress. Neuron, 85, 505–511.

41.

Vanderwal

Eilbott

Castellanos

F. X.

(2018). Movies in the magnet: Naturalistic paradigms in developmental functional neuroimaging. Developmental Cognitive Neuroscience, 36, Article 100600.

42.

Van Essen

D. C.

Smith

S. M.

Barch

D. M.

Behrens

T. E. J.

Yacoub

Ugurbil

, & WU-Minn HCP Consortium. (2013). The WU-Minn Human Connectome Project: An overview. NeuroImage, 80, 62–79.

43.

Vul

Harris

Winkielman

Pashler

(2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4, 274–290.

44.

Woo

C.-W.

Chang

L. J.

Lindquist

M. A.

Wager

T. D.

(2017). Building better biomarkers: Brain models in translational neuroimaging. Nature Neuroscience, 20, 365–377.

45.

Yarkoni

(2009). Big correlations in little studies: Inflated fMRI correlations reflect low statistical power—commentary on Vul et al. (2009). Perspectives on Psychological Science, 4, 294–298.

46.

Yarkoni

Westfall

(2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12, 1100–1122.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

5.96 MB