Need for Psychometric Theory in Neuroscience Research and Training: Reply to Kragel et al. (2021)

Casey B. J. Cannonier T. Conley M. I. Cohen A. O. Barch D. M. Heitzeg M. M. Soules M. E. Teslovich T. Dellarco D. V. Garavan H. Orr C. A. Wager T. D. Banich M. T. Speer N. K. Sutherland M. T. Riedel M. C. Dick A. S. Bjork J. M. Thomas K. M. . . . Dale A. M. (2018). The Adolescent Brain Cognitive Development (ABCD) study: Imaging acquisition across 21 sites. Developmental Cognitive Neuroscience, 32, 43–54. https://doi.org/10.1016/j.dcn.2018.03.001

Cohen A. (2020, June 25). Duke University researchers say every brain activity study you’ve ever read is wrong. https://www.fastcompany.com/90520750/duke-university-researchers-say-every-brain-activity-study-youve-ever-read-is-wrong

Cronbach L. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12(11), 671–684.

Elliott M. L. Knodt A. R. Cooke M. Kim M. J. Melzer T. R. Keenan R. Ireland D. Ramrakha S. Poulton R. Caspi A. Moffitt T. E. Hariri A. R. (2019). General functional connectivity: Shared features of resting-state and task fMRI drive reliable and heritable individual differences in functional brain networks. NeuroImage, 189, 516–532. https://doi.org/10.1016/j.neuroimage.2019.01.068

Elliott M. L. Knodt A. R. Ireland D. Morris M. L. Poulton R. Ramrakha S. Sison M. L. Moffitt T. E. Caspi A. Hariri A. R. (2020). What is the test-retest reliability of common task-functional MRI measures? New empirical evidence and a meta-analysis. Psychological Science, 31(7), 792–806. https://doi.org/10.1177/0956797620916786

Fröhner J. H. Teckentrup V. Smolka M. N. Kroemer N. B. (2019). Addressing the reliability fallacy in fMRI: Similar group effects may arise from unreliable individual effects. NeuroImage, 195, 174–189. https://doi.org/10.1101/215053

Hedge C. Powell G. Sumner P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166–1186. https://doi.org/10.3758/s13428-017-0935-1

Kragel P. A. Han X. Kraynak T. E. Gianaros P. J. Wager T. D. (2021). Functional MRI can be highly reliable, but it depends on what you measure: A commentary on Elliott et al. (2020). Psychological Science, 32(4), 622–626. https://doi.org/10.1177/0956797621989730

Miller K. L. Alfaro-Almagro F. Bangerter N. K. Thomas D. L. Yacoub E. Xu J. Bartsch A. J. Jbabdi S. Sotiropoulos S. N. Andersson J. L. Griffanti L. Douaud G. Okell T. W. Weale P. Dragonu I. Garratt S. Hudson S. Collins R. Jenkinson M. . . . Smith S. M. (2016). Multimodal population brain imaging in the UK Biobank prospective epidemiological study. Nature Neuroscience, 19(11), 1523–1536. https://doi.org/10.1038/nn.4393

Noble S. Spann M. N. Tokoglu F. Shen X. Constable R. T. Scheinost D. (2017). Influences on the test–retest reliability of functional connectivity MRI and its relationship with behavioral utility. Cerebral Cortex, 27(11), 5415–5429. https://doi.org/10.1093/cercor/bhx230

Sugden K. Hannon E. J. Arseneault L. Belsky D. W. Corcoran D. L. Fisher H. L. Houts R. M. Kandaswamy R. Moffitt T. E. Poulton R. Prinz J. A. Rasmussen L. J. H. Williams B. S. Wong C. C. Y. Mill J. Caspi A. (2020). Patterns of reliability: Assessing the reproducibility and integrity of DNA methylation measurement. Patterns, 1(2), Article 100014. https://doi.org/10.1016/j.patter.2020.100014

Van Essen D. C. Smith S. M. Barch D. M. Behrens T. E. J. Yacoub E. Ugurbil K. (2013). The WU-Minn Human Connectome Project: An overview. NeuroImage, 80, 62–79. https://doi.org/10.1016/j.neuroimage.2013.05.041

Zuo X.-N. Xu T. Milham M. P. (2019). Harnessing reliability for neuroscience research. Nature Human Behaviour, 3, 768–771. https://doi.org/10.1038/s41562-019-0655-x