Cognitive tasks borrowed from experimental psychology are often used to assess individual differences. A cardinal issue of this transition from experimental to correlational designs is reduced retest reliability of some well-established cognitive effects as well as speed–accuracy trade-off. The present study aimed to address these issues by examining the retest reliability of various methods for speed–accuracy integration and by comparing between two types of task modeling: difference scores and residual scores. Results from three studies on executive functions show that (a) integrated speed–accuracy scoring is generally more reliable as compared with nonintegrated methods: mean response time and accuracy; and (b) task modeling, especially residual scores, reduced reliability. We thus recommend integrating speed and accuracy, at least for measuring executive functions.
BruyerR.BrysbaertM. (2011). Combining speed and accuracy in cognitive psychology. Psychologica Belgica, 51(1), 5-13. https://doi.org/10.5334/pb-51-1-5
2.
BrysbaertM. (2019). How many participants do we have to include in properly powered experiments ? A tutorial of power analysis with reference tables. Journal of Cognition, 2(1), 1-38. https://doi.org/10.5334/joc.72
3.
CantyA. (2002). Resampling methods in R: The boot package. Newsletter of the R Project, 2(3), 343-344. https://doi.org/10.1159/000323281
4.
CarusoJ. C. (2004). A comparison of the reliabilities of four types of difference scores for five cognitive assessment batteries. European Journal of Psychological Assessment, 20(3), 166-171. https://doi.org/10.1027/1015-5759.20.3.166
5.
CharterR. A.FeldtL. S. (2001). Meaning of reliability in terms of correct and incorrect clinical decisions: The art of decision making is still alive. Journal of Clinical and Experimental Neuropsychology, 23(4), 530-537. https://doi.org/10.1076/jcen.23.4.530.1227
6.
ConleyJ. J. (1984). The hierarchy of consistency: A review and model of longitudinal findings on adult individual differences in intelligence, personality and self-opinion. Personality and Individual Differences, 5(1), 11-25. https://doi.org/10.1016/0191-8869(84)90133-8
7.
CronbachL. J.FurbyL. (1970). How we should measure “change”: Or should we?Psychological Bulletin, 74(1), 68-80. https://doi.org/10.1037/h0029382
8.
DanthiirV.RobertsR. D.SchulzeR.WilhelmO. (2005). Mental speed: On frameworks, paradigms, and a platform for the future. In Handbook of understanding and measuring intelligence (pp. 27-46). Sage. https://doi.org/10.4135/9781452233529.n3
9.
DraheimC.HicksK. L.EngleR. W. (2016). Combining reaction time and accuracy: The relationship between working memory capacity and task switching as a case example. Perspectives on Psychological Science, 11(1), 133-155. https://doi.org/10.1177/1745691615596990
10.
DraheimC.MashburnC. A.MartinJ. D.EngleR. W. (2019). Reaction time in differential and developmental research : A review and commentary on the problems and alternatives. Psychological Bulletin, 145(5), 508-535. https://doi.org/10.1037/bul0000192
11.
EnkaviA. Z.EisenbergI. W.BissettP. G.MazzaG. L.MacKinnonD. P.MarschL. A.PoldrackR. A. (2019). Large-scale analysis of test–retest reliabilities of self-regulation measures. Proceedings of the National Academy of Sciences, 116(12), 5472-5477. https://doi.org/10.1073/pnas.1818430116
12.
GollwitzerM.ChristO.LemmerG. (2014). Individual differences make a difference: On the use and the psychometric properties of difference scores in social psychology. European Journal of Social Psychology, 44(7), 673-482. https://doi.org/10.1002/ejsp.2028
13.
GordonS.TodderD.DeutschI.GarbiD.AlkobiO.ShrikiO.Shkedy-RabaniA.ShaharN.MeiranN. (2020). Effects of neurofeedback and working memory-combined training on executive functions in healthy young adults. Psychological Research, 84(6), 1586-1609. https://doi.org/10.1007/s00426-019-01170-w
14.
GordonS.TodderD.DeutschI.GarbiD.GetterN.MeiranN. (2018). Are resting state spectral power measures related to executive functions in healthy young adults?Neuropsychologia, 108(January), 61-72. https://doi.org/10.1016/j.neuropsychologia.2017.10.031
15.
HedgeC.PowellG.SumnerP. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166-1186. https://doi.org/10.3758/s13428-017-0935-1
16.
HughesM. M.LinckJ. A.BowlesA. R.KoethJ. T.BuntingM. F. (2014). Alternatives to switch-cost scoring in the task-switching paradigm: Their reliability and increased validity. Behavior Research Methods, 46(3), 702-721. https://doi.org/10.3758/s13428-013-0411-5
17.
InselT.CuthbertB.GarveyM.HeinssenR.PineD. S.QuinnK.SanislowC.WangP. (2010). Research domain criteria (RDoC): Toward a new classification framework for research on mental disorders. American Journal of Psychiatry, 167(7), 748-751. https://doi.org/10.1176/appi.ajp.2010.09091379
18.
JewsburyP. A.BowdenS. C.StraussM. E. (2016). Integrating the switching, inhibition, and updating model of executive function with the Cattell-Horn-Carroll model. Journal of Experimental Psychology, 145(2), 220-245. https://doi.org/10.1037/xge0000119
19.
JuradoM. B.RosselliM. (2007). The elusive nature of executive functions: A review of our current understanding. Neuropsychology Review, 17(3), 213-233. https://doi.org/10.1007/s11065-007-9040-z
20.
KarrJ. E.AreshenkoffC. N.RastP.HoferS. M.IversonG. L.Garcia-BarreraM. A. (2018). The unity and diversity of executive functions: A systematic review and re-analysis of latent variable studies. Psychological Bulletin, 144(11), 1147-1185. https://doi.org/10.1037/bul0000160
21.
KlieglR.WeiP.DambacherM.YanM.ZhouX. (2011). Experimental effects and individual differences in linear mixed models : Estimating the relationship between spatial, object, and attraction effects in visual attention. Frontiers in Psychology, 1(January), 1-12. https://doi.org/10.3389/fpsyg.2010.00238
22.
KotovR.KruegerR. F.WatsonD.AchenbachT. M.AlthoffR. R.BagbyR. M.BrownT. A.CarpenterW. T.CaspiA.ClarkL. A.EatonN. R.ForbesM. K.ForbushK. T.GoldbergD.HasinD.HymanS. E.IvanovaM. Y.LynamD. R.MarkonK.. . . ZimmermanM. (2017). The Hierarchical Taxonomy of Psychopathology (HiTOP): A dimensional alternative to traditional nosologies. Journal of Abnormal Psychology, 126(4), 454-477. https://doi.org/10.1037/abn0000258
23.
LiesefeldH. R.FuX.ZimmerH. D. (2015). Fast and careless or careful and slow? Apparent holistic processing in mental rotation is explained by speed-accuracy trade-offs. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41(4), 1140-1151. https://doi.org/10.1037/xlm0000081
24.
LiesefeldH. R.JanczykM. (2019). Combining speed and accuracy to control for speed-accuracy trade-offs(?). Behavior Research Methods, 51(1), 40-60. https://doi.org/10.3758/s13428-018-1076-x
25.
LinnR. L.SlindeJ. A. (1977). The determination of the significance of change between pre- and posttesting periods. Review of Educational Research, 47(1), 121-150. https://doi.org/10.3102/00346543047001121
26.
MagnusB. E.WilloughbyM. T.BlairC. B.KuhnL. J. (2019). Integrating item accuracy and reaction time to improve the measurement of inhibitory control abilities in early childhood. Assessment, 26(7), 1296-1306. https://doi.org/10.1177/1073191117740953
MillerJ.UlrichR. (2013). Mental chronometry and individual differences: Modeling reliabilities and correlations of reaction time means and effect sizes. Psychonomic Bulletin & Review, 20(5), 819-858. https://doi.org/10.3758/s13423-013-0404-5
29.
MiyakeA.FriedmanN. P.EmersonM. J.WitzkiA. H.HowerterA.WagerT. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49-100. https://doi.org/10.1006/cogp.1999.0734
30.
MorrisS. E.CuthbertB. N. (2012). Research domain criteria: Cognitive systems, neural circuits, and dimensions of behavior. Dialogues in Clinical Neuroscience, 14(1), 29-37. https://doi.org/10.31887/DCNS.2012.14.1/smorris
31.
NunnallyJ. C. (1965). Educational measurement and evaluation. British Journal of Educational Studies, 13(2), 212. https://doi.org/10.2307/3118345
32.
PaapK. R.SawiO. (2016). The role of test-retest reliability in measuring individual and group differences in executive functioning. Journal of Neuroscience Methods, 274(December), 81-93. https://doi.org/10.1016/j.jneumeth.2016.10.002
33.
PachellaR. G.FisherD. (1972). Hick’s law and the speed-accuracy trade-off in absolute judgment. Journal of Experimental Psychology, 92(3), 378-384. https://doi.org/10.1037/h0032369
RatcliffR. (2014). Measuring psychometric functions with the diffusion model. Journal of Experimental Psychology: Human Perception and Performance, 40(2), 870-888. https://doi.org/10.1037/a0034954
37.
RevelleW.CondonD. M. (2019). Reliability from α to ω: A tutorial. Psychological Assessment, 31(12), 1395-1411. https://doi.org/10.1037/pas0000754
Rey-MermetA.GadeM.SouzaA. S.von BastianC. C.OberauerK. (2019). Is executive control related to working memory capacity and fluid intelligence?Journal of Experimental Psychology: General, 148(8), 1335-1372. https://doi.org/10.1037/xge0000593
40.
RouderJ. N.HaafJ. M. (2019). A psychometrics of individual differences in experimental tasks. Psychonomic Bulletin & Review, 26(2), 452-467. https://doi.org/10.3758/s13423-018-1558-y
41.
SchmiedekF.OberauerK.WilhelmO.WittmannW. W. (2007). Individual differences in components of reaction time distributions and their relations to working memory and intelligence. Journal of Experimental Psychology: General, 136(3), 414-429. https://doi.org/10.1037/0096-3445.136.3.414
42.
SchubertA.-L.FrischkornG. T.HagemannD.VossA. (2016). Trait characteristics of diffusion model parameters. Journal of Intelligence, 4(3), Article 7. https://doi.org/10.3390/jintelligence4030007
43.
ShaharN.TeodorescuA. R.UsherM.PeregM.MeiranN. (2014). Selective influence of working memory load on exceptionally slow reaction times. Journal of Experimental Psychology: General, 143(5), 1837-1860. https://doi.org/10.1037/a0037190
44.
SnyderH. R.MiyakeA.HankinB. L. (2015). Advancing understanding of executive function impairments and psychopathology: Bridging the gap between clinical and cognitive approaches. Frontiers in Psychology, 6, Article 328. https://doi.org/10.3389/fpsyg.2015.00328
45.
StraussE.ShermanE. M. S.SpreenO. (2006). A compendium of neuropsychological tests: Administration, norms, and commentary. American Chemical Society.
46.
StraussG. P.AllenD. N.JorgensenM. L.CramerS. L. (2005). Test-retest reliability of standard and emotional Stroop tasks: An investigation of color-word and picture-word versions. Assessment, 12(3), 330-337. https://doi.org/10.1177/1073191105276375
47.
TownsendJ. T.AshbyF. G. (1983). Stochastic modeling of elementary psychological processes. American Journal of Psychology, 98(3), 480-484. https://doi.org/10.2307/1422636
48.
VandierendonckA. (2017). A comparison of methods to combine speed and accuracy measures of performance: A rejoinder on the binning procedure. Behavior Research Methods, 49(2), 653-673. https://doi.org/10.3758/s13428-016-0721-5
49.
VandierendonckA. (2018). Further tests of the utility of integrated speed-accuracy measures in task switching. Journal of Cognition, 1(1), 1-16. https://doi.org/10.5334/joc.6
50.
WasC. A.WoltzD. J. (2007). Reexamining the relationship between working memory and comprehension: The role of available long-term memory. Journal of Memory and Language, 56(1), 86-102. https://doi.org/10.1016/j.jml.2006.07.008
51.
WatsonD. (2004). Stability versus change, dependability versus error: Issues in the assessment of personality over time. Journal of Research in Personality, 38(4), 319-350. https://doi.org/10.1016/j.jrp.2004.03.001
52.
WeirJ. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. https://doi.org/10.1519/00124278-200502000-00038
53.
WoltzD. J.WasC. A. (2006). Availability of related long-term memory during and after attention focus in working memory. Memory & Cognition, 34(3), 668-684. https://doi.org/10.3758/BF03193587
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
