The Impact of Interference Rate on the Test-Retest Reliability of the Stroop Test

Abstract

Interference tasks (e.g., the Stroop task) exhibit low test–retest reliability, which hinders the measurement of individual differences. This study examined whether controlling the interference rate improves reliability. The interference rate is a metric derived from the Horse-Race Model, defined as the probability of experiencing semantic-color conflict in incongruent Stroop trials (distinct from list-wide congruency, i.e., the proportion of incongruent trials in a block). In Studies 1a and 1b, interference rate was positively correlated with the magnitude of the Stroop effect, indicating that it accounts for meaningful individual variability. In Study 2, the fixed-interference-rate group (IR-fixed) had their interference rate controlled at 75% across two testing sessions (2-week interval) by adjusting the stimulus-onset asynchrony between semantic and color features (i.e., ensuring a 75% probability of experiencing interference in incongruent trials). Compared to the random-interference-rate group (IR-random), the IR-fixed group showed significantly higher test–retest reliability and stronger cross-session predictability of the Stroop effect, demonstrating that controlling the interference rate enhances measurement stability. Studies 1b and 2 also revealed moderate cross-session correlations for interference rate, regardless of whether the interval was a single Stroop task administration or 2 weeks. Furthermore, across all studies, interference rate exhibited poor internal consistency (McDonald’s ω < .60). Overall, the results suggest that interference rate can predict the Stroop effect but is not a stable psychological trait; instead, it fluctuates within a certain range. Therefore, failing to control the interference rate adversely impacts the test–retest reliability of the Stroop task.

Keywords

Stroop effect interference rate test–retest reliability reliability paradox

Get full access to this article

View all access options for this article.

References

Algom

Chajut

(2019). Reclaiming the Stroop effect back from control to input-driven attention and perception. Frontiers in Psychology, 10, 1683. https://doi.org/10.3389/fpsyg.2019.01683

Braem

Bugg

J. M.

Schmidt

J. R.

Crump

M. J. C.

Weissman

D. H.

Notebaert

Egner

(2019). Measuring adaptive control in conflict tasks. Trends in Cognitive Sciences, 23(9), 769–783. https://doi.org/10.1016/j.tics.2019.07.002

Burgoyne

A. P.

Tsukahara

J. S.

Mashburn

C. A.

Pak

Engle

R. W.

(2023). Nature and measurement of attention control. Journal of Experimental Psychology: General, 152(8), 2369–2402. https://doi.org/10.1037/xge0001408

Chen

Bailey

Tiernan

B. N.

West

(2011). Neural correlates of stimulus and response interference in a 2–1 mapping stroop task. International Journal of Psychophysiology, 80(2), 129–138. https://doi.org/10.1016/j.ijpsycho.2011.02.012

Cohen

J. D.

Dunbar

McClelland

J. L.

(1990). On the control of automatic processes: A parallel distributed processing account of the Stroop effect. Psychological Review, 97(3), 332–361. https://doi.org/10.1037/0033-295X.97.3.332

De Houwer

. (2003). The extrinsic affective Simon task. Experimental Psychology, 50(2), 77–85. https://doi.org/10.1026//1618-3169.50.2.77

De Jong

Berendsen

Cools

. (1999). Goal neglect and inhibitory limitations: Dissociable causes of interference effects in conflict situations. Acta Psychologica, 101(2–3), 379–394. https://doi.org/10.1016/S0001-6918(99)00012-8

Diamond

(2013). Executive functions. Annual Review of Psychology, 64(1), 135–168. https://doi.org/10.1146/annurev-psych-113011-143750

Draheim

Hicks

K. L.

Engle

R. W.

(2016). Combining reaction time and accuracy: The relationship between working memory capacity and task switching as a case example. Perspectives on Psychological Science, 11(1), 133–155. https://doi.org/10.1177/1745691615596990

10.

Draheim

Mashburn

C. A.

Martin

J. D.

Engle

R. W.

(2019). Reaction time in differential and developmental research: A review and commentary on the problems and alternatives. Psychological Bulletin, 145(5), 508–535. https://doi.org/10.1037/bul0000192

11.

Draheim

Tshukara

J. S.

Engle

R. W.

(2023). Replication and extension of the toolbox approach to measuring attention control. Behavior Research Methods, 56(3), 2135–2157. https://doi.org/10.3758/s13428-023-02140-2

12.

Draheim

Tsukahara

J. S.

Martin

J. D.

Mashburn

C. A.

Engle

R. W.

(2021). A toolbox approach to improving the measurement of attention control. Journal of Experimental Psychology: General, 150(2), 242–275. https://doi.org/10.1037/xge0000783

13.

Dunbar

MacLeod

C. M.

(1984). A horse race of a different color: Stroop interference patterns with transformed words. Journal of Experimental Psychology: Human Perception and Performance, 10(5), 622–639. https://doi.org/10.1037/0096-1523.10.5.622

14.

Dyer

F. N.

(1971). The duration of word meaning responses: Stroop interference for different preexposures of the word. Psychonomic Science, 25(4), 229–231. https://doi.org/10.3758/BF03329102

15.

Dyer

F. N.

(1973). The Stroop phenomenon and its use in the stlldy of perceptual, cognitive, and response processes. Memory & Cognition, 1(2), 106–120. https://doi.org/10.3758/BF03198078

16.

Egner

Delano

Hirsch

(2007). Separate conflict-specific cognitive control mechanisms in the human brain. NeuroImage, 35(2), 940–948. https://doi.org/10.1016/j.neuroimage.2006.11.061

17.

Eidels

(2012). Independent race of colour and word can predict the Stroop effect. Australian Journal of Psychology, 64(4), 189–198. https://doi.org/10.1111/j.1742-9536.2012.00052.x

18.

Engel

S. A.

Furmanski

C. S.

(2001). Selective adaptation to color contrast in human primary visual cortex. Journal of Neuroscience, 21(11), 3949–3954. https://doi.org/10.1523/JNEUROSCI.21-11-03949.2001

19.

Epp

A. M.

Dobson

K. S.

Dozois

D. J. A.

Frewen

P. A.

(2012). A systematic meta-analysis of the Stroop task in depression. Clinical Psychology Review, 32(4), 316–328. https://doi.org/10.1016/j.cpr.2012.02.005

20.

Eriksen

B. A.

Eriksen

C. W.

(1974). Effects of noise letters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16(1), 143–149. https://doi.org/10.3758/BF03203267

21.

Eysenck

M. W.

Derakshan

(2011). New perspectives in attentional control theory. Personality and Individual Differences, 50(7), 955–960. https://doi.org/10.1016/j.paid.2010.08.019

22.

Faber

L. G.

Maurits

N. M.

Lorist

M. M.

(2012). Mental fatigue affects visual selective attention. PLOS ONE, 7(10), e48073. https://doi.org/10.1371/journal.pone.0048073

23.

Faul

Erdfelder

Buchner

Lang

A.-G.

(2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. https://doi.org/10.3758/BRM.41.4.1149

24.

Friedman

N. P.

Miyake

(2004). The relations among inhibition and interference control functions: A latent-variable analysis. Journal of Experimental Psychology: General, 133(1), 101–135. https://doi.org/10.1037/0096-3445.133.1.101

25.

Gärtner

Strobel

(2021). Individual differences in inhibitory control: A latent variable analysis. Journal of Cognition, 4(1), 17. https://doi.org/10.5334/joc.150

26.

Geng

Schnur

T. T.

Janssen

(2014). Relative speed of processing affects interference in Stroop and picture–word interference paradigms: Evidence from the distractor frequency effect. Language, Cognition and Neuroscience, 29(9), 1100–1114. https://doi.org/10.1080/01690965.2013.846473

27.

Glaser

M. O.

Glaser

W. R.

(1982). Time course analysis of the Stroop phenomenon. Journal of Experimental Psychology: Human Perception and Performance, 8(6), 875–894. https://doi.org/10.1037/0096-1523.8.6.875

28.

Hedge

Powell

Bompas

Sumner

(2022). Strategy and processing speed eclipse individual differences in control ability in conflict tasks. Journal of Experimental Psychology: Learning, Memory, and Cognition, 48(10), 1448–1469. https://doi.org/10.1037/xlm0001028

29.

Hedge

Powell

Sumner

(2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166–1186. https://doi.org/10.3758/s13428-017-0935-1

30.

Hommel

(1993). The relationship between stimulus processing and response selection in the Simon task: Evidence for a temporal overlap. Psychological Research, 55(4), 280–290. https://doi.org/10.1007/BF00419688

31.

Huang

Yan

Tian

Han

(2022). Using relative-speed-of-processing to explain the shielding function of task rules. Quarterly Journal of Experimental Psychology, 75(11), 2177–2196. https://doi.org/10.1177/17470218211069484

32.

Kucina

Wells

Lewis

De Salas

Kohl

Palmer

M. A.

Sauer

J. D.

Matzke

Aidman

Heathcote

(2023). Calibration of cognitive tests to address the reliability paradox for decision-conflict tasks. Nature Communications, 14(1), 2234. https://doi.org/10.1038/s41467-023-37777-2

33.

Lavie

(1995). Perceptual load as a necessary condition for selective attention. Journal of Experimental Psychology: Human Perception and Performance, 21(3), 451–468. https://doi.org/10.1037/0096-1523.21.3.451

34.

Logan

G. D.

Zbrodoff

N. J.

(1998). Stroop-type interference: Congruity effects in color naming with typewritten responses. Journal of Experimental Psychology: Human Perception and Performance, 24(3), 978–992. https://doi.org/10.1037/0096-1523.24.3.978

35.

Luo

C. R.

(1999). Semantic competition as the basis of Stroop interference: Evidence from color-word matching tasks. Psychological Science, 10(1), 35–40. https://doi.org/10.1111/1467-9280.00103

36.

MathWorks. (2021). MATLAB (Version R2021b). The MathWorks, Inc. https://www.mathworks.com/products/matlab.html

37.

McDonald

R. P.

(2013). Test theory: A unified treatment (1st ed.). Psychology Press. https://doi.org/10.4324/9781410601087

38.

McGraw

K. O.

Wong

S. P.

(1996). “Forming inferences about some intraclass correlations coefficients”: Correction. Psychological Methods, 1(4), 390–390. https://doi.org/10.1037/1082-989X.1.4.390

39.

Moretti

Koch

Hornjak

Von Bastian

C. C.

(2025). Quality over quantity: Focusing on high-conflict trials to improve the reliability and validity of attentional control measures. Journal of Experimental Psychology: Learning, Memory, and Cognition, 52(1), 34–54. https://doi.org/10.1037/xlm0001466

40.

Moretti

Koch

Steinhauser

Schuch

(2023). Disentangling task-selection failures from task-execution failures in task switching: An assessment of different paradigms. Psychological Research, 87(3), 929–950. https://doi.org/10.1007/s00426-022-01708-5

41.

Nakagawa

Schielzeth

(2010). Repeatability for Gaussian and non-Gaussian data: A practical guide for biologists. Biological Reviews, 85(4), 935–956. https://doi.org/10.1111/j.1469-185X.2010.00141.x

42.

Parris

B. A.

Hasshim

Wadsley

Augustinova

Ferrand

(2022). The loci of Stroop effects: A critical review of methods and evidence for levels of processing contributing to color-word Stroop effects and the implications for the loci of attentional selection. Psychological Research, 86(4), 1029–1053. https://doi.org/10.1007/s00426-021-01554-x

43.

Posner

M. I.

Snyder

C. R. R.

(2004). Attention and cognitive control. In Balota

D. A.

Marsh

E. J.

(Eds.), Cognitive psychology: Key readings (pp. 205–223). Psychology Press.

44.

Provost

S. C.

Woodward

(1991). Effects of nicotine gum on repeated administration of the stroop test. Psychopharmacology, 104(4), 536–540. https://doi.org/10.1007/BF02245662

45.

R Core Team (2023). R: A language and environment for statistical computing (Version 4.2.3). R Foundation for Statistical Computing. https://www.R-project.org/

46.

Ranger

Kuhn

J. T.

Ortner

T. M.

(2020). Modeling responses and response times in tests with the hierarchical model and the three-parameter lognormal distribution. Educational and Psychological Measurement, 80(6), 1059–1089. https://doi.org/10.1177/0013164420908916

47.

Rey-Mermet

Gade

Oberauer

(2018). Should we stop thinking about inhibition? Searching for individual and age differences in inhibition ability. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(4), 501–526. https://doi.org/10.1037/xlm0000450

48.

Rey-Mermet

Gade

Souza

A. S.

Von Bastian

C. C.

Oberauer

(2019). Is executive control related to working memory capacity and fluid intelligence? Journal of Experimental Psychology: General, 148(8), 1335–1372. https://doi.org/10.1037/xge0000593

49.

Rothermund

Gollnick

Giesen

C. G.

(2022). Accounting for proportion congruency effects in the Stroop task in a confounded setup: Retrieval of stimulus-response episodes explains it all. Journal of Cognition, 5(1), 39. https://doi.org/10.5334/joc.232

50.

Rouder

J. N.

Haaf

J. M.

(2019). A psychometrics of individual differences in experimental tasks. Psychonomic Bulletin & Review, 26(2), 452–467. https://doi.org/10.3758/s13423-018-1558-y

51.

Rouder

J. N.

Kumar

Haaf

J. M.

(2023). Why many studies of individual differences with inhibition tasks may not localize correlations. Psychonomic Bulletin & Review, 30(6), 2049–2066. https://doi.org/10.3758/s13423-023-02293-3

52.

Scheibe

K. E.

Shaver

P. R.

Carrier

S. C.

(1967). Color association values and response interference on variants of the Stroop test. Acta Psychologica, 26, 286–295. https://doi.org/10.1016/0001-6918(67)90028-5

53.

Schönbrodt

F. D.

Perugini

(2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47(5), 609–612. https://doi.org/10.1016/j.jrp.2013.05.009

54.

Seymour

P. H. K.

(1977). Conceptual encoding and locus of the Stroop effect. Quarterly Journal of Experimental Psychology, 29(2), 245–265. https://doi.org/10.1080/14640747708400601

55.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420

56.

Simon

J. R.

Small

A. M.

(1969). Processing auditory information: Interference from an irrelevant cue. Journal of Applied Psychology, 53(5), 433–435. https://doi.org/10.1037/h0028034

57.

Stirling

(1979). Stroop Interference: An input and an output phenomenon. Quarterly Journal of Experimental Psychology, 31(1), 121–132. https://doi.org/10.1080/14640747908400712

58.

Stoet

(2010). PsyToolkit: A software package for programming psychological experiments using Linux. Behavior Research Methods, 42(4), 1096–1104. https://doi.org/10.3758/BRM.42.4.1096

59.

Stoet

(2017). PsyToolkit: A novel web-based method for running online questionnaires and reaction-time experiments. Teaching of Psychology, 44(1), 24–31. https://doi.org/10.1177/0098628316677643

60.

Szűcs

Soltész

White

(2009). Motor conflict in Stroop tasks: Direct evidence from single-trial electro-myography and electro-encephalography. NeuroImage, 47(4), 1960–1973. https://doi.org/10.1016/j.neuroimage.2009.05.048

61.

Tipper

S. P.

(2001). Does negative priming reflect inhibitory mechanisms? A review and integration of conflicting views. Quarterly Journal of Experimental Psychology Section A, 54(2), 321–343. https://doi.org/10.1080/713755969

62.

Tsal

Benoni

(2010). Diluting the burden of load: Perceptual load effects are simply dilution effects. Journal of Experimental Psychology: Human Perception and Performance, 36(6), 1645–1656. https://doi.org/10.1037/a0018172

63.

Van Der Linden

W. J

. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204. https://doi.org/10.3102/10769986031002181

64.

Van Veen

Carter

C. S

. (2005). Separating semantic conflict and response conflict in the Stroop task: A functional MRI study. NeuroImage, 27(3), 497–504. https://doi.org/10.1016/j.neuroimage.2005.04.042

65.

Von Bastian

C. C.

Blais

Brewer

G. A.

Gyurkovics

Hedge

Kałamała

Meier

M. E.

Oberauer

Rey-Mermet

Rouder

J. N.

Souza

A. S.

Bartsch

L. M.

Conway

A. R. A.

Draheim

Engle

R. W.

Friedman

N. P.

Frischkorn

G. T.

Gustavson

D. E.

Koch

Wiemers

E. A

. (2020). Advancing the understanding of individual differences in attentional control: Theoretical, methodological, and analytical considerations. PsyArXiv. https://doi.org/10.31234/osf.io/x3b9k

66.

Voormann

Miller

(2024). Sequential effects on reaction time distributions: Commonalities and differences across paradigms. Journal of Cognition, 7(1), 68. https://doi.org/10.5334/joc.395

67.

Wagenmakers

E.-J.

Brown

(2007). On the linear relation between the mean and the standard deviation of a response time distribution. Psychological Review, 114(3), 830–841. https://doi.org/10.1037/0033-295X.114.3.830

68.

Weissman

D. H.

Egner

Hawks

Link

(2015). The congruency sequence effect emerges when the distracter precedes the target. Acta Psychologica, 156, 8–21. https://doi.org/10.1016/j.actpsy.2015.01.003

69.

Wendt

Kiesel

Geringswald

Purmann

Fischer

(2014). Attentional adjustment to conflict strength: Evidence from the effects of manipulating flanker-target SOA on response times and prestimulus pupil size. Experimental Psychology, 61(1), 55–67. https://doi.org/10.1027/1618-3169/a000227

70.

Wheeler

D. D.

(1977). Locus of interference on the Stroop test. Perceptual and Motor Skills, 45(1), 263–266. https://doi.org/10.2466/pms.1977.45.1.263

71.

Yuan

Ren

Chen

(2020). Caffeine effect on cognitive function during a Stroop task: fNIRS study. Neural Plasticity, 2020, 1–8. https://doi.org/10.1155/2020/8833134

72.

Zhang

Kornblum

(1998). The effects of stimulus–response mapping and irrelevant stimulus–response and stimulus–stimulus overlap in four-choice Stroop tasks with single-carrier stimuli. Journal of Experimental Psychology: Human Perception and Performance, 24(1), 3–19. https://doi.org/10.1037/0096-1523.24.1.3

73.

Zhang

H. (Harry)

Zhang

Kornblum

(1999). A parallel distributed processing model of stimulus–stimulus and stimulus–response compatibility. Cognitive Psychology, 38(3), 386–432. https://doi.org/10.1006/cogp.1998.0703

74.

Zorowitz

Niv

(2023). Improving the reliability of cognitive task measures: A narrative review. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 8(8), 789–797. https://doi.org/10.1016/j.bpsc.2023.02.004

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.12 MB