The use of the Implicit Association Test (IAT) as a measure of individual differences is stymied by insufficient test–retest reliability for assessing trait-level constructs. We assess the degree to which the IAT measures individual differences and test a method to improve its validity as a “trait” measure: aggregating across IATs. Across three studies, participants (total n = 960) completed multiple IATs in the same session or across multiple sessions. Using latent-variable models, we found that half of the variance in IAT scores reflects individual differences. Aggregating across multiple IATs approximately doubled the variance explained with explicit measures compared with a single IAT D-score. These findings show that IAT scores contain considerable noise and that a single IAT is inadequate to estimate trait bias. However, aggregation across multiple administrations can correct this and better estimate individual differences in implicit attitudes.
AgerströmJ.RoothD.-O. (2011). The role of automatic obesity stereotypes in real hiring discrimination. Journal of Applied Psychology, 96(4), 790–805. http://doi.org/10.1037/a0021594
2.
AustF.DiedenhofenB.UllrichS.MuschJ. (2013). Seriousness checks are useful to improve data validity in online research. Behavior Research Methods, 45(2), 527–535.
3.
AxtJ. R. (2018). The best way to measure explicit racial attitudes is to ask about them. Social Psychological and Personality Science, 9(8), 896–906. https://doi.org/10.1177/1948550617728995
4.
BaronA. S. (2015). Constraints on the development of implicit intergroup attitudes. Child Development Perspectives, 9(1), 50–54. https://doi.org/10.1111/cdep.12105
5.
BrickC.LaiC. K. (2018). Explicit (but not implicit) environmentalist identity predicts pro-environmental behavior and policy preferences. Journal of Environmental Psychology, 58, 8–17. https://doi.org/10.1016/j.jenvp.2018.07.003
6.
BrownT. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). Guilford.
7.
BrowneW. M.CudeckR. (1993). Alternative ways of assessing model fit. In BollenK. A.LongJ. S. (Eds.), Testing structural equation models (pp. 136–162). SAGE.
8.
BrownsteinM.MadvaA.GawronskiB. (2019). What do implicit measures measure?WIREs Cognitive Science, 10(5), Article e1501. https://doi.org/10.1002/wcs.1501
9.
ButtrickN.AxtJ.EbersoleC. R.HubandJ. (2020). Re-assessing the incremental predictive validity of Implicit Association Tests. Journal of Experimental Social Psychology, 88, 103941. https://doi.org/10.1016/j.jesp.2019.103941
10.
CarpenterT. P.PogacarR.PulligC.KourilM.AguilarS.LaBouffJ.IsenbergN.ChakroffA. (2019). Survey-software implicit association tests: A methodological and empirical analysis. Behavior Research Methods, 51, 2194–2208. https://doi.org/10.3758/s13428-019-01293-3
11.
CicchettiD. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290.
12.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.
13.
CohenJ.CohenP.WestS. G.AikeL. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Lawrence Erlbaum.
14.
ColegraveN.RuxtonG. D. (2003). Confidence intervals are a more useful complement to nonsignificant tests than are power calculations. Behavioral Ecology, 14(3), 446–447. https://doi.org/10.1093/beheco/14.3.446
15.
ConnorP.EversE. R. K. (2020). The bias of individuals (in crowds): Why implicit bias is probably a noisily measured individual-level construct. Perspectives on Psychological Science, 15(6), 1329–1345.
16.
CronbachL. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.
17.
CunninghamW. A.PreacherK. J.BanajiM. R. (2001). Implicit attitude measures: Consistency, stability, and convergent validity. Psychological Science, 12, 163–170. https://doi.org/10.1111/1467-9280.00328
18.
De HouwerJ. (2001). A structural and process analysis of the Implicit Association Test. Journal of Experimental Social Psychology, 37(6), 443–451. https://doi.org/10.1006/jesp.2000.1464
19.
De HouwerJ. (2014). A propositional model of implicit evaluation. Social and Personality Psychology Compass, 8, 342–353.
20.
De HouwerJ. (2019). Implicit bias is behavior: A functional-cognitive perspective on implicit bias. Perspectives on Psychological Science, 14, 835–840. https://doi.org/10.1177/1745691619855638
21.
EndersC. K.BandalosD. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8, 430–457. https://doi.org/10.1207/S15328007SEM0803_5
22.
FatfoutaR.Schröder-AbéM. (2018). Agentic to the core? Facets of narcissism and positive implicit self-views in the agentic domain. Journal of Research in Personality, 74, 78–82. https://doi.org/10.1016/j.jrp.2018.02.006
23.
FinniganK. M.VazireS. (2018). The incremental validity of average state self-reports over global self-reports of personality. Journal of Personality and Social Psychology, 115(2), 321–337. https://doi.org/10.1037/pspp0000136
24.
ForscherP. S.CoxW. T. L.GraetzN.DevineP. G. (2015). The motivation to express prejudice. Journal of Personality and Social Psychology, 109(5), 791–812.
25.
ForscherP. S.LaiC. K.AxtJ. R.EbersoleC. R.HermanM.DevineP. G.NosekB. A. (2019). A meta-analysis of procedures to change implicit measures. Journal of Personality and Social Psychology, 117(3), 522–559. https://doi.org/10.1037/pspa0000160
26.
FrieseM.WänkeM.PlessnerH. (2006). Implicit consumer preferences and their influence on product choice. Psychology & Marketing, 23, 727–740. https://doi.org/10.1002/mar.20126
27.
GawronskiB.BodenhausenG. V. (2006). Associative and propositional processes in evaluation: An integrative review of implicit and explicit attitude change. Psychological Bulletin, 132(5), 692–731. https://doi.org/10.1037/0033-2909.132.5.692
28.
GawronskiB.BodenhausenG. V. (2011). The associative-propositional evaluation model: Theory, evidence, and open questions. In OlsonJ. M.ZannaM. P. (Eds.), Advances in experimental social psychology (Vol. 44, pp. 59–127). Academic Press.
29.
GawronskiB.BodenhausenG. V. (2017). Beyond persons and situations: An interactionist approach to understanding implicit bias. Psychological Inquiry, 28(4), 268–272.
30.
GawronskiB.LeBelE. P.PetersK. R. (2007). What do implicit measures tell us? Scrutinizing the validity of three common assumptions. Perspectives on Psychological Science, 2(2), 181–193. https://doi.org/10.1111/j.1745-6916.2007.00036.x
31.
GawronskiB.MorrisonM.PhillsC. E.GaldiS. (2017). Temporal stability of implicit and explicit measures: A longitudinal analysis. Personality and Social Psychology Bulletin, 43, 300–312. https://doi.org/10.1177/0146167216684131
32.
GeiserC. (2012). Data analysis with Mplus. Guilford Press.
33.
GeldhofG. J.PreacherK. J.ZyphurM. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138
34.
GervaisW. M.ShariffA. F.NorenzayanA. (2011). Do you believe in atheists? Distrust is central to anti-atheist prejudice. Journal of Personality and Social Psychology, 101(6), 1189–1206. https://doi.org/10.1037/a0025882
35.
GloverD.PallaisA.ParienteW. (2017). Discrimination as a self-fulfilling prophecy: Evidence from French grocery stores. The Quarterly Journal of Economics, 132(3), 1219–1260. http://doi.org/10.1093/qje/qjx006
36.
GreenwaldA. G.BanajiM. R. (1995). Implicit social cognition: Attitudes, self-esteem, and stereotypes. Psychological Review, 102, 4–27. https://doi.org/10.1037/0033-295X.102.1.4
37.
GreenwaldA. G.BrendlM.CaiH.CvencekD.DovidioJ. F.FrieseM.HahnA.HehmanE.HofmannW.HughesS.HusseyI.JordanC.KirbyT. A.LaiC. K.LangJ. W. B.LindgrenK. P.MaisonD.OstafinB. D.RaeJ. R.. . .WiersR. W. (2021). Best research practices for using the Implicit Association Test. Behavior Research Methods. Advance online publication. https://doi.org/10.3758/s13428-021-01624-3
38.
GreenwaldA. G.BrendlM.CaiH.CvencekD.DovidioJ. F.FrieseM.HahnA.HehmanE.HofmannW.HughesS.HusseyI.JordanC.KirbyT. A.LaiC. K.LangJ. W. B.LindgrenK. P.MaisonD.OstafinB. D.RaeJ. R.WiersR. W. (2022). Best research practices for using the Implicit Association Test. Behavior Research Methods, 54(3), 1161–1180.
39.
GreenwaldA. G.LaiC. K. (2020). Implicit social cognition. Annual Review of Psychology, 71, 419–445.
40.
GreenwaldA. G.McGheeD. E.SchwartzJ. L. K. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464–1480. https://doi.org/10.1037/0022-3514.74.6.1464
41.
GreenwaldA. G.NosekB. A.BanajiM. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197–216. https://doi.org/10.1037/0022-3514.85.2.197
42.
HahnA.JuddC. M.HirshH. K.BlairI. V. (2014). Awareness of implicit attitudes. Journal of Experimental Psychology: General, 143(3), 1369–1392. https://doi.org/10.1037/a0035028
43.
HehmanE.CalanchiniJ.FlakeJ. K.LeitnerJ. B. (2019). Establishing construct validity evidence for regional measures of explicit and implicit racial bias. Journal of Experimental Psychology: General, 148(6), 1022–1040. https://doi.org/10.1037/xge0000623
44.
HofmannW.GawronskiB.GschwendnerT.LeH.SchmittM. (2005). A meta-analysis on the correlation between the Implicit Association Test and explicit self-report measures. Personality and Social Psychology Bulletin, 31, 1369–1385. https://doi.org/10.1177/0146167205275613
45.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. https://doi.org/10.1080/10705519909540118
46.
KlineR. B. (2015). Principles and practice of structural equation modeling (4th ed.). Guilford Press.
47.
KurdiB.MannT. C.CharlesworthT. E. S.BanajiM. R. (2019). The relationship between implicit intergroup attitudes and beliefs. PNAS Proceedings of the National Academy of Sciences of the United States of America, 116(13), 5862–5871. https://doi.org/10.1073/pnas.1820240116
48.
LaiC. K.MariniM.LehrS. A.CerrutiC.ShinJ. L.Joy-GabaJ. A.HoA. K.TeachmanB. A.WojcikS. P.KolevaS. P.FrazierR. S.HeiphetzL.ChenE.TurnerR. N.HaidtJ.KesebirS.HawkinsC. B.SchaeferH. S.RubichiS.SartoriG.DialC. M.SriramN.BanajiM. R.NosekB. A. (2014). Reducing implicit racial preferences: I. A comparative investigation of 17 interventions. Journal of Experimental Psychology: General, 143, 1765–1785. https://doi.org/10.1037/a0036260
49.
LaiC. K.SkinnerA. L.CooleyE.MurrarS.BrauerM.DevosT.CalanchiniJ.XiaoY. J.PedramC.MarshburnC. K.SimonS.BlancharJ. C.Joy-GabaJ. A.ConwayJ.RedfordL.KleinR. A.RoussosG.SchellhaasF. M. H.BurnsM.. . .NosekB. A. (2016). Reducing implicit racial preferences: II. Intervention effectiveness across time. Journal of Experimental Psychology: General, 145, 1001–1016. https://doi.org/10.1037/xge0000179
50.
LaiC. K.WilsonM. E. (2021). Measuring implicit intergroup biases. Social and Personality Psychology Compass, 15(1), Article e12573. https://doi.org/10.1111/spc3.12573
51.
LaneK. A.BanajiM. R.NosekB. A.GreenwaldA. G. (2007). Understanding and using the Implicit Association Test: IV: What we know (so far) about the method. In WittenbrinkB.SchwarzN. (Eds.), Implicit measures of attitudes (pp. 59–102). Guilford Press.
52.
MathesonG. J. (2019). We need to talk about reliability: Making better use of test-retest studies for study design and interpretation. PeerJ, 7, Article e6918. https://doi.org/10.7717/peerj.6918
53.
McDonaldR. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum
54.
MuthénL. K.MuthénB. O. (1998). Mplus user’s guide (7th ed.).
NosekB. A.Bar-AnanY.SriramN.AxtJ.GreenwaldA. G. (2014). Understanding and using the brief implicit association test: Recommended scoring procedures. PLoS ONE, 9(12), e110938. https://doi.org/10.1371/journal.pone.0110938
57.
NosekB. A.GreenwaldA. G.BanajiM. R. (2005). Understanding and using the Implicit Association Test: II. Method variables and construct validity. Personality and Social Psychology Bulletin, 31(2), 166–180.
58.
NunnallyJ. C. (1967). Psychometric theory. McGraw-Hill.
59.
PayneB. K.VuletichH. A.LundbergK. B. (2017). The bias of crowds: How implicit bias bridges personal and systemic prejudice. Psychological Inquiry, 28(4), 233–248. https://doi.org/10.1080/1047840X.2017.1335568
60.
PeruginiM. (2005). Predictive models of implicit and explicit attitudes. British Journal of Social Psychology, 44, 29–45. https://doi.org/10.1348/014466604X23491
61.
PettyR. E.TormalaZ. L.BriñolP.JarvisW. B. G. (2006). Implicit ambivalence from attitude change: An exploration of the PAST model. Journal of Personality and Social Psychology, 90(1), 21–41. https://doi.org/10.1037/0022-3514.90.1.21
62.
PlantE. A.DevineP. G. (1998). Internal and external motivation to respond without prejudice. Journal of Personality and Social Psychology, 75(3), 811–832. https://doi.org/10.1037/0022-3514.75.3.811
63.
RauthmannJ. F.HorstmannK. T.ShermanR. A. (2018). Do self-reported traits and aggregated states capture the same thing? A nomological perspective on trait-state homomorphy. Social Psychological and Personality Science, 10(5), 596–611. https://doi.org/10.1177/1948550618774772
RudmanL. A.PhelanJ. E.HeppenJ. B. (2007). Developmental sources of implicit attitudes. Personality and Social Psychology Bulletin, 33(12), 1700–1713. https://doi.org/10.1177/0146167207307487
66.
RydellR. J.McConnellA. R. (2006). Understanding implicit and explicit attitude change: A systems of reasoning analysis. Journal of Personality and Social Psychology, 91(6), 995–1008. https://doi.org/10.1037/0022-3514.91.6.995
67.
SchwarzN. (2007). Attitude construction: Evaluation in context. Social Cognition, 25, 638–656.
68.
SmithE. R. (1996). What do connectionism and social psychology offer each other?Journal of Personality and Social Psychology, 70(5), 893–912.
69.
SpearmanC. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72–101.
70.
SteyerR.FerringD.SchmittM. J. (1992). States and traits in psychological assessment. European Journal of Psychological Assessment, 8(2), 79–98.
StrackF.DeutschR. (2004). Reflective and impulsive determinants of social behavior. Personality and Social Psychology Review, 8, 220–247. https://doi.org/10.1207/s15327957pspr0803_1
73.
StrackF.DeutschR. (2015). The duality of everyday life: Dual-process and dual system models in social psychology. In MikulincerM.ShaverP. R.BorgidaE.BarghJ. A. (Eds.), APA handbook of personality and social psychology: Vol. 1. Attitudes and social cognition (pp. 891–927). American Psychological Association.
f
YuanK.-H.BentlerP. M. (2000). Three likelihood-based methods for mean and covariance structure analysis with nonnormal missing data. Sociological Methodology, 30(1), 165–200. https://doi.org/10.1111/0081-1750.00078
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
