Implicit measures have contributed to important insights in almost every area of psychology. However, various issues and challenges remain concerning their use, one of which is their considerable variation in reliability, with many implicit measures having questionable reliability. The goal of the present investigation was to examine an overlooked consequence of this liability with respect to replication, when such implicit measures are used as dependent variables in experimental studies. Using a Monte Carlo simulation, the authors demonstrate that a higher level of unreliability in such dependent variables is associated with substantially lower levels of replicability. The results imply that this overlooked consequence can have far-reaching repercussions for the development of a cumulative science. The authors recommend the routine assessment and reporting of the reliability of implicit measures and also urge the improvement of implicit measures with low reliability.
Algom, D., Chajut, E., & Shlomo, L. ( 2004). A rational look at the emotional Stroop phenomenon: A generic slowdown, not a Stroop effect. Journal of Experimental Psychology: General, 133, 323-338.
2.
Arkes, H.R., & Tetlock, P.E. ( 2004). Attributions of implicit prejudice, or would Jesse Jackson ‘fail’ the Implicit Association Test? Psychological Inquiry, 15, 257-278.
3.
Barden, J., Maddux, W.W., Petty, R.E., & Brewer, M.B. ( 2004). Contextual moderation of racial bias: The impact of social roles on controlled and automatically activated attitudes. Journal of Personality and Social Psychology, 87, 5-22.
4.
Bargh, J.A. ( 1999). The cognitive monster: The case against the controllability of automatic stereotype effects. In S. Chaiken & Y. Trope (Eds.), Dual-process theories in social psychology (pp. 361-382). New York, NY : Guilford.
5.
Baugh, F. ( 2002). Correcting effect sizes for score reliability: A reminder that measurement and substantive issues are linked inextricably. Educational and Psychological Measurement, 62, 254-263.
6.
Blanton, H., & Jaccard, J. ( 2006). Arbitrary metrics in psychology. American Psychologist, 61, 27-41.
7.
Blanton, H., & Jaccard, J. ( 2008). Unconscious racism: A concept in pursuit of a measure . Annual Review of Sociology, 34, 277-297.
8.
Blanton, H., Jaccard, J., Gonzales, P., & Christie, C. ( 2006). Decoding the Implicit Association Test: Perspectives on criterion prediction. Journal of Experimental Social Psychology , 42, 192-212.
9.
Blascovich, J., Loomis, J., Beall, A.C., Swinth, K.R., Hoyt, C.L., & Bailenson, J.N. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychology Inquiry, 13, 103-124.
10.
Borkenau, P., & Mauer, N. ( 2007). Well-being and the accessibility of pleasant and unpleasant concepts. European Journal of Personality, 21, 169-189.
11.
Borkenau, P., Paelecke, M., & Yu, R. ( 2009). Personality and lexical decision times for evaluative words. European Journal of Personality, 24, 123-136.
12.
Bosson, J.K., Lakey, C.E., Campbell, W.K., Zeigler-Hill, V., Jordan, C.H., & Kernis, M.H. ( 2008). Untangling the links between narcissism and self-esteem: A theoretical and empirical review. Personality and Social Psychology Compass, 2/3, 1415-1439.
13.
Bosson, J.K., Swann, W.B., & Pennebaker, J.W. (2000). Stalking the perfect measure of implicit self-esteem: The blind men and the elephant revisited? Journal of Personality and Social Psychology, 79, 631-643.
14.
Buchner, A., & Wippich, W. ( 2000). On the reliability of implicit and explicit memory measures . Cognitive Psychology, 40, 227-259.
Charles, E. ( 2005). The correction for attenuation due to measurement error: Clarifying concepts and creating confidence sets. Psychological Methods, 10, 206-226.
17.
Cohen, J. ( 1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
18.
Cronbach, L.J. ( 1951). Coefficient alpha and the internal structure of tests . Psychometrika, 12, 1-16.
19.
Cronbach, L.J. ( 1958). Proposals leading to analytic treatment of social perception scores. In R. Tagiuri & L. Petrullo (Eds.), Person perception and interpersonal behavior (pp. 353-379). Stanford, CA: Stanford University Press.
20.
Cronbach, L.J. ( 2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418.
21.
Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
22.
Cumming, G. ( 2008). Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286-300.
De Houwer, J. ( 2003). The Extrinsic Affective Simon Task. Experimental Psychology, 50, 77-85.
25.
De Houwer, J., & De Bruycker, E. (2007). The identification-EAST as a valid measure of implicit attitudes toward alcohol-related stimuli. Journal of Behavior Therapy and Experimental Psychiatry, 38, 133-143.
26.
De Raedt, R., Schacht, R., Franck, E., & De Houwer, J. ( 2006). Self-esteem and depression revisited: Implicit positive self-esteem in depressed patients? Behaviour Research and Therapy, 44, 1017-1028.
27.
DeShon, R.P. ( 1998). A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3, 412-423.
28.
DeShon, R.P. ( 2004). Measures are not invariant across groups without error variance homogeneity. Psychology Science, 46, 137-149.
29.
Dovidio, J.F., Kawakami, K., & Gaertner, S.L. (2002). Implicit and explicit prejudice and interracial interaction. Journal of Personality and Social Psychology, 82, 62-28.
30.
Duhachek, A., & Iacobucci, D. ( 2004). Alpha’s standard error (ASE): An accurate and precise confidence interval estimate. Journal of Applied Psychology , 89, 792-808.
31.
Edwards, J.R. ( 2002). Ten difference score myths. Organizational Research Methods, 4, 265-287.
32.
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. ( 2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.
33.
Fazio, R.H., Jackson, J.R., Dunton, B.C., & Williams, C.J. (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline ? Journal of Personality and Social Psychology, 69, 1013-1027.
34.
Fazio, R.H., & Olson, M.A. ( 2003). Implicit measures in social cognition research: Their meaning and use. Annual Review of Psychology, 54, 297-327.
35.
Ferguson, M.J., & Bargh, J.A. ( 2007). Beyond the attitude object: Implicit attitudes spring from object-centered contexts. In B. Wittenbrink & N. Schwarz (Eds.), Implicit measures of attitudes: Procedures and controversies (pp. 216-246). New York, NY: Guilford.
Gawronski, B. ( 2002). What does the Implicit Association Test measure? A test of the convergent and discriminant validity of prejudice related IATs. Experimental Psychology, 49, 171-180.
38.
Gawronski, B., Cunningham, W.A., LeBel, E.P., & Deutsch, R. ( 2010). Attentional influences on affective priming: Does categorization influence spontaneous evaluations of multiply categorizable objects?Cognition and Emotion, 24, 1008-1025.
39.
Gawronski, B., Deutsch, R., & Banse, R. (in press). Response interference tasks as indirect measures of automatic associations. In K. C. Klauer, C. Stahl, & A. Voss (Eds.), Cognitive methods in social psychology. New York, NY: Guilford.
40.
Gawronski, B., Deutsch, R., LeBel, E.P., & Peters, K.R. ( 2008). Response interference as a mechanism underlying implicit measures: Some traps and gaps in the assessment of mental associations with experimental paradigms. European Journal of Psychological Assessment, 24, 218-225.
41.
Gawronski, B., LeBel, E.P., & Peters, K.P. ( 2007). What do implicit measures tell us? Scrutinizing the validity of three common assumptions. Perspectives on Psychological Science, 2, 181-193.
42.
Gray, N.S., McCulloch, M.J., Smith, J., Morris, M., & Snowden, R.J. ( 2003). Violence viewed by psychopathic murderers: Adapting a revealing test may expose those psychopaths who are most likely to kill. Nature, 423, 497-498.
43.
Greenwald, A.G., & Banaji, M.R. ( 1995). Implicit social cognition: Attitudes, self-esteem, and stereotypes. Psychological Review, 102, 4-27.
44.
Greenwald, A.G., McGhee, D.E., & Schwartz, J.L.K. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464-1480.
45.
Greenwald, A.G., Nosek, B.A., & Banaji, M.R. ( 2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197-216.
46.
Hopkins, K.D., & Hopkins, B.R. ( 1979). The effect of the reliability of the dependent variable on power. Journal of Special Education, 13, 463-466.
47.
Jaccard, J., & Wan, C.K. ( 1995). Measurement error in the analysis of interaction effects between continuous predictors using multiple regression: Multiple indicator and structural equation approaches. Psychological Bulletin , 117, 348-357.
48.
Johnson, R.E., & Steinman, L. ( 2009). Use of implicit measures for organizational research: An empirical example. Canadian Journal of Behavioural Science , 41, 202-212.
49.
Jordan, C.H., Spencer, S.J., Zanna, M.P., Hoshino-Browne, E., & Correll, J. ( 2003). Secure and defensive high self-esteem. Journal of Personality and Social Psychology, 85, 969-978.
50.
Kashy, D.A., Donnellan, M.B., Ackerman, R.A., & Russell, D.W. ( 2009). Reporting and interpreting research in PSPB: Practices, principles, and pragmatics. Personality and Social Psychology Bulletin, 35, 1131-1142.
51.
Kawakami, K., & Dovidio, J.F. ( 2001). Implicit stereotyping: How reliable is it?Personality and Social Psychology Bulletin, 27, 212-225.
52.
Kopriva, R.J., & Shaw, D.G. ( 1991). Power estimates: The effect of dependent variable reliability on the power of one-factor ANOVAs. Educational and Psychological Measurement, 51, 585-595.
53.
LeBel, E.P., & Gawronski, B. ( 2009). How to find what’s in a name: Scrutinizing the optimality of five scoring algorithms for the name-letter task. European Journal of Personality, 23, 85-106.
54.
Lipsey, M.W., & Wilson, D.B. ( 1993). The efficacy of psychological, education, and behavioral treatment: Confirmation from meta-analysis. American Psychologist , 48, 1181-1209.
55.
Liu, Y., & Zumbo, B.D. ( 2007). The impact of outliers on Cronbach’s coefficient alpha estimate of reliability. Educational and Psychological Measurement, 67, 620-634.
56.
Lord, F.M., & Novick, M.R. ( 1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
57.
Meier, B., & Perrig, W.J. ( 2000). Low reliability of perceptual priming: Consequences for the interpretation of functional dissociations between explicit and implicit memory. Quarterly Journal of Experimental Psychology, 53, 211-233.
58.
Nosek, B.A., & Banaji, M.R. ( 2001). The Go/No-go Association Task. Social Cognition, 19, 625-666.
59.
Nunnally, J.C. ( 1982). Reliability of measurement. In H. E. Mitzel (Ed.), Encyclopedia of educational research (pp. 1589-1601). New York, NY: Free Press.
60.
Nuttin, M.J., Jr. ( 1985). Narcissism beyond gestalt and awareness: The name letter effect. European Journal of Social Psychology, 64, 723-739.
61.
Olson, M.A., & Fazio, R.H. ( 2003). Relations between implicit measures of prejudice: What are we measuring? Psychological Science, 14, 636-639.
62.
Papies, E.K., Stroebe, W., & Aarts, H. ( 2009). Who likes it more? Restrained eaters’ implicit attitudes towards food. Appetite, 53, 279-287.
63.
Paunonen, S.V. ( 1984). The reliability of aggregated measurements: Lessons to be learned from psychometric theory. Journal of Research in Personality, 18, 383-394.
64.
Payne, B.K., Cheng, S.M., Govorun, O., & Stewart, B.D. ( 2005). An inkblot for attitudes: Affect misattribution as implicit measurement. Journal of Personality and Social Psychology , 89, 277-293.
65.
Perez, L.A., Peynircio, Z.F., & Blaxton, T.A. ( 1998). Developmental differences in implicit and explicit memory performance. Journal of Experimental Child Psychology, 70, 167-185.
66.
Robie, C., & Komar, S.G. ( 2007). Simulation, computer approach. In S. G. Rogelberg (Ed.), Encyclopedia of industrial and organizational psychology (pp. 723-724). Thousand Oaks, CA: Sage.
67.
Rogers, W.T., & Hopkins, K.D. ( 1988). Power estimates in the presence of a covariate and measurement error. Educational and Psychological Measurement, 48, 647-56.
68.
Schimmack, U., & Diener, E. ( 2003). Predictive validity of explicit and implicit self-esteem for subjective well-being. Journal of Research in Personality , 37, 100-106.
69.
Schmidt, F.L., & Hunter, J.E. ( 1996). Measurement error in psychological research: Lessons from 27 research scenarios. Psychological Methods, 1, 199-223.
70.
Schmukle, S.C. ( 2005). Unreliability of the dot probe task. European Journal of Personality, 19, 595-605.
71.
Schott, B.H., Henson, R.N., Richardson-Klavehn, A., Becker, C., Thoma, V., Heinze, H.-J., & Düzel, E. (2005). Redefining implicit and explicit memory: The functional neuroanatomy of priming, remembering, and control of retrieval. Proceedings of National Academy Science , 102, 1257-1262.
72.
Spruyt, A., Hermans, D., De Houwer, J., Vandekerckhove, J., & Eelen, P. ( 2007). On the predictive validity of indirect attitude measures: Prediction of consumer choice behavior on the basis of affective priming in the picture-picture naming task. Journal of Experimental Social Psychology, 43, 599-610.
73.
Sutcliffe, J.P. ( 1958). Error of measurement and the sensitivity of a test of significance. Psychometrika, 23, 9-17.
74.
Teige, S., Schnabel, K., Banse, R., & Asendorpf, J.B. (2004). Assessment of multiple implicit self-concept dimensions using the Extrinsic Affective Simon Task . European Journal of Personality, 18, 495-520.
75.
Thye, S.R. ( 2000). Reliability in experimental sociology. Social Forces, 78, 1277-1309.
76.
Uhlmann, E.L., Pizarro, D.A., & Bloom, P. ( 2008). Varieties of unconscious social cognition. Journal for the Theory of Social Behaviour, 38, 293-322.
77.
Williams, R.H., & Zimmerman, D.W. (1989). Statistical power analysis and reliability of measurement. Journal of General Psychology , 116, 359-369.
78.
Wittenbrink, B. ( 2007). Measuring attitudes through priming. In B. Wittenbrink & N. Schwarz (Eds.), Implicit measures of attitudes: Procedures and controversies (pp. 17-58). New York, NY: Guilford .
79.
Wittenbrink, B., Judd, C.M., & Park, B. ( 1997). Evidence for racial prejudice at the implicit level and its relationship with questionnaire measures. Journal of Personality and Social Psychology, 72, 262-274.
80.
Wittenbrink, B., & Schwarz, N. (Eds.). (2007). Implicit measures of attitudes . New York, NY: Guilford.
81.
Woltz, D.J., & Shute, V.J. ( 1993). Individual difference in repetition priming and its relationship to declarative knowledge acquisition. Intelligence, 17, 333-359.
82.
Zeigler-Hill, V. ( 2006). Discrepancies between implicit and explicit self-esteem: Implications for narcissism and self-esteem instability. Journal of Personality, 74, 119-143.
