Sequential evaluation is the hallmark of fair review: The same raters assess the merits of applicants, athletes, art, and more using standard criteria. We investigated one important potential contaminant in such ubiquitous decisions: Evaluations become more positive when conducted later in a sequence. In four studies, (a) judges’ ratings of professional dance competitors rose across 20 seasons of a popular television series, (b) university professors gave higher grades when the same course was offered multiple times, and (c) in an experimental test of our hypotheses, evaluations of randomly ordered short stories became more positive over a 2-week sequence. As judges completed repeated evaluations, they experienced more fluent decision making, producing more positive judgments (Study 4 mediation). This seemingly simple bias has widespread and impactful consequences for evaluations of all kinds. We also report four supplementary studies to bolster our findings and address alternative explanations.
AlterA. L.OppenheimerD. M. (2009). Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review, 13, 219–235. doi:10.1177/1088868309341564
2.
AlterA. L.OppenheimerD. M.EpleyN.EyreR. N. (2007). Overcoming intuition: Metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology: General, 136, 569–576. doi:10.1037/0096-3445.136.4.569
3.
BornsteinR. F. (1989). Exposure and affect: Overview and meta-analysis of research, 1968–1987. Psychological Bulletin, 106, 265–289. doi:10.1037/0033-2909.106.2.265
4.
BornsteinR. F.D’AgostinoP. R. (1994). The attribution and discounting of perceptual fluency: Preliminary tests of a perceptual fluency/attributional model of the mere exposure effect. Social Cognition, 12, 103–128. doi:10.1521/soco.1994.12.2.103
5.
ChoudhryN. K.FletcherR. H.SoumeraiS. B. (2005). Systematic review: The relationship between clinical experience and quality of health care. Annals of Internal Medicine, 142, 260–273. doi:10.7326/0003-4819-142-4-200502150-00008
6.
CsikszentmihalyiM.RathundeK. (1993). The measurement of flow in everyday life: Toward a theory of emergent motivation. In JacobsJ. E. (Ed.), Current theory and research in motivation, Vol. 40. Nebraska Symposium on Motivation, 1992: Developmental perspectives on motivation (pp. 57–97). Lincoln: University of Nebraska Press.
7.
DanzigerS.LevavJ.Avnaim-PessoL. (2011). Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences, USA, 108, 6889–6892. doi:10.1073/pnas.1018033108
8.
de BruinW. B. (2005). Save the last dance for me: Unwanted serial position effects in jury evaluations. Acta Psychologica, 118, 245–260. doi:10.1016/j.actpsy.2004.08.005
9.
DreyfusH. L.DreyfusS. E. (1986). Mind over machine: The power of human intuition and expertise in the era of the computer. New York, NY: Free Press.
10.
EricssonK. A. (2007). An expert-performance perspective of research on medical expertise: The study of clinical performance. Medical Education, 41, 1124–1130. doi:10.1111/j.1365-2923.2007.02946.x
11.
FiskeS. T.TaylorS. E. (2013). Social cognition: From brains to culture. Thousand Oaks, CA: SAGE.
12.
FuF.-L.SuR.-C.YuS.-C. (2009). EGameFlow: A scale to measure learners’ enjoyment of e-learning games. Computers & Education, 52, 101–112. doi:10.1016/j.compedu.2008.07.004
13.
HarteisC.BillettS. (2013). Intuitive expertise: Theories and empirical evidence. Educational Research Review, 9, 145–157. doi:10.1016/j.edurev.2013.02.001
14.
HayesA. (2013). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. New York, NY: Guilford Press.
15.
JiangY.HongJ. (2014). It feels fluent, but not right: The interactive effect of expected and experienced processing fluency on evaluative judgment. Journal of Experimental Social Psychology, 54, 147–152. doi:10.1016/j.jesp.2014.05.004
16.
KahnemanD.FrederickS. (2005). A model of heuristic judgment. In HolyoakK. J.MorrisonR. G. (Eds.), The Cambridge handbook of thinking and reasoning (pp. 267–293). Cambridge, England: Cambridge University Press.
17.
KahnemanD.KleinG. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64, 515–526. doi:10.1037/a0016755
18.
KahnemanD.TverskyA. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131. doi:10.1126/science.185.4157.1124
19.
LevavJ.HeitmannM.HerrmannA.IyengarS. S. (2010). Order in product customization decisions: Evidence from field experiments. Journal of Political Economy, 118, 274–299. doi:10.1086/652463
20.
LitmanL.RobinsonJ.AbberbockT. (2017). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49, 433–442. doi:10.3758/s13428-016-0727-z
21.
LittA.ReichT.MayminS.ShivB. (2011). Pressure and perverse flights to familiarity. Psychological Science, 22, 523–531. doi:10.1177/0956797611400095
22.
MarshH. W. (2007). Do university teachers become more effective with experience? A multilevel growth model of students’ evaluations of teaching over 13 years. Journal of Educational Psychology, 99, 775–790. doi:10.1037/0022-0663.99.4.775
23.
McGloneM. S.TofighbakhshJ. (2000). Birds of a feather flock conjointly (?): Rhyme as reason in aphorisms. Psychological Science, 11, 424–428. doi:10.1111/1467-9280.00282
24.
OppenheimerD. M. (2004). Spontaneous discounting of availability in frequency judgment tasks. Psychological Science, 15, 100–105. doi:10.1111/j.0963-7214.2004.01502005.x
25.
OppenheimerD. M. (2006). Consequences of erudite vernacular utilized irrespective of necessity: Problems with using long words needlessly. Applied Cognitive Psychology, 20, 139–156. doi:10.1002/acp.1178
26.
OppenheimerD. M. (2008). The secret life of fluency. Trends in Cognitive Sciences, 12, 237–241. doi:10.1016/j.tics.2008.02.014
27.
PettyR. E.BriñolP.TormalaZ. L. (2002). Thought confidence as a determinant of persuasion: The self-validation hypothesis. Journal of Personality and Social Psychology, 82, 722–741. doi:10.1037/0022-3514.82.5.722
28.
PreacherK. J.HayesA. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36, 717–731. doi:10.3758/BF03206553
29.
PrenticeD. A.MillerD. T. (1992). When small effects are impressive. Psychological Bulletin, 112, 160–164. doi:10.1037/0033-2909.112.1.160
30.
ReberR.SchwarzN.WinkielmanP. (2004). Processing fluency and aesthetic pleasure: Is beauty in the perceiver’s processing experience?Personality and Social Psychology Review, 8, 364–382. doi:10.1207/s15327957pspr0804_3
31.
SchwarzN. (2004). Metacognitive experiences in consumer judgment and decision making. Journal of Consumer Psychology, 14, 332–348. doi:10.1207/s15327663jcp1404_2
32.
TverskyA.KahnemanD. (1981). The framing of decisions and the psychology of choice. Science, 211, 453–458. doi:10.1126/science.7455683
33.
WatsonD.ClarkL. A.TellegenA. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54, 1063–1070.
34.
WerthL.StrackF. (2003). An inferential approach to the knew-it-all-along phenomenon. Memory, 11, 411–419. doi:10.1080/09658210244000586
35.
ZajoncR. B. (1968). Attitudinal effects of mere exposure. Journal of Personality and Social Psychology, 9(2), 1–27. doi:10.1037/h0025848
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
