Multimodal Cues to Change Your Mind: The Intertwining of Faces,Voices,and Behaviors in Impression Updating

Abstract

Impressions of others are formed from multiple cues, including facial features, vocal tone, and behavioral descriptions, and may be subject to multimodal updating. Four experiments (N = 803) examined the influence of a target’s face or voice on impression updating. Experiments 1a-1b examined whether behavior-based impressions are susceptible to updating by incongruent information conveyed by the target’s face, voice, or behavior (within-participant manipulation). Both faces and voices updated impressions with comparable strength, but less than behaviors. Experiment 2, contrasting faces and voices only (between-participants manipulation), showed that voices outperformed faces regardless of how impressions were formed (i.e., via behavioral vs. nonbehavioral information). Experiment 3 found no difference when comparing faces and voices in a within-participant design and controlling for stimulus attractiveness. Our work highlights the importance of multimodal cues for impression updating and shows that the relative power of faces and voices depends on contextual factors.

Keywords

impression updating face voice behavior multimodal perception

Get full access to this article

View all access options for this article.

References

Ambady

Bernieri

F. J.

Richeson

J. A.

(2000). Toward a histology of social behavior: Judgmental accuracy from thin slices of the behavioral stream. Advances in Experimental Social Psychology, 32, 201–271. https://doi.org/10.1016/S0065-2601(00)80006-4

Asch

S. E.

Zukier

(1984). Thinking about persons. Journal of Personality and Social Psychology, 46(6), 1230–1240. https://doi.org/10.1037/0022-3514.46.6.1230

Barker

Imhoff

(2021). The dynamic interactive pattern of assimilation and contrast: Accounting for standard extremity in comparative evaluations. Journal of Experimental Social Psychology, 97, 104190. https://doi.org/10.1016/j.jesp.2021.104190

Barr

D. J.

Levy

Scheepers

Tily

H. J.

(2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001

Baumeister

R. F.

Bratslavsky

Finkenauer

Vohs

K. D.

(2001). Bad is stronger than good. Review of General Psychology, 5(4), 323–370. https://doi.org/10.1037/1089-2680.5.4.323

Brambilla

Carraro

Castelli

Sacchi

(2019). Changing impressions: Moral character dominates impression updating. Journal of Experimental Social Psychology, 82, 64–73. https://doi.org/10.1016/j.jesp.2019.01.003

Brambilla

Sacchi

Rusconi

Goodwin

G. P.

(2021). The primacy of morality in impression development: Theory, research, and future directions. Advances in Experimental Social Psychology, 64, 187–262. https://doi.org/10.1016/bs.aesp.2021.03.001

Cone

Ferguson

M. J.

(2015). He did what? The role of diagnosticity in revising implicit evaluations. Journal of Personality and Social Psychology, 108(1), 37–57. https://doi.org/10.1037/pspa0000014

Cone

Mann

T. C.

Ferguson

M. J.

(2017). Changing our implicit minds: How, when, and why implicit evaluations can be rapidly revised. Advances in Experimental Social Psychology, 56, 131–199. https://doi.org/10.1016/bs.aesp.2017.03.001

10.

De Gelder

Bertelson

. (2003). Multisensory integration, perception and ecological validity. Trends in Cognitive Sciences, 7(10), 460–467. https://doi.org/10.1016/j.tics.2003.08.014

11.

Fasoli

Maass

Paladino

M. P.

Sulpizio

(2017). Gay-and lesbian-sounding auditory cues elicit stereotyping and discrimination. Archives of Sexual Behavior, 46, 1261–1277. https://doi.org/10.1007/s10508-017-0962-0

12.

Faul

Erdfelder

Lang

A. G.

Buchner

(2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146

13.

Fiske

S. T.

(1992). Thinking is for doing: Portraits of social cognition from Daguerreotype to laserphoto. Journal of Personality and Social Psychology, 63(6), 877–889. https://doi.org/10.1037/0022-3514.63.6.877

14.

Foo

Y. Z.

Sutherland

C. A. M.

Burton

N. S.

Nakagawa

Rhodes

(2021). Accuracy in facial trustworthiness impressions: Kernel of truth or modern physiognomy? A meta-analysis. Personality and Social Psychology Bulletin, 48, 1580–1596.

15.

Freeman

J. B.

Stolier

R. M.

Brooks

J. A.

(2020). Dynamic interactive theory as a domain-general account of social perception. Advances in Experimental Social Psychology, 61, 237–287. https://doi.org/10.1016/bs.aesp.2019.09.005

16.

Funder

D. C.

Ozer

D. J.

(2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://doi.org/10.1177/2515245919847202

17.

Gluszek

Dovidio

J. F.

(2010). The way they speak: A social psychological perspective on the stigma of nonnative accents in communication. Personality and Social Psychology Review, 14(2), 214–237. https://doi.org/10.1177/1088868309359288

18.

Goller

Leder

Cursiter

Jenkins

(2018). Anchoring effects in facial attractiveness. Perception, 47(10–11), 1043–1053. https://doi.org/10.1177/0301006618802696

19.

Green

MacLeod

C. J.

(2016). SIMR: An R package for power analysis of generalized linear mixed models by simulation. Methods in Ecology and Evolution, 7(4), 493–498. https://doi.org/10.1111/2041-210X.12504

20.

Hansen

Rakić

Steffens

M. C.

(2017). Competent and warm? Experimental Psychology, 64(1), 27–36. https://doi.org/10.1027/1618-3169/a000348

21.

Hansen

Rakić

Steffens

M. C.

(2018). Foreign-looking native-accented people: More competent when first seen rather than heard. Social Psychological and Personality Science, 9(8), 1001–1009. https://doi.org/10.1177/1948550617732389

22.

Jiang

Yang

Liu

Yue

. . . Xue

(2024). Comparison of face-based and voice-based first impressions in a Chinese sample. British Journal of Psychology, 115(1), 20–39. https://doi.org/10.1111/bjop.12675

23.

Johnson

K. L.

Lick

D. J.

Carpinella

C. M.

(2015). Emergent research in social vision: An integrated approach to the determinants and consequences of social categorization. Social and Personality Psychology Compass, 9(1), 15–30. https://doi.org/10.1111/spc3.12147

24.

Klofstad

C. A.

Anderson

R. C.

Peters

(2012). Sounds like a winner: Voice pitch influences perception of leadership capacity in both men and women. Proceedings of the Royal Society B: Biological Sciences, 279(1738), 2698–2704. https://doi.org/10.1098/rspb.2012.0311

25.

S. J.

Judd

C. M.

Stapel

D. A.

(2009). Stereotyping based on voice in the presence of individuating information: Vocal femininity affects perceived competence but not warmth. Personality and Social Psychology Bulletin, 35(2), 198–211. https://doi.org/10.1177/0146167208326477

26.

Kraus

M. W.

(2017). Voice-only communication enhances empathic accuracy. American Psychologist, 72(7), 644–654. https://doi.org/10.1037/amp0000147

27.

Kruse

Degner

(2021). Spontaneous state inferences. Journal of Personality and Social Psychology, 121(4), 774–791. https://doi.org/10.1037/pspa0000232

28.

Kuznetsova

Brockhoff

P. B.

Christensen

R. H. B.

(2017). LmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–26. https://doi.org/10.18637/jss.v082.i13

29.

Lakens

(2022). Sample size justification. Collabra: Psychology, 8(1), 33267. https://doi.org/10.1525/collabra.33267

30.

Lan

Peng

Zhao

Chen

Liu

Yang

(2022). Facial attractiveness is more associated with individual warmth than with competence: Behavioral and neural evidence. Social Neuroscience, 17(3), 225–235. https://doi.org/10.1080/17470919.2022.2069152

31.

Latinus

Belin

(2011). Human voice perception. Current Biology, 21(4), R143–R145. https://doi.org/10.1016/j.cub.2010.12.033

32.

Lavan

Mileva

Burton

A. M.

Young

A. W.

McGettigan

(2021). Trait evaluations of faces and voices: Comparing within- and between-person variability. Journal of Experimental Psychology: General, 150(9), 1854–1869. https://doi.org/10.1037/xge0001019

33.

Lenth

R. V.

(2022). Emmeans: Estimated marginal means, aka least-squares means (R package, Version 1.8.1-1). https://CRAN.R-project.org/package=emmeans

34.

Liu

Sommer

Yue

(2023). Dominance of face over voice in human attractiveness judgments: ERP evidence. Psychophysiology, 60(11), Article e14358. https://doi.org/10.1111/psyp.14358

35.

D. S.

Correll

Wittenbrink

(2015). The Chicago face database: A free stimulus set of faces and norming data. Behavior Research Methods, 47(4), 1122–1135. https://doi.org/10.3758/s13428-014-0532-5

36.

Masi

Fasoli

(2022). When fluency matters: The interplay between categorization fluency and gender atypicality on gaydar judgments. Journal of Language and Social Psychology, 41(6), 746–759. https://doi.org/10.1177/0261927X221111382

37.

Masi

Mattavelli

Fasoli

Brambilla

(2022). Cross-modal impression updating: Dynamic impression updating from face to voice and the other way around. British Journal of Social Psychology, 61(3), 808–825. https://doi.org/10.1111/bjso.12511

38.

McAleer

Belin

(2018). The perception of personality traits from voices. In Fruholz

Belin

(Eds.), The Oxford handbook of voice perception (pp. 585–606). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780198743187.013.26

39.

McAleer

Todorov

Belin

(2014). How do you say ‘hello’? Personality impressions from brief novel voices. PLoS ONE, 9(3), Article e90779. https://doi.org/10.1371/journal.pone.0090779

40.

McArthur

L. Z.

Baron

R. M.

(1983). Toward and ecological theory of social perception. Psychological Review, 90(3), 215–238. https://doi.org/10.1037/0033-295X.90.3.215

41.

McCarthy

R. J.

Skowronski

J. J.

(2011). What will Phil do next? Spontaneously inferred traits influence predictions of behavior. Journal of Experimental Social Psychology, 47(2), 321–332. https://doi.org/10.1016/j.jesp.2010.10.015

42.

Mileva

Lavan

(2023). Trait impressions from voices are formed rapidly within 400 ms of exposure. Journal of Experimental Psychology. General, 152(6), 1539–1550. https://doi.org/10.1037/xge0001325

43.

Mileva

Tompkinson

Watt

Burton

A. M.

(2018). Audiovisual integration in social evaluation. Journal of Experimental Psychology: Human Perception and Performance, 44(1), 128–138. https://doi.org/10.1037/xhp0000439

44.

Mileva

Tompkinson

Watt

Burton

A. M.

(2020). The role of face and voice cues in predicting the outcome of student representative elections. Personality and Social Psychology Bulletin, 46(4), 617–625. https://doi.org/10.1177/0146167219867965

45.

Moskowitz

G. B.

Olcaysoy Okten

Schneid

(2022). The updating of first impressions. In Balcetis

Moskowitz

G. B.

(Eds.), The handbook of impression formation (pp. 348–392). Routledge.

46.

Mussweiler

(2003). Comparison processes in social judgment: Mechanisms and consequences. Psychological Review, 110(3), 472–489. https://doi.org/10.1037/0033-295X.110.3.472

47.

Newman

L. S.

(1996). Trait impressions as heuristics for predicting future behavior. Personality and Social Psychology Bulletin, 22(4), 395–411. https://doi.org/10.1177/0146167296224006

48.

Olivola

C. Y.

Todorov

(2010). Elected in 100 milliseconds: Appearance-based trait inferences and voting. Journal of Nonverbal Behavior, 34(2), 83–110. https://doi.org/10.1007/s10919-009-0082-1

49.

Perugini

Gallucci

Costantini

(2014). Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science, 9(3), 319–332. https://doi.org/10.1177/1745691614528519

50.

Rakić

Steffens

M. C.

Mummendey

(2011). When it matters how you pronounce it: The influence of regional accents on job interview outcome. British Journal of Psychology, 102(4), 868–883. https://doi.org/10.1111/j.2044-8295.2011.02051.x

51.

Rezlescu

Duchaine

Olivola

C. Y.

Chater

(2012). Unfakeable facial configurations affect strategic choices in trust games with or without information about past behavior. PLOS ONE, 7(3), Article e34293. https://doi.org/10.1371/journal.pone.0034293

52.

Rezlescu

Penton

Walsh

Tsujimura

Scott

S. K.

Banissy

M. J.

(2015). Dominant voices and attractive faces: The contribution of visual and auditory information to integrated person impressions. Journal of Nonverbal Behavior, 39(4), 355–370. https://doi.org/10.1007/s10919-015-0214-8

53.

Rougier

De Houwer

(2024). Updating stereotypical attributions in light of new information: The attractiveness halo effect changes when attractiveness changes. European Journal of Social Psychology, 54(1), 364–379. https://doi.org/10.1002/ejsp.3017

54.

Rule

N. O.

Sutherland

S. L.

(2017). Social categorization from faces: Evidence from obvious and ambiguous groups. Current Directions in Psychological Science, 26(3), 231–236. https://doi.org/10.1177/0963721417697970

55.

Rule

N. O.

Tskhay

K. O.

Brambilla

Riva

Andrzejewski

S. A.

Krendl

A. C.

(2015). The relationship between anti-gay prejudice and the categorization of sexual orientation. Personality and Individual Differences, 77, 74–80.

56.

Schneid

E. D.

Crawford

M. T.

Skowronski

J. J.

Irwin

L. M.

Carlston

D. E.

(2015). Thinking about other people: Spontaneous trait inferences and spontaneous evaluations. Social Psychology, 46(1), 24–35. https://doi.org/10.1027/1864-9335/a000218

57.

Shen

Maiolatesi

A. J.

Mann

T. C.

Ferguson

M. J.

(2023). The Relation between updated implicit evaluations and the Trust Game. Social Psychological and Personality Science, 14(8), 911–921. https://doi.org/10.1177/19485506221141661

58.

Shen

Mann

T. C.

Ferguson

M. J.

(2020). Beware a dishonest face? Updating face-based implicit impressions using diagnostic behavioral information. Journal of Experimental Social Psychology, 86, 103888. https://doi.org/10.1016/j.jesp.2019.103888

59.

Skowronski

J. J.

Carlston

D. E.

Hartnett

(2008). Spontaneous impressions derived from observations of behavior: What a long, strange trip it’s been (and it’s not over yet). In Ambady

Skowronski

J. J.

(Eds.), First impressions (pp. 313–333). Guilford Publications.

60.

Snyder

Stukas

A. A.

Jr. (1999). Interpersonal processes: The interplay of cognitive, motivational, and behavioral activities in social interaction. Annual Review of Psychology, 50, 273–303. https://doi.org/10.1146/annurev.psych.50.1.273

61.

Sutherland

C. A.

Young

A. W.

(2022). Understanding trait impressions from faces. British Journal of Psychology, 113(4), 1056–1078. https://doi.org/10.1111/bjop.12583

62.

Todorov

(2008). Evaluating faces on trustworthiness: An extension of systems for recognition of emotions signaling approach/avoidance behaviors. Annals of the New York Academy of Sciences, 1124(1), 208–224. https://doi.org/10.1196/annals.1440.012

63.

Todorov

Olivola

C. Y.

Dotsch

Mende-Siedlecki

(2015). Social attributions from faces: Determinants, consequences, accuracy, and functional significance. Annual Review of Psychology, 66, 519–545. https://doi.org/10.1146/annurev-psych-113011-143831

64.

Todorov

Uleman

J. S.

(2002). Spontaneous trait inferences are bound to actors’ faces: Evidence from a false recognition paradigm. Journal of Personality and Social Psychology, 83(5), 1051–1065. https://doi.org/10.1037/0022-3514.83.5.1051

65.

Uleman

J. S.

Kressel

L. M.

(2013). A brief history of theory and research on impression formation. In Carlston

D. E.

(Ed.), The Oxford handbook of social cognition (pp. 53–73). Oxford University Press.

66.

Vives

M. L.

Frances

Baus

(2023). Automating weighing of faces and voices based on cue saliency in trustworthiness impressions. Scientific Reports, 13(1), 20037.

67.

Vonk

(1994). Trait inferences, impression formation, and person memory: Strategies in processing inconsistent information about persons. European Review of Social Psychology, 5(1), 111–149.

68.

Willis

Todorov

(2006). First impressions: Making up your mind after a 100-ms exposure to a face. Psychological Science, 17(7), 592–598. https://doi.org/10.1111/j.1467-9280.2006.01750.x

69.

Young

A. W.

Frühholz

Schweinberger

S. R.

(2020). Face and voice perception: Understanding commonalities and differences. Trends in Cognitive Sciences, 24(5), 398–410. https://doi.org/10.1016/j.tics.2020.02.001

70.

Zebrowitz

L. A.

(2017). First impressions from faces. Current Directions in Psychological Science, 26(3), 237–242. https://doi.org/10.1177/0963721416683996

71.

Zuckerman

Sinicropi

(2011). When physical and vocal attractiveness differ: Effects on favorability of interpersonal impressions. Journal of Nonverbal Behavior, 35(2), 75–86. https://doi.org/10.1007/s10919-011-0106-5

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB