The present study investigated how acoustic and phonetic characteristics of synthetic and natural voices affect personality impressions of the voices. To this end, we conducted a personality rating experiment in which 30 native Korean speakers judged the perceived personality of natural Korean utterances and their synthetic counterparts (voice clones) using the Big-Five personality model. Various acoustic parameters, including measures of voice quality, F0, and articulation rate, were then extracted from the speech, and Intonational Phrase boundary tones were annotated. The ratings of the Big-Five personality traits were reduced to two dimensions (P1: agreeableness, conscientiousness, and emotional stability; P2: extraversion and openness) using a principal component analysis. The results suggest that the acoustic differences between state-of-the-art synthetic speech and its original counterpart can produce varying effects on personality perception. For example, speech produced with a narrower F0 range received lower scores on P1 and P2, but for male speakers, this effect was only observed in synthetic voices, likely due to the less-natural intonational patterns used. The intonation analysis further demonstrates that across speech type, using context-appropriate tones or those conveying positive attitudes improves the overall impression of the voice (both P1 and P2). The results also suggest that a less-modal voice enhances the personality scores overall, but specific voice qualities (i.e., breathiness and creakiness) and voice pitch seem to affect P1 and P2 differently. The present study demonstrates a range of acoustic and phonetic characteristics that should be considered when designing personas for AI voices or developing more likable synthetic voices.
AndersonR. C.KlofstadC. A.MayewW. J.VenkatachalamM. (2014). Vocal fry may undermine the success of young women in the labor market. PLoS ONE, 9(5), Article e97506.
2.
AppleW.StreeterL. A.KraussR. M. (1979). Effects of pitch and speech rate on personal attributions. Journal of Personality and Social Psychology, 37(5), 715–727.
3.
AronovitchC. D. (1976). The voice of personality: Stereotyped judgments and their relation to voice quality and sex of speaker. The Journal of Social Psychology, 99(2), 207–220.
4.
BarrickM. R.MountM. K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44(1), 1–26.
5.
BatesD.MächlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48.
6.
BausC.McAleerP.MarcouxK.BelinP.CostaA. (2019). Forming social impressions from voices in native and foreign languages. Scientific Reports, 9(1), 414.
7.
BelinP.BestelmeyerP. E.LatinusM.WatsonR. (2011). Understanding voice perception. British Journal of Psychology, 102(4), 711–725.
BolingerD. (1978). Intonation across languages. In GreenbergJ. H. (Ed.), Universals of human language (Vol. 2, pp. 471–524). Stanford University Press.
10.
BolingerD.BolingerD. L. M. (1986). Intonation and its parts: Melody in spoken English. Stanford University Press.
11.
BorkowskaB.PawlowskiB. (2011). Female voice frequency in the context of dominance and attractiveness perception. Animal Behaviour, 82(1), 55–59.
12.
ChenY.NiuZ.MaZ.DengK.WangC.ZhaoJ.YuK.ChenX. (2024). F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv. https://arxiv.org/pdf/2410.06885
13.
DanielescuA.ChristianG. (2018, April21–26). A bot is not a polyglot: Designing personalities for multi-lingual conversational agents [Paper presentation]. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.
14.
DavidsonL. (2019). The effects of pitch, gender, and prosodic context on the identification of creaky voice. Phonetica, 76(4), 235–262.
15.
EdwardsC.EdwardsA.StollB.LinX.MasseyN. (2019). Evaluations of an artificial intelligence instructor’s voice: Social Identity Theory in human-robot interactions. Computers in Human Behavior, 90, 357–362.
16.
EricksonD.KawaharaS.RilliardA.HayashiR.SadanobuT.LiY.DaikuharaH.De MoraesJ.ObertK. (2020, May). Cross cultural differences in arousal and valence perceptions of voice quality [Paper presentation]. Proceedings of the 10th International Conference on Speech Prosody 2020, Tokyo.
GallardoL. F.WeissB. (2016, September8–12). Speech likability and personality-based social relations: A round-robin analysis over communication channels [Paper presentation]. Interspeech, San Francisco, CA, United States.
19.
GarellekM.KatzW. F.AssmannP. F. (2019). The phonetics of voice. In KatzW. F.AssmannP. F. (Eds.), The Routledge handbook of phonetics (pp. 75–106). Routledge.
20.
GarellekM.SamlanR.GerrattB. R.KreimanJ. (2016). Modeling the voice source in terms of spectral slopes. The Journal of the Acoustical Society of America, 139(3), 1404–1410.
21.
GoblC.ChasaideA. N. (2003). The role of voice quality in communicating emotion, mood and attitude. Speech Communication, 40(1–2), 189–212.
22.
GoblC.ChasaideA. N. (2010). 11 voice source variation and its communicative functions. In HardcastleW. J.LaverJ.GibbonF. E. (Eds.), The handbook of phonetic sciences (Vol. 50, p. 378). Wiley.
23.
GoldbergL. R. (1992). The development of markers for the Big-Five factor structure. Psychological Assessment, 4(1), 26–42.
24.
GreenbergJ. H. (1963). Universals of language. MIT Press.
25.
HodariZ.WattsO.KingS. (2019). Using generative modelling to produce varied intonation for speech synthesis. arXiv. https://arxiv.org/pdf/1906.04233
26.
Hodges-SimeonC. R.GaulinS. J.PutsD. A. (2010). Different vocal parameters predict perceptions of dominance and attractiveness. Human Nature, 21, 406–427.
27.
HollidayJ. J.WalkerA.JungM.ChoE. (2023). Bringing indexical orders to non-arbitrary meaning: The case of pitch and politeness in English and Korean. Laboratory Phonology: Journal of the Association for Laboratory Phonology, 14(1), 1–24.
28.
IshiC.IshiguroH.HagitaN. (2010). Analysis of the roles and the dynamics of breathy and whispery voice qualities in dialogue speech. EURASIP Journal on Audio, Speech, and Music Processing, 2010, 1–12.
29.
JohnO. P.NaumannL. P.SotoC. J. (2008). Paradigm shift to the integrative big five trait taxonomy. Handbook of Personality: Theory and Research, 3(2), 114–158.
30.
JunS. A. (1996). The phonetics and phonology of Korean prosody: Intonational phonology and prosodic structure. Routledge.
JunS. A. (Ed.). (2005). Prosodic typology: The phonology of intonation and phrasing. OUP Oxford.
33.
KaiserH. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20(1), 141–151.
34.
KawaharaH.Masuda-KatsuseI.De CheveigneA. (1999). Restructuring speech representations using a pitch-adaptive time–frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds. Speech Communication, 27(3-4), 187–207.
35.
KimS. Y.KimJ. M.YooJ. A.BaeK. Y.KimS. W.YangS. J.ShinI. S.YoonJ. S. (2010). Standardization and validation of big five inventory-Korean version (BFI-K) in elders. Korean Journal of Biological Psychiatry, 17(1), 15–25.
36.
KlofstadC. A.AndersonR. C.NowickiS. (2015). Perceptions of competence, strength, and age influence voters to select leaders with lower-pitched voices. PLoS ONE, 10(8), Article e0133779.
37.
KreimanJ.GerrattB. R.GarellekM.SamlanR.ZhangZ. (2014). Toward a unified theory of voice production and perception. Loquens, 1(1), Article e009.
38.
KreimanJ.SidtisD. (2011). Foundations of voice studies: An interdisciplinary approach to voice production and perception. John Wiley & Sons.
39.
KringA. M.SmithD. A.NealeJ. M. (1994). Individual differences in dispositional expressiveness: Development and validation of the Emotional Expressivity Scale. Journal of Personality and Social Psychology, 66(5), 934–949.
40.
KuznetsovaA.BrockhoffP. B.ChristensenR. H. B. (2017). LmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82, 1–26.
41.
LadefogedP. (1971). Preliminaries to linguistic phonetics. University of Chicago Press.
42.
LamerisH.GustafsonJ.SzékelyÉ. (2024). CreakVC: A voice conversion tool for modulating creaky voice. In 25th Interspeech Conference 2024, Kos Island, Greece, September 1-5, 2024 (pp. 1005–1006). International Speech Communication Association. https://www.isca-archive.org/interspeech_2024/lameris24_interspeech.pdf
LeeE. J.NassC.BraveS. (2000, April). Can computer-generated speech have gender? An experimental test of gender stereotype. In CHI’00 extended abstracts on Human factors in computing systems (pp. 289–290). https://dl.acm.org/doi/pdf/10.1145/633292.633461
45.
LeeH. Y. (1990). The structure of Korean prosody. University of London, University College London.
46.
LeemannA.KollyM. J.BritainD. (2018). The English Dialects App: The creation of a crowdsourced dialect corpus. Ampersand, 5, 1–17.
47.
LenthR.LenthM. R. (2018). Package “lsmeans.”The American Statistician, 34(4), 216–221.
48.
LinE.HornibrookJ.OrmondT. (2012). Evaluating iPhone recordings for acoustic voice assessment. Folia phoniatrica et logopaedica, 64(3), 122–130.
49.
LovedayL. (1981). Pitch, politeness and sexual role: An exploratory investigation into the pitch correlates of English and Japanese politeness formulae. Language and Speech, 24(1), 71–89.
50.
MalloryE. B.MillerV. R. (1958). A possible basis for the association of voice characteristics and personality traits. Communications Monographs, 25(4), 255–260.
51.
MandeelA. R.Al-RadhiM. S.CsapóT. G. (2023, October25–27). Modeling irregular voice in end-to-end speech synthesis via speaker adaptation [Paper presentation]. 2023 International Conference on Speech Technology and Human-Computer Dialogue (SpeD), Bucharest.
52.
ManfrediC.LebacqJ.CantarellaG.SchoentgenJ.OrlandiS.BandiniA.DeJonckereP. H. (2017). Smartphones offer new opportunities in clinical voice research. Journal of Voice, 31(1), 111.e1–111.e7.
53.
McAleerP.TodorovA.BelinP. (2014). How do you say ‘Hello’? Personality impressions from brief novel voices. PLoS ONE, 9(3), Article e90779.
54.
McCraeR. R.Costa, JrP. T. (1997). Personality trait structure as a human universal. American Psychologist, 52(5), 509–516.
55.
MilevaM.LavanN. (2023). Trait impressions from voices are formed rapidly within 400 ms of exposure. Journal of Experimental Psychology: General, 152(6), 1539–1550.
56.
MillerN.MaruyamaG.BeaberR. J.ValoneK. (1976). Speed of speech and persuasion. Journal of Personality and Social Psychology, 34(4), 615–624.
57.
MohammadiG.OrigliaA.FilipponeM.VinciarelliA. (2012, October29). From speech to personality: Mapping voice quality and intonation into personality differences [Paper presentation]. Proceedings of the 20th ACM International Conference on Multimedia, Nara, Japan.
58.
NassC.LeeK. M. (2001). Does computer-synthesized speech manifest personality? Experimental tests of recognition, similarity-attraction, and consistency-attraction. Journal of Experimental Psychology: Applied, 7(3), 171.
59.
NassC.MoonY. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56(1), 81–103.
60.
NassC.MoonY.GreenN. (1997). Are machines gender neutral? Gender-stereotypic responses to computers with voices. Journal of Applied Social Psychology, 27(10), 864–876.
61.
OhalaJ. J. (1983). Cross-language use of pitch: An ethological view. Phonetica, 40, 1–18.
62.
OhalaJ. J. (1984). An ethological perspective on common cross-language utilization of F₀ of voice. Phonetica, 41(1), 1–16.
63.
ParkM. J. (2012). The meaning of Korean prosodic boundary tones. Brill.
64.
Penton-VoakI. S.PoundN.LittleA. C.PerrettD. I. (2006). Personality judgments from natural and composite facial images: More evidence for a “kernel of truth” in social perception. Social Cognition, 24(5), 607–640.
65.
PolzehlT.MöllerS.MetzeF. (2010, December12–15). Automatically assessing acoustic manifestations of personality in speech [Paper presentation]. 2010 IEEE Spoken Language Technology Workshop, Berkeley, CA, United States.
66.
PolzehlT.MöllerS.MetzeF. (2011, August27–31). Modeling speaker personality using voice [Paper presentation]. Twelfth Annual Conference of the International Speech Communication Association, Florence.
67.
PonsotE.BurredJ. J.BelinP.AucouturierJ. J. (2018). Cracking the social code of speech prosody using reverse correlation. Proceedings of the National Academy of Sciences of the United States of America, 115(15), 3972–3977.
68.
PradhanA.LazarA. (2021, July27–29). Hey Google, do you have a personality? Designing personality and personas for conversational agents [Paper presentation]. Proceedings of the 3rd Conference on Conversational User Interfaces, Bilbao.
69.
PrudkýD.FircA.MalinkaK. (2023, September). Assessing the human ability to recognize synthetic speech in ordinary conversation [Paper presentation]. 2023 International Conference of the Biometrics Special Interest Group (BIOSIG), Bonn.
70.
PutsD. A.HodgesC. R.CárdenasR. A.GaulinS. J. (2007). Men’s voices as dominance signals: Vocal fundamental and formant frequencies influence dominance attributions among men. Evolution and Human Behavior, 28(5), 340–344.
71.
RammstedtB.JohnO. P. (2007). Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41(1), 203–212.
72.
RenY.HuC.TanX.QinT.ZhaoS.ZhaoZ.LiuT. Y. (2020). FastSpeech 2: Fast and high-quality end-to-end text to speech. arXiv. https://arxiv.org/pdf/2006.04558
73.
RenY.RuanY.TanX.QinT.ZhaoS.ZhaoZ.LiuT. Y. (2019). Fastspeech: Fast, robust and controllable text to speech. Advances in Neural Information Processing Systems, 32.
RiggioH. R.RiggioR. E. (2002). Emotional expressiveness, extraversion, and neuroticism: A meta-analysis. Journal of Nonverbal Behavior, 26, 195–218.
76.
SamlanR. A.StoryB. H.BuntonK. (2013). Relation of perceived breathiness to laryngeal kinematics and acoustic measures based on computational modeling. Journal of Speech, Language, and Hearing Research, 56, 1209–1223.
77.
SchererK. R. (1978). Personality inference from voice quality: The loud voice of extroversion. European Journal of Social Psychology, 8(4), 467–487.
78.
ShueY. L.KeatingP.VicenikC.YuK. (2011). VoiceSauce: A program for voice analysis [Paper presentation]. Proceedings of the ICPhS XVII, Hong Kong.
79.
SigurgeirssonA. T.KingS. (2023, June4–10). Do prosody transfer models transfer prosody? [Paper presentation]. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island.
80.
SmithB. L.BrownB. L.StrongW. J.RencherA. C. (1975). Effects of speech rate on personality perception. Language and Speech, 18(2), 145–152.
81.
SohnH. M. (2001). The Korean language. Cambridge University Press.
82.
SongJ.KimM.ParkJ. (2023). Acoustic correlates of perceived personality from Korean utterances in a formal communicative setting. PLoS ONE, 18(10), Article e0293222.
83.
StarrR. L. (2015). Sweet voice: The role of voice quality in a Japanese feminine style. Language in Society, 44(1), 1–34.
84.
TabachnickB. G.FidellL. S.UllmanJ. B. (2013). Using multivariate statistics (Vol. 6, pp. 497–516). Pearson.
85.
TehT. H.HuV.MohanD. S. R.HodariZ.WallisC. G.IbarrondoT. G.TorresquinteroA.LeoniJ.GalesM.KingS. (2023, June4–10). Ensemble prosody prediction for expressive speech synthesis [Paper presentation]. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island.
86.
TigueC. C.BorakD. J.O’ConnorJ. J.SchandlC.FeinbergD. R. (2012). Voice pitch influences voting behavior. Evolution and Human Behavior, 33(3), 210–216.
87.
VinciarelliA.SalaminH.PolychroniouA.MohammadiG.OrigliaA. (2012). From nonverbal cues to perception: Personality and social attractiveness. In EspositoA.EspositoA. M.VinciarelliA.HoffmannR.MüllerV. C. (Eds.), Cognitive behavioural systems: COST 2102 International Training School, Dresden, Germany, February 21-26, 2011, revised selected papers (pp. 60–72). Springer Berlin Heidelberg.
88.
WaaramaaT.LukkarilaP.JärvinenK.GeneidA.LaukkanenA. M. (2021). Impressions of personality from intentional voice quality in Arabic-speaking and native Finnish-speaking listeners. Journal of Voice, 35(2), 326.e21–326.e28.
89.
WangC.ChenS.WuY.ZhangZ.ZhouL.LiuS.ChenZ.LiuY.WangH.LiJ.HeL.ZhaoS.WeiF. (2023). Neural codec language models are zero-shot text to speech synthesizers. arXiv. https://arxiv.org/pdf/2301.02111
90.
WangY.Skerry-RyanR. J.StantonD.WuY.WeissR. J.JaitlyN.YangZ.XiaoY.ChenZ.BengioS.LeQ.AgiomyrgiannakisY.ClarkR.SaurousR. A. (2017). Tacotron: Towards end-to-end speech synthesis. arXiv. https://arxiv.org/pdf/1703.10135
91.
WeissB.TrouvainJ.Barkat-DefradasM.OhalaJ. J. (Eds.). (2020). Voice attractiveness: Studies on sexy, likable, and charismatic speakers. Springer Nature.
92.
WinterB.GrawunderS. (2012). The phonetic profile of Korean formal and informal speech registers. Journal of Phonetics, 40(6), 808–815.
93.
XuY. (2019). Prosody, tone and intonation. In KatzW. F.AssmannP. F. (Eds.), The Routledge handbook of phonetics (pp. 314–356). Routledge.
94.
XuY.LeeA.WuW. L.LiuX.BirkholzP. (2013). Human vocal attractiveness as signaled by body size projection. PLoS ONE, 8(4), Article e62397.
95.
YamamotoR.SongE.KimJ. M. (2020, May4–8). Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram [Paper presentation]. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona.
96.
YanushevskayaI.GoblC.Ní ChasaideA. (2013). Voice quality in affect cueing: Does loudness matter?Frontiers in Psychology, 4, 335.
97.
YeonJ.BrownL. (2013). Korean: A comprehensive grammar. Routledge.
98.
ZhangC.LiB.ChenS.YangY. (2018, September2–6). Acoustic analysis of whispery voice disguise in Mandarin Chinese [Paper presentation]. Proceedings of the Interspeech 2018, Hyderabad.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
