While acoustic analysis methods have become a commodity in voice emotion research, experiments that attempt not only to describe but to computationally manipulate expressive cues in emotional voice and speech have remained relatively rare. We give here a nontechnical overview of voice-transformation techniques from the audio signal-processing community that we believe are ripe for adoption in this context. We provide sound examples of what they can achieve, examples of experimental questions for which they can be used, and links to open-source implementations. We point at a number of methodological properties of these algorithms, such as being specific, parametric, exhaustive, and real-time, and describe the new possibilities that these open for the experimental study of the emotional voice.
AkuzawaK.IwasawaY.MatsuoY. (2018). Expressive speech synthesis via modeling expressions with variational autoencoder. Retrieved from arXiv:1804.02135
2.
AnikinA. (2019a). Soundgen: An open-source tool for synthesizing nonverbal vocalizations. Behavior Research Methods, 51(2), 778–792.
3.
AnikinA. (2019b). The perceptual effects of manipulating nonlinear phenomena in synthetic nonverbal vocalizations. Bioacoustics. Advance online publication. https://doi.org/10.1080/09524622.2019.1581839
4.
AriasP.BelinP.AucouturierJ. J. (2018). Auditory smiles trigger unconscious facial imitation. Current Biology, 28(14), R782–R783.
5.
AriasP.SoladieC.BouafifO.RobelA.SeguierR.AucouturierJ. J. (2018). Realistic transformation of facial and vocal smiles in real-time audiovisual streams. IEEE Transactions on Affective Computing. Advance online publication. https://doi.org/10.1109/TAFFC.2018.2811465
6.
ArmstrongM. M.LeeA. J.FeinbergD. R. (2019). A house of cards: Bias in perception of body size mediates the relationship between voice pitch and perceptions of dominance. Animal Behaviour, 147, 43–51.
7.
ArnalL. H.FlinkerA.KleinschmidtA.GiraudA. L.PoeppelD. (2015). Human screams occupy a privileged niche in the communication soundscape. Current Biology, 25(15), 2051–2056.
8.
AucouturierJ. J.JohanssonP.HallL.SegniniR.MercadiéL.WatanabeK. (2016). Covert digital manipulation of vocal emotion alter speakers emotional states in a congruent direction. Proceedings of the National Academy of Sciences, 113(4), 948–953.
9.
BachorowskiJ. A.OwrenM. J. (1995). Vocal expression of emotion: Acoustic properties of speech are associated with emotional intensity and context. Psychological Science, 6(4), 219–224.
10.
BanseR.SchererK. R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70(3), 614–636.
11.
BarthelH.QuenéH. (2015). Acoustic-phonetic properties of smiling revised: Measurements on a natural video corpus. In The Scottish Consortium for ICPhS 2015 (Ed.), Proceedings of the 18th International Congress of Phonetic Sciences (pp. 1–5). Glasgow, UK: The University of Glasgow.
12.
BelinP.BoehmeB.McAleerP. (2017). The sound of trustworthiness: Acoustic-based modulation of perceived voice personality. PLoS ONE, 12(10). https://doi.org/10.1371/journal.pone.0211282.
13.
BlumsteinD. T.RecapetC. (2009). The sound of arousal: The addition of novel non-linearities increases responsiveness in marmot alarm calls. Ethology, 115(11), 1074–1081.
14.
BoersmaP.WeeninkD. (2002). PRAAT, a system for doing phonetics by computer. Glot International, 5(9–10), 341–345.
15.
BõhmT.AudibertN.Shattuck-HufnagelS.NémethG.AubergéV. (2008). Transforming modal voice into irregular voice by amplitude scaling of individual glottal cycles. Journal of the Acoustical Society of America, 123(5). https://doi.org/10.1121/1.2935816
16.
BoidronL.BoudeniaK.AvenaC.BoucheixJ. M.AucouturierJ. J. (2016). Emergency medical triage decisions are swayed by computer-manipulated cues of physical dominance in callers voice. Scientific Reports, 6. https://doi.org/10.1038/srep30219
17.
BonadaJ.BlaauwM. (2013). Generation of growl-type voice qualities by spectral morphing. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 6910–6914). New York, NY: IEEE.
18.
BonebrightT. L.ThompsonJ. L.LegerD. W. (1996). Gender stereotypes in the expression and perception of vocal affect. Sex Roles, 34(5–6), 429–445.
19.
BowlingD.GarciaM.DunnJ.RuprechtR.StewartA.FrommoltK. H.FitchW. (2017). Body size and vocalization in primates and carnivores. Scientific Reports, 7(41070), 1–11. https://doi.org/10.1038/srep41070
20.
BradyM. C. (2005). Synthesizing affect with an analog vocal tract: Glottal source. In Toward Social Mechanisms of Android Science: A CogSci 2005 Workshop, Stresa, Italy, 25–26 July 2005 (pp. 45–49). Cognitive Science Society.
21.
BruckertL.BestelmeyerP.LatinusM.RougerJ.CharestI.RousseletG. A.. . . BelinP. (2010). Vocal attractiveness increases by averaging. Current Biology, 20(2), 116–120.
22.
BurleighT.SchoenherrJ.LacroixG. (2013). Does the uncanny valley exist? An empirical test of the relationship between eeriness and the human likeness of digitally created faces. Computers in Human Behavior, 29(3), 759–771.
23.
BurredJ. J.PonsotE.GoupilL.LiuniM.AucouturierJ. J. (2019). CLEESE: An open-source audio-transformation toolbox for data-driven experiments in speech and music cognition. PLoS One, 14(4). https://doi.org/10.1371/journal.pone.0205943
24.
CasadevallA.FangF. C. (2008). Descriptive science. Infection and Immunity, 76(9), 3835–3836.
25.
CharltonB. D.EllisW. A.BrummJ.NilssonK.FitchW. T. (2012). Female koalas prefer bellows in which lower formants indicate larger males. Animal Behaviour, 84(6), 1565–1571.
26.
ChenB.KitaokaN.TakedaK. (2016). Impact of acoustic similarity on efficiency of verbal information transmission via subtle prosodic cues. EURASIP Journal on Audio, Speech, and Music Processing, 2016(1). https://doi.org/10.1186/s13636-016-0097-6
27.
ChongC. S.KimJ.DavisC. (2018). Disgust expressive speech: The acoustic consequences of the facial expression of emotion. Speech Communication, 98, 68–72.
28.
DattorroJ. (1997). Effect design. Part 2: Delay line modulation and chorus. Journal of the Audio Engineering Society, 45(10), 764–788.
29.
De BoerB. (2000). Self-organization in vowel systems. Journal of Phonetics, 28(4), 441–465.
30.
DegottexG.LanchantinP.RoebelA.RodetX. (2013). Mixed source model and its adapted vocal tract filter estimate for voice transformation and synthesis. Speech Communication, 55(2), 278–294.
31.
DrahotaA.CostallA.ReddyV. (2008). The vocal communication of different kinds of smile. Speech Communication, 50(4), 278–287.
32.
DupouxE.KakehiK.HiroseY.PallierC.MehlerJ. (1999). Epenthetic vowels in Japanese: A perceptual illusion?Journal of Experimental Psychology: Human Perception and Performance, 25(6), 1568–1578.
33.
El HaddadK.DupontS.d’AlessandroN.DutoitT. (2015). An HMM-based speech-smile synthesis system: An approach for amusement synthesis. In Proceedings of the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (Vol. 5, pp. 1–6). New York, NY: IEEE.
34.
EybenF.WeningerF.GrossF.SchullerB. (2013). Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In Proceedings of the 21st ACM International Conference on Multimedia (pp. 835–838). New York, NY: ACM.
35.
FeeM. S.ShraimanB.PesaranB.MitraP. P. (1998). The role of nonlinear dynamics of the syrinx in the vocalizations of a songbird. Nature, 395, 67–71.
36.
FitchW. T. (1997). Vocal tract length and formant frequency dispersion correlate with body size in Rhesus macaques. The Journal of the Acoustical Society of America, 102(2), 1213–1222.
37.
FitchW. T.NeubauerJ.HerzelH. (2002). Calls out of chaos: The adaptive significance of nonlinear phenomena in mammalian vocal production. Animal Behaviour, 63(3), 407–418.
38.
GentilucciM.ArdaillonL.LiuniM. (2019). Composing vocal distortion: A tool for real-time generation of roughness. Computer Music Journal, 42(4), 26–40.
39.
GoblC.ChasaideA. N. (2010). 11 voice source variation and its communicative functions. The handbook of phonetic sciences1: 378.
40.
GoblC.ChasaideA. N. (2012). Voice source variation and its communicative functions. In HardcastleW. J.LaverJ.GibbonF. E. (Eds.), The handbook of phonetic sciences (Vol. 119, pp. 378–423). John Wiley & Sons.
41.
GoldstoneR. L.LupyanG. (2016). Discovering psychological principles by mining naturally occurring data sets. Topics in Cognitive Science, 8(3), 548–568.
42.
GoodfellowI.Pouget-AbadieJ.MirzaM.XuB.Warde-FarleyD.OzairS.. . . BengioY. (2014). Generative adversarial nets. In Annual Conference on Neural Information Processing Systems 2014, Montreal, Quebec, Canada, 8–13 December 2014 (pp. 2672–2680).
43.
GovindD.PrasannaS. M. (2013). Expressive speech synthesis: A review. International Journal of Speech Technology, 16(2), 237–260.
44.
HainT. C.BurnettT. A.LarsonC. R.KiranS. (2001). Effects of delayed auditory feedback (daf) on the pitch-shift reflex. The Journal of the Acoustical Society of America, 109(5), 2146–2152.
45.
HienzR. D.JonesA. M.WeertsE. M. (2004). The discrimination of baboon grunt calls and human vowel sounds by baboons. The Journal of the Acoustical Society of America, 116, 1692–1697.
46.
HsuW. N.ZhangY.GlassJ. (2017). Learning latent representations for speech generation and transformation. Retrieved from arXiv:1704.04222
47.
IlieG.ThompsonW. F. (2006). A comparison of acoustic cues in music and speech for three dimensions of affect. Music Perception, 23(4), 319–329.
48.
JackR. E.SchynsP. G. (2017). Toward a social psychophysics of face communication. Annual Review of Psychology, 68, 269–297.
49.
JiangX.PellM. D. (2017). The sound of confidence and doubt. Speech Communication, 88, 106–126.
50.
JohnstoneT.SchererK. R. (1999). The effects of emotions on voice quality. In Proceedings of the 14th International Congress of Phonetic Sciences, San Francisco, 1–7 August 1999 (pp. 2029–2032). International Phonetic Association.
51.
JuilleratN.Schubiger-BanzS.ArisonaS. M. (2008). Low latency audio pitch shifting in the time domain. In Proceedings of the IEEE 2008 International Conference on Audio, Language and Image Processing (pp. 29–35). New York, NY: IEEE.
52.
JuslinP. N.LaukkaP. (2003). Communication of emotions in vocal expression and music performance: Different channels, same code?Psychological Bulletin, 129(5), 770–814.
53.
KamilogluR.FischerA.SauterD. A. (2020). Good vibrations: A review of vocal expressions of positive emotions. Psychonomic Bulletin & Review, 27, 237–265.
54.
KawaharaH. (1997). Speech representation and transformation using adaptive interpolation of weighted spectrum: Vocoder revisited. In Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (Vol. 2, pp. 1303–1306). New York, NY: IEEE.
55.
KawaharaH.MatsuiH. (2003). Auditory morphing based on an elastic perceptual distance metric in an interference-free time-frequency representation. In Proceedings of the 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing (Vol. 1, pp. 256–259). New York, NY: IEEE.
56.
KitayamaS.MesquitaB.KarasawaM. (2006). Cultural affordances and emotional experience: Socially engaging and disengaging emotions in Japan and the United States. Journal of Personality and Social Psychology, 91(5), 890–903.
57.
LartillotO.ToiviainenP. (2007). A MATLAB toolbox for musical feature extraction from audio. In Proceedings of the 10th International Conference on Digital Audio Effects, Bordeaux, France, 10–15 September 2007 (pp. 237–244). DAFx.
58.
LasarcykE.TrouvainJ. (2008). Spread lips + raised larynx + higher F0 = smiled speech? – An articulatory synthesis approach. Proceedings of the 8th International Seminar on Speech Production (pp. 43–48). Nancy, France: Institut National Polytechnique de Lorraine.
59.
LatinusM.McAleerP.BestelmeyerP. E.BelinP. (2013). Norm-based coding of voice identity in human auditory cortex. Current Biology, 23(12), 1075–1080.
LeeE.KarrerT.BorchersJ. O. (2007). An analysis of startup and dynamic latency in phase vocoder-based time-stretching algorithms. In Proceeding International Computer Music Conference, Copenhagen, Denmark, 27–31 August. International Computer Music Association.
62.
LeeY.RabieeA.LeeS. Y. (2017). Emotional end-to-end neural speech synthesizer. Retrieved from arXiv:1711.05447
63.
LoscosA.BonadaJ. (2004). Emulating rough and growl voice in spectral domain. In Proceedings of the International Conference on Digital Audio Effects, Naples, Italy, 5–8 October 2004 (pp. 49–52). DAFx
64.
LuceyP.CohnJ. F.KanadeT.SaragihJ.AmbadarZ.MatthewsI. (2010). The extended Cohn–Kanade Dataset (ck+): A complete dataset for action unit and emotion-specified expression. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, San Francisco, CA, USA, 13–18 June 2010 (pp. 94–101). New York, NY: IEEE.
65.
LuoZ.ChenJ.TakiguchiT.ArikiY. (2017). Emotional voice conversion with adaptive scales F0 based on wavelet transform using limited amount of emotional data. In Proceedings of the 2017 Interspeech Conference, Stockholm, Sweden, 20–24 August 2017 (pp. 3399–3403). International Speech Communication Association.
66.
MaW.ThompsonW. F. (2015). Human emotions track changes in the acoustic environment. Proceedings of the National Academy of Sciences of the USA, 112(47), 14563–14568.
67.
MaliszZ.HenterG. E.Valentini-BotinhaoC.WattsO.BeskowJ.GustafsonJ. (2019). Modern speech synthesis for phonetic sciences: A discussion and an evaluation. In Proceedings of the 19th International Congress of Phonetic Sciences, Melbourne, Australia, 5–9 August 2019 (pp. 487–491). Australasian Speech Science and Technology Association.
68.
MatsumotoD. (2002). Methodological requirements to test a possible in-group advantage in judging emotions across cultures: Comment on Elfenbein and Ambady (2002) and evidence. Psychological Bulletin, 128(2), 236–242.
69.
MohammadiS. H.KainA. (2017). An overview of voice conversion systems. Speech Communication, 88, 65–82.
70.
MoulinesE.LarocheJ. (1995). Non-parametric techniques for pitch-scale and time-scale modification of speech. Speech Communication, 16(2), 175–205.
71.
NarendranathM.MurthyH. A.RajendranS.YegnanarayanaB. (1995). Transformation of formants for voice conversion using artificial neural networks. Speech Communication, 16(2), 207–216.
72.
NowickiS.MitaniJ. C.NelsonD. A.MarlerP. (1989). The communicative significance of tonality in birdsong: Responses to songs produced in helium. Bioacoustics, 2(1), 35–46.
73.
OhalaJ. J. (1980). The acoustic origin of the smile. The Journal of the Acoustical Society of America, 68(S1), S33.
PellM. D. (2001). Influence of emotion and focus location on prosody in matched statements and questions. The Journal of the Acoustical Society of America, 109(4), 1668–1680.
PellM. D.MonettaL.PaulmannS.KotzS. A. (2009). Recognizing emotions in a foreign language. Journal of Nonverbal Behavior, 33(2), 107–120.
78.
PierceB. C.BenjaminC. (2002). Types and programming languages. Cambridge, MA: MIT Press.
79.
PonsotE.BurredJ. J.BelinP.AucouturierJ. J. (2018). Cracking the social code of speech prosody using reverse correlation. Proceedings of the National Academy of Sciences of the USA, 115(15), 3972–3977.
RachmanL.LiuniM.AriasP.LindA.JohanssonP.HallL.. . . AucouturierJ. J. (2018). DAVID: An open-source platform for real-time transformation of infra-segmental emotional cues in running speech. Behavior Research Methods, 50(1), 323–343.
82.
RandA. S.DudleyR. (1993). Frogs in helium: The anuran vocal sac is not a cavity resonator. Physiological Zoology, 66(5), 793–806.
83.
RebyD.McCombK.CargneluttiB.DarwinC.FitchW. T.Clutton-BrockT. (2005). Red deer stags use formants as assessment cues during intrasexual agonistic interactions. Proceedings of the Royal Society B: Biological Sciences, 272(1566), 941–947.
84.
RuinskiyD.LavnerY. (2008). Stochastic models of pitch jitter and amplitude shimmer for voice modification. In Proceedings of the 2008 Convention of Electrical and Electronics Engineers in Israel (pp. 489–493). New York, NY: IEEE.
85.
SchererK. R. (1972, April). Acoustic concomitants of emotional dimensions: Judging affect from synthesized tone sequences. Paper presented at the Eastern Psychological Association Meeting, Boston, MA.
86.
SchererK. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40(1–2), 227–256.
87.
SchererK. R.BanseR.WallbottH. G. (2001). Emotion inferences from vocal expression correlate across languages and cultures. Journal of Cross-Cultural Psychology, 32(1), 76–92.
88.
SchererK. R.OshinskyJ. S. (1977). Cue utilization in emotion attribution from auditory stimuli. Motivation and Emotion, 1(4), 331–346.
89.
SeshadriS.JuvelaL.AlkuP.RäsänenO. (2019). Augmented cyclegans for continuous scale normal-to-Lombard speaking style conversion. In Proceedings of the 2019 Interspeech Conference, Graz, Austria, 15–19 September 2019 (pp. 2838–2842). International Speech Communication Association.
90.
SimonyanK.HorwitzB. (2011). Laryngeal motor cortex and control of speech in humans. The Neuroscientist, 17(2), 197–208.
91.
StylianouY. (2009). Voice transformation: A survey. In Proceedings of the 2009 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 3585–3588). New York, NY: IEEE.
92.
SueurJ.AubinT.SimonisC. (2008). Seewave, a free modular tool for sound analysis and synthesis. Bioacoustics, 18(2), 213–226.
93.
SunP.AnumanchipalliG. K.ChangE. F. (2019). Brain2char: A deep architecture for decoding text from brain recordings. Retrieved from arXiv:1909.01401
94.
TartterV. C. (1980). Happy talk: Perceptual and acoustic effects of smiling on speech. Perception & Psychophysics, 27(1), 24–27.
95.
TartterV. C.BraunD. (1994). Hearing smiles and frowns in normal and whisper registers. The Journal of the Acoustical Society of America, 96(4), 2101–2107.
96.
TchernichovskiO.MitraP. (2004). Sound Analysis Pro user manual. New York, NY: CCNY.
97.
TiceD. M. (1992). Self-concept change and self-presentation: The looking glass self is also a magnifying glass. Journal of Personality and Social Psychology, 63(3), 435–451.
TysonR. B.NowacekD. P.MillerP. J. (2007). Nonlinear phenomena in the vocalizations of North Atlantic right whales (Eubalaena glacialis) and killer whales (Orcinus orca). The Journal of the Acoustical Society of America, 122(3), 1365–1373.
100.
Van den OordA.DielemanS.ZenH.SimonyanK.VinyalsO.GravesA.. . . KavukcuogluK. (2016). Wavenet: A generative model for raw audio. Retrieved from arXiv:1609.03499
101.
Van DoornE. A.HeerdinkM. W.van KleefG. A. (2012). Emotion and the construal of social situations: Inferences of cooperation versus competition from expressions of anger, happiness, and disappointment. Cognition & Emotion, 26(3), 442–461.
102.
VermaA.KumarA. (2005). Introducing roughness in individuality transformation through jitter modeling and modification. In Proceedings of the 2005 IEEE International Conference on Acoustics, Speech, and Signal Processing (Vol. 1, pp. 1–5). New York, NY: IEEE.
103.
WangW.Alameda-PinedaX.XuD.FuaP.RicciE.SebeN. (2018). Every smile is unique: Landmark-guided diverse smile generation. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (pp. 7083–7092). New York, NY: IEEE.
104.
WangY.Skerry-RyanR.StantonD.WuY.WeissR. J.JaitlyN.. . . SaurousR. A. (2017). Tacotron: Towards end-to-end speech synthesis. Retrieved from arXiv:1703.10135
105.
WildenI.HerzelH.PetersG.TembrockG. (1998). Subharmonics, biphonation, and deterministic chaos in mammal vocalization. Bioacoustics, 9(3), 171–196.
106.
WuY. C.HayashiT.TobingP. L.KobayashiK.TodaT. (2019). Quasi-periodic wavenet vocoder: A pitch dependent dilated convolution model for parametric speech generation. Retrieved from arXiv:1907.00797
107.
XuT.ZhanJ.GarrodO. G.TorrP. H.ZhuS. C.InceR. A.SchynsP. G. (2018). Deeper interpretability of deep networks. Retrieved from arXiv:1811.07807
108.
YoshieM.SauterD. A. (2019). Cultural norms influence nonverbal emotion communication: Japanese vocalizations of socially disengaging emotions. Emotion. Advance online publication. https://doi.org/10.1037/emo0000580
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
