This paper presents the results of three perceptual experiments investigating the role of auditory and visual channels for the identification of statements and echo questions in Brazilian Portuguese. Ten Brazilian speakers (five male) were video-recorded (frontal view of the face) while they produced a sentence (“Como você sabe”), either as a statement (meaning “As you know.”) or as an echo question (meaning “As you know?”). Experiments were set up including the two different intonation contours. Stimuli were presented in conditions with clear and degraded audio as well as congruent and incongruent information from both channels. Results show that Brazilian listeners were able to distinguish statements and questions prosodically and visually, with auditory cues being dominant over visual ones. In noisy conditions, the visual channel improved the interpretation of prosodic cues robustly, while it degraded them in conditions where the visual information was incongruent with the auditory information. This study shows that auditory and visual information are integrated during speech perception, also when applied to prosodic patterns.
ArantesP. (2015). Time-normalization of fundamental frequency contours: A hands-on tutorial. In MeirelesA. (Ed.), Courses on speech prosody (pp. 98–123). Newcastle upon Tyne, UK: Cambridge Scholars Publishing.
2.
BarbosaP. (2013). Semi-automatic and automatic tools for generating prosodic descriptors for prosody research. Proceedings of Interspeech satellite event Tools and Resources for the Analysis of Speech Prosody, Aix-en-Provence, France, pp. 86–89.
3.
BarkhuysenP.KrahmerE.SwertsM. (2010). Cross-modal and incremental perception of audiovisual cues to emotional speech. Language and Speech, 53(1), 3–30.
4.
BenoîtC.Le GoffB. (1998). Audiovisual speech synthesis from French text: Eight years of models, designs and evaluation at the ICP. Speech Communication, 26, 117–129.
5.
BoersmaP.WeeninkD. (2016). Praat: Doing phonetics by computer. Computer program (Version 5.1.05) available at: http://www.praat.org/.
6.
Borràs-ComesJ.PrietoP. (2011). Seeing tunes. The role of visual gestures in tune interpretation. Journal of Laboratory Phonology, 2(2), 355–380.
7.
CavéC.GuaïtellaI.BertrandR.SantiS.HarlayF.EspesserR. (1996). About the relationship between eyebrow movements and F0 variations. Proceedings of Fourth International Conference on Spoken Language Processing, Philadelphia, Pennsylvania, USA. DOI: 10.1109/ICSLP.1996.607235
8.
CookeM. (2006). A glimpsing model of speech perception in noise. Journal of the Acoustical Society of America, 119(3), 1562–1573.
9.
CostaM. J.DanielR. C.dos SantosS. N. (2011). Reconhecimento de sentenças no silêncio e no ruído em fones auriculares: Valores de referência de normalidade. CEFAG, 13(4), 685–691.
10.
CoutoL. R.SilvaC. G.MirandaL. S. (2017). Prosódia dos enunciados declarativos e interrogativos totais nas variedades de Salvador, Fortaleza e Rio de Janeiro. Revista de Estudos da Linguagem, 25(3), 1105–1142. DOI: http://dx.doi.org/10.17851/2237-2083.25.3.1105-1142.
11.
Crespo SendraV.KalandC.SwertsM.PrietoP. (2013). Perceiving incredulity: The role of intonation and facial gestures. Journal of Pragmatics, 47, 1–13.
12.
CruzM.SwertsM.FrotaS. (2015). Variation in tone and gesture within language. Proceedings of Eighteenth International Congress of Phonetic Sciences, Glasgow, UK. Retrieved from: http://www.icphs2015.info/pdfs/Papers/ICPHS0452.pdf.
13.
CruzM.SwertsM.FrotaS. (2017). The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties. Laboratory Phonology, 8(1), 23. DOI: http://doi.org/10.5334/labphon.110.
DohenM.LoevenbruckH. (2009). Interaction of audition and vision for the perception of prosodic contrastive focus. Language and Speech, 52(2–3), 177–206.
16.
EkmanP.FriesenW.HagerJ. (2002). The facial action coding system. 2nd ed (CD-ROM). Salt Lake City, UT: Research Nexus.
17.
FisherC. G. (1969). The visibility of terminal pitch contour. Journal of Speech and Hearing Research, 12(2), 379–382.
18.
FitzpatrickM.KimJ.DavisC. (2015). The effect of seeing the interlocutor on auditory and visual speech production in noise. Speech Communication, 74, 37–51.
19.
FrotaS.CruzM.SvartmanF.CollischonnG.FonsecaA.SerraC.VigárioM. (2015). Intonational variation in Portuguese: European and Brazilian varieties. In FrotaS.PrietoP. (Eds.), Intonation in romance (pp. 235–283). Oxford, UK: Oxford University Press.
20.
Gili FivelaB. (2018). Multimodal analyses of audio-visual information: Some methods and issues in prosody research. In FeldhausenI.FliessbachJ.VanrellM. M. (Eds.), Methods in prosody: A Romance language perspective (Studies in Laboratory Phonology 4) (pp. 83–122). Berlin, Germany: Language Science Press.
21.
GrafH. P.CosattoE.VolkerS.HuangF. J. (2002). Visual prosody: Facial movements accompanying speech. Proceedings of Fifth IEEE International Conference on Automatic Face and Gesture Recognition, Washington, DC, USA. DOI: 10.1109/AFGR.2002.1004186
22.
GranströmB.HouseD.SwertsM. (2002). Multimodal feedback cues in human-machine interaction. Proceedings of First International Conference on Speech Prosody, Aix-en-Provence, France, pp. 347–350.
23.
HadarU.SteinerT. J.GrantE. C.Clifford RoseF. (1983). Head movement correlates of juncture and stress at sentence level. Language and Speech, 26, 117–129.
24.
HirstD.EspesserR. (1993). Automatic modelling of fundamental frequency using a quadratic spline function. Travaux de l’Institut de Phonétique d’Aix, 15, 75–85.
25.
HouseD. (2002). Intonational and visual cues in the perception of interrogative mode in Swedish. Proceedings of Seventh International Conference on Spoken Language Processing, Denver, Colorado, USA, pp. 1957–1960.
26.
HussonF.LêS.PagèsJ. (2017). Exploratory multivariate analysis by example using R. Chapman and Hall/CRC.
27.
KimJ.SironicA.DavisC. (2011). Hearing speech in noise: Seeing a loud talker is better. Perception, 40(7), 853–862.
28.
KrahmerE.RuttkayZ.SwertsM.WesselinkW. (2002). Pitch, eyebrows and the perception of focus. Proceedings of First International Conference on Speech Prosody, Aix-en-Provence, France, pp. 443–446.
29.
KrahmerE.SwertsM. (2007). The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language, 57(3), 396–414.
30.
LambrechtK. (1994). Information structure and sentence form. Cambridge, UK: Cambridge University Press.
31.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
32.
MassaroD.CohenM. (1983). Evaluation and integration of visual and auditory information in speech perception. Journal of Experimental Psychology Human Perception & Performance, 41(5), 751–775.
MirandaL. S. (2015). Análise da entoação do português do Brasil segundo o modelo IPO. Master’s thesis, Faculdade de Letras da Universidade Federal do Rio de Janeiro, Brazil. Retrieved from: http://www.letras.ufrj.br/posverna/mestrado/MirandaLS.pdf.
35.
MirandaL. S.MoraesJ. A.RilliardA. (2019). Audiovisual perception of wh-questions and wh-exclamations in Brazilian Portuguese. Proceedings of Nineteenth International Congress of Phonetic Sciences, Melbourne, Australia, pp. 2941–2945.
36.
MoraesJ. A. (1998). Intonation in Brazilian Portuguese. In HirstD.Di CristoA. (Eds.), Intonation systems: A survey of twenty languages (pp. 179–194). Cambridge, UK: Cambridge University Press.
37.
MoraesJ. A. (2008). The pitch accents in Brazilian Portuguese: Analysis by synthesis. Proceedings of Fourth Conference on Speech Prosody, Campinas, Brazil, pp. 389–397.
38.
MoraesJ. A.MirandaL. S.RilliardA. (2012). Facial gestures in the expression of prosodic attitudes in Brazilian Portuguese. Proceedings of Seventh GSCP International Conference Speech and Corpora, Belo Horizonte, Brazil, pp. 157–161.
39.
NicholsonK. G.BaumS.KilgourA.KohC. K.MunhallK. G.CuddyL. L. (2003). Impaired processing of prosodic and musical patterns after right hemisphere damage. Brain and Cognition, 52, 382–389.
40.
OuniS.CohenM.IshakH.MassaroD. (2006). Visual contribution to speech perception: Measuring the intelligibility of animated talking heads. EURASIP J. AUDIO SPEECH MUSIC PROC., (2006) 2007: 047891. DOI: https://doi.org/10.1155/2007/47891.
41.
PeresD. O.Raposo de MedeirosB.Ferreira NettoW.BaiaM. F. A. (2011). The role of the visual stimuli in the perception of prosody in Brazilian Portuguese. Proceedings of Fifth Conference on Laboratory Approaches to Romance Phonology, Somerville, MA, USA, pp. 136–141.
42.
PierrehumbertJ. (1980). The phonology and phonetics of English intonation. PhD dissertation, Massachusetts Institute of Technology, Cambridge, MA, USA.
RosenblumL. D. (2005). Primacy of multimodal speech perception. In PisoniD. B.RemezR. (Eds.), Handbook of speech perception (pp. 52–78). Oxford, UK: Blackwell Publishing.
45.
SrinivasanR.MassaroD. (2003). Perceiving prosody from the face and voice: Distinguishing statements from echoic questions in English. Language and Speech, 46(1), 1–22.
46.
SteenekenH.GeurtsenF. W. M. (1988). Description of the RSG-10 noise database. TNO Institute for Perception, report IZF, 1988–3.
47.
SumbyW.PollackI. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.
48.
SwertsM.KrahmerE. (2008). Facial expressions and prosodic prominence: Effects of modality and facial area. Journal of Phonetics, 36(2), 219–238.
49.
TorreiraF.ValterssonE. (2015). Phonetic and visual cues to questionhood in French. Phonetica, 72, 20–42.
50.
TubauS.González-FuenteS.PrietoP.EspinalM. A. (2015). Prosody and gesture in the interpretation of yes-answers to negative yes/no-questions. The Linguistic Review, 32(1), 115–142.
51.
Vatikiotis-BatesonE.BarbosaA. V.ChowC. Y.ObergM.TanJ.YehiaH. C. (2007). Audiovisual Lombard speech: Reconciling production and perception. Proceedings of International Conference on Auditory-Visual Speech Processing, Hilvarenbeek, Twhe Netherlands, pp. 45–50.
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
