Perceiving Prosody from the Face and Voice: Distinguishing Statements from Echoic Questions in English

Abstract

We examined the processing of potential auditory and visual cues that differentiate statements from echoic questions. In Experiment 1, four natural speech statement-question pairs were identified by participants, and then analyzed to determine which characteristics were ecologically valid. These characteristics were tested in subsequent experiments to determine if they were also functionally valid. In Experiment 2, the characteristics of the most discriminable utterance pair were successfully extended to the other utterance pairs. For Experiment 3, an auditory continuum (varying in F0, amplitude, duration) was crossed with a visual continuum (varying in eyebrow raise, head tilt), using synthetic speech and a computer-animated head. Participants judged five levels along each of these two speech continua between a prototypical statement and prototypical question, in an expanded factorial design. Experiments 4 and 5 were unable to appreciably enhance the weak visual effect relative to the strong auditory effect (from Experiment 3). Overall, we found that both auditory and visual cues reliably conveyed statement and question intonation, were successfully synthesized, and generalized to other utterances. However, the weak visual effect relative to the robustly strong auditory effect precluded optimal integration and conclusive examination of information processing through model-fitting.

Keywords

bimodal perception prosody question synthesis

Get full access to this article

View all access options for this article.

References

Bernstein, L.E. ,& Eberhardt, S.P. (1986). Johns Hopkins lip-reading corpus video-disc set . Baltimore, MD: Johns Hopkins University .

Granstrom, B. , House, D. , & Lundeberg, M. (1999). Prosodic cues in multimodal speech perception . Proceedings of the International Congress of Phonetic Sciences (ICPhS 99), 655-658.

Granstrom, B. , House, D. , & Swerts, M.G. (2002). Multimodal feedback cues in human-machine interactions . In B. Bel & I. Marlien (Eds.), Proceedings of the Speech Prosody 2002 Conference, 347-350.

't Hart, J. , & Collier, R. (1975). Integrating different levels of intonation analysis . Journal of Phonetics, 235-255.

House, D. (2002). Intonation and visualcues in the perception of interrogative mode in Swedish. Proceedings of ICSLP 2002, 1957-1960.

House, D. , Beskow, J. , & Granstrom, B. (2001). Timing and interaction of visual cues for prominence in audiovisual speech perception. Proceedings of Eurospeech 2001, 387-390.

Kadlec, H. (1999). Statistical properties of d'and beta estimates of signal detection theory. Psychological Methods, 22-43.

Lansing, C.R. , & McConkie, G.W. (1999). Attention to facial regions in the segmental and prosodic visual speech perception tasks. Journal of Speech, Language, and Hearing Research, 526-539.

Lieberman, P. (1967). Intonation, perception and languages. MIT Press: Cambridge, MA.

10.

Majewski, W. , & Blasdell, R. (1969). Influence of fundamental frequency cues on the perception of some synthetic intonation contours. Journal of the Acoustical Society of America, 450-457.

11.

Massaro, D.W. (1987). Speech perception by ear and eye: A paradigm for psychological inquiry. Hillsdale, NJ: Erlbaum.

12.

Massaro, D.W. (1994). A pattern recognition account of decision making . Memory and Cognition, 616-627.

13.

Massaro, D.W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. MIT Press: Cambridge, MA.

14.

Massaro, D.W. , & Beskow, J. (2002). Multimodal speech perception: A paradigm for speech science. In B. Granstrom , D. House , & I. Karlsson (Eds.), Multilmodality in language and speech systems (pp.45-71). Kluwer Academic Publishers, Dordrecht, The Netherlands.

15.

Massaro, D.W. , & Cohen, M.M. (1983). Evaluation and integration of visual and auditory information in speech perception. Journal of Experimental Psychology: Human Perception and Performance, 753-771.

16.

Massaro, D.W. , & Cohen, M.M. (1993). The paradigm and the fuzzy logical model of perception are alive and well. Journal of Experimental Psychology: General , 115-124.

17.

Massaro, D.W. , Cohen, M.M. , Beskow, J. , Cole, R.A. (2000). Developing and evaluating conversational agents . In J. Cassell , J. Sullivan , S. Prevost , E. Churchill (Eds.), Embodied conversational agents(pp.287-318). MIT Press: Cambridge, MA.

18.

Miller, J. (1996). The sampling distribution of d'. Perception & Psychophysics, 65-72.

19.

O'Shaughnessy, D. (1979). Linguistic features in fundamental frequency patterns . Journal of Phonetics, 119-145.

20.

Ryalls, J. , Dorze, G. , Lever, N. , Oullet, L. , & Larfeuil, C. (1994). The effects of age and sex on speech intonation and duration for matched statements and questions in French. Journal of the Acoustical Society of America, 2274-2276.

21.

Sjolander, K. , & Beskow, J. (1999). WaveSurfer-an open source speech tool. Stockholm, Sweden: Center for speech Technology (CTT) at KTH. <http://www.speech.kth.se/wavesurfer>.

22.

Studdert-Kennedy, M. , & Hadding, K. (1973). Auditory and linguistic processes in the perception of intonation. Language and Speech, 293-313.

23.

Taylor, P.A. , Black, A. , & Caley, C. (1998). The architecture of the Festival speech synthesis system. Proceedings of Third ESCA Workshop in Speech Synthesis, 147-151.

24.

Terken, J. , & Lemeer, G. (1988). Effects of segmental quality and intonation on quality judgments for texts and utterances. Journal of Phonetics , 453-457.

25.

Wouters, J. , Rundle, B. , & Macon, M. (1999). Authoring tools for speech synthesis using the sable markup standard. Proceedings of Eurospeech , 963-966.