Segmenting Speech by Mouth: The Role of Oral Prosodic Cues for Visual Speech Segmentation

Abstract

Adults are able to use visual prosodic cues in the speaker’s face to segment speech. Furthermore, eye-tracking data suggest that learners will shift their gaze to the mouth during visual speech segmentation. Although these findings suggest that the mouth may be viewed more than the eyes or nose during visual speech segmentation, no study has examined the direct functional importance of individual features; thus, it is unclear which visual prosodic cues are important for word segmentation. In this study, we examined the impact of first removing (Experiment 1) and then isolating (Experiment 2) individual facial features on visual speech segmentation. Segmentation performance was above chance in all conditions except for when the visual display was restricted to the eye region (eyes only condition in Experiment 2). This suggests that participants were able to segment speech when they could visually access the mouth but not when the mouth was completely removed from the visual display, providing evidence that visual prosodic cues conveyed by the mouth are sufficient and likely necessary for visual speech segmentation.

Keywords

Speech segmentation visual prosody audiovisual speech language acquisition multisensory integration

Get full access to this article

View all access options for this article.

References

Barenholtz

Mavica

Lewkowicz

D. J.

(2016). Language familiarity modulates relative attention to the eyes and mouth of a talker. Cognition, 147, 100–105.

Boersma

Weenink

(2011). Praat: Doing phonetics by computer (Version 5.2.11) [Computer Software]. http://www.praat.org/

Cunillera

Càmara

Laine

Rodríguez-Fornells

(2010). Speech segmentation is facilitated by visual cues. The Quarterly Journal of Experimental Psychology, 63, 260–274.

Cutler

(2012). Native listening: Language experience and the recognition of spoken words. MIT Press.

Cvejic

Kim

Davis

(2012). Recognizing prosody across modalities, face areas and speakers: Examining perceivers’ sensitivity to variable realizations of visual prosody. Cognition, 122, 442–453.

de la Cruz-Pavía

Gervain

Vatikiotis-Bateson

Werker

J. F.

(2019). Finding phrases: On the role of co-verbal facial information in learning word order in infancy. PLOS ONE, 14(11), Article e0224786.

Dohen

Loevenbruck

Hill

(2006). Visual correlates of prosodic contrastive focus in French: Description and inter-speaker variability. Proceedings of Speech Prosody, 2006, 221–224.

Fernandes

Ventura

Kolinsky

(2007). Statistical information and coarticulation as cues to word boundaries: A matter of signal quality. Perception & Psychophysics, 69, 856–864.

Graf

P. H.

Cosatto

Strom

Huang

F. J.

(2002, May 21). Visual prosody: Facial movements accompanying speech [Conference session]. Fifth IEEE International Conference on Automatic Face and Gesture Recognition, Washington, DC, United States.

10.

Grant

K. W.

Seitz

P. F.

(2000). The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustic Society of America, 108, 1197–1208.

11.

Green

J. R.

Nip

I. S. B.

Wilson

E. M.

Mefferd

A. S.

Yunusova

(2010). Lip movement exaggerations during infant-directed speech. Journal of Speech, Language, and Hearing Research, 53, 1529–1542.

12.

Hollich

Newman

R. S.

Jusczyk

P. W.

(2005). Infants’ use of synchronized visual information to separate streams of speech. Child Development, 76, 598–613.

13.

Johnson

E. K.

Jusczyk

P. W.

(2001). Word segmentation by 8-month-olds: When speech cues count more than statistics. Journal of Memory and Language, 44, 548–567.

14.

Jordan

T. R.

Thomas

S. M.

(2011). When half a face is as good as a whole: Effects of simple substantial occlusion on visual and audiovisual speech perception. Attention, Perception, & Psychophysics, 73, 2270–2285.

15.

Kim

Cvejic

Davis

(2014). Tracking eyebrows and head gestures associated with spoken prosody. Speech Communication, 57, 317–330.

16.

Kuhl

P. K.

Meltzoff

A. N.

(1982). The bimodal perception of speech in infancy. Science, 218, 1138–1141.

17.

Lansing

C. R.

McConkie

G. W.

(1999). Attention to facial regions in segmental and prosodic visual speech perception tasks. Journal of Speech, Language, and Hearing Research, 42, 526–539.

18.

Lansing

C. R.

McConkie

G. W.

(2003). Word identification and eye fixation locations in visual and visual-plus-auditory presentations of spoken sentences. Perception and Psychophysics, 65, 536–552.

19.

Lewkowicz

D. J.

Hansen-Tift

A. M.

(2012). Infants deploy selective attention to the mouth of a talking face when learning speech. Proceedings from the National Academy of Sciences of the United States of America, 109, 1431–1436.

20.

Lusk

L. G.

Mitchel

A. D.

(2016). Differential gaze patterns on eyes and mouth during audiovisual speech segmentation. Frontiers in Psychology, 7, Article 521.

21.

MacDonald

Andersen

Bachmann

(2000). Hearing by eye: How much spatial degradation can be tolerated? Perception, 29, 1155–1168.

22.

Mani

Schneider

(2013). Speaker identity supports phonetic category learning. Journal of Experimental Psychology: Human Perception and Performance, 39, 623–629.

23.

Massaro

D. W.

(1998). Perceiving talking faces: From speech perception to a behavioral principle. MIT Press.

24.

Mitchel

A. D.

Christiansen

M. H.

Weiss

D. J.

(2014). Multimodal integration in statistical learning: evidence from the McGurk illusion. Frontiers in Psychology, 5, Article 407.

25.

Mitchel

A. D.

Weiss

D. J.

(2010). What’s in a face? Visual contributions to speech segmentation. Language and Cognitive Processes, 25, 456–482.

26.

Mitchel

A. D.

Weiss

D. J.

(2011). Learning across senses: Cross-modal effects in multisensory statistical learning. Journal of Experimental Psychology: Learning, Memory, & Cognition, 37, 1081–1091.

27.

Mitchel

A. D.

Weiss

D. J.

(2014). Visual speech segmentation: Using facial cues to locate word boundaries in continuous speech. Language, Cognition & Neuroscience, 29, 771–780.

28.

Munhall

K. G.

Jones

J. A.

Callan

D. E.

Kuratate

Vatikiotis-Bateson

(2004). Visual prosody and speech intelligibility head movement improves auditory speech perception. Psychological Science, 15, 133–137.

29.

Munhall

K. G.

Kroos

Jozan

Vatikiotis-Bateson

(2004). Spatial frequency requirements for audiovisual speech perception. Perception & Psychophysics, 66, 574–583.

30.

Rosenblum

L. D.

(2008). Speech perception as a multimodal phenomenon. Current Directions in Psychological Science, 17, 405–409.

31.

Saffran

J. R.

(2003). Statistical language learning: Mechanisms and constraints. Directions in Psychological Science, 12, 110–114.

32.

Saffran

J. R.

Newport

E. L.

Aslin

R. N.

(1996). Word segmentation: The role of distributional cues. Journal of Memory and Language, 35(4), 606–621.

33.

Sanders

L. D.

Neville

H. J.

(2000). Lexical, syntactic, and stress-pattern cues for speech segmentation. Journal of Speech, Language, and Hearing Research, 43, 1301–1321.

34.

Scarborough

Keating

Mattys

S. L.

Cho

Alwan

(2009). Optical phonetics and visual perception of lexical and phrasal stress in English. Language and Speech, 52(2-3), 135–175.

35.

Sell

A. J.

Kaschak

M. P.

(2009). Does visual speech information affect word segmentation? Memory and Cognition, 37, 889–894.

36.

Shukla

Nespor

Mehler

(2007). An interaction between prosody and statistics in the segmentation of fluent speech. Cognitive Psychology, 54(1), 1–32.

37.

Smith

M. R.

Cutler

Butterfield

Nimmo-Smith

(1989). The perception of rhythm and word boundaries in noise-masked speech. Journal of Speech, Language, and Hearing Research, 32, 912–920.

38.

Sumby

W. H.

Pollack

(1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.

39.

Swerts

Krahmer

(2008). Facial expression and prosodic prominence: Effects of modality and facial area. Journal of Phonetics, 36, 219–238.

40.

Teinonen

Aslin

R. N.

Alku

Csibra

(2008). Visual speech contributes to phonetic learning in 6-month-old infants. Cognition, 108, 850–855.

41.

Thiessen

E. D.

(2010). Effects of visual information on adults’ and infants’ auditory statistical learning. Cognitive Science, 34, 1093–1106.

42.

Thiessen

E. D.

Saffran

J. R.

(2003). When cues collide: Use of stress and statistical cues to word boundaries by 7- to 9-month-old infants. Developmental Psychology, 39(4), 706–716.

43.

Thomas

S. M.

Jordan

T. R.

(2004). Contributions of oral and extraoral facial movement to visual and audiovisual speech perception. Journal of Experimental Psychology: Human Perception and Performance, 30, 873–888.

44.

Tyler

M. D.

Cutler

(2009). Cross-language differences in cue use for speech segmentation. The Journal of the Acoustical Society of America, 126, 367–376.

45.

Vatikiotis-Bateson

Ostry

D. J.

(1995). An analysis of the dimensionality of jaw motion in speech. Journal of Phonetics, 23, 101–117.

46.

Weatherhead

White

K. S.

(2017). Read my lips: Visual speech influences word processing in infants. Cognition, 160, 103–109.

47.

Weikum

W. M.

Vouloumanos

Navarra

Soto-Faraco

Sebastián-Gallés

Werker

J. F.

(2007). Visual language discrimination in infancy. Science, 316(5828), Article 1159.

48.

Yehia

Rubin

Vatikiotis-Bateson

(1998). Quantitative association of vocal-tract and facial behavior. Speech Communication, 26, 23–43.

49.

Yehia

H. C.

Kuratate

Vatikiotis-Bateson

(2002). Linking facial animation, head motion and speech acoustics. Journal of Phonetics, 30, 555–568.