We report four experiments designed to determine whether visual information affects judgments of acoustically-specified nonspeech events as well as speech events (the “McGurk effect”). Previous findings have shown only weak McGurk effects for nonspeech stimuli, whereas strong effects are found for consonants. We used click sounds that serve as consonants in some African languages, but that are perceived as nonspeech by American English listeners. We found a significant McGurk effect for clicks presented in isolation that was much smaller than that found for stop-consonant-vowel syllables. In subsequent experiments, we found strong McGurk effects, comparable to those found for English syllables, for click-vowel syllables, and weak effects, comparable to those found for isolated clicks, for excised release bursts of stop consonants presented in isolation. We interpret these findings as evidence that the potential contributions of speech-specific processes on the McGurk effect are limited, and discuss the results in relation to current explanations for the McGurk effect.
Best, C.T. (1984). Discovering messages in the medium: Speech and the prelinguistic infant. In H. E. Fitzgerald , B. Lester, & M. Yongman (Eds.), Advances in pediatric psychology, Vol.2 (pp. 97 -145). New York: Plenum .
2.
Best, C.T. (1994). The emergence of native-language phonological influences in infants: A perceptual assimilation model. In J. C. Goodman & H. C. Nusbaum (Eds.), The development of speech perception: The transition from speech sounds to spoken words (pp. 167-224). Cambridge: MIT Press.
3.
Best, C.T. (1995). A direct realist perspective on cross-language speech perception. In W. Strange (Ed.), Cross-language speech perception (pp. 171-204). Baltimore: York Press.
4.
Best, C.T., & Avery, R.A. (1999). Left hemisphere advantage for click consonants is determined by linguistic significance. Psychological Science , 10, 65-70.
5.
Best, C.T., McRoberts, G.W., & Sithole, N.M. (1988). Examination of perceptual reorganization for non-native speech contrasts: Zulu click discrimination by English-speaking adults and infants. Journal of Experimental Psychology: Human Perception and Performance, 14, 345-360.
6.
Calvert, G.A. (2001). Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex, 11, 1110-1123.
Calvert, G.A., & Campbell, R. (2003). Reading speech from still and moving faces: The neural substrates of visible speech. Journal of Cognitive Neuroscience, 15, 57-70.
9.
Calvert, G.A., Campbell, R., & Bremmer, M.J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology, 10, 649-657.
10.
Campbell, R., MacSweeney, M., Surguladze, S., Calvert, G., McGuire, P., Suckling, J., Brammer, M.J., & David, A.S. (2001). Cortical substrates for the perception of face actions: An fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Cognitive Brain Research, 12, 233-243.
11.
De Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and eye. Cognition and Emotion, 14, 289-311.
12.
Diehl, R.L., & Kluender, K.R. (1989). On the objects of speech perception. Ecological Psychology, 1, 121-144.
13.
Driver, J. (1996). Enhancement of selective listening of illusory mislocation of speech sounds due to lip-reading. Nature, 381, 66-68.
14.
Fant, G. (1973). Speech sounds and features. Cambridge, MA: MIT Press.
15.
Fodor, J.A. (1983). The modularity of mind: An essay on faculty psychology . Cambridge, MA: MIT Press.
16.
Fowler, C.A. (1986). An event approach to the study of speech perception from a direct-realist perspective. Journal of Phonetics, 14, 3-28.
17.
Fowler, C.A. (1994). Speech perception: Direct realist theory. In Encyclopedia of language and linguistics (pp. 4199-4203). Oxford: Pergamon Press.
18.
Fowler, C.A., & Dekle, D.J. (1991). Listening with eye and hand: Crossmodal contributions to speech perception. Journal of Experimental Psychology: Human Perception and Performance, 17, 816-828.
19.
Gaver, W.W. (1993). What in the world do we hear? An ecological approach to auditory event perception. Ecological Psychology, 5, 1-29.
20.
Gibson, J.J. (1966). The senses considered as perceptual systems. Boston, MA: Houghton-Mifflin.
21.
Gibson, J.J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.
22.
Green, K.P., Kuhl, P.K., Meltzoff, A.N., & Stevens, E.B. (1991). Integrating speech information across talkers, genders and sensory modality: Female faces and male voices in the McGurk effect . Perception and Psychophysics, 50, 524-536.
23.
Green, K.P., & Norrix, L.W. (1997). Acoustic cues to place of articulation and the McGurk effect: The role of release bursts, aspiration, and formant transitions . Journal of Speech, Language, and Hearing Research, 40, 646-665.
24.
Grossman, E., Donnelly, M., Price, R., Pickens, D., Morgan, V., Neighbor, G., & Blake, R. (2000). Brain areas involved in perception of biological motion. Journal of Cognitive Neuroscience, 12, 711-720.
25.
Jones, J.A., & Callan, D.E. (2003). Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. NeuroReport, 14, 1129-1133.
26.
Ladefoged, P., & Maddieson, I. (1996). The sounds of the world's languages. Oxford, U.K.: Blackwell Publishers.
27.
Ladefoged, P., & Traill, A. (1994). Clicks and their accompaniments. Journal of Phonetics, 22,33-64.
28.
Liberman, A.M., Delattre, P.C., Cooper, F.S., & Gerstman, L.J. (1954). The role of consonant-vowel transitions in the perception of the stop and nasal consonants. Psychological Monographs, 68, 1-13.
29.
Liberman, A.M., & Mattingly, I.G. (1985). The motor theory revised. Cognition, 21, 1 - 36.
30.
MacDonald, J., & McGurk, H. (1978). Visual influences on speech perception. Perception and Psychophysics, 24, 253-257.
31.
Manuel, S.Y., Repp, B.H., Liberman, A.M., & Studdert-Kennedy, M. (1983) Exploring the "McGurk effect."Talk presented at the 24th meeting of the Psychonomic Society, November, 1983.
32.
Massaro, D.W. (1987). Speech perception by ear and eye: A paradigm for psychological inquiry. Hillsdale, NJ: Lawrence Erlbaum Associates.
33.
Massaro, D.W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press.
34.
Massaro, D.W., & Cohen, M.M. (1993) Perceiving asynchronous bimodal speech in consonant-vowel and vowel syllables. Speech Communication, 13, 127-134.
35.
Massaro, D.W., & Egan, P.B. (1996). Perceiving affect from the voice and face. Psychonomic Bulletin and Review, 3, 215-221.
36.
McGurk,H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746 - 748.
37.
Möttönen, R., Krause, C.M., Tiippana, K., & Sams, M. (2002). Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research, 13, 417-425.
38.
Olson, I.R., Gatenby, J.C., & Gore, J.C. (2002). A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Cognitive Brain Research, 14, 129-138.
39.
Paulesu, E., Perani, D., Blasi, V., Silani, G., Borghese, N.A., de Giovanni, U., Sensolo, S., & Fazio, F. (2003). A functional-anatomical model for lipreading. Journal of Neurophysiology, 90, 2005-2013.
40.
Puce, A., Allison, T., Bentin, S., Gore, J.C., & McCarthy, G. (1998). Temporal cortext activation in humans viewing eye and mouth movements. Journal of Neuroscience, 18,2188-2199.
41.
Radeau, M. (1994). Auditory-visual spatial interactions and modularity . Cahiers de Psychologie Cognitive, 13, 3-51.
42.
Rizzolatti, G., Fogassi, L., & Gallese, V. (2002). Motor and cognitive functions of the ventral premotor cortex, Current Opinion in Neurobiology, 12, 149-154.
43.
Rosenblum, L.D. (1987). Towards an ecological alternative to the motor theory. Perceiving-Acting Workshop, 2, 25-28.
44.
Rosenblum, L.D. (1994). How special is audiovisual speech integration?Cahiers de Psychologie Cognitive, 13, 110-116.
45.
Rosenblum, L.D., & Fowler, C.A. (1991). An audio-visual investigation of the loudness/effort effect for speech and nonspeech events. Journal of Experimental Psychology: Human Perception and Performance, 17, 976-985.
46.
Rosenblum, L.D., & Saldaña, H.M. (1992). Discrimination tests of visually influenced syllables . Perception and Psychophysics, 52, 461-473.
47.
Rosenblum, L.D., & Saldaña, H.M. (1996). An audiovisual test of kinematic primitives for visual speech perception. Journal of Experimental Psychology: Human Perception and Performance, 22, 318-331.
48.
Rubin, P.E. (1995). HADES: A case study on the development of a signal analysis system. In A. Syrdal, R. Bennett, & S. Greenspan (Ed.), Applied signal technology. (pp. 501-520) Florida : CRC Press.
49.
Saldaña, H.M., & Rosenblum, L.D. (1993). Visual influences on auditory pluck and bow judgments. Perception and Psychophysics, 54, 406-416.
50.
Summerfield, Q., & McGrath, M. (1984). Detection and resolution of audio-visual incompatibility in the perception of vowels. Quarterly Journal of Experimental Psychology, 36A, 51-74.
