A series of experiments measured the audiovisual stimulus onset asynchrony (SOAAV), yielding facilitative multisensory integration. We evaluated (1) the range of SOAAV over which facilitation occurred when unisensory stimuli were weak; (2) whether the range of SOAAV producing facilitation supported the hypothesis that physiological simultaneity of unisensory activity governs multisensory facilitation; and (3) whether AV multisensory facilitation depended on relative stimulus intensity. We compared response-time distributions to unisensory auditory (A) and visual (V) stimuli with those to AV stimuli over a wide range (300 and 20 ms increments) of SOAAV, across four conditions of varying stimulus intensity. In condition 1, the intensity of unisensory stimuli was adjusted such that d′ ≈ 2. In condition 2, V stimulus intensity was increased (d′ > 4), while A stimulus intensity was as in condition 1. In condition 3, A stimulus intensity was increased (d′ > 4) while V stimulus intensity was as in condition 1. In condition 4, both A and V stimulus intensities were increased to clearly suprathreshold levels (d′ > 4). Across all conditions of stimulus intensity, significant multisensory facilitation occurred exclusively for simultaneously presented A and V stimuli. In addition, facilitation increased as stimulus intensity increased, in disagreement with inverse effectiveness. These results indicate that the requirements for facilitative multisensory integration include both physical and physiological simultaneity.
AlvaradoJ. C.VaughanJ. W.StanfordT. R.SteinB. E. (2007). Multisensory versus unisensory integration: Contrasting modes in the superior colliculus. Journal of Neurophysiology, 97, 3193–3205. doi:10.1152/jn.00018.2007
2.
BarlowR. B.SnodderlyD. M.SwadlowH. A. (1978). Intensity coding in primate visual system. Experimental Brain Research, 31, 163–177. doi:10.1007/BF00237597
3.
CallanD. E.CallanA. M.KroosC.Vatikiotis-BatesonE. (2001). Multimodal contribution to speech perception revealed by independent component analysis: A single sweep EEG case study. Cognitive Brain Research, 10, 349–353. doi:10.1016/S0926-6410(00)00054-9
4.
CalvertG. A. (2001). Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex, 11, 1110–1123. doi:10.1093/cercor/11.12.1110
5.
CampbellF. W.GreenD. G. (1965). Monocular vs. binocular visual acuity. Nature, 208, 191–192. doi:10.1038/208191a0
6.
CelesiaG. G.PulettiF. (1971). Auditory input to the human cortex during states of drowsiness and surgical anesthesia. Electroencephalography and Clinical Neurophysiology, 31, 603–609. doi:10.1016/0013-4694(71)90076-9
7.
ClarkV. P.HillyardS. A. (1996). Spatial selective attention affects early extrastriate but not striate components of the visual evoked potential. Journal of Cognitive Neuroscience, 8, 387–402. doi:10.1162/jocn.1996.8.5.387
8.
DiederichA.ColoniusH. (2004). Bimodal and trimodal multisensory enhancement: Effects of stimulus onset and intensity on reaction time. Perception and Psychophysics, 66, 1388–1404. doi:10.3758/BF03195006
9.
EggermontJ. J. (1998). Azimuth coding in primary auditory cortex of the cat: Relative latency and interspike interval representation. Journal of Neurophysiology, 80, 2151–2161
10.
FosterD. H.BischofW. F. (1991). Bootstrap estimates of the statistical accuracy of thresholds obtained from psychometric functions. Spatial Vision, 11, 135–139
11.
FoxeJ. J.SimpsonG. V. (2002). Flow of activation from V1 to frontal cortex in humans: A framework for defining “early” visual processing. Experimental Brain Research, 142, 139–150. doi:10.1007/s00221-001-0906-7
12.
FoxeJ. J.StrugstadE. C.SehatpourP.MolholmS.PasiekaW.SchroederC. E.McCourtM. E. (2008). Parvocellular and magnocellular contributions to the initial generators of the visual evoked potential: High-density electrical mapping of the “C1” component. Brain Topography, 21, 11–21. doi:10.1007/s10548-008-0063-4
13.
FrassinettiF.BologniniN.LadàvasE. (2002). Enhancement of visual perception by crossmodal visuo–auditory interaction. Experimental Brain Research, 147, 332–343. doi:10.1007/s00221-002-1262-y
14.
GiardM. H.PeronnetF. (1999). Auditory–visual integration during multi-modal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11, 473–490. doi:10.1162/089892999563544
15.
HershensonM. (1962). Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology, 63, 289–293. doi:10.1037/h0039516
16.
HolmesN. P. (2007). The law of inverse effectiveness in neurons and behavior: Multisensory integration versus normal variability. Neuropsychologia, 45, 3340–3345. doi:10.1016/j.neuropsychologia.2007.05.025
17.
HolmesN. P. (2009). The principle of inverse effectiveness in multisensory integration: Some statistical considerations. Brain Topography, 21, 168–176. doi:10.1007/s10548-009-0097-2
18.
HolmesN. P.SpenceC. (2005). Multisensory integration: Space, time and superadditivity. Current Biology, 15, R762–R764. doi:10.1016/j.cub.2005.08.058
19.
IacoboniM.ZaidelE. (2003). Interhemispheric visuo-motor integration in humans: The effect of redundant targets. European Journal of Neuroscience, 17, 1981–1986. doi:10.1046/j.1460-9568.2003.02602.x
20.
JaskowskiP. (1985). The effect of visual adaptation on simple motor reaction time: Pt. 1. Studia Psychologica, 27, 191–201
21.
JeffressL. A. (1948). A place theory of sound localization. Journal of Comparative and Physiological Psychology, 41, 35–39. doi:10.1037/h0061495
22.
JeffreysD. A.AxfordJ. G. (1972). Source locations of pattern-specific components of human visual evoked potentials. II. Component of extrastriate cortical origin. Experimental Brain Research, 16, 22–40. doi:10.1007/BF00233372
23.
LakatosP.ChenC. M.O'ConnellM. N.MillsA.SchroederC. E. (2007). Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron, 53, 279–292. doi:10.1016/j.neuron.2006.12.011
24.
LeggeG. E. (1984). Binocular contrast summation II. Quadratic summation. Vision Research, 24, 385–394. doi:10.1016/0042-6989(84)90064-6
25.
MansfieldR. J. W. (1973). Latency functions in human vision. Vision Research, 13, 2219–2234. doi:10.1016/0042-6989(73)90224-1
26.
MaunsellJ. H. R.GhoseG. M.AssadJ. A.McAdamsC. J.BoudreauC. E.NoeragerB. D. (1999). Visual response latencies of magnocellular and parvocellular LGN neurons in macaque monkeys. Visual Neuroscience, 16, 1–14
27.
MeredithM. A. (2002). On the neuronal basis for multisensory convergence: A brief overview. Cognitive Brain Research, 14, 1–40. doi:10.1016/S0926-6410(02)00059-9
MeredithM. A.NemitzJ. W.SteinB. E. (1987). Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. Journal of Neuroscience, 7, 3215–3229
30.
MeredithM. A.SteinB. E. (1986). Visual auditory and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neuroscience, 56, 640–662
31.
MillerJ. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14, 247–279. doi:10.1016/0010-0285(82)90010-X
32.
MillerJ. (1986). Timecourse of coactivation in bimodal divided attention. Perception and Psychophysics, 40, 331–343. doi:10.3758/BF03203025
33.
MiniussiC.GirelliM.MarziC. A. (1998). Neural site of the redundant target effect: Electrophysiological evidence. Journal of Cognitive Neuroscience, 10, 216–225. doi:10.1162/089892998562663
34.
MolholmS.RitterW.MurrayM. M.JavittD. C.SchroederC. E.FoxeJ. J. (2002). Multisensory auditory–visual interactions during early sensory processing in humans: A high-density electrical mapping study. Cognitive Brain Research, 14, 115–128. doi:10.1016/S0926-6410(02)00066-6
35.
MolholmS.SehatpourP.MehtaA. D.ShpanerM.Gomez-RamirezM.OrtigueS.FoxeJ. J. (2006). Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. Journal of Neurophysiology, 96, 721–729. doi:10.1152/jn.00285.2006
36.
MordkoffJ. T.YantisS. (1991). An interactive race model of divided attention. Journal of Experimental Psychology, 17(2), 520–538. doi:10.1037/0096-1523.17.2.520
37.
MurrayM. M.FoxeJ. J.HigginsB. A.JavittD. C.SchroederC. E. (2001). Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: A high-density electrical mapping study. Neuropsychologia, 39, 828–844. doi:10.1016/S0028-3932(01)00004-5
38.
OttoT. U.MamassianP. (2012). Noise and correlations in parallel perceptual decision making. Current Biology, 22, 1–6. doi:10.1016/j.cub.2012.05.031
39.
PhillipsD. P. (1998). Factors shaping the response latencies of neurons in the cat's auditory cortex. Behavioural Brain Research, 93, 33–41. doi:10.1016/S0166-4328(97)00139-3
40.
PiéronH. (1952). The Sensations: Their Functions, Processes and Mechanisms. London: Frederick Muller Ltd
41.
PrestrudeA. M. (1971). Visual latencies at photopic levels of retinal illuminance. Vision Research, 11, 351–361. doi:10.1016/0042-6989(71)90246-X
42.
RaabD. H. (1962). Statistical facilitation of simple reaction times. Transactions of the New York Academy of Sciences, 24, 574–590. doi:10.1111/j.2164-0947.1962.tb01433.x
43.
RossL. A.Saint-AmourD.LeavittV. M.JavittD. C.FoxeJ. J. (2007). Do you see what I'm saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex, 17, 1147–1153. doi:10.1093/cercor/bhl024
44.
RoufsJ. A. J. (1963). Perception lag as a function of stimulus luminance. Vision Research, 3, 81–91. doi:10.1016/0042-6989(63)90070-1
45.
SenkowskiD.Saint-AmourD.HöfleM.FoxeJ. J. (2011). Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. NeuroImage, 56, 2200–2208. doi:10.1016/j.neuroimage.2011.03.075
46.
SerinoA.FarnèA.RinaldesiM. L.HaggardP.LàdavasE. (2007). Can vision of the body ameliorate impaired somatosensory function?Neuropsychologia, 45, 1101–1107. doi:10.1016/j.neuropsychologia.2006.09.013
47.
SongS.MillerK. D.AbbottL. F. (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nature Neuroscience, 3, 919–926. doi:10.1038/78829
48.
SteinB. E.MeredithM. A. (1993). The Merging of the Senses. Cambridge: MIT Press
49.
StufflebeamS. M.PoeppelD.RowleyH. A.RobertsT. P. L. (1998). Peri-threshold encoding of stimulus frequency and intensity in the M100 latency. NeuroReport, 9, 91–94
50.
SupekS.AinesC. J.RankenD.BestE.FlynnE. R.WoodC. C. (1999). Single vs. paired visual stimuli: Superposition of early neuromagnetic responses and retinotopy in extrastriate cortex in humans. Brain Research, 830, 43–55. doi:10.1016/S0006-8993(99)01316-5
51.
TurrigianoG. G.LeslieK. R.DesaiN. S.RutherfordL. C.NelsonS. B. (1998). Activity-dependent scaling of quantal amplitude in neocortical neurons. Nature, 391, 892–896. doi:10.1038/36103
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
