In spite of accumulating evidence for the spatial rule governing cross-modal interaction according to the spatial consistency of stimuli, it is still unclear whether 3D spatial consistency (i.e., front/rear of the body) of stimuli also regulates audiovisual interaction. We investigated how sounds with increasing/decreasing intensity (looming/receding sound) presented from the front and rear space of the body impact the size perception of a dynamic visual object. Participants performed a size-matching task (Experiments 1 and 2) and a size adjustment task (Experiment 3) of visual stimuli with increasing/decreasing diameter, while being exposed to a front- or rear-presented sound with increasing/decreasing intensity. Throughout these experiments, we demonstrated that only the front-presented looming sound caused overestimation of the spatially consistent looming visual stimulus in size, but not of the spatially inconsistent and the receding visual stimulus. The receding sound had no significant effect on vision. Our results revealed that looming sound alters dynamic visual size perception depending on the consistency in the approaching quality and the front–rear spatial location of audiovisual stimuli, suggesting that the human brain differently processes audiovisual inputs based on their 3D spatial consistency. This selective interaction between looming signals should contribute to faster detection of approaching threats. Our findings extend the spatial rule governing audiovisual interaction into 3D space.
BachD. R.NeuhoffJ. G.PerrigW.SeifritzE. (2009). Looming sounds as warning signals: The function of motion cues, International Journal of Psychophysiology, 74: 28–33.
3.
BachD. R.SchächingerH.NeuhoffJ. G.EspositoF.Di SalleF.LehmannC.HerdenerM.SchefflerK.SeifritzE. (2008). Rising sound intensity: An intrinsic warning cue activating the amygdala, Cerebral Cortex, 18: 145–150.
4.
BallW.TronickE. (1971). Infant responses to impending collision: Optical and real, Science, 171: 818–820.
5.
BeverleyK. I.ReganD. (1979). Visual perception of changing size: The effect of object size, Vision Research, 19: 1093–1104.
6.
CappeC.ThelenA.RomeiV.ThutG.MurrayM. M. (2012). Looming signals reveal synergistic principles of multisensory integration, The Journal of Neuroscience, 32: 1171–1182.
7.
CappeC.ThutG.RomeiV.MurrayM. M. (2009). Selective integration of auditory–visual looming cues by humans, Neuropsychologia, 47: 1045–1052.
8.
ChenL.VroomenJ. (2013). Intersensory binding across space and time: A tutorial review, Attention, Perception, & Psychophysics, 75: 790–811.
9.
ChenL.YuanX.XuQ.WangY.JiangY. (2016). Subliminal impending collision increases perceived object size and enhances pupillary light reflex, Frontiers in Psychology, 7: 1897.
10.
ChenY.-C.SpenceC. (2017). Assessing the role of the ‘unity assumption’ on multisensory integration: A review, Frontiers in Psychology, 8: 445.
11.
CooperE. A.JiangH.VildavskiV.FarrellJ. E.NorciaA. M. (2013). Assessment of OLED displays for vision research, Journal of Vision, 13: 1–16.
12.
DosherB. A.SperlingG.WurstS. A. (1986). Tradeoffs between stereopsis and proximity luminance covariance as determinants of perceived 3D structure, Vision Research, 26: 973–990.
13.
FinlaysonJ. J.RemingtonR. W.GroveP. M. (2012). The role of presentation method and depth singletons in visual search for objects moving in depth, Journal of Vision, 12: 8, 1–9.
14.
FranconeriS. L.SimonsD. J. (2003). Moving and looming stimuli capture attention, Perception & Psychophysics, 65: 999–1010.
15.
FreemanE.DriverJ. (2008). Direction of visual apparent motion driven solely by timing of a static sound, Current Biology, 18: 1262–1266.
16.
FreydJ. J.FinkeR. A. (1984). Representational momentum, Journal of Experimental Psychology: Learning, Memory, & Cognition, 10: 126–132.
17.
GhazanfarA. A.SchroederC. E. (2006). Is neocortex essentially multisensory?Trends in Cognitive Sciences, 10: 278–285.
18.
GirardS.PellandM.LeporeF.CollignonO. (2013). Impact of the spatial congruence of redundant targets on within-modal and cross-modal integration, Experimental Brain Research, 224: 275–285.
19.
GrassiM.DarwinC. J. (2006). The subjective duration of ramped and damped sounds, Perception & Psychophysics, 68: 1382–1392.
20.
GuskiR. (1992). Acoustic tau: An easy analogue to visual tau?Ecological Psychology, 4: 189–197.
21.
GuttmanS. E.GilroyL. A.BlakeR. (2005). Hearing what the eyes see: Auditory encoding of visual temporal sequences, Psychological Science, 16: 228–235.
22.
HassanO.ThompsonP.HammetS. T. (2016). Perceived speed in peripheral vision can go up or down, Journal of Vision, 16: 1–7.
23.
HeffnerR. S.KoayG.HeffnerH. E. (1996). Sound localization in chinchillas III: Effect of pinna removal, Hearing Research, 99: 13−21.
24.
HidakaS.ManakaY.TeramotoW.SugitaY.MiyauchiR.GyobaJ.IwayaY. (2009). Alternation of sound location induces visual motion perception of a static object, PLoS One, 4: 12.
25.
HowardI. P.FujiiY.AllisonR. S. (2014). Interactions between cues to visual motion in depth, Journal of Vision, 14: 1–16.
26.
HubbardT. (2005). Representational momentum and related displacements in spatial memory: A review of the findings, Psychonomic Bulletin & Review, 12: 822–851.
27.
ItoH.OgawaM.SunagaS. (2013). Evaluation of an organic light-emitting diode display for precise visual stimulation, Journal of Vision, 13: 1–21.
28.
JainA.SallyS. L.PapathomasT. V. (2008). Audiovisual short-term influences and aftereffects in motion: Examination across three sets of directional pairings, Journal of Vision, 8: 1–13.
29.
JonesJ. A.JarickM. (2006). Multisensory integration of speech signals: The relationship between space and time, Experimental Brain Research, 174: 588–594.
30.
KadunceD. C.VaughanJ. W.WallaceM. T.BenedekG.SteinB. E. (1997). Mechanisms of within- and cross-modality suppression in the superior colliculus, Journal of Neurophysiology, 78: 2834–2847.
31.
KingS. M.DykemanC.RedgraveP.DeanP. (1992). Use of a distracting task to obtain defensive head movement to looming visual stimuli by human adults in a laboratory setting, Perception, 21: 245–259.
32.
KitagawaN.IchiharaS. (2002). Hearing visual motion in depth, Nature, 416: 172–174.
33.
KitagawaN.ZampiniM.SpenceC. (2005). Audiotactile interactions in near and far space, Experimental Brain Research, 166: 528–537.
34.
LeeJ.SpenceC. (2015). Audiovisual crossmodal cuing effects in front and rear space, Frontiers in Psychology, 6: 1086.
35.
LeoF.RomeiV.FreemanE.LadavasE.DriverJ. (2011). Looming sounds enhance orientation sensitivity for visual stimuli on the same side as such sounds, Experimental Brain Research, 213: 193–201.
36.
LewaldJ.GuskiR. (2003). Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli, Cognitive Brain Research, 16: 468–478.
37.
LiQ.YangH.SunF.WuJ. (2015). Spatiotemporal relationships among audiovisual stimuli modulate auditory facilitation of visual target discrimination, Perception, 44: 232–242.
MaierJ. X.ChandrasekaranC.GhazanfarA. A. (2008). Integration of bimodal looming signals through neuronal coherence in the temporal lobe, Current Biology, 18: 963–968.
40.
MatuschekH.KlieglR.VasishthS.BaayenH.BatesD. (2017). Balancing Type I error and power in linear mixed models, Journal of Memory and Language, 94: 305–315.
41.
Náñez, J., Sr., & Yonas, A. (1994). Effects of luminance and texture motion on infant defensive reactions to optical collision. Infant Behavior and Development, 17, 165–174.
42.
NeuhoffJ. G. (1998). Perceptual bias for rising tones, Nature, 395: 123–124.
43.
NeuhoffJ. G. (2001). An adaptive bias in the perception of looming auditory motion, Ecological Psychology, 13: 87–110.
44.
NoelJ. P.GrivazP.MarmaroliP.LissekH.BlankeO.SerinoA. (2015). Full body action remapping of peripersonal space: The case of walking, Neuropsychologia, 70: 375–384.
45.
OccelliV.SpenceC.ZampiniM. (2011). Audiotactile interactions in front and rear space, Neuroscience & Biobehavioral Reviews, 35: 589–598.
46.
PeirceJ. W. (2007). PsychoPy–psychophysics software in Python, Journal of Neuroscience Methods, 162: 8–13.
47.
PeirceJ. W. (2009). Generating stimuli for neuroscience using Psychopy, Frontiers in Neuroinformatics, 2: 10.
48.
QianJ.PetrovY. (2016). A depth illusion supports the model of general object constancy: Size and depth constancies related by a same distance-scaling factor, Vision Research, 129: 77–86.
49.
ReganD.BeverleyK. I. (1978). Illusory motion in depth: After effect of the adaptation to changing size, Vision Research, 18: 209–212.
50.
RomeiV.De HaasB.MokR. M.DriverJ. (2011). Auditory stimulus timing influences perceived duration of co-occurring visual stimuli, Frontiers in Psychology, 2: 215.
51.
SanabriaD.Lupiá̃ezJ.SpenceC. (2007). Auditory motion affects visual motion perception in a speeded discrimination task, Experimental Brain Research, 178: 415–421.
52.
SchiffW.CavinessJ. A.GibsonJ. J. (1962). Persistent fear responses in rhesus monkeys to the optical stimulus of “looming”, Science, 136: 982–983.
53.
SchlauchR. S.RiesD. T.DiGiovanniJ. J. (2001). Duration discrimination and subjective duration for ramped and damped sounds, The Journal of the Acoustic Society of America, 109: 2880–2887.
ShiraiN.YamaguchiM. K. (2004). Asymmetry in the perception of motion-in-depth, Vision Research, 44: 1003–1011.
57.
Soto-FaracoS.KingstoneA.SpenceC. (2003). Multisensory contributions to the perception of motion, Neuropsychologia, 41: 1847–1862.
58.
Soto-FaracoS.LyonsJ.GazzanigaM.SpenceC.KingstoneA. (2002). The ventriloquist in motion: Illusory capture of dynamic information across sensory modalities, Cognitive Brain Research, 14: 139–146.
59.
Soto-FaracoS.Morein-ZamirS.KingstoneA. (2005). On audiovisual spatial synergy: The fragility of the phenomenon, Perception & Psychophysics, 67: 444–457.
60.
Soto-FaracoS.SpenceC.LloydD.KingstoneA. (2004). Moving multisensory research along motion perception across sensory modalities, Current Directions in Psychological Science, 13: 29–32.
61.
SpenceC. (2013). Just how important is spatial coincidence to multi-sensory integration? Evaluating the spatial rule, Annals of the New York Academy of Sciences, 1296: 31–49.
62.
SpenceC.LeeJ.Van der StoepN. (2017). Responding to sounds from unseen locations: Crossmodal attention orienting in response to sounds presented from the rear, European Journal of Neuroscience. 1–14. doi:10.1111/ejn.13733.
63.
StefanucciJ. K.ProffittD. R. (2009). The roles of altitude and fear in the perception of height, Journal of Experimental Psychology: Human Perception and Performance, 35: 424–438.
64.
SutherlandC. A.ThutG.RomeiV. (2014). Hearing brighter: Changing in-depth visual perception through looming sounds, Cognition, 132: 312–323.
65.
TakeshimaY.GyobaJ. (2013). High-Intensity sound increases the size of visually perceived objects, Attention, Perception, & Psychophysics, 75: 501–507.
66.
TanakaS.FujitaI. (2015). Computation of object size in visual cortical area V4 as a neural basis for size constancy, The Journal of Neuroscience, 35: 12033–12046.
67.
TyllS.BonathB.SchoenfeldM. A.HeinzeH.-J.OhlF. W.NoesseltT. (2013). Neural basis of multisensory looming signals, NeuroImage, 65: 13–22.
68.
VagnoniE.LourencoS. F.LongoM. R. (2012). Threat modulates perception of looming visual stimuli, Current Biology, 22: 826–827.
69.
Van der StoepN.NijboerT. C. W.Van der StigchelS.SpenceC. (2015). Multisensory interactions in the depth plane in front and rear space: A review, Neuropsychologia, 70: 335–349.
70.
Van der StoepN.NijboerT.Van der StigchelS. (2016). Audiovisual integration and spatial alignment in azimuth and depth, Journal of Vision, 16: 865.
71.
Van der StoepN.SerinoA.FarnèA.Di LucaM.SpenceC. (2016). Depth: The forgotten dimension in multisensory research, Multisensory Research, 29: 493–524.
72.
Van UlzenN. R.SeminG. R.OudejansR. R.BeekP. J. (2008). Affective stimulus properties influence size perception and the Ebbinghaus illusion, Psychological Research, 72: 304–310.
73.
VaseyM. W.VilenskyM. R.HeathJ. H.HarbaughC. N.BuffingtonA. G.FazioR. H. (2012). It was as big as my head, I swear! Biased spider size estimation in spider phobia, Journal of Anxiety Disorders, 26: 20–24.
74.
VatakisA.SpenceC. (2007). Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli, Perception & Psychophysics, 69: 744–756.
75.
WallaceM. T.RobersonG. E.HairstonW. D.SteinB. E.VaughanJ. W.SchirilloJ. A. (2004). Unifying multisensory signals across time and space, Experimental Brain Research, 158: 252–258.
76.
WannJ. P.PoulterD. R.PurcellC. (2011). Reduced sensitivity to visual looming inflates the risk posed by speeding vehicles when children try to cross the road, Psychological Science, 22: 429–434.
77.
WelchR. B.WarrenD. H. (1980). Immediate perceptual response to intersensory discrepancy, Psychological Bulletin, 88: 638–667.
78.
WhitakerD.McGrawP. V.PearsonS. (1999). Non-veridical size perception of expanding and contracting objects, Vision Research, 39: 2999–3009.
79.
ZahorikP. (2002). Assessing auditory distance perception using virtual acoustics, Journal of the Acoustic Society of America, 111: 1832–1846.
ZampiniM.TorresanD.SpenceC.MurrayM. M. (2007). Auditory–somatosensory multisensory interactions in front and rear space, Neuropsychologia, 45: 1869–1877.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
