Spatial modulation of audiovisual integration in the sound-induced flash illusion: A hierarchical drift diffusion modeling approach

Abstract

The sound-induced flash illusion (SIFI) is a classic phenomenon in audiovisual integration research, reflecting the interactive mechanisms in multimodal information processing. This study aimed to recruit 30 healthy participants to explore the effects of sound and flash presentation frequencies and spatial location of stimuli on the perceptual decision-making. Previous studies have used illusion rate or accuracy as the core measurement indices, which fail to reflect the underlying cognitive processes of perceptual decision-making and thus have certain limitations. Therefore, this study aims to model the process of behavioral responses in the task. Among the perceptual decision-making models—Only the Hierarchical Drift Diffusion Model (HDDM) is suitable for the experimental task in this study. We used the drift rate, which reflects the speed of perceptual information accumulation in the model, as the core indicator. The results showed that both the sound-flash combinations and spatial location information significantly influenced evidence accumulation. Furthermore, the overall drift rate for right-side spatial presentation conditions was higher than for left-side conditions. The left-side advantage in B1F1 may reflect the right hemisphere's superiority in spatial attention, while the right-side advantage in B2F1 may reflect hemispheric differences in audiovisual integration. This study is the first to introduce HDDM in analyzing the spatial modulation mechanism within the SIFI paradigm, confirming that spatial location information not only has an independent main effect but can also interact with sensory combination factors to dynamically influence perceptual decision-making.

Keywords

sound-induced flash illusion hierarchical drift diffusion model multisensory integration decision-making perception

Get full access to this article

View all access options for this article.

References

Apthorp

Alais

Boenke

L. T.

(2013). Flash illusions induced by visual, auditory, and audiovisual stimuli. Journal of Vision, 13(5), 3–3. https://doi.org/10.1167/13.5.3

Bidelman

G. M.

Heath

S. T.

(2019). Enhanced temporal binding of audiovisual information in the bilingual brain. Bilingualism: Language and Cognition, 22(4), 752–762. https://doi.org/10.1017/S1366728918000408

Bizley

J. K.

Shinn-Cunningham

B. G.

Lee

A. K. C.

(2012). Nothing is irrelevant in a noisy world: Sensory illusions reveal obligatory within-and across-modality integration. Journal of Neuroscience, 32(39), 13402–13410. https://doi.org/10.1523/JNEUROSCI.2495-12.2012

Carland

M. A.

Thura

Cisek

(2019). The urge to decide and act: Implications for brain function and dysfunction. The Neuroscientist, 25(5), 491–511. https://doi.org/10.1177/1073858419841553

Chan

J. S.

Connolly

S. K.

Setti

(2018). The number of stimulus-onset asynchronies affects the perception of the sound-induced flash illusion in young and older adults. Multisensory Research, 31(3-4), 175–190. https://doi.org/10.1163/22134808-00002605

Chandrasekaran

(2017). Computational principles and models of multisensory integration. Current Opinion in Neurobiology, 43, 25–34. https://doi.org/10.1016/j.conb.2016.11.002

Chang

Wang

Yang

Luan

Wang

Zhang

(2023). Differences in eccentricity for sound-induced flash illusion in four visual fields. Perception, 52(1), 56–73. https://doi.org/10.1177/03010066221136670

Chau

Murray

C. A.

Shams

(2021). Hierarchical drift diffusion modeling uncovers multisensory benefit in numerosity discrimination tasks. PeerJ, 9, e12273. https://doi.org/10.7717/peerj.12273

Chen

Y.-C.

Maurer

Lewis

T. L.

Spence

Shore

D. I.

(2017). Central–peripheral differences in audiovisual and visuotactile event perception. Attention, Perception, & Psychophysics, 79(8), 2552–2563. https://doi.org/10.3758/s13414-017-1396-4

10.

Cohen

(2009). Statistical power analysis for the behavioral sciences (2. ed., reprint). Psychology Press.

11.

Corbetta

Shulman

G. L.

(2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews Neuroscience, 3(3), 201–215. https://doi.org/10.1038/nrn755

12.

DeLoss

D. J.

Andersen

G. J.

(2015). Aging, spatial disparity, and the sound-induced flash illusion. PLOS One, 10(11), e0143773. https://doi.org/10.1371/journal.pone.0143773

13.

Diederich

(2008). A further test of sequential-sampling models that account for payoff effects on response bias in perceptual decision tasks. Perception and Psychophysics, 70(2), 229–256. https://doi.org/10.3758/PP.70.2.229

14.

Downing

C. J.

(1988). Expectancy and visual-spatial attention: Effects on perceptual quality. Journal of Experimental Psychology: Human Perception and Performance, 14(2), 188. https://doi.org/10.1037/0096-1523.14.2.188

15.

Drugowitsch

DeAngelis

G. C.

Angelaki

D. E.

Pouget

(2015). Tuning the speed-accuracy trade-off to maximize reward rate in multisensory decision-making. Elife, 4, e06678. https://doi.org/10.7554/eLife.06678

16.

Drugowitsch

DeAngelis

G. C.

Klier

E. M.

Angelaki

D. E.

Pouget

(2014). Optimal multisensory decision-making in a reaction-time task. Elife, 3, e03005. https://doi.org/10.7554/eLife.03005

17.

Falchier

Clavagnier

Barone

Kennedy

(2002). Anatomical evidence of multimodal integration in primate striate cortex. Journal of Neuroscience, 22(13), 5749–5759. https://doi.org/10.1523/JNEUROSCI.22-13-05749.2002

18.

Faul

Erdfelder

Lang

A.-G.

Buchner

(2007). G*power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146

19.

Fetsch

C. R.

Noppeney

(2023). How the brain controls decision making in a multisensory world. Philosophical Transactions of the Royal Society B: Biological Sciences, 378, 1886. https://doi.org/10.1098/rstb.2022.0332

20.

Forstmann

B. U.

Ratcliff

Wagenmakers

E.-J.

(2016). Sequential sampling models in cognitive neuroscience: Advantages, applications, and extensions. Annual Review of Psychology, 67(1), 641–666. https://doi.org/10.1146/annurev-psych-122414-033645

21.

Foss-Feig

J. H.

Kwakye

L. D.

Cascio

C. J.

Burnette

C. P.

Kadivar

Stone

W. L.

Wallace

M. T.

(2010). An extended multisensory temporal binding window in autism spectrum disorders. Experimental Brain Research, 203(2), 381–389. https://doi.org/10.1007/s00221-010-2240-4

22.

Gavin

Hirst

R. J.

McGovern

D. P.

(2022). The magnitude of the sound-induced flash illusion does not increase monotonically as a function of visual stimulus eccentricity. Attention, Perception, & Psychophysics, 84(5), 1689–1698. https://doi.org/10.3758/s13414-022-02493-4

23.

Gingras

Rowland

B. A.

Stein

B. E.

(2009). The differing impact of multisensory and unisensory integration on behavior. Journal of Neuroscience, 29(15), 4897–4902. https://doi.org/10.1523/JNEUROSCI.4120-08.2009

24.

Harris

C. R.

Millman

K. J.

Van Der Walt

S. J.

Gommers

Virtanen

Cournapeau

Wieser

Taylor

Berg

Smith

N. J.

Kern

Picus

Hoyer

Van Kerkwijk

M. H.

Brett

Haldane

Del Río

J. F.

Wiebe

Peterson

Oliphant

T. E.

(2020). Array programming with NumPy. Nature, 585(7825), 357–362. https://doi.org/10.1038/s41586-020-2649-2

25.

Haß

Sinke

Reese

Roy

Wiswede

Dillo

Oranje

Szycik

G. R.

(2017). Enlarged temporal integration window in schizophrenia indicated by the double-flash illusion. Cognitive Neuropsychiatry, 22(2), 145–158. https://doi.org/10.1080/13546805.2017.1287693

26.

Hirst

R. J.

McGovern

D. P.

Setti

Shams

Newell

F. N.

(2020). What you see is what you hear: Twenty years of research using the sound-induced flash illusion. Neuroscience & Biobehavioral Reviews, 118, 759–774. https://doi.org/10.1016/j.neubiorev.2020.09.006

27.

Innes-Brown

Crewther

(2009). The impact of spatial incongruence on an auditory-visual illusion. PLOS One, 4(7), e6450. https://doi.org/10.1371/journal.pone.0006450

28.

Ito

Matsumoto

Kobayasi

K. I.

(2023). Subjective audibility modulates the susceptibility to sound-induced flash illusion: Effect of loudness and auditory masking. Multisensory Research, 36, 613–629. https://doi.org/10.1163/22134808-bja10109

29.

Jeffreys

(1998). The theory of probability. OuP Oxford.

30.

Keil

(2020). Double flash illusions: Current findings and future directions. Frontiers in Neuroscience, 14, 298. https://doi.org/10.3389/fnins.2020.00298

31.

Keil

Müller

Hartmann

Weisz

(2014). Prestimulus beta power and phase synchrony influence the sound-induced flash illusion. Cerebral Cortex, 24(5), 1278–1288. https://doi.org/10.1093/cercor/bhs409

32.

Kleiner

Brainard

Pelli

(2007). What’s new in psychtoolbox-3?

33.

Kumpik

D. P.

Roberts

H. E.

King

A. J.

Bizley

J. K.

(2014). Visual sensitivity is a stronger determinant of illusory processes than auditory cue parameters in the sound-induced flash illusion. Journal of Vision, 14(7), 12–12. https://doi.org/10.1167/14.7.12

34.

Lee

Noppeney

(2014). Music expertise shapes audiovisual temporal integration windows for speech, sinewave speech, and music. Frontiers in Psychology, 5, 868. https://doi.org/10.3389/fpsyg.2014.00868

35.

Liesefeld

H. R.

Janczyk

(2019). Combining speed and accuracy to control for speed-accuracy trade-offs(?). Behavior Research Methods, 51(1), 40–60. https://doi.org/10.3758/s13428-018-1076-x

36.

Macmillan

N. A.

(2002). Signal detection theory. Stevens’ Handbook of Experimental Psychology: Methodology in Experimental Psychology, 3, 43–90. https://doi.org/10.1146/annurev.ps.39.020188.001125

37.

McKinney

(2015). Pandas, python data analysis library (pp. 3–15). URL Http://Pandas.Pydata.Org.

38.

Mercier

M. R.

(2020). The interplay between multisensory integration and perceptual decision making .

39.

Mishra

Martinez

Hillyard

S. A.

(2008). Cortical processes underlying sound-induced flash fusion. Brain Research, 1242, 102–115. https://doi.org/10.1016/j.brainres.2008.05.023

40.

Mishra

Martinez

Sejnowski

T. J.

Hillyard

S. A.

(2007). Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27(15), 4120–4131. https://doi.org/10.1523/JNEUROSCI.4912-06.2007

41.

Murray

C. A.

De Larrea-Mancera

E. S. L.

Glicksohn

Shams

Seitz

A. R.

(2020). Revealing multisensory benefit with diffusion modeling. Journal of Mathematical Psychology, 99, 102449. https://doi.org/10.1016/j.jmp.2020.102449

42.

Narinesingh

Goltz

H. C.

Wong

A. M. F.

(2017). Temporal binding window of the sound-induced flash illusion in amblyopia. Investigative Opthalmology & Visual Science, 58(3), 1442. https://doi.org/10.1167/iovs.16-21258

43.

Odegaard

Wozny

D. R.

Shams

(2015). Biases in visual, auditory, and audiovisual perception of space. PLOS Computational Biology, 11(12), e1004649. https://doi.org/10.1371/journal.pcbi.1004649

44.

Pan

Geng

Zhang

Fengler

Frank

M. J.

Zhang

R.-Y.

Chuan-Peng

(2025). dockerHDDM: A user-friendly environment for Bayesian hierarchical drift-diffusion modeling. Advances in Methods and Practices in Psychological Science, 8(1), 25152459241298700. https://doi.org/10.1177/25152459241298700

45.

Powers

A. R.

Hillock

A. R.

Wallace

M. T.

(2009). Perceptual training narrows the temporal window of multisensory binding. Journal of Neuroscience, 29(39), 12265–12274. https://doi.org/10.1523/JNEUROSCI.3501-09.2009

46.

Retsa

Naish

Bekinschtein

Bak

T. H.

(2016). Temporal judgments in multi–sensory space. Neuropsychologia, 88, 101–112. https://doi.org/10.1016/j.neuropsychologia.2016.02.011

47.

Rockland

K. S.

Ojima

(2003). Multisensory convergence in calcarine visual areas in macaque monkey. International Journal of Psychophysiology, 50(1-2), 19–26. https://doi.org/10.1016/S0167-8760(03)00121-1

48.

Seabold

Perktold

(2010). Statsmodels: Econometric and statistical modeling with python (pp. 92–96). https://doi.org/10.25080/Majora-92bf1922-011

49.

Shams

Iwaki

Chawla

Bhattacharya

(2005). Early modulation of visual cortex by sound: An MEG study. Neuroscience Letters, 378(2), 76–81. https://doi.org/10.1016/j.neulet.2004.12.035

50.

Shams

Kamitani

Shimojo

(2000). What you see is what you hear. Nature, 408(6814), 788–788. https://doi.org/10.1038/35048669

51.

Shams

Kamitani

Shimojo

(2002). Visual illusion induced by sound. Cognitive Brain Research, 14(1), 147–152. https://doi.org/10.1016/S0926-6410(02)00069-1

52.

Shulman

G. L.

Pope

D. L. W.

Astafiev

S. V.

McAvoy

M. P.

Snyder

A. Z.

Corbetta

(2010). Right hemisphere dominance during spatial selective attention and target detection occurs outside the dorsal frontoparietal network. Journal of Neuroscience, 30(10), 3640–3651. https://doi.org/10.1523/JNEUROSCI.4085-09.2010

53.

Sun

Zhou

Liu

Wang

Yue

Zhang

(2022). Sound-induced flash illusion is modulated by the depth of auditory stimuli: Evidence from younger and older adults. Attention, Perception, & Psychophysics, 84(6), 2040–2050. https://doi.org/10.3758/s13414-022-02537-9

54.

Tremblay

Champoux

Voss

Bacon

B. A.

Lepore

Théoret

(2007). Speech and non-speech audio-visual illusions: A developmental study. PLOS One, 2(8), e742. https://doi.org/10.1371/journal.pone.0000742

55.

Vallat

(2018). Pingouin: Statistics in python. Journal of Open Source Software, 3(31), 1026. https://doi.org/10.21105/joss.01026

56.

Vanes

L. D.

White

T. P.

Wigton

R. L.

Joyce

Collier

Shergill

S. S.

(2016). Reduced susceptibility to the sound-induced flash fusion illusion in schizophrenia. Psychiatry Research, 245, 58–65. https://doi.org/10.1016/j.psychres.2016.08.016

57.

Waskom

(2021). Seaborn: Statistical data visualization. Journal of Open Source Software, 6(60), 3021. https://doi.org/10.21105/joss.03021

58.

Wiecki

T. V.

Sofer

Frank

M. J.

(2013). HDDM: Hierarchical Bayesian estimation of the drift-diffusion model in python. Frontiers in Neuroinformatics, 7, 14. https://doi.org/10.3389/fninf.2013.00014

59.

Zhou

Huang

Yang

Wang

Zhang

(2023). Sound-induced flash illusions at different spatial locations were affected by personality traits. Attention, Perception, & Psychophysics, 85(2), 463–473. https://doi.org/10.3758/s13414-022-02638-5

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.32 MB

0.00 MB