Although previous studies have shown that semantic multisensory integration can be differentially modulated by attention focus, it remains unclear whether attentionally mediated multisensory perceptual facilitation could impact further cognitive performance. Using a delayed matching-to-sample paradigm, the present study investigated the effect of semantically congruent bimodal presentation on subsequent unisensory working memory (WM) performance by manipulating attention focus. The results showed that unisensory WM retrieval was faster in the semantically congruent condition than in the incongruent multisensory encoding condition. However, such a result was only found in the divided-modality attention condition. This result indicates that a robust multisensory representation was constructed during semantically congruent multisensory encoding with divided-modality attention; this representation then accelerated unisensory WM performance, especially auditory WM retrieval. Additionally, an overall faster unisensory WM retrieval was observed under the modality-specific selective attention condition compared with the divided-modality condition, indicating that the division of attention to address two modalities demanded more central executive resources to encode and integrate crossmodal information and to maintain a constructed multisensory representation, leaving few resources for WM retrieval. Additionally, the present finding may support the amodal view that WM has an amodal central storage component that is used to maintain modal-based attention-optimized multisensory representations.
BaddeleyA. (1996). Exploring the central executive. The Quarterly Journal of Experimental Psychology Section A, 49(1), 5–28. https://doi.org/10.1080/713755608
4.
BizleyJ. K.CohenY. E. (2013). The what, where and how of auditory-object perception. Nature Reviews Neuroscience, 14(10), 693–707. https://doi.org/10.1038/nrn3565
5.
CohenM. A.EvansK. K.HorowitzT. S.WolfeJ. M. (2011). Auditory and visual memory in musicians and nonmusicians. Psychonomic Bulletin & Review, 18(3), 586–591. https://doi.org/10.3758/s13423-011-0074-0
6.
CohenM. A.HorowitzT. S.WolfeJ. M. (2009). Auditory recognition memory is inferior to visual recognition memory. Proceedings of the National Academy of Sciences of the United States of America, 106(14), 6008–6010. https://doi.org/10.1073/pnas.0811884106
7.
CourtneyS. M.UngerleiderL. G.KeilK.HaxbyJ. V. (1997). Transient and sustained activity in a distributed neural system for human working memory. Nature, 386(6625), 608–611. https://doi.org/10.1038/386608a0
8.
Della SalaS.BaddeleyA.PapagnoC.SpinnlerH. (1995). Dual-task paradigm: A means to examine the central executive. Annals of the New York Academy of Sciences, 769(1), 161–171. https://doi.org/10.1111/j.1749-6632.1995.tb38137.x
9.
DeloguF.RaffoneA.BelardinelliM. O. (2009). Semantic encoding in working memory: Is there a (multi)modality effect?Memory (Hove, England), 17(6), 655–663. https://doi.org/10.1080/09658210902998054
10.
D'EspositoM.DetreJ. A.AlsopD. C.ShinR. K.AtlasS.GrossmanM. (1995). The neural basis of the central executive system of working memory. Nature, 378(6554), 279–281. https://doi.org/10.1038/378279a0
11.
FristonK. (2010). The free-energy principle: A unified brain theory?Nature Reviews Neuroscience, 11(2), 127–138. https://doi.org/10.1038/nrn2787
12.
FrtusovaJ. B.PhillipsN. A. (2016). The auditory-visual speech benefit on working memory in older adults with hearing impairment. Frontiers in Psychology, 7, 490. https://doi.org/10.3389/fpsyg.2016.00490
13.
FrtusovaJ. B.WinnekeA. H.PhillipsN. A. (2013). ERP Evidence that auditory-visual speech facilitates working memory in younger and older adults. Psychology and Aging, 28(2), 481–494. https://doi.org/10.1037/a0031243
14.
GaoZ.WuF.QiuF.HeK.YangY.ShenM. (2017). Bindings in working memory: The role of object-based attention. Attention, Perception & Psychophysics, 79(2), 533–552. https://doi.org/10.3758/s13414-016-1227-z
15.
GazzaleyA.NobreA. C. (2012). Top-down modulation: Bridging selective attention and working memory. Trends in Cognitive Sciences, 16(2), 129–135. https://doi.org/10.1016/j.tics.2011.11.014
LooseR.KaufmannC.AuerD. P.LangeK. W. (2003). Human prefrontal and sensory cortical activity during divided attention tasks. Human Brain Mapping, 18(4), 249–259. https://doi.org/10.1002/hbm.10082
19.
MastroberardinoS.SantangeloV.BottaF.MarucciF. S.BelardinelliM. O. (2008). How the bimodal format of presentation affects working memory: An overview. Cognitive Processing, 9(1), 69–76. https://doi.org/10.1007/s10339-007-0195-6
MeredithM. A.SteinB. E. (1983). Interactions among converging sensory inputs in the superior colliculus. Science (New York, N Y ), 221(4608), 389–391. https://doi.org/10.1126/science.6867718
22.
MishraJ.GazzaleyA. (2012). Attention distributed across sensory modalities enhances perceptual performance. Journal of Neuroscience, 32(35), 12294–12302. https://doi.org/10.1523/JNEUROSCI.0867-12.2012
23.
MolholmS.RitterW.JavittD. C.FoxeJ. J. (2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex (New York, N Y , 14(4), 452–465. https://doi.org/10.1093/cercor/bhh007
NikolinS.LaufS.LooC. K.MartinD. (2018). Effects of high-definition transcranial direct current stimulation (HD-tDCS) of the intraparietal sulcus and dorsolateral prefrontal cortex on working memory and divided attention. Frontiers in Integrative Neuroscience, 12, 64. https://doi.org/10.3389/fnint.2018.00064
27.
OdegaardB.WoznyD. R.ShamsL. (2016). The effects of selective and divided attention on sensory precision and integration. Neuroscience Letters, 614, 24–28. https://doi.org/10.1016/j.neulet.2015.12.039
28.
RoheT.NoppeneyU. (2018). Reliability-Weighted integration of audiovisual signals can be modulated by top-down attention. eNeuro, 5(1). https://doi.org/10.1523/ENEURO.0315-17.2018
29.
SantangeloV.MacalusoE. (2013). The contribution of working memory to divided attention. Human Brain Mapping, 34(1), 158–175. https://doi.org/10.1002/hbm.21430
30.
SchmidC.BuchelC.RoseM. (2011). The neural basis of visual dominance in the context of audio-visual object processing. Neuroimage, 55(1), 304–311. https://doi.org/10.1016/j.neuroimage.2010.11.051
31.
SchneiderB. A.Pichora-FullerM. K. (2000). Implications of perceptual deterioration for cognitive aging research. https://doi.org/10.4324/9780203837665.
32.
SimonS. S.TuschE. S.HolcombP. J.DaffnerK. R. (2016). Increasing working memory load reduces processing of cross-modal task-irrelevant stimuli even after controlling for task difficulty and executive capacity. Frontiers in Human Neuroscience, 10, 380. https://doi.org/10.3389/fnhum.2016.00380
33.
SnodgrassJ. G.VanderwartM. (1980). A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 6(2), 174–215. https://doi.org/10.1037//0278-7393.6.2.174
34.
SteinB. E.StanfordT. R.RamachandranR.PerraultT. J.JrRowlandB. A. (2009). Challenges in quantifying multisensory integration: Alternative criteria, models, and inverse effectiveness. Experimental Brain Research, 198(2-3), 113–126. https://doi.org/10.1007/s00221-009-1880-8
35.
SuiedC.BonneelN.Viaud-DelmonI. (2009). Integration of auditory and visual information in the recognition of realistic objects. Experimental Brain Research, 194(1), 91–102. https://doi.org/10.1007/s00221-008-1672-6
36.
TalsmaD. (2015). Predictive coding and multisensory integration: An attentional account of the multisensory mind. Frontiers in Integrative Neuroscience, 9, 19. https://doi.org/10.3389/fnint.2015.00019
37.
TalsmaD.DotyT. J.WoldorffM. G. (2007). Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration?Cerebral Cortex (New York, N Y , 17(3), 679–690. https://doi.org/10.1093/cercor/bhk016
38.
TatzJ. R.UndorfM.PeynirciogluZ. F. (2020). Effect of impoverished information on multisensory integration in judgments of learning. Journal of Experimental Psychology Learning Memory and Cognition, 47(3), 481–497. https://doi.org/10.1037/xlm0000953
39.
ThelenA.TalsmaD.MurrayM. M. (2015). Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition, 138, 148–160. https://doi.org/10.1016/j.cognition.2015.02.003
40.
ThompsonV. A.PaivioA. (1994). Memory for pictures and sounds: Independence of auditory and visual codes. Canadian Journal of Experimental Psychology = Revue Canadienne De Psychologie Experimentale, 48(3), 380–398. https://doi.org/10.1037/1196-1961.48.3.380
UncapherM. R.RuggM. D. (2005). Effects of divided attention on fMRI correlates of memory encoding. Journal of Cognitive Neuroscience, 17(12), 1923–1935. https://doi.org/10.1162/089892905775008616
43.
UncapherM. R.HutchinsonJ. B.WagnerA. D. (2011). Dissociable effects of top-down and bottom-up attention during episodic encoding. Journal of Neuroscience, 31(35), 12613–12628. https://doi.org/10.1523/JNEUROSCI.0152-11.2011
44.
van de RijtL. P. H.RoyeA.MylanusE. A. M.van OpstalA. J.van WanrooijM. M. (2019). The principle of inverse effectiveness in audiovisual speech perception. Frontiers in Human Neuroscience, 13, 335. https://doi.org/10.3389/fnhum.2019.00335
45.
XiY.LiQ.GaoN.HeS.TangX. (2019). Cortical network underlying audiovisual semantic integration and modulation of attention: An fMRI and graph-based study. PLoS One, 14(8), e0221185. https://doi.org/10.1371/journal.pone.0221185
XieY.XuY.BianC.LiM. (2017). Semantic congruent audiovisual integration during the encoding stage of working memory: An ERP and sLORETA study. Scientific Reports, 7(1), 5112. https://doi.org/10.1038/s41598-017-05471-1
48.
YangW.LiS.XuJ.LiZ.YangX.RenY. (2020). Selective and divided attention modulates audiovisual integration in adolescents. Cognitive Development, 55, 100922. https://doi.org/10.1016/j.cogdev.2020.100922
49.
YangW.RenY.YangD. O.YuanX.WuJ. (2016). The influence of selective and divided attention on audiovisual integration in children. Perception, 45(5), 515–526. https://doi.org/10.1177/0301006616629025
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
