Semantically Congruent Bimodal Presentation with Divided-Modality Attention Accelerates Unisensory Working Memory Retrieval

Abstract

Although previous studies have shown that semantic multisensory integration can be differentially modulated by attention focus, it remains unclear whether attentionally mediated multisensory perceptual facilitation could impact further cognitive performance. Using a delayed matching-to-sample paradigm, the present study investigated the effect of semantically congruent bimodal presentation on subsequent unisensory working memory (WM) performance by manipulating attention focus. The results showed that unisensory WM retrieval was faster in the semantically congruent condition than in the incongruent multisensory encoding condition. However, such a result was only found in the divided-modality attention condition. This result indicates that a robust multisensory representation was constructed during semantically congruent multisensory encoding with divided-modality attention; this representation then accelerated unisensory WM performance, especially auditory WM retrieval. Additionally, an overall faster unisensory WM retrieval was observed under the modality-specific selective attention condition compared with the divided-modality condition, indicating that the division of attention to address two modalities demanded more central executive resources to encode and integrate crossmodal information and to maintain a constructed multisensory representation, leaving few resources for WM retrieval. Additionally, the present finding may support the amodal view that WM has an amodal central storage component that is used to maintain modal-based attention-optimized multisensory representations.

Keywords

audiovisual integration semantic congruency working memory divided-modality attention modality-specific selective attention

Get full access to this article

View all access options for this article.

References

Allen

R. J.

Hitch

G. J.

Baddeley

A. D.

(2009). Cross-modal binding and working memory. Visual Cognition, 17(1-2), 83–102. https://doi.org/10.1080/13506280802281386

Baddeley

(1992). Working memory. Science (New York, N Y ), 255(5044), 556–559. https://doi.org/10.1126/science.1736359

Baddeley

(1996). Exploring the central executive. The Quarterly Journal of Experimental Psychology Section A, 49(1), 5–28. https://doi.org/10.1080/713755608

Bizley

J. K.

Cohen

Y. E.

(2013). The what, where and how of auditory-object perception. Nature Reviews Neuroscience, 14(10), 693–707. https://doi.org/10.1038/nrn3565

Cohen

M. A.

Evans

K. K.

Horowitz

T. S.

Wolfe

J. M.

(2011). Auditory and visual memory in musicians and nonmusicians. Psychonomic Bulletin & Review, 18(3), 586–591. https://doi.org/10.3758/s13423-011-0074-0

Cohen

M. A.

Horowitz

T. S.

Wolfe

J. M.

(2009). Auditory recognition memory is inferior to visual recognition memory. Proceedings of the National Academy of Sciences of the United States of America, 106(14), 6008–6010. https://doi.org/10.1073/pnas.0811884106

Courtney

S. M.

Ungerleider

L. G.

Keil

Haxby

J. V.

(1997). Transient and sustained activity in a distributed neural system for human working memory. Nature, 386(6625), 608–611. https://doi.org/10.1038/386608a0

Della Sala

Baddeley

Papagno

Spinnler

(1995). Dual-task paradigm: A means to examine the central executive. Annals of the New York Academy of Sciences, 769(1), 161–171. https://doi.org/10.1111/j.1749-6632.1995.tb38137.x

Delogu

Raffone

Belardinelli

M. O.

(2009). Semantic encoding in working memory: Is there a (multi)modality effect? Memory (Hove, England), 17(6), 655–663. https://doi.org/10.1080/09658210902998054

10.

D'Esposito

Detre

J. A.

Alsop

D. C.

Shin

R. K.

Atlas

Grossman

(1995). The neural basis of the central executive system of working memory. Nature, 378(6554), 279–281. https://doi.org/10.1038/378279a0

11.

Friston

(2010). The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11(2), 127–138. https://doi.org/10.1038/nrn2787

12.

Frtusova

J. B.

Phillips

N. A.

(2016). The auditory-visual speech benefit on working memory in older adults with hearing impairment. Frontiers in Psychology, 7, 490. https://doi.org/10.3389/fpsyg.2016.00490

13.

Frtusova

J. B.

Winneke

A. H.

Phillips

N. A.

(2013). ERP Evidence that auditory-visual speech facilitates working memory in younger and older adults. Psychology and Aging, 28(2), 481–494. https://doi.org/10.1037/a0031243

14.

Gao

Qiu

Yang

Shen

(2017). Bindings in working memory: The role of object-based attention. Attention, Perception & Psychophysics, 79(2), 533–552. https://doi.org/10.3758/s13414-016-1227-z

15.

Gazzaley

Nobre

A. C.

(2012). Top-down modulation: Bridging selective attention and working memory. Trends in Cognitive Sciences, 16(2), 129–135. https://doi.org/10.1016/j.tics.2011.11.014

16.

Heikkilä

Alho

Tiippana

(2017). Semantically congruent visual stimuli can improve auditory memory. Multisensory Research, 30(7-8), 639–651. https://doi.org/10.1163/22134808-00002584

17.

Heikkila

Alho

Hyvonen

Tiippana

(2015). Audiovisual semantic congruency during encoding enhances memory performance. Experimental Psychology, 62(2), 123–130. https://doi.org/10.1027/1618-3169/a000279

18.

Loose

Kaufmann

Auer

D. P.

Lange

K. W.

(2003). Human prefrontal and sensory cortical activity during divided attention tasks. Human Brain Mapping, 18(4), 249–259. https://doi.org/10.1002/hbm.10082

19.

Mastroberardino

Santangelo

Botta

Marucci

F. S.

Belardinelli

M. O.

(2008). How the bimodal format of presentation affects working memory: An overview. Cognitive Processing, 9(1), 69–76. https://doi.org/10.1007/s10339-007-0195-6

20.

Matusz

P. J.

Wallace

M. T.

Murray

M. M.

(2017). A multisensory perspective on object memory. Neuropsychologia, 105, 243–252. https://doi.org/10.1016/j.neuropsychologia.2017.04.008

21.

Meredith

M. A.

Stein

B. E.

(1983). Interactions among converging sensory inputs in the superior colliculus. Science (New York, N Y ), 221(4608), 389–391. https://doi.org/10.1126/science.6867718

22.

Mishra

Gazzaley

(2012). Attention distributed across sensory modalities enhances perceptual performance. Journal of Neuroscience, 32(35), 12294–12302. https://doi.org/10.1523/JNEUROSCI.0867-12.2012

23.

Molholm

Ritter

Javitt

D. C.

Foxe

J. J.

(2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex (New York, N Y , 14(4), 452–465. https://doi.org/10.1093/cercor/bhh007

24.

Moran

Z. D.

Bachman

Pham

Cho

S. H.

Cannon

T. D.

Shams

(2013). Multisensory encoding improves auditory recognition. Multisensory Research, 26(6), 581–592. https://doi.org/10.1163/22134808-00002436

25.

Mozolic

J. L.

Hugenschmidt

C. E.

Peiffer

A. M.

Laurienti

P. J.

(2008). Modality-specific selective attention attenuates multisensory integration. Experimental Brain Research, 184(1), 39–52. https://doi.org/10.1007/s00221-007-1080-3

26.

Nikolin

Lauf

Loo

C. K.

Martin

(2018). Effects of high-definition transcranial direct current stimulation (HD-tDCS) of the intraparietal sulcus and dorsolateral prefrontal cortex on working memory and divided attention. Frontiers in Integrative Neuroscience, 12, 64. https://doi.org/10.3389/fnint.2018.00064

27.

Odegaard

Wozny

D. R.

Shams

(2016). The effects of selective and divided attention on sensory precision and integration. Neuroscience Letters, 614, 24–28. https://doi.org/10.1016/j.neulet.2015.12.039

28.

Rohe

Noppeney

(2018). Reliability-Weighted integration of audiovisual signals can be modulated by top-down attention. eNeuro, 5(1). https://doi.org/10.1523/ENEURO.0315-17.2018

29.

Santangelo

Macaluso

(2013). The contribution of working memory to divided attention. Human Brain Mapping, 34(1), 158–175. https://doi.org/10.1002/hbm.21430

30.

Schmid

Buchel

Rose

(2011). The neural basis of visual dominance in the context of audio-visual object processing. Neuroimage, 55(1), 304–311. https://doi.org/10.1016/j.neuroimage.2010.11.051

31.

Schneider

B. A.

Pichora-Fuller

M. K.

(2000). Implications of perceptual deterioration for cognitive aging research. https://doi.org/10.4324/9780203837665.

32.

Simon

S. S.

Tusch

E. S.

Holcomb

P. J.

Daffner

K. R.

(2016). Increasing working memory load reduces processing of cross-modal task-irrelevant stimuli even after controlling for task difficulty and executive capacity. Frontiers in Human Neuroscience, 10, 380. https://doi.org/10.3389/fnhum.2016.00380

33.

Snodgrass

J. G.

Vanderwart

(1980). A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 6(2), 174–215. https://doi.org/10.1037//0278-7393.6.2.174

34.

Stein

B. E.

Stanford

T. R.

Ramachandran

Perrault

T. J.

Jr Rowland

B. A.

(2009). Challenges in quantifying multisensory integration: Alternative criteria, models, and inverse effectiveness. Experimental Brain Research, 198(2-3), 113–126. https://doi.org/10.1007/s00221-009-1880-8

35.

Suied

Bonneel

Viaud-Delmon

(2009). Integration of auditory and visual information in the recognition of realistic objects. Experimental Brain Research, 194(1), 91–102. https://doi.org/10.1007/s00221-008-1672-6

36.

Talsma

(2015). Predictive coding and multisensory integration: An attentional account of the multisensory mind. Frontiers in Integrative Neuroscience, 9, 19. https://doi.org/10.3389/fnint.2015.00019

37.

Talsma

Doty

T. J.

Woldorff

M. G.

(2007). Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration? Cerebral Cortex (New York, N Y , 17(3), 679–690. https://doi.org/10.1093/cercor/bhk016

38.

Tatz

J. R.

Undorf

Peynircioglu

Z. F.

(2020). Effect of impoverished information on multisensory integration in judgments of learning. Journal of Experimental Psychology Learning Memory and Cognition, 47(3), 481–497. https://doi.org/10.1037/xlm0000953

39.

Thelen

Talsma

Murray

M. M.

(2015). Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition, 138, 148–160. https://doi.org/10.1016/j.cognition.2015.02.003

40.

Thompson

V. A.

Paivio

(1994). Memory for pictures and sounds: Independence of auditory and visual codes. Canadian Journal of Experimental Psychology = Revue Canadienne De Psychologie Experimentale, 48(3), 380–398. https://doi.org/10.1037/1196-1961.48.3.380

41.

Tulving

(1981). Similarity relations in recognition. Journal of Verbal Learning and Verbal Behavior, 20(5), 479–496. https://doi.org/10.1016/S0022-5371(81)90129-8

42.

Uncapher

M. R.

Rugg

M. D.

(2005). Effects of divided attention on fMRI correlates of memory encoding. Journal of Cognitive Neuroscience, 17(12), 1923–1935. https://doi.org/10.1162/089892905775008616

43.

Uncapher

M. R.

Hutchinson

J. B.

Wagner

A. D.

(2011). Dissociable effects of top-down and bottom-up attention during episodic encoding. Journal of Neuroscience, 31(35), 12613–12628. https://doi.org/10.1523/JNEUROSCI.0152-11.2011

44.

van de Rijt

L. P. H.

Roye

Mylanus

E. A. M.

van Opstal

A. J.

van Wanrooij

M. M.

(2019). The principle of inverse effectiveness in audiovisual speech perception. Frontiers in Human Neuroscience, 13, 335. https://doi.org/10.3389/fnhum.2019.00335

45.

Gao

Tang

(2019). Cortical network underlying audiovisual semantic integration and modulation of attention: An fMRI and graph-based study. PLoS One, 14(8), e0221185. https://doi.org/10.1371/journal.pone.0221185

46.

Xie

Y. J.

Y. Y.

Xie

Y. Y.

Peng

(2019). The neural basis of complex audiovisual objects maintenances in working memory. Neuropsychologia, 133, 107189. https://doi.org/10.1016/j.neuropsychologia.2019.107189

47.

Xie

Bian

(2017). Semantic congruent audiovisual integration during the encoding stage of working memory: An ERP and sLORETA study. Scientific Reports, 7(1), 5112. https://doi.org/10.1038/s41598-017-05471-1

48.

Yang

Ren

(2020). Selective and divided attention modulates audiovisual integration in adolescents. Cognitive Development, 55, 100922. https://doi.org/10.1016/j.cogdev.2020.100922

49.

Yang

Ren

Yang

D. O.

Yuan

(2016). The influence of selective and divided attention on audiovisual integration in children. Perception, 45(5), 515–526. https://doi.org/10.1177/0301006616629025

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.32 MB

0.50 MB