Spatial Decoding for Gaze Independent Brain–Computer Interface Based on Covert Visual Attention Shift Using Electroencephalography

Abstract

The gaze-independent brain–computer interface (BCI) device is used to re-establish interaction for individuals who have abnormal eye movement. It may be possible to control the BCI by shifting your attention spatially. However, spatial attention is rarely employed to increase the effectiveness of target detection and is typically used to provide a simple “yes” or “no” response to the target recognition inquiry. To improve the effectiveness of detecting target, it is crucial to take advantage of the possible advantages of spatial attention. N2-posterior-contralateral (N2pc) component reflects correlates of visual spatial attention and is used to determine target position. In this study, a long-short-term memory (LSTM) network is used to answer “yes/no” questions by decoding covert spatial attention based on N2pc characteristics using EEG signals. The proposed LSTM-based model's average decoding accuracy is 92.79%. The target detection efficiency was successfully increased by about 4% when compared to conventional machine learning algorithms. The proposed model is tested on the independent dataset to validate its performance. The results of this work show that N2pc characteristics can be employed in gaze-independent BCIs for tracking covert attention shifts, which may help persons with poor eye mobility to connect with their environment.

Keywords

covert attention LSTM EEG N2pc

Get full access to this article

View all access options for this article.

References

James

. The Principle of Psychology. New York: Dover Publications; 1890.

Moran

Desimone

. Selective attention gates visual processing in the extrastriate cortex. Science. 1985;229)(1475):782-784.

Carrasco

. Visual attention: the past 25 years. Vision Research. 2011;51(13):1484-1525. doi: https://doi.org/10.1016/j.visres.2011.04.012

Posner

. Orienting of attention. Q J Exper Psychol. 1980;32(1):3-25.

Dosher

. Noise exclusion in spatial attention. Psychol Sci 2000;11(2):139-146.

Green

McDonald

. Electrical neuroimaging reveals timing of attentional control activity in human brain. PLoS Biol. 2008;6(4):e81.

Thut

Nietzel

Brandt

Pascual-Leone

. α-Band electroencephalographic activity over occipital cortex indexes visuospatial attention bias and predicts visual target detection. J Neurosci. 2006;26(37):9494-9502.

Wyart

Tallon-Baudry

. Neural dissociation between visual awareness and spatial attention. J Neurosci 2008;28(10):2667-2679.

Luck

Hillyard

. Spatial filtering during visual search: evidence from human electrophysiology. J Exper Psychol Hum Percept Perform. 1994;20(5):1000-1014.

10.

Mazza

Turatto

Caramazza

. Attention selection, distractor suppression and n2pc. Cortex. 2008;45(7):879-890.

11.

Woodman

Luck

. Electrophysiological measurement of rapid shifts of attention during visual search. Nature. 1999;400(6747):867–869.

12.

Awni

Norton

JJS

Umunna

Federmeier

Bretl

. “Towards a brain computer interface based on the N2pc event-related potential.” In: Proceedings of the International IEEE/EMBS Conference on Neural Engineering, NER, San Diego, CA, 2013: 1021–1024. doi: 10.1109/NER.2013.6696110.

13.

Wang

Nakanishi

, et al. Fast detection of covert visuospatial attention using hybrid N2pc and SSVEP features. J Neural Eng. 2016;13:(6):066003. doi: https://doi.org/10.1088/1741-2560/13/6/066003

14.

Matran-Fernandez

Poli

. Brain-computer interfaces for detection and localization of targets in aerial images. IEEE Trans Biomed Eng 2017;64(4):959-969. doi: https://doi.org/10.1109/TBME.2016.2583200

15.

Reichert

Dürschmid

Bartsch

Hopf

Heinze

Hinrichs

. Decoding the covert shift of spatial attention from electroencephalographic signals permits reliable control of a brain-computer interface. J Neural Eng 2020;17(5):056012.

16.

Reichert

Ceja

Sweeney-Reed

Heinze

Dürschmid

. Impact of stimulus features on the performance of a gaze-independent brain-computer interface based on covert spatial attention shifts. Front Neurosci. 2020:14:591777.

17.

Thiery

Lajnef

Jerbi

Arguin

Aubin

Jolicoeur

. Decoding the locus of covert visuospatial attention from EEG signals. PLoS one. 2016;11(8):e0160304.

18.

Fahrenfort

Grubert

Olivers

CNL

Eimer

. Multivariate EEG analyses support high-resolution tracking of feature-based attentional selection. Sci Rep 2017;7(1):1886. doi: https://doi.org/10.1038/s41598-017-01911-0

19.

Alhagry

Aly Fahmy

El-Khoribi

. Emotion recognition based on EEG using LSTM recurrent neural network. Int J Adv Comput Sci Appl. 2017;8(10)

20.

Tang

Zheng

W-L

B-L

. Emotion recognition using multimodal residual LSTM network. In: Proceedings of the 27th ACM international conference on multimedia, 2019: pp. 176-183.

21.

Tal

Friedman

. “Recurrent neural networks for P300-based BCI.” arXiv preprint, 2019:arXiv:1901.10798.

22.

Wang

Jiang

Liu

Shang

Zhang

. LSTM-based EEG classification in motor imagery tasks. IEEE Trans Neural Syst Rehabil Eng 2018;26(11):2086-2095.

23.

Lotte

Congedo

Lécuyer

Lamarche

Arnaldi

. A review of classification algorithms for EEG-based brain–computer interfaces. J Neural Eng 2017;4(2):R1.

24.

Oikonomou

Vangelis P.

Georgiadis

Kostas

Liaros

George

Nikolopoulos

Spiros

Kompatsiaris

Ioannis

. “A comparison study on EEG signal processing techniques using motor imagery EEG data.” In: 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS), 2017: pp. 781–786. IEEE.

25.

Delorme

Makeig

. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 2004;134(1):9-21.

26.

Hochreiter

Schmidhuber

. Long short-term memory. Neural Comput 1997;9(8):1735-1780.

27.

Hasib

M. M.

Nayak

Huang

, “A hierarchical LSTM model with attention for modeling EEG non-stationarity for human decision prediction.” In: Proceedings of the 2018 IEEE EMBS International Conference on Biomedical & Health Informatics(BHI), Las Vegas, NV, USA, 2018.

28.

Lipton

Kale

Elkan

Wetzel

. “Learning to diagnose with LSTM recurrent neural networks.” arXiv preprint 2015;arXiv:1511.03677.

29.

Understanding LSTM Networks. https://colah.github.io/posts/2015-08-Understanding-LSTMs/ (2015) Accessed 17 Feb 2020.

30.

Chung

Ofori

Misra

Hess

Vaillancourt

. Beta-band activity and connectivity in sensorimotor and parietal cortex are important for accurate motor performance. Neuroimage. 2017;144:164-173.

31.

Kumar

Bhuvaneswari

. Analysis of electroencephalography (EEG) signals and its categorization–a study. Procedia Eng 2012;38(25):25-36.

32.

Halder

Rea

Andreoni

, et al. An auditory oddball brain-computer interface for binary choices. Clin Neurophysiol. 2010;121(4):516-523.

33.

Müller-Putz

Pokorny

Klobassa

Horki

. A single switch BCI based on passive and imagined movements: toward restoring communication in minimally conscious patients. Int J Neural Syst 2013;23(2):1250037. doi: https://doi.org/10.1142/S0129065712500372

34.

Brouwer

Van Erp

. A tactile P300 brain-computer interface. Front Neurosci 2010;4:1440. Hickey C, Di Lollo V, McDonald JJ. Electrophysiological indices of target and distractor processing in visual search. J Cogn Neurosci. 2009: 21(4):760–775.

35.

Treder

Bahramisharif

Schmidt

, et al. Brain-computer interfacing using modulations of alpha activity induced by covert shifts of attention. J Neuro Engineering Rehabil. 2011;8(1):1-10.

36.

Farahat

Reichert

Sweeney-Reed

Hinrichs

. Convolutional neural networks for decoding of covert attention focus and saliency maps for EEG feature visualization. J Neural Eng 2019;16(6):066010.

37.

Brisson

Jolicœur

. A psychological refractory period in access to visual short-term memory and the deployment of visual–spatial attention: multitasking processing deficits revealed by event-related potentials. Psychophysiology. 2007;44(2):323-333.