Sage Journals: Discover world-class research

Abstract

A brain-computer interface (BCI) system can recognize the mental activities pattern by computer algorithms to control the external devices. Electroencephalogram (EEG) is one of the most common used approach for BCI due to the convenience and non-invasive implement. Therefore, more and more BCIs have been designed for the disabled people that suffer from stroke or spinal cord injury to help them for rehabilitation and life. We introduce the common BCI paradigms, the signal processing, and feature extraction methods. Then, we survey the different combined modes of hybrids BCIs and review the design of the synchronous/asynchronous BCIs. Finally, the shared control methods are discussed.

Keywords

BCI EEG MI SSVEP P300

1 Introduction

Electroencephalogram (EEG) is one of the most common used approach in brain-computer interface systems (BCIs). An EEG-based BCI system can rebuild the neuromuscular bypass by an external device. The brain potentials recorded by electrode placed on the scalp are transformed into commands to control the robotic arm, exoskeleton, wheelchair or other robot. There are many paradigms in EEG-based BCIs, such as motor imagery (MI) based on event-related desynchronization/synchronization (ERD/ERS) referred to as sensory-motor rhythms (SMR) [1–4], sensation imagery based on somatosensory, attentional orientation (SAO) potentials [5, 6], steady state visual evoked potentials (SSVEPs), steady-state somatosensory evoked potentials (SSSEPs) [7–10], P300 potentials [11–13], and slow cortical potentials (SCPs) [14–16]. The different EEG patterns can be recognized by the algorithms of feature extraction and classification, which can generate commands to control robot. Those paradigms also can be combined in series or parallel to constitute hybrid BCIs. Furthermore, the BCIs are divided into synchronous and asynchronous systems in the signal processing.

This review aims at the design of EEG-based BCIs for controlling intelligent systems and is organized as follows: Section 2 reviews the methods of EEG signal processing. Section 3 surveys the design of hybrid BCIs. Then, synchronous and asynchronous systems are depicted in Section 4. Finally, Section 5 describes the shared control method.

2 EEG signal processing

There are several popular EEG paradigms for the design of BCIs. The algorithms of signal processing for three common paradigms including MI, SSVEP and P300 are reviewed in this paper. MI paradigm considered as spontaneous EEG is based on the changes of rhythmic activity in ERD/ERS. The subjects usually imagine the movement of right hand or left hand according to the two-dimensional cursor in screen. Then the patterns of EEG can be recognized by signal processing algorithm. In a SSVEP BCI paradigm, the frequency spectrum of the EEG relates to the flickering stimulus frequency on which the subject focuses. SSVEP is evoked response EEG that has the advantage of no training requirement and high pattern classification accuracy. P300 evoked potentials happen about 300 ms after intending to some less probable stimulus such as infrequent auditory, visual, or somatosensory stimuli. The typical P300 paradigm usually comprises a matrix of letters, numbers, or other symbols or commands. The subject gazes at the desired symbol when the rows or columns of this matrix are flashed at random. P300 potentials can be elicited once the desired row or column flashes. In short, the MI-based EEG is one of spontaneous BCI that has the advantages of operation at free will, sensory organs affection and cursor control easiness application. However, this paradigm needs too much time training and not all the subjects have good performance. SSVEP and P300 are both evoked BCI that have many advantages such as no need training, high bit rate and comparatively higher classification accuracy. But they rely on the external stimuli and may cause tiredness. The methods of EEG signal processing are introduced as follows.

2.1 Signal preprocessing

In order to obtain high accuracy of classification, the raw EEG signals should be preprocessed before the feature extraction due to low signal-to-noise ratio.

(1) Channel selection

For the MI-based BCIs, some EEG sampling channels are closely related to the sensorimotor rhythms, such as CP6, CP4, CP2, C6, C4, C2, FC6, FC4, FC2, CPZ, CZ, FCZ, CP1, CP3, CP5, C1, C3, C5, FC1, FC3 and FC5. Removing unrelated channels can improve the spatial feature extraction. SSVEP and P300 are related to the visual cortex, so the corresponding channels can be selected, such as P7, PO7, O1, Oz, O2, PO8, and P8.

(2) Time window setting

In order to recognize the EEG pattern more precisely, proper length of signal segment should be cut out according to the mental activity tasks. Generally speaking, for MI-based EEG, the intermediate part of sampling signals in the procedure of motor imagery may be propitious to feature extraction of classification.

(3) Artifacts removal

Artifacts are undesirable signals that can reduce the performance of EEG-based BCIs, which include electrooculography (EOG), electromyography (EMG), electrocardiography (ECG) and technical artifacts like power-line noises. Linear filtering is a common method to remove artifacts since the most usage of EEG frequency bands referred to as delta (δ), theta (θ), alpha (α), beta (β), and gamma (γ) concentrates the range from 4 Hz to 30 Hz. Moreover, Fatourechi et al. [17] employed principal component analysis (PCA) to remove the EMG and EOG artifacts. Independent component analysis (ICA) as a statistical procedure is another artifacts removal method for EEG signal preprocessing. Flexer et al. [18] applied ICA to remove ocular artifacts in EEG recorded from blind subjects. Gao et al. [19] also used ICA for automatic removal of eye-movement and blink artifacts from EEG signals.

2.2 Feature extraction of MI-based EEG

The features describe the similarity and discrimination of the EEG signals to distinguish their different patterns. They can be extracted in the temporal, spatial or frequency spectrum space.

Autoregressive (AR) components spectral estimation is a common modeling method for signal processing. It can be considered as a linear time invariant filter. The filter coefficients are the signal features. Krusienski et al. [20] applied AR algorithm for MI-BCI pattern recognition. Then, Wang et al. [21] proposed a multivariate adaptive AR (MVAAR) method for the MI classification. The experimental results showed that their method was very effective for feature extraction of MI-based EEG signal processing. Wavelet transform (WT) is also a modeling method that reveals the non-stationary time variations of brain signals. Demiralp et al. [22] proposed a fast wavelet transform (FWT) algorithm to analyze the time frequency feature of event-related potentials. Farina et al. [23] employed a discrete wavelet transformation (DWT) method to classify the movement related cortical potentials. The coefficients of DWT were considered as the feature of MI pattern recognition.

Common spatial pattern (CSP) algorithm was designed as a spatial filter that could discriminate the multichannel EEG signals by highlighting differences and minimizing similarities. Ramoser et al. [24] employed the CSP to analyze two-class MI-based EEG that imagined left/right hand movement for feature extraction. Then, Grosse-Wentrup et al. [25] extended the CSP for multiclass BCIs. Their experiments showed that CSP had a better performance than other algorithms in the aspect of spatial resolution. Therefore, many extension methods improving the original CSP method were proposed by fusing other spatiospectral features, such as common spatiospectral pattern (CSSP) [26], common sparse spectral spatial pattern (CSSSP) [27], sub-band common spatial pattern (SBCSP) [28], filter bank common spatial pattern (FBCSP) [29], wavelet common spatial pattern (WCSP) [30], and separable common spatiospectral patterns (SCSSP) [31]. These improved algorithms obtained better classification accuracy than original CSP to some extent. Moreover, many other methods of MI-based BCI analysis were proposed. Zhang et al. [32] applied linear dynamical systems (LDSs) for MI signal modeling. Spatiospectral dual-feature matrix could be resulted simultaneously without much preprocess or post-process. Then, low-rank linear dynamical systems (LR-LDS) [33] were proposed by decomposing feature subspace of LDSs on finite Grassmannian space and obtained a good performance. With the rapid development of deep learning for big data training, this method is used more and more widely in EEG pattern recognition. An et al. [34] applied deep neural networks to classify left- and right-hand MI EEG patterns. Tan et al. [35–39] employed transfer learning method to solve the small sample training problem for MI-based BCI. They transform the raw signals into EEG optical flow and design a deep neural network containing a transfer network and a classification network. Experimental results demonstrated the high robustness and accuracy.

2.3 Feature extraction of SSVEP

SSVEP feature extraction mainly needs to find the special frequency of EEG response. Power spectral density analysis (PSDA) is a traditional method to detect the SSVEP frequency from a specific time window by using fast Fourier transform (FFT)-based spectrum estimation. However, PSDA is easily effected by noise. Thus, some better algorithms are proposed.

Lin et al. [40] firstly applied canonical correlation analysis (CCA) for SSVEP frequency recognition. Experimental results showed that the CCA method could find the maximal correlation coefficient that was considered as the SSVEP frequency between the reference and test signals. Comparing with PSDA, CCA obtained a better performance. Then, many improved methods [41–43] were proposed to raise the accuracy of CCA recognition.

Cecotti et al. [44] applied convolutional neural network (CNN) with embedded Fourier transform for SSVEP classification. The hidden layers switched from the time domain to the frequency domain by using Fourier transform.

2.4 Feature extraction of P300

The feature of P300 is about evoked potentials happening about 300 ms after event. Rivet et al. [45] proposed an xDawn algorithm to define a P300 subspace and designed a spatial filter for P300 detection. Cecotti et al. [46] presented the CNN algorithm for P300 detection.

2.5 Design of classifier

Classifier aims to recognize the user's intentions based on the feature vector. Linear discriminant analysis (LDA) and support vector machine (SVM) are common linear classifiers for the EEG signal processing which are successfully applied in the MI-based BCI [47, 48] and P300 speller [49, 50]. Nonlinear classifiers including k-NN [51, 52] and ANN [53–55] also have a high performance. Generally speaking, big samples training and learning can improve the classification accuracy.

3 Hybrid BCIs

A hybrid BCI usually combines different types of BCIs in series or parallel mode according to their advantages [56]. The main purpose of hybrid BCIs is to improve the accuracy of pattern recognition. However, not all of the combinations are effective and feasible. Furthermore, in a hybrid BCI, an EEG-based BCI can combine other type of BCI such as magnetoencephalogram (MEG), electro-corticogram (ECoG), functional magnetic resonance imaging (fMRI), and near infrared spectroscopy (NIRS). Sometimes, the EEG-based BCI may combine none BCI-based system such as electromyogram (EMG), electrooculography (EOG), etc. In this paper, three major paradigms of EEG-based BCI are considered: MI, SSVEP and P300. Then, some other hybrid systems are introduced in brief.

3.1 MI-SSVEP hybrid BCIs

This hybrid BCI combines the MI and SSVEP EEG paradigms to improve the BCIs application. Zhang et al. [57] presented a pipeline of series hybrid EEG-based BCI for robot grasping. SSVEP was detected to select the target and MI was used to control the moving direction. Allison et al. [58] proposed a novel parallel combination of ERD and SSVEP tasks to gain more accurate than conventional BCIs. The subjects simultaneously imagined moving the left or right hand and focused on one of the two oscillating visual stimuli. The results showed that the hybrid condition yielded the highest classification accuracy (81.0%), followed by SSVEP (76.9%) and ERD (74.8%). Moreover, the hybrid BCI could supply subjects more BCI approach. Pfurtscheller et al. [56] presented a hybrid BCI composed of an MI-based switch and a SSVEP. The ERS brain switch could activate and control the four-step SSVEP-based orthosis. The contrast test showed that this hybrid BCI resulted a much lower rate of FPs per minute than the SSVEP BCI alone. Duan et al. [59] designed a multimodal hybrid BCI taking advantage of both SSVEP and motor imagery. Alpha rhythms were considered as a switch of SSVEP and motor imagery. They used three SSVEP signals to control the robot movement. One motor imagery signal was applied to control the robot to grasp object. These researches showed that proper combination of MI and SSVEP could improve the accuracy of classification.

3.2 P300-SSVEP Hybrid BCIs

Panicker et al. [60] proposed a hybrid BCI that combined P300 and SSVEP. In this system, the P300 paradigm was designed as a 6 × 6 speller matrix. The background was flashed with a frequency for the SSVEP detection. There were ten subjects in the online experiment. The result of average classification accuracy was 94% and the control state detection accuracy was 88.15%. Mouli and Palaniappan [61] developed a hybrid BCI based on SSVEP and P300 responses. The SSVEP paradigm was generated by visual stimuli of four independent green LED rings flashing at different frequencies. The P300 was generated by random red LED flashes locating inside each of the four rings, which were marked as events along with the recorded SSVEP EEG. The results showed that the hybrid stimulus method could improve the reliability and accuracy of BCI applications. Li et al. [62] proposed a hybrid BCI system combining P300 and SSVEP to improve the performance of asynchronous control. They designed four groups of flickering buttons in the graphical user interface. SSVEP was evoked by buttons flashing at a fixed frequency. Meanwhile, P300 potential was generated by the four large buttons in the groups intensified through shape and color changes in a random order. This hybrid BCI was applied to control the wheelchair to go or stop. The experimental results were illustrated that combining P300 and SSVEP could improve the detection accuracy and response time of the BCI system.

3.3 MI-P300 hybrid BCIs

Su et al. [63] combined P300 and MI to produce control commands in a virtual environment. MI by imagination of left and right hand movement was used for navigation. P300 oddball paradigm was applied to switch the system state. Riechmann et al. [64] proposed a P300 and ERD hybrid BCI in robotic control decision applications. Bhattacharyya et al. [65] proposed a series hybrid EEG-based BCI to control the robot arm. MI was used for the motion control and P300 was applied to stop the motion of the robot on reaching the goal position. The experimental results were showed that the proposed control method was suitable for designs of prosthetics in rehabilitative applications. However, it is very difficult to design a parallel hybrid BCI combining P300 and MI. The subjects felt hard to pay attention to the P300 stimuli meanwhile they were imagining movement.

3.3 MI-SAO hybrid BCIs

Yao et al. [5] found that somatic attention without real tactile stimulation could be applied as a novel BCI paradigm called as somatosensory attentional orientation (SAO). Then, they [66] proposed a stimulus-independent hybrid BCI based on motor imagery and SAO paradigms. The experimental results indicate that the hybrid mode combining two of the tasks such as L-SAO and R-MI performed higher classification accuracy than MI mode alone. Nowadays, many researchers are devoting to the new EEG paradigm exploration.

4 Synchronous and asynchronous BCIs

BCI can be divided into synchronous and asynchronous modes based on the input data processing modality. The synchronous BCI is predefined fixed variable time windows. Therefore, the subject is only allowed to do the given brain tasks according to the preset paradigm all the time which is designed in advance and associated with a specific cue or trigger stimulus. Unlike the synchronous BCI, an ideal asynchronous BCI system has no cue stimulus. The subject can operate the BCI by mental activities at any time if he wants. The asynchronous BCI provides a more natural human-machine interaction mode than synchronous BCI. However, the brain signals should be analyzed and classified all the time. Mental events are detected and transformed into commands as soon as possible. Therefore, it needs more computation demanding.

In the beginning, BCIs are usually designed to be synchronous. With the development of computer power, asynchronous BCI has become more popular in recent years. Mason et al. [67] designed an asynchronous BCI switch which first evaluated an asynchronous device in EEG. Receiver operating characteristics (ROC) curves were applied for evaluating the asynchronous performance during MI paradigm. Then, Townsend et al. [68] defined upper and lower thresholds for the classification of “resting periods” and “mental periods” to detect the mental activities. Borisoff et al. [69] proposed a two-state asynchronous direct brain switches for self-paced control applications for the patients with high-level spinal-cord injuries. Their experimental results by four subjects were with a mean true positive (TP) rate of 73% for false positive (FP) rate of 2%. Chae et al. [70] developed an asynchronous direct-control system for humanoid robot navigation by using MI-based EEG. Lisi et al. [71] proposed an asynchronous BCI and analyzed the EEG signal associated with gait speed changes. They assessed performance based on the logistic model probability output by means of the ROC and the respective area under the curve (AUC).

Synchronous or asynchronous BCIs should be designed based on the concrete application scene. Generally speaking, asynchronous systems are convenient for the user. However, the difficulty is the detection of brain activities by BCI all the time.

5 Shared control strategy

There are two control components in the BCIs: robotic automatic control and mental control by BCI. At first, a switch of the automatic and BCI control was often designed to change the control mode. Geng et al. [72, 73] presented a switch that could change the control models between automatic control and mental control modes to control a simulated robot by a self-paced online BCI. This switching method was proved very efficient by the experiment. However, there was only one independent control mode at any control time. The major problem of control by BCI is that the accuracy of mental pattern recognition can hardly reach 100%. Therefore, the commands of BCI may be wrong from the real mental activities.

In order to reduce and avoid these errors, shared control method is proposed and applied in asynchronous BCIs widely. Sawaragi et al. [74] proposed a shared control strategy for a mobile robot teleoperation system. Ivanisevic et al. [75] designed a shared control system combining robotic intelligence and human experience in the teleoperation task. Millan et al. [76] proposed an EEG-based asynchronous BMI to control a wheelchair. The shared control strategy combining the intelligent wheelchair and BMI is applied to help the subject for driving. Their method is proved efficient for the subjects to drive the wheelchair by mental control. Su et al. [77] presented a dual control path method for the asynchronous BCIs. They combined two control paths for one data path and obtain a good performance. Liu et al. [78] designed a shared controller referring to fuzzy discrete event system (FDES) theory for driving an intelligent wheelchair. Sun et al. [79] presented a novel shared control method called fused fuzzy Petri nets (FFPNs) to control a robotic arm and hand with high degree of freedom to grasp objects. On the one hand, FFPNs could reduce the bad impacts of wrong EEG pattern recognition; on the other hand, this Petri net method is easy to design and realize the optimal control. Both MATLAB simulation and practical robotic experiments showed that the shared control method could improve the performance and robustness for the BCIs.

The shared control method mainly deals with the relation of BCI control and robotic control. However, the accuracy of mental command can hardly reach 100%. Therefore, the mental commands from BCI are not reliable.

6 Conclusions

This paper has reviewed fundamental design of EEG-based BCIs. BCI researches have achieved many breakthroughs since the past 10 years. Various methods of signal feature extraction and classification algorithms for EEG pattern recognition have been proposed to improve the information bit rate and classification accuracy. Novel designs of EEG-based BCIs for widespread applications in the daily life of disabled people, such as wheelchair control, words spelling or prostheses manipulating, are presented year by year.

In spite of the recent development in the BCIs design, many problems still need to be solved for the system design. Firstly, the algorithms improving the classification accuracy and reducing the user training should be further developed. Multi-class recognition with high accuracy is still a very hard problem. Secondly, more optimal combination modes of BCI paradigms should be designed to maximum advantages. Thirdly, asynchronous BCIs need more efficient and precise algorithms to detect the mental activities. Finally, proper shared control methods combining the BCI commands and robotic control should be designed to improve the robustness and safety.

Recently, the sampling channel amount and accuracy of EEG devices develop fast. New BCI applications boost the BCI research. Therefore, the design of EEG-based BCIs is vital for the user manipulation. More and more innovative methods should be proposed to better BCIs.

Footnotes

This work was supported by the Innovation Cultivating Fund Project (No. 17-163-12-ZT-001-019-01).

All contributing authors have no conflict of interests.

Wenchang Zhang was born in Shandong Province, China, in 1982. He received the B.S. degree of computer science from Shandong University of Technology, China, in 2004 and the M.S. degree of biomedical engineering from Academy of Military Medical Science, Beijing, China, in 2008. In 2018, He received the PhD degree from the Department of Computer Sciences and Technology, Tsinghua University. Also, He is a research assistant at Academy of Military Science. His research interests include artificial intelligence, robotics and brain-computer interface.

Chuanqi Tan is a Ph.D. candidate in Department of Computer Sciences and Technology, Tsinghua University. Before this, he worked at baidu.com in 2014 and in 2012. His research interests include artificial intelligence, deep learning, transfer learning and brain-computer interface.

Fuchun Sun was born in Jiangsu Province, China, in 1964. He received Ph.D. degree from the Department of Computer Science and Technology, Tsinghua University, Beijing, China, in 1998. From 1998 to 2000 he was a Postdoctoral Fellow of the Department of Automation at Tsinghua University, Beijing, China. Now he is a professor in the Department of Computer Science and Technology, Tsinghua University, Beijing, China. His research interests include intelligent control, networked control system and management, neural networks, fuzzy systems, and nonlinear systems and robotics.

Hang Wu was born in Sichuan Province, China, in 1991. He received the B.S. degree of mechanical engineering from Tianjin University, Tianjin, China, in 2012 and the PhD degree of biomedical engineering from Academy of Military Medical Science, Beijing, China, in 2017. Currently, he is a research assistant at Academy of Military Science. His research interests are robot vision and graphics processing, including visual terrain classification, remotely sensed image analysis, and pattern recognition.

Bo Zhang received the degree from the Department of Automatic Control, Tsinghua University, in 1958. He is currently a professor with the Department of Computer Science and Technology, Tsinghua University. He has published over 150 papers and four monographs. His current research interests include artificial intelligence, pattern recognition, neural networks, and intelligent control. He is the fellow of the Chinese Academy of Sciences.

References

Allison

B.Z.

, Brunner

, Kaiser

, Mullerputz

G.R.

, Neuper

, Pfurtscheller

Toward a hybrid brain-computer interface based on imagined movement and visual attention. J Neural Eng 2010, 7: 026007.

Vidal

J.J.

Real-time detection of brain events in EEG. Proc IEEE 1977, 65(5): 633–641.

Kalcher

, Flotzinger

, Neuper

, Gölly

, Pfurtscheller

Graz brain-computer interface II: towards communication between humans and computers based on online classification of three different EEG patterns. Med Biol Eng Comput 1996, 34(5): 382–388.

Farwell

L.A.

, Donchin

Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol 1988, 70(6): 510–523.

Yao

, Sheng

X.J.

, Zhang

D.G.

, Jiang

, Farina

, Zhu

X.Y.

A BCI system based on somatosensory attentional orientation. IEEE Trans Neural Syst Rehabil Eng 2017, 25(1): 81–90.

Yao

, Sheng

X.J.

, Mrachacz-Kersting

, Zhu

X.Y.

, Farina

, Jiang

Decoding covert somatosensory attention by a BCI system calibrated with tactile sensation. IEEE Trans Biomed Eng 2018, 65(8): 1689–1695.

Muller-Putz

C.R.

, Scherer

, Neuper

, Pfurtscheller

Steady-state somatosensory evoked potentials: Suitable brain signals for brain-computer interfaces?

IEEE Trans Neural Syst Rehabil Eng 2006, 14(1): 30–37.

Ahn

, Ahn

, Cho

, Chan Jun

Achieving a hybrid brain-computer interface with tactile selective attention and motor imagery. J Neural Eng 2014, 11(6): 066004.

Breitwieser

, Pokorny

, Gernot

R.M.

, Müller-Putz

A hybrid three-class brain-computer interface system utilizing SSSEPs and transient ERPs. J Neural Eng 2016, 13(6): 066015.

10.

Allison

B.Z.

, Faller

, Neuper

C.H.

BCIs that use steady-state visual evoked potentials or slow cortical potentials. In Brain-Computer Interfaces: Principles and Practice. Wolpaw

, Wolpaw

E.W.

, Eds. Oxford: Oxford University Press, 2012.

11.

Birbaumer

, Ghanayim

, Hinterberger

, Iversen

, Kotchoubey

, Kübler

, Perelmouter

, Taub

, Flor

A spelling device for the paralysed. Nature 1999, 398(6725): 297–298.

12.

Sellers

, Arbel

, Donchin

BCIs that uses P300 event-related potentials. In Brain-Computer Interfaces: Principles and Practice. Wolpaw

, Wolpaw

E.W.

, Eds. Oxford: Oxford University Press, 2012.

13.

Sutter

E.E.

The brain response interface: communication through visually-induced electrical brain responses. J Microcomput Appl 1992, 15(1): 31–45.

14.

Vidal

J.J.

Toward direct brain-computer communication. Annu Rev Biophys Bioeng 1973, 2: 157–180.

15.

Allison

B.Z.

, Wolpaw

E.W.

, Wolpaw

J.R.

Brain-computer interface systems: Progress and prospects. Expert Rev Med Devices 2007, 4(4): 463–474.

16.

Brunner

, Allison

B.Z.

, Krusienski

D.J.

, Kaiser

, Müller-Putz

G.R.

, Pfurtscheller

, Neuper

Improved signal processing approaches in an offline simulation of a hybrid brain-computer interface. J Neurosci Methods 2010, 188(1): 165–173.

17.

Fatourechi

, Bashashati

, Ward

R.K.

, Birch

G.E.

EMG and EOG artifacts in brain computer interface systems: A survey. Clin Neurophysiol 2007, 118(3): 480–494.

18.

Flexer

, Bauer

, Pripfl

, Dorffner

Using ICA for removal of ocular artifacts in EEG recorded from blind subjects. Neural Netw 2005, 18(7): 998–1005.

19.

Gao

J.F.

, Yang

, Lin

, Wang

, Zheng

C.X.

Automatic removal of eye-movement and blink artifacts from EEG signals. Brain Topogr 2010, 23(1): 105–114.

20.

Krusienski

D.J.

, McFarland

D.J.

, Wolpaw

J.R.

An evaluation of autoregressive spectral estimation model order for brain-computer interface applications. In Proceedings of 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, New York, NY, USA, 2006, pp 1323–1326.

21.

Wang

, Xu

G.Z.

, Wang

, Zhang

H.Y.

Feature extraction of brain-computer interface based on improved multivariate adaptive autoregressive models. In Proceedings of the 2010 3rd International Conference on Biomedical Engineering and Informatics, Yantai, China, 2010, pp 895–898.

22.

Demiralp

, Yordanova

, Kolev

, Ademoglu

, Devrim

, Samar

V.J.

Time-frequency analysis of single-sweep event-related potentials by means of fast wavelet transform. Brain Lang 1999, 66(1): 129–145.

23.

Farina

, do Nascimento

O.F.

, Lucas

M.F.

, Doncarli

Optimization of wavelets for classification of movement-related cortical potentials generated by variation of force-related parameters. J Neurosci Methods 2007, 162(1-2): 357–363.

24.

Ramoser

, Muller-Gerking

, Pfurtscheller

Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng 2000, 8(4): 441–446.

25.

Grosse-Wentrup

, Buss

Multiclass common spatial patterns and information theoretic feature extraction. IEEE Trans Biomed Eng 2008, 55(8): 1991–2000.

26.

Lemm

, Blankertz

, Curio

, Muller

K.R.

Spatiospectral filters for improving the classification of single trial EEG. IEEE Trans Biomed Eng 2005, 52(9): 1541–1548.

27.

Dornhege

, Blankertz

, Krauledat

, Losch

, Curio

, Muller

K.R.

Combined optimization of spatial and temporal filters for improving brain-computer interfacing. IEEE Trans Biomed Eng 2006, 53(11): 2274–2281.

28.

Novi

, Guan

C.T.

, Dat

T.H.

, Xue

Sub-band common spatial pattern (SBCSP) for brain-computer interface. In Proceedings of the 2007 3rd International IEEE/EMBS Conference on Neural Engineering, Kohala Coast, USA, 2007, pp 204–207.

29.

Ang

K.K.

, Chin

Z.Y.

, Zhang

H.H.

, Guan

C.T.

Filter bank common spatial pattern (FBCSP) in brain-computer interface. In Proceedings of 2008 IEEE International Joint Conference on Neural Networks, Hong Kong, China, 2008, pp 2390–2397.

30.

Mousavi

E.A.

, Maller

J.J.

, Fitzgerald

P.B.

, Lithgow

B.J.

Wavelet common spatial pattern in asynchronous offline brain computer interfaces. Biomed Signal Process Control 2011, 6(2): 121–128.

31.

Aghaei

A.S.

, Mahanta

M.S.

, Plataniotis

K.N.

Separable common spatio-spectral patterns for motor imagery BCI systems. IEEE Trans Biomed Eng 2016, 63(1): 15–29.

32.

Zhang

W.C.

, Sun

F.C.

, Tan

C.Q.

, Liu

S.B.

Linear dynamical systems modeling for EEG-based motor imagery brain-computer interface. In Proceedings of the 3rd International Conference on Cognitive Systems and Signal Processing, Beijing, China, 2016.

33.

Zhang

W.C.

, Sun

F.C.

, Tan

C.Q.

, Liu

S.B.

Low-rank linear dynamical systems for motor imagery EEG. Comput Intell Neurosci 2016, 2016: 2637603.

34.

, Kuang

D.P.

, Guo

X.J.

, Zhao

Y.L.

, He

L.H.

A deep learning method for classification of EEG data based on motor imagery. In Proceedings of the 10th International Conference on Intelligent Computing in Bioinformatics, Taiyuan, China, 2014, pp 203–210.

35.

Tan

C.Q.

, Sun

F.C.

, Zhang

W.C.

, Kong

Electroencephalography classification in brain-computer interface with manifold constraints transfer. In Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Honolulu, HI, USA, 2018.

36.

Tan

C.Q.

, Sun

F.C.

, Zhang

W.C.

, Kong

, Yang

, Zhang

X.Y.

Adaptive adversarial transfer learning for electroencephalography classification. In Proceedings of 2018 International Joint Conference on Neural Networks, Rio de Janeiro, Brazil, 2018.

37.

Tan

C.Q.

, Sun

F.C.

, Zhang

W.C.

Deep transfer learning for EEG-based brain computer interface. In Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada, 2018.

38.

Tan

C.Q.

, Sun

F.C.

, Zhang

W.C.

, Chen

J.H.

, Liu

C.F.

Multimodal classification with deep convolutional-recurrent neural networks for electroencephalography. In Proceedings of the 24th International Conference on Neural Information Processing, Guangzhou, China, 2017.

39.

Tan

C.Q.

, Sun

F.C.

, Zhang

W.C.

, Liu

S.B.

, Liu

C.F.

Spatial and spectral features fusion for EEG classification during motor imagery in BCI. In Proceedings of 2017 IEEE EMBS International Conference on Biomedical & Health Informatics, Orlando, FL, USA, 2017, pp 309–312.

40.

Lin

Z.L.

, Zhang

C.S.

, Wu

, Gao

X.R.

Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans Biomed Eng 2006, 53(12): 2610–2614.

41.

Pan

, Gao

X.R.

, Duan

, Yan

, Gao

S.K.

Enhancing the classification accuracy of steady-state visual evoked potential-based brain-computer interfaces using phase constrained canonical correlation analysis. J Neural Eng 2001, 8(3): 036027.

42.

Zhang

, Zhou

G.X.

, Zhao

Q.B.

, Onishi

, Jin

, Wang

X.Y.

, Cichocki

Multiway canonical correlation analysis for frequency components recognition in SSVEP-Based BCIs. In Proceedings of the 18th International Conference on Neural Information Processing, Shanghai, China, 2011, pp 287–295.

43.

Zhang

, Zhou

G.X.

, Jin

, Wang

X.Y.

, Cichocki

Frequency recognition in SSVEP-based BCI using multiset canonical correlation analysis. Int J Neural Syst 2014, 24(4): 1450013.

44.

Cecotti

, Graeser

Convolutional neural network with embedded Fourier transform for EEG classification. In Proceedings of the 2008 19th International Conference on Pattern Recognition, Tampa, FL, USA, 2008, pp 1–4.

45.

Rivet

, Souloumiac

, Attina

, Gibert

xDAWN algorithm to enhance evoked potentials: application to brain-computer interface. IEEE Trans Biomed Eng 2009, 56(8): 2035–2043.

46.

Cecotti

, Graser

Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE Trans Pattern Anal Mach Intell 2011, 33(3): 433–445.

47.

Pfurtscheller

Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin Neurophysiol 1999, 110(11): 1842–1857.

48.

Bostanov

BCI competition 2003-data sets Ib and IIb: feature extraction from event-related brain potentials with the continuous wavelet transform and the t-value scalogram. IEEE Trans Biomed Eng 2004, 51(6): 1057–1061.

49.

Kaper

, Meinicke

, Grossekathoefer

, Lingner

, Ritter

BCI competition 2003-data set IIb: Support vector machines for the p300 speller paradigm. IEEE Trans Biomed Eng 2004, 51(6): 1073–1076.

50.

Rakotomamonjy

, Guigue

, Mallet

, Alvarado

Ensemble of SVMs for improving brain computer interface p300 speller performances. In Proceedings of the 15th International Conference on Artificial Neural Networks, Warsaw, Poland, 2005.

51.

Schlögl

, Lee

, Bischof

, Pfurtscheller

Characterization of four-class motor imagery EEG data for the BCI-competition 2005. J Neural Eng 2005, 2(4): L14–L22.

52.

Kayikcioglu

, Aydemir

A polynomial fitting and k-NN based approach for improving classification of motor imagery BCI data. Pattern Recognit Lett 2010, 31(11): 1207–1215.

53.

Haselsteiner

, Pfurtscheller

Using time-dependent neural networks for EEG classification. IEEE Trans Rehabil Eng 2000, 8(4): 457–463.

54.

Hazrati

M.K.

, Erfanian

An online EEG-based brain-computer interface for controlling hand grasp using an adaptive probabilistic neural network. Med Eng Phys 2010, 32(7): 730–739.

55.

Felzer

, Freisieben

Analyzing EEG signals using the probability estimating guarded neural classifier. IEEE Trans Neural Syst Rehabil Eng 2003, 11(4): 361–371.

56.

Pfurtscheller

, Solis-Escalante

, Ortner

, Linortner

, Muller-Putz

G.R.

Self-paced operation of an SSVEP-based orthosis with and without an imagery-based “brain switch”: A feasibility study towards a hybrid BCI. IEEE Trans Neural Syst Rehabil Eng 2010, 18(4): 409–414.

57.

Zhang

W.C.

, Sun

F.C.

, Liu

C.F.

, Su

W.H.

, Tan

C.Q.

, Liu

S.B.

A hybrid EEG-based BCI for robot grasp controlling. In Proceedings of 2017 IEEE International Conference on Systems, Man, and Cybernetics, Banff, Canada, 2017, pp 3278–3283.

58.

Allison

B.Z.

, Brunner

, Kaiser

, Muller-Putz

G.R.

, Neuper

, Pfurtscheller

Toward a hybrid brain-computer interface based on imagined movement and visual attention. J Neural Eng 2010, 7(2): 026007.

59.

Duan

, Lin

D.X.

, Li

W.Y.

, Zhang

Design of a multimodal EEG-based hybrid BCI system with visual servo module. IEEE Trans Auton Ment Dev 2015, 7(4): 332–341.

60.

Panicker

R.C.

, Puthusserypady

, Sun

An asynchronous P300 BCI with SSVEP-based control state detection. IEEE Trans Biomed Eng 2011, 58(6): 1781–1788.

61.

Mouli

, Palaniappan

Hybrid BCI utilising SSVEP and P300 event markers for reliable and improved classification using LED stimuli. In Proceedings of 2017 IEEE Symposium on Computer Applications & Industrial Electronics, Langkawi, Malaysia, 2017, pp 127–131.

62.

Y.Q.

, Pan

J.H.

, Wang

, Yu

Z.L.

A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control. IEEE Trans Biomed Eng 2013, 60(11): 3156–3166.

63.

, Qi

, Luo

J.X.

, Wu

, Yang

, Li

, Zhuang

Y.T.

, Zheng

X.X.

, Chen

W.D.

A hybrid brain-computer interface control strategy in a virtual environment. J Zhejiang Univ Sci C 2011, 12(5): 351–361.

64.

Riechmann

, Hachmeister

, Ritter

, Finke

Asynchronous, parallel on-line classification of P300 and ERD for an efficient hybrid BCI. In Proceedings of the 2011 5th International IEEE/EMBS Conference on Neural Engineering, Cancun, Mexico, 2011, pp 412–415.

65.

Bhattacharyya

, Konar

, Tibarewala

D.T.

Motor imagery, P300 and error-related EEG-based robot arm movement control for rehabilitation purpose. Med Biol Eng Comput 2014, 52(12): 1007–1017.

66.

Yao

, Sheng

X.J.

, Zhang

D.G.

, Jiang

, Mrachacz-Kersting

, Zhu

X.Y.

, Farina

A stimulus-independent hybrid BCI based on motor imagery and somatosensory attentional orientation. IEEE Trans Neural Syst Rehabil Eng 2017, 25(9): 1674–1682.

67.

Mason

S.G.

, Birch

G.E.

A brain-controlled switch for asynchronous control applications. IEEE Trans Biomed Eng 2000, 47(10): 1297–1307.

68.

Townsend

, Graimann

, Pfurtscheller

Continuous EEG classification during motor imagery-simulation of an asynchronous BCI. IEEE Trans Neural Syst Rehabil Eng 2004, 12(2): 258–265.

69.

Borisoff

J.F.

, Mason

S.G.

, Birch

G.E.

Brain interface research for asynchronous control applications. IEEE Trans Neural Syst Rehabil Eng 2006, 14(2): 160–164.

70.

Chae

, Jeong

, Jo

Toward brain-actuated humanoid robots: asynchronous direct control using an EEG-based BCI. IEEE Trans Rob 2012, 28(5): 1131–1144.

71.

Lisi

, Morimoto

EEG single-trial detection of gait speed changes during treadmill walk. PLoS One 2015, 10(5): e0125479.

72.

Geng

, Gan

J.Q.

, Hu

H.S.

A self-paced online BCI for mobile robot control. Int J Adv Mech Syst 2010, 2(1-2): 28–35.

73.

Geng

, Gan

J.Q.

Motor prediction in brain-computer interfaces for controlling mobile robots. In Proceedings of the 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vancouver, Canada, 2008, pp 634–637.

74.

Sawaragi

, Takayuki

, Akashi

Foundations for designing an ecological interface for mobile robot teleoperation. Robot Auton Syst 2000, 31(3): 193–207.

75.

Ivanisevic

, Lumelsky

V.J.

Configuration space as a means for augmenting human performance in teleoperation tasks. IEEE Trans Syst Man Cybern Part B Cybern 2000, 30(3): 471–484.

76.

Millan

D.R.J.

, Galan

, Vanhooydonck

, Lew

, Philips

, Nuttin

Asynchronous non-invasive brain-actuated control of an intelligent wheelchair. In Proceedings of 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, USA, 2009, pp 3361–3364.

77.

, Shen

, Wang

Z.Y.

, Wang

Y.R.

, Huang

L.B.

, Shi

DCP: Improving the throughput of asynchronous pipeline by dual control path. In Proceedings of the 2013 IEEE 10th International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing, Zhangjiajie, China, 2013.

78.

Liu

, Wang

Y.X.

, Zhang

An FDES-based shared control method for asynchronous brain-actuated robot. IEEE Trans Cybern 2016, 46(6): 1452–1462.

79.

Sun

F.C.

, Zhang

W.C.

, Chen

J.H.

, Wu

, Tan

C.Q.

, Su

W.H.

Fused fuzzy petri nets: a shared control method for brain computer interface systems. IEEE Trans Cogn Dev Syst, in press, DOI: 10.1109/TCDS.2018.2818173.

A Review of EEG-Based Brain-Computer Interface Systems Design

Abstract

Keywords

1 Introduction

2 EEG signal processing

2.1 Signal preprocessing

(1) Channel selection

(2) Time window setting

(3) Artifacts removal

2.2 Feature extraction of MI-based EEG

2.3 Feature extraction of SSVEP

2.4 Feature extraction of P300

2.5 Design of classifier

3 Hybrid BCIs

3.1 MI-SSVEP hybrid BCIs

3.2 P300-SSVEP Hybrid BCIs

3.3 MI-P300 hybrid BCIs

3.3 MI-SAO hybrid BCIs

4 Synchronous and asynchronous BCIs

5 Shared control strategy

6 Conclusions

Footnotes

References