Sage Journals: Discover world-class research

Abstract

The advancement in neuroscience and computer science promotes the ability of the human brain to communicate and interact with the environment, making brain–computer interface (BCI) top interdisciplinary research. Furthermore, with the modern technology advancement in artificial intelligence (AI), including machine learning (ML) and deep learning (DL) methods, there is vast growing interest in the electroencephalogram (EEG)‐based BCIs for AI‐related visual, literal, and motion applications. In this review study, the literature on mainstreams of AI for the EEG‐based BCI applications is investigated to fill gaps in the interdisciplinary BCI field. Specifically, the EEG signals and their main applications in BCI are first briefly introduced. Next, the latest AI technologies, including the ML and DL models, are presented to monitor and feedback human cognitive states. Finally, some BCI‐inspired AI applications, including computer vision, natural language processing, and robotic control applications, are presented. The future research directions of the EEG‐based BCI are highlighted in line with the AI technologies and applications.

Keywords

electroencephalogram (EEG)brain–computer interface (BCI)artificial intelligence computer vision natural language processing robot controls

1 Introduction

Generally, electroencephalography (EEG) signals that measure cortical electrical activities directly with high temporal resolution [1 –3] are widely used in many non‐invasive brain–computer interfaces (BCIs). As shown in Fig. 1, the EEG‐based BCI aims to investigate a bi‐direction communication channel between the human brain and external devices. By touching the EEG electrodes within the head cortex region, it allows us to extract multiple non‐overlapping frequency band signals, such as delta, theta, alpha, and beta, to interact with different human behavioral states and present various corresponding patterns.

Fig. 1.

EEG‐based BCI: a bi‐direction communication channel between the human brain and external devices.

The BCI close to the EEG’s cerebral cortex dynamics helps humans to understand cognitive activities and assist in the advancement of computer science and engineering. Particularly the most two popular application areas where the EEG‐based BCI is widely used include: (1) promoting human–computer interaction (HCI) and related entertainment games [4]. For example, the general approach of conventional HCI is mostly limited to manual operations and communications. However, the BCI has changed the way of HCI applications in complex and demanding operating environments and may become a revolution strategy of the HCI. Besides, the compact wireless BCI headset mainly developed for the gaming industry is wearable and flexible and can be installed without pre‐settings. Although they are not as efficient as other state‐of‐the‐art BCI equipment used in clinical experiments, they are still suitable for entertainment purposes and used for developing interactive games; (2) playing an important role in pattern recognition and machine learning (ML) of brain dynamics.

Recently, some review studies show that artificial intelligence (AI) technologies, such as the ML and deep neural networks are used to qualify brain activities from various learning algorithms [5, 6]. The findings of these studies suggest that designs of the adaptive BCI framework and extractions of the reliable neural patterns are useful in improving the performance of BCI learning. However, they only consider some traditional EEG experiments, such as audial, visual stimulations, to support fatigue, emotion, or motor‐imagery assessment. The current and future advancement expect the EEG‐based BCI to support the mainstream applications of AI, such as computer vision (CV), natural language processing (NLP), robotics, and data visualizations.

In this review study, we provide a comprehensive EEG‐based BCI review to summarize the AI approaches, including ML (supervised/unsupervised learning, transfer learning, and reinforcement learning) and deep learning (DL), and to concentrate the connections to spotlights of the AI applications, as addressed in Fig. 2. Sections 2, 3, and 4 address them point‐by‐point.

Fig. 2.

The graph of artificial intelligence (machine learning and deep learning) and their applications.

2 Machine learning for EEG‐based BCI

2.1 Overview

Machine learning primarily aims to automatically discover the brain patterns in a specific task without using traditional statistical methods. The ML technologies in EEG‐based BCI applications are usually classified into two major categories: classification, and individual adaptive tasks.

Before applying the ML technologies to EEG signals, we require to preprocess the EEG signals and then extract features, such as power spectral and functional connectivity [7] from the preprocessed EEG data. Fig. 3 shows a genetic pipeline of data processing in the EEG‐based BCI in a compact and relevant way, which includes data acquisition, data preprocessing, pattern extraction, and classification techniques.

Fig. 3.

The genetic pipeline of data processing in the EEG‐based BCI.

In terms of classification tasks, we usually apply the following two conventional types of ML: supervised learning and unsupervised learning [8]. Specifically, in supervised learning, the data is usually divided into two subsets: the training and testing sets. The testing set is used to judge the type of event by using the knowledge learned in the training phase with label sets. To classify or predict future events, supervised learning can be used for regression or classification tasks. The data used for training here is neither classified nor labeled; instead, we apply unsupervised learning.

In terms of individual adaptive tasks, we generally refer to transfer learning, exhibiting distinctive classification problems of consistency. The above ML strategy, such as supervised learning, assumes that the training and the testing datasets used to evaluate the classifier geared with the same feature space and probability distribution. However, the above assumption is usually violated in practical applications because the human body is different [9] in most cases. Therefore, transfer learning can be used to overcome the purpose of dealing with data that violates the above hypothesis by using the knowledge obtained while learning a given task to solve another but related tasks.

Furthermore, reinforcement learning (RL) features rewards through the interaction between learning and the environment. It uses feedback signals to build adaptability that is suitable for EEG‐based BCI. The RL learning method helps us to understand better, and then use brain activities to interact with computers and other devices to improve the performance of several EEG‐based BCI applications.

2.2 Supervised and unsupervised learning

The traditional supervised learning models include linear models, neural networks (NNs), Bayesian classifiers, neighbor‐distance classifiers, and classifier combinations [10]. For linear models, such as linear discriminant analysis (LDA), regularized LDA, and linear support vector machine (SVM), the discriminative EEG patterns are classified using the linear decision boundary between the feature vectors of each class. Furthermore, for NNs, they assemble artificial neuron layers to border on any non‐linear decision boundary. A common type in EEG‐based BCI applications is a multi‐layer perceptron, using only one or two hidden layers. In the next section, we will discuss deep NNs for DL. Then, in terms of non‐linear Bayesian classifiers, such as Bayesian quadratic classifiers and Hidden Markov Models, it measures the probability distribution of each category and uses Bayesian rules to select the EEG patterns. Besides, the nearest‐neighbor classifier, such as the k‐nearest‐neighbor (kNN) algorithm, considering the physical distance of each class, proposes to assign classes to the EEG patterns according to its nearest‐neighbor. Finally, a combination strategy, such as voting or stacked‐based combination method, is applied in classification tasks by mixing the predicted classes from multi‐classifiers.

Based on the EEG‐based BCI classification algorithms in the past ten years [11], the classifier options that go with different kinds of EEG patterns/features were reviewed as follows:

EEG features with noise and/or outliers: we suggest choosing regularized classifiers, such as the SVM, which is suitable for handling outliers. The discriminative classifiers [12] may be a back‐up choice, as it showed better performance compared to generative classifiers in the presence of noise or outliers.

EEG features with high dimensions: The SVM may be the most suitable classifier for processing high‐dimensional feature vectors. Alternatively, using dimensionality reduction techniques for feature selection may solve this problem as well.

EEG features with non‐stationarity: since combination strategy classifiers reduce the information variance, some stable linear classifiers, such as LDA and SVM, are suitable for testing the classification performance.

2.3 Transfer learning

The EEG patterns are the potential to be passed between different domains. For example, selecting one type of human activity is transmitted based on the similarity of the EEG pattern between the target and source domains. If both domains are very similar, and the EEG pattern is semblable, the discriminatory information should be transmitted. However, for the cases that domains are diverse, and there exists some standard information between the target and source domains, where some unchanged or persistent information can be transferred via a constant system.

In EEG‐based BCI applications, the covariate shift is the most common case, which input distributions of the training and test samples differently [13]. The BCI‐based transfer learning is divided into three categories: (1) transfer from tasks to tasks, (2) transfer from subjects to subjects, and (3) transfer from sessions to sessions. For example, given a set of training data sets, such as source tasks, subjects, or sessions, the covariate shift tries to find a transformation space where training the model could facilitate the classification or prediction of new samples.

Especially, subject to subject transfer is the leading category in the EEG‐based BCI transfer learning. The BCI may need to collect a small amount of data, called preliminary data, to overcome differences between subjects, before applying individual EEG features to a training model. But transfer learning can overcome the above matter. For example, the study of driving drowsiness detection [14] evaluated inter‐subject and intra‐subject variability. The transfer was verified by performing hierarchical clustering on large‐scale EEG data sets collected from a large number of subjects.

Furthermore, interdisciplinary transfer learning can also be used to reduce the training data collection time, which is the least square transformation (LST) approach [15]. The experiments conducted to confirm the performance of the LST approach using interdisciplinary visual stimulation data demonstrate that the number of training templates can be reduced. Besides, when labeling data is not possible, the inter‐subject and inter‐subject transfer learning can be applied under unsupervised conditions. For instance, a recent study [16] is used to directly align the EEG trajectories between different objects in Euclidean space with increasing similarity. Experimental results confirm the potential for learners to transfer learning using unsupervised ECI‐based BCI.

2.4 Reinforcement learning

RL aims to learn and optimize the agent’s actions and behaviors based on reward signals during task execution. These reward signals are usually designed by human experience. Still, these rewards are often difficult to encode in complex situations, making the EEG activities from a BCI system a suitable substitution. Furthermore, it is a novel approach to capture subjective aspects and criteria unique to each user. For example, the EEG signals are considered a reward reference to accelerated robot movement and learning [17] and applied in neurofeedback mindfulness system optimization [18].

3 Deep learning for EEG‐based BCI

3.1 Overview

Deep learning is a series of specific ML algorithms, which refers to the system of deep NNs, and it is used to learn the EEG patterns [19]. The typical characteristic of the DL systems is based on a large number of neurons in multiple cascades, allowing them to learn features highly related to the human cognitive level. The typical systems for DL include convolutional neural network (CNN), recurrent neural network (RNN), and generative adversarial network (GAN). For the EEG‐based BCI applications, the DL is widely used to extract EEG features and then combine with the traditional ML technology to achieve a stateof‐the‐art classification or regression task. In this section, spontaneous EEG‐based BCI applications with CNN system, the use of GAN, the process and application of RNN, especially long short‐term memory (LSTM), and the spatiotemporal combination of CNN and LSTM are introduced.

3.2 Convolutional neural networks

Convolutional neural network is a model of feedforward deep NNs, where input information flows the EEG patterns through the essential three stacked layers: a convolutional layer, pooling layer, and fully connected layer [20]. The convolutional layer convolves the tensor with the shape, and then pooling layer simplifies the necessary calculations to reduce the data dimension. Finally, the fully connected layer connects each neuron in the previous layer to a new layer for predicting or classifying the target goals. This system of DL with stacked layers aims to reduce the input data and capture unique spatial dependence of the EEG patterns.

The significant progress of CNN in classification tasks is vital to the development of the BCI, such as improvement of EEG‐based fatigue assessment. For example, an EEG‐based system of the spatial‐temporal convolutional neural network was proposed [21]. The advance of this system extracts temporal dependencies. It combines with dense layers for spatial‐temporal EEG information process and classification, which achieved 97% accuracy in the classification of EEG driving fatigue levels. Another recent study [22] also assessed diverse fatigue levels in a visual EEG experiment. It proposed a system of multi‐scale CNN that uses a space‐time-frequency combined features extraction strategy to extract raw EEG signals from an inception structure.

3.3 Recurrent neural networks and long short‐term memory

The RNN, featuring with a temporal sequence to record the memory, can address human reasoning by forming a directed graph between neurons. However, the generic network of RNN could only look at the new dependencies to avoid retaining some long‐term information. A particular type of RNN, LSTM networks, was proposed to handle learning long‐term dependencies capability.

The studies of the RNN or LSTM framework in EEG‐based BCIs increased substantially in the past years with findings proposing that the results of RNN or LSTM‐based methods outperform traditional ML. The reason is that they can extract hidden EEG feature representations, and RNN combined with CNN can optimize feature representations by the integration of both temporal and spatial information of EEG signals. The framework is helpful for motion identification, emotion recognition, and sleep staging assessment [23]. For example, Supratak et al. [24] investigated automatic sleep stage scoring by employing an LSTM bidirectional algorithm for transitional rules learning of hidden time‐series EEG features and CNN to extract time-invariable EEG features.

3.4 Generative adversarial networks

In some cases, the real‐world EEG data is limited, which could be vulnerability when we train some ML or DL models. To overcome this problem, the GAN, including two synchronously trained NNs, called “generator networks” and “discriminator networks”, was recently proposed primarily for data augmentation purposes. It claims GAN generates new samples to mimic real‐world data from implying generative models [25].

For example, a novel GAN model [26] was recently proposed, which learned the statistical characteristics of the EEG and increased the size of datasets by generating synthesis samples to improve classification performance. Also, a new EEG‐based framework [27], named Conditional Wasserstein GAN, was proposed to enhance emotion recognition by generating high‐quality synthetic EEG data based using the SEED and DEAP datasets.

4 EEG‐based BCI inspired by artificial intelligence applications

Here, we list the three mainstreams of AI applications integrating EEG knowledge and CV, NLP, and robotic controls.

4.1 Computer vision

Computer vision explores digital images from cameras and videos and uses the ML to identify and classify objects to represent what a human sees accurately.

The EEG signals were recently investigated to see if they could be used as effective ways to transform human visual abilities into the CV. For example, an experiment of previous work [28] used 128‐channel electrodes to record the EEG signals, while six subjects were meant to view 2000 images from 40 types of object classes. Then, the authors proposed a visual object classifier driven by the EEG signals, which use an RNN to learn discriminative brain activities manifolds of visual categories. Also, they trained a CNN‐based regression model to project the image onto the learned manifold for classifications with 83% accuracy. Furthermore, a follow‐up study [29] used the learned EEG manifold to adjust the GAN’s image generation. During the inference process, the GAN reads the EEG signals and converts it into an image, while the generated image is real and very similar to the image of GAN’s EEG signals.

Another recent study [30] introduced a brain sourcing framework which measured EEG responses to perform a recognition task where visual features of faces (gender, hair color, age, and smile) are recognized better from a group of subjects instead of the individual subject. It may open opportunities for more general applications using the BCI in a crowdsourced setting.

4.2 Natural language processing

The NLP is a theory‐motivated range of computation methods for automatic analysis and representation of human language, where distributional semantics describe the meaning of words and sentences using vectorized representations.

We assume that our brain processes language and generates cognitive processing data, such as the EEG signals, when we read words and sentences. A recent work [31] evaluated word representations from reflecting the semantic representations in the EEG and associated physiological signals. It proposed a multi‐modal framework, called CogniVal, for evaluating English word representations based on cognitive lexical semantics. Also, six types of word embeddings were evaluated by fitting them to 4 EEG datasets during language processing.

Machine translation is a critical element of NLP, which aims to build a reliable encoder-decoder system. Recently, Chang’s team [32] released an EEG‐based text translation study showing how to decode EEG signals with high accuracy and natural speech rate. They trained an RNN to encode each sentence length sequence of neural activity into an abstract representation, and then decode this representation verbatim into English sentences, and achieved a 3% average word error rate.

4.3 Robotic controls

For a human who cannot control such robots through peripheral nerves and muscles, the EEG‐based BCIs are instead proven suitable means to communicate with the robots. The early stage of the EEG‐based robotic controls was driving a wheelchair from one destination to the other by using visual simulations or motor imagery. Some recent studies are motivated to address the safety risks in controlling robots. Li et al. [33] proposed a shared control strategy between a brain–machine and autonomous control modes. In the autonomous control mode, the synthesis of angle‐based potential field and vision‐based simultaneous localization and mapping was used to guide the robot for obstacle avoidance and keep the robot safe. Then, Tonin et al. [34] proposed a control method of motor‐imagery BCI to ensure robot safety by investigating inherent predictive nature and constraint handling capabilities to reduce the false‐positive rate. Besides, the RL, especially reinforce information from hierarchical critics [35], can also be another suitable method to propose a shared control approach to address the safety and performance issues of brain‐controlled robots. This work provides an enabling platform to enhance the future research and development of brain‐controlled robots.

5 Conclusions

In this study, the recent trends of ML and DL and relevant research areas of AI applications, including the CV, NLP, and robotic controls, are systematically reviewed for EEG‐based BCIs. Moreover, since the training process of current AI models cannot be fully emphasized, we assume that human cognitive computing from the EEG signals can provide natural insights to multimedia, semantic, and clinical applications. Also, applying various ML and DL models help us to learn reliable brain cortex features to understand hidden EEG feature representations. They also improve the EEG‐based BCI of research and industry impacts through relevant AI applications.

Footnotes

Conflict of interests

The authors declared that they have no conflict of interests.

References

Ramadan

Vasilakos

. Brain computer interface: control signals review. Neurocomputing. 2017, 223: 26–44.

Zander

Brönstrup

Lorenz

, et al.Towards BCI-based implicit control in human–computer interaction. In Advances in Physiological Computing. Human–Computer Interaction Series. Fairclough

Gilleade

, Eds. London, UK: Springer, 2014.

Aricò

Borghini

Di Flumeri

, et al. Passive BCI beyond the lab: current trends and future directions. Physiol Meas. 2018, 39(8): 08TR02.

Sree Shankar

Rai

. Human factors study on the usage of BCI headset for 3D CAD modeling. Comput - Aided Des. 2014, 54: 51–55.

Cao

Jolfaei

, et al. EEG-based brain–computer interfaces (BCIs): A survey of recent studies on signal sensing technologies and computational intelligence approaches and their applications. arXiv preprint. 2020: arXiv: 3021522.

Zhang

Yao

Wang

, et al. A survey on deep learning-based brain–computer interface: Recent advances and new frontiers. arXiv preprint. 2019: arXiv: 1905.04149.

Daly

Nasuto

Warwick

. Brain computer interface control via functional connectivity dynamics. Pattern Recognit. 2012, 45(6): 2123–2136.

Kasabov

. Evolving fuzzy neural networks for supervised/unsupervised online knowledge-based learning. IEEE Trans Syst Man Cybern B Cybern. 2001, 31(6): 902–918.

Azab

Toth

Mihaylova

, et al.A review on transfer learning approaches in brain–computer interface. In Signal Processing and Machine Learning for Brain-Machine Interfaces Tanaka

Arvaneh

, Eds. IET Digital Library, 2018.

10.

Kotsiantis

Zaharakis

Pintelas

. Machine learning: a review of classification and combining techniques. Artif Intell Rev. 2006, 26(3): 159–190.

11.

Lotte

Congedo

Lécuyer

, et al. A review of classification algorithms for EEG-based brain–computer interfaces. J Neural Eng. 2007, 4(2): R1–R13.

12.

Chiappa

Bengio

. HMM and IOHMM modeling of EEG rhythms for asynchronous BCI systems. In European Symposium on Artificial Neural Networks ESANN, Bruges, Belgium, 2004.

13.

Shimodaira

. Improving predictive inference under covariate shift by weighting the log-likelihood function. J Stat Plan Inference. 2000, 90(2): 227– 244.

14.

Wei

Lin

Wang

, et al. A subject-transfer framework for obviating interand intra-subject variability in EEG-based drowsiness detection. Neuroimage. 2018, 174: 407–419.

15.

Chiang

Wei

Nakanishi

, et al. Cross-subject transfer learning improves the practicality of real-world applications of brain–computer interfaces. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), San Francisco, USA, 2019, pp 424–427.

16.

. Transfer learning for brain–computer interfaces: a euclidean space data alignment approach. IEEE Trans Biomed Eng. 2020, 67(2): 399–410.

17.

Akinola

Wang

Shi

, et al. Accelerated robot learning via human brain signals. arXiv preprint. 2019: arXiv: 1910.00682.

18.

Wang

You

. MindGame: Mediating people’s EEG alpha band power through reinforcement learning. In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings, New York, USA, 2018, 5–6.

19.

LeCun

Ranzato

. Deep learning tutorial. In Tutorials in International Conference on Machine Learning (ICML13), Atlanta, USA, 2013.

20.

Cecotti

Gräser

. Convolutional neural networks for P300 detection with application to brain–computer interfaces. IEEE Trans Pattern Anal Mach Intell. 2011, 33(3): 433–445.

21.

Gao

Wang

Yang

, et al. EEG-based spatio–temporal convolutional neural network for driver fatigue evaluation. IEEE Trans Neural Networks Learn Syst. 2019, 30(9): 2755–2763.

22.

Yue

Wang

. EEG-based 3D visual fatigue evaluation using CNN. Electronics. 2019, 8(11): 1208.

23.

Patnaik

Moharkar

Chaudhari

. Deep RNN learning for EEG based functional brain state inference. In 2017 International Conference on Advances in Computing, Communication and Control (ICAC3), Mumbai, India, 2017, pp 1–6.

24.

Supratak

Dong

, et al. DeepSleepNet: a model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Trans Neural Syst Rehabil Eng. 2017, 25(11): 1998–2008.

25.

Hartmann

Schirrmeister

Ball

. EEG-GAN: Generative adversarial networks for EEG brain signals. arXiv preprint. 2018: arXiv: 1806.01875.

26.

Abdelfattah

Abdelrahman

Wang

. Augmenting the size of EEG datasets using generative adversarial networks. In 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil, 2018, pp 1–6.

27.

Luo

BL. EEG

data augmentation for emotion recognition using a conditional Wasserstein GAN. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, USA, 2018, pp 2535–2538.

28.

Spampinato

Palazzo

Kavasidis

, et al. Deep learning human mind for automated visual classification. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, 2017, pp 4503–4511.

29.

Palazzo

Spampinato

Kavasidis

, et al. Generative adversarial networks conditioned by brain signals. In 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017, pp 3430–3438.

30.

Davis

KM III

Kangassalo

Spapé

, et al. Brainsourcing: Crowdsourcing recognition tasks via collaborative brain–computer interfacing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, USA, 2020, pp 1–14.

31.

Hollenstein

De la Torre

Langer

, et al. CogniVal: A framework for cognitive word embedding evaluation. arXiv preprint. 2019: arXiv: 1909.09001.

32.

Makin

Moses

Chang

. Machine translation of cortical activity to text with an encoder-decoder framework. Nat Neurosci. 2020, 23(4): 575–582.

33.

Zhao

Duan

, et al. Human cooperative wheelchair with brain–machine interaction based on shared control strategy. IEEE/ASME Trans Mechatronics. 2017, 22(1): 185–195.

34.

Tonin

Bauer

del R Millán

. The role of the control framework for continuous teleoperation of a brain–machine interface-driven mobile robot. IEEE Trans Robotics. 2020, 36(1): 78–91.

35.

Cao

Lin

. Reinforcement learning from hierarchical critics. arXiv preprint. 2020: arXiv:1902. 03079.

A review of artificial intelligence for EEG‐based brain−computer interfaces and applications

Abstract

Keywords

1 Introduction

2 Machine learning for EEG‐based BCI

2.1 Overview

2.2 Supervised and unsupervised learning

2.3 Transfer learning

2.4 Reinforcement learning

3 Deep learning for EEG‐based BCI

3.1 Overview

3.2 Convolutional neural networks

3.3 Recurrent neural networks and long short‐term memory

3.4 Generative adversarial networks

4 EEG‐based BCI inspired by artificial intelligence applications

4.1 Computer vision

4.2 Natural language processing

4.3 Robotic controls

5 Conclusions

Footnotes

Conflict of interests

References