Thoughts on neurophysiological signal analysis and classification

Abstract

Neurophysiological signals are crucial intermediaries, through which brain activity can be quantitatively measured and brain mechanisms are able to be revealed. In particular, non‐invasive neurophysiological signals, such as electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI), are welcomed and frequently utilised in various studies since these signals can be non‐invasively recorded without harming the human brain while they convey abundant information pertaining to brain activity. The recorded neurophysiological signals are analysed to mine meaningful information for the understanding of brain mechanisms or are classified to distinguish different patterns (e.g., different cognitive states, brain diseases versus healthy controls). To date, remarkable progress has been made in both the analysis and classification of neurophysiological signals, but scholars are not feeling complacent. Consistent effort ought to be paid to advance the research of analysis and classification based on neurophysiological signals. In this paper, I express my thoughts regarding promising future directions in neurophysiological signal analysis and classification based on current developments and accomplishments. I will elucidate the thoughts after brief summaries of relevant backgrounds, accomplishments, and tendencies. According to my personal selection and preference, I mainly focus on brain connectivity, multidimensional array (tensor), multi‐modality, multiple task classification, deep learning, big data, and naturalistic experiment. Hopefully, my thoughts could give a little help to inspire new ideas and contribute to the research of the analysis and classification of neurophysiological signals in some way.

1 Background

The brain is one of the most complex organs in the human body and plays a critical role in our lives. It is significantly involved in generating cognition, experiencing complicated emotions, and coordinating precise behaviors. The outstanding capacity and function of the human brain enable human beings to differ from animals, which is the reason why human beings stand out among other creatures on the planet. As the brain is so special to us and yet is still mysterious, we are keen to understand it by diverse means. According to Gödel’s incompleteness theorem [1], we might infer that the brain may be so simple that its mental resources are not enough for it to understand itself, or the brain may be so complex that the brain is too complex to understand itself. However, this paradox might not apply to the human brain because the brain can create new things by combining several basic concepts or manipulating afferent information to generate new knowledge. For example, different combinations of the binary numbers 0 and 1 can be used to express any number on a computer. Similarly, basic elements or concepts might be combined in the brain to generate new abstracts.

Thanks to the advancement of neuroimaging techniques, brain activity can be recorded and represented with neuroimaging signals, which allows us to investigate brain mechanisms. The first electroencephalogram (EEG) recording from the human scalp was performed by Hans Berger in 1924 using a simple galvanometer [2], and since then, novel EEG recording devices have been developed to make EEG recordings more precise and higher in resolution. Later, neuroimaging techniques other than EEG were utilised to record brain activity, such as magnetic resonance imaging (MRI), which can produce images of the brain’s anatomy as well as oxygen consumption in tissues. Although recording principles for the acquisition of these signals are quite diverse, they all can reflect underlying brain activity and can be utilised to investigate brain mechanisms. The type of neuroimaging signal that is employed in a study generally depends on the study’s purpose and how well a neuroimaging signal meets the requirement of the study. For instance, EEG and functional MRI (fMRI) are frequently employed in cognitive studies because they capture temporal information that reflects the dynamic cognitive processes of the brain.

Neuroimaging signals represent the underlying activity of the brain. However, brain activity cannot be observed visually by merely looking at neuroimaging signals since the content that we usually intend to observe is masked by strong background noise or is contaminated by unwanted signals or surrounding interference. Therefore, the signals are usually pre‐processed to remove the effects of the above factors before further processing, namely artifacts removal. This initial processing is followed by feature extraction. The extracted features (or biomarkers) can be analysed using statistical methods or fed into a machine learning model for classification (also known as identification or recognition). Both data analysis and classification require good features that intrinsically stand for investigatory ones in order to accomplish brain mechanism discovery or brain pattern differentiation. Data analysis is commonly performed before data classification. Data analysis aims to explore recorded data from different views to understand brain activity and then reveal brain mechanisms behind the activity. When a brain mechanism is revealed, or differences between brain patterns are found, a machine learning model can learn based on these findings so that the trained model will be able to classify new data automatically.

In the past, numerous features derived from the temporal domain (e.g., amplitude, mean, peak), spectral domain (e.g., power spectral density, phase), and spatial domain (e.g., interactions between brain regions) have been used in data analysis. A lot of methods have been proposed to process data. Similarly, a variety of models have been developed for data classification, ranging from simple models (e.g., k‐Nearest Neighbours) to complex models (e.g., deep learning). These methods have helped us understand a lot about the brain and enable successful diagnosis of brain diseases, which are associated with the success of data analysis and classification. Although considerable accomplishments have been attained, there are still many aspects of the brain that we do not understand. Continuous effort is required to reveal brain mechanisms and increase our understanding of the brain. As we progress further in the journey of brain investigation, more difficult problems will emerge. New views and novel methods are required to produce additional successes in the field regarding brain investigation in the next decade. In this paper, I would like to express a few thoughts about promising potential directions in data analysis and classification. I will focus on the following topics: brain connectivity, multidimensional array (tensor), multi‐modality, multiple task classification, deep learning, big data, and naturalistic experiment (see Fig. 1). Some of these topics have been exhibiting an obvious trend, to which research concentration is being shifted from the past, while the others seem to appear but are unclear.

Fig. 1.

Overview of promising developments in the neurophysiological signal analysis and classification. The topics discussed in this paper are shown in the right panel, which are brain connectivity, multidimensional array (tensor), multi‐modality, multiple task classification, deep learning, big data, and naturalistic experiment.

The remainder of the paper is organised as follows. Each of the above topics is expanded to detail my thoughts in the following separate sections. These sections are followed by a conclusion, where the key information of the paper is recalled, and the relationships among the topics are briefly addressed. Finally, conflict of interests, financial support, and acknowledgements are stated.

2 Brain connectivity

Since neuroimaging techniques were utilised to record brain activity, the brain was considered as an organ, of which each region is structurally integrated and functionally independent. This concept is based on the observations of early experiments, where a brain region is predominantly activated by a particular task/stimulus [see the illustrations of the features extracted from individual brain regions in Fig. 2(A)]. Within this context, many specialised brain regions have been determined. For example, the occipital region is devoted to visual information processing, and the motor region is in charge of movement coordination. The parietal region, which is located between the occipital region and motor region, plays an intermediate role in information transmission. These regions can be further divided into smaller regions, which perform subordinate functions. For instance, the occipital region is divided into V1, V2, V3, and V4, through which visual information is processed and becomes more abstract. The above examples appear to demonstrate that brain regions are functionally independent and separately oversee specialised processing. Brain regions likely support each other and perform cooperatively, although brain regional cooperation might not be observed since the cooperation is minuscule compared to dominant regional activation. Cooperation can generally be observed when viewing on a larger scale (e.g., brain hemisphere); for example, large‐scale cooperation is usually visible during a cognitive task that involves several brain regions.

Fig. 2.

(A) Information extracted from individual brain regions. Typical extracted features are shown on the right. Feature examples are shown above the text. (B) Connectivity within the brain and between the brains. Typical methods of connective strength estimation are listed on the right.

Due to the nature of brain interregional cooperation in cognitive tasks, more and more studies investigate interactions between brain regions and attempt to reveal how brain regions work together. With the help of brain connectivity analysis, brain interregional connective patterns concerning different brain states/statuses are revealed. Rissman et al. explored functional connectivity to reveal that a greater connection between brain hemispheres in the motor area was observed during sequence-tapping task [3]. Functional connectivity could also be changed from one mental state to another [4 –7] or switched from one condition to another condition [8, 9]. It is believed that functional connectivity is relevant to numerous types of cognitive processing [10 –13] and emotion perception [14]. In studies involving patients, functional connectivity was also found to be closely related to a range of brain diseases such as Alzheimer’s disease [15], Parkinson’s disease [16], depression [17], and schizophrenia [18, 19]. In general, brain disease disrupts functional connectivity or changes connective strengths between regions. For instance, schizophrenia leads to wide disconnectivity, especially between brain hemispheres [20]. Dysconnectivity, or the disruption of functional connectivity, results in the alteration of brain organization toward inefficiency from a “smallworld”optimal organisation (small‐world organisation is an efficient network, in which information can be quickly transported from one region to another via short paths). Moreover, graph metrics such as shortest path length are also used to depict topological properties of brain organisation. Additional detailed information about these metrics can be found in ref. [21], which contains definitions and interpretations of each frequently used metric.

In studies concerning neurophysiological signal classification, the features of brain connectivity are increasingly utilised (a recent review can be found in ref. [22]). Those features represent interactions or relationships between brain regions and contribute to the discrimination of mental states [23, 24] and the detection of brain diseases [20, 25, 26]. Changes in brain state would accompany alterations in functional connectivity. The information contained in functional connectivity is believed to be supplementary to the information derived from individual brain regions. Incorporating functional connectivity features into the feature pool improves classification performance. This point has been confirmed in both movement category classification [27] and mental fatigue identification [28]. Furthermore, the features extracted based on dynamic functional connectivity could provide additional discriminative information for classification because dynamic functional connectivity reflects the underlying changes of brain activity over time. Such temporal information might not be contained in static functional connectivity because the temporal information could be canceled out when collapsing connectivity across time in connective strength estimations. It has been observed that dynamic functional connectivity is significantly different between brain states [11, 29, 30]. Through the use of the features derived from dynamic functional connectivity, traumatic brain injury has been successfully classified [31]. To date, dynamic functional connectivity is a straightforward extension of static functional connectivity. A long signal that is used in the strength estimation of static functional connectivity is divided into smaller segments, and each segment has its strength estimated independently. Then, dynamic functional connectivity is attained based on these strengths. Currently, the division is arbitrary. A division coinciding with underlying neural processing is preferable so that dynamic connectivity can more accurately reflect the pace of brain activity. Alternatively, a brand‐new method that estimates brain connectivity without signal division is desired.

In addition to ordinary functional connectivity (namely, low‐order functional connectivity), other kinds of functional connectivity emerged to provide additional tools for neurophysiological signal analysis and classification. The study of mild cognitive impairment detection demonstrated that detection performance significantly improved when high‐order functional connectivity features were used [32, 33]. High‐order functional connectivity was extended to between‐frequency high‐order functional connectivity in a drowsiness assessment study [4]. In this study, they also proposed a global metric to quantify the aggregated effect of functional connectivity. These new types of functional connectivity could be used to extract high‐level connectivity patterns to enrich data analysis and classification beyond ordinary functional connectivity. High‐order functional connectivity is a measure of similarity between topographical profiles of brain regions, while associated high‐order functional connectivity shows the similarity between topographical profiles and high‐order topographical profiles (see Fig. 2 in ref. [8] for the illustrations and explanations). All of these types of functional connectivity can be extended to dynamic types by taking temporal information into account when estimating connective strengths.

New methods of estimating connective strengths will be developed either to improve estimation precision or to meet emerging requirements. Finer data analysis and classification also require new methods to acquire connectivity information from brand-new angles. I believe these imminent methods will expand the current toolkit to empower us to perform more comprehensive connectivity-based analysis and facilitate classification using connectivity features. Another trend is to continue to broaden the usage of connectivity. This would not be restricted to brain‐related studies and could also expand into the

investigations of other organs such as the heart. Nowadays, hyperscanning is an attractive technique to simultaneously record neurophysiological signals from two or more persons, by which cooperation or competition between persons could also be investigated [34, 35]. Functional connectivity is also applicable to more than one brain, thus inter‐brain connectivity could also be investigated [See Fig. 2(B)]. In a hyperscanning study of emotion perception, significantly different connections between positive emotion and negative emotion were observed not only within each brain but also between the brains [36]. Currently, the amount of studies investigating between‐brain connectivity is very limited. I believe these types of studies will dramatically increase eventually because cognitive investigation should be performed in the context of multi‐person interactions, since we are living in a society. Two potential limitations may be the lack of techniques supporting the setup of such experiments and impractical experimental control for excluding confounding factors to achieve the required experimental purposes.

3 Multidimensional array (tensor)

In the past, features were usually arranged as vectors, one feature by another feature, which were further refined or fed into a classifier. For example, features extracted from each EEG channel or MRI voxel are concatenated into a vector without consideration of channel order or voxel position. In this case, the relationships between channels or voxels could be destroyed, resulting in a loss of potential discriminative information for classification. A promising solution is to arrange features as a multidimensional array (tensor) [37] so that the relationship information between channels or voxels can be better preserved (see Fig. 3 for the illustrations of tensors and their decomposition).

Fig. 3.

(A) Prevalent third‐order tensor, fourth‐order tensor, and a general nth‐order tensor. (B) A high‐order tensor is decomposed into components. This figure was adapted from Fig. 1 in ref. [43] with the permission of IEEE.

In a study on stroke recovery training [38], EEG data were transformed into a time‐frequency representation of the signal and were then formed into a third‐order tensor based on which rehabilitation course was tracked. To date, a number of the published papers demonstrated that tensor representation had an effective and useful role in the EEG classification [39 –42]. In these studies, tensor representation was decomposed into components to extract discriminative features. An additional classifier is then required to complete classification. However, this is not always the case. The separate classification step can be merged into tensor decomposition. Therefore, EEG decomposition and classification can be achieved at a time without the involvement of a classifier, as shown in ref. [43].

Overall, tensor has not been extensively applied to neurophysiological signals. The vast majority of the existing studies that employ tensor for data analysis and classification are applications of tensor theory without adaption according to the characteristics of neurophysiological signals. Novel tensor algorithms will be developed to meet the application requirements of neurophysiological signals in the future. Moreover, higher‐order tensor decomposition will be applied to neurophysiological signals beyond the currently prevalent third-order tensor and fourth‐order tensor. When using a higher‐order tensor, more information could be incorporated into decomposition, which could benefit data analysis and classification or expedite data processing.

4 Multi‐modality

Brain activity can be recorded using a few different techniques. These techniques are of their own strengths and some of their strengths are complementary. Fig. 4 depicts EEG recording, MRI recording, and concurrent EEG/MRI recording. EEG has high temporal resolution while MRI has high spatial resolution. These two modalities (i.e., EEG and MRI) can be simultaneously recorded to gain high temporal and spatial resolution (a relevant review can be found in ref. [44]). Full information can be acquired by taking advantages of multimodal recordings. Although the use of multiple modalities brings benefits, it also raises new challenges. One challenge is the interference that originates from the respective modalities. While two or more modalities are simultaneously recording, a modality could adversely influence other modalities by introducing noise thereby degrading the quality of recorded signals. To remove this noise, noise removal methodologies have been developed. For concurrent EEG/fMRI recording, the EEG recording produces a negligible effect on the fMRI but the fMRI recording significantly influences the EEG and introduces gradient artifacts into the EEG. Average artifact subtraction (AAS) [45], optimal basis set (OBS) [46], and canonical correlation analysis‐based method [47] were proposed to remove gradient artifacts from EEG signals. Another challenge is accurate alignment of the timing across modalities. This is essential to effectively combine different modalities to enhance data analysis and classification. This is particularly important for the investigation of cognition because time‐locked exploration is very prevalent in revealing cognitive mechanisms concerning outside stimuli or internal particular processing.

Fig. 4.

(A) Single EEG recording using a multi‐channel cap and a signal amplifier. (B) Single MRI recording using an MRI scanner. (C) Concurrent EEG/MRI recording using both types of signal equipment.

In a study on assessing brain involvement during focal epileptic activity [48], EEG was utilised to capture pathological temporal characteristics (e.g., epileptic spike) and fMRI was used to help localise generators corresponding to such pathological temporal characteristics. This instance is a good example of how EEG could be utilised to inform fMRI analysis. In contrast, fMRI can be utilised to inform the EEG for source estimation [49]. Both of the above manners integrate EEG and fMRI by relying on one modality to guide the analysis of the other modality. Alternatively, modalities could be treated equally to explore correlations between modalities [50]. In addition to the fact that multiple modalities measure the same neural activity, each modality could also measure completely different physiological activities. For instance, EEG was used to measure brain activity and EMG was used to measure muscular activity. The correlations between these two modalities were examined to reveal the relationship between the brain and lower limbs during the process of walking [51]. In the future, different modalities respectively measuring diverse physiological activities will be employed to investigate cognition, emotion, and behavior to a more extensive degree. This could facilitate studies that aim to understand relationships between organs or interactions between different parts of the human body.

5 Multiple task classification

Usually, different cognitive levels (e.g., workload levels) or brain statuses (e.g., diseased versus healthy) within the same context are classified. It is called single task classification (see Fig. 5). The single task classification can be divided into two categories: within‐subject classification and cross‐subject classification. In the within-subject classification, data recorded from each subject (participant) are split into the training set and testing set. A classifier trained using the data from a subject applied to classify the data in the testing set of the same subject. In the case of cross‐subject classification, a trained classifier is used to classify the data of another subject whose data are not used for training the classifier. Generally speaking, single task classification is relatively easier compared to multi‐task classification. Herein, multi‐task does not refer to several tasks, where each task only has one level/condition. It refers to several tasks and each of them has two or more levels/conditions. These tasks could be either analogous or dissimilar (heterogeneous). For example, mental workload and mental fatigue are analogous, while mental workload and motor movement are dissimilar. According to the data division, the multi‐task classification comprises of task‐mixed classification and cross‐task classification. Task‐mixed classification means that training data consist of the data from all tasks. If a classifier trained using the data from a task is used to classify the data of other tasks, it called cross‐task classification. Similar to the taxonomy in the single task classification, multi‐task classification can also be categorised into within‐subject classification and cross‐subject classification (see Fig. 5).

Fig. 5.

Taxonomy of classification problems. Blue and red colors represent training data and testing data, respectively. Letters and roman numerals are used to label subjects.

According to the taxonomy depicted in Fig. 5, no study of the multi‐task classification has been done until now, as far as I know. Multitask classification might receive increasing attention because this is meaningful to practical applications based on neurophysiological signals. With the multi‐task classification, a general classifier is capable of the discrimination of different tasks or detection of multiple diseases, which has to be achieved by respective classifiers trained for each task or each disease. In other words, neurophysiological signal recorded at one time could be used to understand diverse aspects of the human body. For example, mental fatigue level and mental workload level are simultaneously tracked based on EEG by a classifier.

In the future, more and more offline classifications will be converted to online classifications (namely, real‐time classification). Meanwhile, techniques solving emerging challenges in the development of online classification will be invented, such as portable signal acquisition equipment and speedy signal processing methods. In order to achieve better performance of real‐time classification, a classifier should be updated to learn the information from the latest data. Classifier adaption algorithms will be developed to meet this new requirement. I think this point is one of the promising directions in the research of the real‐time classification.

6 Deep learning and big data

Deep learning is a revival of the conventional neural network, which suffers from some drawbacks (e.g., local minima). With the advancements in novel network architectures (e.g., long short‐term memory architecture [52]) and training strategies (e.g., batch normalization [53]), the neural network is upgraded to deep learning while eliminating its initial drawbacks. Nowadays, deep learning is almost overwhelmingly involved in data recognition due to its excellent performance in a variety of classification problems [54], including image object recognition, speech recognition, as well as EEG and fMRI data classification.

As the number of layers of a deep learning model increases, the learning capacity of the model increases so that more information can be learnt by the model. Massive learning capacity might be one of the prerequisites to accomplish the aforementioned multi‐task classification. This is because the volume of information a model must learn for multi‐task classification is much larger than what is needed for single‐task classification. Deep learning models have numerous weights to empower the learning of the large volume of information, which appears impossible to do with a simple conventional model such as support vector machine. Numerous weights empower the learning capacity of a deep learning model, but can also lead to training complications. As we know, deep learning training is very time‐consuming. This shortcoming could be resolved by using high-performance computers. An alternative solution is a pre‐training strategy, which has been proposed in the processing of image and speech data [55]. The real challenge is the lack of definite training rules, by which a good model can be guaranteed. Currently, we have to tune model parameters blindly and cannot predict whether a parameter setting would result in a good model. This parameter tuning obstacle becomes even more severe when a deep learning model contains complicated architecture (e.g., a high number of layers and connections between neurons within the same layer). Further research addressing this difficulty should be conducted in the future.

A great deal of data is usually required to train a deep learning model being good performance. This is associated with the concept of big data. As neurophysiological signals can be recorded more easily and portably than ever, more and more data can be gathered, forming a large volume of data. The increase in data size is also related to the topics I mentioned above. The use of multiple modalities and the involvement of multi‐task undoubtedly lead to an increase in data size. Large data size enhances the probability of the use of tensor arrangements.

7 Naturalistic experiment

Most experiments are conducted in laboratory-based environments. The experiments are strictly controlled to ensure the exclusion of as many confounding factors as possible. Although strict control of experiments benefits and facilitates these studies, the assumptions in the experiments are not always accurate. In this case, the observations derived from the experiments could be distorted and might significantly differ from the real phenomena that should occur. For instance, emotional images or videos are frequently presented to subjects in order to induce emotional responses. The assumption underpinning such an experiment is that these images or videos can successfully induce the same emotion as shown in the images or videos. However, this assumption is not always plausible. The emotional response of a subject is influenced by several factors, including baseline emotional status, and cannot be uniquely determined by emotional stimuli. Therefore, it would be better to conduct experiments in a naturalistic environment. Ideally, the experiment environment should be as identical to the real scenario as possible. For example, a walking experiment carried out in a real corridor would be better than that one carried out on a treadmill [56].

In the future, naturalistic experiments should extensively replace laboratory‐based experiments. With the emergence of portable recording equipment and relevant technological advancement, producing naturalistic experiments will become more and more feasible. Undoubtedly, many challenges will arise during the implementation of naturalistic experiments. For instance, there may be some difficulties in synchronizing multiple signal streams in a mobile manner. However, the advantages gained from these naturalistic experiments will be worth the cost paid to solve these challenges.

8 Conclusions

In this paper, I discussed and expressed my thoughts on the topics of brain connectivity, multidimensional array (tensor), multi‐modality, multiple task classification, deep learning, big data, and naturalistic experiment. In my opinion, these topics are both crucial and promising for the future advancement of neurophysiological signal analysis and classification. The topics are separately discussed, but they are inherently pertinent. Brain connectivity enhances data exploration from a different perspective within the same data modality, and multiple modalities provide more data from different angles. These are devoted to multiple task classification and make it more feasible since big data (a huge amount of data in different modalities) and more feature representations will benefit multiple task classification. For such a complex classification, a model with high learning capacity (e.g., deep learning) is required to learn based on massive amounts of data. More flexible deep learning models are expected in multiple task classification. Decisive outcomes (also known as classification outputs) are not necessarily made at the last layer of a deep learning model. Some of them could be generated in the middle layers if sufficient information has been available for the classification decision. I believe that a more flexible deep learning model is key for complex multi‐task classification. Finally, I discussed naturalistic experiment, which ensures quality data for neuroscience investigations and engineering applications.