Sage Journals: Discover world-class research

Abstract

This article describes a methodology to recognize emotional states through an electroencephalography signals analysis, developed with the premise of reducing the computational burden that is associated with it, implementing a strategy that reduces the amount of data that must be processed by establishing a relationship between electrodes and Brodmann regions, so as to discard electrodes that do not provide relevant information to the identification process. Also some design suggestions to carry out a pattern recognition process by low computational complexity neural networks and support vector machines are presented, which obtain up to a 90.2% mean recognition rate.

Keywords

EEG affective computing emotions neural networks support vector machines Brodmann regions arousal valence

Introduction

The recognition of emotional states through a computer has been a widely studied topic in affective computing, since Rosalind W Picard proposed the basis to analyze the physiological responses of an emotion;¹ however, most of these studies have been focused on analyzing the peripheral reactions produced by emotions, such as vocal or facial expression, and some researchers suggest that these responses could be manipulated by common users and they suggest the implementation of bio-medical signals as more reliable data sources, according to the William James theory:² emotional states could produce disturbances in one or more of the basic human functions, such as changes in heart rate, muscle responses or temperature changes, when a person is facing a real or an imaginary emotional stimulus. There are many bio-medical signals that could be evaluated to recognize emotional states, such as heart rate or body temperature variations; however, our interest is focused on the analysis of signals that were originated by brain activity, particularly electroencephalography (EEG) due to the fact that this technique does not require extensive technical knowledge to implement and implementation costs are relatively low.

Related work

Some of the most outstanding work in emotion recognition by an EEG signals analysis, was presented by Dr M Murugappan of Perlis University in Malaysia and by Dr Sander Koelstra at Queen Mary University of London in the UK. Dr Murugappan proposed a mathematical model to infer emotional states from EEG analysis and Dr Koelstra developed the DEAP database, which is an extensive collection of physiological signal records of emotional stimulation processes, and both works demonstrates the feasibility of the establishment of a relationship between the electrical activity in brain cortex and emotional states.^3
–5 A summary of related work is presented in Table 1.

Table 1.

Related work in classification and identification techniques of emotional states and associated identification rate (IR).

Technique	IR (%)	Reference
KNN, LDA	81.25	Murugappan (2011)⁴
MPC	64.00	Yuan et al. (2007)⁶
KNN	82.27	Heraz at al. (2007)⁷
SVM	66.70	Takahashi (2004)⁸
SVM	93.50	Li and Lu (2009)⁹
SVM	77.80	Rozgic et al. (2012)¹⁰
NN	43.14	Attabi and Dumouchel (2012)¹¹
NN	60.00	Razak et al. (2005)¹²
NN	93.30	Gunler et al. (2012)¹³

KNN: k-Nearest Neighbors algorithm; LDA: Linear discriminant analysis; MPC: Multi-way Polarity Classification; SVM: support vector machines; NN: neural networks.

Emotions

Disambiguation

Words such as affect, feeling and emotion are commonly used synonymously; however, each of them has a very different meaning, in their etymology and in the physical and mental reactions they cause.^14
-16

Emotions are the manifested reactions to those affective conditions that due to their intensity, move us to some kind of action with slight or intense, concomitant or subsequent, repercussions upon several organs, that can set up partial or total blocking of logical reasoning.

Affect could be defined as a grouping of psychic phenomena manifesting under the form of emotions, feelings or passions, always followed by impressions of pleasure or pain, satisfaction or discontentment, like, dislike, joy or sorrow.^{17, ii}

Feelings are seen as affective states with a longer duration, causing less intensive experiences, with fewer repercussions upon organic functions and lesser interference on reasoning and behavior.ⁱⁱⁱ

Emotion theories

Over centuries, philosophers, physicians and psychologists have studied affectivity phenomena by questioning their origin, their role upon psychic life, their action with regards to favoring or hindering adaptation and their neuro-physiological concomitants;¹⁸ however as established by Scherer (Scherer (2005)) “even the simple question, of what emotions are?, hardly get the same answer”.¹⁹

There are classical and antagonistic theories of emotions:

Classical theories are supported by the Darwinian theory of emotions, which states that affective reactions are innate patterns designed to orient behavior and promote adaptation,^18,20,21 and by recent theories which suggest that emotions are complex phenomena initiated by a central process as result of internal or external causes, that can be observed as an organismic alteration.^19,22
-29

Antagonistic theories are led by the Claparede theory,¹⁴ that defines emotions as useless, non-adaptive and harmful phenomena, since according to this theory emotions are characterized by a sudden disruption of an affective balance (mostly for short episodes), with slight or intense, concomitant or subsequent, repercussions upon several organs, that can set up partial or total blocking of logical reasoning in the affected subject.

It is the definition provided by Scherer (Scherer (2005)), which best fits an engineering task, defining an emotion as: an episode of interrelated and synchronized changes in most of the organismic subsystems(note call 4 and 5), in response to the evaluation of an external or internal stimulus event.

Data sources

The lack of a standardized database is one of the main problems with carrying out a study about the physiological responses of an emotional state, since standardized databases such as the International Affective Picture System (IAPS) or the International Affective Digitized Sound system (IADS) can be only implemented to perform the stimuli processes, which implies that if two different research groups perform an experimental setup that associates the same stimulus and even the same participants, the results will vary due the environmental conditions.^30,31

However, there are some data collections such as bu-3DFE, PhysioNet, DEAP and the Ibug project that can help with this problem.

bu-3DFE is a collection of several physiological signal records, associated with a wide range of emotional expressions.³²

PhysioNet is a collection of physiological signals, time series and images, constructed to perform a behavior analysis.³³

The Ibug project is a collection of bio-metric data associated with affective behaviors.³⁴

DEAP (Database for Emotion Analysis using Physiological Signals) is a wide collection of biosignals generated by several specialized experimental setups under arousal and valence stimuli.⁵

Database for emotion analysis using physiological signals

is a large collection of physiological signals which are directly associated to an emotional stimulus in a multi-modal dataset for the analysis of human affective states, where the EEG and other peripheral physiological signals of 32 participants were recorded as each watched 40 one-minute long excerpts of music videos. Participants also rated each video in terms of the levels of arousal, valence, like/dislike, dominance and familiarity they experienced.⁵

This is the database that we implemented for the experimental process presented in this paper, because it is to our best knowledge the most comprehensive and reliable source of data.

Data bounding methodology

This work proposes a methodology that reduces the amount of the processed data, by defining a model that establishes a correlation criteria between emotional activity and it responses in the brain cortex and excludes regions that could not provide significant performance to the recognition process.

The Brodmann regions

Dr Korbinian Brodmann sub-divided the brain cortex into regions that appeared to have micro-structural differences and associated these regions with specific cognitive functions, such as motor processing, speech, hearing or sight. Since our work is focused on analyzing an emotional process evoked by audio-visual stimuli, we are proposing that only the electrodes that are strongly related to the audition, visual, sensory and motor^vi regions^vii should be considered for the digital signal processing task (see Table 2).^18,35
–40

Table 2.

Regions of the cortex with its associations with the cerebral cortex and its relationship with Brodmann regions.

Processes	Brodmann regions	Cortex regions
Visual	17, 18, 19, 20, 21, 37	Temporal lobe
		Occipital lobe
Audition	22, 41, 42	Temporal lobe
Sensory	1, 2, 3, 4, 5, 7, 22, 37, 39, 40	Parietal lobe
Motor	4, 6, 44, 9, 10, 11, 45, 46, 47	Temporal lobe
		Frontal lobe

• The audition processes are related to electrodes T7, T8, F7, F8, P7 and P8.

• The visual processes are related to electrodes O1 and O2.

• The sensory processes are related to electrodes CP5, CP1, CP2 and CP6.

• The motor processes are related to electrodes FC1, FCz and FC.

Selected electrodes

Only 15 electrodes were set as active elements, while 22 were setted as non active elements as can be observed in Figure 1 and Table 3. This provides a data reduction of 11,264 samples per second (considering a nominal sampling rate of 512 Hz), which is a significant data reduction and consequently also a computational burden reduction.^viii

Table 3.

Electrodes associated with the Brodmann regions.

Associated regions	Electrodes
Frontal Temporal 7	F7
Frontal Cortex 5	FC5
Frontal Cortex 5	C5
Frontal Cortex 6	C6
Frontal Temporal 8	F8
Parietal Cortex 5	CP5
Parietal Occipital 4	P4
Parietal Occipital 8	P8
Parietal Occipital 7	P7
Parietal Occipital 3	P3
Parietal Cortex 6	CP6
Occipital 1	O1
Occipital 2	O2
Temporal 7	T7
Temporal 8	T8

Figure 1.

Graphical representation of the active and non-active electrodes (green electrodes are considered as active electrodes and black marked as non-active electrodes).

Signal conditioning

A very wide variety of phenomena could affect the performance of the analysis of physiological data, such as a wide variety of noise, or the large amount of resources required to process these types of signals.

Noise filtering

The Laplacian filter described by Murugappan (equation (1)), was implemented to mitigate the problem that EEG signals are naturally contaminated with noise and artifacts (i.e. eye movement(EOG), muscular movement (EMG), vascular movements (ECG) and kinetic artifacts)⁴

x_{new} = [x (t) - 1 / N_{E}] [\sum_{i = 1}^{N_{E}} x_{i} (t)]

(1)

where $x_{new}$ is the filtered signal, $X_{t}$ the raw signals and $N_{E}$ is the number of neighbor electrodes.

Signal bounding

A band-pass filter with cutoff frequencies of 0.5 Hz to 47 Hz, was implemented to exclude all frequencies that are not associated with the brain rhythms model: delta (0.2 to 3.5 Hz), theta (3.5 to 7.5 Hz), alpha (7.5 to 13 Hz), beta (13 to 28 Hz) and gamma ( $>$ 28 Hz).^41,42

Blind source separation

A blind source separation (BSS) algorithm was implemented to remove redundancy between active elements but preserve information of non-active elements. Since these cases are generally considered as a multi-channeling problem and the signal $y (n)$ , its components $y_{i} (n)$ , could be defined as:

p_{Y} (y (n)) = \prod_{i = 1}^{m} p_{y} (y i (n)) \forall n

(2)

where $p_{(Y)}$ is the probability distribution set, $p_{y} (y i (n))$ is the marginal distribution and $m$ refers to the predefined independent components, to separate each element of our gating region as shown in Figure 2, where $S_{i} = e_{i} + \sum_{m = 1}^{n} r_{m}$ , it is the sum of overlapped signals that occur when you try to read an specific electrode and $S_{b i} = e_{i} + λ_{i}$ represents the information of a specific an electrode without redundancy of the active electrodes.

Figure 2.

Process model of the blind source separation by independent component analysis implemented.

Feature extraction

A common question about the implementation of the wavelet transformation is ‘Why does it not use Fourier traditional methods?’. The answer is that there are two important differences between Fourier and wavelet analysis.

The first is that due to the Fourier basis, functions are localized in frequency but not in time (a small frequency change in the Fourier transform could produce changes in all parts in the time domain), unlike the wavelet transform which presents resolutions in frequency (through expansion) and in time (through translations).

The second is that many kinds of functions that can be represented by wavelets in a more compact form (i.e. functions with discontinuities and features with sharp spikes usually need fewer functions when they are analyzed on a methodology based on wavelets) and due to this, large data sets can be easily and quickly transformed by the discrete wavelet transform (the counterpart of the discrete Fourier transform) by encoding the data as wavelet coefficients, which implies a higher processing speed since the computational complexity of the fast Fourier transform is $O (n Â \cdot l o g 2 (n))$ , while for the wavelet transform this is reduced to $O (n)$ .⁴³

Feature selection

Each level of the discrete wavelet transform is calculated by passing the signal through a series of filters; samples are passed through a low pass filter $g$ and simultaneously a high pass filter $h$ , as^ix

y [n] = (x * g) [n] = \sum_{k = - \infty}^{\infty} x [k] g [n - k]

(3)

Also the energy content in a specified region of the signal $f (t)$ , can be decomposed as

f (t) = \sum_{j} \sum_{k} d_{j, k} ψ_{j, k} (t) = \sum_{j} f_{j} (t)

(4)

where $j$ , $k \in Z$ y $ψ (t)$ is the mother wavelet and the coefficients $d_{j, k}$ represent the inner product (equation (5))

d_{j, k} = 〈 f (t), ψ_{j, k} (t) 〉 = \frac{1}{\sqrt{2^{j}}} \int f (t) ψ {(2}^{- j} t - k) d t

(5)

$d_{j, k}$ represents the energy of the detail coefficients^x from $f$ at level $j$ , and the classification input array can be obtained by

E_{j} = \sum_{k} d_{j, k}^{2}

(6)

This same procedure can be implemented to extract the approximation coefficients.

The total coefficients energy can be expressed as

E_{t} = \sum_{i = 1}^{e} E_{j}

(7)

where $e$ , are the electrodes associated with the wavelet analysis and $j$ is the corresponding energy percentile level calculated as

ε_{j} = \frac{E_{j}}{E_{t}} \times 100

(8)

The translation and dilation coefficients, can be implemented directly as the features in the classification problem.

Classes

Our proposal to establish an appropriate model that establishes a clear distance between each emotional tag, is the assignment of a distance parameter between the emotional tags according to the Ekman, Russell and Scherer models. Since the Ekman model provides the basic tags to identify each class, while the Russell and Sceherer models define emotional states as arousal and valence levels.

This model can be observed in Figure 3, which encompasses all similar emotional states according to their arousal and valence levels (i.e. all emotions that have high levels of arousal and high levels of valence are correlated to states of happiness, states which have a low valence and high arousal are correlate to anger, low arousal and low valence to sadness, and low arousal and high valence to relaxation states).

Figure 3.

Emotional states distribution model, based on arousal and valence levels and the discrete tags of the Ekman model.

Class 1: (HA-HV) high arousal, high valence.

Class 2: (HA-LV) high arousal, low valence.

Class 3: (LA-HV) low arousal, high valence.

Class 4: (LA-LV) low arousal, low valence.

Also this model defines the orientation of each of the evoked potentials according to their characteristics to define the elements of each of the classes that would be implemented as references in the identification model, as can be observed in Figures 4 and 5, where each element of the experimental process belongs to a particular class.

Figure 4.

The location of the experimental cases, according to their emotional stimuli responses obtained by the user ratings (the input elements to the classification process are created based on this information). Arousal: it includes features that define idle or alert states (i.e. disinterested, boring, alert, excited). Valence: ranges from unpleasant (i.e. sad, stressed) and nice (i.e. happy, euphoric). Domination: ranging from a feeling of helplessness and weak (uncontrolled) to one feeling empowered (in control of everything).

Figure 5.

Distribution of evoked potential trials in an arousal and valence space. As seen there are ten experiments related to each class and a greater emphasis are given to three of them in relation to its distance to the other cases to ensure a greater separability between classes, according to their labeling.

Identification process

Support vector machines and neural networks, are the techniques that according to literature review have shown the best recognition rates (Table 1). Therefore, these two techniques were implemented in this work in order to corroborate the performance of our methodology.

Experimental setup and inputs

The experimental configuration presented in Figure 6, was designed to evaluate the identification performance of each of the combinations produced by implementing the clustering Algorithm 8.1, which ensures a consistent experimental process distribution for each of the elements of the classes by considering at least one experimental class associated with each class (i.e. each of the cross validations evaluate distinct cases of the same class).^xi

Figure 6.

Class selection model to perform the identification performance task.

Algorithm 1: Feature input arrangement algorithm.
Input: Features $d_{j, k}$ arrays
Output: Matrix arrangement $E_{k}$
Selecting features to provide input classes;
for $i \leftarrow 1$ to $3$ do
e= electrodes array);
for $j \leftarrow 1$ to $15$ do
$C = [e (b); d_{j, k}]$
‘a’ is the parameter that defines the number of emotional states that would be analyzed;
for $k$ to $a$ do
for $l \leftarrow 1$ to $30$ do
$E_{k} = [l; C]$ ;
$E_{k}$ is the arrangement of features of the three case studies of each emotional state and all users

8.1.1 Classification inputs

To ensure that each case study contains the information from more than one user at the same time and to corroborate the existence of a correlation between different study subjects, each case is also assessed by means of the following classification scheme:

a: Contains the information of a single user.

b: Contains information of two users.

c: Contains information of three users.

d: Contains information for all users.

Neural networks

Artificial neural networks (NN) are computational techniques that can be trained to find solutions, recognize patterns, classify data or forecast events by defining the way its individual computing elements are connected, which automatically adjusts their configurations to solve specific problems according to a specified learning rule. Due to the fact that EEG data are considered as chaotic signals, many researchers have proposed the implementation of NN, as one of the most appropriate tools to carry out the analysis, since NN have a remarkable ability to derive meaning from complicated or imprecise data. Besides NN are inspired by the natural behavior of a neuron, and EEG signals are signals produced by neurons.

Implementation

The implementation of a highly complex NN was our initial proposal to carry out the pattern recognition task; however, as can be observed in Figure 7, the error rates have a direct correlation to the network architecture (i.e. if network architecture increases, so to does the error), contrary to our initial thought and that the implementation of low complexity architecture will be the best option for this particular case, as shown in Figure 8, where a 10-fold cross-validation was performed to evaluate 20 configurations of low complexity networks, in order to determine the amount of units per layer that would be necessary to obtain the most accurate and stable recognition rate.

Figure 7.

An identification process performed by the same NN configuration, but incrementing by one the number of layers in each iteration up to 50.

Figure 8.

Performance of low complexity NN configurations. Considering only NN with less than 20 hidden layers, the validation process chose the number of units per layer this network would have by considering that 20 would be the maximum number of elements per layer.

Based on the presented considerations, the following NN was implemented to perform the presented experimental results:^xii

resilient back-propagation algorithm;

11-layer topology, 2/3/4 units in the input layer, 18 hidden units per layer and 2/3/4 units on the output;

200 maximum epoch;

maximum square error (MSE), Goal = $0.002$ ;

70% for training data;

15% to validate;

15% to test.

Support vector machines

Support vector machines (SVM) provide separability to non-linear regions by implementing kernel functions that avoid the local minimum issues by implementing quadratic optimization, so that, unlike NN, this technique is more related to an optimization algorithm rather than to a greedy search algorithm. Also when the classification problems do not present a simple separating criterion, there are several mathematical approaches that could be applied to the SVM strategy, in order to retain all the simplicity of hyperplane separation.

Implementation

Three transformation kernels were implemented to perform the SVM identification processes:

Gaussian or radial basis $G (x, y) = \exp (- (x - y)^{'} (x - y) / 2 σ^{2}))$ ;

polynomial $G (x, y) = (1 + x^{'} y)^{d}$ ;

multi-layer perceptron $G (x, y) = \tanh (p_{1} x y + p_{2})$ .

Besides the fact that the training algorithm function implements an optimization method to identify vectors $s_{j}$ , weights $α_{j}$ and bias $b$ , which is used to classify the vectors $x$ according to equation (9), the results of the training process are also considered in the optimization of the classification process. This condition is known as the condition of Karush– Kuhn– Tucker (KKT), which is analogous to the condition where the gradient must be zero or at least modified to consider its limitations as

c \sum_{i} α_{i} k (s_{i}, x) + b

(9)

where $k$ is the kernel function and for the lineal case and if c ⩾ 0, $x$ is considered as a member of the first group and otherwise as a member of the second group, also restricted by the Lagrangian conditions, as

L (x, λ) = f (x) + \sum λ_{g, i} g_{i} (x) + \sum λ_{h, i} h_{i} (x)

(10)

where $f (x)$ is the objective function, $g (x)$ is the conditioning function vector when $g (x) ⩽ 0$ and $h (x)$ is the the conditioning function vector when $h (x) = 0$ . $λ$ vector, is then a concatenation of the $λ_{g}$ and $λ_{h}$ values of the Lagrange multiplier, with the the conditions of $\nabla_{x} L (x, λ) = 0$ , $λ_{g, i} g_{i} (x) = 0 \forall i$ , $g (x) ⩽ 0$ , $h (x) = 0$ and $λ_{g, i} ⩾ 0$ , to reduce the computational burden in the identification process, and the support vectors are defined by a $n_{s v} x p$ matrix, where $n_{s v}$ is the number of support vectors (maximum size of the training sample) and $p$ are the elements of the $β$ vector (which is the numeric vector of linear predictor coefficients), having a length equal to the number of predictors used to train the model and $α$ values are vectors of $n_{s v}$ elements (which can be very large for data sets that contains many features).

Results

Separability

The first thing that may be noticed is that this methodology provides a significant reduction of the computational complexity, since, as can be observed in Figures 9 to 17, the degree of separability obtained by implementing it can be easily observed and some of the behavioral tendencies can be noticed without a computational identification process.

Figure 9.

Distribution comparison between classes 3 and 1. A separation between the coefficients associated with these classes can be easily observed, even before the classification and recognition processes (class 3: relaxation/calm, class 1: happiness/elation).

Figure 10.

Distribution comparison between classes 3 and 2. A very clear separation between the coefficients associated with these classes can be easily observed, even before the classification and recognition processes (class 3: relaxation/calm, class 2: anger/hostile).

Figure 11.

Distribution comparison between classes 3 and 4. A clear separation between the coefficients associated with these classes can be easily appreciated, even before the classification and recognition processes (class 3: relaxation/calm, class 4: boredom/sadness).

Figure 12.

Distribution comparison between classes 1 and 4. For this case the separation between classes is not as clear as the three previous cases; however, that the separation process can be carried out with simple classification techniques can still be observed (class 1: happiness/elation, class 4: boredom/sadness).

Figure 13.

Distribution comparison between classes 1 and 2. For this case the separation between classes is not as clear as the three previous cases; however, that the separation process can be carried out with simple classification techniques can still be observed (class 1: happiness/elation, class 4: anger/hostile).

Figure 14.

Distribution comparison between classes 4 and 2. For this case the relationship between the classes is considered very close, which could indicate very similar behavior in both classes (class 1: boredom/sadness, class 4: anger/hostile).

Figure 15.

Distribution comparison between classes 1, 2 and 3. This representation shows the level of complexity for the three-class identification process (class 1: happiness/elation, class 2: anger/hostile, class 3: relaxation/calm).

Figure 16.

Distribution comparison between classes 1, 3 and 4. This representation shows the level of complexity for the three-class identification process (class 1: happiness/elation, class 3: relaxation/calm, class 4: boredom/sadness).

Figure 17.

Distribution comparison between all classes. This representation shows the level of complexity for the four-class identification process (class 1: happiness/elation, class 2:anger/hostile, class 3: relaxation/calm, class 4: boredom/sadness).

Implementation of class three as reference

Figures 9, 11 and 10 show feature distribution models that implement class 3 as reference. A very weak correlation between classes 2, 4 and 3 can be observed, while a sightly higher correlation can be observed for classes 1 and 3 (i.e. the relaxation state is more related to happiness than to anger or sadness).

Implementation of class one as reference

Figures 12 and 13 show the feature distribution models that implement class 1 as reference. It can be observed that the correlation between classes 1 and 4 could be considered weak, while the correlation between classes 1 and 2 could be considered strong (i.e. the happiness state is more related to anger than to sadness).

Implementation of class two as reference

Figure 14 shows a feature distribution model that provides a comparison of classes 2 and 4; the relationship between these emotional states is considered very close.

Implementation of a three-classe comparison

In Figures 15 and 16 it can be seen that even though there is clearly overlap between classes, they are mostly distinguishable to the naked eye, except for those combinations labeled with a close relationship, such as anger and sadness.

Implementation of a four-class comparison

Figure 17 provides an overview of the four emotion recognition process that was implemented in this work; it can be observed that these emotions are closely related and represent a considerable increase in the complexity of identification.

NN performance

The implemented NN obtains up to 98% mean recognition rate for the trivial case (by recognizing a single emotional state), while for the binary and the multi-class scheme (two, three and four emotions) the mean identification rates were up to 90.2%, 84.2% and 80.9%, respectively.

The performance of the NN for the two classes recognition process can be observed in Figures 18 and 19, in which can be appreciated the recognition rate by class and a graph showing ten validations, showing a stable performance.

Figure 18.

The 10-fold cross-validation performance of the NN for the two-class identification process.

Figure 19.

Performance of the NN for the two-class identification process.

Figures 20 and 21 present the identification performance for the three-class recognition scheme, which obtains up to an 84.2% mean identification rate and the 10-fold cross-validation process to evaluate the stability of this identification scheme.

Figure 20.

The 10-fold cross-validation performance of the NN for the three-class identification process.

Figure 21.

Performance of the NN for the three-class identification process.

Figures 22 and 23 presents the four-class recognition scheme which obtains up to an 80.9% mean identification rate and the 10-fold cross-validation process to evaluate the stability of this identification scheme.

Figure 22.

The 10-fold cross-validation performance of the NN for the four-class identification process.

Figure 23.

Performance of the NN for the four-class identification process.

SVM performance

The results obtained by implementing the support vector methodology and the proposed signal conditioning strategy are shown in Tables 4, 5 and 6.

Table 4.

SVM average identification rates for the Gaussian experimental cases.

	Gaussian
	Identification Rates (%)
	Case 1	Case 2	Case 3	Case 4
a	89.59	83.61	79.20	82.61
b	88.78	83.22	81.19	80.56
c	86.61	81.22	78.53	81.23
d	84.03	84.85	83.88	83.38

Table 5.

SVM average identification rates for the polynomial experimental cases.

	Polynomial
	Identification Rates (%)
	Case 1	Case 2	Case 3	Case 4
a	90.19	82.34	78.76	82.67
b	89.45	83.76	81.56	81.23
c	89.12	82.21	79.50	82.47
d	85.47	84.73	84.49	85.15

Table 6.

SVM average identification rates for the multi-layer experimental cases.

	Multi-layer
	Identification Rates (%)
	Case 1	Case 2	Case 3	Case 4
a	89.89	83.43	78.16	81.11
b	89.85	82.98	83.12	81.64
c	87.68	82.47	77.93	80.51
d	85.48	86.64	83.13	83.18

SVM identification performance

As can be observed in Table 7, the performance obtained from implementing this identification technique and the proposed signal conditioning strategy remains in competitive range when compared with the rates reported in the literature, although it could be considered that some other authors obtain slightly higher identification rates,^9,13 despite the fact that their implementations do not consider multi-class problems, while works that do consider multi-class schemes obtain significantly lower identification rates than those presented in this paper.^4,5

Table 7.

Performance comparison of the implemented classification techniques.

Overall Performance (%)
Cases	Case 1	Case 2	Multi-class
SVM Gaussian	87.25	83.22	81.32
SVM Polynomial	88.55	83.26	81.97
SVM Multi-Layer	88.22	83.88	81.11
NN	90.2	84.2	80.9

Conclusions

We present a strategy to carry out an emotion identification process, by analyzing the electroencephalographic activity of users when they are experiencing one or more emotional stimuli by implementing a strategy that reduces the amount of electrodes that have to be analyzed, which can be translated as an a priori removal of large amounts of information from the outset, retaining competitive recognition rates. In addition, some considerations that allow for the reduction of the computational burden required for the recognition and identification process are presented.

Most of the comparatives representations presented in this work contain information sets of up to 16 participants, and for all of them the information appears to be grouped into classes, which suggest the existence of relationships between the signal behavior of emotional states.

Another important aspect that may be noticed is that even though the amount of the initial information is reduced, we did not get any significant penalty in the recognition rates and our identification rates are comparable to those presented by some of the most important researchers in the field, such as Dr Muruarapan. However, the lack of a standard methodology to perform an emotion recognition task is one of the main problems that we faced in the development of this research, since most works implement distinct approaches and even distinct data sources, resulting in a very wide range of experimental procedures and results, making comparison between them unfeasible. The proposals presented by Dr Muruarapan and Dr Scherer underpin the efforts of the community by supporting the importance of affective computing and its impact on technological developments. Also the development of affective computing techniques are becoming more viable with the development of a wide variety of portable devices which facilitate information gathering processes, as well as digital processing techniques.

Because we work with a group of specialists focused on the physical rehabilitation of high performance athletes, the implementation of this algorithm in a real-time platform, combined with a micro-expressions recognizing technique is contemplated as future work to serve as a support tool to monitor patient progress. This is important because experts suggest that some of the athletes endanger their physical integrity, whether by their competitive desire or by frustration. Also almost any modern device has capability to collect and process large amounts of information, and therefore the possibility of developing systems to recognize the emotional states of a user is of growing interest.

Footnotes

Acknowledgements

I would like to thank CONACYT for making this project possible and the Technological Institute of Tijuana for providing necessary technical support for the implementation of this project.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Picard

RW.

Affective computing. 1st ed. Cambridge, MA: MIT Press, 2000.

James

What is an emotion?

Mind 1884; 9(34): 188–205.

Murugappan

Rizon

Nagarajan

et al . Lifting scheme for human emotion recognition using EEG. In: International symposium on information technology, Kuala Lumpur, Malaysia, 26–28 August 2008, Vol. 2, pp.1–7. IEEE.

Murugappan

Human emotion classification using wavelet transform and NN. In: International conference on pattern analysis and intelligent robotics (ICPAIR), Putrajaya, 28–29 June 2011, Vol. 1, pp.148–153. IEEE.

Koelstra

Muehl, Soleymani

et al . DEAP: A database for emotion analysis using physiological signals,. IEEE T Affective Comput 2012; 3: 18–31.

Lin

Wang

et al . Multilayer perceptron for EEG signal classification during listening to emotional music. In: TENCON 2007 – 2007 IEEE region 10 conference, Taipei, 30 October–2 November 2007, pp.1–3. IEEE.

Heraz

Razaki

Frasson

Using machine learning to predict learner emotional state from brainwaves. In: Seventh IEEE international conference on advanced learning technologies, Niigata, 18–20 July 2007, pp.853–857. IEEE.

Takahashi

Remarks on SVM-based emotion recognition from multi-modal bio-potential signals. In: 13th IEEE international workshop on robot and human interactive communication, 20–22 September 2004, pp.95–100. IEEE.

. Emotion classification based on gamma-band EEG. In: Annual international conference of the IEEE engineering in medicine and biology society, Minneapolis, MN, 3–6 September 2009, pp.1223–1226. IEEE.

10.

Rozgic

Ananthakrishnan

Saleem

et al . Ensemble of SVM trees for multimodal emotion recognition. In: Asia-Pacific signal information processing association annual summit and conference, Hollywood, CA, 3–6 December 2012, pp.1–4. IEEE.

11.

Attabi

Dumouchel

Emotion recognition from speech: WOC-NN and class-interaction. In: 11th international conference on information science, signal processing and their applications, 2–5 July 2012 pp.126–131. E-ISBN : 978-1-4673-0380-4 Print ISBN: 978-1-4673-0381-1INSPEC Accession Number: 13033317 Conference Location : Montreal, QC DOI: 10.1109/ISSPA.2012.6310487 Publisher: IEEE.

12.

Razak

Komiya

Izani

et al . Comparison between fuzzy and NN method for speech emotion recognition. In: Third international conference on information technology and applications, Montreal, QC, 4–7 July 2005, Vol. 1. pp.297–302. IEEE.

13.

Gunler

Tora

Emotion classification using hidden layer outputs. In: International symposium on innovations in intelligent systems and applications, Trabzon, 2–4 July 2012, pp.1–4. IEEE.

14.

Claparede

Feelings and emotions. Feelings and Emotions: The Wittenberg Symposium 1928; 16: 124–139.

15.

Cannon

WB.

The James– Lange theory of emotion: A critical examination and an alternative theory. Am J Psych 1927; 39: 10–124.

16.

Asada

Development of artificial empathy. Neuroscience Res 2015; 90: 41–50.

17.

Calvo

D’Mello

Gratch

et al . The Oxford Handbook of affective computing. 1st ed. New York, NY: Oxford Library of Psychology, 2014.

18.

do Amaral

de Oliveira

. Limbic system: The center of emotions, http://www.healing-arts.org/n-r-limbic.htm (accessed January 2016).

19.

Scherer

. What are emotions? and how can they be measured? In: Trends and developments: research on emotions, 2005, Vol. 44, pp.695–729. Thousand Oaks, CA and New Delhi: Social Science Information and SAGE Publications.

20.

Darwin

The expression of the emotions in man and animals. 3rd ed. New York, NY: Oxford University Press, 1998.

21.

Black

Darwin in the world of emotions. J Roy Soc Med 2002; 95: 311–313.

22.

Russell

JA.

A circumplex model of affect. J Pers Soc Psychol 1980; 39: 1161–1178.

23.

Lehmann

Bahçesular

Brockmann

et al . Subjective experience of emotions and emotional empathy in paranoid schizophrenia. Psych Res 2014; 220: 825–833.

24.

Roseman

IJ.

Phenomenology, behaviors, and goals differentiate discrete emotions. J Pers Soc Psychol 1994; 67: 206–221.

25.

Roseman

IJ.

Appraisal processes in emotion: Theory, methods, research. 7th ed. Oxford: Oxford University Press, 2001.

26.

Ekman

Friesen

O’Sullivan

et al . Universals and cultural differences in the judgments of facial expressions of emotion. In: J Pers Soc Psychol 1987 53: 712–717.

27.

Ekman

Friesen

WV.

Constants across cultures in the face and emotions. J Pers Soc Psychol 1971; 17: 124–129.

28.

Clore

Ortony

Psychological construction in the OCC model of emotion. Emotion Rev 2013; 5: 335–343.

29.

The cognitive structure of emotions. Andrew Ortony, Gerald Clore and Alan Collins. New York, NY: University of Cambridge, 1988. no. of pages: 207. isbn 0-521-35364-5. Applied Cognitive Psychology 1992; 6: 181–182.

30.

Lang

Bradley

Cuthbert

. International affective picture system (IAPS): Technical manual and affective ratings. In: NIMH Center for the Study of Emotion and Attention, FL: The Center for Research in Psychophysiology, University of Florida. Retrieved from http://csea.phhp.ufl.edu/Media.html#midmedia

31.

Lang

Bradley

Cuthbert

. International affective digitized sounds (IADS): Stimuli, instruction manual and affective ratings. In: The Center for Research in Psychophysiology, University of Florida. Retrieved from http://csea.phhp.ufl.edu/Media.html#midmedia

32.

Yin

Wei

Sun

et al . A 3D facial expression database for facial behavior research. In: 7th international conference on automatic face and gesture recognition, Southampton, 2–6 April 2006, pp.211–216. IEEE.

33.

Moody

Mark

Goldberger

Physionet: A web-based resource for the study of physiologic signals. IEEE Eng Med Biol Mag 2001; 20: 70–75.

34.

Petridis

Martinez

Pantic

The {MAHNOB} laughter database. Image Vision Comput 2013; 31: 186–202.

35.

Laurence

Garey

. Brodmann’s localisation in the cerebral cortex. 1st ed. New York, NY: Spinger Science+Bussiness Media, 2006.

36.

Jacobs

KM.

Brodmann’s areas of the cortex. In: Encyclopedia of clinical neuropsychology. New York: Springer New York, 2011, pp.459–459.

37.

Sanchez-Panchuelo

Besle

Beckett

et al . Within-digit functional parcellation of Brodmann areas of the human primary somatosensory cortex using functional magnetic resonance imaging at 7 tesla. J Neurosci 2012; 32: 15815–15822.

38.

Isaacson

RL.

Limbic system. New York, NY: John Wiley and Sons, 2001.

39.

LeDoux

JE.

Emotion circuits in the brain. Ann Rev Neuroscience 2003; 23: 154–184.

40.

MacLEAN

The limbic system (‘ visceral brain’) and emotional behavior. AMA Arch Neurol Psychiatry 1955; 73: 130–134.

41.

Sherwood

Human physiology from cells to system. 8th ed. Boston, MA: Brooks/Cole, Cengage Learning, 2013.

42.

Fox

SI.

Human physiology. 7th ed. New York, NY: McGraw-Hill, 2002.

43.

Vidakovic

Mueller

Wavelets for kids: A tutorial introduction. Technical report, Duke University, 1991.

Emotional states recognition,implementing a low computational complexity strategy

Abstract

Keywords

Introduction

Related work

Emotions

Disambiguation

Emotion theories

Data sources

Database for emotion analysis using physiological signals

Data bounding methodology

The Brodmann regions

Selected electrodes

Signal conditioning

Noise filtering

Signal bounding

Blind source separation

Feature extraction

Feature selection

Classes

Identification process

Experimental setup and inputs

8.1.1 Classification inputs

Neural networks

Implementation

Support vector machines

Implementation

Results

Separability

Implementation of class three as reference

Implementation of class one as reference

Implementation of class two as reference

Implementation of a three-classe comparison

Implementation of a four-class comparison

NN performance

SVM performance

SVM identification performance

Conclusions

Footnotes

Acknowledgements

Declaration of Conflicting Interests

Funding

Notes

References