Sage Journals: Discover world-class research

Abstract

BACKGROUND:

Mental workload is one of the contributing factors to human errors in road accidents or other potentially adverse incidents.

OBJECTIVE:

This research probes the effects of mental workload on the electroencephalographic (EEG) and electrocardiogram (ECG) of subjects in visual monitoring tasks, based on which a comprehensive evaluation model for mental workload is established effectively.

METHODS:

Three degrees of mental workload were obtained by monitoring tasks with different levels of difficulty. 20 healthy subjects were selected to take part in the research.

RESULTS:

The subjective scores showed a significant increase with the increase of task difficulty, meanwhile the reaction time (RT) increased and the accuracy decreased significantly, which proved the validity of three degrees of mental workload induced. For the EEG parameters, a significant decrease of $\theta$ energy was found in Frontal, Parietal and Occipital with the increase of level of mental workload, as well as a significant decrease of $\alpha$ energy in Frontal, Central and Occipital, meanwhile a significant increase of $\beta$ energy occurred in Frontal and Occipital. There was a significant decrease of $\alpha$ / $\theta$ in Occipital, and significant increases of $\theta$ / $\beta$ and ( $\alpha+\beta)/\theta$ in Frontal, Central and Occipital, meanwhile ( $\alpha+\theta)/\beta$ and WPE decreased significantly in Frontal and Occipital. Among the ECG parameters, it was shown that Mean RR, RMSSD, HF_norm and SampEn decreased significantly with the increase of task difficulty, while LF_norm and LF/HF showed significant increases. These EEG indictors in Occipital and ECG indictors were chosen and constituted a multidimensional original sample. Principal Component Analysis (PCA) was used to extract the principal elements and decreased the dimension of sample space in order to simplify the calculation, based on which an effective classification model with accuracy of 80% was achieved by support vector machine (SVM).

CONCLUSION:

This study demonstrates that the proposed algorithm can be applied to mental workload monitoring.

Keywords

Mental workload EEG ECG PCA SVM

1. Introduction

The modern man-machine system is highly automated with the development and application of technology. Therefore, cognitive monitoring tasks are more and more common in systems, especially in the actual production, power plant control, transportation, aerospace and other fields of work. As the amount of information faced and processed by operators increases dramatically, the roles of humans in the human-computer system gradually begins to turn to monitoring and making decisions. It has been shown that 80% of the information is obtained through vision, so the operators must be highly focused, and thus bear great mental workload, which may easily cause safety accidents. Therefore, the study on the assessment of mental workload draws great attention.

There is no consistent of definition for mental workload, and the definitions vary from field to field. Eggemeier et al. [1] indicated that mental workload referred to the information processing ability of an operator to complete a specific task in a man-machine interaction system. The assessment of mental workload is very important and meaningful in ergonomic domain. Mehler [2] thought that mental workload was the load imposed limited brain resources by a specific task. Some scholars proposed that mental workload described the total amount of information that the brain could process in a certain time, namely rate of information processing [3, 4]. It is considered that mental workload is related to task requirement, time pressure, individual ability, effort and performance [5]. There are three principal measurements for the evaluation of mental workload: subjective measurement, behavior performance measurement and physiological measurement [6]. Questionnaires are the main forms of subjective measurement, such as Cooper-Harper, SWAT, and NASA-TLX [7, 11]. However, the disadvantage of subjective evaluation is that the measurement results are easily affected by differences between individuals, such as memory loss and cultural difference. Performance evaluation mainly includes primary task measurement and secondary task measurement based on indicators as reaction time (RT) and accuracy. However, the comparison of workloads between different environments is unworkable by primary task measurement. Meanwhile, it is easy to change the priority of primary and secondary tasks and lead to wrong conclusions in secondary task measurement if the test design is improper. In comparison to the above two measurements, the advantage of physiological evaluation is that it can measure the workload degree directly and continuously. The electroencephalogram (EEG) and electrocardiogram (ECG) are the two mostly used signals for the physiological evaluation of mental workload [12, 14].

Numerous experiments have been conducted in the past to quantify the effects of mental workload on the EEG and ECG, but the results have been mixed. Previous studies showed that alpha power varied with mental workload, as it is associated with arousal level, idling and cortical inhibition [15, 16]. Theta is shown to be mediated by intellectual efforts and task demands [17, 18]. In addition, beta also proved to be associated with mental workload [19, 20]. ECG is another physiological signal that can quantitatively reflect the effect of mental workload on the human body [21, 22]. It was shown that the heart rate (HR) increased when the mental workload increased [23]. Meanwhile, the HRV proved to respond more quickly than HR to changes in momentary workload, i.e. the HRV decreased as the mental workload increased [24]. However, the challenge for physiological measurement is its reliability, for it is assumed that the changes of mental workload are the only factors causing changes of physiological indicators, but in fact, many other factors unrelated to mental workload may also cause these changes. Another limitation is that different work takes up different mental resources and thus produce different physiological responses, i.e. one physiological indicator is applicable to one type of work and may not be applicable to another type of work.

Considering the multi-dimension characteristics of mental workload, single measurement for mental workload is limited. In contrast with previous studies, multiple measurements were combined to study mental workload in the current study, i.e. various measurements complemented each other and avoided their own shortcomings. Along with subjective and performance measurements, effective EEG and ECG characteristic evaluation indicators of mental workload were obtained and an effective high-dimensional vector was constructed. PCA was used to reduce the dimension of the data and a new set of samples was formed, based on which the method of SVM was subsequently used to establish an effective model for evaluating mental workload.

2. Materials and methods

2.1 Subjects

Twenty healthy and young males (Mean age $=$ 24.7, SD $=\pm$ 2.45) participated in the experiment (compared with females, the scalp impedance for males with shorter hair is more easily reduced in order to better collect EEG signals). All subjects had normal or corrected vision and were used to a computer keyboard. All subjects had a normal sleep schedule and did not work night shifts. None of them reported a history of prior neurological disorder, heart disease or other medical contraindications. Each subject was required to sleep more than seven hours and keep off alcohol, tea, or other caffeinated substances, as well as smoking the day before the formal experiment. All subjects were informed of their rights and provided written informed consent prior to the experiment and were paid $50 after completion of the experiment.

Figure 1.

Interfaces of instrument-monitoring tasks with different difficulties.

2.2 Task design

The brain thinking mechanism, especially the process of information processing, thinking and decision-making is analyzed by studying the process of human-computer interaction in modern systems and extracting the corresponding cognitive model. According to the complex information processing process, the monitoring task can be divided into three phases: perceiving, judging and decision-making, and executing. Instrument monitoring is one of the most typical monitoring tasks, which covers the three phases fully. According to the attention resource theory, when people are engaged in monitoring work, the more information sources they need to perceive, the more attention resources they will allocate [25]. Mental workload is the extent of attention resources occupied of the person at work. Therefore, an instrument monitoring software with the amount of instruments settable was designed to induce three different levels of mental workload. Figure 1a–c show the interfaces for the low difficulty (L), medium difficulty (M) and high difficulty (H) instrument-monitoring tasks respectively. Each task is composed of multiple trials, each trial is presented for 6,000 ms, and, at most, one target appears with one instrument parameter beyond the normal range (ratio of target stimulus to non-target stimulus is 1:1). The normal ranges of instrument parameters, and the corresponding operations when they are out of the range are decided in advance. During the experiment, the subjects are required to observe and judge whether the corresponding parameters of the instruments on the screen are in normal range and if not, they are asked to respond with the corresponding button as quickly and correctly as possible, or otherwise to respond by pressing the “spacebar” button. The software records the RT and accuracy during the experiment automatically.

Figure 2.

Experiment procedure.

2.3 Experiment procedure

The experiment was carried out in an electrically shielded and soundproof room. The tasks were shown on a high-resolution computer 70 cm in front of the subjects. Figure 2 shows the experiment procedure, which is consisted by three tasks with different difficulties. To avoid the influence of single task sequence on experiment results, the sequence of three tasks with different difficulties was shuffled and designed according to Latin square design (e.g. the task sequence for the first subject was L-M-H, and then the sequence for the second subject should be M-H-L, and so on). Each task lasted for 6 min. To avoid the effect of fatigue induced by continuous task execution on experimental results, subjects were required to take a break for 5 minutes after completing each task, and then proceeded with the next task. EEG and ECG signals were recorded during the tasks, and the software recorded RT and accuracy of each task automatically. Meanwhile subjects were asked to score their workload levels according to subjective evaluation scale shown in Table 1 after completing each task.

Table 1
Subjective evaluation scale for mental workload

Descriptions for 1 ${}^{\text{st}}$ index	Descriptions for 2 ${}^{\text{nd}}$ index	Score
Difficulty level is acceptable	Very easy	1
	Easy	2
	Normal	3
Difficulty level is a little high	Low but annoying	4
	Medium and objectionable	5
	Very objectionable but endurable	6
Difficulty level is higher	Requiring extreme effort to reduce the error to a medium level	7
	Requiring extreme effort to avoid countless mistakes	8
	Requiring extreme effort to complete task, but still with countless mistakes	9
Difficulty level is highest	Unable to complete the task	10

2.4 Physiological signal recording and processing

2.4.1 EEG

The EEG signal was amplified and recorded with a Brain Products 32-channel System (ANT, Germany). This system consisted of 32 Ag/AgCl active electrodes which are located on a scalp according to the International 10–20 System. The vertical and horizontal channels of electrooculograms were recorded. Linked-mastoid (M1 and M2) were used as reference electrodes. The electrode impedance was controlled no more than 10 k $\Omega$ during the experiment. The EEG signals were digitalized continuously with a sampling rate of 512 Hz with a 0.05–100 Hz band-pass filter and 50 Hz notched.

Considering different brain regions associated with different cognitive functions, 12 channels EEG signals from 4 brain regions, i.e. Frontal (F3, Fz, and F4), Central (C3, C4, and Cz), Parietal (P3, P4, and Pz), and Occipital (O1, O2, and Oz), were analyzed in the current study. Since there were many artifacts in the raw EEG signals, of which the major contributor was ocular artifacts, Independent Component Analysis (ICA) was adopted to suppress EEG artifacts in this study. Wavelet Packet Transform (WPT) was chosen to detect EEG rhythms $\delta$ (0.5–4 Hz), $\theta$ (4–8 Hz), $\alpha$ (8–16 Hz) and $\beta$ (16–32 Hz) and to research on the varying of energy spectrum activities in different levels of workloads. The EEG signals were processed by Wavelet packet analysis with six-octave wavelet decomposition, among which dB4 was used as the mother wavelet which was considered to be the most suitable function. The wavelet packet decomposition tree and corresponding EEG rhythms are shown in Fig. 3. Then the energy for each rhythm could be obtained by summing the squares of wavelet coefficients, based on which the relative energy for EEG rhythms, four equations: $\alpha$ / $\theta$ , $\beta$ / $\theta$ , ( $\alpha+\beta$ )/ $\theta$ and ( $\alpha+\theta$ )/ $\beta$ could be obtained. Meanwhile wavelet entropy (WPE) was also obtained, which is often used as a disorder quantifier in the wavelet domain. These parameters were assumed as the indicators of mental workload and the validity would be examined.

Figure 3.

Schematic diagram of wavelet packet decomposition.

2.4.2 ECG

The ECG data were obtained through the Breath Wear-ECG device (wearable wireless heart patch). Six channels were recorded by placing the electrodes on the designated locations of body. Raw ECG signals were digitalized with a sampling rate 1000 Hz, and filtered by a low pass Gaussian filter with cut-off frequency of 40Hz, while IIR Zero-Phase Filter was used to attenuate baseline wander with a cutoff frequency of 0.5 Hz. Three methods, that is, time domain analysis, frequency domain analysis and nonlinear analysis were used to process the preprocessed ECG signals. The HRV time domain parameters (Mean RR, SDNN, SDANN, RMSSD, SDNNI, NN50 and PNN50), frequency domain parameters (LF_norm, HF_norm, LF/HF) and SampEn were chosen in the current study as the assumed indicators of mental workload for the later analysis. These parameters were obtained through the package software of the ECG device.

Figure 4.

Subjective scores and performance results.

3. Results

3.1 Subjective results

Mean subjective scores for the three degrees of mental workload can be found in Fig. 4a. One-way ANOVA was chosen to analyze the subjective scores of mental workloads among the three tasks with different mental workloads. The results showed a significant increase of scores with the increase of task difficulty ( $F=$ 115.976, $P<$ 0.01). Bonferroni corrected post-hoc tests were used and the results showed significant differences between any two tasks with different degrees of difficulty (all $P<$ 0.05).

3.2 Performance results

Mean performance indicators (RT and Accuracy) for the three levels of difficulty can been found in Fig. 4b and c, from which it could be seen that the accuracy decreased when the task difficulty increased. Meanwhile the correctly-hitting RT, correctly-denying RT and mean RT all increased when the task difficulty increased, which were obviously due to the increase of the amount of perceived signal sources when the task difficulty increased. One-way ANOVA was used to analyze the accuracy among the three tasks with different difficulties. The results showed a significant decrease of accuracy when the task difficulty increased ( $F=$ 11.641, $P<$ 0.05). Bonferroni corrected post-hoc tests were chosen and the results showed significant differences between any two tasks with different difficulty (all $P<$ 0.05).

3.3 EEG data

The artifact-removed EEG signals from the first 180 s of each task were chosen for analysis in order to speed up data processing. The WPT and WPE were used to compute the previously assumed EEG indicators for each 1-s interval, and then averaged these 180 1-s intervals and the final values were considered as the parameters of the task. Therefore, the relative energies of $\alpha$ , $\theta$ , $\beta$ , $\alpha$ / $\theta$ , $\beta$ / $\theta$ , ( $\alpha+\beta$ )/ $\theta$ , ( $\alpha+\theta$ )/ $\beta$ and WPE in four brain regions (F, C, P and O) were obtained.

One-way ANOVA was used to analyze each parameter in each brain region, and the results are shown in Table 2. Figure 5 shows the histogram of comparisons of EEG parameters among three levels difficulties in different brain regions. As shown in Table 2 and Fig. 5, significant decreases of the relative energy of $\theta$ occurred in Frontal, Parietal and Occipital (all $P<$ 0.05); the $\alpha$ relative energy decreased significantly in Frontal, Central and Occipital (all $P<$ 0.05); the relative energy of $\beta$ increased obviously in Frontal and Occipital (all $P<$ 0.05); $\alpha$ / $\theta$ decreased significantly in Occipital ( $P<$ 0.05); $\theta$ / $\beta$ and ( $\alpha+\beta$ )/ $\theta$ showed significant increases in Frontal, Central and Occipital (all $P<$ 0.05), meanwhile ( $\alpha+\theta$ )/ $\beta$ and WPE decreased significantly in Frontal and Occipital (all $P<$ 0.05). Figure 6 shows the brain topography of one subject who was chosen randomly, which indicates the activities of the three rhythms in the tasks with three different difficulties levels, and blue denotes low activity and red denotes high activity. As shown in the topography, the activity of $\theta$ decreased significantly in Parietal and Occipital; the activity of $\alpha$ wave decreased significantly in Central, and also in downtrends in both Frontal and Occipital; the $\beta$ activity increased significantly in the Occipital. These results were consistent with the results of statistical analysis of the data in Table 2, and were observed in most subjects (65%).

Table 2
One-way ANOVA results of EEG parameters in different brain regions

EEG parameters/brain regions	F	C	P	O
$\theta$	$<$ 0.05	/	$<$ 0.05	$<$ 0.05
$\alpha$	$<$ 0.05	$<$ 0.05	/	$<$ 0.05
$\beta$	$<$ 0.05	/	/	$<$ 0.05
$\alpha/\theta$	/	/	/	$<$ 0.05
$\theta/\beta$	$<$ 0.05	$<$ 0.05	/	$<$ 0.05
( $\alpha+\beta$ )/ $\theta$	$<$ 0.05	$<$ 0.05	/	$<$ 0.05
( $\alpha+\theta$ )/ $\beta$	$<$ 0.05	/	/	$<$ 0.05
WPE	$<$ 0.05	/	/	$<$ 0.05

Figure 5.

The comparisons of EEG characteristic parameters among three tasks with different degrees of difficulty.

Figure 6.

One subject’s brain topography.

3.4 ECG data

One-way ANOVA was used to analyze the ECG parameters (Seven HRV time domain parameters and four frequency domain parameters) obtained during the three tasks with different difficulties. For the analysis results in $P<$ 0.05, pairwise comparison was conducted through Bonferroni corrected post-hoc tests. The analysis results are shown in Table 3, from which it can be seen that both Mean RR and RMSSD decreased significantly with the increase of task difficulty (all $P<$ 0.05), and pairwise comparison results indicate that obvious differences existed between the tasks with low difficulty and high difficulty (all $P<$ 0.05), but there were no obvious changes between the tasks with low difficulty and medium difficulty (all $P>$ 0.05), as well as tasks with medium difficulty and high difficulty (all $P>$ 0.05); LF_norm showed a significant increase when task difficulty increased ( $P<$ 0.01), and the difference between tasks with low difficulty and high difficulty was significant ( $P<$ 0.01); HF_norm showed a significant decrease with the increase of task difficulty ( $P<$ 0.01), and the difference between tasks with low difficulty and high difficulty was significant ( $P<$ 0.01), as well as the tasks with medium difficulty and high difficulty ( $P<$ 0.05); LF/HF increased significantly when the task difficulty increased ( $P<$ 0.01), and the difference between tasks with low difficulty and high difficulty was significant ( $P<$ 0.01), as well as the tasks with medium difficulty and high difficulty ( $P<$ 0.01); SampEn decreased significantly with the increase of task difficulty ( $P<$ 0.05), and pairwise comparison results showed significant differences between the tasks with low difficulty and high difficulty in the both two parameters ( $P<$ 0.01). Therefore, Mean RR, RMSSD, LF_norm, HF_norm, LF/HF, and SampEn proved to be effective indictors for the evaluation of mental workload.

Table 3
Statistical analysis results of ECG parameters

Parameters	L		M		H		P		P (L-M)	P (L-H)	P (M-H)
	863.75	(76.38)	850.05	(71.26)	846.84	(110.64)	$<$	0.05	0.11	$<$ 0.05	1.	0
SDNN	143.11	(36.45)	148.44	(35.88)	126.75	(36.40)		0.167	/	/	/
SDANN	133.77	(25.04)	136.31	(26.11)	124.52	(27.79)		0.355	/	/	/
RMSSD	42.47	(8.97)	35.40	(10.44)	33.28	(9.17)	$<$	0.05	0.078	$<$ 0.05	1.	0
SDNNI	26.46	(10.56)	25.73	(6.63)	30.12	(10.63)		0.436	/	/	/
NN50	38.00	(15.24)	27.65	(19.44)	36.00	(22.17)		0.767	/	/	/
PNN50	22.30	(6.58)	20.53	(7.02)	19.08	(6.05)		0.392	/	/	/
LF_norm	43.19	(12.65)	49.69	(13.02)	59.20	(11.89)	$<$	0.01	0.347	$<$ 0.01	0.	068
HF_norm	43.89	(7.49)	42.56	(8.36)	36.37	(5.03)	$<$	0.01	1.0	$<$ 0.01	$<$ 0.	05
LF/HF	1.01	(0.33)	1.18	(0.23)	1.65	(0.38)	$<$	0.01	0.329	$<$ 0.01	$<$ 0.	01
SampEn	2.55	(0.35)	2.30	(0.44)	2.18	(0.37)	$<$	0.05	0.068	$<$ 0.01	0.	353

3.5 Multi-physiological classifier

It can be concluded from the above analysis results that EEG signals and ECG signals can effectively evaluate mental workload. However, considering the reliability of physiological signals, the effectiveness of using a single physiological signal to evaluate mental workload is low. Therefore, the EEG and ECG indictors were combined to establish an effective classifier in the current study, which could be used to classify three degrees: low difficulty, medium difficulty and high difficulty. There are many classification algorithms, such as Bayesian Network, Neural Network, and Decision tree, etc., all of which need to follow Empirical Risk Minimization principle, but need sufficient data guarantee. However, the sample size of this study is small, therefore the above algorithms don’t apply here. Especially high-dimensional feature inputs in this study, insufficient samples will result in the inability to obtain better generalization ability. Therefore, Support Vector Machine (SVM) is proposed and used in this study as it is a classification algorithm based on the Structural Risk Minimization (SRM) principle, which has advantages in solving small sample, nonlinearity and high dimensions pattern recognition [25]. However, the resource for calculating time and space will increase exponentially as the samples increase in numbers, which is a disadvantage for SVM. Therefore, Principal Component Analysis (PCA) was used to eliminate correlation of data, extract the principal elements and decrease the dimension of sample space in order to simplify calculation and save resource space. Then the dimension-reduced data was used as the input of SVM classification.

Eight EEG indictors [ $\theta$ , $\alpha$ , $\beta$ , $\alpha$ / $\theta$ , $\beta$ / $\theta$ , ( $\alpha+\beta$ )/ $\theta$ , ( $\alpha+\theta$ )/ $\beta$ and WPE] in Occipital and six ECG indictors (Mean RR, RMSSD, LF_norm, HF_norm, LF/HF, and SampEn) were chosen and constituted a multidimensional original sample. After standardizing the original sample to unified dimension, SPSS 19.0 was used to conduct PCA analysis and the results indicated that the cumulative frequency of the first five principal components were greater than 80%. Therefore, the final indictors were reduced from 14 to 5 dimensions, and a new sample set were developed as the input of SVM. The LIBSVM ToolBox of Matlab was used to complete the classification for three degrees of workload. The kernel function plays a key role in transforming input space into high dimensional feature space [26], Polynomial, Radial Basis Kernel and Sigmoid Kernel Functions were chosen as kernel functions in the current study respectively.

Table 4
Training results of the classification models

Kernel function	Loss function parameter ( $c$ )	Regularization parameter ( $g$ )	Accuracy
Polynomial	5.773	0.608	89.2%
Radial basis	5.628	0.562	92.2%
Sigmoid	4.375	0.563	87.4%

Figure 7.

Optimization of radial basis kernel function parameters.

The SVM classification process was done as follows [27]:

Normalization of eigen vector sample. After PCA analysis, [20 $\times$ 5] set data were obtained for each level of workload, and 75% data of each level was chosen for the training sample which was expressed as train_data [15 $\times$ 5], meanwhile the corresponding classification was expressed as train_group [15 $\times$ 5] (Low, Middle and High degrees were expressed by 1, 2, 3 respectively). Similarly, the rest 25% data was considered as testing sample and expressed as test_data [5 $\times$ 5], and the classification was expressed as train_group [5 $\times$ 5].

Preprocessing. For improving the model classification accuracy effectively, all samples were normalized to [0, 1], and the processing method was as follows:

$\displaystyle f:x\rightarrow y=\frac{x-x_{\min}}{x_{\max}-x_{\min}}\quad(x,y% \in R^{n})$

Kernel parameter optimizing. Loss function parameter ( $c$ ) and regularization parameter ( $g$ ) are two important parameters for kernel function, and the choice of them determines the classification effect and performance [28]. In the study, the grid search algorithm was used for parameter optimization ( $c, g$ ), while the K-fold cross validation (K-CV) was used for cross validation, among which the range of values for log ${}_{2}(c)$ and log ${}_{2}(g)$ were [ $-$ 4, 4], K $=$ 3.

Classifier training and testing. Based on the optimum parameters ( $c, g$ ), the training sample was trained and the optimum classifier was obtained. Then the testing sample was classified and predicted by the classifier with a testing accuracy, which was considered as the measure of the performance of the classification model.

The optimization results for the three classification models are shown in Table 4, from which it can be seen that the model constructed by Radial Basis Kernel function obtained the best classification effect (Best_ACC $=$ 92.2%) with optimum parameters (Best_c $=$ 5.628, Best_g $=$ 0.562) comparing with other classifiers, and the parameter searching results of the optimization process (contour map and 3D view) are shown in Fig. 7. Based on these parameters, the SVM classifier was built and verified, and the classification testing result (Predict_ACC $=$ 80%, 12/15) is shown in Fig. 8. Therefore, the established classifier based on the fusion of EEG and ECG parameters in this paper has a good performance.

Figure 8.

Actual and predictive classifications of test set.

4. Discussion

Many studies have examined mental workload, and most of them indicated that the evaluation effect of comprehensive assessment with multi-methods is superior to the assessment with single method [29, 31], which may be due to the multi-dimensional characteristics of mental workload. In the present study, multiple measurements were combined to study mental workload. Subjective and performance evaluations were combined to verify the validity of task difficulty degrees of classification. The subjective evaluation results indicated that the mental workload of the subjective experience increased significantly when the task difficulty increased. The RT increased and accuracy decreased significantly with the increase of task difficulty. Therefore, it can be concluded that the instrument-monitoring tasks with different degrees of difficulty induced different degrees of workload effectively.

EEG measurements have proved to be reliable for the evaluation of mental workload. Most previous studies showed that EEG rhythms ( $\theta$ , $\alpha$ , $\beta$ ) were all sensitive to the changes of mental workload [32, 34], that is, when persons were in a state of high vigilance, the activity of $\beta$ rhythm was strengthened; however, the activity of $\alpha$ would be motivated when they were in a waking state but with low vigilance; the $\theta$ activity was strengthened when they were in sleepy state. In this study, with the increase of task difficulty level, it was shown that $\theta$ activity showed a significant decrease in Frontal, Parietal significantly in Frontal and Occipital. Obviously, the increase of task difficulty degrees would induce high vigilance and pressure for operators to try to reduce the performance impaired. When the workload degrees induced are not enough to induce obvious energy changes of the EEG rhythms, suitable combinations of these EEG rhythms may magnify the energy variation for better distinguishing workload degrees. Therefore, four formulas, $\alpha$ / $\theta$ , $\beta$ / $\theta$ , ( $\alpha+\beta$ )/ $\theta$ and ( $\alpha+\theta$ )/ $\beta$ were proposed and studied. $\alpha$ / $\theta$ decreased significantly in Occipital; $\theta$ / $\beta$ and ( $\alpha+\beta$ )/ $\theta$ showed significant increases in Frontal, Central and Occipital, meanwhile ( $\alpha+\theta$ )/ $\beta$ decreased significantly in Frontal and Occipital. The WPE reflects the energy distribution information of the signal frequency space [35]. WPE decreased significantly in Frontal and Occipital, which indicated that the distribution of the four rhythms became increasingly narrow with lower disorder because of increasing slow wave energy and decreasing fast wave energy with the increase of mental workload, which demonstrated that the brain activities would be restrained in high workload. It could be found these EEG parameters all changed in the Occipital, and this might be due to the correlation of Occipital and visual cognition function. In addition, the Occipital-Parietal junction is associated with complex perception, attention, and thinking [35]. The instrument-monitoring tasks required a variety of complex perception, and most information was obtained through vision. Therefore, Occipital was the most sensitive region for mental workload comparing with other regions.

ECG is another physiological measurement for the evaluation of mental workload, which is used frequently in previous studies. HRV is considered to be sensitive to mental workload, which is most closely associated with autonomic nervous system (ANS) [36, 37]. In the present study, it was shown that Mean RR, RMSSD, HF_norm and SampEn decreased significantly when the task difficulty increased, while LF_norm and LF/HF increased significantly. Even though the results of these indicators did not show a positive correlation, it was found that mental workload caused a reduction of HRV in general. Some researchers thought this might be because the heart’s sympathetic activity increased as the workload increased [38]. Increased sympathetic activity could account for the decrease in Mean RR and the increase in LF_norm and LF/HF. However, other studies reported that mental workload might inhibit parasympathetic activity which can also cause reduction of HRV [39]. RMSSD that characterizes rapid changes in RR time series, as well as HF_norm which reflects the activity of parasympathetic, decreased with the increase of mental workload, which indicated the reduction of parasympathetic activity [40]. Therefore, Tobias Heine [41] speculated that mental workload, as well as concentration degrees, were both characterized by growing sympathetic activity firstly, and then weakening parasympathetic activity according to the degree of mental workload. Meanwhile, the decrease of SampEn indicated that the complexity of HRV signal reduced when the task difficulty increased.

Considering the reliability of physiological signals, the EEG and ECG indictors were combined to establish an effective evaluation model based on SVM. Before the classification, PCA was used to extract the principal elements and decrease the dimension of sample space in order to simplify the calculation. An effective classification model with accuracy of 80% based on SVM was achieved, which indicated that the proposed algorithm can be applied for mental workload monitoring.

5. Conclusion

Our findings demonstrate that EEG and ECG could be effectively used to evaluate different levels of mental workload in monitoring tasks. Three EEG rhythms ( $\theta$ , $\alpha$ and $\beta$ ), four equations [ $\alpha$ / $\theta$ , $\beta$ / $\theta$ , ( $\alpha+\beta$ )/ $\theta$ , ( $\alpha+\theta$ )/ $\beta$ ] and WPE all changed sensitively when the mental workload changed in specific brain regions. Meanwhile ECG parameters, Mean RR, RMSSD, HF_norm, SampEn, LF_norm and LF/HF all showed significant changes with the changes of mental workload. Therefore, all these physiological parameters proved to be effective indicators of mental workload. Based on these indicators, the method of SVM based on PCA was used to establish an effective classification model with an accuracy of 80% for three classes. This study demonstrated that the proposed algorithm could be applied to mental workload monitoring.

However, there were some limitations to the current study. Firstly, the experiment was only conducted during visual monitoring tasks, and whether the conclusions obtained apply to other types of tasks (e.g. memory, calculation) needs further verification. Secondly, the research was still in the preliminary stage, and how the online monitoring of mental workload should be realized needs further exploration. Moreover, except for EEG and ECG, other physiological measurements, such as EMG and EOG, should be included in the comprehensive assessment of mental workload.

Footnotes

Acknowledgments

This research was funded by NQI (2017YFF0206601), Presidential Fund of CNIS (522018Y-5943) and National Key R&D Program of China (2017YFF0206604).

Conflict of interest

None to report.

References

Fastenmeier

Gstalter

. Driving task analysis as a tool in traffic safety research and practice. Safety Science.2007; 45: 952-979.

Mehler

Reimer

Coughlin

, et al. Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transportation Research Record.2009; 2138(2138): 6-12.

Sanders

McCormick

. Human factors in engineering and design. Industrial Robot. 1998; 25(2): 153-159.

Wickens

. Information processing, decision making, and cognition. In Salvendy

. (Ed.), Cognitive engineering in the design of human-computer interaction and expert systems. Amsterdam: Elsevier. 1987.

Paas

Van Merriënboer

Adam

. Measurement of cognitive load in instructional research. Percept Mot Skills.1994; 79: 419-430.

Weinger

Herndon

Zornow

, et al. An objective methodology for task analysis and workload assessment in anesthesia providers. Anesthesiology.1994; 80(1): 77-92.

Reid

Nygren

. The subjective workload assessment technique: A scaling procedure for measuring mental workload. Advances in Psychology.1988; 87: 185-218.

Hart

Staveland

. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology.1988; 32: 139-183.

Gao

Wang

Song

, et al. Mental workload measurement for emergency operating procedures in digital nuclear power plants. Ergonomics.2013; 56(7): 1070-1085.

10.

Wanyan

Zhuang

Zhang

. Improving pilot mental workload evaluation with combined measures. Bio-medical Materials and Engineering.2014; 24(6): 2283-2290.

11.

Hart

Staveland

. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology.1988; 52: 139-183.

12.

Gevins

Smith

Leong

McEvoy

, et al. Monitoring working memory load during computer-based tasks with EEG pattern recognition methods. Humman Factors.1998; 40: 79-91. doi: 10.1518/001872098779480578.

13.

Hoover

Singh

Muth

. Real-time detection of workload changes using heart rate variability. Biomed. Signal Process.2011; 7: 333-341. doi: 10.1016/j.bspc.

14.

Ranganathan

Rangarajan

Bindhu

. Estimation of heart rate signals for mental stress assessment using neuro fuzzy technique. Appl. Soft Comput.2012; 12: 1978-1984. doi: 10.1016/j.asoc.

15.

Pfurtscheller

Stancak

Neuper

. Event-related synchronization (ERS) in the alpha band – an electrophysiological correlate of cortical idling: A review. Psychophysiol.1996; 24: 39-46.

16.

Fink

Grabner

Neuper

. EEG alpha band dissociation with increasing task demands. Cognitive Brain Research.2005; 24(2): 252-259.

17.

Pesonen

Hamalainen

Krause

. Brain oscillatory 4–30 Hz responses during a visual n-back memory task with varying memory load. Brain Res.2007; 1138: 171-177.

18.

Esposito

Aragri

Piccoli

Tedeschi

. Distributed analysis of simultaneous EEG-fMRI time-series: Modeling and interpretation issues. Magnetic Resonance Imaging.2009; 27(8): 1120-1130.

19.

Laine

Bauer

Lanning

. Selection of input features across subjects for classifying crewmember workload using artificial neural networks. Systems Man and Cybernetics.2002; 32(6): 691-704.

20.

Michels

Bucher

Lüchinger

. Simultaneous EEG-fMRI during a working memory task: Modulations in low and high frequency bands. Plos One.2010; 5(4): 187-197.

21.

Dishman

Nakamura

Garcia

. Heart rate variability, trait anxiety, and perceived stress among physically fit men and women. International Journal of Psychophysiology.2000; 37(2): 121-133.

22.

Salahuddin

Jeong

Kim

. Dependence of Heart Rate Variability on Stress Factors of Stress Response Inventory. In International Conference on E-health Networking, Application & Services. 2007. pp. 236-239.

23.

Lenneman

Backs

. Diagnosticity of cardiac modes of autonomic control elicited by simulated driving and verbal working memory dual-tasks. In Harris

. (Ed.), Engineering Psychology and Cognitive Ergonomics, Lecture Notes in Artificial Intelligence. Springer-Verlag, Berlin. 2007. pp. 541-550.

24.

Chen

Liaw

Chang

, et al. Combined heart rate variability and dynamic measures for quantitatively characterizing the cardiac stress status during cycling exercise. Computers in Biology and Medicine. 2015; 133-142.

25.

Roy

Charbonnier

Campagne

, et al. Efficient mental workload estimation using task-independent EEG features. Journal of Neural Engineering.2016; 13(2): 230-239.

26.

Fleury

Vacher

Noury

. SVM-based multimodal classification of activities of daily living in Health Smart Homes: Sensors, algorithms, and first experimental results. IEEE Transactions on Information Technology in Biomedicine. 2010; 14(2): 274-283.

27.

Ziegler

Kraft

Krein

, et al. Sensing and Assessing Cognitive Workload Across Multiple Tasks. In International Conference on Foundations of Augmented Cognition. 2016. pp. 440-450.

28.

Yan

. Combining active learning and semi-supervised learning to construct SVM classifier. Knowledge-Based Systems. 2014; 44(1): 121-131.

29.

Wilson

. An Analysis of mental workload in pilots during flight using multiple psychophysiological measures. The International Journal of Aviation Psychology.2002; 12(1): 3-18.

30.

Longo

. A defeasible reasoning framework for human mental workload representation and assessment. Behavior & Information Technology.2015; 34(8): 758-786.

31.

Wei

Sourina

Liu

, et al. EEG-based mental workload recognition related to multitasking. In International Conference on Information. Singapore. 2015. p. 112.

32.

Borghini

Vecchiato

Toppi

Astolfi

. Assessment of mental fatigue during car driving by using high resolution EEG activity and neurophysiologic indices. In 34th Annual International Conference of the IEEE EMBS. 2012. pp. 6442-6445.

33.

Joseph

Rosa

Chan

. Evaluation of Mental Workload in Visual-Motor Task: Spectral Analysis of Single-Channel Frontal EEG. In Conference of the Industrial Electronics Society. 2013. pp. 8426-8430.

34.

Paxion

Galy

Berthelon

, et al. Mental workload and driving. Frontiers in Psychology.2014; 5(1344): 1344-1349.

35.

Fan

Zhao

Luo

. Electroencephalographic Study of Mental Fatigue Induced by Prolonged Vigilance Task. Journal of Medical Imaging and Health Informatics.2018; 8: 1349-1357.

36.

Paxion

Galy

Berthelon

. Mental workload and driving. Front. Psychol.2014; 5(1344): 1-11.

37.

Chandra

Sharma

, et al. Workload regulation by Sudarshan Kriya: An EEG and ECG perspective. Brain Informatics.2017; 4(1): 13-25.

38.

Hirshfield

Chauncey

Gulotta

, et al. Combining Electroencephalograph and Functional Near Infrared Spectroscopy to Explore Users’ Mental Workload. In International Conference on Foundations of Augmented Cognition. 2009. pp. 239-247.

39.

Hjortskov

Rissen

Blangsted

, et al. The effect of mental stress on heart rate variability and blood pressure during computer work. European Journal of Applied Physiology. 2004: 84-89.

40.

Zimmermannviehoff

Thayer

Koenig

Herrmann

. Short-term effects of espresso coffee on heart rate variability and blood pressure in habitual and non-habitual coffee consumers – A randomized crossover study. Nutritional Neuroscience.2016; 19(4): 169-175.

41.

Tobias

Gustavo

Patrick

. Electrocardiographic features for the measurement of drivers’ mental workload. Applied Ergonomics.2017; 61: 31-43.

Assessment of mental workload based on multi-physiological signals