Mental workload characteristics of manipulator teleoperators with different spatial cognitive abilities

Abstract

The main research on manipulator teleoperation includes robust of high-degree of freedom manipulators, sensor measurement accuracy, time delay, and mechanical structure design. Increased mental capacity requirements for complex assignments result in an increased mental workload. Spatial cognitive ability was considered to be the key factor affecting teleoperation performance. To accomplish this, we had 50 participants performed teleoperation while recorded their electroencephalogram. Electroencephalogram data of each task were divided into two periods, which correspond to the observation and large-scale transfer stages of teleoperation, respectively (period 1) and adjust the attitude of the manipulator to approach and align with the target stage (period 2). Brain topographic maps of period 1 (period 1 wavelet packet energy minus resting state wavelet packet energy) and period 2 (period 2 wavelet packet energy minus resting state wavelet packet energy) show that the frontal, central, and occipital regions are the main working areas of low spatial cognitive operators in period 1, while the frontal, central, and occipital regions are the main working areas of high spatial cognitive operators in period 1. The main changes in period 2 were frontal, central, parietal, and occipital regions. This study has implication for the analysis of electroencephalogram signal characteristics of mental workload in different populations to improve operators’ well-being and safety at teleoperation work.

Introduction

Teleoperation is a form of operation in which operator remotely operates a robot or a manipulator to accomplish a task using spatial image information to remotely control the manipulator. A typical teleoperation system is commonly composed of four parts: operator, master robot, communication channel, slave robot, and the task environment. ^{1 –3} The display, also called communication channel, extends the operator’s ability to remotely manipulate by providing the same scene of real-time information to the operator’s remote object on the display, ¹ and such hand–eye alignment has been shown to increase mental workload, task time, and errors ^{4 –6} and bring challenges to geometric reasoning and spatial cognitive ability of operators. ⁷ Teleoperation robot can play an important role in earthquake relief, space exploration, and energy development, which are inaccessible to human beings. Relying on the telepresence technology, teleoperation robot enables people to control remote robots while working but also get feedback from the perception of the robot, such as hearing, vision, force, and so on, so that the operator can have a feeling of immersion.

The scene accuracy and automation control precision of teleoperation system are getting higher and higher. More and more of human–computer interfaces reasonably from the level of human–computer interaction in teleoperation system are designed. ² In special environment, human operation is more reliable than robot’s complete autonomous action, but the robot must have some local autonomous functions to cooperate with human operation; to ensure that the robot moves safer in dangerous and complex environment, there is a growing need to develop effective user interfaces that minimize operator error and improve task efficiency. The application of teleoperation robot technology in the field of medical diagnosis includes minimally invasive positioning operation, noninvasive diagnosis and treatment, and so on. The application in the field of space exploration mainly refers to orbit assembly, satellite maintenance, space station maintenance, and in the military field mainly refers to security, mine clearance, and rescue. These teleoperation techniques require the operator to have a rich prior knowledge of the remote environment to establish an accurate virtual reality teleoperation environment. In recent years, researches on teleoperation mainly includes robust remote teleoperation of high degree of freedom (DOF) manipulators, ⁸ improvement of sensor measurement accuracy and effect of time delay on field sensor system stability of teleoperation robot, ^3,9 mechanical structure design of robot body in special environment, ¹⁰ research on master–slave teleoperation robot control system based on virtual reality technology, ^11,12 above-mentioned teleoperation robot technology improves the safety of operation accuracy, makes full use of the inherent sense and reaction ability of operator, this degree of flexibility gives users the maximum level of control and performance, the size of the workspace can become a burden that can increase the likelihood of operator error, ⁸ and also improves the intelligent and autonomous operation ability of the robot, so that the robot can complete tasks that were difficult or impossible in the past in unknown or complex environments. ³

In the teleoperation mode of manipulator, operator mainly relies on the image information captured by the local camera and the global camera mounted on the manipulator to obtain the perception of the external working environment of the manipulator. In the operation process, it is necessary to recognize the processing and processing of the pictures provided by multiple local (follow-up) and global (fixed) cameras, ¹³ and operators need to pay comprehensive attention to other operational status information of manipulator. Therefore, teleoperation is a complex task to test the operator’s comprehensive cognitive and operational ability. The evaluation of its operational level and performance has also become an important issue in the selection training. ¹⁴ Teleoperation performance largely lies on operators, human-related factors, such as that spatial cognitive ability plays a key role in teleoperation process. ^7,15 Some scholars have studied the relationship between spatial cognitive ability and operational performance and pointed out that some components of spatial ability are important factors for rendezvous and docking tasks, which can be used to predict operational performance. ¹⁶ A validated and standardized task enables trainees to learn and quantify skills in a safe, low-stress environment, which translates easily to the real-world operating suite. However, studies have shown that training may regulate the relationship between cognitive ability and task performance, but at different levels of training, spatial cognitive ability may have different predictive effects on performance. ^17,18

Previous studies have shown that teleoperation performance of people with high spatial cognitive ability is collided less, more fluidly, and more effective. ^{19 –21} Ensuring the stability of teleoperation system under the action of estimating operator and environmental forces, ²² the study of simulation experiment of laparoscopic surgery in teleoperation also shows that with the training, spatial cognitive ability still affects the performance index of laparoscopic surgery. ^18,23 Formulating reasonable personnel selection mechanism, providing operators with effective spatial cognitive, motion control clues, and exploring spatial cognitive performance are the focus of research in recent years. In many studies, task performance has been proved to be closely related to cognitive ability, ^15,24,25 but the role of cognitive factors in skilled performance will be greater in complex operations. ¹⁷ In the complex operation process, researchers began to pay more attention to operator’s mental workload. ²⁶ Mental workload is usually related to task demand and individual’s capacity, and it is affected by many factors such as specific work requirement, ²⁷ operator’s effort, ability and environment, and to a degree the environmental context in which the performance occurs. ^28,29 Mental workload is a multidimensional construct determined by characteristics of the task of the operator. ²⁶ Electroencephalogram (EEG) is the most sensitive and promising workload indicator for dynamic monitoring of operator’s mental workload. ^30,31 It has been applied in many fields such as brain load detection, attention distribution, and situational awareness. The development of EEG technology also makes it available in the field of operation. Primitive EEG signal analysis mainly includes two methods: one is based on event-related potentials (ERPs) to measure mental workload ^32,33 and the other is based on spontaneous EEG to measure mental workload. ^31,34,35 Because of the intrusiveness of ERPs to the main task and the stringent demand for time, ERPs cannot be used as an indicator of real-time monitoring of mental workload. EEG requires less time, and linear indicators in frequency domain can be sensitive to the state of mental workload, which is the most commonly used method at present.

Differences in cognitive processing strategies and task processing abilities of different individuals result in differences in EEG signal characteristics and subjective perceived mental workload under the same task. ³⁶ However, previous studies have shown that people with high spatial cognitive ability have better collided less, more fluidly, and more effective teleoperation performance, ^{19 –21} so human spatial cognitive ability plays a key role in teleoperation. ¹⁵ However, there are not many studies on mental workload of operators with different spatial cognitive abilities in teleoperation, therefore, we used EEG measures to evaluate mental workload. The primary aims of this study were (i) to study the mental workload of operators with different spatial cognitive abilities during teleoperation of manipulators, (ii) to analyze the mental workload indicators of operators with different spatial cognitive abilities based on EEG signals, and (iii) the study can be used to select and train operators with different spatial cognitive abilities in teleoperation. Operator’s mental workload index and trend change provide the basis.

Methods

Subjects

A total of 50 male undergraduates participants with an average age of 22.85 (s = ±1.18) were recruited in this experiment. They are right-handed and had no experience in manipulator teleoperation. They were non-color-blind and had normal visual acuity. Spatial cognitive ability was tested using Hegarty’s Perspective-taking Test (spatial orientative ability) and the revised Vandenberg mental rotation test (mental rotation). ^15,37 Participants shall ensure adequate sleep time and a normal diet 2 days before the experiment. Drinking stimulating drinks such as tea or coffee 24 h before the test is forbidden. Each participant should wash their hair before the experiment, and during the experiment, they should avoid unnecessary speech and unnecessary head movement.

Experimental platform

The experimental platform builds a simulation platform based on the Virtual Robot Experimentation Platform and constructs a single display, three views, and six DOFs (three translational DOFs and three attitude DOFs) manipulator model. Participants need to observe the real-time image information provided by the virtual camera in the simulation scene on the display, and control the two manipulators of the left and right hand to adjust the distance and attitude, so that avoid two obstacles and eventually aligns its end ball with the hole in the target cube, and based on the end coordinate system of the manipulator in the global image to complete the task.

EEG acquisition equipment includes a Brain Products amplifier, a 32-channel EEG system to record EEG data, and a synchronous acquisition system of EEG. A synchronous acquisition system is used to collect, record, and store EEG data by the, while performing tasks through integrated signal amplification and acquisition software. Scalp electrodes were placed according to the modified International 10–20 system. The reference electrode was set to Fz, and the prefrontal lobe was grounded. The sampling rate was 500 Hz and electrode impedances were kept below 5 kΩ. The filter range of the amplifier was set to 0.1 Hz–100 Hz.

Experimental procedure

Before the experiment, each participant was provided with information about the experiment and EEG measurements, after which they performed three preexperiments to familiarize itself with the process. Before the formal experiment, participant needs to wear EEG acquisition equipment, and they were asked to complete task, each task will be completed three times. A 5-min rest period between each task was given to each participant, with the experimental sessions at the same time of the day assessing effects of teleoperation. To prevent errors and failure during the measurement procedure, the experimenter asked participants to make the least possible number of movements and avoid talking to others while physiological indices were recorded.

Data processing

Preprocessing of EEG data

The original EEG data collected by EEG recording system need to reset the reference electrode firstly, band-pass filters with frequencies of 1–50 Hz was selected to remove EMG and voltage drift, and 50 Hz notch filters were used to eliminate power frequency interference. Independent component analysis was used to correct vertical eye movements, blinks, and pulse artifacts. ³⁸ To analyze the data of multiple tasks uniformly, it is necessary to remove the baseline drift caused by the interception of data segments and to correct the baseline of data segments. The range of baseline correction is set to −300 ms –0 ms before the mark point. Because the frequency of attention, decision-making, and other neural activities in brain cognitive are generally below 40 Hz, after converting the time domain signal of EEG data into frequency domain, then the signal is filtered by the recursive infinite-length impulse-response (IIR) filter. The frequency range is 0.1–40 Hz. EEG data of each task were divided into two periods and processed separately to detect the changes of EEG of participants. The two periods correspond to the observation and large-scale transfer stages of teleoperation, respectively, and adjust the attitude of the manipulator to approach and align with the target stage.

Processing of EEG data

Previous studies have shown that EEG delta, beta bands, and gamma bands are sensitive to changes in mental workload. ^39,40 Different rhythm characteristics of EEG are significantly different under different states, so different rhythms can be used to reveal the internal relationship between the different states; in the common types of tasks such as working memory, the energy of common EEG frequency bands such as δ, θ, α, β, and γ can reflect the changes of mental workload. ^41,42 Wavelet transform was used, and in 32-channels, we only analyzed the EEG data of 14 channels in frontal region (F3, Fz, F4), middle frontal region (FC5, FC6), central region (C3, Cz, C4), parietal region (P3, Pz, P4), and occipital region (O1, Oz, O2). Multiscale decomposition of preprocessed EEG data was done using wavelet transform multiresolution analysis, extracting five sub-frequencies of δ(0.5–4Hz), θ(4–8Hz), α(8–13Hz), β(14–30Hz), and γ(31–40Hz). Entropy analysis method is used to determine that when the wavelet base is db6, the wavelet entropy is the smallest and the reconstructed signal is the closest to the original signal. ⁴³

The calculation method of wavelet entropy is as follows

$H\left(x\right)=-{\displaystyle \sum }_{i=0}^{L}p\left(i\right){\text{log}}_{10}p\left(i\right)$

1

L is the signal sub-band and p(i) is the probability of the occurrence of the sub-band.

The implementation steps of the wavelet packet energy analysis algorithm are as follows:

• 1) The time domain EEG signal is converted into frequency domain, and five sub-band signals are extracted. To reduce the time and space complexity of wavelet packet decomposition, the sampling rate of EEG signal is reduced from 1000 Hz to 256 Hz. Then, seven-level wavelet decomposition was conducted, in which each wavelet packet node represents a sub-band.

• 2) Calculate the energy of wavelet packet. Assuming that $C_j^i\left(k\right)$ is the wavelet decomposition coefficient corresponding to the i wavelet node in the j layer, then the energy value of the node is $E_j^i$ , as follows

$E_j^i\text{=}{\displaystyle \sum }_k{{\mathrm{(C}}_j^i\mathrm{(k))}}^{\text{2}}$

2

Equation (2) is used to calculate the wavelet packet energy of three frequency bands, as follows

$\begin{array}{l}{E}_{\text{θ}}=E_6^1\\ \\ E_{\text{α}}=E_6^2+E_7^6\\ \\ E_{\text{β}}=E_7^8+E_6^4+E_6^5+E_6^6+E_7^{14}\end{array}$

3

• 3) Calculate the sum of energy in three frequency bands, that is, the total energy, $E_{total}$ , as follows

$E_{total}=E_{\text{θ}}+E_{\text{α}}+E_{\text{β}}$

4

• 4) Calculate the wavelet packet energy of frequency parameters. The relative wavelet packet energy values of six frequency parameters k (k = δ, θ, α, β, γ) were calculated: $P_{\text{k}}=E_{\text{k}}/E_{total}$ , and then the relative wavelet packet energy mean values of all subjects in each lead will be calculated respectively.

Two-way repeated-measures analyses of variances were carried out to investigate condition-by-task interaction effects. Multiple comparisons were performed using the Bonferroni method. If the Mauchl’y sphericity test is not met, then the correction is performed using Greenhouse–Geisser.

Results

Mental workload with task load based on EEG

Figures 1 and 2 show the changes for δ, θ, α, β, and γ in different channels of operators. Figures 3 and 4 show wavelet packet energy difference brain topographic mapping of period 1 and period 2 cycles of EEG in different spatial cognitive ability operators.

Figure 1.

Contrast of wavelet packet energy of EEG between period 1 and period 2 in low spatial cognitive ability operators. Note: The black histogram represents period 1, and the white histogram represents period 2. In each column of data, the value is average across all participants with different spatial cognitive abilities for each task. EEG: electroencephalogram.

Figure 2.

Contrast of wavelet packet energy of EEG between period 1 and period 2 in high spatial cognitive ability operators. EEG: electroencephalogram.

Figure 3.

(a to e) Wavelet packet energy difference brain topographic mapping of period 1 and period 2 cycles of EEG in low spatial cognitive ability operators. EEG: electroencephalogram.

Figure 4.

(a to e) Wavelet packet energy difference brain topographic mapping of period 1 and period 2 cycles of EEG in high spatial cognitive ability operators. EEG: electroencephalogram.

Results of sample entropy comparison chart of five frequency bands in different teleoperation stages of Figure 1, the difference of sample entropy characteristics in different teleoperation stages of Figure 3, brain topographic map of Figure 3, and paired T test, increased wavelet packet energy in period 1, δ (F3, Pz, O1, O2, Oz), θ (F3, O2, Oz), α and β frequency segments F3, Pz, Oz.

Figure 2 is a comparison of five wavelet packet energies of operators with high spatial cognition. It can be seen that compared with period 1, the wavelet packet energy of all leads in period 2 δ, θ, α, and β frequency bands increases. From the brain topographic map and the paired sample T test results in Figure 4, we concluded that the wavelet packet energy of the leads F3, Fz, C3, Pz, O1, O2, Oz in the period 1 δ, θ, α, and β frequency segments was significantly larger than period 2. The most complex regions of the period with the cerebral cortex are the frontal, central, apical, and occipital regions of the operator with high spatial cognitive ability.

δ (F3, Pz, O1, O2, Oz) and θ (F3, O2, Oz) power of participant with low spatial cognitive ability increases with the increase of teleoperation time, while participant with high spatial cognitive ability has little change of θ power, which shows that with the increasing difficulty of task, mental demand and mental workload increase for the participant who has low spatial cognitive ability, as is reported that θ power during mentally demanding states has been reported at the frontal cortex. ^44,45 The results suggest that the decline of processing abilities as a consequence of high mental effort. α power of participant with low spatial cognitive ability increases with the increase in teleoperation time, while α power of participant with high spatial cognitive decreases. β of low spatial cognitive ability operators central region (C3, Cz, C4) and parietal region (P3, Pz, P4) increases but decreases in high spatial cognitive ability operators. γ of low spatial cognitive ability operators in frontal region (F3, Fz, F4) and middle frontal region (FC5, FC6) decreases but increases in high spatial cognitive ability operators.

Discussion

There are not many studies on the mental workload of operators with different spatial cognitive abilities in teleoperation, and this study aimed at to evaluate the mental workload of operators in teleoperation with different spatial cognitive abilities. Therefore, we used EEG measures to evaluate mental workload.

As the increase of teleoperation time, and it is difficult to succeed for a long time and accompanied by sleepiness, multiple operations are unsuccessful, participant feels more difficult, θ power increases continuously, ⁴⁶ especially participants with low spatial cognitive ability are relatively difficult to complete tasks, and mental workload is increasing.

It is generally believed that psychological stress, active thinking, and attention will cause EEG activity to move to higher frequency bands and inhibit the activity of α power. The evidence of α power variations at different workload levels may lie in its link to arousal level and idling, ⁴¹ but as the difficulty of task increases, participant’s psychological stress, active thinking, and attention may vary, not all participants’ α power will continue to change, which is consistent with a continuous increase of α activity has not been observed, not even in long-lasting experiments. ^47,48 The change of β power decreases as the difficulty of task increases, which is not related to the level of spatial cognitive ability, which also shows that β power changes consistently in different populations, especially in participants with low spatial cognitive ability.

The indicators α, β of participant with low spatial cognitive ability increased with the increase of operation, but the opposite of high spatial cognitive ability, this indicates that the main active brain region of participants with low cognitive region in teleoperation is the parietal region, which is considered to be the main active region of attention. ⁴⁹ It has been shown that increased α activity is related to mental states in which errors become more probable, ⁵⁰ with continuous measurements it has been shown that only δ power increases continuously in extended tasks, so people with different spatial cognitive abilities change consistently in θ power, but a continuous increase of α activity has not been observed, not even in long-lasting experiments. ^47,48

The results show that the complexity of EEG signals in frontal and occipital regions of low spatial cognitive operators is higher. Combining with the behavioral characteristics of the second stage, it shows that in the second stage, operators not only need to maintain correct operation selfishness but also need to constantly adjust the focus of sight and adjust the attitude of the manipulator to approach and target the eyes. This requires the brain to allocate more resources. The EEG power spectrum energy in the left hemisphere increased significantly, which is quite different from that in the right hemisphere. This is closely related to the task characteristics and operation modes of arm teleoperation. The right hemisphere plays an important role in spatial reasoning.

The level of brain thinking activity is directly related to the level of spatial cognition, people with low spatial cognitive ability are more alert, emotionally tense and stressed, and their EEG energy is mainly concentrated in β band with lower amplitude and higher frequency and α band is suppressed. Participants with high spatial cognitive ability have lower arousal level and mental workload level, enhanced activity of EEG in α band at lower frequencies, decreased activity in β band, so β changed less, which is consistent with the fact that the frontal region is mainly responsible for movement control, reasoning, and emotion. ^51,52 The increased mental workload level at the end of teleoperation is mainly reflected in the time and task refinement operation pressure, especially the task visual information perception is more refined, so in the occipital region, the high spatial cognitive ability of the participant’s related indicators changes significantly more than in the frontal region, which is consistent with the occipital region is closely linked to visual information. ⁵³

Performing teleoperation tasks mainly involves visual processing, decision-making, and motion control; therefore, θ, α, and β on participants have a good sensitivity to the mental load level. Because α activity of EEG is obvious when people are awake and need less demand, ⁵⁴ participants with low spatial cognitive ability clearly feel the difficulty of the task, and at this time, their mental need increases, resulting in sleepiness, the main source of the operator’s mental workload is the perception of relative deviation, which requires higher control accuracy, so in the frontal region, α decreased obviously.

Parietal region is the representative region of proprioceptive projection of muscle, which is known as the motor sensory region, it is the representative region of proprioceptive projection of muscle. In the teleoperation process, visual imagery imagination and trajectory prediction are firstly needed, which are a combination of visual imagery and kinesthetic imagery, visual image will significantly activates the visually relevant region and its parietal region, and kinesthetic imagine can activate motor-related regions and the inferior and superior parietal lobules. ⁵⁵ The parietal region contains secondary cortical motor regions, which play an important role in the learning of motor conditions. Participants with high spatial cognition can quickly activate the reflection mechanism of ontology and body surface.

In conclusion, when operators with low spatial cognitive ability are stimulated by external and internal stimuli, the inhibition and excitation in the cortex are in a state of diffusion. The desynchronization of the parietal and frontal regions results in the incoordination of technical actions and the reduction of the accuracy and accuracy of operation. These phenomena are results of the excitation and diffusion of cortical cells in the brain. Therefore, in the future research, the change of EEG signal can also be one of the methods to test the spatial cognitive ability of operators.

In the human-computer-interaction (HCI) system of the space station robot arm, the deep sea detector robot arm, the industrial production robot arm, and so on, the method of the present research can monitor the operator’s EEG changes in real time, determine the operator’s working state, and formulate the operation plan in time. Guide operators to be more refined and efficiently perform teleoperation tasks.

Some limitations should be mentioned. Long-term training will result from learning and training through the functional reorganization of the reticular structure between neurons. When this pathway is retained, it may have an important impact on the motor control mechanism of the brain and cerebellum, but teleoperation performance of analog manipulator reflects the performance of operators at a specific level of proficiency, whose task performance is also theoretically influenced by spatial cognitive ability. Besides, the spatial cognitive ability test in this article has not been repeated, and the use of student participants rather than real operators may limit the result of this study.

Conclusion

Human–machine interaction teleoperation robot telepresence technologies such as force perception, force feedback and large time delay force control, virtual reality technology, and teleoperation robot system design all improve the efficiency and reliability of teleoperation. It provides an effective way to reduce the burden on the operator. This study started with the mental load of the operator’s operator process and the spatial cognitive ability that affects teleoperation and investigated EEG frequency bands of participant in teleoperation. Results show that mental workload of participant with different spatial cognitive abilities change significantly, which can be used as a measurement standard of mental workload of operators. At the same time, those with high spatial cognitive ability have more advantages in working with teleoperation, and participants with low spatial cognitive ability show a regularity change trend in 14 channels of 5 regions of δ, θ, α, β, and γ. In space exploration, ²¹ navigation, ²⁵ and other applications of complex human–machine interaction systems, there are more complex environments and more intensive complex information processing and decision-making needs. More research is needed on assessing operators of varying skill levels as well as with more complicated tasks, combine the spatial cognitive process and related elements in the manipulator task unit, the EEG signals of each sub-module of perception, cognition and operation of operators with different cognitive abilities are analyzed, in addition to dynamic task allocation between human and machine, real-time task allocation among human task team members is also required based on dynamic monitoring operator’s mental workload, these findings can be utilized as a potential indicator, and help choosing suitable operator according to the task difficulty, which can improve the efficiency of human-machine and human-to-human collaboration.