Exploring Cognitive Workload Changes in Teleoperators: Collaboration with On-Site Workers and Robots in Assembly Tasks

Abstract

Teleoperation is commonly employed to perform industrial tasks in remote or inaccessible areas. However, there is a noticeable gap in evaluating cognitive workload of teleoperators in collaboration environments. This study compared the variation of cognitive workload for teleoperators guiding an on-site participant to complete a wire assembly task in two scenarios: one aided by a robot arm (tRH) and another without any robot assistance (HH). Additionally, the task demands for on-site participants were manipulated to measure its impact on teleoperator’s workload. NASA-TLX and EEG activity were utilized to assess workload. The results indicated that EEG theta activity was significantly higher for the HH group than tRH group, potentially showing lower workload for teleoperators in the scenario with robot assistance. Task difficulty did not affect any of the workload measures. The study highlights the importance of cognitive workload assessment in human-robot collaborations to optimize human cognitive demands in complex settings.

Driven by advances in communication and robotic technologies, teleoperation—enabling robots to be operated remotely—holds potential for broader adoption in the industrial sector, especially in the wake of the post-COVID era (Hu et al., 2022). This mode of collaboration (between human operators and robot systems) can enable the execution of industrial tasks across long distances and/or in environments that are inaccessible or hazardous to humans (Guo et al., 2022; Ito et al., 2021). However, teleoperated robots can present challenges, particularly in terms of the cognitive load on teleoperators who are inexperienced (Labonte et al., 2010).

In potential work scenarios that involve collaboration between a teleoperator and an on-site worker operating alongside a teleoperated robot, the complexity of the task can vary. This variation may require adaptations by the teleoperator to accommodate the physical and/or cognitive demands of the on-site worker, potentially leading to an increased cognitive load for the teleoperator. However, existing research has paid limited attention to such collaborative scenarios, particularly regarding the cognitive workload of teleoperators in the presence of on-site workers. Consequently, monitoring the workload of teleoperators can be imperative, given that both excessive and insufficient cognitive loads have been observed to impede performance.

While previous research has explored the use of physiological measures such as eye tracking, heart rate and electroencephalogram (EEG) to assess the cognitive workload of teleoperators (Fernandez Rojas et al., 2020; Nenna et al., 2023), there remains a notable gap in evaluating the cognitive workload of teleoperators within collaborative settings. Also, it remains uncertain if elevated task demands for the on-site participant can impose higher workload for the teleoperator guiding the onsite participant in completing the task. Therefore, as an initial step toward a more comprehensive understanding of the collaboration between a teleoperator and an on-site worker, this exploratory study investigated changes in the teleoperator’s cognitive load during such collaboration. Specifically, we explored scenarios in which a teleoperator, equipped with the expertise to complete an assembly task remotely, provides instructions to an on-site worker performing the task. The teleoperator’s involvement ranged from offering solely verbal instructions to providing a combination of verbal instructions and direct physical assistance via a teleoperated robot. Meanwhile, the on-site worker performed the task under different levels of physical and/or cognitive demands. To assess the cognitive workload experienced by the teleoperator, we used both objective and subjective measures. For the subjective measurement NASA-TLX questionnaire (Hart & Staveland, 1988) were employed and for the objective measurement, real time EEG data were collected from the participants and were processed to obtain power at theta and alpha band as the metrics of cognitive workload. Overall, this study aims to increase an understanding of the dynamic cognitive challenges faced by teleoperators in collaborative settings and to inform the development of strategies that can optimize teleoperation practices for enhanced efficiency and safety.

A convenience sample of 32 gender–balanced participants (mean age: 25 ± 2.9 years) completed the study where they performed a wire assembly task. The task involved placing pins into randomly arranged holes on a wooden wall and attaching wires to these pins following a specific plan. Participants were paired and randomly assigned roles as either teleoperators or on-site operators, with the workplace physically separated. The teleoperator provided verbal guidance to the on-site operator utilizing six cameras placed around the on-site environment.

We employed a between-group design, assigning participant pairs to one of two collaboration scenarios: Human-Human (HH) or teleoperator-Robot-Human (tRH). In the HH scenario, the teleoperator guided the on-site operator verbally through the assembly task without any teleoperated robot assistance. Conversely, in the tRH scenario, the teleoperator provided verbal instructions and additionally operated a robot arm to offer physical assistance to the on-site operator. Task conditions for on-site operators were manipulated to create low and high cognitive loads. In the former condition, the on-site operator performed a secondary cognitive task, whereas in the latter condition, no additional cognitive task was performed.

EEG data were recorded from the teleoperators. Also, teleoperators reported their perceived workload using the NASA-TLX questionnaire. We preprocessed raw EEG data and calculated normalized power in theta and alpha bands for each trial. Between-subject ANOVA tests were performed on normalized theta and alpha power and mean NASA-TLX scores. Statistical significance was determined when p < .05.

There were no significant main and interaction effects of Collaboration condition and Task difficulty on NASA-TLX score. Regarding the EEG normalized power values, we found a significant effect of Collaboration condition on normalized theta (p = .009), where the normalized theta power in HH was 58.2% higher than normalized theta in tRH. While the mean normalized alpha power was 32% higher for HH than tRH, there were no significant main or interaction effects observed for the two independent variables on normalized alpha power.

The lack of significant differences in NASA-TLX scores between two conditions (HH and tRH) might suggest that the subjective measure of workload was not sensitive enough to discriminate between the two collaboration conditions. However, theta activity was higher for the teleoperators when collaborating without the robot, suggesting potential high cognitive demand among teleoperators when assisting on-site operators without using the robot for physical assistance (Borghini et al., 2012). Conversely, the presence of the robot may have mitigated this challenge, as the teleoperator could carry out some aspects of the task by operating the robot, which may not be overly demanding for them.

Finally, the lack of significant differences for the task difficulty condition on each of the NASA-TLX and EEG frequency bands imply that engaging in secondary tasks by the on-site participant did not substantially impact the cognitive workload experienced by the teleoperators. In addition, the results showed that normalized theta EEG power can be more sensitive than NASA-TLX in identifying the subtle differences in cognitive workload in the teleoperator-robot-human collaboration assembly tasks (Cheng, 2018). Overall, our findings showed the interconnected nature of cognitive workloads within collaborative environments, potentially suggesting that optimizing teleoperation strategies must consider not only the specific tasks at hand but also the cognitive demands placed on both teleoperators and on-site workers.