Cognitive Workload Evaluation of Onsite Workers Collaborating with a Teleoperated Robot during Assembly Tasks Using Heart Rate

Abstract

Incorporating collaborative robots in modern industry has given rise to the synergistic collaboration between humans and robots. Working along with a teleoperated robot enables completion of tasks across geographical boundaries and has the potential to address the skills gap in manufacturing by enhancing future distributed manufacturing environments. In this study, we investigated cognitive demands of onsite workers in two different collaboration scenarios: (1) Human-Human (HH) and (2) Teleoperator-Robot-Human (tRH), during wire assembly tasks with varying physical and cognitive workloads. Under an additional cognitive task condition, the perceived effort (measured by NASA-TLX) for HH (57.8 ± 26.4) was significantly higher than for tRH (31.6 ± 26.5). However, no significant differences in heart rate were found between HH and tRH. These findings suggest that tRH might not introduce additional cognitive demands compared to HH collaboration, despite the lack of in-person and/or less direct communication.

Industry 4.0 has ushered in a new era of intelligent, automated, and decentralized manufacturing that merge physical and virtual workplaces. The adoption of collaborative robot technology has become increasingly prevalent in modern industry to enhance and support distributed decentralized manufacturing processes and operations (Bragança et al., 2019; Sherwani et al., 2020). This has given rise to human-robot collaboration (HRC), a paradigm where humans and robots work together in a synergistic relationship to improve human well-being and productivity (Ajoudani et al., 2018; Fryman & Matthias, 2012; Matheson et al., 2019). In this context, robots are often assigned to tasks that involve high repetition or physical demands, even in challenging conditions or hard-to-reach areas. Humans thus can concentrate on complex task planning and problem-solving skills. Additionally, HRC can alleviate labor shortages by providing opportunities for underrepresented groups such as women, disabled individuals, and the elderly to participate in the workforce.

Teleoperation, a type of human-robot interaction, involves remote operation of a robot by a human operator, enabling the completion of tasks irrespective of geographical distances (Murphy & Rogers, 1996). Despite the growing importance of HRC, a limited attention has been given to the broad context of HRC in teleoperation—specifically, interactions and partnerships among the teleoperator, robot, and onsite worker(s) (Lim et al., 2024). This collaborative model, termed teleoperator-Robot-Human (tRH) collaboration hereafter, has a potential to address the skills gap in manufacturing and further enhance distributed manufacturing environments. This particular collaboration setting, however, may introduce cognitive workload challenges for onsite workers due to the communication complexity in a virtual workplace. Coordinating actions, sharing and interpreting information remotely can be more challenging than in-person interactions. Further, onsite workers may need to adapt to the constraints associated with robots’ movements especially for tasks requiring fine motor skills.

Onsite workers operating under high cognitive workload, nearing their cognitive capacity, are prone to suboptimal decisions and human errors. Conversely, low cognitive workloads may lead to errors due to boredom and possible environmental distractions (Fernandez Rojas et al., 2020). Hence, it is crucial to understand a worker’s cognitive workload in order to optimize the utilization of their mental resources during tRH tasks. Nevertheless, the research focusing on identifying the cognitive workloads of onsite workers, specifically in tRH collaboration has been limited. Therefore, this exploratory study examined changes in the onsite worker’s cognitive workload during tRH collaboration. Assessing cognitive workloads quantitatively can be achieved using subjective measurements, such as NASA-TLX questionnaire (Hart & Staveland, 1988) or objective measurements by analyzing their physiological responses (e.g., heart rate, skin conductance response). Heart rate (HR) can be considered as a reliable indicator of cognitive workload due to its responsiveness to the autonomic nervous system, which changes based on the cognitive demands exerted by an individual (Jorna, 1992; Roscoe, 1992). In this study, we investigated cognitive demands of onsite workers using NASA-TLX subscales and HR in two different collaboration scenarios; (1) Human-Human (HH) and (2) teleoperator-Robot-Human (tRH). The HH collaboration solely consisted of an onsite worker and teleoperator, without the involvement of a teleoperated robot. Whereas, tRH scenario included a teleoperated robot that provided physical assistance to the onsite worker.

A convenience sample of 32 gender-balanced participants [mean (SD) age: 25 (2.9) years] were recruited for the study. Participants were paired and randomly assigned to either an onsite or a teleoperator role, in a between-subject design assignment for HH or tRH collaboration. The experimental setup included a wooden wall with randomly placed holes and a middle shelf, a robot arm (Research 3, Franka Emika) and camera views for teleoperators. The onsite participant and the robot were separated by the wooden wall, allowing interaction solely through the exchange of materials via holes. Communication between onsite participants and teleoperators occurred verbally through headsets.

The onsite participant performed two tasks: primary and cognitive. The primary task involved simulated repair tasks, requiring completing a wire assembly, with step-by-step instructions provided verbally by teleoperators. The cognitive task involved memorization. Three independent variables were manipulated: (1) Physical demand, operationalized by the assembly task location, either above or lower the middle shelf; (2) Cognitive demand, with or without the cognitive task of attention-based random sequence memorization; (3) Collaboration type (HH vs. tRH). Subjective measures of mental and physical demands were obtained using NASA-TLX. HR data were obtained using an EmbracePlus smartwatch (Empatica, frequency = 0.02 Hz), worn on the participant’s non-dominant hand. A baseline HR was established during a 2-min rest condition at the start of the experiment. Separate ANOVA tests were performed on HR with baseline as a covariate and subjective scores; Gender was included as a blocking variable. Student’s t-test was used for post-hoc analysis (p < .05).

A main effect of Cognitive (p < .001) was found for HR where participants showed greater HR with the cognitive task. Significant first-order interactions (all p < .05) were observed among Physical demand, Collaboration type, Cognitive demand, and Gender. HR was significantly greater for the more physically demanding condition (i.e., below the middle shelf) within HH (p = .042) and tRH (p = .017) conditions. For the HH condition, presence of cognitive task had a significantly higher (p < .001) HR independent of physical demand conditions. Females (vs. males) had a higher HR (p < .001) in the presence of cognitive task and with less demanding physical condition (p = .033). Whereas a higher HR in males (vs. females) was found with the more demanding physical condition (p = .010). Significant main Cognitive effects were found on some NASA-TLX subscales, including Mental demand (p < .001), Temporal demand (p < .001), Effort (p = .006), and Frustration (p = .003). Presence of cognitive task had a greater score than its absence. No significant main and interaction effects were found for NASA-TLX Physical demand and Performance. In effort, a significant interaction effect was found for Collaboration × Cognitive. It was observed that without the cognitive task, effort of HH (57.8 ± 26.4) was reported significantly higher than tRH (31.6 ± 26.5).

Our lab-based study, which involved simulated manufacturing repair tasks, explored the cognitive workload experienced by onsite workers in collaborative scenarios within a distributed industrial setting. Our findings suggest a positive correlation between HR and perceived cognitive demand levels. Yet, no significant differences were found between HH and tRH conditions for HR, suggesting that incorporating a teleoperated robot into the collaborative process might not introduce additional cognitive challenges, despite the lack of direct, in-person communication.

However, for subjective measurements, there were significant differences found between HH and tRH conditions for Effort. A high effort value indicates that participants perceived the HH task to require substantial mental and/or physical energy expenditure compared to tRH. Although tRH generally reported a lower score than HH collaboration, we did not find any significant differences between them in the subjective scores of Mental, Physical, Temporal, Performance, and Frustration subscales. In the future, we plan to analyze differences based on other objective physiological measures such as electrodermal activity, skin conductance temperature, blood volume pressure (Charles & Nixon, 2019). We will also focus on enhancing the design of virtual workplaces for reducing the onsite participant’s mental and physical demands.