Simultaneous Remote Monitoring of Multiple Automated Vehicles

Abstract

As vehicle automation continues to become a more prevalent part of everyday life, remote operators may be able to contribute to making this technology safer and more efficient. Such operators could be expected to monitor vehicles until it is necessary for them to intervene. In recent years several countries have passed legislation mandating the implementation of remote operation, making it a plausible scenario. For many practical reasons, operators may monitor more than one vehicle. This study provides an initial look into how remote operators might perform when asked to attend to multiple concurrent displays using simulated automated vehicles. The initial results indicate that participants retained a comparable level of accuracy in detecting failures in automation as the number of vehicles they monitored rose from one to two, albeit with a higher workload. The results and the methodology provide insights into potential task designs for remote operators of automated vehicles.

Objectives

Until we achieve high or full automation, human operators will continue to be required in cases of automation failure (SAE International, 2021). Such an operator could be a person inside the vehicle (i.e., driver) or a person remotely monitoring and controlling the vehicle (i.e., remote operator). Remote operation refers to the use of a remote system to interact with and manipulate distant objects (Kebria et al., 2018). When considered for use alongside vehicle automation, remote operation presents a solution for instances where an in-vehicle fallback driver may perform poorly or when no one in the vehicle can serve as a fallback driver. In such a system, a remote operator might closely monitor one or more vehicles until a potential hazard or automation failure is identified, assuming control of the vehicle until it is safe to re-engage automation. Similar systems involving remote operators as fallback drivers have been proposed (Neumeier et al., 2018). Recognizing the potential of remote operation, several countries and regions have mandated its use with automated vehicles (Juliussen, 2021).

Despite the unique technical challenges of remote operation (e.g., latency issues, Chen et al., 2007), it presents potential advantages such as a long trip can be broken into shorter tasks monitored by different operators to mitigate fatigue and vigilance decrement, and an operator could monitor more than one vehicle at once. Indeed, one-to-N operation has been proposed in other domains such as robot operation (Olsen & Woods, 2004) and missile management (Cummings & Guerlain, 2007), and it has also been noted as a potential scenario in the automated vehicle context (Zhang, 2020).

While monitoring multiple vehicles may sound counterintuitive as the task should not overload a remote operator, it could be a solution to the underload problem that we observe from drivers of automated vehicles. Under partial automation, drivers already experience boredom and passive fatigue (Körber et al., 2015) and have been observed to engage in a significant amount of non-driving-related activities (de Winter et al., 2014). This will likely further exacerbate with a higher level of automation and with the increasing reliability of automated vehicles as failures could become less prevalent. Monitoring more than one vehicle may increase the workload to a more desired level; however, this needs to be empirically examined to ensure this does not turn into an overload problem. Furthermore, the task duration that leads to vigilance decrement under different monitoring load conditions (i.e., number of vehicles) also needs to be quantified, as drivers would likely be expected to monitor vehicles over a sustained duration of time.

This study serves as a preliminary step to examine how participants attend to concurrent displays using videos of simulated automated vehicles. Effects of the number of concurrent vehicles on participants’ performance on monitoring and takeover are investigated. Moreover, we also examine how participants’ monitoring performance and behavior change over time.

Approach

A total of seven undergraduates (mean age 21.1 years, three females, all drivers) participated in the study at the time of this article submission. The study is ongoing. The seven participants completed a remote monitoring task in a laboratory set-up with a 27-inch monitor. The viewing distance was controlled at 22 inches via a headrest.

In the remote monitoring task, participants were asked to assume the role of remote operators tasked with monitoring either one, two, or four displays of simulated automated vehicles simultaneously (Figure 1), pressing a button to indicate takeover when an automation failure is spotted. Each task session took about 30 min total, with individual trials ranging from 32 to 125 s. The individual displays with automation failures were chosen from a prior study (Morrison et al., 2022) with matched difficulty among one, two and four display conditions. Once participants pressed the takeover button, they simply reported the hazard type and which displayed vehicle it was observed in (if multiple vehicles). While every trial contained an automation failure event, participants were told that failure may or may not occur to minimize their expectations.

Figure 1.

Examples of each vehicle display condition.

The monitoring task included a total of 18 trials, organized into three blocks of six trials each. Each block had an equal representation of each number-of-display condition and the trial order within blocks was randomized. Within-subject factors were number of vehicles (1, 2, 4) and block (first, second, third). We measured participants’ takeover accuracy, cognitive workload via an adapted scale based on the NASA TLX (Hart & Staveland, 1988). A takeover was coded as correct if a button press took place within the specified event response time window(s) and the participant correctly identified the event and the display.

Findings

The effects of vehicle number (1, 2, 4) and block (first, second, third) on takeover performance and workload were examined. Preliminary findings suggest a significant block effect with takeover accuracy decreasing over time, and also a significant effect of vehicle number with takeover accuracy being worse in the four-vehicle condition compared to the one- or two-vehicle conditions. Workload increased with the number of vehicles presented simultaneously but not over blocks. Specific results are described below.

For takeover accuracy, 3 × 3 repeated measures ANOVAs were conducted with the number of vehicles and blocks being two within-subject variables on three types of participant responses: misses, false alarms, and correct takeovers. No response until the end of a video stimulus was coded as a miss, and a takeover response in non-hazardous situations was coded as a false alarm. For correct responses, participants made fewer correct responses with more vehicles to monitor (F(2,12) = 30.47, p < .001) and over the blocks (F(2,12) = 4.96, p = .027). Subsequent pairwise comparisons indicated similar performance between one- and two-vehicle conditions (p = .522), and between the second and third blocks (p = .788). Among the incorrect takeover responses, both misses and false alarms were higher in the four-vehicle condition than in the one- (misses: p < .001; false alarms: p = .004) or two-vehicle conditions (misses: p = .004; false alarms: p = .005). Misses did not increase over blocks (F(2,12) = 0.55, p = .593) but false alarms did (F(2,12) = 3.95, p = .048).

For workload, NASA TLX scores were collected after each block and at the end of the study retrospectively on the three vehicle number conditions. Results from repeated measures ANOVAs suggest progressively increasing workload with more vehicles (F(2,12) = 93.92, p < .001), but not over blocks (F(2,12) = 0.39, p = .687).

Takeaways

Primary takeaways from this study provide a foundational insight into how remote operators’ capacity to recognize and respond to events requiring takeover may be affected by the number of automated vehicles presented as well as their time on task. Whether monitoring one vehicle or two simultaneously, participants maintained similar accuracy in recognizing automation failures, though at the cost of increased workload. Performance was only impaired once four vehicles were presented concurrently, suggesting it may be possible for trained operators to monitor more than one vehicle effectively but four may be too many. Regarding time on task, fewer correct takeovers and more false alarms were observed in the second and third blocks when compared to the first. This may suggest a harmful effect of fatigue or vigilance decrement, although participants’ self-reported workloads seemed to remain stable over time. This may be a relevant concern when determining how long remote operators should be expected to monitor.

Methodological factors limiting the ecological validity of this study’s findings include the timing of hazardous events, the measure of takeover effectiveness, and the prevalence of hazardous events throughout the experiment. In this study, participants pressed a button to indicate takeover. If differential control devices (e.g., a driving wheel or path drawer), takeover success should be assessed with more comprehensive methods. Furthermore, all trials contained an event requiring a takeover, which may not reflect the rate of similar events occurring in reality. Future research could address the effects of operator training, lower prevalence of hazards, and longer trial durations to explore differential applied contexts.