Understanding Reliance Decisions in Automated Vehicles Using Random Forest Analysis

Abstract

Driver reliance on automated vehicles (AV) is a critical component of safety particularly during high-risk traffic scenarios. Factors that influence reliance, including trust, situation awareness, fatigue, and demographics, have been independently explored; however, few analyses have investigated predicting AV reliance and compared factors comprehensively. The goals of this study were to develop a random forest (RF) model to predict reliance and to analyze the importance of factors for reliance decisions. We leveraged data from a driving simulation study where participants encountered four traffic events including responding to an illegal vehicle crossing, managing construction zones, stopping at a vandalized stop sign, and a pedestrian detection task. The dataset included reliance decisions and subjective assessments of dispositional trust, situational trust, fatigue, and workload. An RF model fit to the dataset using cross validation achieved an average AUC of 0.81 and accuracy of 0.77 and situational trust emerged as the most influential predictor.

Automated vehicle (AV) technologies hold significant promise for reducing costs associated with vehicle crashes (Fagnant & Kockelman, 2015). However, this potential hinges on both the adoption and appropriate use of these technologies. The proper use of AVs can be gauged by the reliance on automation, specifically relying on the technology to drive only in conditions where it performs as well or better than a human driver (Banks et al., 2018; Lee & Kolodge, 2020). Despite this, recent high-profile crashes and the limited adoption of automation suggest that human drivers often misuse the technology by relying on it in environments beyond its capability or disuse it by not relying on the automation at all (Parasuraman & Riley, 1997). This central role of reliance on automation necessitates a thorough investigation into reliance decisions and the factors influencing the choice to depend on automation or revert to human control.

Previous research highlights that various human factors may influence reliance on automation (Hopko et al., 2021; Kridalukmana et al., 2020; Lee & See, 2004; Matthews et al., 2019), but comprehensive studies examining these factors in relation to reliance behavior, especially in the context of automated vehicles, remain scarce. Factors such as trust, situation awareness, workload, fatigue, and driver demographics (e.g., age and sex) are frequently cited. This study aims to develop a model that accurately predicts driver reliance behavior based on these comprehensive factors. Using the Random Forest (RF) method to analyze data from a driving simulation study, we aim to answer two critical questions: the effectiveness of the RF algorithm in predicting driver reliance behavior based on human factors data, and the influence of these factors on reliance behavior.

We conducted a driving simulation study involving 49 participants, each engaging in two drives designed to induce variations in reliance behavior through variable automation performance. These drives included four critical traffic events that necessitated reliance decisions. After each event, participants completed a post-event questionnaire assessing situational trust, fatigue, situation awareness, and workload.

The first event occurred within the first 5 min of the initial drive, where an illegally crossing vehicle at an uncontrolled intersection triggered emergency automated responses to avert a collision. In the second event, the automation unnecessarily stopped the vehicle in response to a construction zone, prompting manual intervention. The third event involved a misleading stop sign that the automation ignored. The fourth event required automated braking to prevent pedestrian collisions, testing whether participants would depend on the automation or intervene manually. Participants’ reliance on automation was measured by their choice to rely on the automation to navigate the event or disengage the automated systems in response to these potential hazards.

Using an RF model, we predicted reliance decisions and estimated the importance of each factor in predicting reliance behavior. We employed 10-fold cross-validation and grid search optimization to determine the model’s best hyperparameters. Feature importance was analyzed using Mean Decrease Accuracy (MDA) to identify the most influential predictors. Additionally, an ANOVA analysis was conducted to assess significant differences in situational trust sub-scales (Holthausen et al., 2020) between reliance and non-reliance behaviors, providing further insights into the role of situational trust on reliance decisions.

The analysis revealed that the RF model achieved an average AUC score of 0.81 and an accuracy of 0.77, indicating its effectiveness in distinguishing between reliance and non-reliance behaviors. Situational trust emerged as the most important feature (importance score of 0.142), followed by age (0.092), driving experience (0.079), mental workload (0.079), situational awareness (0.074), fatigue (0.072), dispositional trust (0.064), and sex (0.019).

Our ANOVA analysis found significant differences across several situational trust sub-scales between reliance and non- reliance behaviors. Specifically, STSAD-Judgement, STSAD-Performance, STSAD-Risky, and STSAD-Trust sub-scales showed significant differences (p < .001) between the two behaviors. Visualizations and coefficients for each scale with significant differences revealed a positive correlation between the scores of these scales and reliance behavior, implying that situational trust importance is linked to the driver’s specific perception of the AV’s judgment, performance, and their perceived risk and trust in the AV.

This study underscores the complexity of driver reliance on AV technology and highlights the critical role of situational trust. The findings suggest that accurate predictions of reliance behavior can be made using human factors data, with situational trust being the most influential factor. These insights are crucial for developing strategies to improve the adoption and appropriate use of AV technology, ultimately enhancing road safety and operational efficiency.