Sage Journals: Discover world-class research

Abstract

The development of autonomous vehicles presents significant challenges, particularly in predicting pedestrian behaviors. This study addresses the critical issue of uncertainty in such predictions by distinguishing between aleatoric (intrinsic randomness) and epistemic (knowledge limitations) uncertainties. Using evidential deep learning (EDL) techniques, we analyze these uncertainties in two key pedestrian behaviors: road crossing and short-term movement prediction. Our findings indicate that epistemic uncertainty is consistently higher than aleatoric uncertainty, highlighting the greater difficulty in predicting pedestrian actions due to limited information. Additionally, both types of uncertainties are more pronounced in crossing predictions compared to destination predictions, underscoring the complexity of future-oriented behaviors. These insights emphasize the necessity for AV algorithms to account for different levels of behavior-related uncertainties, ultimately enhancing the safety and efficiency of autonomous driving systems. This research contributes to a deeper understanding of pedestrian behavior prediction and lays the groundwork for future studies to explore scenario-specific uncertainty factors.

Keywords

AI in human factors pedestrian behaviors uncertainty estimation autonomous driving

Introduction

The rise of autonomous driving technology promises to transform urban mobility by making it safer, more efficient, and more convenient. Among the many challenges in developing autonomous vehicles (AVs), predicting pedestrian behavior is one of the most complex and crucial tasks. This complexity comes from the unpredictable nature of human behavior, which means AVs must not only react to their immediate surroundings but also anticipate what pedestrians might do next (Rasouli & Tsotsos, 2019). Accurate prediction of pedestrian behavior is essential for the safe and effective operation of AVs.

Pedestrian behavior prediction includes tasks such as determining if someone will cross the street, forecasting their future paths, predicting immediate actions, and estimating their movement directions (Deb et al., 2017). Despite advances in AI and deep learning, current models still do not achieve the accuracy needed for reliable AV decision-making. A significant barrier to achieving higher accuracy is the uncertainty inherent in human behavior predictions.

Uncertainty in pedestrian behavior can be divided into two main types: aleatoric and epistemic (Sensoy et al., 2018). Aleatoric uncertainty refers to the inherent randomness and variability in pedestrian behavior. This type of uncertainty captures the natural unpredictability of pedestrians’ decisions and actions. Epistemic uncertainty, on the other hand, comes from the AV’s lack of knowledge or incomplete information. While many studies discuss the uncertainty in behavior prediction, few compare the uncertainties associated with different pedestrian behaviors (Zhang et al., 2023). Additionally, there is limited research on the challenges of computing the uncertainties. Understanding these uncertainties is key to improving AV decision-making systems.

By distinguishing these uncertainties, we aim to make the model’s predictions more understandable and provide valuable insights into pedestrian behavior to promote trust in AV (Zhang et al., 2022). This research focuses on analyzing uncertainties in two specific behaviors: crossing the road and choosing short-term moving positions. The first behavior is future-oriented and may happen several seconds ahead, while the second is present oriented and occurs within a second. We aim to answer two main questions: (a) How are aleatoric and epistemic uncertainties distributed when predicting pedestrian behaviors? This question helps to understand the levels of uncertainty within the behaviors and the extra uncertainties due to limited external information. (b) What is the relationship between uncertainties and different levels of pedestrian behaviors? This question explores whether more complex behaviors, like crossing the road, increase inherent uncertainty and prediction difficulty.

By exploring these questions, this study aims to advance autonomous driving by using uncertainties as valuable features to improve AV decision-making. This approach not only helps AVs interact better with pedestrians but also supports the larger goal of integrating AVs safely and effectively into urban environments. Through this research, we hope to take steps toward a future where humans and machines coexist harmoniously in cities.

Literature Review

Predicting pedestrian behavior is a critical aspect of autonomous driving research, and a variety of approaches have been developed over the years (Papadimitriou et al., 2016). As machine learning techniques evolved, researchers began to employ more sophisticated model to improve prediction accuracy (Gupta et al., 2018; Zhang et al., 2022; Zhang et al., 2023). Recurrent Neural Networks have become popular due to their ability to capture temporal dependencies (Rasouli et al., 2017; Song et al., 2020).

Despite these advancements, predicting pedestrian behavior with high accuracy remains challenging due to the inherent uncertainties in human actions. Uncertainty quantification has therefore become an essential component of modern prediction models (Grote, 2014; Kunze et al., 2019). Aleatoric uncertainty, which represents the inherent variability in pedestrian behavior, has been addressed in several studies. For instance, Kendall and Gal introduced a Bayesian deep learning framework that incorporates aleatoric uncertainty in the prediction of segmentation tasks (Kendall & Gal, 2017).

Epistemic uncertainty, which arises from the model’s lack of knowledge, is equally important. Gal and Ghahramani proposed a method to estimate epistemic uncertainty using Monte Carlo dropout, which involves performing multiple forward passes with dropout enabled at test time (Gal & Ghahramani, 2016). Recent research has started to explore the combination of both types of uncertainties in autonomous driving (Michelmore et al., 2020). McAllister et al. proposed a model that integrates aleatoric and epistemic uncertainties to improve the robustness of predictions in autonomous driving scenarios (McAllister et al., 2017). Their approach demonstrates that considering both uncertainties can lead to more reliable and interpretable predictions.

Despite these efforts, there is still a lack of studies that compare the uncertainties associated with different pedestrian behaviors and the challenges in computing them. This research fills this gap by analyzing the distribution and correlation of aleatoric and epistemic uncertainties in two specific pedestrian behaviors: road crossing and short-term movement prediction.

Method

Dataset

We utilize the PIE dataset, a common benchmark for egocentric view pedestrian behavior prediction (Rasouli et al., 2019). The PIE dataset contains annotated videos of pedestrians captured from the perspective of a moving vehicle, providing rich contextual information for behavior prediction tasks. The dataset includes two specific tasks relevant to our study: Crossing Prediction: Predicting whether a pedestrian will cross the road within a specified time window. Destination Prediction: Predicting the pedestrian’s short-term destination by classifying their position into one of two regions divided by the center line of the x-axis. The PIE dataset offers a variety of scenarios that capture the diverse and dynamic nature of pedestrian behaviors in urban environments. This diversity is crucial for training models that generalize well across different contexts and situations.

Model Architecture

Our model architecture is designed to handle the complexities of pedestrian behavior prediction and uncertainty estimation. We employ Convolutional Neural Networks (CNNs) to extract spatial features from the video frames and Long Short-Term Memory (LSTM) networks to capture temporal dependencies. The EDL framework is integrated into the model to estimate both aleatoric and epistemic uncertainties (Sensoy et al., 2018). The CNN layers are responsible for processing the raw image data, extracting relevant features such as pedestrian positions, movements, and interactions with the environment. These features are then fed into LSTM layers, which model the temporal dynamics of pedestrian behavior. The combination of CNN and LSTM networks allows the model to learn both spatial and temporal patterns, which are crucial for accurate behavior prediction.

In the context of EDL, the model outputs are interpreted as evidence supporting each possible class. This evidence is used to calculate the model’s uncertainty regarding its predictions: Aleatoric Uncertainty: This is calculated by measuring the corrected distance of the evidence in probability to the decision boundary, which is set at 0.5 for binary classification tasks. Aleatoric uncertainty accounts for the inherent variability and unpredictability in pedestrian behavior. Epistemic Uncertainty: This is calculated by dividing the outputted evidence for a class by the sum of the evidence plus a measure of randomness. High epistemic uncertainty indicates a lack of knowledge in the prediction. In our application, this measure helps identify scenarios where the AV’s model is unsure about a pedestrian’s behavior due to limited or ambiguous information.

Training and Evaluation

The model is trained on the PIE dataset using a combination of standard classification loss and evidential loss, which balances the trade-off between prediction accuracy and uncertainty estimation. The training process involves optimizing the model parameters to minimize the evidential loss function, which is defined as:

ℒ_{E D L} = \sum_{i = 1}^{N} (\frac{{(y_{i} - {\overset{⌢}{y}}_{i})}^{2}}{2 σ_{i}^{2}} + \frac{1}{2} \log σ_{i}^{2})

where y_i is the true label, ŷ_i is the predicted label, and ŷ_i² is the estimated aleatoric uncertainty for each prediction i. The model’s performance is evaluated using standard metrics such as accuracy and F1 score for both crossing and destination prediction tasks. Additionally, we analyze the distribution patterns of aleatoric and epistemic uncertainties across both tasks to understand their impact on prediction performance. The evaluation process also includes analyzing the receiver operating characteristic (ROC) curves and area under the curve (AUC) to assess the model’s discriminative ability.

Uncertainty Analysis

To analyze the uncertainties, we calculate the mean and standard deviation of both aleatoric and epistemic uncertainties for each task. We also perform paired t-tests to determine the statistical significance of the differences in uncertainties between crossing and destination predictions. The results of these analyses provide insights into the inherent variability in pedestrian behavior and the challenges faced by the AV’s perception system.

Furthermore, we apply k-means clustering to categorize prediction cases based on their uncertainty profiles. Principal Component Analysis (PCA) is used for dimensionality reduction, allowing us to visualize the diverse uncertainty profiles across different clusters. The clustering results are further analyzed to understand the common characteristics of scenarios with high uncertainty. For instance, we examine the influence of environmental factors such as lighting conditions, occlusions, and pedestrian density on the uncertainty profiles. This analysis helps identify the key factors that contribute to prediction uncertainty and provides guidance for improving model performance in challenging scenarios.

Results

Model Performance

The model’s performance was evaluated on two tasks: destination classification and pedestrian crossing prediction. The results are summarized in Table 1. The destination classification task achieved an accuracy of 0.89 and an F1 score of 0.89, while the crossing prediction task recorded an accuracy of 0.91 and an F1 score of 0.90. These results demonstrate the effectiveness of our model in predicting pedestrian behaviors.

Table 1.

Performance Metrics for the Two Prediction Tasks.

Task	Accuracy	F1 score
Destination classification	0.89	0.89
Crossing prediction	0.91	0.90

Data Analysis

The distribution patterns of epistemic and aleatoric uncertainties were analyzed for both prediction tasks. Figures 1 and 2 show the distributions of epistemic and aleatoric uncertainties for the destination prediction task, respectively. Similarly, Figures 3 and 4 show the distributions of epistemic and aleatoric uncertainties for the crossing prediction task.

Figure 1.

Epistemic uncertainty distribution for destination prediction: The x-axis represents the uncertainty values, while the y-axis denotes the probability.

Figure 2.

Aleatoric uncertainty distribution for destination prediction: The x-axis represents the uncertainty values, while the y-axis denotes the probability.

Figure 3.

Epistemic uncertainty distribution for crossing prediction: The x-axis represents the uncertainty values, while the y-axis denotes the probability.

Figure 4.

Aleatoric uncertainty distribution for crossing prediction: The x-axis represents the uncertainty values, while the y-axis denotes the probability.

The averages and standard deviations of uncertainties for each task are summarized in Table 2.

Table 2.

Averages and Standard Deviations of Aleatoric and Epistemic Uncertainties for Crossing and Destination.

Uncertainty	Average	Standard deviation
Aleatoric (crossing)	0.26	0.28
Epistemic (crossing)	0.65	0.15
Aleatoric (destination)	0.17	0.27
Epistemic (destination)	0.53	0.20

The analysis revealed that within each task, aleatoric uncertainty is significantly lower than epistemic uncertainty, indicating that the inherent randomness or variability in pedestrian behavior is much less than the uncertainty associated with predicting this behavior. Paired t-tests confirmed the statistical significance of these differences: crossing prediction (t = 163.30, p < .001) and destination prediction (t = 186.32, p < .001).

Additionally, both aleatoric and epistemic uncertainties associated with crossing predictions were significantly larger than those for destination predictions. The statistical significance of this difference was underscored by paired t-tests: aleatoric uncertainty (t = 18.21, p < .001) and epistemic uncertainty (t = 40.00, p < .001). These results suggest that crossing, a more complicated behavior, has higher inherent uncertainties and is more challenging to predict.

Figure 5 shows the correlation matrix visualization, which highlights the relationships between different uncertainties for the two tasks. The correlation analysis revealed a strong positive correlation (.98 for crossing and .95 for destination) between the types of uncertainties within each task, indicating that higher unpredictability in pedestrian behavior correlates with increased prediction uncertainty. Additionally, a moderate positive correlation of approximately .5 between tasks suggests a shared underlying complexity in predicting pedestrian behaviors across different scenarios.

Figure 5.

Correlation matrix visualization. This heatmap illustrates the pairwise correlation coefficients between different uncertainties of two tasks, highlighting the strength and direction of relationships.

To better understand the effects of scenarios on behavior uncertainties, we applied k-means clustering on two types of uncertainties, categorizing all cases into four groups. This categorization was informed by the elbow method. After applying Principal Component Analysis (PCA) for dimension reduction, we visualized the diverse uncertainty profiles across the clusters, as shown in Figure 6.

Figure 6.

Cluster plots after PCA dimensionality reduction to 2D. The red crosses represent cluster centers, and each color denotes a different cluster. The first principal component and second explains 75% and 23.41% of the variance, respectively.

Discussion

Our analysis revealed that epistemic uncertainty consistently exceeded aleatoric uncertainty across both tasks. This suggests that the primary challenge in predicting pedestrian behavior lies not in the inherent variability of the behavior itself but in the limitations of the model’s knowledge and information. High epistemic uncertainty indicates areas where the model lacks sufficient data or has encountered scenarios that are significantly different from those in the training set. This finding underscores the importance of improving data diversity and model robustness to reduce epistemic uncertainty and enhance prediction accuracy. In the crossing prediction task, both aleatoric and epistemic uncertainties were higher compared to the destination prediction task. This aligns with our hypothesis that more complex behaviors, such as deciding to cross the road, involve greater uncertainty due to the multitude of factors influencing such decisions. These factors may include pedestrian intention, environmental context, and interactions with other road users, all of which contribute to higher prediction uncertainty.

The strong positive correlation between aleatoric and epistemic uncertainties within each task suggests that scenarios with inherently unpredictable pedestrian behavior also tend to be those where the model’s knowledge is limited. This finding is crucial for developing more effective pedestrian behavior prediction models, as it highlights the need to address both types of uncertainties simultaneously. The moderate positive correlation between uncertainties across different tasks indicates a shared underlying complexity in predicting various pedestrian behaviors. This suggests that improving the model’s performance in one task could potentially benefit other related tasks (e.g., destination prediction) by addressing common sources of uncertainty.

Understanding the distribution and sources of uncertainties in pedestrian behavior prediction has significant implications for AV decision-making. Higher uncertainties in crossing predictions, for instance, indicate that AVs should exercise greater caution when approaching pedestrians near crosswalks or intersections. By incorporating uncertainty estimates into the decision-making process, AVs can better assess the risk associated with their actions and make more informed decisions to enhance safety.

Limitations

There are a few limitations to our study. The PIE dataset, while comprehensive, may not cover all possible pedestrian behaviors and environmental contexts. Therefore, the generalizability of our findings to other datasets or real-world scenarios may be limited. Additionally, the model’s performance is constrained by the quality and diversity of the training data. Future studies should aim to validate these findings using larger and more diverse datasets.

In conclusion, our study underscores the importance of understanding and addressing both aleatoric and epistemic uncertainties in pedestrian behavior prediction. By improving data diversity, model robustness, and integrating uncertainty estimates into AV decision-making, we can enhance the safety and efficiency of autonomous driving systems. These findings provide a foundation for future research aimed at reducing uncertainties and advancing the field of autonomous vehicle technology.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is supported by the National Science Foundation under Grant No. 2145565.

ORCID iD

Renran Tian

References

Deb

Strawderman

DuBien

Smith

Carruth

D. W.

Garrison

T. M.

(2017). Evaluating pedestrian behavior at crosswalks: Validation of a pedestrian behavior questionnaire for the us population. Accident Analysis & Prevention, 106, 191–201.

Gal

Ghahramani

(2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International conference on machine learning (pp. 1050–1059).

Grote

(2014). Adding a strategic edge to human factors/ergonomics: Principles for the management of uncertainty as cornerstones for system design. Applied Ergonomics, 45(1), 33–39.

Gupta

Johnson

Fei-Fei

Savarese

Alahi

(2018). Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2255–2264).

Kendall

Gal

(2017). What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30.

Kunze

Summerskill

S. J.

Marshall

Filtness

A. J.

(2019). Automation transparency: Implications of uncertainty communication for human-automation interaction and interfaces. Ergonomics, 62(3), 345–360.

McAllister

R. T.

Gal

Kendall

Van Der Wilk

Shah

Cipolla

Weller

(2017). Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning. In International joint conferences on artificial intelligence.

Michelmore

Wicker

Laurenti

Cardelli

Gal

Kwiatkowska

(2020). Uncertainty quantification with statistical guarantees in end-to-end autonomous driving control. In 2020 IEEE international conference on robotics and automation (icra) (pp. 7344–7350).

Papadimitriou

Lassarre

Yannis

(2016). Introducing human factors in pedestrian crossing behaviour models. Transportation Research Part F: Traffic Psychology and Behaviour, 36, 69–82.

10.

Rasouli

Kotseruba

Kunic

Tsotsos

J. K.

(2019). Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. In Proceedings of the ieee/cvf international conference on computer vision (pp. 6262–6271).

11.

Rasouli

Kotseruba

Tsotsos

J. K.

(2017). Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior. In Proceedings of the IEEE international conference on computer vision workshops (pp. 206–213).

12.

Rasouli

Tsotsos

J. K.

(2019). Autonomous vehicles that interact with pedestrians: A survey of theory and practice. IEEE transactions on intelligent transportation systems, 21(3), 900–918.

13.

Sensoy

Kaplan

Kandemir

(2018). Evidential deep learning to quantify classification uncertainty. Advances in Neural Information Processing Systems, 31.

14.

Song

Chen

Sun

Hou

Cui

Wang

(2020). Pedestrian trajectory prediction based on deep convolutional lstm network. IEEE Transactions on Intelligent Transportation Systems, 22(6), 3285–3302.

15.

Zhang

Tian

Ding

(2023). Trep: Transformer-based evidential prediction for pedestrian intention with uncertainty. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, pp. 3534–3542).

16.

Zhang

Tian

Duffy

V. G.

(2022). Trust in automated vehicle: A meta-analysis. In Human-automation interaction: Transportation (pp. 221–234). Springer.

17.

Zhang

Tian

Sherony

Domeyer

Ding

(2022). Attention-based interrelation modeling for explainable automated driving. IEEE Transactions on Intelligent Vehicles, 8(2), 1564–1573.

Uncertainty Differences in Computing Hierarchical Pedestrian Behaviors

Abstract

Keywords

Introduction

Literature Review

Method

Dataset

Model Architecture

Training and Evaluation

Uncertainty Analysis

Results

Model Performance

Data Analysis

Discussion

Limitations

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

References