Dynamic Benchmarks: Spatial and Temporal Alignment for Automated Driving Systems Performance Evaluation

Abstract

Deployed SAE level 4 automated driving systems (ADS) without a human driver are currently operating in ride-hailing fleets on surface streets in the U. S. This current use case, and future applications of this technology, will determine where and when the fleets operate, potentially resulting in a divergence from the distribution of driving of the human benchmark population within a given locality. Existing benchmarks for evaluating ADS performance have only done county-level geographical matching of ADS and benchmark driving exposure in crash rates. This study presents a novel methodology for constructing dynamic human benchmarks that adjust for spatial and temporal variations in driving distribution between ADS and the overall human-driven fleet. Dynamic benchmarks were generated using human police-reported crash data, human vehicle miles traveled data, and over 20 million miles of operational data collected from an active ADS deployment across three U. S. counties. The spatial adjustment revealed significant differences across various severity levels in adjusted crash rates compared with unadjusted benchmarks with these differences ranging from 10% to 47% higher in San Francisco, 12% to 20% higher in Maricopa, and 7% lower to 34% higher in Los Angeles counties. The time-of-day adjustment in San Francisco, limited to this region because of data availability, resulted in adjusted crash rates 2% lower to 16% higher than unadjusted rates, depending on severity level. The findings underscore the importance of adjusting for spatial and temporal confounders in benchmarking analysis, which ultimately contributes to a more equitable benchmark for ADS performance evaluations.