Pitfalls of Using R 2 to Evaluate Goodness of Fit of Accident Prediction Models

Abstract

In developing statistical models of traffic accidents, flow, and roadway design, the R² goodness-of-fit measure has been used for many years to (a) determine the overall quality and usability of the model, (b) select covariates for inclusion in the model, (c) make decisions as to whether it would be worthwhile to collect additional covariates, and (d) compare the relative quality of models developed from different studies. The pitfalls of using R² to make these decisions and comparisons are demonstrated through computer simulations of commonly used accident prediction models, including the Poisson and negative binomial regression models. Because accident prediction models are nonnormal and functional forms are typically nonlinear, it is shown that R² is not an appropriate measure to make any of the decisions and comparisons mentioned. Also, three properties are identified as desirable for any alternative measure to appropriately evaluate these models: (a) it should be bounded between 0 and 1—a value of 0 if no covariate is included in the model and a value of 1 if all the necessary covariates are included; (b) it should increase proportionally as equally important, independent covariates are added to the model one at a time, regardless of their order of selection; and (c) it should be invariant with respect to the mean (i.e., the value of the measure should not change by simply increasing or decreasing the value of the intercept term). Finally, two recent research efforts aimed at developing alternative measures with such properties are briefly reported.