Bayesian Training and Committees of State-Space Neural Networks for Online Travel Time Prediction

Abstract

This paper presents the Bayesian evidence framework that enables a unified way of constructing and training committees of an arbitrary number of models. The main contribution the paper makes is an expansion of this framework for recurrent neural networks, which involves analytically deriving the gradient and the Hessian of the network. State-space neural networks (SSNNs), a special type of recurrent neural networks, are compared with feed-forward neural networks (FFNNs), and the effect of the Bayesian framework on both types is investigated using data from a densely used freeway in the Netherlands. From a cross-validation procedure, it can be concluded that, for a short time horizon, both Bayesian training and recurrency do not lead to improvements, but that, for a longer horizon, both techniques are beneficial. It is shown that the use of a committee leads to improved performance; furthermore, the correlation of the evidence factor, which follows from Bayesian model-fitting, and the generalization performance is compared against the training error and the generalization performance. It is found that the evidence has lower correlation, which is an indication that (a) the data set may be too small, (b) bias exists, (c) the mapping between the input and output data is difficult, and (d) the approximation of the evidence is imperfect. Future research will need to resolve these issues. However, the Bayesian framework will already be beneficial to more complex problems and lead to estimations of error bars on the predictions, which may be useful for many applications.