An evaluation of real-time forecasting performance across 10 western U.S. States

Abstract

The recent financial crisis and economic downturn has emphasized the importance of accurate sub-national forecasting models. To judge which models work best, researchers have emphasized the importance of looking at the true real-time performance of models and not simply an analysis of out-of-sample results. In this study we utilize real-time forecasts from the Western Blue Chip Economic Forecast to analyze and evaluate a host of different forecasters and models across time and 10 U.S. states to see if some models and forecasters consistently outperform others. We use the forecast accuracy criteria established by the Blue Chip publication. To evaluate accuracy we develop a scoring procedure based on the number of years that the forecaster/model was closest to actual relative to what we would expect just by random chance. We also utilize standard measures such as the Root Mean Square Error and Theil's inequality coefficient and test the statistical significance of the best forecasts. We then take a closer look at one model that has proven to be very accurate.