Automated data collection from web sources for official statistics: First experiences

Abstract

Summary: As the internet grows National Statistical Institutes (NSI) are increasingly using data from the internet as a source for their statistics. This paper describes the first experiences of Statistics Netherlands in the automated collection of data from web pages.

First, we explore the feasibility of this method of data collection for NSI's. We conclude that from a technical point of view this type of data collection is possible. Furthermore, the use of website data is probably legal. It is also concluded, that methodological applicability will have to be assessed on a case by case basis.

Second, we investigate the desirability of this type of data collection. On the basis of six case studies tentative conclusions are drawn. Our results show that the cost of reprogramming the robots in case of a site change and the initial investment in the manual process are the most important factors. Furthermore, we conclude that efficiency gains are likely to occur due to the learning-by-doing and scale effects. Finally, we conclude that the main advantages of internetrobots are in applications with large amounts of data(leading to increases in quality, frequency and speed). This will lead to new statistics but also advance the methodological aspects of certain measurement issues.