Completing survey questionnaires can be quite burdensome for businesses. Important reasons for this are the retrieval of the required data from internal data sources and the computation of the answers to the questions. With IT technologies, like submitting XML files, web services and Application Programming Interfaces (APIs), this process can be automated. With a growing number of businesses using electronic business information systems, automated data collection becomes more and more in reach. In this paper we will discuss how automated data collection can be implemented in practice, and explore its advantages and challenges, both from the statistical agency as well as the business perspective, based on experiences with cases in tourism, financial data and agriculture at Statistics Portugal and Statistics Netherlands as presented at the International Conference on Establishment Statistics-VII in 2024. The cases show that the technology works, but that doesn’t necessarily mean that it will be adopted by businesses. To successfully implement this data collection method investments by National Statistical Institutes (NSIs) and businesses are needed. Important considerations for businesses are e.g., the information systems they use, their own benefits, and trust in the NSI and the automated collection system. It seems most applicable for stable and recurring data reports, retrieving data from one centralized database.
CouperMPNichollsWLII. The history and development of computer assisted survey information collection methods. In: CouperMPBakerRPBethlehemJClarckCZFMartinJNichollsWLIIO’ReillyJM (eds) Computer assisted survey information collection. New York: Wiley, 1998, pp.1–21.
2.
NicholsWLIIMesenbourgTLJrAndrewsSH, et al.Use of new data methods in establishment surveys. In: Proceedings of the Second International Conference of Establishment Surveys (ICESII). Alexandria, VA: American Statistical Association, 2000, pp.373–382. Available from: https://www.amstat.org/meetings/ices/2000/proceedings/s50.pdf) (accessed 24 February 2025).
3.
BuitenGSnijkersGSaraivaP, et al.Business data collection: toward electronic data interchange. Experiences in Portugal, Canada, Sweden, and The Netherlands with EDI. J Off Stat2018; 34: 419–443. (ICES-5 special issue).
4.
BharosaNvan WijkRde WinneN, et al. (eds). Challenging the Chain. Governing the automated exchange and processing of business information. Amsterdam, the Netherlands: IOS Press, 2015. Available from: https://www.iospress.nl/book/challenging-the-chain/ (accessed 24 February 2025).
5.
EriksonJGiesenDHoubenL,et al.Managing response burden for official statistics business surveys – experiences and recent developments at Statistics Netherlands, statistics Portugal, and Statistics Sweden. In: SnijkersGBavdažMBenderSJonesJMacFeelySSakshaugJWThompsonKJvan DeldenA (eds) Advances in business statistics, methods and data collection. Hoboken: Wiley, 2023, pp.193–223.
6.
CidadeMASnijkersGThorsteinssonK, et al.Establishment-first survey design approach at four national statistical institutes. Surv Methodol2025(to be published)
7.
BavdažMSnijkersGSakshaugJW, et al.Business data collection methodology: current state and future outlook. Stat J IAOS2020; 36: 741–756.
8.
SnijkersGBavdažMBenderS, et al.Advances in business statistics, methods and data collection: introduction. In: SnijkersGBavdažMBenderSJonesJMacFeelySSakshaugJWThompsonKJvan DeldenA (eds) Advances in business statistics, methods and data collection. Hoboken: Wiley, 2023, pp.1–22.
9.
BethlehemJ. Applied survey methods: A statistical perspective. Hoboken: Wiley, 2009.
10.
SnijkersGHaraldsenGJonesJ, et al. (eds). Designing and Conducting Business Surveys. Hoboken: Wiley, 2013.
11.
WillimackDKSnijkersG. The business context and its implications for the survey response process. In: SnijkersGHaraldsenGJonesJWillimackDK (eds) Designing and conducting business surveys. Wiley, Hoboken: Wiley, 2013, pp.39–82.
BavdažM. Measurement Errors and the Response Process in Business Surveys. PhD thesis, University of Ljubljana, Ljubljana, Slovenia, 2007.
14.
HaraldsenGJonesJ. Web and paper questionnaires seen from the business respondent’s perspective. In: Proceedings of the Third International Conference of Establishment Surveys (ICESIII), Alexandria, VA, 2007, pp.1040–1047: American Statistical Association. https://www.amstat.org/meetings/ices/2007/proceedings/ICES2007-000259.PDF (accessed 24 February 2025).
15.
TourangeauR. Cognitive science and survey methods. In: JabineTStrafMTanurJMTourangeauR (eds) Cognitive aspects of survey design. Building a bridge between disciplines. Washington DC, USA: National Academy Press, 1984, pp.73–100.
16.
TourangeauRRipsLJRasinskiK. The psychology of survey response. New York: Cambridge University Press, 2000.
17.
SnijkersG. Cognitive laboratory experiences: on pre-testing computerized questionnaires and data quality. PhD thesis, Utrecht University / Statistics Netherlands, Utrecht / Heerlen, the Netherlands, 2002.
18.
HaraldsenG. Response processes and response quality in business surveys. In: LorencBSmithPABavdažM HaraldsenGNedyalkovaDZhangLZimmermanT (eds) The unit problem and other current topics in business survey methodology. Newcastle upon Tyne, UK: Cambridge Scholars Publishing, 2018, pp.157–177.
19.
HaraldsenG. What computerized business questionnaires and questionnaire management tools can offer. In: SnijkersGBavdažMBenderSJonesJMacFeelySSakshaugJWThompsonKJvan DeldenA (eds) Advances in business statistics, methods and data collection. Hoboken: Wiley, 2023, pp.337–355.
20.
SnijkersGHoubenLDemollinF. Tailoring the design of a new combined business survey: process, methods, and lessons learned. In: SnijkersGBavdažMBenderSJonesJMacFeelySSakshaugJWThompsonKJvan DeldenA (eds) Advances in business statistics, methods and data collection. Hoboken: Wiley, 2023, pp.357–386.
21.
SaraivaPRodriguesSMoreiraA. Automated data collection: experiences in Portugal. Paper presented at the 7th International Conference on Establishment Statistics (ICES-VII), Glasgow (Scotland UK), 17–20 June 2024. https://files.aievolution.com/prd/stt2102/events/17681/1035_0_Santos.pdf (accessed 24 February 2025).
22.
SnijkersGvan WijkD. Automated business data collection: experiences at Statistics Netherlands. Paper presented at the 7th International Conference on Establishment Statistics (ICES-VII), Glasgow (Scotland UK), 17–20 June 2024. https://files.aievolution.com/prd/stt2102/events/17682/1110_0_Snijkers.pdf (accessed 24 February 2025).
23.
MartinsRRodriguesSJordãoM,et al.Tourism Data: Integrated Information System (S2S), sharing data and Official Statistics. Presentation at UNECE Expert Meeting on Statistical Data Collection and Sources, Geneva, Switzerland, 22–24 May 2024. https://unece.org/statistics/events/DC2024 (accessed 24 February 2025).
24.
SnijkersGOnatEVis-VisschersR. The annual structural business survey: developing and testing an electronic form. In: Proceedings of the Third International Conference of Establishment Surveys (ICESIII), Alexandria, VA, 2007, pp.456–463: American Statistical Association. https://ww2.amstat.org/meetings/ices/2007/proceedings/ICES2007-000095.PDF (Accessed 24 February 2025).
25.
SnijkersGTessers-IppelL. What do we already know about the user-friendliness of Revolux questionnaires? (in Dutch: “Wat weten we al over de gebruikersvriendelijkheid van Revolux-vragenlijsten?”. Internal report. Heerlen, the Netherlands: Statistics Netherlands, 2024.
26.
Tessers-IppelLSchalkenNPoulissenD, et al.Revolux IHG IHD and SFO: results from paradata analysis and business visits (In Dutch: Revolux rapportage: paradata en bedrijfsbezoeken IHG, IHD en SFO). Internal report. Heerlen, the Netherlands: Statistics Netherlands, 2024.
27.
GiesenDFietenB. Results of the evaluation of the Structural Statistics Survey (SBS) questionnaires (in Dutch: Resultaten Inventarisatiefase Evaluatie Vragenlijst Productiestatistiek). Internal report. Heerlen, the Netherlands: Statistics Netherlands, 2004.
28.
SnijkersGWielingM. User test of the SBS questionnaire with RCS: set-up, execution, and results (in Dutch: Gebruikerstest PS met RGS: opzet, uitvoering en resultaten). Internal report. Heerlen, the Netherlands: Statistics Netherlands, 2021.
29.
GiesenDDaniëlsPvan DijkJ, et al.Evaluation of the SBS Questionnaires (in Dutch: Evaluatie vragenlijsten PS). Internal report. Heerlen/The Hague, the Netherlands: Statistics Netherlands, 2022.
30.
LamCTessers-IppelLSnijkersG. Business software landscape inventory (in Dutch: Softwarelandschap inventarisatie). Internal report. Heerlen, the Netherlands: Statistics Netherlands, 2025.
31.
SnijkersGWielingM. Pre-testing the 2020 Crop Protection Survey questionnaire: Set-up and results (in Dutch: Pre-test vragenlijst Gewasbescherming 2020: Opzet en resultaten). Internal report. Heerlen, the Netherlands: Statistics Netherlands, 2020.
32.
NieuwenhuijsRHoubenLSnijkersG. A new vision on primary data collection from businesses: “naturally relevant”. Paper presented at the 2022 UNECE Expert meeting on Statistical Data Collection, Rome, 26-28 October 2022. https://unece.org/statistics/events/DC2022 (accessed 24 February 2025).
33.
NieuwenhuijsRSnijkersGvan WijkD. A method to automatically collect financial data from businesses in the SBS questionnaire (in Dutch: Een methode om automatisch financiële data voor de PS te verzamelen). Internal Report. The Hague / Heerlen: Statistics Netherlands, 2021, 2021.
34.
VuikJMatthijsDNieuwenhuijsR. Processing RCFSI data from SME’s in the SBS (in Dutch: Verwerking RGS data van MKB bedrijven in de Productiestatistiek). Internal report. The Hague / Heerlen: Statistics Netherlands, 2019, 2019.
35.
BongersNTessers-IppelLRoosM,et al.User test RCS using speedbooks. (in Dutch: Rapport Gebruikersonderzoek PS-RGS gebruik Speedbooks). Internal Report. The Hague / Heerlen: Statistics Netherlands, 2023.
SnijkersGPuntTDe BroeS,et al.Exploring sensor data for agricultural statistics: the fruit is not hanging as low as we thought. Stat J IAOS2021; 37: 1301–1314.
Gómez PérezJSnijkersG. A new method for automated business data collection of official statistics through APIs. Paper presented at the 2021 European Establishment Statistics Workshop (EESW21), 14-17 September 2021, on-line Workshop, hosted by Statistics Netherlands, The Hague.
40.
PaulussenR. Project description for 2020-NL-AGRI-SISA: agriculture system integration and spatial analysis. Project Technical Description regarding Modernisation of Agricultural Statistics: ESTAT-2020-PA8-E-AGRI. Heerlen/The Hague, the Netherlands: Statistics Netherlands, and Finland: Natural Resource Institute Finland (Luke), 2020.
41.
SnijkersGde JongTBungardtB. System-to-system data communication applied to the CBS Crop Yield Survey using John Deere. AGRI-SISA: WP1 intermediate report. Heerlen/The Hague, the Netherlands: Statistics Netherlands, 2022.
42.
SnijkersGde JongTLamC, et al.System-to-system data communication applied to the Statistics Netherlands Crop Yield Survey using MyJohnDeere. AGRI-SISA: WP1 final report. Heerlen/The Hague: Statistics Netherlands, 2023.
43.
SnijkersGVijftigschildRde JongT, et al.Farm Management Information Systems: A potential data source for agricultural statistics. AGRI-SISA: WP3 intermediate report. The Hague/Heerlen, The Netherlands: Statistics Netherlands, and Finland: Natural Resource Institute Finland (Luke), 2022.
44.
DennemanAVijftigschildRHoogervorstL, et al.Farm Management Information Systems: A potential data source for agricultural statistics. AGRI-SISA: WP3 final report. The Hague/Heerlen, The Netherlands: Statistics Netherlands, and Finland: Natural Resource Institute Finland (Luke), 2023.
45.
NijssenM. FMIS market coverage (in Dutch: Gebruik van FMIS door boeren). Internal memo. The Hague/Heerlen, the Netherlands, 8 February 2024.
46.
KempenaarC. Getting grip on Ag data (in Dutch: Grip op data in de landbouw). Presentation at the workshop on data in smart farming, Agri Innovation Centre at the Aeres University in Applied Sciences, 31 January 2025. Dronten, the Netherlands.
47.
PaulussenR. Use of Farm Management Information Systems (FMIS), administrative registers and earth observation for agricultural statistics under SAIO regulation. Joint project proposal for Modernization of Agricultural Statistics 2023 (2023-NL-AGRI). Statistics Netherlands, DESTATIS, Germany, and the Ministry of Agriculture, France. 2024.
MännikM. Future of Data Collection: Automated Data-based Reporting. Paper presented at the 2025 European Conference of New Techniques and Technologies for official Statistics. Book of Abstracts NTTS 2025, p. 259–262. https://cros.ec.europa.eu/ntts2025 (accessed 14 March 2025).
