Abstract
Official economic statistics are uncertain even if not always interpreted or treated as such. From a historical perspective, this article reviews different categorisations of data uncertainty, specifically the traditional typology that distinguishes sampling from nonsampling errors and a newer typology of Manski (2015). Throughout, the importance of measuring and communicating these uncertainties is emphasised, as hard as it can prove to measure some sources of data uncertainty, especially those relevant to administrative and big data sets. Accordingly, this article both seeks to encourage further work into the measurement and communication of data uncertainty in general and to introduce the Comunikos (COMmunicating UNcertainty In Key Official Statistics) project at Eurostat. Comunikos is designed to evaluate alternative ways of measuring and communicating data uncertainty specifically in contexts relevant to official economic statistics.
References
1.
Abowd, J.M., and M.H. Stinson. 2013. “Estimating measurement error in annual job earnings: A comparison of survey and administrative data.” The Review of Economics and Statistics 95(5): 1451–1467. DOI: http://dx.doi.org/10.2139/ssrn.1894690.
2.
Aruoba, S.B., F.X. Diebold, J. Nalewaik, F. Schorfheide, and D. Song. 2016. “Improving GDP measurement: A measurement-error perspective.” Journal of Econometrics 191(2): 384 – 397. DOI: http://dx.doi.org/10.1016/j.jeconom.2015.12.009.
3.
Australian Bureau of Statistics. 2013. “Types of error.” Available at: https://www.abs.gov.au/websitedbs/a3121120.nsf/home/statistical+language+-+types+of+error (accessed December 2019).
4.
Bagus, P. 2011. “Morgenstern’s forgotten contribution: A stab to the heart of modern economics.” American Journal of Economics and Sociology 70(2): 540–562. DOI: https://doi.org/10.1111/j.1536-7150.2011.00783.x.
5.
Bank of England. 2007. Inflation Report, November 2007 . Available at: https://www.bankofengland.co.uk/-/media/boe/files/inflation-report/2007/november-2007.pdf?la=en&hash=03AA4FB7EE5CA613952B392EFB00411A7D10195A (accessed December 2019).
6.
Bernhard, S. 2016. “A real-time GDP data set for Switzerland.” SNB Economic Studies 9/2016. Available at: https://www.snb.ch/n/mmr/reference/economic_studies_2016_09/source/economic_studies_2016_09.n.pdf (accessed December 2019).
7.
Biemer, P.P., and L.E. Lyberg. 2003. Introduction to Survey Quality . Hoboken, NJ: John Wiley Sons. DOI: https://doi.org/10.1002/0471458740.
8.
Boumans, M. 2012: “Observations in A Hostile Environment: Morgenstern on the Accuracy of Economic Observations.” History of Political Economy 44: 110–131. DOI: https://doi.org/10.1215/00182702-1631806.
9.
Burridge, P., and K.F. Wallis. 1985. “Calculating the variance of seasonally adjusted series.” Journal of the American Statistical Association 80(391): 541–552. DOI: https://doi.org/10.1080/01621459.1985.10478151.
10.
Chambers, R. 2014. “Survey sampling in official statistics - some thoughts on directions.” Proceedings of Statistics Canada Symposium 2014: Beyond traditional survey taking: adapting to a changing world . Available at: https://www.statcan.gc.ca/eng/conferences/symposium2014/program/14251-eng.pdf (accessed December 2019).
11.
Chow, G., and A.-I. Lin. 1971. “Best linear unbiased interpolation, distribution, and extrapolation of time series by related series.” Review of Economics and Statistics 53(4): 372–75. DOI: https://doi.org/10.2307/1928739.
12.
Clements, M.P., and A.B. Galv~ao. 2010. “First announcements and real economic activity.” European Economic Review 54: 803–817. DOI: https://doi.org/10.1016/-j.euroecorev.2009.12.010.
13.
Conti, P.L., D. Marella, and M. Scanu. 2012. “Uncertainty analysis in statistical matching.” Journal of Official Statistics 28: 69–88. Available at: https://www.scb.se/-contentassets/ca21efb41fee47d293bbee5bf7be7fb3/uncertainty-analysis-in-statistical-matching.pdf (accessed December 2019).
14.
Croushore, D. 2011. “Frontiers of real-time data analysis.” Journal of Economic Literature 49: 72–100. DOI: https://doi.org/10.1257/jel.49.1.72.
15.
Croushore, D., and T. Stark. 2001. “A real-time data set for macroeconomists.” Journal of Econometrics 105(1): 111–130. DOI: https://doi.org/10.1016/s0304-4076(01)00072-0.
16.
Croushore, D., and T. Stark. 2003. “A real-time data set for macroeconomists: Does the data vintage matter?” The Review of Economics and Statistics 85: 605–617. DOI: https://doi.org/10.1162/003465303322369759.
17.
Cunningham, A., J. Eklund, C Jeffery, G. Kapetanios, and V. Labhard. 2012. “A state space approach to extracting the signal from uncertain data.” Journal of Business and Economic Statistics 30: 173–80. DOI: https://doi.org/10.2139/ssrn.1080189.
18.
Cunningham, A. and C. Jeffery. 2007. “Extracting a better signal from uncertain data.” Bank of England Quarterly Bulletin 2007Q3 . Available at: https://www.bankofen-gland.co.uk/-/media/boe/files/quarterly-bulletin/2007/extracting-a-better-signal-from-uncertain-data.pdf?la=en&hash=58933100C07CD43ED04DC670F9D6721154AD4F87 (accessed December 2019).
19.
De Jonge, E. (2020). “Communicating uncertainties in official statistics – A review of communication methods.” Available at: https://ec.europa.eu/eurostat/documents/3888793/12135507/KS-TC-20-011-EN-N.pdf/b2ee60fe-ed5b-ad3e-05bd-56f6b87dfec0?t=1608645519272 (accessed May 2021).
20.
Draper, D. 1995. “Assessment and propagation of model uncertainty.” Journal of the Royal Statistical Society. Series B (Methodological) 57(1): 45–97. DOI: https://doi.org/10.1111/j.2517-6161.1995.tb02015.x.
21.
Dungan, J., D. Gao, and A. Pang. 2002. “Definitions of uncertainty.” Available at: ftp://ftp. cse.ucsc.edu/pub/reinas/papers/white.pdf (accessed December 2019).
22.
European Commission. 2010. “Report on Greek government deficit and debt statistics.” Available at: https://ec.europa.eu/eurostat/documents/4187653/6404656/COM_2010_-report_greek/c8523cfa-d3c1-4954-8ea1-64bb11e59b3a (accessed October 2020).
23.
Eurostat. 2015. “Eurostat statistics explained.” Available at: https://ec.europa.eu/eurostat/statistics-explained/index.php/Glossary:Accuracy (accessed December 2019).
24.
Eurostat. 2017. “European statistics code of practice: revised edition 2017.” Available at: https://ec.europa.eu/eurostat/web/quality/principle12 (accessed December 2019).
25.
Eurostat. 2019. “ESS Standard for Quality Reports.” Available at: https://unstats.un.org/unsd/dnss/docs-nqaf/eurostat-esqr_final.pdf (accessed December 2019).
26.
Faust, J., J.H. Rogers, and J. H. Wright. 2005. “News and noise in G-7 GDP announcements.” Journal of Money, Credit and Banking 37(3): 403–420. DOI: https://doi.org/10.1353/mcb.2005.0029.
27.
Frale, C., M. Marcellino, G.L. Mazzi, and T. Proietti. 2011. “EUROMIND: a monthly indicator of the Euro Area economic conditions.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 174(2): 439–470. DOI: https://doi.org/10.1111/j.1467-985x.2010.00675.x.
28.
Fuller, W. 1987. Measurement Error Models . John Wiley and Sons: New York. DOI: https://doi.org/10.1002/jae.3950030407.
29.
Galvao, A.B. 2017. “DSGE models and data revisions.” Journal of Econometrics 196: 215–232. DOI: https://doi.org/10.1016/j.jeconom.2016.09.006.
30.
Galvao, A. and J. Mitchell. 2020. Real-Time Perceptions of Historical GDP Data Uncertainty, EMF Research Papers 35, Economic Modelling and Forecasting Group, Warwick Business School. Available at: https://ideas.repec.org/p/wrk/wrkemf/35.html (accessed September 2020).
31.
Galvao, A. J. Mitchell, and J. Runge. 2019. “Communicating data uncertainty: experimental evidence for UK GDP.” Economic Statistics Centre of Excellence Discussion Paper 2019-20. Available at: https://EconPapers.repec.org/RePEc:nsr:es-coed:escoe-dp-2019-20 (accessed December 2019).
32.
Geweke, J. 1978. “The temporal and sectoral aggregation of seasonally adjusted time series.” In Seasonal Analysis of Economic Time Series edited by A. Zellner. Washington, DC: US Department of Commerce, Census Bureau. Available at: https://www.census.gov/ts/papers/Conference1978/Geweke1978.pdf (accessed December 2019).
33.
Ghysels, E. 1997. “Seasonal adjustment and other data transformations.” Journal of Business and Economic Statistics 15: 410–418. DOI: https://doi.org/10.1080/07350015.1997.10524719.
34.
Giannone, D., J. Henry, M. Lalik, and M. Modugno. 2012. “An Area-Wide Real-Time Database for the Euro Area.” Review of Economics and Statistics 94: 1000–1013. DOI: https://doi.org/10.1162/rest_a_00317.
35.
Goedeme, T. 2013. “How much Confidence can we have in EU-SILC? Complex Sample Designs and the Standard Error of the Europe 2020 Poverty Indicators.” Social Indicators Research 110(1): 89–110. DOI: https://doi.org/10.1007/s11205-011-9918-2.
36.
Granger, C.W.J., and M.H. Pesaran. 2000. “Economic and statistical measures of forecast accuracy.” Journal of Forecasting 19: 537–560. DOI: https://doi.org/10.1002/1099-131x(200012)19:7%3C537:aid-for769%3E3.3.co;2-7.
37.
Gromme, F. Ustek-Spilda, E. Ruppert, and B. Cakici. 2017. “Citizen data and official statistics: Background document to a collaborative workshop.” Available at: https://pdfs.semanticscholar.org/4bc0/6a6504c5086a2e81a299abd95ef42617b77b.pdf (accessed December 2019).
38.
Groves, R.M., and L. Lyberg. 2010. “Total survey error: past, present and future.” Public Opinion Quarterly 74: 849–879. DOI: https://doi.org/10.1093/poq/nfq065.
39.
Hand, D.J. 2018. “Statistical challenges of administrative and transaction data.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 181(3): 555–605. DOI: https://doi.org/10.1111/rssa.12315.
40.
Jacobs, J.P.A.M. and S. van Norden. 2011. “Modeling data revisions: Measurement error and dynamics of “true” values.” Journal of Econometrics 161: 101–109. DOI: https://doi.org/10.1016/j.jeconom.2010.04.010.
41.
Kapetanios, G., M. Marcellino, F. Kempf, G.-L. Mazzi, J. Eklund, and V. Labhard. 2021. “Measuring and communicating uncertainty in official statistics: State of the art and perspectives.” Statistical working papers collection, Eurostat. Available at: https://ec.europa.eu/eurostat/cros/system/files/ks-tc-20-006-en-n.pdf (accessed May 2021).
42.
Kapteyn, A., and J.Y. Ypma. 2007. “Measurement error and misclassification: A comparison of survey and administrative data.” Journal of Labor Economics 25(3): 513–551. DOI: https://doi.org/10.1086/513298.
43.
Kishor, N.K., and E.F. Koenig. 2012. “VAR estimation and forecasting when data are subject to revision.” Journal of Business and Economic Statistics 30: 181–190. DOI: https://doi.org/10.1198/jbes.2010.08169.
44.
Kloprogge, P., J. van der Sluijs, and A. Wardekker. 2007. “Uncertainty Communication , Utrecht: Copernicus Institute.” Available at: http://www.nusap.net/downloads/reports/uncertainty_communication.pdf (accessed December 2019).
45.
Koch, G.G., and D.B. Gillings. 2006. Inference, Design-Based vs. Model-Based . American Cancer Society. DOI: https://doi.org/10.1002/0471667196.ess1235.pub2.
46.
Kuznets, S. 1948. “National income: a new version.” Review of Economics and Statistics 30(3): 151–179. DOI: https://doi.org/10.2307/1926746.
47.
Kuznets, S. 1950. “Review of On the accuracy of economic observations , by Oscar Morgenstern.” Journal of the American Statistical Association 45: 576–79. DOI: https://doi.org/10.2307/2280732.
48.
Laureti, T., and I. Benedetti. 2020. “Measuring and communicating uncertainty of poverty indicators at regional level.” Statistical working papers collection, Eurostat. Available at: https://ec.europa.eu/eurostat/documents/3888793/12137895/KS-TC-20-010-EN-N.pdf/6745684c-c989-b3e5-33ae-7bd7dd89bf92?t=1608634421113 (accessed May 2021).
49.
Lui, S., J. Mitchell, and M. Weale. 2011. “Qualitative business surveys: signal or noise?” Journal of the Royal Statistical Society Series A 174(2): 327–348. DOI: https://doi.org/10.1111/j.1467-985x.2010.00667.x.
50.
Manclossi, S., and V. Ayodele. 2016. “Users understanding of uncertainty measures to describe data quality.” Presented at The 21st Government Statistical Service (GSS) Methodology Symposium Methodology: the key to the door of innovation. Available at: https://gss.civilservice.gov.uk/wp-content/uploads/2016/07/4.4.3-Silvia-Manclossi-Users-understanding-and-use-of-uncertainty-measures-to-descripdf (accessed December 2019).
51.
Mankiw, N.G., and M.D. Shapiro. 1986. “News or noise: An analysis of GNP revisions.” Survey of Current Business (May 1986), US Department of Commerce, Bureau of Economic Analysis : 20–25. DOI: https://doi.org/10.3386/w1939.
52.
Manski, C.F. 2015. “Communicating Uncertainty in Official Economic Statistics: An Appraisal Fifty Years after Morgenstern.” Journal of Economic Literature 53(3): 631–653. DOI: https://doi.org/10.1257/jel.53.3.631.
53.
Manski, C.F. 2016. “Credible interval estimates for official statistics with survey nonresponse.” Journal of Econometrics 191(2): 293–301. DOI: https://doi.org/10.1016/j.jeconom.2015.12.002.
54.
Manski, C.F. 2019. “Communicating uncertainty in policy analysis.” Proceedings of the National Academy of Sciences 116(16): 7634–7641. DOI: https://doi.org/10.1073/pnas.1722389115
55.
McCracken, M.W., and S. Ng. 2016. “FRED-MD: A Monthly Database for Macroeconomic Research.” Journal of Business and Economic Statistics 34(4): 574–589. DOI: https://doi.org/10.1080/07350015.2015.1086655.
56.
McKenzie, R. (2006), “Undertaking revisions and real-time data analysis using the OECD main economic indicators original release data and revisions database.” OECD Statistics Working Papers, 2006/02, OECD Publishing, Paris.
57.
Mehrhoff, J. 2008. “Sources of revisions of seasonally adjusted real time data.” Paper prepared for the Meeting of the OECD Short-term Economic Statistics Working Party (STESWP), 23–24 June 2008, Paris. France. Available at: http://www.oecd.org/sdd/fin-stats/40671433.pdf (accessed December 2019).
58.
Mevik, A.-K. 2004. “Uncertainty in the Norweigan Business Tendency Survey.” Available at: https://www.ssb.no/a/english/publikasjoner/pdf/doc_200410_en/doc_200410_en.pdf (accessed December 2019).
59.
Mitchell, J., R. Smith, M. Weale, S. Wright, and E.L. Salazar. 2005. “An Indicator of Monthly GDP and an Early Estimate of Quarterly GDP Growth.” Economic Journal 115(501): 108–129. DOI: https://doi.org/10.1111/j.0013-0133.2005.00974.x.
60.
Morgenstern, O. 1950. Introduction to survey quality , Princeton University Press.
61.
National Science Foundation 2018. “Data accuracy.” Available at: https://nsf.gov/statistics/2018/nsb20181/report/sections/appendix-methodology/data-accuracy (accessed December 2019).
62.
Oberski, D.L., A. Kirchner, S. Eckman, and F. Kreuter. 2017. “Evaluating the quality of survey and administrative data with generalized multitrait-multimethod models.” Journal of the American Statistical Association 112(520): 1477–1489. DOI: https://doi.org/10.1080/01621459.2017.1302338.
63.
OECD 2015. “Revisions of quarterly GDP in selected OECD countries.” Statistics Brief–July 2015 – No 22 by Zwijnenburg . Available at: https://www.oecd.org/sdd/na/Revisions-quarterly-GDP-selected-OECD-countries-OECDSB22.pdf (accessed December 2019).
64.
ONS 2017a. “Methodology for measuring uncertainty in ONS local authority midyear population estimates: 2012 to 2016.” Available at: https://www.ons.gov.uk/methodology/methodologicalpublications/generalmethodology/onsworkingpaperseries/methodologyformeasuringuncertaintyinonslocalauthoritymidyearpopulationestimates20 (accessed December 2019).
65.
ONS 2017b. “ONS methodology working paper series no. 9 – Guide to calculating standard errors for ONS Social Surveys.” Available at: https://www.ons.gov.uk/methodology/methodologicalpublications/generalmethodology/onsworkingpaperseries/ons-methodologyworkingpaperseriesno9guidetocalculatingstandarderrorsforonssocialsu (accessed December 2019).
66.
ONS 2018a. “Communicating quality, uncertainty and change.” Available at: https://gss.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/ (accessed December 2019).
67.
ONS 2018b. “VAT turnover data in National Accounts: background and methodology.” Available at: https://www.ons.gov.uk/economy/grossdomesticproductgdp/methodologies/vatturnoverdatainnationalaccountsbackgroundandmethodology (accessed December 2019).
68.
ONS 2019. “Gross domestic product (GDP) QMI.” Available at: https://www.ons.gov.uk/economy/grossdomesticproductgdp/methodologies/grossdomesticproductgdpqmi (accessed December 2019).
69.
Orphanides, A., and S. van Norden. 2002. “The unreliability of output-gap estimates in real time.” The Review of Economics and Statistics 84: 569–583. DOI: https://doi.org/10.1162/003465302760556422.
70.
Peters, C.A. 2001. “Statistics for Analysis of Experimental Data.” In Environmental Engineering Processes Laboratory Manual , edited by S.E. Powers. Champaign, IL: AEESP. Available at: https://www.princeton.edu/~cap/AEESP_Statchap_Peters.pdf (accessed December 2019).
71.
Piccolo, D., and R. Simone, 2019. “The class of cub models: statistical foundations, inferential issues and empirical evidence.” Statistical Methods and Applications 28: 389–435. DOI: https://doi.org/10.1007/s10260-019-00461-1.
72.
Plumper, T., and E. Neumayer. 2012. “Population and sample uncertainty.” EPSA 2013 Annual General Conference Paper 166 . Available https://ssrn.com/abstract=2224688 (accessed December 2019).
73.
Ruggeri Cannata, R., and G.-L. Mazzi. 2017. “The trade-off between timeliness and accuracy: the perspective of a statistical agency.” In Handbook on Rapid Estimates , edited by G.L Mazzi. Eurostat and United Nations, : 123–131. DOI: https://doi.org/10.2785/317290.
74.
Seljak, R. 2006. “Estimation of standard error of indices in the sampling business surveys.” Proceedings of Q2006 European Conference on Quality in Survey Statistics. Available at: https://ec.europa.eu/eurostat/documents/64157/4374310/16-ESTIMATION-OF-STANDARD-ERROR-OF-INDICES-IN-THE-SAMPLING.pdf/5721f63b-9ac8-4767-b557-dd30b0378336 (accessed December 2019).
75.
Smith, R.J., M.R. Weale, and S.E. Satchell. 1998. “Measurement Error with Accounting Constraints: Point and Interval Estimation for Latent Data with an Application to U.K. Gross Domestic Product.” Review of Economic Studies 65(1): 109–134. DOI: https://doi.org/10.1111/1467-937x.00037.
76.
Statistics Canada 2017. “Non-sampling error.” Available at: https://www150.statcan.gc.-ca/n1/edu/power-pouvoir/ch6/nse-endae/5214806-eng.htm (accessed December 2019).
77.
Stone, R., D.G. Champernowne, and J. Meade. 1942. “The precision of national income estimates.” Review of Economic Studies 9(2): 111–125. DOI: https://doi.org/10.2307/2967664.
78.
US Census Bureau 2018. “Measures of nonsampling error (chapter 11).” Available at: https://www.census.gov/ts/papers/Conference1978/Geweke1978.pdf (accessed December 2019).
79.
Van der Bles, A.M., D. Spiegelhalter, S. Dryhurst, A. Freeman, M. Pearson, and J. Park. 2018. “Determining and facilitating the clearest ways to visualize uncertainty around estimates, time series and curves.” Report to M2D. Available at:. https://wintoncentre.maths.cam.ac.uk/projects/communicating-uncertainty/ (accessed June 2020).
80.
Van der Bles, A.M., S. van der Linden, A.L.J. Freeman, and D.J. & Spiegelhalter. 2020. “The effects of communicating uncertainty on public trust in facts and numbers.” Proceedings of the National Academy of Sciences 117(14): 7672–7683. DOI: https://doi.org/10.1073/pnas.1913678117.
81.
Van der Bles, A.M., S. van der Linden, A. Freeman, J. Mitchell, A.B. Galvao, L. Zaval, and D. Spiegelhalter. 2019. “Communicating uncertainty about facts, numbers and science.” Royal Society Open Science 6: 1–42. DOI: https://doi.org/10.1098/rsos.181870.
82.
Van der Loo, J. Pannekoek, and L. Rijnveld. 2017. “Computational estimates of data-editing related variance.” United Nations Economic Commission for Europe, Conference of European Statistians, Work Session on Statistical Data Editing (The Hague, Netherlands, 24/26 April 2017). Available at: https://www.unece.org/fileadmin/DAM/stats/documents/ece/ces/ge.44/2017/mtg2/Paper_7_DataEditingBoot-strap.pdf (accessed December 2019).
83.
Verma, V., G. Betti, and F. Gagliardi. 2010. “An assessment of survey errors in EUSILC.” Eurostat: Methodologies and working papers . Available at: https://ec.europa.eu/eurostat/documents/3888793/5848229/KS-RA-10-021-EN.PDF/2b5b31ad-3973-48ad-9a87-8645c1d2c16c (accessed December 2019).
84.
Wallis, K.F. 1982. “Seasonal adjustment and revision of current data: linear filters for the X-11 method.” Journal of the Royal Statistical Society A 145: 74–85. DOI: https://doi.org/10.2307/2981422.
85.
Walton, A. 2016. “International comparison of GDP revisions.” Office for National Statistics : 1–34. Available at: https://www.ons.gov.uk/economy/grossdomesticproductgdp/articles/internationalcomparisonofgdprevisions/2016-04-28 (accessed December 2019).