Abstract
Globalisation, technology, competition and changes in the political agenda have led to a demand for new statistics, but also provided new possibilities in terms of new data sources. However, these developments have challenged statistical institutes’ compliance with values and principles described in quality frameworks for official statistics, such as the European Statistics Code of Practice. In practice there are often trade-offs between the principles of quality frameworks and full compliance is difficult. Official statistics must be relevant, and this may challenge the independence of a statistical institution. Utilisation of new data sources implies challenges for accuracy and reliability, and meeting competition with partnership may harm confidentiality principles and equal treatment. The paper considers some of these challenges, how they are handled in the existing quality frameworks and could be met by the statistical institutes. Reflecting on these challenges may guide the way forward. Statistical professionalism is a key word in this context.
Introduction
New developments in demand for statistics and technology have challenged statistical institutes’ compliance with their values and principles, described in quality frameworks.
In Europe, the European Statistics Code of Practice (CoP) [1] and the Quality Assurance Framework (QAF) [2] constitute the cornerstone of a common framework for quality in official statistics. There are similar frameworks developed by UN, OECD and regional statistical cooperation bodies. They are all inspired by and built on the UN Fundamental Principles [3].
The Code of Practice was developed in 2005 and revised in 2011 and 2017 to adapt to new requirements for statistics, new data sources and technology. The focus of the revisions has been the strengthening of the statistical institutes’ independence and coordination role, the use of administrative and new data sources and innovation in general. Compliance with the CoP among the members of the European Statistical System has been assessed by peer reviews.
In practice, compliance with quality frameworks must to be balanced over their principles as they inter-relate. Even if requirements for professional independence, impartiality and confidentiality are laid down by law, these requirements are challenged by requirements for relevance, cost effectiveness and cooperation.
The paper considers these challenges. The starting point is international, but examples are mainly from Statistics Norway (SSB). However, these are believed to be representative for several statistical institutes.
Balancing quality principles
The trade-offs between quality principles that come most readily to mind are those between accuracy, timeliness and cost effectiveness. Another trade-off is between timeliness and punctuality. These quality dimensions are linked to statistical products or processes. However, there are situations where also principles linked to the institutional environment for official statistics must be considered in relation to other quality criteria, notably professional independence versus the relevance of the statistics. Cooperation, availability or lack of resources may contribute to weakening independence as well, and there may be a temptation to compromise impartiality and objectivity. Statistical confidentiality may also be affected by the demand for relevance, cooperation and cost effectiveness.
Both professional independence and statistical confidentiality are principles that are included, not only in the CoP, but also in legislation (both European and national statistical laws). CoP itself is self-regulatory, i.e. compliance is controlled by the organisations and people within the system.
Principles laid down by law are subject to stronger requirements and should not be compromised. The amended European Statistical Law [4] puts principal demands on the institutional environment such as:
Professional independence of the statistical institution and its head (CoP principle 1) Coordination role of Eurostat and the NSIs (CoP principle 1bis) Mandate for data collection, including access to administrative and other data for statistical purposes (CoP principle 2) Statistical confidentiality and data protection (CoP principle 5) Equal access and release calendars (under CoP principle 6)
The European Statistical Law refers to CoP and its quality criteria for product quality (principles 11–15). Regarding these principles there is always a need to balance compliance between them. Law prescriptions are limited to the use of CoP with its accompanying Quality Assurance Framework as such, aiming at good compliance.
The principles of independence, statistical confidentiality and impartiality are also central in the ethics of statisticians and statistical institutes, as formulated in the International Statistical Institute (ISI) Declaration of Professional Ethics [5]. Here, the principles are summed up as values:
The balance between such basic principles of official statistics is illustrated in Fig. 1 and discussed in the following chapters.
Balancing independence and other principles for European statistics.
What is professional independence?
CoP principle 1 states that
The UN fundamental principles also use the word professional independence, stating in its principle 2:
Again, this refers to
User needs
In the CoP user needs are reflected by the principles linked to the statistical output, such as principle 11 on relevance: “
The question discussed here is how user focus in practise is balanced against professional independence.
Funding
The CoP principle 3 on adequacy of resources states that the
A great part of this funding covers the production of European statistics, following from EU-regulations. These regulations state not only
Other funding directly from the users might be more problematic. The Leadership Group (LEG) on Quality [6] distinguised between statistics provided as a public good (main product of the NSIs) and as a “private” good (tailor-made analyses on the demand from individual customers even if these may be public bodies). The latter type of statistics often results in conflicting requirements.
In SSB, almost 30 percent of the budget stems from work on commission. Such funding offers direct contact with the users, which is valuable. However, user-funded commissions should supplement the primary tasks financed by the state budget, not replace them. The extent of user-funded commissions must not be of such a magnitude that it threatens our primary priorities and hence independence. It is important that the results are statistics or analyses available to everybody at the same time.
Data management
In some cases, users pay for services like data collection and data management. Examples from SSB comprise systems for coordinated data collection and management of data from the municipalities and from the health sector. Data are often used to establish an administrative register. Ljones [7] discusses independence and ethical issues related to the existence of detailed data from registers, and the role of NSIs as responsible for data collection and management for data used for New Public Management (in addition to the role of producers of official statistics). Combining the two roles is efficient. However, the roles should not be mixed. In these cases, SSB uses the legal basis of the relevant Ministry for data collection, and then the Statistical Act for using the data for statistics. There is still a danger for compromising professional independence linked to this or jeopardising the trust in that we really act independently.
The desired development towards more use of administrative data might also imply a shift of the content of statistics in the direction that fits the authorities but not necessarily those who are governed. CoP states that definitions and concepts in administrative data systems should be adapted to those required for statistical purposes, but this is often not so easy in practice.
Analyses
Users often demand analyses. Analyses add value to and make statistics more relevant. Relevance is not only what statistics cover, but also how statistics are presented. Analyses may take various forms. In the context of this paper analysis is a tool to explain the statistics and their impact, by presenting them in a suitable form (tables and graphics), combining data from different sources, interpreting data, and identifying possible causes and effects. Analyses also include modelling and more substantive analyses, such as making projections. An analysis by a statistical institute shall not advocate policies or take partisan positions. But there is still a question on how far the institution shall go in performing analyses.
Experiences from quality reviews in SSB tell us that users almost always ask for better coherence and interpretations of the statistics in a broader context, which can be achieved by simple analyses of trends and comparison with other statistics. More advanced analyses highlighting correlations casting light on political issues can increase the value for the users. But this also calls for caution to safeguard the principle on impartiality and objectivity. Particularly if conclusions from the analysis rely on model dependent assumptions that may be subject to scientific controversies.
Unlike most other NSIs SSB has a research department that carry out economic and demographic analyses (not only research on development of methodology supporting statistics production). This work counts for about 10 percent of SSB’s activity, and the results are not official statistics, thus in principle not subject to compliance with the CoP. However, the principles of research comprise independence, objectivity, sound methodology and transparency as well. The Department of Research in SSB represents a special challenge for the independence, see example below.
Examples
There are a few examples from NSIs where the heads or other statisticians have been exposed to pressure or have resigned because of conflicts with governing authorities linked to or allegedly linked to the central principles for official statistics. Oppeln-Bronikowski et al. [8] mention some examples of pressure from Governments on NSIs not to release or change release time of statistics that were not in the best interest of the Government. In this case the international quality frameworks provided protection for the statisticians. Langkjær-Bain [9] has interviewed former Director Generals (DGs) of Greece and Canada, and the former responsible statistician for the CPI in Argentina. Their cases vary, but they are all linked to professional independence of statistics and in the Canadian case to statistical confidentiality as well.
There was also a case in Statistics Canada in 2010 where the DG resigned because of a disagreement with the authorities on whether a voluntary survey can become a substitute for a mandatory census [10]. His answer was no since a voluntary survey would harm the quality and thus the usefulness of the survey considerably. Just as important as the professional disagreement was that the minister went public with inaccurate claims about the advice he had received from Statistics Canada. To avoid discussing this as a political issue the DG described it as a technical matter, which sounds less dramatic, though a possible breach of the professional independence of official statistics from the Government’s side.
Though different, we can add a Norwegian example to this list. Last year the DG of SSB withdrew after pressure from the Ministry of Finance. There are different opinions about the reasons for this, but independence became an issue. Like other Government institutions, SSB is imposed to be more efficient, and a program for modernisation has been initiated. This brought about some reorganisation. The reorganisation plan comprised the Department of Research, and there were concerns that some macroeconomic analyses could be negatively affected. In addition to providing the main funding to official statistics, the Ministry is also an important customer paying for analyses such as these. Even if authorities and other stakeholders such as the labour market’s organisations were concerned, the conflict between the Minister of Finance and the SSB DG accelerated when it was proposed that one among 25 other researchers (from a total of 75) should move from the Department of Research to one of the departments producing statistics. This researcher had been the key person carrying out cost estimates linked to immigration, including long term forecasts, for a public committee [11]. With a Minister of Finance representing an immigration sceptic party, there were reasons to suspect political reasons behind her lack of confidence for the DG of SSB. The Ministry claimed that it had given strong warnings against the reorganisation program for months. This was disputed by the SSB DG. (Note the similarities with the disagreement on the content of the dialogue between the Canadian Ministry and NSI in 2010).
In the aftermath there has been an inquiry about this in the Parliament,
As we have discussed, independence does not mean that an NSI solely can decide to produce
Georgiou [12] has discussed professional independence of official statistics considering the institutional links between the statistical institute and the executive branch of Government, i.e. ministries with great interest in statistics or where statistics are crucial for assessing their performance. He argues that the production of official statistics should be carried out by a separate part of Government. This is hardly realistic in Norway, but an arm’s length distance between the Ministry of Finance and SSB would not harm.
For several European countries the European peer review teams 2014–2015 recommended a revision of their statistical laws, to strengthen their professional independence and clarify the content of “official statistics” (see for example report on Norway [13]). In Norway a Government committee has just delivered a proposal for a new law amending the Statistical Act from 1989 [14]. The draft has been subject to a public consultation. New proposals include clarification of the role of the DG and the establishment of a multiannual statistical program. The program will define the content of “official statistics” regardless of who produces it, and the mechanisms for quality control of such statistics which like European statistics shall comply with the CoP.
Confidentiality
The CoP principle 5 on confidentiality and data protection may also be affected by some of the principles and indicators linked to statistical processes and products. Here follow some cases where also professional independence might be affected.
Microdata
Statistical institutes are committed to provide microdata for research, but under strict confidentiality rules. This is stated under CoP principle 15 on accessibility and anchored in the statistical laws.
This has normally not been a problem for confidentiality, data protection has had a strong position. Complaints from researchers about use of time and costs of producing microdata, however, have been more frequent, but that is another issue.
In the new CoP data sharing and data integration is promoted (under Principle 9 on Non-Excessive Burden on Respondents), given adherence to confidentiality and data protection. In the new Norwegian statistical law draft data sharing is proposed between producers of official statistics for this and only this purpose. This is cost-effective but may be a challenge for data protection and hence for the trust in the Norwegian statistical system.
Statistics on small groups
There is a question of judgment linked to confidentiality and the size of groups for which statistics should be published. This issue has two aspects: How to prevent disclosure of confidential information linked to individuals, and how to avoid publishing statistics in a way that can harm small and vulnerable groups.
The first issue is more of technical nature, even if there might be a question of considerations of acceptable risks, the second ethical. The challenges connected to technology are linked to the vast amount of available data from other sources than the NSIs, and the possibilities to identify persons and businesses by combining these data with statistics. This calls for more caution than before when disseminating statistics for small groups. There has been an increased demand for statistics on a more detailed level, for example small geographical areas for municipal planning or analyses of business possibilities by using geographical information systems. The extensive use of administrative and new data sources for statistics (e.g. data on social media) makes it easier to produce statistics for small groups and areas. This demonstrates the need to balance relevance with risks of harming privacy protection.
Legal protections relating to statistical confidentiality in most countries only pertain to identifiable microdata, and hence the technical challenge, but not the ethical one.
As quoted above, the UN principles for official statistics refers to ethics in principle 2. Selzer [15] has expressed this in the following way: “
Examples
SSB has a set of ethical guidelines from 2007, where it is stated that carrying out surveys and disseminating results such that individuals or groups are affected negatively should be avoided.
However, attitudes and policies in this area have changed over time, both in the society and in SSB. In the early 90s there was a discussion by the SSB management following a request to estimate costs and value creation by refugees and immigrants. The conclusion was negative since such estimates would be too uncertain to publish them, but SSB could provide a lot of statistics relevant for this issue. Ten years later we were asked to develop projections of the “immigration population”. The conclusion was that SSB did not have enough knowledge on future immigration to make projections with sufficient quality compared to normal and less detailed population projections. However, it was decided to continue the discussions, also on ethical aspects linked to this, and to develop ethical guidelines.
Nevertheless, as mentioned costs linked to immigration were estimated during the past years as input to a public study. Such calculations would hardly have been done by SSB 15 years ago. SSB has recently also published a report on immigration and crime [16] where immigrants have been grouped by origin country. In this work it has been important to explain not only what the statistics show, but also group by variables that normally are correlated with crime, such as age, sex, economy and relation to the labour market. And to ensure that there are reasonably many units in each group.
An example on how attitudes have changed is also how statistics on the Sami population in Norway have been disseminated. In the period before World War II social statistics was characterised by detailed statistical descriptions of marginal population groups including ethnic groups. In the Norwegian census from 1930 the number of the Sami population was published together (same report) [17] with the number of citizens from other countries, blind, deaf, retarded and mentally ill people. Statistics were not published in this way after the war, but now representatives of the Sami population are promoting more statistics about and for their minority.
Cooperation and coordination
Cooperation has two aspects. One is the extended coordination role with the responsibility for the NSI for quality assurance of all official statistics, following European statistical and the new CoP principle 1bis. Though only for European statistics, the same responsibility for coordination is reflected in national statistical laws such as in the new Norwegian one. The statistical institutes will have a role as national data curators/custodians. The other aspect is cooperation to ensure access to source data for development and production statistics, though laid down by law such cooperation is necessary in practice.
Other producers of official statistics normally have some administrative and operative functions as their core business. Producing statistics here constitute a small part of their respective institutions. This may represent a challenge both to professional independence and confidentiality.
To maintain professional independence, it is important that the role of producing European and official statistics is clearly delimited from other roles the producers might have. A statistical release calendar and full transparency about methods and production procedures are concrete requirements and means of ensuring this.
CoP promotes data sharing and integration among producers of European and official statistics to minimise response burden, while adhering to confidentiality and data protection requirements. Even so this data sharing may also increase the risk for confidentiality breaches and weaken public trust of the producers.
In this context it is also worth reflecting on if increased cooperation and data sharing will be followed by increased research demands for access to microdata? And if so, whether quality requirements on microdata then will impose a raised focus on other quality dimensions than statistical accuracy, timeliness etc.? Being a steward of microdata that are made available for public good research and analysis is a different role than the statistics producer’s. Highlighted perspectives on ‘data quality’ parallel to (or in parts of the organisation instead of) ‘statistics quality’ is likely to create new challenges of how to balance quality principles. Cooperation is the key to get access to new also privately held data sources, but this might also affect data protection. Cooperation partners might want to have access to data in return for providing their data. It also represents a challenge for the requirement of equal treatment.
Partnership has become a key word in the strategies of statistical authorities. In the ESS Vision 2020 [18] it is said that
But how could an NSI have a formal partnership with one or some private companies, for example telecommunication companies, and not with others? Are agreements with all necessary? The answer is probably that this kind of cooperation is OK if professional independence and statistical confidentiality are not compromised, and that
SSB is engaged in a few partnerships with both public, academic and private institutions where data are used only to investigate if and how they can be used for different purposes including new statistics.
For example, to investigate new ways of making the household budget survey, SSB have acquired data about purchase transactions from the private sector. To assist in the analysis of these data partnership with a research consortium of both private, public and academic entities are considered.
This is unproblematic in an experimental phase, but it must be clear that data acquired by the NSI is used only for experimenting aiming at and eventually producing official statistics. Access to data on these terms is included in the proposal for the new Norwegian Statistical Act.
Conclusions
A main message from the work on quality in statistics at the end of last century was that quality consists of several features reflecting user needs. This has been taken into account in the development of quality frameworks such as the European Statistical System Code of Practice, comprising principles related to both the statistical institutions, their production processes and the different dimensions of output quality required by the users.
It is difficult or in some cases meaningless to set minimum requirements for compliance with each principle across all statistics, since the principles inter-relate.
The existence of quality frameworks itself provides protection and safeguarding of public trust in official statistics. But assessing compliance with these frameworks is not enough.
Footnotes
Meso denotes middle, i.e. not micro and not macro.