2  Confidentiality in legal acts and ethical codes

2.1 Confidentiality in European statistical laws

Up to the late 1980’s microdata were rarely sent to Eurostat, the statistical office of the European Union. There was a general reliance on submission by National Statistical Institutes (NSIs) of agreed tabular data. National confidentiality rules in some of the European countries made it impossible to harmonise European statistics. This was an unwanted situation for all NSIs, and especially for Eurostat. Therefore, a regulation on the transmission of confidential data to Eurostat has been prepared and was finally adopted by the Council in June 1990 as Regulation 1588/90.

Committee on Statistical Confidentiality
In January 1994, these measures have been defined and formally adopted by the Member States through the Committee on Statistical Confidentiality (CSC). This Committee met at least once a year at the Eurostat office in Luxembourg. This Committee discussed the implementation and evaluation of European Regulations on the dissemination of microdata and tabular data. Also revisions to the basic statistical legal framework were considered. The last meeting of the CSC was held in 2008.

Another relevant Council Regulation is No 322/97 of February 1997. This Regulation defined the general principles governing Community statistics, the processes for the production of these statistics and established detailed rules on confidentiality. This Regulation was considered as the general statistical law of the European Union until 2009 when the Regulation on European statistics entered into force and replaced Council Regulation 322/97.

EU Regulation on European statistics
A new statistical legal framework at the European level was introduced in 2009. One of the new aspects concerns statistical confidentiality: the need to enhance the role of the NSIs and Eurostat for organisational, co-ordination and representation purposes was noted. In this context the former Statistical Programme Committee was replaced by a new Committee, the European Statistical System Committee (ESSC). This new Committee is also entrusted with the functions of the CSC, which thus ceased to exist.

The European Statistical System (ESS) is defined by Regulation 223/2009 on European statistics on 1 April 2009 as the partnership between the Community statistical authority (the Commission (Eurostat)) and all national authorities responsible for the development, production and dissemination of European Statistics (ES). Regulation 223/2009 was amended by Regulation 2015/759 of the European Parliament and the Council in order to further strengthen the governance of the ESS, in particular its professional independence.

Currently, Regulation 223/2009 is being ammended by a more modern version that gives, among others, better possibilities for the NSIs to use privately held data. However, the new regulation will not include substantial changes regarding the situation of Statistical Disclosure Control in the European Union.

The availability of confidential data for the needs of the ESS is of particular importance in order to maximise the benefits of the data with the aim of increasing the quality of European statistics and to ensure a flexible response to the newly emerging EU statistical needs.

The transmission of confidential data between ESS partners is allowed if necessary for the production, development and dissemination of ES and also for increasing the quality of these statistics. The conditions for their further transmission, in particular for scientific purposes, are also strictly defined.

The ESSC is consulted on all draft comitology measures submitted by the Commission in the domain of statistical confidentiality.

The next section gives some background by discussing a few ethical codes and laws. It does not contain any national specialities.

2.2 Ethical codes

Many Member States have an ethical code that forms the basis of the production of official statistics. Additionally, there are internationally recognised ethical codes that are discussed in this section.

ISI Declaration on Professional Ethics
After an intense preparation process taking place from 1979 to 1985, the International Statistical Institute (ISI) adopted the ISI Declaration on Professional Ethics in 1985. A newer version was adopted in 2010 by the ISI Council. Finally, an updated version was endorsed by the ISI Executive Committee. Whilst the 2010 Declaration content remains largely valid, the increasing use of a diversity of data sources, linked data sets and computationally heavy statistical methods has required updates introduced in 2023.

European Statistics Code of Practice
On 24 February 2005 the Statistical Programme Committee adopted the European Statistics Code of Practice. On 17 November 2017 a renewed version of this Code was adopted by the European Statistical System Committee (ESSC). This Code of Practice has the dual purpose of:

  • Improving trust and confidence in the independence, integrity and accountability of both National Statistical Authorities and Eurostat, and in the credibility and quality of the statistics they produce and disseminate (i.e. an external focus);
  • Promoting the application of best international statistical principles, methods and practices by all producers of European Statistics to enhance their quality (i.e. an internal focus).

The Code of Practice is based on 15 Principles. Governance authorities and statistical authorities in the European Union commit themselves to adhering to the principles fixed in this code and to reviewing its implementation periodically by the use of Indicators of Good Practice for each of the 15 Principles, which are to be used as references. Principle 5 concerns statistical confidentiality and is cited below.

Principle 5: Statistical Confidentiality

The privacy of data providers, the confidentiality of the information they provide, its use only for statistical purposes and the security of the data are absolutely guaranteed.

Indicators

  • 5.1 Statistical confidentiality is guaranteed in law.
  • 5.2 Staff sign legal confidentiality commitments on appointment.
  • 5.3 Penalties are prescribed for any wilful breaches of statistical confidentiality.
  • 5.4 Guidelines and instructions are provided to staff on the protection of statistical confidentiality throughout the statistical processes. The confidentiality policy is made known to the public.
  • 5.5 The necessary regulatory, administrative, technical and organisational measures are in place to protect the security and integrity of statistical data and their transmission, in accordance with best practices, international standards, as well as European and national legislation.
  • 5.6 Strict protocols apply to external users accessing statistical microdata for research purposes.

UNECE principles and Guidelines of Good Practice for Managing Statistical Confidentiality and Microdata Access
The 2003 Conference of European Statisticians (CES) of the United Nations Statistical Commission for Europe installed a Task Force, chaired by Dennis Trewin (at that time the Australian Statistician), to draft Principles and Guidelines of Good Practice for Managing Statistical Confidentiality and Microdata Access. In their final report of 2007 the following two key objectives in these guidelines are mentioned:

  • To foster greater uniformity of approach by countries whilst facilitating better access to microdata by the research community for worthwhile papers;
  • Through these guidelines and supporting case studies, to enable countries to improve their arrangements for providing access to microdata.

The sixth United Nations Fundamental Principle of Official Statistics, which was mentioned in Section 1.2 of this handbook, is very clear on statistical confidentiality: “Individual data collected by statistical agencies for statistical compilation, whether they refer to natural or legal persons, are to be strictly confidential and used exclusively for statistical purposes”. Any principles for microdata access must be consistent with this Fundamental Principle.

According to the report by Trewin c.s. the following principles should be used for managing the confidentiality of microdata. Each is discussed in the following paragraphs.

Principle 1: It is appropriate for microdata collected for official statistical purposes to be used for statistical analysis to support research as long as confidentiality is protected.

Principle 2: Microdata should only be made available for statistical purposes.

Principle 3: Provision of microdata should be consistent with legal and other necessary arrangements that ensure that confidentiality of the released microdata is protected.

Principle 4: The procedures for researcher access to microdata, as well as the uses and users of microdata, should be transparent and publicly available.

Making microdata available for research is not in contradiction with the sixth UN Fundamental Principle as long as it is not possible to identify data referring to an individual. Principle 1 does not constitute an obligation to provide microdata. The National Statistical Office should be the one to decide whether to provide microdata or not. There may be other concerns (for example, quality) that make it inappropriate to provide access to microdata. Or there may be specific persons or institutions to which it would be inappropriate to provide microdata.

For Principle 2, a distinction has to be made between statistical or analytical uses and administrative uses. In the case of statistical or analytical use, the aim is to derive statistics that refer to a group (be it of persons or legal entities). In the case of administrative use, the aim is to derive information about a particular person or legal entity to make a decision that may bring benefit or harm to the individual. For example, some requests for data may be legal (a court order) but inconsistent with this principle. It is in the interest of public confidence in the official statistical system that these requests are refused. If the use of the microdata is incompatible with statistical or analytical purposes, then microdata access should not be provided. Ethics committees or a similar arrangement may assist in situations where there is uncertainty whether to provide access or not.

Researchers are accessing microdata for research purposes but to support this research they may need to compile statistical aggregations of various forms, compile statistical distributions, fit statistical models, or analyse statistical differences between sub-populations. These uses would be consistent with statistical purposes. To the extent that this is how the microdata are being used, it could also be said to support research purposes.

With respect to Principle 3, legal arrangements to protect confidentiality should be in place before any microdata are released. However, the legal arrangements have to be complemented with administrative and technical measures to regulate the access to microdata and to ensure that individual data cannot be disclosed. The existence and visibility of such arrangements (whether in law or supplementary regulations, ordinances, etc.) are necessary to increase public confidence that microdata will be used appropriately. Legal arrangements are clearly preferable but in some countries this may not be possible and some other form of administrative arrangement should be put in place. The legal (or other arrangements) should also be cleared with the privacy authorities of countries where they exist before they are established by law. If such authorities do not exist, there may be NGOs who have a “watchdog” role on privacy matters. It would be sensible to get their support for any legal or other arrangements, or at least to address any serious concerns they might have.

In some countries, authorising legislation does not exist. At a minimum, release of microdata should be supported by some form of authority. However, an authorising legislation is a preferable approach.

Principle 4 is important to increase public confidence that microdata are being used appropriately and to show that decisions about microdata release are taken on an objective basis. It is up to the NSO to decide whether, how and to whom microdata can be released. But their decisions should be transparent. The NSO web site is an effective way of ensuring compliance and also for providing information on how to access research reports based on released microdata.

The guidelines of the report were endorsed by the CES plenary session in 2006. They addressed the need to unify the approaches internationally and to agree on core principles for dissemination of microdata. They also suggested moving towards a risk management rather than a risk avoidance approach in the provision of microdata.

The report originally contained an annex with 22 case studies describing good practices in different countries. It is a dynamic document that is updated from time to time.

UNECE Principles and Guidelines on Confidentiality Aspects of Data Integration
In 2007 and 2008 a CES Task Force chaired by Brian Pink, at that time the Australian Statistician, drafted Principles and Guidelines on Confidentiality Aspects of Data Integration.

Data integration is concerned with integrating unit record data from different administrative and/or survey sources to compile new official statistics which can then be released in their own right. In addition these integrated data sets may be used to support a range of economic and social research not possible using traditional sources.

The drafted principles and associated guidelines expand on the sixth UN Fundamental Principle by providing a common framework for assessing and mitigating legislative and other confidentiality aspects of the creation and use of integrated datasets for statistical and associated research purposes. In particular they recognise that the fundamental principles of official statistics apply equally to integrated data sets as to any other source of official statistics.

In developing these principles, it is recognised that integration of statistical data sets has become a normal part of the operations of a number of statistical offices and is generally most advanced in those countries where a heavy reliance is placed on obtaining statistical information from administrative registers. Countries that regularly undertake statistical integration usually already have a strong legislative basis and clear rules about protection of the confidentiality of personal and individual business data irrespective of whether the data has been integrated from different sources or not.

However, for many other countries the notion of integrating data from different sources for statistical and related research purposes is relatively new. The drafted principles and associated guidelines are designed to provide some clarity and consistency of application.

These Principles and Guidelines were endorsed by the CES at their June 2009 meeting.

2.3 European laws

For statistical disclosure control the following two laws are of importance.

Commission Regulation (EC) No 223/2009 of the European Parliament and Council of 11 March 2009 on European statistics. This regulation has entered into force on the 1 April 2009 and amended Regulation 322/97 on Community Statistics and Regulation 1101/2008 (previously 1588/90) on the transmission of data subject to statistical confidentiality.

This Regulation establishes the legal framework for the development, production and dissemination of European statistics, including the rules on confidentiality.

Article 2, clause 1(e) defines statistical confidentiality as "the protection of confidential data related to single statistical units which are obtained directly for statistical purposes or indirectly from administrative or other sources and implying the prohibition of use for non-statistical purposes of the data obtained and of their unlawful disclosure".

Confidential data are defined as "data which allow statistical units to be identified, either directly or indirectly, thereby disclosing individual information. To determine whether a statistical unit is identifiable, account shall be taken of all relevant means that might reasonably be used by a third party to identify the statistical unit".

Chapter V 'Statistical Confidentiality' describes in detail rules and measures that shall apply to ensure that confidential data are exclusively used for statistical purposes and how their unlawful disclosure shall be prevented (Articles 20-26). Article 23, in particular, makes provision for the access to confidential data for scientific purposes.

Commission Regulation (EC) No 557/2013 of 17 June 2013 implementing Regulation (EC) No 223/2009 of the European Parliament and the Council on European Statistics as regards access to confidential data for scientific purposes and repealing Commission Regulation (EC) No 831/2002. This Regulation establishes the conditions under which access to confidential data transmitted to the Commission (Eurostat) may be granted for enabling statistical analyses for scientific purposes, and the rules of cooperation between the Commission (Eurostat) and national statistical authorities in order to facilitate such access.

In Article 3 general principles are described. The Commission (Eurostat) may grant access to confidential data for scientific purposes held by it for the development, production or dissemination of European statistics as referred to in Article 1 of Regulation (EC) No 223/2009, provided that the following conditions are satisfied:

  • access is requested by a recognized research entity;
  • an appropriate research proposal has been submitted;
  • the requested type of confidential data for scientific purposes has been submitted;
  • access is provided either by the Commission (Eurostat) or by another access facility accredited by the Commission (Eurostat);
  • the relevant national statistical authority which provided the data has given its approval.

The original Regulation 831/2002 covered four surveys: Labour Force Survey (LFS), Continuing Vocational Training Survey (CVTS), European Community Household Panel (ECHP) and Community Innovation Survey (CIS). Later other surveys were added and the European Union Statistics on Income and Living Conditions (EU-SILC) replaced the ECHP.

Discussions about the level of detail for the microdata (to which the researchers get access) take place in the relevant Working Groups. Currently, the task to approve this level of detail is delegated to the Directors’ Group on Methodology (DIME). In addition to the DIME two Expert Groups in the ESS exist that deal with Statistical Disclosure Control (SDC): the Expert Group on SDC and the Microdata Access Network Group (MANG).

2.4 References

Commission Regulation (EC) No 831/2002 concerning access to confidential data for scientific purposes
https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX%3A32002R0831

Commission Regulation (EC) No 223/2009 on European statistics
https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=celex%3A32009R0223

Commission Regulation (EC) No 557/2013
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32013R0557

Council Regulation 1588/90
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A31990R1588

Council Regulation 322/97
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A31997R0322

European Statistics Code of Practice
https://ec.europa.eu/eurostat/web/products-catalogues/-/european-statistics-code-of-practice-revised-edition-2017

ISI (1985) Declaration on Professional Ethics
https://isi-web.org/declaration-professional-ethics

Pink, B et al. (2009) Principles and Guidelines on Confidentiality Aspects of Data Integration UNECE United Nations Economic commission for Europe
https://digitallibrary.un.org/record/651518?v=pdf

Trewin, D et al. (2007) Principles and Guidelines of Good Practice for Managing Statistical Confidentiality and Microdata Access UNECE United Nations Economic commission for Europe
https://unece.org/fileadmin/DAM/stats/publications/Managing.statistical.confidentiality.and.microdata.access.pdf