Sage Journals: Discover world-class research

Abstract

Background

Governments across the World Health Organization (WHO) European Region have prioritised dashboards for reporting COVID-19 data. The ubiquitous use of dashboards for public reporting is a novel phenomenon.

Objective

This study explores the development of COVID-19 dashboards during the first year of the pandemic and identifies common barriers, enablers and lessons from the experiences of teams responsible for their development.

Methods

We applied multiple methods to identify and recruit COVID-19 dashboard teams, using a purposive, quota sampling approach. Semi-structured group interviews were conducted from April to June 2021. Using elaborative coding and thematic analysis, we derived descriptive and explanatory themes from the interview data. A validation workshop was held with study participants in June 2021.

Results

Eighty informants participated, representing 33 national COVID-19 dashboard teams across the WHO European Region. Most dashboards were launched swiftly during the first months of the pandemic, February to May 2020. The urgency, intense workload, limited human resources, data and privacy constraints and public scrutiny were common challenges in the initial development stage. Themes related to barriers or enablers were identified, pertaining to the pre-pandemic context, pandemic itself, people and processes and software, data and users. Lessons emerged around the themes of simplicity, trust, partnership, software and data and change.

Conclusions

COVID-19 dashboards were developed in a learning-by-doing approach. The experiences of teams reveal that initial underpreparedness was offset by high-level political endorsement, the professionalism of teams, accelerated data improvements and immediate support with commercial software solutions. To leverage the full potential of dashboards for health data reporting, investments are needed at the team, national and pan-European levels.

Keywords

COVID-19 public reporting of healthcare data public health surveillance health information management WHO European Region dashboard

Introduction

Governments, as the stewards of healthcare systems, have the chief responsibility for protecting and promoting the health and well-being of the population.¹ The stewardship role includes collecting and reporting relevant information and supporting its use as performance intelligence by all health system actors, including the general public.^2,3 This task has taken on new pertinence in the context of a public health emergency such as the COVID-19 pandemic.^4–7 Governments worldwide have prioritised tools for delivering pandemic-related information, most often through the development of public web-based dashboards.^8,9

Dashboards can be characterised as dynamic, visual displays of key performance indicators, arranged on a single screen for viewing at a glance.^10–12 The ubiquitous use of dashboards as public reporting tools during a pandemic is a novel development^13,14, but the use case for dashboards in a pandemic is clear. Contrary to static reporting, through their dynamic nature, they iterate content and its display daily, evolving with the stages of the pandemic.¹⁵ By design, dashboards can manage large datasets and this, together with their near real-time reporting capabilities, makes them highly responsive to the information urgency in a pandemic.⁸ And, when paired with geographic information system (GIS) and interactive drill-downs, dashboards are critical for local monitoring, reporting and decision making.¹²

In the health sector, dashboards have traditionally been used for internal purposes, assisting managers in strategic and operational decision making, particularly in hospitals¹⁶ and supporting clinicians in clinical care and quality improvement.¹¹ There are also notable examples of public web-based dashboards for international health system benchmarking.^17–22 In contrast to COVID-19 dashboards, these have traditionally not been updated in near real-time. Previous studies have explored optimising the design of dashboards in healthcare,¹⁰ their effects on quality in clinical practice¹¹ and their development and implementation cycles.^15,23 In the context of the COVID-19 pandemic, scientific accounts have documented the technical development of dashboards^9,24–28 and their applications in clinical practice.^29,30 From a healthcare performance intelligence perspective, our research group HealthPros³¹ has conducted international comparative research on COVID-19 dashboards, exploring features common to highly actionable dashboards³² and their evolution over time.^33,34

In this context, there are two critical gaps in the available scientific evidence. First, describing the processes of developing COVID-19 dashboards over the course of the pandemic's first year has predominately focussed on individual, anecdotal country accounts (e.g.^35–38). While these provide some insight into the process, scientific methods to describe their development remains needed, offering critical historical intelligence for the future. Several systematic approaches to capture pandemic experiences have already been published, offering important insights from the perspective of healthcare providers,^39–41 patients^42,43 and the general public.^44–46 Second, the anecdotal evidence and our previous COVID-19 dashboard research signal a gap in cross-country collaboration in the development process. This is despite the presence of several international actors (e.g. World Health Organization (WHO), European Centre for Disease Control and Eurostat) that have led initiatives for multi-country COVID-19 surveillance. The absence of established cross-country exchanges suggests there is untapped potential for the sharing of experiences and learning between countries.

Objective and research questions

To support governments during the current pandemic, and to better prepare for future health threats as well as for other potential uses of dashboards, our research group set out to conduct a multi-country study on the process of developing COVID-19 dashboards across Europe and Central Asia. To do so, we partnered with the WHO Regional Office for Europe, a key convening actor in the region and counterpart of our targeted health system stewards. With this aim, our study was guided by two research questions: (1) How can the development process of COVID-19 dashboards during the first year of the pandemic be described from the perspective of the teams responsible for development? (2) What common barriers, enablers and lessons can be derived from their experiences?

Methods

Design

The study adheres to the Consolidated Criteria for Reporting Qualitative Research.⁴⁷ To retrospectively examine the development process, we undertook a series of semi-structured group interviews with COVID-19 dashboard developer teams across the WHO European Region. We employed multiple methods to identify and recruit dashboard teams, using a purposive, quota sampling approach. Group interviews in the local language of teams, to the extent possible, provided rich, collective team reflections on experiences with the process.⁴⁸ To address our research questions, we adapted an approach previously developed by the study team to describe and assess the actionability of COVID-19 dashboards.³² We also drew on the findings of prior COVID-19 dashboard research to help determine the characteristics and features to explore³² (Table 1).

Table 1.

Overview of characteristics and features explored.

Focus by research question	Characteristics and features explored
Development process	Responsible organisations, teams and launch
	Aims, users and content
	Data sources and breakdowns (geographic, population)
	Data display, interpretation and visualisation
	Future plans
Reflections on process	Barriers
	Enablers
	Lessons learned

The study team included researchers from HealthPros and WHO European experts on health data. The multinational nature of the study team ensured broad and complementary expertise on contexts, research, policy and subject matter. Team members conducting interviews (four women and six men) had previously researched COVID-19 dashboards.^32,49 They were trained in health services research and had prior research and professional experience in countries of the WHO European Region. Interviewers were collectively proficient in 13 languages used in the WHO European Region.

The research protocol was developed in accordance with the ethical requirements of our primary research base, the Amsterdam University Medical Centre affiliated with the University of Amsterdam. Participants provided written consent during the recruitment stage and verbally restated their consent at the start of their interview. Confidentiality was assured by assigning each participating dashboard team a random code (e.g. D1) and removing identifying information from verbatim quotes used throughout the paper.

Sample of dashboards and informant recruitment

We defined our target sample of COVID-19 dashboards using five criteria: (1) reporting of key performance indicators related to the pandemic; (2) use of some form of visualisation; (3) public availability in a web-based format; (4) reporting at national level within the WHO European Region; and (5) development by a governmental organisation or appointed authority. To maximise the generalisability of our findings, we set out to recruit a geographically representative sample of COVID-19 dashboards from the region's 53 Member States. We applied the country subgroups from the WHO's European Health for All database⁵⁰ and set a target of 50% representation within each: European Union (EU) members before May 2004 (EU15) (n = 15); EU members after May 2004 (EU13) (n = 13); Commonwealth of Independent States (CIS) members (n = 12); and other European Region countries not included in those groups (n = 13).

To identify COVID-19 dashboards, we consulted the affiliated country data sources of the international COVID-19 Situation Dashboard of the WHO European Region⁵¹ as well as the sample from our previous COVID-19 dashboard work.³² Additionally, we manually web-searched national and governmental COVID-19 webpages. Our target informants were referred to as dashboard teams, specified as members of core teams directly involved in developing and managing national COVID-19 dashboards, ideally right from their inception.

Following the identification of appropriate dashboards, we used multiple methods to recruit dashboard teams for interviews. In cases where contact details were listed, teams were reached directly via email and/or through social media (Twitter, Facebook and LinkedIn). Alternatively, where no direct contact was publicly available, we applied a snowball approach, soliciting the advice of existing networks on health information systems across Europe, including country focal points of the European Health Information Initiative (EHII)⁵² and the Population Health Information Research Infrastructure (PHIRI) project,⁵³ as well as other experts known to the study team. To further the recruitment process, the WHO together with the study team, organised and hosted a public webinar in March 2021, convening EHII and PHIRI focal points, at which the study protocol was presented. Approximately 45 participants attended and, when appropriate, bilateral contacts were made by the study team. For countries in which these methods did not result in direct contact with dashboard teams, support from WHO Country Offices was obtained to contact their respective ministries of health, informing them of the study and soliciting participation. During correspondence with all prospective informants, an overview of the study was provided in English or Russian (online Supplemental material). When possible, correspondence took place in the local language.

Data collection

A detailed interview guide was developed by the first authors (EB and DI) and reviewed by the study team. Once finalised, a training session for interviewers was organised to calibrate the interview process. Interviews consistently explored two main themes: (1) the dashboard's development and (2) reflections on the process over the course of the first year of the pandemic (Table 1). A brief version of the interview guide was prepared in English and Russian and provided to informants in advance of the interview (online Supplemental material). A structured pre-interview process was developed for interviewers to familiarise themselves with the corresponding dashboard. For this, we adapted our descriptive COVID-19 dashboard assessment tool³² and approach to scoring a dashboard's actionability.

Between April and June 2021, 60-minute semi-structured group interviews were conducted with participating dashboard teams, either virtually or in person. Dashboard teams were assigned to interviewers based on their language and context expertise. Interviews were conducted in pairs (a lead interviewer and a second team member), where language competencies were allowed and provided that the lead interviewer was not the first author (online Supplemental material for the distribution of interviewers and languages used). With the agreement of informants, interviews were recorded, transcribed verbatim and, when necessary, translated to English. Transcripts were made available to informants upon request Interview data were stored by the first authors. Fortnightly meetings were organised for interviewers to exchange impressions on the process and to update themes.

Data analysis

The first authors analysed the translated interview transcripts to identify descriptive and explanatory themes using elaborative coding⁵⁴ and thematic analysis⁵⁵ in an Excel tool developed in the approach set out by Meyer and Avery.⁵⁶ The analysis process included familiarisation with the data, development and piloting of a coding framework, independent coding, peer review, mapping and interpretation of results. The coding framework was aligned with the research questions and was developed based on the characteristics and features (Table 1) of the semi-structured interviews (level 1). Additional themes (level 2) were generated through open (unrestricted) coding. The first authors independently coded three test transcripts each, then collaboratively reconciled and revised their coding. The approach was reviewed and discussed during the piloting phase with two other study team members (NK and DK) and an external qualitative researcher. The transcripts were divided between the first authors for independent coding.

Once coding had been peer-reviewed by the second coder, and reconciled by the first, a consolidated dataset for analysis was developed. For analysis of the dataset, the characteristics/features explored were divided equally between the first authors for re-reading, mapping and interpretation. In this process, we iteratively noted recurrent themes, as well as outliers. For reporting on the results by the research question, verbatim quotes were extracted from the transcripts. To ensure the validity of the findings, we employed different techniques, including researcher reflexivity, debriefing with all interviewers and reviews by the full study team. Additionally, validation of the findings with informants was organised through a virtual workshop, again hosted by the WHO and attended by 55 study participants in June 2021.

Results

Sample of participating COVID-19 dashboards

Five WHO European Region Member States (5/53, 9%) did not have an applicable dashboard at the time of sampling. Three COVID-19 dashboards from non-members, yet representing territories within the WHO European Region, were identified through our sampling and included during the recruitment stage (Figure 1).

Figure 1.

Geographic representation of participating COVID-19 dashboard teams.

In total, 33 dashboard teams participated in the study; see online Supplemental material for direct links to each dashboard. The dashboards represented 31 out of 53 WHO European Region Member States, and 65% (33/48) of total Member States with eligible dashboards (Figure 1). Seventeen Member States (17/48, 35%) were unreachable, unavailable or uninterested in participating. We met our target for representation in all but one regional subgroup, the CIS countries, where a quarter of Member States did not have an applicable dashboard (3/12, 25%). Overall, participation rates of eligible dashboards by subgroups ranged from 83% (10/12) of EU13 countries to 73% (11/15) of EU15 countries, to 57% (8/14) of other countries or territories and to 44% (4/9) of CIS countries.

Interviews engaged a total of 80 informants (45 men and 35 women). On average, two informants per dashboard were interviewed (range: 1–8), with three or more informants contributing to 42% (14/33) of dashboard teams. Two dashboard teams provided written responses and two others required a second interview to finalise data collection. One informant was directly involved in the development of two dashboards. Interviews were conducted in eleven languages (online Supplemental material).

Description of the development process

Responsible organisations, teams and launch: The development of dashboards was initiated predominately by high-level officials, namely a country's health minister, prime minister or president. Units of government or ministries of health (16/33, 48%) or national public health institutes (12/33, 36%) were mostly appointed to lead their development (see online Supplemental material for a listing by country/territory). In a few cases, governments appointed other organisations (5/33, 15%), such as NGOs, private companies or academic institutions, to develop the national dashboard, typically due to resource constraints. An awareness of the Johns Hopkins COVID-19 dashboard⁵⁷ and other national COVID-19 dashboards was often cited in connection with the initial request.

The launch dates of participating dashboards were in three main time periods (listed in online Supplemental material): February to May 2020, the first months of the pandemic (24/33, 73%); late 2020, in parallel to the pandemic's second wave across Europe (6/33, 18%); and early 2021, in connection with vaccination campaigns (3/33, 9%). Dashboards launched after the first months of the pandemic shared similar challenges, often data constraints or issues in identifying a responsible organisation. The unprecedented speed and workload involved in launching dashboards was a recurrent theme across teams. Many could vividly recount the initial days, recalling the level of uncertainty that characterised the process. As one informant remarked, ‘We were flying the plane as we were building it’ (D31).

While some teams had prior experience with developing dashboards for internal use, most had never worked on dashboards intended for public reporting. Where possible, internal teams of data management units were re-purposed or new internal teams formed. Often, non-COVID-19 activities were paused. For a quarter of dashboards (8/33, 25%), external teams were contracted to develop the dashboard. In two instances these teams worked on a volunteer basis. Most teams started small, typically with one or two persons, though growing with time to about three to five core persons, and in some contexts, to more than twenty. The importance of multidisciplinary teams was emphasised, involving epidemiologists, public health specialists, information technology professionals, data analysts, policy experts and administrative staff. As teams expanded, additional expertise engaged included business intelligence and analytics experts, GIS specialists, user experience researchers and communication professionals. Support received from private front-end dashboard software suppliers was described as a critical addition to teams, especially in the early stages following the launch.

Aims, users and content: The dashboards were depicted as a vehicle for informing, but also as a tool for partnering with the public to ‘achieve greater participation of people in fighting the pandemic’ (D9). Specific aims and target users were often implied rather than explicitly defined, with many citing the ‘chaotic’ (D2) period that characterised the initial phase as a cause for this. Above all, the dashboards targeted the general public, though no dashboard team described having direct or formal contact with the public in the early development stage. Time constraints were consistently cited as the cause for this: ‘We would normally have done some user engagement to understand user needs, but the pace and expectation and demand to get the information out was so high’ (D31). Other target users included national, regional and municipal officials, health professionals and the media.

In the early stages following a dashboard's launch, the monitoring of user analytics was not pursued, for one common reason: ‘there wasn’t time for deep analysis of user behaviour’ (D1). The intensity of the dashboards’ use was described as an effect of high expectations and ‘insatiable demands [for data]’ (D8), new requests, and questions. While internal feedback mechanisms to dashboard teams were well established, with dedicated pandemic crisis management teams or committees meeting daily, a structured process to manage feedback from the general public was largely absent. Communication teams were described as playing an important role in triaging such comments, predominantly received via email and social media. However, the core dashboard team was typically tasked with providing technical replies – a demanding task was given the magnitude and the work pace.

Most dashboards (27/33, 82%) reported two or more types of content, most often data on spread and death, on health systems or on vaccination (online Supplemental material). More than half (21/33, 64%) had added vaccination data by late 2020 or early 2021, either as new tabs or as separate dashboards. The latter were typically attributable to one of three reasons or a combination: (1) a different organisation was mandated to coordinate and report on vaccinations; (2) existing or new data collection infrastructure for vaccinations differed from the epidemiological system; and (3) a different software solution was used. Beyond the addition of vaccination data, major changes were usually avoided, due either to lack of time or to concerns about reactions from a public accustomed to the dashboard and trusting the original version.

Data sources and breakdowns: ‘In the beginning, there were Excel spreadsheets’ (D24). Many recounted similar intense manual data processing, especially countries or territories with more decentralised, less digitised information systems. The availability, completeness and quality of data ultimately played an important role in determining what indicators could be reported, especially in the initial stages. Many described the trade-off between speed and quality, facing intense demands to publish data in near real-time. This challenge intensified as the volume of data points increased with time. As one informant recounted, ‘Our data are usually ready for deployment 15 to 25 minutes before 4:00 p.m., which is the time at which we usually deploy the data. So we have that much time to curate 40 million records’ (D10). Choosing to report open data was a political decision, typically made with a view to ensuring full transparency. More than half of the dashboards reported open data (21/33, 64%), meaning full datasets could be directly downloaded from the webpage (online Supplemental material). Some went so far as to completely democratise their reporting: ‘The prime minister of [country] sees the data at the same time as the guy down the road’ (D10).

Data protection rules influenced a number of dashboard features, including what indicators were reported, data sources used and the geographic and population breakdowns applied. Clearances about what could be published reportedly caused delays, as one dashboard team noted in reference to vaccination data: ‘We have had it ready for months, but right now the lawyers are debating, writing back and forth with the ministry’ (D16). Fear of exposing personal health data was a repeated issue, with different interpretations of the lowest level of granularity when reporting cases locally, such as groups ‘larger than 20’ (D16), ‘smaller than 10’ (D2) or ‘no less than 5’ (D19).

Many reported that there was great interest in breaking down data to local or municipal levels. This became increasingly relevant with the progression of the pandemic, as infection control measures were being introduced subnationally. Possibilities to report data more locally also improved with time, as data collection processes became increasingly automated and of better quality. Persisting challenges included protecting the privacy and ensuring that declining numbers were not universally interpreted by the public as low levels of risk. Resolving issues of incomplete denominators was also faced by some teams, which described outdated census data and challenges in recording migrants, undocumented persons and seasonal workers. Beyond geographic breakdowns, disaggregations by age and gender were common, but the use of ethnicity- and race-related data and socioeconomic status breakdowns were generally not pursued. In some cases that was due to a lack of data, but more often it was a political decision. Specifically, some informants reported uncertainty about its relevance for decision making in the general public and fears of provoking discrimination.

Data display, interpretation and visualisation: Dashboard teams relied on front-end dashboard display solutions either developed in-house (14/33, 42%) or commercially available (19/33, 58%). Most started with commercial solutions (21/33, 64%), typically ArcGIS™ by Esri (11/33, 33%), which had a ‘COVID dashboard module’ by early March 2020 (online Supplemental material). Selecting a commercial solution was determined by a range of considerations: a team members’ previous experience with the software, which eased the learning curve; available free of charge (often for a limited period), meaning that public procurement processes could be avoided; proactive outreach and support by vendors; comparison with solutions used by other countries; and technical considerations, such as the degree of automation. Despite the speed-to-launch advantages of commercial solutions, they often posed limitations in terms of available templates and customisation. Most notably, the software selected was described as limiting the range and types of visualisations and multilanguage capabilities. Additionally, most commercial solutions were cloud-based, which was described by some as suboptimal, predominately due to data security concerns. For such reasons, four dashboard teams (12%) switched software over time.

Many described the task of visualising data in a clear, understandable way as a challenge. Incorporating policy measures in order to explain data trends was seen by some as beyond their function of reporting facts. Providing detailed explanations and interpretations of the data were rather left to the media or to what were described as ‘data enthusiasts’ (D24) among the public. The dashboard was often part of a larger COVID-19 data and reporting ecosystem. Supplementary reporting efforts, mostly through static weekly situation reports, typically included additional indicators and more sophisticated analytics. These reports accommodated more text than the dashboard, making detailed explanations of data possible.

The dashboard teams described the importance placed on preparing simple, easy-to-understand and interactive visualisations. In the early stages, visualisations were often not prioritised, as one developer described: ‘I can imagine maybe hundreds of other ways to visualise data describing the COVID-19 situation. Unfortunately, because of lack of time, we decided to implement only the simple versions’ (D1). Maps were consistently used to present local information, though privacy considerations also influenced visualisations, with some describing the challenge of avoiding the suggestion that specific addresses were sites of cases and outbreaks.

Future plans: In discussions of what is next for COVID-19 dashboards, four non-mutually exclusive scenarios were identified: (1) continuing to update existing dashboards, though less frequently over time; (2) further developing content (e.g. on vaccines and wastewater studies), data management (e.g. automation, quality and open data), design (e.g. visuals and organisation) and user elements (e.g. low literacy levels and user behaviour studies); (3) exploring non-COVID-19 uses of dashboards for monitoring other communicable diseases (e.g. influenza) and registry data (e.g. cancers); and (4) preparedness planning, including investing in centralised data warehouses, in-house dashboard teams, coordinating across European countries and exploring alternative server and software options.

Barriers and enablers

Six main themes and fifteen subthemes were identified as recurrent barriers for some dashboard teams, yet enablers for others. These are briefly described and listed in Table 2a and b.

Table 2a.

Summary of themes and illustrative quotes describing barriers.

Theme	Subtheme	Barrier	Illustrative quote
Pre-pandemic context	Data infrastructure	Lack of data infrastructure. Outdated, slow processes of data collection and processing. Data ownership and custodianship challenges. Highly decentralised data platforms.	‘The epidemiological surveillance systems were not effectively prepared for a pandemic of this scale, either in the volume of information analysed or in the usability of the information systems themselves’. (D9)
Pre-pandemic context	Privacy regulations	Undefined rules and/or lack of practical experience in publishing health data and in conditions for ensuring anonymity.
Pandemic context	Urgency	Lack of time to analyse user behaviour, manage feedback, improve visualisations and engage across stakeholders.	‘There was quite a lot of shuffling about who was responsible – which institution should be responsible for announcing and disseminating aggregated data’. (D4)
	Leadership	Political instability and influence on dashboard's content.
	Mandate	Lack of common understanding of the purpose of a dashboard, with different interpretations of its use and target users. Unclear responsibilities.
People and processes	Human resources	Huge workload. Lack of human resources and competencies in working with dashboards. Regular work tasks in parallel.	‘At the resource level, we are quite limited. It's complicated to find the right people with the required expertise. It's difficult to recruit, and the team has changed quite a lot over time’. (D17)
	Partnerships	Lack of time and possibilities to engage users and other key stakeholders through ‘traditional’ processes of engagement.
	Processes	Slow and convoluted public procurement processes. Underprepared or nonexistent plans for pandemic situations. Issues with prioritising resources.
Software	Availability	Prohibitive pricing on licensing fees in the immediate and longer term. Slow speed of data processing and publishing.	‘We needed a Venn diagram. Like really needed it.… But we just couldn’t use it because ArcGIS doesn’t have one. They only have bar charts, maps and pie charts’. (D4)
Software	Functionality	Software dictating the look and feel of the dashboard. Limitations in visualisation options.
Data	Availability and quality	Data unavailable and/or not sufficient in timeliness, completeness, structure, consistency or granularity across geographies. Issues to link data from various sources. Labour-intensive, error-prone data processing. Outdated population registration data.	‘One primary point that was a large problem was the exact place where someone lives, which is taken from the national census. And they didn’t want to share it with us because we were not legally able to obtain them.… We have information about [approximately half the population]. The rest of them, we’re not sure where they live now’. (D27)
	Data culture	Data siloed by different data custodians. Different data standards among sources. Lack of open data culture.
	Automation	Challenges in setting update times and cut-offs across data sources and custodians. Demand for real-time data compromises quality.
Users	Target groups	Lack of defined target audience. Broad definition of the user group (e.g. general public, regional public health authorities, national policymakers, media).	‘The biggest thing was the reactions in the media. Kind of, a lot of negativity online about everything you do, and not enough positivity’. (D2)
Users	Information needs, user experience and expectations	Limited or no knowledge of users” information needs. No systematic way of dealing with user feedback. Oversimplifying content, thereby posing the risk of misinterpretation. Users with high expectations and low data literacy. Negative reactions to modifications to content and visualisations.

Table 2b.

Summary of themes and illustrative quotes describing enablers.

Theme	Subtheme	Enablers	Illustrative quote
Pre-pandemic context	Data infrastructure	Electronic, centralised data flows with automated data management. Ability to link data sources.	‘The existing infrastructure, the central health information system and so on, those were definitely enabling factors. If we hadn’t had them, or integration with the labs, with the hospitals, we would have been in the Stone Age’. (D24)
Pre-pandemic context	Privacy regulations	Supportive privacy and security legislation and practice. Enabling state-of-emergency conditions.
Pandemic context	Urgency	End-goal orientation fostering committed and focussed efforts. Not being a perfectionist ‘Once-in-a-lifetime experience’ as a motivator.	‘chain of command was clear. That was an enabler, for sure. We all knew [the dashboard] was the official communication channel. It was, practically considered, an extended arm of the government’. (D15)
	Leadership	Political and upper-management support and endorsement in providing access to sufficient resources. High-level directives.
	Mandate	Clear purpose and mandate. High degree of autonomy. Easy access to data. Skills needed in-house and/or potential to outsource for added capacity.
People and processes	Human resources	Committed, competent, multidisciplinary and proactive team. Prior experience with public reporting, dashboards and visualisations.	‘This then also led, with an agile development, to changing relationships between people, who became much less structured and hierarchical and became much more intellectual and free’. (D12)
	Partnerships	Improvements to intra- and inter-organisational communication and collaboration. Need-based stakeholder collaboration and engagement, including communication specialists and decision makers.
	Processes	Flattened hierarchy. Streamlined and agile internal organisation. Change of mentality towards a more operational one.
Software	Availability	Supportive technological solutions. Commercial software offered free of charge, for a period. Reusing existing solutions.	‘There is no doubt that ten years ago the management of this pandemic would have been much more difficult, and now we have technology that has enabled us to fight the pandemic much more effectively’. (D9)
Software	Functionality	Easy to build and automate. Flexible and easy to maintain once set up. Extensive vendor support.
Data	Availability and quality	Available, accurate, and timely data of sufficient granularity. Ability to link data across sources and organisations.	‘Because we are a data department … it's very easy for us, since we are data managers, all of us, and we have access to all the data. So when we’re asked to do something, we don’t have to ask anybody “Can you get me this data?” This access is very important for quick results’. (D16)
	Data culture	Aligned data standards and methodologies. Culture of data interoperability, open data and secondary data use.
	Automation	Streamlined data processes including automation of collection, processing and reporting.
Users	Target groups	Clearer definition of the target audience. Separate dashboards or modules for different user groups with different information needs. Curious, rather than malicious, users. Partnership with media. Support and readiness for data-driven decision-making at all levels. Dissemination aids, including social media platforms, high-level officials, transitional media, and data champions.	‘We’ve had a good relationship with the media and there's a short communication line from the public to us. So, if there's something the public is insecure about or wants to know more about or wants, [they can easily reach us], and that's good’. (D26)
Users	Information needs, user experience and expectations	Systematic approaches to researching user experience, implementing improvements and managing user feedback. Tradition of public using data for decision-making.

Pre-pandemic context: The existing data infrastructure was a major challenge facing some dashboard teams, exposing the limitations of traditional data collection and processing for near real-time dashboard reporting. In contrast, teams working in settings with more advanced data systems and a culture of data use and reuse credited this as a contributor to their success. Similarly, some described the challenge resulting from highly decentralised data structures and processes, which contributed to a lack of clarity around data ownership and custodianship. The limited level of preparedness for handling privacy regulations was a key barrier for some, whereas prior experience with interpreting privacy and security legislation in the context of public reporting was an enabler for others.

Pandemic context: For most teams, a chronic lack of time caused by the pandemic's urgency and the demand for publishing near real-time data meant that other needs such as analysing user behaviour, managing feedback, improving visualisations and engaging across stakeholders became secondary. However, some teams did cite the constant urgency of the pandemic as an incentive for streamlined processes and end-goal orientation, which in turn fostered committed teams and focussed efforts. The political context, including changes in leadership positions and the content demands of high-level decision makers, was highlighted as a barrier for some. An additional barrier was for some a lack of a common understanding of the purposes and audiences of dashboards and their position in the COVID-19 data ecosystem. Contrary to that experience, some teams credited high-level political endorsements for the development of national COVID-19 dashboards as a key enabler, providing ample resources and direction, yet autonomy, to the teams developing them.

People and processes: Seven-day work weeks, an immense workload and shortages of competent specialists hindered the development of dashboards in the experience of some teams. So did bureaucratic processes, including those involving partnering with stakeholders, accessing and linking data and public procurement. Others, however, considered assets such as having experienced data dashboard and business analytics teams in place, working across departments and organisations and engaging new team members with necessary competencies to be enabling factors. Flattening hierarchical structures and streamlining processes to facilitate decision making were also seen as playing supporting roles.

Software: Front-end dashboard software solutions occasionally impaired the development of dashboards, according to some teams, due to their limited visualisation potential, lack of multilanguage functionality and other customisation options, or prohibitive pricing. For others, the reuse of existing, in-house-developed data analytics and dashboard tools (where available) provided more flexibility. If in-house options were not available, some teams described commercial software as an aid to accelerating the launch of dashboards. Some software vendors waived initial fees and provided direct support at the outset, and this was cited as a key enabling factor.

Data: Publishing reliable, accurate, consistent and timely data proved challenging for many dashboard teams. Agreeing on data standards, including daily cut-off times, and the absence of granular data needed for reporting close to home and broken down by different population subgroups, were described as key hindrances. Conversely, the availability of interoperable data, coordination across data custodians, existing data and methodology standards and cultures of secondary data use and open data publishing were cited by others as benefits.

Users: Managing users was described by some teams as a critical challenge, who recounted intense scrutiny over the content of dashboards, issues of user misinterpretation of data and negative reactions to mistakes or modifications. Some teams also detailed high expectations from users (e.g. for real-time reporting), challenges in explaining methodologies (e.g. to lay audiences) and a lack of systematised processes to handle user feedback. For others, having clearly defined target audiences, knowing their information needs, engaging with media outlets, systematically improving user experience and handling feedback was a way of partnering with users, and hence an overall advantage to the process. Transparency on methods, and acknowledging mistakes that inevitably happen at this volume and speed of work, were perceived as enablers.

Lessons learned

When informants were asked to consider, with the benefit of hindsight, what dashboard teams would have done differently or what advice they would offer others to best prepare for public reporting in the context of a crisis, five recurrent themes were identified. One theme was the importance of simplicity – reporting only essential information, prioritising content that can be easily interpreted and supported by explanations. The second was the importance of trust. Teams described the inevitability of errors, given the urgency and volume of data, and the importance of disclosing errors as they happen. Using open data and prioritising data security and privacy were also important lessons for building user trust Third, the necessity of working in partnerships was emphasised – working in agile and collaborative ways with system leaders as well as across in-house units, with other stakeholders and with the target audience. A fourth theme was the importance of software and data. While this was intuitive and recognised at the outset, teams were continuously confronted with the parameters set by the software chosen and were reminded of the importance of automating processes and investing in quality data. The last theme was about confronting the truly dynamic nature of dashboards, finding ways to learn from others in order to improve, adapt to the stages of the pandemic and embed dashboards within other reporting modalities. Recurrent themes, lessons learned and representative quotes are summarised in Table 3.

Table 3.

Summary of lessons learned, recurrent themes and representative quotes.

Theme	Lesson	Representative quote
Simplicity	Report essential information only	‘From the beginning, one of the main aims was to make it as simple as possible to understand. I avoided providing additional information, so that it was as simple as possible’. (D7)
	Ensure interpretation is straightforward	‘I mean, that's not to say there aren’t a lot of other types of users, but our primary focus is always that this is for the public and therefore anything that is there for the public should be understandable’. (D10)
	Include explanations	‘We learned it is super-extra important to describe all the measures as soon as possible … the first-time people see a number, [that is] the way they understand it and they will continue misinterpreting it forever’. (D4)
Trust	Report errors	‘Be brave enough to try it out, to put stuff out and also be transparent while doing it. And, if you have mistakes, also be transparent about it’. (D24)
	Use open data	‘Publishing open data took a lot of work off our hands and made it a lot easier, more transparent and, yeah, that's one of the lessons from the crisis, definitely’. (D24)
	Prioritise data security and privacy	‘I also noticed issues in terms of data protection.… I think that's another lesson learned, integrating data protection right from the get-go to make it easier later on’. (D17)
Partnership	Involve the right people	‘It was a kind of multidisciplinary team that met to design the needs for the dashboard, the functionalities, the data that is necessary to communicate, the functions of the dashboard’. (D29)
	Ensure high-level endorsement	‘And then with the support and the initiative of the minister's office, the decision was made to come together in collaboration between my agency and the ministry. We resolved to add transparency and to be more efficient in ensuring a dedicated website for coronavirus data’. (D20)
	Listen to your audience	‘We also take into account the needs of politicians: what do they need to make their decisions? So, there is the public health side and the political side that have to be taken into account’. (D17)
Software and data	Choose software wisely	‘The tooling influences a lot how the dashboard looks in the end and what is feasible’. (D2)
	Automate when possible	‘Unlike other diseases, we are talking about a very large volume of data … using more basic tools that, at this point, no longer works. So everything we want to represent must be represented in an automatic way, with a great capacity to go “drinking at the source”. we are talking about a lot of data, millions of millions of data’. (D9)
	Quality data is essential	‘If the data is not collected in a way they can utilise [it], they will not be able to produce dashboards … Try to predict how they are going to grow their data warehouses, because that is one of the problems as the pandemic progresses: you need to respond quickly’. (D7)
Change	Learn from others	‘We looked at many of the versions [from countries] to see which one will be more appropriate for [us] and how we can design it better for the country. So, examples from other countries helped a lot, I think’. (D29)
	Adapt with the situation	‘I would say they⁵⁸ evolved very organically as the pandemic evolved, and the data became available in ways that were meaningful to visualise to the public and to the media’. (D32)
	Embed in reporting ecosystem	‘We never saw the website as a separate entity, but as a central place from which we disseminate information to other communication platforms’. (D15)

Discussion

In this study, we retrospectively explored the development of COVID-19 dashboards over the first year of the pandemic across the WHO European Region. It is a geographically vast region, comprised of countries and territories with widely differing population sizes, health information systems at varying stages of development,^59,60 and diverse data and administrative cultures and traditions,⁴ among other key differences. Nonetheless, we encountered more similarities than differences in the development stories related to the 33 national COVID-19 dashboard teams we met with. The factors that hindered or facilitated the development process, and the resulting lessons learned, also shared many commonalities.

COVID-19 dashboards were developed across the region, albeit with little cross-national cooperation

The overall speed with which governments requested the development of public-facing COVID-19 reporting reflects the WHO European Region's tradition of prioritising good governance.⁶¹ Even so, the ubiquitous use of dashboards for delivering COVID-19 data was driven more so by an international ripple effect than by the activation of pre-existing emergency response plans. The wide uptake of dashboards appears to have been triggered by early adopters (e.g. ⁹) and sustained by a proactive commercial software vendor market. In that light, the observation by Bouckaert et al.⁴ that ‘coping with the crisis has been first and foremost an issue of the national states, whereas the European voice has been weak’ likewise holds true for COVID-19 public reporting tools, including dashboards. To foster more pan-European collaboration, the convening role of international actors needs to be leveraged. That could advance a common approach to public reporting using dashboards and an exchange of lessons across contexts.

Data behind dashboards is crucial, but so are visual presentations and data interpretation

Dashboards feed on relevant, quality data. In the literature, data sources are cited as an essential aspect of dashboard development.^15,62,63 Initially, national health information systems struggled to provide accurate, timely data for COVID-19 dashboards – a key challenge also reported by other observers.^14,64,65 Our findings signal the need for continued investment in national-level health data sources that are integrated and interoperable, and in digital infrastructure that spans systems of secondary, primary and social care.¹⁴ Preparedness to tackle data privacy and security issues, including practical applications of the General Data Protection Regulation (GDPR) in EU contexts, needs further prioritisation, with the guiding aim of reporting data in ‘proportionate, ethical and privacy-preserving’ ways.¹⁴ Cross-national efforts like the European Health Data Space⁶⁶ may advance the development of common data standards, indicator sets and methodologies, thus enhancing both national and pan-European reporting.

While dashboards are an important communication tool, they are, like other digital technologies,¹⁴ not a silver bullet. Previous research has emphasised that the features of dashboards must fit their intended purposes.¹⁵ The actionability of data for end users depends on how clearly and understandably the information is communicated.^67–69 We observed differences in the extent to which teams were exploring, discussing and defining their dashboards’ purposes. Some saw the purpose as solely presenting data (raw numbers) for the public to interpret on its own; others endeavoured to provide explanations using narratives or visual methods. As reported in other studies, the ways in which information is presented may affect not only the subjective perception but also the objective comprehension of the information.^11,70,71 Systematic approaches to exploring user needs and use patterns are necessary if dashboards are to bridge the gap between mere managerial tools and full-fledged public reporting devices.

Dashboards seem to be ‘here to stay’ for monitoring and reporting

The interviewed teams agreed that dashboards – as well as other near real-time, web-based, interactive and visual reporting tools – are the likely future of public reporting. COVID-19 dashboards have served to demonstrate how much can be achieved with limited resources and in high urgency. They also serve to flag imminent areas for improvement and to spot new challenges like potentially harmful misinformation.⁷² Running dashboards in the longer term will likely include further expanding their policy, public health and clinical use into areas such as resilience and recovery plans,⁷³ late complications of COVID-19^74–76 and its influence on specific populations and patient groups, such as people living with HIV⁷⁷ or diabetes,^78–80 but also non-COVID-19 monitoring, like on cancer, seasonal flu and patient safety. More research is needed into the implementation and management costs of the long-term and extended use of dashboards. Past studies have warned that such costs could be prohibitive.^15,16 These were managed (or avoided) during the pandemic through the mobilisation of emergency resources. Ultimately, the continued and expanded use of dashboards will require more intentional resource planning and investment.

Strengths and limitations

Working in partnership with the WHO gave us unique access to the targeted teams from COVID-19 dashboards associated with the national government. The diverse composition of our research team enabled the use of an extensive range of languages during data collection, thus aiding both informant recruitment and the richness of exchanges during the actual interviews. The study captured COVID-19 dashboards at a critical point in their development: teams were actively improving and making adjustments to their dashboards at the time of the interviews. Hence, teams were still deeply immersed in their dashboard work and had little difficulty recalling the processes that took place over the previous year.

We acknowledge several potential limitations. First, the size and composition of core dashboard teams varied across countries or territories, causing some variability per dashboard in the profiles and numbers of informants and, ultimately, in the nature of their experiences. Second, although group interviews stimulated joint reflections across teams and thus enriched data collection, such an approach could have also induced group pressure resulting in socially desirable responses. Third, the findings are a snapshot of the initial year of the COVID-19 pandemic and may not reflect the later stages of the pandemic. Lastly, the study encompassed national, government-associated COVID-19 dashboards in the WHO European Region, and findings may not be generalisable to the experiences of subnational dashboards or of other types of developers, such as academia, independent initiatives, media outlets or industry. They may also not apply to other regions globally, and in particular to low-income countries.

Conclusion

The study revealed more similarities than differences among the 33 participating COVID-19 dashboard teams from across the WHO European Region. The learning-by-doing approach described by the teams reflects the novelty of the use of dashboards as tools for public reporting during a pandemic. The experiences of the dashboard teams show that initial underpreparedness was compensated for by high-level political endorsement, the teams’ own professionalism, accelerated data improvements and commercial software solutions. Recurrent barriers and enablers deriving from the pre-pandemic and pandemic contexts, from people and processes and from software, data and users should inform future investments, both in dashboard teams and at the national and pan-European levels. Many lessons have been learned in relation to the themes of simplicity, trust, partnership, software and data and change. These highlight areas where the action is needed to ensure a data-informed approach to health data stewardship using dashboards.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076221121154 - Supplemental material for The experiences of 33 national COVID-19 dashboard teams during the first year of the pandemic in the World Health Organization European Region: A qualitative study

Supplemental material, sj-docx-1-dhj-10.1177_20552076221121154 for The experiences of 33 national COVID-19 dashboard teams during the first year of the pandemic in the World Health Organization European Region: A qualitative study by Erica Barbazza, Damir Ivanković, Karapet Davtyan, Mircha Poldrugovac, Zhamin Yelgezekova, Claire Willmington, Bernardo Meza-Torres, Véronique L.L.C. Bos, Óscar Brito Fernandes, Alexandru Rotar, Sabina Nuti, Milena Vainieri, Fabrizio Carinci, Natasha Azzopardi-Muscat, Oliver Groene, David Novillo-Ortiz, Niek Klazinga and Dionne Kringos in Digital Health

Footnotes

Acknowledgements

The authors would like to thank Jeanine Suurmond for methodological advice, Michael Dallas for language editing and all the interviewees who generously shared their time to participate in the study: Assel Abakova; Anna Artsruni; Georgi Asatryan; Jonas Bačelis; Simon Bak; Senad Begić; Tania Boa; Lovro Bucić; Stefan Buttigieg; Alan Cahill; Fabrizio Carinci; Gianfranco Costabile; Phillip Couser; Igor Crnčić; Olgeta Dhono; Michaela Diercke; Mykola Dobysh; Marjolein Don; Anna Fumačová; Aram Ghulijanyan; Clare Griffiths; Veaceslav Gutu; Pouria Hadjibagheri; Luc Hagenaars; Scott Heald; Jonas Kähler; Kristjan Kolde; Martin Komenda; Kristina Kovačikova; Tanja Kustec; Terje Lasn; Sonia Leite; Renata Lenhardcziková; Pedro Licínio Pinto Leite; Jana Lepiksone; Mathias Leroy; Avet Manukyan; Elena Martinez; Graham McGowan; Emma McNair; Lorraine McNerney; Matej Mišik; Susana Monge; Teemu Möttönen; Maja Mrzel; Gints Muraševs; Hugo Agius Muscat; Martina Nagyová; Valentin Neevel; Aleksandar Obradović; André Peralta-Santos; Natalia Plugaru; Ane Radović; Anders Rasmussen; Raul Ritea; Leonardo Rocchi; Josep Romagosa; Giulio Siccardi; Fernando Simon; Margita Štāle; Alexandra Ștefănescu; Kristian Sufliarsky; Patrick Suter; Maríanna Þórðardóttir; Nataša Terzić; Vitaliy Trenkenshu; Alexander Ullrich; Zuzana Vallová; Jeroen van Leuken; Jana Vanagė; Giordano Veltro; Olivia Vereha; Solange Vogt; Liina Voutilainen; Martina Vrbiková; Marjana Vrh; Pauline White; Piotr Wlodarczyk; Kıvanç Yilmaz; and Tatiana Zvonnikova.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. DNO, KD and NAM are staff members of the WHO. The authors alone are responsible for the views expressed in this article, and these do not necessarily represent the decisions, policy or views of the WHO.

Contributorship

EB and DI contributed equally as co-first authors. EB, DI, KD, OG, DNO, NK and DK designed the study. All authors contributed to the tool development, sampling and/or recruitment of dashboards. EB, DI, OBF, MP, CW, BMT, VB, ZY, OG and NK conducted interviews and transcribed and translated interview data. EB, DI, NK and DK analysed interview data. EB and DI prepared the manuscript. All authors reviewed drafts of the manuscript and approved the final version.

Ethical approval

Ethical requirements of the primary research affiliation to the Amsterdam University Medical Centres, University of Amsterdam, the Netherlands, have been met, as stipulated in the Dutch ‘Medical Research Act with People’ (Wet medisch-wetenschappelijk onderzoek met mensen (WMO)) (Dutch), in BWBR0009408, W.a.S. Ministry of Health, Editor. 1998: Hague, Netherlands. Accordingly, as no human data were retained, voluntary informed consent was deemed appropriate.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was carried out by the Marie Skłodowska-Curie Innovative Training Network (HealthPros – Healthcare Performance Intelligence Professionals), which has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. 765141.

Guarantor

EB and DI.

ORCID iDs

Erica Barbazza

Mircha Poldrugovac

Véronique L.L.C. Bos

Alexandru Rotar

Supplemental material

Supplemental material for this article is available online.

References

World Health Organization. The World Health Report 2000: Health systems: improving performance. Geneva: World Health Organization, 2000.

Barbazza

Tello

. A review of health governance: definitions, dimensions and tools to govern. Health Policy 2014; 116: 1–11.

Travis

, et al. Towards better stewardship: concepts and critical issues. Geneva: World Health Organization, 2002.

Bouckaert

, et al. European Coronationalism? A Hot Spot Governing a Pandemic Crisis. Public administration review, 2020: p. 10.1111/puar.13242.

World Health Organization. Global surveillance during an influenza pandemic. Geneva: World Health Organization, 2009.

WHO Regional Office for Europe. Strengthening and adjusting public health measures throughout the COVID-19 transition phases. Copenhagen: WHO Regional Office for Europe, 2020.

Boin

McConnell

. Preparing for critical infrastructure breakdowns: the limits of crisis management and the need for resilience. Journal of Contingencies and Crisis Management 2007; 15: 50–59.

Kamel Boulos

Geraghty

. Geographical tracking and mapping of coronavirus disease COVID-19/severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) epidemic and associated events around the world: how 21st century GIS technologies are supporting the global fight against outbreaks and epidemics. Int J Health Geogr 2020; 19: 8.

Dong

Gardner

. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis 2020; 20: 533–534.

10.

Yigitbasioglu

Velcu

. A review of dashboards in performance management: implications for design and research. Int J Account Inf Syst 2012; 13: 41–59.

11.

Dowding

, et al. Dashboards for improving patient care: review of the literature. Int J Med Inform 2015; 84: 87–100.

12.

Kamel Boulos

Geraghty

13.

Zhang

, et al. Mapping the landscape of COVID-19 crisis visualizations, in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Yokohama, Japan: Association for Computing Machinery, 2021, p. Article 608.

14.

Budd

, et al. Digital technologies in the public-health response to COVID-19. Nat Med 2020; 26: 1183–1192.

15.

Ghazisaeidi

, et al. Development of performance dashboards in healthcare sector: key practical issues. Acta Informatica Medica: AIM: Journal of the Society for Medical Informatics of Bosnia & Herzegovina: Casopis Drustva za Medicinsku Informatiku BiH 2015; 23: 317–321.

16.

Weggelaar-Jansen

AMJWM

Broekharst

DSE

de Bruijne

. Developing a hospital-wide quality and safety dashboard: a qualitative research study. BMJ Qual Saf 2018; 27: 1000.

17.

Primary Health Care Performance Initiative. Country data. 2021; Available from: https://improvingphc.org/explore-country-data.

18.

World Health Organization. Global Health Expenditure Database. 2021 [cited 2021]; Available from: https://apps.who.int/nha/database/country_profile/Index/en.

19.

Eurostat. Eurostat regional yearbook 2021. 2021 [cited 2021]; Available from: https://ec.europa.eu/eurostat/statistical-atlas/gis/viewer/?config = config.json&mids = BKGCNT,BKGNT22016,C01M01,CNTOVL&o = 1,1,1,0.7&ch = PEO,C01&center = 50.00754,19.98211,3&.

20.

Institute for Health Metrics and Evaluation. Data visualizations. 2021 [cited 2021]; Available from: http://www.healthdata.org/results/data-visualizations.

21.

Gapminder. Resources. 2021 [cited 2021]; Available from: https://www.gapminder.org/resources/.

22.

Our World in Data. Health. 2021 [cited 2021]; Available from: https://ourworldindata.org/#entries.

23.

Pauwels

, et al.

Dashboards as a service: why, what, how, and what research is needed?

J Serv Res 2009; 12: 175–189.

24.

Florez

Singh

. Online dashboard and data analysis approach for assessing COVID-19 case and death data [version 1; peer review: 2 approved, 1 approved with reservations]. F1000Res 2020; 9: 1–13.

25.

Komenda

, et al. Complex reporting of the COVID-19 epidemic in the Czech Republic: use of an interactive web-based app in practice. J Med Internet Res 2020; 22: e19367.

26.

Ulahannan

, et al. A citizen science initiative for open data and visualization of COVID-19 outbreak in Kerala, India. J Am Med Inform Assoc 2020; 27: 1913–1920.

27.

Berry

, et al. Open access epidemiologic data and an interactive dashboard to monitor the COVID-19 outbreak in Canada. CMAJ 2020; 192: E420.

28.

Dixon

, et al. Leveraging data visualization and a statewide health information exchange to support COVID-19 surveillance and response: application of public health informatics. J Am Med Inform Assoc 2021; 28: 1363–1373.

29.

de Lusignan

, et al. COVID-19 Surveillance in a primary care sentinel network: in-pandemic development of an application ontology. JMIR Public Health and Surveillance 2020; 6: e21434–e21434.

30.

Ibrahim

, et al. Rapid development and utilization of a clinical intelligence dashboard for frontline clinicians to optimize critical resources during COVID-19. Acta Informatica Medica: AIM: Journal of the Society for Medical Informatics of Bosnia & Herzegovina: Casopis Drustva za Medicinsku Informatiku BiH 2020; 28: 209–213.

31.

HealthPros. International Training Network for Healthcare Performance Intelligence Professionals. 2020 [cited 2021]; Available from: https://www.healthpros-h2020.eu/.

32.

Ivanković

, et al. Features constituting actionable COVID-19 dashboards: descriptive assessment and expert appraisal of 158 public web-based COVID-19 dashboards. J Med Internet Res 2021; 23: e25682.

33.

Bos

, et al. Development and actionability of the Dutch COVID-19 dashboard: descriptive assessment and expert appraisal study. JMIR Public Health Surveill 2021; 7: e31161.

34.

Barbazza

, et al. Exploring changes to the actionability of COVID-19 dashboards over the course of 2020 in the Canadian context: descriptive assessment and expert appraisal study. J Med Internet Res 2021; 23: e30200.

35.

Tindemans

. COVID dashboard: ‘Good conversations about data have started’. In: Tindemans

Jongkind

(eds) DeCrisismanager. Arnhem, the Netherlands, 2021 [cited 2021]; Available from: https://decrisismanager.nl/covid-dashboard-er-zijn-goede-gesprekken-over-data-ontstaan.

36.

Jensen

. What we can learn from open data: COVID-19 in the Netherlands. 2021 [cited 2021]; Available from: https://www.tableau.com/about/blog/2021/4/what-we-can-learn-open-data-covid-19-netherlands.

37.

Perkel

Behind the Johns Hopkins University coronavirus dashboard. 2020 [cited 2021]; Available from: https://www.natureindex.com/news-blog/behind-the-johns-hopkins-university-coronavirus-dashboard.

38.

Merrick

. Behind the scenes of the coronavirus dashboard: a record 76.5m views, 2bn rows of data and 100 computers, in iNews. 2021: UK.

39.

Liu

, et al. The experiences of health-care providers during the COVID-19 crisis in China: a qualitative study. Lancet Glob Health 2020; 8: e790–e798.

40.

Bennett

, et al. COVID-19 confessions: a qualitative exploration of healthcare workers experiences of working with COVID-19. BMJ Open 2020; 10: e043949.

41.

Eftekhar Ardebili

, et al. Healthcare providers experience of working during the COVID-19 pandemic: a qualitative study. Am J Infect Control 2021; 49: 547–554.

42.

Zomerdijk

, et al. Experiences and needs of people with haematological cancers during the COVID-19 pandemic: a qualitative study. Psychooncology 2022; 31(3): 416–424.

43.

Drury

Eicher

Dowling

. Experiences of cancer care during COVID-19: phase 1 results of a longitudinal qualitative study. Int J Nurs Stud Adv 2021; 3: 100030.

44.

Williams

, et al. Public perceptions and experiences of social distancing and social isolation during the COVID-19 pandemic: a UK-based focus group study. BMJ Open 2020; 10: e039334.

45.

Mondino

, et al. Public perceptions of multiple risks during the COVID-19 pandemic in Italy and Sweden. Sci Data 2020; 7: 434.

46.

Branquinho

, et al. "Hey, we also have something to say": a qualitative study of Portuguese adolescents’ and young people's experiences under COVID-19. J Community Psychol 2020; 48: 2740–2752.

47.

Tong

Sainsbury

Craig

. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007; 19: 349–357.

48.

Frey

Fontana

. The group interview in social research. Soc Sci J 1991; 28: 175–187.

49.

Ivanković D, Barbazza E, Klazinga N, et al. Comment on verhulsdonck and Shah's lean data visualization: considering actionable metrics for technical communication. J Bus Tech Commun. 2022; 36(1): 105–113.

50.

WHO Regional Office for Europe. European health for all database. 2021 [cited 2021]; Available from: https://gateway.euro.who.int/en/datasets/european-health-for-all-database/.

51.

WHO Regional Office for Europe. COVID-19 situation in the WHO European Region 2021 [cited 2021 18 July]; Available from: https://who.maps.arcgis.com/apps/dashboards/ead3c6475654481ca51c248d52ab9c61.

52.

WHO Regional Office for Europe. European Health Information Initiative (EHII) 2021 [cited 2021 18 July ]; Available from: https://www.euro.who.int/en/data-and-evidence/european-health-information-initiative-ehii.

53.

PHIRI. The PHIRI project. 2021 [cited 2021]; Available from: https://www.phiri.eu/.

54.

Auerbach

Silverstein

. Analyzing your next study using elaborative coding, in Qualitative data: an introduction to coding and analysis. New York: New York University, 2003.

55.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

56.

Meyer

Avery

. Excel as a qualitative data analysis tool. Field Methods 2008; 21: 91–112.

57.

Center for Systems Science and Engineering (CSSE) Johns Hopkins University. COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University 2020 Available from: https://www.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6.

58.

Carinci

, et al. Towards actionable international comparisons of health system performance: expert revision of the OECD framework and quality indicators. Int J Qual Health Care 2015; 27: 137–146.

59.

Saigí-Rubió

, et al. Routine health information systems in the European context: a systematic review of systematic reviews. Int J Environ Res Public Health 2021; 18: 4622.

60.

Bogaert

, et al. Identifying common enablers and barriers in European health information systems. Health Policy 2021; 125(12): 1517–1526.

61.

WHO Regional Office for Europe. Implementation of the Tallinn Charter: final report. Copenhagen: WHO Regional Office for Europe, 2015.

62.

McAvey

. Investing in the data systems we need to create the health system we deserve, in Health Affairs. 2021.

63.

Barnum

, et al. Your data looks good on a dashboard. Online HIMSS 2019. Available from: https://www.himss.org/resources/your-data-looks-good-dashboard (accessed 2021).

64.

Dagliati

, et al. Health informatics and EHR to support clinical research in the COVID-19 pandemic: an overview. Brief Bioinform 2021; 22: 812–822.

65.

Rutter

Wolpert

Greenhalgh

. Managing uncertainty in the COVID-19 era. Br Med J 2020; 370: m3349.

66.

European Union. European Health Data Space. 2021 [cited 2021]; Available from: https://ec.europa.eu/health/ehealth/dataspace_en.

67.

Damman

, et al. An international comparison of web-based reporting about health care quality: content analysis. J Med Internet Res 2010; 12: e8.

68.

Kurtzman

Greene

. Effective presentation of health care performance information for consumer decision making: a systematic review. Patient Educ Couns 2016; 99: 36–43.

69.

Hibbard

Gilburt

. Supporting people to manage their health: an introduction to patient activation. London: The King’s Fund, 2014.

70.

Belle

, et al. Factoring in the human factor: experimental evidence on how public managers make sense of performance information. Int Public Manag J 2021: 1–14.

71.

Karami

Langarizadeh

Fatehi

. Evaluation of effective dashboards: key concepts and criteria. Open Med Inform J 2017; 11: 52–57.

72.

The Lancet Infectious, D. The COVID-19 infodemic. Lancet Infect Dis 2020; 20: 875.

73.

OECD. OECD COVID-19 Recovery Dashboard. 2022 [cited 2022 April 24]; Available from: https://www.oecd.org/coronavirus/en/recovery-dashboard.

74.

SeyedAlinaghi

, et al. Late complications of COVID-19; a systematic review of current evidence. Arch Acad Emerg Med 2021; 9: e14.

75.

American Academy of Physical Medicine and Rehabilitation. AAPM&R Long COVID Dashboard Released. 2021 [cited 2022 April 24]; Available from: https://www.aapmrorg/members-publications/member-news/member-news-details/2021/08/24/aapm-r-long-covid-dashboard-released.

76.

Pledge Times. A dashboard for the long-term complications of “Covid-19” in Abu Dhabi. 2022 [cited 2022 April 23]; Available from: https://pledgetimes.com/a-dashboard-for-the-long-term-complications-of-covid-19-in-abu-dhabi/.

77.

SeyedAlinaghi

, et al. The clinical outcomes of COVID-19 in HIV-positive patients: a systematic review of current evidence. Immun Inflamm Dis 2021; 9: 1160–1185.

78.

Capak

, et al. Prevalence of somatic comorbidities among coronavirus disease 2019 patients in Croatia in the first pandemic wave: data from national public health databases. Croat Med J 2020; 61: 518–524.

79.

Diabetes Center Berne. COVID-19 pandemic and Type 1 Diabetes: Weekly questionnaire to learn more about the experiences of the community during those challenging times. 2020; Available from: https://www.dcberne.com/en/2020/03/covid-19-pandemic-and-type-1-diabetes/.

80.

Cerovečki

Švajda

. COVID-19 pandemic influence on diabetes management in Croatia. Frontiers in Clinical Diabetes and Healthcare 2021; 2: 1–8.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB