Beyond convenience: practical considerations with using routine health data for evaluations

Routine health information systems (RHIS) comprise data collected at regular intervals at public, private, and community-level health facilities and institutions and health programs. This article looks at how this data may be used for evaluations, and the reasons behind why some are optimistic about this and some have concerns.

As digital routine health information systems (RHIS) become more prevalent, there is a growing interest in using them for evaluations. Rakha et al. ¹ used data from an integrated management of childhood illness (IMCI) database to evaluate under-5 mortality and the timing of IMCI implementation in 213 districts in Egypt. Bennett et al. ² used data from Zambia’s RHIS to evaluate an insecticide-treated net program and confirmed outpatient malaria incidence. Lalla-Edward et al. ³ used data collected from nine roadside wellness centers in South Africa to evaluate the services accessed by long-distance truck drivers and the drivers’ sociodemographic characteristics. Brennan et al. ⁴ used data from the National Health Service (NHS) in Leicester to assess hospital cost efficiency.

Some researchers are optimistic about this underutilized data source. Wagenaar et al. ⁵ discussed how certain evaluation designs, such as interrupted time series analysis, are better suited to routine data than to population-based surveys. Others are cautious about routine data’s limitations; Ashton et al. ⁶ expressed ‘concerns over internal validity, completeness and potential bias in estimates of program and intervention impact’, and Bennett et al. ² argued that RHIS data cannot be used for rigorous program evaluation without complex modeling strategies.

MEASURE Evaluation used RHIS data to conduct several process and outcome evaluations in low- and middle-income countries (LMIC). From these, we have insights not yet mentioned in the literature on several topics: RHIS data access, quality, and relevance.

National RHIS belong to national governments, to monitor and evaluate health service delivery, health system outcomes, and health outcomes and to improve lives. Ownership and privacy rights are assumed. Often, though, nongovernmental actors are credentialed to build the systems and capture and access data. Furthermore, systems themselves may be housed on servers paid for by nongovernmental partners. Governments have the right to control what is collected, used, and shared ⁷ and to whom they give access. Patients in some countries, like the United Kingdom, can choose whether their health records are used for evaluation. ⁸

Depending on the type of data (sensitive, such as HIV status or patient-level records, or important for national security, such as birth and death registrations) and the research purpose, the levels of approval can be numerous, complex, and time-consuming. Yet, these elements of a data system and data sharing should not be perceived as mere procedural hurdles. They reflect the ethical integrity of the system and processes.

A central concern of using routine data for research and evaluations is whether the data can be trusted and whether the data systems and the data stewards are trustworthy. Data quality ⁹ comprises accuracy, reliability, precision, completeness, timeliness, integrity, and confidentiality. The most common metrics – accuracy, completeness, and timeliness – contribute significantly to whether reliable conclusions can be drawn from the data. For mature and well-established routine data systems such as the NHS, patient privacy, data security, and public trust are a preoccupation. ⁸ The NHS with its large, consistent, and clean data has been called a ‘goldmine’ for entities interested in voluminous data for pharmaceutical research and machine learning. ¹⁰ In addition, concerns have been raised about the use of public assets for commercial and financial gain. ¹¹

Disease-response definitions, such as confirmed malaria versus suspected malaria, are often conflated in reporting processes. Furthermore, updating and distributing data collection tools such as facility registers can be years behind changes in national policies and indicator definitions. Local contextual changes (e.g. in administrative units) can also affect the data’s scope. Finally, updating the data in electronic systems can introduce errors; changes in program outcome estimates affect the ability to determine trends over time.

An evaluation using RHIS data alone should ensure that those data accurately represent indicators of interest. This requires understanding each indicator’s definition, how it is disaggregated (e.g. by age and sex), its original data source(s) (e.g. registers), the levels where it is aggregated (e.g. facility and district levels), how it is reported and on which forms (e.g. monthly reporting forms), its frequency of collection and reporting, and how it is represented in the RHIS. Countries with robust RHIS have indicator reference documents, but details are usually incomplete and additional steps are required to map indicator numerators and denominators from their original sources of collection to the RHIS. RHIS data elements are often abbreviated and do not intuitively reflect the names on indicator reference sheets and/or source documents. These discrepancies are rarely documented. Furthermore, many indicators in the RHIS require denominators derived from population-level data sets. Subnational unit or subpopulation estimates are often mathematically modeled from national-level data and come with their own data quality issues. Researchers may not have access to the models used to determine denominators and cannot easily assess how appropriate the models are for the variables used.

Wagenaar, Ashton, and Bennett discuss the methods possible with data generated from RHIS and inherent limitations. However, not until the data are accessed and extracted can the final specification of models take place. Although RHIS data provide repeated measures over time, several factors affect their utility for this purpose: changes in indicator definition, availability of population-based denominators, splitting of administrative units, and levels of disaggregation. Multi-country analyses have similar limitations. For example, indicator elements may be defined or calculated dissimilarly. Finally, when new data sets are created, it introduces provenance management or data origination issues which could become problematic if the software used to manage them are not ‘provenance-aware’. ¹²

Using data in hand is inevitably cheaper than collecting new data, but cost is not the only consideration. We have described issues of data access, quality, and relevance for methods and questions. The use of routine data to inform population health decisions should continue to be explored and refined. But the current limitations of routine data inherently restrict the questions these data can answer.