Sage Journals: Discover world-class research

Abstract

This study identifies essential components in the data collection process for public health information systems based on appraisal and synthesis of the reported factors affecting this process in the literature. Extant process assessment instruments and studies of public health data collection from electronic databases and the relevant institutional websites were reviewed and analyzed following a five-stage framework. Four dimensions covering 12 factors and 149 indicators were identified. The first dimension, data collection management, includes data collection system and quality assurance. The second dimension, data collector, is described by staffing pattern, skill or competence, communication and attitude toward data collection. The third, information system, is assessed by function and technology support, integration of different data collection systems, and device. The fourth dimension, data collection environment, comprises training, leadership, and funding. With empirical testing and contextual analysis, these essential components can be further used to develop a framework for measuring the quality of the data collection process for public health information systems.

Keywords

data collection data quality measurement instrument process assessment public health information systems

Introduction

Public health information systems (PHIS), the government-recognized population-based data repositories, are essential for public health management and improvement.¹ PHIS provide nations with health-related data mainly required for monitoring, prevention, and control of diseases and other adverse health conditions.² Data in PHIS must be of sufficient quality to meet public health needs and worthy of data users’ trust.^3–7 The World Health Organization (WHO) has introduced a generic Data Quality Assessment Framework (DQAF) developed by the International Monetary Fund.⁴ The WHO has reinforced that data quality assessment should not only describe the quality status of data but also enable identification of the causes of data quality problems.^4,8 The process of data collection is an essential element of data quality. It includes the generation, assembly, description, and maintenance of data, all of which should be of high quality.^9,10 While data quality problems originating from the process of data collection have been frequently found, research into this topic area is yet to further develop.^7,11

To date, the assessment of the quality of the data collection process in PHIS has not been well considered nor routinely conducted.⁹ The quality improvement effort has been focused on the assessment of the quality of data which have already been captured and stored.^5,9,12 Data quality assessment is mainly focused on the identification and evaluation of the attributes of data quality, including accuracy, completeness, and timeliness of data.^9,13,14 The reason for the lack of attention to the quality of the data collection process could be an insufficient clarification of the essential components for data collection. For example, our review of PHIS data quality assessment showed that only 2 (5%) of 39 studies specified an explicit definition of the quality of data collection.⁹ A variety of quality criteria for data collection were introduced such as data accuracy, data integrity, minimum response burden for data provider practices, and the relevance, simplicity, and layout of the data collection tools.^5,7,15,16

Data collection is a systematic data gathering process,¹⁵ which includes a set of interrelated or interacting activities contributing to the process of transforming inputs into outputs.¹⁷ Organizational, technical, and behavioral factors can affect the performance of the data collection process for the PHIS.^{7,11,13,18–20} They may “take the form of defects in organizational procedures, faulty logic, and reasoning, or human error that result in compromised performance.”²¹ In total, 16 of the 39 implementation studies reviewed in our earlier study, instead of taking a comprehensive picture of the entire process, were centered around some data collection procedures, such as data recording and storage, and on quality control mechanisms.⁹ The unsystematic knowledge about the key factors influencing the quality of the data collection process has impaired the effectiveness and efficiency of data-driven monitoring and performance evaluation mechanisms for public health programs.^7,11,21,22

An interesting question is: what are the exact components to ensure the quality of the data collection process? Researchers have conducted some exploration in this area. At the organizational level, structure, resources, procedures, support services, and culture in an organization can all influence the process quality of data collection.^13,18–20 However, the operational definition or measurement for these factors has yet to be reached. At the technical level, the design of electronic data collection forms and integration of different information systems are the important mechanisms. But technology advancement alone cannot always lead to high-quality data.^7,11,13,21 At the individual behavioral level, a data collector’s motivation and competence to perform a task, though often scrutinized from the lens of data users, have not been clarified in the context of the data collection process.¹³ The lack of a comprehensive understanding of the contribution of these factors leads to challenges in assessing the quality of that process.

Such challenges hence pose the fundamental research question of this study which aims to identify the essential components of quality in the data collection process for PHIS.

Methods

Literature search

We searched peer-reviewed full-text English literature in medical and informatics electronic databases, including Scopus, CINAHL, Medline, PubMed, and Web of Science. The publication dates were from 1 January 2001 (since the principles and practices of PHIS were defined in the discipline of public health informatics)²³ to 31 December 2016. Search terms included words or phrases relating to data collection, process assessment, measurement, data quality, public health, and health information systems. The symbol “*” was used to include the variations of a word. A total of 172 publications were retrieved.

To improve the literature coverage, we further conducted a manual search of the literature and identified papers referenced by the selected publications. Prominent public health institutional websites were also searched, such as those of the WHO and the Australian Institute of Health and Welfare (AIHW). The authors’ research databases on data quality assessment were also searched. Another 52 publications from these sources were included. Two of the authors independently assessed the study quality. Any discrepancies were resolved through discussion and an informal consensus process. Further details are shown in Supplemental Appendix Table 1.

Selection of publications

Articles were assessed based on inclusion and exclusion criteria. The inclusion criteria were articles contributing significantly in the domain of quality of the data collection process in the PHIS and research topics related to the quality of the data collection process. The article types included empirical studies, reviews, guidelines, and work reports.

The exclusion criteria were publications that did not mention factors or components of the quality of the data collection process; those focusing on data use or only measuring the data stored in the PHIS; and those lacking clear definition or without evidence-based information. Editorials, notes, and letters were also excluded.

The above screening activities led to a selection of 107 articles eligible for further study quality evaluation.

We used the Critical Appraisal Skills Programme (CASP) tools to assess the reliability and validity of each selected study.^24,25 The CASP tools provide a set of checklists for evaluation of study quality including the context, subjects, study design, research methods, data collection, data analysis, and conclusions.

We also considered (1) whether the concepts of data quality or the quality of the data collection process matched our understanding of these; (2) whether the cause of poor data quality arising from the data collection process was analyzed; and (3) how the factors contributing to the quality of data collection or data quality were measured. The studies that did not provide adequate information and rigorous research methods were excluded.

Eventually, 45 publications were selected for review. The publication selection and evaluation process is illustrated in Figure 1.

Figure 1.

Publication screening process.

Data extraction and analysis

A five-stage framework was followed for qualitative data extraction, processing, and analysis:²⁶

Stage 1—familiarization with data. Each article was thoroughly read to identify quality issues, concepts, and themes related to the data collection process. Relevant data from the selected studies were extracted and entered into an Excel spreadsheet to facilitate critical evaluation of the results. A total of 453 pieces of relevant text were pre-selected and recorded.

Stage 2—identification of a thematic framework. A process of shortening the extracted text while still preserving the core was conducted for condensing the relevant data. A constant comparison and aggregation process led to the abstraction of 149 first-level codes as indicators relating to the quality of the data collection process. Further comparison, aggregation, abstraction, and classification of the indicators generated 16 factors that were related to the quality of the data collection process. These factors were further abstracted using an approach based on general systems theory²⁷ and advice from public health experts. A four-dimensional (4D) thematic framework was developed, including data collection management, data collector, information system, and the data collection environment.

Stage 3—indexing and validation of the thematic framework. The process of constant comparison, aggregation, and classification was iterated repetitively. Data were rearranged per the appropriate dimension of the thematic framework to which they were likely to belong. Attempts were made to avoid duplication and overlap in semantics and refinement of paraphrasing within the framework. This process led to the reduction of factors from 16 to 12.

Stage 4—charting. The 12 factors were arranged into the appropriate dimension of the thematic framework to which they related. A chart was prepared.

Stage 5—mapping and interpretation. This stage is a process to map the nature and range of the concepts and factors. The associations between the factors were identified to create the typology of the framework. Each indicator was interpreted as either a facilitator or a barrier according to its direction of influence, positive or negative, on the quality of the data collection process. Eventually, the theoretical saturation was reached, and all extracted data were placed into the categories already created (see Table 1).

Table 1.

Data for the 45 included studies.

Reference	Region	Characteristics of study	Themes included (A: Data collection management; B: Data collector; C: Information system; D: Data collection environment)	Association with data quality
Abernethy et al.²⁸	USA	1. Surveyed tuberculosis contact investigation forms from all 50 states, 3 municipalities, and 2 countries. 2. Apply statistics and cluster analysis	A, C	1. Standardization of data elements. 2. Quantitative data analysis
Glèlè Ahanhanzo et al.²⁹	Benin	1. Cross-sectional descriptive and analytical study of factors associated with data quality in the routine health information systems. 2. Data including (a) document review of data, (b) self-administered questionnaire, and (c) focus groups with 116 health workers in 4 municipalities. 3. Apply statistics and qualitative analysis	B	1. Data comprehensiveness, reliability, and accuracy. 2. Significant difference by statistical quantitative data analysis
Aidara et al.³⁰	Eight European countries	A clinical survey was completed for 146 sentinel dentists in 8 European countries and included 2877 clinical examinations and 2877 individual assessments	A, B, D	1. Missing data and time for data collection. 2. Descriptive correlational statistical analysis
Amouzou et al.³¹	Malawi	1. Comparison of children mortality monitoring data reported by 160 community health workers and those computed in a standard household mortality survey. 2. Data analysis using rates, standard errors, and 95% confidence interval	A, D	1. Data accuracy and completeness. 2. Descriptive quantitative statistics
Anderka et al.³²	USA	1. A descriptive study to develop and test a data quality assessment tool for population-based birth defect surveillance programs. 2. Assessment by a self-administered (with no evaluation component) Standards Data Quality Assessment Tool about National Birth Defects Prevention Network (NBDPN)	A, B, D	1. Data completeness, timeliness, and accuracy. 2. No statistical association analysis was conducted
Andersson et al.³³	Sweden	1. Cross-sectional case study in process-oriented healthcare organizations. 2. Data collected (a) from archives, (b) through interviews, (c) by participatory observation, and (d) by conducting a focus group session. 3. Use of descriptive qualitative data	A, B, C, D	Not provided
Bosch-Capblanch et al.³⁴	Forty-one low-income countries	1. Cross-sectional study using the Data Quality Audit (DQA), a WHO-validated, standard methodology to compare data collected from health unit (HU) records of immunizations administered with reports of immunizations at the central level and to collect quality indicators of the reporting system. 2. Statistical analysis of quality scores and data verification results	A, C	1. Data accuracy (verification factor, VF). 2. No significant correlation between data accuracy and national quality scores while significant correlations with districts and health unit’s quality scores
Braa et al.³⁵	United Republic of Tanzania	1. An action research conducted at 10 districts, delivering interventions by quarterly data use workshops focusing on data collection, integration, data quality, teamwork, practical computer skills, and presentation skills. 2. Use of descriptive qualitative data	A, B, C, D	1. General data quality. 2. Improvement of general data quality was supported by descriptive qualitative data
Bradley et al.³⁶	Peru	1. A descriptive study to assess data quality of a perinatal syphilis information system. 2. Data collected by (a) record review in 43 of 156 public hospitals and (b) 8 in-depth interviews with stakeholders. 3. Descriptive statistic data analysis	A, C, D	1. Data quality and completeness. 2. Qualitative reasoning while no statistical association analysis was conducted
Chen et al.⁹	International	1. Comparative study. 2. Use of descriptive qualitative data	A	Not provided
Chisha et al.³⁷	Zambia	Intervention of enhanced surveillance and data feedback loop	A, D	1. Data completeness, accuracy, and consistency. 2. Descriptive correlational statistical analysis
Cibulskis and Hiawalyer³⁸	Papua New Guinea	1. Program report of national health management information systems relating to standardization of the system design, test, implementation, integration of different systems, and using information of a national health information system. 2. Use of descriptive qualitative data	A, B, C, D	Not provided
Corriols et al.³⁹	Nicaragua	1. A nationally representative cross-sectional survey of 3169 persons aged 15 years and older. 2. Review of official evaluations of the pesticide poisoning surveillance system to understand what caused the underreporting. 3. Use of quantitative and qualitative data	B, D	1. Underreporting of data. 2. No statistical association analysis was conducted
Digiacomo et al.⁴⁰	Australia	1. Cross-sectional descriptive study of the implementation of the national guidelines in cardiac rehabilitation. 2. Data collected by semi-structured interviews with 37 health professional’s perspectives of systems-related barriers to implementation of the national guidelines. 3. Use of descriptive qualitative data	A, B, C, D	Not provided
Galvin et al.⁴¹	Ireland	1. Extract data using a software. 2. Evaluate extracted data from 30 practices in line with 12 European quality indicators. 3. Comparative analysis between seasons and regions	C	1. Data accuracy and variation. 2. Descriptive statistics without correlational analysis
Hajdu et al.⁴²	Norway	1. Use USCDC Guidelines for the evaluation of point-prevalence of healthcare-associated infections in hospitals and long-term care facilities for elderly surveillance systems. 2. Data collected by review of surveillance protocol and database, and survey with data mangers and practitioners	A, C, D	1. Data completeness. 2. Descriptive statistics
Haskew et al.⁴³	Kenya	1. Use a two-sample test of proportions pre- and post-implementation of EMR-based data verification. 2. Significant improvements in completeness of the antenatal record were recorded	C	1. Data completeness. 2. Significant difference by statistical quantitative data analysis
Haswell et al.⁴⁴	Australia	1. Intervention of using an indigenous register within a psychiatric hospital information system. 2. Cross-validation of health information system data with the indigenous register over 2 years about 355 indigenous admissions. 3. Use of quantitative data	A, B, C, D	Data accuracy (increased accuracy and reliability of data by descriptive quantitative statistics)
Heidebrecht et al.⁴⁵	Canada	1. A feasibility study of scannable forms in two settings. 2. Assessment of efficiency, data quality, and usability through time observations, record audits, staff interviews, and client surveys. 3. Use of quantitative and qualitative data	C	1. Efficiency of data entry. 2. Agreement (consistency) of data. 3. Quantitative statistics and qualitative data
Heidebrecht et al.¹⁴	Canada	1. Cross-sectional descriptive study of electronic immunization data collection systems in two Ontario public health units. 2. Use of adapted Centers for Disease Control and Prevention surveillance system evaluation guidelines (USCDC Guidelines). 3. Data collected by key informant interviews, logic and completeness assessments, client surveys, and on-site observations. 4. Use of quantitative and qualitative data	A, B, C, D	1. User’s perception of simplicity and flexibility of data input and report access. 2. Data completeness and validity, timeliness. 3. No statistical association analysis was conducted
Ing et al.⁴⁶	Canada	1. Qualitative study of barriers experienced and the techniques used to overcome barriers during investigation of enteric disease cases among expert investigators. 2. Four focus groups, 28 experts involved via teleconference; data from cases were thematically analyzed	C, D	Not provided
Kaposhi et al.⁴⁷	South Africa	1. Cross-sectional descriptive study. 2. Data audits and semi-structured interviews by adapting an international routine data quality assessment tool in 32 facilities in 3 sub-districts. 3. Use of quantitative and qualitative data	A, B, C, D	1. Data accuracy described with quantitative data audit. 2. No statistical association analysis was conducted
Kawonga et al.⁴⁸	South Africa	1. Cross-sectional descriptive study. 2. Data collected by key informant interviews with managers and review of records, observation. 3. Use of quantitative and qualitative data	A	Not provided
Kiilu et al.⁴⁹	Kenya	1. Cross-sectional study of assessing health workers’ skills and current training needs for information management by purposive sampling of 121 health workers. 2. Data collected by self-administered open-ended and closed-ended, structured questionnaires. 3. Quantitative data analysis	B	1. Data accuracy. 2. Status of data quality was not provided
Krenzelok et al.⁵⁰	USA	1. Deliver an intervention of a two-tiered intervention program (passive and interactive training) to enhance the quality of data collection using acetaminophen exposures as a model. 2. Data collected by a quality measurement scorecard	C	Data accuracy of some data items with significant difference by statistical quantitative data analysis
Ledikwe et al.⁵¹	Botswana	1. A qualitative assessment of the health data management and reporting systems that capture and transfer routine monitoring and evaluation (M&E) AIDS data using Routine Data Quality Audit Tool (MEASURE Evaluation). 2. Data collected by 71 in-depth interviews and over 300 reference materials. 3. ATLAS (Version 6.1, Scientific Software Development) was used to assist with systematic analysis of the qualitative data	A, B, C, D	Not provided
Lin et al.⁵²	China	1. Use the USCDC Guidelines to assess the performance of the national HIV sentinel surveillance system (HSS). 2. Data collected using a detailed checklist to review surveillance guidelines, protocols, and relevant documents; conducting self-administered, anonymous surveys with 286 local China CDC staff; and carrying out field observations in 32 sentinel sites in four provinces	A, C, D	1. Data completeness and accuracy. 2. No statistical association analysis was conducted
Lowe et al.⁵³	USA	1. A retrospective review of the introduction of tan electronic health record system. 2. Use descriptive quantitative data	C	1. Efficiency of data collection. 2. Descriptive statistical analysis
Madsen et al.⁵⁴	Europe	Report on the manual of operations	A	Not provided
Melnik et al.⁵⁵	USA	Cross-sectional descriptive study to identify current practices and potential barriers to accurate and complete data collection by birth registrars (BRs) in New York State facilities and data collected by a piloted and tested questionnaire survey on 127 birth registrars from New York City and the Rest of State	A, B, C, D	Data accuracy and completeness with statistical analysis
Mphatswe et al.⁵⁶	South Africa	1. Intervention of prevention of mother-to-child transmission (PMTCT) of the human immunodeficiency virus using a series of strategies including training, feedback, monthly data reviews, and data audits at facilities over 10 months in 8 antenatal clinics and 20 delivery wards (37 urban, 21 rural, and 20 semi-urban) in a province. 2. Data completeness and accuracy were compared before and after intervention. 3. Use of quantitative data	A, D	Data accuracy and completeness with statistical analysis
Murai et al.⁵⁷	Philippine	1. Investigate the mechanisms of errors in the case identification process in the existing routine health information system (RHIS) by cross-sectional descriptive case study and data collected by a structured questionnaire by measuring the level and disparity of 132 health workers’ (who were in charge of the case identification step) understanding of the definitions of 12 health program indicators in 14 selected municipalities in the province of Palawan. 2. Use of quantitative data	A, B, C, D	Data accuracy with statistical analysis of quantitative data
Mykkänen et al.⁵⁸	Finland	1. Description of current status and issues and a set of recommendations related to the standardization of healthcare. 2. Data collected by web-based survey to named experts in different organizations	A, C, D	Not provided
Nicol et al.⁵⁹	South Africa	1. Cross-sectional descriptive study using the Performance of Routine Information System Management (PRISM) tool. 2. Data collected by a self-administered questionnaire with 161 health information personnel in 58 health facilities and 2 district offices from 2 conveniently sampled health districts. 3. Use of quantitative data	B	Not provided
Paterson et al.⁶⁰	Pacific Island Countries and Territories (PICTs)	1. An in-country quantitative and qualitative evaluation in five PICTs using the USCDC guidelines. 2. Data collected by semi-structured key informant interviews, observational techniques including field inspections of raw data and data collection methods, and analysis of syndromic data reported	A, B, C, D	Not provided
Puttkammer et al.⁶¹	Haiti	A mixed-methods evaluation of a multi-site electronic medical record system using a qualitative Delphi process to identify data quality priorities among local stakeholders, and a quantitative data quality assessment (DQA) examined 13 indicators of completeness, accuracy, and timeliness of retrospective data collected from 2005 to 2013	A, C, D	1. Data completeness. 2. No statistical association analysis was conducted
Reilly et al.⁶²	USA	Program report of quality of care and case documentation by a two-part review process: a qualitative case review by independent reviewers and an administrative review of required documentation in the physical case record	A, B, C, D	1. Data timeliness and completeness. 2. Qualitative review and quantitative statistical analysis
Rimando et al.⁶³	USA	Qualitative study of the data collection challenges in dissertation research by sharing experience of eight doctoral students	A, D	1. General data quality. 2. Status of data quality was not provided
Schuurman et al.⁶⁴	South Africa	A pilot study by analysis of the data set to develop injury data collection protocols appropriate to low-resource environments, building on previous developments in the field such as those described previously	A, C, D	Not provided
Soti et al.⁶⁵	Kenya	1. A pilot study by implementation of a system, the web portal to cloud-based information. 2. Evaluate data accuracy, completeness, and timeliness using descriptive statistics	C	1. Accurate, complete, and timely data. 2. Descriptive statistics without correlational analysis
Sychareun et al.⁶⁶	Lao PDR	Retrospective review of data by compiling the data into a standardized excel sheet as well as making copies of health center data for verification	A, B, C	1. Data consistency, accuracy, security, availability, and representation. 2. Simple descriptive statistical analysis
Tamrat and Kachnowski⁶⁷	International	Literature review	C	Not provided
West et al.⁶⁸	Scotland	1. Program report of feedback interviews with 33 primary healthcare staff following service implementation of a computerized information system to collect workload data. 2. Use of qualitative data	B, D	Not provided
Wilkins et al.⁶⁹	International	1. To develop a model for a data-driven health system assessing areas that impede decision makers’ access to information. The Data for Decision Making (DDM) surveillance assessment methodology was applied to six systems in five countries. 2. Data collected by interviews and review of the data collection methods	A, B, C, D	1. Data timeliness and accuracy. 2. Simplicity, flexibility, acceptability, and usefulness of the system. 3. Qualitative analysis
Yourkavitch et al.⁷⁰	Malawi	1. Review integrated community case management data quality and the monitoring and evaluation (M&E) system using adapted Measure Evaluation “data quality audit tool.” 2. Randomly selected sample facilities. 3. Use of quantitative and qualitative data	A, B, C, D	1. Data accuracy, timeliness, completeness, confidentiality, reliability, precision, integrity, and availability. 2. Quantitative and qualitative analysis

WHO: World Health Organization; USCDC: US Centers for Disease Control; EMR: electronic medical record.

Results

The results from the qualitative data processing and analysis provided material for the identification of the essential components of quality in the data collection process for PHIS, including four dimensions that covered 12 factors (see Figure 2). The first dimension, data collection management, includes data collection system and quality assurance. The second dimension, data collector, is described by staffing pattern, skill/competence, communication, and attitude toward data collection. The third, information system, is assessed by function and technology support, integration of different data collection systems, and devices. The fourth, data collection environment, comprises training, leadership, and funding. The 12 factors are characterized by 149 indicators with either positive or negative impacts on the quality of the data collection process (see Supplemental Appendix Table 2).

Figure 2.

The components of quality in the data collection process for public health information systems.

Accuracy, completeness, and timeliness are the most frequently mentioned parameters of quality for evaluating the performance of the data collection process. These three parameters appeared in 24, 16, and 7 studies, respectively. And 14 studies did not define data quality specifically. Reliability, data use, quality of service, and system quality were also addressed.

Data collection management

From an organizational perspective, data collection management is an administrative process by which data are acquired, validated, stored, protected, and processed.^7,11 Effective management requires the application of knowledge, skills, tools, and techniques to data collection activities to meet data quality requirements. The ultimate goal of data collection management is to fulfill every requirement from data users.^11,16 That is the provision of sufficient supervision to personal and systematic process audits to ensure data quality. Of the 45 included articles, 32 assessed the quality of data collection management^{9,14,28,29,31–38,40,42,44,47,48,51,52,54–58,60–64,66,69,70} (see Table 1). Details of facilitators and barriers for data collection management are shown in Supplemental Appendix Table 2.

For the preservation of data integrity, data collection management needs to detect errors that have occurred in the data collection process.¹⁶ Errors may be produced intentionally (i.e. deliberate falsification) or unintentionally (i.e. systematic or random errors).¹⁶ In public health, data collection management primarily focuses on the procedures of data collection, storage, quality control, and data presentation for users.^5,6 They are often presented in a format of guidelines or a set of policies to direct the execution of programs and guide the practice of parties involved in the process.^5,56 We identified two major factors for data collection management in the context of the PHIS. They are data collection system and quality assurance.

A data collection system primarily comprises two sub-components: data collection form and data collection practice. Data collection form is the core component of data collection instruments. A poorly designed data collection form may impair data accuracy;⁶⁹ therefore, data collection form needs to be standardized, well defined, and structured. As one of the major concerns, standardization in data collection form can be facilitated by a series of tactics.⁵⁹ The format of the data collection form needs to be simple, standardized, and complete.^28,32,55 The layout and order of data items of a form need to accord with the workflow of data capture or reporting for easy data entry and retrieval.^33,34

Data collection practice should be well guided, conducted, and documented. They include guideline development, documentation, data backup and security, selection of data collection methods, and a trial of a new process before implementation.^38,41,44,54 A complete record of the data collection process in line with the workflow of data collection is recommended.⁴⁷

Quality assurance for data collection should be in place before the collection begins and it should be focused on quality control.^16,58 The function of quality assurance is to ensure that each process of data collection is traceable, accurate, and timely, and has integrity. Four factors could be utilized to assess the adequacy of a system for quality assurance. These are quality audit, fundamental responsibility, mechanisms for addressing data quality challenges, and a feedback loop.^14,35,37,47 Designated unit or individuals to monitor data quality and prevent data collection mismanagement are recommended.^{44,47,49,53,55} A veteran health worker register could remind data collectors to correct inaccurate data items,^44,62 and provide additional training, supervision, and incentives.³¹ Holding regular meetings with medical or clinical staff and a data registrar is useful to address missing or inconsistent data.^32,57

Data collector

A data collector collects or supplies data for the PHIS. Of the 45 articles, 23 assessed the performance of data collectors^{14,29–31,33,35,38–40,44,46,47,49,51,55,57,59,60,62,66,68–70} (see Table 1). Details of facilitators and barriers for data collector are shown in Supplemental Appendix Table 2.

A data collector is a stakeholder with whom the data user should build up and nurture a relationship. Data collectors’ performance was mainly related to data accuracy.^{44,47,55,57,66,69,70} The association between data quality and the data collector’s certain characteristics such as level of responsibility, level of work engagement, and sector of employment was statistically quantified.²⁹ Four types of factors including the staffing pattern, their data collection skills or competence, communication with clients, and their attitude toward data collection could, directly and indirectly, influence data collectors’ performance. For example, data collector shortage and high turnover could impair data quality.⁶⁶ A data collector needs to have sufficient capability to conduct data collection activities, for example, understanding contextual information and having basic knowledge of the data elements to be collected.^55,57 Proficient data collection skills and good communication with clients are ideal for a data collector.^14,44 Mistakes often originated from data collectors’ attempts to simplify data collection tasks.^11,30,39,66 Provision of training is regarded by higher authorities and upper management as a useful approach to improving data collectors’ capabilities including fundamental medical knowledge and routine data management skills.^35,57,70,71

Information system

An information system is a combination of hardware, software, infrastructure, and trained personnel.⁵ Of the 45 articles, 30 assessed the quality of information systems^{14,28,33–36,38,40–45,47,48,51–53,55,57,58,60–62,64–67,69,70} (see Table 1). A regular system of data quality checks may be more cost effective and reliable to ensure data quality.⁵¹

Characteristics of information systems in PHIS are demonstrated by automatic functions and technology support provided to the users of the system and the integration of different data collection systems and devices. The facilitators and barriers for information systems are described in Supplemental Appendix Table 2.

The functions of information systems in PHIS are automatic data processing, usually via an electronic interface of data collection forms and prompts for data collectors about data collection activities. The systems may automatically check the logic of data, assess the comprehensiveness of required data items, and issue alerts for errors made during data entry.^35,55 These functions serve as an online task reminder to help with task completion and prevent slippage.¹⁴ Use of the “smart chart” technology can prevent a data collector from submitting a record with missing fields. In this manner, the function of an automatic logic check and smart selection of data are integrated into the mandatory fields. It is found that data errors are rare since the introduction of “smart charts.”⁵² If an automatic workflow chart is available in the system, it could guide and standardize the data collection and reporting process. However, changes in the project procedures and system configuration over time may lead to a decline in data quality if deployed against established guidelines or specifications on data collection for PHIS.^11,53

Integration of different systems is important in the PHIS. Multiple systems and files may impair the quality of the data collection process if data are from the various sources.⁵⁵ Therefore, centralizing data in one unique source and use of linked data systems is preferred.^{34,48,51,58,64} For example, the use of external data linkage and collaboration with other jurisdictions can facilitate the generation of a higher level data repository or data sharing platform.^14,38

Devices are the hardware used to store or transmit data such as computers, printers, and other electronic equipment. These devices need to be adapted to the operational system, suitable for use in data collection, and free from computer crashes, viruses, and insecure methods for data backup and storage.^{40,51,58,65,67}

Data collection environment

Data collection environment refers to the context for data collection. In a government context, the PHIS is directly responsible to legislative, regulatory, and policy directives.¹ Of the 45 articles, 30 assessed the quality of the data collection environment^{14,29,31–33,35–40,42,44,46,47,50–52,55–58,60–64,68–70} (see Table 1). Training, leadership, and funding support are the three main factors. Details of facilitators and barriers for data collection environment are shown in Supplemental Appendix Table 2.

Training is imparting information and providing instructions to help trainees attain a required level of knowledge and skill or improve their performance. Such training should be mandated by higher authorities and upper management instead of on a voluntary basis.^57,71 Continuing education and training opportunities should be provided to all data collectors, including frontline health professionals, managers, and specialists. The training should be individualized, measurable, and may focus on communication skills for data collection, and criteria and procedure of health service provision.^{47,56,57,62,63}

Attributes of good leadership include (1) strengthened coordination, cooperation, and communication among government agencies and between healthcare facilities and health professionals; (2) recognition of the importance of data to be collected; (3) provision of sufficient funding; and (4) allocation of full-time staff or specific staff to data collection.^{35,44,51–53,69} Examples of good leadership include the development of a less resource-intensive approach using strategies of decentralization to empower the management team in the field and establishing a multi-level supervision network that includes health departments and healthcare facilities.^51,52 Supervisors can perform real-time field quality assurance and control activities.

Implementation of electronic systems, installation of local system infrastructure, and maintenance of a network across data collection facilities are sometimes costly; therefore, funding is critical for data collection in resource-constrained settings.^38,51 Availability of funding can improve data quality.

Discussion

The quality of the data collection process is a key component of overall data quality in the PHIS.^9,10 Conceptualization of the quality of the data collection process for PHIS is also requisite for reaching public health high data quality goals. A recent evaluation of data quality in the country health information systems by WHO in a global context has found that data management was the weakest component of the system performance.⁷ A lack of knowledge about the key factors influencing the quality of the data collection process for PHIS has hindered data quality improvement and thus has impeded the effectiveness of data-driven monitoring and performance evaluation for public health programs. Effective process assessment of data collection that focuses on how data are collected will help standardize the performance of public health programs by comparing “the specific actions taken, events occurring, and human interactions with accepted standards.”⁷² Prior studies have explored some factors that may affect the quality of the data collection process. But consensus on a comprehensive and systematic assessment of the process has not been reached. Identification of the essential components of the quality of data collection is needed to guide efforts in the development of a quality framework for PHIS.

The most commonly reported quality dimension for the data collection process is data collection management; half of the identified facilitators and barriers belong to this dimension. Key areas demanding an effort for improvement include the design of the data collection form, data collection practice, and data collection quality assurance. Standardization of public health data collection practice is a long-standing issue, together with the integration of different data sources and data collection systems in public health. These key study findings reflected a primary concern with the definitions and characteristics of data collection. A variety of definitions and different quality criteria of the data collection process may contribute to a wide range of factors that affect the quality of the process.^5–7,15

The data collection process is recognized as a systematic process that consisted of the interrelated and interdependent parts. However, the different parts of the process and the interaction between those parts for PHIS have not been well articulated. For example, the quality of the data collection form can contribute to data completeness and standardization and thus the use and validity of data. But the association between the quality of data collection form and data quality has not been quantified. Over-emphasis on the procedures, methods, and quality control parts of data collection and simply automating data collection systems cannot solve all data quality problems.²¹

Data collectors play an important role in the quality of the data collection process. Extant data quality assessment instruments have not paid sufficient attention to data collectors except for their training experience.¹² Gaps existed between actual and recommended practice even though guidelines were available to data collectors.¹¹ These gaps may arise from inefficient communication between data users and data collectors. Seamless translation of data users’ requirements for data quality into the quality of the data collection process is an effective strategy for collecting high-quality data. We suggest more contextual analysis with an emphasis on data quality criteria to meet data users’ needs.

Our study identified the data collection environment as one of the four essential components of the quality of the data collection process. Training, leadership, and funding are the building blocks of a friendly and supportive data collection environment, in addition to the other factors. These factors include whether the relationship with the data collectors “is of the utmost importance” in a data collection setting⁵ and if the data collection process fits with an organization’s service model.¹¹ Barriers to health clients’ participation in health services such as poor communication, cultural safety, and a lack of transport to health facilities could also affect the volume of data available for collection. Adding the data collection environment to the essential components would better inform data quality assessment in troubleshooting the factors that affect the quality of the data collection process.

Limitations of existing measurement instruments and studies were also found. Information about data quality was not provided in a third of the studies (14 articles).^{9,33,38,40,46,48,51,54,58–60,64,67,68} The majority of studies used simple descriptive or qualitative data to analyze the relationship between the factors affecting the data collection process and data quality. As the identified components were distilled from qualitative analysis of the published literature, future empirical testing and practical implementation are needed.

Conclusion

Acceptable data quality in the PHIS cannot be achieved without a high-quality data collection process. The identification of the essential components that contribute to the quality of the data collection process is thus vital to ensure that data collection leads to high-quality data. After an extended literature review, this study has identified 4D components of the quality of the data collection process for PHIS. They are data collection management, data collector, information systems, and data collection environment. With empirical testing and contextual analysis, the above identified essential components can be used in future research and practice to develop a quality framework for measuring and improving the quality of the data collection process.

Supplemental Material

Health_informatics_journal-Appendix_Table_2_framework – Supplemental material for Identification of the essential components of quality in the data collection process for public health information systems

Supplemental material, Health_informatics_journal-Appendix_Table_2_framework for Identification of the essential components of quality in the data collection process for public health information systems by Hong Chen, Ping Yu, David Hailey and Tingru Cui in Health Informatics Journal

Footnotes

Author contributions

H.C. searched the literature, performed data extraction, conducted data analysis, and wrote the manuscript. Being a principal investigator, P.Y. led the study, conceptualized, and wrote the manuscript. D.H. advised the conceptualization and revised the manuscript. T.C. reviewed the extracted data and revised the manuscript. All authors read, reviewed, and approved the final manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This research has been conducted with the support of an Australian Government Research Training Program Scholarship.

Supplemental material

Supplemental material for this article is available online.

ORCID iD

Ping Yu

References

O’Carroll

Yasnoff

Ward

, et al. Public health informatics and information systems. New York: Springer Science & Business Media Inc., 2003, pp. 3–38.

World Health Organization. Country health information systems: a review of the current situation and trends. Geneva: World Health Organization, 2011, pp. 3–10.

World Health Organization. Monitoring the building blocks of health system, a handbook of indicators and their measurement strategies. Geneva: World Health Organization, 2010.

World Health Organization. Data quality review: a toolkit for facility data quality assessment. Module 1: framework and metrics. Geneva: World Health Organization, 2017, pp. 5–12.

Canadian Institute for Health Information (CIHI). The CIHI data quality framework. Ottawa, ON: CIHI, 2009, pp. 1–8.

Australian Institute of Health and Welfare. Data governance framework, https://www.aihw.gov.au/getmedia/a10b8148-ef65-4c37-945a-bb3effaa96e3/AIHW-data-governance-framework.pdf.aspx (2014, accessed 24 October 2017).

United States Agency for International Development (USAID)/MEASURE Evaluation. Guidelines for data management standards in routine health information systems, 2015, http://pdf.usaid.gov/pdf_docs/PA00KB8N.pdf

World Health Organization. Monitoring results with health facility information systems: a technical consultation. Geneva: World Health Organization, 2014.

Chen

Hailey

Wang

, et al. A review of data quality assessment methods for public health information systems. Int J Environ Res Public Health 2014; 11(5): 5170–5207.

10.

Karr

Sanil

Banks

. Data quality: a statistical perspective. Stat Methodol 2006; 3: 137–173.

11.

Australian Institute of Health and Welfare (AIHW). The nKPI data collection: data quality issues working paper 2012–2014. Canberra, ACT: AIHW, 2015.

12.

World Health Organization. Data quality review: a toolkit for facility data quality assessment. Module 3: data verification and system assessment. Geneva: World Health Organization, 2017.

13.

Aqil

Lippeveld

Hozumi

. PRISM framework: a paradigm shift for designing, strengthening and evaluating routine health information systems. Health Policy Plan 2009; 24(3): 217–228.

14.

Heidebrecht

Kwong

Finkelstein

, et al. Electronic immunization data collection systems: application of an evaluation framework. BMC Med Inform Decis Mak 2014; 14: 5.

15.

US National Library of Medicine National Institutes of Health. Data collection, https://www.ncbi.nlm.nih.gov/mesh/?term=data+collection (accessed 28 October 2017).

16.

Office of Research Integrity. Responsible conduct in data management, https://ori.hhs.gov/education/products/n_illinois_u/datamanagement/dctopic.html (2005, accessed 18 April 2017).

17.

International Organization for Standardization (ISO). ISO 9001: 2015 quality management systems—requirements. Geneva: ISO, 2015.

18.

Batini

Cappiello

Francalanci

, et al. Methodologies for data quality assessment and improvement. ACM Comput Surv 2009; 41: 1–52.

19.

Kamper-Jorgensen

. Public health information systems—and EUPHIX. Scand J Public Health 2008; 36(7): 673–675.

20.

Heisey-Grove

Wall

Helwig

, et al. Using electronic clinical quality measure reporting for public health surveillance. MMWR Morb Mortal Wkly Rep 2015; 64(16): 439–442.

21.

Carney

Weber

. Public health intelligence: learning from the Ebola crisis. Am J Public Health 2015; 105(9): 1740–1744.

22.

Davis

Morgans

Burgess

. Information management in the Australian aged care setting: an integrative review. HIM J 2017; 46: 3–14.

23.

Yasnoff

O’Carroll

Koo

, et al. Public health informatics: improving and transforming public health in the information age. J Public Health Manag Pract 2000; 6(6): 67–75.

24.

Aveyard

. Doing a literature review in health and social care: a practical guide. 2nd ed. Maidenhead: McGraw-Hill/Open University Press, 2010.

25.

Critical Appraisal Skills Programme. CASP checklist, http://www.casp-uk.net/casp-tools-checklists (2017, accessed 12 February 2017).

26.

Pope

Ziebland

Mays

. Analysing qualitative data. BMJ 2000; 320: 114–116.

27.

von Bertalanffy

. An outline of general system theory. Emerg Complex Organ 2008; 10: 103–123.

28.

Abernethy

DeRimer

Small

. Methods to identify standard data elements in clinical and public health forms. AMIA Annu Symp Proc 2011; 2011: 19–27.

29.

Glèlè Ahanhanzo

Ouedraogo

Kpozehouen

, et al. Factors associated with data quality in the routine health information system of Benin. Arch Public Health/Archives belges de santé publique 2014; 72(1): 25.

30.

Aidara

Pitts

Markowska

, et al. Quality of data gathered with International Caries Detection and Assessment System (ICDAS) assessment and dentists’ perceptions of completion of dental records quality of data gathered with ICDAS assessment. Int Dent J 2011; 61: 314–320.

31.

Amouzou

Banda

Kachaka

, et al. Monitoring child mortality through community health worker reporting of births and deaths in Malawi: validation against a household mortality survey. PLoS ONE 2014; 9(2): e88939.

32.

Anderka

Mai

Romitti

, et al. Development and implementation of the first national data quality standards for population-based birth defects surveillance programs in the United States. BMC Public Health 2015; 15: 925.

33.

Andersson

Hallberg

Timpka

. A model for interpreting work and information management in process-oriented healthcare organisations. Int J Med Inform 2003; 72(1–3): 47–56.

34.

Bosch-Capblanch

Ronveaux

Doyle

, et al. Accuracy and quality of immunization information systems in forty-one low income countries. Trop Med Int Health 2009; 14(1): 2–10.

35.

Braa

Heywood

Sahay

. Improving quality and use of data through data-use workshops: Zanzibar, United Republic of Tanzania. Bull World Health Organ 2012; 90(5): 379–384.

36.

Bradley

Tapia

Kamb

, et al. Can the Perinatal Information System in Peru be used to measure the proportion of adverse birth outcomes attributable to maternal syphilis infection. Rev Panam Salud Publica 2014; 36(2): 73–79.

37.

Chisha

Larsen

Burns

, et al. Enhanced surveillance and data feedback loop associated with improved malaria data in Lusaka, Zambia. Malar J 2015; 14: 222.

38.

Cibulskis

Hiawalyer

. Information systems for health sector monitoring in Papua New Guinea. Bull World Health Organ 2002; 80(9): 752.

39.

Corriols

Marin

Berroteran

, et al. The Nicaraguan pesticide poisoning register: constant underreporting. Int J Health Serv 2008; 38(4): 773–787.

40.

Digiacomo

Davidson

Taylor

, et al. Health information system linkage and coordination are critical for increasing access to secondary prevention in Aboriginal health: a qualitative study. Qual Prim Care 2010; 18(1): 17–26.

41.

Galvin

Callan

Cormican

, et al. Improving antimicrobial prescribing in Irish primary care through electronic data collection and surveillance: a feasibility study. BMC Fam Pract 2015; 16: 77.

42.

Hajdu

Eriksen

Sorknes

, et al. Evaluation of the national surveillance system for point-prevalence of healthcare-associated infections in hospitals and in long-term care facilities for elderly in Norway, 2002–2008. BMC Public Health 2011; 11: 923.

43.

Haskew

Saito

, et al. Implementation of a cloud-based electronic medical record for maternal and child health in rural Kenya. Int J Med Inform 2015; 84(5): 349–354.

44.

Haswell

Wheeler

Wargent

, et al. Validation and enhancement of Australian Aboriginal and Torres Strait Islander psychiatric hospitalisation statistics through an Indigenous Mental Health Worker register. Rural Remote Health 2013; 13(1): 2002.

45.

Heidebrecht

Quach

Pereira

, et al. Incorporating scannable forms into immunization data collection processes: a mixed-methods study. PLoS ONE 2012; 7(12): e49627.

46.

Ing

Lee

Middleton

, et al. A focus group study of enteric disease case investigation: successful techniques utilized and barriers experienced from the perspective of expert disease investigators. BMC Public Health 2014; 14: 1302.

47.

Kaposhi

Mqoqi

Schopflocher

. Evaluation of antiretroviral treatment programme monitoring in Eastern Cape, South Africa. Health Policy Plan 2015; 30(5): 547–554.

48.

Kawonga

Blaauw

Fonn

. Aligning vertical interventions to health systems: a case study of the HIV monitoring and evaluation system in South Africa. Health Res Policy Syst 2012; 10: 2.

49.

Kiilu

Okero

Muiruri

, et al. Human resource capacity for information management in selected public healthcare facilities in Meru County, Kenya. Pan Afr Med J 2015; 20: 334.

50.

Krenzelok

Reynolds

Dart

, et al. A model to improve the accuracy of US Poison Center data collection. Clin Toxicol (Phila) 2014; 52(8): 889–896.

51.

Ledikwe

Grignon

Lebelonyane

, et al. Improving the quality of health information: a qualitative assessment of data management and reporting systems in Botswana. Health Res Policy Syst 2014; 12: 7.

52.

Lin

Chen

Seguy

, et al. Is the HIV sentinel surveillance system adequate in China? Findings from an evaluation of the national HIV sentinel surveillance system. Western Pac Surveill Response J 2012; 3(4): 76–85.

53.

Lowe

Griffin

Hart

. Analysis of STAT laboratory turnaround times before and after conversion of the hospital information system. Respir Care 2014; 59(8): 1275–1280.

54.

Madsen

Gudnason

Pajak

, et al. Population-based register of acute myocardial infarction: manual of operations. Eur J Cardiovasc Prev Rehabil 2007; 14(Suppl. 3): S3–S22.

55.

Melnik

Guldal

Schoen

, et al. Barriers in accurate and complete birth registration in New York State. Matern Child Health J 2015; 19(9): 1943–1948.

56.

Mphatswe

Mate

Bennett

, et al. Improving public health information: a data quality intervention in Kwazulu-Natal, South Africa. Bull World Health Organ 2012; 90(3): 176–182.

57.

Murai

Lagrada

Gaite

, et al. Systemic factors of errors in the case identification process of the national routine health information system: a case study of Modified Field Health Services Information System in the Philippines. BMC Health Serv Res 2011; 11: 271.

58.

Mykkänen

Korhonen

Porrasmaa

, et al. A national study of eHealth standardization in Finland—goals and recommendations. Stud Health Technol Inform 2007; 129(Pt 1): 469–473.

59.

Nicol

Bradshaw

Phillips

, et al. Human factors affecting the quality of routinely collected data in South Africa. Stud Health Technol Inform 2013; 192: 788–792.

60.

Paterson

Kool

Durrheim

, et al. Sustaining surveillance: evaluating syndromic surveillance in the Pacific. Glob Public Health 2012; 7(7): 682–694.

61.

Puttkammer

Baseman

Devine

, et al. An assessment of data quality in a multi-site electronic medical record system in Haiti. Int J Med Inform 2016; 86: 104–116.

62.

Reilly

McKelvey-Walsh

Freundlich

, et al. Training and technology: improving the quality and timeliness of service plans and case documentation. Admin Soc Work 2011; 35: 207–222.

63.

Rimando

Brace

Namageyo-Funa

, et al. Data collection challenges and recommendations for early career researchers. Qual Rep 2015; 20: 2025–2036.

64.

Schuurman

Cinnamon

Matzopoulos

, et al. Collecting injury surveillance data in low- and middle-income countries: the Cape Town Trauma Registry pilot. Glob Public Health 2011; 6(8): 874–889.

65.

Soti

Kinoti

Omar

, et al. Feasibility of an innovative electronic mobile system to assist health workers to collect accurate, complete and timely data in a malaria control programme in a remote setting in Kenya. Malar J 2015; 14: 430–438.

66.

Sychareun

Hansana

Phengsavanh

, et al. Data verification at health centers and district health offices in Xiengkhouang and Houaphanh Provinces, Lao PDR. BMC Health Serv Res 2014; 14: 255.

67.

Tamrat

Kachnowski

. Special delivery: an analysis of mHealth in maternal and newborn health programs and their outcomes around the world. Matern Child Health J 2012; 16(5): 1092–1101.

68.

West

Farmer

Whyte

. Implementing computerised workload data collection in rural primary health care. Aust J Rural Health 2004; 12(1): 11–16.

69.

Wilkins

Nsubuga

Mendlein

, et al. The data for decision making project: assessment of surveillance systems in developing countries to improve access to public health information. Public Health 2008; 122(9): 914–922.

70.

Yourkavitch

Zalisk

Prosnitz

, et al. How do we know? An assessment of integrated community case management data quality in four districts of Malawi. Health Policy Plan 2016; 31(9): 1162–1171.

71.

Juran

. Juran on leadership for quality: an executive handbook. New York: The Free Press, 1989.

72.

National Center for Biotechnology Information US National Library of Medicine. Process assessment, 1992, https://www.ncbi.nlm.nih.gov/mesh/68017064

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB