Abstract
Objective.
We compared the quality of data reported to New York City’s immunization information system, the Citywide Immunization Registry (CIR), through its real-time Health Level 7 (HL7) Web service from electronic health records (EHRs), with data submitted through other methods.
Methods.
We stratified immunizations administered and reported to the CIR in 2014 for patients aged 0–18 years by reporting method: (1) sending HL7 messages from EHRs through the Web service, (2) manual data entry, and (3) upload of a non-standard flat file from EHRs. We assessed completeness of reporting by measuring the percentage of immunizations reported with lot number, manufacturer, and Vaccines for Children (VFC) program eligibility. We assessed timeliness of reporting by determining the number of days from date of administration to date entered into the CIR.
Results.
HL7 reporting accounted for the largest percentage (46.3%) of the 3.8 million immunizations reported in 2014. Of immunizations reported using HL7, 97.9% included the lot number and 92.6% included the manufacturer, compared with 50.4% and 48.0% for manual entry, and 65.9% and 48.8% for non-standard flat file, respectively. VFC eligibility was 96.9% complete when reported by manual data entry, 95.3% complete for HL7 reporting, and 87.2% complete for non-standard flat file reporting. Of the three reporting methods, HL7 was the most timely: 77.6% of immunizations were reported by HL7 in <1 day, compared with 53.6% of immunizations reported through manual data entry and 18.1% of immunizations reported through non-standard flat file.
Conclusion.
HL7 reporting from EHRs resulted in more complete and timely data in the CIR compared with other reporting methods. Providing resources to facilitate HL7 reporting from EHRs to immunization information systems to increase data quality should be a priority for public health.
Immunization information systems (IISs) are funded by the Centers for Disease Control and Prevention (CDC), states, and local jurisdictions and operated by health departments in 49 of 50 U.S. states, five cities, and the District of Columbia. 1 The primary purposes of an IIS are to collect and consolidate reports of vaccinations administered by all providers in a jurisdiction and to allow easy sharing of these records with other providers, health plans, schools, and parents. Ultimately, better sharing of records should reduce extra immunizations and improve immunization rates. In recent years, health departments have begun relying on IISs to perform immunization program core functions, including management of and accountability for publicly purchased vaccines distributed through the Vaccines for Children (VFC) program. 2,3 The effectiveness of IISs for these purposes depends on capturing complete and timely data, which is a challenge for many IISs. 1,3 –7
Electronic Health Records (EHRs) Incentive Programs were established by the Health Improvement Technology for Economic and Clinical Health (HITECH) Act of 2009. This Act created financial incentives under Medicaid and Medicare programs for health-care providers to adopt certified EHRs and demonstrate their meaningful use to improve health-care delivery and patient care. 8 –11 Starting in 2011, Stage 1 of the EHR Incentive Programs included a public health menu of objectives, of which one option was to report immunizations from an EHR to an IIS using Health Level 7 (HL7) messaging standards. 9 –11 These standards define the packaging of information sent from one health information system to another by prescribing the structure and types of data needed to enable seamless integration between the systems. 12 When a health-care provider documents an immunization in a certified EHR, the information is sent directly to the IIS. Meeting the meaningful use immunization objective in Stage 1 required only the sending of a test message. Starting in 2014, Stage 2 of the EHR Incentive Programs required successful ongoing submission of immunization reports to the IIS in the provider’s jurisdiction, thus moving immunization reporting to a meaningful use core objective. 13 By giving providers an incentive to report immunizations to their local IIS using HL7 messages sent from their EHR, the meaningful use objective offered an opportunity to improve the quality and completeness of IIS data. CDC has urged IIS managers to support HL7 reporting as a minimal functional standard since 2001, and most have done so.1
In 1994, the New York City Health Code was amended to require all providers to report childhood immunizations. 14 Shortly thereafter, the city’s IIS, the Citywide Immunization Registry (CIR), was created and implemented in 1997 by the New York City Department of Health and Mental Hygiene (DOHMH). Methods of reporting that were available to providers from the beginning were flat file upload from billing systems (the source changed to EHRs in later years), manual data entry, and paper forms (which ended in 2008). In 2009, the New York City DOHMH developed CIR’s real-time HL7 Web service. It was not until 2011, when the meaningful use immunization objective began, that providers started reporting from their EHRs through the CIR’s HL7 Web service. At that time, DOHMH established stricter data quality requirements for HL7 reporting than those for non-standard flat files.
New York City’s CIR has achieved high standards for data completeness, as evidenced by being selected as one of six IIS sentinel sites used by CDC to monitor national trends in immunization practices and vaccination coverage. 15 As of December 31, 2014, 90% of childhood immunization facilities were reporting to the CIR regularly (i.e., reporting >1 immunization during the previous six months), and an estimated 90% of immunizations administered to children aged 0–18 years in New York City were being captured in the CIR. 16 By the second half of 2014, a total of 636 of 1,990 (32%) childhood immunizing facilities reporting to the CIR had met the meaningful use immunization objective by using EHRs to connect and regularly report to the CIR’s HL7 Web service. In addition, 226 pharmacy sites reported childhood immunizations to the CIR via the HL7 Web service during that time. Combined, these facilities represented 862 of 1,990 (43%) childhood immunizing facilities reporting to the CIR in the second half of 2014, and accounted for nearly half of all immunizations reported in the city. 16
The purpose of this study was to compare the quality of data reported to the CIR through the HL7 Web service from EHRs with data submitted using the other two reporting methods: manual entry using the CIR’s Online Registry Web application and upload of a non-standard flat file. We examined the completeness of reporting of vaccine lot number, vaccine manufacturer, and patient VFC eligibility, which are necessary for immunization programs to meet CDC’s goal of VFC dose-level accountability. 17 We also examined the timeliness of reporting. Immunization providers need up-to-date vaccination histories to determine which vaccines patients need at visits and to avoid administering extra immunizations. Immunization programs require timely data to monitor vaccine uptake, especially for outbreak control and pandemic influenza preparedness and response.
Methods
We examined all immunizations administered and reported to the CIR in 2014 for patients aged 0–18 years. We excluded all immunizations administered in the past by a provider other than the reporting provider (i.e., historical immunizations) because they did not include vaccine lot number, vaccine manufacturer, or patient VFC eligibility. Typically, such information is only known to the immunizing provider. Also, historical immunizations might have been administered weeks or months before the time of report and, therefore, could not be considered in the timeliness analysis.
We stratified all immunizations reported to the CIR by the three reporting methods: (1) sending HL7 messages from EHRs through the HL7 Web service, (2) manual data entry, and (3) non-standard flat file upload from EHRs. We examined the total number and percentage of reported immunizations for each method. We assessed completeness by measuring the percentage of immunizations reported with vaccine lot number, vaccine manufacturer, and patient VFC eligibility for each reporting method. We assessed timeliness by measuring the number of days from the date the immunization was administered to the date it was entered into the CIR. For each reporting method, we calculated the percentage of immunizations reported in <1 day and <30 days of administration. To determine the significance of differences in timeliness and completeness found between data reported through the HL7 Web service and the other two reporting methods, we used Pearson’s χ2 test to conduct pair-wise comparisons (i.e., HL7 Web service compared with Online Registry, and HL7 Web service compared with non-standard flat file for each measure). We performed all analyses using SAS® version 9.2. 18
Results
During January 1–December 31, 2014, a total of 3,796,973 immunizations were reported to the CIR for patients aged 0–18 years. The largest percentage of immunizations was reported using the HL7 Web service (46.3%), followed by non-standard flat file (28.6%) and manual data entry (25.2%). Of all immunizations reported, 76.8% included vaccine lot number, 68.9% included vaccine manufacturer, and 93.4% included patient VFC eligibility. More than half (54.6%) of all immunizations were reported in <1 day and 94.8% were reported in ≤30 days of administration (Table).
Completeness and timeliness of immunizations administered and reported to the New York City Citywide Immunization Registry for patients aged 0–18 years, by method of reporting, 2014
aSending HL7 messages from electronic health records through the HL7 Web service
bManual data entry using the Citywide Immunization Registry’s Online Registry Web application
cUpload of a non-standard flat file from electronic health records
dSignificantly different from HL7 Web service at p<0.001
HL7 = Health Level 7
VFC = Vaccines for Children program
The HL7 Web service was the method through which the most complete data were reported, except for VFC eligibility, which was slightly more complete for manual data entry at 96.9% compared with 95.3% for the HL7 Web service. Of immunizations reported through the HL7 Web service, 97.9% included vaccine lot number and 92.6% included vaccine manufacturer; these data fields were more complete when reported through the HL7 Web service than when reported through the other two reporting methods. The HL7 Web service was the timeliest method, with 77.6% of immunizations reported in <1 day of administration. In contrast, 53.6% of immunizations reported through manual data entry and 18.1% of immunizations reported through the non-standard flat file were reported in <1 day. Results of the pair-wise comparisons between the HL7 Web service and the other two reporting methods were significant for all three data completeness and two timeliness measures (p<0.001). The HL7 Web service was the superior reporting method for all measures except VFC eligibility (Table).
Discussion
The results of this study show that data reported through the HL7 Web service from EHRs were overall more complete and timely than data reported through manual entry using the Online Registry and non-standard flat file uploads from EHRs. These findings are important in that the HL7 Web service was the method through which the largest percentage of immunizations (46.3%) was reported in 2014. Also, the use of HL7 messaging to report immunizations in New York City is trending upward and will almost surely increase in the years ahead, propelled by implementation of Stage 2 and Stage 3 of the EHR Incentive Programs. During July–December 2015, 57.8% (1,872,923/3,238,745) of immunizations administered to children aged 0–18 years were reported through the HL7 Web service by 57.9% (1,205/2,080) of all facilities reporting to the CIR during that period. 12
For the VFC eligibility measure, both the HL7 Web service and manual data entry were highly complete (95.3% and 96.9%, respectively). VFC eligibility was slightly more complete when reported via manual data entry, likely because this field must be populated for the immunization report to be accepted through this reporting method. The HL7 Web service outperformed manual data entry on all other data quality measures and was superior to non-standard flat file reporting on all measures.
One reason the HL7 Web service reporting performed better than the other two methods is likely that reporting using an interface between an EHR and the CIR’s HL7 Web service is automatic and does not require extra effort from the provider, particularly in contrast to the time required to report individual patients’ immunizations manually. In addition, the source of reporting through the HL7 Web service is a meaningful use-certified EHR that the provider uses to clinically manage patients and document details related to vaccine administration, including vaccine manufacturer and lot number, as required by federal law. 19 EHRs are also the source of non-standard flat file reporting for most facilities using that method, but the data quality requirements for flat files were established before the meaningful use immunization objective and, therefore, are not as strict. Most importantly, the meaningful use immunization objective gave the CIR the opportunity to raise data quality standards and require providers and EHR vendors to comply with these higher reporting standards to meet the immunization objective and qualify for incentives.
Our findings of higher-quality data reported using HL7 are consistent with the results of a similar study conducted by the Wisconsin IIS. 20 A study of the Washington State IIS concluded that increasing HL7 reporting would be unlikely to improve data completeness, but would substantially improve timeliness. 7 Hence, all three studies showed the benefits of HL7 reporting from EHRs to IISs for improving data timeliness, and two studies demonstrated its value for increasing data completeness. In New York City, creating electronic connections between EHRs and IISs for HL7 reporting is time consuming and labor intensive, but the effort is well worth it. The value of IISs for facilitating interventions to increase immunization rates and strengthen management of the VFC program is well documented. 3,21
Limitations
This study’s main limitation was that completeness of immunizations reported was assessed for only three measures. Further research is needed to assess the completeness of other data elements, such as vaccine expiration date and parent (or patient) e-mail addresses and mobile telephone numbers. Vaccine expiration dates are used by IISs to facilitate better vaccine management and to help providers prevent administration of expired vaccines to patients. Capturing e-mail addresses and mobile telephone numbers enables IISs to send reminder or recall messages when immunizations are due. Reminder/recall has been shown to be effective in increasing immunization rates. 22
Conclusion
HL7 reporting from EHRs resulted in more complete and timely data in the CIR. Providing resources to facilitate HL7 reporting from EHRs to IISs to increase IIS data quality should be a priority for public health.
Footnotes
Acknowledgements
The Institutional Review Board of the New York City Department of Health and Mental Hygiene (NYC DOHMH) determined that this study was exempt from the federal regulations under 45 C.F.R.