Abstract
Background:
Identifying pregnancy episodes and accurately estimating their beginning and end dates are imperative for observational maternal vaccine safety studies using electronic health record (EHR) data.
Methods:
We modified the Vaccine Safety Datalink (VSD) Pregnancy Episode Algorithm (PEA) to include both the International Classification of Disease, ninth revision (ICD-9 system) and ICD-10 diagnosis codes, incorporated additional gestational age data, and validated this enhanced algorithm with manual medical record review. We also developed the new Dynamic Pregnancy Algorithm (DPA) to identify pregnancy episodes in real time.
Results:
Around 75% of the pregnancy episodes identified by the enhanced VSD PEA were live births, 12% were spontaneous abortions (SABs), 10% were induced abortions (IABs), and 0.4% were stillbirths (SBs). Gestational age was identified for 99% of live births, 89% of SBs, 69% of SABs, and 42% of IABs. Agreement between the PEA-assigned and abstractor-identified pregnancy outcome and outcome date was 100% for live births, but was lower for pregnancy losses. When gestational age was available in the medical record, the agreement was higher for live births (97%), but lower for pregnancy losses (75%). The DPA demonstrated strong concordance with the PEA and identified pregnancy episodes ⩾6 months prior to the outcome date for 89% of live births.
Conclusion:
The enhanced VSD PEA is a useful tool for identifying pregnancy episodes in EHR databases. The DPA improves the timeliness of pregnancy identification and can be used for near real-time maternal vaccine safety studies.
Plain Language Summary
Introduction
Post-licensure monitoring of the safety of vaccination during pregnancy is important because pregnant women have historically been excluded from vaccine clinical trials. Inactivated influenza vaccination and tetanus-diphtheria-acellular pertussis vaccination are currently recommended during pregnancy, but other vaccinations can be administered during or around the time of pregnancy, sometimes inadvertently before a pregnancy has been confirmed or in certain high-risk individuals.1–3 In the post-licensure setting, the safety of vaccination during pregnancy is evaluated either through passive surveillance systems (e.g. the Vaccine Adverse Event Reporting System), 4 pregnancy registries, 5 or by conducting observational studies within large linked databases derived from electronic health records (EHRs), like the Vaccine Safety Datalink (VSD), 6 the Food and Drug Administration Sentinel System’s Post-Licensure Rapid Immunization Safety Monitoring Program, 7 or the UK Clinical Practice Research Database. 8 Identifying pregnancy episodes and precisely estimating the beginning and end dates of these episodes can be challenging in these data systems. In addition, it is often difficult to link mothers’ and infants’ records to evaluate birth outcomes such as congenital anomalies.
The VSD is a long-standing collaboration of multiple integrated care delivery systems and the Centers for Disease Control and Prevention (CDC).6,9 The data-contributing VSD sites currently include: Denver Health (Colorado); HealthPartners (Minnesota); the Colorado, Northern California, Northwest, Southern California, and Washington regions of Kaiser Permanente; Marshfield Clinic (Wisconsin). Harvard Pilgrim participates as a non-data-contributing VSD site, providing subject matter expertise. Denver Health was added as a VSD site in 2017, and thus did not contribute data to some of the analyses described below. On a weekly basis, VSD sites collect healthcare data, including diagnoses, procedures, and vaccinations, to support vaccine safety surveillance and studies; patient demographics and health plan enrollment data are updated monthly or quarterly. These files are supplemented with linked birth and death certificate records annually.
Since 2012, VSD investigators have used a validated pregnancy episode algorithm (PEA) to identify retrospectively pregnant women for vaccine safety studies. 10 The PEA generates a pregnancy data file annually using combined healthcare and birth record data from each participating site; each annual file contains data on pregnancy episodes dating from 2002. In a previous validation study of pregnancies ending in 2002 through 2006 identified by the original PEA, 75% of these pregnancies ended in live births, 12% in spontaneous abortions (SABs), 9% in induced (therapeutic or elective) abortions (IABs), and 4% other pregnancy outcomes. 10 We manually reviewed a sample of these episodes and found high agreement in outcome type, outcome dates, and gestational ages, especially for live births. We found lower agreement between the algorithm and medical record review for IABs (e.g. 70% agreement in outcome date).
The VSD PEA was designed to identify pregnancy-related International Classification of Disease, ninth Revision (ICD-9) diagnosis and procedure codes, and therefore the algorithm code became obsolete when participating sites switched to ICD-10 coding on 1 October 2015. In addition, the original VSD PEA relied on gestational age data from linked birth records for live births, which resulted in a data lag of at least 9 months depending on birth record availability. For non-live birth outcomes or live births without a birth record, the original VSD PEA imputed the start of the pregnancy episode by applying a median gestational age for a given outcome (e.g. live births estimated at 40 weeks’ gestation, SABs estimated at 10 weeks’ gestation).
The objective of this study was to update and enhance our ability to conduct rigorous safety studies of maternal vaccination by modifying the original VSD PEA to include both ICD-9 and ICD-10 codes and incorporate additional sources of gestational age data, and to validate the enhanced VSD PEA with manual medical record review. We also developed a new algorithm, the Dynamic Pregnancy Algorithm (DPA), to identify ongoing pregnancy episodes to enhance our ability to conduct more timely safety assessments.
Methods
PEA
The original VSD PEA has been described in detail previously and the ‘look-up tables’ containing lists of ICD-9 pregnancy-related diagnosis and procedure codes are available from the authors upon request. 10 The algorithm is used to create an annual data file containing pregnancy episodes dating from 2002. In brief, the algorithm searches VSD diagnosis and procedure code files to identify possible indicators of pregnancy (Figure 1). Some of these indicators are codes related to a pregnancy outcome (e.g. Z37.0 delivery of single live birth), while others represent pregnancy-related diagnoses and care (e.g. Z34.90 encounter for supervision of normal pregnancy). These data are then hierarchically processed to identify a pregnancy end date and assign an outcome to the pregnancy episode. Pregnancy outcome types include live birth, SAB, IAB (e.g. therapeutic or elective abortion), stillbirth (SB), ectopic pregnancy, trophoblastic disease, or nondefinitive outcome (e.g. abortion of unknown type or early loss of unknown type). Pregnancies ending with a procedure like a dilation and curettage are classified as IABs by the PEA. Multiple gestation pregnancies are identified by the PEA, and if, for example, the pregnancy results in a live birth and an SB, the algorithm assigns live birth and still birth as the outcome. A second data processing step identifies gestational age at pregnancy outcome date and then calculates a pregnancy start date [i.e. estimated last menstrual period (LMP) date]. When gestational age is not available, it is imputed using published outcome-specific medians or definitions (e.g. live births estimated as 40 weeks’ gestation).11–13
Description of the VSD PEA and DPA.
As an initial step in modifying the original PEA, we used the Centers for Medicare and Medicaid Services general equivalent maps and forward backward mapping methods to update the diagnosis codes included in the PEA look-up tables. 14 The study team discussed and adjudicated discrepancies in the code lists generated by the mapping method, as well as ICD-9 and ICD-10 codes identified from manual code book reviews, to form a final list of look-up table codes (available from the authors by request). We then modified the original VSD PEA SAS program to utilize the new look-up tables and incorporate new gestational age data as described below.
Gestational age estimation
The original PEA included gestational age data from linked birth records when available, but otherwise assigned an imputed outcome-specific estimate. To expand the capture of gestational age data, sites now create standardized data files including estimated date of delivery (EDD) and LMP dates from the EHRs. ICD-10 codes for prenatal care are an additional new source of gestational age data as they now include a fifth digit specifying the gestational age or trimester. For example, Z34.01 represents a first trimester encounter for the supervision of a normal first pregnancy, and Z34.02 represents a similar encounter occurring in the second trimester.
We modified the original PEA to incorporate these new sources of gestational age data. The enhanced PEA estimates the pregnancy start date based on data availability using the following hierarchy: (a) linked birth records (live births only); (b) EDD; (c) gestational-age specific ICD-10 codes; (d) LMP. If none of these data are available, the enhanced PEA uses the imputed outcome-specific gestational age to assign a start date to the episode.
Validation of the enhanced PEA
To validate the enhanced PEA, each site reviewed medical records for a sample of pregnancy episodes with outcomes from 1 January through 30 June 2016. From each participating site, among women with continuous enrollment in the health system during their pregnancy episodes, we randomly sampled 15 live births, 15 SABs, 15 IABs, and 15 SBs (60 total per site). Sites with fewer than 15 episodes in a single category reviewed all that were available. We first confirmed that a pregnancy episode was noted in the medical record and then compared the PEA outcome with that noted by the abstractor. We calculated agreement (within ±30 days) between the VSD PEA and abstracted pregnancy beginning and end dates and described the distribution of the differences in the algorithm-assigned and abstracted dates by outcome. The study team and clinical adjudicator (KKV) reviewed and adjudicated cases where the abstractor and PEA differed on outcome type.
DPA
As the PEA only identifies pregnancies that have ended, we developed a new algorithm, the DPA, to identify new and ongoing pregnancy episodes within a specified time period, such as an influenza season. The DPA searches the specified time period for the pregnancy indicator codes in the PEA look-up tables (Figure 1). When a code associated with a pregnancy outcome (e.g. Z37.0 delivery of single live birth) is identified, the DPA uses the PEA logic to identify the episode, outcome type, and pregnancy start and end dates. When no outcome code is available, the DPA estimates the pregnancy start date and projected outcome date using the hierarchy of EDD, gestational age specific ICD-10 codes, and LMP. If gestational age data are not available, the DPA output is limited to the dates of the first and most recent pregnancy indicator codes and the number of pregnancy indicator codes identified during the study period.
We conducted two comparisons of pregnancy episodes identified by the DPA with those identified by the enhanced PEA. To assess how quickly the DPA identifies ongoing pregnancy episodes, we identified live birth and non-live birth outcomes in December 2016 using the enhanced PEA. Next, we described the month these episodes were first identified using the DPA. We also described the concordance between the episodes identified by the DPA and the PEA by following episodes identified by the DPA in February 2016 through the end of the calendar year.
Human subjects’ protection
The study protocol was reviewed and approved by the institutional review boards at all participating sites: Harvard Pilgrim (non-data-contributing), HealthPartners Institute, Marshfield Clinic Research Institute, Denver Health, Kaiser Permanente of Colorado, Kaiser Permanente Northwest, Kaiser Permanente Washington Research Institute, Northern California Kaiser Permanente, and Southern California Kaiser Permanente. All sites were granted a waiver of informed consent as the study met 45 CFR 46.116(d) criteria. The CDC received a determination of non-engagement from their institutional review board.
Results
Enhanced PEA
The most recent data file generated by the enhanced PEA included 2,485,410 pregnancy episodes from 1 January 2002 through 31 December 2018 from eight VSD sites. Most (74.7%) of the pregnancy outcomes identified were live births, followed by SABs (12.2%), IABs (10.2%), SBs (0.4%), and pregnancies of other or uncertain outcome (2.6%) (data not shown).
In the subset of pregnancy episodes ending in 2018, the source of gestational age data used by the enhanced PEA to estimate the pregnancy start date varied by pregnancy outcome type as well as by study site (Table 1). Linked birth records were only available for live births and the availability of these data varied by site ranging from 49% to 96%; overall, 81% of gestational age data were obtained from birth records for live births, 14% from EDD, 4% from ICD-10 codes, and less than 1% from both LMP and imputed estimates. The proportions of imputed gestational age estimates were generally higher for non-live birth episodes compared with the live birth episodes; 58% of IAB, 31% of SAB, and 11% of SB gestational ages were imputed.
Sources of gestational age by pregnancy outcome type and study site for pregnancy episodes ending in 2018.
EDD, estimated date of delivery; IAB, induced abortion; ICD-10, International Classification of Disease, tenth revision; LMP, last menstrual period; SAB, spontaneous abortion; SB, stillbirth.
Validation of the enhanced PEA
A total of 53,534 pregnancy episodes ending in live birth, SAB, IAB, or SB were identified by the PEA among women with continuous health plan enrollment from 1 January through 30 June 2016; 375 of these episodes were sampled for review. The abstractor confirmed a pregnancy episode in the medical record of most of the sampled episodes; however, the confirmation rate for IABs was lower than other outcomes at 77% (Table 2). For most, the abstractor found documentation in the medical record that the woman wanted a referral for an IAB or was planning an IAB, but there was no record of the actual procedure in the record. Many of these episodes were identified by insurance claims for IABs performed outside the health system, which the abstractors were unable to confirm using EHR data.
Validation of the enhanced Vaccine Safety Datalink Pregnancy Episode Algorithm.
Pregnancy episodes ending January 2016 through June 2016 with continuous enrollment from 92 days before pregnancy began to 31 days after the pregnancy outcome.
Pregnancy Episode Algorithm and abstractor-identified dates agreed within ±30 days.
Includes all non-live birth outcomes sampled.
Agreement on outcome type and date was 100% for live births but was lower for the individual non-live birth outcomes. Many of the discrepancies were associated with pregnancy losses occurring around 19–20 gestational weeks, which could be considered late SABs or early SBs as 20 weeks’ gestation is a common cut point for defining these outcomes. 15 In addition, many of the SABs ended with a procedure like a dilation and curettage, which was flagged by the PEA as an IAB. To minimize some of these challenges, we created post-hoc a pregnancy loss category which included all sampled SABs, IABs, and SBs. The agreement between the abstracted outcome type (live birth versus non-live birth) and date using the combined pregnancy loss category for the non-live births was 99% and 98%, respectively.
When gestational age was available in the medical record, the agreement ±30 days between the abstracted gestational age and the PEA assigned gestational age was high for live births (97%), but lower for pregnancy losses (75%) (Table 2). After excluding 23 SABs, 35 IABs, and 8 SBs with imputed gestational age values, agreement ±30 days between the abstracted and PEA assigned gestational age for non-live birth outcomes increased to 88% (data not shown). Among pregnancy episodes with agreement in gestational age, the difference between the PEA assigned and abstracted dates was within ±7 days for 78% of live births, 58% of SABs, 39% of IABs, and 86% of SBs (Figure 2).
Differences in algorithm assigned and abstracted gestational ages by pregnancy outcome type.
DPA
The enhanced PEA identified 8971 live births that occurred in December 2016. The DPA identified 89% of these live birth episodes ⩾6 months prior to the outcome date (Figure 3). The PEA identified 4194 non-live birth outcomes in December 2016. The DPA identified 47% of these pregnancy episodes ⩾1 month prior to the outcome date (Figure 3). Most were identified at or near the time of the outcome date.
Month of detection by the DPA for pregnancies ending in live birth or non-live birth in December 2016.
When restricted to the 86,147 pregnancies identified by the DPA in February 2016 (i.e. episodes that should have had an outcome in 2016), 92% had subsequent outcomes identified by the PEA by the end of 2016. Among those without a subsequent PEA outcome, 70% had incomplete health plan enrollment in 2016 and were likely lost to follow up.
Discussion
Like its predecessor, the enhanced VSD PEA is a useful tool for identifying pregnancy episodes and associated dates for studies of maternal vaccine safety. The additional sources of gestational age (ICD-10 codes, EDDs, and LMPs) included in the enhanced PEA improved the estimation of pregnancy start dates and reduced reliance on imputed values, especially for the non-live birth outcomes, which were all imputed with the original algorithm. Furthermore, the DPA enhances the VSD infrastructure for evaluating the safety of maternal vaccination by improving the timeliness of pregnancy identification and can be used to support near real-time safety and vaccination coverage assessments.
We have used the VSD PEA to conduct several studies of vaccine safety during pregnancy,16–20 and the enhanced algorithm provides even greater precision for identifying and dating live births, which means that large observational safety studies can rely on the coded medical records data without requiring labor-intensive and costly manual record review. While we feel confident about the algorithm’s ability to identify live births accurately and precisely, most of our studies involving non-live birth outcomes have relied on manual medical record review to collect precise information about the timing of vaccine exposures related to pregnancy beginning and end dates.21,22 The enhanced PEA accurately identifies non-live birth outcomes, but additional review is needed to determine the precise type of pregnancy loss. Even with careful medical record review, it may still be difficult to correctly categorize pregnancy losses occurring around 19–20 weeks’ gestation.
We have presented the agreement between the chart-abstracted and algorithm-derived outcomes and dates rather than designating the abstracted data as the ‘gold standard’ and calculating the sensitivity and specificity of the algorithm. While most VSD data are derived from EHRs, insurance claims data are also included in the diagnosis and procedure code files used by the VSD PEA and these records are often not available to abstractors in the EHR. In the case of IABs, the PEA may be considered the ‘gold standard’ because it likely identifies procedures performed outside the health systems through claims that are not available for abstraction in the EHR.
In addition to enhancing the PEA with additional sources of gestational age data, we also developed the DPA to support more rapid assessments of maternal vaccine safety. The DPA generally identifies pregnancy episodes earlier than the PEA, but the trade-off for improved timeliness means that some key information about a pregnancy episode, including the outcome and gestational age, may be incomplete. The DPA also identifies pregnancies that ultimately do not have outcomes in the health system EHR or claims data. This can occur in women who have a change in health plan membership before delivery. These women have previously been excluded from retrospective studies using the PEA because of incomplete data. We are currently using the DPA to monitor influenza and COVID-19 vaccination coverage during pregnancy, to assess COVID-19 reactogenicity in pregnant women, and to identify severe acute respiratory syndrome coronavirus 2 infection during pregnancy. 23
In conclusion, the enhanced VSD PEA is a useful tool for evaluating the safety of vaccination during pregnancy. The inclusion of additional sources of gestational age data allows us to estimate more precisely the beginning of pregnancy episodes, especially for non-live birth outcomes which were all imputed with the original algorithm. To improve our ability to monitor rapidly the safety of maternal vaccination, we developed the DPA. The enhancements we have made to the VSD infrastructure will allow us to better evaluate the safety of current and future vaccinations administered during or around the time of pregnancy, to conduct disease surveillance in pregnant women, and to estimate maternal vaccination coverage.
Footnotes
Acknowledgements
The authors would like to thank: Stacy Harsh for chart abstraction, and Ashley Stoneburner and Sanchita Sengupta for their early contributions to data analysis (Kaiser Permanente Northwest); Alison Kawai for providing methods expertise (Harvard Pilgrim); Sophia Newcomer for providing methods expertise and Kate Burniece for chart abstraction (Kaiser Permanente Colorado); Nidia Golla, Claire Park, and Denison Ryan for chart abstraction (Kaiser Permanente Southern California); Jim Donahue for providing content and methods expertise (Marshfield Clinic Research Institute).
Conflict of interest statement
ALN has received research funding from Pfizer for unrelated work. NPK has received research funding from GlaxoSmithKline, Pfizer, Sanofi Pasteur, and Merck Protein Science (now Sanofi Pasteur) for unrelated work.
The VSD Project is funded by the CDC. The funders of this study had no role in the design of the study, data collection, data analysis, data interpretation, or writing of the report. The manuscript was approved by the CDC clearance process.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Disclaimer
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC.
