Sage Journals: Discover world-class research

Abstract

Objectives:

Having accurate influenza vaccination coverage estimates can guide public health activities. The objectives of this study were to (1) validate the accuracy of electronic health record (EHR)–based influenza vaccination data among pregnant women compared with survey self-report and (2) assess whether survey respondents differed from survey nonrespondents by demographic characteristics and EHR-based vaccination status.

Methods:

This study was conducted in the Vaccine Safety Datalink, a network of 8 large medical care organizations in the United States. Using EHR data, we identified all women pregnant during the 2018-2019 or 2019-2020 influenza seasons. Surveys were conducted among samples of women who did and did not appear vaccinated for influenza according to EHR data. Separate surveys were conducted after each influenza season, and respondents reported their influenza vaccination status. Analyses accounted for the stratified design, sampling probability, and response probability.

Results:

The survey response rate was 50.5% (630 of 1247) for 2018-2019 and 41.2% (721 of 1748) for 2019-2020. In multivariable analyses combining both survey years, non-Hispanic Black pregnant women had 3.80 (95% CI, 2.13-6.74) times the adjusted odds of survey nonresponse; odds of nonresponse were also higher for Hispanic pregnant women and women who had not received (per EHR data) influenza vaccine during current or prior influenza seasons. The sensitivity, specificity, and positive predictive value of EHR documentation of influenza vaccination compared with self-report were ≥92% for both survey years combined. The negative predictive value of EHR-based influenza vaccine status was 80.5% (95% CI, 76.7%-84.0%).

Conclusions:

EHR-based influenza vaccination data among pregnant women were generally concordant with self-report. New data sources and novel approaches to mitigating nonresponse bias may be needed to enhance influenza vaccination surveillance efforts.

Keywords

influenza vaccination pregnancy misclassification electronic health records nonresponse bias

Influenza viruses cause substantial morbidity and mortality every year in the United States and globally,^1-4 including among pregnant women who encounter higher risk of severe disease compared with nonpregnant women of similar age.^5-7 Vaccination is the primary means of influenza prevention,^8,9 and improving influenza vaccination coverage is a national public health priority.¹⁰ Influenza vaccination coverage among pregnant women in the United States is primarily monitored through in-person,^11,12 telephone,¹³ and internet-based surveys,^5,14,15 with respondents asked to report their vaccination status. While survey-based methods for assessing influenza vaccination coverage have strengths, important limitations are operant.^13,16 Self-report of vaccination may be inaccurate by mistake or because of social desirability bias, survey response rates are generally falling, and survey-based methods are subject to nonresponse bias.¹⁷ In addition, if survey response is more likely from vaccinated people than from unvaccinated people,¹⁸ influenza vaccination coverage based on national surveys could overestimate true coverage.

Electronic health record (EHR)¹⁹ and claims data²⁰ have also been used to estimate influenza vaccination coverage during pregnancy, an approach that may mitigate some limitations of survey-based methods. EHR data can be used to reliably identify pregnant women,²¹ assess other risk factors for influenza disease,^8,9 and obtain demographic information such as race and ethnicity. In addition, if EHR data indicate that a pregnant woman was vaccinated during the current influenza season, the positive predictive value (PPV) is high compared with self-report.¹⁸ However, if EHR data do not indicate influenza vaccination, it is possible that a pregnant woman was vaccinated outside the medical home, such as at a workplace or pharmacy.^18,22 In this circumstance, EHR data would underestimate true coverage.

Having accurate and timely influenza vaccination coverage estimates, particularly in groups at increased risk of serious influenza disease,^5-7 is necessary to guide public health activities^8,9 toward achieving national public health goals.¹⁰ The objectives of the current study were to (1) assess by survey the validity of EHR-based influenza vaccination data among pregnant women and (2) assess whether survey respondents differed from survey nonrespondents by demographic characteristics and EHR-based vaccination status.

Methods

Study Setting

This study was conducted in the Vaccine Safety Datalink (VSD), a collaboration between the Centers for Disease Control and Prevention and 8 large medical care organizations (referred to as “sites”).^23-25 Participating VSD sites included Marshfield Clinic, HealthPartners, Denver Health, Kaiser Permanente (KP) Washington, KP Northwest, KP Northern California, KP Southern California, and KP Colorado. Denver Health cares for publicly insured, privately insured, and uninsured patients^26,27; the remaining VSD sites primarily serve insured patients.^23-25 The demographic and socioeconomic characteristics of the VSD population are similar to those of the US population,²⁵ while influenza vaccination coverage among pregnant women in the VSD is generally higher than coverage indicated by national estimates.¹⁹

Vaccination data at VSD sites, which are used for clinical and research purposes, are derived from several data sources. Vaccines ordered and administered within VSD sites represent a high proportion of available vaccine records. In addition, vaccine data from claims and state immunization information systems (at 6 VSD sites) are added to a patient’s EHR.^19,28,29 Influenza vaccines administered in pharmacies would be integrated into EHR-based vaccination data if a claim were submitted to insurance or if the pharmacy submitted vaccine data to a state immunization information system. In the current study, the term “EHR-based vaccination data” refers to all available electronic vaccination data integrated into an individual’s record.

Study Overview

We identified all women at VSD sites who were pregnant during the 2018-2019 or 2019-2020 influenza season. We then conducted surveys among a sample of pregnant women who did and did not appear vaccinated during the same season according to EHR data. Pregnant women were surveyed during only 1 influenza season. Respondents were asked to report their influenza vaccination status for the current season, which was compared with EHR-based vaccination data. We considered vaccination self-report as the criterion standard.

Study Population: Identification of Pregnant Women

The VSD has developed and validated an algorithm to identify pregnancies with a high degree of accuracy.²¹ The algorithm can identify current pregnancies in near-real time, using weekly updates of VSD EHR-based data.³⁰ The following data sources are used: pregnancy-related International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes,³¹ prenatal procedure codes, and estimated delivery and last menstrual period dates from obstetric flowsheets.^21,30

Using this algorithm, we identified all women aged 18-49 years who were pregnant at any time from August 1 through January 31 of the respective influenza season (eg, for the 2018-2019 survey, pregnant from August 1, 2018, through January 31, 2019). We required women to have continuous health insurance enrollment through the influenza season (for Denver Health, 1 outpatient visit was used as a proxy for health insurance enrollment). We excluded women with an ICD-10-CM code indicating an adverse pregnancy outcome such as spontaneous abortion, stillbirth, or anencephaly. We excluded from analysis women with multiple pregnancy episodes during a single influenza season, women with a diagnosis code for a vaccine allergy, and women with presumed influenza vaccine data errors (eg, receipt of a nonindicated vaccine, such as high-dose influenza). For the 2018-2019 survey, 439 of 107 126 pregnant women were excluded for vaccine allergy or invalid vaccine data; for the 2019-2020 survey, 409 of 97 661 pregnant women were excluded.

Survey Design and Administration

Sampling procedures

After identifying all eligible pregnant women in the VSD as our sampling frame, we randomly sampled women for survey administration. We stratified sampling for each survey year by VSD site and EHR-based influenza vaccination status, with women who appeared unvaccinated in EHR data oversampled. The total sample of pregnant women was 1247 for the 2018-2019 survey and 1748 for the 2019-2020 survey. For the 2018-2019 survey, we sampled 999 unvaccinated (per EHR data) and 248 vaccinated (per EHR data) women; for the 2019-2020 survey, the numbers sampled were 1460 unvaccinated and 288 vaccinated. In the 2018-2019 survey, non-Hispanic Black women were less likely to respond than women from other racial and ethnic groups; consequently, non-Hispanic Black women were oversampled for the 2019-2020 survey. With a sample of 999 EHR-unvaccinated women (ie, the 2018-2019 survey), a 50% response rate, and 60% of EHR-unvaccinated women reporting they were unvaccinated, the study was powered to achieve a CI width of 8.8% around the estimated proportion confirming their vaccination status.

Survey instrument

We developed and pilot-tested a survey instrument with 8 individuals, and then revised it accordingly. We based survey questions on published survey instruments; whenever possible, we used exact wording from prior instruments.^32-38 Survey content included whether the respondent received an influenza vaccine and, if so, at what location; whether influenza vaccine had been recommended and offered by a health professional; if the respondent was not vaccinated, the primary reason why not; and attitudes about influenza vaccination. We also asked women to confirm that they were currently or had recently been pregnant; we excluded from analyses women who did not confirm pregnancy on the survey.

Survey administration

For the 2018-2019 influenza season, survey administration began March 15, 2019; for the 2019-2020 influenza season, survey administration began February 18, 2020. Surveys were in the field for 15 weeks. Women received up to 3 mailed surveys, up to 5 emails with a unique hyperlink to an internet-based survey, and up to 2 automated telephone reminders. Outreach stopped once the survey was completed or a person requested no further contact. In nearly all aspects, survey administration was consistent across survey years and VSD sites. One VSD site required that participants receive a pre-survey letter with an opportunity to opt out from the survey and did not permit email contact; consequently, at this site, participants received an additional mailed survey. For the 2018-2019 influenza season, a Spanish-language version of the survey was sent to women with an EHR designation of preferred language Spanish. Because of resource constraints, the 2019-2020 survey was available in English only. The internet form of the survey used Research Electronic Data Capture.³⁹ Respondents received a $20 gift card for completing the survey.

Analytic Methods

Pregnant women were considered survey respondents if they answered the primary survey question of “Since July 1 of [the current influenza season] have you had a flu vaccination?” We used the Pearson χ² test to compare respondents with nonrespondents, with P < .05 considered significant. Self-reported vaccination status was treated as the criterion standard for all analyses. We accounted for the stratified sampling design, included a finite population correction, and incorporated inverse probability weighting for sampling and survey response probabilities. For the 2018-2019 season analyses, sampling weights accounted for VSD site and EHR vaccination status; for the 2019-2020 season analyses, weighting also accounted for oversampling non-Hispanic Black women. Weighted percentages for survey responses and EHR vaccination validity measures (sensitivity, specificity, PPV, and negative predictive value [NPV]) were reported with Clopper–Pearson 95% CIs.⁴⁰ In the context of this study, sensitivity is interpreted as follows: among all individuals who self-report influenza vaccination, what percentage have EHR documentation of influenza vaccination? When presenting descriptive frequencies, we combined survey results across influenza seasons. We estimated κ coefficients and associated CIs by using bootstrap replication variance estimation.

Two multivariable logistic regression models were developed: a model assessing factors associated with survey nonresponse (among the entire sample) and a model assessing factors associated with self-reporting influenza vaccination (among women unvaccinated according to EHR data). We explored correlation within VSD site and tested for the interaction between race and ethnicity and vaccination status. Covariates, identified a priori based on prior publications,^11,14,15,19 included the following: age group (18-24, 25-34, 35-49 y), race and ethnicity (non-Hispanic White, non-Hispanic Black, Hispanic, non-Hispanic Asian, non-Hispanic Other [multiracial, Hawaiian, American Indian/Alaska Native, and unspecified other race and ethnicity]), chronic health condition (yes/no), gravidity (first pregnancy, pregnant before), vaccinated for influenza in prior season (yes/no), health care provider vaccination recommendation (yes/no), health insurance type (private, Medicaid, none), education (≤high school, some college/college graduate, advanced degree), annual household income (<$75 000, ≥$75 000), and VSD site. Covariates were retained in the final models regardless of significance. Multivariate analyses accounted for the study design and sampling strategy,⁴¹ study year, VSD site, EHR vaccination status, and race and ethnicity. VSD site was treated as a fixed effect. In addition, we conducted a subanalysis among pregnant women unvaccinated according to EHR data, examining factors associated with self-reporting influenza vaccination (eg, EHR false negatives). We conducted all analyses using SAS version 9.4 (SAS Institute, Inc).

The KP Colorado Institutional Review Board approved this study, and other study sites ceded oversight to KP Colorado. Written consent was not required for survey administration, and survey participants could opt out in writing or by telephone.

Results

Survey Sample

For the 2018-2019 influenza season, a total of 106 687 survey-eligible pregnant women were identified, of whom 1247 (1.2%) were sampled for survey administration (Table 1). For the 2019-2020 influenza season, a total of 97 244 survey-eligible pregnant women were identified, of whom 1748 (1.8%) were sampled.

Table 1.

Characteristics of respondents and nonrespondents to surveys administered to women who were pregnant during the 2018-2019 and 2019-2020 influenza seasons, Vaccine Safety Datalink study sites^a

Characteristic^b	2018-2019 Respondents, no. (%)	2018-2019 Nonrespondents, no. (%)	P value^c	2019-2020 Respondents, no. (%)	2019-2020 Nonrespondents, no. (%)	P value^c
Total	630 (100.0)	617 (100.0)		721 (100.0)	1027 (100.0)
Age, y			<.001			<.001
18-24	86 (13.7)	176 (28.5)	<.001	116 (16.1)	248 (24.1)	<.001
25-34	405 (64.3)	342 (55.4)		432 (59.9)	582 (56.7)
35-49	139 (22.1)	99 (16.0)		173 (24.0)	197 (9.2)
Race and ethnicity^d			<.001			<.001
Non-Hispanic White	398 (63.2)	244 (39.5)	<.001	299 (41.5)	211 (20.5)	<.001
Non-Hispanic Black	28 (4.4)	77 (12.5)		234 (32.5)	550 (53.6)
Hispanic	107 (17.0)	164 (26.6)		99 (13.7)	165 (16.1)
Non-Hispanic Asian	38 (6.0)	46 (7.5)		37 (5.1)	31 (3.0)
Non-Hispanic Other	18 (2.9)	32 (5.2)		15 (2.1)	15 (1.5)
Missing	41 (6.5)	54 (8.8)		37 (5.1)	55 (5.4)
Chronic health condition^e			.29			.12
Yes	126 (20.0)	109 (17.7)	.29	174 (24.1)	282 (27.5)	.12
No	504 (80.0)	508 (82.3)		547 (75.9)	745 (72.5)
Vaccinated for influenza in current season (per EHR)			<.001			.004
Yes	169 (26.8)	79 (12.8)		141 (19.6)	147 (14.3)
No	461 (73.2)	538 (87.2)		580 (80.4)	880 (85.7)
Vaccinated for influenza in prior season (per EHR)			.002			<.001
Yes	160 (25.4)	111 (18.0)	.002	206 (28.6)	207 (20.2)	<.001
No	470 (74.6)	506 (82.0)		515 (71.4)	820 (79.8)

Abbreviation: EHR, electronic health record.

Vaccine Safety Datalink^23-25 sites are located in California, Colorado, Minnesota, Oregon, Washington, and Wisconsin.

Individuals unvaccinated for influenza in the current season (per EHR) were oversampled in both survey years; non-Hispanic Black women were oversampled in the 2019-2020 survey.

Using the Pearson χ² test, with P < .05 considered significant.

For respondents and nonrespondents, data on race and ethnicity were obtained from the EHR. Other race and ethnicity included 37 multiracial, 15 Hawaiian, and 9 American Indian/Alaska Native women, as well as 19 women with an unspecified other race and ethnicity.

Chronic health conditions (not including pregnancy) that increased the risk of influenza-related morbidity and mortality^9,42 were identified from EHR encounters using International Classification of Diseases, Tenth Revision, Clinical Modification³¹ diagnosis codes.

Survey Response

For the 2018-2019 influenza season, 630 of 1247 (50.5%) women responded; for the 2019-2020 influenza season, 721 of 1748 (41.2%) women responded (Table 1). Nearly all respondents (94.9% in 2018-2019, 97.1% in 2019-2020) confirmed they were or had been pregnant. Response rates were highest among non-Hispanic White pregnant women (62.0% in 2018-2019, 58.6% in 2019-2020) and lowest among non-Hispanic Black pregnant women (26.7% in 2018-2019, 29.9% in 2019-2020). Respondents differed significantly from nonrespondents by age group, race and ethnicity, and EHR-documented influenza vaccination in current and prior influenza seasons (all P < .001).

In multivariable analyses, non-Hispanic Black pregnant women had 3.80 (95% CI, 2.14-6.74) times the adjusted odds of not responding to the survey compared with non-Hispanic White pregnant women (both survey years combined) (Table 2). The adjusted odds of nonresponse were higher for Hispanic pregnant women than for non-Hispanic White pregnant women and for those who had not received (per EHR data) influenza vaccine during current or prior seasons compared with those who had received the influenza vaccine.

Table 2.

Multivariable analyses of characteristics associated with nonresponse to influenza vaccination surveys administered to women who were pregnant during the 2018-2019 and 2019-2020 influenza seasons, Vaccine Safety Datalink study sites^a

Characteristic	2018-2019 Influenza season, aOR (95% CI)^b	2019-2020 Influenza season, aOR (95% CI)^b	Both seasons combined, aOR (95% CI)^b
Age, y
18-24	1.85 (0.75-4.54)	1.03 (0.26-4.05)	1.28 (0.61-2.70)
25-34	1 [Reference]	1 [Reference]	1 [Reference]
35-49	0.75 (0.36-1.56)	0.50 (0.17-1.46)	0.59 (0.31-1.13)
Race and ethnicity^c
Non-Hispanic White	1 [Reference]	1 [Reference]	1 [Reference]
Non-Hispanic Black	4.67 (2.10-10.39)	2.79 (1.20-6.48)	3.80 (2.14-6.74)
Hispanic	4.62 (2.07-10.33)	2.68 (0.77-9.41)	3.49 (1.54-7.88)
Non-Hispanic Asian	1.57 (0.58-4.26)	0.45 (0.14-1.44)	0.84 (0.35-2.03)
Non-Hispanic Other^d	4.12 (1.26-13.44)	2.28 (0.60-8.72)	2.42 (0.78-7.51)
Missing	1.69 (0.40-7.08)	1.88 (0.74-4.79)	1.27 (0.45-3.63)
Chronic health condition^e
Yes	1 [Reference]	1 [Reference]	1 [Reference]
No	2.10 (0.89-4.96)	0.87 (0.30-2.52)	1.30 (0.69-2.44)
Vaccinated for influenza in current season (per EHR)
Yes	1 [Reference]	1 [Reference]	1 [Reference]
No	1.87 (1.10-4.96)	1.20 (0.52-2.76)	1.46 (0.92-2.30)
Vaccinated for influenza in prior season (per EHR)
Yes	1 [Reference]	1 [Reference]	1 [Reference]
No	1.47 (0.68-3.18)	2.52 (0.82-7.69)	1.93 (1.02-3.64)

Abbreviations: aOR, adjusted odds ratio; EHR, electronic health record.

Vaccine Safety Datalink^23-25 sites are located in California, Colorado, Minnesota, Oregon, Washington, and Wisconsin.

Each column represents a separate multivariable regression model; models were adjusted for all variables listed and for Vaccine Safety Datalink site.

For respondents and nonrespondents, data on race and ethnicity were obtained from the EHR.

Other race and ethnicity included multiracial, Hawaiian, and American Indian/Alaska Native women and women with an unspecified other race and ethnicity.

Net Implied Bias in Coverage Estimates

In the survey sample for both years combined, the sampling probability weighted prevalence of having received influenza vaccine (per EHR data) was 70.6% (95% CI, 66.2%-74.6%). The sampling probability weighted prevalence of vaccination (per EHR data) among survey respondents was 75.9% (95% CI, 71.0%-80.3%). The net implied bias in coverage would therefore be 5.3% overall. Stratified by race and ethnicity, the net implied bias in coverage would be 6.5% among non-Hispanic Black, 5.2% among non-Hispanic White, and 1.1% among Hispanic pregnant women.

EHR Vaccination Data Compared With Self-Report

Combining results across both survey years, among pregnant women who appeared vaccinated according to EHR data, 8 of 310 self-reported they were not vaccinated (weighted percent false positives: 0.6%; 95% CI, 0.1%-2.3%). Combining both survey years, among pregnant women who appeared unvaccinated according to EHR data, 304 of 1041 self-reported they were vaccinated (weighted percent false negatives: 19.5%; 95% CI, 16.0%-23.3%).

The sensitivity, specificity, and PPV of EHR documentation of influenza vaccination were high for 2018-2019, 2019-2020, and both years combined (Table 3). For example, the sensitivity for both years combined was 92.4% (95% CI, 89.9%-94.5%). The NPV was the lowest of the validity measures in 2018-2019, 2019-2020, and both years combined. The κ coefficient showed strong agreement (κ, both years combined, 0.84; 95% CI, 0.81-0.87) between EHR documentation of influenza vaccination and self-report.

Table 3.

Sensitivity, specificity, positive and negative predictive values, and κ coefficient for EHR documentation of influenza vaccination compared with self-reported vaccination status among women who were pregnant during the 2018-2019 and 2019-2020 influenza seasons, Vaccine Safety Datalink^a,b

Characteristic	No.	Sensitivity	Specificity	Positive predictive value	Negative predictive value	κ coefficient
Year 1 survey (2018-2019)	630	92.7 (89.2-95.4)	98.8 (96.1-99.8)	99.6 (97.0-100.0)	82.0 (76.8-86.4)	0.86 (0.82-0.89)
Year 2 survey (2019-2020)	721	92.1 (87.4-95.5)	97.4 (92.5-99.5)	99.1 (95.9-100.0)	78.8 (72.7-84.1)	0.83 (0.78-0.88)
Years 1 and 2 combined	1351	92.4 (89.9-94.5)	98.1 (95.7-99.4)	99.4 (97.7-99.9)	80.5 (76.7-84.0)	0.84 (0.81-0.87)
Years 1 and 2 combined, by specific strata
Age group, y
18-24	202	91.7 (80.5-97.6)	97.8 (91.1-99.8)	98.6 (86.7-100.0)	86.9 (78.4-93.0)	0.87 (0.78-0.97)
25-34	837	92.2 (88.8-94.8)	97.7 (94.0-99.4)	99.3 (96.8-100.0)	79.0 (73.8-83.6)	0.83 (0.79-0.87)
35-49	312	93.5 (88.5-96.8)	99.7 (96.9-100.0)	99.9 (95.6-100.0)	80.7 (72.4-87.4)	0.86 (0.80-0.92)
Race and ethnicity^c
Non-Hispanic White	697	91.7 (87.5-94.9)	97.8 (92.5-99.7)	99.2 (96.2-100.0)	79.0 (73.1-84.1)	0.83 (0.78-0.88)
Non-Hispanic Black	262	83.6 (75.4-89.9)	97.7 (92.3-99.7)	96.7 (89.0-99.5)	88.3 (82.3-92.8)	0.82 (0.75-0.90)
Hispanic	206	95.4 (88.1-98.8)	97.8 (92.6-99.7)	99.3 (90.6-100.0)	87.4 (79.8-92.8)	0.90 (0.84-0.95)
Non-Hispanic Asian	75	90.4 (75.9-97.7)	100.0 (NA)	100.0 (NA)	67.7 (49.5-82.7)	0.76 (0.61-0.90)
Non-Hispanic Other^d	33	98.2 (76.4-100.0)	100.0 (NA)	100.0 (NA)	87.1 (67.2-97.2)	0.92 (0.83-1.00)
Missing	78	87.0 (72.7-95.5)	100.0 (NA)	100.0 (NA)	68.7 (51.0-83.1)	0.75 (0.59-0.91)
Chronic health condition^e
Yes	300	94.2 (88.9-97.4)	97.0 (91.3-99.4)	99.3 (93.9-100.0)	79.1 (69.7-86.6)	0.84 (0.77-0.90)
No	1051	91.8 (88.4-94.4)	98.4 (95.4-99.7)	99.4 (97.3-100.0)	80.9 (76.5-84.7)	0.84 (0.81-0.88)
Vaccinated for influenza in prior season (per EHR)
Yes	366	95.4 (92.3-97.5)	93.9 (80.8-99.1)	99.6 (97.1-100.0)	53.9 (41.7-65.7)	0.66 (0.56-0.77)
No	985	87.4 (81.7-91.8)	98.7 (96.2-99.7)	98.8 (95.4-99.9)	85.7 (81.8-89.0)	0.85 (0.80-0.89)

Abbreviations: EHR, electronic health record; NA, not applicable.

Vaccine Safety Datalink^23-25 sites are located in California, Colorado, Minnesota, Oregon, Washington, and Wisconsin.

All values are percentage (95% CI) unless otherwise indicated. Point estimates and 95% CIs accounted for sample design strata, sample and response weights, and finite population correction; κ coefficient and associated 95% CIs were estimated with bootstrap replication variance estimation; 95% CIs for all other measures were estimated with Clopper–Pearson.

Data on race and ethnicity were obtained from the EHR.

Other race and ethnicity included multiracial, Hawaiian, and American Indian/Alaska Native women and women with an unspecified other race and ethnicity.

The validity of EHR documentation of influenza vaccination was similar across age groups and chronic health condition status (Table 3). The NPV was lower among women who (per EHR data) had received influenza vaccine the prior season (53.9%; 95% CI, 41.7%-65.7%) than among women who were unvaccinated (per EHR data) in the prior season (85.7%; 95% CI, 81.8%-89.0%).

We conducted a subanalysis among pregnant women unvaccinated according to EHR data to examine factors associated with self-reporting influenza vaccination (eg, EHR false negatives). In multivariate analyses, for both years combined, the pregnancy being the first pregnancy, having influenza vaccination recommended by a health care provider, and having received (per EHR data) influenza vaccine in the prior season were positively associated with self-reporting influenza vaccination (Table 4).

Table 4.

Multivariable analyses of characteristics associated with failure to capture data on influenza vaccination receipt within EHR data (EHR false negatives) among women who were pregnant during the 2018-2019 and 2019-2020 influenza seasons, Vaccine Safety Datalink^a

Characteristic	2018-2019 Influenza season, aOR (95% CI)^b	2019-2020 Influenza season, aOR (95% CI)^b	Both seasons combined, aOR (95% CI)^b
Age, y
18-24	0.73 (0.26-2.06)	0.99 (0.34-2.95)	0.70 (0.30-1.64)
25-34	1 [Reference]	1 [Reference]	1 [Reference]
35-49	0.56 (0.18-1.78)	0.68 (0.26-1.81)	0.54 (0.27-1.08)
Race and ethnicity^c
Non-Hispanic White	1 [Reference]	1 [Reference]	1 [Reference]
Non-Hispanic Black	0.29 (0.08-1.10)	1.01 (0.40-2.50)	0.72 (0.34-1.49)
Hispanic	0.80 (0.18-3.52)	1.07 (0.37-3.08)	0.98 (0.42-2.27)
Non-Hispanic Asian	2.30 (0.60-8.77)	2.71 (0.62-11.83)	2.27 (0.88-5.86)
Non-Hispanic Other^d	0.80 (0.05-14.01)	0.47 (0.10-2.15)	0.38 (0.09-1.54)
Missing	0.79 (0.21-3.06)	3.31 (1.09-10.09)	1.86 (0.81-4.29)
Chronic health condition^e
Yes	1.32 (0.47-3.72)	0.96 (0.35-2.67)	1.21 (0.65-2.27)
No	1 [Reference]	1 [Reference]	1 [Reference]
Gravidity
First pregnancy	3.01 (1.26-7.20)	2.18 (0.91-5.24)	2.70 (1.48-4.92)
Pregnant before	1 [Reference]	1 [Reference]	1 [Reference]
Rather not say	0.01 (0.01-0.01)	0.46 (0.03-7.31)	0.87 (0.14-5.46)
Vaccinated for influenza in prior season (per EHR)
Yes	4.67 (1.97-11.03)	7.97 (3.40-18.65)	7.30 (4.00-13.31)
No	1 [Reference]	1 [Reference]	1 [Reference]
Health care provider recommended influenza vaccination
Yes	2.36 (0.71-1.09)	8.08 (2.51-26.00)	4.46 (1.63-12.17)
No	1 [Reference]	1 [Reference]	1 [Reference]
Rather not say	0.24 (0.03-2.12)	0.46 (0.03-7.31)	0.34 (0.05-2.39)
Type of health insurance
Private	1 [Reference]	1 [Reference]	1 [Reference]
Medicaid	0.43 (0.17-1.09)	0.65 (0.25-1.66)	0.56 (0.28-1.10)
No coverage	0.08 (0-8.86)	0.36 (0.03-3.84)	0.12 (0-4.16)
Rather not say/not sure	0.45 (0.11-1.95)	1.46 (0.23-9.20)	1.13 (0.33-3.93)
Education
≤High school	0.79 (0.13-4.91)	1.01 (0.31-3.28)	1.13 (0.40-3.22)
Some college/college graduate	1 [Reference]	1 [Reference]	1 [Reference]
Advanced degree	4.77 (1.59-14.27)	2.13 (0.79-5.72)	2.97 (1.40-6.30)
Rather not say	0.14 (0.01-2.33)	0.84 (0.04-19.23)	0.94 (0.11-8.05)
Annual household income, $
<75 000	0.47 (0.18-1.24)	0.67 (0.24-1.86)	0.56 (0.29-1.07)
≥75 000	1 [Reference]	1 [Reference]	1 [Reference]
Rather not say	0.71 (0.14-3.70)	1.19 (0.34-4.20)	0.94 (0.36-2.44)

Abbreviations: aOR, adjusted odds ratio; EHR, electronic health record.

Vaccine Safety Datalink^23-25 sites are located in California, Colorado, Minnesota, Oregon, Washington, and Wisconsin.

Each column represents a separate multivariate regression model; models were adjusted for all variables listed and Vaccine Safety Datalink site.

Data on race and ethnicity were obtained from the EHR.

Other race and ethnicity included multiracial, Hawaiian, and American Indian/Alaska Native women and women with an unspecified other race and ethnicity.

Additional Vaccine-Related Information

Among pregnant women who reported they were vaccinated, 92.6% (95% CI, 86.5%-96.5%) reported receiving a recommendation from a health professional for influenza vaccination; among unvaccinated pregnant women, 83.9% (95% CI, 79.4%-87.8%) reported receiving a recommendation. Among pregnant women who reported they were vaccinated, 85.2% (95% CI, 75.7%-92.0%) reported being offered an influenza vaccine at a health care visit; among unvaccinated pregnant women, 84.4% (95% CI, 80.0%-88.2%) reported being offered a vaccine.

All respondents who self-reported receiving influenza vaccine were asked the location of their vaccination. For women unvaccinated per EHR data who reported vaccination (EHR false negatives), 35.6% (95% CI, 25.9%-46.4%) were vaccinated in a workplace, 24.2% (95% CI, 15.3%-34.9%) in a hospital, and 36.1% (95% CI, 27.4%-45.6%) in a physician’s office, clinic, or health center. For women vaccinated per EHR data who reported vaccination (EHR true positives), 1.1% (95% CI, 0.3%-3.1%) were vaccinated in a workplace, 18.7% (95% CI, 9.5%-31.2%) in a hospital, and 79.4% (95% CI, 67.1%-88.7%) in a physician’s office, clinic, or health center.

Discussion

To achieve high influenza vaccination coverage overall¹⁰ and among groups at high risk from influenza,^5-7 accurate and timely data on vaccination coverage are needed to guide public health activities.^8,9 However, no single data source can provide a complete picture of coverage.¹⁶ In this study, using self-report as the criterion standard, we found relatively close agreement between influenza vaccine receipt in EHR-based data and self-report, with high sensitivity, specificity, PPV, and κ. The NPV was lower at 80.5%, indicating that 19.5% of women who appeared unvaccinated in EHR data self-reported receiving influenza vaccine during the current season. In addition, using EHR data to characterize survey nonrespondents, we detected substantial risk of nonresponse bias across race and ethnicity and vaccination status, a finding with important implications for survey-based assessments of influenza vaccination coverage.

Aside from an investigation after the 2009 H1N1 influenza outbreak in the United States⁴³ and a report from Australia,⁴⁴ we are unaware of other studies assessing the accuracy of EHR-based influenza vaccination data among pregnant women. Accuracy has been assessed in other US populations, with some studies using self-report as the criterion standard^18,22 and others treating EHR⁴⁵ or immunization information system^46,47 data as the standard. Sy et al¹⁸ compared EHR-based influenza vaccination data with self-report among 50- to 79-year-old members of a large health care organization during the 2007-2008 influenza season: the NPV of EHR data was 79.5%. This value is surprisingly close to the estimate of 80.5% from our study. We anticipated finding a higher NPV among pregnant women in the current study, given the expansion of immunization information systems and the improved capacity of these systems to send immunization data to EHR systems.⁴⁸ Workplace vaccination programs may contribute to the lower-than-anticipated NPV; integrating workplace-based influenza vaccination into state immunization information systems may be an important strategy to improve accuracy.

Our findings also highlight the risk of nonresponse bias in surveys of influenza vaccination. Non-Hispanic Black and Hispanic pregnant women and the unvaccinated (per EHR data) were significantly less likely to respond to the survey than non-Hispanic White pregnant women. Non-Hispanic Black people, including those who are pregnant, have consistently lower influenza vaccination rates^11,12,15 and express less confidence in influenza vaccine safety and efficacy⁴⁹ than non-Hispanic White people; similar findings have also been observed among Hispanic people.^12,49 Because our survey was introduced as concerning “the flu (which is also called influenza) and flu vaccination,” unvaccinated individuals and those with low vaccine confidence might have been less willing than vaccinated individuals and those with high vaccine confidence to complete the survey. National surveys, particularly if immunization-focused, may encounter similar behaviors. This type of nonresponse bias could lead to an overestimation of influenza vaccination coverage, and weighting to a referent population may not necessarily correct for this bias.

Limitations

Our study findings had several limitations. First, survey nonresponse may have influenced our results; differential nonresponse to the survey could have negatively or positively affected the validity measures we assessed. Second, while we treated self-report as the criterion standard, self-reported vaccination can be inaccurate, and we could not review outside records (such as from a workplace) to verify respondents’ self-report. Third, misclassification of pregnancy status could have occurred, although nearly all survey respondents confirmed their pregnancy status. Fourth, patients at VSD sites are largely insured (Denver Health also cares for uninsured individuals), and the VSD population may not be generalizable to the overall population of pregnant women in the United States.

Conclusions

The current study can be viewed as part of a longer-term goal of enhancing the nation’s ability to monitor influenza vaccination coverage. The study highlights the strengths of using EHR-derived data for this purpose, including the ability to identify large populations of pregnant women, as well as the limitations, including missing some influenza vaccination occurring outside the medical home. Current EHR-based systems are also not representative of all states and may underrepresent important sociodemographic groups such as the uninsured. Integrating data from multiple sources, including from surveys,^5,11-15 EHR-based networks,¹⁹ and immunization information systems,⁴⁸ continues to be needed to monitor influenza vaccination coverage.

Footnotes

Acknowledgements

The authors acknowledge Bradley Crane, MS, and Stephanie Irving, MHS, at the Center for Health Research, Kaiser Permanente Northwest, Portland, Oregon, for their contributions to this work.

Authors’ Note

Matthew F. Daley, Liza M. Reifler, and Jo Ann Shoup had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Disclaimer

The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Liza M. Reifler has an immediate family member who owns stock in Merck, Abbott Laboratories, and AstraZeneca. Allison L. Naleway has received research support from Pfizer unrelated to the current study. Michael L. Jackson has received research support from Sanofi Pasteur unrelated to the current study. Huong McLean has received research support from Seqirus unrelated to the current study. Nicola Klein reports research support from GlaxoSmithKline, Sanofi Pasteur, Protein Science (now Sanofi Pasteur), Merck, and Pfizer unrelated to the current study.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: this research was funded by the Centers for Disease Control and Prevention, through a Task Order (contract no. 200-2012-53582-21454), issued as part of the Vaccine Safety Datalink project (contract no. 200-2012-53582). Coauthors from the Centers for Disease Control and Prevention were involved in design and conduct of the study, interpretation of data, and review and approval of the article.

ORCID iDs

Matthew F. Daley, MD

Hilda Razzaghi, PhD, MSPH

References

Centers for Disease Control and Prevention. Estimates of deaths associated with seasonal influenza—United States, 1976-2007. MMWR Morb Mortal Wkly Rep. 2010;59(33):1057-1062.

Reed

Chaves

Daily Kirley

, et al. Estimating influenza disease burden from population-based surveillance data in the United States. PLoS One. 2015;10(3):e0118369. doi:10.1371/journal.pone.0118369

Centers for Disease Control and Prevention. Disease burden of flu. 2021. Accessed March 16, 2021. https://www.cdc.gov/flu/about/burden/index.html

Rolfes

Foppa

Garg

, et al. Annual estimates of the burden of seasonal influenza in the United States: a tool for strengthening influenza surveillance and preparedness. Influenza Other Respir Viruses. 2018;12(1):132-137. doi:10.1111/irv.12486

Lindley

Kahn

Bardenheier

, et al. Vital signs: burden and prevention of influenza and pertussis among pregnant women and infants—United States. MMWR Morb Mortal Wkly Rep. 2019;68(40):885-892. doi:10.15585/mmwr.mm6840e1

Mertz

Geraci

Winkup

Gessner

Ortiz

Loeb

Pregnancy as a risk factor for severe outcomes from influenza virus infection: a systematic review and meta-analysis of observational studies. Vaccine. 2017;35(4):521-528. doi:10.1016/j.vaccine.2016.12.012

Prasad

Huang

Wood

, et al. Influenza-associated outcomes among pregnant, postpartum, and nonpregnant women of reproductive age. J Infect Dis. 2019;219(12):1893-1903. doi:10.1093/infdis/jiz035

Grohskopf

Alyanak

Broder

, et al. Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2020-21 influenza season. MMWR Recomm Rep. 2020;69(8):1-24. doi:10.15585/mmwr.rr6908a1

Grohskopf

Alyanak

Broder

Walter

Fry

Jernigan

DB.

Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2019-20 influenza season. MMWR Recomm Rep. 2019;68(3):1-21. doi:10.15585/mmwr.rr6803a1

10.

Office of Disease Prevention and Health Promotion. Healthy People 2030: increase the proportion of people who get the flu vaccine every year—IID-09. 2021. Accessed March 19, 2021. https://health.gov/healthypeople/objectives-and-data/browse-objectives/vaccination/increase-proportion-people-who-get-flu-vaccine-every-year-iid-09

11.

Ding

Kahn

Black

O’Halloran

Williams

WW.

Influenza vaccination coverage among pregnant women in the U.S., 2012-2015. Am J Prev Med. 2019;56(4):477-486. doi:10.1016/j.amepre.2018.11.020

12.

Hung

O’Halloran

, et al. Seasonal influenza vaccination coverage trends among adult populations, U.S., 2010-2016. Am J Prev Med. 2019;57(4):458-469. doi:10.1016/j.amepre.2019.04.007

13.

Kennedy

Ahluwalia

Ding

Singleton

Bridges

CB.

Monitoring seasonal influenza vaccination coverage among pregnant women in the United States. Am J Obstet Gynecol. 2012;207(3 suppl):S9-S16. doi:10.1016/j.ajog.2012.06.069

14.

Kahn

Black

Ding

, et al. Influenza and Tdap vaccination coverage among pregnant women—United States, April 2018. MMWR Morb Mortal Wkly Rep. 2018;67(38):1055-1059. doi:10.15585/mmwr.mm6738a3

15.

Razzaghi

Kahn

Black

, et al. Influenza and Tdap vaccination coverage among pregnant women—United States, April 2020. MMWR Morb Mortal Wkly Rep. 2020;69(39):1391-1397. doi:10.15585/mmwr.mm6939a2

16.

Santibanez

Williams

, et al. Surveillance of influenza vaccination coverage—United States, 2007-08 through 2011-12 influenza seasons. MMWR Surveill Summ. 2013;62(4):1-28.

17.

Tourangeau

Plewes

, eds. Nonresponse in Social Science Surveys: A Research Agenda. National Academies Press; 2013.

18.

Liu

Solano

, et al. Accuracy of influenza vaccination status in a computer-based immunization tracking system of a managed care organization. Vaccine. 2010;28(32):5254-5259. doi:10.1016/j.vaccine.2010.05.061

19.

Groom

Henninger

Smith

, et al. Influenza vaccination during pregnancy: influenza seasons 2002-2012, Vaccine Safety Datalink. Am J Prev Med. 2016;50(4):480-488. doi:10.1016/j.amepre.2015.08.017

20.

Hong

Lindley

Zhou

Coverage and timing of influenza vaccination among privately insured pregnant women in the United States, 2010-2018. Public Health Rep. Published June 23, 2021. doi:10.1177/00333549211026779

21.

Naleway

Gold

Kurosky

, et al. Identifying pregnancy episodes, outcomes, and mother–infant pairs in the Vaccine Safety Datalink. Vaccine. 2013;31(27):2898-2903. doi:10.1016/j.vaccine.2013.03.069

22.

Greene

Shi

Dutta-Linn

, et al. Accuracy of data on influenza vaccination status at four Vaccine Safety Datalink sites. Am J Prev Med. 2009;37(6):552-555. doi:10.1016/j.amepre.2009.08.022

23.

Baggs

Gee

Lewis

, et al. The Vaccine Safety Datalink: a model for monitoring immunization safety. Pediatrics. 2011;127(suppl 1):S45-S53. doi:10.1542/peds.2010-1722H

24.

McNeil

Gee

Weintraub

, et al. The Vaccine Safety Datalink: successes and challenges monitoring vaccine safety. Vaccine. 2014;32(42):5390-5398. doi:10.1016/j.vaccine.2014.07.073

25.

Sukumaran

McCarthy

, et al. Demographic characteristics of members of the Vaccine Safety Datalink (VSD): a comparison with the United States population. Vaccine. 2015;33(36):4446-4450. doi:10.1016/j.vaccine.2015.07.037

26.

Gabow

Eisert

Wright

Denver Health: a model for the integration of a public hospital and community health centers. Ann Intern Med. 2003;138(2):143-149. doi:10.7326/0003-4819-138-2-200301210-00016

27.

Eisert

Durfee

Welsh

Moore

Mehler

Gabow

PA.

Changes in insurance status and access to care in an integrated safety net healthcare system. J Community Health. 2009;34(2):122-128. doi:10.1007/s10900-008-9136-2

28.

Mullooly

Drew

DeStefano

, et al. Quality of HMO vaccination databases used to monitor childhood vaccine safety. Vaccine Safety Datalink Team. Am J Epidemiol. 1999;149(2):186-194. doi:10.1093/oxfordjournals.aje.a009785

29.

McCarthy

Gee

Weintraub

, et al. Monitoring vaccine safety using the Vaccine Safety Datalink: utilizing immunization registries for pandemic influenza. Vaccine. 2011;29(31):4891-4896. doi:10.1016/j.vaccine.2011.05.003

30.

Naleway

Crane

Irving

, et al. Vaccine Safety Datalink infrastructure enhancements for evaluating the safety of maternal vaccination. Ther Adv Drug Saf. 2021;12:20420986211021233. doi:10.1177/20420986211021233

31.

American Medical Association. International Classification of Diseases, 10th Revision, Clinical Modification. American Medical Association Press; 2018.

32.

Robbins

Boulet

Morgan

, et al. Disparities in preconception health indicators—Behavioral Risk Factor Surveillance System, 2013-2015, and Pregnancy Risk Assessment Monitoring System, 2013-2014. MMWR Surveill Summ. 2018;67(1):1-16. doi:10.15585/mmwr.ss6701a1

33.

Hill

Elam-Evans

Yankey

Singleton

Dietz

Vaccination coverage among children aged 19-35 months—United States, 2015. MMWR Morb Mortal Wkly Rep. 2016;65(39):1065-1071. doi:10.15585/mmwr.mm6539a4

34.

Hill

Elam-Evans

Yankey

Singleton

Kang

Vaccination coverage among children aged 19-35 months—United States, 2016. MMWR Morb Mortal Wkly Rep. 2017;66(43):1171-1177. doi:10.15585/mmwr.mm6643a3

35.

Parsons

Moriarity

Jonas

Moore

Davis

Tompkins

Design and estimation for the National Health Interview Survey, 2006-2015. Vital Health Stat 2. 2014;(165):1-53.

36.

Williams

O’Halloran

, et al. Surveillance of vaccination coverage among adult populations—United States, 2015. MMWR Surveill Summ. 2017;66(11):1-28. doi:10.15585/mmwr.ss6611a1

37.

Centers for Disease Control and Prevention. National early-season flu vaccination coverage, United States, November 2017. 2017. Accessed May 17, 2018. https://www.cdc.gov/flu/fluvaxview/nifs-estimates-nov2017.htm

38.

Centers for Disease Control and Prevention. National early-season flu vaccination coverage, United States, November 2016. 2016. Accessed May 17, 2018. https://www.cdc.gov/flu/fluvaxview/nifs-estimates-nov2016.htm

39.

Harris

Taylor

Minor

, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208. doi:10.1016/j.jbi.2019.103208

40.

Korn

Graubard

BI.

Confidence intervals for proportions with small expected number of positive counts estimated from survey data. Surv Methodol. 1998;24:193-201.

41.

Qiu

Statistical inference of the relative concentration index for complex surveys. Stat Med. 2019;38(21):4083-4095. doi:10.1002/sim.8284

42.

Grohskopf

Sokolow

Broder

Walter

Fry

Jernigan

DB.

Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2018-19 influenza season. MMWR Recomm Rep. 2018;67(3):1-20. doi:10.15585/mmwr.rr6703a1

43.

Specker

Wey

Fuller

, et al. 2009 H1N1 and seasonal influenza immunization among pregnant women: a comparison of different sources of immunization information. Matern Child Health J. 2014;18(3):681-687. doi:10.1007/s10995-013-1293-y

44.

Regan

Mak

Moore

, et al. Surveillance of antenatal influenza vaccination: validity of current systems and recommendations for improvement. BMC Public Health. 2015;15:1155. doi:10.1186/s12889-015-2234-z

45.

Rolnick

Parker

Nordin

, et al. Self-report compared to electronic medical record across eight adult vaccines: do results vary by demographic factors? Vaccine. 2013;31(37):3928-3935. doi:10.1016/j.vaccine.2013.06.041

46.

Irving

Donahue

Shay

Ellis-Coyle

Belongia

EA.

Evaluation of self-reported and registry-based influenza vaccination status in a Wisconsin cohort. Vaccine. 2009;27(47):6546-6549. doi:10.1016/j.vaccine.2009.08.050

47.

King

McLean

Belongia

EA.

Validation of self-reported influenza vaccination in the current and prior season. Influenza Other Respir Viruses. 2018;12(6):808-813. doi:10.1111/irv.12593

48.

American Immunization Registry Association. Landscape analysis: IIS adult vaccination data capture and data utilization. May 2021. Accessed July 19, 2021. https://repository.immregistries.org/files/resources/60958446cdad0/adultlandscape_final.pdf

49.

Dudley

Limaye

Salmon

, et al. Racial/ethnic disparities in maternal vaccine knowledge, attitudes, and intentions. Public Health Rep. 2021;136(6):699-709. doi:10.1177/0033354920974660

Influenza Vaccination Among Pregnant Women: Self-report Compared With Vaccination Data From Electronic Health Records,2018-2020 Influenza Seasons

Abstract

Objectives:

Methods:

Results:

Conclusions:

Keywords

Methods

Study Setting

Study Overview

Study Population: Identification of Pregnant Women

Survey Design and Administration

Sampling procedures

Survey instrument

Survey administration

Analytic Methods

Results

Survey Sample

Survey Response

Net Implied Bias in Coverage Estimates

EHR Vaccination Data Compared With Self-Report

Additional Vaccine-Related Information

Discussion

Limitations

Conclusions

Footnotes

Acknowledgements

Authors’ Note

Disclaimer

Declaration of Conflicting Interests

Funding

ORCID iDs

References