Abstract
Objectives:
Identifying the best and worst states for coverage of cancer-preventing vaccines (hepatitis B [HepB] and human papillomavirus [HPV]) may guide public health officials in developing programs, such as promotion campaigns. However, acknowledging the imprecision of coverage and ranks is important for avoiding overinterpretation. The objective of this study was to examine states’ vaccination coverage and ranks, as well as the imprecision of these estimates, to inform public health decision making.
Methods:
We used data on coverage of HepB and HPV vaccines among adolescents aged 13-17 from the 2011-2015 National Immunization Survey-Teen (n = 103 729 from 50 US states and Washington, DC). We calculated coverage, 95% confidence intervals (CIs), and ranks for vaccination coverage in each state, and we generated simultaneous 95% CIs for ranks using a Monte Carlo method with 100 000 simulations.
Results:
Across years, HepB vaccination coverage was 92.2% (95% CI, 91.8%-92.5%; states’ range, 84.3% in West Virginia to 97.0% in Connecticut). HPV vaccination coverage was 57.4% (95% CI, 56.6%-58.2%; range, 41.8% in Kansas to 78.0% in Rhode Island) for girls and 31.0% (95% CI, 30.3%-31.8%; range, 19.0% in Utah to 59.3% in Rhode Island) for boys. States with the highest and lowest ranks generally had narrow 95% CIs; for example, Rhode Island was ranked first (95% CI, 1-1) and Kansas was ranked 51st (95% CI, 49-51) for girls’ HPV vaccination. However, states with intermediate ranks had wider and more imprecise 95% CIs; for example, New York was 26th for girls’ HPV vaccination coverage, but its 95% CI included ranks 18-35.
Conclusions:
States’ ranks of coverage of cancer-preventing vaccines were imprecise, especially for states in the middle of the range; thus, performance rankings presented without measures of imprecision could be overinterpreted. However, ranks can highlight high-performing and low-performing states to target for further research and vaccination promotion programming.
Keywords
The Advisory Committee on Immunization Practices recommends that children receive 2 vaccines that prevent cancer as a part of routine care: hepatitis B (HepB) and human papillomavirus (HPV) vaccines. 1 HepB vaccine, administered to infants, 1 protects against infection with HepB, a virus that causes up to 50% of hepatocellular carcinoma in the United States. 2 HPV vaccine, administered to adolescents, 1 protects against infection with high-risk strains of HPV, which cause about 25 000 cases of anogenital cancers each year. 3
As of 2015, coverage of 3 doses of HepB vaccine reached 93% for children aged <3 years 4 and 91% for adolescents aged 13-17. 5 In the same year, initiation of the HPV vaccine series among adolescents aged 13-17 was only 63% for adolescent girls and 50% for adolescent boys. 5 For HepB and HPV vaccines, coverage was relatively equal across demographic subgroups; however, large differences existed by geography. HepB vaccination among infants ranged from 87% (Virginia) to 97% (Connecticut) in 2015. 4 HPV vaccination initiation among girls ranged from 48% (Wyoming) to 88% (Rhode Island), and among boys from 35% (Kentucky) to 81% (Rhode Island). 5
Ranking states on their performance by public health indicators such as vaccination coverage is an intuitive way to focus attention on modifiable health outcomes and motivate action. 6,7 Programs such as the County Health Rankings 8 and America’s Health Rankings 9 have generated interest from policy and public stakeholders, 6,7,10 especially for geographic areas with very high or very low ranks. However, ranks are often presented as error-free, thereby ignoring the imprecision involved in their generation or in the calculation of the underlying health indicators. These statistics are often derived from surveys of population-based samples, which can introduce imprecision through a combination of many potential sources of error, including sampling, self-report, and other random sources of error variance. 11,12 Ignoring these sources of imprecision can result in overinterpretation of performance rankings and, perhaps, misallocation of limited resources. The relative importance of the imprecision in ranks has been debated in the scientific literature, 13 –18 but few studies have empirically estimated imprecision around ranks or examined its implications for public health messaging. 7,14 –16,19
This study had 2 objectives: (1) rank states on coverage of cancer-preventing vaccines among adolescents in 2011-2015, including subgroups by race/ethnicity and poverty level; and (2) examine the underlying imprecision in calculating the ranks. The findings from this study can inform public health messaging and interventions, especially when comparing ranks for coverage of these vaccines across states.
Methods
Data Source
National Immunization Survey (NIS)-Teen is an annual population-based survey with 2 stages of data collection. 20 In the first stage, NIS-Teen staff members administer a telephone survey using random-digit dialing (including landline and cellular telephone numbers) to parents of adolescents aged 13-17, gathering information on the vaccination history of the index child. In the second stage, NIS-Teen staff members mail questionnaires to health care providers of the index child to verify their vaccination history with medical records. Each year, NIS-Teen collects provider-verified vaccination data on approximately 20 000 adolescents and develops sampling weights to account for non-equal probability of selection into the complex survey. Additional details on the design and administration of NIS-Teen are available elsewhere. 20 We used data from NIS-Teen survey years 2011-2015 for all 50 US states and Washington, DC (hereinafter referred to collectively as states). This analysis did not require institutional review board approval because it involved secondary analysis of publicly available, de-identified data.
Measures
We developed 5-year estimates of state coverage for 2 cancer-preventing vaccines. HepB vaccination coverage estimates reflect the proportion of adolescents with provider-verified receipt of all 3 doses of HepB vaccine. HPV vaccination coverage estimates reflect the proportion of adolescents with provider-verified receipt of at least 1 dose of HPV vaccine (HPV vaccine began as a 3-dose regimen 1 and recently changed to a 2-dose regimen for young adolescents, 21 given that receiving 2 doses provides substantial protection against HPV infection 22 ). Because the Advisory Committee on Immunization Practices adopted recommendations for routine HPV vaccination for females in 2007 and males in 2011, 23,24 we examined HPV vaccination coverage separately by adolescent sex. In addition, we generated estimates of state coverage for these 3 vaccine outcomes by year, race/ethnicity, and poverty level. We calculated coverage proportions as percentages of adolescents aged 13-17 who were vaccinated.
In the NIS-Teen telephone survey, 20 parents reported adolescent sex (female or male), race/ethnicity, and household characteristics. Although NIS-Teen collects detailed data on race/ethnicity, we categorized this variable as non-Hispanic white or racial/ethnic minority because small cell sizes precluded more precise analyses. Poverty level was based on parent-reported annual household income and household size (below federal poverty level; above federal poverty level and household income ≤$75 000; or above federal poverty level and household income >$75 000). Participants’ state of residence was used in the sampling scheme and confirmed with parents during the NIS-Teen telephone survey.
Statistical Analysis
We generated unweighted frequencies and weighted percentages of adolescent characteristics by year and across the 5 survey years. Then, using the 5-year estimates, we generated choropleths 25 to depict state coverage for each vaccine and created scatterplots to reflect state vaccination coverage and 95% confidence intervals (CIs). We evaluated rankability (ie, the portion of variance due to true difference) of states’ vaccination coverage using I 2 calculated according to the Pearson χ2 statistic. 16 I2 indicates the proportion of the true (ie, non-error) between-state variability over the total observed variation. It typically varies from 0 to 1, with higher values representing greater between-state variability. I2 values >0.75 generally reflect high true variability, low error variability, and adequate rankability.
To calculate ranks and 95% CIs, we implemented a simulation analysis, 15 iteratively estimating state vaccination coverage given the NIS-Teen parameters, ranking the states in each simulation, and using the distribution of simulated ranks to calculate 95% CIs, separately for each vaccination outcome. We assumed that the vaccination outcome for each state was a realization of a random process with a truncated normal distribution, bounded between 0 and 1, with the mean and standard error equal to the estimates from NIS-Teen. A simulated value outside of the boundary of (0,1) would be rejected and another normal variate would be redrawn; however, when we used the truncated normal distribution, none of the originally simulated estimates fell outside of the boundary. We used a Monte Carlo method with 100 000 replications to simulate the proportion of adolescents vaccinated in each state given the observed mean and standard error. Then, we ranked the states in each simulation. Based on the 100 000 ranks generated, we calculated the 95% CIs for each state’s rank for coverage. We used the individual 95% CI to examine the state’s uncertainty measure. After we determined the ranks and 95% CIs for all the states, we used the simultaneous 95% CI for comparison across all the states. 15 As a test of the statistical differences among states’ ranks, we examined whether the 95% CIs for ranks overlapped; although this test is extremely conservative, 26 it is a straightforward approach to making multiple comparisons. Finally, we created scatterplots to reflect the rank and 95% CI for each state. Preliminary analyses found that assuming the NIS-Teen coverage estimates were derived from normal, truncated normal, binomial, or lognormal distributions resulted in equivalent ranks and 95% CIs (data not shown).
As a sensitivity analysis, we repeated these analyses and calculated states’ vaccination coverage, ranks, and 95% CIs by year, race/ethnicity, and poverty level. To examine the consistency of the ranks, we calculated Spearman rank correlation coefficients for the pairwise comparisons of each state’s rank produced in the main analysis and the ranks produced in each sensitivity analysis. We also calculated Spearman rank correlation coefficients for the pairwise comparisons of the ranks across states for each of the 3 outcomes (eg, correlation of states’ ranks for HepB and HPV vaccination among girls). Spearman rank correlation coefficients range from –1 (perfect disagreement) to 1 (perfect agreement) and indicate the degree of agreement among the sets of ranks. 27 For the comparison of the ranks produced in the main analysis with the ranks produced in the sensitivity analysis, these Spearman correlations were expected to be high, because the data for each set of ranks in the sensitivity analysis composed the full data for the ranks in the main analysis.
Estimates accounted for the complex survey design of NIS-Teen and incorporated sampling weights to account for non-equal probability of selection into the survey. We conducted analyses using SAS version 9.3. 28 We used 95% CIs to assess the imprecision in the point estimates, with α = .05 considered significant.
Results
Adolescents surveyed (n = 103 729) were nearly evenly split by sex (48.8% female, 51.2% male), were mostly non-Hispanic white (55.0%), and predominately lived in households at or above the federal poverty level (71.8%).
HepB Vaccination Among All Adolescents
National HepB vaccination coverage for 2011-2015 was 92.2% (95% CI, 91.8%-92.5%) (Table). The I2 statistic for between-state HepB vaccination variability was 0.94, indicating adequate rankability. State HepB vaccination coverage ranged from 84.3% in West Virginia to 97.1% in Georgia (Figures 1A, 2). Thus, across states, Georgia was ranked first for HepB vaccination, and its 95% CI overlapped with 4 other ranks (95% CI, 1-5), whereas West Virginia was ranked 51st (95% CI, 49-51) (Table, Figure 2). Maryland had the middle rank (26th), with a 95% CI overlapping with 23 other ranks (95% CI, 11-34). These wide 95% CIs indicated large imprecision in the ranks; for example, the 95% CI for Maryland’s rank overlapped with the 95% CIs for ranks of 36 other states (ie, only 14 states had a rank with a 95% CI that excluded Maryland’s 95% CI), indicating that their ranks were not necessarily statistically different.
Coverage and ranks for vaccination coverage among adolescents aged 13-17, by state, National Immunization Survey-Teen, 2011-2015a
Abbreviations: HepB, hepatitis B; HPV, human papillomavirus; NA, not applicable.
aCoverage estimates incorporate survey weights from National Immunization Survey-Teen. 20 Ranks and their 95% CIs were calculated using a simulation analysis, which generated 100 000 sets of simultaneous ranks and examined the distribution of the ranks across iterations.
bProvider-verified receipt of 3 doses of HepB vaccine.
cProvider-verified receipt of at least 1 dose of HPV vaccine.
State coverage among adolescents aged 13-17 for (A) hepatitis B (HepB) vaccination, (B) human papillomavirus (HPV) vaccination among girls, and (C) HPV vaccination among boys, National Immunization Survey-Teen, 20 2011-2015.
State hepatitis B vaccination coverage and ranks among adolescents aged 13-17, National Immunization Survey-Teen, 20 2011-2015. Error bars are 95% confidence intervals.
HPV Vaccination Among Girls
National HPV vaccination coverage among girls for 2011-2015 was 57.4% (95% CI, 56.6%-58.2%) (Table). The I2 statistic for between-state girls’ HPV vaccination variability was equal to 0.90, indicating adequate rankability. State HPV vaccination coverage ranged from 41.8% in Kansas to 78.0% in Rhode Island (Figures 1B, 3). Thus, Rhode Island was ranked first for girls’ HPV vaccination, and its 95% CI did not overlap with other states (95% CI, 1-1), whereas Kansas was ranked 51st (95% CI, 49-51) (Table, Figure 3). New York had the middle rank (26th), with a 95% CI overlapping with 17 other ranks (95% CI, 18-35). The 95% CI for New York’s rank overlapped with the 95% CIs for ranks of 41 other states, indicating that their ranks were not necessarily statistically different.
State human papillomavirus (HPV) vaccination coverage and ranks among adolescent girls aged 13-17, National Immunization Survey-Teen, 20 2011-2015. Error bars are 95% confidence intervals.
HPV Vaccination Among Boys
National HPV vaccination coverage among boys for 2011-2015 was 31.0% (95% CI, 30.3%-31.8%) (Table). The I2 statistic for between-state boys’ HPV vaccination variability was 0.94, indicating adequate rankability. State HPV vaccination coverage ranged from 19.0% in Utah to 59.3% in Rhode Island (Figures 1C, 4). Thus, Rhode Island was ranked first for boys’ HPV vaccination across states, and its 95% CI overlapped with no other states (95% CI, 1-1), whereas Utah was ranked 51st (95% CI, 44-51) (Table, Figure 4). Oregon had the middle rank (26th), with a 95% CI overlapping with 19 other ranks (95% CI, 16-35). The 95% CI for Oregon’s rank overlapped with the 95% CIs for ranks of 32 other states, indicating that their ranks were not necessarily statistically different.
State human papillomavirus (HPV) vaccination coverage and ranks among adolescent boys aged 13-17, National Immunization Survey-Teen, 20 2011-2015. Error bars are 95% confidence intervals.
States’ ranks for HPV vaccination among boys were highly correlated with states’ ranks for vaccination among girls (Spearman rank correlation coefficient = 0.83), whereas states’ ranks for HPV vaccination among either sex and states’ ranks for HepB vaccination were not highly correlated (Spearman rank correlation coefficient for HepB and HPV vaccination among girls = 0.24; among boys = 0.26).
Sensitivity Analyses
When ranking states for vaccination coverage separately by year (5 groups; data available from author), race/ethnicity (2 groups; data available from author), and poverty level (3 groups; data available from author), the results of the rankings were similar overall for each subset and were fairly consistent with the findings from the main analysis. Spearman rank correlation coefficients that compared the ranks produced in the main analysis and the ranks produced in each sensitivity analysis ranged from 0.40 to 0.95, and all coefficients were significant. Generally, the sensitivity analyses that leveraged the smaller subgroups (ie, racial/ethnic minority [45% of the sample] and adolescents living in households below the federal poverty level [23% of the sample]) had the widest 95% CIs, as expected, and the lowest correlation coefficients. However, the lowest correlation coefficient, 0.40, was found for the comparison of the state ranks of HPV vaccination coverage among boys in 2011 and the ranks for that outcome in the main analysis.
Discussion
When we compared coverage of HepB and HPV vaccination during 5 years among more than 100 000 adolescents in 51 states, considerable variability in states’ ranks emerged, especially among those states in the middle of the observed ranges. The 95% CIs of many of these intermediate states overlapped with the ranks of numerous other states. For example, Maryland was ranked 25th for HepB vaccination coverage, but its 95% CI ranged from 11 to 34 and overlapped with the 95% CIs for the ranks of 36 other states. However, for states at the extremes of each outcome, clear differences emerged. For example, Rhode Island ranked first (95% CI, 1-1) for both girls’ and boys’ HPV vaccination coverage, indicating a distinct difference in this state’s rank compared with other states for these outcomes. Thus, public health messaging and resource allocation based on states’ ranks for vaccination coverage may be valid for states only at the extreme ends (very high or very low) of the distribution. Descriptively, the statistical differences in states’ ranks at the extremes of the observed range were not attributable to differences in sample sizes across states, which were fairly similar.
To our knowledge, only one other study examined the precision of states’ ranks of vaccination coverage, analyzing coverage of a series of early childhood immunizations. 19 That study, using 1 year of National Immunization Survey data, found that 90% CIs (which, statistically, are more narrow than 95% CIs) for states’ ranks overlapped considerably. The authors’ analysis constructed 90% CIs using a similar parametric simulation procedure to the one used in our study, and they compared all states with a reference state (Georgia), finding that only 3 states were significantly different from the reference. Our study expands on this emerging evidence base by using data from several years of NIS-Teen surveys and evaluating multiple vaccination outcomes related to cancer prevention. Taken together, the implication from both studies is clear: ranks are imprecise and should be accompanied by uncertainty estimates (eg, 95% CIs).
As expected, the 5-year estimates of HepB coverage differed substantially from the 5-year estimates of HPV coverage: 92% of all adolescents were vaccinated against HepB, whereas 57% of girls and 31% of boys were vaccinated against HPV. At least 2 processes could underpin these differences. First, missed opportunities for HPV vaccination are relatively common 29,30 and vary across geography. 31 Adolescents could miss more opportunities for HPV vaccination during their health care visits than infants miss for HepB vaccination during their health care visits. A reduction in the number of missed opportunities could substantially improve HPV vaccination coverage 32 and narrow the gap across states between HPV vaccination coverage and HepB vaccination coverage. Second, because the Advisory Committee on Immunization Practices recommendations for routine HPV vaccination are newer than the recommendations for routine HepB vaccination, 1 state patterns in HPV vaccination coverage could change and differences could diminish as HPV vaccination becomes more normalized. 33 The Advisory Committee on Immunization Practices’ adoption of recommendations for HPV vaccination for boys 4 years after its adoption of recommendations for girls may also partially explain the difference we observed in HPV vaccination between girls and boys. In addition, HPV vaccination varied across states: the difference in coverage among girls between the first-ranked state and the last-ranked state was almost twofold (states’ range, 42%-78%) and among boys was almost threefold (states’ range, 19%-59%). Ongoing surveillance of states’ vaccination coverage and ranks can provide more insight into processes underpinning differences in coverage between vaccines and variation in coverage across states.
Comparing ranks between vaccines, top-ranking states in HepB vaccination coverage (for infants) did not necessarily translate into high ranks in HPV vaccination coverage (for adolescent boys or girls), as indicated by the relatively low Spearman rank correlation coefficients for these comparisons (both <0.30). For example, Georgia ranked first in HepB vaccination, but it ranked 30th and 19th for girls’ and boys’ HPV vaccination initiation, respectively. Conversely, Rhode Island ranked first for both girls’ and boys’ HPV vaccination coverage yet ranked eighth in HepB vaccination coverage. Although states’ vaccination coverage ranks were dissimilar between vaccines, the results of our sensitivity analyses demonstrated considerable consistency in ranks within vaccines by year, race/ethnicity, and poverty level. The correlation coefficients for ranks in subgroups and overall ranks in the main analysis were all above 0.60, with 1 exception (HPV vaccination among boys in 2011, coverage for which was low; Spearman rank correlation coefficient = 0.40). Therefore, state ranks were dissimilar between vaccination outcomes, but they were relatively similar across population subgroups. This latter pattern implies that the processes that give rise to coverage rates and ranks across states operate relatively similarly across subgroups.
Ranking states may be a practical tool to guide public health messaging 6,7,10 ; however, our study highlights potential problems with overinterpretation of these data given the substantial imprecision in ranks. Creating state-level estimates using 5 years of data allowed us to use generous sample sizes. In contrast, many decisions for public health programming are made at sub-state or county levels 7 ; to the extent that sub-state or county ranks use survey-based estimates, they may have correspondingly smaller sample sizes and generally wider 95% CIs than in our study. Thus, it will be even more challenging to precisely rank health outcomes in these smaller geographic areas than it was in our study, which used state data. Although we were able to demonstrate precision in ranking states at extreme ends of the vaccination coverage range, our findings underscore the need to acknowledge the imprecision and variability in interpreting ranks. Supporters of geographic health rankings suggest that concerns about the reliability of ranks hinder public health surveillance and dissemination efforts, 13 whereas critics argue that the appearance of certainty in ranks obscures the imprecision inherent in their creation. 34 Conceptually, the imprecision in these ranks can come from many sources. Empirically, we have shown that, except at the extremes, ranks of state vaccination coverage may not be distinguishable statistically.
Strengths and Limitations
This study had several strengths. First, we used a large, high-quality national data set with provider verification of vaccination status. Second, we used several years of data to evaluate stable trends in state coverage and ranks for 2 vaccination outcomes. Third, we estimated 95% CIs using rigorous methodology. Additionally, our results were robust to sensitivity analyses in that we were able to identify similar patterns for the overall ranks in the main analysis compared with the ranks in demographic subgroups.
This study also had several limitations. First, the study included data that reflected the Advisory Committee on Immunization Practices recommendation for routine HPV vaccination for boys, 24 which is a more recent recommendation than that for girls, 23 which could make these outcomes seem noncomparable. However, coverage rates for HPV vaccination among girls and boys are converging. 5 Future research should continue to monitor HPV vaccination coverage among both girls and boys in the United States. Analysis of vaccination coverage at finer geographic levels (eg, at the county or city level) or more detailed subgroups (eg, various racial/ethnic groups) could be more relevant to public health decision making, but data scarcity precluded this approach. In addition, the use of overlapping 95% CIs to evaluate significant differences among states’ ranks is, as noted previously, an extremely conservative approach; alternate tests of statistical differences could have found different results, especially for pairs of states whose 95% CIs overlapped only slightly. Finally, the coverage estimates analyzed in our study were averages during a 5-year period; coverage and ranks could have changed during this period, but analyzing year-specific coverage would have resulted in smaller sample sizes and correspondingly larger standard errors and wider 95% CIs.
Conclusions
Ranking jurisdictions according to HepB or HPV vaccination coverage can be a valuable tool in identifying states that are achieving high rates of coverage of cancer-preventing vaccines and states that need more support for vaccination systems and promotion. However, wide 95% CIs for the vaccination coverage ranks resulted in substantial overlap and imprecision. Creating an understanding of the extent of imprecision and the true differences in ranks for health behaviors and outcomes among state populations—and especially populations at smaller geographic units or subpopulations—will require nuanced communication with various stakeholders in public health and policymaking. However, lessons learned from states at the extremes of vaccination coverage could inform public health messaging and decision making in other states to promote HepB and HPV vaccination and, ultimately, reduce the burden of cancer among adolescents throughout their lifetime.
Footnotes
Acknowledgments
This research was conducted as part of active duty at the National Cancer Institute. This article was prepared or accomplished by the authors in their personal capacity. The opinions expressed in this article are the authors’ own and do not reflect the view of the National Institutes of Health, the US Department of Health and Human Services, or the US government.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.