Abstract
Objectives
Coordinated measurement strategies are needed to inform collaborative approaches to improve access to and quality of care for persons with sickle cell disease (SCD). The objective of our study was to develop a multilevel measurement strategy to assess improvements in access to and quality of care for persons with SCD in 4 US regions.
Methods
From 2014 through 2017, regional grantees in the Sickle Cell Disease Treatment Demonstration Program collected administrative and patient-level electronic health record (EHR) data to assess quality improvement initiatives. Four grantees—covering 29 US states and territories and an SCD population of 56 720—used a collective impact model to organize their work. The grantees collected administrative data from state Medicaid and Medicaid managed care organizations (MCOs) at multiple points during 2014-2017 to assess improvements at the population level, and local patient-level data were abstracted from site-level EHRs at regular intervals to track improvements over time.
Results
Administrative data were an important source of understanding population-level improvements but were delayed, whereas patient-level data were more sensitive to small-scale quality improvements.
Conclusions
We established a shared measurement approach in partnership with Medicaid and Medicaid MCO stakeholders that can be leveraged to effectively support quality improvement initiatives for persons with SCD in the United States.
Sickle cell disease (SCD) affects approximately 100 000 persons in the United States, leads to more than 83 000 hospitalizations, costs $488 million annually, and is the most commonly detected genetic disorder in the nation. 1 -3 SCD is a group of disorders, in which red blood cells become sickle shaped and lead to acute and chronic clinical complications. 4 Since medical advances in the 1980s, SCD childhood mortality has decreased as much as 68% (from 2.42 per 100 000 in 1999-2002 to <0.78 per 100 000 in 1983-1986 among children aged 0-3 y), 5,6 but persons with SCD have limited access to comprehensive care, compared with access for other genetic disorders, such as cystic fibrosis. 7
Access to high-quality health care is critical to prevent complications and early SCD-related mortality, yet many persons with SCD are unable to obtain quality care. For example, hydroxyurea, the only drug approved by the US Food and Drug Administration to treat SCD until the approval of L-glutamine in 2017 and the approvals of Adakveo and Oxbryta in November 2019, 8 is often underprescribed because of concerns among health care providers and patients about potential toxicity. 9 Another example of limited access to high-quality health care is the inadequate number of knowledgeable health care providers; patients with SCD often seek specialty care in emergency departments, which is costly, or from primary care providers who may not have experience treating SCD.
The SCD Treatment Demonstration Program (SCDTDP), funded by the Maternal and Child Health Bureau of the Health Resources and Services Administration, has 3-year funding cycles. 10 -12 In each previous funding cycle, grantees developed measures to assess improvements in quality of care for persons with SCD. During the 2014-2017 funding cycle, grantees and a group of experts developed and piloted a measurement strategy that captured data and described progress in improving clinical performance at the regional, state, and practice levels. One objective of the 2014-2017 funding cycle was to develop a multilevel measurement strategy to assess improvements in access to and quality of care for persons with SCD in 4 US regions. This article describes the work of the national coordinating center (NCC), the National Institute for Children’s Health Quality, and the regional coordinating centers (RCCs), as well as lessons learned and recommendations.
Methods: Components of the Measurement Strategy
History of the SCDTDP
In response to disparities in access to effective SCD treatment strategies, Congress authorized a demonstration program in the Sickle Cell Treatment Act of 2003. 13 -15 The goals of the program are to (1) improve care coordination and service delivery for persons living with SCD, (2) improve access to services, and (3) improve and expand patient and health provider education.
For the 2014-2017 funding cycle, the Health Resources and Services Administration established a regional collaborative model that organized efforts across regional networks to reach a broad portion of the SCD population and coordinate improvement efforts. The Health Resources and Services Administration selected 4 grantees to become RCCs to create networks across states that covered 6 of the 10 Health Resources and Services Administration regions, with the potential to affect an estimated 56 720 persons with SCD. Each regional network was led by an RCC using a hub-and-spoke model. 16 The networks comprised academic medical centers, state Medicaid offices, Medicaid managed care organizations (MCOs), federally qualified health centers, and community-based organizations. The 4 RCCs are Sickle Cell Improvement Across the Northeast Region Through Education, in Baltimore, Maryland; Sickle Treatment and Outcomes Research in the Midwest, in Cincinnati, Ohio; the Pacific Sickle Cell Regional Collaborative, in Oakland, California; and the Heartland Sickle Cell Disease Treatment Network, in St Louis, Missouri (Figure).

Twenty-nine states and territories included in the 2014-2017 Sickle Cell Disease Treatment Demonstration Program in green with the 4 Regional Coordinating Centers and the National Coordinating Center in Boston. The 4 regions are represented in green with the following regional coordinating centers: Sickle Cell Improvement Across the Northeast Region Through Education, in Baltimore, Maryland; Sickle Treatment and Outcomes Research in the Midwest, in Cincinnati, Ohio; the Pacific Sickle Cell Regional Collaborative, in Oakland, California; and the Heartland Sickle Cell Disease Treatment Network, in St Louis, Missouri.
Each RCC created a regional learning collaborative focused on 3 high-leverage, interrelated aims to maximize collaboration within and across regions. These aims are to (1) increase the number of health care providers treating patients with SCD, (2) increase the number of health care providers prescribing hydroxyurea, and (3) increase the number of patients obtaining care from health care providers knowledgeable about SCD. The NCC provided infrastructure to operationalize a collective impact model, support the RCCs’ collaborative networks, and serve as a data hub.
The SCDTDP regional model used the collective impact model—an approach to facilitating coordination and collaborative work that includes the following key components for successful implementation: a common agenda, a shared measurement system, mutually reinforcing activities, continuous communication, and a backbone support organization. 17 The work of the Health Resources and Services Administration aligned with the work of other federal agencies, such as the 2014 expert panel report of the National Heart, Lung, and Blood Institute, on evidence-based treatment and management of SCD, which highlighted underused disease-modifying treatments. 6 The expert panel report included strong evidence and recommendations for appropriate treatment guidelines for using hydroxyurea. 6 Hydroxyurea, approved by the US Food and Drug Administration for use by adults and children with SCD, reduces the frequency of acute chest syndrome and painful episodes and can reduce mortality. 18,19
To demonstrate progress toward SCDTDP aims at the local, state, and regional levels, the program needed a shared measurement strategy. The quality improvement (QI) metrics for previous funding cycles of the SCDTDP were based on small (~20 patients per month) samples designed to inform local QI initiatives, whereas the aims of the 2014-2017 funding cycle needed metrics to assess improvements at a regional population level. In November 2014, representatives from the 4 RCCs and the NCC, federal partners, and other experts convened a meeting, a data summit, to develop a shared measurement strategy. They prioritized building a framework that would use data to demonstrate the effect of the SCDTDP on the number of persons with SCD and progress toward achieving the 3 project aims. The data summit attendees considered various data sources, including local registries, electronic health records (EHRs), and administrative data. In considering patient privacy, technical complexity, population size, and project resources, experts recommended collaboration with state Medicaid offices and Medicaid MCOs, because this approach was most likely to support project aims 1 and 2. For project aim 3, experts recommended a separate qualitative evaluation strategy to understand regional efforts in health care provider education. 11 This study describes the measurement strategy for aims 1 and 2. This project was approved by the NCC’s institutional review board (IRB).
Administrative Data
An analysis of hospital claims data suggested that approximately two-thirds of persons with SCD are insured by Medicaid. 2 Therefore, state Medicaid claims data are population-based and comprehensive. State-level Medicaid data span multiple health care provider types and allow for analysis across pediatric and adult health care providers, including specialty and primary care. Some state Medicaid programs are managed by MCOs, so the strategy would need to include collaboration with both types of organizations; the data requests from the organizations are the same and hereinafter are referred to collectively as Medicaid/MCO. Although state-to-state differences exist, systems of common nomenclature and coding of Medicaid data exist to help standardize claims data across states and regions, reducing the need for a new standardization process in a national initiative. During the data summit, RCC teams and measurement experts suggested that state-level Medicaid claims data had the potential to monitor QI efforts by providing quarterly data that could not be obtained from national sources because of substantial lags in data access. Therefore, experts recommended that state Medicaid/MCOs report data quarterly.
The NCC and RCCs developed administrative data measures on the basis of project aims, literature review, evidence-based practice and guidelines for quality SCD care, previously developed metrics, expert opinion, and measurement strategies proposed by the RCCs. 20 -25 Content experts and grantees evaluated all measures through a modified Delphi technique, a structured process used to achieve expert consensus through multiple rounds of feedback (Table 1). 26
Administrative data measures included in a multilevel measurement strategy to assess improvements in access to and quality of care for persons with sickle cell disease in 4 US regions, Sickle Cell Disease Treatment Demonstration Program, funding cycle 2014-2017 a
Abbreviation: SCD, sickle cell disease.
aThe 4 regions are represented by the following regional coordinating centers: Sickle Cell Improvement Across the Northeast Region Through Education, in Baltimore, Maryland; Sickle Treatment and Outcomes Research in the Midwest, in Cincinnati, Ohio; the Pacific Sickle Cell Regional Collaborative, in Oakland, California; and the Heartland Sickle Cell Disease Treatment Network, in St. Louis, Missouri.
The expert group determined that MCOs might be incentivized to participate in this collaborative effort by the potential cost savings provided by improvements in the availability and quality of SCD care for their enrollees. The program used MCO data for comparison across MCOs and to inform descriptive uptake of patient care initiatives (eg, health care providers seeing patients ≥2 times annually) and treatments (eg, use of hydroxyurea) among the various data sources.
Patient-Level EHR Data
The second component of the measurement strategy was patient-level EHR data collected by each RCC at the level of health care provider and institution (Table 2). The RCCs designed the EHR database to complement the state- and regional-level measures. Whereas administrative data are available several months to years after being collected, EHR data provide immediate, real-time patient-level data and are more sensitive than administrative data in driving improvement efforts.
Measures from patient-level electronic health records included in a multilevel measurement strategy to assess improvements in access to and quality of care for persons with sickle cell disease in 4 US regions, Sickle Cell Disease Treatment Demonstration Program, funding cycle 2014-2017 a
Abbreviations: ED, emergency department; SCD, sickle cell disease.
aThe 4 regions are represented by the following regional coordinating centers: Sickle Cell Improvement Across the Northeast Region Through Education, in Baltimore, Maryland; Sickle Treatment and Outcomes Research in the Midwest, in Cincinnati, Ohio; the Pacific Sickle Cell Regional Collaborative, in Oakland, California; and the Heartland Sickle Cell Disease Treatment Network, in St. Louis, Missouri.
The multistep process of creating and using patient-level EHR data began with leveraging existing SCD resources, including the PhenX Toolkit (consensus measures for Phenotypes and eXposures), 27 and previous SCDTDP measures. Frequent conversations between the NCC and RCCs aligned measure definitions, data collection specifications, and reporting processes (eg, Medicaid cross-references, data dictionaries, and Research Electronic Data Capture). 28 RCCs shared lessons learned about state-level IRB and data use agreements. The process allowed RCCs to learn from each other’s successes and challenges, while ensuring data consistency across regions. The RCCs reviewed patient records, including EHR progress notes and local patient registries, after obtaining patient consent, to collect relevant data during the study period. All RCCs used a customized online data collection platform, the National Institute for Children’s Health Quality Collaboratory, to report, annotate, and compare reported measures on anonymous, aggregated data across regions.
Results
The 2014-2017 SCDTDP captured data on 8714 children and adults with SCD in the EHR data, and these data were available as soon as EHR reviews were completed. During the same period, data on 21 873 patients were captured in the SCDTDP administrative data. Using the administrative data and the EHR data together, RCCs were able to assess improvement on a population level and a more granular level in 2 critical domains: (1) access to care (ie, health care practices and health care providers seeing persons with SCD) and (2) provision of a hydroxyurea prescription.
Administrative Data
RCCs obtained baseline administrative data from 28 of the 29 US states and territories, as well as at least 1 quarterly data update from 7 of those states.
RCCs and states worked with state Medicaid/MCOs to establish standard algorithms and data requests to obtain the SCD measures from Medicaid administrative claims databases. The program used Medicaid/MCO data collection requests to monitor changes, document improvements in care, and provide data for state- and regional-level educational activities. Aggregate data obtained by the RCCs on health care provider encounters with patients with SCD and hydroxyurea prescriptions were stored in local databases, and RCCs submitted these data to the NCC quarterly for each state Medicaid/MCO in their region.
The RCCs recognized the need to build systems and infrastructure to manage data received from states. Staff member time and expertise were necessary for cleaning, analyzing, and compiling data from the state data sets before submitting them to the NCC. State data sets were large and complex, requiring extensive work using analytic applications in regions that received nonaggregated data. For nonaggregated data, personnel who had statistical programming experience or experience working with Medicaid data were required to manage and prepare the data for the NCC. RCCs collected Medicaid/MCO administrative claims data in most states in their regions (Table 3). When statewide data could not be obtained from either source, the inability was often due to resource or funding issues.
Administrative data collection details including source, time range, and barriers to data collection for the Sickle Cell Disease Treatment Demonstration Program, funding cycle 2014-2017
Abbreviations: —, none noted; FFS, fee for service; MCO, managed care organization; RCC, regional coordinating center.
aThe Iowa and Missouri data from the Illinois Medicaid office.
bIncludes data from patients living in Indiana, Iowa, Missouri, and Wisconsin, but receiving care from providers in Illinois.
cVirginia MCO data were received but suppressed due to low numbers and differences in measure definitions.
Patient-Level EHR Data
We obtained patient-level EHR data from 26 sites and ≥1 quarterly data update from 16 of those sites, allowing the RCCs to use these baseline data for state team learning, QI, and immediate assessment of improvements in project aims 1 and 2.
Although RCCs aimed to create common measures, they were unable to define and collect data uniformly for some measures. The lack of common measures and uniform data collection caused difficulties in creating and comparing data across regions. In addition, the RCCs developed the approach for using patient-level EHR data after they had designed the initial, administrative data-focused project measurement strategy. By then, resources were limited, and time was limited for data collection, entry, and standardization of the final submitted aggregated patient-level EHR data. The RCCs experienced setbacks while obtaining data use agreements and IRB approvals (eg, a single IRB process covering multiple entities and a local IRB process covering 1 entity have different timelines), but eventually, the RCCs were able to obtain consent and enroll patients without offering incentives (ie, the Pacific RCC obtained consent from 407 of 417 families approached for participation) and gather data on most or all network-wide EHR measures. In the Heartland, all 4 states submitted EHR data: Iowa, Kansas, Missouri, and Nebraska. In the Midwest region, 5 states submitted EHR data: Illinois, Indiana, Michigan, Minnesota, and Ohio. In the Northeast region, 8 states submitted EHR data: Delaware, District of Columbia, Maryland, New Jersey, New York, Pennsylvania, Virginia, and West Virginia. In the Pacific region, 7 states submitted EHR data: Alaska, Arizona, California, Idaho, Nevada, Oregon, and Washington. However, the consistency of reporting by state, period, and measure varied. 11
Discussion
To demonstrate the QI outcomes of a treatment demonstration program, administrative data can be useful, although the time (ie, potential 3-year time delay) and funding limitations for infrastructure development at the state level need to be carefully considered. Patient-level EHR data provide a complementary data source for driving real-time improvement at the local and health care provider levels. Developing patient-level EHR measures collaboratively between the RCCs and NCC using a collective impact model facilitated cross-regional learning. For example, the adoption of similar data collection methods and instruments across regions was critical in ensuring comparable measures and reducing burden through sharing best practices.
Although population-level and patient-level data are both needed, the work of the SCDTDP RCCs and NCC demonstrates the need to have an overall measurement strategy that aligns both administrative data and data at the level of health care providers, health care organizations, public health entities, and patients. In addition, rigorous methods are needed to ensure quality measures can be used equally at national, state, and site levels. These findings are reflected in the current landscape of national efforts to develop metrics to assess quality of care for SCD patients across agencies, including the National Heart, Lung and Blood Institute’s Sickle Cell Disease Implementation Consortium and the Agency for Healthcare Research and Quality’s Pediatric Quality Measurement Program.
Administrative data for SCD have strengths and limitations for determining clinical use and identifying health care providers caring for persons with SCD. Administrative data can be useful for SCD surveillance and to better understand where patients receive clinical care. They can be used to measure access and use on a national scale for a rare disease. However, administrative data are not currently available in real time, making it difficult to link to ongoing QI activities, which require continuous, current data to inform change.
We found challenges in the standardization of administrative data despite a centrally developed set of clearly defined measure specifications and a data dictionary. The MCO partner organizations interpreted the measure definitions in various ways (eg, definitions of “provider” could include transport services, pharmacies, or durable medical equipment in some states). In addition, the methods for successfully obtaining Medicaid data in 1 state do not necessarily translate to another state. For example, cost, timeliness, and availability of claims data may vary among state agencies and MCOs. Some states rely on third-party MCOs, academic centers, or claims processing firms to administer claims, and extra care and resources (eg, a statistical programmer with experience using Medicaid data) are required to ensure that collected data are comparable across states and regions. Lessons for establishing a data collection strategy de novo ranged from the administrative data challenges to delays in obtaining individual state IRB approvals and data use agreements.
Although Medicaid data provide a record of care received, clinical information is limited, and no information on care needed is provided. RCCs also found that the quality of Medicaid data varied; for example, some data indicated incorrect diagnosis codes. Another limitation of MCO data requests is that some MCOs with research institutes require that their personnel are granted co-principal investigator status with salary support and authorship agreements before agreeing to provide data. A final limitation was that some MCOs had too few SCD claims to warrant the time required to set up the data request.
Initially, the NCC and RCCs also pursued the idea of collecting national administrative data directly through the Centers for Medicare & Medicaid Services’ Research Data Assistance Center and through a contracted supplier of Centers for Medicare & Medicaid Services data outputs. Unfortunately, given the cost (approximately $150 000 per annum) and a substantial lag in data access (ie, several years between when an event takes place, when a claim is made, and when these data become available in the national data system), they decided that although access to these measures long term would be an asset to understanding SCD at a national level, resources were insufficient to contract with the Research Data Assistance Center during the 2014-2017 funding cycle. 11
The collection of administrative data and the collection of EHR data posed different challenges. Many challenges arose from the retroactive alignment of data collection and analysis strategies developed locally to drive site-level QI. Furthermore, the consent and IRB approval process created challenges and delays, because patient-level EHR data were not included in the original project measurement strategy or required grant funding. Finally, because of limited resources, the EHR data included only patients from specific sites, leaving major gaps in a broad understanding of the reach and effectiveness of the SCDTDP. Much of the SCD population receives care outside the specialty centers that contributed EHR data.
A major limitation of this measurement strategy was that the expert group, NCC, and RCCs developed quality measures via a modified Delphi method that were not otherwise validated or tested for feasibility. Using a modified Delphi method was a way to ensure timely buy-in from all stakeholders so that data collection could commence as quickly as possible. In addition, the project timeline necessitated a short turnaround time that did not allow for additional feasibility and validity testing.
A strategy to address the measurement concerns would be to create a population-based national data registry for persons with SCD. Registries can have different purposes—research and population-based disease management, surveillance, and QI—and can be sponsored by a government agency, nonprofit organization, health care facility, or private company. However, to establish an SCD registry, many questions need to be answered in collaboration with a broad range of stakeholders, including registry objectives, type of data collected, patient identification, data access, funding sources, and, most importantly, how a registry would benefit stakeholders including patients, families, health care providers, and researchers.
Practice Implications
SCD requires an effective, simple, shared measurement approach because of the small numbers of SCD patients and the complexity of care delivery. To drive QI efforts in future SCDTDP funding cycles, patient-level EHR measures will be used, and RCCs have a structure for collectively developing additional measures as needed. This expanded data set will provide high-quality data, encourage use of existing systems and common metrics, and increase the ability of data to validate key processes and outcomes at regional and national levels. Medicaid data will not be used because of the cost, time, and effort required to access these data and the inconsistencies in data accuracy.
RCCs implemented real-time quality metrics to assess whether the QI interventions led to improvements in access to quality and care (aim 1) and increased use of hydroxyurea (aim 2). These interventions included telehealth-based approaches and expanding collaborative networks to include government partners and other stakeholders. RCCs developed education activities (eg, publications, presentations, clinical decision-making tools) for health care providers and patient education materials (eg, brochures, shared decision-making tools) to increase the use of hydroxyurea. Further details and a compendium of tools and resources can be found in the 2017 congressional report. 11
By targeting a limited number of well-defined metrics and building partnerships with stakeholders invested in improving care and experienced with managing complex data, the SCDTDP established a shared measurement approach that can be leveraged, and improved, to effectively support national improvement initiatives for SCD.
Footnotes
Acknowledgments
The authors thank Susan T. Paulukonis, MA, MPH; Karen Kalinyak, MD; Michael Rowland, MPH; Rosalyn W. Stewart, MD, MS, MBA; and John J. Strouse, MD, MPH, for their contributions to the development of the measurement strategy, data collection, submission, and analysis for the SCDTDP.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Health Resources and Services Administration, Maternal and Child Health Bureau, US Department of Health and Human Services, contract no. HHSH50201 400026C, Sickle Cell Disease Treatment Demonstration Program.
