Improving HRSA Programs Through Research and Evaluation

Most people know the Health Resources and Services Administration (HRSA) as a part of the US Department of Health and Human Services (HHS) that provides health care to people who are geographically isolated, economically vulnerable, or medically vulnerable. Comprising 5 bureaus and 11 offices, HRSA provides leadership and financial support to state agencies, local governments, community and faith-based organizations, academic institutions, and health care providers in every state and US territory. People recognize HRSA for administering the health center program, the federal organ donation and transplantation program, the Ryan White HIV/AIDS Program, the National Health Service Corps and other health workforce programs, and the Title V Maternal and Child Health Services Block Grant Program, among other programs. But many people do not know that HRSA also conducts research.

HRSA does indeed conduct research, and HRSA staff members frequently author or coauthor published research studies on HRSA’s programs. ^{1 –4} In this Executive Perspective, we—2 senior scientists at HRSA—describe the research and evaluation work that our agency does and how we use the results to inform program planning and decision making, improve operations, and ensure that we effectively and efficiently address the public health needs of the populations we serve.

Background Efforts

HRSA’s research work is not new. The agency has a long history of conducting research and evaluating its programs. For example, since 1963, the Maternal and Child Health Bureau has supported investigations, through its Office of Epidemiology and Research, that have influenced the clinical management, organization, and delivery of health care services, preventive care, and early intervention for maternal and child health populations, including children with special health care needs. ⁵ In addition, starting in the 1980s, HRSA supported research on the causes, diagnosis, prevention, and treatment of Hansen’s disease (leprosy) aimed at global elimination of the disease. ⁶ Finally, the Ryan White HIV/AIDS Program has conducted research on innovative models of care for underserved, underinsured, and uninsured populations (eg, homeless people) since the mid-1990s. ⁷

More recently, HRSA has made research and evaluation a higher priority and has worked to ensure its quality and rigor, in part to inform decisions about costs, interventions, and quality of care. In 2010, HRSA reorganized its planning and evaluation functions and created an agency-wide Office of Research and Evaluation (ORE) within the Office of Planning, Analysis and Evaluation housed in the HRSA Office of the Administrator. ⁸ ORE conducts internal research and evaluation studies of HRSA’s programs to inform program and agency-level decision making; provides consultation, technical assistance, education, and review services to HRSA bureaus and offices on their research and evaluation protocols, tools, and activities; and pursues continuous quality-improvement initiatives to strengthen HRSA program management and operations.

Although most of ORE’s activities are internal behind-the-scenes efforts, some efforts are more visible. For example, in September 2010, HRSA established the Community Health Applied Research Network (CHARN), a nationwide network of 17 health centers established to conduct patient-centered outcomes research among underserved patient populations. CHARN has resulted in partnerships between community clinicians and health care researchers to build capacity to conduct rigorous multisite patient-centered outcomes research to improve patient care. These researchers have used the CHARN data warehouse registry to answer important questions about health conditions in underserved populations. For example, a recent study found that effective medication-assisted treatment may be substantially underprescribed among health center patients identified as having opioid or alcohol use disorders. ⁹ The results from this analysis support the Substance Abuse and Mental Health Services Administration-HRSA Center for Integrated Health Solutions in targeting technical assistance and training to HRSA-funded safety net providers, including health centers, that are building capacity for the treatment of addictions. ¹⁰

Our office also funded an external formative and summative evaluation of the Healthy Weight Collaborative, an initiative implemented during 2011-2013 and charged with creating partnerships among primary care, public health, and community-based organizations to discover sustainable ways to promote healthy weight and eliminate disparities in US communities at the organizational, environmental, and individual levels. This evaluation assessed the planning, implementation, and refinement of key Healthy Weight Collaborative components and sustainability efforts and whether the project achieved its overall goals. ^11,12

One of ORE’s roles is to build research and evaluation capacity among HRSA staff members. To that end, beginning in 2014, ORE began sponsoring an annual Research & Innovation Symposium, an internal 1-day event held at HRSA headquarters. Several hundred HRSA staff members have participated in this event, showcasing their research results, innovations, and best practices through panel sessions and poster displays, similar to a professional meeting. In addition, ORE provides targeted training to program staff members through regular “EvalChats” (ie, hour-long seminars with in-person and online participants) and disseminates evaluation resources with practical information on topics ranging from designing logic models and developing surveys, to demonstrating how program evaluation skills infuse performance measurement and enhance organizational learning.

HRSA’s Research and Evaluation Portfolio: From Surveys to Research Networks

Although it is difficult to quantify HRSA’s budget for research and evaluation, this type of work is conducted both in-house by HRSA staff members (intramural) and out of house by funded grantees and contractors (extramural). Most research projects are extramural. In fiscal year 2016, we identified 280 research studies being conducted (or “in process”) at HRSA, 248 of which were supported by extramural funding. Extramural research at HRSA also includes research centers funded by the Maternal and Child Health Bureau, the Bureau of Health Workforce, and the Federal Office of Rural Health Policy. HRSA’s research and evaluation activities vary from conducting household surveys, to analyzing outcomes using administrative data, to supporting innovative and complex research methods and study approaches in academic centers and in the field, as illustrated by the following examples.

Surveying Children’s Health Needs

HRSA administers the National Survey of Children’s Health, the only nationally representative survey that considers children’s health and well-being in the context of family and community. The National Survey of Children’s Health measures children’s experiences with the health care system and provides national and state prevalence estimates for various child health and health care indicators (eg, oral health status, access to primary care). Research from this survey informs programs such as the Title V Maternal and Child Health Services Block Grant Program. For example, survey results help the Maternal and Child Health Bureau establish baseline estimates for block grant performance measures and provide data for states’ needs assessments. ¹³ In addition, data from the National Survey of Children’s Health are available to researchers through the Data Resource Center for Child & Adolescent Health, a national data resource that provides national, state, and regional data. ¹⁴ The Data Resource Center promotes the use of these data to facilitate child health needs assessments, program planning and evaluation, policy and standards development, and applied research efforts.

Analyzing Disparities in HIV Outcomes Using Administrative Data

Administrative data provide a wealth of information for HRSA program evaluations. The Ryan White HIV/AIDS Program collects client-level data on demographic and human immunodeficiency virus (HIV) clinical outcomes from more than 2000 grant recipients that can be analyzed to inform improvements in care and treatment. A recently published study led by HRSA’s HIV/AIDS Bureau examined changes in viral suppression among clients served by the Ryan White HIV/AIDS Program from 2010 to 2014. ¹ The results uncovered reductions in disparities in viral suppression between black/African American and white populations, between younger and older clients, and between clients living in the South and clients living in other US regions. The study highlighted important information for refining and implementing approaches to reducing disparities.

Partnering With Academic Researchers

Another source of research information is the annual administrative data reported by health center grantees to the agency. ¹⁵ Partnering with academic researchers, HRSA can apply complex research methodologies and analyze these data to answer important policy questions. For example, in 2016, researchers at HRSA and the Johns Hopkins University examined the correlation between achieving recognition as a patient-centered medical home and the quality of care delivered at health centers. ³ They found that after controlling for patient, provider, and practice characteristics, health centers with recognition as a patient-centered medical home reported significantly better performance on asthma-related pharmacologic therapy, diabetes control, Papanicolaou testing, prenatal care, and tobacco cessation intervention than did health centers without such recognition. Such studies support the association between transformation into a patient-centered medical home and clinical quality improvement.

Supporting Rural Health Research

HRSA’s Federal Office of Rural Health Policy funds research through its Rural Health Research Center cooperative agreement, the only federal program dedicated to producing policy-relevant research on rural health. Research topics pursued by Rural Health Research Centers include assessing access to care and health care utilization among rural residents, identifying barriers and disparities among rural populations, and determining the impact and effectiveness of telehealth, critical access hospitals, and other service delivery models. A recent study using a mixed-methods approach examined why obstetric units in rural hospitals were closing. The study found that the closures were more common in smaller hospitals and communities with a limited obstetric workforce, raising concerns about access to care. ¹⁶ This type of research regularly informs HRSA rural health programs and policymaking and is widely disseminated through HRSA’s Rural Health Research Gateway (https://www.ruralhealthresearch.org).

Future HRSA Research and Evaluation: Challenges and Opportunities

Building HRSA’s research and evaluation capacity has its challenges, including limited resources and data. As a services agency, HRSA and its grantees must place the highest priority on program implementation, whether delivering health center services to pregnant women or early intervention or treatment services to children with special needs or HIV-positive people. In addition, limitations in the extent and quality of available data can sometimes preclude answering certain important research questions that can inform data-driven decision making. Adequate program funding is a priority; sustained funding commitments to support the collection of quality data and conduct additional research and evaluation, however, may help improve programs and enhance lessons learned.

Moving forward, HRSA will identify new areas of research and evaluation to accurately and effectively assess the impact of agency programs, as well as further understand differences in health care access, supply, delivery, and outcomes. HRSA is also continuing efforts to build internal capacity to conduct rigorous research and evaluation. Ultimately, HRSA’s commitment to the use of evidence from our research and evaluation studies for decision making will have multiple benefits. These benefits include providing better information on which HRSA programs are working and which are not working, a greater chance that successful policies and programs will be implemented at our agency, increased workforce productivity, and, ultimately, a more efficient and effective use of public funding. ¹⁷