A Longitudinal Evaluation of a Novel Short-Term Staffing and Telehealth Program: A Retrospective Observational Study

Abstract

Introduction:

Primary care is foundational to health, but provider shortages impact access. In the Veterans Health Administration (VHA), Clinical Resource Hubs (CRH) provide remote clinicians to alleviate short term staffing gaps.

Methods:

A longitudinal analysis of primary care clinics from October 1, 2020, through September 30, 2024, was conducted. CRH user clinics were those with ≥10 CRH visits/month in 2 consecutive months. A time-varying measure of minimal threshold CRH use (ie, ≥10 CRH visits/month) was also constructed. Outcomes were measured monthly, including established and new patient wait time, third next available appointment (ie, average time to the third open appointment in provider schedule), timely care, and an indicator that 80% of new patients were seen within 20 days. Using a within-between model framework, generalized linear mixed models with robust standard errors assessed the association between CRH use and access to care.

Results:

Overall, 37.1% of VHA primary care clinics (N = 366) were CRH users In adjusted models, clinics meeting the time-varying minimal threshold for CRH use versus not was associated with an increase in established patient wait time (β = .35 day, SE = 0.07, P < .001) and new patient wait time (β = .69 day, SE = 0.20, P < .001), and a decrease in timely care (β = −1.40%, SE = 0.21, P < .001).

Conclusions:

When CRH was engaged, clinics had marginally longer wait times. Though CRHs have provided remote staffing, additional efficiencies may be needed to close access gaps.

Keywords

primary care telemedicine veterans rural veterans United States Department of Veterans Affairs

Introduction

The Veterans Health Administration (VHA) provides care to over 9 million veterans in the United States and its territories.¹ Primary care is a foundational service. Primary care providers play a crucial role in care coordination for complex patients and primary care access has been shown to improve health outcomes, lower mortality rates, and facilitate subspecialty access.^2-4 To provide adequate and timely care, primary care providers need to be available; however, VHA faces a provider shortage.^5-8 These provider shortages will likely worsen as practicing physicians retire,^9-11 fewer providers choose primary care as their specialty,¹² and the general population ages increasing demand. Additionally, rural areas have difficulty recruiting and retaining providers.^13,14

VHA piloted the Virtual Integrated Multisite Patient Aligned Care Teams (V-IMPACT) program from 2013 to 2018 to deliver primary care when VHA sites experienced gaps in primary care staffing,¹⁵ with care predominantly delivered using clinic-to-clinic synchronous telemedicine. Research has shown V-IMPACT delivered high quality diabetes care similar to that delivered in-person,¹⁶ increased telehealth primary care visits,¹⁷ and had higher penetration rates among rural veterans and sites.¹⁵ From this program, the national Clinical Resource Hub (CRH) program originated in 2019.¹⁸ Like V-IMPACT, the CRH program utilizes a hub-and-spoke model; it is organized within 18 Veterans Integrated Service Networks (VISN; geographic regions that implement national policy and manage VA medical centers) to provide care to outlying clinics during staffing gaps, but covered services expanded to include mental health and specialty providers. VA medical centers or community-based outpatient clinics request coverage based on the lack of staffing or increasing veteran volumes, while CRH leaders prioritize fulfillment based on veteran need and CRH staff availability.¹⁸ This organization allows program flexibility responsive to facility needs. The primary program goal is to increase or maintain access to care through contingency staffing (ie, provision of remote staff when existing staff are insufficient) with an emphasis on clinics that are most underserved.¹⁹

Preliminary data prior to the COVID-19 pandemic showed no significant differences in wait times when comparing clinics implementing CRH versus not.²⁰ This initial analysis included 80 CRH user clinics, however the program has since engaged fourfold as many clinics. Further, the COVID-19 pandemic led to expanded telehealth options.^21,22 This shift has persisted with video-based visits accounting for 11% of all VHA outpatient care.^23,24

The objective of this study was to longitudinally evaluate the association between CRH utilization and primary care access through the assessment of primary care wait times.

Methods

Study Design

A time series analysis using a retrospective observational cohort of clinics using (CRH user) and not using CRH (control clinics) from October 1, 2020, through September 30, 2024, was conducted. CRH user clinics were those with ≥10 primary care CRH visits per month for 2 consecutive months. Control clinics were those not meeting this criterion. This cutoff was selected to balance meaningful engagement and variable clinic sizes based on statistical measures, graphical approaches, and team discussions. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline was followed.²⁵ This work was designated non-research since it was carried out as a quality improvement evaluation under the terms of a signed attestation of non-research from the University of Iowa Institutional Review Board [IRB #202002500]. The authors had full access to and take full responsibility for the integrity of the data.

Data Sources

VHA administrative data and electronic health records were accessed using the Corporate Data Warehouse (CDW) to obtain clinic- and patient-level data. Clinic-level data, including facility type, was obtained from the VHA Site Tracking System. Clinic size was obtained from the Primary Care Management Module, a quarterly census of active primary care patients. The number of primary care full-time equivalents (FTEs) in VHA clinics or by CRH providers and the 4 outcomes (ie, established and new patient wait time, time to third next available^26,27 appointment, proportion of new patients seen within 20 days) were obtained from the VHA Support Services Center and aggregated to the clinic-month. Lastly, the timely care success rate was calculated using outpatient and appointment tables from CDW.²⁸

Patient-level data, including demographics (ie, age, sex, race, rural residency, and VHA Nosos risk adjustment score, which uses a scoring algorithm based on patient diagnoses, age, sex, pharmacy records, VA priority status, and VA-computed costs²⁹), and the date, location, and delivery method of outpatient primary care visits were obtained from the CDW. The Area Deprivation Index, a measure of neighborhood socioeconomic disadvantage,³⁰ was spatially merged to the fiscal year-specific latitude and longitude of each veteran’s home address using 2020 Census Bureau census block TIGER/Line shapefiles.

Outcomes

Five measures of access to care were used: (1) established patient wait time; (2) new patient wait time; (3) third next available appointment,^26,27 (4) timely care,²⁸ and (5) a binary indicator ≥80% of new patients were seen in 0 to 20 days. Established patients are those with a VHA primary care appointment in the past 3 years. Established patient wait time was measured from the date the patient and provider agree is clinically indicated for care to appointment completion date. New patient wait time was measured from the date the appointment was created to appointment completion date. Third next available and timely care are measures of clinic capacity. Third next available was the average time to the third open appointment in a provider’s clinic schedule.^26,27 Timely care was defined as the provision of an appointment within 48 h of any patient requesting the clinic’s next available appointment or walking in to receive care.²⁸ These measures were averaged at the clinic level by month.

Covariates

Patient access is affected by a clinic’s patient mix and its physical environment. In addition to our exposure of interest, a time-varying measure of minimal threshold CRH use (ie, ≥10 CRH visits/month) was constructed; it was measured monthly and matched in time to each outcome. Clinic variables included facility type and size, categorized gap metric,³¹ and VISN (VHA designation of region). Clinic size, as determined by the number of primary care patients assigned per year in the primary care management module, was categorized as small (450-2399), medium (2400-9999), and large (10 000+). The categorized gap metric, the ratio of the CRH-corrected expected number of primary care patients to the observed number of primary care patients, was calculated monthly and categorized as understaffed (<1.0), marginally staffed (1.0-1.2), and adequately staffed (≥1.2).

Patient variables were aggregated to the clinic-month and included average patient age, proportion male, proportion white, proportion rural,³² and average Nosos score²⁹ anchored to October 1, 2022 (ie, the midpoint of the study period), and average Area Deprivation Index.³⁰

Statistical Analyses

We describe the clinic and patient mix characteristics, as well as study outcomes stratified by clinic-level CRH user status. Group comparisons were 2-sided with a .05 significance level using a t-test or chi-square test, as appropriate. We summarize the proportion of understaffed clinic-months in which the clinic met the minimum threshold for CRH use (ie, ≥10 CRH visits/month) by VISN.

The primary analysis used a within-between model framework, which partitions estimates into within-group and between-group effects. Each outcome was modeled separately using a generalized linear mixed model with a random intercept for clinics and adjusted using the covariates previously described. Model estimates, standard errors, and an indicator of significance are reported. The Hausman specification test assessed whether between and within effects were the same. Models were assessed for multicollinearity using the adjusted generalized standard error inflation factor (aGISF), which can accommodate categorical variables with more than 2 levels.³³

A sensitivity analysis considering only the most recent 2 years of data (eg, October 1, 2022, through September 30, 2024) was performed, as the program’s ability to provide primary care when a clinic was understaffed improved over time.

All analyses were conducted using R (version 4.4.0) via the panelr package.^34,35 The study was approved by the University of Iowa Institutional Review Board [IRB # 202002500] and the Iowa City VA Healthcare System Research and Development Committee. We conducted this work without direct patient contact using routinely collected data deemed of minimal risk. A waiver of informed consent was obtained.

Results

From October 1, 2020, through September 30, 2024, 37.1% of primary care clinics (N = 366) were defined as a CRH user (ie, used ≥10 primary care CRH visits for 2 consecutive months; Table 1). CRH user clinics compared to controls were more likely to serve 10 000+ patients (28.4% vs 10.0%; P < .001), and slightly more likely to be understaffed (15.8% vs 14.8%; P < .001). CRH users had, on average, longer mean [SD] wait times compared to controls for both established (5.8 [4.0] vs 5.1 [5.1] days) and new patients (21.6 [14.2] vs 20.1 [13.5] days). CRH users also experienced longer mean [SD] times for the third next available visit (13.7 [7.2] vs 11.4 [6.8] days) compared to control clinics, with 67.6% vs 71.1% of requests for timely care being fulfilled.

Table 1.

Characteristics of Clinical Resource Hub (CRH) User and Control Clinics.

	CRH user^a	Control^b	P-value
Characteristic	N = 366	N = 620	P-value
Clinic characteristics, N (%)
Number of months with identified gap in primary care provider staffing			<.001
Understaffed	2760 (15.8%)	4297 (14.8%)
Adequately staffed	3863 (22.2%)	9159 (31.5%)
Overstaffed	10 802 (62.0%)	15 646 (53.8%)
Facility type			<.001
Community-based outpatient clinic	233 (63.7%)	428 (69.0%)
VHA Medical Center/HCC	103 (28.2%)	61 (9.9%)
Other outpatient services	30 (8.2%)	131 (21.1%)
Clinic size			<.001
Small (450-2399)	81 (22.1%)	244 (39.4%)
Medium (2400-9999)	181 (49.5%)	314 (50.6%)
Large (10 000+)	104 (28.4%)	62 (10.0%)
Clinic characteristics^c, mean (SD)
Primary care (PC) provider FTEs	8.5 (7.7)	4.8 (5.3)	<.001
CRH-corrected PC FTE	8.2 (7.6)	4.7 (5.3)	<.001
CRH minimum threshold months, N (%)	5826 (33.4%)	0 (0.0%)	—
Access to care^d, mean days (SD)
Established patient wait time	5.8 (4.0)	5.1 (5.1)	<.001
New patient wait time	21.6 (14.2)	20.1 (13.5)	<.001
Third next available	13.7 (7.2)	11.4 (6.8)	<.001
Timely care	67.6% (16.8%)	71.1% (17.3%)	<.001
80% of new patients seen in 0-20 days	15.9% (36.5%)	26.3% (44.0%)	<.001
Patient mix characteristics^c, mean (SD)
Average baseline age	63.4 (3.8)	64.0 (4.4)	<.001
Male sex, % (SD)	90.7% (4.7%)	91.5% (6.7%)	<.001
White race, % (SD)	64.1% (17.0%)	66.1% (16.0%)	<.001
Rural residing, % (SD)	46.6% (34.3%)	53.3% (36.9%)	<.001
Average Nosos score	1.0 (0.3)	0.9 (0.2)	<.001
Area Deprivation Index	55.9 (19.6)	58.5 (19.4)	<.001

CRH user clinics must have a minimum of 2 consecutive months with ≥10 primary care CRH visits between October 2020 and September 2023.

Control clinics are those not meeting the definition to be a CRH user clinic.

Reported within the 3 months prior to study start (ie, July-September 2020).

Reported within the prior fiscal year (ie, October 2019-September 2020).

Within clinic, on average, the models estimated an increase of 0.36-day (SE = 0.07; P < .001) and 0.76 day (SE = 0.20; P < .001), respectively, for established and new patient wait times when a clinic delivered ≥10 CRH primary care visits in a month versus not (Table 2).

Table 2.

Within-Between Models Estimates (Standard Errors) for Wait Time Outcomes.

Model variables		Established patient wait time^a	New patient wait time^a
Model variables		Estimate (SE)
Measures of clinical resource hub (CRH) use
Minimum threshold CRH use in month met
	Within Clinics	0.36 (0.07)^‡	0.76 (0.20)^‡
	Between Clinics	0.96 (0.63)	2.04 (1.69)
CRH user status	Between Clinics	0.00 (0.28)	1.01 (0.75)
Time (month of study),	Within Clinics	0.02 (0.00)^‡	0.12 (0.01)^‡
Clinic characteristics
Gap staffing metric (ref: understaffed)
Marginally staffed	Within Clinics	−0.25 (0.07)^‡	−2.46 (0.22)^‡
	Between Clinics	−0.51 (0.45)	−2.77 (1.20)*
Overstaffed	Within Clinics	−0.34 (0.08)^‡	−4.04 (0.24)^‡
	Between Clinics	−1.54 (0.36)^‡	−6.97 (0.95)^‡
Facility type (ref: community-based outpatient clinic),	Between Clinics
	VAMC/HCC	0.57 (0.37)	0.38 (0.98)
	Other Outpatient Services	0.38 (0.28)	0.28 (0.74)
Clinic size (ref: small: 450-2399),	Between Clinics
	Medium: 2400-9999 patients	0.74 (0.25)^†	0.51 (0.68)
	Large: 10 000+ patients	0.51 (0.39)	1.15 (1.05)
Patient mix characteristics
Average baseline age, mean	Within Clinics	−0.05 (0.04)	−0.45 (0.11)^‡
	Between Clinics	0.06 (0.03)	−0.41 (0.09)^‡
Male sex, %	Within Clinics	−0.03 (0.03)	−0.64 (0.09)^‡
	Between Clinics	−0.05 (0.02)^†	−0.09 (0.05)
White, %	Within Clinics	0.03 (0.01)^‡	0.09 (0.02)^‡
	Between Clinics	−0.02 (0.01)*	−0.02 (0.03)
Rural residing, %	Within Clinics	−0.06 (0.02)^†	−0.19 (0.06)^†
	Between Clinics	0.00 (0.00)	−0.01 (0.01)
Nosos risk score, mean	Within Clinics	0.01 (0.22)	0.71 (0.83)
	Between Clinics	0.32 (0.57)	−3.39 (1.54)*
Average ADI, mean	Within Clinics	0.08 (0.03)^†	−0.32 (0.08)^‡
	Between Clinics	−0.01 (0.01)	0.01 (0.02)

Abbreviation: ADI, Area Deprivation Index.

The generalized linear mixed model was estimated using a normal distribution with identity link function and random intercept at the clinic-level; the between model also adjusted for VISN, but results for these fixed effects are not reported in the table.

P < .05. ^†P < .01. ^‡P < .001.

No significant associations with meeting the minimum threshold for CRH use per month were noted for the third next available appointment or when a clinic saw at least 80% of new patients within 20 days. (Table 3). However, an average decrease in timely care of 1.41% (SE = 0.21; P < .001) is noted within clinics, when ≥10 CRH primary care visits are provided in a month versus not.

Table 3.

Association of Months When CRH Engaged With Access Within and Between Clinics in National Sample of VA Clinics From October 2020 Through September 2024.

Model variables^a		Third next available^a	Timely care^a	New patients seen ≤20 Days^‖
Model variables^a		Estimate (SE)		OR (95% CI)
Measures of CRH use
Minimum threshold CRH use in month met
	Within Clinics	0.12 (0.08)	−1.41 (0.21)^‡	0.81 (0.71, 0.92)
	Between Clinics	−0.75 (1.02)	−2.23 (2.69)	0.71 (0.34, 1.46)
CRH user status,	Between Clinics	0.57 (0.45)	−0.97 (1.20)	0.94 (0.68, 1.28)
Time (month of study),	Within Clinics	0.03 (0.00)^‡	−0.12 (0.01)^‡	0.95 (0.94, 0.95)
Clinic characteristics
Gap staffing metric (ref: understaffed)
Marginally staffed	Within Clinics	−0.90 (0.09)^‡	−0.01 (0.22)	1.05 (0.94, 1.18)
	Between Clinics	0.05 (0.72)	−1.11 (1.92)	0.81 (0.49, 1.32)
Overstaffed	Within Clinics	−1.21 (0.10)^‡	−0.34 (0.24)	1.24 (1.10, 1.39)
	Between Clinics	−0.83 (0.57)	2.97 (1.52)	1.33 (0.90, 1.96)
Facility type (ref: community−based outpatient clinic),	Between Clinics
VAMC/HCC		0.61 (0.59)	−1.23 (1.56)	0.23 (0.15, 0.35)
Other outpatient services		−0.49 (0.44)	−1.84 (1.17)	1.67 (1.25, 2.22)
Clinic size (ref: small: 450−2399),	Between Clinics
Medium: 2400−9999 patients		0.96 (0.41)*	−2.91 (1.08)^†	0.44 (0.33, 0.57)
Large: 10 000+ patients		4.10 (0.63)^‡	−2.69 (1.67)	0.16 (0.11, 0.26)
Patient mix characteristics
Average baseline age, mean	Within Clinics	−0.52 (0.05)^‡	1.05 (0.11)^‡	1.07 (1.01, 1.14)
	Between Clinics	−0.02 (0.06)	0.05 (0.15)	1.05 (1.01, 1.09)
Male sex, %	Within Clinics	−0.06 (0.04)	−0.47 (0.09)^‡	1.25 (1.18, 1.32)
	Between Clinics	−0.08 (0.03)^†	0.20 (0.08)*	1.03 (1.01, 1.05)
White, %	Within Clinics	0.05 (0.01)^‡	−0.16 (0.02)^‡	0.96 (0.95, 0.97)
	Between Clinics	0.02 (0.02)	0.06 (0.04)	0.99 (0.98, 1.00)
Rural residing, %	Within Clinics	0.04 (0.02)	−0.23 (0.06)^‡	1.04 (1.00, 1.07)
	Between Clinics	−0.02 (0.01)*	0.00 (0.02)	1.01 (1.00, 1.01)
Nosos risk score, mean	Within Clinics	2.15 (0.24)^‡	2.55 (0.67)^‡	0.74 (0.45, 1.21)
	Between Clinics	4.28 (0.91)^‡	−2.59 (2.42)	1.63 (0.84, 3.14)
Average ADI, mean,	Within Clinics	−0.01 (0.03)	−0.08 (0.08)	0.98 (0.94, 1.02)
	Between Clinics	−0.04 (0.01)^†	0.10 (0.03)^†	1.00 (0.99, 1.01)

Abbreviation: ADI, Area Deprivation Index.

P < .05. ^†P < .01. ^‡P < .001.

Sensitivity Analysis

Overall, the CRH program’s ability to provide primary care gap coverage at clinics with identified staffing gaps improved over time (Supplemental eTable 1). Clinical characteristics of CRH users and control clinics were similar in the 2-year period (eg, October 1, 2022, thru September 30, 2024) as in the full study (Supplemental eTable 2). Estimates of the within clinic effects of the time-varying minimum threshold for CRH use were similar, though slightly greater in magnitude for established patient time (β = .46 day, SE = 0.11, P < .001; Supplemental eTable 3) and timely care (β = −1.55 day, SE = 0.34, P < .001; Supplemental eTable 4), while no significant association was identified for new patient time.

Conclusions

VHA has a long history of providing care to America’s veterans.^36-41 The expanded role of telehealth in clinical settings, influenced heavily by the COVID-19 pandemic, will likely persist. CRH fulfills the need for short- and long-term provider coverage by delivering remote providers to areas with provider shortages to ensure access to care remains available.¹⁸ In this longitudinal evaluation of the CRH program, we found (1) measures of access were, on average, worse at CRH user clinics compared to control clinics, and (2) CRH users had marginally longer wait times. Ultimately, though the CRH program has provided remote staffing to avoid interruptions in care, these findings suggest additional efficiencies may be needed.

Provider shortages in rural and underserved areas are projected to grow.^6,42,43 The Health Professional Shortage Area (HPSA) program has attempted to improve health outcomes for rural and underserved communities through loan repayment assistance, increased Medicare reimbursement, J-1 visa waivers, and more,⁴² but with no significant impact on mortality or physician density.⁴⁴ Despite programs offering financial incentives for primary care providers to move to underserved areas, few physicians relocated after establishing their practice and for those that did, more moved from HPSA to non-HPSA areas.⁴⁵ Importantly, the CRH program is not the only innovative telehealth program to address geographic disparities in provider availability and patient need. Like the CRH, these programs are conducted with remotely located medical doctors who work alongside in-person support (eg, nursing staff and physician assistant).^46-50 However, they are typically implemented for specific specialties (eg, hospitalist, pediatric gastroenterology, and mental health)^46-50 rather than offering a tableau of services.^18,51 This paper addresses an important way rural and underserved areas could facilitate increased access to care – through providers living in non-HPSA areas, but who provide telehealth when staffing gaps exist.

Despite this potential, established and new patient wait times increased, albeit by less than a day, when a clinic met the minimum threshold for monthly CRH use. This could be a consequence of a clinic’s existing staffing gaps and subsequent need to engage with the CRH. A lack of primary care providers may lead to longer wait times, which further creates a backlog of appointments that must be addressed before wait times can improve. This increase in wait times may also be related to previous attempts to increase primary care access through the provision of VHA-purchased community care, which reduces the need to maintain appointment availability for veterans receiving care outside VHA facilities. Notably, previous research has found wait time for VHA-purchased primary care in the community was longer compared to that provided by VHA facilities, often exceeding the modest half-day differences we detected when CRH is engaged.^52,53

We note further that some control clinics also experienced understaffed months but did not sufficiently engage with the CRH to be considered a CRH user. Some of these clinics may have experienced short-term needs prohibitive to using the CRH program, which requires the time to coordinate CRH providers with VHA staff at spoke locations. Additionally, the choice not to engage with the program could be due to difficulties staffing non-provider (eg, nurse, pharmacist, and medical assistant) clinic roles.⁵⁴ Finally, decreased use may be due to lack of awareness or resources (eg, training, infrastructure, staffing, etc.) needed to implement the program.

The study’s strengths include its large national study population, but it does have some limitations. While our study population was diverse regarding rurality, race, and socioeconomic status, the majority were older men reducing generalizability. Additionally, because clinics choose to engage with the CRH program based on their need for support, selection bias may have impacted our results. Though the longitudinal nature of the data is a strength, this retrospective observational study can only illustrate associations, not causation, and it is constrained by the accuracy of the retrospective data reported in the electronic medical record. Additionally, the results were not broken down based on the geographic location of the clinic though previous studies have shown geography can play a significant role in access to care.⁵⁵ Lastly, patient outcomes were not tracked to determine if the care delivered remotely was adequate for patient need.

In conclusion, we assessed access in primary care clinics stratified by meaningful CRH engagement. CRH user clinics were more likely to have reduced access and marginally longer wait times. Findings suggest CRH has provided care in the presence of staffing shortages. However, further research into implementation barriers in smaller and more rural clinics should be considered.

Supplemental Material

sj-docx-1-jpc-10.1177_21501319261424758 – Supplemental material for A Longitudinal Evaluation of a Novel Short-Term Staffing and Telehealth Program: A Retrospective Observational Study

Supplemental material, sj-docx-1-jpc-10.1177_21501319261424758 for A Longitudinal Evaluation of a Novel Short-Term Staffing and Telehealth Program: A Retrospective Observational Study by Paige Carlson, Bjarni Haraldsson, Matthew R. Augustine, Peter J. Kaboli and Amy M. J. O’Shea in Journal of Primary Care & Community Health

Footnotes

Author Note

The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the United States Government.

ORCID iDs

Paige Carlson

Bjarni Haraldsson

Matthew R. Augustine

Peter J. Kaboli

Amy M. J. O’Shea

Ethical Considerations

The study was approved by the University of Iowa Institutional Review Board [IRB # 202002500] and the Iowa City VA Healthcare System Research and Development Committee.

Consent to Participate

We conducted this work without direct patient contact using routinely collected data deemed of minimal risk. A waiver of informed consent was obtained.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon work supported (or supported in part) by the VHA Primary Care Analytics Team (PCAT), funded by the VHA Office of Primary Care, as well as the Department of Veterans Affairs, Veterans Health Administration, Office of Rural Health, Veterans Rural Health Resource Center – Iowa City, and the Office of Research and Development, Health Services Research and Development (HSR&D) Service through the Comprehensive Access and Delivery Research and Evaluation (CADRE) Center (CIN 13-412). The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. This manuscript is not under review elsewhere and there is no prior publication of manuscript contents. The preliminary results of this manuscript were presented at the Society of General Internal Medicine Meeting; Boston, Massachusetts in May 2024 and the Annual Meeting of Academy Health; Baltimore Maryland in June 2024.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

Due to US Department of Veterans Affairs (VA) regulations and our ethics agreements, the analytic data sets used for this study are not permitted to leave the VA firewall without a Data Use Agreement. This limitation is consistent with other studies based on VA data. However, VA data are made freely available to researchers with an approved VA study protocol. For more information, please visit or contact the VA Information Resource Center at VIReC@va.gov.

Supplemental Material

Supplemental material for this article is available online.

References

Heyworth

Kirsh

Zulman

Ferguson

Kizer

KW.

Expanding access through virtual care: the VA’s early experience with Covid-19. N Engl J Med. 2020;1(4). doi:10.1056/CAT.20.0327

Starfield

Shi

Macinko

Contribution of primary care to health systems and health. Milbank Q. 2005;83(3):457-502. doi:10.1111/j.1468-0009.2005.00409.x

Fortney

Steffick

Burgess

Jr Maciejewski

Petersen

LA.

Are primary care services a substitute or complement for specialty and inpatient services?

Health Serv Res. 2005;40(5 Pt 1):1422-1442. doi:10.1111/j.1475-6773.2005.00424.x

Ohl

Richardson

Kaboli

Perencevich

Vaughan-Sarrazin

Geographic access and use of infectious diseases specialty and general primary care services by veterans with HIV infection: implications for telehealth and shared care programs. J Rural Health. 2014;30(4):412-421. doi:10.1111/jrh.12070

Government Accountability Office. Improved Succession Planning Would Help Address Long-Standing Workforce Problems. GAO; 2019.

Zhang

Lin

Pforsich

Lin

VW.

Physician workforce in the United States of America: forecasting nationwide shortages. Hum Resour Health. 2020;18(1):8. doi:10.1186/s12960-020-0448-3

Malayala

Vasireddy

Atluri

Alur

RS.

Primary care shortage in medically underserved and health provider shortage areas: lessons from Delaware, USA. J Prim Care Community Health. 2021;12:2150132721994018. doi:10.1177/2150132721994018

Moldestad

Sayre

Rinne

, et al Perspectives on training and working in the VHA: implications for primary care physician recruitment and retention. Acad Med. 2022;97(8):1175-1183.

Hedden

Ahuja

Lavergne

, et al How long does it take patients to find a new primary care physician when theirs retires: a population-based, longitudinal study. Hum Resour Health. 2021;19(1):92. doi:10.1186/s12960-021-00633-9

10.

Khan

Teoh

Islam

Hassard

Psychosocial work characteristics, burnout, psychological morbidity symptoms and early retirement intentions: a cross-sectional study of NHS consultants in the UK. BMJ Open. 2018;8(7):e018720. doi:10.1136/bmjopen-2017-018720

11.

Hodkinson

Zhou

Johnson

, et al Associations of physician burnout with career engagement and quality of patient care: systematic review and meta-analysis. BMJ. 2022;378:e070442. doi:10.1136/bmj-2022-070442

12.

West

Dupras

DM.

General medicine vs subspecialty career plans among internal medicine residents. Jama. 2012;308(21):2241-2247. doi:10.1001/jama.2012.47535

13.

Johnson

Wong

Lampman

, et al Comparing patient-centered medical home implementation in urban and rural VHA clinics: results from the patient aligned care team initiative. J Ambul Care Manage. 2018;41(1):47-57. doi:10.1097/jac.0000000000000212

14.

Weeks

Yano

Rubenstein

LV.

Primary care practice management in rural and urban Veterans Health Administration settings. J Rural Health. 2002;18(2):298-303. doi:10.1111/j.1748-0361.2002.tb00890.x

15.

Scott

Chow

, et al Rural and urban differences in the implementation of Virtual Integrated Patient-Aligned Care Teams. J Rural Health. 2023;39(1):272-278. doi:10.1111/jrh.12676

16.

Gunzburger

Glorioso

, et al Impact of longitudinal virtual primary care on diabetes quality of care: longitudinal virtual primary care on diabetes quality of care. J Gen Intern Med. 2021;36(9):2585-2592. doi:10.1007/s11606-020-06547-x

17.

Gujral

Scott

Ambady

, et al A primary care telehealth pilot program to improve access: associations with patients’ health care utilization and costs. Telemed J E Health. 2022;28(5):643-653. doi:10.1089/tmj.2021.0284

18.

Burnett

Stockdale

Yoon

, et al The clinical resource hub initiative: first-year implementation of the veterans health administration regional telehealth contingency staffing program. J Ambul Care Manage. 2023;46(3):228-239. doi:10.1097/jac.0000000000000468

19.

Rubenstein

Curtis

Wheat

, et al Learning from national implementation of the Veterans Affairs Clinical Resource Hub (CRH) program for improving access to care: protocol for a six year evaluation. BMC Health Serv Res. 2023;23(1):790. doi:10.1186/s12913-023-09799-5

20.

O’Shea

AMJ

Haraldsson

Augustine

Shahnazi

Mulligan

Kaboli

. Impact of a remote primary care telehealth staffing model on primary care access in the veterans health administration. J Gen Intern Med. 2024;39:2771-2779. doi:10.1007/s11606-024-08835-2

21.

Baum

Kaboli

Schwartz

MD.

Reduced in-person and increased telehealth outpatient visits during the COVID-19 pandemic. Ann Intern Med. 2021;174(1):129-131. doi:10.7326/m20-3026

22.

Mann

Chen

Chunara

Testa

Nov

COVID-19 transforms health care through telemedicine: evidence from the field. J Am Med Inform Assoc. 2020;27(7):1132-1135. doi:10.1093/jamia/ocaa072

23.

Ferguson

Wray

Van Campen

Zulman

DM.

A new equilibrium for telemedicine: prevalence of in-person, video-based, and telephone-based care in the veterans health administration, 2019-2023. Ann Intern ed. 2024;177(2):262-264. doi:10.7326/M23-2644

24.

O’Shea

AMJ

Howren

Mulligan

Haraldsson

Shahnazi

Kaboli

. Quantifying the digital divide: associations of broadband internet with tele-mental health access before and during the COVID-19 pandemic. J Gen Intern Med. 2023;38(3):832-840. doi:10.1007/s11606-023-08120-8

25.

von Elm

Altman

Egger

, et al The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61(4):344-349. doi:10.1016/j.jclinepi.2007.11.008

26.

Hempel

Hilton

Stockdale

, et al Defining access management in health care delivery organizations. J Ambul Care Manage. 2021;44(3):218-226.

27.

Shah

Latifovic

Meaney

, et al Association between clinic-reported third next available appointment and patient-reported access to primary care. JAMA Netw Open. 2022;5(12):e2246397. doi:10.1001/jamanetworkopen.2022.46397

28.

Batten

Augustine

Nelson

Kaboli

PJ.

Development of a novel metric of timely care access to primary care services. Health Serv Res. 2020;55(2):301-309. doi:10.1111/1475-6773.13255

29.

Wagner

Stefos

Moran

, et al Risk Adjustment: Guide to the V21 and Nosos Risk Score Programs: Technical Report 30. Health Economics Resource Center; 2016.

30.

University of Wisconsin School of Medicine and Public Health. 2015 Area Deprivation Index v2.0. Accessed February 4, 2021, https://www.neighborhoodatlas.medicine.wisc.edu/

31.

O’Shea

AMJ

Haraldsson

Shahnazi

Sterling

Wong

Kaboli

. A novel gap staffing metric for primary care in the veterans health administration and implications for rural and urban clinics. J Ambul Care Manage. 2023;46(1):25-33. doi:10.1097/JAC.0000000000000429

32.

University of Washington Rural Health Research Center and USDA Economic Research Service (ERS). Rural-Urban Commuting Area Codes. University of Washington. 2010. Accessed December 13, 2018. http://depts.washington.edu/uwruca/ruca-uses.php

33.

Fox

Monette

Generalized collinearity diagnostics. J Am Stat Assoc. 1992;87(417):178-183. doi:10.1080/01621459.1992.10475190

34.

R: A Language and Environment for Statistical Computing . R Foundation for Statistical Computing, Vienna, Austria. 2024. Accessed April 24, 2024. https://www.R-project.org

35.

panelr: Regression Models and Utilities for Repeated Measures and Panel Data. R package . (Version 0.7.3. 2020). 2020. https://cran.r-project.org/package=panelr

36.

Lum

Nearing

Pimentel

Levy

Hung

WW.

Anywhere to anywhere: use of telehealth to increase health care access for older, rural veterans. Public Policy Aging Rep. 2019 2019;30(1):12-18. doi:10.1093/ppar/prz030

37.

Darkins

Ryan

Kobb

, et al Care Coordination/Home Telehealth: the systematic implementation of health informatics, home telehealth, and disease management to support the care of veteran patients with chronic conditions. Telemed J E Health. 2008;14(10):1118-1126. doi:10.1089/tmj.2008.0021

38.

Girard

Military and VA telemedicine systems for patients with traumatic brain injury. J Rehabil Res Dev. 2007;44:1017-1026.

39.

Griffin

Agarwal

Amberker

, et al An Initiative to improve 30-day readmission rates using a transitions-of-care clinic among a mixed urban and rural veteran population. J Hosp Medi. 2021;16(10):583-588. doi:10.12788/jhm.3659

40.

Moreau

Cordasco

Young

, et al The use of telemental health to meet the mental health needs of women using department of veterans affairs services. Womens Health Issues. 2018;28(2):181-187. doi:10.1016/j.whi.2017.12.005

41.

Naditz

Telemedicine at the VA: VistA, MyHealtheVet, and other VA programs. Telemed e-Health. 2008;14:330-332. doi:10.1089/tmj.2008.9973

42.

Arredondo

Touchett

Khan

Vincenti

Watts

BV.

Current programs and incentives to overcome rural physician shortages in the United States: a narrative review. J Gen Intern Med. 2023;38(3):916-922. doi:10.1007/s11606-023-08122-6

43.

Gemelas

JC.

Post-ACA Trends in the US primary care physician shortage with index of relative rurality. J Rural Health. 2021;37(4):700-704. doi:10.1111/jrh.12506

44.

Markowski

Wallace

Ndumele

CD.

After 50 years, health professional shortage areas had no significant impact on mortality or physician density. Health Aff. 2023;42(11):1507-1516. doi:10.1377/hlthaff.2023.00478

45.

Kandrack

Martsolf

Reid

Friedberg

MW.

Primary care physician migration patterns and their implications for workforce distribution. J Gen Intern Med. 2019;34(7):1108-1109. doi:10.1007/s11606-019-04872-4

46.

Boltz

Cuellar

Cole

Pistorese

Comparing an on-site nurse practitioner with telemedicine physician support hospitalist programme with a traditional physician hospitalist programme. J Telemed Telecare. 2019;25(4):213-220. doi:10.1177/1357633X18758744

47.

Waasdorp Hurtado

Neigut

Hoffenberg

, et al Telehealth coverage for a pediatric gastroenterology consult service at a regional hospital: an accepted, useful, and sustainable model of care. J Pediatr. 2023;262:113341. doi:10.1016/j.jpeds.2023.01.015

48.

Owen

Woodward

Drummond

, et al Using implementation facilitation to implement primary care mental health integration via clinical video telehealth in rural clinics: protocol for a hybrid type 2 cluster randomized stepped-wedge design. Implement Sci. 2019;14(1):33. doi:10.1186/s13012-019-0875-5

49.

Chandra

Hertz

Khurana

Doerfler

ME.

Collaboration between tele-ICU programs has the potential to rapidly increase the availability of critical care physicians—our experience was during coronavirus disease 2019 nomenclature. Crit Care Explor. 2021;3(3):e0363. doi:10.1097/CCE.0000000000000363

50.

Hilker

Mancho

Srivatsava

, et al A proof-of-concept model for implementing a “smart-NICU” to improve infant mortality. J Intensive Care Med. 2024;39(10):974-979. doi:10.1177/08850666241247532

51.

Tisdale

Burnett

Rogers

Nelson

Heyworth

Zulman

DM.

An approach to evaluating the impact of virtual specialty care: the Veterans Health Administration’s clinical resource hub as case study. JAMIA Open. 2025;8(3):ooaf038. doi:10.1093/jamiaopen/ooaf038

52.

Rosen

Beilstein-Wedel

Shwartz

Davila

Gurewich

Racial and ethnic and rural variations in access to primary care for veterans following the MISSION act. JAMA Health Forum. 2024;5(6):e241568. doi:10.1001/jamahealthforum.2024.1568

53.

Penn

Bhatnagar

Kuy

, et al Comparison of wait times for new patients between the private sector and United States Department of Veterans Affairs Medical Centers. JAMA Netw Open. 2019;2(1):e187096.

54.

Terry

Peck

Baker

Schmitz

The rural nursing workforce hierarchy of needs: decision-making concerning future rural healthcare employment. Healthcare. 2021;9(9):1232. doi:10.3390/healthcare9091232

55.

Feyman

Asfaw

Griffith

KN.

Geographic variation in appointment wait times for US military veterans. JAMA Netw Open. 2022;5(8):e2228783. doi:10.1001/jamanetworkopen.2022.28783

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB