Abstract
Telehealth has been shown as a tool to improve health access and outcomes in rural areas. There is less literature examining the usability and utility of telehealth by rural and low-income populations. Existing literature focuses on examining telehealth usability for specific telehealth platforms and specific use cases. There is minimal literature broadly examining telehealth usability in rural and low-income populations. Using the Telehealth Usability Questionnaire in conjunction with demographic questions, we examined telehealth usability in the Wyoming Medicaid population. Additionally, we explored whether factors such as rurality, age, gender, and whether the visit was behavioral, affected usability scores. (1) We found that overall, usability scores were high in all cases. (2) Wyoming Medicaid clients with behavioral visits demonstrated statistically significant increased usability scores compared to the already positive scores in patients with nonbehavioral visits. (3) The other variables we examined did not show any statistically significant differences in usability scores. (4) These results demonstrate broad usability of telemedicine in a rural and low-income population and may be used to justify expansion of telehealth services.
Introduction
Rural Americans are a group with high healthcare needs. Sixty-one million Americans live in rural areas. 1 They tend to be older, poorer, and have worse health outcomes compared to their urban counterparts. Unfortunately, these same rural Americans have less access to medical care. 2 Barriers to healthcare in rural areas include geographic and transportation issues, fewer healthcare providers, and increased financial costs to obtain care. These financial considerations are not limited to the costs of travel, absenteeism from work, and the cost associated with employing caregivers. 3 Financial hardships can be particularly burdensome for rural Americans with limited financial resources, like the Wyoming Medicaid population.
The state of Wyoming is overwhelmingly rural. According to the 2020 Census, the median household income is $72 495. Additionally, for persons over the age of 25, 93.9% were high school graduates or higher and 29.0% had a bachelor's degree or higher. Wyoming is the tenth largest state in the United States by area, but is the least populated, with 581 381 residents in 2022. The 2020 Census showed that 22.2% were under the age of 18 and 19.2% were over the age of 65. Of its 23 counties, 4 are considered rural (6-12 people per square mile) and 17 are frontier (fewer than 6 people per square mile). According to the US Health Resources and Services Administration (HRSA), Wyoming has 47 primary care Designated Health Professional Shortage Areas (HPSA). Additionally, the entire state is considered a HPSA for mental health. 4
One approach to addressing healthcare access in rural, low-income populations is with telehealth. The US Department of Health and Human Services Centers for Medicare and Medicaid Services (CMS) defines telehealth as “professional services given to a patient through an interactive telecommunications system by a practitioner at a distant site.” 5 Conceptually, telehealth could address issues of fewer local healthcare providers and geographic and transportation problems. Telehealth improves access to healthcare, health services, and access to providers, especially in rural, geographically isolated settings6,7 where there is often a lack of specialized health services. 8 Behavioral health accessibility is notably augmented through telehealth both during the pandemic 9 and specifically for rural populations.10,11 Telehealth usage in rural settings is associated with decreased direct and indirect costs to the patient in terms of travel cost and time, 12 as well as decreasing staffing costs and improving physician recruitment and retention. 13
As a rural state, Wyoming Medicaid has long valued telehealth, covering services at full-parity with in-person visits since 2002. All outpatient visits may be provided by telehealth. With strong legislative support, Wyoming has focused on strategic initiatives to advance telehealth including provider training, community outreach, a statewide telehealth network, and various collaborations with key stakeholders. Wyoming Medicaid is a fee-for-service program administered by the Division of Healthcare Financing within the Wyoming Department of Health. In 2022, Medicaid enrollment in Wyoming was 86 574. These included 43 701 children and 4709 pregnant people. Wyoming has not expanded its Medicaid eligible population. Like other health plans across the country, Wyoming Medicaid saw its telehealth usage balloon during the COVID pandemic. The highest usage was 72 654 visits in 2021 and 43 409 visits in 2022. For context, the overall claim count for outpatient services was 198 052 in 2020, 213 796 in 2021 and 239 859 in 2022. Furthermore, in 2022, there were 618 providers of telehealth services and 11 347 unique Medicaid recipients of the services.
Previous studies have supported telehealth use in the Medicaid population. A recent study utilizing data from 2011 of Medicaid recipients in multiple states demonstrated those in rural areas were more likely to use telehealth than their urban counterparts. Nonetheless, the overall usage rates in this study population were low. 14 In addition, the Office of Health Policy notes the highest rates of telehealth visits were among Medicaid recipients, but that disparities existed among audio versus video service utilization. For patients that are low-income, elderly, or rural, barriers of access to quality telemedicine continue to persist. 15
This presents a question of usability of telehealth in the rural and low-income population. Potential barriers might include broadband and technology access, technological literacy, challenges for those with disability and potential negative attitudes toward telehealth. Broadband access continues to pose a challenge, with rurality, lower income, and advanced age associated with decreased broadband adoption. 16 According to the Federal Communications Commission, in Wyoming, there are several areas in the state lacking broadband and cellular coverage. Infrastructure offered by cellular providers is not uniform. The 2020 Census stated that 88.2% of households had a broadband internet subscription. As telehealth develops as a modality to provide healthcare, it is important to understand how patients in demographics that lag in broadband access view their experiences to improve upon existing telehealth practice with the aim of ensuring long-term use and expansion.
Previous studies have assessed usability with the Telehealth Usability Questionnaire (TUQ). The TUQ is a validated instrument to assess user usability of the telehealth platform, including usefulness, ease of use and learnability, interface quality, interaction quality, reliability (error troubleshooting), and satisfaction and future use. 17 The TUQ is the most reported instrument in the published literature for this purpose. 18 Previous studies have used the TUQ to evaluate telehealth systems, 19 telehealth-based programs,20,21 and telehealth for a variety of different specialty areas and aspects of healthcare.22,23 To date, no study has specifically evaluated the rural, low-income population on such a broad scale. The current study expands on the existing literature and provides a large-scale evaluation of telehealth usability and satisfaction among a rural and low-income population. We sought to perform a cross-sectional observational study to determine satisfaction and usability of telehealth in the Wyoming Medicaid population as primary outcomes. Additionally, we sought to determine factors that correlated with usability as secondary outcomes.
Methods
The target population and the study population included Wyoming Medicaid recipients or their guardians who received at least 1 telehealth visit in the 36 months prior to the survey date. This was determined using claims data. In 2020, 2021, and 2022, there were 9220, 7303, and 3595 patients, respectively, who received 2 or more telehealth visits. The respondent population included those who responded to the survey. A text message was sent to the telephone number(s) on file asking the recipient or guardian to follow a link to a web-based survey. The text was sent 3 times at 1-week intervals. The web-based survey included the TUQ, demographic questions, and if the visit was a behavioral health visit. The survey was adapted for a handheld experience for mobile users. The survey was shared only in English, as only 5% of the population reports speaking Spanish, the second most spoken language. The survey was completely anonymous and optional, and posed little to no risk to respondents. The name and phone number of an appropriate person to contact for more information about the study appeared on the survey. Informed consent was not required.
Demographics included age, self-identified race and gender and rurality. Rurality was defined using the Office of Management and Budget definitions of Metropolitan, Micropolitan, or neither. 24
Summary statistics were calculated for the demographics of survey respondents. For each respondent, TUQ questions were scored on a scale of 1 (strongly disagree) to 7 (strongly agree) and averaged for each usability component and for the whole scale. Missing responses were allowed in scoring (if a respondent left a question blank, their other responses were used to produce averages). Following scoring of individual responses, the researchers obtained the median, interquartile range (IQR), and bootstrapped confidence intervals (CIs) of the median (using the percentile method) for each TUQ usability score and the overall score. Kruskal-Wallis tests
25
were performed on each TUQ usability component and the overall score for each demographic to test for differences in response distributions between demographic groups. As needed, the Dunn test
26
was used as a post hoc measure to identify which specific demographic groups responded differently to the survey. Dunn tests incorporated the Bonferroni correction, which adjusts the
Results
After review of the potential study population, 12 031 recipients identified as receiving at least 1 telehealth visit during the study period. Texts were sent with 10 452 (86.9%) identified as received and 1579 rejected due to an unreachable number. Of those that received the request, 490 recipients or guardians responded (4.7%). The average time to complete the survey was 3 min and 17 s (no median value was obtained).
Compared to the population overall, survey respondents were more likely to be female, Caucasian, and older (Table 1). Women represented 77.8% of the respondent population compared to 61% of the survey population. This may be due to women being respondent guardian caregivers to children and disabled Medicaid clients. Caucasians represented 74.5% of the respondent population and only 57.2% of the survey population. All measured levels of rurality were well-represented by survey respondents, with 31.6% from a city population of 10 000 or less, 37.3% from a city population of 10 000 to 50 000, and 28.9% from a city population of over 50 000 (Table 1). Most respondents had not used telehealth for a behavioral health visit (65.0%; Table 1). And 7475 original survey invitations were linked to behavioral telehealth encounters.
Demographic Characteristics of TUQ Survey Respondents Compared to the Population of Wyoming Medicaid TH Participants.
Note: TUQ respondents came from the population of Wyoming Medicaid TH participants and used telehealth in the previous 36 months.
Items with a dash (-) did not have data available. TUQ survey respondents had different responses choices for age than they were categorized by in Medicaid data, preventing direct comparisons. Medicaid data did not include information on city population size or behavioral health visits.
Abbreviations: TH, telehealth; TUQ, Telehealth Usability Questionnaire.
Respondents generally gave high scores for the TUQ overall and to individual usability components. On a scale of 1 to 7, respondents gave the TUQ an overall median score of 5.76 (CI: 5.48, 5.95). The highest scores were related to interaction quality and satisfaction, both tied with median scores of 6.00 and CIs of 6.00, 6.25. Respondents gave the lowest score to the reliability usability component (median: 5.00, CI: 4.67, 5.33; Table 2).
Median Score, IQR, n, and 95% CI for TUQ Survey Responses From Wyoming Medicaid Participants Who Used Telehealth in the Previous 36 Months.
Scores were calculated by usability component and overall.a
TUQ component scores are on a scale of 1 to 7, with 7 being the most positive.
CIs are obtained through percentile bootstrapping.
Abbreviations: CI, confidence interval; IQR, interquartile range; TUQ, Telehealth Usability Questionnaire.
When comparing TUQ overall and component score distributions by demographic, there were statistically significant differences seen among those who had a behavioral health visit (Table 3). For those who had a behavioral health visit versus a nonbehavioral health visit, there were differences in the distributions of the overall score, interaction quality, satisfaction, usefulness, interface quality, and reliability (Table 3). Medicaid recipients who used telehealth for a behavioral health visit had consistently higher scores on the TUQ than for Medicaid recipients who used telehealth for nonbehavioral health visits, although scores were generally high for both groups (Table 4). The study found no significant differences in the TUQ scores between survey respondents living in metropolitan, micropolitan, or other (more rural) areas.
Kruskal-Wallis Test Results Comparing Differences in TUQ Usability Component Scores and Overall Scores by Demographic.
Respondents were Wyoming Medicaid participants who used telehealth in the previous 36 months.
Gender: male, female.
Race/ethnicity: Caucasian, Hispanic, Native American, all other races/ethnicities, unknown.
Age: 18-34, 35-44, 45-54, 55-64, 65-74, 75, and over, unknown.
City size: 10 000 or less, 10 000-50,000, 50 000 or more, unknown.
Behavioral health visit: Yes (used telehealth for behavioral health), no, unknown.
*Statistically significant at a 0.05 level.
People identifying as “other” for gender were excluded due to their small group size (n = 5).
Abbreviation: TUQ, Telehealth Usability Questionnaire.
Differences in TUQ Component Scores a and Overall Scores by Whether or Not the Person Received a Behavioral Health Visit.
Respondents were Wyoming Medicaid participants who used telehealth in the previous 36 months.
*Statistically significant at a 0.05 level.
TUQ component scores are on a scale of 1 to 7, with 7 being the most positive.
CIs were obtained through percentile bootstrapping.
Abbreviations: CI, confidence interval; TUQ: Telehealth Usability Questionnaire.
No differences in the score distributions were observed for any specific age group during post hoc testing once the Bonferroni correction for multiple comparisons was applied. There were no statistically significant differences in TUQ scores based on gender, race/ethnicity, or city size (Table 3). For gender, the overall median TUQ score was 5.76 for both males and females. For race/ethnicity, the overall median TUQ score was 5.81 for Caucasian respondents, 6.12 for Hispanic respondents, 5.24 for Native American respondents, and 5.33 for respondents of all other races/ethnicities. For age, the overall median TUQ score was 5.90 for respondents aged 18-34, 6.12 for respondents aged 35-44, 6.07 for respondents ages 45-54, 5.32 for respondents aged 55-64, 5.5 for respondents aged 65-74, and 4.70 for respondents aged 75 or older. Although the highest age range (75+) tended to have lower median scores on the TUQ, this was likely not significant due to a small group size (n = 23).
Discussion
Our study shows positive ratings of telehealth usability across all categories in the survey, regardless of rurality, age, gender, and other factors. This suggests that Wyoming Medicaid telehealth users are generally satisfied with their experiences. Positive responses from people over the age of 65 is meaningful since there are significant barriers to telemedicine usage faced by older adults including hearing difficulty, language barriers and difficulty with technology. 27 The similarity in ratings across age groups underscores the utility telehealth provides to rural Medicaid patients. There are nonsignificant differences between age groups and rurality. Areas with 50 000 or more had higher scores than the other 2 categories, and older adults had lower scores in multiple categories. This could be due to factors such as poor connectivity or barriers to telemedicine faced by older individuals as discussed earlier. Increased sample size may help to clarify the relationship between age and telehealth rating.
Respondents that used telehealth for behavioral health visits responded more positively on the TUQ than other respondents in every category outside of ease of use and learnability. This is consistent with previous studies demonstrating a preference for and usability of behavioral health visits delivered over telehealth.28,29 An important point to note is that behavioral healthcare often involves a higher number of sessions over time, which may lead to increased familiarity in turn leading to a higher level of satisfaction with telehealth. Efforts to expand telehealth-based behavioral health services to rural and low-income patients may be worthwhile from a patient perspective given the high usability scores demonstrated in our study.
This study reinforces the use of telehealth as a practical modality to provide health services to rural and low-income populations. Policies encouraging the use of telehealth during the COVID-19 pandemic should be maintained, as they directly impact patient access to care. At the system level, further efforts are needed to facilitate the expansion of telehealth in rural areas. Further efforts to facilitate telehealth in rural areas can be made. Looking toward the future, investments in broadband infrastructure might improve access and user satisfaction due to uninterrupted, high-quality telehealth services. Furthermore, parity reimbursement for telehealth services, and medical practice licensure for interstate telehealth providers will be key in long-term telehealth sustainability. Additionally, increased awareness of telehealth options among rural residents and providers is necessary to promote its continued adoption. Last, focusing on patient satisfaction with telehealth is essential to increase the likelihood that patients utilizing these services continue to do so.
Strengths of this study include the large number of respondents and the novelty of using the TUQ to examine a rural and low-income population. These strengths lay a strong foundation for further telehealth satisfaction research in other rural areas. The study also supports existing literature and broadens the scope of documented telehealth satisfaction. Its findings contribute to the practical implementation of telehealth and its expansion within modern healthcare. This broad look at telehealth usability and satisfaction suggests that patients find the services relatively useful, easy to learn, and are overall satisfied with telehealth services.
Limitations
The 4.7% response rate can lead to a nonresponse bias. The respondents were older, white, and female than the overall study population. This might have implications for the generalizability of the study. Additionally, there are inherent limitations in survey studies such as ours where survey respondents self-selected, and invitations were sent to smartphones. The respondents may have been more technologically savvy than the study population which may have inflated scores. One way to improve the response rate in future studies is to utilize multiple survey modalities or offer an incentive to participate. Other factors leading to nonresponse bias may be loss of cellular access or language barriers. Given these limitations, we observed that satisfaction by gender was equal between men and women, older individuals in our study were less satisfied than younger age groups, and Caucasians were less satisfied than Hispanics but more satisfied than Native Americans.
Since visits from 36 months prior were included in the study, some participants may have exhibited recall bias. In future studies, satisfaction surveys can be sent immediately following the visit. The patients in this study may have established relationships with their physicians which may lead them to view their care through telehealth more favorably. Telehealth platforms used by survey participants were not uniform. Certain platforms and modalities may lead to better or worse user experiences. Respondents were not asked to report how many telehealth visits they had. Patients with a greater number of visits and therefore familiarity may have increased satisfaction. The study did not examine urban Medicaid populations or privately insured populations. Future directions could include utilizing the TUQ to examine these populations and exploring any potential differences between rural usability.
Conclusions
Wyoming Medicaid recipients who had a telehealth visit and responded to the TUQ had overall positive views of telehealth. There were no statistically significant differences in ratings regarding rurality, gender, age, or race/ethnicity. The results demonstrated people accessing telehealth for behavioral visits gave higher scores in most usability categories, consistent with previous studies. This study provides evidence that should bolster the confidence of policy makers when expanding telehealth for rural Medicaid populations. The authors believe that efforts to expand telehealth awareness and availability in rural communities would greatly improve health outcomes.
Footnotes
Data Availability Statement
Please contact paul.johnson@wyo.gov for data used in this study.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Ethical Approval
This study was approved by the Wyoming Department of Health Internal Review Board IRB00003618/ FWA00005562.
Statement of Human and Animal Rights
No procedures were conducted on humans or animals.
Statement of Informed Consent
Informed consent was exempt due to the survey being anonymous and participation completely voluntary.
