Abstract
Objective:
The purpose of this study was to explore telemedicine use and obtain actionable recommendations to improve telemedicine user experience from a diverse group of patients and providers.
Methods:
We interviewed adult patients and primary care providers (PCPs) across three National Patient-Centered Clinical Research Network (PCORnet) sites in New York City, North Carolina, and Florida. Both patients and providers could participate via phone or videoconferencing; patients could complete the interview in English or Spanish. Spanish interviews were conducted by a member of the research team who spoke Spanish fluently. Interviews were audio-recorded, transcribed verbatim, and when necessary, professionally translated.
Results:
We interviewed 21 PCPs and 65 patients between March and October 2021. We found that patients' and providers' perspectives on ways to improve the telemedicine experience focused on three recommendation themes: (1) expectations of care provided via telemedicine, (2) innovations to support usability, and (3) alleviation of physician burden. Key recommendations were related to expectations regarding (1) care provided, for example, adding educational content for the patients, and clarity about long-term payment models; (2) support innovation to improve telemedicine usability, for example, providing patients with remote monitoring devices, integrating in-home testing and nursing evaluation; (3) and reduce physician burden, for example, virtual rooming, reimbursement of time spent outside of the telemedicine encounter.
Discussion:
Primary care patients and providers see merit in telemedicine. However, both groups recommended novel ways to improve the quality of care and user experience. Findings from this article suggest that policymakers would be best served by addressing current gaps in patient digital literacy by creating technical support strategies, and gaps in telemedicine reimbursement to present an equitable form of payment.
Introduction
Telemedicine use has rapidly expanded, and it is now an integral part of health care delivery. 1 It has proven to be instrumental in connecting physicians with patients during the COVID-19 pandemic. 1 Benefits of telemedicine include increased access to care, reduced waiting and travel time, and providing more options and flexibility for patients. 2 For providers, it can reduce crowding in waiting rooms and allow them to care for a wider patient population including those in remote areas. Although telemedicine has been useful in connecting physicians and patients, its sudden large-scale adoption in the face of the COVID-19 pandemic exposed challenges associated with its utilization, which can result in poor user experiences.
One barrier is the lack of telemedicine training and education for physicians, health care workers, and patients. A second important barrier involves inequitable access to broadband internet and devices to use it so that telemedicine may worsen health inequities. 3 Both these barriers contribute to the user (both patient and provider) experience of telemedicine.
User experience is defined as “a person's perceptions and responses that result from the use or anticipated use of a product, system or service.” 4 Positive user experience is critical for the adoption, acceptability, and effectiveness of telemedicine.5,6 User experience is closely associated with patient satisfaction, an important quality-of-care indicator. Multiple studies have identified important factors associated with improved patient satisfaction with telemedicine such as ease of use, low cost, better communication, and decreased travel time.2,7–14
Another study reported that while patients found telemedicine to be less stressful, associated challenges included time lag, video freezing, uncertainty in virtual waiting room, technology problems (some needing transfer to phone), and unclear expectations leading to poor patient satisfaction.15,16
In parallel, studies have shown that provider satisfaction with telemedicine is associated with having administrative support and reliable technology, being able to provide input in its development, ease of use, and adequate reimbursement.17–21 Provider satisfaction is closely linked to provider acceptance, which has been found to be the most important factor determining success of telemedicine. 22 Providers have been less satisfied with telemedicine as compared with in-person visits due to perceived reduced doctor-patient communication. 23 The perceived ease of use and usefulness of telemedicine services are dominant factors affecting provider satisfaction. 24
Despite prior studies analyzing provider and patient user experiences with telemedicine, there is a critical gap around actionable recommendations from end-users to improve the user experience. Improving user experience for both providers and patients in various settings (urban, suburban, rural) and age groups is critical for wider telemedicine acceptance and success. The purpose of this study was to explore telemedicine use and obtain actionable recommendations to improve telemedicine user experience from a diverse group of patients and providers.
Methods
Participants
We identified adult patients and primary care providers (PCPs) across three National Patient-Centered Clinical Research Network (PCORnet) sites in New York City (urban), North Carolina (suburban), and Florida (rural). Using a definition adapted from the Medicare specialty designation, we defined adult primary care as practices in the fields of general practice, family practice, ambulatory internal medicine, preventive, and geriatric medicine.
Based on a sampling frame of 250 primary care practices and with the help of recruited clinician champions, we recruited participants (i.e., providers and patients) between March and October 2021 through several methods, including emails, patient registries, flyers, clinician referrals, and snowball referrals from participants. The clinician champions did not participate in the interviews.
Eligible patients were 18 years, English- or Spanish-speaking, able to participate via telephone or videoconferencing, and had at least one chronic disease diagnosis. Eligible PCPs worked in primary care at one of the recruitment sites. Maximum variation sampling 25 was used to sample participants of different ages, races, ethnicities, geographic locations, and levels of technology experience.
To ensure diverse representation, the study team developed a screening checklist to ensure eligibility assessments were being conducted uniformly across sites and quotas to avoid over-sampling certain groups. The study protocol was approved by the Biomedical Research Alliance of New York institutional review board.
Measures
Semi-structured interview guides were developed in collaboration with our stakeholder board, which included patients, providers, payers, and information technology experts. Research staff conducted individual interviews asking about participants' experiences with telemedicine during the pandemic. Both patients and providers could participate via phone or videoconferencing; patients could complete the interview in English or Spanish. Spanish interviews were conducted by a member of the research team who spoke Spanish fluently. Interviews were audio-recorded, transcribed verbatim, and when necessary, professionally translated.
Analysis
Coders developed code keys for providers and patient transcripts based on a priori domains from the interview guides and emergent codes. Three researchers (K.A., K.A.M., J.J.L.) coded transcripts independently and met to compare codes and resolve discrepancies.
Stakeholder board did not aid in the coding or results interpretation. Data were analyzed using interpretive description, 26 an approach previously applied to health care experiences. 27 Iterative analysis was conducted in parallel with recruitment, and recruitment concluded when data saturation was achieved. 28 Final codes were captured using Dedoose Version 9.0.46 (Los Angeles, CA).
For the current study, we retrieved all text with codes related to recommendations for improvement in telemedicine. Two domain expert researchers (S.K., P.C.) independently ranked the novelty of all codes on a 3-point scale (1 = low, 3 = high). Novelty was defined as innovative ideas, based on domain experts assessment, that can improve the telemedicine user experience. Then, the average scores were calculated and those codes with three were included in this analysis, which represented high novelty.
The mean was used instead of the median since there were no outliers in the ranking since only two domain experts ranked the codes. Those codes were then categorized into themes based on patient and provider-based reported outcomes from the Benson framework. 29 The Benson framework is a comprehensive taxonomy of short generic measures covering both patient-reported and provider-reported outcomes, which enables the categorization of participant responses into mutually exclusive themes.
Results
We interviewed 21 PCPs and 65 patients between March and October 2021. Of the patients, 60% were female and 42% self-identified as White, 25% as Black, 23% as Hispanic, 9% as other, and 1% as Asian. Half were between the ages of 41–65 years, 26% were <40 years, and 22% were >65 years. Two of the interviews were conducted in Spanish. Of the PCPs, 62% were female and 48% self- identified as White, 24% as Asian, 14% as Hispanic, 9% as Black, and 5% as other.
The majority were between 41 and 60 years, with 29% <40 years and 14%> 60 years (Table 1). Patients and PCPs were recruited uniformly from each of the three sites in New York, Florida, and North Carolina. Among the 21 PCPs, 7 were recruited from New York, 8 from Florida and 6 from North Carolina. Among patients,—were recruited from New York,—from Florida, and—from North Carolina. On average, patient interviews lasted 20–25 min, whereas provider interviews ranged from 30 to 40 min.
Demographics of Participants
Only patient participants were asked about their primary language, and only provider participants were asked about the type of practice they are working in. Patient participants were asked about their primary language and whether they preferred to take the interview in English, or Spanish. Four participants with a “Spanish” primary language selected to do the interview in English, while two selected to do the interview in Spanish. Two participants selected “Other” as their primary language but felt comfortable conducting the interview in English.
FQHC, Federally Qualified Health Centers.
We found that patients' and providers' perspectives on ways to improve the telemedicine experience focused on three recommendation themes: (1) expectations of care provided via telemedicine, (2) innovations to support usability, and (3) alleviation of physician burden (Table 2, Table 3). Four patients and five providers contributed to theme 1, six patients and five providers contributed to theme 2, and six providers contributed to theme 3.
Recommendations from Patients to Improve the User Experience Categorized by Themes: (1) Care Provided, (2) Innovation, and (3) Individual Care
Recommendations from Medical Providers to Improve the User Experience Categorized by themes: (1) Care Provided, (2) Innovation, and (3) Individual Care
AVS, after visit summary; EHR, electronic health records.
Theme 1: recommendations around the expectations for the care provided via telemedicine (Benson framework: care provided)
Four patients and five providers contributed to this theme. One provider recommended educational videos, such as “YouTube videos” (P12) for patients to improve their understanding of the telemedicine visit, especially while waiting for the telemedicine visit to begin. In addition, patients, especially older adults, may be unable to remember all the information discussed with the provider during the telemedicine visit, and with telemedicine visits, they do not receive a printed after-visit summary. One provider offered a solution to mitigate this by reminding patients to write important instructions: “can you have pen and paper ready? Because this is my main thing that I want you to do, number one, number two, number three” (P13).
The PCPs also emphasized the importance of managing patient expectations regarding insurance coverage and billable visits since telemedicine visits may not always be covered by insurance. As providers stated, the “patient gets a big bill” (P11) and “then patients are not going to be particularly happy with that” (P14). Uncertainty in telemedicine reimbursement can also hinder long-term investment to build and support a robust telemedicine infrastructure; as one provider noted, “we don't know how long we're going to be able to be reimbursed at the same rate, people are kind of unwilling to invest in that right now” (P15).
Clarity about long-term payment models for telemedicine visits from the government and private insurance companies will ensure long-term sustainability and development of innovative telemedicine platforms and improve patient care and health care access.
Telemedicine can be a means to assess home safety situations for patients. Providers voiced that educating PCPs of best-practices and integrating more telemedicine functions, beyond audio/video functions, can expand the unique opportunities offered by telemedicine visits compared with routine clinic visits. Patients offered several recommendations to support PCPs: “if the provider said turn your volume down; I'm going to ask you something—are you in a safe place and you can nod your head or shake your head” (P9).
Another patient suggested using “chat function [to] gauge people's safety situations” (P10). Patients also expressed interest in staying connected with their PCP via “virtual telemedicine” (P6) while travelling and “group meetings” (P7) with the ability for other family members to join remotely.
Theme 2: recommendations to support innovations to improve telemedicine usability (Benson framework: innovation)
Five providers and six patients contributed to this theme. Several providers recommended that patients be provided with devices (“some sort of universal physical exam technology that everybody could get” or a “kit of easy-to-use devices” [P22]) to facilitate remote monitoring of their chronic conditions. These could be “the tools to track their chronic conditions at home” (P20). Similarly, one patient recommended an affordable vital monitoring kit that “can read the temperature off the screen [and] read your heart rate” (P8) to improve telemedicine visit care. Providing these tools to patients will also enable providers to conduct more comprehensive investigation; as a provider mentioned, with such tools, the “doctor would be more comfortable” (P22).
Patients also expressed that integrating in-home testing and nursing evaluation into the telemedicine visit would improve the telemedicine experience, especially for patients with “limited mobility or disabilities” (P1). Patients also saw benefits with an in-home visit by a “nurse to check your vitals, check your lungs at home before your appointment” (P3). One patient recommended the addition of a “narrative interpretation” (P2) of test results to make them easier for patients to understand.
During telemedicine visits, ancillary staff may not be available for pre-visit medication reconciliation. This causes providers to spend valuable and limited telemedicine visit time reviewing patients' home medications. One provider suggested using an automated system for virtual rooming to “get everything all set up.… they could even check their medications or put in the chief complaint” (P16).
Another important recommendation from providers is integrating telemedicine interfaces with institutional electronic health records (EHRs) to make the interface more user-friendly and reduce click burden. One provider recommended, “as much as [telemedicine] can be integrated into the EHR and again one click” (P18). There is also a need for innovation in designing and creating intuitive “care pathways” (P19) built into visits that can also serve as a visual reminder of “important things without having to think of it every time, like oh, we haven't done inhaler teaching” (P19).
On the other hand, patients recommended telemedicine functionality that would allow for “sharing screens more often” (P5), such as lab results (“perhaps they could put the screenshot in my lab results” [P4]) and links for information provided to patients during a visit: “So, if somebody needs a pamphlet for something, they will just have those links accessible” (P5).
Theme 3: recommendations to alleviate physician burden with telemedicine (individual care)
Six providers contributed to this theme. Technology issues can negatively impact the telemedicine user experience and make it more expensive if providers need to provide “tech support” (P17) during the visit. Similarly, inadequately trained care assistants may not be “successful in offering support for how to use the interface” (P21) to the patients. Improving technical training for care assistants for simple issues and adequate technical support for complex issues can improve telemedicine visits and care.
Although physical examination is an important part of clinical evaluation, the ability to perform detailed physical examinations is limited in telemedicine. The current technologies used for physical examination are inadequate. This adds to the physician's burden by spending more time on subjective and observational assessments to compensate for physical examination limitations.
One provider stated, “I don't think that there's a significant comfort amongst doctors for physical exam technology” (P25). Effective integration of state-of-the-art remote examination technologies can improve providers' trust in and satisfaction with doing telemedicine well.
The PCPs do not always get credit for time spent on important patient care activities like reviewing labs, imaging, and updating patients. One provider noted that the ability to set aside time for things like “telehealth billed phone calls to talk over test results” (P26) can allow providers to see fewer patients. Building a system that “gives the provider credit for their time” (P23) spent taking care of the patient outside the clinical encounter can facilitate better focus on quality (over quantity) and can significantly reduce provider burnout.
Back-to-back telemedicine visits do not allow PCPs additional time to address patients' complaints. Also, intermittent notifications about the next visit during an “emotionally charged visit” (P24) can distract providers from addressing patients' concerns. One provider suggested that having “more breaks built in between visits” (P24) can help address this issue.
Discussion
This multi-site study investigated the perceptions of patients and providers regarding ways to improve the user experience during telemedicine encounters. We report actionable recommendations from end-users based on their experiences with telemedicine visits during the COVID-19 pandemic. Recommendations were categorized into three main themes (1) expectations for the care provided via telemedicine, (2) innovations to improve telemedicine usability, and (3) alleviation of physician burden in telemedicine.
To enhance the quality of care provided, patients suggested rethinking the current telemedicine visit protocol to include domestic violence situations. Recommendations for providers to always ask the patient if they are in a safe place and then, to allow patients to communicate back in verbal or non-verbal cues (head shake, type in chat, etc.) were introduced. Another recommendation was to provide at-home lab options to enable patients to send their results to their provider before their telemedicine visit.
At-home kits can help providers better understand the patient's condition and hence, improve their decision-making abilities in the virtual space. The ability for patients to access provider notes and after-visit summary can improve adherence to the care plan.
Patients reported that more innovation is still needed to improve telemedicine usability. Patients stated that the ability to screen share during a telemedicine visit can empower patients by allowing providers to share educational resources to help with self-management or lab results to educate patients about their health status. Providers also suggested that integrating the telemedicine platform into the EHR can improve workflow automation and reduce documentation burden.
With regards to recommendations for reducing physician burden when using telemedicine, providers suggested integrating breaks in between telemedicine visits to allow providers to recalibrate especially after telemedicine visits where unpleasant news were discussed. Also, providers recommended reimagining how systems could better assess providers' time spent in patient care for reimbursement purposes. For instance, time spent on the phone for team coordination or resolving technical issues are not billable although they are clinically relevant.
To ensure meaningful implementation of telemedicine, decision makers and policy makers are encouraged to reconsider how providers' time is accounted for in the reimbursement cycle. It is not enough to bill for only the time spent in the telemedicine encounter without accounting for the supplemental tasks pre- and post-visit.
Another recommendation to reduce physician burden was to provide telemedicine usability support such as having ancillary staff call patients before their telemedicine visit to review medications and obtain necessary information. However, care assistants are often not trained to provide technical assistance to patients with limited digital literacy. Several providers indicated that lack of usability support for patients hinders the telemedicine experience and has cost implications due to prolonged visit duration. More ancillary support will allow physicians to utilize their valuable time for clinical evaluation, counseling, and patient support.
Previous studies that offered recommendations on telemedicine best-practices were based on expert opinions and mainly focused on implementation, policy, and visit etiquette.30–32 However, they were lacking in direct recommendations from end-users on the optimization of the telemedicine user experience. We report specific recommendations to improve the telemedicine user experience by improving the quality of care provided (i.e., adding educational content for the patients, managing patients' expectations regarding insurance coverage, clarity about long-term payment models, establishment of protocols to assess domestic violence); supporting innovation to improve telemedicine usability (i.e., providing patients with remote monitoring devices, integrating in-home testing and nursing evaluation); and to reduce physician burden (i.e., virtual rooming, more ancillary as well as technical support, reimbursement of time spent outside of the telemedicine encounter).
This study has several limitations. Although we worked systematically to identify and recruit diverse patient and provider participants, our patients' perspectives may reflect those of individuals more engaged with the health system. In addition, despite our efforts to recruit Spanish-speaking patient participants, we were only able to conduct two interviews in Spanish and did not include other languages as an option. Thus, we cannot describe the experiences of other patients whose communication might be even more affected in virtual settings.
Though our study findings are based on qualitative interviews and might not be generalizable to a population beyond primary care, they provide meaningful insights into patients' and providers' experience and suggestions to improve telemedicine. Of the three sites, only one site recruited telemedicine champions and no information was obtained about the number of sites they represented. It is plausible that recruitment through champions may introduce bias in the sample at that site.
Similarly for the other two sites, no data were obtained regarding how many practices were represented. For each provider, we recorded the practice type but not the exact practice that the providers came from as it was not responsive to the research question.
Conclusion
In summary, primary care patients and providers see merit in telemedicine. However, both groups recommended novel ways to improve the quality of care and user experience. Key recommendations were related to expectations regarding (1) care provided, for example, adding educational content for the patients, and clarity about long-term payment models; (2) support innovation to improve telemedicine usability, for example, providing patients with remote monitoring devices, integrating in-home testing and nursing evaluation; (3) and reduce physician burden, for example, virtual rooming, reimbursement of time spent outside of the telemedicine encounter.
Findings from this article suggest that policymakers would be best served by addressing current gaps in patient digital literacy by creating technical support strategies, and gaps in telemedicine reimbursement to present an equitable form of payment to providers.
Footnotes
Acknowledgment
We thank the members of the telehealth stakeholder board and those we interviewed for their expertise and time.
Author Disclosure Statement
No competing financial interests exist.
Funding Information
This work was funded by the Patient-Centered Outcomes Research Institute (PCORI), grant COVID-2020C2-10791 (Ancker and Kaushal, MPIs). The statements in this work are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee. The funder/sponsor did not participate in the work.