Abstract
The coronavirus disease 2019 (COVID-19) pandemic has caused a rapid and widespread application of telemedicine services in the outpatient setting. Prior to COVID-19, patient satisfaction was measured with Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician & Group Survey (CG-CAHPS) and was then measured with the Press Ganey telemedicine survey. Both surveys ask about a patient’s likelihood to recommend a particular medical practice, which is a useful, but imperfect, surrogate for overall satisfaction. The purpose of this analysis was to identify any changes in patient satisfaction scores with the implementation of telemedicine services. A retrospective analysis of our institution’s experience during the early months of the COVID-19 pandemic compared to the months immediately prior to the pandemic was conducted. The percent of patients with a “Top box” response to survey questions regarding their likelihood to recommend a medical practice were compared. A total of 14 430 CG-CAHPS results collected in November 2019 through February 2020 were compared to 22 009 telemedicine survey results collected between March and May 2020. In general, most medical specialties incorporated telemedicine but suffered a decrease in their patient’s likelihood to recommend a medical practice during the first few months of the pandemic. However, the magnitude of this decrease was variable by medical specialty. Physical medicine and rehabilitation and pain medicine had relatively poor scores prior to the pandemic which did not statistically change. Oncology was the sole medical specialty that continued to have unchanged high patient satisfaction scores. These data provide insights for the refinement of telemedicine.
Introduction
On March 11, 2020, the World Health Organization declared that the rapid rise of COVID-19 cases constituted a global pandemic (1). In order to mitigate the spread of the virus in outpatient medical settings, many social distancing practices were implemented or expanded in the United States, including the broader use of telemedicine services. The Centers for Medicare and Medicaid Services defines telemedicine as “a service that seeks to improve a patient’s health by permitting 2-way, real-time interactive communication between the patient and the physician at a distant site” (2). As the number of cases increased in certain geographic areas, there was a concurrent increase among the general population’s interest in telemedicine (3).
Telemedicine has several advantages and disadvantages compared to in-person visits which could alter patient satisfaction with the delivered care. Telemedicine has the ability to reduce the amount of time needed to deliver medical care, increase patient convenience, and decrease the risk of exposure to infectious agents, such as COVID-19 (4). However, there is also a lack of physical contact, technological challenges, difficulty in perceiving nonverbal cues, and challenges in performing physical examinations, all of which patients could perceive negatively. As outpatient visits with different medical specialties involve different degrees of participation, telemedicine could impact these practices in different and yet unknown ways. For example, patients seeing an orthopedic surgeon via telemedicine for knee pain may be very happy about not having to ambulate long distances but dissatisfied about not having as thorough a physical examination of their knee.
Prior to COVID-19, our institution monitored satisfaction with outpatient care by the 35-question Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) survey (5). After the onset of COVID-19, there was an abrupt decline with in-person outpatient clinic visits and a concurrent increase in the utilization of telemedicine services. Satisfaction with telemedicine was monitored by the 13-question Press-Ganey Telemedicine Survey (6). The 2 surveys are substantially different in both the satisfaction domains assessed and the language used. However, the language in questions from both surveys assessing overall satisfaction was similar; CG-CAHPS asked “Would you recommend this provider’s office to your friends and family?” while the Press-Ganey Telemedicine Survey asked, “[what is] the likelihood of recommending our service to others?” This represents a Net Promoter Score which is frequently used in surveys as a surrogate for customer loyalty, likelihood of recurrent use of services, or overall satisfaction. Within both surveys, these questions are meant to examine overall satisfaction rather than satisfaction with one particular domain. The similar language and focus allow for a reasonable comparison of these 2 questions over time.
It is not clear how COVID-19 and the increased utilization of telemedicine services have altered overall patient satisfaction with outpatient care in different medical specialties. The purpose of this article is to examine changes in overall patient satisfaction with outpatient clinical visits measured in the 4 months before the onset of COVID-19 with questions from CG-CAHPS compared to the 4 months after the onset of COVID-19 measured by questions from the Press-Ganey Telemedicine Survey stratified by medical specialty.
Methods
This survey data analysis study was approved by the School of Medicine Institutional Review Board (IRB#00251710). Survey data were aggregated from large, tertiary care, academic network composed of 5 hospitals in multiple states and multiple outpatient primary and specialty clinics. The survey data provided was the percent of survey respondents who answered with an affirmative “top-box” response to the Agency for Healthcare Research and Quality CG-CAHPS Version 2 and Telemedicine Surveys and stratified by medical specialty. Surveys were administered to all eligible persons who received medical care in either the outpatient setting or via telemedicine. The CG-CAHPS questionnaire allows for several responses to questions. “Top box” scoring indicates the percentage of survey respondents who have chosen the highest available response. Surveys are available in multiple languages. Surveys completed by patients seen by pediatric practices were not analyzed in this study.
Both surveys are electronic surveys administered with similar methods. Two days following an outpatient visit or telemedicine visit, the patient first receives a text message on their mobile phone asking them to complete the survey. Two days after that, the patient will receive an email to remind them to complete the survey if they have not already done so, and a final reminder email is sent 5 days after the first. Historically, patients at our institution complete surveys within one month of their clinical visit.
The 4 months immediately prior to the onset of COVID-19 (November 2019 through February 2020) surveyed patients with CG-CAHPS and are referred to as pre-COVID. The question “Would you recommend this provider’s office to your friends and family?” was used as a measure of overall satisfaction with the patients’ experience. The 4 months immediately following the onset of COVID-19 and the increased use of telemedicine (March 2020 to June 2020) examined the Telemedicine Survey (Press-Ganey) and are referred to as post-COVID. The data were reported as the percentage of either survey respondents who answered this question with an affirmative “top box” response.
The number of respondents to each survey was recorded. Medical specialties that did not have an average survey response rate of at least 25 per month were not analyzed.
Data for additional months during the COVID-19 pandemic were not available at the time of analysis. Low response rates of Telemedicine survey prior to March 2020 and CG-CAHPS after March 2020 precluded direct comparison of the same survey. On average, there were less than 5 telemedicine surveys per month prior to March 2020. Additionally, after March 2020, there were less than 10 CG-CAHPS surveys completed per month for the majority of medical specialties although an increase was observed in some specialties starting in June. The limited number of months and aggregate nature of the data also precluded meaningful use of comparison instruments, such as Bland-Altman plots.
Statistical Methods
Statistical analysis was carried out with GraphPad Prism, version 7.0 and Stata, version 14. Data were stratified by medical specialty. The percentage of respondents providing top-box responses in the pre-COVID and post-COVID times was compared using t tests. A P value of less than .05 was considered statistically significant. For each group, the 95% CI was determined. The percentage change from pre-COVID to post-COVID was determined by calculating 100 × (Pre-COVID−Post-COVID)/Pre-COVID.
Results
A total of 14 430 CG-CAHPS surveys that met inclusion criteria were collected between November 2019 and February 2020 and 22 009 with a response rate of 20.2% Telemedicine surveys were collected between March 2020 and June 2020 with a response rate of 26.5%.
Twenty-three medical specialties met the inclusion requirements for analysis: allergy and immunology, cardiology, dermatology, endocrinology, gastroenterology, general surgery, geriatrics, infectious disease, internal medicine, nephrology, neurosurgery, obstetrics/gynecology, oncology, ophthalmology, orthopedics, otolaryngology, pain medicine, physical medicine and rehabilitation, plastic surgery, pulmonology, rheumatology, sleep medicine, and urology. Only 5 medical specialties did not have statistically significant decreases in overall satisfaction including oncology, ophthalmology, pain medicine, physical medicine and rehabilitation, and plastic surgery. There were no statistical differences in the overall average number of survey responses per month overall or for any medical specialty.
The average result of the likelihood to recommend question from CG-CAHPS Pre-COVID was 82.319% (95% CI: 80.51-84.13) top box. The average result of the likelihood to recommend question Telemedicine Post COVID was 69.45% (95% CI: 59.76-79.16) top box. This is significantly lower than pre-COVID overall satisfaction (P = .029). There was a general decrease in overall satisfaction top box reporting by 15.62%. The average Pre- and Post-COVID scores, percentage change, and the number of survey responders’ overall satisfaction are provided in Table 1 for providers who primarily practice nonprocedurally based medical specialties and in Table 2 for providers who practice procedurally based medicine.
Percent of Patients With a “Top Box” Response When Questioned About Their Likelihood to Recommend a Noninterventional, Nonsurgical Medical Practice.a
Abbreviation: CG-CAHPS, Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey.
a Data for Pre-COVID and Post-COVID scores are reported as mean and 95% CI. % change in score is determined by 100 × (Pre-COVID−Post-COVID).
Percent of Patients With a “Top Box” Response When Questioned About Their Likelihood to Recommend an Interventional or Surgical Medical Practice.a
Abbreviation: CG-CAHPS, Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey.
a Data for Pre-COVID and Post-COVID scores are reported as mean and 95% CI. % change in score is determined by 100 × (Pre-COVID − Post-COVID).
Conclusions
The results indicate that most medical specialties were rated lower on overall patient satisfaction survey questions when telemedicine surveys were used compared to in CG-CAHPS. However, the vast majority of patients continued to have positive opinions of the overall care that they received. The degree of difference varied by medical specialty. Two specialties, oncology and ophthalmology, experienced a nonsignificant decrease in overall patient satisfaction, but remained very high in all time points. Three other medical specialties, plastic surgery, pain medicine, and physical medicine and rehabilitation, had Pre-COVID low overall satisfaction questions top box results, and while they experienced a decrease, it was not statistically different from baseline.
The use of telemedicine during the COVID-19 pandemic represents a viable and necessary tool for continuing to deliver care to patients who have been generally well received by patients (7). During the pandemic, individual medical specialties have examined how telemedicine has changed patient satisfaction but comparison across specialties has not been done (8 –10). There have also been previous attempts to evaluate patient satisfaction with telemedicine services in different outpatient settings prior to COVID-19 which, again, provided generally positive ratings (11 –13). In 2016, the specialties that most frequently self-reported use of telemedicine were radiology (39.5%), psychiatry (27.8%), and cardiology (24.1%), while the specialties with the lowest use were allergy and immunology (6.1%), gastroenterology (7.9%), and obstetrics/gynecology (9.3%) (14). Studies have identified a number of barriers to the widespread implementation of telemedicine before COVID-19 including a lack of awareness by patients who telemedicine services were even available and reluctance on the part of the patient and physician (4). Since the onset of COVID-19, our and others’ data suggest that there is now an abundant awareness of this technology and that it has been firmly embedded into the health care landscape (3).
This study has several limitations. Similar questions from 2 unique surveys were used. The CG-CAHPS survey has fewer possible responses than the Telemedicine Survey, which could increase the chances of a “top box” score in the CG-CAHPS survey. Given the relatively low response rate of the telemedicine survey pre-COVID and the equally poor response rate of CG-CAHPS post-COVID, meaningful comparison of agreement in the 2 instruments was not possible. However, the language used in both survey questions to collect information on overall satisfaction was similar enough to provide insights. The language used in more granular questions of specific facets of patient care were markedly divergent and the authors felt that reasonable comparisons could not be made. Additionally, the incorporation of specific patient comments may provide greater insight but these were not captured in our dataset. As a retrospective study, it can only describe associations but not causations. Individual patient characteristics, such as age, gender, socioeconomic status, education, could not be assessed and adjusted for the given aggregate nature of the data.
In conclusion, the recent rise in the use of telemedicine due to COVID-19 has allowed for safe health care delivery with increased convenience and decreased risk of disease transmission. Other benefits such as decreased cost and time needed to receive medical care have also been previously observed (4,7). Conversely, there are potentially dissatisfying qualities to telemedicine such as a lack of physical contact between patient and provider and decreased nonverbal communication. In general, patients surveyed with the Telemedicine Survey reported a high overall likelihood to recommend the practice which can be interpreted as high satisfaction with the outpatient care they received. However, many medical specialties experienced a decrease in responses to questions that measure overall satisfaction when pre-COVID CG-CAHPS measurements are compared with the Telemedicine Survey. The relationship between these 2 instruments will be the subject of ongoing research. Additional research will also define more granular aspects of patient satisfaction responses and identify areas that can be improved upon, particularly to tease out if satisfaction is directly related to telemedicine visits and what aspects impact patient satisfaction.
Footnotes
Authors’ Note
All procedures in this study were conducted in accordance with the Johns Hopkins Institutional Review Board. (Approval ID: 00251710). The IRB waived the requirement for informed patient consent as no identifiable data was used in the analysis. This study did not involve the use of animals.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article: None of the contributing authors have any declared conflicts of interest within 36 months of submission. This study was not associated with any grants, sponsors, or funding including institutional support, NIH, Wellcome Trust, Howard Hughes Medical Institute, or departmental or institutional funding. No copyrighted material from outside sources was used in preparation of this manuscript. This study has not been previously presented.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
