Abstract
The purpose of the current study is to examine how nonmodifiable sociodemographic, disease, appointment, management, and survey factors correlate with provider rating. This was a retrospective cross-sectional study conducted on 29 857 patient Clinician and Group Consumer Assessment of Healthcare Providers and Systems surveys collected from January 2017 to January 2019 at a tertiary eye center. We included surveys of patients aged 18 years or older, who answered at least 4 of 6 subfield questions, and completed the survey within 90 days of the appointment. The main outcome was the odds of receiving top box score (TBS) of 10/10 on the survey question regarding overall provider rating. The results showed that the variables with higher odds of TBS included higher overall appointment attendance (odds ratio [OR]: 2.66 [95% CI: 1.23-5.75], P = .013); older patient age (OR 2.44 [95% CI: 2.08-2.87], P < .001]; higher percentage of survey questions completed (OR: 2.02 [95% CI: 1.79-2.27], P < .001); better best corrected visual acuity (OR: 1.85 [95% CI: 1.3-2.64], P = .001); optometry clinic visit (OR: 1.25 [95% CI: 1.15-1.36], P < .001); having procedures (OR: 1.19 [95% CI: 1.04-1.36], P = .013), surgery scheduled (OR: 1.18 [95% CI: 1.03-1.36], P = .020], or refraction done (OR: 1.16 [95% CI: 1.08-1.25], P < .001); being seen by male providers (OR: 1.11 [95% CI: 1.04-1.17], P = .001); and having additional eye testing performed (OR: 1.06 [95% CI: 1.00-1.13], P = .048). Variables associated with lower odds of TBS included longer time to complete survey (OR: 0.42 [95% CI: 0.3-0.58], P = .001); new patient encounter (OR: 0.62 [95% CI: 0.58-0.65], P < .001); and glaucoma (OR: 0.66 [95% CI: 0.59-0.75], P < .001), cornea (OR: 0.79 [95% CI: 0.71-0.87], P < .001), or comprehensive clinic visits (OR: 0.86 [95% CI: 0.79-0.94], P < .001). Thus, nonmodifiable factors may affect the provider rating, and these factors should be studied further and accounted for when interpreting the results of patient experience surveys.
Introduction
Patient experience is the sum of all interactions patients have within the health care system, the quality of which contributes to the financial and reputational success of individual providers and medical institutions (1). Patient assessment of their experience affects patient engagement, compliance, readmission rates, and is progressively used to quantify hospital and provider reimbursements (2). The increasing importance of patient experience after the Affordable Care Act (3) has led to a wealth of studies in a variety of fields to identify the modifiable areas of improvement, including not only in physician communication, but also in hospital renovations (4), provider attire (5,6), and survey delivery methodology (7). However, the subjective nature of patient experience makes it difficult to assess its true determinants, and more studies are needed to identify new factors of interest and their relationships to each other (8).
It is important to study patient experience in ophthalmology for several reasons. There exists significant differences in the patient experience across specialties, which suggests specialties themselves may be a factor of interest (9). Expectations may be higher in ophthalmology, as vision is often a high priority for patients, and vision impairment is a significant source of social and economic burden in the United States (10). Ophthalmology clinics, like many others, face conflicting pressures to increase patient volume in the face of an aging population while maintaining quality of care and accessibility. Ophthalmology commands a greater proportion of Medicare reimbursements (payments for providers in return for services rendered to Medicare beneficiaries) compared to other specialties, with implications for both revenue and the demographic impact of age on patient satisfaction (11). In addition, ophthalmologists may serve as the first point of contact for potential new patients in a health system, providing a unique setting to improve patient engagement and general health (12).
Several existing studies in ophthalmic settings have demonstrated that such modifiable factors as communication skills (13), wait times (14,15), time spent with patients (16), accessibility (17), team cohesion (18), among others were significantly associated with better patient experience. Recognizing modifiable factors has led to implementation of various programs such as those focusing on effective communication skills (19). However, nonmodifiable factors can have significant impact on ratings as well and examining them helps us to better understand physician and health care system–patient interactions. Notably, beneficial aspects of patient satisfaction on visual outcome, such as improved patient compliance and subsequent treatment response, have been reported (20).
Many prior patient experience studies are limited by the scope of factors examined, care settings, sample size, and inconsistent methodology across studies. There have been efforts from national and global organizations to address these weaknesses. Of these, the Consumer Assessment of Healthcare Providers and Systems (CAHPS) family of surveys provides a standardized assessment of the patient experience and has been adopted by many medical institutions. Of these surveys, the Clinician and Group (CG)-CAHPS survey measures patient perceptions of care delivered by a provider specifically in an office setting. These kinds of surveys allow for more valid comparisons of the patient experience between health care institutions.
The goal of this study is to utilize the CG-CAHPS to understand how nonmodifiable factors affect provider rating in a high volume, multispecialty tertiary eye center. In particular, the current study examined how sociodemographic, disease, appointment, management, and survey variables might affect the overall provider rating.
Methods
Study Design
A comprehensive retrospective cross-sectional study was performed for 41 529 ophthalmology patients surveyed at a single tertiary ophthalmology center from January 2017 to January 2019. The study obtained approval from the Institutional Review Board. All study-related procedures were performed in accordance with the Health Insurance Portability and Accountability Act, adhered to the tenets of the Declaration of Helsinki, and complied with all federal and state laws. All returned patient surveys with a score for overall provider rating and available corresponding visit data were included. An approved modified version of the standardized CG-CAHPS survey was conducted (21). Overall provider rating was answered on a rating scale (0-10). Patients younger than 18 years were excluded. Surveys submitted more than 90 days after the visit or with less than 4 of 6 CG-CAHPS survey questions answered were also excluded. The primary end point examined was the odds of receiving an overall provider rating of 10 of 10, the “top box score” (TBS).
Patient demographic variables examined included age, gender, race, distance in kilometers from home zip code to the clinic zip code for instate patients, income level in US dollars expressed in unit folds of the poverty level, and insurance payor (Medicaid, Medicare, or commercial insurance). Income was determined by instate zip code and US census data for poverty levels ($24 600 for a family of 4) (22). Best corrected visual acuity (BCVA) of the better-seeing eye was converted from Snellen to Early Treatment of Diabetic Retinopathy Study (ETDRS) letters. Appointment-specific variables examined included provider gender, being seen in Same Day Access (SDA) clinic which serves patients with acute issues, being rescheduled from an existing appointment, being a new patient to a provider, patient outpatient appointment attendance overall and within ophthalmology, and the subspecialty clinic the patient was seen in. Attendance was defined as the difference between 100% and the rate of cancellations and no-shows.
Treatment variables included orders for consultation with another provider, dilation, eye testing/imaging, laboratory testing, medical imaging, minor outpatient procedures, refraction, and whether or not surgery was scheduled during the visit.
Survey-related variables examined included the number of days it took the patient to respond to the survey after the appointment, and percentage of survey questions completed.
Patients seen in all subspecialty clinics were included in the data set. However, only departments with greater than 3 providers were examined as variables in the final analysis to minimize the effect of individual providers.
Statistical Analysis
Statistical analysis was done in JMP Pro (version 14.2), an SAS Institute software. Relationships between individual categorical measures and TBS were quantified using contingency tables for univariate analysis. Odds ratios (ORs) were reported along with 95% CIs. Significance was set at P < .05 and was reported using the Pearson chi-square statistic due to large sample sizes. Relationships between individual continuous measures and TBS were analyzed using logistic regression for univariate analysis. Unit ORs and overall ORs were reported. The receiver operating characteristic (ROC) area under the curve (AUC) was also reported with values ranging from 0.5 to 1, with 1 indicating that the variable was a robust predictor of TBS. Predictors of TBS were also analyzed in multivariate analysis using nominal binary logistic regression. Because there appeared to be correlation between starting variables, we utilized a conservative approach of including all variables into initial multivariate analysis followed by backward elimination of nonsignificant variables to improve biasing and instability (23). Odds ratios were reported along with 95% CIs. Significant variables were reported using the Wald chi-square statistic due to large sample size, and the ROC AUC of the model was also reported.
Results
In total, 29 857 of the initial surveys fulfilled criteria to be included in the study; 73.4% of the included surveys scored a TBS of 10 for overall provider rating.
Supplemental Figure and Table illustrate the univariate ORs of TBS. Multivariate analyses are provided in Figure 1 and Table 1.
Forest plot of final multivariate analysis of patient, disease, and appointment factors on top box score provider rating.
Summary of Final Multivariate Analysis of Patient, Disease, and Appointment Factors on Top Box Score Provider Rating.a
Abbreviations: AUC, area under the curve; BCVA, best corrected visual acuity; ETDRS, Early Treatment of Diabetic Retinopathy Study; ROC, receiver operating characteristic.
a n = 29 200. ROC AUC = 0.601.
In the final multivariable model, higher overall appointment attendance (OR: 2.66 [95% CI: 1.23-5.75], P = .013), older age (OR: 2.44 [95% CI: 2.08-2.87], P < .001), and higher percentage of survey filled (OR: 2.02 [95% CI: 1.79-2.27], P < .001) were the top 3 factors associated with higher odds of TBS. Other factors predictive of TBS included better BCVA (OR: 1.85 [95% CI: 1.3-2.64], P = .001), being seen in optometry clinic (OR: 1.25 [1.15-1.36], P < .001), having a procedure done (OR: 1.19 [95% CI: 1.04-1.36], P = .013), having a surgery scheduled (OR: 1.18 [95% CI: 1.03-1.36], P = .020), undergoing refraction [OR: 1.16 [95% CI: 1.08-1.25], P < .001), being seen by a male provider (OR: 1.11 [95% CI: 1.04-1.17], P = .001), and undergoing eye tests (OR: 1.06 [95% CI: 1.00-1.13], P = .048).
Variables associated with lower odds of TBS included longer time to complete a survey (OR: 0.42 [95% CI: 0.3-0.58], P = .001); new patient encounter (OR: 0.62 [95% CI: 0.58-0.65], P < .001); and being seen in glaucoma (OR: 0.66 [95% CI: 0.59-0.75], P < .001], cornea clinic (OR: 0.79 [95% CI: 0.71-0.87], P < .001), and comprehensive clinic (OR: 0.86 [95% CI: 0.79-0.94], P < .001).
Factors that were not statistically significant included patient gender, race, income level, insurance type, and distance from home to clinic; retina or uveitis clinic visits; being seen in SDA clinic; rescheduled appointments; ophthalmology appointment attendance; and having another consultation requested, dilation, laboratory testing requiring phlebotomy, or medical imaging ordered during the visit. The ROC AUC of the final model was 0.601.
Discussion
Patient experience and perception of care have both clinical and nonclinical implications. Prior patient experience studies are often difficult to compare due to inconsistent methodology and limited sample size. Advantages of the current study include a large sample size at a large-volume multispecialty ophthalmology practice, examination of 24 different variables that may be associated with overall provider rating, and utilization of the CG-CAHPS survey specifically designed for outpatient settings (21). The current study focuses on nonmodifiable variables and their association with the overall provider rating.
Of all sociodemographic variables in our study, older patient age was the only variable that remained significant in the final multivariate analysis. Outside the ophthalmic setting, Drain reported older age was correlated with more generous provider ratings in addition to lower rates of provider defection in the primary care setting (24). Abtahi et al also found age as a significant factor for higher patient satisfaction when examining nonmodifiable patient characteristics in the orthopedic setting (25). Although there is concern that older patients may have a higher response rate, the distribution of ages in our respondents (62.9 ± 14.4 years) is not unreasonable given a higher percentage of ophthalmology patients qualify for Medicare (11).
New patient–physician encounters carried lower odds of receiving a TBS. Established patient may have more continuity and increased accessibility to quality care which may lead to many unmeasured improvements in patient experience. Additionally, provider familiarity may temper patient expectations of providers in terms of scheduling difficulties and wait times. This also supports several studies highlighting the importance of multiple aspects of the patient–physician relationship, which is more likely to develop over the course of several visits (8,16 –18,26).
In this study, patients seen in the Comprehensive, Cornea, and Glaucoma clinics were significantly less likely to provide TBSs. Subspecialty departments were used as a surrogate for disease type and these departments had comparable average BCVA of 81.38, 80.69, and 81.11 ETDRS letters (all approximately 20/25 in Snellen), respectively. High patient expectations of diagnosis or treatment for chronic conditions like dry eye syndrome may be a potential explanation (27). Long-term glaucoma treatment can be a burden for patients, and Lemij et al showed the development of surface disease in glaucoma patients was associated with higher dissatisfaction rates in patients (28). The exhaustive testing, in particular perimetry, required for glaucoma patients may also be poorly tolerated (29). Planned surgery carried higher odds to TBS in our model, suggesting that these departments might derive some of their patient satisfaction scores from surgeries. Removing the variable of planned surgery from the analysis slightly improves the lower odds to TBS in these departments but does not eliminate the significant difference. Cataract surgery is a common staple of these 3 fields and is generally associated with positive patient outcomes and satisfaction (30). These results suggest that each subspecialty department within ophthalmology might experience unique challenges which influence patient experience, highlighting the important role of the provider in establishing patient expectations (31). In addition, some issues can be actionable; for example, efforts are being made to optimize perimetry administration times by reducing the number of visual field locations tested to make these examinations more tolerable for patients (32).
There have been multiple studies of optometric services and their relationship with the patient experience, including contact lens comfort (33), contact lens counseling (34), prescribing patterns (35), and comparisons of refractive interventions (36). In addition, optometrists can serve as primary eye care providers and have an important role in the screening, management, and referral for insidious diseases like glaucoma or diabetic retinopathy (37). Patients seen in optometry clinics in our study were significantly more likely to rate a TBS in the final multivariate analysis. Comparing shared-care systems from the United Kingdom and Australia demonstrates that optometrist visits are associated with greater patient satisfaction due to shorter wait times, therefore alleviating this burden for both ophthalmologists and patients (38,39). Optometrists are generally more accessible than ophthalmologists, tend to have less volume and complex disease patients. Although our results support the value of optometric providers on the patient experience, more studies are needed to understand what other aspects of optometric care affect satisfaction outcomes.
Better BCVA was significantly associated with higher provider rating in the final multivariate analysis, suggesting that patients’ existing functional visual ability may play a role in determining their satisfaction. In addition, BCVA can estimate disease severity in some, but not all, ocular conditions. Analyzing BVCA with patient experience has not been previously examined in ophthalmic patients to our knowledge, although there have been studies which relate improved BCVA to patient satisfaction post-cataract surgery (40,41). There are many possible explanations from increased patient expectations of providers when presenting with worse disease to more profound vision loss affecting patients’ emotional well-being (42).
Orders placed for each visit were generalized into categories and analyzed to determine their effect on patient satisfaction. Dilation increases wait time and is often poorly tolerated, but it was not a significant factor in our multivariate regression model (43). Many of the different adjunctive eye tests were generalized into a single category, but each test might be perceived differently by patients (29). Although eye tests contribute to overall appointment time and wait time, patients in our study with eye tests ordered were significantly more likely to be fully satisfied with their providers, potentially as they feel they are receiving a higher level of care. Both contact lens and glasses wearers generally view glasses as a safe and effective refractive intervention, and refraction generally results in good outcomes, which may explain the significance of refraction as predictor of TBS (36).
Although not previously examined in the ophthalmic setting, male provider gender was significantly associated with higher odds of receiving a TBS in our study. Female providers have been associated with lower patient satisfaction when controlling for provider access and physician assessments in gynecology, as well as in other specialty settings (44,45). Yet, there is evidence that female providers have better patient outcomes in the hospital setting (46). Another study showed male ophthalmologists are significantly more likely than women to be reported for malpractice activity (47). Although there are many studies on physician implicit bias effects on care delivery, there is paucity of research on the effects of patient explicit and implicit bias on provider rating (48). It has been argued that provider gender, among other characteristics, may need to be controlled for in future patient satisfaction studies, including in ophthalmology (49,50).
Our study has a number of limitations. The retrospective nature of the study makes it inherently vulnerable to nonresponse bias, although this has not been a consistent concern across CAHPS studies (51). Furthermore, the experiences at a tertiary referral academic center, as in our study, may not be easily generalizable to other practices. Our decision to use TBS analysis was principally derived from the convention set by the Agency for Healthcare Research and Quality, but TBS may overrepresent extremely satisfied patients while minimizing extremely dissatisfied patients (4,24). We also defined TBS more conservatively as a score of 10 due to statistical limitations, but TBS has been previously defined as 9 and 10 together (52).
General institutional appointment attendance rate, but not specifically ophthalmology appointment attendance rate, was significant predictor of TBS in our analysis. The distribution of attendance was skewed in our population, with an over 99% average general attendance rate with little deviation. This highlights a possible bias, as patients with better attendance rates may be more responsive to completing surveys, but dissatisfaction may cause patient disengagement and decrease attendance. The same applies to the finding that patients who completed more portions of the survey were more likely to give a TBS and those who took longer to complete a survey were less likely to give a TBS. This self-selection highlights the fact that volunteer surveys might not represent all the patients who receive care and should be taken into consideration when interpreting survey results. Conversely, patients who take longer to complete the survey may be less likely to rate a TBS due to late-response bias. Further work needs to be done to assess nonrespondent ophthalmology patients. Previous work shows that among diabetic patients, nonresponders to a patient satisfaction survey incurred higher health care costs and utilization and had lower medication adherence (53). Ultimately, identification of barriers those patients face might elucidate the patient experience of all the patients receiving care.
There is also potential survivorship bias when examining new versus established patients, as only patients with positive experiences may return to the same provider. Despite this, patients with acute complaints seen in SDA clinic by rotating and often unfamiliar ophthalmic providers did not have significantly different odds of rating a TBS. Rescheduling was also not significant, although it may comment on provider flexibility and accessibility (16,17). Lastly, it is important to note that this is a cross-sectional study, and patient experiences, including their expectations and health care engagement, may change with time.
Of note, few patient experience studies report a measure of the predictability of their statistical models. Our multivariate model produced a ROC AUC of 0.601, which could be interpreted to mean that our model could correctly predict a provider receiving a TBS 60.1% of the time, only 10.1% greater than a coin toss. This model strength was similar to one by Ziaei et al, who examined determinants of patient satisfaction with ophthalmic services, although they primarily examined the associations of survey items with overall satisfaction using a different validated survey (17). The unreliability of these models is not unexpected given the subjective and mercurial nature of survey-based studies. It is likely that there are many factors that influence patient satisfaction on the individual level that are not adequately captured in most methods of measurement or statistical modeling.
Conclusion
The results of our study highlight that several nonmodifiable factors can affect provider rating. These factors should be studied further and be taken into consideration when interpreting the survey results. Although these factors are not modifiable, they still might be actionable. These are the questions that can be pursued in further studies. For instance, how can patient expectation be met when it comes to a new visit? What is the difference between comprehensive ophthalmology and optometry practice, and what tools can be given to the providers to improve the experience of their patients? How can implicit bias affect patient perspective and what community education can help alleviate these biases? Another important next step would be to analyze the patient population that receives the randomly assigned surveys but does not complete them. Identifying the barriers to survey completion might improve the care the patients receive and make the results of these surveys more valid.
Supplemental Material
Supplemental Material, sj-docx-1-jpx-10.1177_23743735211033750 - Ophthalmology Provider Ratings and Patient, Disease, and Appointment Factors
Supplemental Material, sj-docx-1-jpx-10.1177_23743735211033750 for Ophthalmology Provider Ratings and Patient, Disease, and Appointment Factors by Michael M. Han, Jessica Hsueh, Andrew X. Chen, Tyler E. Greenlee, Thais F. Conti, Susannah L. Rose, Rishi P. Singh and Aleksandra V. Rachitskaya in Journal of Patient Experience
Supplemental Material
Supplemental Material, sj-docx-2-jpx-10.1177_23743735211033750 - Ophthalmology Provider Ratings and Patient, Disease, and Appointment Factors
Supplemental Material, sj-docx-2-jpx-10.1177_23743735211033750 for Ophthalmology Provider Ratings and Patient, Disease, and Appointment Factors by Michael M. Han, Jessica Hsueh, Andrew X. Chen, Tyler E. Greenlee, Thais F. Conti, Susannah L. Rose, Rishi P. Singh and Aleksandra V. Rachitskaya in Journal of Patient Experience
Supplemental Material
Supplemental Material, sj-jpg-1-jpx-10.1177_23743735211033750 - Ophthalmology Provider Ratings and Patient, Disease, and Appointment Factors
Supplemental Material, sj-jpg-1-jpx-10.1177_23743735211033750 for Ophthalmology Provider Ratings and Patient, Disease, and Appointment Factors by Michael M. Han, Jessica Hsueh, Andrew X. Chen, Tyler E. Greenlee, Thais F. Conti, Susannah L. Rose, Rishi P. Singh and Aleksandra V. Rachitskaya in Journal of Patient Experience
Supplemental Material
Supplemental Material, sj-pdf-1-jpx-10.1177_23743735211033750 - Ophthalmology Provider Ratings and Patient, Disease, and Appointment Factors
Supplemental Material, sj-pdf-1-jpx-10.1177_23743735211033750 for Ophthalmology Provider Ratings and Patient, Disease, and Appointment Factors by Michael M. Han, Jessica Hsueh, Andrew X. Chen, Tyler E. Greenlee, Thais F. Conti, Susannah L. Rose, Rishi P. Singh and Aleksandra V. Rachitskaya in Journal of Patient Experience
Footnotes
Authors’ Note
The individual subject consent was not required for this study.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: MH: None. JH: None. AXC: None. TG: None. TC: None. SLR: None. RPS: Genentech/Roche (personal fees), Alcon/Novartis (personal fees), Apellis (grant), Graybug (grant), Zeiss (personal fees), Bausch + Lomb (personal fees), Ophthea (personal fees), Regeneron Pharmaceuticals, Inc. (personal fees). AVR: AR: Alcon, Novartis, Zeiss, Allergan, Regeneron, Genentech, Samsara (all personal fees); Novartis, Genentech (research).
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
