Abstract
Background:
The increased focus on patient satisfaction has led to growth in the use of physician rating websites.
Purpose:
To analyze the factors associated with online 5-star patient reviews for orthopaedic sports medicine surgeons.
Study Design:
Cross-sectional study.
Methods:
A total of 70 orthopaedic sports medicine surgeons were randomly selected from the AOSSM website. A search was performed for these surgeons on Yelp.com. All reviews other than 5 stars (of a possible 5 stars) were excluded from the study. Each review was categorized as referring to a surgical or nonsurgical aspect of care, and each comment within the review was categorized as being clinically or nonclinically related. Comments were further subcategorized by specific features such as bedside manner, clinical outcomes, and patient education. Categorical variables were analyzed using the chi-square test.
Results:
Overall, 400 five-star reviews were included in the study, comprising 1225 total positive comments. Of the 400 five-star reviews, 200 (50%) were from surgically treated patients, and 200 (50%) were from nonsurgically treated patients. Of the 1225 positive comments, 505 (41%) were clinically related, and 720 (59%) were nonclinical. The most common positive clinical comments were for clear treatment plans (191 reviews [48%]), good outcomes (173 reviews [43%]), and providing alternative treatment plans (55 reviews [14%]). The most common positive nonclinical comments were for good physician bedside manner (287 reviews [72%]), friendly/professional staff (194 reviews [49%]), and ease of scheduling (68 reviews [17%]).
Conclusion:
The majority of 5-star patient reviews left positive comments regarding nonclinical aspects of care such as physician bedside manner and friendly staff. The most common positive comments regarding clinical aspects concerned good outcomes and clear treatment plans. The overall most common positive comment, in both surgically and nonsurgically treated patients, referred to good bedside manner.
In recent years, there has been an increased focus on patient satisfaction. In 2010, the Affordable Care Act allowed the public to access data on patient satisfaction with hospitals and providers, which offered valuable information to help patients make educated choices about their health care. 2,17,27 While government-funded websites such as HealthData.gov and the US Department of Health and Human Services Protect Public Data Hub are publicly available, only 14% of patients know about these resources. 12,24 Patients more commonly rely on public physician review websites (PRWs), such as Yelp, Healthgrades, Vitals, and others, to make choices regarding their health care. 9 Specifically, a recent study reported that 59% of patients regard physician ratings as being important in their decision to choose a provider. 9 Furthermore, 54% of young adults rely on PRWs before going to see a physician, and up to 80% of patients trust both personal and online recommendations. 21 Because of their ease of use and accessibility, it stands to reason that online rating sites will continue to influence patient decisions in terms of choosing providers.
Because of the value that patients place on PRWs, it is important that physicians understand the impact that these platforms may have on their practices. Despite the widespread usage of PRWs by patients, surgeons remain skeptical of their utility. Additionally, some argue that it may not be a true metric to measure patient satisfaction. 3 Also, despite the objections to PRWs, they continue to have a significant impact on providers. One study found that 35% of patients selected providers based on good reports and that another 37% avoid providers based on negative reports. 5,22
Given the significance of online reviews, it is important to understand the underlying factors that are associated with positive online reviews. While studies have analyzed reviews among orthopaedic sports surgeons, none, to our knowledge, has investigated the elements that contribute to an extremely positive review in orthopaedic sports medicine surgery. 10,14,19,26 This is particularly relevant, given the practice pattern of sports medicine involving a large percentage of elective cases.
The purpose of this study was to analyze the factors that contribute to a 5-star review on PRWs among orthopaedic sports medicine surgeons.
Methods
Institutional review board approval was not needed for this study. Overall, 70 surgeons were randomly selected from the AOSSM Find a Doctor tool on the AOSSM website using public information. All search criteria on the Find a Doctor tool were left blank, generating a list of 3585 AOSSM-accredited surgeons. 1 A random number generator (1-3585) was utilized to determine which 70 surgeons would be selected. The number of surgeons chosen for this study was based on a prior study analyzing negative online reviews of orthopaedic sports surgeons. 20 A search was then performed for all reviews listed under the selected surgeons on Yelp.com. This website was chosen because it has been reported as the most utilized and trusted PRW by consumers. 4 All reviews other than 5 stars (of a possible 5 stars) were excluded from the study. A diagram of the inclusion and exclusion criteria is shown in Figure 1.
Flowchart of inclusion and exclusion criteria for this study.
The included 5-star reviews were than classified as either surgical or nonsurgical. Reviews with a direct reference to a surgical aspect of care were assumed to be from patients who underwent surgery and were classified as surgical. Additionally, the comments within each review were classified as clinically or nonclinically related. Clinically related comments included those with a reference to good treatment results, alternative treatment methods, pain, diagnosis, or treatment plans, and nonclinical comments included physician bedside manner/professionalism, midlevel (physician assistant or nurse practitioner) professionalism, staff professionalism, wait time, time with the provider, cost, ease of billing, facilities, or ease of scheduling. The above categories were determined before analyzing the comments. All categorizations were conducted independently by 2 authors (M.E.N. and Z.L.). A third author (E.H.R.) was utilized to resolve conflicting classifications or categorizations between the initial 2 reviewers.
Univariate analysis was performed to determine means and 95% CIs. The association between categorical variables was examined with the chi-square test and Fisher exact test. An alpha value of .05 was used to determine statistical significance. The ratio of the rate of nonsurgical reviews divided by the rate of surgical reviews (rate ratio) was determined for each category and used to determine whether that category was more often being reviewed by a surgical or nonsurgical patient. In addition, the interrater reliability of the 2 reviewers was calculated. Statistical analysis was performed using Excel (Microsoft) version 16.73.
Results
A total of 1029 reviews for 70 surgeons were selected from Yelp. Of the initial 1029 reviews, 581 (57%) were identified as 5-star reviews, with the remaining excluded. Another 181 were excluded, as they were unable to be classified as 5-star reviews with either no comments or only short comments (eg, “good”). Finally, 400 five-star reviews were included for analysis, of which 200 (50%) were from surgically treated patients and 200 (50%) were from nonsurgically treated patients. These reviews were found to have 1225 positive comments, or “compliments,” with each review having an mean of 3.06 compliments. Of these compliments, 505 (41%) were clinically related, and 720 (59%) were nonclinical. The interrater reliability of the categorizations indicated 96.5% agreement, with a Cohen kappa statistic of 0.667. There was no statistically significant difference in the types of comments in terms of a clinical or nonclinical nature between the surgically and nonsurgically treated patients (Table 1).
Total Number and Mean Per Review of Compliments
a Nonsurgical/surgical reviews.Dash indicates unable to calculate p value.
The most common positive clinical comments were for clear treatment plans (191 reviews [48%]), good outcomes (173 reviews [43%]), and providing alternative treatment options to surgery (55 reviews [14%]) (Figure 2A). The most common positive nonclinical comments were in relation to favorable physician bedside manner (287 reviews [72%]), friendly/professional staff (194 reviews [49%]), and ease of scheduling that contributed to no delays in care (68 reviews [17%]) (Figure 2B). The total number of compliments was larger than the number of reviews, as each review could contain multiple categorized comments (Table 2).
Types of (A) clinical and (B) nonclinical compliments of orthopaedic sports medicine surgeons on Yelp.com.
Types of Comments for Compliments a
a Data are reported as n (%) unless otherwise indicated. Boldface P values indicate a statistically significant difference between groups (P < .05).Dashes indicate unable to calculate the p value.
b Nonsurgical/surgical reviews.
The most common positive clinical comments from surgically treated patients were in regard to good outcomes (131 reviews [66%]), clear treatment plans (91 reviews [46%]), and well-controlled pain (41 reviews [21%]). The most common positive nonclinical comments from surgical patients were related to good bedside manner of the physician (141 reviews [71%]), friendly/professional staff (95 reviews [48%]), ease of scheduling that contributed to no delays in care (33 reviews [17%]), and enough time spent with the provider (21 reviews [11%]).
The most common positive clinical comments from nonsurgically treated patients were for clear treatment plans (100 reviews [50%]), good outcomes (42 reviews [21%]), and providing alternative treatment options to surgery (40 reviews [20%]). The most common positive nonclinical comments from nonsurgical patients were for good bedside manner of the physician (146 reviews [73%]), friendly/professional staff (99 reviews [50%]), and ease of scheduling that contributed to no delays in care (35 reviews [18%]) (Figure 3).
Comparison of compliments according to the review type.
The difference in the number of compliments for surgically treated and nonsurgical patients was statistically significant (P < .05) for good outcomes (131 vs 42 reviews, respectively), providing alternative treatment options to surgery (15 vs 40 reviews, respectively), well-controlled pain (41 vs 11 reviews, respectively), short wait times (16 vs 34 reviews, respectively), clear billing and insurance practices (12 vs 3 reviews, respectively), and impressive facilities (8 vs 17 reviews, respectively) (Table 2).
Discussion
The intent of this study was to investigate the attributes of a patient’s episode of care that led to positive reviews of orthopaedic sports medicine surgeons. Nonclinical aspects of care were more frequently commented on than clinical aspects (59% vs 41%, respectively). Nonclinical compliments most frequently referred to bedside manner, professional/friendly staff, and ease of scheduling, while clinical compliments focused on clear treatment plans, good outcomes, and providing alternative treatment options. Surgical patients tended to focus on good outcomes and control of pain, whereas nonsurgical patients commented on alternative treatment options and short wait times.
In previous studies, positive reviews tended to relate to the nonclinical aspects of patient care such as staff, convenience of the location, and the patient-physician relationship. 8,11,16 Conversely, poor staff interactions and physician bedside manner tended to produce negative reviews. 13,15,20,26 Furthermore, excellent bedside manner was one of the most commented on factors in 5-star reviews of physicians. It appears that patients may depend on nonclinical aspects of care for evaluating overall quality in the medical field. 18 Therefore, when hiring staff or interacting with patients, it is important that surgeons take into consideration the way that staff members and their behaviors are perceived.
Even though patients focus on nonclinical aspects of care, having friendly, accommodating staff and good bedside manners are not the only characteristics seen in positive reviews. Perceived technical competence has previously been reported as a common factor in higher physician ratings. 16,19 In our study, 5-star ratings referenced clinical skills in 41% of the compliments, most frequently focusing on clear treatment plans and good outcomes. While previous studies have argued that PRWs have a poor correlation with a physician’s clinical capability, our study finds that patients still placed a high value on favorable clinical outcomes. 6,22,23 However, it is important to note that PRWs are inherently subjective and typically do not objectively reflect health care quality.
Additionally, orthopaedic surgeons need to be aware of any differences between reviews left by surgical versus nonsurgical patients. For negative reviews, 1 study showed that nonsurgical patients were more likely to leave poor reviews than surgical patients. 20 Furthermore, surgical patients focused on their surgical course, with comments pertaining to complications, reoperations, and readmissions compared to nonsurgical patients, who focused on unclear treatment plans and misdiagnoses. 20 Both surgical and nonsurgical patients emphasized uncontrolled pain, delays in care, and bedside manner in negative reviews. For positive reviews, there were a few categories that surgical patients complimented more frequently than nonsurgical patients including good outcomes, well-controlled pain, and clear billing. Nonsurgical patients had more comments pertaining to alternative treatment plans to surgery and short wait times. Both surgical and nonsurgical patients focused on good physician bedside manner, clear treatment plans, and friendly staff. Thus, orthopaedic sports medicine surgeons should be cognizant of their interactions with patients, communication of treatment plans, and controlling pain, with acknowledgment that surgical and nonsurgical patients are equally likely to leave positive reviews.
Previous reports have expressed concerns about the legitimacy of online reviews and the possibility of false reviews tarnishing a physician’s practice and reputation. 18 Although Yelp was selected for its reputation as the most utilized and trusted PRW by consumers, up to 20% of its reviews have been shown to be fraudulent. 4,26 In fact, Lagu et al 13 were able to identify reviews written by physicians themselves when investigating PRWs. However, Yelp.com does offer software that will identify and hide reviews that were solicited by the business, but it does not prevent fraudulent reviews. 25 As physician reputation is becoming increasingly considered by patients, there is a financial incentive for providers to increase their online rating. Additionally, there are reports of companies dedicated to posting positive reviews as well as fake negative reviews, making it even more difficult to maintain the validity of PRWs. 7 Future studies could assess the validity of online reviews and propose ways to ensure their integrity moving forward. One such way would be to compare online hospital scores (verified) versus PRW online scores. With the future increased trustworthiness of PRWs, physicians would have confidence that their efforts to provide excellent care, practice good bedside manner, and promote a friendly staff environment will be rewarded with extremely positive reviews.
Limitations
There were several limitations to this study. Selection bias may be present, as we limited our online review platform to Yelp.com. This may not accurately reflect the components of positive reviews if other platforms are visited by different patients. We attempted to minimize this issue by using a platform that is the most utilized and trusted by consumers. 4 Selection bias may also be present because we chose 70 orthopaedic sports medicine surgeons of a possible 3585, potentially limiting the external validity of this study. By using a randomized generator, we hoped to increase our relatability and reduce our impact on external validity.
Conclusion
The majority of 5-star reviews complimented nonclinical aspects of care such as physician bedside manner and friendly staff. The most common clinical compliments were related to good outcomes and clear treatment plans. The overall most common compliment referred to good bedside manner from both surgical and nonsurgical patients.
Footnotes
Final revision submitted February 22, 2023; accepted March 20, 2023.
One or more of the authors has declared the following potential conflict of interest or source of funding: S.T. has received education payments from ImpactOrtho. A.C. has received education payments from Arthrex and consulting fees from Zimmer Biomet. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval was not sought for the present study.