Does Patient Experience Correlate With Overall Hospital Quality Ratings?

Abstract

Publicly reported patient experience (PE) scores and overall hospital quality and safety ratings (QSR) increasingly influence consumers' healthcare choices and reimbursement, but their correlation remains unclear. We analyzed 2,384 adult US acute care hospitals’ public data from Google® (Google), the Leapfrog Group® (Leapfrog), Hospital Compare^® (HC), US News and World Report^® (USN), and Healthgrades^® (HG). We abstracted QSR from three sources (Leapfrog, HC, and USN) and PE from four sources (HC, USN, HG, and Google). We found significant differences in PE scores across QSR categories in Leapfrog and HC (Kruskal-Wallis Test, all P < 0.001) and between USN-ranked and non-ranked institutions (Mann-Whitney U test, P < 0.001), indicating better PE in hospitals with higher QSR scores. We performed a logistic regression that showed that some but not all QSR were independently associated with PE. In conclusion, our study adds to the evidence that PE and quality often, though not always, correlate.

Introduction

Hospitals in the United States are increasingly leveraging online marketing to deliver services. Personal recommendations, insurance network constraints, and publicly available data influence patients’ hospital and provider choices. Meanwhile, patients’ engagement with public data sources is rising rapidly.^1,2

Two categories of online data accessible to consumers are patient experience (PE) information and quality and safety ratings (QSR) generated by public and private organizations. PE data are derived from unscripted, purely evaluative online reviews and standardized tools such as the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) Survey. ³ The HCAHPS surveys aim to measure experiences by asking objective questions about care processes to reduce subjectivity.

Previous research generally supports a positive association between PE and quality-of-care outcomes. ⁴ A systematic review found a positive association between higher PE scores and higher quality-of-care and patient-safety ratings across most of the 55 studies. ⁵ However, these studies have limitations, including small sample sizes, a focus on specific conditions, and an emphasis on primary care settings. We identified a gap in the research: the need to include a large number of hospitals and to simultaneously compare multiple sources of QSR and PE. We, therefore, included all U.S. acute care hospitals to examine the correlation between various PE sources, including unscripted evaluative online reviews, and the association between PE and QSR across several rating systems.

Methods

Search Strategy

We obtained the list of acute care hospitals from the American Hospital Directory, which compiles information from publicly available sources. ⁶ We excluded specialty, pediatric, and critical access hospitals because they were outside the scope of our intended study population. We also excluded facilities with no data entries in any of the five PE and QSR datasets.

Between February 1, 2023, and October 3, 2023, four authors (TN, SI, AE, and JR) independently queried five online databases: Google® (Google), the Leapfrog Group® (Leapfrog), Hospital Compare® (HC), U.S. News & World Report® (USN), and Healthgrades® (HG). To achieve temporal alignment, all QSR and PE data for a given hospital were recorded on a single day, recognizing that there will always be variable lags between sources; i.e., for a given hospital, two QSR sources might be assessing different time periods.

Databases Searched

We gathered data from five sources: Google, HC, HG, Leapfrog, and USN.

Google is a large technology company focusing primarily on online searches. Users can leave unscripted reviews and rate their experience on a scale of 1 to 5, with 5 being the highest rating. The review aggregation is therefore random and unadjusted. HC is CMS’s tool for presenting data from hospitals participating in the Medicare program. HC assigns scores on a 1 to 5-star scale. HG is a private U.S. company that evaluates hospitals based on risk-adjusted mortality and in-hospital complications, converting data from publicly available sources into 1 to 5-star scores. Leapfrog is a not-for-profit organization dedicated to advancing patient safety in healthcare. Each hospital voluntarily completes a survey twice a year. After validating the data, the Leapfrog organization assigns an overall safety grade (A, B, C, D, or F; F is the lowest) based on various safety and quality domains. USN is a private digital media company that publishes regional, national, and specialty procedures & conditions-specific hospital rankings.

Variables Recorded

For each hospital, we recorded the location, number of beds, total average yearly discharges, gross revenue, and patient days. For the QSR data, we noted Leapfrog’s overall safety grade, HC’s total number of quality and safety stars, and USN’s hospital regional or national ranking status. For the PE results, we documented the average ratings on Google, HC, and USN, along with the percentage of patients who rated a hospital either a 9 or 10 in HG.

Statistical Analysis

We present continuous data as median [25-75% Interquartile range] and categorical data as numbers (percentage). We calculated the Spearman correlation coefficient to assess the strength of the correlation between the different PE scores. We analyzed differences in patient PE across quality ratings and compared the values using the Mann-Whitney U and Kruskal-Wallis tests.

We performed a linear logistic regression to assess the independent association between the PE scores and QSR. We entered the following variables: number of beds, total average yearly discharges, patient days, gross revenue, and PE for HC, HG, USN, and Google in one step. We then uncovered evidence of collinearity among the non-PE variables (number of beds, total discharges, patient days). We then added interaction terms for the three variables and re-ran the model. We entered all variables in a single step and considered them significant at p < 0.05. All analyses were conducted using SPSS software version 28.0, IBM (Chicago, IL, USA).

Results

Hospitals Characteristics

The American Hospital Directory database included 3,871 hospitals. Of these, 2,384 met the inclusion criteria for our study. The average number of beds per hospital was 197 (±237), the average number of yearly discharges was 7,754 (±10,835), and the total patient days were 37,936 (±64,212). The number of Google reviews varied widely, ranging from close to 50 to over 1,000.

Correlation Between the Different PE Scores

We found a very weak but statistically significant correlation between Google and HC (0.17 [0.13-0.21]), USN (0.14 [0.1-0.18]), and HG (0.28 [0.25-0.32]) (all p < 0.001). There were very strong correlations between HC-USN 0.75[0.73-0.77], HC-HG 0.76[0.74-0.77] and USN-HG 0.75[0.73-0.76] (all P <0.001). Table 1 summarizes these findings in a Correlation Matrix.

Table 1.

Correlation Matrix Between the Patient Experience Ratings

Google	1	​	​	​
HC	0.17	1	​	​
USN	0.14	0.75	1	​
HG	0.28	0.76	0.75	1
​	Google	HC	USN	HG

Correlation coefficient interpretation: 0-0.4; weak, 0.41-0.79; moderate, 0.80-1; strong.

We present the correlation coefficients between the studied PE sources. The correlation is strong between HC, HG, and USN, and very weak between Google and the other three sources.

PE Scores and QSR

We found significant differences in PE scores from HC, USN, Google, and HG across Leapfrog and HC categories (Kruskal-Wallis Test, all P < 0.001), and between USN-ranked and non-ranked institutions (Mann-Whitney U test, P < 0.001), i.e., better PE experience was associated with higher QSR grades (Figure 1). Appendix Table 1 presents the average PE scores for the different categories, expressed as Median [IQR].

Figure 1.

Patient experience scores in HC, HG, USN, and google stratified by leapfrog or HC grades

The logistic regression showed that QSR in Leapfrog was statistically independently associated with PE in Google (P=0.044), HG (P=0.004), and USN (P=0.032) and was close to statistical significance in HC (P=0.054); QSR in HC was statistically independently associated with PE in HC (P=0.002), HG(P<0001), and Google (P=0.009) but not USN. QSR in USN was not statistically associated with PE, with a close result with HC (P=0.053). Appendix Tables 2–4 presents the complete results of our Linear logistic regression analysis.

Discussion

Our analysis of five databases demonstrated that PE and QSR correlate to varying degrees. The dramatic increase in patients’ use of ratings and reviews to over 75% in recent surveys ² underscores the growing relevance of our subject.

The strong correlation of PE scores among HC, HG, and USN is expected and likely arises from the use of HCAHPS survey data. Despite statistical significance, the crowdsourced database (Google) and the three other PE sources show little correlation, a finding that aligns with published research, such as that of Ellenbogen et al., ⁷ and may be due to differences in methodology. HCAHPS’s experiential measurement with random sampling and risk adjustment of surveys sent only to patients discharged home ⁸ contrasts with purely evaluative online review platforms accessible to all to express experiences beyond the constraints of standardized questions. Unsolicited reviews are at risk of self-selection bias, in which a vocal minority, either irate or ecstatic about their experience, posts online. ⁹

Our results regarding the correlation between QSR and PE were mixed, consistent with prior evidence. For example, research using CMS quality measures has not consistently shown meaningful correlations with Google reviews,^7,10 and patient-reported outcomes from a national spine registry correlated poorly with physician ratings on platforms such as HG, Vitals, WebMD, and Google. ¹¹ In some studies, lower 30-day readmissions were associated with better PE [^12-15], but not in others.^10,16 Regarding mortality, Trzeciak et al. ¹⁵ reported lower in-hospital mortality with better PE, and a Norwegian study found that Facebook ratings significantly correlated with 30-day mortality. ¹⁷

The lack of correlation between QRS in USN and PE across all sources can be explained by the uniqueness of the USN rating, which heavily rewards reputation and complex specialty outcomes rather than consumer sentiment after routine care. A crowded hospital with an unpleasant emergency room experience might have low PE scores but still receive a USN ranking for, say, advanced neurosurgical procedures that only a few patients receive.

The practical implications of our findings are significant. Hospital leaders must look beyond improving HCAHPS scores, even though these scores directly affect reimbursement under Medicare’s Hospital Value-Based Purchasing Program. A hospital must adopt a dual-track strategy in which clinical quality initiatives will improve its Leapfrog grade, while service excellence and reputation management are required to improve Google ratings. One is not always a proxy for the other.

The financial consequences of poor quality or PE ratings are considerable. In a 2015 survey of 2,360 physicians and other healthcare providers, nearly 55% used online ratings to derive measures to improve patient care, ¹⁸ and this number has likely increased over the past decade. Yet, performance-based reimbursement faces skepticism. In a 2017 survey of 4197 practicing physicians in Rhode Island, only 2% felt that Medicare sites were “very accurate” in depicting physician quality, just over one-third felt that performance-based quality measures were “helpful”, and a similar percentage reported that patient reviews were “helpful” for patients choosing a physician. ¹⁹ One reason for this skepticism may be PE’s emphasis on non-clinical factors, such as bedside manner and communication skills, as highlighted in a recent study of one-star reviews of pediatric orthopedic surgeons. ²⁰ Our findings suggest a correlation between PE and QSR, but are not sufficient to reduce that skepticism and can only reinforce the need for a hospital to pay attention to multiple PE and QSR sources.

Our study has several strengths, including the number of hospitals we included and the diversity of PE sources we queried, which mirrors the multi-platform search behavior of modern healthcare consumers. We do not believe excluding specialty, pediatric, and critical-access hospitals is a limitation of our study, given their stark differences from acute adult hospitals.

Limitations

First, we explore limitations due to temporal alignment. PE and quality ratings are dynamic and could change our conclusions if the study were repeated. Along similar lines, the published data across different QSR and PE sources is not in real time, with variable lags between sources. Our data collection spanned eight months, and we could not verify the extent of change within the collection timeframe. Second, we chose not to exclude hospitals with a handful of potentially unstable Google ratings. Third, we used Google as the sole source of crowdsourced PE. Patients may value different platforms unequally, and assuming equal influence could misrepresent real-world behavior. Future research should first explore the patient-preferred and most widely used crowdsourced sites. Fourth, the correlations we found do not establish causation. In our logistic regression, we were unable to adjust for the case mix index or Medicaid data.

Conclusion

In our study of 2,384 acute care hospitals in the USA, the correlation between QSR of Leapfrog, HC, USN, and PE in Google, HC, HG, and USN was frequent but not uniform.