Abstract
Hospitals commonly seek to improve patient experience as measured by the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, yet there are limited data to guide improvement efforts. The HCAHPS survey was developed for interhospital comparisons, whereas its use in intrahospital comparisons has not been validated. We sought to better understand the validity of utilizing intrahospital score comparisons and to identify the factors that may predict top-box HCAHPS scores. We performed a retrospective observational cohort study at an academic urban safety-net hospital examining 4898 HCAHPS surveys completed by hospitalized patients. We found that while most Patient-Mix Adjustment factors for which HCAHPS scores are adjusted were associated with top-box scores on intrahospital comparisons, few additional variables were associated with top-box scores. Further, HCAHPS questions pertaining to nurse and doctor communication were highly correlated with overall hospital rating, suggesting that communication-related factors may influence a patient’s hospital experience more strongly than do administrative factors.
Introduction
The Centers for Medicare and Medicaid Services (CMS) and the Agency for Healthcare Research and Quality developed the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey to measure and compare how patients perceive their care across health care organizations (1 –3), The survey is scored as percentages of top-box (ie, most positive) responses and is utilized to determine hospitals’ financial bonuses or penalties from CMS. For the fiscal year 2020, 25% of the approximately US$1.9 billion in available value-based incentive payments was based upon HCAHPS scores (4). Multiple studies have positively linked patient experience as measured by HCAHPS with higher quality of care (5 –10). Although the HCAHPS survey is rigorously validated (1,3,11), there are limited data on how to improve patient experience as measured by HCAHPS scores (12).
The HCAHPS was designed for interhospital comparisons, and CMS does not endorse its use for intrahospital comparisons, such as among wards or providers (4,13). Intrahospital comparisons are unreliable due to prohibitively small sample sizes (14). Nonetheless, many institutions already use HCAHPS for intrahospital comparisons. Understanding whether intrahospital comparisons are valid could allow HCAHPS data to be utilized effectively to target quality improvement efforts.
Centers for Medicare and Medicaid Services adjust HCAHPS scores for Patient-Mix Adjustment (PMA) factors, which are unrelated to a hospital’s performance but influence patients’ responses and impact scores (2,14,15). Prior studies have evaluated additional variables suspected to impact patient experience (10,16 –27). As hospitals seek to improve patient experience, understanding whether there are additional nonperformance-related factors for which HCAHPS scores should be adjusted may help target performance-related variables amenable to quality improvement initiatives.
We aimed to: (1) understand whether factors known to impact scores at the interhospital comparison level (ie, PMA factors) also impact scores at the intrahospital level, (2) determine which, if any, patient-level variables are associated with top-box scores for overall hospital rating and nurse and doctor communication composites, and (3) determine which HCAHPS questions most impact overall hospital rating.
Methods
Study Design
We performed a retrospective analysis of administrative and clinical data from adult medical and surgical patients hospitalized between October 1, 2010 and September 30, 2014, who completed the HCAHPS survey in English or Spanish. The study was performed at Denver Health, a 525-bed university-affiliated regional safety-net hospital, and approved by the Colorado Multiple Institutional Review Board. Data were acquired via electronic query of the Denver Health Data Warehouse. Inclusion and exclusion criteria are provided in Figure 1.
Inclusion and exclusion criteria. *Approximate values based on institutional estimates.
During the study period, HCAHPS surveys were administered to a random sample of patients via telephone by a third-party vendor per CMS guidelines (28). When patients completed more than one survey (for additional visits), we analyzed only the survey pertaining to their first hospitalization.
Because of the large sample size and differences in processes and care models for medical and surgical patients, we analyzed medical and surgical patients separately.
Variables and Measurement
Performance-related and nonperformance-related variables that may impact patient experience were selected a priori based on literature review and the authors’ collective clinical expertise. Table 1 lists these variables grouped into 7 domains: HCAHPS PMA factors, patient demographics, quality, utilization, patient flow, provider, and patient care.
HCAHPS Patient-Mix Adjustment Factors, Demographics, and Variables Postulated to be Predictive of Top-Box HCAHPS Scores.
Abbreviations: DC, discharge; ED, emergency department; GED, general equivalency diploma; HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems ICU, intensive care unit; IQR, interquartile range; MICU, medical intensive care unit; NPO, nothing by mouth; SD, standard deviation; SICU, surgical intensive care unit.
All categorical variables were categorized per HCAHPS guidelines. Response percentile is a rank order of the time between discharge and survey completion and was calculated per HCAHPS guidelines (29).
A patient’s address and payor were used to determine homelessness. Substance use disorders and psychiatric illness were determined from International Classification of Disease, Ninth Revision codes for the index hospitalization and all outpatient encounters during the preceding 3 years.
Hours spent in the emergency department (ED) included time in both the waiting room and an examination room. Number of bed changes included all movement except from the ED to the ward at admission and to and from the operating room, when applicable. Clinical orders were queried to determine discharging team, diet, medications (nonopioids, opioids), and restraint use.
Number of Physicians Index (NPI; the number of unique physicians caring for a patient) and Usual Provider of Continuity index (the greatest number of a patient’s encounters with a single physician, divided by the total number of encounters) were calculated from physician billing data using methods described by Turner et al (30).
Charlson Comorbidity Index score was calculated based on comorbidities identified at the time of discharge. Facility and physician billing charges were used to calculate average total charges per day, number of diagnostic or therapeutic activities, and number of procedures.
We conducted multiple validations of data warehouse queries via direct chart review and modified query criteria until queries accurately captured the data.
Statistical Methods
Data were analyzed with SAS Enterprise Guide 5.1 (SAS Institute, Inc).
Outcomes
Presence or absence of top-box HCAHPS scores (ie, 9 or 10 on a 10-point numeric Likert scale or “always” on a 4-point Likert scale of “always” to “never”) for overall hospital rating, doctor communication composite, and nurse communication composite were examined (28). For the composites, if all 3 included questions were answered, “always,” the composite was coded as top-box; otherwise the composite was coded as not-top-box.
Patient-Mix Adjustment Factors and Association With Top-Box Scores
Per HCAHPS guidelines, hospital-level survey results are adjusted for PMA factors. The purpose of this is both to adjust for nonperformance-related variables that may impact patient experience and to account for response bias. In advance of multivariable modeling, unadjusted associations between PMA factors and each top-box score outcome were assessed using logistic regression.
Predictors of Top-Box Scores
Unadjusted associations between independent covariates were assessed using a Student t test or Wilcoxon Rank Sum test for continuous variables depending on results of normality tests and by a chi-square test for categorical variables. These covariates were then adjusted for PMA factors (Table 2) (2,15).
Final Multivariable Logistic Regression Models, Adjusted for HCAHPS PMA Factors.
Abbreviations: HCAHPS PMA, Hospital Consumer Assessment of Healthcare Providers and Systems Patient-Mix Adjustment; SE, standard error.
a Binary variables are coded 0 for no or 1 for yes.
Independent adjusted covariates were then assessed for collinearity within each domain before including them in multivariable logistic regression models using a McNemar test for categorical variables and either a Pearson product-moment correlation coefficient or Spearman rank correlation coefficient for continuous variables. The more predictive variable was retained for multivariable modeling if any variables were considered highly collinear.
To identify the most significant variables, a multivariable logistic regression model was constructed for each domain using backward variable deletion of the least significant risk factors using a P value exceeding .05 as the criterion for variable removal, after adjustment for PMA factors. All variables remaining in each domain’s model were subsequently modeled together.
Following selection of a final main effects model, 2-way interaction terms that were considered clinically relevant a priori and had an unadjusted P <.25 were tested in the main effects model using backward deletion of the least significant terms. If a final model was selected, goodness-of-fit or calibration was assessed using the Hosmer-Lemeshow test. The area under the receiver operating characteristic curve (ie, the logistic regression c index statistic) was calculated to quantify the predictive accuracy or discrimination of the final model. Influence diagnostics were used to examine the role of individual subjects, specifically the presence of extreme observations not well explained by the model, in the final models selected.
Hospital Consumer Assessment of Healthcare Providers and Systems Questions and Impact on Overall Hospital Rating
To assess the strength of correlation between overall hospital rating and the nurse or doctor communication composite score, Pearson partial correlation coefficients were calculated while adjusting for the effect of PMA factors. Composite scores were calculated by converting individual nurse and doctor communication questions to continuous scores and finding the mean score across the 3 questions constituting each composite.
Missing Data
If a variable was included in a specific statistical test, then subjects with missing data were excluded in that analysis (ie, complete case analysis). The pain score variables were only available after January 2013 and thus analysis using those covariates was limited to patients discharged after January 2013. The surgical service at our institution has only teaching teams, so no analyses of teaching versus nonteaching team were conducted for surgical patients.
Results
From October 2010 to September 2014, 57016 patients were admitted to the general medical or surgical services, and 4898 of these completed an HCAHPS survey (Figure 1). Our institutional HCAHPS survey response rate during this time period was approximately 25%. Except for pain score and NPI for surgical patients, only 12 of the 41 variables had missing data, and for none of these variables did the amount of missing data exceed 10%; 37% of the data needed to calculate NPI for surgical patients were missing because of limitations in physician billing data. Comparisons between medical and surgical patients for the analyzed variables are summarized in Table 1.
Patient-Mix Adjustment Factors and Association With Top-Box Scores
Most PMA factors were associated with top-box scores, except for service line and mean response percentile (Table 2).
Predictors of Top-Box Scores
For medical patients, after PMA adjustment, no variables were found to predict top-box overall hospital rating (Figure 2). For surgical patients, after PMA adjustment, factors associated with top-box overall hospital rating included white race, medically indigent status, and having higher average daily charges; factors associated with not-top-box overall hospital rating included having Medicare insurance; being discharged with a psychiatric diagnosis; and returning for inpatient, urgent, or emergent care before completing the survey (Figure 2).
Predictive variables for top-box HCAHPS overall hospital score odds ratio (OR) with 95% CI * = no data as all surgical services are teaching teams. HCAHPS indicates Hospital Consumer Assessment of Healthcare Providers and Systems.
For both medical and surgical patients, several additional variables were associated with nurse and doctor communication scores, but most variables were not (Supplemental Appendix Figures 1 and 2).
After PMA adjustment, modeling for each domain, and testing relevant interaction terms, no covariates emerged as being significantly associated with overall hospital rating among medical patients (Supplemental Appendix Table 1). The number of subjects with complete data included in each model is shown in Supplemental Appendix Table 2. Additional variables were retained as significant covariates in models for overall hospital rating for surgical patients and for nurse and doctor communication. However, while the Hosmer-Lemeshow test for each model suggested the models fit the data, the c-statistics suggested poor predictive accuracy for all models except the overall hospital rating model for surgical patients, for which the predictive accuracy was fair. Additionally, the influence diagnostics for all models indicated the presence of extreme observations not well explained by the model, suggesting inadequate model fit.
Hospital Consumer Assessment of Healthcare Providers and System Questions and Impact on Overall Hospital Rating
For both medical and surgical patients, both nurse and doctor communication were significantly positively correlated with overall hospital rating after controlling for PMA factors. Partial correlation coefficients between overall hospital rating and doctor communication were 0.54 and 0.57 for medical and surgical patients, respectively, (P < .0001). Partial correlation coefficients between overall hospital rating and nurse communication were 0.59 and 0.64 for medical and surgical patients, respectively (P < .0001).
Discussion
Key findings of this study include: (1) the majority of PMA factors utilized by CMS to adjust HCAHPS scores at the interhospital level also influenced scores at the intrahospital level; (2) most other patient-level variables hypothesized to influence HCAHPS scores were not associated with top-box scores; (3) for the few variables that were significantly correlated with top-box scores, multivariable models were limited in utility due to poor fit and low predictive accuracy; and (4) both nurse and doctor communication scores were highly correlated with overall hospital rating at the intrahospital level.
The HCAHPS survey has been extensively studied and validated (1,3,11) and CMS adjusts for nonperformance-related factors known to impact HCAHPS responses when comparing hospitals (2). Our findings suggest that HCAHPS case-mix approaches perform well when applied at the patient level within an institution, and that single-institution sample sizes may be large enough so as not to invalidate intrahospital comparisons. Although CMS recommends against using HCAHPS data for intrahospital comparisons, many institutions nonetheless already do so. Our findings suggest this may be a useful strategy, but further guidelines on how to utilize these data most effectively are needed.
Although we found close associations with top-box scores for PMA factors, we did not find associations for most other variables. We found no association between overall hospital rating and number of procedures, Charlson Comorbidity Index, or length of stay, although in surgical patients, higher average daily cost was associated with increased odds of a top-box score. Cost may be a proxy for severity of illness, and it is possible that Charlson Comorbidity Index did not capture the acuity of illness in surgical patients who, at our level-one trauma center, are often younger and healthier at baseline. We found no correlation between overall hospital rating and readmissions in medical patients, but in surgical patients, overall hospital rating was negatively correlated with having a subsequent inpatient, urgent, or emergency encounter before being surveyed. Prior work examining length of stay and/or illness complexity has yielded mixed results. Although one study found no relationship between satisfaction and either length of stay or comorbidity index, others found that longer length of stay, higher illness complexity, and readmissions are associated with lower satisfaction scores (18,19,25 –27,31 –34). However, not all of these studies account for PMA factors.
Two systematic reviews found consistent positive associations between patient experience and clinical outcomes, best practices, safety culture, and appropriate resource utilization (5,6). We did not find correlations between top-box scores and process-related or resource-related variables such as outside hospital transfers, number of bed changes, ED wait times, discharges before noon, isolation status, or restraint use.
We found that neither provider continuity nor team structure (teaching vs nonteaching) was associated with top-box overall hospital rating. We did find trends suggesting that less provider continuity negatively affected nurse and doctor communication scores, but these associations did not persist in multivariable modeling. Turner et al similarly found a trend toward lower satisfaction with doctor communication when there was less provider continuity (30). Wray et al found that nonteaching services yielded higher patient satisfaction compared to teaching services but did not utilize the HCAHPS survey (35). Lappe et al found higher Press Ganey scores, but no difference in HCAHPS scores, for physician performance-related questions among patients cared for by a solo hospitalist as compared to a team including residents or advance practice providers (20).
We have examined an extensive list of possible predictors of patient experience, seeking to identify modifiable factors that remain after adjusting for nonperformance-related factors known to influence patients’ ratings and thus to target improvement interventions. Our findings suggest that unmeasured interpersonal factors, such as communication and managing expectations, likely influence patient satisfaction more so than easily measured variables.
Communication is key to excellent inpatient care. We previously found that communication among care teams and utilizing words that patients can understand are critical for optimizing patient experience (36). Similarly, we previously demonstrated that a real-time feedback initiative aimed at improving communication improved HCAHPS scores (37). Jackson et al described 2 models that predicted a patient’s overall experience at an urgent care clinic, with key variables being the patient’s age, patient–doctor interaction, patient functioning, and symptom improvement, but these models explained only 40% or less of the variance observed, suggesting similar influence of unmeasured factors (38). Kravitz et al found that meeting patients’ expectations was associated with better patient experience (39), and Tackett et al found etiquette-based medicine physician behaviors correlated with higher physician-specific Press-Ganey ratings (40). Communication training has been the target of several interventions aimed at improving patient satisfaction scores, with mixed results (41 –43).
Although factors such as NPO status, number of bed changes, isolation status, and discharge time did not directly correlate with satisfaction scores, we suspect that communication and expectation setting around these factors do matter. Consistent with prior work by Elliott et al (44) and Iannuzzi et al (18), we found that nurse and doctor communication composite scores were strongly correlated with top-box overall hospital rating. This suggests that effective communication plays a substantial role in patients’ experience of their hospital care. Our study was not designed to detect unmeasured factors such as expectations or the provider–patient interaction, and future research should target these areas.
Our study has several limitations. First, because we assessed only dichotomized top-box versus not-top-box scores, the analyzed variables could have affected patient experience without being detected in our analysis. Nonetheless, we designed our analysis this way to reflect how CMS reports and utilizes HCAHPS data. Second, HCAHPS survey response rates are typically under 30%, and our institution’s was approximately 25% during the study period (45). We were unable to analyze nonresponders, as data about them were not available. However, because PMA adjustment eliminates most observed nonresponse bias (15,46), and in following CMS methodology, we did not attempt to weigh for nonresponse. Third, subjects with missing data were excluded from analyses (ie, complete case analysis). Because of the relatively low level of missing data, we choose not to utilize an imputation approach (47), although there are benefits and weaknesses to different approaches (48). Notably, the 2 variables with the most missing data—pain scores and NPI (for surgical patients)—were not missing data at random, but rather due to systematic issues with data availability. Fourth, our data came from our data warehouse. Although this type of data has many known limitations, we conducted multiple validations to maximize accuracy. Fifth, our study was conducted at the patient level. Since this was a single-hospital study, there may not be enough variability in some of the processes studied to determine whether they affect patient experience. Nonetheless, many of the factors already known to affect HCAHPS scores were similarly found to affect scores in our study. Finally, our single-institution results may not be generalizable.
Our study has several strengths. Data were examined at the patient level, which may yield a more differentiated analysis than data examined at the interhospital level. Further, our analysis included a large number of HCAHPS surveys. Finally, we rigorously validated and revised our data queries to maximize accuracy.
Conclusion
We found strong associations between top-box HCAHPS scores and many of the PMA factors known to impact scores; however, we found no additional performance-related or nonperformance-related variables that were strongly and consistently associated with top-box scores. Nurse and doctor communication scores most impacted overall hospital rating. It is likely that unmeasured factors pertaining to communication and interpersonal interactions influence a patient’s hospital experience more strongly than measured variables.
Supplemental Material
Supplemental Material, sj-pdf-1-jpx-10.1177_23743735211034342 - Predictors of a Top-Box Patient Experience: A Retrospective Observational Study of HCAHPS Data at a Safety Net Institution
Supplemental Material, sj-pdf-1-jpx-10.1177_23743735211034342 for Predictors of a Top-Box Patient Experience: A Retrospective Observational Study of HCAHPS Data at a Safety Net Institution by Kimberly A. Indovina, Angela Keniston, Venkata Manchala and Marisha Burden in Journal of Patient Experience
Footnotes
Authors’ Note
Kimberly A. Indovina and Angela Keniston are co-first authors. This study was approved by the Colorado Multiple Institutional Review Board. There are no human subjects in this article and informed consent is not applicable. This article does not contain any studies with human or animal subjects.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
