Abstract
Introduction
PROMIS® Computer Adaptive Testing (CAT) is a testing tool for tracking patient-reported outcome measures (PROM) with a goal to improve individual and population outcomes. Patients’ health literacy (HL) may affect how they are able to complete PROM. We hypothesized that patients with low HL were less likely both to complete surveys and to do so via online patient portal (OPP).
Methods
A retrospective cohort study of 3926 orthopaedic patients in a rural state was performed. Completion rate and completion method both were subjected to log-binomial regressions with patient demographics and HL as predictor variables.
Results
PROM surveys were completed by 2166 (55.2%) of patients, including 512 completed via OPP and 1654 completed via in-clinic tablet (ICT). Compared to high HL patients, low HL patients had a 23% higher non-completion rate, and low HL completers were 63% less likely to use OPP. Age and gender had significant (P < 0.05) associations with completion method, but not completion rate, whereas Area Deprivation Index (ADI) had significant associations with both. Compared to White patients, Black patients had a 25% higher non-completion rate, and Black completers were 49% less likely to use OPP.
Discussion
Our analysis shows that health literacy, demographics, and socioeconomic status affect both whether and how patients fill out PROM surveys. Patients with low HL were less likely to complete PROM surveys, and less likely to use the OPP when they did.
Conclusion
Our results demonstrate that patients’ health literacy, demographics, and socioeconomic status affect both whether they complete their PROMIS® CAT and what method they complete it with. Additional efforts should be made to understand these factors, accommodate patients, and facilitate accurate and complete PROM responses, especially in hospitals that serve diverse and socioeconomically disadvantaged patients.
Keywords
Introduction
Patient-reported outcome measures (PROM) are a vital aspect in modern healthcare, as they provide objective measures that provide healthcare teams information to better understand patient’s health status, quality of life, functional status, and comorbid factors. The advent of the Patient-Reported Outcomes Measurement Information System Computer Adaptive Testing (PROMIS® CAT) has enabled greater efficiency in collection of PROM. PROMIS® consists of a series of domain-based surveys that provide a more universal assessment of health outcomes than legacy disease-specific PROM instruments. 1 The CAT format substantially shortens survey length, as the survey adapts based on each subsequent patient answer to avoid unnecessary questions. Uncertainty remains regarding the best way to capture outcomes scores for all patients, especially in the outpatient setting. Current strategies used to obtain PROM include online patient portals (OPPs), where an email/phone number links patients to a web connection for survey completion prior to a clinical visit. Alternatively, the patient can complete the survey at the time of visit using an in-clinic tablet (ICT).2,3 Completion of questionnaires before clinic visits using an OPP is advantageous for clinic flow and patient visit length due to less time being spent completing PROM surveys at the time of visit.
Health literacy (HL) is defined as “the ability to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”4,5 Health literacy and other patient demographics may play a role in the ability of patients to complete forms and utilize electronic methods for data collection. 6 Older patients have trouble with both digital and functional literacy due to age-related cognitive declines, compounding the effect these disparities have on the health literacy of elderly patients. 7 Patients with lower health literacy may not be as able to understand medical language and may have difficulties filling out medical forms.4,5,7 Health literacy is complex, and understanding may fluctuate between healthcare interactions and the delivery of healthcare information.4,5,7,8 Demographic and socioeconomic factors may also play a role in the ability of the patient to fill out electronic forms; patients in rural states may have not have a smartphone or have no internet access, preventing them from filling out forms online.
The aim of our study was to examine factors associated with successful completion of PROM questionnaires by orthopaedic patients. Our two hypotheses were as follows: (1) patients with low HL are less likely to complete PROMIS® CAT forms, and (2) patients with low HL who do complete the PROMIS® forms are less likely to use the OPP and more likely to use an ICT. We also sought to determine which baseline demographic and patient factors contribute to both whether the patient completed the PROM survey and what method they used.
Methods
After obtaining institutional review-board approval, patient data (age, gender, race, address, and health literacy) were obtained through our institution’s central repository. All 4601 hip, knee, and spine patient encounters from our outpatient orthopaedic clinics were collected over a 1-year period (January 1 to December 31, 2020). Our institution, a tertiary medical center in a rural state, first deployed PROMIS® CAT to the spine and total joint clinics, which is why these patient groups were selected for this study. Patients that were given a PROMIS® intake assessment as part of their outpatient orthopedic clinic visit were selected, and all duplicate patient encounters were removed so that each patient was only recorded once and those that were out of state were excluded due to differences in state-level Area Deprivation Index (ADI) score for a total of 3926 total patients included in this study (see Chart 1). If patients were seen on more than one encounter, we included only their first PROMIS® survey-completion method. Strobe Flow Chart for Patient Study Inclusion Based on Residency.
As part of standard intake clinic workflow, patients were asked to complete a PROMIS® CAT in two domains: Pain Interference v. 1.1 and Physical Function v. 2.0. All patients with an email address on record were emailed a secure link, via OPP, to complete PROMIS® questions prior to their appointment. This was an optional (but encouraged) survey collected when patients logged in or set up their new appointment, and if they did not complete before their scheduled appointment, they were given a tablet in the clinic and encouraged to complete the survey in-person. Patients were given instructions from the front desk staff on how to complete the PROM surveys as well as information on their clinical purpose. The OPP used in this study was MyChart, part of the Epic electronic health record system (Verona, WI).
Health literacy (HL) was determined using the Single Question Health Literacy Screening Tool question. 9 This questionnaire asks patients: “How confident are you filling out medical forms by yourself?” Responses range from “Extremely” to “Not at all,” with each being assigned a numerical value of 1-5, respectively. Answers from this questionnaire were dichotomized into high-HL (1-2) or low-HL (3-5). 9 This methodology has been validated in previous health literacy studies. 10 The HL screening is required in our institution every 3 years for any patient seen in the inpatient or outpatient setting as a best-practice notification, and only patients who answered this question were included in our analysis.
Using the University of Wisconsin’s Neighborhood Atlas, patient addresses were used to score each patient’s ADI for both the state and national levels. 11 For clarity, a lower ADI score indicates better socioeconomic status. Because a PO box would provide a null value, the patients that had only a post office box or “bad address” listed in the address section in the original patient list were cross-referenced by medical record number (MRN) in the electronic health record database and the address listed on the photo ID provided at the latest encounter was used to search for ADIs. Patients were omitted if ADI scores were unavailable because of no address listed, having no valid address on their photo ID, or being from out-of-state to keep continuity in state ADI levels.
To examine non-completion rate statistically, we divided patients by whether they completed vs Did Not Complete (“DNC”) the PROMIS® CAT. To examine completion method statistically, we took the patients that completed the PROMIS® Assessment Survey and subdivided them by whether they used the OPP or an ICT to complete it. We considered completion of at least one of the two PROMIS® CATs as sufficient for inclusion. All analysis was performed with SAS v9.4 software (SAS Institute Inc., Cary, NC, USA), and all comparisons employed a 2-sided alpha = 0.05 significance level.
Our initial analysis proceeded as follows. First, we summarized the continuous factors (age, national ADI scores, and state ADI scores) by completion status (DNC vs completed) as means and standard deviations (SDs) and compared them for status-group differences with Wilcoxon’s rank-sum test. Then we summarized and compared them the same way by completion method (OPP vs ICT). The categorical factors [factor levels] were Gender [male and female], Race [Black, White, and Other], dichotomized HL [low and high], and Subspecialty [hip/knee and spine]. For each factor, first we summarized the data within factor levels as the number (percentage) DNC and the number (percentage) that completed the survey, and used Pearson’s chi-square test to assess DNC-rate differences between factor levels among all participants. Then we repeated this process to assess OPP-rate differences between factor levels among participants that completed the survey.
After this initial analysis, we conducted multivariable analysis using log-binomial regression to examine whether HL in particular was independently significant for either DNC rate or OPP rate after adjusting for all other factors being studied.12,13 We chose log-binomial regression over the more commonly used logistic regression in order to compare rates of DNC and OPP use directly instead of through odds ratios. In both of our multivariable models, we included national ADI but excluded Arkansas ADI because of the high collinearity between them. To address the question of whether the patterns of multivariable association with either DNC rate or OPP-use rate change between subspecialties, we undertook interaction analysis. Briefly, we augmented the original or “main effects” multivariable model with five additional interaction terms, one for the Subspecialty interaction with each of the five other factors. Then we compared the augmented model to the main-effects model via Likelihood Ratio Test (LRT) for the improvement in how well the augmented model fits the data. If the LRT yielded a statistically significant result, then we used the augmented model to estimate separate rate ratios (95% confidence intervals) for the hip/knee submodel and the spine submodel, and displayed these estimates side-by-side accompanied by interaction P-values to indicate which multivariable rate ratios differed significantly between subspecialties.
Results
Sociodemographic Characteristics of Participants Versus (1) Whether They Completed and (2) How They Completed Their PROMIS Assessment Survey.
DNC, did not complete; OPP, online patient portal; ICT, in-clinic tablet; ADI, area deprivation index. Higher ADIs are associated with lower socioeconomic status.
†P-values are from either Pearson’s chi-square test or ‡Wilcoxon’s rank-sum test; P-values in
aPercentage of the Group Total.
bPercentage of the patients that completed the PROMIS survey.
With respect to the other sociodemographic factors (Table 1), average ages were only 0.3 years different between those who completed the questionnaire and those who did not (P = 0.228), but within the completers, the average age was 1.7 years lower among those who used the OPP (P = 0.024). Similarly, the survey completion rate in all patients was only 2% different between males and females (P = 0.210), but among those who completed it, the rate of OPP use was almost 4% lower in males (P = 0.033). In terms of race, fewer than half of Black patients (46.5%) completed the assessment compared to 57.4% of White patients and 57.3% of Other races (P < 0.001). Among the completers, the rates of OPP use were 13.3% for Black patients, 26.1% for White patients, and 19.4% for members of Other races (P < 0.001). Relative to patients who completed the questionnaire, participants who did not had higher average ADI scores, i.e., more than 4 percentiles higher on National ADI and almost half a decile higher on Arkansas ADI (P < 0.001 for both). Among the patients who completed it, those who did so with OPP scored an average of 8.1 percentiles lower on National ADI and 0.96 deciles lower on Arkansas ADI than those who used ICT (P < 0.001 for both). Finally, Table 1 shows that the survey non-completion rate was 1.46-fold higher among spine patients (59.0%) than among hip/knee patients (40.4%) (P < 0.001), but also shows a 1.81-fold higher rate of OPP use among survey completers that were spine (37.4%) vs hip/knee (20.7%) (P < 0.001).
Log-Binomial Regressions of Rate of DNC (Did Not Complete) on Sociodemographic Factors (N = 3296 Participants).
DNC, did not complete. ADI, area deprivation index. Rate Ratio, the ratio of DNC rates.
§Regression results are from Univariable and Multivariable log-binomial regressions of DNC rate on the sociodemographic factors. The Univariable Rate Ratios for categorical factors are the ratios of the corresponding DNC rates from Table 1.
†95% Confidence Interval.
‡P-value.
P-values in
Multivariable Log-Binomial Regression “Submodels” by Subspecialty of DNC Rates Versus Sociodemographic Factors, With Tests for Factor Interactions With Subspecialty.
DNC, did not complete. ADI, area deprivation index. Rate Ratio, the ratio of DNC rates.
§Submodel estimates are from the Multivariable log-binomial regression of DNC rate on sociodemographic factors when the regression model includes the interactions of Subspecialty with the five other factors. The Rate Ratio, †95% Confidence Interval, and ‡P-value describes the sociodemographic factor’s relationship to the DNC rate within each subspecialty, whereas the ⁑Interaction P-value indicates whether that sociodemographic factor’s relationship changes significantly between subspecialties. P-values in
Log-Binomial Regressions of Rate of OPP (Online Patient Portal) Use Versus Sociodemographic Factors Among the 2166 Participants That Completed the PROMIS Assessment Survey.
OPP, online patient portal. ADI, area deprivation index. Rate Ratio, the ratio of OPP-use rates.
§Regression results are from Univariable and Multivariable log-binomial regressions of OPP-use rate on the sociodemographic factors. The Univariable Rate Ratios for categorical factors are the ratios of the corresponding OPP-use rates from Table 1. †95% Confidence Interval. ‡P-value. P-values in
Discussion
Our study found that health literacy may be an important factor in determining if and how patients choose to complete PROM. Patients with low health literacy were 1.23-fold less likely to complete the PROM survey in any manner, and when they did complete it, they were 2.71-fold less likely to use the OPP. Previous studies have shown that patients with poor health literacy have worse patient reported outcomes, worse health status, and lower quality of life. 14
The impact of health literacy on successful completion of electronic PROM instruments has not been well reported. With the advent of the PROMIS® CAT, a digital device is required to enter questionnaire responses, which may bias results when tracking surgical outcomes. Providers taking care of patients in rural states with lower socioeconomic status and higher ADI may find it more difficult to collect results. As insurers and Medicare shift to reimbursement models based on PROM collection rates, this may influence the ability of physicians to demonstrate value in less affluent clinical practices. While electronic patient portals can foster greater patient satisfaction and better communication for patients outside of the ambulatory visit, a greater reliance on digital tools has the potential to increase discrepancies in engagement with those that have limited access or understanding, thereby exacerbating existing health disparities. 15 Recognizing that those who did not fill out their PROM at all, and those who did not fill it out via OPP, as both being more likely to have poorer health literacy could allow for accommodations in order to hopefully achieve better physician-patient communication and improved health outcomes. The literature regarding disparities in digital health is new and evolving, but a set of practical action steps has been proposed by Smith and Magnani. They advocate intuitive design of digital health interfaces, emphasizing ease of use that incorporates patient feedback on usability. Clinic staff should make a specific effort to ask patients regarding their preferred method of electronic communication.
The number of patients who did not complete PROMIS® CAT surveys was notable. Our findings that this group of patients had lower HL, worse ADI, and were more likely to be Black suggest that, even with our efforts to provide multiple methods for PROM completion, we were still missing these crucial data on a large subset of at-risk patients. Each institution likely will face unique challenges to achieving 100% PROM completion, but we do think our findings are likely common to many hospitals. Further educating front-facing clinic staff to help patients understand the reasons for PROM completion and helping them navigate the ICT interface are strategies to mitigate lack of response.
Effective provider-patient communication is essential for patients to share information more openly with their healthcare providers, leading to more proper diagnoses, better adherence to treatment plans, and higher patient satisfaction. 16 Early recognition of patients with potentially lower health literacy may facilitate better communication and use of clinical time during patient encounters. Strategies to achieve this goal include having prepared printed materials, scheduling slightly longer appointments, and allocating time for the patient-provider teach-back method, which involves asking patients to repeat key information in their own words to help patients retain what they have learned during their visit.4,17 Identifying patients who haven’t completed their PROM survey via OPP, i.e., completed it before their visit, and recognizing the potential link of not completing it before their visit to lower health literacy as well as lack of access to broad-band internet service, may allow adequate preparation ahead of scheduled appointments to allocate resources to provide more effective communication as well as provide a functioning device (the ICT) that they can complete their PROM survey. Further investigation is needed to improve PROM survey collection in low health literacy groups. This may include better patient portals with easier access and further education as to the importance of patient portal use.
A previous study by Long et al shows there are multiple distinct patient-level factors that affect PROM completion. Failure to consider these factors in PROM design and implementation may lower completion rates or prevent accurate completion, undermining PROM validity. In our population, unappreciated factors such as patient vision, print literacy, and aptitude with digital platforms likely contributed to one’s ability to answer PROM questions and follow written instructions. Given our similar patient demographics, these factors affect minority populations at disproportionate rates, which may contribute to widening existing health disparities. 18
Bernstein et al showed that the rate of completion of PROM varies according to sociodemographic variables. This variability could bias clinical outcomes research in orthopaedic surgery. 19 Our results also show that socioeconomic factors affect PROM collection methods. Average ADI was shown to be approximately a full decile lower for those who submitted via OPP than those who submitted via ICT for both the state and national level (Table 1). This could be the result of confounding with variables tested as well as those not tested, such as computer and internet access. It has been reported that 15-24% of Americans lack broadband and internet connectivity. Up to 40% of low-income households lack an at-home internet subscription, requiring them to use cellular data plans (that may be limited) or public Wi-Fi to access the internet, thereby limiting their ability to use online health technology.15,20 We also noted that Black and Other non-White patients completed PROM via OPP less frequently than Caucasian patients. Previous literature has shown that access to internet is a social determinant of health and that minorities are less likely to have broadband internet accesses, in both urban and rural settings. 20 Lower socioeconomic status and higher ADI have both been shown to be predictive of hospital readmission rates, and the association of impaired health literacy in these statistics should not be discounted.21,22
Our analysis also found that, among those who completed the PROM assessment, each 10-year increase in age led to a 6% decrease in the probability of using the OPP. This could be explained by age-related changes in cognitive function that lower the reading ability of older age groups, and health literacy can be markedly lower in older patients, thereby leading to worse functional health. 7 Digital literacy has been shown to be a predictor of health as healthcare becomes more digitally based; thus the essential skills that a patient must have to benefit from information technology multiply and become more complex.15,23 Recognizing that elderly patients can have greater difficulties with functional literacy, digital literacy, and health literacy can allow for implementing measures to help them make more informed decisions about their health.
We found that spine patients completed PROM via the OPP almost 50% more frequently than hip and knee patients. We also noticed that spine patients tended to be almost 7 years younger and with a slightly higher composition of White patients than hip/knee patients, which may help explain why the effects of age and race on non-completion rate were stronger in the hip/knee subgroup. This is reminiscent of what was found in a Danish study investigating the characteristic differences in hip, knee, and spine pain patients. 24 Their study also found that more back-pain patients had a <3-month time between onset of pain and seeking treatment than knee/hip patients (19% vs 5-7%, respectively). 24 Moreover, spine patients in their study had a higher level of education. 24 Wright et al identified that social deprivation does not affect PROMIS® scores in patient populations across orthopaedic subspecialties equally. Notably, spine patients did not show significant differences in their PROs based on social deprivation, while total joints patients did. Understanding that different pathology subgroups of orthopaedic patients may have different health literacy and digital literacy capacities merits consideration in patient care, as well as further research investigation into why.
Our study has several limitations. We collected data during part of the COVID-19 pandemic. This could have created a situation in which prospective patients may have canceled appointments, been more motivated to use the OPP over the ICT, or even refused the survey all together due to hygiene concerns, possibly skewing results. Patients had to “op in” to set up a portal either on a smartphone or computer, and provided an email address to be contacted via the OPP. We do not have the information that would allow us to know if all patients set up this feature. Our study did not account for the lack of broadband or email access that plagues rural southern states. 25 The “2020 Broadband Deployment Report” produced by the Federal Communications Commission found that 22% of the rural population does not have access to viable broadband internet service, regardless of ability to pay for service. 26 Therefore even if email correspondences were attempted to be sent to all patients, it is likely that not all patients received such messages and thus were unable to set up online portal access. Inability to access the internet is a major determinate in a patient’s ability to use OPPs, and racial minorities and elderly patients are more likely to live in areas that have poor broadband internet access, and therefore be less likely to access an OPP. 24 Sisodia et al showed that during the COVID-19 pandemic, profound racial disparities occurred due to relying on portal use for PROM completion. 27
Further possible limitations to the study include patients with marginal reading ability that may not acknowledge they need help with reading comprehension. Patients may be in denial or ashamed of their need for help with their literacy problem, or patients just may not understand the question being asked.4,28 This may influence their response to the validated health literacy screening tool we utilized. Finally, the rate of PROM non-response was quite high; we attribute this to reluctance of patients to undertake a new aspect of their clinic experience (including touching a tablet in clinic during the pandemic), however, this provided an opportunity to evaluate demographics of “non-responses” in a large, real-world study population.
Conclusion
Our results demonstrate that patients’ health literacy, demographics, and socioeconomic status affect both whether they complete their PROMIS® questionnaire and what method they use when they do. Additional efforts should be made to understand these factors, accommodate patients, and facilitate accurate and complete PROM responses; this is especially true in hospitals serving diverse and socioeconomically disadvantaged patients.
Footnotes
Authors’ Note
Institutional IRB #262252 at The Univeristy of Arkasnas for Medical Sciences; accepted 6/25/21.
Acknowledgements
Statistical analysis was performed by Mr Eric Siegel (UAMS statistician).
Statements and Declarations
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The project described was supported by the Translational Research Institute (TRI), grant UL1 TR003107 through the National Center for Advancing Translational Sciences of the National Institutes of Health (NIH). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. In addition, the Bill and Betty Petty Research Fund at the University of Arkansas for Medical Sciences supported this work.
Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
IRB Approval
UAMS approval # 262252.