Detecting professional interpreter use among patients with limited English proficiency: Derivation and validation study

Abstract

Objectives:

The objective of this derivation and validation study was to develop and validate a search strategy algorithm to detect patients who used professional interpreter services.

Methods:

We identified all adults who had at least one intensive care unit admission during their hospital stay across the Mayo Clinic Enterprise between 1 January 2015 and 30 June 2020. Three random subsets of 100 patients were extracted from 60,268 patients to develop the search strategy algorithm. Two physician reviewers conducted gold standard manual chart review and any discrepancies were resolved by a third reviewer. These results were compared with the search strategy algorithm each time it was refined. Sensitivity and specificity were calculated during each phase by comparing the search strategy results to the reference gold standard for both derivation cohorts and the final validation cohort.

Results:

The first search strategy resulted in a sensitivity of 100% and a specificity of 89%. The second revised search strategy achieved a sensitivity of 100% and a specificity of 87%. The final version of the search strategy was applied to the validation subset and sensitivity and specificity were 100% and 89%, respectively.

Conclusion:

We derived and validated a search strategy algorithm to assess interpreter use among hospitalized patients. Using a search strategy algorithm with high sensitivity and specificity can reduce the time required to abstract data from the electronic medical records compared with manual data abstraction.

Introduction

The number of residents in the United States with Limited English Proficiency (LEP) has grown in recent decades. ¹ According to the 2017 US Census, more than 64 million people aged 5 years and older speak a language other than English at home, and more than 25 million of the US population are classified as “speaking English less than very well” or having LEP.^2,3 Furthermore, approximately 60,000 to 85,000 patients travel to prestigious medical centers in the United States every year for treatments not available in their native countries and frequently these patients also have LEP.^4,5

Although there is a federal mandate for language services deployment when patients with LEP navigate the healthcare system, interpreters are underused. Negative health outcomes and suboptimal healthcare quality have been documented for patients with LEP.^6–14 However, integrating medical interpreting standards into clinical practice has been challenging.^15–17 Research demonstrates the benefits of using interpreter services in clinical practice. These include improved communication, improved patient and family satisfaction, and improved adherence to treatment plans. Other beneficial healthcare outcomes include the reduction of complications and length of hospital stay, as well as increased use of preventive healthcare services.^{11–13,18–21}

Despite evidence of the advantages of using interpreters, physicians still try to “get-by” with the use of their own limited language skills or those of family members as interpreters.^15,22–24 Physician behaviors may be influenced by the lack of access and availability of professional interpreter services based on organizational structures and resources. ²⁵ Evidence suggests that professional interpreters provide higher quality interpretation than family members and therefore should be used.^26,27

It is important to understand the use of interpreters in the clinical setting to be able to optimize systems to increase access to interpreters for patients who need one. This knowledge will be a useful step toward sustained change to address disparities among those with LEP.

The adoption of electronic health records (EHRs) in place of traditional paper charts has provided unprecedented amounts of information (“Big data”) that allows researchers to evaluate larger cohorts of patients than traditional research approaches.^28,29 Harnessing the potential of the EHR and the vast amounts of data within is challenging but developing and applying automated search strategies is a helpful approach. By improving validated algorithms to identify when an interpreter is utilized, we can support quality improvement initiatives, enhance clinical practice, and improve our research accuracy and efficiency.

Methods

Setting and study design

This is a derivation and validation cohort study which was approved by the Mayo Clinic Institutional Review Board (IRB). The IRB reviewed the study and the procedures and deemed the study exempt. This study did not include patient contact. We only reviewed and included the EHR of patients who had provided research authorization in accordance with Minnesota state statutes if applicable. ³⁰

Study participants

We included (1) consecutive adult patients (⩾18 years), (2) with research authorization if the state required it, (3) admitted to the intensive care units (ICU) across the Mayo Clinic enterprise (Minnesota, Wisconsin, Arizona, and Florida), (4) between 1 January 2015 and 30 June 2020 (5½ years). The search strategies to identify when an interpreter was used incorporated the interpreter flag indicating an interpreter was needed and a combination of a preferred language other than English and interpreter flow sheets. Patients who did not need an interpreter or whose primary language was listed as English were randomly selected for conducting the derivation and validation. If a patient was admitted several times during this period, only the first admission including an ICU stay was included.

Our search strategy was designed to detect those who used professional interpreters including phone, video, or in-person interpreters. Since this study focused on the use of professional interpreters, we excluded encounters in which interpretation by family, friends, and healthcare team members occurred. Three random subsets of 100 patients were used for the derivation and validation phases (see Figure 1). No sample size calculation was performed. This size cohort has been used in previous similar derivation and validation studies we have conducted and is considered the accepted and standard size for this type of work.^28,29

Figure 1.

Enrollment of admissions to derivation and validation cohort.

Manual data extraction strategies (reference gold standard)

Before developing the automated search strategy and to formulate the gold standard for interpreter use, members of the study team (J.S., A.B., and S.F.) manually reviewed the EHRs of random patients classified as having a preferred language other than English and those patients whose EHR indicated that an interpreter was required. The study team examined the different ways that the “interpreter used” was documented by healthcare providers in the EHR including within flowsheets, progress, and encounter notes.

During the derivation and validation process, the reference gold standard comparison involved EHR manual review by two physician researcher reviewers. They examined the electronic medical record between the specified admission dates to assess whether an interpreter was used. They did this by reviewing the medical note documentation and the flowsheets as well as the patient-provided information. Two reviewers (J.S. and S.F.) conducted gold standard electronic medical record reviews and any disagreements were resolved by a third reviewer (A.B.). We have used this approach in other studies in which manual chart review is the gold standard. ³¹

Automated electronic search strategy

Although “interpreter flags” appear in the patient-provided information of the EHR, this does not specify whether an interpreter was actually used, simply that an interpreter was needed or that the primary language was not English. The search strategy algorithm was developed in several stages. The first derivation used patient-provided information in the EHR specifically “Interpreter Indicator” (which indicates if a patient needs Interpreter Services) and “Preferred Language.” We also used applicable interpreter flowsheets from the EHR. The second derivation used modifications to the flowsheet search and interpreter needed indicator. The preferred language was not included in the second search strategy. We used the interpreter indicator search in the ICU DataMart. DataMart is an extensive data warehouse containing a near-real-time normalized replica of Mayo Clinic’s EHR. ³² We accessed the DataMart warehouse and searched the data by using JMP software. DataMart contains patient demographic characteristics, diagnoses, laboratory results, and clinical flow sheets, gathered from various sources within the institution. ³² The data within DataMart has been validated and is reliable.^32–34

We also used an interpreter indicator search in Mayo Clinic’s Advanced Cohort Explorer (ACE), an electronic retrieval query database within Mayo Clinic’s Unified Data Platform (UDP). ACE is a powerful web-based software toolset that enables the search of the EHR by specific text phrases or terms in specific parts of the clinical notes. All data extracted by ACE can be exported to Excel to enable further statistical analysis. ²⁸ Each subset sample contained 100 patients who were randomly selected from DataMart. (Figure 1) These cohorts consisted of both those likely to use interpreter services and those not likely to use interpreter services (see Figure 1). In order for patients to be categorized as “Interpreter used” using the automated search strategy, the patients needed to have both an interpreter indicator “Yes” within their demographic information as well as an applicable flowsheet articulating that an interpreter was used at the time of the encounter we were examining.

Following the manual review of each of the derivation subsets, A.M. refined the electronic search strategy through several iterations of evaluation and refinement of the electronic search algorithm in the derivation cohorts. Once the search strategy was assessed and modified in the derivation cohorts, it was validated in the third cohort.

Statistical analysis

An overall percent agreement between the electronic search algorithm and the manual EHR review was calculated. Sensitivity and specificity were calculated by comparing the results to the reference gold standard for each derivation and validation subsets. JMP statistical software (version 10.0.0; SAS Institute Inc., Cary, NC) was used for all analyses. Furthermore, we calculated a kappa statistic to assess the agreement between reviewers 1 and 2 conducting gold standard manual EHR reviews.

Results

The cohort of those with hospital admissions and at least one ICU admission across the Mayo Clinic Enterprise between 1 January 2015 and 30 June 2020 was 60,268 admissions.

Primary outcome

The primary outcome of the study was to derive and validate a search strategy to identify when an interpreter was used by patients with LEP by assessing sensitivity and specificity with classification performance.

In the first derivation cohort, our manual review resulted in a sensitivity of 100% (95% confidence interval [CI] 93.5–100) and a specificity of 89% (95% CI 75.9–96.3). The supervised algorithm was used for the second derivation subset. The second derivation achieved a sensitivity of 100% (95% CI 92.6–100) and a specificity of 87% (95% CI 74.2–94.4). The final version of the search strategy for interpreter use was applied to the validation subset and this achieved a sensitivity of 100% and a specificity of 89% (95% CI 75.9–96.3) (Table 1). Kappa agreement between reviewers with the first derivation was 0.90, and with the second derivation and validation were 0.86, and 0.88, respectively. These can be construed as near perfect agreement ³⁵ (Table 1).

Table 1.

Sensitivity and specificity of subset groups.

	Sensitivity (%) (95% CI)	Specificity (%) (95% CI)
1st derivation cohort	100% (93.5–100)	89% (75.9–96.3)
2nd derivation cohort	100% (92.6–100)	87% (74.2–94.4)
Validation cohort	100% (93.5–100)	89% (75.9–96.3)

CI: confidence interval.

Discussion

In this study, we developed a very sensitive and specific automated search strategy for identifying patients who used an interpreter. The final validated search algorithm was able to identify interpreter utilization within a cohort with 100% sensitivity and 89% specificity.

Currently, almost all healthcare institutions in the United States have adopted an EHR to integrate distributed data sources.^36–38 The large amount of data within the EHR presents great potential if strategies to examine the data can be developed. Otherwise, the large amount of data may present barriers to clinical research.^39,40 Accessing big data usefully through the EHR and abstracting needed information for clinical research using manual data extraction methods is time-consuming and inefficient.^29,41 The time required to conduct manual EHR review to identify if a patient used an interpreter is about 10 min per patient which is not feasible for researchers studying large data sets.

Strengths and limitations

Strengths of the study include the following. We used robust approaches during the derivation and validation processes with an experienced team. We did gold standard EHR review by having two reviewers and a third reviewer to resolve discrepancies. Our kappa statistics are 0.86–0.90 and therefore demonstrate near perfect agreement. Despite the high sensitivity, which remained at 100% in the final cohort, the specificity of the validation subset did not increase beyond 89%, meaning there is a small possibility of false-positive results. The differences between the first, second, and third cohorts are not statistically significant.

Based on estimated times required to conduct manual EHR review during this study, we believe our search strategy will prove very useful for future identification of patients who used an interpreter across our healthcare enterprise. Furthermore, it will provide useful foundational knowledge to build more real-time algorithms to identify patients who would benefit from an interpreter. Other institutions that use EPIC may also be able to leverage this approach to identify patients who used or need an interpreter. Based on 2019 data, EPIC has a market share of almost one-third of acute care multi-specialty hospital EHRs. It is becoming increasingly dominant and being adopted by healthcare systems nationwide. ⁴²

The study has some limitations. It is worth noting that we were able to leverage institutional software and specific data sets (ACE and DataMart) to conduct our study and these may not be available in all institutions. Therefore, external validation of this search strategy is needed especially if different electronic infrastructures exist in those institutions. Other institutions with diverse computational infrastructure may need to modify our search strategy within their patient data sets and EHRs.

We wanted to focus on professional interpreter use in this study as there is evidence that unless absolutely necessary, or in an emergency situation, interpretation by family and friends should be avoided.^43,44 Using family and friends as interpreters threatens the accuracy and completeness of interpretation. This can lead to vital information being inadvertently or deliberately omitted or misinterpreted.^42–47 Using clinical team members who do not have sufficient language skills can also lead to information being inaccurately interpreted compromising communication.^44,46

In comparison to other papers describing validation and derivation of automated digital algorithms in which sensitivity varied from 77% to 100%, and specificity ranged from 91% to 99.7%, the results of this study are favorable.^28,33,34 Amra et al.’s ²⁸ automated electronic search algorithm sensitivity ranged from 94% to 97% and specificity ranged from 93% to 99%, while Singh et al. achieved a sensitivity between 91% to 100% and specificity from 98% to 100%. ³³ Others automated electronic queries demonstrated sensitivity ranging from 77% to 100% and specificity ⩾96%. ³⁴

The purpose of this study was to develop an electronic automated search algorithm to accurately and reliably detect patients who used professional interpreter services. This could then be used to reduce the time and effort needed to identify those who used interpreter services in a large data set. It is important to better understand professional interpreter use to foster strategies to increase timely and appropriate interpreter use in near real-time. Interpreter use can mitigate disparities experienced by patients with LEP.^43–46 Interpreters can reduce cultural, language, and literacy barriers by improving communication between patients and clinicians. ⁴⁸

Conclusion

We have successfully derived and validated an “Interpreter used” search strategy that identifies if a patient used a professional in-person interpreter, video-linked interpreter, or telephone interpreter. It can be deployed in our enterprise EHR demonstrating a sensitivity of 100% and a specificity of 89%. This method of electronic data extraction by an automated algorithm through institutional software connected to EHR is accurate, time-saving, and cost-effective.