Outcome After Self-Triage App Referral in Urgent Direct-to-Consumer Telemedicine Encounter

Abstract

Background:

The quantification of self-triage effectiveness, guided by mobile applications, in urgent direct-to-consumer telemedicine (TM) encounters requires further investigation. The objective of this study was to evaluate the outcomes of referral guidance provided by a symptom-based self-management mobile application decision algorithm in the context of remote urgent care assessments.

Methods:

An observational retrospective single-center study was conducted from May 2022 to December 2023. The inclusion criteria encompassed individuals aged >18 years old, and those spontaneously seeking virtual emergency care through the EINSTEIN CONECTA application. Patients experiencing connectivity issues, preventing completion of the encounter, were excluded. The primary outcomes included the rate of patient concurrence with the algorithm's recommendation for seeking in-person emergency care and the referral rate to face-to-face assessment among cases evaluated through TM. The application's algorithm employs scientific evidence based on symptoms to recommend referrals to emergency departments (EDs).

Results:

Out of 88,834 patients connected to the TM Center, self-triage obviated the need for virtual physician assessment in 53,302 (60%) encounters. A total of 35,532 patients were remotely evaluated by 316 on-duty physicians, resulting in 1,125 ICD-coded diagnoses. Among these, 21,722 (61.1%) were initially advised by self-triage to visit the ED, with subsequent medical assessment leading to in-person referrals in 6,354 (29.3%) of the evaluations. Of the 13,810 patients recommended to continue with virtual care post-self-triage, 157 (1.1%) were referred for in-person assessment.

Conclusions:

Self-triage effectively reduced the need for physician encounters in approximately three-fifths of TM consultations. Despite being based on scientific evidence, symptom-based referral algorithms demonstrated high sensitivity but poor correlation with physician decision-making.

Introduction

Episodic care centers, incorporating a variety of health care modalities including direct-to-consumer telemedicine (TM) enabled by smartphones, are an emerging aspect of health care delivery. ¹ The introduction of mobile health (m-Health) technologies has significantly expanded their diversity and functional capabilities. However, the effectiveness of these online tools in enhancing health care delivery within systems remains limited. ²

m-Health applications are distinguished by their user-friendliness and strong inter-rater reliability. ³ Symptom assessment is crucial in medical evaluations, making the accuracy of symptom checker software in patient self-management a key focus of recent investigations. ⁴ A comprehensive analytical review of 31 studies, examining patient self-management of chronic conditions, identified a wide range of methodologies employed by mobile applications, with diverse interaction modalities. Despite extensive research, consensus on the most effective approach for optimizing health care is yet to be reached. Nevertheless, some studies suggest a slight advantage in managing populations with high needs and high costs. ⁵

Effective virtual triage is essential for clinical risk management, especially in safely and efficiently managing patient flow in situations where demand exceeds available resources. ^6,7 In acute care settings, the triage recommendations provided by symptom checker applications are generally conservative; however, they face limitations in triage accuracy and diagnostic capabilities. ⁸

This study aims to examine the effectiveness of self-triage algorithms integrated within TM connection applications, with a focus on early referrals to emergency departments (ED) in high-risk scenarios. Current knowledge on the efficacy of symptom-based self-triage, especially immediately after a virtual medical consultation, is limited. We hypothesize that patients adhere to app guidance for red-flag detections in self-triage even after seeking virtual physician consultation. Therefore, our research aims to assess the impact of referral guidance from a symptom-based self-management mobile application algorithm in the context of remote urgent care assessments.

Methods

STUDY DESIGN AND PARTICIPANTS

This single-center retrospective study was conducted at the Telemedicine Center of Hospital Israelita Albert Einstein, in São Paulo, Brazil. The study and consent waiver (the data were analyzed anonymously) were approved by the Hospital Israelita Albert Einstein Review Board (registration number CAAE 74197023.2.0000.0071), and were designated the TeleConnect study. Access to all data is available through the institutional digital records. Data collection and confidential storage were managed by TM physicians, with all analyses performed by the TM Center coordinating the study. The article was collaboratively written by all authors, who also decided to submit it for publication and attest to the integrity of the data.

The study population comprised patients aged >18 years, who spontaneously sought care at the Virtual Emergency Department from May 2022 to December 2023. Inclusion was universal for patients presenting with any condition. The exclusion criterion was limited to connectivity issues preventing the creation of medical records, as these participants could not complete a medical assessment, leaving no data record in the database.

MOBILE APPLICATION CONNECTION AND REFERRAL ALGORITHMS

The EINSTEIN CONECTA app, a proprietary mobile application, was designed and developed adhering to current best practices and standards for software engineering and end-user experience. Its self-triage algorithm is based on a logical decision tree, computationally implemented and maintained by a team of Information Technology and TM specialists. Figures 1 and 2 provide an example of the decision tree algorithm and the EINSTEIN CONECTA user interface, respectively.

Fig. 1.

Self-triage algorithm for COVID-19 translated into English, showing the decision tree structure of the EINSTEIN CONECTA app (a full version is available in the Supplementary File S1). The application's homepage prompts the user for the consultation reason, displaying options such as COVID-19, diarrhea, nausea, abdominal discomfort, headache, and musculoskeletal pain. It includes questions related to COVID-19 symptoms, previous consultations, and exposure to confirmed cases, as well as red-flag queries about breathlessness and chest pain, requiring yes or no responses.

Fig. 2.

Self-triage app layout example assessed by smartphone (Portuguese language). (1) The application's home page requests the reason for the consultation. This image shows COVID-19, diarrhea, nausea, abdominal discomfort, headache, and low back, muscle, or joint pain. (2) Questions regarding the condition related to COVID-19: I have suggestive symptoms; I have already had a consultation, and it got worse; I have already been to consultations, and I have doubts; I am asymptomatic and have had contact with someone who has COVID-19. (3) Question A to red-flag identification: Do you have breathlessness? Yes or No answer. (4) Question B to red-flag identification: Do you have chest pain? Yes or No answer.

PRIMARY OUTCOMES

The primary outcomes measured were the rate of patient agreement with the algorithm's recommendation for seeking emergency in-person care and the referral rate to emergency in-person care among cases evaluated by a physician through TM.

DATA EXTRACTION

Self-triage data were extracted from the transactional database of the EINSTEIN CONECTA app. By matching patient identification codes with timestamps marking the completion of user navigation flows, we queried the institutional database for TM consultations corresponding with EINSTEIN CONECTA navigation paths. This process yielded a comprehensive data set for self-triage and TM consultations, facilitating subsequent statistical analysis.

STATISTICAL ANALYSIS

Statistical analyses were performed using IBM SPSS Statistics for Windows, Version 22.0. The analysis was purely descriptive, with categorical variables presented as counts and percentages. There were no missing data.

Results

A total of 88,834 patients connected to TM Center and the self-triage prevented virtual assessment by physician in 53,302 (60%) encounters. In total, 35,532 patients completed the connection to the TM Center using the EINSTEIN CONECTA app after answered the self-triage symptom-based questions proposed by the algorithm. They were remotely evaluated by 316 doctors on duty and received 1,125 ICD-coded diagnoses. Of those, 21,722 (61.1%) had previously been referred to the ED by self-triage and medical assessment led to a referral in 6,354 (29.3%) evaluations. Regarding 13,810 (38.2%) patients guided to continue to a TM encounter by app triage, 157 (1.13%) were referred for face-to-face assessment in the ED (Fig. 3).

Fig. 3.

Outcome after recommendation of the self-triage app in urgent direct-to-consumer telemedicine encounter. ED, emergency department; TM, telemedicine.

Discussion

Digital health technologies, including m-Health, diverge from many traditional medical products and services by being often sold directly to consumers. This direct-to-consumer approach facilitates access to health-related data and services, potentially improving care accessibility for those facing barriers to traditional health services. However, this model circumvents standard health care system safeguards, eliciting concerns over the proliferation of low-value or potentially harmful products. ⁹ Thus, optimizing TM encounters is paramount for patient safety, ensuring assertive referrals for in-person evaluations in EDs, and promoting rational resource use. ¹⁰

Virtual triage is essential for managing clinical risks, allowing for effective and appropriate patient influx management, particularly regarding early referrals for face-to-face ED evaluations. ¹¹ Currently, symptom checking by a physician or through self-triage by an app represents the primary method for TM triage. ¹² As of January 2024, 97% of Americans own mobile phones, with 85% owning smartphones and 15% being smartphone-only internet users. ¹³

Globally, the user base of mobile phones is estimated at ∼4.68 billion, catalyzing a substantial increase in m-Health technologies designed to improve various life aspects. Despite projections of a 41% annual growth in the medical app market from 2015 to 2020, trends indicate reduced app downloads and diminished user engagement. ¹⁴ It is crucial to note that the unavailability of most m-Health apps on iOS or Android app stores limits their effectiveness, positioning m-Health as a strategy worth exploring for health care system optimization. ¹⁵

The TM Center in this study, accessed by patients through the EINSTEIN CONECTA app developed by the company, formulated a self-triage algorithm based on scientific evidence aimed at high sensitivity in red-flag detection, facilitating early ED referral recommendations. Surprisingly, almost three-fifths of patients followed the immediate ED referral recommendation, not awaiting virtual medical evaluation—a result not anticipated given factors such as local ED overcrowding, misconceptions about symptom severity, limited prior experience with self-triage, and cultural beliefs. Previous evidence suggests a preference among patients for TM consultations with doctors with whom they have established relationships. ¹⁶

This study also found minimal correlation between the guidance for immediate ED referral and the advice provided by the TM physician on duty, allowing exploration of the appropriateness of referrals by the self-triage algorithm for both immediate ED demands and continuation in TM. Only 29.3% of patients were referred by physicians for face-to-face evaluations, indicating that although symptoms are crucial in remote assessments and self-triage algorithms derive from guidelines also used by doctors, medical assessments in virtual care reclassified two-thirds of patients to safer scenarios. Conversely, only 1.13% of patients were incorrectly screened, underscoring the high sensitivity of self-triage.

Previous evidence on deploying self-triage tools in TM offers mixed results. Although they achieve an average accuracy of 58%, these tools have advantages over traditional triage methods, especially for individuals reluctant to use telephone triage or unable to visit physical health care centers. However, the accuracy of these tools does not necessarily translate to improved patient outcomes or efficient service use, often due to their conservative risk-averse design. ⁸ The integration of self-triage applications in TM presents opportunities and challenges.

For instance, a study conducted at Marburg University Hospital involving 378 patients demonstrated a promising safety rate of 94.7% when comparing the Ada app's recommendations with the Manchester Triage System, suggesting self-triage tools could potentially reduce unnecessary ED visits. ¹⁷ However, the varying accuracy of online symptom checkers, as highlighted by Riboli-Sasco et al., underscores the need for cautious application and further research to validate these tools. ⁴ An observational study in Japan within the pediatric population used a mobile app for self-triage by 59,375 patients (parents or guardians) presenting acute complaints, providing insights into the clinical profile of complaints without addressing patient behavioral conduct. ¹⁸

The real-world impact of self-triage tools on health care utilization remains incompletely understood. The paradoxical increase in primary care visits and hospital admissions after the introduction of online personal health records highlights the complexity of health care behaviors and utilization patterns. Consequently, there is an urgent need for comprehensive research to elucidate how these tools are utilized, the user response to their recommendations, and their ultimate impact on health care resources and patient outcomes. ¹⁹

A notable challenge with self-triage tools is their inability to replicate the nuanced clinical rapport characteristic of traditional triage methods, such as telephone-based triage. Furthermore, these tools often fail to accurately capture the subtleties of patient symptom descriptions and symptom prioritization, which can vary significantly between patients and health care professionals. ²⁰ Until advancements in technologies, such as affective computing, are incorporated, self-triage tools may not fully supplant the need for human triage in specific scenarios. ²

Despite these limitations, certain self-triage tools have demonstrated sufficient promise to justify further investigation through randomized trials. Such studies could facilitate the integration of self-triage tools into a multichannel triage strategy, thereby enhancing both their effectiveness and acceptance within the health care system. In addition, the development of an evidence base around the predictive performance of human symptoms could substantially contribute to clinical education and practice, underscoring the significance of this line of research. ²¹

User perception and behavior play critical roles in the effectiveness of self-triage tools. The observed lack of awareness and varied attitudes toward symptom checkers, as reported by Aboueid et al., suggest that public education and engagement are essential for the broader acceptance and effective use of these technologies. ²² Self-triage tools in TM have shown potential in enhancing health care efficiency and patient safety. The high safety rate noted in studies, such as that by Cotte et al., is promising, yet this optimism must be tempered with an understanding of the variability in accuracy and reliability among different symptom checkers. ¹⁵

Projections from the study by Semigran et al. estimate that ∼50 million people self-screen annually. ²³ The accumulated evidence and findings of this study advocate for the development of new self-screening strategies, potentially restricting scenarios and diseases to ensure real cost-effectiveness and safety. The outcomes of this research, focusing on the use of self-screening in the evaluation of direct-to-consumer TM, are constrained by limitations including its retrospective and unicentric design, as well as the unique social, cultural, and health system characteristics that may differ significantly from those in other countries, and the predominance of low-risk diseases.

Conclusion

Self-triage was able to reduce approximately three-fifths of the TM encounters with physicians. But still many patients do not follow the referral to the ED provided by the self-triage app after first contact with the TM Center. Despite scientific-based apps, symptom-based referral algorithms appear very sensitive and correlate poorly with physician management. Future investigations are needed to improve this strategy.

Authors' Contribution

Conception, planning, analysis, and interpretation of data by T.A.D.A., F.T.M., and A.A.E. The writing of the article or its critical intellectual review by T.A.D.A., F.T.M., K.F.K., and K.D.A.L. Responsibility for the final approval for publication by R.A.M. and C.H.S.P.