Abstract
Objective:
Evaluate if artificial intelligence (AI)-based virtual triage (VT) and care referral technology integrated with a live nurse triage workflow can improve care acuity alignment by appropriately altering patient-member post-triage health care intent and care seeking.
Methods:
Data were extracted from Infermedica’s AI-based call center triage application, implemented by the Médis health plan in their VT-informed nurse triage service over a 26-month period, from January 2022 through March 2024. Patient-member care seeking intent pre- and post-VT-informed nurse triage, as well as nurse triage recommendation, were grouped into five care acuity levels, and responses were compared to the output care recommendation of the VT process, including four kinds of medical consultation available for nurses to schedule. Pre- and post-triage care intent and care seeking behavior were compared and differences assessed for statistical significance. Analyses were conducted on a dataset of eligible patient-members interviews (N = 54,587) to examine if the use of VT influenced patient-member care seeking behavior. We examined if post-triage care seeking behavior aligned with that recommended, and if it changed as a result of triage and in what direction. To determine statistical significance of differences in care intent pre- and post-VT-informed nurse triage, Z-tests were performed.
Results:
The impact of VT-informed nurse triage recommendations was high with 83.9% of encounters influencing patient-member care seeking behavior, and 22.8% changing their care seeking intent as a result. Of these, 62.2% (14.2% of all patient-members) de-escalated care intent to a lower acuity care service, while 37.8% (8.6% of all patient-members) escalated to higher acuity care (p = 0.05). There was a substantial post-triage increase in patient-members intending to engage in self-care (+5.5 percentage points or PP or +39.5%), and a decrease in patient-members with a pre-triage intent to seek an urgent outpatient consultation within 24 h (−5.0 PP or −8.4%) (p = 0.05). The largest group of 14,109 patient-members (35.6%) decided to instead schedule a telemedicine consultation. One-fifth of encounters occurred outside usual operating hours, when patient-members were nearly twice as likely to initially intend to visit an emergency department, indicating strong after-hours patient-member care need and demand, with 48.4% of these cases de-escalated to lower care acuity settings.
Conclusions:
Integration of AI-based VT with a live triage nurse workflow was effective in yielding much improved care acuity alignment by changing patient post-triage care intent and care seeking, particularly in de-escalating care appropriately from in-person outpatient care to telemedical/virtual care delivery or self-care.
Introduction
In recent years, there has been a growing shortage of skilled health care practitioners worldwide. 1 The World Health Organization (WHO) has identified the shortage of health care workers, including nurses, as a significant limitation to the advancement of health care provision and health care development worldwide.1,2 In 2013, the WHO estimated that there was a global shortage of ∼7.2 million health care workers, and predicted that this number will increase to 12.9 million worldwide by 2035.2,3 This shortage may disproportionately affect the number of nurses available in emergency departments (EDs) and clinics, as nurses account for around half of the global health care workforce. 4 New technology solutions, such as artificial intelligence (AI)-based virtual triage VT, (commonly known as symptom checkers), along with integrated care referral, are being deployed to bolster the nurse triage resources available to health plans, hospitals, and health systems.
AI-based VT, such as Infermedica’s VT platform, are digital tools that patient-members and practitioners can access 24/7/365 to assist in the evaluation of symptoms and determine appropriate care referral. 5 In recent years, online symptom checkers have been utilized increasingly to increase the efficiency and accessibility of health care services and reduce patient volume in EDs and clinics, while saving time for patients and health care practitioners.3,5 In addition, a literature review on AI-based VT has suggested that VT allows patient-members to play a more active role in their health care, resulting in more positive patient experiences and potentially better health care outcomes. 6 Thus, VT may have a potentially synergistic effect, improving patient-member satisfaction and care acuity alignment, while also improving the efficiency of health care, such as nurse call centers.5,6
Research on the consistency and accuracy of live nurse triage call centers has shown that the concurrence of disposition between call center nurses and computer-based triage algorithms was around 80%. 7 A 1-year analysis of after-hours telephone triage found that 21% of callers were instructed to seek urgent evaluation. 8 When such a substantial proportion of callers may require urgent care, care acuity alignment and automated referral may not only improve operational and financial performance but can also be a matter of life and death.
The addition of AI-based symptom checkers to existing nurse triage can potentially benefit nurses working in call centers and patient-members seeking care. This study evaluates the utility of an AI-based VT and care referral solution integrated with live nurse triage within a large Portuguese health insurance company, Médis (Grupo Ageas Portugal). We assessed whether nurse triage can benefit from the addition of AI-based VT and care referral to improve patient care access and care-acuity alignment.
Materials and Methods
Study objectives
Evaluate if an AI-based VT and care referral technology integrated with live nurse triage can improve care acuity alignment by appropriately altering patient-member post-triage health care intent and care seeking behavior within a large Portuguese health insurance plan.
Study setting and design
Médis has been a leading health insurance plan in Portugal for more than two decades. It operates a comprehensive network of over 110 hospitals and more than 1,100 clinics, ensuring access to quality care for its membership of over 900,000 insured individuals. With over 16,400 physicians, Médis delivers a full spectrum of health care services, including dental clinics, vision care, and partnerships with over 850 health and wellness service providers. The Médis Line, is a 24/7 access service staffed by experienced nurses that utilizes AI-driven VT capabilities to assess symptom presentation and convey evidence-based care guidance. This integration of advanced technology ensures efficient, acuity appropriate care recommendations, improved care accessibility, and high patient-centric care.
Data for this analysis were extracted from Infermedica’s AI-based call center triage application, which was implemented by Médis in the VT-informed nurse triage service Médis Line over a 26-month period, from January 2022 through March 2024. The pre-VT informed nurse triage included asking patient-members about their care intent across five care acuity levels, and was compared with the output care recommendation of the VT process, including four kinds of medical consultation available.
Patient-member care seeking intent pre- and post-VT-informed nurse triage, as well as nurse triage recommendation, were grouped into five care acuity levels or options, as follows: (1) self-care: symptoms do not require professional medical care, patient-members can try to manage the condition at home; (2) non-urgent outpatient consultation: symptoms require medical evaluation, a routine consultation with a health care professional is recommended; (3) urgent outpatient consultation within 24 h, where symptoms required medical evaluation and consultation with a health care professional within 24 h; (4) emergency care self-transport, where symptoms were sufficiently concerning clinically that patient-members were advised to self-transport to the nearest ED as soon as possible; and (5) emergency care with ambulance transport, where symptoms were of sufficient clinical concern and urgency that patient-members were advised to call an ambulance. Pre- and post-triage care intent and care seeking behavior were compared and differences assessed for statistical significance.
Description of intervention/VT engine utilized
The Infermedica VT engine completes evidence-driven analyses focusing on 800 illnesses, 1,500 symptoms, and 300 risk factors. Utilizing AI, machine learning, and natural language processing, the triage engine assesses patient-member reported symptoms and suggests the most likely diseases from the medical history and current clinical presentation. Interaction with the technology begins with a question to specify the patient’s gender and age and elements of past medical history, followed by a prompt to list symptoms and complaints the user is experiencing. At the start of the VT encounter, patient-members are also queried regarding their care intent. Subsequently, the AI generates a list of yes/no, single choice, and multiple-choice questions, and after reaching a confidence threshold, the interview concludes. VT then refers the patient-member to the safest and most clinically suitable care setting. There are no set protocols or decision trees, and in light of new data/information reported, the VT engine examines numerous clinical hypotheses and possibilities just as a physician does.
Patient responses are analyzed on a current basis, and VT rapidly conveys a symptom analysis, explaining probable causative factors, severity, and suggesting acuity appropriate needed health care. Unlike traditional VT algorithms employing a decision tree design, AI-based VT enables patients to report multiple symptoms at the commencement of an interview. The system analyses patient’s responses in the context of all complaints and demographic information provided. This allows the VT engine to explore multiple pathways before furnishing the patient-user with a list of potential causes and an acuity-appropriate urgency assessment. AI-based VT offers the added advantage of detecting critical symptom combinations and diseases that may be the cause of the patient-user’s complaints. Each identified potential disease state is linked to a corresponding triage acuity-level recommendation.
To operate independently, VT engines are typically optimized to comprehend and employ low health literacy language, bridging the gap between medical jargon and layman terms. Developers commonly integrate multiple languages into these tools, facilitating rapid health assessments in the patient’s mother tongue. VT applications may use a probabilistic algorithm assessing symptoms using an underlying medical knowledge base and inference algorithm that determines the next question to ask based on prior patient responses, demographic information, past medical history, and disease risk factors. Follow-up questions are generated based on these data. Certain VT engines, including that deployed in this care setting, use AI validated by practicing multispecialty physicians, which processes the data and displays a list of diseases that may be causing the patient-user’s symptoms.
Infermedica’s free public VT platform, Symptomate, is available in 24 languages and has had over 18 million VT encounters since 2012. To protect patient confidentiality/anonymity, Symptomate data is analyzed only in the de-identified aggregate in order to refine the AI within Infermedica’s VT and care referral engine.
In Europe, VT technologies are considered medical device class I according to Medical Device Directive (93/42/EEC), and fall under the Food, Drug and Cosmetic Act in the United States. The Food and Drug Administration (FDA) currently exercises enforcement discretion, meaning the technology is not required to comply with FDA regulations related to medical devices.
Data for this analysis was extracted from VT encounters engaged by patient-members using the Infermedica VT engine deployed within Médis. Portuguese was the language of this implementation.
VT engine clinical validity
AI-based VT engines require rigorous validation to ensure safety and minimize mistriage. VT focuses on common diseases by design, with AI built to err on the side of over-triage to higher acuity care rather than possibly missing and misguiding a patient with acute care needs. VT accuracy varies across clinical specialties and settings, as determined by the depth of disease-specific data used to train the triage AI. VT validity has been evaluated using clinical vignettes prepared by physicians of various patient symptomatic presentations in different clinical settings.7–12
In one study, Infermedica’s VT engine provided safe recommendations in 97.8% of instances. 10 Published studies, while providing a point in time evaluation, quickly become obsolete due to the rapid evolution of AI-based VT. For example, since the 2019 Gilbert et al. analysis, 10 there have been 29 releases of the Infermedica medical content model and 17 updates to core functionality, including new epidemiological models and triage algorithms, each designed to improve accuracy and safety.
Prior to deploying Infermedica VT in 2021, a large ambulatory care delivery system in Massachusetts completed an independent clinical validation of its diagnostic accuracy involving 407 test cases or vignettes for patients 12 and older. 13 The diagnostic accuracy was reported at 92%. A comparison of the triage referral accuracy of AI-based VT to rules-based triage protocols (RBTP) from Schmitt-Thompson used in live telephonic triage found that both modalities achieved >70% triage accuracy, with identical safety performance of 91%. 14 AI-based VT was more accurate in care referral for emergency and nonemergency care and overtriaged to emergency care 50% less frequently than RBTP but was less accurate than RBTP in self-care vignettes (neither statistically significant). Both modalities demonstrated decreased sensitivity as care urgency/acuity decreased, more pronounced in AI-based VT than RBTP. 14
Respondent selection and characteristics
Data for this analysis was extracted from Infermedica’s AI-powered Call Center Triage application, implemented by Médis in the VT informed nurse triage service Médis Line. Data was analyzed from January 1, 2022, to March 30, 2024, with the following inclusion or eligibility criteria: (1) all Médis VT encounters (71,945); (2) encounters using Infermedica’s VT (59,777), excluding all decision tree interviews; and (3) complete encounters with nurse care recommendations captured (54,587).
Data captured and analyses completed
Analyses were conducted on a dataset of eligible Médis patient-members encounters (n = 54,587) to examine if the use of VT influenced care seeking behavior. Data were analyzed using Microsoft Excel and Google Sheets (Online Spreadsheet Editor). Outcomes analyzed included pre-VT care intention and post-VT care seeking action. Care seeking intent and nurse triage recommendation were grouped into the five care acuity levels described above. To evaluate the influence of VT-informed nurse triage on patient behavior, member care seeking intent pre- and post-VT were compared. We assessed whether post-triage care seeking behavior was aligned with that recommended by VT, or if changed as a result of triage, in what direction. Statistical significance of differences in patient-members’ care intent pre- and post-VT-informed nurse triage was evaluated using Z-tests. The time of day of the triage encounter was assessed to determine if it had any impact on care seeking behavior.
Consultations scheduled by patient-members were analyzed and grouped by whether the consultation was with the member’s regular attending physician in an office or clinic setting; was a home visit consultation with a physician; or a telemedical online consultation with a primary or specialty care physician.
Results
Overall impact of VT-informed nurse triage on care seeking intent
VT-informed nurse triage was effective in engaging plan patient-members and aligning their care intent to the most clinically appropriate and needed care acuity. The impact of VT-informed nurse triage was high, with 83.9% of encounters influencing patient-member care seeking behavior. Almost one-fourth of patient-members (22.8%) changed their care seeking intent as a result of the VT-informed nurse triage recommendation (Table 1). Of those changing their care seeking intent, 62.2% (or 14.2% of all eligible patient-members) de-escalated their intent to lower acuity care, while 37.8% (or 8.6% of all patient-members) escalated to higher acuity care, as shown in Table 1. These changes were statistically significant (p = 0.05).
Overall Impact of Virtual Triage Informed Nurse Triage Recommendation on Patient-Member Care Seeking Intent
VT, virtual triage.
Comparison of patient-member care seeking intent before and after triage guidance
Following VT-informed nurse triage, there were decreases in patient-members with a pre-triage intent to seek an urgent outpatient consultation within 24 h (−5.0 percentage points or PP or −8.4%), and to seek emergency care with self-transport (−1.4 PP or −10.6%) (Table 2). There was a substantial post-triage increase in patient-members with an intention to engage in self-care (+5.5 PP or + 39.5%), and some increase in patient-members with a clinically appropriate intent to call an ambulance for transport to emergency care (+0.4PP or +170.6%) (Table 2). These changes were statistically significant (p = 0.05).
Comparison of Patient-Member Care Seeking Intent Pre- and Post-VT-Informed Nurse Triage Guidance
Specific changes in patient-member care seeking intent following triage
Substantial changes in care seeking intent occurred among patient-members who intended initially to seek outpatient consultation within 24 h but changed to pursue self-care following triage (5.0% of all patient-members) (Table 3). After triage, fewer patient-members who intended initially to seek emergency care did so, instead choosing outpatient consultation within 24 h (2.9% of all patient-members). Care acuity escalated among patient-members who intended initially to seek urgent outpatient consultation within 24 h, and who instead pursued emergency care (3.5% of all patient-members). These changes were statistically significant (p = 0.05).
Specific Changes in Patient-Member Care Seeking Intent Following VT-Informed Nurse Triage
Impact of VT-informed nurse triage on outpatient consultation
Analysis of consultations scheduled by patient-members (Table 4) showed that 5,640 or 14.2% of patient-members who initially intended to access an outpatient consultation changed their care seeking action and did not do so after triage. The largest group of 14,109 patient-members (35.6%) decided to instead schedule a telemedicine consultation, while the least frequent realignment were patient-members seeking in-person care with their attending physician (in 1,975 or 5.0% of patient-members), as seen in Table 4.
Impact of VT-Informed Nurse Triage on Type and Volume of Outpatient Consultation Accessed
Changes in care seeking intent following triage by encounter time
As shown in Table 5, one-fifth of encounters (20.2%) occurred outside usual business working hours (8:30 am–8:30 pm), indicating strong after hours patient-member need and demand. Patient-members were nearly twice as likely to report an initial pre-triage intent to access emergency care when the encounter occurred outside of business hours (20.5%), compared with encounters occurring during business hours (11.0%). This difference was statistically significant (p = 0.05). Nearly half of cases (48.4%) outside business hours were successfully de-escalated to lower care acuity settings. Patient-members were also more likely to escalate intent to access emergency care outside business hours (6.5%) compared with during business hours (4.6%) (p = 0.05).
Changes in Care Seeking Intent Following VT-Informed Nurse Triage by Encounter Time of Day
Discussion
VT-informed nurse triage improved care acuity alignment between patient pre-triage care intent and actual clinical need, with almost one in four patient-members altering their care seeking intent as a result of triage. This may yield a net reduction in avoidable medical costs across the member population. Improved care acuity alignment can reduce care delays, which often result in greater hospital length of stay, and potentially avoidable intensive care unit admissions.15–19 De-escalation of care acuity occurred more frequently than escalation (62.2% of all changed acuity and 14.2% of all eligible patient-members), which may increase care clinical- and cost-effectiveness, and reduce avoidable or unnecessary care. However, for 37.8% of those changing care acuity or 8.6% of all patient-members escalating care acuity, VT can potentially help triage nurses facilitate early disease detection and potentially reduce care delays among those needing a rapid escalation in care acuity. This finding replicates the same effect observed for standalone, fully automated VT.19,20 While VT-informed nurse triage significantly impacted the post-triage care seeking intent of patient-members, there remains further potential for de-escalation to self-care and non-urgent outpatient consultation.
The greatest magnitude of care intent change among patient-members was much increased intent to use self-care (+39.5%), derived largely from decreased intent to pursue emergency care with self-transport (−10.6%) and urgent outpatient consultation (−8.4%). The combined impact of all changes observed in care seeking was a substantial overall reduction of inappropriate medical care utilization, and associated care and administrative costs, as well as recovering clinical care capacity to accelerate appropriate treatment of higher acuity patients with more urgent care needs. This likely improves clinical outcomes and financial performance, and further research is needed to validate the full impact of improved care acuity alignment. These findings suggest that VT-informed nurse triage—through its ability to appropriately realign patient-member care intent with the real level of care acuity needed—delivers clinical and operational value by reducing avoidable, higher than needed acuity ED and urgent outpatient care utilization (and costs).
In addition, aligning patient care intent with actually needed care acuity can increase early detection of serious, evolving illnesses which urgently require emergency care with ambulance transport, as occurred in 215 patients of this cohort. This finding is intuitive, given that most patients with a severe, evolving, and potentially life-threatening pathology would not tend to utilize VT. However, evidence suggests that patients may often be unaware of prodromal symptoms that may be evolving toward life-threatening conditions, such as myocardial infarction, stroke, and pulmonary embolism.19,20 In addition, the de-escalations of care acuity were incremental or stepwise, with initial intent of emergency care changing to urgent outpatient consultation within 24 h, and from the latter to self-care.
VT-informed triage enabled a reduction in unnecessary in-person consultations, while facilitating patient-member engagement of more appropriate and lower cost care, such as virtual/telemedicine in-person visits (35.6%). This creates dual benefits of reduced individual and total costs to serve the patient-member population, and reduced wait times for those requiring in-person care (which can help improve patient-member experience/satisfaction and care efficiency).
The findings around VT-informed nurse triage utilization during hours and days when medical offices are closed compellingly demonstrate the importance of providing 24/7/365 access to VT and care referral. This enables the plan to more effectively ensure continuity of clinical care while also better aligning patient-members’ care intent with the most clinically appropriate level of care acuity. One in five triage encounters occurred at night when outpatient services are closed. As might be expected, triage use among patient-members with an intent to seek ED care was nearly twice as high during nighttime (20.6%) versus daytime operating hours (11.2%). Importantly, almost half of nighttime encounters were de-escalated to lower acuity settings, again conveying value to the health plan with respect to reducing inappropriate ED use for nonurgent care, and increasing appropriate use of self-care and nonurgent outpatient consultation.
In interpreting these findings, it is important to bear in mind that the final care acuity decision of patient-members may often be influenced by their current plan coverage, access to resources for self-care, and availability of non-urgent outpatient care. Systematically evaluating how changes in final selection of care acuity impact care utilization and health outcomes should constitute the next phase of research on the integration of VT into existing triage and care referral workflows.
Summary Table
Compliance with VT-informed nurse triage recommendations was high (83.9%). Nearly one-fourth of patient-members (22.8%) changed their care seeking intent as a result of the VT-informed nurse triage recommendation (p = 0.05).
Of those changing their care seeking intent, 62.2% (or 14.2% of all eligible patient-members) de-escalated their intent to lower acuity care services, while 37.8% (or 8.6% of all patient-members) escalated to higher acuity care (p = 0.05).
There was a substantial post-triage increase in patient-members with an intent to engage self-care (+5.5 PP or + 39.5%) and a decrease in those with a pre-triage intent to seek urgent outpatient consultation within 24 h (−5.0 PP or −8.4%) (p = 0.05).
The largest group of 14,109 member-patients (35.6%) instead scheduled a telemedicine consultation.
One-fifth of encounters (20.2%) occurred outside usual working hours (8:30 am–8:30 pm), indicating strong after hours member need and demand, with nearly half (48.4%) of cases outside business hours de-escalating to lower acuity care.
Footnotes
Authorship Contribution Statement
G.A.G., T.P., and A.K.K. designed the study methodology and interpreted the data. G.A.G., T.P., and G.L.G. wrote and edited all drafts of the article. T.P., A.K.K., G.A.G., P.G., S.M.G., D.A.C., and P.M.O. reviewed the data, validated the data analyses, and co-authored the results interpretation and the discussion sections. G.L.G. and D.A.C. assisted in project management. G.L.G. assisted with writing, literature search, and reference identification and integration.
Availability of Study Data and Analyses
The authors will make the study data and detailed analytic files from this report available upon reasonable request.
Author Disclosure Statement
G.A.G. is a medical advisor to Infermedica. T.P., A.K.K., and P.M.O. are employees of Infermedica. P.G., S.M.G., and D.A.C. are employees of Médis. G.L.G. is a contractor for Infermedica.
Funding Information
No specific funding supported this research.