Beyond Algorithm: Emergency Department Professionals’ Perspectives on Machine Learning-Based Triage Integration—A Qualitative Study

Abstract

Emergency department (ED) overcrowding has necessitated more efficient triage processes. Traditional methods can struggle to keep up with increasing patient volumes, and interest in machine learning-based triage systems is increasing. However, the perspectives of emergency department professionals, who play a critical role in triage decision-making, are often overlooked in ML development. This qualitative study explores emergency department professionals’ perspectives on the potential for machine learning-based triage to enhance triage processes in emergency departments. Semi-structured interviews were conducted with 13 ED professionals (9 physicians, 4 nurses) from 6 hospitals in Istanbul. A grounded theory approach was used to analyze the data, identifying themes related to current triage challenges, attitudes toward machine learning-based triage, and suggestions for ML algorithm development. Three main themes emerged: (i) patient and public interaction with triage, (ii) technology and ML in triage, and (iii) triage processes and challenges. Healthcare professionals expressed optimism about the potential of machine learning-based triage but raised concerns about the accuracy of current technology and the need for ML models to integrate complex clinical judgments, particularly regarding pain assessment and patient behavior. While machine learning-based triage has the potential to significantly enhance ED triage, emergency department professionals’ experiential insights are crucial for the development of more accurate and usable ML models which focus on incorporating human expertise.

Introduction

Background

Emergency department (ED) overcrowding—driven by rising patient numbers and intensive medical interventions—significantly jeopardizes care quality and patient safety.^1,2 Effective triage is crucial for ensuring that high-acuity patients receive prompt care and that resources are allocated efficiently.^3,4 However, traditional triage methods often struggle with both accuracy and speed under high-volume conditions, leading to suboptimal patient outcomes and operational inefficiencies.^5,6

Recent advances in machine learning (ML) and artificial intelligence (AI) offer promising solutions to these challenges. By leveraging diverse datasets—including patient history, symptoms, and real-time clinical metrics—ML algorithms can enhance the precision and speed of patient categorization, ultimately predicting acuity levels and resource needs more effectively.^{7 -9} This potential is further underscored by the surge in U.S. Food and Drug Administration (FDA) approvals for ML-based healthcare devices, reflecting growing confidence in these technologies. ¹⁰ Integrating ML into ED triage processes thus represents a significant step toward improving patient categorization, operational efficiency, and overall patient safety.

Problem Statement

Research indicates that ML-based triage algorithms can outperform traditional methods by providing more accurate patient prioritization, which may lead to improved resource allocation and reduced waiting times. ¹¹ For instance, ML models have demonstrated superior performance in predicting hospital admissions compared to conventional tools such as the Emergency Severity Index, ¹² and they significantly enhance the prediction of critical outcomes, thereby improving patient safety and departmental efficiency.^13,14 Moreover, ML’s potential to reduce under triage is crucial for ensuring that high-risk patients receive timely care.

Despite these advantages, the success of ML-based triage systems depends heavily on the quality and diversity of input data, their integration into clinical workflows, and continuous validation against evolving patient populations.^12,13 In practice, the performance of these algorithms is significantly influenced by the variety of input variables. Research shows that incorporating both structured data—such as vital signs (pulse, heart rate, respiratory rate, blood pressure, temperature) along with other demographic and contextual factors (age, sex, arrival mode)—and unstructured data, such as clinical notes, markedly improves predictive capabilities.^14,15 However, structured data alone may not capture the nuanced decision-making processes of experienced ED staff, whose clinical judgment is critical for accurately assessing and prioritizing patients.

While growing research has demonstrated the technical feasibility of ML-based triage systems, limited qualitative investigation has explored healthcare professionals’ perspectives on their implementation in emergency settings. Although studies have examined clinicians’ general attitudes toward AI in healthcare,^{16 -19} the specific challenges and workflow considerations unique to ED triage decision-making remain largely unexplored. This represents a critical knowledge gap, as successful ML implementation requires understanding not only technical capabilities but also the experiential insights and concerns of frontline healthcare professionals who will ultimately use these systems.

Proposed Solution

Clinical expertise and intuitive decision-making, honed through years of patient care, represent a valuable knowledge base that can significantly enhance the precision and reliability of ML-based triage algorithms when properly integrated. ²⁰ This underscores the necessity of embedding the empirical insights of ED professionals alongside structured and unstructured data to develop a more robust and context-aware triage system. Although ML-based triage shows considerable promise, its effectiveness is currently limited by the underutilization of clinical expertise. This study aims to address that gap by exploring healthcare professionals’ insights on how ML systems can be optimized for triage in ED settings.

Materials and Methods

We investigated the perceptions of ED professionals regarding the implementation of ML algorithms in triage processes. The main research question addressed factors perceived as influential inpatient triage and how these factors could be integrated into ML models. In this study, data saturation was achieved when no new themes, insights, or patterns emerged from the interviews. This point was reached after conducting the 12th interview, as subsequent interviews did not provide additional information or themes relevant to the study’s objectives. To ensure thoroughness, one additional interview was conducted, confirming that data saturation had been reached. We used convenience sampling to select 17 healthcare professionals (13 emergency physicians and 4 nurses) across 6 hospitals in Istanbul, Turkiye (Tables 1 and 2). To ensure that participants had sufficient expertise and familiarity with the dynamics of triage, only healthcare professionals with a minimum of 5 years of experience in ED were included. In the Turkish emergency department system studied, triage is primarily conducted by nurses who perform initial patient assessment, vital sign measurement, and Manchester Triage System scoring, while emergency physicians are actively involved through supervision of triage decisions, review of complex cases, and re-evaluation of patient priorities based on clinical judgment. It is important to note that while Turkey has developed its own triage system (HETS - Hacettepe Emergency Triage System) which is more widely implemented across the country, all 6 hospitals included in this study had adopted the Manchester Triage System protocol. This provides a consistent and internationally recognized framework for triage decision-making among participants, though it represents an exception to typical Turkish emergency department practice. "Active involvement" was defined as regular participation in either direct triage assessment (primarily nurses) or triage decision oversight and clinical re-evaluation (primarily physicians) within the past 6 months. Participants who declined to participate or were unable to join due to workload constraints during the interview period were excluded. 4 emergency physicians did not participate separately due to their workload and cases requiring urgent intervention during interview.

Table 1.

Characteristic features of the participants.

Participant	Age	Triage Experience
ED Specialist 1	41	15
ED Specialist 2	55	21
ED Specialist 3	46	14
ED Specialist 4	44	16
ED Specialist 5	44	18
ED Specialist 6	50	25
ED Specialist 7	40	17
ED Specialist 8	43	18
ED Specialist 9	39	13
ED Nurse 1	29	7
ED Nurse 2	24	5
ED Nurse 3	32	12
ED Nurse 4	36	14

Table 2.

Overview of the Interviewed Emergency Service Staff.

Category	Subcategory	Frequency (n)
Role	Emergency Department Supervisor	5
	Emergency Specialist	4
	Supervisor Nurse	4
Profession	Physician	9
	Nurse	4
Experience	5-10 Years	2
	10-15 Years	4
	15-20 Years	5
	20+ Years	2
Age Group	25-30	2
	30-35	1
	35-45	7
	45-55	3

Individual interviews were conducted face-to-face and utilized a semi structured format. The structured component consisted of 10 questions that prepared by all authors designed to elicit broad perspectives on the current triage processes and the potential for ML enhancements. The interview guide was not pilot tested due to time constraints. All interviews were conducted one-on-one between the researcher (male) and participants, with no other individuals present during the sessions. No prior relationships existed between the researcher and participants before study commencement. No formal field notes were taken during interviews as the focus was on comprehensive audio recording for data capture. Additional unstructured questions were introduced as needed to explore emerging themes. The interviews varied in length from 11-75 minutes. Data collection started on 2024 January and finished 2024 March. All interviews were audio-recorded with the participants’ consent after being informed about the study’s aims and their rights. Participants did not provide feedback on the findings as the transcripts were directly analyzed without further validation from participants.

To analyze the transcripts, we employed a modified version of the grounded theory approach.^21,22,23 Grounded theory was specifically chosen for this research question because it is particularly well-suited for exploring complex phenomena where limited theoretical frameworks exist. ²² Given the nascent nature of machine learning implementation in emergency department triage processes, there is insufficient existing theory to guide a deductive analytical approach. The exploratory nature of our research question—examining healthcare professionals’ perceptions of factors influential in triage and how these could be integrated into ML models—required an inductive methodology capable of generating theoretical insights from empirical data. ²¹

Grounded theory’s systematic approach to theory development was essential for our study objectives. The methodology’s emphasis on allowing themes and theoretical frameworks to emerge from the data, rather than testing pre-existing hypotheses, was particularly appropriate given the limited prior research on clinicians’ perspectives regarding ML-based triage systems. Furthermore, grounded theory’s iterative process of data collection and analysis allowed us to refine our understanding of participants’ perspectives throughout the research process, ensuring that our theoretical insights were firmly grounded in the lived experiences and professional expertise of emergency department staff. ²⁴

The modified approach we employed retained grounded theory’s core principles of constant comparison, theoretical sampling concepts (within the constraints of our convenience sampling), and iterative analysis, while adapting the methodology to fit the practical constraints of our healthcare setting and timeline. ²² This methodological choice enabled us to develop a nuanced understanding of the complex factors influencing healthcare professionals’ perceptions of ML in triage, ultimately contributing to theory development in this emerging field.

Qualitative analyses were performed on the transcribed interviews using MAXQDA 24. The coding process was conducted in 3 stages. Initially, line-by-line coding was applied to generate tentative open codes closely tied to the raw data. Following this, focused coding was used to develop higher-level categories and subcategories from the initial open codes. In the final stage, selective coding was employed to identify relationships among the various categories, resulting in a comprehensive codebook.

To enhance the credibility and dependability of our analysis, we implemented several additional measures. First, 2 coders independently coded the data, and discrepancies were resolved through regular team debriefing sessions, ensuring consensus on code definitions and thematic interpretations. Second, an audit trail was maintained throughout the coding process to document key decisions, revisions, and the evolution of the codebook, thereby increasing transparency. Although member checking was not conducted, the combined use of inter-coder reliability and systematic debriefings provided a robust check on the consistency and reliability of our findings.

The collaboratively reviewed codebook resulted in the emergence of 3 main themes and 18 associated codes (Table 3). Thematic analysis was subsequently employed to explore in-depth patterns and relationships within these themes, and comparative analysis was used to examine variations in perceptions based on roles, professions, experience levels, and age groups. Participant quotations were incorporated to illustrate the themes, with each quotation attributed to a specific participant using a unique identifier (eg, ED Specialist 1, ED Nurse 3) to maintain clarity and authenticity.

Table 3.

Thematic Area and Codes.

Thematic area	Codes
Patient and public interaction	• Facility use and management• Patient education• Satisfaction with current process
Technology and machine learning in triage	• General awareness of machine learning• Lack of experience with machine learning• Limitations of current technology• Potential machine learning capabilities• Suggestion for machine learning inputs
Triage process and challenges	• Challenges in triage• Communication barriers in triage• Critical triage factors• Decision making in triage• Diagnostic challenges• Documentation• Dynamic nature of triage• Training and experience of health staff• Triage protocol and flow• Vital sign collection

Ethical approval for this study was granted by the Acibadem University and Acibadem Health Institutions Medical Research Ethics Board on January 11, 2024 (Approval Number: 2023- 21/757). Each participant signed a volunteer participation form, which outlined the study’s purpose, the voluntary nature of participation, confidentiality measures, and the use of data for scientific publication. The participants were informed of their right to withdraw from the study at any time without consequence.

This study was conducted and reported in accordance with established qualitative research reporting standards, with adherence documented using the COREQ (Consolidated Criteria for Reporting Qualitative Research) checklist to ensure methodological rigor and transparency.

Our study benefits from the inclusion of participants with various roles within the ED, including physicians and nurses with diverse levels of experience and a wide age distribution. This diversity is crucial for selecting robust inputs for ML-based triage algorithms and for assessing the overall need and perception of such algorithms among ED staff. By capturing their expertise and attitudes toward ML, this research aims to determine whether there is a genuine need for these algorithms and how they are perceived by those on the front lines of emergency care. These insights provide a comprehensive understanding of the challenges and opportunities in integrating ML systems into clinical workflows. The findings were interpreted and evaluated considering this demographic diversity, offering more comprehensive and inclusive recommendations for improving ML-based triage algorithm applications.

The primary investigator is a doctoral candidate in Health Informatics with a systems-oriented background in healthcare process optimization but lacking direct clinical experience in emergency departments. First, the researcher approached this study with algorithmic, deterministic assumptions—believing that machine learning could fully capture triage decision-making through structured data. However, after consulting with clinical experts, this technical orientation was fundamentally challenged, catalyzing a paradigmatic shift toward an interpretive understanding of clinical practice. Throughout data collection, preconceived notions about algorithmic predictability encountered the chaotic nature of emergency care, necessitating continuous reflexive adjustment facilitated by regular debriefing sessions with the doctoral supervisor for investigator triangulation. The researcher’s explicit positioning as a health informatics outsider created a collaborative dynamic where clinicians assumed an educative role, facilitating authentic disclosure; however, this outsider status may have limited access to nuanced professional culture and tacit knowledge. The absence of systematic reflexive journaling, member validation procedures, and prolonged engagement represent methodological limitations that may have compromised reflexive analysis depth and overall trustworthiness.

Study Limitations

Several limitations should be considered when interpreting these findings. First, convenience sampling may limit generalizability and could introduce selection bias, as participants volunteered based on availability and interest. Second, participants’ prior knowledge, attitudes, and experience with machine learning and artificial intelligence were not systematically assessed, which may have influenced their responses. Third, all participants were recruited from a single geographical region (Istanbul, Turkey) within the same healthcare organization, limiting transferability to other healthcare systems and contexts.

Regarding methodological trustworthiness, several limitations exist: prolonged engagement was not conducted as all interviews were single sessions; member checking was not performed; and while investigator triangulation occurred through regular debriefings, data source and methodological triangulation were not implemented. As noted in the reflexivity statement, the absence of systematic reflexive journaling and formal member checking may have impacted analysis depth and trustworthiness.

Additionally, while all participating hospitals used the Manchester Triage System, this standardized international protocol is not widely implemented across Turkey, where the Hacettepe Emergency Triage System (HETS) is more commonly used. This may limit the transferability of findings to emergency departments using different triage protocols, particularly those using the more prevalent HETS system in Turkish healthcare settings.

Our study did not explicitly capture concerns around ML–human triage discordance, namely the risks of over‑triage (high sensitivity at the cost of excessive workload) versus under‑triage (high specificity at the cost of missed critical cases). Although participants did not raise this theme spontaneously, prior work has shown that optimizing sensitivity in algorithmic triage can overwhelm staff with false positives, while prioritizing specificity may delay care for true high‑risk patients.^25,26

Finally, the sample composition was weighted toward physicians (13 physicians vs 4 nurses), which may not adequately represent perspectives of nurses who often lead triage processes in many healthcare systems. These limitations highlight areas for future research to enhance generalizability and trustworthiness of findings.

Result

Specifically, we aimed to explore the perceptions and experiences of ED staff regarding the adoption and use of ML-based triage systems. Through qualitative analysis of the interviews, 3 main themes emerged: patient and public interaction, technology and ML in triage, and the triage process and challenges. These themes encapsulate the diverse viewpoints of the participants and highlight the multifaceted nature of integrating ML into clinical workflows. The findings provide valuable insights into both the practical and the attitudinal factors that influence the successful implementation of such technologies in emergency care settings.

Patient and Public Interaction

The theme of patient and public interaction encompasses 3 distinct codes: facility use and management, patient education, and satisfaction with the current process.

The Facility Use and Management Code addresses various challenges and suggestions for optimizing the use of triage rooms and other physical spaces in ED. The participants emphasized the current usage patterns and physical constraints, as well as the importance of dedicated and purpose-specific spaces for improving triage processes. This code provides critical insights into enhancing triage efficiency and managing patient flow more effectively.

The Patient Education Code refers to the impact of patients’ health literacy and education level on the accurate implementation of triage, their compliance with healthcare processes, and their communication with healthcare professionals in ED. The participants emphasized that the public often lacks an understanding of the purpose of emergency services, which negatively affects the triage process. Issues such as poorly informed or uneducated patients making inappropriate demands, attempting to manipulate the triage process, and showing impatience during long waiting times were highlighted. Additionally, low health literacy could undermine trust in healthcare services and decrease the overall quality of care. The participants stressed the need for increased focus on public education, suggesting that such educational efforts should begin at an early age to address these challenges effectively.

Satisfaction with the current process code reflects the overall content among ED staff with the current triage procedures. The participants consistently noted that the existing triage system functions well, with no significant issues or areas requiring immediate improvement. One physician explicitly stated that they do not perceive any substantial management difficulties within the current process and that they cannot think of any necessary enhancements. This sentiment is echoed by another doctor, who mentioned that the current criteria used for triage seem sufficient, particularly when patient vital signs are taken, and complaints are assessed to determine the appropriate triage level.

In high-volume settings, however, there is an acknowledgment of the inherent challenges in managing large patient flows. One physician highlighted that while triage works well in smaller facilities with fewer patients, scaling the process to handle 1,200 to 1,500 patients daily in larger high-intensity EDs would be significantly more difficult. Despite these challenges, the consensus among participants is that the current triage process effectively meets the demands of the patient population, with no urgent need for modifications (Table 4).

Table 4.

Illustrative Quotes on Patient and Public Interaction.

Codes	Quotes
Facility use and management	• “We have a triage room, but we use it for multiple purposes. For instance, when we’re very busy, we also use it for treatments.” (ED Specialist 2)• “When there isn’t a nurse available to do triage, it causes delays in the triage process. This results in patients who need to be triaged having to wait. We don’t have a specific room for this. Normally, the area where triage is done should be a room right at the entrance of the emergency department. However, we don’t have such a room.” (ED Specialist 6).
Patient education	• “There is a perception that immediate intervention is necessary. Unfortunately, it is difficult to change this. That’s our biggest problem. Otherwise, it stems from education. If this kind of information were taught to children during their schooling years, they might grow up to be more understanding, but what they experience and see is not like that.” (ED Specialist 1).• “Public awareness needs to be raised on this issue. This is the most important change that needs to happen.” (ED Nurse 1).
Satisfaction with current process	• “Right now, no, there is not anything specific related to the current triage process. Based on the criteria used, it seems sufficient to me. You take the patient’s vitals, listen to their complaint, and then place them in the appropriate category based on that. It seems adequate.” (ED Specialist 3).• “Honestly, there’s nothing in this triage process that needs to be changed or improved. My emergency department sees about 40 patients a day. I could handle triage myself just standing at the door. But in high-volume emergency departments, it’s impossible to properly organize triage under the conditions in Turkey. Because with daily numbers of 1,200, 1,400, or 1,500 patients in any emergency department, it’s impossible to conduct triage correctly and objectively. There’s just not enough human resources for that.” (ED Specialist 9).

Technology and Machine Learning in Triage

This theme encapsulates the growing recognition of ML technologies, particularly regarding their evolving role in healthcare and triage (Table 5). The participants expressed varied levels of awareness, ranging from initial skepticism to gradual acceptance of AI and ML in medical practices. One participant highlighted how doubts about robotic triage systems shifted toward an acknowledgment of their potential after observing advancements in AI. Similarly, others noted the significant role of AI in medical imaging, where its capacity to increase diagnostic accuracy and efficiency was evident. Despite concerns about AI potentially supplanting human judgment, there is a consensus that AI can significantly support healthcare professionals by improving the precision and speed of medical procedures. This reflects a nuanced understanding among healthcare professionals of the current capabilities of ML and its future potential within the medical field.

Table 5.

Illustrative Quotes on Technology and Machine Learning in Triage.

Codes	Quotes
General awareness of machine learning	• “At that time, I thought, ‘No, this isn’t going to happen, such a thing is impossible, people need human touch,’ but after recent developments in AI, I became one of those who believe that it might actually be necessary.” (ED Specialist 3)• “I am not against AI in medicine. I believe that AI should advance in medicine. However, as I mentioned, in fields where the human factor is crucial, I think there should be a new model of AI that works in collaboration with humans. AI can diagnose, prescribe medications, and do everything. It can take a patient’s blood results, MRI, tomography, and other scans and instantly provide a diagnosis. But I believe this should be done in partnership with the human factor. You can’t remove the human factor, because no one wants to learn about their illness from a machine, like withdrawing money from an ATM. Imagine giving all your information to a machine that tells you, ‘You have lung disease.’ Imagine it giving you a paper saying, ‘You have stage 4 prostate cancer, with a 5% chance of surviving the next 10 years.’ Would you want to deal with such a machine?” (ED Specialist 9).
Lack of experience with machine learning	• “I have no experience with artificial intelligence. Nor have we used any platform in our emergency department. However, some of my colleagues have worked on this subject. I think it would be better, but I don’t know. I have no idea how the current methods are working. Recently, I had no knowledge or experience on how methods in AI have been transferred to tablets or how they are used.” (ED Specialist 6).• “It never happened. In the medical field? No, it didn’t.” (ED Nurse 4)
Limitations of current technology	• “Well, machine learning, of course, technology, it does a lot for us, but in terms of patient evaluation, I find it a bit difficult for machines to perform. In terms of fully evaluating a patient. Someone will input the vitals, and their complaints, so a human will input them. Is a machine going to examine a patient’s throat? I will evaluate and write it down. It will act according to my evaluation. I might write it down incorrectly. In the end, there is still an error. The margin of error remains with the human. Of course, the margin of error remains with the human. So, I’ll input the values correctly, but how can the machine evaluate everything from top to bottom?” (ED Specialist 1).• “Let me say this: I have never interacted with Artificial Intelligence until now. I believe nursing is entirely dependent on human effort. For example, when a patient’s triage is 3, they are taken to the emergency room and treated. During the intervention, the patient’s triage can change to 2 or even 1. Can artificial intelligence handle that?” (ED Nurse 3)
Potential capabilities for machine learning	• “You know what could happen? I can imagine an AI for triage that can visually detect a patient’s phenotype and behavior patterns. For example, it could alert us if a patient is clutching their chest and sweating, indicating a possible heart attack. Or when someone enters the ER holding their right side, it could identify that it’s likely a kidney stone, assigning them a red alert. Instead of just analyzing data and giving us results, it would be incredible if AI in triage could combine those data points with visual human phenotypes and behaviors.” (ED Specialist 9).• “We’ve started implementing treatment protocols with it. Both the treatment protocol and the way the patient is examined, all are moving towards an algorithmic approach. Therefore, I think artificial intelligence will be very effectively used in the initial reception of patients. I believe AI can easily handle the tasks I mentioned or the work of healthcare professionals. When I say easily, I mean it will be one of the most effectively utilized areas.” (ED Specialist 3).
Suggestion for machine learning inputs	• “There could be pain assessment. Pain assessment. The onset time of the complaint could be considered.” (ED Nurse 1)• “I’m not sure if it can be added, but visual data like the patient’s skin color, whether they are sweating, their cornea color, or if there is any cyanosis on the lips—this could be assessed by a camera system.” (ED Specialist 2).• “What else did we mention, why might it be processed? The form of the complaint, I mean, which complaint, and whether it has remained the same. And the spread. I mean the order of the complaint—how it started, how it progressed, what the final state of the complaint is.” (ED Specialist 4).• “If the patient arrives with chest pain early in the morning, you can consider this patient to be more serious. There’s a big difference between a patient who says, ‘My chest hurts’ while sitting in the evening and goes to bed, and one who wakes up from sleep with chest pain.” (ED Specialist 1).

There is growing recognition among healthcare professionals of the potential of ML in streamlining triage processes and improving patient outcomes, although familiarity with its applications varies widely. Some practitioners are beginning to see how transformative ML could be in making triage procedures more efficient. However, despite this increasing awareness, many healthcare workers have not yet engaged directly with ML tools, leading to a reliance on traditional methods. This lack of hands-on experience represents a significant barrier to the widespread adoption of ML in clinical settings, underscoring the need for more training and exposure to these technologies.

Healthcare professionals also recognize significant limitations in current technology, including concerns about the accuracy of ML models and the essential need for human oversight to ensure patient safety. These limitations suggest that while ML holds considerable promise, there are still challenges that need to be addressed before it can be widely adopted in triage settings.

Despite these challenges, there is considerable optimism about the potential capabilities of ML in triage. Healthcare professionals envision a future where ML enhances the speed and precision of patient assessment, helping to quickly identify and prioritize cases on the basis of symptoms and vital signs. This potential is particularly valuable in high-pressure environments where rapid decision-making is crucial.

To harness the full potential of ML in triage, it is crucial to enhance algorithms by integrating diverse data inputs, including behavioral and contextual information. Such enhancements would help ML models better mimic human judgment, leading to more accurate and nuanced decision-making and ultimately improving patient care in emergency settings.

In the context of developing a ML algorithm for triage, participants emphasized the critical importance of including pain assessment as a variable because of its significant impact on triage decisions. They highlighted that variables such as the onset time, intensity, and location of the pain could greatly influence the triage process. Additionally, they suggested that visual data, such as skin color, sweating, and other physical symptoms, should also be considered. These factors, they argued, could improve the accuracy of the algorithm by providing a more comprehensive understanding of the patient’s condition.

ED specialists emphasize that pain is a critical variable in determining triage levels. However, previous ML-based triage research has largely overlooked key pain characteristics, such as type; onset time; contributing and relieving factors; and location, frequency, and intensity, which are essential for accurate assessment.

Triage Process and Challenges

The triage process in ED is designed to prioritize patients based on the urgency of their conditions, ensuring that critical cases receive immediate attention. However, this process is inherently dynamic and presents multiple challenges (Table 6). As one emergency specialist highlighted, “In the emergency department, there are normally 32 parameters used to evaluate patients as emergency cases. Based on these criteria, patients are categorized into red, yellow, and green zones. The yellow and green zones are somewhat intertwined in our system. It mostly comes down to red and green zones. Chest pain, high fever — these are critical emergencies categorized into the red zone, whereas the rest fall into yellow and green zones” (ED Specialist 1). This form of categorization is common in many triage systems, including the widely utilized Manchester Triage System (MTS), which has been shown to facilitate rapid identification of critically ill patients. ²⁷

Table 6.

Illustrative Quotes on Triage Process and Challenges.

Codes	Quotes
Triage protocol and flow	• “When the patient first comes to the emergency room, they are met by the triage nurse who takes their complaint and vital signs. If the nurse considers it a critical case, they will place the patient in the red zone for immediate care. Otherwise, they will be placed in yellow or green zones based on their condition.” (ED Specialist 9).• “We use the 5-level triage system here, the Manchester system. Based on the patient’s complaint, the triage nurse greets them. We have 2 different entries, one for ambulances and one for walk-ins. At both entry points, our nurses are present, all of whom are competent in triage. These nurses evaluate the patient based on their condition according to the triage scale and take them to the appropriate area for further assessment.” (ED Nurse 1).• “The first person the patient meets are the triage nurse, who assesses the complaint and vital signs. The nurse then categorizes the patient into red, yellow, or green zones. Afterward, the doctor makes the final decision.” (ED Nurse 4).
Vital signs collection	• “Based on the patient’s complaints, such as chest and back pain or severe headaches, we evaluate the patient’s vitals. We check their blood pressure, temperature, and pulse. If there is an emergency, these signs are already evaluated. In trauma cases, for example, an arterial cut is treated as an emergency.” (ED Specialist 1).• “In the file, there is an anamnesis form, but the first vital signs we check include pulse, respiratory rate, oxygen saturation, blood pressure, and temperature. In addition, we ask about their pain, past medical history, and other relevant information.” (ED Nurse 4).
Critical triage factors	• “In my experience, I believe the patient’s history is critical—by history, I mean probing the complaint. What is the complaint? What are the symptoms? What type is it? Where is it located? Is there any radiation? I believe the history largely determines the triage outcome. Of course, vital parameters also play a role, but a good history already gives you a clear picture.” (ED Specialist 4).• “Pain is the biggest factor. If you take a good history, there’s no problem. But if you don’t take a good history, the triage scale can vary greatly depending on the type of pain and its onset. If there’s new onset of back pain, you might consider dissection and give it a triage level of two, or it could even be a level 4. It’s important to ask about pain—its onset, radiation, and so on. When you thoroughly assess these, you can more accurately assign the patient to the correct triage category.” (ED Specialist 6).
Decision making in triage	• “Vital signs are taken during the first triage—blood pressure, pulse, respiratory rate, saturation, temperature. Of course, these are all done simultaneously. Apart from that, past medical history is crucial, but the primary focus is the patient’s complaint. We often look for ‘red flags’ in the complaint, such as chest pain, shortness of breath, altered consciousness, or epigastric pain that might mimic cardiac issues. These factors help determine whether we’re dealing with a serious case or not.” (ED Specialist 4).• “I would say the correct anamnesis, the correct history from the patient, is the most important. You can see the patient’s vital signs for yourself, and you can physically evaluate the patient, but in the anamnesis part, it’s crucial that the patient provides accurate information and explains what’s happening clearly.” (ED Nurse 4).
Challenges in triage	• “One of the difficulties is abdominal pain, for example. Because abdominal pains are subjective. The patient describes it. You might catch some findings during your examination, but it’s usually sensitivity. . . Determining the severity of an abdominal pain complaint, its triage, is difficult.” (ED Specialist 4).• “Certainly, the crowd is a factor in triage. . . The more patients there are, the harder it becomes to triage. Secondly, who is doing the triage matters.” (ED Specialist 5).• “Interference from the patient’s relatives can be a challenge during triage. Poorly taken vital signs, or vital signs taken by rote, can also be a problem during triage.” (ED Specialist 7).• “Sometimes patients don’t want to reveal their actual complaint, or they try to hide it. They won’t tell the doctor or the nurse, and this complicates things.” (ED Nurse 4)
Communication barriers in triage	• “However, for instance, the patient doesn’t state their complaint. We must extract it from them. Sometimes, they don’t tell us at first. When the results come out, you ask, ‘Is this it?’ Only then do they admit it. This is very common in Turkish emergency departments.” (ED Specialist 4).• “I could say that patients’ reluctance to provide too much information contributes to this.” (ED Nurse 4).
Diagnostic challenges	• “Yes, young patients can have chest pain. Personally, you can take chest pain seriously, but if you dismiss it because a young patient isn’t likely to have a heart attack, it can still lead to complications later. That’s why it’s important to carefully ask about the patient’s initial complaint—pain, its intensity, and type. You need to question the duration well. The evaluation of someone who’s had chest pain for 5 days is not the same as someone who has had it for just an hour. So, this is certainly important.” (ED Specialist 1).• “In our case, for example, the most frequent mix-up is between stomach problems and heart problems. A patient comes in with heartburn, and if you don’t differentiate it, you can follow that course, but the patient might be having a heart attack. The triage changes between these two.” (ED Specialist 2).
Documentation	• “We usually write detailed notes in the medical history. For example, the characteristics of the pain, its type, and intensity. We also have a numeric pain scale in our system. Below that, we note the type of pain and whether it radiates. If it does, we record where it radiates to. We fill this out for every patient and enter it into our Cerebral system. We also write it in the patient’s history, as it's an important group of data. We document this information to justify our thought process and to provide supporting evidence.” (ED Specialist 4).
Dynamic nature of triage	• “Therefore, patients aren’t directly assigned to red, yellow, or any other triage category with rigid lines. It’s not as definite as ‘this is red, this is yellow,’ because illness is a process. A patient may start in one triage category and transition to another.” (ED Specialist 2).• “I’ll give an example. A patient says they feel short of breath. We assess them, and their blood pressure is normal, oxygen saturation is normal, temperature is normal, everything seems fine. But when we listen to the lungs during a physical exam, we find one lung completely silent, no breath sounds. At that moment, we change the triage. A patient we initially classified as green; we now move to red based on the physical exam. Likely a pneumothorax, and they’ll need a chest tube.” (ED Specialist 7).
Training and experience of health staff	• “As I said, everything is about the patient's complaint and its evaluation. The content is knowledge and experience.” (ED Specialist 1).• “Of course, ideally, the triage should be done by a doctor — in emergency departments, or anywhere in the world, the most competent person should do it. But it’s not feasible for the most competent emergency doctor to always be at the front doing triage. That’s why it’s valuable that the nurse, as the first person to greet the patient, has more experience and has seen more cases in this regard.” (ED Nurse 4).• “In the past, we had someone dedicated only to that role [triage]. They would sit separately at the front, doing the welcoming. Then the numbers decreased, and now they both take care of patients and do triage when available. Whoever is free at that moment does the triage and informs the available doctor. Also, I don’t know if there are any questions, but in our case, the triage team doesn’t receive formal training. In my opinion, there should be proper training.” (ED Specialist 4).

In practice, the initial triage is performed by a nurse, who evaluates the patient’s complaints and vital signs. As one nurse explained, “When the patient first comes to the emergency room, they are met by the triage nurse, who takes their complaint and vital signs. If the nurse considers it a critical case, they will place the patient in the red zone for immediate care. Otherwise, they will be placed in yellow or green zones based on their condition” (ED Nurse 4). This approach is supported by the literature, which emphasizes the role of nurses in making critical first-line decisions in triage. ²⁸ However, while nurses handle the initial evaluation, the final decision is typically made by the attending physician, reinforcing the collaborative nature of triage within the healthcare team.

Vital signs play a crucial role in determining a patient’s triage category. A specialist commented, “Based on the patient’s complaints, such as chest and back pain or severe headaches, we evaluate the patient’s vital signs. We checked their blood pressure, temperature, and pulse. If there is an emergency, these signs are already evaluated. In trauma cases, for example, an arterial cut is treated as an emergency” (ED Specialist 1). The importance of promptly assessing vital signs, including blood pressure, oxygen saturation, and pulse, is well documented in studies of emergency care. Vital signs provide a quantitative measure of the patient’s status, enabling healthcare providers to make informed decisions quickly. ²⁹ As another specialist noted, “Vital signs such as blood pressure, oxygen saturation, and general condition based on the patient’s mental status, or whether the patient is conscious or cooperative, all contribute to the triage process. Pain is also part of the triage scale” (ED Specialist 2). This underscores the multidimensional nature of triage assessments, where both physiological data and patient-reported symptoms must be integrated for an accurate evaluation.

One of the primary difficulties highlighted during the interviews was the complexity of assessing subjective complaints such as abdominal pain. This issue aligns with findings in the literature that emphasize the variability in patients’ pain perception and expression, which can complicate the triage process. ³⁰ Additionally, the specialist mentioned that “the healthcare provider evaluating the patient is key. This process can vary depending on the person making the evaluation. Of course, I think the decision of the healthcare provider is more important because they might prioritize a less serious patient over one with a potentially serious condition” (ED Specialist 1). This comment reflects broader concerns in the literature about the subjectivity in triage decision-making and the potential for bias based on the evaluator’s experience. ³¹

The issue of crowding in ED also complicates the triage process. One specialist noted, “Certainly, the crowd is a factor in triage. . . The more patients there are, the harder it becomes to triage. Second, who is doing the triage matters” (ED Specialist 5). This observation is consistent with previous research, which has shown that ED overcrowding can significantly impair the effectiveness of triage, leading to delayed care for critically ill patients. ³² The challenge is further compounded by the shortage of beds, as one nurse described: “The biggest challenge we face is the shortage of beds. Additionally, overcrowding in wards is an issue. Even if we do the triage, we might not have a place to admit the patient” (ED Nurse 1). This speaks to systemic issues within healthcare facilities, where resource limitations, such as bed availability, directly affect patient care. ³³

Another critical factor in the triage process is the dynamic nature of patient conditions. As one emergency specialist explained, “Patients aren’t directly assigned to red, yellow, or any other triage category with rigid lines. It is not as definite as ‘this is red, this is yellow’, because illness is a process. A patient may start in one triage category and transition to another” (ED Specialist 2). This observation aligns with the notion of dynamic triage, where ongoing reassessment is essential as patients’ conditions evolve over time. ³⁴

Studies have shown that formal triage training improves decision-making and reduces errors in patient categorization. ³⁵ However, as the specialist observed, “the triage team does not receive formal training. In my opinion, there should be proper training” (ED Specialist 4), pointing to a gap in current practice that could be addressed to improve outcomes.

Professional Role Perspectives

Comparative analysis of physician and nurse perspectives revealed both convergent views and distinct professional priorities regarding ML implementation in triage processes. While both professional groups demonstrated cautious optimism about ML potential and emphasized the critical importance of human oversight, notable differences emerged in their focus areas and primary concerns.

Both physicians and nurses expressed similar levels of cautious optimism about ML’s potential to enhance triage efficiency and accuracy. Participants from both professional groups consistently emphasized the irreplaceable value of human clinical judgment and the complexity of subjective assessments, particularly pain evaluation. Both professional groups acknowledged the challenges of incorporating subjective clinical variables into algorithmic systems and stressed the necessity of maintaining human oversight in critical decision-making processes

Analysis revealed distinct professional focus areas between physicians and nurses. Physician participants demonstrated primary concern with clinical decision-making authority and diagnostic accuracy, emphasizing systematic approaches to triage. ED Specialist 1’s comment that “the healthcare provider evaluating the patient is key. . .the decision of the healthcare provider is more important” reflects this emphasis on clinical authority and diagnostic responsibility. Physicians also focused more extensively on the technical capabilities of ML systems and their integration with clinical reasoning processes, particularly regarding the incorporation of complex clinical variables such as pain characteristics and patient behavioral cues.

In contrast, nurse participants prioritized operational and workflow considerations, demonstrating acute awareness of resource constraints and practical implementation challenges. ED Nurse 1’s identification of “shortage of beds” and “overcrowding in wards” as primary challenges illustrates this operational focus. Nurses emphasized the practical aspects of triage workflow, with ED Nurse 4 providing detailed description of the patient flow process: “When the patient first comes to the emergency room, they are met by the triage nurse, who takes their complaint and vital signs.” This operational perspective highlighted concerns about how ML implementation would affect day-to-day workflow efficiency and resource allocation.

These professional perspective differences suggest that successful ML integration in triage requires addressing both clinical decision-making concerns and operational workflow considerations. Physicians’ emphasis on diagnostic accuracy and clinical authority indicates the need for ML systems that enhance rather than replace clinical judgment, while nurses’ focus on operational challenges suggests that ML implementation must address workflow efficiency and resource optimization to gain acceptance across the healthcare team. The complementary nature of these perspectives underscores the importance of involving both professional groups in ML system design and implementation processes.

Discussion

Our findings highlight that while healthcare professionals are cautiously optimistic about the potential of ML in triage, they underscore the continued need for human oversight, particularly in areas such as pain assessment, which remains difficult for machines to interpret accurately. This finding is consistent with the study by Ilicki, ³⁶ which highlighted that ML models frequently face challenges in accounting for subjective symptoms, such as pain—an essential factor in triage decision-making.

Our participants, especially physicians, echoed this concern, emphasizing that triage involves not only physiological metrics but also nuanced clinical judgment based on a patient’s behavior, nonverbal cues, and historical data. In line with the findings of Chiu et al, ³⁷ integrating such unstructured data into ML systems—such as clinical notes or even visual cues from patients—could increase the accuracy of these models.

Moreover, our study contributes to the growing conversation around the limitations of current technology. While AI applications, such as those used in medical imaging, have shown high diagnostic accuracy, ³⁸ these systems typically rely on structured data that are less dynamic than the real-time decision-making required in triage. Studies by Chen et al ³⁹ support this, suggesting that while ML algorithms handle static data well, the dynamic nature of ED patients makes full automation of triage unlikely without major advances in adaptability. Our participants consistently indicated that real-time adjustments during triage, such as downgrading or upgrading a patient’s status based on evolving symptoms, are difficult to mimic in current ML systems.

While there is optimism about the potential of ML, the skepticism surrounding its integration into ED workflows is substantial. Previous studies, such as Ahmed et al, ⁴⁰ have argued that healthcare professionals’ resistance to ML adoption is often rooted in a lack of exposure and training. This finding resonates with our findings, where many participants acknowledged their lack of direct experience with AI or ML technologies. This unfamiliarity not only creates a barrier to trust but also reveals the importance of training and organizational support in the successful deployment of ML systems.

Another critical issue raised in both our study and the literature is the quality of the data used to train ML models. According to Parets et al, ⁴¹ poor data quality, incomplete records, and biased datasets can lead to inaccurate predictions, which could exacerbate disparities in healthcare access. Our findings align with this concern, as healthcare professionals noted that ML models need to incorporate a wider range of data, including both structured and unstructured forms, to truly mimic the depth of human judgment in triage. For example, physiological data alone (vital signs such as blood pressure and heart rate) may not be sufficient to capture the full clinical picture, and incorporating behavioral cues such as a patient’s demeanor or nonverbal indicators of pain could significantly increase model accuracy.

Emergency staff highlighted the need for models to account for contextual factors such as social determinants of health, which are often overlooked in ML-based systems but are known to impact patient outcomes. Studies by Siddique et al ⁴² have shown that ML models trained without considering socioeconomic factors may produce biased results, further complicating their use in high-stakes environments such as the ED. Our participants suggested that future models should be trained with datasets that reflect the diversity of real-world ED populations, including variables related to race, socioeconomic status, and comorbidities, to avoid biased outcomes.

The distinct professional perspectives observed in our study align with emerging literature on interprofessional differences in AI technology adoption within healthcare settings. Research indicates that nurse practitioners remain at the “developmental” or “experimental” stage regarding AI, rather than at the “adoption” or “assimilation” stage, ⁴³ which may explain the operational and workflow-focused concerns expressed by nurses in our study. Studies have shown that nurses appreciate AI’s potential to streamline workflows and reduce administrative burdens, while also expressing concerns about dehumanization of care and job displacement, ⁴⁴ reflecting the dual perspective we observed where nurses emphasized both efficiency gains and human-centered care considerations.

Literature suggests that AI implementation will spur “a fundamental alteration in provider roles, shifting the anchor of professional identities from the possession of a specific fund of knowledge toward expertise in accessing, assessing, and applying that information,” ⁴⁵ which may explain physicians’ emphasis on maintaining clinical decision-making authority observed in our findings. Research emphasizes that successful AI integration requires profession-specific approaches to training and implementation, acknowledging that different healthcare professionals have distinct needs and concerns. ⁴⁶ This supports our finding that physicians focused on diagnostic accuracy and algorithm capabilities while nurses prioritized operational challenges and resource optimization.

The importance of interprofessional collaboration in AI implementation has been highlighted, with experts noting that “learning about, from, and with one another adds rich perspectives that will enable health professions to anticipate and mitigate perils and amplify the promise of AI.” ⁴⁵ Our findings demonstrate this complementary nature of professional perspectives, suggesting that successful ML-based triage implementation requires addressing both the clinical decision-making concerns raised by physicians and the operational workflow considerations emphasized by nurses. This interprofessional approach aligns with recommendations for collaborative development involving both data scientists and clinicians to ensure that AI tools meet diverse professional needs and integrate seamlessly into clinical workflows. ⁴⁷

To ensure that ML models align with the needs of healthcare professionals, future research should focus on collaborative development involving both data scientists and clinicians. The incorporation of clinician feedback during model development could lead to the creation of algorithms that not only predict patient outcomes but also integrate seamlessly into clinical workflows. Moreover, training programs must be developed to increase clinicians’ familiarity with ML systems, addressing the current knowledge gap that limits adoption. This approach is supported by the findings of Nasarian et al, ⁴⁷ who argue that involving clinicians in the development process fosters trust and ensures that ML tools meet clinical needs.

A key area not directly explored in our interviews is the balance between sensitivity and specificity in ML-based triage — in other words, the risk of over-triaging versus under-triaging when compared to human classifiers. Over-triage (high sensitivity) can lead to excessive alerts and increased clinician workload, whereas under-triage (high specificity) risks delayed care for genuinely critical patients.^26,27 Although our participants focused on accuracy and clinical integration, they did not spontaneously discuss these nuanced trade-offs, which suggests this may be an underappreciated consideration among ED staff. We recommend that future studies explicitly evaluate ML algorithms across a range of sensitivity-specificity thresholds, measure rates of over- and under-triage against human benchmarks, and assess downstream effects on staff workload, patient outcomes, and clinical workflow sustainability.

Further research should also investigate hybrid models, where human oversight complements algorithmic predictions. This would address healthcare professionals’ concerns about the reliability of fully automated systems in critical environments such as EDs. Additionally, longitudinal studies are needed to evaluate the long-term impact of ML-based triage on patient outcomes, workflow efficiency, and clinician satisfaction. Such studies could also explore the role of dynamic triage systems, where continuous patient monitoring (eg, through wearable technology) updates triage categories in real time.

Finally, as suggested by several participants, future ML systems should be designed to account for subjective variables such as pain intensity and behavioral cues, which are often omitted in current models. Expanding datasets to include unstructured data—such as clinicians’ notes, patient speech, and nonverbal cues—could significantly improve the accuracy and utility of these systems, a hypothesis supported by recent advances in natural language processing.

Conclusion

In conclusion, this study highlights the importance of integrating healthcare professionals’ nuanced perspectives into the development and implementation of ML algorithms in triage systems. One critical area highlighted by healthcare professionals is pain assessment, which is a major factor in triage decisions. The characteristics of pain—including its onset time, intensity, location, and radiation—are vital indicators of a patient’s condition and can significantly influence the urgency assigned during triage. For example, whether a patient experiences chest pain upon waking up or at rest can change the entire course of their treatment. These nuanced factors are essential for accurately determining the severity of a patient’s condition and must be integrated into any ML-based triage system.

Furthermore, the time of pain onset is particularly crucial, as it can be the difference between a routine case and a life-threatening emergency, such as an acute myocardial infarction or an aortic dissection. Ignoring such detailed information in ML algorithms would lead to a significant gap between real-world clinical needs and the algorithm’s outputs, potentially resulting in suboptimal patient outcomes.

Another key insight from this study is that the development of ML algorithms must be tailored to the specific needs of healthcare professionals rather than merely utilizing available data for the sake of creating a model. Designing an algorithm based solely on the data at hand, without considering the actual needs and workflows of ED staff, would render the tool less effective or even irrelevant in practice. An algorithm that fails to capture critical factors such as pain characteristics or the dynamic nature of patient conditions will be of limited use. A needs-driven approach, as opposed to a data-driven approach, is essential to ensure that these tools are adopted and trusted by clinicians in the fast-paced environment of emergency care.

Supplemental Material

sj-pdf-1-inq-10.1177_00469580251376921 – Supplemental material for Beyond Algorithm: Emergency Department Professionals’ Perspectives on Machine Learning-Based Triage Integration—A Qualitative Study

Supplemental material, sj-pdf-1-inq-10.1177_00469580251376921 for Beyond Algorithm: Emergency Department Professionals’ Perspectives on Machine Learning-Based Triage Integration—A Qualitative Study by Güvey Emre M., Esen Fevzi M. and Onganer Efe in INQUIRY: The Journal of Health Care Organization, Provision, and Financing