Sage Journals: Discover world-class research

Abstract

Background:

Patients performing self-triage for chest pain need to pick an algorithm which includes questions pertinent to chest pain to triage accurately.

Objective:

Our study reviews patient self-triage choices and outcomes for patients who completed an online self-triage encounter before being triaged by a nurse for chest pain.

Methods:

Patients who underwent telephone nurse triage for chest pain, and also had an online self-triage encounter within the prior 24 hours were reviewed for the frequency with which they chose “chest pain” as their symptom to triage for during self-triage, whether they reported chest pain during the self-triage encounter (if asked as part of the algorithm for their chosen self-triage symptom) and whether the patient had any follow up evaluation within 1 week.

Results:

There were 70 self-triage and nurse triage dyads during the study period where triage nurses chose “chest pain” as the primary symptom to be triaged for during the telephone triage encounter. Of these, only 5 (7.1%) patients chose the “chest pain” self-triage option during online self-triage. During the self-triage encounter, 50 (71%) reported chest pain, 10 (14%) denied chest pain and 10 chose to self-triage their symptoms with an algorithm that did not include a question on whether chest pain was present. Of the 70 total dyads 59 (84%) had a follow up emergency department (40 patients) or office visit (19 patients) documented.

Conclusion:

Most patients chosen to be triaged for chest pain by triage nurses did not choose the online self-triage algorithm for chest pain even though the majority did report chest pain during their self-triage encounter (when asked). This is concerning from a safety perspective as a non-chest pain self-triage option chosen by the patient may not always ask about the presence or absence of chest pain and thus could increase the risk of being triaged inaccurately. Implications for improvement of self-triage could include diagrams for patients to choose the symptom area.

Keywords

telemedicine chest pain online triage symptom checkers self-triage

Introduction

In healthcare, the goal of triage is to match appropriate patient care to patient symptoms using a preset algorithm.¹ Recently healthcare systems have utilized online assessment for a variety of uncomplicated illnesses^2
-4 and development of online self-triage and self-scheduling has progressed dramatically since the COVID pandemic to meet increased demand for access to healthcare.^5
-9 While self-triage is becoming more widely used, systematic reviews have found the evidence for patient safety to be weak with variable results in diagnostic accuracy and appropriateness of triage.^10,11

Online self-triage checkers can be used to triage patients with chest pain.^12,13 Because chest pain can be a symptom of potentially life-threatening disease processes including acute coronary syndrome (ACS), pulmonary embolism and aortic dissection^14
-18 expedient and accurate triage is important to identify patients who need urgent evaluation for their chest pain. For self-triage checkers to safely and appropriately triage patients with chest pain, the algorithm used in the self-triage needs to include questions pertinent to triage chest pain. For the algorithm to include pertinent questions, either the patient must choose “chest pain” to self-triage for or the chosen algorithm would need to include questions on chest pain and associated symptoms. Though it would seem logical that patients with chest pain would choose “chest pain” as the symptom to self-triage for, patients with multiple symptoms or those not understanding how to identify a symptom as chest pain may choose an entirely different symptom to triage for. This could potentially miss patients with chest pain (if the chosen symptom algorithm does not include a question on whether chest pain is present) as well as increase the risk of inaccurate triage of patients with chest pain (if the algorithm does not include questions pertinent to the triage of chest pain).

To ascertain how often online self-triage led to missed chest pain triage, we explored the frequency with which patients who report chest pain to a telephone triage nurse also reported chest pain during an online self-triage encounter performed less than 24 hours previously. We also reviewed follow up care recommendations and outcomes for these patients.

Methods

Description of Mayo Clinic Online Self-Triage

Patients who are empaneled to a Mayo Clinic primary care provider and who have a patient portal can log into the patient portal to access a menu of possible symptoms to perform self-triage for (Supplemental Table 1). Our self-triage tool does not allow patients to pick multiple symptoms. Rather, the patient chooses one symptom that they feel best covers their concern(s). The online self-triage then asks a series of algorithmic questions to arrive at an endpoint for care recommendations. Some of the non-chest pain symptom menu options still include a question on whether chest pain is present or not. Care recommendations can include a recommendation for home care, to contact nurse triage, to schedule a visit or to access emergency care.

Description of Nurse Telephone Triage

Patients empaneled to a primary care provider have 24/7 access to a nurse triage line that uses the Schmitt-Thompson triage decision support tool.¹⁹ During nurse triage, the nurse chooses a primary symptom to triage for based on their training and discussion with the patient over the phone.

We reviewed all patients who performed online self-triage followed by nurse phone triage within 24 hours of their self-triage for the period between September 1, 2022, and September 30, 2023. All self-triage/nurse triage dyads where the nurse triaged for the symptom of chest pain were included in this study (Figure 1). For these dyads we reviewed the electronic medical record (EMR) for the following information: the symptom chosen by the patient during their online self-triage encounter, whether the patient reported chest pain during the self-triage encounter (if asked as part of the algorithm for their self-triage symptom choice) and whether the patient had any follow up evaluation within 1 week for their symptoms. For patients that had follow up in our medical system we reviewed if the follow up encounter included hospitalization or emergency department (ED) visit, whether the patient reported chest pain during the follow up encounter, whether evaluation for acute coronary syndrome (ACS) was performed and diagnosis at the encounter to review for potentially life-threatening diagnoses. We also electronically captured data for the time between the self and nurse triage encounter and whether any patients died within 1 month of their triage. This study was deemed exempt by the Mayo Clinic Institutional Review Board (IRB 20-006809). Only patients with institutional written research authorization were included in this study.

Figure 1.

Flow sheet showing strategy used to identify self-triage/nurse triage dyads for study inclusion during study period September 1, 2022 through September 30, 2023.

Results

Seventy dyads where a triage nurse chose to triage the patient’s symptom(s) as chest pain were eligible for study. Of these 70 dyads, only 5 (7.1%) were self-triaged as chest pain by patients during online self-triage. Table 1 shows the list of symptoms patients chose to triage for during self-triage.

Table 1.

Self Triage Symptoms Chosen by Patients During Online Self-Triage for 70 Encounters Triaged as Chest Pain by Nursing Triage.

Self-triage symptom chosen by patient	Number (%)
Covid and strep symptom testing	32 (45.7)
Belly pain	11 (15.7)
Nausea or vomiting	7 (10)
Chest pain	5 (7.1)
Anxiety or depression	4 (5.7)
Cough	3 (4.3)
Arm or hand concerns	2 (2.9)
Shoulder concerns	2 (2.9)
Anal or rectal concerns	1 (1.4)
Low back pain	1 (1.4)
Sore throat only	1 (1.4)
Vaginal bleeding or menstrual concerns	1 (1.4)

Median time between self-triage and nurse triage was 16 minutes (interquartile range 8-107 minutes) with 53 (75%) of dyads having less than 2 hours between self and nurse triage. Mean age of patients was 40 years (range 15-74 years old) and 51 (72.9%) of patients were female. A slight majority of the self-triage encounters were performed outside of regular clinic hours (52.9%).

Of the 70 total dyads 59 (84%) had a follow up emergency department (40 patients) or office visit (19 patients) within 1 week that was documented in the EMR for additional care. Table 2 shows the diagnosis for these patients.

Table 2.

Diagnosis for 59 Dyads Seen for Follow Up Care.

Diagnosis	Number
Cough/covid/respiratory infection	17
Atypical chest pain	8
Chest pain	8
Chest wall pain/costochondritis	8
GERD/GI pain	6
Anxiety	2
Pleurisy	2
Asthma	1
Pneumonia	1
Dysmenorrhea	1
Iron deficiency anemia	1
Palpitations	1
Pericarditis	1
Herpes zoster	1
Dizziness	1

We also reviewed how many dyads reported chest pain at the time of their online self-triage and found that 50 (71%) reported chest pain, 10 (14%) denied chest pain and 10 chose self-triage symptoms whose algorithm did not include a question on chest pain. All 10 patients that chose self-triage symptom algorithms that did not ask about chest pain did have a follow up encounter. Of these 10 patients, eight did report chest pain at their follow up encounter while one denied chest pain and one patient did not have any documentation to indicate whether they reported chest pain.

Eighteen patients had evaluation for acute coronary syndrome during an emergency department evaluation, and, of these 18 patients, none were found to have evidence for acute coronary syndrome. Only one patient, found to have severe iron deficiency anemia related to menorrhagia, was hospitalized within 24 hours of their self and nurse triage. Of the 59 patients with follow up care, 56 reported chest pain at their follow up encounter while 2 denied chest pain and, for one patient, as reported above, there was not any documentation regarding the presence or absence of chest pain at the follow up encounter.

Interestingly only one of the 18 patients who had evaluation for acute coronary syndrome chose to self-triage their symptom as “chest pain” with the others triaging as shoulder concerns (2), Anxiety or depression (3), belly pain (4), Covid and strep throat symptoms (7), nausea or vomiting (1).

No patients died within 1 month of their triage encounters.

Discussion

Most patients triaged for chest pain by a telephone triage nurse did not choose the online self-triage algorithm option for chest pain. Of the 60 patients who chose a self-triage algorithm that included a question on whether chest pain was present, 50 patients did report chest pain as a symptom during the self-triage encounter. Our finding that the majority (when asked) did report chest pain during their self-triage encounter but did not choose the chest pain self-triage algorithm is concerning from a safety perspective. Specifically, not all of our self-triage algorithms include a question on whether chest pain is present and, indeed, 10 (14%) patients in this study chose a self-triage algorithm choice that did not include a query on whether chest pain was present or not.

Potentially even more concerning is the fact that only one of the 18 patients who ultimately were felt to have symptoms warranting evaluation for acute coronary syndrome chose to self-triage their symptoms with the chest pain algorithm. Though our study cannot determine the reason that patients would not choose the chest pain symptom when using online self-triage to triage their symptoms in this scenario, one possibility includes that they may have had atypical presentations of ACS (such as epigastric or jaw pain)²⁰ however, all 18 of these patients did report chest pain during their ED encounter suggesting against this as a contributing factor. Additionally, all 18 patients were triaged as chest pain by triage nurses also suggesting chest pain was present. Another possible reason people with chest pain may not choose to triage their symptom as chest pain is if the chest pain is part of a symptom complex (such as a respiratory infection) they may understandably feel triaging for COVID is more appropriate than triaging for chest pain. Indeed, one pattern that emerged in our study was that almost half of the patients in this study chose the symptom category of “COVID symptoms” to self-triage for. This algorithm does ask about the presence or absence of chest pain and includes a possible endpoint for emergency care (depending on severity and other clinical factors) so that patients with respiratory symptoms and chest pain would receive appropriate triage. This heterogeneity in patient choice for what to self-triage for echos a finding in a previous study we performed that showed patients chose symptoms that were not always a best match for those chosen by trained triage nurses.²¹

In contrast to the above, some of the patient choices to self-triage for were difficult to reconcile with chest pain syndromes including one use of “anal/rectal concerns” and “sore throat only.” Additionally, one concerning pattern seen in our review was that there were two different patients who chose to self-triage their symptoms as “shoulder concerns.” The shoulder self-triage algorithm does not ask about chest pain and both self-triaged to home care. Though triaged to home care, both chose to contact nurse phone triage and, during nurse triage, both reported chest pain and had a triage endpoint of emergency care evaluation. Both patients presented to the ED where it was felt that ACS evaluation was indicated. Fortunately, both patients ruled out for ACS. Of note, the phone triage for these two patients occurred 14 and 17 minutes after their self-triage encounter suggesting that likely their symptoms were similar at the time of self and nurse triage.

Fortunately, no patients in our study were diagnosed with ACS. A study on the prevalence of life-threatening diagnosis in patients with chest pain presenting for evaluation to the ED was found to be overall 5.5% with 5.1% of potentially life-threatening causes attributed to ACS and the other 0.4% associated with less common diagnoses. Prevalence of ACS was found to be higher in older age groups.¹⁴ Our study design does not allow us to determine if the relative younger mean age (40 years) of our study population may have impacted the finding that none of the patients in our study were diagnosed with ACS. The lack of any diagnosis of ACS also could be related to the relatively small sample size of our study. It is also possible that patients with chest pain related to ACS may differ in presentation severity to the extent that they may not use an online symptom checker to assess their symptoms.

Many evaluations of the diagnostic and triage accuracy of online symptoms checkers use clinical vignettes rather than actual patients.^11,22 Our study highlights the value in using real world scenarios to evaluate symptom checkers as outcomes of actual patients use of symptom checkers may differ when compared with clinical vignettes.

Because of the heterogeneity in what symptoms patients choose to self-triage for, as well as variability in how they report their symptoms, symptom checkers may need greater overlap and redundancy in algorithm questions used for assessment. Concerns may be raised that increased redundancy could increase the risk of over triage, which could result in misallocated utilization of resources.²² Interestingly, a recent systematic review by Riboli-Sasco et al²³ found that under triage was also a problem for some online symptom checkers with the potential for poor outcomes for patients. Ceney et al²⁴ also investigated the accuracy of symptom checkers and reported that accuracy is lower than desired and symptom checkers appear to be more accurate for less urgent concerns. That a significant number of patients in our study were deemed to need ACS evaluation supports redundancy to improve self-triage. Further studies could look at whether over or under triage occurs with increased redundancy.

Limitations

Limitations to our study include the possibility that patient’s symptoms may have progressed or changed between using the online self-triage tool and contacting nurse triage though the short median time (16 minutes) between triage types would suggest this may not be overly impactful. Additionally, the fact that 71% reported chest pain during their self-triage encounter is supportive that most patients did have the symptom of chest pain at the time of their self-triage encounter even though they did not choose to triage their symptoms with the chest pain self-triage algorithm. Another limitation is that we are cognizant of the fact that using the nurse triage choice of primary symptom as a “gold standard” with which to compare the presence (or absence) of chest pain to may be subject to error as evidenced by two patients denying the presence of chest pain at their follow up encounter. Eleven patients did not have any documented follow-up care in our system but may have been seen in other health systems. However, given they are a small minority of the total, we believe the impact on our findings would be minimal. Additionally, although we included all eligible self-triage/nurse triage dyads that met our study criteria during the study period, our sample size was small which may limit conclusions. Another limitation is that our study was not designed to assess how social drivers of health, digital literacy, health literacy, educational background, or comorbidities (or lack thereof) may have contributed to patient self-triage symptom choices. Finally, explanatory models such as the Health Belief Model suggest that patients may avoid or misreport symptoms due to their belief about their own health.²⁵ For example, how patients perceive their susceptibility to a heart attack might alter how they interact with the symptom checker. Our study design prevents us from being able to assess for this possibility.

Conclusions

Our study illustrates that most patients triaged for chest pain by a telephone triage nurse did not choose the online self-triage algorithm option for chest pain. One potential method to improve self-triage could include diagrams for patients to review and choose the area of their symptoms thus improving the ability of patients to be guided to appropriate algorithm questions. Also, artificial intelligence, already being explored to aid Emergency Department triage,²⁶ could help improve symptom checker accuracy via real-time patient risk assessment and adaptive questioning. Including interface designs that allow the ability to select multiple symptoms might also enhance online self-triage tools. Future studies could look at how these interventions affect patient safety and diagnostic accuracy of online symptom checkers.

Supplemental Material

sj-docx-1-his-10.1177_11786329251385022 – Supplemental material for To Choose or Not to Choose: Patients with Chest Pain Often Do Not Choose to Triage for Chest Pain During Online Self-Triage

Supplemental material, sj-docx-1-his-10.1177_11786329251385022 for To Choose or Not to Choose: Patients with Chest Pain Often Do Not Choose to Triage for Chest Pain During Online Self-Triage by Jennifer L. Pecina, Elizabeth N. Curry, Nathaniel E. Miller, Matthew C. Thompson and Frederick North in Health Services Insights

Footnotes

ORCID iDs

Jennifer L. Pecina

Elizabeth N.Curry

Nathaniel E. Miller

Matthew C. Thompson

Frederick North

Ethical Considerations

This study was deemed exempt by the Mayo Clinic Institutional Review Board (IRB 20-006809).

Consent to Participate

The Mayo Clinic Institutional Review Board waived written consent for patients to participate in this study due to impracticability in obtaining written consent, however all patients included in this study had signed, approved institutional research authorization.

Author Contributions

JLP, ENC: Study concept. JLP, ENC: Study design. MCT: Data collection. JLP, ENC, NEM, MCT, FN: Data analysis and interpretation. JLP, ENC: First draft. JLP, ENC, NEM, MCT, FN: Final draft and editing. JLP, ENC, NEM, MCT, FN: Final review and approval.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

Our study uses an institutional research authorization which does not include authorization to publicize patient level data.

Supplemental Material

Supplemental material for this article is available online.

References

Iserson

Moskop

JC.

Triage in medicine, part I: concept, history, and types. Ann Emerg Med. 2007;49(3):275-281.

North

Jensen

Pecina

, et al Online self-triage of ear or hearing concerns in a patient portal: comparison of subsequent diagnoses and hospitalizations to national emergency department and national ambulatory ear or hearing visits. Health Serv Res Manag Epidemiol. 2023;10:1-8.

Penza

Murray

Myers

Maxson

Furst

Pecina

JL.

Treating pediatric conjunctivitis without an exam: an evaluation of outcomes and antibiotic usage. J Telemed Telec. 2020;26(1-2):73-78.

Penza

Murray

Myers

Furst

Pecina

JL.

Management of acute sinusitis via e-visit. Telemed e-Health. 2021;27(5):532-536.

Galmiche

Rahbe

Fontanet

, et al Implementation of a self-triage web application for suspected COVID-19 and its impact on emergency call centers: observational study. J Med Internet Res. 2020;22(11):e22924.

Judson

Odisho

Neinstein

, et al Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19. J Am Med Inform Assoc. 2020;27(6):860-866.

Haase

Bearman

Brodersen

Hoeyer

Risor

‘You should see a doctor’, said the robot: reflections on a digital diagnostic device in a pandemic age. Scand J Public Health. 2021;49(1):33-36.

North

Jensen

Stroebel

, et al Self-triage use, subsequent healthcare utilization, and diagnoses: a retrospective study of process and clinical outcomes following self-triage and self-scheduling for ear or hearing symptoms. Health Serv Res Manag Epidemiol. 2023;10:1-11.

North

Nelson

Majerus

Thompson

Knutson

Crum

BA.

Self-scheduling process efficiency and utilization of online self-scheduling of lab tests: a retrospective analysis of self-scheduled appointments for COVID testing. Health Serv Res Manag Epidemiol. 2022;9:1-10.

10.

Chambers

Cantrell

Johnson

, et al Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review. BMJ Open. 2019;9(8):e027743.

11.

Semigran

Linder

Gidengil

Mehrotra

Evaluation of symptom checkers for self diagnosis and triage: audit study. Br Med J. 2015;351:h3480.

12.

Hourizadeh

Zeltser

Makaryus

Druz

Real-world evaluation of a chest pain digital triage platform at the point of care. South Med J. 2023;116(11):857-862.

13.

Joury

Alshathri

Alkhunaizi

Jaleesah

Pines

JM.

Internet websites for chest pain symptoms demonstrate highly variable content and quality. Acad Emerg Med. 2016;23(10):1146-1152.

14.

Hsia

Hale

Tabas

JA.

A national study of the prevalence of life-threatening diagnoses in patients with chest pain. JAMA Intern Med. 2016;176(7):1029-1032.

15.

Cayley

Jr.

Diagnosing the cause of chest pain. Am Fam Physician. 2005;72(10):2012-2021.

16.

McConaghy

Sharma

Patel

Acute chest pain in adults: outpatient evaluation. Am Fam Physician. 2020;102(12):721-727.

17.

Safdar

D’Onofrio

Women and chest pain: recognizing the different faces of angina in the emergency department. Yale J Biol Med. 2016;89(2):227-238.

18.

Lenfant

Chest pain of cardiac and noncardiac origin. Metabolism. 2010;59(Suppl. 1):S41-S46.

19.

Content S-TC. Accessed January 7, 2025. https://www.stcc-triage.com/the-guidelines

20.

DeVon

Mirzaei

Zegre-Hemsey

Typical and atypical symptoms of acute coronary syndrome: time to retire the terms?

J Am Heart Assoc. 2020;9(7):e015539.

21.

Curry

Miller

North

Thompson

Pecina

JL.

Congruency of symptom choice between patients performing online self-triage and nurse phone triage. Telemed J E Health. 2025;31(9):1127–1136. doi:10.1089/tmj.2024.0589

22.

Wallace

Chan

Chidambaram

, et al The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. NPJ Dig Med. 2022;5(1):118.

23.

Riboli-Sasco

El-Osta

Alaa

, et al Triage and diagnostic accuracy of online symptom checkers: systematic review. J Med Internet Res. 2023;25:e43803.

24.

Ceney

Tolond

Glowinski

Marks

Swift

Palser

Accuracy of online symptom checkers and the potential impact on service utilisation. PLoS One. 2021;16(7):e0254088.

25.

Champion

Skinner

C. S.

Health Behavior and Health Education: Theories, Research, and Practice. John Wiley & Sons; 2008.

26.

Da’Costa

Teke

Origbo

Osonuga

Egbon

Olawade

DB.

AI-driven triage in emergency departments: a review of benefits, challenges, and future directions. Int J Med Inform. 2025;197:105838.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB