Feasibility study of using mobile application to support triage and diagnosis clinical decisions for pediatricians: User-centered design approach

Abstract

Background

While there is some evidence in the literature demonstrating success in using a triage software application in ED, none of the solution was developed specifically to support a holistic decision of pediatricians in triage and diagnosis purposes to initiate the first treatment properly. To explore the usefulness and possibility of employing a digital-based solution to enhance clinician performance, the mobile application was developed and then assessed in different perspectives.

Objective

The primary objective of this study is to contribute implementation practice of an application to support pediatric triage and diagnoses. The secondary objective is to present the results of the preliminary evaluation of the application.

Methods

The application called Pedicmeter was developed. Formative tests with revisions were applied throughout the development phase. A number of summative extensive evaluations were also conducted to investigate the efficacy of the proposed method. The evaluation focused on measuring the ability of an application to support a pediatric staff’s decision to determine an overall severity level and disease diagnosis. Finally, the user’s (clinician's) satisfaction of using the application was measured.

Results

The application Pedicmeter enables clinicians to make more accurate decisions in determining emergency level of pediatric patients by 6.66%. The application accurately diagnosed a disease with 73.08% accuracy and 66.67% accuracy for respiratory and infectious diseases, respectively. The diagnostic information that the application suggested shows that it does have an influence on a clinician’s diagnosis. Using the app showed improvements in diagnostic accuracy for asthma, croup, sepsis, but it showed a decrease in the accuracy of a clinician's decision for pneumonia. The benefit of the application that satisfies the pediatricians the most is the helpfulness of the features of the application (86%), while the least satisfying factor was the required number of inputs (63%).

Conclusion

The developed application conceptually shows a promising opportunity to enhance clinicians’ decisions from the pilot study. However, the study also reveals further tweaks are required and unveils challenging issues and the concerns of clinician users when use the application. Further research will be conducted to investigate and determine the limiting factors and specific issues revealed by this study. Longitudinal data collection and analysis also need to be conducted to investigate the clinical implications.

Keywords

Mobile application pediatric triage pediatric diagnosis decision making human factors

Introduction

Globally, the 24-hour mortality rate for children admitted to hospitals remains high at 32–66%.^1,2,3 In low-income countries, the under-5 mortality rate was 67 deaths per 1000 live births versus 5 deaths per 1000 live births in high-income countries.⁴ The causes of death vary slightly depending on what classification system is used, but the major causes of death are generally the same. According to Zhu et al.,¹ the top three causes of death are respiratory diseases (14.8%), neuromuscular disorders (14.3%), and cardiovascular disease (12.6%), whereas in Jofiro et al.,² cardiovascular diseases are the leading cause of death (27.8%), followed by respiratory diseases 78 (26.2%), and infectious diseases (25.5%). More specific causes of death were discovered by Lahmini³ who ranked pneumonia and meningitis as the top two causes of death.

The sooner a correct diagnosis is obtained, and the urgency of the patient's condition is established, the sooner appropriate treatment or a resuscitation process can begin. For example, some studies suggested the deaths from respiratory tract infections are readily preventable through timely access to appropriate public health or medical interventions, including vaccination, passive immunization, and antibiotic treatment.^5,6 Similarly, the early recognition of the sepsis and septic shock enabled the early goal-directed therapy to improve the survival rate significantly in pediatric patients.^7,8

Although early screening is crucial in a hospital's Emergency Department (ED), it can be especially difficult to start proper treatments on pediatric patients. The retrospective review from the previous research for common malpractice in ED revealed that one of the most frequent malpractices caused by inaccurate diagnosis was pediatric fever/meningitis.⁹

Many systems and processes for triage have been proposed in the literature and have shown promising results compared to paper and memory-based triage processes. For example, reducing disagreements among nurses¹⁰; increasing accuracy¹¹; and increasing confidence in triage difficult and rare situations.¹² However, viable solutions for pediatric groups are still limited. This paper presents the findings of research into a new application.

A systematic review of mobile triage applications recently found that majority of application implied to applicable to general ages.¹³ In addition, the literature on triage application in pediatric patients has largely focused on solutions for specific diseases or looking at specific purposes for triage such as abdominal pain investigation,¹⁴ sepsis-focused triage,^15,16 or in telehealth setting used by non-medical triagers.¹⁷

Because the proposal of using a digital tool that comprehensively includes triage and diagnosis information to support pediatricians is a relatively new approach, this research focuses on conducting a feasibility study to investigate the effectiveness and feasibility of the approach in a clinical setting. It contributes to the literature by presenting implementation details from a user-centered design (UCD) approach, as well as the results of an evaluation at Maharaj Nakorn Hospital, a teaching hospital affiliated with Chiang Mai University in Thailand.

Methods

The nature of this study is empirical research to investigate the impact of providing pediatricians with a new approach to triage and diagnosis in a teaching hospital setting. The study was conducted at the pediatric Chest and ICU unit, department of pediatrics, Chiang Mai University, also known as Maharaj Nakorn Hospital, Chiang Mai between July 2019 and March 2021. The study was conducted in two major phases which are the software development phase and the evaluation phase.

During the software development phase, several research approaches were combined to create a workable solution. For example, archival research was carried out in order to identify and extract medical knowledge about triage and diagnosis from medical guidelines. In order to synchronize the solution with clinic practice, additional survey research was conducted with the expert to fine-tune the results of the archival research.

During the evaluation phase, experimental research was conducted to investigate the impact of the proposed solution's effectiveness, especially how the application influences pediatricians’ decisions. To assess pediatricians’ technology adoption, survey research was undertaken to explore user satisfaction in both quantitative and qualitative aspects.

Developmental phase

The application entitled “Pedicmeter” operated using an Android operating system platform, which was developed. The implementation focuses on a report by Elmasllari and Reiners,¹⁸ who reports lesson learnt from the failures found in many e-triage assessment that lack involvement from the healthcare staffs. This research, on the other hand, maintained a close collaboration between the software engineering team and the clinician team throughout the entire project. Therefore, the software implementation of this project adheres to the engineering paradigm known as UCD, which defines an approach in which end-users have a significant impact on the shape of a design. The purpose of the UCD methodology is to ensure that the developed software is consistent with clinical practices. In this phase of the study, the research involved defining the implementation's requirements and scope, as well as designing and developing software.

Defining requirements and scope of the implementation

At the early stage of the application implementation for this research, requirements are elicited from the pediatrician users, clear scope of the application is defined, and what factors pediatrician users considered to determine the level of emergency from patient's conditions and diagnosis are investigated.

The requirement study is the first activity of this research; the requirements were extracted from a staff member at the Department of Pediatrics through meetings. The ultimate goal of the project is to develop a triage system that can enhance pediatric triage residents in ED settings so that proper treatment can start earlier.

As “Pedicmeter” has an objective to provide a function as an e-triage tool, the key activity and procedure should follow standard triage processes stringently. The Department of Pediatrics, Chiang Mai University uses a pediatrics’ early warning score (PEWS) as routine practice. At the Requirement Elicitation phase, PEWS are reassessed by clinician target users. When interviewed, they voiced that one of the significant difficulties is that the approach presently being used relies on the subjective justification of the triagers. For this research, it was decided that the application should adopt the modified version of PEWS, which is entitled the modified pediatric early warning score (MPEWS) instead. The MPEWS rely more on objective factors, such as heart rate, respiratory rate, and body temperature. These biometric measures are standard healthcare practice and are easy for the triager to evaluate.¹⁹ Using the MPEW score alone is theoretically possible, but pediatricians will face another significant challenge in memorizing the standard range of all vital signs criteria, which vary by patient age, and pediatricians must determine the severity score associated with those vital signs measured.

Another important requirement had been raised by the clinician who has supervisory role in pediatrician training. The requirement is that when giving a diagnostic suggestion, the system needs to present all possible answer possibilities to the triagers. Triagers need to decide on a final and definitive diagnosis by themselves; they may or may not act on the suggestion.

Setting up a clear scope of which disease diagnosis to include in the application—is another important task that the project predefined at the early stage of the project. A disease selection based on the prevalence, and medical records review has indicated that respiratory-related and infectious diseases were the major cause of hospitalization by accounting for 43% of the total cases at the Department of Pediatrics, Chiang Mai University. So, four important diseases which are asthma, croup, pneumonia, and sepsis are selected to focus. Those four have standard medical guidelines for investigation. Details of diseases and their standard diagnosis guidelines are presented in Table 1.

Table 1.

Disease and standard diagnosis guidelines.

Diseases	Standard guideline
Asthma	PRAM score²⁰
Croup	Downes and Raphaely croup score²¹
Pneumonia	WHO classification of childhood pneumonia 2014²²
Sepsis	Pediatric sepsis trigger tool²³

PRAM: preschool respiratory assessment measure.

It was concluded that the MPEW scores should be used to determine what severity level pediatric patients are in, and a multiple disease screening metric will be used to determine what disease patients are likely to have.

In order to enable an application to provide triage and diagnosis suggestions, a set of inputs from the medical investigation are required to proceed to generate decisional output. At this stage, it is necessary to investigate the required inputs, what logical rules clinicians associate when making diagnostic decisions as well as the output characteristics and their interpretation(s). That information is defined and can be extracted from the diagnosis guidelines listed in Table 1.

It was found there are mainly three categories of required inputs. These are vital signs, scoring based on the investigation, and specific signs that might signify a symptom. For the outputs, there are two major possibilities, which are the score-based results and Boolean results. The main inputs and output characteristics are outlined in Table 2.

Table 2.

Inputs and output from diagnostics guidelines.

Guidelines	Major inputs	Output
MPEW	Vital signs: Score based on O₂ therapy Score based on Level of consciousness score	Range of 0–4 and above to representing level of severity
Asthma	Score based on retraction and contraction Score based on lung signs Score based on O2Sat	≥9 Severe asthma ≥5 Moderate asthma <5 Mild asthma
Sepsis	Vital signs Specific sign such as suspected infectious, pulse, capillary refill, risk factors	True/Falsa logical Condition for Suspected sepsis Suspected septic shock
Pneumonia	Vital signs Score based on retraction and contraction Specific signs: suspected infection, nasal flaring, grunting	True/Falsa logical Condition for Suspected sepsis Suspected septic shock
Croup	Specific signs such as suspected infection, nasal flaring, stridor	True/Falsa logical Condition for Suspected croup

Software design and implementation

The details of how two major challenges in the application development process are addressed are elaborated here. These are (a) how the application generates a decision output, and (b) how the user interface (UI) and interaction design process is designed.

Formulating decision outputs

The Pedicmeter application uses a set of inputs that triagers supply to enter decision modules to compute for triage and diagnostic decisions. The decision modules are formulated from the medical guidelines presented in Table 1. There are two major approaches to determine outputs that the application adopts. The first approach is using an arithmetical approach in calculation. In this approach, the module interprets each input to a numeric score. The summation of scores for every related factor is a total score to determine the severity or how likely a patient is present with a specific disease. The outputs that take an arithmetic approach are the calculation of MPEW to determine overall severity and the preschool respiratory assessment measure (PRAM) score to diagnose asthma. MPEW uses five numeric factors, which are heart rate by age; respiratory rate per minute by age; body temperature; oxygen saturation; oxygen therapy (L/minute); and consciousness level. Possible score ranges to indicate the severity in each factor are 0, 1, 2, and 4. The PRAM score is a score-based metric used for detecting signs of asthma. Five important signs that this measurement focus are: suprasternal retractions, scalene muscle contraction, air entry, wheezing sound, and SpO₂. The score range for each factor is from 0 to 3, where the magnitude of each factor is based on the presentation of symptoms or physical examination. The total score indicates the level of severity patient attributes to asthma. A score greater than or equal to 9, for example, indicates a sign of severe asthma.

Another approach to determine decision outputs is using a logical approach. The disease diagnosis guidelines employing this approach include sepsis, pneumonia, and croup screening. In this approach, the signs and symptoms presented are monitored. All related signs and symptoms indicated by the standard guidelines are checked, and the data is used to formulate a logical expression, which in turn indicates the approximate disease stage. If there are indications of a specific disease, and there is a need for examination, it will prompt a suggested diagnosis based on that condition. Typically, several indicators are needed to formulate a compound logical expression, and in some cases, there are several possible expressions to determine a disease stage. An example of a suspected condition, such as pneumonia, is shown in an expression (1).

Nasal flaring AND (Subcostal retraction OR Suprasternal retraction) AND (heart rate AND respiratory rate) AND suspect infection (1)

UI design

Although the triage criteria used in this research were extracted from multiple standard guidelines, they could not be simply taken and applied to the Pedicmeter application straight away because some criteria were redundant which needed to be removed or grouped them together. Also, with the interest in making the design applicable to the healthcare operation time of Maharaj Nakorn Chiang Mai Hospital, the sequence of triage should be considered. Information architecture (IA) is the art of organizing and labeling an information space for optimal understanding and use. This research captured all triage parameters from the MPEWS, and specific disease screening guideline mentioned in Table 1, then using a card sorting technique to find the order of triage; consolidating parameter; categorizing triage parameters; and seek out a label that exactly match user’s understandings.

The card sorting was conducted with four pediatric residents through a mini workshop. Before the workshop, cards of parameters were prepared and shuffled, and at the workshop, the residents are asked to group the parameters together and give a title for each category. It took three iterations of sorting before participant could gain consensus on the card organization. Cards were arranged based on the sequence of routine diagnosis, which generated four kinds of diagnosis categories which are: initial impression; risk factors; vital signs; and physical examination. Figure 1 shows the card sorting activity conducted for this project.

Figure 1.

Card sorting activity.

The output from the IA procedure was then applied to the UI design. Each diagnosis category represents a step of determining diagnosis, containing diagnosis item limits with approximately 10 questions to each UI to reduce the burden of a user’s cognitive workload. A user, as a triager, is asked to evaluate the condition of a patient, following a checklist, that is, whether a specific symptom is present. The application design maintains consistency of the control actions required by presenting a toggled switch of “yes/no” for each question. On the diagnosis result page, which is in the last step, the system output two important information the MPEW score to show the overall severity and presumptive diagnosis with the possible following justification: suspected sepsis/suspected septic shock/suspected mild asthma/suspected moderate asthma/suspected severe asthma/suspected pneumonia/suspected moderate pneumonia/severe pneumonia and suspected croup. The UI of the application is illustrated in Figure 2.

Figure 2.

User interface of the application.

Evaluation phase

Two kinds of usability evaluation are employed. Formative evaluations are implemented throughout the development phases to detect and fix any problems before the application is released. Summative evaluation is conducted to evaluate the performance of the system after implementation. There were several summative evaluations conducted in this research, each of the evaluation has a specific testing objective and detailed as follows:

Beta version test

The first release of Pedicmeter was tested by six senior pediatricians who have at least five years of pediatric experience. The objectives of this evaluation were twofold: (a) to evaluate the overview experience and to test the UI of the application and (b) to focus the test on the novel concept of the disease identification feature. Six experienced pediatricians, which this research as medical experts, were given a mobile phone with the application. The research team gave a quick introduction about the application, but no tutorial was offered to them. The experts were required to use the application to assist in diagnosis. For the 10 cases that had been prepared, they were encouraged to “think aloud” when using the application.

From this beta version test, some UI issues were detected, such as an unclear system status; visibility and conspicuity issues of some design items; and the speed of response.

The test found that most of the experts can use an application to finish the evaluation. However, three significant UI problems were detected: (a) many of the participants could not use the slider UI, where the user was required to use his/her finger to press and drag the status to supply the progressive input smoothly. Later we changed the slider to a series of buttons, where the user was required to use a simpler click to address the problem. (b) The interface contained technical abbreviations to indicate the level of consciousness of a patient. Some participants (users) were asked to repeat the definition of each choice, and later we added a short definition to give extra information on the screen and (c) the original UI for input of patient's age used a dropdown menu to indicate the age, where the default shows the unit as “Year.” Some users accidentally missed the unit of “month” for baby patients, causing the system to incorrectly compute the severity score. Later we made the option more prominent using a radio button.

One substantial recommendation to improve the accuracy of the system is for pneumonia diagnosis. The original guideline suggests observing the sign of “Nasal flaring.” However, this observation is never performed to inspect patients in many clinicians. So, one of the medical experts recommended inspecting the lung sign for crepitation instead.

Based on the feedback for the beta version evaluation, the enhancements were made to the Pedicmeter application with a view to releasing the revised version. The revised version was then tested for usability by residents and interns. With the limitation of conducting the test in a real hospital environment where all the stakeholders and evaluators are busy with their routine duties, it is difficult to allocate a long session to evaluate the application in its entirety in one session. The evaluation was therefore split into three sessions over three days. Each session was limited to 1.30 hour. Clear and explicit evaluation purposes were determined for each evaluation.

Evaluation of diagnosis feature

This evaluation has the objective of testing the disease identification feature of the proposed application, as far as we know, the integration of diagnosis to the triage application is relatively new. It is also the first time that a multiple disease screening metric has been considered simultaneously. Verification is required to see whether the approach is feasible to be used to support clinician’s decision. This evaluation is broken down into stages: (a) validity test and (b) test for efficiency in decision support.

Validity test

The validity test is a test that validates the performance of the disease determination feature of the application. The test was made by using an application to determine a disease from historical medical records. The steps of the validity test are as follows:

Retrieve cases from retrospective data from Pediatric Outpatients’ Medical Records within 12 months of patient discharge to make a pool of data sets for the test.

To prevent bias, only one nurse used the application to diagnose all the cases in the data sets in 1) above. Only the application was used to determine a disease by input-given data.

Record a list of all possible diseases that the application might suggest.

A pediatrician reviews and checks the application's decisions for diagnosis accuracy.

Influence to pediatrician’s decision-making

The purpose of this evaluation is to determine how the information that clinicians are exposed to impacts their clinical decision-making. This evaluation follows this experimental protocol:

Prepare the questions, in a format similar to an exam paper, where each question gives a list of diagnostic factors that could lead to a diagnosis for the four diseases included in this study. There are 4 cases for each disease, with a total of 16 cases. Two copies of question sets were prepared: Set A and Set B.

Triagers complete the Set A test. They are asked to justify a disease from the given information purely based on their knowledge and expertise. Then submit the answer.

Next, they take a turn to complete the Set B test. This time, triagers are asked to use the Pedicmeter to support their triage decision. The application is likely to provide presumptive diseases from the given information. Triager may or may not decide to follow the guide answers. They then give their final answer.

After completing both tests, the accuracy of diseases’ identifications are compared.

Evaluation of triage feature

This evaluation mainly focuses on how effective the Pedicmeter can support a decision in determining a level of urgency in triage. The following protocol is used for this evaluation.

Prepare the questions, in a format like an exam paper, the questions are devised to determine the urgency level of hypothetical pediatric patients from the given information extracted from real medical records. There are 15 questions shuffled according to different levels of urgency. Level 7 questions are “severe-condition cases,” and eight questions are “moderate-condition cases.” Two copies of the question sets are prepared: Set A and Set B.

Triagers complete the Set A test. They are asked to determine the level of emergency, which is classified according to three levels: “Mild moderate” and “Severe” based on their knowledge and expertise. They submit their answers.

Next, they take a turn to complete the Set B test. This time, triagers are asked to use the Pedicmeter to support their triage process.

After completing both tests, the accuracies of the disease identifications are compared.

Evaluation of user satisfaction

This evaluation measures the general satisfaction of users and gathers feedback opinions on the perceived effectiveness after a user has had experience with the application.

Organize a public engagement event to introduce the application Pedicmeter at Maharaj Nakorn Chiang Mai Hospital. Pediatric staff are invited to join a half-day event and are encouraged to download a trial version of the application to evaluate the application.

The questionnaires with the five statements below are distributed to staff who have experience with the application. The level of agreement for each aspect is measured using 5-likert scale. A space for extra comments is also provided in the questionnaire.

“ I think the features of this application are helpful.”

“I think the procedure of using this application in a clinical setting makes sense.”

“The application helps me gain more confidence in diagnosis.”

“The input required for the application is just the right amount.”

“I like the overall appearances of the User Interface.”

Analyzing data collected from the questionnaire.

Results

The research explores three important aspects of triage, which are the efficiency of the triage feature to the decision-making: validity of the diagnosis feature; impact on decision-making; and overall user satisfaction. The results are presented below.

Efficiency of triage feature to the decision-making

Five residents participated in the evaluation session. It was found that with the supplement of MPEW scores information from the mobile application, pediatrician trainees could make triage decision more accurately when determining the emergency level of pediatric patients by approximately 6.66%. The median of the pretest was 9 (8–10.5) from 15 or equivalent to 60%, while the median of the post-test was 10 (8.5–12.5) from 15 or equivalent to 66.66%. The result of pre- and post-supplement of digital MPEW from five participants is presented in Figure 3.

Figure 3.

Efficacy of the application when being used to facilitate emergency level.

Validity of the diagnosis feature

The case retrieval process was carried out using archived medical records which had not been categorized by disease. The clinical member of the research team was then able to categorize them according to disease for the purposes of the research.

Six major groups of diseases were found, which are: respiratory diseases; cardiovascular diseases; gastrointestinal diseases; neurologic diseases; infectious diseases; and a miscellaneous group. The study extracted the cases that were categorized in respiratory diseases and infectious diseases groups (n = 88). Of the 88 participants, 56 patients were in the respiratory group and 32 patients were in the infectious group. We conducted a retrospective test by using an application to predict a disease from the known information. It was found that the Pedicmeter is able to give a correct diagnosis for respiratory diseases 73.08% of the time and 66.67% for infectious diseases. The results are shown in Table 3. It should be noted that the sample sizes of infectious diseases are relatively smaller than the respiratory groups.

Table 3.

Accuracy of the pedicmeter application for diseases diagnosis.

Disease group	Total cases (n = 88)	Cases detected from APP (n = 64)	Correct (n = 46)	Incorrect (n = 18)	Accuracy of cases detection (%)
Respiratory diseases	56	52	38	14	73.08
Infectious diseases	32	12	8	4	66.67
Total	88	64	46	18	71.88

Impact to diagnosis decisions

This study investigates whether the provision of a second opinion influences the outcome of a decision. The Content Validity Index (CVI) was used to measure the decision agreement to disease diagnosis by residents. Five residents attended this evaluation. Each of them gave a diagnosis of the written scenario twice by following the experimental protocol 1B. From Table 4, it can be seen that after supplying the second opinion information, the residents had stronger corroboration to the correct diagnosis in most diseases. Exceptional results were found in the pneumonia diagnosis. When supplying the second opinion, residents showed less agreement to the correct diagnosis.

Table 4.

The efficiency of the application by comparing performance.

Dx	Set A (pre)	Set B (post)
Asthma	0.866	0.966
Pneumonia	0.666667	0.5
Croup	0.666667	0.708333
Sepsis	0.666667	0.75
CVI	0.7165	0.731083

User satisfaction

There were 38 participants (3 pediatricians, 12 pediatric residents, 7 interns, 6 externs, and 10 nurses) who returned the questionnaire. The satisfaction results that were in 5-scale Likert scale were consolidated. The given score from the participants was interpreted and recorded as “agree,” Those who gave a score of 1–3 were grouped and interpreted as “disagree.” The results from the questionnaires are illustrated in Table 5. It was found that the participants who gave the higher scores of agreements as “helpful” aspects of the application and the least agreement of the system was onto the amount of the required inputs.

Table 5.

Level of satisfaction of the application by the participants.

Questions	Percent
Questions	Agree	Disagree
I think the features of this application are helpful	86%	14%
I think the use of this application in this clinical setting is make sense	68%	32%
The application helps me gain more confidence in diagnosis	69%	31%
The input required for the application was just the right amount	63%	37%
I like the overall appearances of the user interface	71%	29%

The questionnaires provided an opportunity for participants to leave their comments. There were two substantial challenges indicated by the participants to further improve the Pedicmeter application. Can the number of required inputs be further reduced? and Can the application advise suitable treatment and healthcare protocol based on the triage and diagnosis results found on the application.

Discussion

The results of this study highlight some effectiveness of the Pedicmeter application that has two key features in providing triage and diagnosis advice to support a pediatrician’s decision-making. The approach sheds some light on promoting the solution to use in mainstream healthcare settings. However, the implementation has also revealed challenges that require further revision.

In order to serve the triage purposes, the application introduced the digital MPEW, which was developed from the conventional MPEW. The application thus relieves the need for triagers to memorize the vital signs standard which is in age-specific, when determining patient’s severity. For example, they are not required to remember whether “Is the respiratory rate of xx normal for this three year’s old boy.”

Moreover, the Pedicmeter application assists triagers to assess the patient in a wholistic manner step-by-step and avoids subjective justifications, resulting in more accurate triage outcomes.

However, the study found some constraints that limit the clinicians utilizing this triage information in clinical settings effectively. First. the scale of MPEW may be too broad in range. For example, the MPEW score of 4 and 14 are both for emergency and are symbolized by the color “red.” In clinical practice, patients who receive an assessment score of 4 and 14 have significantly different levels of severity. This implies clinicians seem to need a finer scale of severity classification when they justify a patient's conditions.

Another issue is that there has not yet been standardized treatments and care protocol defined for each score. When the score is obtained, there are no clear guidelines for the required healthcare operational action or a standard treatment protocol to enforce. This healthcare setting can vary the practice of care and treatment for different clinicians.

In order to support the diagnosis purpose, it was shown from the result of the study that the Pedicmeter solution is a feasible option for early identification of diseases. The application predicts correct diagnosis more than 70% of the time.

Another potential advantage of using a computer-based solution in triage is that clinical biases can be reduced, since in a real-world setting, clinician's decision-making is sometimes influenced by non-medical influence.²⁴

Focusing on the screening of each specific disease, infectious diseases showed the lowest accuracy from the disease diagnosis feature of the Pedicmeter application because there are a great variety of diseases in this category. These include dengue fever; hemorrhagic fever; infectious diarrhea; others bacterial infections; and sepsis. This research focuses on only a small section of the infectious diseases category.

In the respiratory diseases category, we identified two diseases that are also common but outside the scope of this research. These are the viral-induced wheezing and the bronchiolitis. When we investigated clinician practices from clinicians who acted on the application’s advice, we found that clinicians are selectively acting on the advice. When a group of pediatricians are provided with the same explicit medical suggestion, they are likely to have a similar diagnosis. For example, for the asthma, croup and sepsis diagnoses, the research result indicates that clinicians have a higher CVI after receiving the output from the application. However, there was also one situation where a second opinion from the application caused diagnosis disagreement between clinician’s shown by lower down the CVI in the case of Pneumonia diagnosis. This could have been due to a complexity of symptoms resulting diagnosis could not be determined easily by the standard Pneumonia diagnosis guideline. This finding also suggests that the knowledge and experience of pediatricians are still necessary in disease diagnosis. The only implication shown from the result of this study is “with highly accurate information supplied, medical staff will make better decisions.”

The mechanism underpinning the disease diagnosis feature of the application relies on checking whether specific signs or symptoms defined in a disease screening guideline are presented; there was also the case that two or more different diseases, that cause patient having similar symptom, for example, pneumonia and asthma, use the same indicators. This means that the application is unable to pinpoint a disease prediction for similar diseases. This led to the interesting notice that in medical practice some tacit knowledge to differentiate two similar diseases is still in need. Identifying the accurate key indicator for a certain disease accurately will lead to early and accurate diagnosis.

After reviewing the performance and usefulness of disease diagnosis feature in clinical settings, a major issue that the team members of this research are reconsidering is how significantly the diagnostic feature contributes to medical treatment. The trade-off to make this feature perform accurately is to request users supply more inputs. In turn, these can reduce the motivation of triager when using an application especially when considering that the application is likely to be used in ED that the time-to-decision is the major requirement. Another issue is, there are far more the number of disease screening guidelines. It is impractical to include them all and difficult to justify which one deserves to be included in the application to make the application have a wider reasonable scope. The research team is considering an alternative option to output this information in a reduced “resolution” way, for example, output a disease category not a precise disease diagnosis if this information is adequate for initial treatment and care.

Focusing the discussion on the overall user satisfaction of participants is investigated. Participants mostly agree that having an application to supply information is helpful as indicated by the result from Question 1. However, when taking a closer look at the possible technical advantages of the system at “this procedure is making sense (Question 2)” and whether the application “lift- up confidence to individual (Question 3),” it does not seem unanimous agree to the usefulness score. A possible cause of disagreement could be the relatively low score of agreement on the number of inputs required to the system (Question 4) which was also commented explicitly by one participant. This issue has been reviewed by revisiting the literature and found that “too much information” was the common complaint of the Clinical Decision Support Systems.¹² Another known suggestion was related to the system need to provide explicit interpretation from the result. From the result in this section, further investigation should be required to confirm the requirements and identify other aspects of the application that impact user’s satisfaction.

The high score in evaluating the level of satisfaction of the application from user implies the effectiveness of e-triage and e-diagnosis in comparison to the memory-driven paper-based approach. The result is in line with the findings of Dong¹⁰ who found that the key success of the e-triage/diagnosis system is due to the approach solving the vital pain-points in clinical practice: minimizing discrepancy of clinical decision in large group clinicians due to the variation in skill level and minimizing a decisional bias from over-influencing the factor associated to the current state of occupancy in the ED.

This study presents the findings from the initial implementation of e-triage/diagnosis. The study discovered significant issues during the pilot implementation in clinical practice. From the standpoint of software engineering, the revision to improve the performance of the application from this version is required and ongoing. The ultimate goal is to deliver an application that contributes accurate clinical decisions while also being easy to use for clinicians.

Upon completion of the revision, not only will this pediatric e-triage/diagnosis be applicable for use at the pediatric chest and ICU unit of the Maharaj Nakorn Hospital in Chiang Mai, as was originally intended, but it will also be applicable for telemedicine. The significant advantage is that healthcare providers who do not have a full-time pediatrician on staff can use the application for screening, thereby reducing unnecessary referrals. The discussion of e-triage and telemedicine which is termed as tele-triage in different setting is well outlined in the study of Minervini.²⁵

Research limitations

The limitations of this research occurred in two important research activities. In the development phase Pedicmeter was initially implemented for the Android operating system; this limitation sometimes incurred usability issue to iOS-users who had a different experience when navigating the application (as mentioned in the usability evaluation section). Another limitation is the process of evaluation. The evaluation was intended to prove the viability of the method when the application was used by clinicians. The retrospective case evaluation has the advantage to frame the scope of the test by preparing the case that was only relevant to the application to test and when the exact triage was known. However, this method was highly time-consuming for evaluators. It was not possible to allocate continued and long sessions of evaluation because all pediatricians who participated in the evaluation session were engaging with medical routine services in the hospital. Another issue that may affect the time required to complete the evaluation was the fact that cases were prepared in an exam-similar format. An evaluator who is a medical resident may feel pressured to get “the answer corrects.” Therefore, they might spend a longer time to review and evaluate the given information.

Conclusions

The paper discusses the development and evaluation of the application “Pedicmeter” to support pediatricians to perform better triage. This application has two important features, which are supplying MPEWS to indicate severity levels and guiding disease identification. The research results showed that both features of the pedicmeter application deliver fairly promising accuracy.

To the viability study whether the information generated from the application impacts pediatricians’ decision or not, the study provides the implication that clinicians selectively intake information and ultimately use their own justification. Also, the more accurate the application, the more likely medical staff will make better decisions.

Nevertheless, the study revealed that the application has some significant issues that need to be enhanced. Modifications to the application will need to be made, and evaluations will need to be carried out with a larger number of participants.

Footnotes

Acknowledgements

We would like to thank all pediatric nurses and all members of pediatric department of Chiang Mai University for participating in this research.

Contributorship

NC and SK researched literature and conceived the study. NC took fully responsible for mobile application development, while SK facilitates and coordinates medical stakeholder in the requirement formulation. NC plans and designs the research and data acquisition and wrote the first draft of the manuscript. SK was involved in gaining ethical approval, participant recruitment and data analysis. NC wrote the first draft of the manuscript. SK reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The ethics committee of faculty of medicine, Chiang Mai university approved this study (REC number: AF/04-021/05.0). Patient informed Consent is a not requirement of this research. Triage and diagnostic information used during the retrospective study are pre-existing medical records, none of data was newly collected for this research.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Research Council of Thailand via Faculty of Public health, CMU (Spearhead) [grant number 2562/].

Guarantor

Noppon Choosri.

ORCID iD

Noppon Choosri

References

Zhu

Liang

, et al. The mortality of patients in a pediatric emergency department at a tertiary medical center in China: an observational study. World J Emerg Med 2015; 6(3): 212-216. DOI: 10.5847/wjem.j.1920-8642.2015.03.009.

Jofiro

Jemal

Beza

, et al. Prevalence and associated factors of pediatric emergency mortality at Tikur Anbessa specialized tertiary hospital: a 5 year retrospective case review study. BMC Pediatric 2018; 18(1): 316. DOI: 10.1186/s12887-018-1287-4.

Lahmini

Bourrous

. Mortality at the pediatric emergency unit of the Mohammed VI teaching hospital of Marrakech. BMC Emerg Med 2020; 57(20): 1-6. DOI: 10.1186/s12873-020-00352-9.

WHO. World health statistics: monitoring health for the SDGs, sustainable development goals. Geneva, Switzerland, 2023.

Hardelid

Dattani

Cortina-Borja

, et al. Contribution of respiratory tract infections to child deaths: a data linkage study. BMC Public Health 2014; 1191(14): 1-10. DOI: 10.1186/1471-2458-14-1191.

Verfürden

Fitzpatrick

Holder

, et al. Deprivation and mortality related to pediatric respiratory tract infection: a cohort study in 3 high-income jurisdictions. CMAJ Open 2020; 8(2): 273-281. DOI: 10.9778/cmajo.20190074.

Han

Carcillo

Dragotta

, et al. Early reversal of pediatric-neonatal septic shock by community physicians is associated with improved outcome. Pediatrics 2003; 112(4): 793-799. DOI: 10.1542/peds.112.4.793.

De Oliveira

DSF

Gottschald

AFC

, et al. ACCM/PALS haemodynamic support guidelines for paediatric septic shock: an outcomes comparison with and without monitoring central venous oxygen saturation. Intensive Care Med 2008; 34: 1065-1075. DOI: 10.1007/s00134-008-1085-9.

Ferguson

Geralds

Petrey

, et al. Malpractice in emergency medicine—a review of risk and mitigation practices for the emergency medicine provider. J Emerg Med 2018; 55(5): 659-665. DOI: 10.1016/j.jemermed.2018.06.035.

10.

Dong

Bullard

Meurer

, et al. Emergency triage: comparing a novel computer triage program with standard triage. Acad Emerg Med 2005; 12(6): 502-507. DOI: 10.1197/j.aem.2005.01.005.

11.

Savatmongkorngul

Yuksen

Suwattanasilp

, et al.

Is a mobile emergency severity index (ESI) triage better than the paper ESI?

Intern Emerg Med 2017; 12: 1273–1277 . DOI: 10.1007/s11739-016-1572-x.

12.

Holmström

Kaminsky

Lindberg

, et al. Registered nurses’ experiences of using a clinical decision support system for triage of emergency calls: a qualitative interview study. J Adv Nurs 2020; 76: 3104-3112. DOI: 10.1111/jan.14542.

13.

Montano

de la Torre Díez

López-Izquierdo

, et al. Mobile triage applications: a systematic review in literature and play store. J Med Syst 2021; 45: 86-97. DOI: 10.1007/s10916-021-01763-2.

14.

Farion

Michalowski

Rubin

, et al. Prospective evaluation of the MET-AP system providing triage plans for acute pediatric abdominal pain. Int J Med Inform 2008; 77(3): 208-218 . DOI: 10.1016/j.ijmedinf.2007.01.004.

15.

Mawji

Komugisha

, et al. Smart triage: triage and management of sepsis in children using the point-of-care pediatric rapid sepsis trigger (PRST) tool. BMC Health Serv Res 2020; 20: 493-506. DOI: 10.1186/s12913-020-05344-w.

16.

Lee

Dunsmuir

Businge

, et al. Evaluation of a digital triage platform in Uganda: a quality improvement initiative to reduce the time to antibiotic administration. PLoS One 2020; 15(10): 1-15. DOI: 10.1371/journal.pone.0240092.

17.

Katayama

Kiyohara

Hirose

, et al. A mobile app for self-triage for pediatric emergency patients in Japan: 4 year descriptive epidemiological study. JMIR Pediatric and Parenting 2021; 4(2): 1-15. DOI: 10.2196/27581.

18.

Elmasllari

Reiners

. Learning from non-acceptance: design dimensions for user acceptance of e-triage systems. In: proceedings of the international ISCRAM conference, 2017.

19.

Vredebregt

Moll

Smit

, et al. Recognizing critically ill children with a modified pediatric early warning score at the emergency department, a feasibility study. Eur J Pediatr 2019; 178: 229-234. DOI: 10.1007/s00431-018-3285-9.

20.

Ducharme

Chalut

Plotnick

, et al. The pediatric respiratory assessment measure: a valid clinical score for assessing acute asthma severity from toddlers to teenagers. J Pediatr 2008; 152(4): 229-234. DOI: 10.1016/j.jpeds.2007.08.034.

21.

Downes

Raphaely

. Pediatric intensive care. J Am Soc Anesthesiol 1975; 43: 238–250.

22.

WHO/UNICEF. Revised WHO classification and treatment of pneumonia in children at health facilities: evidence summaries.

23.

Davis

Carcillo

Aneja

, et al. American College of Critical Care Medicine clinical practice parameters for hemodynamic support of pediatric and neonatal septic shock. Crit Care Med 2017; 18(9): 884-890. DOI: 10.1097/CCM.0000000000002425.

24.

Hajjaj

Salek

Basra

MKA

, et al. Non-clinical influences on clinical decision-making: a major challenge to evidence-based practice. J R Soc Med 2010; 103(5): 178-187. DOI: 10.1258/jrsm.2010.100104.

25.

Minervini

Russo

Herford

, et al. Teledentistry in the management of patients with dental and temporomandibular disorders. BioMed Res Int; 2022: 1-7. DOI: 10.1155/2022/7091153.