Abstract
Objective:
In this survey, a protocol-based Chemotherapy Prescription Decision Support System (CPDSS) was designed and evaluated to reduce medication errors in the chemotherapy process of children with ALL.
Methods:
The CPDSS algorithm was extracted by the software development team based on the protocol used by doctors to treat children with ALL. The ASP.Net MVC and SQL Server 2016 programming languages were used to develop the system. A 3-step evaluation (technical, retrospective, and user satisfaction) was performed on CPDSS designed at 2 children’s hospitals in Tehran. The data were analyzed using descriptive statistics. At the technical evaluation step, users provided recommendations included in the system.
Results:
In the retrospective CPDSS evaluation step, 1281 prescribed doses of the drugs related to 30 patients were entered into the system. CPDSS detected 735 cases of protocol deviations and 57 (95%, CI = 1.25-2.55) errors in prescribed chemotherapy for children with ALL. In the user satisfaction evaluation, the users approved two dimensions of the user interface and functionality of the system.
Conclusions:
With the provision of alerts, the CPDSS can help increase compliance with chemotherapy protocols and decrease the chemotherapy prescribing errors that can improve patient safety.
Keywords
Background
Leukemia is a type of malignancy that begins in hematopoietic tissues. This type of cancer accounts for about 8% of all human cancers and is recognized as the fifth most common cancer in the world. Acute Lymphoblastic Leukemia (ALL) is the most common type of cancer among children aged 1 to 16 years, and its peak incidence is in children aged 3 to 7 years.1-5 The treatment of pediatric ALL is one of the most outstanding achievements achieved in cancer science.6,7 Progress in the understanding and treatment of the disease in the 1990s led to approximately 90% improvement in the survival rate of pediatric ALL patients.5,8 The primary treatment for children with ALL is chemotherapy that lasts 2 to 2.5 years. They receive chemotherapy based on complex chemotherapy protocols. 9 The complexity of the chemotherapy protocols in children makes it vulnerable to errors. The complexity of the protocol, coupled with the potential for interrupting the implementation of the protocols (due to unstable physiological conditions), increases the probability and effect of errors and generates more critical complications in a vulnerable population (ie, Children).10,11
Chemotherapy errors are a significant concern for healthcare systems. Chemotherapy, as a specialized medication process, includes the risk of errors in the calculation of the drug dose in children, the risks associated with narrow therapeutic indexes, and the high potential for acute and cumulative toxicity of chemotherapy agents.10,12,13 These errors can occur at any stage of the chemotherapy process (prescription, preparation, dispensing, and administration), especially in the prescribing stage.14-17 In one comprehensive pediatric oncology medication error article, 74% prescription errors, 13.5% administration errors, 9.4% preparation errors, and 3.1% transcription errors were reported. 18 More significantly, the risk of errors with the potential for harm is 3 times higher for children, especially those receiving chemotherapy. 19 Institute of Medicine (IOM) has suggested using evidence to guide the best practice and minimize variation to achieve the goal of providing safe care for pediatric patients. Part of this evidence is protocols and therapeutic guidelines. 20 Although chemotherapy protocols are the best evidence for cancer treatment, compliance with these protocols is low. 21 One of the reasons for this low compliance with protocols is that oncologists need to keep track of many multi-dimensional and complex paper-based protocols (with limited user-friendliness) in inherently long-term chemotherapy process and to adjust them to each specific patient who appears to be difficult for them and leads to chemotherapy errors.22,23
It has been progressively recognized that some clinical decision support systems (CDSSs) have the potential to overcome the problems of paper-based protocols, improve compliance with chemotherapy protocols and decrease chemotherapy prescription errors.21,24-27 As Collins et al 24 pointed out: “Oncology is a very data-rich and protocol-driven specialty, making it perfect for the use of a CDSS.” Such CDSSs are coupled with CPOE (Computerized Provider Order Entry) can provide special alerts and recommendations based on standardized chemotherapy protocols for the patient at the time of the prescription and increase patient safety,28,29 particularly while considering user interest in the early stage of the CDSS design process. Basically, the poor outline of a framework interface can promptly back off Healthcare specialists as they may invest more energy in looking for relevant data than in investigating suggestions.24,30
Specifications of chemotherapy prescription decision support system (CPDSS) such as dose/drug/order/regimen recommendation according to the respective patient, performing a complex calculation of the dose and adjusting the dosage based on the various parameters (such as cumulative dose, kidney, and liver function tests, calculations based on age, height, weight and etc.) And drug interaction, allergy, abnormal weight, height, BMI, and body surface area (BSA) alerts were described in a recent review study. 31
The purpose of the present study was to design a CPDSS that was evaluated using a 3-step retrospective evaluation study to assess its effect on compliance with protocols and user satisfaction of CPDSS.
Methods
The method includes software development, 3-step retrospective evaluation, and data analysis description.
Development
A multidisciplinary team consisting of 2 oncologists and 3 information system specialists experienced in CDSS developed the CPDSS algorithm based on the IC-BFM 2002 protocol. The rules defined in this protocol can be used as “IF-Then” in decision support systems. These rules are translated into the computer-readable format as the algorithms of CPDSS.
The CPDSS checks the orders with the standard protocol agreed by the departmental experts and warns if they do not comply. The doctor can correct this deviation or ignore the alert. If the user determines that deviation from the protocol is necessary, the system displays a message and asks the user to justify the reason for deviating from the protocol according to the patient’s particular circumstances. However, all the alert drug/order characteristics, such as the type of alert, the prescriptive response to the alert, reasons of rejected alert, dose, frequency, or other modified profile of the drug, are recorded in the system’s database.
The design and architecture of CPDSS were intended to check and verify the parameters listed in Table 1 and provide the necessary alerts.
CPDSS alerts.
The system was developed in ASP.NET MVC with SQL Server as a database management system.
Evaluation
This study was evaluated at 2 teaching children’s hospitals in Tehran with an oncology ward that admitted children up to 18 years of age.
Information required for the technical and the retrospective clinical validation was extracted from the paper records and consisted of all the children with ALL between the ages of 1 and 18. Six physicians treated these patients from January to December 2019. Patients had received chemotherapy according to Protocol I/I’Phase 1, Protocol I/I’ Phase 2, Protocol Mm/M of the IC-BFM 2002 main protocol. The software was carried out at the hospital for 6 months.
A validation strategy developed by Scheepers-Hoeks et al 32 was modified and used for this study. The development and validation strategy developed by Scheepers-Hoeks consisted of 4 steps. The first step was a retrospective technical validation based on which CDSS was investigated to confirm that parameters are correctly linked to other related data. In the second step, an expert team examined all alerts in terms of clinical relevance, feasibility, and usefulness. In step 3, to ensure the accuracy of the alerts, CDSS was used in daily clinical practice. The fourth step was the CDSS improvement in practice, which was carried out by constant maintenance after implementing the system. In this study, the third to fourth steps were left out, and user satisfaction assessment of the CPDSS as another step was added to the evaluation process.
Technical evaluation
The technical evaluation step aimed to determine if the CPDSS algorithm worked technically and correctly. To perform this step, 4 oncologists examined the parameters and alerts in the algorithm.
Retrospective evaluation
At this evaluation step, the ability of the CPDSS to detect protocol deviation was studied using prescriptions of the study population. Initially, patients’ data such as the type of ALL, other diseases, allergies, height, age, weight, and patient tests during the treatment with chemotherapy were collected from paper records and entered into CPDSS. Subsequently, the chemotherapy prescriptions of these patients were entered into CPDSS; their compliance with the chemotherapy protocol was compared, and in case of non-compliance, the system alerted. The specifications of the alerts were stored in the system database.
User Satisfaction evaluation
Data related to user satisfaction were collected by a checklist in the form of a questionnaire in Persian. The questions of this questionnaire were designed based on previous studies and the opinions of a team of experts. The validity of the questionnaire was examined and approved by 5 medical informatics experts. The reliability of the questionnaire based on Cronbach’s alpha was estimated to be .90.
The questionnaire consisted of 2 parts (user interface, functionality) and 15 questions. User satisfaction evaluation of CPDSS was conducted on 2 aspects of user interface (8 yes/no questions) and functionality (7 yes/no questions). This questionnaire was distributed among 10 oncologists who had experience working with the CPDSS.
Data analysis
At first, the characteristics of patients were examined. Subsequently, the number of deviations from the protocol was determined for the parameters of the drug, dose, unit, frequency, route, solvent, rate, and date of drug delivery to the patient based on the alerts stored in the CPDSS database. In addition, the number of drug allergies alerts was examined. Two oncologists (who were different from prescribing oncologists) reviewed these alerts. The number of cases of protocol deviation and chemotherapy errors was determined, followed by an evaluation of users’ satisfaction with this system.
Ethical considerations
The medical center ethics committee approved the study and waived the need for specific patient consent due to its retrospective nature.
Results
Results of technical evaluation
In the technical evaluation of the software, three of the alert messages were corrected, changes were made to the user interface, and the warning was suggested if the doctor prescribed 10% more or less than the recommended dosage of the medication protocol.
Results of retrospective evaluation
In the retrospective evaluation step, 1281 doses prescribed for 30 patients (15 patients in each hospital) were entered into the system. The study was conducted on children aged 1 to 18 years old, of which the sample age was between 1 and 14. The number of drug doses in each protocol is summarized in Table 2.
Number of drug doses per protocol for all patients.
Table 3 presents the frequencies and percentages of alerts related to drug, dose, and day parameters in each of the treatment protocols for 30 patients. This Table shows that hospital A has 5 protocol deviations of the drug parameter from the Protocol I/I’ Phase 2, which is 0.076% of all prescriptions for this hospital in this Phase. Hospital B has 1 protocol deviation of the drug parameter from the Protocol I/I’ Phase 1 and one from the Protocol I/I’ Phase 2, which is 0.013% to 0.015% of its total prescriptions in each Phase, respectively. Also, CPDSS found 8 cases of the protocol deviation of the dose parameter in the Phase 1, 63 cases in the Phase 2, and 24 cases in the Phase 3, constituting 0.094%, 0.318%, 0.123% of the total dose in each Phase regarding the hospital A. For hospital B, 8 cases of the protocol deviation of the dose parameter were found in the first Phase, 24 cases in the Phase 2, and 30 cases in the Phase 3, constituting 0.1%, 0.121%, 0.154% of the total dose in each Phase. Furthermore, CPDSS found 126 cases of protocol deviation for day parameter in the Phase 1, 132 in the Phase 2, and 36 in the Phase 3, constituting 0.494%, 0.667%, 0.185% of total prescriptions of hospital A, respectively. Hospital B had 99, 108, 24 protocol deviation of day parameter from the first, second, third Phase, constituting 0.123%, 0.545%, 0.413% of the total prescriptions of this hospital prescriptions in each protocol respectively.
Frequency of the stored alerts in CPDSS database.
With the implementation of CPDSS, oncologists prescribed unit, frequency, route, solvent, and rate of the drugs in compliance with the protocol. No patient allergy to the drug was mentioned in the patients’ files, and the system did not record any alert about this parameter.
Evaluation results of user satisfaction
The results of the user satisfaction assessment provided by 10 oncologists are shown in Tables 4 and 5. The CPDSS user interface was rated 0.86, and CPDSS functionality was 0.96, which were both acceptable. At this point, users offered suggestions like showing a preview of the selected section of the protocol on the CPDSS page and creating links to more resources in the alerts messages to increase user satisfaction.
The results of evaluating the user satisfaction of the CPDSS-User Interface.
The results of evaluating the user satisfaction of the CPDSS-Functionality.
Results of data analysis
In the data analysis step, during the meetings, all the alerts recorded in the CPDSS database by 2 oncologists were analyzed. Experts have appraised all the cases of deviation from the protocol found by the CPDSS. Besides, based on the patient’s situation and paper records, they determined which of the protocol deviations were errors. Table 6 demonstrates the number of dose errors broken down by protocol for each hospital.
Frequency of prescription dose errors.
Abbreviation: CI, confidence interval.
According to experts, the CPDSS correctly found 42 medication errors occurred in hospital A, which accounted for 0.065% of the total doses prescribed by this oncologist (95%, CI = 0.92-2.88), as well as 15 drug dosing errors occurred in hospital B, which accounted for 0.024% of all doses prescribed in that hospital (95%, CI = 1.33-2.47). Based on the reasons given in Table 7, none of the deviations from the protocol in the drug parameter and drug day and 132 cases of the dosing parameter deviations that were detected by the CPDSS were documented as errors. CPDSS did not record deviations from the protocol for other parameters that led to an error. These findings showed that only the dose parameter in prescriptions for the 30 patients was erroneous; there was no error in other parameters of the chemotherapeutic agents.
List of reasons provided by oncologists.
Discussion
In this paper, a prescription chemotherapy CDSS for children with ALL was designed and evaluated. In the retrospective evaluation, 1281 doses of chemotherapy administered to 30 patients were prescribed with the CPDSS. Most of the 735 alerts recorded in the CPDSS database belonged to day, dose, and drug parameters with 525 (71%), 189 (26%), and 21 (3%), respectively. There was no alert for other parameters (including unit, frequency, route, solvent, rate, and allergy). As other studies had considered using the CDSSs to increase compliance with the treatment protocol,21,28,33,34 the CPDSS accurately identified 100% of the deviations from the treatment protocol. The results of this study demonstrated the ability of the CPDSS to find cases of deviation from the ALL chemotherapy protocols in drug, dose, unit, frequency, route, solvent, rate, and day parameters for children.
Several studies have pointed to the role of various CDSSs in reducing drug errors.10,24,35-38 Especially, CPOE coupled with CDSS has been able to significantly reduce therapeutic medication errors based on the rules derived from the therapeutic protocols.26,38-41 The results of this study on alert cases deviating from the chemotherapy protocol by CPDSS showed that this system could help find chemotherapy drug errors. All errors in prescription chemotherapy drugs occurred in the dose parameter (57 cases [30%] of the 189 cases deviated from the protocol [95%, CI = 1.25-2.55]). This is also supported by other studies14,42,43 in which dose errors were the most reported medication errors in the pediatric population, particularly at the stage of drug prescription.
The evaluation results related to user satisfaction from CPDSS also indicated that users were satisfied with the system based on 2 criteria of the user interface and system functionality. Even though in the evaluation of the previous step, many deviations from the chemotherapy protocol were not known as errors, especially in the case of the field of the drug and day, in assessing user satisfaction, from the perspective of a CPDSS user, the alert in these fields was useful. The users also confirmed the ability of the system to find errors in the prescription of chemotherapy drugs, especially regarding dose errors. They acknowledged that the system had played an active role in the decision-making process, and it has been instrumental in ensuring and subsequently improving patient safety. At this point, users provided suggestions to increase user satisfaction, such as showing a preview of the selected section of the protocol on the CPDSS page and creating links to more resources in alerting messages. As noted in other studies, 44 by demonstrating scientific documentation for chemotherapy protocols, alerts can play a significant role in learning the treatment process with evidence-based medicine and, users accepted the role of CPDSS in teaching and decision-making, especially for less experienced physicians.
Limitations of this research include lack of control of some factors by CPDSS, such as the use of other drugs, the simultaneous implementation of different interventions, and the cumulative dose of chemotherapy drugs. The CPDSS only makes good decisions about patients who are not in exceptional circumstances and, if the patient is in a particular condition, CPDSS will be able to decide while instructing the physicians.
The shortage of resources in the hospital, such as minimum hardware needed for such programs (eg, tablets that can be carried by physicians easily), and a comprehensive hospital information system which provides complete information on the treatment of patients, were also the limitations of this research. Since these preliminary results are encouraging, further investigation with providing complete information for CDSS and a larger sample is recommended to generalize these results.
Conclusion
The overall outcome indicated the ability of the CPDSS to find cases of deviation from the chemotherapy protocol on drug, dose, unit, frequency, route, solvent, rate, and day of drug administered to the patients, which led to the discovery of prescription errors in the chemotherapy regimen. The system can contribute to increasing the matching of chemotherapy protocols by providing alerts that lead to an increase in patients’ safety.
The results of the user satisfaction evaluation showed that the CPDSS was acceptable in terms of user interface features. Additionally, the system’s function in contributing to decision-making, awareness of the protocol, reducing the errors in prescribing chemotherapy drugs, reducing drug name errors, and ultimately, users’ confidence and improved patient safety were accepted. The results show that the system can reduce the prescribing errors of chemotherapy.
Footnotes
Funding:
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Declaration of conflicting interests:
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Authors Contributions
All authors contributed to the conception and design of the work. HM, AK, and RR performed data collection, conducted the analysis, and drafted the manuscript. HM, AK, and RR contributed to the analysis and interpretation of data and revised the manuscript critically. GrB, KAR, and FR contributed to the interpretation of data and revised the manuscript critically. All authors approved the final version of the submitted manuscript.
