Development and Evaluation of an Automated Protocol Recommendation System for Chest CT Using Natural Language Processing With CLEVER Terminology Word Replacement

Abstract

French

Purpose: To evaluate the clinical performance of a Protocol Recommendation System (PRS) automatic protocolling of chest CT imaging requests. Materials and Methods: 322 387 consecutive historical imaging requests for chest CT between 2017 and 2022 were extracted from a radiology information system (RIS) database containing 16 associated patient information values. Records with missing fields and protocols with <100 occurrences were removed, leaving 18 protocols for training. After freetext pre-processing and applying CLEVER terminology word replacements, the features of a bag-of-words model were used to train a multinomial logistic regression classifier. Four readers protocolled 300 clinically executed protocols (CEP) based on all clinically available information. After their selection was made, the PRS and CEP were unblinded, and the readers were asked to score their agreement (1 = severe error, 2 = moderate error, 3 = disagreement but acceptable, 4 = agreement). The ground truth was established by the readers’ majority selection, a judge helped break ties. For the PRS and CEP, the accuracy and clinical acceptability (scores 3 and 4) were calculated. The readers’ protocolling reliability was measured using Fleiss’ Kappa. Results: Four readers agreed on 203/300 protocols, 3 on 82/300 cases, and in 15 cases, a judge was needed. PRS errors were found by the 4 readers in 1%, 2.7%, 1%, and 0.7% of the cases, respectively. The accuracy/clinical acceptability of the PRS and CEP were 84.3%/98.6% and 83.0%/99.3%, respectively. The Fleiss’ Kappa for all readers and all protocols was 0.805. Conclusion: The PRS achieved similar accuracy to human performance and may help radiologists master the ever-increasing workload.

Visual Abstract

This is a visual representation of the abstract.

Keywords

natural language processing computed tomography chest protocols

Introduction

Computed tomography (CT) imaging is a mainstay diagnostic tool for a wide variety of diseases. The ability of a radiologist to create an accurate interpretation is dependent on the CT images’ clear visualization of the associated pathology. For this reason, image quality remains an important research focus. The CT protocol used for an imaging procedure refers to the selection of various parameters of the scan which, for chest CT as well as other procedure types, are tailored to produce an image that is well-suited for the clinical indication. Some examples of these parameters include administration of contrast bolus, scan range, patient position, slice thickness, and CT tube settings.¹ The appropriate selection of these parameters is integral to producing a diagnostic-quality image, while inappropriately selected parameters can produce an unusable image and may even introduce health risks to the patient. CT imaging accounts for the majority of medical imaging-related radiation exposure, which underscores the importance of managing the radiation dose—image quality tradeoff.² Use of contrast medium may possibly pose risks as the kidney function can be impacted, in particular in patients with renal insufficiency,³ although this is increasingly debated. Some patients may even have a history of allergic-like reactions to the contrast agent. In light of these factors, selecting appropriate CT imaging parameters is clinically important and requires expertise, necessitating that optimal protocolling be performed by trained radiologists in a clinical setting.

Protocolling is a monotonous and time-consuming task and can create disruptions for radiologists in a clinical setting.⁴ While a single imaging request may only take around 1 minute to protocol on its own, the large volume of CT imaging procedures exacerbates the task. Machine learning techniques have become increasingly broadly applied to a large variety of medical fields and can achieve near-expert performance for a well-defined task.⁵ The clinical workflow for protocolling, which involves free-text image request documents that are examined and then protocolled by a radiologist through the Radiology Information System (RIS), creates a fertile dataset for supervised natural language processing (NLP) techniques.⁶ Automated protocolling systems have been proposed before, in MRI imaging,^6,7 as well as abdominal CT,⁸ and have shown potential to increase both the efficiency and quality of protocolling when used as a support tool for a radiologist.⁹ We developed a protocol recommendation system (PRS) for chest CT and tested its performance against an optimized, reader-agreement reference standard. We hypothesize that such a system can help support the protocolling radiologist, making the task more efficient as well as reducing inter-reader variability.

Methods

Dataset Selection

With research ethics board approval, a total of 322 387 chest CT records were extracted from the RIS from 2017 to 2022. Each record contained several different information fields pertaining to the imaging request; for this study, we aimed to predict the “Protocol” field exclusively from the free text associated with the request contained in the Hospital Information System (HIS) field. Returning patient records were not removed as patients may have shifting or changing clinical indications (eg, a cancer patient with acute chest pain may undergo a chest pulmonary angiography [CTPA] for suspected pulmonary embolism), and the comment field is manually entered by the referring physician in free-text style. Records that were missing either of these fields were removed, for instance, when no clinical information or comment was provided or the assigned protocol was not documented (n = 55 488), and records with infrequently occurring protocols (more than 150 protocols with less than 100 instances, n = 3503) were also removed from the dataset, leaving 18 unique protocols reflecting current clinical practice (Table 1). Three hundred randomly selected records were then held-out as an evaluation dataset, resulting in a total of 263 096 records available to train the model.

Table 1.

List of generic imaging protocols in chest CT and their institutional abbreviated protocol names.

CT imaging protocol	Protocol name (clinically used)
Routine CT without i.v. contrast material	R
Routine CT with i.v. contrast material	R + C
Routine CT in low-dose technique	Low dose CT (LDCT)
Routine CT in low-dose technique, followed by CT pulmonary angiography (CTPA)	PE-ILD (LDCT + PE)
CT pulmonary angiography (CTPA)	PE (CTPA)
CT pulmonary angiography (CTPA) with iodine subtraction	PE-iodine subtraction
CT pulmonary angiography (CTPA) followed by Routine CT	PE + R + C
CT pulmonary angiography (CTPA) and CT angiography (CTA) for aortic dissection (chest only)	PE + dissection, chest only
CT angiography (CTA) for aortic dissection (without and with i.v. contrast, chest only)	Aorta-dissection (non-con + post-con), chest only
CT angiography (CTA) for aortic dissection (without and with i.v. contrast, chest and abdomen)	Aorta-dissection (non-con + post-con), chest + abdomen
Esophageal cancer (routine CT with i.v. and oral contrast)	Esophageal cancer (R + C + oral)
Lung cancer (routine CT with i.v. contrast, plus abdomen)	Lung cancer (R + C, plus abdomen)
Mediastinal mass with iodine subtraction	Mediastinal mass (iodine subtraction)
Lung transplant (routine CT in low-dose + expiratory scan)	Lung transplant (LDCT + exp)
Superior vena cava obstruction with bilateral i.v. contrast injection	SVCO (bilateral contrast)
Trauma	Trauma-Chest
Trachea (routine CT and thin-section reconstructions for 3D)	Trachea (R + thin section for 3D)
Interstitial lung disease, baseline assessment (routine CT, thin-section reconstruction, plus expiratory and prone scan)	ILD baseline (R + HighRes + exp + prone)

Free-Text Pre-Processing and Classification Model

Firstly, a regular expression was used to remove the standardized text from the report template. CLEVER (CL-inical EV-ent R-ecognizer) word substitutions¹⁰ were then applied to the remainder of the free text. The CLEVER text analysis tool has been clinically validated for information extraction from electronic medical records and has been used in similar works.^11,12 In this case, the word substitutions help to reduce the size of the vocabulary by replacing different phrases with similar meanings with a single token across the dataset. Punctuation was subsequently removed. The text was then tokenized, and stopwords were removed using the nltk Python package. Very low-frequency words were then removed from the text, with the best threshold experimentally determined to be <50 occurrences in the entire dataset. Frequently occurring word pairs were concatenated with a hyphen to become a single token. The best-performing threshold was >1000 occurrences in the dataset, wich was also experimentally determined. A main motivator for this step was to preserve variations of the phrase “no contrast” as a single feature for the otherwise order-agnostic bag-of-words model. The bag-of-words model was then created using the term frequency-inverse document frequency weighting scheme.¹³ The resultant sparse vector-space encoding was then used as the input to a multinomial logistic regression model with the “saga” solver and using the l2 penalty, trained using 10-fold cross-validation and implemented using the sklearn python package. A grid search was performed during the 10-fold cross-validation to determine the best-performing parameters for the pre-processing steps. The best-performing parameters were then used to train the model on the entire dataset. Figure 1 depicts the data processing pipeline.

Figure 1.

Flow chart depicting the data processing steps from the database to the classifier.

Clinical Evaluation

To evaluate the accuracy of the PRS and the clinically executed protocol (CEP) selected by a radiologist, the 300 cases held-out from the training dataset were used as an evaluation dataset. These cases were randomly sampled from the training dataset, but with the condition that each protocol occurs at a minimum of 10 times in the evaluation set. The distribution of protocols was different between the training and evaluation sets, as matching the distribution as closely as possible while also ensuring that each protocol is adequately represented would require the validation set to be much larger than 300 cases. Four fellowship-trained chest radiologists with 6, 15, 21, and 33 years of clinical experience, respectively, were asked to read through the clinical information for each of the evaluation cases and select one of the 18 protocols. After a reader made a selection, both the PRS’ and the clinically executed protocols were unblinded to the reader, who was then asked to grade both protocol selections against their own selection on a scale from 1 to 4 (1 = severe error, 2 = moderate error, 3 = disagreement but acceptable, 4 = agreement). The readers were allowed to change their minds after the PRS and the CEP were uncovered, and the change was recorded. The reference standard protocol for each case was eventually determined as the most frequently selected protocol by the 4 readers. In cases of a tie (eg, 2 readers select protocol A, 2 readers select protocol B), a judge reader (senior staff) would be used as a tiebreaker.

The protocol selections by the PRS and the CEP were defined as accurate when they matched the protocol of the reference standard. The accuracies of the PRS and the CEP were statistically compared using McNemar’s test. Two measures were determined from readers’ evaluation scores: the “correctness,” defined as the proportion of cases where the majority of readers assessed a grade of 4; and the “acceptability,” defined as the proportion of cases where the majority of readers assessed a grade of at least 3. The clinical performance of the PRS was assessed based on the readers’ evaluation scores, and the readers’ scores 1 and 2 were recorded to calculate the mistake rates of both the PRS and the CEP. The reliability of the readers’ evaluations was measured using Fleiss’ Kappa.

Results

Initially, the 300-case evaluation dataset contained a minimum of 10 cases for all 18 protocols. After the “reader-agreement” reference standard was created, all cases historically assigned to 2 of these 18 protocols (Lung Cancer, Mediastinal Mass) were re-labelled as another protocol by all readers, effectively removing those 2 protocols from the evaluation set. Similarly, 3 protocols (Trachea, PE-ILD, and ILD baseline) lost 1, 2, and 2 cases, respectively, in the final evaluation set because all readers re-labelled them with another protocol. The final distribution of protocols in both the training/test dataset, and the evaluation dataset (as determined by the “reader-agreement” reference standard) is shown in Table 2.

Table 2.

Overview of the Distribution of Protocols in Both the Train/Test Data and the Held-Out Evaluation Dataset, the Accuracy of the PRS and Clinically Executed Protocol Compared to the Ground Truth.

Protocol	Train/test	Evaluation	Accuracy
Protocol	Train/test	Evaluation	PRS	CEP	P
R	121 709	68	0.91	0.84	.195
R + C	65 672	65	0.92	0.63	<.0001
Low dose CT (LDCT)	33 480	23	0.96	0.91	.55
PE (CTPA)	23 246	19	1.00	0.79	.034
Esophageal cancer (R + C + oral)	3133	14	0.93	1.00	.309
Lung transplant (LDCT + exp)	6086	13	1.00	0.92	.308
Aorta-dissection (non-con + post-con), chest + abdomen	1496	11	0.82	0.91	.534
PE-iodine subtraction	273	11	0.82	0.91	.534
Trauma-Chest	138	11	0.00	0.91	n/a
SVCO (bilateral contrast)	127	10	0.60	0.90	.121
Aorta-dissection (non-con + post-con), chest only	1346	10	0.90	0.90	1
PE + dissection, chest only	416	10	0.70	1.00	.06
PE + R+C	421	10	0.20	0.70	.025
Trachea (R + thin section for 3D)	207	9	1.00	1.00	n/a
PE-ILD (LDCT + PE)	698	8	0.75	1.00	.131
ILD baseline (R + HighRes + exp + prone)	2979	8	0.88	0.88	1
Lung cancer (R + C, plus abdomen)	1814	—	—	—	—
Mediastinal mass (iodine subtraction)	155	—	—	—	—
Total/all protocols	263 396	300	0.84	0.83	.66

Only one reader changed their mind in 3 cases and selected different protocols after they had seen the PRS and CEP protocols. The accuracy of both the PRS and CEP for all cases was 84.3% and 83.0%, respectively. Table 2 shows the accuracies for each protocol. The PRS was found to perform significantly better in one protocol (“R + C”), with P = .0003, and the CEP protocol was significantly better in 2 protocols (“Trauma-Chest” and “PE + R + C”), with P = .0044 and P = .0077, respectively.

Both the “correctness” and “acceptability” of the PRS and the CEP selection, calculated from the evaluation grades, are shown in Figure 2. Table 3 summarizes the performance of the PRS based on the readers’ scores, both overall and broken down by individual readers, as well as the readers’ classification of mistakes made by the PRS an CEP. For all protocols, the correctness of the PRS and the CEP was 84.3% and 83%, and the acceptability 98.6% and 99.3%, respectively. One reader classified 8 PRS selections (2.7%) as mistakes, and another reader classified 6 CEP selections (2%) as mistakes; a detailed breakdown of all 4 readers is also shown in Table 3.

Figure 2.

PRS-selected and clinically-executed protocols, as graded by readers. “Correctness” was defined as the proportion of cases where the majority of readers gave a score of 4. “Acceptability” was defined as the proportion of cases where the majority of readers gave a score of 3 or 4.

Table 3.

Performance of the PRS Based on the Evaluation Scores, Overall and by Individual Readers. Mistakes of the PRS and CEP, as Scored by the Readers, Vary Between 0.7% and 2.7%.

Evaluation	PRS		Mistakes
	Correctness (%, only score 4)	Acceptability (%, scores 3 and 4)	PRS	CEP
	Correctness (%, only score 4)	Acceptability (%, scores 3 and 4)	(Scores 1 and 2)
Overall	84.3	98.6	—	—
Reader 1	70.6	99.0	3 (1.0%)	6 (2.0%)
Reader 2	85.3	97.3	8 (2.7%)	5 (1.7%)
Reader 3	85.7	99.0	3 (1.0%)	3 (1.0%)
Reader 4	87.7	99.3	2 (0.7%)	2 (0.7%)

Figure 3 shows the level of agreement between readers for each case in the evaluation dataset, split by protocol. Overall, all 4 readers assigned the same protocol in only 203 out of 300 cases (67.6%), 3 readers agreed on the same protocol in 83 out of 300 protocols (28%), and in 15 cases, a judge reader needed to break the tie (5%). The Fleiss Kappa for all readers and protocols was 0.805. Supplement 1 shows example cases from the evaluation dataset.

Figure 3.

Level of agreement between readers, split by protocol.

Discussion

The overall accuracy of the automated PRS for chest CT matched the human performance. While the PRS’ overall accuracy of 84.3% against the reader-agreement reference standard leaves room for improvement, this performance was still slightly better than the clinically executed protocol (CEP) with an 83% accuracy, albeit not significantly. The CEP accuracy being as low as 83% is surprising, implying that a panel of radiologists will advise for a different protocol than was clinically executed in 17% of cases. The acceptability score of the CEP, however, was much higher at 99.3%, implying that these cases were not necessarily medical errors. The performance differed on a per-protocol basis, with the PRS performing significantly better on a routinely chosen chest protocol “R + C” (PRS: 92.3% vs CEP: 62.1%), but significantly worse on the protocol for chest trauma “Trauma- Chest” (PRS: 0% vs CEP: 90.9%) and for the biphasic protocol “PE + R + C” (PRS: 20% vs CEP: 70%).

The PRS’ superior performance for the routine chest with contrast protocol is surprising as labels in the training dataset were determined by the radiologist-chosen, clinically executed protocol. The differences between the “reader-agreement” reference standard and the clinically executed protocol may be caused by inter-reader variances or possibly by shifting clinical standards over time, as the evaluation dataset was sampled randomly from the training dataset across the entire 5-year window. The poorer performance for the 2 other protocols may be attributed to their lower frequency in the training dataset and the tendency of the PRS to default to the most commonly used protocol when the clinical history is sparse or of limited clinical relevance.

The training dataset (and evaluation dataset) was highly imbalanced, where the top protocol chest without contrast (“R”) accounted for approximately 40% of the dataset, whereas 11 of the 18 protocols each individually comprised less than 1% of the total. The PRS largely struggled with these lower-frequency protocols (such as “Trauma-Chest”—a rarely used protocol as our institution is not a trauma centre) but performed comparably or better than the clinically-executed protocol on the more frequent protocols.

Providing a meaningful clinical history and concise clinical query is fundamental to any patient transfer and imaging request.^14-16 This is especially critical for imaging studies that can be conducted using various imaging protocols.¹⁷ Additionally, certain protocols can significantly affect patient scheduling due to factors such as exam prioritization, equipment selection, scan time requirements, and patient preparation, underscoring the clinical importance of accurate protocolling.¹⁸ While machine learning demonstrates superiority in uncertain scenarios, inadequate training data can render its output ineffective.^19,20

We observed notable concurrent and temporal variability in radiologists’ protocol selection that should not be disregarded. First, the 4 readers who established the concurrent reference standard could only agree on the same protocol in 68% of the clinical requests. Second, 2 out of 18 protocols dropped out of the evaluation dataset because all cases were re-labelled to other protocols than the original, clinically executed protocols. Instead of using the historically executed protocols with cross-validation, which only one radiologist selected at that time, we opted to create a validation dataset based on a four-reader-agreement as a more robust reference standard for testing the PRS and human performance. The purpose was to mitigate against potential variability and errors in the historically executed protocols. Thus, the overall model performance would likely be enhanced with a sufficiently large dataset of cases annotated with this “reader-agreement” reference standard being available for training.

The process of protocolling incoming imaging requests is inherently repetitive and charitably monotonous, leading to a heightened risk of errors due to fatigue. Moreover, radiologists are frequently interrupted during image reading sessions to handle ad-hoc protocols or address emergency requests, which can divert their focus and potentially result in inaccurate image interpretation or protocol selection.⁴ Despite the fact that protocolling a single CT examination may take less than a minute, the cumulative demand of timely protocolling all imaging requests prior to execution imposes a substantial burden on radiologists. Chung et al. conducted a multi-year evaluation of an automated protocolling system and demonstrated that it significantly reduced the radiologists’ protocolling workload and decreased the time from order entry to protocol assignment without increasing protocolling errors.²¹

As medical imaging is integral to operations in high-reliability organizations (HROs), it is difficult to envision that relying solely on a machine learning model within a single-operator paradigm would address the key challenges of HROs, particularly the complexity, unpredictability, and constant change.²² An automated PRS could help mitigate some of these risks when employed as a second opinion, akin to the use of a Computer-Aided Detection (CAD) system.²³ Alternatively, a “model-protocols-then-reader-validates” approach may alleviate the cognitive burden by simplifying the protocolling task to a validation task. More importantly, this approach has the potential to reduce variability in protocolling, ensuring that similar cases are managed consistently with minimal reliance on individual reader biases.

The PRS model necessitates continuous training to enhance its accuracy, alongside ongoing data curation efforts to adapt to the dynamic requirements of HRO environments. We observed a higher model performance for protocols extensively represented in the training dataset, indicating potential benefits from employing data augmentation techniques.⁸ Protocols with fewer than 100 occurrences over a 5-year period were excluded from the analysis, yet these instances collectively constitute a substantial portion of the dataset, underscoring the importance of augmenting or curating available data resources.

In summary, the automated protocol recommendation system (PRS) for chest CT imaging requests achieved similar accuracy to human performance and may help radiologists master the ever-increasing workload by alleviating the burden of monotonous protocol selection, though challenges with less frequent protocols remain to be solved.

Supplemental Material

sj-docx-1-caj-10.1177_08465371241280219 – Supplemental material for Development and Evaluation of an Automated Protocol Recommendation System for Chest CT Using Natural Language Processing With CLEVER Terminology Word Replacement

Supplemental material, sj-docx-1-caj-10.1177_08465371241280219 for Development and Evaluation of an Automated Protocol Recommendation System for Chest CT Using Natural Language Processing With CLEVER Terminology Word Replacement by Patrik Rogalla, Jennifer Fratesi, Sonja Kandel, Demetris Patsios, Farzad Khlavati and Sean Carey in Canadian Association of Radiologists Journal

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Patrik Rogalla

Farzad Khalvati

Supplemental Material

Supplemental material for this article is available online.

References

Little

. Approach to chest computed tomography. Clin Chest Med. 2015;36(2):127-145. doi:10.1016/j.ccm.2015.02.001

Kubo

Lin

PJP

Stiller

, et al. Radiation dose reduction in chest CT: a review. Am J Roentgenol. 2008;190(2):335-343. doi:10.2214/AJR.07.2556

Hsieh

Chiu

How

, et al. Contrast medium exposure during computed tomography and risk of development of end-stage renal disease in patients with chronic kidney disease: a nationwide population-based, propensity score-matched, longitudinal follow-up study. Medicine (Baltimore). 2016;95(16):e3388. doi:10.1097/MD.0000000000003388

Schemmel

Lee

Hanley

, et al. Radiology workflow disruptors: a detailed analysis. J Am Coll Radiol. 2016;13(10):1210-1214. doi:10.1016/j.jacr.2016.04.009

Deo

. Machine learning in medicine. Circulation. 2015;132(20):1920-1930. doi:10.1161/CIRCULATIONAHA.115.001593

Chen

. Essential elements of natural language processing: what the radiologist should know. Acad Radiol. 2020;27(1):6-12. doi:10.1016/j.acra.2019.08.010

Wong

Hatef

Ryu

Nguyen

Makary

Prevedello

. An artificial intelligence tool for clinical decision support and protocol selection for brain MRI. Am J Neuroradiol. 2023;44(1):11-16. doi:10.3174/ajnr.A7736

Xavier

Chen

. Natural language processing for imaging protocol assignment: machine learning for multiclass classification of abdominal CT protocols using indication text data. J Digit Imaging. 2022;35(5):1120-1130. doi:10.1007/s10278-022-00633-8

Kalra

Chakraborty

Fine

Reicher

. Machine learning for automation of radiology protocols for quality and efficiency improvement. J Am Coll Radiol. 2020;17(9):1149-1158. doi:10.1016/j.jacr.2020.03.012

10.

GitHub. CLEVER. Accessed August 17, 2024. https://github.com/stamang/CLEVER/blob/master/res/dicts/base/clever_base_terminology.txt

11.

Ling

Kurian

Caswell-Jin

Sledge

Shah

Tamang

. Using natural language processing to construct a metastatic breast cancer cohort from linked cancer registry and electronic medical records data. JAMIA Open. 2019;2(4):528-537. doi:10.1093/jamiaopen/ooz040

12.

Tamang

Hernandez-Boussard

Ross

Patel

Gaskin

Shah

. Enhanced quality measurement event detection: an application to physician reporting. EGEMS (Wash DC). 2017;5(1):5. doi:10.13063/2327-9214.1270

13.

Sammut

Webb

, eds. TF–IDF. In: Encyclopedia of Machine Learning. Springer US; 2011:986-987. doi:10.1007/978-0-387-30164-8_832

14.

Cohen

. Accuracy of information on imaging requisitions: does it matter? J Am Coll Radiol. 2007;4(9):617-621. doi:10.1016/j.jacr.2007.02.003

15.

Starmer

Spector

Srivastava

, et al. Changes in medical errors after implementation of a handoff program. N Engl J Med. 2014;371(19):1803-1812. doi:10.1056/NEJMsa1405556

16.

Leslie

Jones

Goddard

. The influence of clinical information on the reporting of CT by radiologists. Br J Radiol. 2000;73(874):1052-1055. doi:10.1259/bjr.73.874.11271897

17.

Trattner

Pearson

GDN

Chin

, et al. Standardization and optimization of CT protocols to achieve low dose. J Am Coll Radiol. 2014;11(3):271-278. doi:10.1016/j.jacr.2013.10.016

18.

Brown

Marotta

. A natural language processing-based model to automate MRI brain protocol selection and prioritization. Acad Radiol. 2017;24(2):160-166. doi:10.1016/j.acra.2016.09.013

19.

Redman

. If your data is bad, your machine learning tools are useless. 2018. 2018. Accessed July 13, 2024. https://hbr.org/2018/04/if-your-data-is-bad-your-machine-learning-tools-are-useless

20.

Jarrahi

. Artificial intelligence and the future of work: human-AI symbiosis in organizational decision making. Bus Horiz. 2018;61(4):577-586. doi:10.1016/j.bushor.2018.03.007

21.

Chung

Demers

Tiberio

, et al. Implementation of an institution-wide rules-based automated CT protocoling system. Am J Roentgenol. 2024;222(4):e2329806. doi:10.2214/AJR.23.29806

22.

Vogus

Weick

Sutcliffe

. Doing no harm: enabling, enacting, and elaborating a culture of safety in health care. SSRN Electron J. Published online 2010. doi:10.2139/ssrn.1904620

23.

Doi

. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imaging Graph. 2007;31(4-5):198-211. doi:10.1016/j.compmedimag.2007.02.002

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB