Sage Journals: Discover world-class research

Abstract

Radiology reporting is narrative, and its content depends on the clinician’s ability to interpret the images accurately. A tertiary hospital, such as anonymous institute, focuses on writing reports narratively as part of training for medical personnel. Nevertheless, free-text reports make it inconvenient to extract information for clinical audits and data mining. Therefore, we aim to convert unstructured breast radiology reports into structured formats using natural language processing (NLP) algorithm. This study used 327 de-identified breast radiology reports from the anonymous institute. The radiologist identified the significant data elements to be extracted. Our NLP algorithm achieved 97% and 94.9% accuracy in training and testing data, respectively. Henceforth, the structured information was used to build the predictive model for predicting the value of the BIRADS category. The model based on random forest generated the highest accuracy of 92%. Our study not only fulfilled the demands of clinicians by enhancing communication between medical personnel, but it also demonstrated the usefulness of mineable structured data in yielding significant insights.

Keywords

information extraction natural language processing radiology reporting rule-based text mining

Introduction

Electronic medical records (EMRs) are collection of the patient's medical information stored on a computer system that allow data tracking over time and monitor and improve the quality of care within the practice.^1,2 EMRs can come in structured and unstructured data, rich in valuable real-world data on the patient, clinical findings, and genomic information.³ A radiology report in electronic form is an example of an unstructured EMR.

A breast radiology report contains the interpretation of a breast imaging study prepared by a radiologist, which plays a vital role in communication between radiologists and clinicians.^4,5 The free-text radiology report is often associated with excessive variability in language and format when reported by different radiologists. This may decrease clarity and increase the difficulty for clinicians to highlight information from the report.⁶ Conversely, structured reporting, an alternative reporting method, helps minimize the variability and maximize the clarity of the report by improving the consistency and communication of information. Additionally, structured reporting has more potential for data analysis by using a machine learning-based techniques compared to unstructured datasets.⁷ Machine learning-based analysis requires well-curated input data, which is typically absent from the EMRs. When utilizing the data in the radiology report, the structured and clean data required to derive the machine learning-based predictive models can be extracted by using natural language processing (NLP), thereby minimizing the cost of manually labelling of training data.

Text mining is an artificial intelligence (AI) technique based on NLP to structure the input text, extracting meaningful, non-trivial patterns or knowledge from unstructured texts.^8,9 There are two main approaches used in NLP: rule-based and machine learning-based techniques.¹⁰ Previously, researchers have utilized the NLP techniques to extract specific information from the EMRs. However, the majority of clinical NLP mainly focuses on a single task, such as, smoking status and hip fracture classification,¹¹ thromboembolic disease detection,¹² actionable appendicitis findings extraction,¹³ skeletal site-specific fractures identification,¹⁴ ureteric stones identification,¹⁵ stroke detection,¹⁶ section label generation,¹⁷ prostate cancer severity classification,¹⁸ and cancer morphology classification.¹⁹

Our goal is to develop a text mining program based on rule-based NLP techniques to generate structured and summarized breast radiology reports from the anonymous institute. The extracted information in a structured data format allows clinicians to audit and secondary use in machine learning data analysis, for example, predicting Breast Imaging Reporting and Data System (BIRADS) category value using the extracted response for mammogram findings.

Method

NLP algorithm to extract important variables from breast radiology report

Study setting

The Medical Research Ethics Committee (MREC) has approved this study to develop a point-of-care data capture for the Institutional Breast Cancer Registry. The dataset consists of 327 de-identified breast radiology reports from the years 2015 to 2019. These reports were written by various radiologists from the anonymous institute and were randomly retrieved from the radiology information system (RIS) in the Department of Radiology. The random retrieval of the radiology reports was performed by the radiologist involved in the study with the aim of minimizing dataset bias. The third-party RIS implemented in the anonymous institute restricts access for non-medical staff and does not allow direct export of breast radiology reports. Therefore, the radiology reports were manually copied from the RIS to another editable format by the radiologist, which increased the difficulty in retrieving the radiology reports within the time constraints.

The structure for mammography or ultrasound reports, the lexicon for breast imaging findings, and the final assessment category are based on the 5th edition of the American College of Radiology (ACR) BIRADS classification system.²⁰ However, each breast radiology report may only contain either mammogram or ultrasound examination findings.

During algorithm development, the radiology reports were randomly divided into 229 training data and 98 radiology reports testing data by using the randomized tools provided by Microsoft Excel. The proposed workflow is illustrated in Figure 1. Table 1 shows the number of mammogram and ultrasound reports involved in this study. Table 2 illustrates the significant data elements identified from the breast radiology report by the radiologists in the anonymous institute.

Figure 1.

Overview of study.

Table 1.

Number of mammogram and ultrasound reports used in this study.

Radiology report	Training data	Testing data
Mammogram finding	39	32
Ultrasound finding	70	19
Mammogram and ultrasound finding	120	47
Total	229	98

Table 2.

Significant data elements.

Data elements	Description
Mammogram
• Examination date	- Date of mammogram examination
• Lesion type	- ACR BIRADS classification mammography lexicon
• Quadrant position	- Breast quadrant where lesion locate
• Breast laterality	- Breast side where lesion located
• BIRADS density	- Levels of density of breast
Ultrasound
• Examination date	- Date of ultrasound examination
• Lesion type	- ACR BIRADS classification ultrasound lexicon
• Clock position	- 1-12 o’clock position where lesion located
• Breast laterality	- Breast side where lesion located
• Lesion size	- Measurement in either, three-dimensional, two-dimensional or one-dimensional
Final assessment
• BIRADS category	- Final assessment to describe findings and results

Rule-based NLP algorithm

R programming language version 3.6.1 was used to create the proposed algorithm.²¹ Based on training dataset, a set of rules was developed for identifying the desired data elements, guided by the radiologist’s suggestions. The NLP algorithm underwent refinement after radiologist verified the correctness of response extracted by the NLP algorithm. Figure 2 shows the overall algorithm design.

Figure 2.

Algorithm design.

In the first preprocessing step, all characters were converted into lowercase to prevent case-sensitive problems in the subsequent extraction process. Unnecessary characters generated during the exportation of the report from the working platform were removed. The Roman numerals previously used to denote BIRADS category value were replaced with Arabic numerals to ensure format uniformity. Therefore, the report was divided into mammogram and ultrasound sections. The segmentation step was conducted by recognizing the occurrence of the terms “mammogram” and “ultrasound” as the keyword. The successive preprocessing steps enabled the use of a subset of sections with specific rules to maximize efficiency. Boundary detection was performed to further split the report into sub-rows coinciding with each new sentence. This approach allowed semantic analysis to be performed on each sentence individually.

For each mammogram finding, the extracted response included lesion type, breast quadrant, and breast laterality. The size and position of the lesion were not often reported in mammogram section, as mammography could not be precisely determined them.^22,23 The presence of at least one of the following data elements in a sentence such as breast quadrant, breast laterality, or a term that describing the lesion location is mandatory as a keyword to allow the NLP algorithm to proceed with further semantic analysis for extracting the lesion type. The lesion type is extracted by matching the text to the predefined R list. Equation (1) illustrates the structure of the R list, which contains a set of the mammography lexicons used for pattern matching.

{PredefinedList}_{l e s i o n T y p e} = c (“ {lexicon}_{1} ”, “ lexicon_n ”),

(1)

The lexicon used for lesion type is illustrated as a hierarchy tree in Figure 3. During algorithm refinement, terms that were not present in the ACR BIRADS classification system were added to reduce false negatives in the training data. Table 3 illustrates the regular expressions used for identifying the lesion type.

Figure 3.

Mammogram lexicons. Adapted from ACR BIRADS classification system 5^th edition.

Table 3.

Regular expressions used for the mammogram variables.

Variable	Regular expression
• Breast quadrant	- Inner upper; outer upper; inner lower; outer lower; upper mid inner; upper mid outer; lower mid inner; lower mid outer; upper mid; lower mid; inner mid; outer mid; upper; outer; lower; mid
• Breast quadrant	* Where each keyword is in no particular order. (Inner upper or upper inner is acceptable)
• Breast laterality	- Left; right; both breast
• Other terms describing location	- Axillary; periareolar; retroareolar; retromammary

The response to be extracted in the ultrasound section includes lesion type, breast clock face, breast laterality, and lesion size. The extraction process was performed in the same manner as in the mammogram section. However, the presence of the lesion size is mandatory as a keyword for further semantic analysis. The algorithm prioritized three-dimensional measurements over two-dimensional and one-dimensional measurements when extracting the lesion size to avoid duplicate extractions. When reporting the lesion’s clock face position, radiologists may not describe it using the clock position (C) but rather within a range of two clock positions (R) or using anatomical direction terms (D). The presence of a keyword “clock” (CK), is critical in differentiating the clock value from other numeric values. The rule that can extract the precise clock position or a clock range was determined as in equation (2). The regular expressions used for each data element are listed in Table 4. Figure 4 shows the terms used for lesion type in ultrasound. The pseudocode for the mammogram and ultrasound findings extraction is shown in the supplementary section.

ClockPosition = {[(C \cup R) \cap CK] \cup D},

(2)

Table 4.

Regular expressions used for the ultrasound variables.

Variable	Regular expression
• Size of lesion	- [numeric][units]? X [numeric][units]? X [numeric][units];
	[numeric][units]? X [numeric][units];
	[numeric][units]?
	*Where symbol “?” indicated that the value is optional
• Clock face of lesion	- 1 to 12 o’clock position\| a range of clock
• Breast laterality	- Left; right; both breast
• Other terms describing location	- Axillary; periareolar; retroareolar; retromammary

Figure 4.

Ultrasound lexicons. Adapted from ACR BIRADS classification system 5^th edition.

Dates can be reported in various forms, with the year, month, and day written in different sequences and expressed in numeric or alphabetical format. To simplify the extraction process, the R package “lubridate” was utilized.²⁴ The algorithm extracts dates and converts them into the Year/Month/Day format to ensure consistency. The keyword “birads” (B) is used to identify the value for BIRADS category and BIRADS density. The algorithm extracts the alphabetical value (A) or numerical value (N) that follows the keyword “birads” as shown in Equations (3) and (4).

BIRADS category = {B \cap N},

(3)

BIRADS density = {B \cap A},

(4)

However, some radiologists prefer to describe the BIRADS density in a narrative format instead of using the density score. Therefore, additional rules to detect the BIRADS density in such cases. These additional rules consisted of two steps. Firstly, a rule was implemented to identify the narrative BIRADS description within a sentence. Following that, another rule was applied to match the narrative description with the corresponding BIRADS density score. Table 5 provides a list of keywords for identifying the BIRADS density score in the narrative format.

Table 5.

Narrative keywords for BIRADS density score.

BI-RADS density	Narrative description
A	- Fatty
B	- Fibroglandular density
C	- Heterogeneously dense
D	- Extremely dense

The outcomes of the information extraction step consist of a set of responses, along with their corresponding data elements, which are stored in a data frame for further utilization in predictive analytics. Using the extracted information, a grammatically correct sentence was generated to produce a synoptic breast radiology report.

Graphical user interface for rule-based NLP algorithm

A simple graphic interface for the text mining program was developed using Python 2.7.18²⁵ along with the Tkinter module.²⁶ This interface aims to assist clinicians in auditing reports by presenting the information in a summarised format. Figure 5 shows the design of the interface.

Figure 5.

Text mining program interface design.

Evaluation of the NLP algorithm

The information extraction ability of the NLP algorithm was assessed using a human validation method. A radiologist was provided with both free-text reports and structured radiology reports generated by the proposed algorithm. During the evaluation, the radiologist determined the correctness of the algorithm-extracted findings and also identified any information missed by the algorithm. In information extraction tasks, the emphasis is on accurately identifying and extracting specific information of interest. The evaluation metrics used, such as precision, recall, and F1 score, measure the accuracy and completeness of the extracted information compared to the ground truth. True negatives, which represent correctly identified absence of information, are not typically included in these metrics. The focus is on maximizing the extraction of relevant information rather than evaluating the absence of it. To measure the algorithm's performance in information extraction, five metrics known as precision, recall, false discovery rate (FDR), false negative rate (FNR), and F1 scores were used.

Precision = \frac{True positive}{True positive + False positive},

(5)

Recall = \frac{True positive}{True positive + False negative},

(6)

False discovery rate = \frac{False positive}{True positive + False positive},

(7)

False negative rate = \frac{False negative}{True positive + False negative},

(8)

F 1 score = \frac{2 \times Precision \times Recall}{Precision + Recall},

(9)

Borderline cases can occur when there are amendments made to the free-text report. These amendments are typically added at the bottom of the report. In such cases, the extraction process can be less accurate and potentially result in increased false-negative or false-positive values. To address this issue, a reminder message will be displayed when the keyword “amendment” is detected in the report. This serves as a prompt for clinicians to pay closer attention to any potential issues or changes that have been made.

Predicting BIRADS category using variables extracted by NLP algorithm

Datasets and variables

The second part of the study focuses on predicting the BIRADS categories using the algorithm-extracted data. For this purpose, a dataset consisting 229 structured mammogram records in CSV format was utilized to develop and train the predictive model. In order to enhance the accuracy of the model, the data element “lesion type” was further divided and stored into 12 sub-variables, as outlined in Table 6, based on the ACR BIRADS classification system, which was automatically generated by the NLP algorithm. Additionally, two additional features from the mammogram findings were considered essential for the model training, namely the number of findings and the BIRADS density, both identified by the radiologist as important in determining the BIRADS categories value. Figure 6 provides an overview of the predictive analytics process, which involved automating the initial steps of requirement collection, data collection, and data analysis using the rule-based NLP algorithm. Subsequently, the focus shifted to the crucial step of feature selection to develop a robust and suitable predictive model.

Table 6.

Variables involved in predictive analytics.

	Variables
Sub-variables of lesion type	Composition; symmetrical; malignancy; density; margin; mass; morphology; shape; calcification; distribution; lesion; associated
Others	Number of finding; BIRADS density

Figure 6.

Overview of predictive analytics process.

Feature selection

Feature selection is a critical step in developing a predictive model. Utilizing only the features with higher importance scores can enhance the accuracy of the model. The R package “caret” provides the recursive feature elimination (RFE) method, which facilitates the removal of unimportant features.²⁷ By eliminating non-significant attributes, the chance of overfitting during model training is reduced. RFE ranks feature importance using the mean decrease of accuracy (MDA) metric, which calculates the accuracy loss after randomly permuting the values of each input in each iteration.²⁸ Consequently, variables with higher importance result in a greater decline in the prediction accuracy. As a result, the variables with the lowest importance are eliminated, resulting in a minimal set of variables with the best predictive ability.

Predictive model

Five machine learning-based predictive models were selected for this study: decision tree, random forest, multinomial logistics regression, extreme gradient boosting, and multiple layer perceptron. The dataset was randomly split into a training set and a testing set using the data splitting function provided by the R package “caret”. The training set consisted of 70% of the data (160 mammogram findings), while the testing set comprised the remaining 30% (69 mammogram findings).

Decision tree

The decision tree model for BIRADS category prediction was generated using the classification and regression tree (CART) function provided by the R package “tree”.²⁹ CART builts the decision tree by recursively partitioning the dataset into subsets based on different predictor variables. At each split, the algorithm selects the predictor variable that best separates the data into homogeneous subsets. This process continues until a stopping criterion is met, resulting in terminal nodes. Each terminal node in the decision tree represents a specific combination of predictor variables and provides a prediction for the target variable, which, in this case, is the BIRADS category.³⁰ By using the CART function from the “tree” package, the decision tree model captures the relationships between predictor variables and the BIRADS category, enabling accurate predictions for new, unseen data.

Random forest

The random forest model, developed using the R package “RandomForest,” is based on Breiman and Cutler's original Fortran code.³¹ The algorithm generates multiple independent decision trees using various random subsets of the training set. By generating multiple trees with varying subsets, the random forest model captures a range of possible relationships between the input variables and the target variable. During prediction, the model combines the outputs of these trees and determines the final prediction by selecting the most frequently occurring output among the individual decision trees.³² This ensemble approach enhances the robustness and accuracy of the predictions in a wide range of applications.

Multinomial logistics regression

The R package “nnet” was utilized to develop the multinomial logistics regress model.³³ Multinomial logistics regression is a classification method that generalizes logistics regression to handle multiple discrete outcomes. During operation, one of the categories was chosen as the base category, and the remaining categories are individually regressed against the base category to determine their probabilities. The coefficients of the regression model represent the influence of each predictor variable on the log-odds of each outcome. By fitting the multinomial logistic regression model, predictions and classifications of new instances into one of the discrete outcome categories can be made. This approach is particularly useful when dealing with categorical response variables that have more than two levels.

Extreme gradient boosting

The gradient boosting machine is an ensemble supervised learning technique that accepts matrices for model evaluation. It is a powerful supervised learning method that is capable of handling both regression and classification problems. It is particularly effective when dealing with complex and diverse types of variables. The technique works by sequentially building a series of weak prediction models, with each subsequent model focusing on minimizing the errors made by the previous models. This iterative process targets areas where the previous models performed poorly, gradually improving the overall accuracy. The final prediction is obtained by combining the predictions of all the weak models.^34,35 The R package “xgboost” was used to perform the gradient boosting machines.³⁶

Multiple layer perceptron

This study utilized multiple layer perceptron, which is a feed-forward and supervised learning technique for artificial neural networks, implemented using the R package “h2o”.³⁷ The multiple layer perceptron consists of at least three layers, namely input, hidden, and output layers. It is known as universal approximator, as demonstrated by Cybenko's theorem, which states that a neural network with a single hidden layer and a sufficient number of neurons can approximate any continuous function. The multiple layer perceptron works by propagating input data forward through the network, applying weights to the connections between neurons, and activating each neuron using an activation function. During training, the network adjusts the weights based on the difference between the predicted and actual output, optimizing its ability to make accurate predictions.³⁸ This technique is effective in capturing complex relationships and patterns in the data, making it suitable for various regression and classification tasks.

Model evaluation

Similar to information extraction tasks, the concept of true negatives is not commonly utilized in multi-class classification scenarios. True negatives are primarily applicable to binary classification problems, where the goal is to correctly classify instances into two categories. In multi-class classification, the emphasis is on assessing the model's ability to accurately identify and differentiate between multiple classes or categories. Therefore, evaluation metrics in multi-class classification typically focus on measures such as accuracy, precision, recall, and F1 score using R package “caTools”,³⁹ which provide insights into the model's overall performance across the different classes. In the model’s performance assessment, the accuracy of each model was evaluated by:

Accuracy = \frac{SUM (outcome predicted correctly)}{SUM (number of inputs)},

(10)

In addition, the agreement between the predicted outcomes and the actual outcomes was further evaluated using the kappa coefficient. The kappa coefficient measures the level of agreement between two raters or evaluators, taking into account both the observed agreement and the agreement expected by chance. It is calculated as:

Kappa coefficient = \frac{(Po - Pe)}{(1 - Pe)},

(11)Where Po represents the observed proportion of agreement and Pe represents the proportion of agreement expected by chance. The resulting kappa value provides an indication of the model's performance in terms of inter-rater agreement, with higher values indicating better agreement between the predicted and actual outcomes.

Result

NLP algorithm to extract important variables from breast radiology report

The radiologist involved in this study identified 11 variables as keywords for potential data mining, which are listed in Tables 7 and 8. Overall, the NLP algorithm achieved an F1 score of 0.970 (precision=0.979; recall=0.962) on the training dataset. For the testing data, the algorithm has achieved an F1 score of 0.957 (precision=0.942; recall=0.949). Tables 7 and 8 show the extracted scores obtained by the rule-based NLP algorithm using training data and testing data, respectively.

Table 7.

Evaluation result of training data.

Extracted results	Precision	Recall	FDR	FNR	F1-score
• Mammogram date	1.000	0.980	0.000	0.020	0.990
• Type of lesion under mammography	0.977	0.916	0.023	0.084	0.946
• Breast quadrant where lesion located under mammography	0.971	0.925	0.029	0.075	0.948
• Breast laterality where lesion located under mammography	0.987	0.943	0.013	0.057	0.964
• Ultrasound date	1.000	1.000	0.000	0.000	1.000
• Type of lesion under ultrasound	0.958	0.907	0.042	0.093	0.932
• Breast clock position where lesion located under ultrasound	0.987	0.974	0.013	0.026	0.980
• Breast laterality where lesion located under ultrasound	0.949	0.984	0.051	0.016	0.966
• Size of lesions observed from ultrasound	0.998	0.998	0.002	0.002	0.998
• BIRADS density	1.000	1.000	0.000	0.000	1.000
• BIRADS category	1.000	0.996	0.000	0.004	0.998
Overall	0.979	0.962	0.021	0.039	0.970

Table 8.

Evaluation result of testing data.

Extracted results	Precision	Recall	FDR	FNR	F1-score
• Mammogram date	1.000	0.972	0.027	0.027	0.986
• Type of lesion under mammography	0.891	0.872	0.128	0.128	0.881
• Breast quadrant where lesion located under mammography	0.918	0.893	0.107	0.107	0.905
• Breast laterality where lesion located under mammography	0.940	0.933	0.067	0.067	0.936
• Ultrasound date	1.000	1.000	0.000	0.000	1.000
• Type of lesion under ultrasound	0.978	0.897	0.103	0.103	0.935
• Breast clock position where lesion located under ultrasound	0.982	0.933	0.067	0.067	0.957
• Breast laterality where lesion located under ultrasound	0.947	0.967	0.033	0.033	0.957
• Size of lesions observed from ultrasound	1.000	1.000	0.000	0.000	1.000
• BIRADS density	1.000	1.000	0.000	0.000	1.000
• BIRADS category	1.000	1.000	0.000	0.000	1.000
Overall	0.957	0.940	0.043	0.063	0.948

Rule-based NLP algorithm

The integration of the R package “lubridate” eased the process of examination date extraction. However, in some cases, false extraction of the date can occur. For instance, in the case of a date reported as “1.2.18”, the month value can be interpreted as 1 or 2, depending on whether the month or the day is checked first. Consequently, this leads to an increase in the false positive, resulting in reduced precision.

During the lesion type extraction process, the NLP algorithm performs pattern matching within the input text using predefined lists. Here are some examples of false positive and false negative cases that occurred throughout development of the NLP algorithm (Figure 7). In Example 1, the algorithm incorrectly extracted the lesion type as “with calcifications” for a lesion with a size of 1.3 x 0.8 x 0.8 cm in S6. However, the correct extraction should be “hypoechoic lesions (S2) with calcifications (S6)”. In another false positive example (Example 2), the sentence boundary detection failed to separate the lesion type for two lesions because only one breast laterality was mentioned. As a result, the two types were combined as “heterogenous fibroglandular density heterogenous architectural distortion scattered”. In Example 3, the algorithm was unable to extract the type due to the term “dermal fat graft” not being included in the ACR BIRADS lexicon and rarely appearing in the training data. Hence, it was not considered in the predefined list. Frequently occurring keywords that were not in the ACR BIRADS lexicon were added to the rules based on the training data to minimize false negatives. In Example 4, the reporting itself can be unclear, as the keyword “septated cyst” is mentioned between lesion at the 6 o’clock and 5 o’clock positions, leading the algorithm to identify both lesions as “cyst with septation”.

Figure 7.

Example of sentences in free-text report.

Multiple lesions can be present on the same breast, and the radiologist may only mention the breast laterality once before reporting the multiple lesions observed. In Example 1, there are two same-type lesions with different sizes located on the same breast as reported in S4 and S5. The response for the data element breast laterality is stated in S1, while the lesion type and the clock position are reported in S2 and S3. To correctly extract the responses presented in different sentences, the algorithm stores the most recently identified terms that describe the laterality and clock position from the sentences into a temporary space. Therefore, when the responses for laterality and clock position are not available in the sentence that contains other information about the lesion, such as the lesion size, the value held by the algorithm will be set to the current lesion.

In some cases, radiologists report the BIRADS density in numerical or narrative format instead of using the category system from A to D. Henceforth, the algorithm is designed to automatically convert the numeric value to an alphabetic value in the sentence where the keywords “density” and “BIRADS” are present. For the BIRADS density described in narrative format, the algorithm converts it into an alphabetic value by searching for keywords that describe each BIRADS density, as shown in Table 5.

The proposed NLP algorithm is capable of analyzing sentences with more complex structure, including those that involve multiple lesions with different sizes, position, and type within the same sentence. Figure 8 demonstrates an example where the algorithm successfully identifies the boundary between two different lesions when there are two measurements and the keyword “and” is present between two clock positions in a single sentence.

Figure 8.

Multiple size, type and position extraction in same sentence.

Graphical user interface for rule-based NLP algorithm

The interface of the text mining program consists of two main panels (Figure 9). The left panel is responsible for displaying the original inputted breast radiology report, while the right panel shows the summarized synoptic breast radiology report generated by the NLP algorithm. Although the displayed input report may contain some meaningless characters due to the exporting process across different platforms, this does not affect the extraction process. These unwanted characters are removed in the preprocessing step.

Figure 9.

Graphical user interface of rule-based NLP algorithm.

Predicting BIRADS category using variables extracted by NLP algorithm

Feature selection with RFE with 5-fold cross validation

RFE has identified that all 14 features are significant in predicting the target variable, BIRADS category value. These features collectively contribute to the best predictive model with the highest accuracy of 0.86 and a kappa value of 0.81. Table 9 shows the details of the accuracy and kappa value of each set of variables after RFE. The 14 significant features identified are:

1. Calcification;

2. Composition;

3. BIRADS density;

4. Number of finding;

5. Density;

6. Calcification distribution;

7. Associated;

8. Margin;

9. Symmetrical;

10. Lesion;

11. Morphology;

12. Malignancy;

13. Shape;

14. Mass.

Table 9.

Resampling performance over subset size.

Variables	Accuracy	Kappa	Accuracy SD	Kappa SD	Selected
1	0.4763	0.1881	0.0551	0.0965
2	0.5896	0.4051	0.0451	0.0633
3	0.6640	0.5164	0.0429	0.0515
4	0.7556	0.6665	0.0303	0.0403
5	0.7554	0.6698	0.0287	0.0493
6	0.7643	0.6796	0.0291	0.0471
7	0.8078	0.7405	0.0326	0.0416
8	0.8167	0.7513	0.0157	0.0185
9	0.8122	0.7487	0.0582	0.0771
10	0.8209	0.7590	0.0180	0.0251
11	0.8209	0.7574	0.0230	0.0328
12	0.8426	0.7870	0.0258	0.0323
13	0.8514	0.7985	0.0310	0.0400
14	0.8603	0.8100	0.0144	0.0207	*

These variables have been determined to have a strong association with the target variable and play an important role in accurately predicting the BIRADS category value.

Model evaluation

After features selection, various predictive models are trained using these features extracted from mammogram findings. The performance of each predictive model is illustrated in Table 10, showcasing their respective evaluation metrics and results that assess the accuracy and effectiveness of the models in predicting the BIRADS category value.

Table 10.

Model accuracies.

Model	Accuracy	Kappa	Precision	Recall	F1 score
Decision tree	0.7826	0.6918	0.7140	0.8288	0.7671
Random forest	0.9275	0.9017	0.9478	0.9322	0.9399
Multinomial logistics regression	0.8261	0.7530	0.8882	0.8103	0.8475
Extreme gradient boosting	0.8841	0.8479	0.9448	0.8850	0.9139
Multiple layer perceptron	0.8500	0.7975	0.8787	0.8600	0.8692

Among the five predictive models, the random forest model achieved the highest accuracy of 0.9275, followed by the extreme gradient boosting model (0.8841), multiple layer perceptron models (0.85), and multinomial logistic regression model (0.8261), and the decision tree model with the lowest accuracy of 0.7826. These predictive models were trained to predict the BIRADS category value ranging from 1 to 5. The BIRADS category value 0 and BIRADS category value six were excluded from the analysis for two main reasons. Firstly, datasets with BIRADS categories 0 and six comprised a small portion of the overall dataset. Secondly, BIRADS category 0 indicated an incomplete mammogram, while category six required confirmation through a biopsy. Overall, these findings demonstrate the performance of the predictive models in accurately predicting the BIRADS category value, while considering the specific exclusions and limitations of the dataset.

Discussion

The rise of EMRs has generated enormous data that is rich in information. However, most medical reports are written in narrative format and separated into different sections, rather than reporting each data element in distinct categories. Consequently, clinicians require more time to interpret and determine the subsequent treatment process. Clinical data stored in free-text have different database structures from structured data, which increases the difficulty of performing data mining. Therefore, NLP, whether rule-based or machine learning-based techniques, provide an opportunity in the clinical domain to reduce the costly manual training data annotation processes.^8,9

The highly unstructured radiology reports have limited opportunities for data mining due to the requirement of structured data for most data mining methods. Henceforth, we propose a rule-based algorithm that aimed to reduce the workloads of the clinicians in the keypoint highlighting process and provides an opportunity for further analysis with minimal human interference. With the aid of the graphical user interface, clinicians can utilize the rule-based NLP algorithm to generate a synoptic breast radiology report in the auditing process without requiring any programming skills.

In order to facilitate the auditing and research purpose, achieving a high F1 score (with a value of 1 being the best and 0 being the worst) was crucial.⁴⁰ Our NLP algorithm yielded promising results in information extraction tasks, with a test set F1 score of 0.947. Data elements with straightforward and minimal ambiguity, such as a dates, BIRADS category, and BIRADS density, showed higher F1 score. The formats used to write these variables are limited, which eliminates the need for complex rules to extract the required information. Conversely, the lesion type, which can be described in a variety of terms, shows the lowest F1 score in both training and test data. The complexity and ambiguous reporting style and lexicon when describing a lesion increased the false positive and false negative values, consequently impacting the overall accuracy.

Lately, the increasing use of machine learning techniques has shown great potential in achieving higher accuracy for automated outcome classification, especially when large volume of data is available. Spandorfer et al. conducted a similar study where they structured unstructured computed tomography pulmonary angiography (CTPA) report. In the study, 91.6% of statement were correctly labelled by the algorithm.¹⁷ Although the algorithm showed promise in achieving a high accuracy in the automated labelling process, it primarily focused on providing section labels to sentences, which is considered the most fundamental structure. However, this approach necessitated the manual annotation of section labels on a large amounts of training data by a clinical expert.

The rule-based NLP algorithm continues to be prominent in clinical NLP, despite the growing popularity of machine learning approaches in the broader NLP community. Clinical NLP has distinct requirements compared to other domains, as machine learning-based NLP primarily focused on prediction, estimation, and association mining. In contrast, the rule-based system offers simplicity and facilitates interactive refinement through clinician feedback, making it easier to debug. Meanwhile, machine learning-based systems often operate as black boxes, making interpretation and correction more challenging. Consequently, rule-based NLP remained favoured in the clinical field.⁴¹

In a study carried on by Hammami et al., rule-based NLP was employed to extract cancer morphology codes from pathology reports, achieving an impressive accuracy of 98.14%.¹⁹ Another example of rule-based NLP is the extraction of family history information, including associations with diseases and the living status of the family members, from the EMRs.⁴² Bozkurt et al. compared different NLP algorithms, including deep learning, rule-based, and hybrid models for the classification of prostate cancer severity. Their findings indicated that rule-based model (accuracy=0.86) is outperformed by the deep model (accuracy=0.73) and a hybrid model (accuracy=0.75).¹⁸ Rule-based NLP is not limited to classification tasks alone; it can also be utilized in medical diagnosis systems to suggest diagnosed diseases from user’s self-input.⁴³

Although rule-based NLP relies on handcrafted rules, it offers the advantage of incorporating domain knowledge from knowledge bases or experts. In contrast, machine learning-based NLP requires a large set of manually annotated and well-structured training data to ensure the accurateness in a given task. Rule-based NLP, on the other hand, reduces the need for costly manual annotation process required in machine learning-based techniques. For example, the labels for training data in a model that performs smoking status and hip fracture classification are generated by the rule-based NLP.¹¹

In this study, we demonstrated the real-world application of our NLP algorithm in assisting the predictive analytics process. We developed five predictive models for BIRADS category prediction based on mammogram features extracted using the NLP algorithm. All variables were found to be significant, resulting in an accuracy of 0.8603 after performing features selection using RFE methods. In addition, the kappa value of 0.81 indicated strong agreement with the data reliability.⁴⁴ Among the models, the random forest model has successfully predicted the BIRADS category value with the highest accuracy of 92.75%. The other models also demonstrated accuracies ranging from 78% to 88%. These successful predictions of the BIRADS category provide further evidence of the effectiveness and feasibility of our NLP algorithm in data mining research.

The main limitation of this study is the use of small dataset from a single institution. The use of a closed-architectural third-party RIS makes obtaining the breast radiology report troublesome and laborious. The rules generated from the dataset, which is limited to a single hospital may not cover all possible scenarios. Besides, different terminology is applied in different medical centers, which can increase the false negative value. Therefore, further research is needed to minimize human workload by automating the process of retrieving and annotating the breast radiography report from the RIS. Furthermore, the exploration of increasing the dataset from multiple sources would improve generalizability and minimize algorithm bias.

Conclusion

The study presented demonstrates the practical application of NLP in routine clinical practice. The ideas presented here not only digitize free-text reports into a minable form, but also prospectively digitize data collection. NLP enables the conversion of narrative reports into a structured format automatically. However, human review and validation are still necessary on utilizing the NLP application in these clinical reports, which is significant for clinical documentation for care and medico-legal purposes. Take Europe as an example, the European Commission published a white paper emphasizing human dignity and privacy protection in medical AI. It aligns with their AI approach based on fundamental rights and values, recognizing the importance of these aspects in using medical AI technologies.⁴⁵

The utilization of AI in clinical domain including radiology field will soon be getting more popular in future. However, human expertise is still indispensable. Henceforth, a balance and cooperation between AI and human intelligence is important in solving problem with legal requirements. Sorantin et al. have conducted a comprehensive discussion on the distinctions between AI and human intelligence in problem-solving, highlighting the significance of achieving a balance and fostering cooperation. They introduce the concept of “explainable AI” as a means to exemplify this approach.⁴⁶

In conclusion, this work assists clinicians by extracting significant points from free-text reports and serves as a low-cost preliminary step in machine learning-based analysis or data mining. It enables the automated annotation process of training data, facilitating efficient and accurate data labelling. By leveraging NLP, this study contributes to the advancement of clinical practice, improving the extraction of meaningful insights from free-text reports. Furthermore, the integration of NLP techniques in machine learning-based studies opens up new possibilities for automated annotation and analysis of clinical data, providing valuable support to healthcare professionals and researchers.

Supplemental Material

Supplemental Material - Natural language processing in narrative breast radiology reporting in University Malaya Medical Centre

Supplemental Material for Natural language processing in narrative breast radiology reporting in University Malaya Medical Centre by Wee Ming Tan, Wei Lin Ng, Mogana Darshini Ganggayah, Victor Chee Wai Hoe, Kartini Rahmat, Hana Salwani Zaini, Nur Aishah Mohd Taib, and Sarinder Kaur Dhillon in Health Informatics Journal

Footnotes

Acknowledgements

The authors thank the Breast Cancer Research group (patients, physicians, nurses, technical staff, research assistants and hospital administration staff at UMMC) for providing relevant data to conduct this study.

Author contributions

Conceptualization, W.-M.T., W.-L.N., M.D.G., N.A.T., V.C.-W.H., K.R., H.S.Z. and S.K.D.; methodology, W.-M.T. and S.K.D. validation, W.-M.T., W.-L.N., M.D.G.; formal analysis, W.-M.T.; resources, H.S.Z. and W.-L.N.; writing—original draft preparation, W.-M.T.; writing—review and editing, W.-M.T. and S.K.D.; supervision, S.K.D.; project administration, S.K.D., K.R. and N.A.T. All authors have read and agreed to the published version of the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical statement

ORCID iD

Sarinder Kaur Dhillon

Supplemental Material

Supplemental material for this article is available online.

References

National Cancer Institute . Definition of electronic medical record - NCIDictionary of Cancer Terms. https://www.cancer.gov/publications/dictionaries/cancer-terms/def/electronic-medical-record (nd, accessed 15-Nov-2021)

Peter

Joshua

. EMR vs EHR – what is the difference? - health ITBuzz, 2011. www.healthit.gov/buzz-blog/electronic-health-and-medical-records/emrvs-ehr-difference (accessed: 15 Nov 2021).

Shinozaki

. Electronic medical records and machine learning inapproaches to drug development. Artificial intelligence in oncology drug discovery and development. London, England: IntechOpen, 2020.

Wallis

Mccoubrie

. The radiology report - are we getting the messageacross? Clin Radiol 2011; 66: 1015–1022.

Ganeshan

Duong

Probyn

, et al. Structured reporting in radiology. Acad Radiol 2018; 25: 66–73.

Kalra

Tizhoosh

. Automatic classification of pathology reports using TF-IDF features. arXiv 2019; arXiv:1903.07406.

Hornberger

. Electronic health records: a guide for clinicians and administrators. JAMA 2009; 301: 110.

Kumar

Bhatia

. Text mining: concepts, process and applications. J.Glob. Res. Comput. Sci 2013; 4: 2013.

Kim

Ohk

Moon

C-S

. trend analysis by using text mining ofjournal articles regarding consumer policy. New Phys. Sae Mulli 2017; 67: 555–561.

10.

Jain

Prajapati

. NLP/deep learning techniques in healthcare for decision making. Prim Health Care Open Access 2021; 11: 1–4.

11.

Wang

Sohn

Liu

, et al. Aclinical text classification paradigm using weak supervision and deeprepresentation. BMC Med Inf Decis Making 2019; 19: 1.

12.

Pham

Névéol

Lavergne

, et al. Natural language processing of radiology reports for the detectionof thromboembolic diseases and clinically relevant incidental findings. BMC Bioinf 2014; 15: 266.

13.

Rink

Roberts

Harabagiu

, et al. Extracting actionable findings of appendicitis fromradiology reports using natural language processing. AMIA Jt. Summits Transl.Sci. proceedings 2013; 2013: 221.

14.

Wang

Mehrabi

Sohn

, et al. Natural language processing of radiology reports for identification of skeletal site-specific fractures. BMC Med Inf Decis Making 2019; 19: 73.

15.

Elliot

. Natural language processing to identify ureteric stones inradiology reports. J. Med. Imaging Radiat. Oncol 2019; 63: 307–310.

16.

Lang

Deng

, et al. Analysis of stroke detection during the COVID-19 pandemic using natural language processing of radiologyreports. Am J Neuroradiol 2020; 42: 429–434.

17.

Spandorfer

Branch

Sharma

, et al. Deep learning to convert unstructured CT pulmonary angiographyreports into structured reports. Eur. Radiol. Exp 2019; 3: 37.

18.

Bozkurt

Paul

Coquet

, et al. Phenotyping severity of patient‐centered outcomes using clinicalnotes: A prostate cancer use case. Learn. Heal. Syst 2020; 4: e10237.

19.

Hammami

Paglialonga

Pruneri

, et al. Automated classification of cancer morphology from Italianpathology reports using natural language processing techniques: a rule-basedapproach. J Biomed Inf 2021; 116: 103712.

20.

American College of Radiology . Breast imaging reporting & data system. www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Bi-Rads (2013, accessed 18 May 2021).

21.

R Core Development Team . R: a language and environment for statisticalcomputing. Vienna, Austria: R Foundation for Statistical Computing, 2013.

22.

Gruber

Rueckert

Kagan

, et al. Measurement of tumour size with mammography,sonography and magnetic resonance imaging as compared to histologicaltumour size in primary breast cancer. BMC Cancer 2013; 13: 328.

23.

Zhao

Zou

Geng

, et al. Limitations of mammography in thediagnosis of breast diseases compared with ultrasonography: a single-centerretrospective analysis of 274 cases. Eur J Med Res 2015; 20: 49.

24.

Grolemund

Wickham

. Dates and times made easy with lubridate. J Stat Software 2011; 40: 1–25.

25.

Python Software

Foundation.

Python language reference. version 2.7.18. www.python.org/downloads/release/python-2718/ (2020, accessed 26 Oct 2021).

26.

Lundh

F. 24.1.

Tkinter — Python interface to Tcl/Tk — Python 2.7 2 documentation, https://python.readthedocs.io/en/v2.7.2/library/tkinter.html (2013, accessed 27 Oct 2021).

27.

Kuhn

Wing

Weston

, et al. Caret: classification and regression training, http://cran.nexr.com/web/packages/caret/index.html (2017, accessed 27 October 2021).

28.

Bahl

Hellack

Balas

, et al. Recursive feature elimination in random forestclassification supports nanomaterial grouping. NanoImpact 2019; 15: 100179.

29.

Brian

Classification and regression trees. http://cran.nexr.com/web/packages/tree/index.html (2016, accessed 2 Jun 2022).

30.

Chen

Xie

Wang

, et al. Acomparative study of logistic model tree, random forest, and classification andregression tree models for spatial prediction of landslide susceptibility. Catena 2017; 151: 147–160.

31.

Liaw

Wiener

. Classification and regression by randomForest. R News 2002; 2: 18–22.

32.

Breiman

Cutler

. Random forests - classification description. https://www.stat.berkeley.edu/˜breiman/RandomForests/cc_home.html (2004, accessed 27 Mar 2021).

33.

Venables

Ripley

. Linear statistical models. Modern AppliedStatistics with S. New York: Springer, 2002, pp. 139–181.

34.

Osman

AIA

Ahmed

Chow

, et al. Extremegradient boosting (Xgboost) model to predict the groundwater levels inSelangor Malaysia. Ain Shams Eng J 2021; 12: 1545–1556.

35.

Mustapha

Saeed

. bioactive molecule prediction using extremegradient boosting. Molecules 2016; 21: 983.

36.

Chen

Benesty

, et al. xgboost: ExtremeGradient Boosting. R package version 1.3.2.1. http://cran.nexr.com/web/packages/xgboost/index.html (2018, accessed: 21 May 2021).

37.

LeDell

Gill

Aiello

, et al. h2o: R interface for the 'H2O' Scalable machine learning platform. https://docs.h2o.ai/h2o/latest-stable/h2o-r/docs/index.html (nd, accessed: 21 May 2021). https://Rproject.org/package=h2o

38.

Hornik

Stinchcombe

White

. Multilayer feedforward networks areuniversal approximators. Neural Network 1989; 2: 359–366.

39.

Jarek

caTools: tools: moving window statistics, GIF, Base64, ROC AUC. Etc. https://cran.r-project.org/web/packages/caTools/index.html (2021, accessed 01 Dec 2021)

40.

Grandini

Bagli

Visani

. Metrics for multi-class classification: AnOverview. arXiv 2020; arXiv:2008.05756.

41.

Wang

Rastegar-Mojarad

, et al. Clinical information extraction applications: a literature review. J Biomed Inf 2018; 77: 34–49.

42.

Almeida

Matos

. Rule-based extraction of family history informationfrom clinical notes. In: Proceedings of the 35th Annual ACM Symposium on Applied Computing, Brno, Czech Republic, 30 March - 3 April 2020, pp. 670–675. New York, Association for Computing Machinery.

43.

Omoregbe

NAI

Ndaman

Misra

, et al. Text messaging-based medical diagnosis using natural language processingand fuzzy logic. J. Healthc. Eng 2020; 2020: 1–14.

44.

McHugh

. Interrater reliability: the kappa statistic. Biochem Med 2012; 22: 276.

45.

Stöger

Schneeberger

Holzinger

. Medical artificial intelligence: the european legal perspective. Commun ACM 2021; 64(11): 34–36.

46.

Sorantin

Grasser

Hemmelmayr

, et al. The augmented radiologist: artificial intelligence in the practice of radiology. Pediatr Radiol 2022; 52(11): 2074–2086.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.11 MB