Sage Journals: Discover world-class research

Abstract

The current study reduced the time lag between performing a diagnostic assessment and identifying a critical finding in CT and MRI exams through improving radiographers’ abilities to identify those critical findings. Radiographers’ diagnostic assessments in CT and MRI exams were used to develop a mobile training application with the aim to improve radiographers’ awareness of critical findings. The current research used data analytics to examine radiographers’ interpretation of imaging studies from a privately owned medical group in Israel. During the project, the radiographers’ ability to identify critical findings improved. Implementation of the mobile training program yielded positive results where the knowledge gap was reduced and time to identify critical cases was decreased. Specifically, this study showed that radiographers can be trained in ways that enhance their involvement with radiologists to provide high quality services and improve treatment Ultimately, this gives patients higher quality of care and safer treatment.

Keywords

CT Scans mobile training MRI scans radiologists radiographers

Introduction

Healthcare researchers and practitioners suggest improved patient care can be derived from access to¹ large volumes of medical data coupled with improved data analytics.² The effort to increase quality of care can be further improved with medical test classification and management. For instance, one potentially beneficial use of big data applications is to reduce errors in diagnostic assessments by healthcare providers. Researchers have noted the existence of errors in radiological diagnostic assessments and offer suggestions for improvements.^3–7 The current study developed a mobile training application with the potential to reduce lag time between performing tests and identifying critical test results by improving radiographers’ diagnostic assessments of CT and MRI Scans. The current research used data analytics to identify gaps in radiographers’ ability to recognize critical findings in imaging studies from a privately owned medical group in Israel then assessed the effect of intervention on the radiographers’ ability to identify critical tests results. Hence the purpose of this study was to map differences between radiographer and radiologist classification regarding severity level of imaging tests, and to reduce these differences by improving radiographers’ abilities to rapidly identify critical findings in the test.

The study was driven by three research hypotheses:

There is a difference between classifications done by radiographers and radiologists in CT scans.

There is a difference between classifications done by radiographers and radiologist in MRI scans.

Intervention through integrated training technology reduces differences between classifications done by radiographers and radiologists in imaging tests.

Changes in the diagnostic procedure in radiology

Radiological tests such as CT and MRI scans are fundamental to medical diagnosis. The tests are performed by radiographers (technicians) and later read by radiologists (physicians). In the past, radiologists in hospitals were involved in imaging tests prior to an interpretation phase. Radiologists would review the referrals and authorize them prior to performing the scans, and occasionally be present alongside the radiographer during screening. ^4,5 In addition, reading results often took place in direct contact with the clinical radiographer in a relatively short time span. This collaboration enabled the radiographer to learn more about performing tests and appraising the results.⁶

Generally, radiologists currently encounter CT and MRI scan images for the first time when they are tasked with making a clinical diagnosis. Unlike radiographers, radiologists usually make the interpretation without physically seeing the patient. Likewise, they exert no influence on the work procedures done beforehand. Radiographers, who act as hospital representatives, read the referrals, and are required to use their discretion regarding the test methods and processes, especially when the referrals come from physicians without sufficient expertise in clinical imaging. The radiographers are not responsible for identifying abnormal findings in the tests before they transfer the relevant information to the radiologists.⁷ However, they do have the right to initiate an expedited processing request for any concerning readings. The current study posits that radiographers can help identify critical cases sooner without being held responsible for the diagnosis. This could be especially helpful in ensuring critical cases receive priority attention or in situations where additional unexpected findings are revealed and require adjustments or immediate actions.

Early detection of critical ailments certainly will improve patient care. Although radiographers are trained in subjects such as human anatomy, physiology and pathology,^8–11 and in using safety protocols for imaging test procedures,¹² their evaluation of findings remains an informal activity. They are not trained to provide documentation for an evaluation. In many such cases, radiographers would have been able to detect and ensure early interventions for deadly diseases such as cancer.¹³ Another limitation of the radiographer’s role is a short patient interaction time. On average, a radiographer spends about five to ten minutes per patient. Short interactions can lead to inadequate review of test results.¹⁴ Therefore, researchers have noted that training radiographers to identify critical conditions in test results can help with early identification of pathological findings prior to the radiologist’s access to test results.^15–17

For many severe medical conditions, time factors play an especially critical role. This is particularly true in situations where waiting for a radiologist’s reading is not feasible and when an immediate life-saving identification may be necessary. The patient’s primary benefit is an efficient early identification of the condition which can enhance early-stage treatment. For instance, recent research showed that immediate identification of a pulmonary embolism by a radiographer shortened the time span between the radiographic test and meeting with the physician from 1.5 days to only 26 min¹⁸ As such, training programs to improve the diagnostic assessments of severe medical conditions by radiographers can lead to early life-saving interventions.^19–21

An additional benefit of radiographers’ diagnostic accuracy is financial savings. Based on 1000 simulated patients, radiographers’ reports reduced the diagnosis costs by £8500.¹⁵ After adding treatment-related costs, faster reporting by radiographers was still more cost effective than radiologists' reports and resulted in 1.4 Quality-adjusted Life Year per each 1000 patients who had undergone a radiological Chest X-Ray examination. A probability analysis suggests that radiographers' reports are more cost effective than radiologists’ reporting. The cost of diagnosis and treatment by radiographers was lower at £2,137,983 as compared to diagnosis and treatment costs by physicians at £2,142,299. The conclusion was that radiographers' reports were a cost effective alternative in lung cancer, and provided quick diagnosis to patients and saved costs of expensive treatments at more advanced stages of the disease, thanks to early diagnoses.¹⁵

In other examinations, such as hysterosalpingographies, the radiographer’s involvement can help the physician to be able to conduct and read the examinations in real time. This can complement the physician’s work, and thus shorten waiting times and reduce patient anxieties and uncertainties.²² Radiographers can utilize Diagnostic Reference Levels (DRL) values for chest, brain, and other X-rays after appropriate training programs.²³ or they can help interpret/decipher various mammograms and ultrasound images.⁴ Radiographers' interpretations of CT results can, for example, be precise, especially after receiving appropriate training.^24,25 Roe et al.²⁶ provided additional support for the claim that a radiographer’s involvement could be important and suggested the need for appropriate training. In that study, a system designed to check the precision of a radiographer’s notes regarding the test, vis-à-vis a physician’s notes, found a 98.5% precision rate in the radiographer’s notes. These notes from radiographers could help reduce medical errors and support physicians’ readings.²⁶

The preceding literature review emphasizes the value that radiographers can bring to patients in identification of crucial situations in real time. Despite such needs, the general, foundation education or training for the radiographers did not address the interpretation of radiography scans. In addition, it is important to note that the case for further training radiographers through IT-based tools does neither intend to reduce physicians’ expertise or skills nor attempt to replace physicians, but rather to enhance the rapid and accurate diagnosis interpretation in critical moments. In the digital age, training radiographers on the job using digital means can be an effective method. Research indicates that learning environments supported by digital imaging or simulations can be very effective.^27,28 Furthermore, research indicates that using mobile technology for imaging tests enables radiographers to conduct tests with high precision (87.69%). For example the ANIMATI DICOM Viewer enables accurate visual imaging.²⁹ Therefore, the current study seeks to close the gap by suggesting a safe way to improve radiographers’ diagnoses, therefore improving timely and accurate treatment, ultimately leading to high quality care and better patient outcomes.

Method

Background and preliminary actions

The study was conducted at the Assuta Medical Centers, a network of healthcare centers in Israel that includes eight hospitals and outpatient centers. The network serves an average of 92,000 patients each month. Two medical imaging tests are included in this study: computed tomography (CT) and magnetic resonance imaging (MRI). In a typical month, Assuta performs 17,000 and 7000 such tests, respectively. The study used a database, which exhibits Big Data characteristics in terms of data volume and variety, to identify the characteristics of the tests in which differences between radiographer and radiologist classification existed.

The current study comprised five stages. The first three stages are regarding data collection and analysis, which are described below. The fourth and fifth stages are regarding the radiographers training, which are described later in Mobile Training Development. In the first stage, radiographers (N = 118) were asked to perform an early evaluation of test results, before these tests were passed to the radiologists for interpretation (N = 98). The radiographers were asked to classify the tests into three types of findings: (1) “Normal range,” (2) “There is a presence of finding” (non-critical), and (3) " There is a critical finding." The classification was performed at the end of the test and documented in the patient’s clinical record. For documentation, a new dedicated field was built in electronic medical record (EMR).

At the end of the interpretation process, the radiologists classified the test into an assessment tool that rated the severity of the findings in five categories: (1) “Normal test”; (2) “Test has a finding, but no urgency to report to the referring physician”; (3) “Test contains a critical finding and must be reported to the referring physician within a week”; (4) “Test contains a critical finding and must be reported to the attending physician or patient within 24 h”; and (5) “Test contains a critical finding and immediate treatment is necessary.” In the usual procedure, in cases where the radiographer suspects or identifies a critical finding, s/he should contact the radiologist on duty immediately, describe the case thoroughly, and prioritize the order of tests based on criticality. The radiographer must also keep the patient remained at the testing facility until the radiologist answers.

In the second stage, at an organized conference for radiographers, we presented a new process of classifying findings in tests and the rationale for the program. As part of the training program, clinical cases were presented. Also, in the conference, we conducted a real-time interactive simulation game, where images of pathological CT scans were presented, and participants were asked to answer questions related to findings and classification. The game was a preview of a training program based on a mobile app being developed. The conference was attended by over 100 radiographers throughout the Assuta network, on two separate dates.

In the third stage, we collect and analyze data to understand the relationship between radiographers’ and radiologists’ classification. Data Collection to Data Validation and Anonymization delineates what we have done in the third stage.

Data collection

The dataset, derived from a large data repository, included 252,957 CT scans and 108,590 MRI scans over a period ranging from July 2018 to January 2020, from all Assuta centers. Information about the scans was extracted from the RIS® system (electronic medical imaging files), and from Tafnit®, an administrative system.³⁰ The data was retrieved and compiled into a SQL database.

Data cleaning

From the initial set, we performed veracity check. In the CT scans, we removed cardiology scans because radiologists were present with the radiographers while the scans were conducted; therefore, there was no potential early detection of findings by radiographers. We also excluded virtual colonoscopies due to data uniformity (all scans resulted in no findings), and hip comparisons and angular measurements in infants, which are for measurement purposes only (N = 5711). We omitted records that have no radiologist classification (N = 21), missing the radiographer’s name (N = 114), and the experimental scans performed by computer unit personnel (N = 326). Similarly, in MRI scans, we removed scans that were performed specifically for other studies, missing finding’s severity (N = 664), and missing radiologist’s classification (N = 815). After cleaning the data, the study retained 246,785 CT scans and 107,111 MRI scans.

Data Validation and Anonymization

Data went through a final validation process using two complementary methods. In the first method, we selected a random number of scans. These data were verified against the existing information in the patients’ clinical and administrative files by comparing manually data. The second method involved comparing quantities of various tests against existing operating reports. The comparison of information retrieved by two different parties indicated the number of retrieved records was correct.

Once validated, the information underwent anonymization (hiding identifying information about patients) and analyzed using Python³¹ and SPSS software.³² The analyzed information was subsequently transferred to Excel, and from there to InDesign for the purpose of simulating the data.³³

Tables 1 and 2 show the breakdown of CT and MRI tests and the types of classifications used by group.

Table 1.

Breakdown of radiographers’ assessment and radiologists’ classifications for CT Scans.

		Radiologist’s CT Scan Classification
Radiographer’s CT Scan Classification		1	2	3	4	5	Total Across Radiographer’s CT Scan Classification
Radiographer’s CT Scan Classification		No finding	Test with finding, no reporting	Test with finding, one-week reporting	Test with finding 24 hours reporting	Test with finding, immediate treatment	Total Across Radiographer’s CT Scan Classification
A	# of Cases	11,290	42,390	1,613	775	41	56,109
A	% of Total	4.57%	17.18%	0.65%	0.31%	0.02%	22.74%
B	# of Cases	21,791	156,565	6,031	5,428	422	190,237
B	% of Total	8.83%	63.44%	2.44%	2.20%	0.17%	77.09%
C	# of Cases	9	196	25	126	83	439
C	% of Total	0.00%	0.08%	0.01%	0.05%	0.03%	0.18%
Total Across Radiologist’s CT Interpretation	# of Cases	33,090	199,151	7,669	6,329	546	246,785
Total Across Radiologist’s CT Interpretation	% of Total	13.41%	80.70%	3.11%	2.56%	0.22%	100.00%

Table 2.

Breakdown of radiographers’ assessment and radiologists’ classifications for MRI Scans.

		Radiologist MRI’s Interpretation
Radiographer MRI Classification		1	2	3	4	5	Total Across Radiographer’s CT Scan Classification
Radiographer MRI Classification		No finding	Test with finding, no reporting	Test with finding, one-week reporting	Test with finding 24 hours reporting	Test with finding, immediate treatment	Total Across Radiographer’s CT Scan Classification
A	# of Cases	8,009	10,256	386	103		18,754
A	% of Total	7.48%	9.58%	0.36%	0.10%		17.51%
B	# of Cases	19,001	65,418	2,808	991	20	88,238
B	% of Total	17.74%	61.07%	2.62%	0.93%	0.02%	82.38%
C	# of Cases	15	75	11	14	4	119
C	% of Total	0.01%	0.07%	0.01%	0.01%	0.00%	0.11%
Total Across Radiologist’s CT Interpretation	# of Cases	27,025	75,749	3,205	1,108	24	107,111
Total Across Radiologist’s CT Interpretation	% of Total	25.23%	70.72%	2.99%	1.03%	0.02%	100.00%

Each test type used a scale created in the EMR with different purposes. These were mapped into a common scale for the current study. Chi-square tests validated the significant differences between the two groups after conversion (p < 0.01). Data regarding the tests include unique patient number, unique exam number, location and time of the exam, type of tests (Body, Body and Neurological, Musculoskeletal, and General), the primary radiographer and his/her classification, and the radiologist and his/her classification. As for the radiologists and radiographers, profile data collected includes gender, age, years of experience out of medical school, years of professional experience in hospitals, and specialties. It is important to note that the current study analyzes only the characteristics of the radiographers since the training implementation will focus on their actions.

Classification mapping between the radiologists and radiographers provided unprecedented insight into training needs and this was used in development of the mobile training application. Tables 3 and 4 show crosstab tables between radiographers’ and radiologists’ classification using only (1) No Finding, (2) Presence of Finding, and (3) Critical Finding for CT and MRI, respectively.

Table 3.

Breakdown of radiographers’ assessment mapped to radiologists’ classifications for CT Scans, where (1) is No Finding, (2) is Presence of Finding, and (3) is Critical Finding.

		Radiologist Matched Code
Radiographer Assessment		1	2	3	Total
1	# of Cases	53,680	1,613	816	56,109
1	% of Total	21.75%	0.65%	0.33%	22.74%
2	# of Cases	178,356	6,031	5,850	190,237
2	% of Total	72.27%	2.44%	2.37%	77.09%
3	# of Cases	205	25	209	439
3	% of Total	0.08%	0.01%	0.08%	0.18%
Total	# of Cases	232,241	7,669	6,875	246,785
Total	% of Total	94.11%	3.11%	2.79%	100.00%

Table 4.

Breakdown of radiographers’ assessment mapped to radiologists’ classifications for MRI where (1) is No Finding, (2) is Presence of Finding, and (3) is Critical Finding.

		Radiologist Matched Code
Radiographer Assessment		1	2	3	Total
1	# of Cases	18,265	386	103	18,754
1	% of Total	17.05%	0.36%	0.10%	17.51%
2	# of Cases	84,419	2,808	1,011	88,238
2	% of Total	78.81%	2.62%	0.94%	82.38%
3	# of Cases	90	11	18	119
3	% of Total	0.08%	0.01%	0.02%	0.11%
Total	# of Cases	102,774	3,205	1,132	107,111
Total	% of Total	95.95%	2.99%	1.06%	100.00%

Tables 3 and 4 depict the total accuracy of the respective tests and provides a visualization of the radiographer’s classifications based on the radiologists' findings. For example, in the CT table, we see the radiologist interpreted 6875 cases as grade 3 (2.79% of the total cases). From this number, the radiographer classified 816 as grade 1, 5850 as grade 2, and 209 as grade 3 to total 6785 cases. The values comparing the 2 groups shown in Tables 3 and 4 were statistically significant based on chi-square tests (p < 0.01). Given the MRI scans are more difficult to interpret than CT scans, a slight increase in accuracy was anticipated since a higher level of training had been administered. All misclassifications are candidates for improvement. However, to make the most effective training program, the targeted program should address the most pressing need.

The tables indicate where training can have the most impact: where radiographers render a score of “1–No Finding” while the radiologist rated the test result with a score of “3–Critical Finding,” as highlighted in red in Table 3 and Table 4. In the case of CT scans, 816 of such cases were identified, making up 0.33% of the total scans. Similarly, for MRI scans, this misclassification was recorded in 103 cases, making up 0.10% of the cases. These cases classified as life threatening by the radiologist and as normal by the radiographer were of primary interest. Even though the number of cases was low, correcting these mismatches are potentially lifesaving because it allows prompt responses by radiologists. Therefore, the training program focused on improving the rate of early identification of such cases.

Mobile training development

In the fourth stage of the project, a study group was established in which radiographers were presented with anonymous cases for discussion. The mobile training app consisted of two parts: pathology identification and rating of the finding severity. In other words, the participant was asked to diagnose the clinical situation that was presented to him in the mobile app, and then classify the severity of the finding. For example, the radiographer taking the survey had to see a tumor and classify whether to recommend urgent treatment. In this scenario, the radiographer would receive an image on the mobile device with a tumor situation represented. The training would prompt him to first identify the situation (in this example, a tumor) and then secondly require its classification in terms of severity. The severity could be A—no finding; B—there is a finding; or C—there is a life-threatening finding. If the finding is “C," the training would recommend that in real life, the radiographer should call the immediately radiologist and request that the patient stays at the hospital.

A sample case, which included images and the appropriate responses for participants as well as other pertinent information, was developed and presented via a hyperlink distributed to a dedicated WhatsApp group of radiologists. Figure 1 displays the technical description of the mobile training program.

Figure 1.

Technical description of building a mobile training program using big data analysis.

The group was established for the purpose of training and was built based on information produced in a dedicated report developed by Assuta human resources. The group included 118 radiographers and the head of the imaging department. After a few days, the case answer was provided to the group. Figure 2 provides a sample screenshot from the mobile app.

Figure 2.

Experiential mobile learning group for radiographers in the Assuta network.

After the proof of concept was ensured, the fifth stage of the project began. Here, the number of training cases were expanded. Ten cases were selected based on the informational analysis of the imaging tests and in collaboration with the Head of the Imaging and Radiology Division in Assuta. These cases were mismatches between classifications of radiographers and radiologists in critical situations. Two were general cases and eight were from the neurological field. In these cases, a representative image of the test was extracted from the PACS, Picture Archiving and Communication System, and two questions were formulated for the radiographer: (1) What is the clinical condition in the image? (e.g., bleeding or tumor); and (2) What is the severity of the findings? (e.g., “No findings,” “There are findings,” or “There are critical findings”). After characterizing the cases, any remaining identifying details of the patient were removed, and an application was built in the Nemala software environment that included presenting the case and questions to the radiographer.³⁴

The mobile test interventions through integrated technology were tested to determine if it reduced the differences between the classifications of radiographers and radiologists in imaging tests. The radiographers were asked to classify an EMR presented to them. This classification was compared with that of the radiologist. The radiographer was then trained to identify the differences and congruences with the radiologist’s classification of the EMR.

The intervention regarding the ten cases was continued for 17 months. Each radiographer completed the training during this time period. Classification of test findings is not part of the radiographers’ job definition, but the study was designed to influence the radiographer ‘s correct classification of critical findings, and therefore sample size was calculated based on the assumption that at least 50% of radiographers would diagnose critical findings correctly. This was used to calculate the required sample size with a significance level of 5% and an intensity of 80%. Analysis showed that 385 radiographer classifications were required. In practice, 88 radiologists of 118 on staff classified tests 444 times with the method of response being distributed as described in Table 5. It follows that 10 training cases for examining integrated technology means that tests have been classified 444 times. This provides the power required.

Table 5.

Rate of identification of the findings and appropriate classification by the radiographers.

Survey num	Respondents num	Type of finding	Classification	Num of radiographers which answer finding answer finding question	Num/% correct findings answer	Num of radiographers which Classified The finding	Num/% correct classification answers
1	88	Obstruction	Life-threatening	88	63 / 72%	87	35 / 40%
2	62	Tumor	There is a finding	60	16 / 27%	60	28 / 47%
3	35	Malignancy	There is a finding	35	31 / 89%	35	27 / 77%
4	22	Tumor	There is a finding	22	12 / 55%	22	15 / 68%
5	53	Pulmonary embolism	Life-threatening	53	38 / 72%	53	38 / 72%
6	30	Tracheal mass	Life-threatening	30	28 / 93%	30	22 / 73%
7	55	Brain bleeding with mass effect	Life-threatening	54	33 / 61%	54	49 / 91%
8	47	Stent in the descending frontal aorta	There is a finding	47	29 / 62%	47	26 / 55%
9	32	Spinaloid sinusitis	Life-threatening	32	22 / 69%	32	5 / 16%
10	20	Acute subvascular infarction of large blood vessels	Life-threatening	20	14 / 70%	20	18 / 90%
Total	444			441	286 / 64%	440	263 / 59%

Results

As described in the methods section of this article significant differences in the rates of detection in critical CT and MRI findings existed between the groups of radiographers and radiologists. This information was used to develop the mobile training application and focus on the most impactful types of cases. Therefore, H1 and H2 were accepted.

H3 was evaluated considering administration of the ten mobile training surveys. The response rate for the surveys was approximately 37%. An average of 64% identified the findings correctly and 59% classified the findings of severity correctly.

Data was collected to determine if misclassified critical findings (MCF) results showed a steady decrease in the misclassified critical findings (MCF) throughout the study period. This was found to be the case with MCFs dropping from 15.3% to 4.5% by the end of the study period. Also, the rate of cases in which radiologists identified critical findings and acted with the radiologist to perform rapid interpretation increased from 84.7% at the beginning of the study to 95.5% at the end.

Differences in the rate of detection were compared using a chi-square test. The test compared critical findings between the first half year of the study and the half year of the study. Significant differences were found between the groups (χ2 = 204.9, p = 0.000). Therefore, H3 was accepted. Figure 3 provides the results timeline.

Figure 3.

Rate of misclassified critical findings versus rate of adjustments in CT scans on the study timeline.

In a follow-up analysis, a multivariate logistic regression model was developed to determine if critical CT scans detection were significantly influenced by types of tests, hospitals, test shifts, gender of the radiographer, age of the radiographer, professional seniority, and the phenomenon of non-compliance in CT scans existed. First, Spearman’s rho correlation test was used to test for multicollinearity. This test indicated a high correlation between the variables: age of the radiographer and professional seniority. (R² = 0.727, p < 0.001, N = 235.978). Therefore, the logistic model will exclude the professional seniority in the model. Table 6 presents the results of the logistic regression model. Likewise, the same model is conducted for MRI, which results in only two significant variables: hospitals and age of the radiographer. The result is shown in Table 7.

Table 6.

Significant variables impacting identification of critical CT scans.

Variable		Sig.	Risk ratio	C.I.f 95%
Variable		Sig.	Risk ratio	Min	Max
	Scan type	0.009	1.024	0.996	1.053
	Hospital	0.000	0.961	0.955	0.967
	Shift	0.000	0.704	0.624	0.793
	Radiographer gender	0.000	1.951	0.642	2.318
	Radiographer age	0.000	1.04	1.032	1.049

Table 7.

Significant variables impacting identification of critical MRI scans.

Variable		Sig.	Risk ratio	C.I.f 95%
Variable		Sig.	Risk ratio	Min	Max
	Hospital	0.928	1.000	0.992	1.009
	Radiographer age	0.009	1.769	1.155	2.708

Discussion

Today, a radiologist’s encounter with an imaging scan is done at the time of interpretation and has no effect on the preceding work processes. Prior to the official clinical interpretations, the radiographer administers and reads the initial test but is required to exercise discretion as to their opinion on test findings. The radiographer’s assessment activity is informal and undocumented. Sometimes, there are cases where critical treatment could begin faster and perhaps before the patient’s release had findings been identified earlier. The current study was motivated by this possibility: using a radiographer as an early warning mechanism to increase speed with which critical issues are identified. Therefore, this study found there are differences between radiographer and radiologist classification in reading CT and MRI scans and there is a need to reduce the differences and improve radiographers’ awareness of critical findings via a training app. The study is of great importance because it has identified the most significant areas of potential instruction for radiographers based on data analysis and then provided opportunities for focused training to raise improvement identification of tests with critical findings.

The study found a gap in the interpretation of CT and MRI scans between radiographers’ and radiologists’ interpretations. For each scan, the overwhelming majority cases are as follows: 77.09% of the CT scans were marked by the radiographer as having findings while the radiologists classified 94.11% of the tests as having no findings. A similar picture was obtained for MRI scans: 82.38% and 95.95%, respectively. The findings showed that the discrepancies in the radiographer’s reports represent a tendency to report the safest option. In other words, in a state of uncertainty, radiologists prefer to mark “existing findings” rather than make a critical mistake and mark one of the two extreme options (“no findings” or “critical findings”).

As discrepancies were found between the reports, the study focused mainly on cases where radiologists classified “no findings,” while radiologists interpretated the scan as a case with a critical finding, which is a critical finding and should be reported to the attending physician/patient within 24 h. This is the most significant factor when it comes to the quality of patient care and therefore the study chose to focus on this section of the tests.

To do this, the study pulled out the tests in which the radiologist interpretation included a critical finding. The distribution of radiographer classifications was then examined. In 12% of CT scans with critical findings (816 out of 6785) and 11% of MRI scans with critical findings (103 out of 896 tests), radiographers classified the tests as normal. This is a significant and important revelation that points to the most important gap between radiologists and radiologists.

Between hospitals, the rate of discrepancy was mainly distributed between two out of six hospitals (Hospital 1 with 49% and Hospital 3 with 25% of all discrepancies). We continue to explore the differences found here. Timewise, in the morning shift, the rate of discrepancies was the highest (Morning shift—15%, evening shift—10%, night shift—7% of all tests with critical findings). Radiographers on evening and night shifts appear to be more available to detect critical findings. A possible explanation pertains to the flow of patients in a particular part of the day. Mornings are much busier with movement of people around and inside the examination room (incoming and outgoing secretaries/families, etc.) in a way that may cause distraction and a decrease in the level of alertness. In the evening and at night, there is fewer administrative staff in the test area and the patients tend to arrive as a single party.

Radiographers-characteristic-wide, the rate of misclassification was slightly higher among female (women 13%, men 12%). This is a small difference to which we have yet to find an explanation. The rate of misclassification among radiographers in the age group over 31.6 years was twice as high (18%) as in the age group below 31.6 years (9%). The highest rate of discrepancies was among radiographers with over 5 years of professional experience (professional experience more than 5 years—21%, professional experience up to 3 years—8%, professional experience between 3 and 5 years—7%). These findings point out the need for special attention in future training for adult radiographers with over 5 years of professional experience.

For MRI scans, a multivariate analysis of the independent variables found that hospital and age of the radiographer, which again is highly correlated with seniority, significantly affected the variable depending on the phenomenon of mismatch in the scans. In hospital number 5, the rate of misclassification was 17% of the total tests with critical findings in the hospital, and in hospital number 1, 11%, respectively. For the age of the radiographer, in the age group above 35.6 years the rate of non-compliance was 11%, compared with 7% in the age group below 35.6 years. These findings reinforce the need to initiate a training program that considers the hospital and the age of the radiographer.

Another finding relates to the fact that the CT field in neurology is the clinical field with the most tests. This was found to be the area with the highest rate of discrepancy between the radiographer classification and the radiologist classification. This is the most important and central finding on which a mobile-based training program was built that presents critical cases in the field of neurology. The training radiologists was unique because it was based on a big data analysis that could focus attention on knowledge gaps in a central area where the phenomenon of non-compliance is most severe. Previous studies have dealt with clinical situations^15–18 that have focused on specific areas of training that are ranked according to specific clinical importance and not necessarily from analyzing data that identify knowledge gaps of the radiologists in the overall process.

Overall, the study determined that mobile training intervention can help reduce discrepancies in critical case identification. The literature shows that training a radiographer while working with the help of digital means can be an effective way to achieve the training goal.^27,28 Evidence that use of mobile technology in favor of raising the quality of medical care is beginning to accumulate.²⁹ The use of mobile technology for the purposes of training the radiographers in this study, adds to this evidence.

At the time this study was published, the app had been active for 17 months. The response rate to the training program in the mobile app was 37%. The training was performed using CT scans but was given to both CT and MRI radiographers. The differences in response rate may have been influenced by test type. The MCF rate was examined in CT scans during the study timeline. The MCF rate gradually decreased from 15.3% on July 2018 to 4.5% at the end of the study period, January 2020. Also, the proportion of cases in which the radiographer identified critical findings and acted with the radiologist to perform rapid interpretation increased from 84.7% at the beginning of the study to 95.5% at the end. The differences were found to be significant in Chi-square test (p = 0.000).

The study showed that radiographers learned to detect critical cases on CT and report them faster to a radiologist. Moreover, from informal conversations with radiologists, it appears that the radiographers have become more involved and more knowledgeable about test results and that the number of referrals regarding patients has increased significantly and radiologists have contacted them much more often than before. This finding was found to be suitable for parallel works in the literature that showed that interpretations of CT results performed by radiographers can be accurate, especially after receiving appropriate training.^24,25

Many studies show the need to validate the artificial intelligence recommendation (whether there is a finding). Validation is performed by radiologists and contributes to the ongoing learning processes of artificial intelligence.^35–37 During this study, a new process of testing the classification was implemented by the radiographers and a training system for detecting abnormal conditions in the test was implemented. In this way, the professional composition of the factors that help validate artificial intelligence systems can be expanded.

Radiographers can also help deepen the feedback process between artificial intelligence and the human factor: a radiographer who recognizes test findings can help validate the algorithms, and a radiographer who does not recognize-the algorithm can help him deepen knowledge. All of these advance the human-machine interface in the world of radiology towards an era in which re-feeding is an integral part of the changing work processes in the world of radiology. Recently, as part of a commercial collaboration, Assuta tested a system of a commercial company that aims to help the radiologist identify abnormal findings in tests in differentiated clinical situations. The radiographer classification is another tool for assessing the system’s effectiveness.

Conclusion

Advances in medical practice and technology have resulted in greater demand for both CT and MRI scans. Correspondingly, an increase in tests with critical findings that need to be identified quickly has emerged. The current study presented a series of data analysis steps to identify areas can be improved with training. The training method proposed in this study shortens the response time in critical cases and leverages radiographers’ knowledge as well as assists in implementation of AI-based tools in the world of radiology. The mobile training app exposed radiographers to new situations and provided insight into critical case identification to encourage quick intervention steps. The feedback loop between radiographer and radiologist can be reduced in critical situations ultimately saving lives where time reduction is imperative.

It is important to remember that the study neither intends to suggest that a radiographer become a radiologist nor to ask them to take on higher levels of responsibility for interpretating tests. However, their involvement is important for providing high quality service, improving medical care and patient satisfaction.

Likewise, it is important to continue investigating the findings of the mobile training and its effects on the accuracy of the radiographer’s assessment. In the future it is also important to investigate the influence of the variance in readings done by various radiologists and strive to achieve a gold standard in this area.

Limitations of the study

It is possible that gaps discovered between radiographer’s classifications and radiologist’s readings stem from clinical optimizations; the radiologists may tend to classify presence of findings only in cases of significant findings as opposed to the radiographers who may classify every finding. It would be appropriate to build an operational model of the ideal radiologist that would provide the ability to assess and compare between different radiologists and different types of findings. In addition, different radiologists exhibit variances in classifying a scan, but in view of this study’s random distribution and the fact that there is no affinity between the radiologists and the radiographers, we would expect to see a correlation between them. In any case, the analysis of the variance among the radiologists is a well-known issue and remains outside the scope of this study. Another limitation is regarding contact time. Previous literature suggests that short contact with the patients, five to ten minutes on average per patient, may lead to an insufficient review of the test results.¹⁴ This research did not account for the duration due to incomplete data record. However, it appears that despite the short time documented in the literature, appropriate training can have a significant effect on identifying the findings in the test.

Future extensions

Artificial intelligence is becoming more present in the world of radiology and its purpose is to accelerate or improve the accuracy of radiologists in the interpretation process. The main strengths of artificial intelligence-automation, accuracy, and objectivity-are expressed in learning and evolving algorithms, which are embedded in the radiologist’s work environment. Because it is supposed to integrate naturally into the work environment, and be alive and developing, it must include processes for receiving feedback from radiologists for continuous learning and improving application.^35,36 Feedback includes independent verification of data the algorithm has not seen before and in-depth test of false and negative results.³⁷

Documentation of the test findings by the radiographers, and the processes of training the radiologists to identify the findings as presented in this work, have the power to contribute to the learning and re-feeding processes of artificial intelligence algorithms and give validity to the radiologists’ classification of the test.

Footnotes

Author contributions

All authors have made a substantial, direct, intellectual contribution to this study.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical approval

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2013.

ORCID iDs

Ace Vo

Iris Reychav

References

Burgoon

Stoner

Bonito

, et al. Trust and deception in mediated communication. In: Nunamaker

(ed). Proceedings of the 36th Annual Hawaii International Conference on System Sciences, NW Washington, DC, January 6–9, 2003; 2003, pp. 1–11.

Reimer

Madigan

. Veracity in big data: how good is good enough. Health Informatics J 2019; 25: 1290–1298.

Ringler

Goss

Bartholmai

. Syntactic and semantic errors in radiology reports associated with speech recognition software. Health Informatics J 2017; 23: 3–13.

Culpan

. Radiographer involvement in mammography image interpretation: A survey of United Kingdom practice. Radiography 2016; 22: 306–312.

Mangahy

Altolub

Batarfi

, et al. Assessment of clinical competences and self-confidence of radiologist and technologist at asir region hospitals, Saudi Arabia. Egypt J Hosp Med 2018; 71: 2544–2552.

Culpan

A-M

Docherty

, et al. Radiographer reporting: a literature review to support cancer workforce planning in England. Radiography 2019; 25: 155–163.

Lam

Egan

Baird

. The Radiographer’s impact on improving clinical decision-making, patient care and patient diagnosis: a pilot study. Radiographer 2004; 51: 133–137.

Ehrlich

Coakes

. Patient care in radiography-e-book: with an introduction to medical imaging. Amsterdam, Netherlands: Elsevier Health Sciences, 2016.

of Radiology ESR

. Patient safety in medical imaging: a joint paper of the European Society of Radiology (ESR) and the European Federation of Radiographer Societies (EFRS). Insights Imaging 2019; 10: 45.

10.

Botwe

Anim-Sampong

Obeng-Nkansah

, et al. Ethical commitments of radiographers in a teaching hospital in West Africa: patients’ perspective: peer reviewed original article. South African Radiogr 2016; 54: 24–27.

11.

Lundgren

Lundén

Andersson

. Radiography-How do students understand the concept of radiography? Radiography 2015; 21: e68–e73.

12.

Sloane

Miller

. Informing radiography curriculum development: The views of UK radiology service managers concerning the ‘fitness for purpose’of recent diagnostic radiography graduates. Radiography 2017; 23: S16–S22.

13.

Pinto

Brunese

. Spectrum of diagnostic errors in radiology. World J Radiol 2010; 2: 377–383.

14.

Pollard

Lincoln

Nisbet

, et al. Patient perceptions of communication with diagnostic radiographers. Radiography 2019; 25: 333–338.

15.

Bajre

Pennington

Woznitza

, et al. Expanding the role of radiographers in reporting suspected lung cancer: a cost-effectiveness analysis using a decision tree model. Radiography 2017; 23: 273–278.

16.

Woznitza

Piper

Rowe

, et al. Immediate reporting of chest X-rays referred from general practice by reporting radiographers: a single centre feasibility study. Clin Radiol 2018; 73: 507e1–507e8.

17.

Sharma

Sarma

Agarwal

. Assessment of diagnostic reference level in radiography of neonatal chest anteroposterior examination: A hospital-based study. J Med Phys 2018; 43: 200–203.

18.

Kilburn

Iddles

Carrington

. Radiographer screening for incidental pulmonary emboli on routine contrast-enhanced computerised tomography scans at a cancer centre. Clin Radiol 2018; 73: 219e1–219e7.

19.

Dendy

Heaton

. Physics for diagnostic radiology. Boca Raton, FL: CRC Press, 2011.

20.

Princy

Rajeswari

. Assessement of competency of diagnostic radiographers. Int J Mech Prod Eng Res Dev 2018; 8: 844–850.

21.

van de Venter

ten Ham-Baloyi

. Image interpretation by radiographers in South Africa: a systematic review. Radiography 2019; 25: 178–185.

22.

Kinkaid

Eagle

McFadden

. Implementing a holistic approach to a radiographer-led Hysterosalpingogram (HSG) service: a review of impact and patient perception. Radiography 2019; 25: 365–373.

23.

Rawashdeh

McEntee

Zaitoun

, et al. Knowledge and practice of computed tomography exposure parameters amongst radiographers in Jordan. Comput Biol Med 2018; 102: 132–137.

24.

Lauridsen

Lefere

Gerke

, et al. Comparison of the diagnostic performance of CT colonography interpreted by radiologists and radiographers. Insights Imaging 2013; 4: 491–497.

25.

Wuni

A-R

Courtier

Kelly

. Opportunities for radiographer reporting in Ghana and the potential for improved patient care. Radiography 2020; 26: e120–e125.

26.

Roe

Harris

Lambie

, et al. Radiographer Workforce role expansion to improve patient safety related to nasogastric tube placement for feeding in adults. Clin Radiol 2017; 72: 518e1–518e7.

27.

Cosson

Willis

. Student radiographer perspectives on using a screen based computer simulator in diagnostic radiography. Upper Montclair, NJ: Shaderware Limited. Retrieved from Shaderware website.http//www shaderware com/distrib/etc/Whitepaper201211-StudentRadiographerPerspectivesOnUsingAScreenBasedComputerSimulatorInDiagnosticRadiography pdf

28.

Morcos

Suwais

. Computer assisted learning for the interpretation of the chest radiograph. Heal Informatics 1996; 2: 146–148.

29.

Venson

Bevilacqua

Berni

, et al. Diagnostic concordance between mobile interfaces and conventional workstations for emergency imaging assessment. Int J Med Inform 2018; 113: 1–8.

30.

Tafnit by Matrix . Tafnit Homepage. https://www.tafniterp.co.il/(2021, accessed 19 August 2021).

31.

McKinney

Python for data analysis: Data wrangling with Pandas, NumPy, and IPython. Newton, MA: ‘ O’Reilly Media, Inc.’, 2012.

32.

Abu-Bader

SH,

others. Using statistical methods in social science research: With a complete SPSS guide.USA: Oxford University Press, 2021.

33.

French

. InDesign Type: Professional Typography with Adobe InDesign. San Francisco, CA: Adobe Press, 2018.

34.

Top Solutions . ANT , 2021, https://www.topsolutions.co.il/nemala-plus/(accessed 19 August 2021.

35.

Pianykh

Langs

Dewey

, et al. Continuous learning AI in radiology: Implementation principles and early applications. Radiology 2020; 297: 6–14.

36.

Dikici

Bigelow

Prevedello

, et al. Integrating AI into radiology workflow: Levels of research, production, and feedback maturity. J Med Imaging 2020; 7: 016502.

37.

Mahajan

Venugopal

Murugavel

, et al. The algorithmic audit: Working with vendors to validate radiology-AI algorithms—how we do it. Acad Radiol 2020; 27: 132–135.

Development of a mobile training app to assist radiographers’ diagnostic assessments