Abstract
Keywords
Introduction
The majority of patients diagnosed in the clinic and emergency department (ED) with acute pulmonary embolism (PE) are normotensive. Accurate assessment and risk stratification of patients with acute PE is of critical importance. Evidence of right ventricular (RV) dysfunction, in particular, can risk stratify this population and help guide site-of-care and treatment decisions. 1 Computed tomography pulmonary angiography (CTPA), which is used to diagnose PE in approximately 95% of cases, and transthoracic echocardiography (TTE), which is considered the gold standard in dynamic assessment of cardiac function, can reveal evidence of RV dysfunction and be used to differentiate between low and intermediate-risk patients. 2 A recent large, patient-level meta-analysis found that RV dysfunction (assessed by any method) was associated with short-term death in otherwise low-risk patients with acute PE. The authors recommended that confirming the absence of RV dysfunction could help identify low-risk patients who may be candidates for outpatient management. 3
Findings such as septal flattening or inferior vena cava contrast reflux on CT and decreased tricuspid annular plane systolic excursion or diminished inspiratory collapsibility of the inferior vena cava on TTE are all indicators of RV dysfunction.2,4 All of these findings, however, require a fair degree of clinical skill to ascertain and interpret. One finding that is strongly correlated with RV dysfunction that is relatively simple to obtain and can be readily assessed by CTPA is RV dilatation.5,6 Compared with TTE, CTPA has been found to be sensitive for RV dilatation, but less specific. 1 CTPA, however, is often more readily available to outpatient and emergency clinicians than TTE.
Radiology reports of CTPA results, however, do not routinely include a description of the RV/left ventricle (LV) diameter ratio, often leaving the identification of RV dysfunction to the ordering clinician. Primary care and emergency physicians are not conventionally trained in evaluating RV dilatation on CT images. A retrospective cohort study found that emergency medicine physicians could be trained to accurately measure the RV/LV diameter ratio on CTPAs of ED patients with acute PE. 7 The measurement of RV/LV diameter ratio has also been found to be reproducible by internal medicine residents with brief training. 8 We inferred from these resident studies that evaluation of the ratio was relatively easy and that nonradiologists at any level of training could be taught to do this reliably. To test this hypothesis, we undertook a post hoc analysis using CT images from a prior retrospective cohort study of patients undergoing management of PE at a large network of U.S. community hospitals. If our hypothesis were confirmed, it could show that a basic level of proficiency for certain applications of CT can easily and effectively be taught to students in their pre-clinical years.
Methods
Study Design and Setting
We performed post hoc analysis of a prior retrospective cohort study of adult patients undergoing management of acute PE at 21 community medical centers across Kaiser Permanente Northern California (KPNC). 9 KPNC is one of the largest community-integrated health systems in Northern California, serving over 4 million members and totaling over 1.2 million ED visits per year. Kaiser Permanente members include approximately a third of the population in areas served and are highly representative of the diverse surrounding and statewide population. 10 KPNC is a healthcare system with an applied research agenda and is supported by a comprehensive electronic health record (Epic, Verona, WI). The Research Determination Committee for KPNC decided that the project did not meet the regulatory definition of research involving human subjects per 45 CFR 46.102(d). Administrative permissions were required, and obtained, to access the raw data. The study was conducted in accordance with the principles of the Declaration of Helsinki.
Study Aim
The aim of this study was to evaluate the accuracy of medical student measurements of RV dilatation on CTPA, after brief training, compared with those of an experienced radiologist. Our primary outcome was the presence of RV dilatation as defined by an RV/LV ratio of 1.0 or greater in the setting of acute PE.
Selection of Participants
We accessed health records of participants with acute PE managed at 21 EDs within KPNC from January 2013 to April 2015. The inclusion and exclusion criteria were the same as the original retrospective study. 9 From the original dataset of 2387 patients who underwent CTPA, we divided patients into 5 PE Severity Index (PESI) classes, which correlate with 30-day all-cause mortality and initial site-of-care (home vs observation vs hospital). 11 Within each class, we determined the proportion of patients who were discharged home either directly from the ED or within 24 h from ED registration (n = 496) and kept the same proportion of patients within each class. 12 This was done to ensure a sufficient number of scans within each category rather than to reflect the actual distribution of patients in each PESI class from the sample population (which fell predominantly into the 3 lowest PESI classes). We sought to include 100 CTPAs, so set our sample size at 120 CTPAs to allow for the possibility that some image files would be corrupted or otherwise inaccessible. The rationale for the number of scans was based on our biostatistician's assessment of the number that would be needed to accurate calculate interrater reliability using Cohen's Kappa and Bland-Altman plots. Assuming that the proportion of RV strain was 0.75, a large sample 2-sided test of the null hypothesis at the 0.05 significance level with an intraclass kappa of 0.4 would have 80% power to detect an alternative kappa of 0.7 with a sample size of 94. We then assembled a randomly selected stratified sample of 24 patients from each PESI class (total N = 120). The 120 images were accessed and reviewed.
The images reviewed were non-ECG-gated axial angiograms performed on a 64-slice multidetector CT scanner. Although many of the patients in our study had TTEs performed during their hospital stay, the reviewers were blinded to any clinical or demographic data, including the performance of additional testing and their results.
As this was a student-led project the student co-authors were involved in the study from its early stages and no outside volunteers were recruited to serve as study subjects. As such, we did not think of them as subjects for research. Additionally, each of the student co-authors had applied to be a part of our specialty research network of their own volition and voluntarily signed up for the project as is typical for projects within our research network. None of our co-authors were supervised by us in a clinical or academic capacity, so their participation in this study had no potential influence on any grades or evaluations.
Our research determination office determined that this project did not fit the definition of human subjects research, so IRB approval and informed consent were not required. None of the prior similar studies, including the one on which we based our own study involving student learners, considered their image-reviewing co-authors to be study subjects or required informed consent from them. 8 Nevertheless, we did take precautions to ensure anonymity. The students were not aware of their individual performance and had access only to the anonymized results of all the reviewers. Only the principal investigator (PI), senior author, and biostatistician had access to the full data.
Training
The study team consisted of 2 emergency medicine attending physicians, 1 experienced radiologist, 3 medical students of various training levels, and 1 premedical student. None of the students had prior radiology training. Prior to beginning data collection, all students attended 3 training sessions led by the same PI to ensure that each was taught the same material in a similar fashion. The students were instructed to find the image with the maximum transverse diameter of each ventricle, identify the endocardial border, and measure the distance between ventricular endocardium of the free wall and the interventricular septum parallel to the tricuspid or mitral valve for RV and LV, respectively. This method is similar to that described in other studies measuring RV strain. 8
The first session consisted of a group lecture on the measurement of ventricle size on CT scan. Each student then independently completed a series of 10 practice cases. During the second individual training session the sample slide set was reviewed and teaching points were reiterated. The students were then given a second sample set to complete. For the third training session the students met with the PI and radiologist, reviewed the second case set, and learned how to use the Picture Archiving and Communications System. Each session lasted approximately 45 min for a total training time of just over 2 hours.
Image Review
The student reviewers had access only to images. They did not have access to the electronic health record and remained blind to clinical data, including the original radiology report. The study radiologist reviewed all the images and his review was used as the gold standard.
Statistical Analysis
Interrater reliability for the assessment of the primary outcome (RV/LV ratio ≥ 1.0) was determined using Cohen's kappa statistic, with agreement determined as none (value < 0), slight (0.00-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80), or almost perfect (0.81-1.00). 13 In comparing each of the raters with the radiologist, we used Bland-Altman plots to represent the mean difference in RV/LV ratio measurements between the radiologist and each of the 4 raters. The Bland-Altman analysis is a simple and accurate way to quantify agreement between 2 raters. The Bland-Altman plot is a scatterplot in which the X-axis represents the average [(M1 + M2)/2], and the Y-axis represents the difference (M1 – M2) of 2 measurements. The scatterplot can be evaluated according to the scatter pattern. When the scattering of points is relatively close to the horizontal line, it indicates good agreement between 2 raters. Pearson correlation coefficients were also used to represent the correlations in RV/LV ratio measurements between the radiologist and each of the 4 raters. We compared patients’ demographic and clinical characteristics by RV dilatation. Comparisons involving categorical variables were performed using the chi-square or Fisher's exact test. Normally distributed continuous variables were compared using Student's t test. All analyses were performed using SAS 9.4 (SAS Institute, Cary, NC).
Results
From the initial sample of 120 images, 12 studies were excluded because the image files were corrupted or inaccessible. The remaining 108 angiograms underwent analysis. The proportions of home/short-stay patients in each PESI class were shown in Table 1. We report patient demographic and clinical characteristics in Table 2. The study radiologist determined that 79 patients (73%) showed RV dilatation on CT and 29 (27%) did not. Patients with RV dilatation had a higher mean age (64.4 ± 15.8 years) than those without RV dilatation (54.0 ± 16.7 years, P = .004). There were no other significant inter-group differences in patient characteristics and comorbidities.
Disposition of Emergency Department Patients With Acute Pulmonary Embolism Stratified by Severity Index Class.
Abbreviations: PESI: Pulmonary Embolism Severity Index; CTPA: computed tomography pulmonary angiography.
Characteristics of Adult Patients with Acute Pulmonary Embolism Whose Computed Tomography Images were used in Student Training, Stratified By Right Ventricular Dilatation.
Abbreviations: PESI, Pulmonary Embolism Severity Index; BMI, body mass index; IQR, interquartile range; F, female; M, male. All cells are reported as n (%), except where noted.
† Two-group t-test
‡ Chi-square test
§ Fisher's exact test
In comparing the 4 raters to the radiologist, all except 1 showed moderate agreement for the presence of RV dilatation (Table 3). The Pearson correlation coefficients for the measurements of each of the raters compared with the radiologist were 0.87, 0.80, 0.74, and 0.78 for raters 1–4, respectively. The Bland-Altman plots (Table 4 and Figures 1–4) show that the mean differences in RV/LV ratio between the raters and the radiologist were relatively small for 3 of the 4 raters (−0.04, −0.05, 0.04, 0.24 for raters 1–4, respectively).
Bland and Altman analysis of the RV/LV ratio measured by rater 1 versus radiologist. Abbreviations: RV, right ventricle; LV, left ventricle.
Bland and Altman analysis of the RV/LV ratio measured by rater 2 versus radiologist. Abbreviations: RV, right ventricle; LV, left ventricle.
Bland and Altman analysis of the RV/LV ratio measured by rater 3 versus radiologist. Abbreviations: RV, right ventricle; LV, left ventricle.
Bland and Altman analysis of the RV/LV ratio measured by rater 4 versus radiologist. Abbreviations: RV, right ventricle; LV, left ventricle.
Cohen's Kappa Statistics of the Experienced Radiologist Reviewer and the 4 Students.
Difference Between Rater and Radiologist RV and LV Ratio Measurements (in mm).
Abbreviations: IQR, interquartile range; RV, right ventricle; LV, left ventricle.
Discussion
In this retrospective cohort study, we found that after brief training, medical and premedical students could measure RV dilatation with moderate accuracy compared with an experienced radiologist. Our study is the first to evaluate the ability of learners at this level of training to interpret these findings on CT.
Studies of nonradiologist physicians and resident physicians on the evaluation of CT dilation on CTPA have shown near-perfect agreement between measurements performed by nonradiologists and experienced radiologists. There are several likely reasons why our findings differ. First, our student raters were much earlier in their medical training than the senior resident physicians. Although the residents in the reference study were nonradiologists, basic interpretation of diagnostic imaging would have been part of their training and routine medical practice during residency. Interestingly, the interrater agreement between the students and the radiologist in our study was similar to that between radiologists performing RV strain measurements in another recent study. 14
Second, it is not possible to make a direct comparison between the training for RV dilatation in our study and that in the reference study. The residents in the reference study were noted to have undergone “simple instruction,” in addition to receiving written instructions from an experienced radiologist. The extent and content of this initial instruction is unclear. Our training was also limited. It is possible that had we expanded it, our students could have achieved a higher level of accuracy.
Finally, a much higher proportion of our patients (73%) had RV dilatation compared with those in the reference study (42%). Our sample was stratified to contain an equal number of patients in each PESI class, so our cohort contained more patients in the higher PESI classes than a random, non-stratified sample. When looking at a dichotomous outcome (RV dilatation), small differences in measurements can significantly change dichotomous results.
Our study had several limitations. As noted above, the high number of positive results relative to expectations may have biased the student reviewers to underreport positive findings (expectation bias). Additionally, because the students had sufficient training to know that an RV/LV diameter ≥ 1 constituted a positive finding, precision may have been seen as less important in measurements with obviously large ratios, which could account for the larger disagreement on Bland-Altman plots for patients with large RV/LV ratios. Finally, because this is essentially a proof-of-concept study that involved only 4 students and is the only one, to our knowledge, to evaluate the ability of learners with that level of training to measure RV/LV diameter, it would be difficult to generalize these findings to the larger population of undergraduate medical students or describe implications for medical school curricula.
Prior studies evaluating the ability of medical students to interpret specific findings on diagnostic imaging after training have shown mixed results, with generally high accuracy for some ultrasound applications and poor accuracy for interpretation of CT.15–17 In one study, nearly two-thirds of interns reported being frequently expected to make preliminary imaging study interpretations, but fewer than half felt confident in their ability to interpret basic CT images. 18 There is consensus among residency program directors that adequate radiology skills should be a part of undergraduate medical education. 19 Our study shows that a basic level of proficiency for certain applications of CT can easily and effectively be taught to students.
Conclusion
The findings of this study imply that although the identification of RV dilation on CTPA may be relatively accurate when performed by nonradiologists and resident physicians, the same level of accuracy may not extend to medical students. Medical and premedical student learners were able to achieve moderate accuracy with limited training. It is possible that with additional focused training, student accuracy could approach that of resident physicians.
Footnotes
Acknowledgments
The authors would like to thank Juleon Rabbani of the Kaiser Biostatistical Consulting Unit for his help and advice in the early stages of this study. A preprint of this manuscript submitted elsewhere is available online.
Authors’ contributions
ED contributed to the study concept and design, acquisition of the data, analysis the and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, statistical expertise, obtaining funding, administrative, technical, or material support, and study supervision. SF contributed to the study concept and design, acquisition of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. DE contributed to the study concept and design, acquisition of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. LF contributed to the study concept and design, acquisition of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. JS contributed to the study concept and design, acquisition of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. YH contributed to the study concept and design, acquisition of the data, analysis, and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, statistical expertise. JC contributed to the study concept and design, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. BSR contributed to study concept and design, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. DV contributed to the study concept and design, acquisition of the data, analysis, and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, statistical expertise, administrative, technical, or material support, and study supervision.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author upon request.
Consent for Publication
Not applicable.
Ethical Approval
The Research Determination Committee for KPNC decided that the project did not meet the regulatory definition of research involving human subjects per 45 CFR 46.102(d), so consent for participation was not required. The study was conducted in accordance with the principles of the Declaration of Helsinki.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by the Permanente Medical Group Delivery Science Research Initiative.
