Determining the Extent of Agreement for Determining the Echographic Ejection Fraction Determined by Emergency Medicine Residents Compared with a Cardiologist

Abstract

Objective:

The aim of this study was to determine the extent of agreement on the measured echographic ejection fraction measured by emergency medicine (EM) residents compared with a cardiologist.

Materials and Methods:

This study was conducted with cardiac patients admitted to a hospital emergency department in Tehran, Iran. A total of 156 patient cases were reviewed. A group of trained EM residents evaluated the ejection fraction (EF) of these patients utilizing three different methods: Simpson’s method (N = 50), E-Point Septal Separation (EPSS, N = 56), and a visual technique (N = 50). A final assessment was made for all patients by a cardiologist.

Results:

In 69.3% of these patient cases, there was a significant correlation between the EM residents’ report and the cardiologists, with Cohen’s Kappa coefficient of .746 (P < .001). The Kappa coefficient for the visual technique, Simpson’s method, and EPSS was .697, .467, and .487, respectively. The sensitivity, specificity, positive and negative predictive values of EF measurements, provided by the EM residents, were 0.89%, 64.2%, 81.6%, and 76.5% compared with the cardiologist, respectively.

Conclusion:

This description study was done to determine the agreement of echocardiographic measurement of EF, assessed by trained EM residents, compared with a cardiologist’s review. In this case, the results demonstrated moderate to good agreement.

Keywords

inter-rater reliability ventricular ejection fraction reporting echocardiography emergency medicine cardiac evaluation

In this recent decade, the mortality rate caused by cardiovascular diseases has increased drastically in Middle Eastern countries compared with other parts of the world. Studies have also revealed that the mortality caused by cardiovascular disorders in the Middle East increased by 171% from 1990 to 2020. The epidemiological studies in Iran and the United Arab Emirates reflect an estimated prevalence of 5.9% for cardiovascular diseases in the future.¹ In response to this rise in cardiovascular disease, goal-directed, emergency medicine departments (EDs) have used a point-of-care ultrasound (POCUS) curriculum to try to increase a physician trainee’s self-confidence and performance.²

As the most important indicator of cardiac function, obtaining the cardiac ejection fraction (EF) is vital in deciding on an at-risk patient’s clinical management, daily clinical examinations, and ultimate prognosis.³ The cardiac EF serves as reliable evidence of total heart function, and changes in EF can indicate various congenital and acquired heart disorders.⁴ The advantages of obtaining this measurement include easy implementation, safety, the minimal need for preparing the patient, repeatability, noninvasiveness, and cost-effectiveness. Hence, physicians highly use echocardiography a noninvasive diagnostic tool to measure EF.⁵

A heart attack or other cardiac disease can negatively influence the EF.⁶ The accurate assessment of left ventricular ejection fraction (LVEF) is essential for managing patients with acute diseases, such as those with chest pain, cardiogenic shock, or severe respiratory distress.⁷ Physicians who are not cardiologists can learn to conduct a basic echocardiogram and assess cardiac EF.⁸ An echocardiogram allows emergency medicine (EM) physicians to obtain diagnostic information on many acute cardiovascular diseases, such as tamponade, pulmonary embolism (PE), cardiogenic shock, and cardiogenic pulmonary edema.⁹ Currently, the rapid ultrasound for shock and hypotension (RUSH) examination can be conducted in the ED, at the bedside of critically ill patients, to determine the cause of cardiogenic shock. This exam includes an assessment of EF, while can assess cardiac function and the probability of PE, tamponade, and pericardial effusion.^8,10 However, only a few studies have been conducted to validate the accuracy of EF calculations in the ED. Therefore, this study was composed to determine the reliability and potential accuracy of these measurements, made in the ED. To this end, a reliability and accuracy test was conducted, of the echographic EF measurements, made by EM residents compared with a cardiologist.

Materials and Methods

Study Design

This study was designed to determine concordance between diagnostic test results and a gold standard. The study design was presented to the Ethics Committee of Iran University of Medical Sciences and was approved (ethics code: IR.IUMS.FMD.REC.1398.070). After providing a complete explanation of the study, informed written consent was obtained from all patients. It was important stress to the patients that participation in the study was optional and medical services would still be provided regardless of their willingness to participate in the study. There were no additional expenses charged to the patient for additional interpretations. The researchers abided by the Helsinki Convention principles during this study. Each patient provided verbal consent and was assured of medical confidentiality.

Study Cohort and Sample Size

This study was conducted, in 2020, with qualified patients between the ages of 18 and 80 years of age and presenting with dyspnea, to the ED at Rasoul Akram Hospital. Exclusion criteria included those patients who were intubated, pregnant, elevated cardiac biomarkers, hemodynamically unstable, and patients with atrial fibrillation. These exclusions were made due to the possible influence on echocardiographic EF measurements.

In this study, sample measurements were gathered using a convenient sampling technique. A total of 156 participants were enrolled in the study.

Measurement Technique

In this study, five senior EM residents received theoretic content (4 hours) and practical training (5 hours on 15 patients) from a skilled and experienced echocardiologist.

Following the referral of patients, eligible individuals underwent an echocardiogram that was conducted on a Samsung HM70A ultrasound equipment system (Samsung Healthcare.com). Each EM resident measured the LVEF using one of the following three diagnostic methods: Simpson’s method, E-Point Septal Separation (EPSS), and a visual assessment. Simpson’s method requires tracking the endocardium in apical four-chamber and two-chamber views, during end-systole and end-diastole. The left ventricle’s end-diastolic volume (EDV) and the end-systolic volume (ESV) were calculated. Afterward, the left ventricular EF is calculated via the following formula

E F = \frac{V o l_{e d} - V o l_{e s}}{V o l_{e d}} \times 100

Normal function is defined as an EF of more than 55%. Mild dysfunction is an EF between 45% and 54%. Moderate and severe dysfunctions are considered 30%–44% and less than 30%, respectively.

The EPSS is an M-mode method measured in the parasternal long-axis (PLAX) view. The shortest distance between the apex of the anterior leaflet of the mitral valve and the ventricular septum is measured in millimeters, called EPSS. A distance less than 8.0 mm is defined as a normal function (EF >55%), between 8 and 14 is mild dysfunction (EF = 45%–54%), between 14 and 17 is moderate dysfunction (EF = 30%–44%), and more than 17 is considered as severe dysfunction (EF <30%).

Using a visual method (sometimes referred to as “eyeballing”), the EF was estimated based on personally detecting LV contraction in the PLAX view. Each resident looked for an inward movement of the endocardium, thickening of the myocardium, the motion of the anterior mitral valve leaflet, and geometry of the left ventricle (e.g., size and aneurysms). This method classified dysfunction in comparison with Simpson’s and EPSS method of measurement.

Finally, a cardiologist assessed all the patients using the same offline approach and device to assess the LVEF. The cardiologist’s report was considered the diagnostic gold standard for interpretation. It is also worth noting that the cardiologist was unaware of the results reported by the EM residents. Also, one cardiologist conducted all the echocardiographic readings. There was a maximum interval of 1 hour between two echocardiography examinations conducted by the EM resident and the cardiologist. As mentioned, all patients were in stable condition, and two echocardiograms did not hinder or postpone the diagnostic or therapeutic measures. All the cardiac function information was recorded as well as their age, gender, blood pressure, heart rate, respiratory rate, and body mass index (BMI). These data were all collected on a predesigned checklist. Finally, all data points were analyzed statistically.

Data Analysis

The analysis was conducted using SPSS statistical software (IBM Corp., Armonk, NY, USA). The quantitative data were descriptively defined by the mean and standard deviations. The qualitative data were described using percentages and frequency. Cohen’s Kappa coefficient, Mann-Whitney U test, and Spearman’s correlation tests were performed on this data set. The diagnostic sensitivity, specificity, positive and negative predictive values of the resident’s measured EF were preformed (see Table 1). As previously mentioned, the reported measurements made by the cardiologist were considered the diagnostic gold standard. The statistical significance was set a priori at P < .05.

Table 1.

Evaluation of Sensitivity, Specificity, Positive Predictive Value, and Negative Predictive Value of the Measured Ejection Fraction by Emergency Medicine Residents.

Groups	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Value
Visual technique	94.8%	72.7%	92.5%	80.0%
E-Point Septal Separation	85.1%	65.5%	69.6%	82.6%
Simpson’s method	85.2%	56.2%	80.5%	64.2%
Total	89.0%	64.2%	81.6%	76.5%

Results

The aim of this study was to determine the extent of agreement on the measured echographic EF measured by EM residents compared with a cardiologist. The study participants included 156 patients. According to the cardiologist’s report, 57 patients had normal cardiac function based on their EF measurement. Demographic characteristics are provided in Table 2. The mean blood pressure, respiratory rate, and heart rate values are also provided (see Table 3).

Table 2.

The Study Cohort’s Demographic Characteristics.

Abbreviations: M, mean; SD: standard deviation.

Table 3.

The Mean Blood Pressure, Heart Rate, and Respiratory Rate for the Study Cohort.

Variables	M ± SD	Minimum	Maximum
Systolic blood pressure in mmHg	128.49 ± 23.49	80	190
Diastolic blood pressure in mmHg	75.51 ± 9.10	60	110
Heart rate	80.50 ± 12.83	45	110
Respiratory rate	17.39 ± 1.78	12	25

Abbreviations: M, mean; SD: standard deviation.

As shown in Table 4, the comparison between the EF measurements made by the EM residents and the cardiologist was analyzed qualitatively and quantitatively.

Table 4.

The Correlation Coefficient Between the Ejection Fraction Measured by Cardiologists and Emergency Medicine Residents.

	N = 156		Cardiologists’ ReportFrequency (%)
	N = 156		Normal Function	Mild Dysfunction	Moderate Dysfunction	Severe Dysfunction
Qualitative	Emergency medicine residents’ report Frequency (%)	Normal function	36 (23.1)	11 (7.0)	0 (0.0)	0 (0.0)
		Mild dysfunction	19 (12.2)	42 (26.9)	5 (3.2)	1 (0.6)
		Moderate dysfunction	1 (0.6)	5 (3.2)	16 (10.3)	5 (3.2)
		Severe dysfunction	0 (0.0)	0 (0.0)	1 (0.6)	14 (9.0)
	Result of statistical test		Kappa contingency coefficient = .746 P = .000
Quantitative	Result of statistical test		Spearman correlation coefficient = .841 P = .000

In the qualitative section, the agreement between the EM resident and the cardiologist reports was at 69.3% for all the cases. A moderately strong Cohen’s Kappa coefficient of .746 (P < .001) was statistically significant. The results of the quantitative analysis were also compared with moderately strong comparison between the EM residents’ reports and the cardiologist’s reports (P < .001).

The Kappa contingency coefficient and Spearman’s coefficient for each diagnostic method are shown separately in Table 5. In this cohort of patients, there were significant Cohen’s Kappa coefficient and high correlation between the EM residents and the cardiologist based on all the diagnostic techniques to measure EF.

Table 5.

The Kappa Contingency Coefficient and Spearman Correlation Coefficient for Each of the Diagnostic Methods to Assess Cardiac Ejection Fraction.

	Kappa Contingency Coefficient	Spearman Correlation Coefficient
Visual technique	.697	.859
E-Point Septal Separation	.487	.782
Simpson’s method	.467	.847

Generally, a Kappa of less than .4 is considered poor (a Kappa of 0 means there is no difference between the observers and chance alone). Kappa values of .4–.75 are considered moderate to good, and a Kappa of >.75 represents excellent agreement. A Spearman’s correlation is based on a perfect correlation approaching 1.0.

Based on these reported findings, the highest agreement and correlation between the EM residents and the cardiologist’s measurements, as well as the highest specificity, sensitivity, and positive and negative predictive values, were reported when using the person visualization technique. In addition, investigating the agreements between EM residents and cardiologists indicated that for all five trained EM residents, the agreements were significant and very similar (81.3%, 80.6%, 83.8%, 77.4%, and 77.4%, respectively).

Discussion

Based on the findings of this study, the overall agreement between the EM residents and the cardiologists was 69.3%, which was significant. Besides, the sensitivity, specificity, and positive and negative predictive values of the EF measurements by the EM resident were 89.0%, 64.2%, 81.6%, and 76.5%, respectively, compared with the cardiologist’s report, which was considered the gold standard.

In an investigation, Unlüer et al. studied the visual (eyeballing) technique for assessing echocardiographic EF. In that study, the sensitivity, specificity, and positive and negative predictive values of the patient’s bedside EF measurements, made by an EM specialist compared with a cardiologist, were 98.7%, 86.2%, 90.4%, and 90.4%, respectively. In the Unlüer et al., study, the reported values were also higher than the total values in this study. However, Unlüer et al.¹¹ only used the visual technique for assessment and used EM physicians to provide the reporting. The sensitivity, specificity, and positive and negative predictive values resulting from the visual technique in this study were 94.8%, 72.7%, 92.5%, and 80.0%, respectively. These values are almost similar to the values reported by Unlüer et al.¹¹ Similar to the study by Unlüer et al., the lowest value reported in this study was the specificity of the bedside EF calculation. In both studies, the highest level of disagreement was in patients with a normal left ventricular EF diagnosed with dysfunction or low EF by the EM residents/specialists.

In a study by Randazzo et al., the sensitivity of the left ventricular EF in the EPSS method was 100%. However, its specificity was below 51.6%.¹² Unlike this study, Randazzo et al. observed the highest level of agreement between the EM specialists and echocardiography specialists in the normal function category. This difference between these studies could be attributed to the difference between their inclusion and exclusion criteria and their patient selection methods. In the study by Randazzo et al.,¹² most participants were patients suffering from low blood pressure, and only 5.7% of the 115 participants had dyspnea. In contrast, one of the inclusion criteria in this study was the presence of dyspnea.

A meta-analysis conducted by Albaroudi et al. assessed the visual estimation technique by EM physicians, using point-of-care echocardiography, compared with expert sonographers. The sensitivity, specificity, and positive and negative likelihood ratios of 89%, 85%, 5.98, and 0.13, respectively, were between clinician sonographers and expert sonographers for normal and abnormal left ventricular systolic function.¹³ These findings compare very nicely with the results of this study.

In comparison with this study, the work published by Weekes et al.,¹⁴ van Royen et al.,¹⁵ and Amico et al.¹⁶ demonstrated that the visual technique was superior to the other relevant diagnostic methods in estimating the EF. Jensen-Urstad et al.¹⁷ and van’t Hof et al.¹⁸ indicated that the visual technique offered a high level of accuracy in the estimation of EF. Gudmundsson et al. indicated that the visual technique was like all the other EF calculation methods, including Simpson’s method. None of these evaluative methods for detecting EF is considerably superior to the others.¹⁹ In addition, a systematic review by McGowan et al.²⁰ indicated that Simpson’s method, EPSS, and the visual technique have similar advantages and disadvantages.

In the study by McKaigney et al., they reported that the visual technique has a weak correlation with the actual output of the left ventricle. In contrast, the EPSS method strongly correlates with the actual LVEF. Unlike this study, in that report, the EPSS method was considerably superior to the visual technique. They also referred to its advantages, including faster deployment, less need for the patient’s cooperation, and fewer views.²¹

In another study, Rasooli et al. compared the echocardiography results between an EM resident and cardiologists. The study showed an acceptable agreement between the results of the EM resident and cardiologists’ echocardiography in assessing regional wall motion abnormalities in acute coronary syndrome.²²

One of the distinguishing features of this study was the assessment of the EF by five different EM residents, which indicated that well-trained EM residents could provide acceptable measurements of EF, within a short time in the ED. As for all five trained residents, the agreements between the residents and the cardiologists were significant and almost similar.

Limitations

The major limitation to this study was the research design and as such the pre-experimental nature and nonrandomization of the patient’s evaluated hamper the results due to threats to internal and external validity. These reliability results are unique to these EM residents and the patients that they examined. Another concern was that this was a single-center trial that used convenience sampling for patient selection. Future work in this area should be conducted across multiple health care centers with a variety of EM residents.

In addition, further limitations exist due to the difference in the time of assessment by EM residents and cardiologists. Although it is very unlikely, the patient’s systolic function may change through long intervals in some cases and due to changes in the patient’s condition, such as the development of acidosis, electrolyte derangements, or the intake of vasopressors. Therefore, future reliability studies should account for the time interval between the bedside EF assessment and the measurement by a cardiologist as a determining variable. In addition, EPSS is an M-mode method that is no longer a reliable measure to assess the EF. Although, in this study, “eyeballing” provided the highest correlation between physicians, it is not considered a scientific method. Moreover, the inclusion criteria of this study did not include a wide range of patients with different conditions.

Conclusion

The results of this reliability study showed acceptable agreement between the diagnostic measurement of EF by trained EM residents and a cardiologist. Furthermore, this study also revealed that that the visual technique had more consistency between EM residents and a cardiologist. Thus, an effort should be considered to educate EM residents on conducting cardiac sonogram and how to make reliable measurements of EF in patients presenting to the ED. Care should be taken in educating residents on using the visual technique for assessing EF, as published study results may not be generalizable or applicable in certain patient populations. Although, in general, the goal would be to promote Simpson’s method because of its accuracy. Therefore, there may be value in instructing and using different diagnostic methods for the measurement of EF among EM residents and ED physicians. Further research with more rigorous methodologies is needed to confirm the results of this study.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Ethics Approval

The study design was presented to the Ethics Committee of Iran University of Medical Sciences and was approved (ethics code: IR.IUMS.FMD.REC.1398.070).

Informed Consent

Informed consent was not sought for the present study because all case data were de-identified and/or aggregated and followed ethics committee or IRB guidelines (also referred to as the Honest Broker System).

Animal Welfare

Guidelines for humane animal treatment did not apply to the present study because no animals were used during the study.

Trial Registration

Not applicable.

ORCID iDs

Seyedamir Tabibzadeh Dezfooli

Samira Vaziri

References

Yang

E-S

Yeo

J-I

, et al: Video clip training improved emergency medicine residents’ interpretation ability of visual ejection fraction. Hong Kong J Emerg Med. 2020;27(1):15–23.

Shi

Liu

Zhu

: Evaluation of a new goal-directed training curriculum for point-of-care ultrasound in the emergency department: impact on physician self-confidence and ultrasound skills. Eur J Trauma Emerg Surg. 2021;47(2):435–444.

Mansouri

Kheyrollahi

Mokhtari

: Comparison of cardiac output measured by phonocardiography and transthoracic echocardiography in patients with valvular disease and congenital heart anomalies. J Isfahan Med Sch. 2016;34(372):143–147.

Malakmohamadi

Marage

: Echocardiographic versus Fick method for determination of cardiac output in children with various congenital heart diseases. J Shahrekord Univ Med Sci. 2003;5(3):12–20.

Soroor-Azimzadeh

Shamsadini

: Estimation of ejection fraction with echocardiographic and cardioangiographic methods in 50 cardiac patients. J Kerman Univ Med Sci. 2003;10(3):166–171.

Ünlüer

Karagöz

Bayata

Akoğlu

: An alternative approach to the bedside assessment of left ventricular systolic function in the emergency department: displacement of the aortic root. Acad Emerg Med. 2013;20(4):367–373.

Shahgaldi

Gudmundsson

Manouras

Brodin

L-Å

Winter

: Visually estimated ejection fraction by two dimensional and triplane echocardiography is closely correlated with quantitative ejection fraction by real-time three dimensional echocardiography. Cardiovascular Ultrasound. 2009;7:41.

Duvall

Croft

Goldman

: Can hand-carried ultrasound devices be extended for use by the noncardiology medical community? Echocardiography. 2003;20(5):471–476.

Nguyen

Paschal

Saul

, et al: Pilot study: ejection fraction estimation by emergency medicine residents using limited bedside echocardiography in the emergency department. Ann Emerg Med. 2013;4(62):S80–S81.

10.

Dresden

Mitchell

Rahimi

, et al: Right ventricular dilatation on bedside echocardiography performed by emergency physicians aids in the diagnosis of pulmonary embolism. Ann Emerg Med. 2014;63(1):16–24.

11.

Unlüer

Karagöz

Akoğlu

Bayata

: Visual estimation of bedside echocardiographic ejection fraction by emergency physicians. West J Emerg Med. 2014;15(2):221–226.

12.

Randazzo

Snoey

Levitt

Binder

: Accuracy of emergency physician assessment of left ventricular ejection fraction and central venous pressure using echocardiography. Acad Emerg Med. 2003;10(9):973–977.

13.

Albaroudi

Hadda

Albaroudi

Abdel-Rahman

Jarman

Harris

: Assessing left ventricular systolic function by emergency physician using point of care echocardiography compared to expert: systematic review and meta-analysis. European J Emerg Med. 2022;29(1):18–32.

14.

Weekes

Tassone

Babcock

, et al: Comparison of serial qualitative and quantitative assessments of caval index and left ventricular systolic function during early fluid resuscitation of hypotensive emergency department patients. Acad Emerg Med. 2011;18(9):912–921.

15.

van Royen

Jaffe

Krumholz

, et al: Comparison and reproducibility of visual echocardiographic and quantitative radionuclide left ventricular ejection fractions. Am J Cardiol. 1996;77(10):843–850.

16.

Amico

Lichtenberg

Reisner

Stone

Schwartz

Meltzer

: Superiority of visual versus computerized echocardiographic estimation of radionuclide left ventricular ejection fraction. Am Heart J. 1989;118(6):1259–1265.

17.

Jensen-Urstad

Bouvier

Höjer

, et al: Comparison of different echocardiographic methods with radionuclide imaging for measuring left ventricular ejection fraction during acute myocardial infarction treated by thrombolytic therapy. Am J Cardiol. 1998;81(5):538–544.

18.

van’t Hof

Schipper

Gerritsen

Reiffers

Hoorntje

: Comparison of radionuclide angiography with three echocardiographic parameters of left ventricular function in patients after myocardial infarction. Int J Card Imaging. 1998;14(6):413–418.

19.

Gudmundsson

Rydberg

Winter

Willenheimer

: Visually estimated left ventricular ejection fraction by echocardiography is closely correlated with formal quantitative methods. Int J Cardiol. 2005;101(2):209–212.

20.

McGowan

Cleland

: Reliability of reporting left ventricular systolic function by echocardiography: a systematic review of three methods. Am Heart J. 2003;146(3):388–397.

21.

McKaigney

Krantz

La Rocque

Hurst

Buchanan

Kendall

: E-point septal separation: a bedside tool for emergency physician assessment of left ventricular ejection fraction. Am J Emerg Med. 2014;32(6):493–497.

22.

Rasooli

Bagheri

Sadatnaseri

Ashraf

Bahreini

: Comparison of emergency echocardiographic results between cardiologists and an emergency medicine resident in acute coronary syndrome. Arch Acad Emerg Med. 2021;9(1):e53.