Sage Journals: Discover world-class research

Abstract

Introduction

Extracorporeal membrane oxygenation (ECMO) emergencies require skilled clinical specialist (CS) who manage ECMO circuits. While tools for assessing CS skills have been published, there is significant variation in protocols and circuit design. This study aims to further develop these checklists to produce a generalizable ECMO skill assessment with adequate validity evidence to support its use as a summative evaluation tool.

Methods

An initial survey determined variation in ECMO circuit components and configurations, and the original checklists and simulations were altered through a modified Delphi process. The finalized checklist and simulation were then assessed for validity and reliability. Three trained raters assessed ten simulations from five subjects at two different institutions using two circuit designs. Data analysis was conducted using a fully crossed subject x rater x circuit generalizability (G) and decision (D) study.

Results

The G-study coefficient was 0 with 0% variance across subject and circuit. The greatest variance was among raters (28.7%). Significant variance was also associated with the subject and pump type relationship (27%).

Conclusion

Despite the rigorous process used to modify the assessment, generalizability was poor. Lack of familiarity with center-specific circuit design played a key role. Future endeavors in ECMO skill assessment should focus either on developing and validating site-specific tools or standardizing circuit designs.

Keywords

ECMO simulation checklist assessment generalizability

Get full access to this article

View all access options for this article.

References

Halamek

. Simulating ECMO: rationale and genesis. In: Johnston

(eds) Comprehensive healthcare simulation: ECMO simulation: a theoretical and practical guide. Cham: Springer International Publishing, 2021, pp. 11–20.

Anderson

Murphy

Boyle

, et al. Simulating extracorporeal membrane oxygenation (ECMO) emergencies, Part II: qualitative and quantitative assessment and validation. Simulat Healthc J Soc Med Simulat 2006; 1: 228–232.

Alphonso

Auerbach

Bechtel

, et al. Development of a child abuse checklist to evaluate prehospital provider performance. Prehosp Emerg Care 2017; 21: 222–232. DOI: 10.1080/10903127.2016.1229824.

Berg

Riesenberg

, et al. The development of a validated checklist for thoracentesis: preliminary results. Am J Med Qual 2013; 28: 220–226. DOI: 10.1177/1062860612459881.

Faudeux

Tran

Dupont

, et al. Development of reliable and validated tools to evaluate technical resuscitation skills in a pediatric simulation setting: resuscitation and emergency simulation checklist for assessment in pediatrics. J Pediatr 2017; 188: 252–257. DOI: 10.1016/j.jpeds.2017.03.055.

Hartman

Wittler

Askew

, et al. Delphi method validation of a procedural performance checklist for insertion of an ultrasound-guided internal jugular central line. Am J Med Qual 2016; 31: 81–85. DOI: 10.1177/1062860614549762.

Muratore

Beilman

John

, et al.

Extracorporeal membrane oxygenation credentialing: where do we stand?

Am J Surg 2015; 210: 655–660.

Abulebda

Hocutt

Gray

, et al. Development of validated checklists to evaluate clinical specialists in pediatric ECMO emergencies using Delphi method. Am Soc Artif Intern Organs J 2020; 66: 314–318. DOI: 10.1097/MAT.0000000000000988.

Messick

. Meaning and values in test validation: the science and ethics of assessment. Educ Res 1989; 18: 5–11.

10.

Adler

Kshetrapal

Calhoun

. Convince me! How a well-constructed validity argument supports the decisions we make about learners. J Pediatr 2022; 245: 11–12. DOI: 10.1016/j.jpeds.2022.03.029.

11.

Calhoun

Scerbo

. Preparing and presenting validation studies: a guide for the perplexed. Simulat Healthc J Soc Med Simulat 2022; 17: 357–365. DOI: 10.1097/SIH.0000000000000667.

12.

Downing

. Validity: on meaningful interpretation of assessment data. Med Educ 2003; 37: 830–837. DOI: 10.1046/j.1365-2923.2003.01594.x.

13.

Brennan

. Performance assessments from the perspective of generalizability theory. Appl Psychol Meas 2000; 24: 339–353.

14.

Keszei

Novak

Streiner

. Introduction to health measurement scales. J Psychosom Res 2010; 68: 319–323.

15.

Alexander

PMA

Habet

Barbaro

. Realizing potential: pediatric extracorporeal membrane oxygenation needs common adverse event definitions to improve outcomes. Pediatr Crit Care Med 2023; 24: 528–530. DOI: 10.1097/PCC.0000000000003270.

16.

Johnston

. Neonatal and pediatric ECMO simulation scenarios. Berlin: Springer, 2023.

17.

Said

Cooley

Moore

, et al. Development of a standardized assessment of simulation-based extracorporeal membrane oxygenation educational courses. ATS Sch 2022; 3: 242–257. DOI: 10.34197/ats-scholar.2021-0068OC.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.06 MB