Abstract
OBJECTIVES
The objectives of this study were to standardize airway management among critical care fellows and to evaluate whether the completion of a web-based preintubation airway preparation module improves their knowledge and behaviors in the identification and preparation of difficult airways.
METHODS
Critical care experts used international guidelines to develop the module, which contained mandatory readings, brief lectures, and a case-based activity. We measured learner satisfaction, improvements in fellows’ preintubation preparation knowledge, and safety-oriented behavior. The paired t-test was used to compare knowledge assessment scores and the chi-square test was used to compare the categorical variables in the evaluation of the behavior construct.
RESULTS
All trainees (N = 14) completed the module and were satisfied with its contents and structure. Fellows logged 114 intubations during the study period. The mean score on the knowledge test increased (pre 79% vs post 90%, P = .02) postmodule and there was a significant increase in documentation of airway risk stratification in fellows’ procedure notes (65.9% vs 72.9%, P = .049). All respondents were confident that they would be able to apply what they learned in the module into clinical practice and that their patients would likely benefit from their new knowledge.
CONCLUSION
The implementation of an asynchronous web-based module on airway assessment and intubation preparation was feasible. The module was engaging, enhanced the knowledge of our trainees, and improved procedural documentation.
Introduction
Respiratory failure accounts for a significant proportion of the disease burden in medical intensive care units (MICUs) where tracheal intubation and mechanical ventilation are needed by 20% to 40% of patients. 1 These patients are at risk of hemodynamic collapse and severe hypoxemia during intubation because their physiologic reserve is low.2,3 The risk of procedural complications increases with subsequent attempts making precision in difficult airway identification crucial for first-pass intubation success. 3 In a qualitative study that explored factors that contribute to unanticipated difficult airways, insufficient airway assessment was identified as a main contributor to adversity. 4 The authors inferred that if the operator had assessed the airway, then they would have anticipated difficulty and consequently prepared for it. 4 Preintubation airway assessment is a necessary step in identifying and preparing for difficult intubations, which accounts for its emphasis in standard airway courses.5‐7 However, critical care fellows may possess inadequate skills and knowledge to assess and prepare for a difficult airway. As a result, there is a need for comprehensive education in preintubation airway assessment.
There is a paucity of literature on how the topic of airway management is taught in Critical Care Medicine (CCM) and Pulmonary and CCM (PCCM) fellowship programs since most of the relevant literature involves Anesthesiology and Emergency Medicine (EM) trainees. 8 The Accreditation Council for Graduate Medical Education (ACGME) specifies that PCCM fellowship programs must have trainees achieve endotracheal intubation and airway management competency, though the methods of acquiring competency are not clearly described.8‐10 The COVID-19 pandemic has further highlighted the importance of critical care trainees acquiring competency in airway management. A targeted needs assessment performed through informal interviews of fellows and faculty identified gaps in fellows’ knowledge in assessing preintubation airway assessment.
International guidelines emphasize the importance of operator training to improve the quality of care for vulnerable patients and using technology to teach this offers several advantages. 3 Web-based learning provides flexibility of access, ease of content updating, low cost, and appeal to adult learning preferences. 6 Despite its reputation for being unengaging 11 integrating cases with the online learning material has the potential to draw learners and make learning more interactive. 12 We used the Analysis, Design, Development, Implementation, and Evaluation instructional design framework to inform the development of a web-based module titled “Prepare to Intubate” that teaches fellows how to assess airway difficulty and prepare for it. 13 We selected this framework due to its thorough step-by-step approach and because it aligns with the Kern six steps of curriculum design used to develop this module.13,14 In this article, we describe the pilot design, implementation, and evaluation of this module.
Methods
Study population and setting
This was a single-center quasi-experimental pilot study. The project met the criteria for institutional review board exemption by Baystate Health (BH 1769805-2) and the Johns Hopkins University Homewood (HIRB00013374) Institutional Review Boards because it was conducted in established educational settings. Fellows provided their consent electronically at the beginning of the online survey. These were completed voluntarily and analyzed anonymously. We included all our enrolled critical care (N = 8) and pulmonary and critical care (N = 6) fellows for the academic year 2021-2022. Six first-year fellows, 6 second-year fellows, and 2 third-year fellows participated in this module. While completion of the postmodule survey was optional, fellowship program directors required participation from fellows in this educational initiative since competency in the content of this module is required by the ACGME.8,9
The study was conducted at Baystate Medical Center in Springfield, MA, which is a regional campus of the University of Massachusetts Chan Medical School. The PCCM division includes a 3-year PCCM fellowship and a 2-year CCM fellowship with a total of 14 fellows who have overlapping responsibilities and training. Most of the MICU intubations are performed by the ICU fellow or attending with some performed by anesthesiology residents or staff.
Module development
Following our needs assessment, we designed a short web-based module that was launched in July 2021 and fellows were expected to complete it before the end of August 2021. Faculty with expertise in adult CCM (FA, MT, ED) developed the learning objectives and content. All content-contributing faculty were critical care board certified and we sought special input from an anesthesiology-critical care expert. The module learning objectives are summarized in Table 1.
Learning objectives of the module.
Abbreviations: A = learning objective in the affective domain, K = learning objective in the knowledge domain, S = learning objective in the skill domain.
We noted that all fellows were familiar with some airway assessment tools, though they were less familiar with the MACOCHA score despite it being uniquely suitable for the patient population they care for. We used the 7-item MACOCHA score as the standardized framework for the identification of difficult airways in the critically ill because it is recommended by international guidelines.3,15 The items of the MACOCHA score are what make up its name:
The first section of the module focused on concept acquisition presented through narrated videos and short readings. The second section was a 10-part single case-based activity (Appendix 1) where fellows applied what they learned to answer prompt questions related to a patient scenario and ensuing hypothetical complications. It was designed to be problem-centered and the content to be immediately relevant in that it applied to learners’ daily work. We used Canvas, a publicly available learning management system (LMS), to deliver the content. The work was self-directed and asynchronous. An email invitation was sent to each fellow and they were expected to register on the website to access the module. The module was divided into sections that contained required readings, optional readings, multimedia presentations, and case-based activities (Table 2). The sequence of these sections mirrored the Royal College of Anaesthetists guidelines for the management of tracheal intubation in critically ill adults, though we added “prepare for difficulty” to emphasize the importance of this concept and to introduce fellows to what resources were available to them at our institution. 15 Faculty verified via the LMS that the module was completed. Email reminders were sent to ensure timely registration and completion of the learning module. Faculty were also available to answer questions regarding the material. Fellows were expected to complete a multiple-choice question (MCQ) quiz before and after the module. The questions were constructed by an intensivist and were reviewed by other divisional faculty for content validity. Both pre- and postquizzes contained eleven MCQs and were scored as a percentage correct by the LMS. Each item included three distractors and one correct answer. Their content was focused on demonstrating knowledge of the learning outcomes. To allow for greater flexibility around schedules, fellows were permitted two months to complete the module. They could either complete the module in one sitting or over several days and there were no restrictions on what rotations they were on to complete this.
Module sections and descriptions.
Outcome measures
The module was evaluated using the Kirkpatrick evaluation model by analyzing three main constructs: learner satisfaction (reaction), improvement in learner knowledge, and behavior change. 16 The fellows’ enhancement of safety-focused behaviors was measured by two modalities. The first was reported as a proportion using data from the postmodule survey to the question: “How likely will you change what you document in your procedural notes to reflect what you have learned?.” The second was a record of airway risk stratification in procedural notes. To obtain evidence of validity and consequences of testing (ie, congruency in action and self-report), we reported on the proportion of procedural notes (recorded in patient charts and authored by our learners) that accurately comment on airway risk stratification 3 months before and 3 months after the module. This is not described in the literature as a metric for the measurement of attitudinal change; however, upon discussion with experts in the field, this was an agreed-upon surrogate for behavior and safety-based action. Another indirect measure of behavior was learner self-efficacy in intubation. Albert Bandura described self-efficacy as how confident a learner perceives their ability to perform a specific task.17,18 Self-efficacy is a motivational construct that is linked to behavior and persistence with a challenging task.17‐19 We recorded intubation skill (first-pass success) but did not include it as an outcome because intubation skill is highly psychomotor and the module focused on knowledge of preintubation preparation.
Data collection
Data collection began when the fellows commenced the module, and both quantitative surveys and pre/postmodule quizzes were collected. At the end of the module, learners were surveyed for their satisfaction and reaction to the module by using selectively adapted questions from the short Demand-Driven Learning Model survey which combines quantitative and qualitative items. 20 We adapted the survey tool to meet our content-specific terminology. This survey was accessible to learners immediately after completing the module. This evaluation tool was selected because it fits into the web-based framework and because of its wide application. The measure provided rapid feedback to module developers to allow for timely intervention. 20
To measure behavior change, we manually reviewed the procedural documentation of every intubation performed by our fellows 3 months pre (3/1/2021-6/30/2021) and 3 months post (9/1/2021-12/31/2022) implementation of this module. Operators were required to document the number of attempts it took to achieve success at intubation. We did not include data from July and August, as this was used as the run-in period for module completion. We identified the fellow-performed intubations by reviewing their procedure logs available on New Innovations, a residency/ fellowship management software, and then grouped them by date (pre vs post module). The fellow was given credit for first-pass success at intubation based on the number of attempts (1 vs more than 1) listed in their patient chart documentation. We reviewed the same notes for documentation of intubation risk in the individual patients. The fellow had to include the highest scoring MACOCHA items in their note to be credited for documenting intubation risk. Two additional reviewers audited the data by each selecting five random procedure documentation notes for agreement.
Statistical Analysis
Learner satisfaction and self-efficacy in intubation skills were summarized using descriptive statistics to report on the Likert scale postcourse module survey results. For knowledge assessment, we compared the overall pre versus postmodule quiz scores using a paired t-test and considered P values <.05 to be statistically significant. We used the chi-square test to compare the differences in proportions between categorical variables. We compared the aggregate percentage of first-pass success three months before the module with the proportions 3 months after using the Fischer exact test. IBM SPSS software was used for our analysis of the quantitative data.
Results
All 14 fellows participated in the module and 11 completed the post-module survey (78.6% response rate). The majority of the fellows were graduates of Internal Medicine residency (11/14), and the rest were EM-trained. Module participation was confirmed by reviewing the activity page on the LMS, and we noted that it took learners a median time of 64.5 min (SD 51.5) to complete the module. All respondents were either very satisfied (64%) or somewhat satisfied (36%) that the module contributed to their learning. All were satisfied with both the organization of the module and the resources provided, while most (91%) were satisfied with the LMS itself (Table 3).
Postmodule survey evaluation.
The mean premodule quiz score was 79% and that increased to 90% in the postmodule quiz (P = .02). Participants unanimously agreed that they would use what they learned in clinical practice, and they all indicated that it would be very likely that their patients would benefit from their new knowledge (Table 3). Self-reported confidence in preintubation preparation knowledge increased among learners. The mean confidence rating on a 5-point Likert self-report scale was 3.82 premodule, SD 0.57 and 4.18 postmodule, SD 0.72. All fellows indicated that they were confident that they would be able to apply what they learned in the module to clinical practice (Table 3).
Nearly all learners (91%) indicated that they would likely change what they document in their procedural notes based on what they learned in the module. This behavior change was also observed in the fellows’ intubation note documentation. The percentage of procedure notes that included airway risk stratification significantly increased from 65.9% to 72.9% (P = .049). One learner added in the open response part of the postsurvey, “I am incorporating more objective data into [my] documentation.” They reported starting to use several tools such as the MACOCHA score and Cormack-Lehane grading in their documentation. 21
Learners self-reported high confidence in being able to successfully intubate at the first attempt after taking the module (Mean 4.18, SD 0.72, of a maximum 5 on a Likert scale). A total of 114 intubations were logged by the fellows in the 6-month study period (44 before and 70 after module completion). First-pass success did not significantly increase (88.6% vs 94.3%, P = .89).
The postcourse survey also included other open-response questions. A selection of participant responses to the question (what was the most valuable aspect of the module) include:
“Learning the different tools and medications” “[There were a] variety of modalities to keep [the] information engaging. I learn best from questions and videos, so this was great!” “[It was] presented in a very straightforward fashion” “The resources were great; the videos were full of knowledge yet concise” “It was really helpful to understand what resources were available for me at this facility. In addition to that, what are specific things I should be calling the DART [Difficult Airway Response Team] for instead of trying to intubate the patient myself.”
Learners suggested that the module could be improved by using spaced learning approaches and by learning more about advanced techniques for difficult airway management. They also indicated more hands-on training (eg, simulation) would have augmented their learning.
Discussion
We found that an online learning module was associated with improved critical care fellows’ knowledge, confidence and procedural documentation of preintubation airway. We found that participant reaction to the module was very positive despite it being self-regulated and web-based. This was supported by evaluations from the learners who pointed out that module strengths included the ease and convenience of technology use, its concise nature, and clear organization. The fellows’ evaluations echoed George et al's findings in their systematic review of online learning where they found that 67% of studies reported no differences in attitudes to e-learning compared to traditional methods. 22 They concluded that online learning was just as effective, if not more effective than traditional methods, defined as “face–to–face learning that takes place in a classroom environment.” 22 Regarding the content-specific application of online learning (ie, airway management), web-based methods of instruction have been demonstrated to increase postcourse knowledge scores and satisfaction among anesthesiology residents.7,23
Conversely, online education has a reputation for being boring. 11 This is particularly evident when nonengaging didactics are simply placed on an online learning platform called e-learning. 11 We believe that our strategy of using questions, and problem-solving of engaging cases paired with short learning videos helped to motivate our learners. This is supported by the trainees’ open response comments to what they thought to be the most valuable aspect of the module. The principles of case/problem-based learning (PBL) and more broadly those of adult learning theory are acknowledged by Gaupp et al and are commonly accepted and used in the healthcare field. 12 The appeal of PBL is in keeping with adult learning preferences as described by Knowles et al, where the focus is problem-centered and less content-oriented. 24 The popularity of PBL in medical education could be due to its mirroring of physicians’ daily problem-solving-based practice. Online learning has the additional benefit of meeting the adult learning principle of self-direction because learning can be asynchronous and adjusted to the pace of the learner. 25
Even though learner satisfaction is critical to the quality of learner-centered curricula, learner satisfaction remains subjective and does not necessarily represent an improvement in knowledge or application. 25 We added to module reaction with data points on knowledge transfer along with other learning and behavior constructs in keeping with the Kirkpatrick evaluation model. 16 Baseline knowledge of the content matter was high, but these results were likely skewed by the inclusion of senior fellows (second and third years) in the analysis (Figure 1). Since the test content relates directly to the module, we would expect a score increase in the test when taken shortly after lesson completion. We also acknowledge that the score increase cannot be completely attributed to the module and may have occurred because of some uncontrolled factors including the use of other learning resources.
Knowledge scores of first-year fellows in pre and post-module quizzes compared to senior fellows (second- and third-year fellows combined).
Our form of online instruction was completed by learners independently which requires motivation and self-efficacy. Self-efficacy can impact learner behavior and greater self-efficacy has been associated with persistence in learning.26,27 The positive response on confidence as reported by our learners could explain the improvement in knowledge and reported practice behaviors. While there is probable bias with self-reporting of practice change, it can represent actual behavior change in the context of measuring outcomes related to medical training programs. 28 The majority of learners indicated that they would change how they document airway risk assessment in their procedural notes and this was in fact what we observed in the procedural notes. All fellows use templates for procedural documentation, though these do not require the inclusion of the components of the MACOCHA. Improving these templates could also have the potential of conditioning the fellows into needing to think of airway assessment risk in a more systematic way.
In the evaluation process, we sought to determine how the module's content was applied to daily clinical practice and what barriers existed to fellows implementing its content to clinical practice. Some potential barriers included patient differences such as the variations in patient populations. An intubation plan must be individualized, and one cannot teach every nuanced scenario that may arise in a short web module. Different disease processes may need different airway preparation, which includes allowing airway experts to take over. This was echoed by one of the participants who pointed out that there were circumstances where airway intubation simply should only be attempted by the DART team. Another barrier to implementation was remembering the content to apply each time. One of the participants suggested, “A one-page cheat sheet would be helpful as a quick guide.” Visual aids to help remember technical concepts such as ramping may also be beneficial. Several short refresher modules spaced throughout the year could also help reinforce the practices.
Intubation is an essential skill in critical care training, yet there is a shortage of publications that focus on trainees in this specialty.5,10 Much of the literature on this topic is related to trainees in anesthesiology and emergency medicine. While there is significant overlap among the disciplines, there are limitations to directly applying that literature to critical care fellows because of contextual variations and specialty competencies. The critical care training environment is characterized by the complexity of patients, the need for time-sensitive decision-making, and a team-based approach to intubations. Given these factors, learning strategies that are specifically designed for critical care trainees are needed to address these unique challenges.
Limitations
Using an experimental design for this study would not have been appropriate partly because our convenience sample of participants was small and as such, a power analysis was not conducted. A large multicentred experimental design would have protected against threats to internal validity. However, since there was the potential to improve patient care by improving learner behavior and knowledge, we did not want to withhold the educational content from a potential control group.
We identified some factors as potential confounders that may limit the ability to measure the attributable efficacy of the module and its generalizability. The factors included experience in the procedure as reflected by postgraduate year (higher year = more experience) and prior residency training (EM vs Internal Medicine) since EM residents would have more intubation experience than their internal medicine counterparts. This possibly explains the high pretest scores and baseline first-pass success rates. Additionally, known difficult airways would be reserved for the DART team and this may have limited the number of truly difficult airways included in our analysis. Another limitation of this study was our assumption of attributable educational efficacy. This was difficult to study because while we assumed that all fellows completed the online module and learned from it, we could not control for what was learned independently. Outcomes including the number of intubation attempts, successful intubations, and procedural complications were not assessed as study outcome measures. Relying on procedural documentation, self-reported first-pass success, and self-reported confidence lends itself to social desirability and reporting bias. It may be that the fellows only submit procedure notes and New Innovations logs where they were successful in completing the procedure. One way to improve this is by corroborating findings with an observer checklist grading of the fellows’ performance, though this is resource exhaustive. Another possibility would be to have a respiratory therapist record the number of intubation attempts. We also could not confirm the accuracy of fellows’ airway risk stratification beyond the confines of the module assessment. Corroborating fellow real-life findings with an expert's findings would increase the reliability of the findings. Since the assessment and surveys were completed shortly after the module, this may only assess their immediate recall which might result in recall bias.
Conclusion
This report describes the successful design and implementation of a web-based preintubation airway assessment and preparation module. Following completion of the course, our learners have a greater appreciation of the importance of structured steps and preparation before intubation. Learning about the patient population that is unique to the intensive care unit, their needs, special considerations and the resources available helped drive a safety-focused culture at our institution. Further multicentered studies looking at methods of acquiring competency in intubation skills in ICU airway management are needed.
Supplemental Material
Supplemental Material
sj-docx-1-mde-10.1177_23821205231210066 - Supplemental material for The Impact of a Web-Based Preintubation Preparation Module on the Knowledge, Confidence, and Behavior of Critical Care Fellows: A Single-Center Pilot Study
Supplemental material, sj-docx-1-mde-10.1177_23821205231210066 for The Impact of a Web-Based Preintubation Preparation Module on the Knowledge, Confidence, and Behavior of Critical Care Fellows: A Single-Center Pilot Study by Fahad Alroumi, Ernest Dinino, Mark Tidswell, Rebecca Blanchard and John Shatzer in Journal of Medical Education and Curricular Development
Supplemental Material
sj-pdf-2-mde-10.1177_23821205231210066 - Supplemental material for The Impact of a Web-Based Preintubation Preparation Module on the Knowledge, Confidence, and Behavior of Critical Care Fellows: A Single-Center Pilot Study
Supplemental material, sj-pdf-2-mde-10.1177_23821205231210066 for The Impact of a Web-Based Preintubation Preparation Module on the Knowledge, Confidence, and Behavior of Critical Care Fellows: A Single-Center Pilot Study by Fahad Alroumi, Ernest Dinino, Mark Tidswell, Rebecca Blanchard and John Shatzer in Journal of Medical Education and Curricular Development
Supplemental Material
sj-pdf-3-mde-10.1177_23821205231210066 - Supplemental material for The Impact of a Web-Based Preintubation Preparation Module on the Knowledge, Confidence, and Behavior of Critical Care Fellows: A Single-Center Pilot Study
Supplemental material, sj-pdf-3-mde-10.1177_23821205231210066 for The Impact of a Web-Based Preintubation Preparation Module on the Knowledge, Confidence, and Behavior of Critical Care Fellows: A Single-Center Pilot Study by Fahad Alroumi, Ernest Dinino, Mark Tidswell, Rebecca Blanchard and John Shatzer in Journal of Medical Education and Curricular Development
Footnotes
Acknowledgments
We would like to thank the participating fellows, all divisional faculty, and the Department of Medicine Leadership for supporting the implementation of this module. We would also like to thank the Program Director and Course Instructors at the Masters of Education in the Health Professions (MEHP) Program at the Johns Hopkins University School of Education.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
