Abstract
Objective:
Bag mask ventilation (BMV) is fundamental to airway management. Simulation is effective in airway management training, but its effectiveness for difficult BMV training is less clear. We evaluated the difference between type of training (simulation vs on patients) and the pass rate on a post-test on patients.
Design:
A single center pilot study was performed with 32 medical students randomized to participate in difficult BMV training on simulators or patients. Pre- and post-training tests on the simulator and on patients were recorded. Surveys of trainee confidence level were collected. The primary goal was to estimate the difference between type of training (simulation vs on patients) and the pass rate on the post-test on patients with an improvement of 10% or more in passing rate considered as a meaningful improvement. Secondary outcomes included whether or not participants passed the simulator post-test, post-test on patient confidence, and pre- and post-test confidence.
Measurements and main results:
Participants trained on the simulator had 13% higher passing rate on the post-test on patients compared to participants trained on patients (88% vs 75%). In addition, subjects that passed the simulator post-test had 11 times the odds of passing a post-test on patients relative to subjects that did not pass the simulator post-test (P = 0.023, odds ratio = 11.0, 95% confidence interval [CI] = 1.48-81.6). Post-training confidence levels were higher among those who passed the simulator pre-test and post-test and received simulator training.
Conclusions:
Simulation training for difficult BMV led to a higher passing rate on a post-test on patients compared to those trained on patients. This finding will need to be confirmed in larger randomized controlled trials. Successfully completing difficult BMV training on a simulator with a passing grade correlated with passing a test on difficult BMV on patients.
Keywords
Introduction
Difficult bag mask ventilation (BMV) has been defined as an inability to establish ventilation with a bag mask due to inadequate mask seal, excessive air leak, or excessive resistance to the ingress or egress of air. 1 It is relatively common, especially in patients with upper or lower respiratory tract pathology. 2 Bag mask ventilation is life-saving for first responders in cases of cardiac and/or pulmonary arrest, yet difficult BMV lacks standardized training or evaluation methods. Bag mask ventilation is taught briefly during Advanced Cardiac Life Support training, but the manikins (ie, Rususci Anne) are not difficult to ventilate, and there are inherent risks in teaching difficult BMV on patients. Simulation-based education is now widely implemented in medical training 3 and may offer a solution. 4
By using a computerized patient simulator (CPS) such as SimMan®, increased airway resistance can be created to represent difficult BMV. A previous study validated a simulation model for difficult BMV training as a method for teaching and differentiating skill levels; 5 however, the effectiveness of this training model has not been evaluated on difficult BMV for real patients. We tested the hypothesis that successfully completing difficult BMV training on a simulator with a passing grade was predictive of passing a test on difficult BMV on patients.
Methods
Study subjects
Eligible subjects were third-year medical students without previous training in difficult BMV who volunteered and provided informed consent while on their anesthesia elective. At the Medical University of South Carolina (MUSC), it is an educational standard that medical students are trained in BMV during electroconvulsive therapy (ECT) procedures while on anesthesia electives, therefore, patient consent was not required by the Institutional Review Board (IRB). Anesthesiologists staff all ECT procedures at MUSC and perform BMV while patients receive general anesthesia. The IRB at MUSC approved this study. Along with providing informed consent, students were informed of the following. As an educational intervention and not as as part of their class grade, they would be pre-tested, taught BMV, and then post-tested. Because the study was on live patients, patient safety would be a priority, and an anesthesiolgist may take over BMV at any time. Before and during pre-testing, no instruction would be provided. During either pre- or post-testing, they would only have supplies to perform BMV and they could ask for assistance commensurate with what could be provided by health care provider not trained in difficult BMV.
Study design
This pilot study at MUSC included 32 medical students randomized into 2 groups. Simple randomization using a pseudo-random number generator was performed by the study coordinator (Figure 1). Both groups were pre-tested on 1 patient in the ECT suite predicted to be difficult to BMV and on simulated patients. Group 1 trained in difficult BMV on patients during ECT, and group 2 trained in difficult BMV on a simulator. Following the pre-test, group 1 observed an attending anesthesiologist perform BMV and then ventilated 5 ECT patients with direct feedback (ie, traditional procedure training). A post-test was administered on a patient in the ECT suite and on the simulator. After pre-testing, group 2 underwent the Basic Emergency Airway Management (BEAM) training course and their performance was evaluated with post-testing in the same manner as group 1.
Study design.
The BEAM training course uses a CPS (SimMan®) modified by inserting air flow restrictors into the inspiratory pulmonary limbs in a standardized manner (Figures 2–4) along with a programmed scenario to support formative and summative feedback. 6 The same programmed scenario was shown to every student. One physician taught the course to 1-3 students at a time. The course began with a 15 minute tutorial on difficult BMV followed by 45 minutes of hands-on practice per learner with direct instructor feedback until proper technique was achieved. Testing on the simulator was scored as Pass (P) for final SpO2 ⩾ 80% or Fail for final SpO2 < 80. The SpO2 on the simulator was based on the airflow generated through the system. The test was for 2 minutes and failure was immediate for any SpO2 ⩽ 50%. Students were instructed not to discuss the testing with any one. Debriefing was performed as part of the BEAM course during the hands-on practice. Aspects discussed included actions that went well, actions that could be improved, participants’ confidence levels, and participants’ questions about BMV techniques.
The manikin’s airway connected to the sensor, no restrictor.
A restrictor in place.
The location of the sensors.
All ECT procedures were staffed by board certified anesthesiologists who identified patients for difficult BMV if at least 1 of the following independent predictors was present: obesity, age > 55, history of snoring, lack of teeth, a beard, Mallampati Class III or IV, or limited mandibular protrusion.7-9 The “MOANS” mnemonic is the most commonly cited clinical assessment tool for predicting difficult BMV and incorporates these factors. “M” represents mask seal, specifically factors that make for a difficult mask seal, such as a beard, blood, or facial injuries. “O” represents obstruction or obesity (body mass index [BMI] > 30), “A” represents aged patients (>55). “N” represents no teeth. “S” represents stiff lungs or chest. The presence of 1 of these 5 features was necessary for a patient to be included in study. The same anesthesiologist scored each participant as “pass” if all 3 of these criteria were met: (1) maintenance of SpO2 ⩾ 92%; (2) maintenance of end-tidal CO2, as measured by capnography, at a level no more than 10% greater than the baseline value; and (3) the presence of chest rise during BMV for the entire ECT procedure. If any of these parameters were not met, the anesthesiologist took over BMV and the student was scored as fail. Participants were permitted to place an oral-pharyngeal airway or ask for help performing 2-person BMV if necessary. Surveys recorded confidence levels prior to and after training, whether participants had prior BMV training, and if students believed BMV is important skill master. The attending anesthesiologist was blinded as to whether the trainee had completed the BEAM course.
Outcome measurements
The primary goal of this pilot study was to estimate the difference between type of training (simulation vs on patients) and the pass rate on the post-test on patients. Secondary outcomes included whether or not participants passed the simulator post-test, ordinal score on the simulator and ECT post-test, and pre- and post-test confidence. Additional information collected on each participant included sex, whether or not they had prior BMV training, whether or not they considered it an important skill, score on the pre-test on the simulator, whether or not they passed the pre-test on a patient, whether or not they used the 2-person technique for BMV, hand size as measured from outstretched tip of thumb to tip of fifth finger, hand size as measured from the midline of the most proximal part of palm to the tip of middle finger, hand area which was evaluated by multiplying the 2 hand sizes together, and level of confidence prior to and after completing the training (on a 5-point ordinal scale: very confident = 1, somewhat confident = 2, neutral = 3, somewhat uncomfortable = 4, very uncomfortable = 5). Participants completed de-identified survey forms which were collected prior to and at the completion of training.
Statistical analysis
This study was a pilot study to compare post-test passing rates between participants receiving simulation vs ECT training for BMV and was designed to provide preliminary evidence of differences in post-test differences. A priori sample size calculations were done to estimate the width of a 95% confidence interval (CI) around the difference in the proportion of participants who pass the post-test by training (simulation vs ECT training on patients). An improvement of 10% or more in passing rate on the ECT post-test between traditional versus simulator training would be considered a meaningful improvement in passing rate. Assuming 75% of participants in the ECT training group and 85% of those receiving simulation training are expected to pass the post-test, a sample size of 16 subjects per group (32 total) produces a 2-sided 95% CI for a difference of 10% in the proportion of subjects who pass after simulation versus ECT training with a width that is equal to 55%, or a difference of 10 + 22.5%. If the difference in proportions is larger, meaning the simulation group has a passing rate greater than 85%, the CI will be tighter.
Descriptive statistics for participant characteristics across all participants and by training type were estimated. Comparisons of participant characteristics by training type were also examined using Fisher’s exact tests for categorical variables and 2-sample t test or Wilcoxon rank sum tests for continuous variables to determine if randomization worked appropriately. No formal hypothesis testing of primary or secondary outcomes was conducted due to the pilot nature of the study. Differences and 95% CIs for passing rate on the simulator and ECT post-tests between the 2 different training groups and for other categorical variables were estimated using Clopper-Pearson exact CIs. Differences in continuous variables between those who passed or failed the ECT post were evaluated by estimating the mean difference or median difference and 95% CIs. Confidence intervals for difference in the median were estimated using Hodges-Lehmann CIs. All analyses were conducted in SAS v. 9.4.
Results
Of 32 participants, 53% were male and 47% were female. Ninety-seven percent of participants felt BMV was an important skill, and 34% indicated that they had received prior BMV training, though none had received training in difficult BMV. There were no notable differences between the 2 groups in sex, level of training, or hand size (Table 1). Prior to training, there were no significant differences between the groups in confidence level, passing the simulator pre-test, placement of an oral airway in pre-test, achievement of chest rise, or conversion to a 2-person technique prior to receiving training. However, participants in the SimMan simulation group had a significantly higher passing rate on the simulator post-test compared to the group trained on live patients. Comparisons of performance metrics and confidence prior to and after training in the 2 training groups are shown in Table 2.
Characteristics of the study population across all study participants and within group.
Abbreviations: ECT, electroconvulsive therapy; BMV, bag mask ventilation.
Categorical variables are reported as n (%) and continuous variables are reported as mean (SD).
One participant in the SimMan group did not answer this question.
Performance metrics and participant confidence before and after the training intervention across all study participants and within group.
Abbreviations: ECT, electroconvulsive therapy.
Categorical variables are reported as n (%) and ordinal variables are reported as median (25%, 75%).
Most (81%) participants passed the ECT post-test, and participants trained on the simulator had an 88% passing rate on the post-test on patients relative to a 75% passing rate among participants trained on patients (difference in ECT post-test pass rate of 12.5%, 95% CI = 24.7%-47.3%). Ninety-two percent of participants that passed the simulator post-test also passed the ECT post-test compared to only 50% of those who did not pass the simulator post-test (difference in ECT post-test pass rate of 41.7%, 95% CI = –33.2%-74.0%). All participants that passed achieved correct oral airway placement while all those that failed did not. In addition, 89.5% of participants that used 2-person conversion in the post-test passed the ECT post-test compared to only 69% of those who did not use 2-person conversion (difference in ECT post-test pass rate of 20.2%, 95% CI = –14.6%-52.7%). There were no notable differences in ECT post-test passing rate by sex, having received prior BMV training, pre-training confidence level, passing the simulator pre-test, passing the ECT pre-test, type of training received, conversion to the 2-person technique, placement of an oral airway, or hand size. Comparisons for passing the ECT post-test are provided in Tables 3 and 4.
Comparison of passing rate on the ECT post-test for participant characteristics by outcome on the ECT post-test.
Abbreviations: ECT, electroconvulsive therapy; CI, confidence interval; BMV, bag mask ventilation.
Passing rates by group are reported as n (% passing) and the difference in passing rate by participant characteristic with 95% CI. Confidence intervals for the differences in passing rates are estimated using Clopper-Pearson exact 95% CIs.
Comparison of continuous participant characteristics by outcome on the ECT post-test.
Abbreviations: ECT, electroconvulsive therapy; CI, confidence interval.
Values reported include the mean (SD) or median (25%, 75%) among those who passed and failed and the mean difference (95% CI) between those who pass versus fail. The confidence intervals for the median difference are estimated using Hodges-Lehmann 95% confidence intervals.
Most (75%) participants passed the simulator post-test. All participants in the simulation training group passed the simulator post-test compared to only 50% of those who received traditional training. In addition, 92% of participants that passed the simulator post-test achieve appropriate chest rise compared to only 50% of participants that failed the simulator post-test. There were not any notable differences in passing rates on the simulator post-test by gender, having prior BMV training, pre-training confidence level, whether or not they passed the simulator pre-test, whether or not they passed the ECT pre-test, use of the 2-person conversion technique either pre- or post-test, or hand size. Comparisons for passing the simulator post-test are provided in Table 5.
Comparison of passing rate on the simulator post-test for participant characteristics by outcome on the ECT post-test.
Abbreviations: ECT, electroconvulsive therapy; CI, confidence interval; BMV, bag mask ventilation.
Passing rates by group are reported as n (% passing) and the difference in passing rate by participant characteristic with 95% confidence interval. Confidence intervals for the differences in passing rates are estimated using Clopper-Pearson exact 95% confidence intervals.
We also examined comfort level of participants prior to and after training on a 5-point ordinal scale with 1 representing very comfortable and 5 representing very uncomfortable. Prior to training, almost 70% of participants reported being somewhat uncomfortable with BMV, 18.8% reported feeling neutral, and 12.5% reported feeling somewhat confident prior to training. Prior to training, males had a median confidence level of 4 (somewhat uncomfortable) compared to females who reported a median confidence of 5. Participants that reported having prior formal BMV training also reported higher pre-training confidence (median comfort of 3 vs 4). Participants that passed the simulator pre-test also reported higher confidence levels prior to training (median comfort of 2.5 vs 4). Participant pre-training confidence level was not notably different by whether or not subjects passed the ECT pre-test, conversion to the 2-person technique in the pre-test, placement of an oral airway during the pre-test, or whether or not they achieved appropriate chest rise during the pre-test.
After training, most participants reported feeling at least somewhat confident (75.0%). In addition, greater than 90% of participants indicated an increase in confidence level after training across both groups. Post-training confidence level were higher among those with higher pre-training confidence, those who passed the simulator pre-test and post-test, and among those who received simulator training. Subjects that passed the simulator pre-test had a median post-training confidence of 1 compared to a median of 2 among those who did not pass the simulator pre-test. In addition, participants that passed the simulator post-test had a median post-training confidence of 2 compared to 2.5 among those who did not. In the group that received traditional training, 87.5% reported feeling at least somewhat confident after training compared to 62.5% in the traditional training group. Post-training confidence level did not differ with having received prior BMV training, whether or not they passed the ECT pre-test or post-test, conversion to the 2-person technique in the post-test, placement of an oral airway during the post-test, or whether or not they achieved appropriate chest rise during the post-test. Figure 5 shows boxplots of the distribution of self-reported confidence level prior to and after training by training group.
Confidence level by training group.
Discussion
This study has several important findings. First, there was a higher passing rate on the post-test on patients among those trained on the simulator versus those receiving traditional training in this pilot study. Specifically, participants in the simulator training group showed a 12.5% increase in ECT post-test passing rates which is similar to the proposed clinically meaningful improvement of 10%. While larger randomized controlled trials are needed to prove a statistically significant difference in pass rates, our results suggest that use of outcomes driven simulation may be as effective or more effective than traditional training. Second, successfully completing difficult BMV training on a CPS with a passing grade was predictive of passing a test on difficult BMV on patients. Finally, while both groups exhibited an overall increase in confidence after receiving training, participants trained on the simulator reported higher confidence levels.
Previously, simulation training for difficult BMV was validated as a teaching technique on a CPS; 6 however, this is the first study to suggest that there is predictive validity of this training, such that passing a post-test on a simulated patient predicts passing a similar test on a real patient. In addition, while outcomes of multiple studies have supported the effectiveness of simulation training in airway management,4,5,10-14 these studies did not focus exclusively on simulated difficult BMV as in this study. This study is timely as high fidelity mechanical ventilation simulation in medical education is gathering momentum. Spadaro et al compared computer screen–based simulation with mannequin-based simulation for teaching mechanical ventilation to residents in the intensive care unit. Residents were more satisfied with the mannequin/simulation approach compared to the computer screen alone. They also performed better on the high fidelity simulation scenarios if they had the mannequin training compared to the computer training. 15
In practicality, up to 8 trainees can be simultaneously trained safely in BMV using 4 simulators and cooperative, mastery-based training methods in less than 1 hour by 1 faculty. Traditional training by anesthesiologists in the operating room is performed with 1 trainee per room per attending, and time costs for average operating rooms run upward of $ 150/minute.
While there was higher likelihood of passing a post-test on patients among participants trained by simulation (88% pass rate) compared to the group trained on patients (75% pass rate), this pilot study was not powered to show statistical significance. However, passing the simulator post-test appeared clinically meaningful as it was associated with 11 times the odds of passing the post-test on patients relative to subjects that did not pass the simulator post-test (P = 0.023, 11.0, 95% CI = 1.48-81.6), and passing on patients was significantly associated with simulator post-test score across both training groups (P = 0.002). Since most of both study groups passed on the ECT post-test, this suggests that both training modalities were effective; however, simulation training accomplished its objective without posing risk to patients. In addition, the simulation model allows for the incorporation of tutorials and repetitive practice in the same setting which is arguably a more comprehensive training modality than the apprenticeship model which can be hurried due to clinical and safety demands. Training initially on patients may also introduce patient variability due to the type of patients scheduled during a given trainee rotation. Thus, a trainee may only see patients who can be easily ventilated with BMV, whereas simulation can control level of difficulty.
Participants trained on the simulator reported higher levels of confidence possibly due to the ability to practice recovering from severe oxygen desaturations which was not allowed based on the parameters set for patients receiving ECT. These were patient rather than learner centric and mandated that the anesthesiologist take over BMV at a lower threshold than what would stop the simulator. Simulation training still required adequate oxygen saturations at the end of 2 minutes to pass, but transient drops in SpO2 were allowed as long as they were not ⩽50%. This provided an important opportunity for trainees to trouble shoot and learn to adjust mask seal or hand position, add an oral or nasal airway, or transition to a 2-person technique (critical interventions for success in difficult BMV).
Experience plays a central role in a physician’s learning process and differentiates the experiential approach (ie, hands-on training) from other learning theories. Arguably, it was less experiential for an attending anesthesiologist to “rescue” the learner compared to when the learner had to trouble shoot their technique. In addition, the simulation training program was standardized, has been validated in a prior study, 6 and is potentially reproducible. By first training in a clinical setting with rotating attending physicians, there may be heterogeneity in teaching styles, some not accounting for different learning styles of the participants. 16
There were limitations in the study design. Due to the pilot nature of the study and the small sample size, we did not evaluate the statistical significance in the trend toward higher passing rate on patients with simulation training, and we could not state non-inferiority as either design would have required a much larger sample size than what was feasible for this pilot study. In addition, this study uses low fidelity resistance bands to create airway resistance. While calibrated in a standardized and reproducible manner, the resistance of the lung could have been represented by a lung simulator connected to the SimMan mannequin. Such devices could simulate different ranges of resistance while maintaining high fidelity simulation throughout the system.
This was a single center pilot study conducted to provide initial estimates of the effectiveness of simulator training for difficult BMV which can be used to design a larger study for a definitive assessment of the effectiveness of the proposed simulator training protocol. Thus, future directions include a multicenter study as the simulation component of the intervention, SimMan®, is now readily available and operational at most major medical centers. The simulation training model described here could be tested in non-physician groups and could become standard training as it is transportable and reproducible. Ultimately, this strategy has the potential to improve patient care without posing as much risk to patients.
Conclusion
In this randomized controlled pilot study, completing difficult BMV training on a simulator and passing a post-test led to a high passing rate on a post-test for difficult BMV in patients. Although this study was not powered to show statistical significance, there was a higher pass rate among those trained on the simulator versus those trained by the traditional method. Simulation training improved confidence levels of trainees embarking on BMV in live patients. Larger multicenter studies should be considered to further evaluate the effectiveness and implementation of this strategy to teach this life-saving skill.
Footnotes
Funding:
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Declaration of conflicting interests:
Dr Pastis received salary support as a percent effort from a Laerdal Foundation Grant to perform this research. Dr Tobin received salary support as a percent effort from a Laerdal Foundation Grant to perform this research. Dr Wolf received salary support as a percent effort from a Laerdal Foundation Grant to perform this research. Dr Schaefer receives royalties from the Laerdal Corporation for 2 patents in simulation. Dr Schaefer’s role in this study was as a mentor only and he did not participate in data collection or analysis per Medical University of South Carolina COI plan.
Author Contributions
NJP contributed to the concept of this trial, training and testing on the computerized patient simulator, writing of the manuscript, and further edits. CDT contributed training and testing on real patients and writing of the manuscript. BJW performed statistical analysis for the trial and participated in writing the manuscript. JGR provided editing and contributed to the writing of the manuscript. JJS provided consultation on the concepts and logistics of conducting this trial including the development of the resistance placed into the airway simulators.