Abstract
Background:
Simulation-based-mastery-learning (SBML) is an effective method to train nephrology fellows to competently insert temporary, non-tunneled hemodialysis catheters (NTHCs). Previous studies of SBML for NTHC-insertion have been conducted at a local level.
Objectives:
Determine if SBML for NTHC-insertion can be effective when provided at a national continuing medical education (CME) meeting. Describe the correlation of demographic factors, prior experience with NTHC-insertion and procedural self-confidence with simulated performance of the procedure.
Design:
Pre-test – post-test study.
Setting:
2014 Canadian Society of Nephrology annual meeting.
Participants:
Nephrology fellows, internal medicine residents and medical students.
Measurements:
Participants were surveyed regarding demographics, prior NTHC-insertion experience, procedural self-confidence and attitudes regarding the training they received. NTHC-insertion skills were assessed using a 28-item checklist.
Methods:
Participants underwent a pre-test of their NTHC-insertion skills at the internal jugular site using a realistic patient simulator and ultrasound machine. Participants then had a training session that included a didactic presentation and 2 hours of deliberate practice using the simulator. On the following day, trainees completed a post-test of their NTHC-insertion skills. All participants were required to meet or exceed a minimum passing score (MPS) previously set at 79%. Trainees who did not reach the MPS were required to perform more deliberate practice until the MPS was achieved.
Results:
Twenty-two individuals participated in SBML training. None met or exceeded the MPS at baseline with a median checklist score of 20 (IQR, 7.25 to 21). Seventeen of 22 participants (77%) completed post-testing and improved their scores to a median of 27 (IQR, 26 to 28; p < 0.001). All met or exceeded the MPS on their first attempt. There were no significant correlations between demographics, prior experience or procedural self-confidence with pre-test performance.
Limitations:
Small sample-size and self-selection of participants. Costs could limit the long-term feasibility of providing this type of training at a CME conference.
Conclusions:
Despite most participants reporting having previously inserted NTHCs in clinical practice, none met the MPS at baseline; this suggests their prior training may have been inadequate.
What was known before
Simulation-based-mastery-learning (SBML) is an effective method to train nephrology fellows to insert temporary, non-tunneled hemodialysis catheters (NTHCs). Prior studies of SBML for NTHC-insertion have been conducted at a local level.
What this adds
SBML for NTHC-insertion can be effective when provided in the setting of a continuing medical education (CME) conference.
Background
The ability to competently insert a temporary non-tunneled hemodialysis catheter (NTHC) is a requirement of nephrology training in Canada [1]. Nonetheless, evidence suggests that many Canadian nephrology trainees do not feel adequately trained to perform the procedure [2]. This is important because improper NTHC-insertion is associated with mechanical and infectious complications that are serious and potentially avoidable [3–5].
Simulation-based-mastery-learning (SBML) is a rigorous form of competence-based education in which learners complete pre-testing, deliberate skills practice with feedback and post-testing until they meet or exceed a predetermined minimum passing score (MPS) [6,7]. Learners who do not initially achieve the MPS undergo further deliberate practice and are retested until they achieve the MPS. In mastery learning, practice time may vary between learners but educational outcomes are uniform. Use of this model (and a stringent MPS) assures that all learners demonstrate their competency in a simulated environment before performing the procedure during actual clinical care [6,7]. SBML has been shown to be effective for multiple clinical skills, including NTHC-insertion [8,9]; however, most previous studies of SBML procedural-training have occurred at the level of a single training program and no information is available regarding the use of SBML in a continuing medical education (CME) setting.
The Canadian Society of Nephrology (CSN) annual meeting provides a unique opportunity to gather current and future nephrology fellows for specialized training during its pre-conference educational program. As such, we sought to determine if SBML for internal jugular (IJ) NTHC-insertion would be effective in this CME setting. The specific aims of this study were to: 1) evaluate the feasibility and effectiveness of NTHC-insertion SBML sessions at the CSN annual meeting; 2) describe the correlations of demographic factors, prior experience with NTHC-insertion and procedural self-confidence with simulated performance of the procedure; and 3) assess learners' perceptions of SBML as part of a national CME conference.
Methods
Study design
The study was a pre-test – post-test (before-after) design [10], assessing an NTHC-insertion SBML pre-conference course during the 2014 CSN annual meeting in Vancouver. Participants were Canadian medical trainees who registered for the CSN annual meeting pre-conference educational program or precourse. The precourse took place on April 24 and 25, 2014 in a conference room of the same hotel that hosted the CSN annual meeting. The Northwestern University Institutional Review Board (Chicago, IL, USA) and the Ottawa Hospital Research Ethics Board (Ottawa, ON, Canada) approved the study and all participants provided informed consent.
Procedure
Study participants provided demographic information including age, gender, year of post-graduate medical training; prior NTHC-insertion clinical experience (number of prior IJ NTHC-insertions) and; a rating of their self-confidence to perform NTHC-insertion competently using a scale of 0–100 (0 = not confident, 100 = extremely confident). Subsequently, all participants underwent an NTHC-insertion clinical skills baseline examination (pre-test). Pre-test clinical skills examinations were conducted using a previously described IJ NTHC-insertion checklist [9]. Over approximately 2 hours, all participants together completed a lecture, video presentation and ultrasound training. Subsequently, groups of 10 or less sequentially underwent approximately 2 hours of deliberate NTHC skills practice using the simulator with directed feedback. Practice sessions involved two participants and one faculty member (JHB, EC, CE, RM, JJP, MS) with expertise in teaching and performing NTHC-insertion at each of the 5 practice-and-testing stations. Participants were scheduled to return the following day for a repeat clinical skills examination (post-test). Both pre- and post-testing was organized in 30-minute increments during which 5 learners were tested simultaneously with each learner being evaluated by one trainer at their own practice-and-testing station. Consistent with the mastery model, participants who did not meet or exceed the MPS would then complete additional deliberate practice and retesting until the MPS was achieved. After post-testing, participants completed a course evaluation questionnaire. A random sample of one-third of pre- and post-test clinical skills examinations were video-recorded and rescored to assess inter-rater agreement between instructors.
We conducted the NTHC SBML course using the CentraLineMan™ (Simulab, Seattle, WA, USA) simulator that provides a realistic representation of the anatomy of the right upper torso and head. This simulator has IJ and subclavian (SC) veins and carotid and SC arteries. In addition, the simulator features an arterial pulse, different coloured venous and arterial ‘blood’, appropriate venous and arterial blood pressures and realistic tissues with self-sealing veins and skin into which needles, dilators and guidewires may be inserted. It is compatible with ultrasound so as to produce highly realistic ultrasound images. A portable ultrasound machine (SonoSite Inc., Chicago, IL, USA) was made available at each practice and testing station. In addition, NTHC kits, sterile gowns, sterile gloves, sterile drapes and sterile CVC-insertion trays (Cardinal Health, LLC., McGaw Park, IL, USA) were available in unopened packaging at each practice and testing station.
To ensure standardization of training and grading of checklist items, faculty instructors (EC, CE, JJP, MS, RM) underwent a four-hour ‘train-the-trainers' session led by an instructor with extensive NTHC SBML experience (JHB) the day before the trainees' precourse. During this ‘train-the-trainers' session, instructors reviewed the SBML curriculum and approaches to providing feedback, and practiced scoring simulated NTHC-insertions using the 28-item checklist.
Measurement
A 28-item skills checklist was used to complete clinical skills evaluations (Table 1). This was adapted from a previously published NTHC-insertion skills checklist [9]. Each skill or action required for safe IJ NTHC-insertion was listed in order, given equal weight and scored dichotomously (done correctly/done incorrectly). The MPS was previously established at 79% by a multidisciplinary panel [9]. If the SC approach was used, the carotid artery was punctured or more than 2 needle passes (punctures of the skin) occurred, the simulation was terminated and the remaining checklist items were marked incorrect.
Temporary dialysis catheter insertion checklist: percentage of participants who performed each checklist item correctly
To assess inter-rater agreement, video-recorded clinical skills evaluations were rescored by one of two authors (JHB, JJP) who were blinded to pre- or post-testing status and the original learners checklist score.
Statistical analysis
We evaluated differences between pre- and post-test scores using the Wilcoxon signed-rank test to assess the impact of the intervention. Spearman's rank correlation coefficients were calculated to assess relationships between demographic factors, prior procedural experience and self-reported procedural-confidence with pre-test scores. The difference between pre-test scores of nephrology fellows versus those of the other trainees was assessed using the Wilcoxon rank-sum test. Inter-rater agreement was assessed by evaluating the percentage of items upon which the two raters agreed. A Kappa coefficient was not calculated because the checklist was previously validated and there was complete agreement on several items. We evaluated differences between the original scores of the video-recorded clinical evaluations with the scores obtained through video review using the Wilcoxon rank-sum test. We performed all statistical analyses using IBM SPSS version 22 (Chicago, IL).
Results
Twenty-two participants (11 nephrology fellows, 10 internal medicine residents and 1 medical student) enrolled in the precourse and completed pre-testing and training. Table 2 reports demographics, NTHC-insertion experience and procedural self-confidence. The median pre-test score was 20 (IQR, 7.25 to 21) checklist items correct out of a possible of 28 with no participants attaining the MPS. Seventeen participants participated in post-testing sessions on Day 2. The median post-test number of correct items increased by 7 (p = 0.001) to 27 (IQR, 26 to 28) with all participants achieving the MPS on their first attempt. Figure 1 depicts participants' pre- and post-test clinical skills examination performance. The percentage of participants who performed each of the 28 checklist items correctly is detailed in Table 1.
Baseline demographics, prior experience and procedural self-confidence (n = 22)
Note: Values are expressed as mean ± SD or number (%).
Clinical skills examination (checklist) performance by 22 participants before simulation-based training and 17 participants after simulation-based training.
The 5 participants who did not return for post-testing were nephrology fellows who each reported performing at least 20 prior IJ NTHC-insertions in clinical practice. Their median pre-test score of 19 (IQR, 3.5 to 20) was similar to overall group performance. Overall, no significant difference in pre-test scores were observed for nephrology fellows (n = 11) compared with trainees who had less medical training (n = 11): the fellows' median score was 14 (IQR, 5 to 21) compared to 19 (IQR, 3.5 to 20) for the residents and student (p = 0.44).
Spearman's rank correlation coefficients demonstrated no significant correlations between pre-test scores and age (r = −0.19; p = 0.40), gender (r = −0.42; p = 0.06), years of post-graduate medical training (r = −0.31; p = 0.89), clinical experience (r = 0.90; p = 0.69) or procedural self-confidence (r = 0.17; p = 0.45).
For the evaluations video-recorded in order to assess inter-rater agreement (n = 13), raters agreed on 331/364 (91%) of the rescored checklist items. The difference between the median of scores obtained by initial grading (22 (IQR, 20.5 to 27.5)) and the median of scores obtained from video review (23 (IQR, 18.5 to 27)) was not significant (p = 0.88).
Of the 17 participants who completed a post-training survey, 14 ‘strongly agreed’ (5/5 on the Likert scale) and 3 ‘agreed’ (4/5 on the Likert scale) that SBML-training improved their ability to competently insert NTHCs. As well, 15 ‘strongly agreed’ and 2 ‘agreed’ that this type of training should be a required component of nephrology fellowship training.
Discussion
This study shows that SBML can be used to train current and future nephrology fellows in NTHC insertion at a national CME meeting. Consistent with the findings of earlier research [9], this program significantly improved trainee NTHC-insertion skills. No participant met or exceeded the MPS at baseline, however all met or exceeded the MPS on their first attempt after the educational intervention. Participants strongly endorsed that SBML improved their NTHC-insertion skills and that training of this type should be a required component of nephrology fellowship.
We believe that one of the strengths of this training intervention was the maintenance of a high level of realism during pre-testing, post-testing and deliberate-practice sessions. Anecdotally, many participants commented that they had received ‘partial’ simulation training for NTHC-insertion in the past, typically involving the use of a patient-simulator and ultrasound machine to repeatedly insert the finder needle into the IJ vein. Forcing participants to complete the entire procedure, using all necessary supplies in their original packaging, allowed them to encounter and overcome difficulties they will face in clinical practice but not during less realistic simulations: for example, the challenge of maintaining sterile technique while preparing the cover for the ultrasound probe only becomes evident during highly realistic simulation.
In addition to failing to achieve the MPS at pre-test, most trainees committed potentially serious errors. Notably, over-advancing the guidewire (beyond 20 cm) can provoke serious ventricular arrhythmias, particularly in patients with acute kidney injury [11]. This is a concerning finding as nearly all participants indicated having inserted at least 3 NTHCs in clinical practice before completing the simulation-based intervention. This confirms the findings of a recent survey in which many Canadian nephrology trainees reported inserting NTHCs despite feeling inadequately trained to do so [2]. Taken together, results of this study and others [2,8,9] suggest that providing SBML-training for NTHC insertion to current and incoming nephrology fellows at the annual CSN meeting could improve patient-safety. We suggest that training be mandatory because procedural self-confidence and prior experience inserting NTHCs did not correlate with proficiency.
SBML has many potential clinical benefits for Canadian nephrology training programs. SBML for CVC insertion reduces the number of needle passes per insertion [12,13], arterial punctures [13], line malpositions [13], insertion failures [13] and central line-associated bloodstream infections (CLABSI) [14]. Nonetheless, some experts have suggested that it may be prohibitively time-consuming and labour-intensive to require training, of any type, for NTHC-insertion, across all nephrology training programs [15]. Our findings suggest that a national CME meeting can be an effective venue to train a relatively large group of trainees from multiple centres, thus allowing training resources to be pooled. While there is evidence to suggest that substantial savings might result from reduced rates of CLABSIs [16,17], additional costs associated with conducting this type of training at a conference include accommodation expenses, flights, shipping costs, and costs related to trainers' travel time. These additional costs may be mitigated somewhat if a large number of potential trainers and trainees plan to attend a conference for reasons other the training course alone. Companies may also be more willing to donate supplies for training in the context of a national meeting. As such, the longer-term feasibility of providing this type of training to nephrology fellows from across Canada annually would depend upon the results of a formal evaluation of its cost-effectiveness which was beyond the scope of this study. In the case of this study, all supplies and trainers' time was donated leaving only travel expenses. This limits the generalizability of our findings with respect to providing similar programs at future conferences and in other settings. Training of all Canadian nephrology fellows at a CME meeting might not be necessary as some trainees may be able to receive similar training in a more cost-effective manner at their local training sites. SMBL-training for NTHC-insertion could be particularly helpful for filling in gaps in this training on a national level, particularly for nephrology fellows from centres where SMBL-training for NTHC-insertion may not be offered. One particular advantage of SBML training is that is enables standardization of procedure competence across many types of learners (e.g. nephrology fellows, nephrologists, internists) working in different settings.
This study has several other important limitations. The small sample size and self-selection of participants limit the generalizability of our findings. It is unknown if those who participated are more or less likely to be proficient at NTHC-insertion than their colleagues who did not elect to participate. We did not assess skill retention although this has been reported elsewhere [8]. Additional long-term studies are required to better define the duration for which procedural skills are retained following SBML, particularly when these skills are not used frequently in clinical practice. Use of a CME setting may allow for this important study at future CSN meetings. Finally, 5 of the 22 (23%) participants did not return for scheduled post-tests. The need for participants to devote a relatively large block of time to NTHC-insertion training while other conference events were taking place may be responsible for this finding. Because pretest scores and self-confidence were similar between those who dropped out and those who completed the program, we strongly believe SBML training should be mandatory.
Conclusions
NTHC-insertion SBML during a national CME conference is an effective method to train current and future nephrology fellows. The long-term feasibility of providing this type of training at a national conference on a recurring basis would depend upon a formal evaluation of its cost effectiveness. Most participants reported having significant experience previously inserting NTHCs. However, pretest performance on a clinical skills examination was poor. SBML should be considered for incoming nephrology fellows prior to performing actual NTHC-insertions on patients.
Footnotes
Abbreviations:
JHB and DBW's institution has received educational grants from ‘Medical Error Reduction and Certification, Inc.’, Seattle, WA, USA.
EC conceived of the study, participated in its design, co-ordination and data collection, and drafted the manuscript. DBW conceived of the study and participated in its design, and helped to draft the manuscript. CE, RM, JJP, MS and SH participated in the co-ordination of the study and data collection, and helped draft the manuscript. JHB conceived of the study, participated in its design, co-ordination and data collection, and helped to draft the manuscript. All authors read and approved the final manuscript.
Acknowledgements
The authors thank the medical trainees who participated in this program for their dedication to education and high quality patient care.
This study was facilitated through the donation of supplies and use of equipment, provided at no charge, by: Simulab, Seattle, WA, USA; Sonosite Inc., Chicago, IL, USA; Cardinal Health, LLC., McGaw Park, IL, USA.