Progressive Medical Simulation: An Analysis of the Integration of Progressive and Personalized Learning in Central Line Simulators

Abstract

Progressive learning gradually increases task difficulty as students advance in their education. One area that can benefit from it is medical education since it can optimize medical trainees’ skill acquisition. While progressive learning can allow for skill transfer to patient encounters, personalized learning increases the efficiency and effectiveness of learning. However, it is not well understood the number of practice trials needed to reach proficiency. To evaluate whether progressive and personalized learning can enhance medical trainees’ learning gains, the learning interface of the Dynamic Haptic Robotic Trainer (DHRT) for Central Venous Catheterization was assessed. Results showed that residents’ performance on the DHRT did not differ based on task difficulty and residents’ performance was as effective with less number of trials. The findings imply a need to integrate progressive and personalized learning on the DHRT simulator to ensure that residents are fully prepared for any patient scenario in a real-life encounter.

Keywords

medical simulation medical training progressive learning personalized learning simulation-based training skill acquistion medical education proficiency in medical procedure central venous catheterization simulators

Introduction

Over the last two decades, medical education has shifted from the apprenticeship model of “see one, do one, teach one” to the concept of “see one, simulate many, do one competently and teach everyone” (Vozenilek,2004). This shift is due to the advancements in medical simulators as a means of training to advance clinical skill acquisition of medical students (Dawson,2007). While research has shown simulators to be effective for medical skills acquisition (Evans et al., 2010), training should also consider the most effective method for students’ learning process and its impact on successful skill acquisition.

One learning technique for obtaining the required and necessary skills that could be implemented in medical simulation-based training is progressive learning. Progressive learning is based on the concept that as student’s skills advance through training, there should be gradual modifications in the simulator characteristics which include increasing task difficulty and the content of information provided (Brydges et al., 2010). This implies that trainees should start with easier tasks and then progress to more challenging ones that integrate another set of skills, thus building skills as the trainee advances their knowledge. In addition, research has shown that progressive learning optimizes skill transfer to realistic patient encounters, where the trainee must coordinate all of their skills during clinical practice (Kneebone R et al., 2006) and leads to greater skill mastery (Issenberg, 2005). Brydges et al., (2010) studied the effect of increasing the level of difficulty using progressive learning on undergraduate medical students’ performance in intravenous (IVcatheterization during simulation training. Results showed that the cohort using progressive learning had better clinical performance scores compared to students that did not complete the tasks using a progressive learning approach (Brydges et al., 2010).

While the integration of progressive learning enhances learning gains and performance, one area that has yet to be explored is if the task is representative of their level of difficulty. This is particularly important to ensure that progressive learning can be effectively integrated. Previous research has effectively used opinions from 11 subject matter experts (SME) across different hospitals to understand and identify 15 essential problems associated with the endoscopy medical procedure (Hildebrand, 2011). Furthermore, another study used 24 SMEs from 16 surgical societies in the USA and 9 internationally to reach agreement on the validity and a wide range of perspectives needed for simulation-based curriculums (Zevin et al., 2012). This implies that SMEs’ opinions can be beneficial in providing useful knowledge for medical education. However, limited research has studied whether SMEs opinions are effective in determining tasks with different levels of difficulty which can be used to integrate progressive learning.

Another form that has emerged in medical education is personalized learning. Chatti et al. (2010) defines personalized learning as the flexibility of the learners to approach their education in a manner they feel that works best for them. The key benefits of this approach are that it increases learner’s satisfaction, speed of learning, and efficiency of learning (Gomez et al, 2014). One form of personalized learning that has been studied in medical simulation is self-guided learning which refers to individuals owning their knowledge by being active in their behavior and metacognition (Brydges, 2009). Research has shown that medical students who underwent personalized learning using a self-guided approach during simulator training achieved better retention of skills (Brydges, 2009). However, no research has studied if all medical learners need the same number of trials during simulation training, or if integrating personalized learning by having the number of trials based on their learning needs can potentially improve performance. This is particularly important because residents have expressed that an 80 work-hour restriction has impacted their education negatively (Krug,2017). In an attempt to improve upon this, simulation training using personalized learning could help them use their 80 work hours more efficiently while positively impacting their performance.

To our knowledge, this is the first study to analyze whether personalized learning with regards to the number of trials completed can be beneficial in improving medical resident’s performance during simulation training. This paper also builds upon the existent limited research by exploring if progressive learning can be integrated using predetermined patient profile cases by only three SMEs from only one medical institution. Specifically, we sought to answer these questions through an exploration on a Dynamic Haptic Robotic Trainer (DHRT) for Central Venous Catherization (CVC). The DHRT was designed to improve upon current training methods for CVC. While the DHRT simulator has been proven to distinguish between novice residents and experienced surgeon’s skill acquisition (Yovanoff, 2017), investigating whether the skill gained by residents are achieved in a progressive manner has not yet been done.

The patient profile cases used for skill training on the DHRT include three difficulty levels which are easy, medium or difficult. The difficulty of the patient cases was based on anatomical characteristics, multiple needle insertions and prior catherization based on three expert opinions from one medical institution (Gonzalez-Vargas, 2020). The easy patient profile is considered a thin patient with no pre-existing conditions. An average patient profile might depict a patient with head injuries and one that requires hypertonic saline. The difficult patient profile includes a patient that has preexisting conditions such as minimal change disease and requires dialysis. The last proficiency trial, also known as the verification of proficiency (VOP) on the DHRT, which illustrates the resident’s competency in CVC, could present a patient with cardiac problems requiring coronary bypass grafting and has hypotension.

To further improve the DHRT simulator, this study was designed to evaluate whether the level of difficulties previously assigned are effective in integrating a progressive learning approach. Furthermore, it was designed to determine whether the implementation of a personalized learning approach with a more personalized number of trials in medical simulators, can be a useful means of increasing performance of medical trainees.

Research Objectives

The goal of the current study is to investigate and evaluate the integration of progressive and personalized learning and its impact on the performance of residents on a CVC medical simulator – the DHRT. Specifically, the study was designed to answer the following research questions (RQ):

RQ1: Do residents perform the patient profiles on the DHRT simulator in a progressive order of difficulty? One way to validate whether progressive learning is integrated on the DHRT is to compare the performance scores of residents on the different patient profiles with varying difficulty levels. The data from the 2021 cohort will be used and includes participants who completed all the required patient profiles with the different levels of difficulty. We hypothesize that the performance scores of residents’ surgeons on the DHRT simulator will differ based on the difficulty of each case. More specifically, as the task complexity of the patient case increases, their performance scores should, especially when controlling for previous trials. Performance of residents on the difficult case should be the lowest, if cases are performed in a progressive manner. This is because based on the challenge point framework, as the level of task difficulty increases, performance is adversely impacted in the shorter term, but improves over time (Guadagnoli,2012).

RQ2: Does personalized learning impact final verification of proficiency (VOP) performance on the DHRT system? One way of determining whether the implementation of a personalized learning experience can be effective on the DHRT simulator, is to compare the VOP performance scores of residents in 2021 after completing a set number of trials, with residents in the 2022 cohort who had more flexibility in the number of trials they were required to complete before attempting the VOP. In 2021, residents were required to complete 5 mandatory insertion trials of varying difficulty during their CVC training before attempting the final VOP trial. In 2022, residents had to complete only two mandatory insertion trials before they were able to attempt the VOP trial. This was to provide them with a more flexible and personalized learning experience. This question was developed to understand and draw insight on whether decreasing the mandatory number of trials required by residents from 2021 to 2022 affected their training performance on the verification of proficiency (VOP) trial on the DHRT system. We hypothesize that VOP performance scores of residents after completing five mandatory trials in 2021 will not differ from the VOP performance scores of residents in 2022 with the reduced number of trial requirements. This is because prior research indicated that medical trainees have the ability to effectively gauge one’s own progress in acquiring a new surgical skill and obtain inherent awareness of when their performance has ceased to advance (Jowett et al., 2007).

Methods

There were two datasets used for the current study that were collected at Hershey Medical Center (HMC) over the years 2021 and 2022.

Participants

All participants in the study were residents at Hershey Medical Center (HMC). In total, there was 86 participants: 51 male and 22 female residents in the 2021 cohorts in post graduate year (PGY) level 1. Of these, 13 general surgery, 23 internal medicine, 16 anesthesia, 34 in other fields. 73 residents have not placed any internal jugular (IJ) insertions, 7 placed one, 1 placed two, 5 placed more than 2. 43 had no previous central line training, 17 in subclavian/femoral, 20 other. In the 2022 cohort, there were 78 participants: 44 males and 34 females in PGY level 1, with a few in PGY level 3 and 4. Of these, 16 general surgery, 23 internal medicine, 17 anesthesia, 13 emergency medicine, 3 plastic surgery, 6 in other fields. 55 residents have not placed any internal jugular (IJ) insertions, 11 placed one, 3 placed two, 9 placed more than 2. Finally, 36 had no previous central line training, 25 in subclavian/femoral, 17 other training.

Procedure

Before the study was conducted, all residents were enrolled in an online procedural training where informed consent was achieved following IRB guidelines. Residents were then required to complete the online procedural training which was around an hour long. The online procedural training included a pre-test to assess residents CVC baseline knowledge and a self-efficacy survey to assess residents’ level of confidence regarding CVC placement. Residents then went through eight videos about CVC information, complications, and management. Residents then had to complete a post-test to assess CVC knowledge after completing the online procedural training. Once they completed the online training, the residents were ready for the in-person training. First, each resident completed another self-efficacy survey to assess and determine their perception of their level of confidence in CVC placement. Residents then were required to perform CVC needle insertions on the DHRT.

For the 2021 cohort, residents were required to complete five mandatory needle insertions before attempting the final VOP trial, each associated with a patient profile. After completion of the six patient profiles in the DHRT, residents took a post self-efficacy test to assess their confidence level after completing the needle insertions in the DHRT. For the 2022 cohort, before residents completed their first needle insertion in the DHRT, they went through a walkthrough tutorial on how to use the DHRT. Moreover, for the 2022 cohort, residents were required to complete only two mandatory patient cases on the DHRT. If residents were successful in these two patient cases, then residents were able to attempt the final proficiency (VOP) trial. Meanwhile, if residents were not successful on their first two patient cases on the DHRT, then they were required to complete a third patient case. If they were successful on their third patient case, then residents could attempt the VOP. Moreover, if residents were not successful on their first three patient cases on the DHRT, then they were required to complete another patient case. If they were successful on their fourth patient case, then residents could attempt the VOP. This procedure continues the same until reaching mastery of the procedure which is successfully passing the VOP. However, if residents did not successfully pass the VOP, then they had to reattempt the VOP until they successfully passed it or had reached six patient cases.

Data Analysis

In order to answer our research questions, quantitative analysis was performed on the data that was collected. In order to answer our research questions, several quantitative metrics were computed. This section serves to highlight the metric computations.

DHRT Performance Metrics

One feature of the DHRT system is the Graphical User Interface (GUI) that informs residents their performance during their insertions on the different patient cases on the DHRT. Residents are evaluated in this GUI through the following metrics: angle of insertion, number of insertions, percentage of aspiration, distance to center of vein, and backwall. This study uses resident’s performance scores on each of these metrics individually, which were collected by the DHRT. Aspiration relates to the percentage of time the resident was aspirating the syringe. This is important because if air seeps into the catheter during insertion, this can lead to fatality which means that the catheter ports should always be closed (McGee and Gould,2003). Number of insertions relates to how many times the needle was inserted and Mansfield et al, (1994) has proven that more than two insertion attempts increases the rate of complications by 43.2%. Angle of insertion relates to the angle in which the resident inserts the needle and a range 30°– 45° was considered a success since this is the required angle of insertion for the procedure (Wheeler, 2009). Distance to center relates to how close the needle was inserted to the center of the vein. Placing the needle outside of the internal jugular vein during CVC can cause the drawing of blood into the syringe therefore, the optimal location of the needle is in the center of the vein (Blaivas et al, 2009). Backwall relates to whether they punctured the backwall of the vein or not and a success was they did not puncture the backwall. This is because puncturing the back wall can increase the risk of hematoma and hemorrhage (Bannon et al, 2011).

Results and Discussion

This section aims to illustrate the results obtained for each of our research questions. All statistical analysis was performed on SPSS (v.26.0) using a significance level of 0.05.

RQ1: Do residents perform the patient profiles on the DHRT simulator in a progressive order of difficulty?

Our first research question was developed to investigate whether the patient profiles on the DHRT are completed by participants in an increasing level of task difficulty. To answer this question, an ANCOVA was run to analyze whether the performance scores on the continuous metrics (aspiration, number of insertions, distance to center of vein) differ based on whether the patient cases was classified as easy, medium or hard, while controlling for previous trials.

For each of the metrics, there was a linear relationship between the number of trials and each performance metric for each level of difficulty, as assessed by visual inspection of scatterplots. Standardized residuals for the trial scores (interventions) were approximately normally distributed, as assessed by a normal Q-Q plot. There was homogeneity of regression slopes as the interaction term was not statistically significant, F(1, 413) = 1.577, p = .210. The assumption of homogeneity of variances was violated, as assessed by Levene's test for equality of variances (p <0.001). There were no outliers in the data, as assessed by no cases with standardized residuals greater than ±3 standard deviations. Adjusted means are presented, unless otherwise stated. The mean aspiration scores for the easy case, medium and hard cases were (M = 0.6406, SE =0.342)(M =0.6749, SE =0.328)(M =0.785, SE =0.2768) respectively. After adjustment for the number of trials, there wasn't a statistically significant difference in the aspiration score between the three different levels of difficulty, F (2, 414) = 0.523, p =0.593, partial η² = .003.The mean distance to center scores for the easy case, medium and hard cases were (M = 1.8662, SE =3.57 (M = 1.09, SE =2.41) (M = 1.1847, SE =2.32) respectively. After adjustment for the number of trials, there was not a statistically significant difference in the distance to center scores between the three different levels of difficulty, F (2, 414) = 0.523, p =0.254, partial η² = .007.The mean number of insertions for the easy case, medium and hard cases were (M = 5.13, SE =7.67 (M =2.29, SE =3.49) (M =2.97, SE =4.08) respectively. We can observe that the mean number of insertions was actually more for the easy case, and less for the harder cases. After adjustment for the number of trials, there was a statistically significant difference in the number of insertions between the three different levels of difficulty, F (2, 414) = 6.930, p =0.001, partial η² = 0.032.

The results indicate that the DHRT is not currently utilizing progressive learning by presenting the cases to participants in an increasing level of difficulty. Overall, there was no statistically significant difference between the scores on different patient cases. This may be due to the fact that the patient cases difficulty was rated by three experts from two different medical fields and only one institution which may have led to some subjectivity in the ratings (Gonzalez-Vargas,2020). The patient profiles on the DHRT need to include tasks with more effective levels of task difficulty because research has shown that progressive learning optimizes skill transfer to real-life patient scenarios (Kneebone R, 2010). One way is by using a larger number of experts from different medical institutions both in the USA and internationally, where all possible factors influencing task difficulty can be determined and used to effectively integrate progressive learning. This is because research has shown that this can result in beneficial and useful information that can be used to guide medical education (Hildebrand, 2011; Zeven, 2012).

RQ2: Does personalized learning impact the last proficiency trial (VOP) performance on the DHRT system?

The goal of our second research question was to understand whether personalized learning – or flexibility in terms of the number of trials residents were required to complete on the DHRT training system in the 2022 cohort impacted VOP scores compared to the residents in the 2021 cohort with a set number of trials. The VOP scores for each of the performance metrics on the DHRT (angle, aspiration, number of insertion attempts, distance to center and backwall) after residents completed the five-mandatory trials from the 2021 cohort data was compared to the VOP scores of residents after they completed either 2,3,4 or 5 trials in 2022, depending on whether they failed each of the VOP attempts.

A Wilcoxon signed rank test was used for the number of insertions, aspiration, distance to center performance metrics. The differences in scores between the VOP after the 5^th trial in 2021 and VOP scores after the 2^nd, 3^rd, 4^th or 5^th trial in 2022 were approximately symmetrically distributed, as assessed by a histogram with superimposed normal curve. All tests showed no statistical significance as shown in table 1.

Table 1.

VOP trial after 5 mandatory trials in 2021, compared to a flexible 2,3,4,5, trials in 2022.

	# of trials in 2022	Z-value	P-Value	Statistical Significance?
Number of insertions	2	1.744	0.081	No
	3	1.254	0.210	No
	4	1.744	0.081	No
	5	–0.154	0.878	No
Aspiration	2	1.428	0.153	No
	3	1.037	0.30	No
	4	0.991	0.322	No
	5	–0.452	0.651	No
Distance to center	2	–1.007	0.314	No
	3	–1.567	0.117	No
	4	0.991	0.322	No
	5	–0.888	0.375	No

For the angle and backwall metrics a two-sided paired sample proportion test was conducted to compare the number of residents who had a successful angle during the insertion and assumptions were verified. There was no statistically significant difference between the number of successes for the angle metric the VOP trial after completing 5 trials in 2021 compared to after completing 2 trials (z=-0.447, p=0.655, 95% CI: [-0.254,0.163], difference in proportions= 0.048),3 trials(z=-0.229, p=0.819, 95% CI: [-0.2,0.160], difference in proportions= 0.021), 4 trials (z=0.125, p=0.815, 95% CI: -0.272,0.215], difference in proportions= 0.03) or 5 trials (z=0.218, p=0.827, 95% CI: [-0.187,0.231], difference in proportions= 0.023) in 2022.

There was no statistically significant difference between the number of successes for the backwall score on the VOP trial after completing 5 trials in 2021 compared to after completing 2 trials (z=-2.449, p=0.014, 95% CI: [-0.256, -0.17], difference in proportions= 0.143),3 trials(z=0, p=1, 95% CI: [-0.136,0.136], difference in proportions= 0), 4 trials (z=0, p=1, 95% CI: -0.192,0.192], difference in proportions= 0) or 5 trials (z=-0.378, p=0.705, 95% CI: [-0.144,], difference in proportions= 0.023) in 2022.

These results indicate that when participants were able to ‘test out’ of the additional trials in 2022, they performed just as well as participants who were required to complete a set number of trials. These results support our hypothesis that personalized learning can be appropriately integrated into medical simulation-based training.

Conclusion

The goal of this study was to investigate whether progressive learning approach is being integrated on the DHRT simulator. Furthermore, the effect of a personalized learning approach on performance was also studied and evaluated through the use of quantitative analysis. The main results of this study are as follows: 1) the DHRT is not currently making use of progressive learning strategies, 2) individuals who used the personalized learning system were just as effective on the VOP even with fewer trials. The findings from this study imply that there is a need to integrate progressive learning on the DHRT simulator to ensure that residents who undergo CVC training are fully prepared for any patient scenario in a real-life setting. This should be implemented by asking a larger number of experts across different medical fields, institutions and international experts, in order to determine all possible factors that can influence task difficulty in medical procedures thus, ensure effective training for medical procedures. Furthermore, implementing a personalized learning approach with regards to the number of trials can achieve the same performance with less number of trials and therefore, this can provide more time for residents to focus on other learning needs, especially with the 80 work hour restrictions in residency programs. Although further investigation and validation is required, this knowledge can be considered for simulation-based training curriculums to optimize learning outcomes and skill acquisition. Future work will further study and validate the effect of tasks with a more representative level of difficulty on skill acquisition using a large number of experts from different fields, hospital and countries and whether personalizing the number of trials can lead to improvements in performance over an extended period of time.

While this study provides valuable knowledge that can be used to improve simulation-based training, there are a few limitations that need to be studied in future work. Firstly, the analysis for the personalized learning approach did not take into account how reducing the number of trials can impact performance over an extended period of time. Furthermore, the residents in the 2021 cohort did not have a walkthrough tutorial compare to the 2022 cohort. Future work should focus on controlling any factors between the two cohorts that may influence performance.

Footnotes

Acknowledgements

This work was supported by National Heart, Lung, and Blood Institute of the National Institutes of Health (NIH) under Award No. ROHL127316. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Coauthors Dr. Moore and Miller owns equity in Medulate, which may have a future interest in this project. Company ownership has been reviewed by the University's Individual Conflict of Interest Committee and is currently being managed by the University.

References

Vozenilek J, Huff JS, Reznek M, Gordon JA. See one, do one, teach one: advanced technology in medical education. Academic Emergency Medicine 2004;11:1149-54.

Dawson, D. L., Meyer, J., Lee, E. S., & Pevec, W. C. (2007). Training with simulation improves residents’ endovascular procedure skills. Journal of vascular surgery, 45(1), 149-154.

Evans LV, Dodge KL, Shah TD, et al. Simulation training in central venous catheter insertion: improved performance in clinical practice. Academic Medicine 2010;85:1462-9

Brydges, Ryan PhD; Carnahan, Heather PhD; Rose, Don PhD; Rose, Louise PhD; Dubrowski, Adam PhD. Coordinating Progressive Levels of Simulation Fidelity to Maximize Educational Benefit. Academic Medicine: May 2010 - Volume 85 - Issue 5 - p 806-812 doi: 10.1097/ACM.0b013e3181d7aabd

Hildebrand, E. A., Branaghan, R. J., Neuman, B. L., Jolly, J., Garland, T. B., Taggart, M., … & Brown, V. (2011, September). An expert perspective of errors in endoscope reprocessing. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 55, No. 1, pp. 748-752). Sage CA: Los Angeles, CA: SAGE Publications.

Kneebone R, Nestel D, Yadollahi F, et al. Assessing procedural skills in context: Exploring the feasibility of an integrated procedural performance instrument (IPPI). Med Educ. 2006;40:1105–1114

Thórisson, K. R., Bieger, J., Li, X., & Wang, P. (2019). Cumulative learning. In Artificial General Intelligence: 12th International Conference, AGI 2019, Shenzhen, China, August 6–9, 2019, Proceedings 12 (pp. 198-208). Springer International Publishing.

Barry Issenberg, S., Mcgaghie, W. C., Petrusa, E. R., Lee Gordon, D., & Scalese, R. J. (2005). Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review. Medical Teacher, 27(1), 10-28. https://doi.org/10.1080/01421590500046924

Zevin, B., Levy, J. S., Satava, R. M., & Grantcharov, T. P. (2012). A consensus-based framework for design, validation, and implementation of simulation-based training curricula in surgery. Journal of the American College of Surgeons, 215(4), 580-586.

10.

Brydges, R., Carnahan, H., Rose, D., Rose, L., & Dubrowski, A. (2010). Coordinating progressive levels of simulation fidelity to maximize educational benefit. Academic Medicine, 85(5), 806-812.

11.

Chatti, M. A., Jarke, M., & Specht, M. (2010). The 3P learning model. Journal of Educational Technology & Society, 13(4), 74–85.

12.

Gómez, S., Zervas, P., Sampson, D. G., & Fabregat, R. (2014). Context-aware adaptive and personalized mobile learning delivery supported by UoLmP. Journal of King Saud University - Computer and Information Sciences, 26(1), 47–61. .10.1016/j.jksuci.2013.10.008

13.

Brydges, Ryan, Heather Carnahan, Oleg Safir, and Adam Dubrowski. "How effective is self-guided learning of clinical technical skills? It’s all about process." Medical education 43, no. 6 (2009): 507-515.

14.

Krug, M. F., Golob, A. L., Wander, P. L., & Wipf, J. E. (2017). Changes in Resident Well-Being at One Institution Across a Decade of Progressive Work Hours Limitations. Academic medicine : journal of the Association of American Medical Colleges, 92(10), 1480–1484. https://doi.org/10.1097/ACM.0000000000001675

15.

Yovanoff, M., Pepley, D., Mirkin, K., Moore, J., Han, D., & Miller, S. (2016, September). Improving medical education: Simulating changes in patient anatomy using dynamic haptic feedback. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 60, No. 1, pp. 603-607). Sage CA: Los Angeles, CA: SAGE Publications.

16.

Guadagnoli M, Morin MP, Dubrowski A. The application of the challenge point framework in medical education. Med Educ. 2012;46:447–453

17.

Jowett N, LeBlanc V, Xeroulis G, MacRae H, Dubrowski A. Surgical skill acquisition with self-directed practice using computer-based video training. Am J Surg. 2007;193:237–242.

18.

Gonzalez-Vargas, J. M., Tzamaras, H. M., Martinez, J., Brown, D. C., Moore, J. Z., Han, D. C., … & Miller, S. R. (2022). Going the (social) distance: Comparing the effectiveness of online versus in-person Internal Jugular Central Venous Catheterization procedural training. The American Journal of Surgery, 224(3), 903-907.

19.

Gonzalez-Vargas, J. M., Brown, D. C., Moore, J. Z., Han, D. C., Sinz, E. H., Sonntag, C. C., & Miller, S. R. (2020, December). Objective assessment metrics for central line simulators: an exploration of causal factors. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 64, No. 1, pp. 2008-2012). Sage CA: Los Angeles, CA: SAGE Publications.

20.

McGee, D. C., & Gould, M. K. (2003). Preventing complications of central venous catheterization. New England journal of medicine, 348(12), 1123-1133.

21.

Mansfield, P. F., Hohn, D. C., Fornage, B. D., Gregurich, M. A., & Ota, D. M. (1994). Complications and failures of subclavian-vein catheterization. New England journal of medicine, 331(26), 1735-1738.

22.

Wheeler, D. S., Wong, H. R., & Shanley, T. P. (2009). Resuscitation and stabilization of the critically ill child (pp. 1-30). London: Springer.

23.

Blaivas, Michael MD, RDMS; Adhikari, Srikar MD, RDMS. An unseen danger: Frequency of posterior vessel wall penetration by needles during attempts to place internal jugular vein central catheters using ultrasound guidance*. Critical Care Medicine 37(8):p 2345-2349, August 2009. | DOI: 10.1097/CCM.0b013e3181a067d4

24.

Bannon, M. P., Heller, S. F., & Rivera, M. (2011). Anatomic considerations for central venous cannulation. Risk management and healthcare policy, 27-39.

25.

J. MacDonald, R.G. Williams, D.A. Rogers Self-assessment in simulation-based surgical skills training Am J Surg, 185 (2003), pp. 319