Emergency medicine staff’s perception of SimWars: A Singapore view

Abstract

Background:

SimWars is an onstage competition among teams of health care providers in front of an audience. Participants demonstrate communication, teamwork, clinical management, and problem-solving skills. Upon completion, judges debrief participants’ clinical actions and team dynamics.

Purpose:

The Society for Emergency Medicine in Singapore hosted the inaugural SimWars in 2014. This study investigated whether SimWars was effective in teaching emergency medicine staff, explored how the competitive nature of SimWars affected the performance and learning of participants, and identified strategies to implement SimWars effectively in Singapore.

Methods:

A descriptive qualitative design was selected owing to the exploratory nature of the questions. Thirteen SimWars participants, six audiences, and three judges participated in the study.

Results:

(1) Twenty-one out of 22 considered SimWars useful for learning; (2) 14 out of 19 agreed that SimWars closely resemble clinical practice compared to clinical practice; (3) 16 out of 19 agreed that competition induced stress from participants and enhanced performance; (4) 16 out of 22 said SimWars was psychologically safe; (5) 18 said the team worked well together; (6) 19 said that debriefing helped them to improve through clarification, discussion, and reflection; debriefing should be personalized, longer, more structured, more detailed, and in a more private manner; (7) 13 said their knowledge and skills on the content areas will change; 21 said what they learned will be transferred to clinical practice; (8) the useful parts of SimWars included practicing under stress, debriefing, teamwork, and critical thinking; and (9) future improvements included avoiding technical problems and providing clear guidelines.

Conclusion:

Our responders perceived SimWars to be effective for professional development.

Introduction

SimWars is an onstage competition among teams of health care providers in front of a large audience. They demonstrate communication, teamwork, clinical management, and problem-solving skills through caring for a simulated patient. Upon completion of each simulated clinical encounter, judges assess and discuss the clinical actions and team dynamics of participants. ¹ Most often, the debriefing occurs on stage and in public and the audiences vote on a winner via an audience response system. The winners move on to the next round of the competition. Since its inception, SimWars have been staged at international meetings such as the International Meeting on Simulation in Healthcare (IMSH), the American Society for Academic Emergency Medicine (SAEM), the American College of Emergency Physicians (ACEP) as well as locally in the form of citywide competitions across different specialties including emergency medicine, pediatrics, and obstetrics and gynecology.

The Society for Emergency Medicine in Singapore (SEMS) hosted the inaugural SimWars in 2014. Teams managed clinical case scenarios in a high-fidelity simulation environment. The winning team received a $500 cash prize and the SEMS SimWars Challenge Trophy. The SEMS SimWars was a single-elimination challenge, including the preliminary round and the final round. The preliminary round was a broadcast event while the final competition was live.

Dong et al. ² investigated the perception of Emergency Medicine residents of SimWars in a large United States (US) academic medical center, and found that SimWars can be effective in residents’ training when debriefings are guided by well-structured rubrics. How do emergency medicine (EM) staff members perceive SimWars in Singapore? This study investigated whether SimWars is effective in teaching EM staff and identified strategies to implement SimWars effectively in Singapore. Second, how does the competitive nature of SimWars affect performance and learning? The results assess the value of SimWars as well as areas for improvement. Third, SimWars is new in Asia, and our findings will be informative to guide future SimWars efforts in Asia.

Methods

A descriptive qualitative design was selected owing to the exploratory nature of the research questions. On the other hand, it is a widely used qualitative research method to gather information about a particular situation. ³ The focus of this study was on the experience of participants who performed in SimWars, audiences who watched the competition, and judges who facilitated and debriefed the session. They are consultants, residents, and nurses from EM.

Questions

The questions included: effectiveness of SimWars, features helpful for learning, simulation fidelity, knowledge transfer, debriefing, competition, and teamwork. These questions were used in the study by Dong et al. ² See Appendix 1 for the questions. These key components will be explained below. In a SimWars session, participants collaborated and interacted with one another while providing care for simulated patients. The goal was for participants to demonstrate and gain clinical knowledge, teamwork and communication, and patient management skills. ¹ The constructive process involves not only participants’ prior knowledge and experience in clinical scenarios,^4,5 but also the psychosocial factors such as motivation, anxiety, and stress, which have been shown to affect performance.^6,7 The outcome is optimal when they are free from threat, the least defensive, and want to learn.^5,8 The SimWars experience theoretically combines collaboration with peers and competition that requires both psychological safety and simultaneously induces stress. ¹ With regards to collaboration in health care settings, the US Agency for Healthcare Research and Quality (AHRQ) and the Joint Commission (TJC) have identified four competencies for teamwork for clinical education: leadership, communication, situational monitoring, and mutual support. ⁹ These competencies are based on research on the crew resource management and human factors principles that improve patient safety and clinical performance. ¹⁰ Effective clinical teamwork and collaboration also require psychological safety and a culture of learning. ⁸

Three sets of questions were developed depending on the individual’s role in the SimWars session, i.e., Participant, Audience, or Judge. Questions for judges focused on how the team cared for the patients and clinical accuracy. Questions for participants and audiences were similar, with a focus on teamwork, attitudes during SimWars, levels of motivation, and responses to judges’ feedback.

Administration of the questions

The questions were designed for individual interview. Because of a delay in the institutional review board (IRB) approval, it was difficult for us to recruit interviewees, so we decided to upload the questions to SurveyMonkey, an online platform. Upon the IRB approval, those who attended the SEMS 2014 meeting were invited to complete the online questions with the link included in the email. The email was sent out by co-authors RG and SGG, who were organizers of the SimWars. A reminder email was sent two weeks after the initial invitation was sent, and the link was closed one week later.

SimWars scenarios and participants

Two SimWars scenarios were included, one for Fournier’s gangrene with sepsis and hyperkalemia with ventricular fibrillation (VF) collapse, and the second one for trauma in pregnancy with maternal shock and delivery of baby. The semifinal included eight teams; the two winning teams competed in the final. Each team included five emergency medicine residents and nurses coming from different hospitals in Singapore. The seven judges were emergency medicine consultants from different hospitals in Singapore and overseas (one from the US, one from Turkey). See Figure 1 for the SimWars overview. There were about 300 audience members, including emergency medicine consultants and residents. The data came from thirteen participants, six audience/observers and three judges, and none of them reported experience in SimWars. For the clinical scenarios, thirteen did not have prior knowledge, and nine had limited knowledge.

Figure 1.

Overview of the SimWars at SEMS.

Data analysis

The quantitative questions were analyzed using SPSS. The qualitative data obtained from the open-ended questions were coded and analyzed by authors CD and YK. The content analysis method was used to analyze the data. The method involved coding and developing categories. ¹¹ CD and YK read the data repeatedly for thorough understanding of how the participants reacted to the questions. Data coding was based on the meaning of the sentence, and then individually grouped into categories by CD and YK. CD and YK coded the data independently, and then met to compare the coding. When disagreement and overlapping of categories occurred, consensus was reached through re-examining the data and discussion. This was conducted to ensure the holistic capture of the perceptions of the research team. ¹¹ Atlas.ti, a qualitative data analysis software, was used by the first author, CD. YD analyzed the data manually. After the consensus was reached on data categories, CD used Atlas.ti to generate frequencies and quotations of codes, which were presented at the following session.

Results

The results were based on nine main categories emerging from the data analysis, i.e., usefulness for learning, realism, impact of competition, psychological safety, teamwork, debriefing, knowledge and skills transfer, useful part of SimWars, and future improvements. Table 1 summarizes these categories and subcategories as well as quotations from participants.

Table 1.

Main categories, subcategories and quotations from content analysis.

Category	Subcategory	Quotations
Usefulness of SimWars	Useful (n = 21)	“Very useful with high fidelity and with the competition we really prepared for it.”
		“I felt it helped me consolidate my learning.”
	Not really (n = 1)	“It was more like a show than an educational activity.”
Realism of SimWars	Real (n = 14)	“It was quite high fidelity, good representation of real life.”
		“Definitely tries to mimic the real life situation & fairly faithful too.”
	Moderate (n = 2)	“While it may be high fidelity, there are certain things that are just impossible to realistically portray.”
		“Sort of real in a sense but fast forwarded.”
	Not much (n = 3)	“Relatively low fidelity as there was clear segregation of p1 n p3 case and the p3 case was hardly addressed by any of the teams.”
		“Not so real. in real situation, we would have more manpower.”
Impact of competition	More stress (n = 11)	“More stress, especially in a large audience, but that’s reality as real-life scenarios are stressful too.”
		“Increased tension (but not realistic as on the shop floor what we fight against is time and tissue survival as well as anxious relatives rather than opponent teams).”
	Positive (n = 5)	“Motivated us to do better.”
		“Will be alright if taken in the spirit of learning.”
	No impact (n = 3)	“Initially, the fact that it was a competition made me more anxious, that I had to be against the others. However, once the simulation started, and I was in it, that all faded away, and I was just focusing on the case itself.”
Whether psychological safety was preserved	Safe (n = 16)	“Relatively safe but with the high stakes still, there was really a sense of trying to make sure things are right.”
		“Definitely safe as there are no real consequences perhaps only embarrassment.”
		“Better than making them in real life.”
	Not really (n = 6)	“As it is a competition, naturally we are afraid of making mistakes.”
Teamwork	Good (n = 18)	“They were all reasonably competent although some communicated better than most.”
		“Different team dynamics differ and the way they handle situation is different. Overall, great effort and teamwork.”
	Average (n = 1)	“Situational awareness could be improved on, failed to pick up signs intentionally.”
Debriefing	Importance	“Point out mistakes that the competitors made – which we would have likely made as juniors ourselves anyway.”
		“Learn where improvements can be made.”
		“It clarifies clinical issues of the scenarios as well as conduct of resuscitation.”
	Future improvement	“A more personalized debrief for every team would be useful.”
		“Subjective based on who the assessors were might be better if all 4 judges watched every team’s performance.”
		“To give individual feedback directly after each scenario.”
		“I think more specific team by team feedback would be good for our learning (which is what certain judges did).”
Learning outcomes	Knowledge and skills change (n = 13)	“Be prepared for the unexpected.”
		“It reinforces the knowledge and clinical competency.”
		“I learnt to be more targeted and observant.”
	Transfer to clinical practice (n = 21)	“Being more confident in running resuscitation as a team.”
		“To a large extent, especially the soft skills such as closed loop communication.”
		“How to better react to emergency situations.”
		“Everything that I have learnt can be put into good use.”
Useful part of SimWars	Practice under stress (n = 11)	“During the resuscitation we all are under stress. So practice makes things better.”
		“All of it – including making the diagnosis based on clinical scenario, changing management as the scenario evolves, and teamwork”
		“Being involved in the actual simulation/scenario.”
		“The competition itself. It is a great way to learn how to handle emergencies”
	Debriefing (n = 6)	“The session itself and the debrief as you learn about things that you didn’t know that we’re not entirely accurate.”
		“The debrief to find out what we could have done better.”
	Teamwork (n = 3)	“Teamwork.”
	Critical thinking (n = 2)	“I think the high fidelity simulation was very helpful as it is an opportunity that does not come by often. Watching other groups perform also enhanced my thinking and learning.”

Usefulness for learning

Responders commented on whether participating in the SimWars was useful for learning. Among 22 responders, 21 answered “Yes” to this question. However, one judge commented that “it was more like a show than an educational activity” because of limitations such as on-stage competition.

Realism

Realism refers to how real the SimWars was when compared to clinical practice. This question was targeted for audiences and participants. Fourteen out of 19 replied “real,” two “moderately real,” and three “not much.”

Impact of competition

Competition is a distinguishing feature of SimWars. What impacts did competition cause on participants? Among those who replied to this question, 11 (58%) said it induced a lot of stress. Five (26%) indicated a positive impact on performance, while three (16%) said no impact.

Psychological safety

Psychological safety refers to the feelings that team members have about performance in certain context. Psychological safety has been shown to be critical for the success of patient care.^7,12 Among 22 who responded to this question, 16 commented the session was “psychologically safe” and six “not really.”

Teamwork

Teamwork is one of the key areas that participants can practice through SimWars. Three skipped this question. Among 19 who responded to this question, 18 said the teams worked well together. One participant commented on lacking situational awareness.

Debriefing

Debriefing is a key feature of SimWars, similar to many other simulation interventions. Overall 19 responders commented on the impact of debriefing. All said that debriefing helped them to improve through clarification, discussion and reflection. Responders also commented on what can be done to improve, including: more personalized debriefing, using videos, longer debriefing, more structured, more details, more private manner, and all judges watching all the performances. In addition, one responder mentioned that debriefing in public prevented judges from providing negative feedback. The reason was explained as:

“I think the debriefing should be in a very private environment and just for the specific team. I was wondering the team’s stress during the debriefing in all auditorium … I do not think this is a good environment for them. And also, for judges too because we were protecting ourselves to say more realistic feedbacks to them, it may block us to say some negative parts of the management in front of everybody to the teams. Therefore, we talked about more positive things I guess and try to generalise our comments etc….”

Knowledge and skills transfer

For the participants’ change of knowledge and skills after the SimWars, 13 said there will be changes, with two “limited changes” and one “no.” For transfer to clinical practice, 21 said what they learned from SimWars will be transferred to clinical practice in various aspects, such as teamwork skills, managing an emergency situation, being a better leader, and being more confident.

Useful parts of SimWars

Among 19 who responded to the question, 11 stated that practicing under stress was most helpful, six for debriefing, three for teamwork, and two for the critical thinking part.

Future improvements

There were 21 responders providing suggestions for future improvements, mostly on “avoiding technical problems” as well as “providing clear rules and guidelines.” Table 2 includes quotations for these two aspects.

Table 2.

Improvement to conduct future SimWars sessions.

Improvement	Quotations
Avoid technical problems	“Cases must be coordinated. Check session with other person would be better to avoid hiccups during a real SimWar.”
Clear rules and guidelines	“Same numbers of physicians, equal elimination environment, patient should never die in front of audience.”
	“Equal elimination environment for all teams.”
	“Enough time frame to evaluate and discuss the team performances.”
	“Private feedback sessions for each team.”
	“More specific goals in evaluation forms and achieved, partially achieved, and not attempt sections for each evaluation goals.” “Finals can be in different room than auditorium, but these activities can be livestream to the auditorium.”
	“The Nearpod or similar online technologies can be used for audience real votes to evaluate the teams. Numbers talk…”

Discussion

Competition in SimWars

Competition is a distinctive feature of SimWars, as teams compete on the stage on the same clinical scenario. Among those who answered this question, 84% indicated that competition in SimWars induced stress from participants, and ultimately enhanced performance and learning. Competition motivates participants to try their best. A couple of them answered that competition did not affect performance because Emergency Medicine is a high-stress specialty, and Emergency Medicine doctors and nurses are well trained to work under stress. As a result, the stress due to competition should not affect their patient care. For the audience, competition definitely added spice to the scenario, and the audience felt more engaged.

The impact of competition is aligned well with stress participants experienced. During SimWars, on-stage competition induced stress. Being assessed by the judges as well as audiences in public also added stress. It is interesting that nobody in our study commented on the negative impact of stress on performance. Psychology research has demonstrated that stress induces an inverted bell-curve impact on performance. ¹³ In other words, too little stress does not have an impact while too much stress backfires on the performance. But it also depends on the individual’s ability to cope with stress. For high-stress professions, training under close-to-real stressful settings helps future practice. ¹⁴ Nevertheless, competition and stress induced from competition in SimWars were shown to be a welcoming factor for our audience, participants, and judges.

Effective strategies for debriefing in SimWars

Debriefing has shown to be key if the aim of SimWars is for knowledge and skill acquisition, which is aligned with previous research on medical simulation. ¹⁵ However, debriefing needs to be performed in a constructive manner, which requires clinical teachers to be trained in conducting debriefing. One judge did point out that debriefing in public in SimWars made it challenging to comment on the negative part of the performance. This was not an issue for the study the first author conducted in the US. It is typical in Asia that people are afraid of losing face in public, ¹⁶ which made it difficult for judges to give feedback in public. To make debriefing work in Asian culture, it is necessary to debrief the participants privately, followed by a public debriefing. Private debriefing should be tailored to SimWars participants, with an emphasis on gaps in performance and suggestions for future improvement. Public debriefing is targeted to everybody present at the SimWars session including observers, starting with a review of the case scenario, followed by overall management, and ending with future improvements.

Teamwork

Teamwork training has been recognized as an important educational goal by quality and regulatory agencies as well as academic societies, medical boards, and residency review committees. Our results indicated that the teams worked well together. Although it is not clear whether SimWars helped them to practice in a health care team, we can predict that their teamwork skills would not improve significantly because they already demonstrated a high level of teamwork skills. The AHRQ defines teamwork competencies through the TeamSTEPPS program, including leadership, mutual support, communication, and situational awareness. ⁹ Without TeamSTEPPS training, responders did not understand the nuisances of the four competencies, which explained why no responders addressed these competencies separately, and they just simply answered that the team worked well together.

Transfer of knowledge and skills

The outcome of medical simulation has been defined as four levels, such as in the simulation laboratory, patient care, patient outcomes, and collateral effects such as cost. ¹⁷ Although our responders reported translational outcome from SimWars, further research is required to collect objective data to demonstrate immediate knowledge gain and skill acquisition as well as transfer to clinical settings.

Conclusions

According to our responders, SimWars was effective for professional development due to the fact that SimWars happened during the annual conference. We speculate that SimWars probably would work better than traditional professional development programs such as workshops or courses in certain topics. However, further investigation is needed to confirm its value on this matter. Second, this was the first time that SimWars was conducted in Singapore. It is not clear whether the results were due to the novel nature of SimWars. It will be interesting to address these questions after SimWars becomes a norm in Singapore medical education. Third, larger sample size and more extensive data will facilitate our understanding of SimWars for formal teaching as well as for professional development.