Abstract
Simulation exercises, whether discussion- or operations-based, are an essential component of public health emergency preparedness and response—so much so that the ability to conduct exercises is included as a key indicator for measurement of emergency response operations in the Joint External Evaluation and as a requirement of the International Health Regulations. Exercises allow ministry of health partners to actually practice and demonstrate their learned capacities and show progress toward those global health security goals. In recent years, the general technique in the global health security arena has been for external partners to design, facilitate, and evaluate public health emergency exercises for ministry of health partners. However, in 2017, the Centers for Disease Control and Prevention (CDC) Global Response Preparedness Team (GRPT) piloted regional training for ministry of health representatives from 4 countries in Latin America to build their internal capacity to design, facilitate, and evaluate simulation exercises using the US Homeland Security Exercise Evaluation Program (HSEEP). Using this program as the source material, the team modified the content for application to an international audience, facilitated it as a 4-day workshop, and designed an evaluation protocol to measure participants' satisfaction with the course as well as their ability to apply concepts learned in their day-to-day jobs. Representatives of 3 of the 4 ministries of health that attended the workshop designed exercises to implement in their home countries, with 2 having completed the facilitation and evaluation of their exercises at the time of this publication.
I
Simulation exercises have been used for decades around the world as a means to practice and demonstrate learned capacities. These exercises, whether discussion- or operations-based, are an essential component of public health emergency preparedness and response—so much so that the ability to conduct exercises is included as a key indicator for measurement of emergency response operations in the Joint External Evaluation (JEE) and a requirement for compliance with the International Health Regulations (IHR). 2 These exercises, with the evaluation component, not only complete the preparedness cycle and inform emergency planning updates, but they can also be used as a tool for measuring progress in capacity development. 3 They offer an opportunity to practice before the real emergency strikes and to identify potential weaknesses that can be corrected in order to improve a response. 4 In the field of public health, external partners such as CDC, the World Health Organization (WHO), the US Department of Defense, and private contractors, among others, have largely been responsible for designing, conducting, and evaluating public health emergency management exercises.
In order to reach full IHR compliance, ministries of health cannot rely solely on external partners to conduct their exercises. The ability to design, conduct, and evaluate exercises must be cultivated in the ministry of health public health emergency management programs. While these types of exercises are not a new concept, they are not developed or conducted in a standardized manner across the globe. 5 Therefore, their effectiveness in comparing performance across countries or regions is limited. “The Homeland Security Exercise and Evaluation Program (HSEEP) provides a set of guiding principles for exercise programs, as well as a common approach to exercise program management, design and development, conduct, evaluation and improvement planning.” 6 Developed as part of the 2007 National Strategy for Homeland Security, HSEEP seeks to address the key issue of standardization. While the program was designed for audiences in the United States, the concepts can be applied internationally. With modifications to various terms and activities, an internationalized version of HSEEP can be an effective tool for ministries of health to be able to develop, conduct, and evaluate their own simulation exercises in a standardized and effective manner.
Methods
Internationalization of US-Based Materials
CDC's Global Response Preparedness Team (GRPT), housed in the Emergency Response and Recovery Branch (ERRB), focuses on building the public health emergency preparedness capacity of partner ministries of health around the world. Their focus on internationalizing standard emergency management practices used in the United States allows for the flexibility of partner ministries of health to implement and adapt those practices to work best for their context. GRPT consists of a Master Exercise Practitioner (MEP) certified by the US Department of Homeland Security (DHS) and additional subject matter experts in exercise execution and evaluation—all with extensive international experience. 7 This combination of specialties uniquely positions GRPT to internationalize public health emergency management curricula, specifically HSEEP, making them more relevant to an international public health audience while maintaining course rigor.
GRPT's internationalization process of the HSEEP materials began with a review of the course objectives to ensure their applicability to a non-US audience. From there, a detailed review of all slides was conducted, and GRPT removed any information that was only applicable to US audiences and unnecessary to meet the course objectives. For example, specific regulatory references to Homeland Security Presidential Directives (HSPDs) and the National Incident Management System (NIMS), while helpful for US audiences to understand the legal framework behind exercise requirements, is not applicable to international public health institutes, as they are not US federal agencies and not subject to US policy.
The overall course structure remained the same, but additional opportunities to apply the concepts through small group activities were added and the overall course length was increased to 4 full days, compared to the original 2-day course length. 8 Photos were changed throughout the course materials to focus more on international public health scenarios, rather than stock FEMA emergency management photos. While general changes to the language and visuals were minimal, extensive changes were made to the application of concepts through small group activities. In the original HSEEP materials, small groups work to develop an exercise for a simulated jurisdiction in “Americana.” 6 In order to increase applicability, the references to Americana were removed and replaced with general instructions to use the current context from the relevant country. The intention of the internationalized version of the course is not to develop a general exercise for a simulated jurisdiction, but rather to have participants arrive with copies of their national plans, protocols, and procedures and have the opportunity to develop a country-specific exercise that is later executed for additional practical experience. Upon the completion of the comprehensive material review and editing process, bilingual members of GRPT translated all materials to Spanish for use in Latin America.
Course Evaluation Planning
In an effort of continuous improvement, GRPT designed an evaluation methodology for the international HSEEP course based on the Kirkpatrick model of training evaluation. 9 GRPT targeted 3 of the 4 levels of evaluation through a combination of immediate course evaluations, pre- and post-course exams, targeted activities, a delayed participant survey, and direct observation of exercise implementation.
Level 1: Reaction
Immediate participant reactions and satisfaction with the course are captured through an 18-item paper-based course evaluation form distributed to all participants immediately upon conclusion of the instruction session. Learner-focused quantitative measures, on a 1 (strongly disagree) to 5 (strongly agree) Likert scale, evaluate the participants' satisfaction with the course content, the relevance of activities, the effectiveness of facilitators, and their comfort with the training space. Qualitative open-ended questions allow participants to identify elements that they would want to add or remove from the course, as well as offer them space for any additional comments.
Level 2: Learning
The degree to which participants gained knowledge from the workshop is measured through the same 15-item pre- and post-course exam to allow for a quantitative look at knowledge increase, as well as to identify areas for improvement in course materials or facilitator instruction. Additionally, to incorporate a “New World” approach to level 2 evaluation, or “the degree to which participants acquire the intended knowledge … based on their participation in the training,” targeted discussion questions were added throughout the course materials to gauge participant understanding, and the team used a Jeopardy-style review game at the conclusion of the instruction period. 9
Level 3: Behavior
The most important element of sustainable global health security capacity development is the evaluation of behavior or “the degree to which participants apply what they learned during training when they are back on the job.” 9 For this workshop in particular, GRPT uses a combination of a delayed survey of participants and direct observation of skills application. The delayed survey was sent electronically to all participants 6 months after the completion of the workshop. It is limited to 5 questions in order to encourage participation, but it directly asks participants how they have applied the skills they have learned, to provide examples, and, if they have not applied the skills, why they feel they could not do so. Direct observation of participants' facilitating and evaluation simulation exercises allows GRPT to gauge critical behaviors and confirm whether participants gained the necessary knowledge and skills to successfully execute a simulation exercise.
Conducting a Regional Pilot Workshop
GRPT received funding through an interagency agreement between CDC and the US Agency for International Development (USAID) to build public health emergency management capacity in 5 priority countries in Latin America in response to the Zika outbreak: Barbados, Colombia, Dominican Republic, Guatemala, and Panama. 10 Because they share a common language and capability level, Colombia, Dominican Republic, Guatemala, and Panama participated in a series of regional training workshops, including GRPT's internationalized HSEEP. Because of international policies, GRPT could not request workshop participants by name. Instead, letters of invitation outlining the necessary qualifications and specific job duties for participants were sent to ministries of health to ensure that participants chosen to attend would be those responsible for executing a national-level exercise after the workshop.
The workshop took place over the course of 4 days in June 2017 in Panama City, Panama. A total of 16 participants from across the 4 invited ministries of health attended. Not all GRPT instructors speak Spanish; therefore, most of the course instruction was in English with simultaneous translation into Spanish. Bilingual GRPT staff conducted periodic spot checks of the translations services for technical accuracy and clarity. The instruction was conducted by GRPT staff facilitating selected units based on their expertise. Small groups were based on participants' countries of representation, and bilingual staff rotated as facilitators among the small groups to provide additional guidance and answer questions.
Results
Course Evaluation
Overall, the course was very well received, with a mean score of 92% (n = 16) in the immediate reaction level course evaluation survey. Comments were received from the participants of all 4 countries that were in attendance. Items regarding the importance of the concepts for participants' jobs and effectiveness of the facilitators received highest marks, while the lowest scoring item indicated that participants experienced various distractions throughout the course. Qualitative data suggest that participants wanted additional course time, as well as the ability to participate in a sample tabletop exercise in order to practice their skills. Multiple comments suggested adding a fifth day to the course for the tabletop exercise and additional review.
Participant knowledge scores increased by 10% on completion of the 4-day course. A paired t-test was conducted to compare the overall performance on the course exam before the course was conducted (pretest) and upon course completion (posttest). There was a significant increase in posttest scores (p = 0.0072), suggesting an effective transfer of knowledge to participants (pretest mean = 58.8%; posttest mean = 68.8%).
Perhaps most encouraging are the delayed survey results. The delayed survey had a 50% response rate (n = 8) with largely positive results. Six months after the course, 87.5% of respondents still felt that the information presented during the course was applicable to their jobs; 75% felt that taking the course was a good use of their time; and 87.5% of respondents have applied the concepts learned in their day-to-day jobs. When asked how they applied the concepts, answers included conducting a workshop on exercises with 11 health districts, developing and conducting tabletop and full-scale exercises to validate protocols and procedures, and using management principles for projects.
Exercise Facilitation
The Department for Health Contingency Situations (DSSC in Spanish) of the Panamanian Ministry of Health (MINSA in Spanish) developed, facilitated, and evaluated their first public health emergency management tabletop exercise on August 22, 2017. DSSC staff managed all aspects of the tabletop exercise, while GRPT attended as observers and additional evaluators. The tabletop exercise was a successful test of their newly developed public health emergency operations center activation procedure in the event of an infectious disease outbreak. It included a plausible and detailed scenario, injects through audiovisual cues, and participation from across MINSA, including the minister of health. DSSC also led the evaluation of the exercise, with key areas for improvement noted and highlighted in an after-action report. DSSC has expressed interest in additional, more complex exercises in the near future.
The National Institute of Health (INS in Spanish) of Colombia developed, facilitated, and evaluated a public health emergency management full-scale exercise July 26-28, 2017. The exercise consisted of scenarios at 4 different sites:
The arrival and notification of suspected cholera cases at the Cartagena port; The activation of the public health emergency operations center in Bogota; The deployment of a rapid response team from Bogota to Cartagena and conducting a field investigation; and The collection, shipment, and testing of samples from the field to the national laboratory in Bogota.
GRPT managed the scenario and evaluation development for the activation of the public health emergency operations center, while HSEEP workshop participants designed, controlled, and evaluated the scenarios at the 3 other sites. The full-scale exercise was a major success in demonstrating INS's ability to detect and respond to a potential public health emergency, both at the national and local levels. INS is currently developing an annual exercise program to be managed by the Rapid Response and Risk Communication Group in the Epidemiologic Surveillance Department.
The Risk Management Unit of the Ministry of Health of Guatemala developed a tabletop exercise to test emergency management capacity, but due to leadership changes, the facilitation of the exercise has been put on hold.
Discussion
Challenges
While the course was well received and participants have clearly demonstrated the ability to use their new skills, building capacity, especially when done in another language, will always present challenges. The field of emergency management has an extensive lexicon of terms with specific definitions that have been agreed on and are used consistently. Despite current efforts to modify this lexicon for public health emergency management, the focus is to develop this lexicon in English. Eventually, it will be translated into other languages, but translation into Spanish, where country-specific dialects may result in different meanings for the same word, will present a challenge. However, this standardized lexicon is much needed in Spanish to avoid confusion and allow for easier translation of public health emergency management concepts for Latin American audiences.
Capacity development at the national level will always be subject to various political issues. Most invited participants from the Dominican Republic were unable to attend the workshop because of the late arrival of invitation letters and their inability to secure visas in time to travel. Workshop planners should be aware of such restrictions during the planning phase and ensure that invitation letters are sent to participants well in advance. Additionally, continued project implementation is dependent on senior leaderships' buy-in and support. Transitions or changes in leadership with little to no warning are common in various countries within the region and can often result in implementation delays. While there is little that can be done to avoid such delays, implementation teams should maintain flexibility and be able to communicate to funders why projects may need to be rescheduled.
Regional workshops are far more cost-effective in terms of human and financial capital than individual national workshops. The ability to bring multiple countries to 1 location reduces the need for travel, venue rental, and translation services in each individual country, not to mention the sheer number of person-hours needed to conduct a 4-day workshop in 4 countries as opposed to 1. However, the tradeoff is a reduced number of participants from each country. In this case, ministries of health could send up to 5 individuals to the course, based on available funding. For some topics, this format may not be the best option, but for the HSEEP course, participants were carefully selected based on their skills and expertise. This limit on the number of participants did not negatively affect the quality of the exercises that were designed and implemented as a result of the course.
Next Steps
GRPT will be replicating the international HSEEP course in Santo Domingo, specifically for ministry of health representatives in the Dominican Republic who were unable to attend the regional workshop, with the same goal that ministry of health participants will design a tabletop exercise during the course of the workshop to be facilitated and evaluated in the following months. GRPT will continue to work with the ministry of health of Guatemala to reschedule their planned tabletop exercise, and it will participate as observers when it is scheduled.
GRPT will integrate additional source material from the recently released WHO Simulation Exercise Manual. 11 While the concepts of exercise design, conducting, and evaluation are the same, many tools and templates have already been adapted for a public health audience. GRPT has provided copies of the WHO Simulation Manual to partners as an additional resource, but the information can be further integrated into the international HSEEP course. Based on participant feedback, a fifth day will be added to the course in which participants will take part in a sample tabletop exercise. This addition will be piloted at the Dominican Republic version of the international HSEEP course.
Impact on Global Health Security
Upon completion of their exercises, both the ministries of health of Panama and Colombia would have higher JEE scores for response evaluation item R.2.3, reaching Limited Capacity—2 and Developed Capacity—3, respectively, if the JEE process had been completed prior to the training. While an increase in 1 indicator may not seem significant, the ability of a ministry of health to practice and demonstrate learned skills is the goal of capacity development. Additionally, these exercises are highly publicized through both traditional and social media, and they demonstrate to the public that their ministries of health are actively improving and preparing for public health emergencies. The exercises designed by the ministries of health of Panama and Colombia focused on key elements of potential public health emergencies of international concern (PHEICs) and not only highlighted strengths but also areas for improvement. Each ministry of health developed detailed after-action reports and improvement plans, with clearly articulated action items to address gaps identified during the exercises.
Capacity development will continue to be the bedrock of global health security efforts. The evaluation of the capacity development efforts, to better understand the gaps as well as to determine how to effectively implement training approaches into routine practice, is an important focus of CDC's global health security efforts. By building the capacity of partners in ministries of health to design, facilitate, and evaluate their own simulation exercises, we reduce the reliance on international cooperation funds from partners to practice and demonstrate their learned capacities. The adoption and replication of internationalized HSEEP program concepts, including the evaluation of these concepts, will build sustainable progress toward our global health security goals.
Footnotes
Acknowledgments
This manuscript was made possible through support provided by the Office of Infectious Disease, Bureau for Global Health, US Agency for International Development, under the terms of an interagency agreement with CDC. The opinions expressed herein are those of the authors and do not necessarily reflect the views of the US Agency for International Development or the US Centers for Disease Control and Prevention.