Sage Journals: Discover world-class research

Abstract

Objectives:

Virtual tabletop exercises (VTTXs) simulate disaster scenarios to help participants improve their emergency-planning capacity. The objectives of our study were to (1) evaluate the effectiveness of a VTTX in improving preparedness capabilities specific to children’s needs among pediatricians and public health practitioners, (2) document follow-up actions, and (3) identify exercise strengths and weaknesses.

Methods:

In February 2017, we conducted and evaluated a VTTX facilitated via videoconferencing among 26 pediatricians and public health practitioners from 4 states. Using a mixed-methods design, we assessed participants’ knowledge and confidence to fulfill targeted federal preparedness capabilities immediately before and after the exercise. We also evaluated the degree to which participants made progress on actions through surveys 1 month (n = 14) and 6 months (n = 14) after the exercise.

Results:

Participants reported a greater ability to identify their state’s pediatric emergency preparedness strengths and weaknesses after the exercise (16 of 18) compared with before the exercise (10 of 18). We also observed increases in (1) knowledge of and confidence in performing most pediatric emergency preparedness capabilities and (2) most dimensions of interprofessional collaboration. From 1 month to 6 months after the exercise, participants (n = 14) self-reported making progress in increasing awareness for potential preparedness partners and in conducting similar pediatric exercises (from 4-7 for both).

Conclusions:

Participants viewed the VTTX positively and indicated increased pediatric emergency preparedness knowledge and confidence. Addressing barriers to improving local pediatric emergency preparedness—particularly long term—is an important target for future tabletop exercises.

Keywords

pediatric preparedness tabletop exercise emergency preparedness disasters interprofessional collaboration

Children aged ≤18 comprise 25% of the US population, yet incorporation of their needs into emergency planning at local, state, and federal levels remains incomplete.^1
-3 Nationally representative data indicate that nearly 1 in 7 US children has experienced a natural or man-made disaster during their lifetime.⁴ Recent crises, including the 2010 Haiti earthquake, the 2014 enterovirus D68 outbreak, and 2015 Zika virus outbreak, are reminders of children’s often adverse outcomes in emergencies (eg, disproportionate incidence of infectious disease), rooted in distinct physiological, social, and developmental characteristics that enhance vulnerability.

Partnerships between pediatric health care providers and public health entities are essential for including children in disaster preparedness, management, and recovery planning.^5

-9 Although emergency preparedness exercises have promoted local collaboration and enhanced preparedness capacity,^10
-12 most exercise reports have been prepared for a single state or selected groups of stakeholders. Few describe large-scale, multistate exercises that are relevant to children.^9,13 Emergencies frequently necessitate coordination across sectors and state boundaries^5,9,14,15; accordingly, exercises that span multiple jurisdictions and target pediatric considerations of children’s needs in emergencies (eg, family reunification after disasters) are warranted.

A promising approach to promoting awareness and planning in disasters is the use of virtual tabletop exercise (VTTX) models implemented by the Federal Emergency Management Agency (FEMA), which simulate emergency scenarios to help participants generate discussion and identify strategies and gaps in their preparedness, response, and recovery planning.¹⁶ In February 2017, the American Academy of Pediatrics (AAP) and Centers for Disease Control and Prevention (CDC) hosted an exercise that simulated a multistate disease outbreak among children. Because hosting an in-person multistate exercise can be costly and time-consuming,¹⁷ the organizations developed and piloted a VTTX. Adapted from an in-person tabletop exercise,¹⁸ the VTTX aimed to replicate a realistic multijurisdictional response in which state leaders would remain in their own states and communicate electronically. We evaluated the VTTX to understand its effectiveness, follow-up actions taken by state teams, and areas of strength and weakness to inform future exercises.

Methods

Virtual Tabletop Exercise

The CDC/AAP planning team selected participants from FEMA Region VII (Iowa, Kansas, Missouri, and Nebraska) to participate in the VTTX; the AAP and CDC had not previously conducted exercise activities in this region. Each state team included at least 2 pediatricians chosen by the AAP state chapter and 2 local public health officials recommended by CDC-funded Public Health Emergency Preparedness state health department contacts. Participants attended orientation telephone calls to review expectations, although we withheld the exercise content until the exercise itself.

We facilitated the 4-hour exercise from CDC headquarters, and states joined remotely via Zoom software version 3.5.¹⁹ The exercise consisted of 3 modules depicting an outbreak of smallpox (Box).²⁰ The exercise facilitator tasked teams with addressing 3 objectives that targeted communication and collaboration (Table 1). To meet these objectives, state teams deliberated on plans and procedures and each sector’s response role after they received new information in the scenario (ie, injects) and watched CDC expert videos. AAP Disaster Preparedness Advisory Council members (ie, pediatricians) and public health experts facilitated debriefings among all state teams after the completion of each module.

Box

Description of content of a Centers for Disease Control and Prevention/American Academy of Pediatrics virtual tabletop exercise to enhance pediatric emergency preparedness, Atlanta, Georgia, February 2017^a

On February 2, 2017, the American Academy of Pediatrics (AAP) and the Centers for Disease Control and Prevention (CDC) conducted a virtual tabletop exercise with a team from each of 4 states (Iowa, Kansas, Missouri, and Nebraska) committed to enhancing pediatric emergency preparedness through strategic collaboration and discussion. Steven Krug, MD, of the AAP Disaster Preparedness Advisory Council (DPAC) moderated the opening session and welcomed the state team participants and observers to the exercise. He introduced Georgina Peacock, MD, MPH, division director of CDC’s Division of Human Development and Disability, who provided opening remarks. Dr Peacock acknowledged the longstanding partnership between AAP and CDC’s Children’s Preparedness Unit (CPU) through pediatric emergency preparedness and response and shared the exercise background as an adaptation of a previous in-person exercise in 2016.

Module 1: David Schonfeld, MD, of the AAP DPAC, presented an exercise overview including the purpose of the exercise and the 3 objectives. State teams gave introductions before launching a video about CPU’s work from Eric Dziuban, MD, DTM, chief medical officer for CDC’s Division of Human Development and Disability (https://www.youtube.com/watch? v=dnBT48xjrYA). Dr Schonfeld explained the ground rules for the exercise and then transitioned to the first exercise module. Module 1 took participants through the first 21 days of the scenario, in which a high school student at summer science camp mistakenly breaks a vial of smallpox and unknowingly infects herself and another student. She begins to show symptoms and receives a diagnosis of a viral illness by a physician on day 14. She grows progressively ill, all the while not quarantined, and receives a diagnosis of smallpox during the third week postexposure. National media discover the story, and misinformation starts to spread quickly. The local hospital contacts the public health department, and affected states are asked to implement ring vaccination to all contacts who might have been exposed to smallpox. Using this strategy, the contacts of confirmed smallpox patients are vaccinated, as are persons in close contact with those contacts, as well as anyone who has been, or could have been, exposed to the patient who had smallpox. The state teams then had approximately 30 minutes to deliberate about whom to contact and how, whom to work with in the community, which persons to vaccinate, what information to provide publicly, and what actions to take. Each team gave a brief report of their actions and decisions to the group.

Module 2: Module 2 began with a video presentation from smallpox subject matter expert Brett Petersen, MD, medical officer for CDC’s Poxvirus and Rabies Branch in the National Center for Emerging and Zoonotic Infectious Diseases. The scenario continues at day 22 and describes the second student who is infected without being quarantined, and the decision is made to broaden ring vaccination to multiple states. A third video was shown about the Strategic National Stockpile, presented by Susan Gorman, PharmD, associate director for science for CDC’s Division of Strategic National Stockpile in the Office of Public Health Preparedness and Response. State teams then had another 30 minutes to discuss 3 questions concerning what groups to include in vaccination plans, recommendations for potentially exposed relatives, and how to ensure that the limited supply of smallpox vaccine is appropriately dispensed among children. After each team reported on its proposed course of action, a live call was made to AAP leadership at its national headquarters, and they explained the procedures AAP would undertake in a real scenario.

Module 3: Dr Schonfeld presented the final module to participants. By day 25 in the scenario, affected states had taken various courses of action, and hospital capacity was overwhelmed. Concern arose as to whether school should resume as scheduled. The questions for the third module asked state teams what advice to provide to school systems about closures or delayed openings, what strategies to use to respond to social media rumors, what approaches to use to brief health care providers, and how to create plans for handling up to 25 cases of smallpox in their state. After each state team reported their recommendations, a member of the planning team read 2 scenario-ending options—one described a positive outcome and the other described an unpleasant, but not implausible, outcome.

An exercise “hotwash,” or after-action discussion, was conducted immediately after the final module and offered an opportunity for participants to reflect on the exercise objectives and discussions and report on what went well, what was learned, and areas for improvement in future exercises. Dr Peacock then closed by thanking state teams and the several groups and individuals who played an instrumental role in successfully executing the exercise.

^aA tabletop exercise is an activity that simulates a real-life disaster scenario to promote discussion for agencies and professionals involved in emergency preparedness. Virtual tabletop exercises convene participants over videoconference technology.

Table 1.

Objectives and affiliated measures designated by the planning team for the Centers for Disease Control and Prevention/American Academy of Pediatrics virtual tabletop exercise to enhance pediatric emergency preparedness, Atlanta, Georgia, February 2017^a

Objective	Performance Measure
Communication: Improve communication between public health and pediatric leaders in the preparation for and response to public health emergencies to sufficiently address pediatric needs	Determine actions that states could take to improve communication specific to children’s issues
Collaboration: Share collaborative strategies that AAP chapters, pediatric clinicians, and public health leaders could implement in disaster situations to advance pediatric inclusion in responses at the state and local level	Identify discrete collaborative strategies in an emergency response to connect AAP chapter leaders, pediatric clinicians, and public health leaders
Follow-up actions: Identify ways for public health and pediatric clinicians to have continuous collaborative activities to optimize preparations for caring for children in public health emergencies	Articulate clear next steps in states to work together for continued preparedness for caring for children in public health emergencies

Abbreviation: AAP, American Academy of Pediatrics.

^a A virtual tabletop exercise is an activity that simulates a real-life disaster scenario to promote discussion for agencies and professionals involved in emergency preparedness. Participants from 4 states (Iowa, Kansas, Missouri, and Nebraska) met virtually using videoconference technology.

Evaluation Approach

Following exercise evaluation best practices,²¹ we applied a mixed-methods design informed by the Kirkpatrick Training Evaluation Model²² and US Department of Homeland Security Exercise Evaluation Program.²³ The planning team, which comprised public health experts, Disaster Preparedness Advisory Council members, and state representatives, iteratively convened to develop evaluation questions:

Exercise effectiveness: To what extent and how does participating in the VTTX result in improved pediatric preparedness capabilities at the state or local level?

Follow-up actions: To what extent and in what ways do participants engage in actions that support improved planning as a result of participating in the VTTX?

Strengths and weaknesses: What aspects of the VTTX structure, process, and/or logistics helped or hindered meeting VTTX objectives?

To optimize the evaluation’s utility, accuracy, and feasibility,²⁴ the planning team vetted the overarching evaluation questions, pilot-tested surveys for clarity and relevance, and member-checked summary reports to facilitate interpretation.

Data Sources and Data Collection

To document how we would evaluate the exercise, we developed an evaluation methods grid²⁴ aligning evaluation questions with methods, data sources, evaluators, and analyses (available upon request). To operationalize pediatric emergency preparedness, we considered participants’ (1) self-rated knowledge and confidence to fulfill key preparedness capabilities (Likert scale response), (2) perceptions of state strengths and weaknesses in pediatric emergency preparedness (Likert scale response), (3) collaboration and communication strategies suggested after participation (open-ended response), (4) intended changes after participation (open-ended response), (5) perceived degree of collaboration between sectors (Likert scale response), (6) perceptions of the exercise’s most and least beneficial aspects (open-ended response), and (7) set of questions probing participants about the extent to which stated exercise objectives were met (Likert scale response). We developed a survey covering these topics and administered the survey at 2 points: 24 hours before the VTTX and 24 hours after the VTTX.

For the survey section on self-rated knowledge and confidence to fulfill key preparedness capabilities, we developed a list of 12 capabilities. Because no consensus guidelines on pediatric emergency preparedness planning exist, the planning team reviewed US Department of Homeland Security²³ and CDC²⁵ standards and rank-ordered 12 capabilities that were most applicable to children’s needs and reflected the spectrum of disaster preparedness and response (Box). Survey response options were given on a 10-point Likert scale, where 1 = least knowledgeable or confident and 10 = most knowledgeable or confident.

For the Likert scale question on perceptions of state strengths and weaknesses in pediatric emergency preparedness, we asked participants to respond to the following statement: “I am able to identify my jurisdictions’ own strengths and weaknesses when it comes to pediatric preparedness planning” (1 = strongly disagree, 5 = strongly agree). For the open-ended question on collaboration and communication strategies suggested after participation, we requested, “Please state additional communication or collaboration strategies you would suggest that public health and pediatric staff could consider for improving pediatric preparedness at the state level.” For the open-ended question on intended changes after participation, we requested, “Please share what specific actions you are considering taking, especially regarding communication and collaboration.”

To assess intersectoral collaboration, we used the Strategic Alliance Formative Assessment Rubric (SAFAR),²⁶ a validated assessment tool for evaluating collaborative goals and strategies across sectors. Participants rated themselves on 4 dimensions: (1) goals and purpose; (2) activities, strategies, and tasks; (3) decision making and leadership; and (4) interagency communication and information sharing. The SAFAR uses a 6-point Likert scale, where 0 = least integrated and 5 = most integrated.

We evaluated (1) responses to the surveys, (2) observation forms completed by independent evaluators, (3) notes from the postexercise participant debriefing (ie, hotwash) about strengths and weaknesses of the exercise, (4) notes from an after-action review of planning committee members to debrief about the exercise, and (5) follow-up surveys collected 1 month and 6 months after the exercise (details available on request). We administered all surveys via SurveyMonkey. Response-rate optimization strategies²⁷ included tailored emails from professional peers, follow-up telephone calls to nonrespondents, and certificates of appreciation. Evaluators collected observation-based data activity without attributing data to individuals to limit observer effects. We enhanced our evaluation’s trustworthiness by soliciting feedback on result summaries from planning team members (ie, member checking²⁸), maintaining an audit trail outlining analytic decisions, and using multiple analysts.²⁹

Data Analysis

We used Stata version 14.1³⁰ to calculate descriptive statistics and 2-tailed nonparametric Wilcoxon signed-rank tests to compare median scores across time points. We handled missing data by using available-case analysis.

We qualitatively analyzed raw, open-ended, discussion-based, and observational data collectively by using the framework method.²⁸ In the framework method, 3 investigators (M.S., J.L.F., K.C.-O.) developed a priori codes based on evaluation objectives, read through the data to assign initial codes, iteratively modified the code list until they attained a complete understanding of possible codes, and recorded reflective notes²⁸ throughout. We resolved coding discrepancies via discussion until consensus was achieved. Finally, we clustered codes into categories to identify recurrent themes, and we assessed both basic frequencies and the degree to which a theme emerged across multiple sources to understand which themes were most prominent.

Four state teams, including 10 pediatricians and 16 state public health representatives, participated in the VTTX. The planning team was aware of only 18 participants before the exercise; thus, the 8 participants who joined the exercise ad hoc were included only in the 1- and 6-month follow-up surveys. The AAP Institutional Review Board determined that the evaluation was not subject to human subjects review.

Results

Effectiveness of the VTTX

In the postexercise survey, 13 of 18 participants reported that they determined actions to improve communication in their state; 8 participants agreed that collaborative strategies were determined, and 10 participants agreed that next steps for children’s preparedness were determined. Most participants perceived that the exercise generated productive discussion (n = 16) and that they could apply the knowledge gained from this exercise to their routine work (n = 15). The number of participants who were able to identify their state’s pediatric emergency preparedness strengths and weaknesses rose from 10 of 18 before the VTTX to 16 of 18 after the VTTX (W = 27, P = .01).

Knowledge and confidence to perform capabilities

Average participant scores in 12 pediatric emergency preparedness capabilities improved after the VTTX, 11 of which represented significant changes for both knowledge and confidence (Table 2). The knowledge (W = 33, P = .08) and confidence (W = 34, P = .08) increases for the medical surge activation/deactivation capability were not significant.

Table 2.

Median scores for self-rated knowledge of and confidence in fulfilling 12 pediatric emergency preparedness capabilities among 18 pediatricians and public health practitioners, before and after participating in a Centers for Disease Control and Prevention/American Academy of Pediatrics pediatric virtual tabletop exercise, Atlanta, Georgia, February 2017^a,b

Preparedness Capability^c	Knowledge			Confidence
Preparedness Capability^c	Preexercise Median Score (IQR)	Postexercise Median Score (IQR)	P Value^d	Preexercise Median Score (IQR)	Postexercise Median Score (IQR)	P Value^d
Before an emergency
1. Identify and engage potential partners outside your discipline to prepare for children’s needs in a disaster	6.0 (5.0-7.0)	8.0 (7.0-9.0)	.01	6.0 (5.0-7.0)	8.0 (7.3-8.0)	.001
2. Partner with other professional and community groups to ensure the ability to deliver pediatric health services in the event of an emergency	5.0 (3.5-6.5)	8.0 (7.0-8.8)	.001	6.0 (4.5-7.0)	8.0 (7.3-9.0)	.001
3. Provide guidance to community partners in educating their constituencies about preparedness, response, mitigation, and recovery should an emergency occur	6.0 (3.5-7.5)	8.0 (7.0-9.0)	.004	6.0 (4.0-8.0)	8.5 (6.0-9.0)	.01
4. Engage subject matter experts about medical countermeasures suitable for children	7.0 (5.0-7.5)	8.0 (8.0-9.0)	.01	7.0 (5.5-8.0)	9.0 (8.0-9.0)	.01
5. Determine state’s short- and long-term pediatric health service delivery priorities, including which roles belong to public health providers and which roles belong to medical providers	6.0 (3.0-7.0)	7.5 (7.0-8.8)	.01	6.0 (5.0-7.0)	7.5 (6.3-9.0)	.01
During or after an emergency
6. Solicit input from community partners about pediatric health service recovery needs	6.0 (3.0-7.5)	8.0 (7.0-8.8)	.01	6.0 (4.5-7.0)	8.0 (7.0-9.0)	.01
7. Serve as a resource for developing and sharing key messaging/information to the public, including recommended interventions or treatments	7.0 (5.0-9.0)	8.0 (7.3-9.0)	.046	7.0 (5.0-8.5)	8.0 (6.3-9.0)	.049
8. Coordinate public health and medical emergency management operations specific to pediatric needs	6.5 (2.3-7.8)	8.0 (7.3-9.0)	.003	6.0 (4.0-8.0)	8.0 (7.0-9.0)	.01
9. Coordinate information sharing between public health and medical information systems to maintain situational awareness about pediatric needs	6.0 (3.0-7.5)	8.0 (7.0-9.0)	.01	6.0 (5.0-8.0)	8.0 (6.3-9.0)	.04
10. Coordinate with partners to ensure that health screening, medications, and other services for pediatric populations remain accessible	6.0 (4.0-7.5)	8.0 (7.0-9.0)	.004	5.0 (3.5-7.5)	8.0 (6.3-9.0)	.003
11. Activate and deactivate medical surge as needed, with attention to the medical needs of children	6.0 (3.5-8.5)	8.0 (7.0-9.0)	.08	7.0 (4.0-9.0)	8.0 (7.0-9.0)	.08
12. Conduct meetings including all active partners to identify strengths and deficiencies of response to guide corrective actions, with specific attention to pediatric populations	5.0 (4.0-8.0)	9.0 (7.3-9.0)	.001	7.0 (5.0-8.5)	9.0 (8.0-9.0)	.002

Abbreviation: IQR, interquartile range.

^b Responses were given on a 10-point Likert scale, where 1 = least knowledgeable or confident and 10 = most knowledgeable or confident.

^c Target preparedness capabilities were identified and adapted from the US Department of Homeland Security Exercise Evaluation Program target capabilities list²³ and the Centers for Disease Control and Prevention Public Health Emergency Preparedness National Standards.²⁵

^d Wilcoxon signed-rank tests were used to compare preexercise and postexercise knowledge and confidence levels. P < .05 was considered significant.

Collaboration between pediatrics and public health

Participant self-ratings on the SAFAR dimensions increased significantly in 3 of 4 SAFAR domains from preexercise to postexercise: (1) activities, strategies, and tasks (W = 56, P = .001), (2) decision making (W = 47, P = .004), and (3) interagency communication and information sharing (W = 49, P = .004; Figure). These collaboration ratings were not different across the disciplines (supplemental analyses not shown).

Figure.

Mean self-ratings for each domain of the Strategic Alliance Formative Assessment Rubric (SAFAR) among pediatricians and public health practitioners (n = 18) in a virtual tabletop exercise on improving pediatric emergency preparedness, Atlanta, Georgia, February 2017. A virtual tabletop exercise is an activity that simulates a real-life disaster scenario to promote discussion for agencies and professionals involved in emergency preparedness. Participants from 4 states (Iowa, Kansas, Missouri, and Nebraska) met virtually using videoconference technology. The preexercise survey was conducted 24 hours before the exercise and the postexercise survey 24 hours after the exercise. Both surveys were administered via SurveyMonkey. SAFAR is a validated assessment tool for evaluating collaborative goals and strategies across sectors.²⁶ It is evaluated on a 6-point Likert scale, where 0 = least integrated and 5 = most integrated. Respondents were asked the following question: “Consider how integrated your state is across pediatric and public health sectors for pediatric preparedness. Integration refers to how well these 2 sectors align and collaborate to optimally prepare your state for a potential disaster. On a scale of 0-5, with 0 being not integrated at all and 5 being completely integrated, please rate how much you think your state is integrated across pediatric and public health sectors, specifically regarding [domains 1-4 in the figure].”

Qualitative responses

The main theme underlying participants’ comments on the VTTX’s influence on perceptions of their jurisdiction’s pediatric emergency preparedness knowledge, collaboration, and communication was the benefit of establishing connections between the 2 sectors. One participant wrote, “The connections just from the exercise are already looking to be invaluable moving forward.” Many participants also noted that each group had “the same interest in safeguarding kids.”

Furthermore, several participants noted an increase in knowledge of content areas as the result of conversations during the exercise, including health communication topics (eg, state secure email systems for infectious disease, tailored messaging to vulnerable populations, communication with bordering states), the roles and responsibilities of the other sector’s staff members in emergency preparedness, and state public health law. Some participants recognized their limited personal knowledge of available state resources and existing preparedness plans. One participant explained, “[The] discussion generated made me realize how much I did not know.”

Observation-based data revealed an exchange of information between sectors. Public health staff members queried pediatric providers about the best way to communicate with physicians (eg, Health Alert Network), hospital surge capacity for highly infectious patients in quarantine situations, and hospital surge capacity criteria. Questions from pediatricians to public health personnel concerned the state’s emergency response infrastructure, the state laboratory’s role in confirming disease outbreaks, and communications with bordering states and vulnerable populations.

Follow-up Actions

Intended actions

Immediately after the exercise, all respondents indicated they would be likely or very likely to initiate new partnerships and increase awareness for potential preparedness partners (Table 3). Seventeen participants indicated they would be likely to make changes to pediatric emergency preparedness planning and conduct a similar pediatric exercise in their state. Conversely, participants indicated it was not likely that they would engage in efforts to effect policy changes (to agencies involved in disaster response and to state legislature) to the same extent.

Table 3.

Ratings of likelihood of follow-up action among pediatricians and public health practitioners who participated in a pediatric virtual tabletop exercise and the degree to which respondents perceived that follow-up actions were taken, at 1 month and 6 months after exercise participation, Atlanta, Georgia, February 2017^a

Action	No. of Respondents Reporting Intention to Take Action^b in Survey Immediately After Exercise (n = 18)	No. of Respondents Reporting Progress in Follow-up Surveys^c
Action		1-Month Follow-up (n = 14)	6-Month Follow-up (n = 14)	P Value^d
Initiate new partnerships	18	9	11	.07
Increase awareness for potential preparedness partners	18	4	7	.02
Make changes to disaster preparedness or response planning	17	6	6	.99
Conduct a similar pediatric exercise in their state	17	4	7	.01
Propose new policies to agencies involved in disaster response	13	6	4	.07
Propose new policies to state legislature	5	4	5	.36

^b Intention to take action determined from participants’ response to the question, “Indicate how likely you are to follow up and take any of the specific actions now that you have participated in the exercise” for a given postexercise action on a 4-point Likert scale (where 1 = very unlikely and 4 = very likely). Responses of likely or very likely indicated intention to take a given follow-up action.

^c Completion of follow-up action at 1 month or 6 months postexercise determined from participants’ response to the question, “Please indicate whether you believe your state has made progress since the virtual tabletop exercise” for a given follow-up action.

^d Wilcoxon signed-rank tests were used to compare reported progress from 1 month to 6 months postexercise. P < .05 was considered significant.

Responses to open-ended survey questions and the debriefing elaborated on their ratings. Consistently reported intended actions included organizing similar drills, planning educational programs (eg, grand rounds), and establishing communication between the health department and state AAP chapter. Unique intended actions included reaching out to pediatricians to increase awareness, including a broader range of stakeholders (eg, family physicians), and better integrating pediatricians into local preparedness planning processes.

Completed actions

Of the 14 participants who completed both the 1- and 6-month follow-up surveys, participants at 1 month postexercise indicated making progress in initiating new partnerships (n = 9), making changes in disaster preparedness or response planning (n = 6), proposing new policies to agencies involved in disaster response (n = 6), increasing awareness for potential preparedness partners (n = 4), and conducting a similar pediatric exercise in their state (n = 4). At 6 months postexercise, respondents reported significant progress in increasing awareness for potential preparedness partners (W = 15, P = .01) and conducting similar pediatric exercises in their state (W = 15, P = .01). We did not observe significant changes for other actions from 1 month to 6 months after the exercise (Table 3).

Participants described actions they had pursued by 1 month postexercise that would not have occurred without VTTX participation, including developing connections to conduct a pediatric-centric exercise, broadly enhancing statewide awareness of pediatric emergency preparedness activities to stakeholders, and developing new pediatric policies and pediatric information for existing in-state exercises. At 6 months postexercise, participants described actions to incorporate pediatric considerations (eg, family reunification) into state preparedness plans and implement exercises focusing on pediatric needs. Several public health agency respondents reported improvements in their health department’s relationship with private practice pediatricians, the local AAP chapter, and diverse community partners (eg, Emergency Medical Services for Children).

Exercise Strengths and Weaknesses

Most participants reacted positively to how the VTTX was operated, agreed that their overall experience was positive, and indicated that the pre-VTTX materials helped them understand expectations. Of the 23 participants, 20 reported that the web-based platform ran smoothly, and 19 described the exercise length as appropriate; others suggested the length be reduced. Observers documented attentiveness and engagement during the VTTX, as well as during breaks, and indicated that the communications software enabled continuous communication among participants and administrators.

Participants identified 2 primary benefits of the exercise: (1) meeting and developing relationships with partners in public health and pediatrics and (2) understanding each other’s role in pediatric emergency preparedness. More than half of participants said that the VTTX needed to be improved by adding components that leverage pediatric expertise (ie, the VTTX “was not very pediatric”); the lack of such components made it “difficult for public health partners to determine how AAP could be engaged.” To remedy this issue, several respondents offered concrete examples of content to incorporate into future scenarios: family reunification, preventing disease spread in family networks, preparing childcare settings, pediatric surge capacity, response considerations specific to developmental status (eg, infant patients), and leveraging social media for risk communication. Other areas for improvement suggested by some participants were involving other professionals (eg, education, public information officers) and improving the usefulness of scenario endings and the live teleconferencing call to AAP leadership.

In both follow-up surveys, participants noted barriers to stimulating change in pediatric emergency preparedness at the local level (4 of 23 in the 1-month survey and 4 of 14 in the 6-month survey), including lack of staff member time and financial resources to engage state partners. Furthermore, in the 6-month survey, 2 participants noted that children’s preparedness was not a health department priority (as it is for hospitals); others reported that some VTTX participants had since left the department.

Lessons Learned

The VTTX helped participants clarify roles and responsibilities, facilitate knowledge transfer between sectors, and identify systems-level challenges. These benefits align with evaluations of in-person tabletop exercises,^11,12 including exercises targeting children’s needs.^31,32 However, barriers including resource constraints, staff turnover, and inconsistent planning priorities were noted, consistent with previous studies.⁸ Our findings expand upon limitations of extant research^10
-12 by identifying the degree to which improvements persist and translate to concrete local preparedness actions 6 months later. That participants demonstrated a largely positive, engaged experience as well as clear improvements in knowledge and confidence—even with the virtual format—attests to the VTTX model’s viability.

Interprofessional partnership remains a key knowledge gap⁹ for, and essential component of, preparedness planning for vulnerable populations.^8,14,33 Thus, the inclusion of a validated measure of collaboration that has not previously been demonstrated for pediatric tabletop exercises is notable. The changes documented reflect validated instruments where available (eg, SAFAR) as well as national standards informed by both evidence and legislative directives and applied to the pediatric population in accordance with practitioner expertise.³⁴

Our mixed-method study design enabled us to understand both the degree and content of enacted changes. This robust approach enhances the likelihood that these indicators reflect the capability of states in an actual emergency. Our approach departs from most research on disaster preparedness “lessons learned,” which has historically neglected strategies for measure selection and study design that could augment internal and external validity.^9,10,35

Limitations and Future Directions

This study had several limitations. First, our sampling approach limited the generalizability of our findings to other sectors that might be involved in an emergency response. However, gathering a selected group of professionals reflects both what an actual outbreak response would involve and how most tabletop exercises are conducted.¹¹ Second, we drew qualitative data from open-ended sources, presenting difficulty in understanding respondents’ true intent. This risk was somewhat tempered through measures such as member checking and methodological triangulation.²⁹ Third, because several planning team members were involved in funding decisions for participants’ organizations, social desirability bias was a concern. Fourth, we were unable to include persons who provided data at 1 month postexercise but not 6 months postexercise, which limited the power of our analysis.

Ultimately, a continuous quality improvement approach is essential in emergency preparedness planning.⁶ More rigorous investigations using larger and more professionally diverse samples, and prospective linkages with performance measures after actual emergencies, would further our progress toward understanding and improving the VTTX’s effectiveness. Additional evaluations of VTTXs and other emerging strategies (eg, virtual simulation models¹⁷) are particularly needed as children’s preparedness gains increasing attention^18,36 and could identify which approaches merit broad dissemination.³

Finally, although exercise-planning conventions dictate that exercises be evaluated according to standardized core capabilities,^23,34 none specific to children’s needs exist, limiting the comparability of existing studies.³³ Efforts to validate guidelines using expert consultation or operationalize existing resources (eg, Save the Children Report Card²) into target capabilities could help similar initiatives more objectively measure their impact.

Implications for Practice

When designing future pediatric tabletop exercises, emergency preparedness planners could include content that better leverages pediatric expertise, targets skills important for influencing state or local policy environments, focuses on medical surge activation/deactivation, or intentionally facilitates the 2 sectors in aligning goals. They might also consider ongoing technical assistance after exercise administration, because respondents reported progress in only 2 of 6 actions assessed from 1 month to 6 months postexercise, indicating that barriers to long-term change in pediatric preparedness persist.

Health departments can implement mechanisms (eg, quality improvement teams) to facilitate collaborative planning and policy development for pediatric emergency preparedness, given staff member time and funding as documented barriers to change. Health departments can also examine topics discussed by the 2 sectors to guide tailored communication and outreach to clinicians. Finally, pediatric providers and regional coalitions can consider how to incorporate VTTXs to build preparedness capacity while facilitating relationship building between public health and physicians.

A central concept reported by participants was the need for substantive inclusion of children’s unique medical and developmental characteristics in the VTTX beyond simply placing them in the scenario. Knowledge about children’s vulnerabilities is well-documented by academics,^6,36,37 government agencies,^1,38 and authoritative organizations safeguarding children.^2,5 In combination with suggestions proposed herein, this literature can inform future exercise content.

Conclusions

Tabletop exercises have limited reach because they often require in-person participation or fail to consider at-risk populations. We determined that a half-day pediatric VTTX was feasible and effective in promoting pediatric emergency preparedness capabilities and interprofessional collaboration. With proper attention to potential dissemination and sustainment barriers, such an exercise could enhance local preparedness capacity and protect more of the nation’s children during crises.

Footnotes

Acknowledgments

The authors thank the pediatric virtual tabletop exercise participants; Laura Aird, Kate Klein, Sean Diederich, and Breanna Smith (American Academy of Pediatrics); Rebecca Leeb, Frank Long, Scott Wilson, Brett Petersen, Susan Gorman, and Dan Sosin (Centers for Disease Control and Prevention); and the pediatric virtual tabletop exercise Planning Committee (list of members available at: ).

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This publication was supported by cooperative agreement #5NU38OT000167-04, funded by CDC. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC or the US Department of Health and Human Services.

ORCID iD

Marvin So, MPH

References

National Commission on Children and Disasters. 2010 Report to the President and Congress. AHRQ Pub No 10-M037. Rockville, MD: Agency for Healthcare Research and Quality; 2010.

Save the Children. Still at Risk: U.S. Children 10 Years After Hurricane Katrina. 2015 National Report Card on Protecting Children in Disasters. Faifield, CT: Save the Children; 2015.

Dziuban

Peacock

Frogel

. A child’s health is the public’s health: progress and gaps in addressing pediatric needs in public health emergencies. Am J Public Health. 2017;107(S2):S134–S137. doi:10.2105/AJPH.2017.303950

Becker-Blease

Turner

Finkelhor

. Disasters, victimization, and children’s mental health. Child Dev. 2010;81(4):1040–1052. doi:10.1111/j.1467-8624.2010.01453.x

American Academy of Pediatrics Disaster Preparedness Advisory Council, Committee on Pediatric Emergency Medicine. Ensuring the health of children in disasters. Pediatrics. 2015;136(5):e1407–e1417. doi:10.1542/peds.2015-3112

Cicero

Baum

. Pediatric disaster preparedness: best planning for the worst-case scenario. Pediatr Emerg Care. 2008;24(7):478–481. doi:10.1097/PEC.0b013e31817e2f2d

Hinton

Griese

Anderson

. CDC grand rounds: addressing preparedness challenges for children in public health emergencies. MMWR Morb Mortal Wkly Rep. 2015;64(35):972–974. doi:10.15585/mmwr.mm6435a3

Bevc

Simon

Montoya

Horney

. Institutional facilitators and barriers to local public health preparedness planning for vulnerable and at-risk populations. Public Health Rep. 2014;129(suppl 4):35–41. doi:10.1177/00333549141296S406

Khan

Fazli

Henry

. The evidence base of primary research in public health emergency preparedness: a scoping review and stakeholder consultation. BMC Public Health. 2015;15:432. doi:10.1186/s12889-015-1750-1

10.

Beerens

RJJ

Tehler

. Scoping the field of disaster exercise evaluation—a literature overview and analysis. Int J Disaster Risk Reduct. 2016;19:413–446. doi:10.1016/j.ijdrr.2016.09.001

11.

Dausey

Buehler

Lurie

. Designing and conducting tabletop exercises to assess public health preparedness for manmade and naturally occurring biological threats. BMC Public Health. 2007;7:92. doi:10.1186/1471-2458-7-92

12.

Skryabina

Reedy

Amlôt

Jaye

Riley

. What is the value of health emergency preparedness exercises? A scoping review study. Int J Disaster Risk Reduct. 2017:21:274–283. doi:10.1016/j.ijdrr.2016.12.010

13.

Wingate

Perry

Campbell

David

Weist

. Identifying and protecting vulnerable populations in public health emergencies: addressing gaps in education and training. Public Health Rep. 2007;122(3):422–426. doi:10.1177/003335490712200319

14.

Waugh

Jr Streib

. Collaboration and leadership for effective emergency management. Public Adm Rev. 2006;66(suppl 1):131–140. doi:10.1111/j.1540-6210.2006.00673.x

15.

Biddinger

Cadigan

Auerbach

. On linkages: using exercises to identify systems-level preparedness challenges. Public Health Rep. 2008;123(1):96–101. doi:10.1177/003335490812300116

16.

US Department of Homeland Security, Federal Emergency Management Agency. Emergency Management Institute (EMI) Virtual Table Top Exercise (VTTX). https://training.fema.gov/programs/emivttx.aspx. 2016. Accessed December 10, 2016.

17.

Hsu

Bayram

Levinson

Yang

Monahan

. State of virtual reality based disaster preparedness and response training. PLoS Curr. 2013;5. doi:10.1371/currents.dis.1ea2b2e71237d5337fa53982a38b2aff

18.

Chung

Gardner

Schonfeld

. Addressing children’s needs in disasters: a regional pediatric tabletop exercise. Disaster Med Public Health Prep. 2018;12(5):582–586. doi:10.1017/dmp.2017.137

19.

Zoom Video Communications, Inc. Zoom Version 3.5. San Jose, CA: Zoom Video Communications; 2014.

20.

2017 AAP pediatric and public health virtual tabletop exercise [Video]. YouTube. https://youtu.be/nc3KMl_V3D4. September 12, 2017. Accessed December 4, 2017.

21.

Jackson

BA.

The Problem of Measuring Emergency Preparedness: The Need for Assessing “Response Reliability” as Part of Homeland Security Planning. Santa Monica, CA: RAND Corporation; 2008. http://www.rand.org/content/dam/rand/pubs/occasional_papers/2008/RAND_OP234.pdf. Accessed December 28, 2016.

22.

Kirkpatrick

. Evaluating Training Programs: The Four Levels. 3rd ed. San Francisco, CA: Berrett-Koehler; 2006.

23.

US Department of Homeland Security. Homeland Security Exercise and Evaluation Program (HSEEP). Washington, DC: US Department of Homeland Security; 2013. https://www.fema.gov/media-library-data/20130726-1914-25045-8890/hseep_apr13_.pdf. Accessed December 28, 2016.

24.

Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR Recomm Rep. 1999;48(RR-11):1–40.

25.

US Department of Health and Human Services, Centers for Disease Control and Prevention. Public health preparedness capabilities: national standards for state and local planning. https://www.cdc.gov/cpr/readiness/00_docs/DSLR_capabilities_July.pdf. Published March 2011. Accessed February 10, 2019.

26.

Gajda

. Utilizing collaboration theory to evaluate strategic alliances. Am J Eval. 2004;25(1):65–77. doi:10.1016/j.ameval.2003.11.002

27.

Flanigan

McFarlane

Cook

. Conducting survey research among physicians and other medical professionals: a review of current literature. In: Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association; 2008:4136–4147.

28.

Ritchie

Lewis

. Qualitative Research Practice: A Guide for Social Science Students and Researchers. London, UK: Sage; 2003.

29.

Sandelowski

. Rigor or rigor mortis: the problem of rigor in qualitative research revisited. ANS Adv Nurs Sci. 1993;16(2):1–8.

30.

StataCorp

LLC

. Stata Version 14.1. College Station, TX: StataCorp LLC; 2015.

31.

Behar

Upperman

Ramirez

Dorey

Nager

. Training medical staff for pediatric disaster victims: a comparison of different teaching methods. Am J Disaster Med. 2008;3(4):189–199.

32.

Bank

Khalil

. Are pediatric emergency physicians more knowledgeable and confident to respond to a pediatric disaster after an experiential learning experience? Prehosp Disaster Med. 2016;31(5):551–556. doi:10.1017/S1049023X16000704

33.

Subbarao

Lyznicki

Hsu

. A consensus-based educational framework and competency set for the discipline of disaster medicine and public health preparedness. Disaster Med Public Health Prep. 2008;2(1):57–68. doi:10.1097/DMP.0b013e31816564af

34.

Shelton

Nelson

McLees

Mumford

Thomas

. Building performance-based accountability with limited empirical evidence: performance measurement for public health preparedness. Disaster Med Public Health Prep. 2013;7(4):373–379. doi:10.1017/dmp.2013.20

35.

Koenig

Schultz

Gould Runnerstrom

Ogunseitan

. Public health and disasters: an emerging translational and implementation science, not “lessons learned.” Disaster Med Public Health Prep. 2017;11(5):610–611. doi:10.1017/dmp.2017.11

36.

Siegel

Strauss-Riggs

Needle

. Prioritization of pediatric BRNE disaster preparedness education and training needs. Clin Pediatr Emerg Med. 2014;15(4):309–317. doi:10.1016/j.cpem.2014.10.002

37.

Abramson

Park

Stehling-Ariza

Redlener

. Children as bellwethers of recovery: dysfunctional systems and the effects of parents, households, and neighborhoods on serious emotional disturbance in children after Hurricane Katrina. Disaster Med Public Health Prep. 2010;4(suppl 1):S17–S27. doi:10.1001/dmp.2010.7

38.

National Advisory Committee on Children and Disasters. Healthcare Preparedness for Children in Disasters: A Report of the NACCD Healthcare Preparedness Working Group. Washington, DC: US Department of Health and Human Services; 2015.

Extending the Reach of Pediatric Emergency Preparedness: A Virtual Tabletop Exercise Targeting Children’s Needs