Mapping the Expert Mind: Integration Method for Revising the ACES Medical Simulation Curriculum

Abstract

Purpose:

This article shares our experience developing an integrated curriculum for the ACES (Acute Critical Event Simulation) program. The purpose of the ACES program is to ensure that health care providers develop proficiency in the early management of critically ill patients. The program includes multiple different types of educational interventions (mostly simulation-based) and targets both specialty and family physicians practicing in tertiary and community hospitals.

Methods:

To facilitate integration between different educational interventions, we developed a knowledge repository consisting of cognitive sequence maps that make explicit the flow of cognitive activities carried out by experts facing different situations - the sequence maps then serving as the foundation upon which multimodal simulation scenarios would be built. To encourage participation of experts, we produced this repository as a peer-reviewed ebook. Five national organizations collaborated with the Royal College of Physicians and Surgeons of Canada to identify and recruit expert authors and reviewers. Foundational chapters, centered on goals/interventions, were first developed to comprehensively address most tasks conducted in the early management of a critically ill patient. Tasks from the foundational chapters were then used to complete the curriculum with situations. The curriculum development consisted of two-phases each followed by a peer-review process. In the first phase, focus groups using web-conferencing were conducted to map clinical practice approaches and in the second, authors completed the body of the chapter (e.g., introduction, definition, concepts, etc.) then provided a more detailed description of each task linked to supporting evidence.

Results:

Sixty-seven authors and thirty-five peer reviewers from various backgrounds (physicians, pharmacists, nurses, respiratory therapists) were recruited. On average, there were 32 tasks and 15 situations per chapter. The average number of focus group meetings needed to develop a map (one map per chapter) was 6.7 (SD ± 3.6). We found that the method greatly facilitated integration between different chapters especially for situations which are not limited to a single goal or intervention. For example, almost half of the tasks of the Hypercapnic Ventilatory Failure chapter map were borrowed from other maps with some modifications, which significantly reduced the authors’ workload and enhanced content integration. This chapter was also linked to 6 other chapters.

Conclusions:

To facilitate curriculum integration, we have developed a knowledge repository consisting of cognitive maps which organize time-sensitive tasks in the proper sequence; the repository serving as the foundation upon which other educational interventions are then built. While this methodology is demanding, authors welcomed the challenge given the scholarly value of their work, thus creating an interprofessional network of educators across Canada.

Keywords

Education Medical continuing Patient simulation Simulation training Practice management medical Educational technology Competency based education

Introduction

Educators are challenged to develop curriculum structures that are coherent, coordinated, and integrated within and across different educational interventions.^1-3 Program development is complex because educational interventions invariably overlap, are offered at different time periods, by different faculty, and to learners whose knowledge is constantly evolving. Ideally, educational interventions within a curriculum should never be considered in isolation; instead, they should articulate with one another to build on previously acquired knowledge and skills.¹ We would like to share our experience developing an integrated curriculum for the ACES (Acute Critical Event Simulation) program.⁴

The ACES program was developed over 17 years ago and has since been offered nationally and internationally to train learners from various clinical milieus and contexts. The program consists of multiple components including face-to-face didactic sessions, various forms of simulation (theater-based, phone, virtual), and technical skills workshops. It targets both specialty (ie, anesthesia, critical care medicine, emergency medicine, internal medicine, surgery) and family physicians practicing in tertiary and community hospitals. The goal of the ACES program is to ensure that health care providers (both individuals and teams) from various clinical backgrounds build proficiency in the early management of a medical emergency given their individual scope of practice and the clinical settings in which they work.

Based on a needs assessment conducted in 2013, it was decided to update the curriculum and remove all didactic content from the face-to-face component, transforming the ACES program into a purely simulation-based experience. Klein’s^5,6 recognition-primed decision (RPD) model was chosen as the conceptual framework supporting curriculum development. Klein’s model postulates that experts, recognizing a situation as familiar, already have an action plan in mind that informs them about which goals to prioritize, what tasks to perform to achieve these goals, what cues to monitor, and what to expect.⁷ In a crisis, most of the experts’ efforts are centered on implementing their action plan.

By attempting to replicate the RPD approach used by experts and repeatedly immersing learners in clinical situations frequently encountered in real life, we hoped to make the ACES course more clinically relevant, thus promoting learning and retention.⁸

Description of Our Curriculum Development Methods

Curriculum structure

We conducted a 1-day interprofessional meeting and used a modified Delphi method to identify clinical situations that evolved from poorly (eg, undifferentiated hypotension or hypoxemia) to better differentiated (eg, massive pulmonary embolism or septic shock). A total of 30 situations were prioritized based on practice and learning relevance to the various disciplines represented. To emulate the RPD model,^5,6 our initial intent was to select 30 authoring teams to define clinical approaches for these different situations. However, during a test authoring trial for the undifferentiated situations, severe hypoxemia and hypotension, we realized that different situations shared common goals and tasks. Developing a curriculum structured around clinical situations was therefore introducing the risks of redundancy, repetition, and even contradiction, as different teams could author the same tasks independently of one another. Our efforts to make a more relevant simulation-based curriculum using clinical situations as a starting-point were therefore hindering curriculum cohesiveness.

To facilitate curriculum integration and ensure that different educational interventions articulated well with one another, we decided to first develop a knowledge repository consisting of cognitive sequence maps that make explicit the flow of cognitive activities performed by experts facing different situations.⁹ These sequence maps would serve as the foundation upon which multimodal simulation scenarios would then be built. We also decided to directly link tasks to the supporting basic science concepts and clinical evidence, thus facilitating integration between what is done at the bedside and scientific data.¹⁰

To encourage participation of experts, we produced this knowledge repository as an ebook (every goal or intervention becoming a chapter) and developed a peer-review process and dissemination strategy to satisfy the requirements of scholarly work.^11,12 We first developed foundational chapters centered on goals/interventions and then used this material to complete the curriculum with situations. For the foundational chapters, we assumed that patients being managed were undifferentiated (ie, no previous knowledge of the patient). Clinical goals and interventions were selected (eg, personal protection, establishing goals of care, administering fluids, vasopressors, noninvasive ventilation) to comprehensively address most tasks conducted in the early management of a critically ill patient. We excluded goals such as diagnosis and treatment of specific conditions (eg, how to diagnose pneumonia and select antibiotics) given that the ebook was centered on early management.

Preparatory work and collaboration with national societies and authors

Instructions to authors, authoring templates and an exemplar of a completed chapter with its own map were developed. We also developed virtual simulation scenario to demonstrate how the ebook, serving as a knowledge repository, could be used as the foundational work to then create simulation scenarios. Our vision was shared with 5 national organizations: The Canadian Critical Care Society, Canadian Association of Critical Care Nurses, Canadian Society of Hospital Pharmacists, Canadian Society of Respiratory Therapists, and the Infection Prevention and Control Canada. All organizations agreed to participate in the project. The role of these organizations was to help recruit national and international expert authors and reviewers (eg, physicians, nurses, pharmacists, respiratory therapists) who had research and/or clinical expertise in the domains of interest. Potential authors were then invited to participate in a web conference during which we shared our vision, confirmed their participation, explained the authoring process, and set timelines. Participating authors agreed to the reassignment of copyright for the content and were asked to submit a declaration of conflict of interest.

Map development

For each chapter, we conducted a series of focus groups to perform cognitive task analysis and uncover the explicit and implicit knowledge of experts.⁹ We used the program CMAP tools, produced by the Institute of Human and Machine Cognition, to develop these cognitive sequence maps. To simplify map creation, we assumed that enough clinical resources (human and physical) were available, thus avoiding the need to prioritize and allocate tasks because of limited resources. Authors were asked to describe the tasks that should be performed and to identify any situations that would change management and thus need to be recognized. Authors were also informed of specific tasks that had already or were concurrently being developed by other teams to encourage articulation between different chapters. Once all individual chapter authors agreed that the map accurately reflected their practice approach and addressed most situations encountered in real life, the map was sent for peer review. Reviewers’ comments and suggested revisions were then addressed by the authors using web conferencing.

Body of the chapter, task write-up, and peer-review process

Having completed the map peer-review process, development entered a second phase, authoring the body of the chapter and tasks. The body of the chapter included an introduction, definitions, key concepts transcending all tasks (eg, shunting for the chapter on severe hypoxemia), a list of abbreviations, a narration of the map, and a conclusion. Each task was linked to a task description which provided explicit details about what needed to be done and how (ie, subtasks) and a task rationale which presented related basic science concepts and/or clinical evidence (Figure 1 and multimedia supplement—https://drive.google.com/open?id=1QrNgLBebWYKA5eteymz2GpkBqkhXSnUT).

Figure 1.

Map and task display.

The authors attended a web conference during which the editorial team explained and clarified the authoring process and helped assign chapter body and task write-up work to different team members. To facilitate the authoring process, authors accessed an online CMAP (Figure 2) which provided links to authoring templates for both the body of the chapter and tasks. Once completed, the chapters were reviewed by the editing team to ensure compliance with the authoring instructions, verifying that the Task Description was focused on the “What and how” and the Rationale on the “Why.” The completed chapters were then sent to reviewers (usually the same individuals who had reviewed the initial map). Reviewers were reminded to only review documents as the map had already been peer-reviewed. We used an open peer-review process—the authors’ names were known to the reviewers and vice versa.

Figure 2.

Example of an online CMAP.

Results

Authors and reviewer’s recruitment

A total of 67 authors and 35 peer reviewers agreed to participate. Sixty-six percent of invited authors were physicians (n = 44), 27% of whom were residents, 15% were pharmacists, 13% were nurses, and 6% were respiratory therapists. Whenever possible, senior authors were paired with residents to promote academic scholarship and mentored collaboration. Sixty-nine percent of reviewers were physicians, 23% were pharmacists, 6% were nurses, and 3% were respiratory therapists.

Major sections

The ebook consists of 4 different sections: First Things First, Ongoing Resuscitation, Typical Situations, and Clinical Drug Summaries (https://navme.royalcollege.ca). As shown in Figure 3, the section on “First Things First” detailed the tasks required to address 5 separate management goals including managing early physiological anomalies (ie, First Glance), Infection Prevention and Control, Establishing Therapeutic Relationships, Goals of Care, and Transport and Disposition. The Ongoing Resuscitation section included more advanced interventions required to complete the ABCs and comprised 9 chapters. The section on Typical Situations comprised 3 chapters and explained the various sequence of tasks needed in the management of Severe Hypoxemia, Hypercapnic Ventilatory Failure, and Undifferentiated Hypotension/Hypoperfusion. The Clinical Drug Summaries section comprised 7 chapters which reviewed medications commonly used to resuscitate and manage critically ill patients. This latter section did not include any map but was well integrated with other chapters; any drug mentioned in the ebook was linked to its summary.

Figure 3.

eBook structure.

Tasks

The total number of tasks and situations in the ebook were 542 and 250, respectively (average of 32 tasks and 15 situations per chapter). Eighty-five percent of all tasks were linked to a document which described them in more details (ie, subtasks) and provided rationales to integrate clinical actions with basic science and clinical evidence.

Map development

On average, 6.7 (SD ±3.6) focus group meetings using web conferencing were needed to develop a map (1 map per chapter), and the average time spent developing a map was 7.4 (SD ±4.2) hours varying from 3 to as much as 14 hours.

Curriculum articulation

To assess curriculum integration, we examined the proportion of original tasks which needed to be developed to complete a map. We also examined whether tasks borrowed from other chapters had to be edited or could be reused without any modifications. If modified, we assessed whether they contradicted similar tasks from other chapters.

The 5 chapters in First Things First were independent of one another because their goals were complementary yet distinct. As shown in Table 1, for all 5 chapters in the First Things First section, almost all tasks were original (ie, were not copied or modified from another chapter); a single task was borrowed from the Severe Hypoxemia chapter without any modification.

Table 1.

Task source.

Sections and chapters	Proportion of original tasks, %	Proportion of reused tasks with no modification, %	Proportion of reused tasks with modifications, %
First Things First section (5 chapters)	99.8	0.2	0.0
Ongoing Resuscitation section (9 chapters)	94.7	0.0	5.3
Typical situations	80.6	5.7	13.7
Hypercapnic Ventilatory Failure	56.4	2.6	41.0
Severe Hypoxemia	85.4	14.6	0.0
Undifferentiated Hypotension-Hypoperfusion	100.0	0.0	0.0

The 9 chapters in the section on Ongoing Resuscitation related to tasks usually performed after already completing tasks described in the First Things First section. As shown in Table 1, on average, 95% of all tasks were original, 5% being reused from other chapters with some modifications. None of the tasks from the Ongoing Resuscitation chapters contradicted information provided in other chapters.

In the section on Typical Situations, as much as 20% of all tasks were borrowed from other chapters. The Severe Hypoxemia and Undifferentiated Hypotension chapters were the first chapters to be developed and led to a change in our curriculum development method (see Methods—Curriculum structure) which became centered on foundational chapters (ie, centered on goals and interventions) instead of situations. Eighty-five percent of all tasks in the Severe Hypoxemia chapter were original. All tasks in the Undifferentiated Hypotension chapter were original. However, this chapter was linked directly to 4 other foundational chapters essential for the management of hypotensive patients (eg, fluid administration, vasopressors—https://navme.royalcollege.ca/EN/imap.shtml#43). The Hypercapnic Ventilatory Failure chapter was developed once all foundational chapters had been developed. Only 56% of all tasks were original. Of the tasks borrowed from other chapters, 94% were modified. None of the tasks used in the Hypercapnic Ventilatory Failure chapter contradicted tasks from other chapters. This chapter was also linked to 6 other chapters (https://navme.royalcollege.ca/EN/imap.shtml#245).

Discussion

Following a needs assessment, the Royal College of Physicians and Surgeons of Canada decided to revise the ACES program, include precourse online material, and transform the face-to-face component into a multimodal simulation experience. To facilitate curriculum integration, we completed an ebook which is serving as a knowledge repository upon which multimodal simulation scenarios are now being built. In this ebook, the core material, organized through cognitive sequence maps, is first presented, informing learners of what they should consider and do given certain situations. Unlike traditional books, the rationale supporting each task is available on request (ie, the reader can click on the task to obtain additional information). The mapping exercise identified 542 tasks and 150 situations that needed to be recognized because they modify management. The project demanded much preparatory work and led to collaboration between 6 national organizations. In total, 67 authors and 35 reviewers participated in the project. While the development of the ebook demanded much time and effort on the part of the authors as well as the editorial and production teams, the creation of this knowledge repository is facilitating the development of a cohesive multimodal and simulation-based curriculum, in part because it describes a standard approach and defines performance objectives.

The ebook structure was informed by literature on experts’ reasoning and clinical decision-making. Klein⁵ developed the RPD model which describes how experts make decisions in natural settings when situations are rapidly evolving. Klein showed that experts, having recognized a situation as familiar, already have a plan in mind and thus can focus all their cognitive resources on task execution. By linking situations and tasks into clinical management strategy maps, the ebook attempts to replicate the knowledge structure and practice actions used by experts. The use of interpretive graphics focuses attention, fosters psychological engagement, and increases learning.^13,14 The maps typically present fewer than 13 tasks or major steps at one time to avoid overwhelming the cognitive resources of learners.¹⁵

The map development process was challenging; it took, on average, 7.4 hours to create a single map possibly because recognizing a situation is an unconscious process and thus, by definition, difficult to explain or describe. Similarly, the procedural knowledge of experts is often implicit. It has been demonstrated that experts omit as many as 70% of all steps required when asked to describe how they perform routine procedures.¹⁶ While the initial development of maps in the first chapters was demanding, it ultimately led to an economy of production in subsequent chapters especially when developing the Typical Situation chapter which, given their nature, are not limited to a single goal or intervention. For example, the map of the Hypercapnic Ventilatory Failure chapter contained 29 tasks, almost half of these were borrowed from other maps with some modifications, which significantly reduced the authors’ workload and enhanced content integration.

The ebook is not meant as a stand-alone educational intervention; it is complemented by online and face-to-face simulation. We have found that having access to maps and supporting evidence documents greatly facilitates the development of simulation scenarios (this work is now in production) while ensuring curriculum consistency between different educational interventions. In addition, we anticipate that having access to a knowledge structure that is clinically sound will facilitate future updates of course material or customization of content to better match a learners’ practice context or scope of practice (eg, medical students versus residents, tertiary versus rural hospitals).

Learning and curriculum integration are facilitated when newly acquired knowledge articulates with previous knowledge.^3,17 Mapping organizes the knowledge of experts in a manner that makes explicit any connections to previous knowledge. The degree to which maps articulate with one another is best exemplified in the Hypercapnic Ventilatory Failure chapter (https://navme.royalcollege.ca/EN/imap.shtml#245), a chapter developed once all foundational chapters had been completed. This chapter connects with no less than 6 foundational chapters. To the learner, clearly displaying how the content of different chapters articulate with one another builds on previously acquired knowledge and skills and avoids repetition and contradiction between chapters. Furthermore, knowledge is more easily retained and retrieved if it is organized in a manner that reflects how it will be used in real life.⁸ By organizing knowledge using maps that present clinically relevant approaches, learning is reinforced not only because maps have rich interconnections, but also because they are well matched to activities performed at the patients’ bedside.^8,17 In addition, the maps explicitly identify situations that matter because they change management. This replicates the experts’ ability to recognize meaningful patterns which is lacking in novices who often fail to integrate diagnosis and management.^18,19

Eighty-five percent of all tasks were linked to documents that provided supporting evidence to better integrate clinical actions and scientific concepts. It has been proposed that a key goal of medical education should be to integrate clinical facts and the supporting scientific concepts into a coherent package.²⁰ Moreover, having a deeper understanding of the underlying scientific mechanisms facilitates recall and may also improve diagnostic accuracy when pressured by time.²⁰ In addition, this deeper understanding should enable learners to modify their initial approaches in real life given different clinical contexts and evolving situations.

Aware that publishing online learning resources faces greater credibility threats than publishing more traditional sources of information such as scholarly journals,^21,22 the authors of this ebook were carefully selected. They were recommended by national specialty societies, professional organizations, or peers and were further identified through previous scholarly work. This helped to recruit committed, academic-clinician topic experts from various fields. All work was peer-reviewed using a transparent process recommended by the International Committee of Medical Journal Editors (ICMJE) whose guidelines inform the working standards of many high-impact medical journals. In addition, scholarship and editorial rigor was maintained by meeting and often exceeding free open-access medical education (FOAMed) quality assurance standards and validated ratings criteria presently cited.^21-23 Setting high standards probably increased the time and effort demanded of authors. Paradoxically, the use of a rigorous methodology did not appear to discourage authors who remained committed to the project despite their many other responsibilities.²⁴

There are several limitations to our work. The maps may include some of the author’s idiosyncrasies and do not necessarily reflect an approach that is widely shared among experts. While possible, the need to present supporting evidence for each task in addition to the peer-review process likely minimizes this possibility rendering the maps more generalizable. Another limitation would be the relatively narrow scope of the ACES curriculum which is limited to the early management of undifferentiated critically ill patients. Our initial objective was to develop chapters on 30 clinically relevant situations evolving from poorly to better differentiated. However, it became obvious early in the developmental process that our efforts should be centered on the development of foundational chapters. We demonstrated that the methodology described here does lend itself to creating comprehensive cognitive sequence maps that address specific situations. The maps were also developed assuming unlimited access to human and physical resources to manage patients. Clearly, the recommended approaches would need to be modified if applied in a context with more limited resources. We decided to assume full access to resources because it is easier to truncate a map than to create a new one. Furthermore, unless deemed essential and deserving of special emphasis, most crisis resource management tasks were omitted because these skills usually transcend all tasks (eg, good communication is required to perform any task in a crisis) so would have added unnecessary repetition and cluttered the maps.⁷

In summary, we developed a methodology to make explicit situations that must be recognized because they lead to a change in the management of critically ill patients. We displayed clinical approaches using maps that organize time-sensitive tasks in the proper sequence. These maps are richly interconnected and now support the development of a cohesive and well-integrated ACES curriculum. The methodology and peer-review process demanded much of our authors. However, their response would suggest that educators welcomed the challenge of developing this valuable resource and through their efforts have helped create an interprofessional network of educators extending across Canada.

Footnotes

Funding:

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests:

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: P.C., G.B., K.D., A.L., S.B., and K.M. are employees of the Royal College of Physicians and Surgeons of Canada.

Author Contributions

Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work: PC, GB, KDR, SY, AL, AS, SS, SB, KMC and TW. Drafting the work or revising it critically for important intellectual content: PC, GB, KDR, SY, AL, AS, SS, SB, KMC and TW. Final approval of the version to be published: PC and GB

ORCID iD

Pierre Cardinal

References

Australian Medical Council Limited (AMC). Standards for Assessment and Accreditation of Primary Medical Programs by the Australian Medical Council 2012. Kingston, ACT, Australia: AMC; 2012.

Liaison Committee on Medical Education (LCME). Functions and Structure of a Medical School: Standards for Accreditation of Medical Education Programs Leading to the MD Degree. Washington, DC: LCME; 2016.

Brauer

Ferguson

. The integrated curriculum in medical education: AMEE Guide No. 96. Med Teach. 2015;37:312-322. doi:10.3109/0142159X.2014.970998.

Sarti

Sutherland

Landriault

DesRosier

Brien

Cardinal

Understanding of evaluation capacity building in practice: a case study of a national medical education organization. Adv Med Educ Pract. 2017;8:761-767. doi:10.2147/AMEP.S141886.

Klein

. A recognition-primed decision (RPD) Model of rapid decision making. In: Klein

Orasanu

Calderwood

, et al., eds. Decision Making in Action: Models and Methods. Norwood, NJ: Ablex; 1993:138-147.

Klein

GA.

Sources of Power: How People Make Decisions. Cambridge, MA: MIT Press; 1998.

Brindley

Cardinal

Optimizing Crisis Resource Management to Improve Patient Safety and Team Performance. Ottawa, ON, Canada: Royal College of Physicians and Surgeons of Canada; 2017.

Ambrose

Bridges

DiPietro

Lovett

Norman

How Learning Works. San Francisco, CA: Jossey-Bass; 2010.

Crandall

Hoffman

Klein

Working Minds. Cambridge, MA: MIT Press; 2006.

10.

Finnerty

Chauvin

Bonaminio

Andrews

Carroll

Pangaro

Flexner revisited: the role and value of the basic sciences in medical education. Acad Med. 2010;85:349-355. doi:10.1097/ACM.0b013e3181c88b09.

11.

Van Melle

Curran

Goldszmidt

Lieff

Lockyer

St Onge

Toward a Common Understanding: Advancing Es for Clinical Faculty in Canadian Medical Schools. A Position Paper. Ottawa, ON: Canadian Association for Medical Education; 2012. http://www.came-acem.ca/pubs_position_papers_en.Php.

12.

Van Melle

Lockyer

Curran

Lieff

St Onge

Goldszmidt

. Toward a common understanding: supporting and promoting education scholarship for medical school faculty. Med Educ. 2014;48:1190-1200. doi:10.1111/medu.12543.

13.

Clark

Lyons

Graphics for Learning: Proven Guidelines for Planning, Designing, and Evaluating Visuals in Training Materials. San Francisco, CA: John Wiley & Sons; 2010.

14.

Clark

Mayer

RE.

E-Learning and the Science of Instruction. San Francisco, CA: Jossey-Bass; 2003.

15.

Clark

RC.

Building Expertise: Cognitive Methods for Training and Performance Improvement. San Francisco, CA: John Wiley; 2008.

16.

Clark

Pugh

Yates

Inaba

Green

Sullivan

The use of cognitive task analysis to improve instructional descriptions of procedures. J Surg Res. 2012;173:e37-e42. doi:10.1016/j.jss.2011.09.003.

17.

Kulasegaram

Martimianakis

Mylopoulos

Whitehead

Woods

Cognition before curriculum. Acad Med. 2013;88:1578-1585. doi:10.1097/acm.0b013e3182a45def.

18.

Bransford

How People Learn. Washington, DC: National Academies Press; 2004.

19.

Monajemi

Rikers

Schmidt

Clinical case processing: a diagnostic versus a management focus. Med Educ. 2007;41:1166-1172. doi:10.1111/j.1365-2923.2007.02922.x.

20.

Woods

Science is fundamental: the role of biomedical knowledge in clinical reasoning. Med Educ. 2007;41:1173-1177. doi:10.1111/j.1365-2923.2007.02911.x.

21.

Chan

Grock

Paddock

Kulasegaram

Yarris

Lin

Examining reliability and validity of an online score (ALiEM AIR) for rating free open access medical education resources. Ann Emerg Med. 2016;68:729-735.

22.

Grock

Paolo

Free open access medical education: a critical appraisal of techniques for quality assessment and content discovery. Clin Exp Emerg Med. 2016;3:183-185.

23.

Lin

Thoma

Trueger

Ankel

Sherbino

Chan

Quality indicators for blogs and podcasts used in medical education: modified Delphi consensus recommendations by an international cohort of health professions educators. Postgrad Med J. 2015;91:546-550.

24.

Sarti

Sutherland

Landriault

DesRosier

Brien

Cardinal

Exploring the components of physician volunteer engagement: a qualitative investigation of a national Canadian simulation-based training programme. BMJ Open. 2017;7:e014303. doi:10.1136/bmjopen-2016-014303.