Using Intervention Mapping to Develop a Virtual Reality Intervention to Support COVID-19 Vaccination Among Black Families

Abstract

Pediatric uptake of COVID-19 vaccines is low, particularly among Black children. There is a critical need for pediatric clinicians to address COVID-19 vaccine concerns through community-centered, culturally appropriate communication strategies. Virtual reality (VR) may provide a modality to support clinicians’ relational skills related to COVID-19 vaccine by providing a safe, realistic environment to practice preferred communication behaviors. The aim of this project was to use Intervention mapping (IM), a six-step framework, to cocreate a VR intervention with Black families to support COVID-19 vaccine uptake. For IM, step 1 (develop a logic model of the problem) and step 2 (identify program objectives and outcomes), our team used quantitative and qualitative methods to identify caregiver, adolescent, and young adult factors associated with intention and decisions to vaccinate against COVID-19. IM step 3 (program design) consisted of assembling planning board sessions with the Black community stakeholders including adolescents, young adults, and caregivers, along with primary care pediatric clinicians. A modified Delphi technique was used to build consensus for salient COVID-19 vaccine concerns and communication preferences. During IM step 4 (program production), the VR intervention (Virtual Immersive Communication Training on Recommending Immunization COVID-19) and accompanying simulation flowsheets were developed. An implementation plan was developed for IM step 5 (program implementation) and an evaluation plan for IM step 6 (evaluation). Through IM, an intervention centered on the needs and preferences of Black families was successfully developed. Lessons learned using IM included: (1) the importance of collaborating with patients and families on clinical topics where there is limited data, (2) the use of rigorous methods such as the modified Delphi approach to help build consensus, and (3) community relationships are critical to drive culturally informed technology-based interventions, especially when urgency is required.

Introduction

While the COVID-19 vaccine is effective at preventing severe disease¹ and hospitalization² in pediatric populations, only 7% of children, aged 6 months to 17 years were reported to be up-to-date with the COVID-19 vaccine as of January 8, 2025.³ Furthermore, during the 2021/2022 season, COVID-19 was the 8th leading cause of death in children and adolescents, outranking both influenza and pneumonia.⁴ Though COVID-19 vaccination is low across all sociodemographic subgroups, Black children have lower COVID-19 vaccination rates when compared to White and Hispanic children.⁵ This may in part be due to a lack of culturally informed interventions that support vaccination, as individuals of different racial and ethnic backgrounds may have reasons for hesitancy distinct from the general population due to historical exploitation and mistrust.⁶ Lower vaccination rates may contribute to the disproportionately higher COVID-related morbidity and mortality among Black children.⁷ Among racial minority groups, rapid development of the vaccine and government mistrust have surfaced as prevalent concerns leading to vaccine hesitancy.^8,9 Identification of culturally relevant communication strategies to overcome vaccine hesitancy and support the uptake of the COVID-19 vaccine is critical to improving pediatric health outcomes.^6,10,11

COVID vaccine recommendation

Strong and consistent vaccine recommendations from health care providers have been found to be a powerful tool to counter vaccine concerns.^12–19 Specifically, introducing a vaccine in a presumptive approach, using short statements indicating vaccination as the default option (e.g., “The patient is due for vaccines today”), has demonstrated effectiveness at increasing uptake of standard childhood vaccines as well as human papillomavirus (HPV) and influenza vaccines.^20–24 Based on this evidence, the Centers for Disease Control and Prevention and other professional organizations recommended the use of a presumptive approach by health care providers as a strategy to increase COVID-19 vaccine uptake.^25,26 However, the American Academy of Pediatrics urges caution in using the presumptive approach indiscriminately, as it limits opportunities to provide culturally relevant vaccine information that may be necessary to facilitate trust.²⁷ Given the surge in misinformation and polarized vaccine discourse, counseling on the COVID-19 vaccine may require nuanced behaviors distinct from historically identified counseling practices. For instance, motivational interviewing (MI)—a patient-centered communication style to build rapport and support collaborative decision-making²⁸—may play a role in supporting COVID-19 vaccine counseling as it has demonstrated effectiveness in addressing hesitancy around the HPV and influenza vaccines.^29,30

Intervention mapping

Intervention mapping (IM) is an established collaborative process incorporating input from key stakeholders to develop an intervention. IM uses a six-step approach to integrate theory, empirical findings, and information from the target population. The sequential six-step process includes identifying a logic model of the problem (step 1), establishing program outcomes and objectives (step 2), designing a program (step 3), producing the program (step 4), developing an implementation plan (step 5), and evaluating the program (step 6; Table 1).³¹ IM methods have been previously employed to cocreate interventions with community members and providers to increase vaccine uptake including within urban pediatric clinics and federally qualified health centers.^34,35 IM was also used to develop and adapt a storytelling intervention targeting HPV vaccine uptake among Hispanic adolescents.³⁶ Thus, IM has demonstrated the capacity to successfully engage stakeholders in intervention development related to vaccination. However, it has not been applied to virtual reality (VR) interventions targeting pediatric clinician communication, specifically among racial minoritized populations, to promote COVID-19 vaccination.

Table 1.

Intervention Mapping Steps, Methods, and Outcomes for COVID-19 Vaccine Virtual Reality Intervention³¹

Timeline	Intervention mapping steps and methods	Outcomes
Online survey: January 2021 Interviews: December 2021–February 2022	Step 1: Develop a logic model of the problem • Adolescents, young adults, and caregivers recruited to complete an online survey to assess knowledge, attitudes, and intentions to receive COVID-19 vaccine³² • Interviews conducted with caregiver-adolescent dyads to further understand acceptance of the COVID-19 vaccine³³	• Facilitators and barriers associated with COVID-19 vaccine intention • Logic model of COVID-19 vaccine hesitancy
Introduction session to WE-CRAB: October 2021	Step 2: Identify objectives and outcomes • Guided by the logic model of the problem (step 1), we used a brainstorming session with our primarily Black and female planning board, led by a facilitator, to develop curricular change objective and performance objectives	• Logic model of change to guide program development
Community planning board sessions 1–4: January 2022–February 2022 Clinician planning board sessions: February 2022–March 2022	Step 3: Design the program • Using the logic model of change (step 2), we used a modified Delphi technique and design sessions with the planning board to identify: (a) salient sources of COVID-19 vaccine hesitancy for Black individuals, (b) messaging from clinicians to address determinants with Black families	• Salient sources of hesitancy • Messaging for clinicians to address COVID-19 vaccine hesitancy
Community planning board session 5: March 2022	Step 4: Program production • Based on the design sessions, we developed the virtual reality simulation scenarios and flowsheets, training rubrics, and tangible educational materials. • We developed a prototype of the virtual reality curriculum to review with the planning board for feedback	• Tangible educational materials (e.g., sample language) for virtual reality training
	Step 5: Develop implementation plan • We identified potential program barriers to implementation using the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) planning questions worksheet.	• Implementation plan addressing potential barriers
Usability testing of VICTORI-COVID19: August 2022VICTORI-COVID-19 Efficacy pilot: September 2022–January 2023	Step 6: Evaluation • Using the program developed in steps 3 and 4 as well as the implementation plan developed in step 5, we implemented the VR simulations to train health care clinicians in COVID-19 vaccine counseling and assess COVID-19 vaccination rates over time in a clinical setting. • Also, we plan, in the future, to assess the impact of the intervention on Black families’ trust in their health care provider.	• Effectiveness data to assess the virtual reality intervention’s impact on COVID-19 vaccine rates

VICTORI, Virtual Immersive Communication Training on Recommending Immunization; WE-CRAB, West End Community Research Advisory Board.

Virtual reality intervention

VR is a computer-generated environment where a user interacts with a digital environment and digital characters in an authentic and realistic manner. VR interventions can be delivered via a 3D-mounted headset or screen-based.³⁷ Prior studies have demonstrated the capacity for brief VR interventions to enhance communication competencies, specifically MI skills in providers.^29,30,38 This is notable as provider training in MI is often lengthy (5–18 h) including workshop or lecture-based formats that may be insufficient to impart these advanced communication competencies.³⁹ Our team has previously developed VR-based interventions aiming to train providers in evidence-based communication.^29,30,40 These interventions entitled Virtual Immersive Communication Training on Recommending Immunizations (VICTORI) have demonstrated a positive impact on influenza and HPV vaccination rates.^29,30 Within VICTORI, a facilitator screen shares a virtual clinic space with a provider through a teleconferencing software. The virtual clinic space includes a graphical caregiver and child that respond in real time to verbal prompts from the provider. The facilitator drives the characters’ verbal and nonverbal responses based on a provider’s specific behavior to reinforce the use of evidence-based communication. Following participation in a specific scenario, a provider receives immediate feedback from the facilitator regarding their performance with an opportunity to repeat cases until demonstrating skill is mastered. Thus, VICTORI may overcome challenges of traditional training methods by allowing for deliberate practice⁴¹ of communication skills in a safe, immersive environment supporting behavior change.

Purpose

Though the VR intervention platform provides an ideal environment for practicing communication behaviors relevant to COVID-19 vaccine counseling, there is a lack of evidence-based communication strategies to support COVID-19 vaccine uptake among racially minoritized populations. Thus, we sought to adapt the VICTORI intervention to focus on COVID-19 vaccine counseling among Black pediatric populations using IM.^31,36,42,43 The specific objectives included identifying the most relevant sources of COVID-19 vaccine hesitancy, establishing consensus on preferred counseling strategies, and obtaining feedback on storyboards of VR scenarios among a planning board representative of the priority population. By using culturally responsive approaches, interventions may address the social and cultural needs of the patient, promoting positive patient-provider interactions when discussing the COVID-19 vaccine.⁴⁴

Methods

This study occurred at a Midwestern academic medical center and within an urban community setting. It was approved by the Cincinnati Children’s Hospital Medical Center Institutional Review Board (2022–0046). The academic medical center’s adolescent medicine primary care clinic was the target site for future implementation. This clinic completes approximately 5,000 primary care patient visits annually. Of the patients receiving care at the clinic, an estimated 72% of the patients are Black. Adaption of the intervention occurred in the winter of 2021 through the spring of 2022. At that time, an emergency use authorization (May 10, 2021) for the Pfizer-BioNtech COVID-19 vaccine had been expanded to include adolescents aged 12–15 years, followed by authorization for those ages 5–11 years old (October 29, 2021).⁴⁵ In addition, the Delta and Omicron variants of COVID-19 were circulating throughout the United States, with nearly 1 million daily infections reported in January 2022.⁴⁶ Thus, the project was timely to meet the needs of the community.

Planning board participants

To ensure the intervention would be patient-centered as well as appropriate and feasible to providers, we established two planning boards: (1) community and (2) clinician.^47,48 To establish the community planning board, we collaborated with the medical center’s community engagement consultant. The consultant served as the connector to a community research advisory group, West End Community Research Advisory Board (WE-CRAB). WE-CRAB consists of community stakeholders who engage, guide, and collaborate with researchers to support the health goals of the West End neighborhood, located in Cincinnati, Ohio. We had an introductory meeting with WE-CRAB to present our project’s goals, objectives, processes, and outputs. Next, we gauged WE-CRAB’s interest in collaborating. After confirming interest, we established the community planning board with those members of WE-CRAB as well as additional identified community members whose interests were most aligned with the project. The community and research team indicated a preference for virtual planning board meetings for ease of attendance and to align with social distancing recommendations at the time. Prior to participation in the planning board, community and clinician participants indicated consent or assent as appropriate based on the age of the member via an online database. Additionally, at the first community planning board sessions, the facilitators reviewed once again the study information. The participating community members were predominantly Black (n = 15; 94%), non-Hispanic (n = 16; 100%) and female (n = 15; 94%) and included adolescents (n = 6; age range 13–17), young adults (n = 4; age range 20–21), and caregivers (n = 6; age range 33–63).

To establish the clinician planning board, we invited clinicians working at a large urban pediatric primary care clinic via email. This clinic was specifically selected as it was located at the same Midwestern academic medical center as the target adolescent medicine clinic and also provided primary care for adolescent patients. Importantly, 75% of patients seen at this primary care clinic identify as Black or African American. The clinician planning board included a total of five clinicians (4 physicians and 1 nurse) with varying demographics, including Black (n = 2; 40%), non-Hispanic (n = 5; 100%), and males (n = 1; 20%). The nurse participant had 11–15 years of experience. The attending physician participants had a range of experience in their current role including 1–2 years (n = 1), 3–5 years (n = 1), 6–7 years (n = 1), and >15 years (n = 1). A summary of the IM steps is included in Table 1.

Intervention mapping step 1

Prior to the planning boards, we conducted mixed-methods studies assessing adolescents, young adults, and caregivers’ determinants associated with COVID-19 vaccine intentions to complete IM step 1 (develop logic model of the problem).^32,33 In January 2021, we conducted a cross-sectional survey of adolescent (n = 242; mean age [standard deviation (SD)] =14.8 [1.8] years) young adults (n = 333; mean age [SD] = 20.4 [2.0] years), and caregivers (n = 563; mean age [SD] = 35.5 [8.9] years) to identify variables associated with intention to receive a COVID-19 vaccine. Participants were recruited via email from primary care clinics and the medical center’s Office of Clinical and Translational Research. The study found that variables associated with intention to receive the COVID-19 vaccine included general vaccine hesitancy, perceptions about COVID-19 vaccine safety, and normative beliefs regarding the COVID-19 vaccine.³² Next, we explored factors influencing caregiver and adolescents’ COVID-19 vaccine decision-making during the Omicron surge in winter 2021–2022 via qualitative interviews within a large, urban pediatric primary care clinic. Participants identified the perceived risk of COVID-19, the opinions of other family members, and two-way communication with clinicians as factors influencing their COVID-19 vaccine decisions.³³ From these data, we developed a logic model of the problem (Fig. 1) encompassing sources of COVID-19 vaccine hesitancy, other factors impacting intentions to vaccinate, potential communication strategies to overcome hesitancy, and training strategies for clinicians.

FIG. 1.

Logic model of the problem that highlights items included within the modified Delphi.

Intervention mapping step 2

Our primary outcome, based on the population’s needs, was to improve COVID-19 vaccine uptake in the pediatric population. The secondary outcome was to increase clinicians’ skills in using patient-centered vaccine counseling when addressing COVID-19 vaccine hesitancy. Aligned with these outcomes and driven by the determinants identified in step 1, we created program objectives focused on understanding how clinicians should ideally: (1) provide a COVID-19 vaccine recommendation, (2) address COVID-19 vaccine concerns and hesitancy, and (3) build trust with patients and caregivers.

Intervention mapping step 3

To address the identified determinants and accomplish the program objectives, we selected VR as the educational strategy. The VR training facilitates deliberate practice, a personal and goal-oriented approach to skill development derived from Ericsson’s Theory on Expertise.⁴¹ This active learning process is characterized by engaging in the task or behavior (e.g., introducing the vaccine and addressing hesitancy), followed by immediate feedback regarding areas for improvement provided by a human facilitator, and then repeating the task, but changing behavior based on feedback.^29,30,38 Thus, a facilitator confirms a learner has mastered specific behavioral objectives prior to completing the training. Thus, VR may provide substantial benefits over standard training strategies (e.g., didactic learning, workshops, and direct clinical observation).³⁹

Delphi round 1

To adapt the VICTORI VR intervention to address COVID-19 vaccine hesitancy, we engaged in IM step 3—design a program—using a modified Delphi approach with the community and clinician planning boards, respectively.^49,50 This approach is frequently used in health profession education to inform training on complex topics.^51,52 We used a structured survey of factors identified in IM step 1 to inform the modified Delphi instrument including: (1) patients and caregivers’ potential vaccine concerns, (2) clinicians’ language to introduce the vaccine, and (3) MI skills for clinicians to address hesitancy. Planning board members were instructed to vote on keeping, removing, or modifying each of the items or state if they had no opinion.⁵⁰ For each item, planning board members were provided with an option to provide free-text responses to support their decision or suggest item revisions. Responses were analyzed by the authors (F.J.R., A.M., and B.L.R.) and consensus was set at 70% to consider intervention inclusion. For the items that did not reach consensus, we reviewed any comments and suggested revisions and either removed the item or revised to better reflect the community insights. These items were included in the round 2 survey.

Delphi round 2

We presented the results of round 1 to the community planning board members to support transparency and answer questions regarding item revisions. Next, planning board members completed the second survey. The analysis and revision process from round 1 was repeated and produced a final list of recommended intervention components.

Of note, the clinician planning board completed only a single survey round that was informed by the results of the community planning board survey results.

In-person consensus meeting

We presented the results of round 2 to the planning board members, addressing any questions regarding the kept and removed items. The subsequent planning board sessions included the members viewing video demonstrations of a clinician using the skills identified in the Delphi method to introduce and address COVID-19 vaccine hesitancy during a clinical encounter to inform final decisions regarding the training content. Planning board members also provided feedback on exemplar language to be included in the VR training.

Results

The results of the modified Delphi process are included in Table 2. There was a strong preference among planning board members for introducing the vaccine in a participatory approach, using open-ended questions to invite a caregiver to voice an opinion on vaccination (e.g., “Would you be interested in getting the COVID-19 vaccine today?”). This was in opposition to the previously described presumptive format that uses short statements indicating vaccination as the default option (e.g., “You are due for the COVID-19 vaccine today.”). The following COVID-19 vaccine concerns were identified via this consensus-building process as most important to Black families: (1) safety and side effects, (2) timeline of vaccine development, (3) long-term side effects, and (4) lack of efficacy due to breakthrough cases and need for boosters. Members identified the following MI and other relevant communication skills as the most important strategies to counter mistrust and validate the lived experience of Black families: (1) asking permission before providing education as a sign of respect, (2) providing information without medical jargon (i.e., being transparent using understandable language), (3) providing a recommendation for the vaccine using data (i.e., valuing the family’s ability to understand the rationale for the recommendation), and (4) openness to continue discussions if patient declines (i.e., respecting the family’s decision and process) (Table 3).

Table 2.

Modified Delphi Survey Results

	Community Delphi round 1 (n = 15)			Community Delphi round 2 (n = 14)			Clinician Delphi round 1 (n = 5)
	Week of January 10, 2022			Week of January 17, 2022			Week of January 24, 2022
Timeline of survey completion	Keep item	Modify item	Remove item	Keep item	Modify item	Remove item	Keep item	Modify item	Remove item
Item	n (%)			n (%)			n (%)
Sources of COVID-19 vaccine hesitancy
Concerns about multiple vaccines at one visit (flu and COVID-19)	9 (60)	3 (20)	3 (20)	8 (57)	3 (21)	3 (21)
Concerns regarding vaccine safety	12 (80)	2 (13)	1 (7)				5 (100)	0 (0)	0 (0)
Concerns regarding the short-term side effects from the vaccine (for example, body aches, chills, high temperature, headache)^a	11 (73)	4 (27)	0 (0)				5 (100)	0 (0)	0 (0)
Concerned about the unknowns regarding long-term side effects from the vaccine^a	13 (87)	2 (13)	0 (0)				4 (80)	1 (20)	0 (0)
Wanting to “wait and see” before getting the vaccine	9 (60)	4 (27)	2 (13)
Concerns about the vaccine causing COVID-19	8 (53)	3 (20)	4 (27)	9 (64)	1 (7)	4 (29)
Concerns about the vaccine causing death	11 (73)	3 (20)	1 (7)				3 (60)	0 (0)	2 (40)
Concerns that the vaccine was developed too fast^a	13 (87)	1 (7)	1 (7)				5 (100)	0 (0)	0 (0)
Concerns about the science behind the vaccine	10 (67)	3 (20)	2 (13)
Concerns about government’s financial incentives for people to get the vaccine	12 (80) ^b	0 (0)	3 (20)				3 (60)	0 (0)	2 (40)
Concerns that the vaccine does not work	11 (73) ^c	4 (27)	0 (0)
Concerns that the vaccine does not work at preventing COVID-19 (for example, people still get “break-through cases”)^a				12 (86)	2 (14)	0 (0)	5 (100)	0 (0)	0 (0)
Concerns that the vaccine does not work because we need a booster^a				10 (71)	3 (21)	1 (7)	4 (80)	0 (0)	1 (20)
Concerns about side effects from the vaccine because of other medical conditions	11 (73) ^b	3 (20)	1 (7)				3 (60)	0 (0)	2 (40)
Vaccine communication behaviors by clinicians
Doctor offers vaccine as an option by stating, “You/your child are eligible for the COVID-19 vaccine. Would you be interested in getting vaccinated today?”^a	11 (73)	2 (13)	2 (13)	10 (71) ^d			3 (60)^d
Doctor assumes you are ready to vaccinate by stating, “You/your child are/is due for the COVID-19 vaccine.”	6 (40)	3 (20)	6 (40)
Doctor brings up the topic of the vaccine by asking, “Now, have you/has your child gotten the COVID-19 vaccine yet?”	13 (87)	1 (7)	1 (7)	4 (29)^d			2 (40)^d
Other factors impacting intentions to vaccinate
Doctor provides information about you/your child’s risk if you have not received the vaccine by stating, “You/your child have/has asthma and are at a higher risk for severe illness and hospitalization if you are infected with COVID-19.”^e	11 (73) ^c	4 (27)	0 (0)				5 (100)	0 (0)	0 (0)
Doctor provides information about your child’s risk if they have not received the vaccine by stating “Because you/your child has asthma, you/your child is at a higher risk for severe illness/hospitalization if you/ s/he gets COVID-19. The vaccine can not only protect Alex from getting COVID-19 but also decrease the severity of illness and the possibility of hospitalization.”^e				12 (86)	2 (14)	0 (0)
Doctor provides strong recommendation for the COVID-19 vaccine by stating, “I strongly recommend the COVID-19 vaccine.”	12 (80)	2 (13)	1 (7)				5 (100)	0 (0)	0 (0)
Doctor provides strong recommendation for the COVID-19 vaccine from the medical community by saying, “We strongly recommend the COVID-19 vaccine.”	10 (67)	4 (27)	1 (7)
Motivational interviewing skills to address hesitancy
When you show concern or hesitancy about the COVID-19 vaccine, doctor asks questions about your concerns. For example, doctor says, “What specific concerns do you have about the vaccine?”^a	13 (87)	2 (13)	0 (0)				5 (100)	0 (0)	0 (0)
After you provide specific concerns (for example, long-term side effects) about the COVID-19 vaccine, the doctor shows they understand your concerns by stating, “I hear that you are worried about the risk of long-term side effects.”^a	13 (87)	2 (13)	0 (0)				4 (80)	0 (0)	1 (20)
After you provide specific concerns (for example, long-term side effects) about the COVID-19 vaccine, the doctor gives additional insights, “I hear that you are worried about the risk of long-term side effects and that can be scary. You just want to make the best choice for your/your child’s health.”^a	12 (80)	2 (13)	1 (7)				5 (100)	0 (0)	0 (0)
Doctor asks permission to share information to address concerns and hesitancy about the COVID-19 vaccine before providing explanations and data by stating, “Is it alright if I share what I know about the COVID-19 vaccine with you?”^a	11 (73)	4 (27)	0 (0)				5 (100)	0 (0)	0 (0)
Doctor emphasizes your choice as it pertains to what you choose for yourself and your health care by stating, “Your/Your child’s overall health is important to me and I strongly recommend this vaccine. I respect that ultimately it is your decision.”^a	13 (87)	2 (13)	0 (0)				4 (80)	0 (0)	1 (20)
Additional Communication Practices
Doctor answers your specific concerns.^a	14 (93)	1 (7)	0 (0)				5 (100)	0 (0)	0 (0)
Doctor addresses your specific concerns by providing a personal example by saying, “Everyone in my family got vaccinated, including my parents, siblings, spouse, and children.” For parents ““I got all my children, who are aged 11, 13, and 17 years old, vaccinated against COVID-19.”	5 (33)	6 (40)	4 (27)
Doctor asks if you have had any family members or close friends who have been impacted by COVID-19 to better understand your experience (e.g., being in the hospital, dying).^a	11 (73)	3 (20)	1 (7)				5 (100)	0 (0)	0 (0)
Doctor shows awareness of your concerns by stating, “I appreciate you talking about the vaccine with me. You have clearly given this a lot of thought.”^a	12 (80)	2 (13)	1 (7)				5 (100)	0 (0)	0 (0)
Doctor shows compassion and sympathy by stating, “I recognize that this is a hard decision.”	13 (87)	1 (7)	1 (7)				5 (100)	0 (0)	0 (0)
Doctor comes across as authentic and genuine during visit.^a	14 (93)	1 (7)	0 (0)				4 (80)	1 (20)	0 (0)
Doctor offers to answer additional questions or concerns once you leave the office/clinic.	14 (93)	0 (0)	1 (7)				5 (100)	0 (0)	0 (0)
Doctor does not use complicated medical language when explaining the vaccine.^a	12 (80)	2 (13)	1 (7)				4 (80)	1 (20)	0 (0)
Doctor explicitly acknowledges and validates racism within medicine by stating, “Racism is a powerful force in our country. It is not fair how people of color have been treated in medicine.”	6 (40)^c	4 (27)	5 (33)
Doctor explicitly acknowledges and validates racism within medicine and how getting the vaccine is a small act in fighting racism in medicine and states, “I want to make it my duty as a health care provider to work against racism within medicine.				4 (29)	3 (21)	7 (50)
Doctor explains the reason behind their recommendation for the vaccine by stating, “I am recommending the vaccine because this will help protect you/your child from having severe COVID-19 and needing to go to the hospital.”^a	13 (87)	2 (13)	0 (0)				4 (80)	0 (0)	1 (20)
Doctor explains how vaccinating is a way to help protect those around you/your child, such as family and friends.^e	10 (67)	4 (27)	1 (7)	11 (79)	2 (14)	1 (7)
Doctor explains how vaccinating is a way to end the pandemic and get back to “normal life”.	7 (47)	5 (33)	3 (20)
Doctor respectfully addresses your decision to not vaccinate by stating, “I understand and respect that you/your child not want to be vaccinated today. This is a very important topic to me and I want you to know that I care about your/your child’s overall health and would like to talk about the vaccine at the next visit.”	9 (60)^c	4 (27)	2 (13)
Doctor respectfully addresses your/ parent’s decision to not vaccinate by stating, “I’m here to answer your questions and concerns about vaccines if your decision [for your child] changes. At you/your child’s next visit, I hope we can reevaluate if you are open.”^a				12 (86)	0 (0)	2 (14)	4 (80)	1 (20)	0 (0)

Community Planning Board Round 1 was n = 15 due to a missing survey from n = 1; Community Planning Board Round 2 was n = 14 due to a missing survey from n = 2; Phrases you/your child were used interchangeably depending on if survey questions were posed to adolescent, young adult, or caregiver planning board members.

Bold denotes consensus >70% to keep the item by planning board members.

Denotes final item included in VICTORI-COVID19.

Denotes discrepancy between community and clinicians regarding the item in which discussion with planning boards determined if item should be included in the final intervention.

Denotes items that were reworded for subsequent rounds.

Denotes items in which planning board members were asked to choose a preferred COVID-19 vaccine introduction strategy without a response option to modify or remove.

Items were removed from the final intervention due to a lack of evidence supporting these statements at the time of intervention deployment.

Table 3.

Recommendation and Motivational Interviewing Skills Included in the VICTORI-COVID19 Curriculum

Skill	Definition
Recommendation behaviors
Introducing the COVID-19 vaccine using a participatory approach	Exploring a caregiver’s opinion on vaccination rather than presuming agreement to vaccinate (e.g., “How do you feel about getting your child the COVID-19 vaccine today?”)
Providing a recommendation for vaccination	Providing a recommendation without qualifying language regarding the strength of the recommendation (e.g., “I recommend the COVID-19 vaccine” rather than “I strongly recommend the COVID-19 vaccine.”)
Accepting a caregiver’s vaccine decision respectfully and leaving the door open for future vaccine discussions	Use of specific language when a caregiver declines COVID-19 vaccination that emphasizes their autonomy and maintains rapport to discuss vaccination at a later visit. (e.g., “Thank you for discussing the COVID-19 vaccine with me. It is certainly your choice, and I am happy to talk more about it in the future.”)
Motivational interviewing (MI) Skills
Using open-ended questions to explore sources of hesitancy	Open-ended questions allow for a wide range of possible answers to allow for targeting counseling (e.g., “Tell me more about what makes you want to pass on getting your child the COVID-19 vaccine today.”)
Reflecting on patient statement	Conveying understanding of something that a caregiver or patient has said (e.g., “You are worried that the COVID-19 vaccine may have bad side effects.”)
Asking permission	Asking permission to provide information (e.g., “Is it okay if I tell you more about the side effects that can occur with the COVID-19 vaccine?”)
Use affirmative statements	Emphasize a family’s strengths and skills (e.g., “I appreciate how dedicated you are to keeping your child healthy.”)

VICTORI, Virtual Immersive Communication Training on Recommending Immunization.

Intervention mapping step 4

Outputs from this work included the final development of VICTORI-COVID-19 through IM step 4 (program production). It was produced using the Unity gaming platform with C# scripting.⁴⁰ The speech for the graphical characters was created using Revoicer.⁵³ VICTORI-COVID-19 includes four unique VR simulations to allow deliberate practice addressing the community-identified salient COVID-19 vaccine concerns and practicing the preferred MI communication skills. All scenarios took place during routine well-child care for an adolescent patient. Scenario 1 focuses on introducing the vaccine with no specific sources of hesitancy reported. Scenario 2 addresses safety and short-term side effect concerns. Scenario 3 addresses concerns related to rapid development of the vaccine and concerns for long-term side effects (i.e., fertility concerns). Scenario 4 focuses on effectiveness concerns and the need for boosters. The flowsheets for each scenario are structured in such a way to positively and negatively reinforce behaviors based on the user input through verbal and nonverbal language by the graphical character. (Supplementary Materials 1) For example, if a user presumes that a family is agreeable to the COVID-19 vaccine without first assessing their interest in vaccination (as recommended by the planning board members), the graphical character will look away from the user and say, “No way. Alex is not getting the COVID-19 vaccine.” The intervention is designed for a clinician to complete independently and is human-facilitated (Fig. 2) so that a clinician can receive coaching following each scenario. This also allows for corrective feedback on less preferred behaviors, such as overconfidence during counseling. Brief didactic information is included between scenarios to highlight planning board communication preferences. The program objectives (IM step 2) were scaffolded to increase in complexity over the four scenarios and introduced by the facilitator, who provides the exemplar language developed and reviewed by the planning board. The intervention takes approximately 40 min to complete. It can be delivered via a 3D-mounted headset (compatible with all oculus and meta headsets supported by the Meta Quest application⁵⁴) or via a screen using a teleconferencing platform (Supplementary video). The program can be run on any VR-capable computer with preferred specifications of a 2070 or better GPU and 16 Mb of RAM.

FIG. 2.

The human facilitator drives the actions of the graphical characters in real-time, including their verbal and non-verbal language in response to the user’s counseling. The facilitator dashboard provides access to prerecorded verbal language and body animations.

Intervention mapping steps 5–6

Next, the team utilized a mixed-methods approach to complete IM steps 5 and 6³¹ to evaluate the effectiveness and implementation of the intervention via a pilot study at the medical center’s adolescent medicine primary care clinic. Analysis and dissemination of this pilot study are ongoing. Study progress and results have since been reported back in-person to the community planning board through two follow-up presentations.

Discussion

Through IM, we were able to engage and codesign a VR intervention focusing on COVID-19 vaccine hesitancy with Black pediatric families and clinicians. We used surveys, focus groups, videos, and storyboarding to establish consensus among planning board members on a complex topic that had been understudied at the time. Our approach aligned with the Vancouver Statement 2015 and the Accreditation Council for Continuing Medical Education recommendation that training incorporate patients’ autonomous and authentic voices to ensure that care delivery is patient-centered.^55,56 This approach serves as a model for others seeking to develop customized, culturally-relevant extended reality interventions.

Developing providers’ communication skills in MI to address sources of vaccine hesitancy while also respecting and maintaining the patients and families’ autonomy requires deliberate practice of skills in a safe environment. VR provides a method to allow for such deliberate practice and accelerate competencies related to MI, which has proven challenging to train providers in through traditional educational strategies such as lectures and workshops.^29,30,39 Indeed, such workshops often demonstrate immediate attitude change, though do not typically produce sustained behavior change related to MI.^57,58 Though screen-based VR simulation may support decreased spatial presence and immersion compared to use of a 3D-mounted headset, it still allows providers to suspend their disbelief easily in a familiar clinical environment to support the practice of specific communication competencies.⁵⁹ The IM process allowed stakeholders to provide input on VR storyboarding that included providing feedback on the scenario flowsheets, the verbal and nonverbal language of the graphical characters, and the presenting physical characteristics of the graphical environment and characters. As such, beyond allowing for deliberate practice of skills, the use of VR allows investigators to obtain input on nuanced details of the training experience that will be consistently applied across providers. Thus, VR is a particularly advantageous educational strategy as it enables repeatable practice and standardization across learners allowing for assessment of objective performance metrics.

Barriers to scalability of the VR intervention across centers may include hardware costs, internet bandwidth, and the need for technical support. As such, screen-based delivery of the intervention to remote locations via teleconferencing software may support equitable access to such high-quality education, aiming to enhance COVID-19 vaccine rates among patients. Train the trainer models, in which initial adopters undergo facilitator training to educate local staff, are a potential approach to support widespread dissemination. Identification of how to best incorporate VR training into clinical workflows or protected educational time will be necessary to support its effective implementation and sustainability over time. Artificial intelligence (AI) applications may support intervention dissemination by removing the need for human facilitation. Thus, recording and transcribing user interactions with graphical characters during scenarios may support the development of such AI models. Such data should be stored on a secure database with transcription occurring in a timely manner to allow deletion of recordings that contain identifying information of the user (e.g., face or voice). However, future work is needed to understand how automating such VR training may impact the intervention’s acceptability among learners as well as its effectiveness.

This work had several limitations. It was a single-site design taking place in an urban setting that limits the generalizability of the findings. It is also important to note that community planning board members were mostly females. Next steps may include recruiting male caregivers and adolescents to further explore their insights and testing the intervention in various regions to capture more diverse perspectives. To establish the effectiveness of the intervention, a future multisite trial should be conducted in diverse geographical regions and obtain feedback from local caregivers and clinicians to ensure aligned perspectives with the scenario content or identify areas for adaptations. Outcomes of interest may include assessing vaccination rates as well as clinicians’ fidelity to the behavioral skills included in the VR intervention through simulation, direct observation, or recorded patient encounters that are coded using instruments such as the Motivational Interviewing Treatment Integrity tool with established validity evidence.^60,61 It will also be important to obtain survey information from parents regarding their perspectives on clinicians’ COVID-19 counseling.

Conclusions

Using IM resulted in an intervention centered on the needs and preferences of Black families. This work continues to highlight the importance of establishing and maintaining community partnerships to meet the needs and preferences of community members served in clinical practices. Though time and resource-intensive, such approaches are critical when cocreating interventions with communities to ensure patients and families’ autonomous and authentic voices are centered in care delivery.

Footnotes

Acknowledgment

The authors thank the community experts and leaders from the West End Community Research Advisory Board for their collaboration on this project.

Authors’ Contributions

All authors contributed to the article. B.L.R. and F.J.R. conceptualized, designed the study, implemented the research protocol, collected data, drafted the initial article, revised the article, and supervised all aspects of the study. A.M. conceptualized and designed the study, provided administrative support, coordinated and supervised data collection, drafted the initial article, and reviewed and revised the article. J.W. and L.E.C. provided edits to the draft. All authors approved the final article.

Author Disclosure Statement

The authors have no conflicts of interest relevant to this article to disclose.

Funding Information

This work was supported through the Cincinnati Children’s Hospital Medical Center Place Outcomes Award. Cincinnati Children’s Hospital Medical Center played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or decision to submit the article for publication.

Supplementary Material

Abbreviations Used

References

Klein

, Demarco

, Fleming-Dutra

, et al. Effectiveness of BNT162b2 COVID-19 vaccination in children and adolescents. Pediatrics, 2023; 151(5):e2022060894; doi: 10.1542/peds.2022-060894

Marks

, Whitaker

, Anglin

, et al.; COVID-NET Surveillance Team. Hospitalizations of children and adolescents with laboratory-confirmed COVID-19—COVID-NET, 14 States, July 2021-January 2022. MMWR Morb Mortal Wkly Rep, 2022; 71(7):271–278; doi: 10.15585/mmwr.mm7107e4

Prevention CfDCa. COVID-19 Vaccination Coverage and Intent for Vaccination, Children 6 months through 17 years, United States. Available from: https://www.cdc.gov/covidvaxview/weekly-dashboard/child-coverage-vaccination.html?CDC_AAref_Val=https://www.cdc.gov/vaccines/imz-managers/coverage/covidvaxview/interactive/children-coverage-vaccination.html [Last accessed: February 4, 2025].

Flaxman

, Whittaker

, Semenova

, et al. Assessment of COVID-19 as the underlying cause of death among children and young people aged 0 to 19 years in the US. JAMA Netw Open, 2023; 6(1):e2253590; doi: 10.1001/jamanetworkopen.2022.53590

Valier

, Elam-Evans

, Mu

, et al. Racial and ethnic differences in COVID-19 vaccination coverage among children and adolescents aged 5–17 years and parental intent to vaccinate their children - national immunization survey-child COVID Module, United States, December 2020-September 2022. MMWR Morb Mortal Wkly Rep, 2023; 72(1):1–8; doi: 10.15585/mmwr.mm7201a1

Kusma

, Walker-Harding

, Svetaz

, et al. A structural lens approach to vaccine hesitancy and identity. Pediatr Clin North Am, 2023; 70(2):271–282; doi: 10.1016/j.pcl.2022.11.005

Fernandes

, Oliveira

, Guerguis

, et al.; Tri-State Pediatric COVID-19 Research Consortium. Severe acute respiratory syndrome coronavirus 2 clinical syndromes and predictors of disease severity in hospitalized children and youth. J Pediatr, 2021; 230:23–31.e10; doi: 10.1016/j.jpeds.2020.11.016

Nguyen

, Kahn

, Hoehner

, et al. COVID-19 vaccination intent, perceptions, and reasons for not vaccinating among groups prioritized for early vaccination, United States, September 2020. 2021. Available from: https://www.cdc.gov/vaccines/imz-managers/coverage/adultvaxview/pubs-resources/COVID-online-report2020.html

Nguyen

, Srivastav

, Razzaghi

, et al. COVID-19 vaccination intent, perceptions, and reasons for not vaccinating among groups prioritized for early vaccination—United States, September and December 2020. MMWR Morb Mortal Wkly Rep, 2021; 70(6):217–222.

10.

Kricorian

, Turner

. COVID-19 Vaccine Acceptance and Beliefs among Black and Hispanic Americans. PLoS One, 2021; 16(8):e0256122; doi: 10.1371/journal.pone.0256122

11.

Streuli

, Ibrahim

, Mohamed

, et al. Development of a culturally and linguistically sensitive virtual reality educational platform to improve vaccine acceptance within a refugee population: The SHIFA community engagement-public health innovation programme. BMJ Open, 2021; 11(9):e051184; doi: 10.1136/bmjopen-2021-051184

12.

Kimmel

. Adolescent vaccination: Physician challenges and solutions. Journal of Adolescent Health, 2010; 46(2):S16–S23.

13.

Rosenthal

, Weiss

, Zimet

, et al. Predictors of HPV vaccine uptake among women aged 19–26: Importance of a physician’s recommendation. Vaccine, 2011; 29(5):890–895; doi: 10.1016/j.vaccine.2009.12.063

14.

Perkins

, Pierre-Joseph

, Marquez

, et al. Why do low-income minority parents choose human papillomavirus vaccination for their daughters? J Pediatr, 2010; 157(4):617–622; doi: 10.1016/j.jpeds.2010.04.013

15.

Dorell

, Yankey

, Strasser

. Parent-reported reasons for nonreceipt of recommended adolescent vaccinations, national immunization survey: Teen, 2009. Clin Pediatr (Phila), 2011; 50(12):1116–1124; doi: 10.1177/0009922811415104

16.

Pandolfi

, Marino

, Carloni

, et al. The effect of physician’s recommendation on seasonal influenza immunization in children with chronic diseases. BMC Public Health, 2012; 12:984; doi: 10.1186/1471-2458-12-984

17.

Wiley

, Leask

. Respiratory vaccine uptake during pregnancy. Lancet Respir Med, 2013; 1(1):9–11; doi: 10.1016/s2213-2600(13)70024-9

18.

Fisher

, Nguyen

, Fouayzi

, et al. Impact of a physician recommendation on COVID-19 vaccination intent among vaccine hesitant individuals. Patient Educ Couns, 2023; 106:107–112; doi: 10.1016/j.pec.2022.09.013

19.

Ryan

, Goulding

, Borg

, et al. Development and beta-testing of the confidence intervention to increase pediatric COVID-19 vaccination. J Pediatr Health Care, 2023; 37(3):244–252; doi: 10.1016/j.pedhc.2022.11.002

20.

Hofstetter

, Robinson

, Lepere

, et al. Clinician-parent discussions about influenza vaccination of children and their association with vaccine acceptance. Vaccine, 2017; 35(20):2709–2715; doi: 10.1016/j.vaccine.2017.03.077

21.

Opel

, Heritage

, Taylor

, et al. The architecture of provider-parent vaccine discussions at health supervision visits. Pediatrics, 2013; 132(6):1037–1046; doi: 10.1542/peds.2013-2037

22.

Opel

, Mangione-Smith

, Robinson

, et al. The influence of provider communication behaviors on parental vaccine acceptance and visit experience. Am J Public Health, 2015; 105(10):1998–2004; doi: 10.2105/ajph.2014.302425

23.

Sturm

, Donahue

, Kasting

, et al. Pediatrician-parent conversations about human papillomavirus vaccination: An analysis of audio recordings. J Adolesc Health, 2017; 61(2):246–251; doi: 10.1016/j.jadohealth.2017.02.006

24.

Brewer

, Hall

, Malo

, et al. Announcements versus conversations to improve HPV vaccination coverage: A randomized trial. Pediatrics, 2017; 139(1):e20161764; doi: 10.1542/peds.2016-1764

25.

Prevention CfDCa. Available from: https://www.cdc.gov/vaccines/covid-19/downloads/vaccination-strategies.pdf [Last accessed: December 1, 2024].

26.

Stevens

. The PANDEMIC framework for encouraging COVID-19 vaccinations. Am J Manag Care, 2021; 27(5):e137–e140; doi: 10.37765/ajmc.2021.88605

27.

O’Leary

, Opel

, Cataldi

, et al.; COMMITTEE ON BIOETHICS. Strategies for improving vaccine communication and uptake. Pediatrics, 2024; 153(3):e2023065483; doi: 10.1542/peds.2023-065483

28.

Rollnick

, Miller

, Butler

, et al. Motivational interviewing in health care: helping patients change behavior. Taylor & Francis; 2008.

29.

Real

, Ollberding

, Meisman

, et al. Impact of a virtual reality curriculum on human papillomavirus vaccination: A pilot trial. Am J Prev Med, 2022; 63(5):865–873; doi: 10.1016/j.amepre.2022.05.003

30.

Real

, DeBlasio

, Beck

, et al. A virtual reality curriculum for pediatric residents decreases rates of influenza vaccine refusal. Acad Pediatr, 2017; 17(4):431–435; doi: 10.1016/j.acap.2017.01.010

31.

Fernandez

, Ruiter

RAC

, Markham

, et al. Intervention mapping: Theory- and evidence-based health promotion program planning: Perspective and examples. Front Public Health, 2019; 7:209; doi: 10.3389/fpubh.2019.00209

32.

Rosen

, Meisman

, Sun

, et al. Factors associated with racially and ethnically diverse sample of adolescents, young adults, and parents’ intention to receive a COVID-19 vaccine. Am J Health Promot, 2024; 38(5):672–682; doi: 10.1177/08901171241233397

33.

Rosen

, Lee

, Chandler

, et al. Factors influencing adolescent COVID-19 vaccination during the omicron surge. Health Promot Pract, 2025:15248399241311583; doi: 10.1177/15248399241311583

34.

Crawford

, Shegog

, Savas

, et al. Using intervention mapping to develop an efficacious multicomponent systems-based intervention to increase Human Papillomavirus (HPV) vaccination in a large urban pediatric clinic network. J Appl Res Child, 2019; 10(2).

35.

Austin

, Rodriguez

, Savas

, et al. Using intervention mapping to develop a provider intervention to increase HPV vaccination in a federally qualified health center. Front Public Health, 2020; 8:530596; doi: 10.3389/fpubh.2020.530596

36.

Rodriguez

, Roncancio

, Savas

, et al. Using intervention mapping to develop and adapt two educational interventions for parents to increase HPV vaccination among Hispanic adolescents. Front Public Health, 2018; 6:164; doi: 10.3389/fpubh.2018.00164

37.

Bracq

, Michinov

, Jannin

. Virtual reality simulation in nontechnical skills training for healthcare professionals: A systematic review. Simul Healthc, 2019; 14(3):188–194; doi: 10.1097/sih.0000000000000347

38.

Real

, Whitehead

, Ollberding

, et al. A virtual reality curriculum to enhance residents’ behavioral health anticipatory guidance skills: A pilot trial. Acad Pediatr, 2023; 23(1):185–192; doi: 10.1016/j.acap.2022.07.011

39.

Budhwani

, Naar

. Training providers in motivational interviewing to promote behavior change. Pediatr Clin North Am, 2022; 69(4):779–794; doi: 10.1016/j.pcl.2022.04.008

40.

Unity. Available from: https://unity.com/products [Last accessed: January 24, 2023].

41.

Ericsson

, Krampe

, Tesch-Römer

. The role of deliberate practice in the acquisition of expert performance. Psychol Rev, 1993; 100(3):363–406; doi: 10.1037/0033-295x.100.3.363

42.

Kok

, Gottlieb

, Peters

, et al. A taxonomy of behaviour change methods: An Intervention Mapping approach. Health Psychol Rev, 2016; 10(3):297–312; doi: 10.1080/17437199.2015.1077155

43.

Fernandez

, Ten Hoor

, van Lieshout

, et al. Implementation mapping: Using intervention mapping to develop implementation strategies. Front Public Health, 2019; 7:158; doi: 10.3389/fpubh.2019.00158

44.

Betancourt

, Green

, Carrillo

. Cultural competence in health care: Emerging frameworks and practical approaches. 2002. Available from: https://www.commonwealthfund.org/sites/default/files/documents/__media_files_publications_fund_report_2002_oct_cultural_competence_in_health_care__emerging_frameworks_and_practical_approaches_betancourt_culturalcompetence_576_pdf.pdf [Last accessed: October 2, 2023].

45.

Services USDoHah. COVID-19 Vaccines Updated September 13, 2023. Available from: https://www.hhs.gov/coronavirus/covid-19-vaccines/index.html [Last accessed: August 15, 2023].

46.

Centers for Disease Control and Prevention. CDC Museum COVID-19 Timeline. 2023. Available from: https://www.cdc.gov/museum/timeline/covid19.html [Last accessed: August 15, 2023].

47.

Quality AfHRa. Six domains of healthcare quality. 2025. Available from: https://www.ahrq.gov/talkingquality/measures/six-domains.html

48.

Proctor

, Bunger

, Lengnick-Hall

, et al. Ten years of implementation outcomes research: A scoping review. Implement Sci, 2023; 18(1):31; doi: 10.1186/s13012-023-01286-z

49.

Okoli

, Pawlowski

. The Delphi method as a research tool: An example, design considerations and applications. Inf Manag, 2004; 42(1):15–29.

50.

Woodcock

, Adeleke

, Goeschel

, et al. A modified Delphi study to identify the features of high quality measurement plans for healthcare improvement projects. BMC Med Res Methodol, 2020; 20(1):8; doi: 10.1186/s12874-019-0886-6

51.

Boulkedid

, Abdoul

, Loustau

, et al. Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS One, 2011; 6(6):e20476; doi: 10.1371/journal.pone.0020476

52.

Humphrey-Murto

, Varpio

, Gonsalves

, et al. Using consensus group methods such as Delphi and Nominal Group in medical education research. Med Teach, 2017; 39(1):14–19; doi: 10.1080/0142159X.2017.1245856

53.

Revoicer. 2022. Available from: https://revoicer.com/

54.

Meta Quest App. 2025. Available from: https://www.meta.com/help/quest/1517439565442928/?srsltid=AfmBOooQEGavtpLHu5mqzxDwmspQu1Otg5QSmMRYU8q9gTp8Jyf8ztKC

55.

Dunn

. Culture competence and the primary care provider. J Pediatr Health Care, 2002; 16(3):105–111.

56.

Towle

, Farrell

, Gaines

, et al. The patient’s voice in health and social care professional education: The Vancouver statement. International Journal of Health Governance, 2016.

57.

Söderlund

, Madson

, Rubak

, et al. A systematic review of motivational interviewing training for general health care practitioners. Patient Educ Couns, 2011; 84(1):16–26; doi: 10.1016/j.pec.2010.06.025

58.

McAfee

, Schumacher

, Madson

, et al. The status of SBIRT training in health professions education: A cross-discipline review and evaluation of SBIRT curricula and educational research. Acad Med, 2022; 97(8):1236–1246; doi: 10.1097/acm.0000000000004674

59.

Herbst

, Rybak

, Meisman

, et al. A virtual reality resident training curriculum on behavioral health anticipatory guidance: Development and usability study. JMIR Pediatr Parent, 2021; 4(2):e29518; doi: 10.2196/29518

60.

Moyers

, Rowell

, Manuel

, et al. The motivational interviewing treatment integrity code (MITI 4): rationale, preliminary reliability and validity. J Subst Abuse Treat, 2016; 65:36–42; doi: 10.1016/j.jsat.2016.01.001

61.

Real

, DeBlasio

, Ollberding

, et al. Virtual reality informs clinical observation tool. Clin Teach, 2023; 20(3):e13575; doi: 10.1111/tct.13575

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

18.47 MB

0.33 MB

18.47 MB