Mixed methods implementation evaluation of virtual interview training for transition-age autistic youth in pre-employment transition services

Abstract

BACKGROUND:

Autistic transition-age youth are employed at rates far lower than their non-disabled peers as well as youth with other disabilities. Meanwhile, very few studies have evaluated the implementation of job interviewing practices within pre-employment transition services.

OBJECTIVE:

We conducted an initial implementation evaluation as part of a Hybrid Type 1 randomized controlled effectiveness-implementation trial where we trained teachers to deliver Virtual Interview Training for Transition-Age Youth (VIT-TAY) within five pre-employment transition services programs.

METHOD:

We used mixed methods to evaluate leader (n = 5), teacher (n = 15) and autistic transition age youth (n = 48) perceptions of VIT-TAY. We used descriptive statistics and thematic network analysis to evaluate survey data. Mixed methods integration was then performed to make comparisons between quantitative and qualitative results.

RESULTS:

Quantitative survey data revealed that leaders and teachers found VIT-TAY to be highly acceptable and appropriate for pre-employment transition services; findings which were confirmed via thematic network analysis of qualitative interview data. Autistic students reported via quantitative surveys that VIT-TAY was acceptable and usable, which was confirmed via thematic network analysis of open-ended survey data.

CONCLUSION:

This initial implementation evaluation can be used to inform a larger scale implementation evaluation of VIT-TAY in schools.

1 Introduction

Autistic transition-age youth are employed at rates far lower than their non-disabled peers as well as youth with other disabilities (Bureau of Labor Statistics, 2020; Roux et al., 2015). Research has suggested that job interviews are a primary barrier to employment (Macan, 2009; Posthuma et al., 2002; Wilk & Cappelli, 2003). Due to the social nature of job interviews, the challenges with navigating a job interview are exacerbated for autistic individuals (Hendricks, 2010). To help autistic youth and their peers with other disabilities prepare for employment, the Workforce Innovation and Opportunity Act (WIOA, 2014) mandates the delivery of workplace readiness training as one of five Pre-Employment Transition Services (Pre-ETS). Notably, Pre-ETS are commonly delivered within special education settings by special education teachers or vocational rehabilitation counselors contracted by state-level divisions of vocational rehabilitation. Similarly, the Individuals with Disabilities Education Act (IDEA, 2004) mandates transition services (e.g., employment readiness) for students receiving special education services.

Although a recent study suggests that a large proportion of employed autistic youth engaged in Pre-ETS interviewed prior to getting hired (Smith, Sherwood, Blajeski et al., 2021), a paucity of research has evaluated the implementation of job interviewing preparation practices within Pre-ETS. For example, Virtual Interactive Training Agents (ViTA) is one intervention that has demonstrated initial efficacy at improving job interview self-efficacy in a non-randomized, non-controlled lab-based study (Burke et al., 2018) and a non-randomized, non-controlled internet-based study (Burke et al., 2020) among adults with autism and other developmental disabilities. Meanwhile, other technology-based job interview interventions have also been tested in small samples of autistic youth or young adults (e.g., Strickland et al., 2013; Smith et al., 2014; Kumazaki et al., 2017; Kumazaki et al., 2019; Genova et al., 2021). Despite demonstrated efficacy among autistic youth and adults, research has not yet evaluated the effectiveness or implementation processes to deliver these interventions at scale.

Understanding factors related to implementation of evidence-based interventions from multiple stakeholder perspectives (e.g., administrative leaders, teachers, students) is critical to sustained delivery (Proctor et al., 2011). Evaluating implementation during an effectiveness trial can shorten the amount of time needed for translation (Landes et al., 2020). In particular, schools provide a critical setting for evidence-based interventions due to the likelihood that most children receive needed services within schools (Domitrovich et al., 2008). Gersten et al. (2005) present intervention implementation as one of the quality indicators for experimental research within special education. However, to complicate this issue, schools are particularly complex, under-funded organizations, which makes intervention adoption, implementation, and sustainability difficult (Locke et al., 2016).

Additionally, although the Pre-ETS mandate is situated within the WIOA (2014) and is funded through state divisions of vocational rehabilitation, there is a need to evaluate the implementation of these interventions within educational settings due to the collaborative nature of special education transition services and Pre-ETS (Carter et al., 2021; Taylor et al., 2019). While such implementation evaluations in schools are gaining traction (Lyon & Bruns, 2019), little funding is dedicated to services for transition-age autistic youth (IACC, 2019). It is even less common for implementation evaluations to be embedded within a randomized effectiveness trial as demonstrating effectiveness often takes priority and implementation outcomes such as cost, fidelity, feasibility, sustainability, acceptability, and usability are lower priority.

Recently, we partnered with the autism community to adapt the efficacious Virtual Reality Job Interview Training (VR-JIT— originally designed for adults with psychiatric disabilities (Smith et al., 2015)), to meet the needs of transition-age autistic youth (Smith et al., 2020). Specifically, we recruited n = 24 autistic transition-age youth and n = 21 adult stakeholders from the autism community (e.g., parents, teachers, autistic adults) who reviewed VR-JIT and gave feedback to adapt and tailor the tool for the autism community. Suggested recommendations were then reviewed by a community advisory board of stakeholders from the autism community (e.g., autistic youth, parents, clinicians, service providers, service administrators, former members of state board of education).

The adapted version, now called Virtual Interview Training for Transition-Age Youth (VIT-TAY), is an interactive, computerized job interview simulator developed and commercially licensed by SIMmersion LLC (www.simmersion.com). Virtual interviews are led by two virtual hiring managers named “Rita Muniz” and “Travis Bishop” and supported by a virtual job coach named “Kendra” to facilitate the repeated practice of job interviews via speech recognition across three levels of difficulty (i.e., easy, medium, and hard). Notably, Rita and Travis’s personalities are selected at random and their moods (e.g., friendly, professional, rigid) can change based on responses to the interview questions. Based on the job interview literature and feedback from the autism community (Huffcutt, 2011; Smith et al., 2020), VIT-TAY highlights ten interview skills within an e-learning curriculum designed to convey positive attributes of the applicant (e.g., being a hard worker, being easy to work with) and shares job interview preparation tips. Prior to interviewing, trainees complete a job application for one of fourteen positions, which informs the questions asked during the virtual interview. VIT-TAY was designed to facilitate sustainable changes in interview skill via hierarchical learning and scaffolding (Bol & Garner, 2011; Whyte et al., 2015) by implementing four levels of feedback that include: 1) real-time nonverbal cues from “Kendra,” 2) transcript-level feedback on responses to interview questions, 3) a score of 0 to 100, and 4) the score is anchored to a summary performance assessment of the ten interview skills.

Subsequently, we conducted an intent-to-treat randomized controlled trial (RCT) using a Hybrid Type 1 (HT1) effectiveness-implementation design to evaluate VIT-TAY among autistic youth receiving special education transition services and Pre-ETS. We found VIT-TAY was initially effective at enhancing job interview skills, reducing interview anxiety, and increasing access to competitive employment (Smith, Sherwood, Ross et al., 2021). Given that VIT-TAY’s effectiveness from the HT1 trial was reported (Smith, Sherwood, Ross et al., 2021), we turn our attention to a multi-level, mixed methods evaluation of VIT-TAY implementation. A more detailed description of the VIT-TAY intervention and its effectiveness can be found in Smith, Sherwood, Ross et al. (2021).

The objective of the present study was to evaluate multilevel barriers and facilitators associated with salient implementation outcomes (Proctor et al., 2011) for VIT-TAY in five schools using staff (leaders and teachers) and student reports. As this is the first study of VIT-TAY in schools for autistic youth, we focus on outcome constructs that Proctor et al. (2011) identified as salient at “early” stages of implementation, along with usability— a factor that is germane to technology-based interventions for autistic youth (Mazon et al. 2019). We used mixed methods to evaluate leader and teacher perceptions of expected implementation feasibility, training acceptability, pre-implementation VIT-TAY appropriateness and acceptability, implementation context, and post-implementation acceptability and sustainability. Concerning autistic transition age youths’ perceptions, we used mixed methods to evaluate acceptability and usability of VIT-TAY. We hypothesized that school leaders and teachers would find VIT-TAY implementation feasible, VIT-TAY orientation to be acceptable, and VIT-TAY to be appropriate for Pre-ETS. We also hypothesized that leaders and teachers would perceive VIT-TAY to be acceptable and sustainable, and that autistic transition age youth would perceive VIT-TAY to be acceptable and usable.

2 Methods

Using a convergent mixed methods design, our implementation evaluation included simultaneous collection of both quantitative and qualitative data followed by the integration of data during interpretation (Creswell & Plano Clark, 2018). Mixed methods are a well-documented approach for studying implementation in schools (Aarons et al., 2012; Locke et al., 2016). This convergent mixed methods implementation evaluation was particularly well-suited to address our questions as the majority of the implementation outcomes were measured simultaneously both quantitatively and qualitatively post-implementation and were equally weighted in terms of meaningfulness. See Supplementary Figure 1 for a diagram of our implementation evaluation study design. The University of Michigan Institutional Review Board approved the study protocol (HUM00129575 on June 2, 2017). All participants provided informed consent or assent in accord with the ethical standards established by the Declaration of Helsinki.

2.1 Recruitment

We recruited five schools (n = 5) from Michigan and Ohio (three public, one private, and one charter) to participate in the RCT that evaluated VIT-TAY among n = 71 autistic transition-age youth (now referenced as students). Administrative leaders (n = 5; e.g., director of transition services, executive director, site lead) and teachers (n = 15) were recruited from the schools. Notably, three leaders, one leader also serving as a teacher, and 15 teachers were trained to deliver VIT-TAY. Given that randomization did not occur at the classroom level, school leaders requested that we train more teachers than were necessary to account for potential teacher absences and scheduling conflicts (i.e., so teachers could cover for one another if they were unavailable to implement VIT-TAY). Thus, only five teachers and one leader (also serving as a teacher) actively implemented VIT-TAY, while the remaining 10 teachers and 4 leaders did not implement VIT-TAY.

Students were randomly assigned to either Pre-ETS+VIT-TAY (n = 48) or Pre-ETS only (n = 23) at a ratio of 2:1. Student inclusion and exclusion criteria were reported in Smith, Sherwood, Ross et al. (2021).

2.2 Sample characteristics

The average age of participating leaders and teachers was 49.75 years (SD = 10.77; n = 8); 88.9%(n = 8) were vocational teachers. The teachers were primarily female (88.9%; n = 8) and White (66.7%; n = 6), with 11.1%(n = 1) identifying as Latinx. They had master’s degrees (88.9%; n = 8), and bachelor’s degrees (11.1%; n = 1). One leader did not complete the background information survey and one teacher did not report their age or race. Table 1 displays the demographics for nine of the 10 leaders and teachers who supervised or directly implemented VIT-TAY. For more details on the RCT recruitment methods, see Smith, Sherwood, Ross et al. (2021).

Table 1

Leader and teacher demographics (n = 9)

	Mean (SD) or percentage	n
Age	49.5 (10.77)	8
Sex
Female	88.9%	8
Male	11.1%	1
Race
White	66.7%	6
African American/Black	11.1%	1
Asian	11.1%	1
Don’t know/refuse	11.1%	1
Ethnicity
Latinx	11.1%	1
Non-Latinx	77.8%	7
Don’t know/refuse	11.1%	1
Level of education
Master’s degree	88.9%	8
Bachelor’s degree	11.1%	1
Has a disability	0.0%	0
Has a child with a disability	22.2%	2
Vocational teacher	88.9%	8

Note: One leader did not complete a demographic survey and one teacher did not report their age or race.

2.3 Procedures

Teachers completed surveys on VIT-TAY implementation in-person or electronically via REDCap. Autistic students completed implementation surveys in person at post-test. See Smith, Sherwood, Ross et al. (2021) for a complete description of the study protocol procedures and data collection schedule.

2.3.1 Training orientation adherence checklist

During VIT-TAY training orientation, an adherence checklist (Appendix) was completed by a research team member (who themselves were trained to fidelity [with the same checklist] to lead orientation). Following the orientation, teachers were asked to navigate all sections of VIT-TAY independently (e.g., e-learning content, job application, virtual interviews) until they felt comfortable to orient students. Teachers then used the adherence checklist to lead student training orientation sessions and were supervised by an on-site research team member. The adherence checklist assessed the adherence and quantity (completeness) dimensions of the broader fidelity construct. The adherence checklist included ten sections: 1) Introduction, 2) Interview Basics, 3) The Application, 4) The Interview, 5) Interview Interface, 6) Interview Assessment and Learning Goals, 7) Transcript, 8) After-Interview Questionnaire, 9) Starting a New Interview, and 10) Your Rewards.

2.3.2 VIT-TAY implementation procedures

Following the training orientation, teachers facilitated students completing virtual interviews over 4–6 weeks. During this time, leaders were involved in three primary ways: 1) contributed to and approved the final implementation design, 2) had final say on which teachers would participate, and 3) supervised to make sure implementation was running smoothly. Based on prior research with VR-JIT (e.g., Smith et al., 2015), the completion of at least 15 virtual interviews was recommended. To promote hierarchical learning, students were asked to progress through VIT-TAY’s three difficulty levels (i.e., easy, medium, hard). If students achieved a score of 90 or higher (out of 100) in the first three ‘easy’ interviews, they advanced to ‘medium’; if not, they had two more attempts to achieve 90 or higher. Students automatically advanced to ‘medium’ after five completed ‘easy’ interviews, regardless of score. Next, students continued with three to five interviews on ‘medium’ using the same progression to determine when they were ready for ‘hard.’ Students were then asked to perform ‘hard’ interviews for the remainder of their training. Teachers were instructed to help students review their virtual interview transcript and the student’s performance assessment.

2.4 Study measures

2.4.1 Leader and teacher-level quantitative measures

VIT-TAY orientation acceptability, appropriateness, and expected feasibility (pre-implementation): Immediately following VIT-TAY training orientation, leaders (n = 4) and teachers (n = 15) completed the VIT-TAY training orientation evaluation. One leader did not complete the pre-implementation surveys. Given that 10 teachers did not implement VIT-TAY, they did not complete any additional study measures. This survey consisted of 7 items (e.g., “How acceptable were the training materials?”). Item responses were on a 5-point Likert scale (e.g., 0 = Not at all satisfied to 4 = Very satisfied; or 0 = Not at all acceptable to 4 = Very acceptable). Internal consistency was high (α= 0.93). Leaders and teachers also completed the VIT-TAY appropriateness and expected implementation feasibility surveys immediately following the orientation. The appropriateness subscale consisted of 5 items (e.g., “How well do you think VIT-TAY fits with students’ goals for job training?”). Item responses were on a 5-point Likert scale (e.g., 0 = Not at all effective to 4 = Very effective; or 0 = Not at all well to 4 = Very well). Internal consistency was high (α= 0.87). The expected implementation feasibility subscale consisted of 9 items (e.g., “How prepared do you feel you are to train students on VIT-TAY?”). Item responses were on a 5-point Likert scale (e.g., 0 = Not at all prepared to 4 = Very prepared; or 0 = Not at all confident to 4 = Very confident). Internal consistency was good (α= 0.89). The above measures were used in prior research (Smith, Smith et al., 2021) and adapted from existing measures (Weiner et al., 2017; Smith et al., 2018).

Implementation context: One leader (serving as a teacher) and five teachers completed the implementation context survey to report on the context in which VIT-TAY was delivered. Implementing teachers completed the implementation context survey following study week two of VIT-TAY implementation and again at post-implementation. The survey consisted of four close-ended items evaluating delivery context (e.g., “What percent of your students needed no guidance, a little guidance, some guidance, or a lot of guidance?,” and “Please enter the percentage of time students used VIT-TAY individually or in a group.”). This survey was adapted from the Stirman adaptation coding taxonomy (Stirman et al., 2017).

VIT-TAY acceptability and sustainability (post-implementation): Leaders (n = 5) and teachers (n = 5) completed the VIT-TAY acceptability and sustainability survey within one week of post-implementation. The acceptability subscale consisted of 10 items (e.g., “How effective does VIT-TAY seem to be in helping students improve their interviewing skills?”). Item responses were on a 4-point Likert scale (e.g., 0 = Not at all to 3 = Very much). Internal consistency was low (α= 0.44); perhaps due to a small sample size as prior studies using this same measure yielded good reliabilities (α>  0.83; [Smith, Smith, et al., 2021]). As an additional measure of acceptability, leaders and teachers were asked one close-ended question during a semi-structured interview (see below; How likely would you be to recommend that other transition programs adopt VIT-TAY in their classrooms?) using a 5-point Likert scale (0 = Not at all to 4 = Very likely).

Sustainability of VIT-TAY was measured with 3 items evaluated independently (e.g., “How motivated are you to continue to deliver VIT-TAY to students?”, “How disruptive will it be to your daily work routine to continue to use VIT-TAY with your students?”, and “How equipped is your school to support the continued delivery of VIT-TAY?”). Item responses were on a 4-point Likert scale (e.g., 0 = Not at all to 3 = Very much). The above measures were used in prior research (Smith, Smith et al., 2021) and adapted from existing measures (Smith et al., 2018).

2.4.2 Leader and teacher-level qualitative measures

Semi-structured interview: Following implementation, leaders (n = 3) and teachers (n = 5) who implemented VIT-TAY completed an interview discussing potential barriers to and facilitators of VIT-TAY implementation. Two leaders were not available for an interview. Consistent with our quantitative measures, the semi-structured interview questions focused on barriers and facilitators that are salient to achieving “early” implementation outcomes as defined by Proctor et al. (2011; e.g., adoption, feasibility, acceptability, etc.) and suggestions for adaptations to the implementation process that would be beneficial to future efforts. The questions were informed by the quantitative measure items, which is consistent with a convergent mixed methods design. The semi-structured interview consisted of 15 open-ended questions (e.g., “What did you struggle with the most getting VIT-TAY up and running in your classroom?” and “How can we make VIT-TAY better for teachers and students to use?”).

2.4.3 Student-level quantitative and qualitative measures

VIT-TAY acceptability, feasibility, and usability: Student acceptability was measured quantitatively using an adapted version of the Treatment Acceptability Rating Form (Reimers & Wacker, 1988). The acceptability scale consisted of 5 items (e.g., “Virtual interviewing was easy to do”, “Virtual interviewing was enjoyable”) rated on a 5-point Likert scale (e.g., 1 = Not at all true to 5 = Very true; α= 0.82). Students also completed 1 survey item evaluating their acceptability of Kendra, the VIT-TAY job coach (“I found Kendra’s hand gestures helpful during the interview (thumbs up, thumbs down, etc.”)), two items evaluating the feasibility of the token economy (e.g., “I used my tokens to talk to Kendra about interviewing”; α= 0.89), and two items evaluating the acceptability of the token economy (e.g., “Talking to Kendra made me want to earn more tokens”; α= 0.91). Item responses were on a 5-point Likert scale (e.g., 1 = Not at all true to 5 = Very true).

Qualitatively, we assessed student-level acceptability via three open-ended questions (e.g., “What was your favorite thing about the VIT-TAY training?”, “What would you make different about the VIT-TAY training so that it helped you more?”).

We measured student-level VIT-TAY usability quantitatively using an adapted version of the System Usability Scale (Brooke, 1986) consisting of 7 items (e.g., “My teacher did a good job helping me learn how to use the virtual interviewing tool”; “I was able to use the virtual interviewing tool on my own”). Item responses were on a 5-point Likert scale (e.g., 1 = Not at all true to 5 = Very true; α= 0.79).

2.5 Data analysis

2.5.1 Quantitative data analysis

Quantitative data was analyzed using SPSS 26.0. We used descriptive analyses (i.e., mean, standard deviation, range) to evaluate our implementation outcomes of pre-implementation teacher acceptability, appropriateness, and feasibility, post-implementation teacher acceptability and sustainability, and student acceptability and usability. Summed scores of the scales were computed prior to analysis. We used item-level mean imputations for scales where items were missing, as long as no more than 50%of survey data was missing. Two teachers missed a single item each on the VIT-TAY orientation acceptability and expected implementation feasibility surveys. One teacher missed two items on the VIT-TAY appropriateness survey. Two teachers did not complete the context and adaptation mid-point survey. One student participant did not complete the acceptability survey; that same participant, along with two other student participants, did not complete the usability survey.

2.5.2 Qualitative data analysis

The interviews with leaders and teachers were transcribed and uploaded into ATLAS.ti, a qualitative data analysis software. We then analyzed the interviews using a 3-phase process (Grinell and Unrau, 2014). Phase 1: reading each transcript and establishing a framework. Phase 2: assigning codes to recurring topics within the interviews, which resulted in a total of 27 codes appearing across the 8 interviews. Using ATLAS.ti, these codes served as our codebook. Phase 3: searching for meaning and relationships between codes and assigning each code to an overarching theme. Finally, based on the method introduced by Attride-Stirling (2001), we conducted a thematic network analysis of themes, codes, and quotes. After reviewing each code and its corresponding quotes the study team then developed a network of global, organizing, and basic themes that encompassed all of the semi-structured interview data in the codebook. Organizing the data in this way helped ensure that the themes reached saturation.

2.5.3 Mixed methods analysis

Following the analysis of both quantitative and qualitative data, we performed an integrated mixed methods analysis of the data focused on convergent validation (triangulation) across methods (Creswell & Plano Clark, 2018). This perspective allowed us to draw conclusions and make comparisons between the quantitative and qualitative results for both the student surveys and interviews and the teacher surveys and interviews. Additionally, we explored convergence between the perspectives of students and teachers on shared constructs to gain a more comprehensive understanding of the aims of this study.

3 Results

3.1 Quantitative results

3.1.1 Leader and teacher-level results

Training orientation adherence checklist: Under the supervision of research staff, teachers delivered the training orientation to students with 100%adherence to the protocol.

Pre-implementation VIT-TAY orientation acceptability, appropriateness, and expected implementation feasibility: Leaders and teachers reported the VIT-TAY orientation was acceptable (M = 21.11, SD = 4.57) and that VIT-TAY was appropriate for transition services (M = 15.84, SD = 2.93). Leaders and teachers also expected that implementation of VIT-TAY would be feasible in their schools (M = 22.55, SD = 7.02). See Table 2 for a summary of descriptive statistics for all survey data.

Table 2

Descriptive statistics for VIT-TAY implementation subscales

Surveys	Mean (SD)	Range
Leaders and teachers
Pre-implementation (n = 19)a
Orientation acceptability	21.11 (4.57)	11–28
VIT-TAY appropriateness	15.84 (2.93)	9–20
Expected implementation feasibility	22.55 (7.02)	9–36
Post-implementation (n = 10)
VIT-TAY acceptability	26.80 (2.25)	24–30
VIT-TAY sustainability
Motivated to continue	2.80 (0.42)	2–3
Equipped to continue	2.60 (0.70)	1–3
Disruptive to routine	2.00 (1.25)	0–3
Recommend VIT-TAY adoption	3.88 (0.35)	3–4
Students (n = 47)
Overall acceptability	20.36 (4.26)	7–25
Overall usability	29.33 (4.92)	11–35
Job coach acceptability	4.17 (1.20)	1–5
Token economy feasibility	5.26 (3.22)	2–10
Token economy acceptability	6.19 (3.13)	2–10

Note: VIT-TAY = Virtual Interview Training for Transition Age Youth. ^aThe 19 leaders and teachers included 9 of the 10 participating teachers and leaders as well as 10 teachers who were trained to serve as back-up implementers but did not implement VIT-TAY after completing their evaluation of VIT-TAY orientation. One leader did not complete the pre-implementation surveys. We included these additional pre-implementation data as they were collected in good faith and optimize variation in the measures collected.

Implementation context: We did not observe any statistical differences using paired-sample t tests between delivery strategies measured at the end of the first two weeks of implementation and again at the end of implementation (all p >  0.10). Thus, we present the means between midpoint and endpoint. As a result, means within categories may not add up to 100%. Reports from four teachers at the end of the first two weeks and six teachers at the end of implementation were used to generate the results. Leaders and teachers reported that 23%of their students needed no guidance when using VIT-TAY, 37%of students needed a little guidance, 29%of students needed some guidance, and 11%of students needed a lot of guidance. Leaders and teachers also reported that all students used VIT-TAY at school, rather than at home, at a job placement, or in another setting. Within schools, teachers reported that all students used VIT-TAY during class rather than during homeroom, study hall, after-school programming, or free periods. Further, leaders and teachers reported that 89%of students used VIT-TAY in group settings with their own devices, 11%used VIT-TAY in private or semi-private rooms with their own devices, and no students used VIT-TAY in group settings with a single device. Teachers reported that most students were receiving some level of typical transition services concurrently with VIT-TAY. Specifically, 77%of students were working on job-skill development, 49%were working on resumes, 43.5%were participating in mock interviews with teachers, and 16%were participating in mock interviews with community employers.

VIT-TAY acceptability and sustainability (post-implementation): Post-implementation, leaders and teachers reported that VIT-TAY was highly acceptable (M = 26.80, SD = 2.25). Leaders and teachers also reported they were motivated (M = 2.80, SD = 0.42) and equipped (M = 2.60, SD = 0.70) to continue VIT-TAY implementation, while noting that VIT-TAY caused little disruption to their daily work routines (M = 2.00, SD = 1.25). Lastly, they reported they would recommend that other transition programs adopt VIT-TAY in their classrooms (M = 3.88, SD = 0.35).

3.1.2 Student-level results

VIT-TAY engagement: Approximately 61%of students (n = 28) progressed through the recommended easy-to-medium-to-hard difficulty levels on the virtual interviews (including the completion of five hard interviews). Meanwhile, 6.5%(n = 3) of students completed four hard interviews and 10.9%of students (n = 5) completed three hard interviews, while 6.5%of students (n = 3) completed one or two hard interviews. Also, 13.6%of students (n = 6) completed no hard interviews, but completed the progression through medium. Finally, one student failed to complete the progression through medium. Notably, two students mistakenly used each other’s “saved” password on their computers, and we were not able to accurately determine their progression.

VIT-TAY acceptability, feasibility, and usability: Post-implementation, students reported VIT-TAY to be acceptable (M = 20.36, SD = 4.26) and usable (M = 29.33, SD = 4.92). Finally, students reported that Kendra the job coach’s real-time feedback was very helpful (M = 4.17, SD = 1.20), they reported that they used the token economy somewhat (M = 5.26, SD = 3.22), and when they did use the token economy, they found it to be somewhat acceptable (M = 6.19, SD = 3.13).

3.2 Qualitative results

3.2.1 Leader and teacher-level results

Semi-structured interview: After analyzing the semi-structured interviews with leaders and teachers who administered VIT-TAY, we discovered one global theme and three organizing themes that helped categorize the data. See Supplementary Figure 2 for the thematic network map.

Under the global theme of acceptability andusability, the first organizing theme Engagement with VIT-TAY describes how the students and teachers engaged with VIT-TAY, highlighting their independence and enjoyment using the tool, as well as how they utilized the feedback to improve their interview skills. Leaders and teachers identified three main aspects of VIT-TAY that encouraged student engagement:

VIT-TAY promotes independence. After teachers had an opportunity to train and implement VIT-TAY with their students, the majority felt that the tool was easy to use and that the students could explore the educational content and practice job-interview training on their own.

Students enjoyed using VIT-TAY. Following implementation, semi-structured interviews with leaders and teachers revealed that students expressed a high level of enjoyment while interacting with VIT-TAY.

VIT-TAY feedback was beneficial. While reflecting upon the VIT-TAY tool, leaders and teachers recognized real time feedback as a valuable factor that increased student engagement. This included feedback from the virtual job coach, the interview transcript, and the ability to have voice replay.

The second organizing theme VIT-TAY presents a real-world job interview scenario illustrates how VIT-TAY helps represent real-world job interview experiences. Leaders and teachers considered VIT-TAY to be a realistic representation of a job interview because of:

Diversity in interview characters. Leaders and teachers thought the interview characters conveyed the authenticity and diversity of a real-life hiring manager, which further prepared them for an actual job interview.

Interview preparation. Many of the leaders and teachers expressed that VIT-TAY was beneficial in getting their students ready for the interview process from start to finish.

The third organizing theme, Inner context and training barriers to implementation in schools, highlights barriers schools may encounter while implementing the VIT-TAY tool. When leaders and teachers identified the most common obstacles with implementing the VIT-TAY tool in their classrooms, the difficulties they experienced fell into two categories:

Inner context barriers to implementation. Inner context barriers consisted of challenges that were directly due to working in a school setting. Leaders and teachers found that the time commitment, scheduling, and space required to successfully implement VIT-TAY in the classroom created barriers to completing VIT-TAY.

Training barriers to implementation. Training barriers represented challenges that could be resolved by additional support resources. Leaders and teachers expressed the desire to have further guidance regarding the importance of certain aspects of VIT-TAY or in-depth training prior to delivering the tool to their students. This additional guidance would promote a smoother implementation process.

See Table 3 for leader and teacher qualitative themes.

Table 3

Teacher qualitative themes

Global theme: Acceptability and usability
Organizing theme: Engagement			Organizing theme: Real world scenarios		Organizing theme: Barriers
Basic theme: Independence	Basic theme: Enjoyment	Basic theme: Feedback	Basic theme: Diversity in interview characters	Basic theme: Interview preparation	Basic theme: Inner context barriers	Basic theme: Training barriers
“At the end, we had a couple of students who wanted to log on independently outside of that time, which they were given permission to do, obviously, because they had already had the orientation and felt comfortable doing it.” “It was excellent. It’s a really good thing. And I think kids got to be very independent.” “They’re very independent, it’s very student-led, which is a great feature of it. It’s quality, too. It’s nice.” “I liked that ...  students can do it when no one is supervising, you can do it independently. You know, you can have a class, once they’re familiar with it, you can have a classroom of students doing it.” “The kids seem to do the interviewing and the whole program on their own, they really didn’t have any problems. They were going right through it.”	“The feedback I got from the staff and students was that it was, again, user-friendly and the students really enjoyed it.” “Most of the students took to it pretty well. They really didn’t have any problems with it. They were pretty happy with it.” “Yeah, they looked forward to it. I was a little bit nervous that it would become mundane to them but I didn’t hear that from any of them. They were excited to do it. Yeah, so it was good, it was good.” “I was really impressed that they did stick with it. And that so many did 15 or more. So I know, [ ...  ]they would tend to come up with excuses, or you know, “I don’t wanna do it” or “I’m done” or “I’m not participating.” But my kids really stuck with it. So there’s something there that really was intriguing to them.”	“[ ... ] I really like the fact that there’s real time feedback for our students. [ ...  ] our students really learn well from that [ ...  ] And I like that it allows students to practice these skills in sort of a safe place. It’s a low stakes setting. It’s not going to make or break anything right now. [ ... ]” “Them being able to look at the transcript, do the playback and see where they scored [ ...  ] they could see some of the other areas where it might have been “Oh, I should have said this”, or “I should have explained this a little bit more”, [ ... ].” “[I would consider giving real-world tokens/rewards such as] extra basketball time in the gym or something like that. [ ...  ] I think that would be very helpful.”	“I think it’s good that you have both men and women, and different cultures, because some kids are, you know if you’re not comfortable, I mean if you’re very white privileged and you’re not really comfortable having to “answer to” an African American man, some might have a difficult time with that. But that’s real. You don’t know who’s going to be interviewing you. That’s real.” “I mean they’re going to come across different people interviewing them their whole life. So you need to have different personalities interviewing on your program. You can’t just always have soft, easy, yeah, you have to have all different kinds of personalities because you never know who you’re going to interview with.”	“I think the interview will also give them a better sense of the process of, you know, this is what I have to do first, before I can even get to the interview, and I have to make sure my application looks correct in order to get the interview.” “We think it’s helpful because then it gets them kind of used to speaking in front of an interviewer. Because you’re not going to have that option to click your answers when you’re in an interview in real time.” “I like how it engaged the students and it was actually like an interview if you were doing an interview with someone... I think it was teaching them what to expect in an interview.”	“Just scheduling and space. The fact that the two groups had to be kind of kept separate. That was a little difficult for us. In a facility where you maybe had multiple computer labs, or areas where everyone could log in at once, it would have been easier.” “I think that was too many to have the talking going on, with 11 kids all saying different answers at the same time. So I think next time maybe have fewer students in a room.” “Until the students met their [ ...  ] goal for a number of interviews, it was sort of time consuming, but not terribly so. That would be the only thing; that it takes a bit of time. [ ... ]” “I think that’s another barrier that allowing the time in the school week to make available to allow the students to work on it on a consistent basis.”	“So if it hadn’t been kind of laid out and explained to the teachers, I think they would have had a hard time being like “OK, how much time do we spend on this section vs. the other section?” So getting that breakdown was really nice, in particular.” “Well, I think just maybe if we had more staff on board that would feel comfortable taking groups... you know we had the one training session, but there were some questions after that. So [ ...  ]a couple of more specific training sessions would have been helpful.” “Maybe if there were more tutorial type sessions. We sort of got an overview when we logged on, and there’s some tutorials online, but maybe a little bit more in depth would be helpful?”

3.2.2 Student-level results

VIT-TAY acceptability and usability: We analyzed the qualitative student acceptability data using the same methods as the semi-structured interviews. We discovered two global themes that helped categorize the data. See Supplementary Figure 3 for the thematic network map.

Under the global theme of Favorite things, the first organizing theme, Interview preparation, describes how the intervention helped students prepare for job interviews, highlighting interview knowledge and skills. The second organizing theme, Ease and enjoyment in usability, illustrates students’ ease and enjoyment in the usability of VIT-TAY. Finally, the third organizing theme, Interacting with interviewers, highlights the interaction component of VIT-TAY. While a few of the students desired more interview characters and variability in their reactions, other students enjoyed the interviewers and being able to interact with them. Under the global theme of Things to Change, the first organizing theme, Performance, identified instances when students wished they had performed better while using VIT-TAY. The second organizing theme, Technical issues, describes technical bugs and glitches within VIT-TAY. Within the third organizing theme, More options, students described three ways VIT-TAY could provide more options, which were sorted into basic themes: 1) more questions/answers (e.g., response optionsduring virtual interviews), 2) more guidance (e.g., hints and explanations within VIT-TAY), and 3) more features (e.g., more interviewers, slower audio). Lastly, the fourth organizing theme, Difficulty, describes students’ recommendation to adjust the difficulty level of VIT-TAY (e.g., more difficult, less difficult). See Table 4 for student qualitative themes.

Table 4

Student qualitative themes

Global theme: Favorite things				Global theme: Things to change
Organizing theme: Interview preparation		Organizing theme: Ease and enjoyment in usability	Organizing theme: Interacting with interviewers	Organizing theme: Performance	Organizing theme: Technical issues	Organizing theme: More options			Organizing theme: Difficulty
Basic theme: Interview knowledge	Basic theme: Interview skills					Basic theme: More questions/answers	Basic theme: More guidance	Basic theme: More features
“It helps you understand what a job interview is.” “Getting a good idea of what was proper to say during an interview.”	“Getting to learn new interview skills.” “To learn skills for a real job interview.”	“It was easy to use.” “Interviews were fun.” “It was fun.”	“Interactive, cool to see what [the]interviewer is thinking. It’s different.” “The way we were able to make contact with the interviewer.” “Talking to the interviewers.”	“Staying on topic and give myself more time to think about what I’m doing.” “I could have scored more on confidence.”	“To remove bugs from it, they weren’t that big a deal but I couldn’t really think of much else I would change.”	“Having more options to pick from.” “My biggest suggestion would be to add more answers to choose from and make it more nuanced.”	“Little hints as far as what responses to choose.” “Explain what some of the words mean.”	“Have more people to interview.” “Make her have more emotions like the palm face slap.” “Slow the talking down.”	“Needs to be slightly more intense (more probing).” “Try to make it easier, the questions, on the medium level. I struggled with that one.”

3.3 Mixed methods integration

Our integrated mixed methods analysis allowed us to triangulate and then compare the qualitative results from leader and teacher semi-structured interviews with their quantitative survey results (Creswell & Plano Clark, 2018). Results from leader and teacher pre-implementation survey subscales (e.g., acceptability, appropriateness, feasibility) were matched with qualitative data from semi-structured interviews (see Fig. 1). For example, quantitatively, leaders and teachers reported VIT-TAY orientation was acceptable (M = 21.11, SD = 4.57), while qualitatively they reported “[ ... ] the training was pretty smooth.” Leaders and teachers also reported, quantitatively, that VIT-TAY was appropriate for students receiving transition services (M = 15.84, SD = 2.93), while qualitatively they reported “I think the interview will [ ... ] give them a better sense of the process [ ... ]”. In addition, leaders and teachers quantitatively reported that they expected implementation of VIT-TAY would be feasible in their schools (M = 22.55, SD = 7.02), while qualitatively they reported “[ ... ] I think it would be a great thing to have them continue to fill that space and just continue to practice”.

Fig. 1

Joint display of integration of leader and teacher implementation subscales and qualitative data.

Similarly, results from the leader and teacher post-implementation survey subscales (e.g., acceptability, sustainability) were matched with qualitative data from semi-structured interviews (see Fig. 1). For example, leaders and teachers quantitatively reported post-implementation that VIT-TAY was highly acceptable (M = 26.80, SD = 2.25), while qualitatively they reported “[ ... ] It’s very age appropriate, it’s very real [ ... ]”. Leaders and teachers also quantitatively reported they were motivated (M = 2.80, SD = 0.42) and equipped (M = 2.60, SD = 0.70) to continue VIT-TAY implementation, while noting that VIT-TAY caused little disruption to their daily work routines (M = 2.00, SD = 1.25). Qualitatively, leaders and teachers reported VIT-TAY “[ ... ] I personally don’t have any major criticisms of the program itself [ ... ].”

Additionally, student post-implementation survey subscales (e.g., VIT-TAY acceptability, usability, and job coach acceptability) were matched with student qualitative data (see Fig. 2). Quantitatively, students reported VIT-TAY to be acceptable (M = 20.36, SD = 4.26) and usable (M = 29.33, SD = 4.92). Qualitatively, students reported VIT-TAY was “Interactive, cool to see what [the] interviewer is thinking [ ... ],” and that “it was easy to use.”

Fig. 2

Joint display of integration of student implementation subscales and qualitative data.

4 Discussion

This study evaluated the implementation of VIT-TAY within an RCT involving five schools. The primary focus on VIT-TAY implementation reflects the critical importance of intervention effectiveness in context. Overall, several factors were identified as facilitators of VIT-TAY implementation. Specifically, we observed that leaders and teachers found VIT-TAY to be highly acceptable, feasible, and sustainable in the school context. Leaders and teachers also reported the appropriateness of VIT-TAY for Pre-ETS. Additionally, we observed that autistic students found VIT-TAY to be acceptable and usable. They reported learning new interview skills and finding VIT-TAY to be fun and easy to use. Meanwhile, in terms of barriers to VIT-TAY implementation, the feasibility of implementing the token economy system was unclear as autistic students generally reported they used it somewhat; though some teachers reported they would link earned VIT-TAY tokens to autistic students receiving real-world rewards (see Table 3). The context of implementation among schools was nearly universal, with all teachers implementing VIT-TAY at school during regular class time set aside for transition services and all autistic students using their own computing device. These results are somewhat surprising given the noted complexity of implementing interventions in schools (Domitrovich et al., 2008).

Leaders and teachers reported multiple barriers to future implementation. For instance, they suggested that additional training prior to implementation would be beneficial to improve their delivery of VIT-TAY. In addition, having more staff and administration on board with the implementation would have benefited the minority of teachers delivering VIT-TAY. Leaders and teachers also reported VIT-TAY implementation to be somewhat time-consuming, although this was possibly due to factors related to the research aspects of the study rather than the actual intervention implementation. Despite these barriers, it is likely that having VIT-TAY incorporated into the Pre-ETS curriculum would provide teachers with more preparation time, and time spent on implementation would be absorbed into regular teaching operations.

This evaluation of early efforts to implement VIT-TAY in schools was a pilot aim of the parent RCT. Thus, these findings contribute to the literature by providing preliminary data on the acceptability, feasibility, and sustainability of a technology-based intervention in Pre-ETS. As the use of technology-based interventions continues to expand in school settings, further examination of implementation outcomes is necessary in order to ensure adoption, appropriateness, and sustainability of the interventions. The field of implementation science in schools has progressed in the years since this study was designed (Cook et al., 2019), including the role outer context variables (e.g., policies) play in the implementation of evidence-based practices in public sector service systems (Lengnick-Hall et al., 2020), which could inform a future research study of VIT-TAY.

4.1 Limitations and future directions

There are both organizational-level and participant-level limitations to this study. Regarding organization-level limitations, this study did not evaluate school organizational factors, which are a known hurdle when implementing interventions in schools (Locke et al., 2016). For example, administrators can have tremendous influence on curricular decisions. The operational dynamics of a school also plays a factor in how the organization functions. Transition services (and Pre-ETS) and special education are highly fluid between states and even within school districts. While some states provide special education through age 22, Michigan provides special education through age 26. In addition, some schools provide more emphasis than others on Pre-ETS as part of their curriculum. For example, one of the schools in our sample was part of an evidence-based internship program where autistic students are on-site at their internship location. Meanwhile, autistic students at other schools were in a traditional public school. In addition, one of the schools served autistic students only, while other schools also served students with other disabilities. Technology accessibility also varies between schools. While each of the schools in our sample was able to provide each autistic student with their own computing device, we realize this is not always possible.

Regarding student-level limitations, individual teaching styles and teacher schedule/availability could impact implementation of interventions in schools (Cook et al., 2019). Teachers in this study may not have had access to the intervention and/or the training prior to the start of the school year, which may have made it difficult for them to plan and incorporate VIT-TAY into their lesson plans. Thankfully, teachers in the present study were flexible and willing to adjust their lesson plans. Additionally, some teachers were less confident in their ability to use and teach VIT-TAY to their students, mostly due to the limited time they had to learn the features within the technology. Some teachers noted that more training and support would have helped the implementation go more smoothly. However, with the ultimate goal of schools and teachers implementing the intervention independently, the existing protocol allowed us to evaluate the potential sustainability of VIT-TAY in schools. Although the schools provided representation of suburban, rural, and suburban communities, the small sample of five participating schools limits the generalizability of the findings. For example, only ten teachers completed the post-implementation acceptability measure, which resulted in a low internal consistency (α= 0.44). The size of the sample may have been too small to generate a stable estimate given that this same measure had a high internal consistency (α= 0.83) among n = 31 teachers in a prior study (Smith, Smith et al., 2021). Finally, teachers and students did not report on the individualized transition services received by students; however, this data would be important for future research to consider.

5 Conclusion

Overall, this initial implementation evaluation is informative for a larger scale study of VIT-TAY in schools. We are especially passionate about how this intervention and the strategies implemented could inform future education and autism policy by increasing the number of interventions available to Pre-ETS programs for TAY with autism. As schools operate with limited funding and resources, dissemination of evidence-based interventions with adequate implementation evaluations is particularly difficult and requires diligence and access to the education pipeline. This issue is especially salient with virtual education taking place around the U.S. during a pandemic, and this study could address the issue of accessible, virtual, evidence-based interventions.