Preliminary data from effects of knowledge translation methods on VR counselors providing pre-ETS to youth with significant disabilities: A brief report

Abstract

BACKGROUND:

Passage of the Workforce Innovation and Opportunity Act (WIOA) of 2014 placed an emphasis on increasing student education and preparation for employment for students as young as 14. When Virginia Commonwealth University (VCU) was awarded a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) to establish a Rehabilitation and Research Training Center (RRTC) on employment for transition-age youth with disabilities, one of the studies focused on knowledge translation methods to develop and deliver a toolkit to assist VR counselors (and others) providing pre-employment transition services (Pre-ETS) to students 14 to 16 years of age with significant disabilities.

OBJECTIVE:

The purpose of this practice brief is to share preliminary findings from Phase 1 and Phase 2 of VCU-RRTC on Employment of Transition-Age Youth with Disabilities’ study, Effects of Knowledge Translation Methods on Vocational Rehabilitation Counselors Providing Pre-ETS to Youth with Significant Disabilities 14–16 years of age.

METHOD:

To prepare this practice brief, program staff conducted a thematic analysis of interview findings from Phase 1 and reviewed program materials from Phase 2 to identify preliminary observations. Program materials included instructor case notes, student reports, student pre- and post-tests, interviews with families, vocational rehabilitation counselors, and school staff after instruction.

RESULTS:

From Phase 1, VR counselors, educators, and families desired early exposure to career exploration and felt that interest-driven experiences and education was generally lacking. From Phase 2, it was observed that lesson content needed to be relevant and engaging to younger students, who often had a very abstract understanding of work and limited self-awareness about career interests, personal interests, and strengths.

CONCLUSION:

Preliminary findings from Phase 1 and Phase 2 of the study underscore the desire and need to increase collaboration between educators, vocational rehabilitation counselors, and families. Observations from Phase 2 demonstrate the value of Pre-ETS instruction to younger students with significant disabilities, and how increased collaboration can prepare students for employment through early career exploration, workplace readiness skills, and community based work experiences.

1 Introduction

Virginia Commonwealth University (VCU) received a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) to establish a Rehabilitation and Research Training Center (RRTC) on employment for transition-age youth with disabilities. The Transition RRTC hosts six studies, with three of the studies conducted by other partnering institutions. This paper focuses on the study entitled Effects of Knowledge Translation Methods on VR Counselors Providing Pre-ETS to Youth with Significant Disabilities 14–16 Years of Age. In this study, VCU partnered with University of Kentucky (UK), who is following the same procedures throughout the state of Kentucky. This paper will provide background about pre-employment transition services (Pre-ETS), a short description of the study, and preliminary findings during Phases 1 and 2 of the study.

1.1 Overview of Pre-ETS

Passage of the Workforce Innovation and Opportunity Act (WIOA) of 2014 placed an emphasis on increasing student education and preparation for employment. Required in WIOA is the use of 15 percent of federal dollars towards the delivery of five Pre-ETS categories: job exploration counseling, work-based learning, postsecondary education counseling, workplace readiness training, and instruction on self-advocacy. Of note, WIOA also changed eligibility requirements for transition services and specified providing services to in-school youth as young as age 14 (WIOA, 2014). While the expansion of eligibility to Pre-ETS services greatly increases the opportunity for youth with disabilities to be better prepared for employment or postsecondary success through earlier involvement with vocational rehabilitation (VR) (Burgess & Cimera, 2014), other studies reveal potential limitations in VR preparation and collaboration (Carter et al., 2011, 2012; Plottner et al., 2019; Wehman, 2013).

Research demonstrates that interagency collaboration between VR and schools are vital to delivering quality Pre-ETS services to students with disabilities (Luecking & Luecking, 2015). Neubert et al. (2018) surveyed VR counselors who rated Pre-ETS services as important, but rated their preparation to deliver these services as significantly lower. A survey of VR counselors in Tennessee identified concerns about ability and time to collaborate, especially with students in general education classes (Awsumb et al., 2020). Concerns about the severity or visibility of a youth’s disability, and parental expectations were also identified as a minor or moderate barrier. This same survey found that VR counselors reported an understanding of Pre-ETS, but half still indicated they had not received sufficient training on implementation or wanted additional training. Additionally, VR counselors in rural areas of Tennessee identified more limited resources than their counterparts in nonrural locations. From an educator’s perspective, 77.9 percent of surveyed educators in Tennessee agreed that they would like more collaboration with VR (Carter, et al., 2021). This same survey indicated a larger difference in current collaboration between high school and middle school staff, with high school educators more likely to collaborate than middle school educators. Similar to VR counselors, educators identified lack of time, training, and resources as barriers. However, they also included providers only servicing certain types of disabilities. For such reasons, the RRTC on Transition chose to focus a study on the delivery of Pre-ETS services to students with significant disabilities aged 14–16.

2 Methods

2.1 Overview

The purpose of the Transition RRTC Study, “Pre-ETS for Youth with Significant Disabilities 14–16 Years of Age” is to use knowledge translation methods to develop and deliver a toolkit to assist VR counselors (and others) providing pre-employment transition services to students 14 to 16 years of age with significant disabilities. The study has defined students with significant disabilities as individuals who meet the Rehabilitation Act of 1973’s definition of significantly disabled, a person with an impairment that will seriously limit one or more functional capacities relevant to employment and will require long-term supports for competitive integrated employment (Rehabilitation Act, 1973). This three-phased study involves collecting information from key stakeholders for the development of an instructional toolkit.

In Phase 1, the study team gathered information from pre-employment counselors, family members/students, and educators across Virginia (VA) and Kentucky (KY). This phase involved the following: (1) interviews with ten pre-employment counselors in each state on the barriers (challenges) and facilitators (positive aspects) of providing Pre-ETS; and (2) interviews with ten family members and their child (optional), as well as ten special education teachers on the barriers and facilitators of receiving Pre-ETS.

Currently in Phase 2, Transition RRTC staff in both states began providing Pre-ETS services. Research facilitated the Pre-ETS process in each of the school systems to obtain student referrals. This phase involves five students referred by VR, through coordination with the schools, to participate in 12 hours of Pre-ETS services. Those services include job exploration (3 one-hour lessons), workplace readiness (6 one-hour lessons) and a community work-based experience (up to 3 hours). Pre- and post-tests are administered and all materials developed by students are shared with families, VR counselors, and educators. At the conclusion of instruction, study staff briefly interview the stakeholders to collect feedback, which will guide the development of a toolkit, to be tested during Phase 3.

Our research questions are as follows:

Phase 1 (already completed):

Research Question 1. What are the barriers and facilitators to providing services on job exploration, workplace readiness and community work-based learning to students 14–16 with significant disabilities?

Phase 2 (currently completing):

Research Question 2. What are the job exploration, workplace readiness, and community work-based learning resource needs reported by 14–16-year-olds with significant disabilities and their families?

Research Question 3. What are the resource needs for Pre-ETS counselors to facilitate job exploration, workplace readiness and community work-based learning with 14–16 year olds with significant disabilities?

Phase 3 (to be completed):

Research Question 4. Is the targeted knowledge translation (KT) strategy superior to the traditional strategy (control) for acquiring and using evidence-based research knowledge (toolkit)?

2.2 Design

Using the Knowledge-to-Action (KTA) method as described by Graham et al. (2006), we will determine the extent of knowledge transfer. Graham and colleagues state that the three phases (with six components) do not necessarily represent a linear procedure. Phase 1 is Knowledge Inquiry, which has been completed. In this phase, all stakeholders participate in providing feedback during the knowledge inquiry process from representatives of all groups, such as Pre-ETS counselors, teachers, students and their families. VCU and UK study staff contacted families via email, and those interviews with students and families were conducted by phone. Students had the option of being included in the interview (they were never interviewed alone), and they were informed that they did not have to answer any questions. Students also had the option of not participating in the interview at all. Interviews with teachers and Pre-ETS counselors were also conducted by phone. VCU and UK study staff have professional relationships with teachers and Pre-ETS counselors across the two respective states, and emailed those individuals to ask for an interview. All email language, interview protocols, and procedures were approved by the institutional review boards.

For Phase 2, termed Knowledge Synthesis by Graham et al. (2006), a maximum of 100 students will be enrolled in the three courses over years two and three. Pre- and post-tests (4-5 items) are given before and after each one-hour lesson to determine if knowledge and skills increased. Responses are dichotomous (yes/no), with a request to answer 2-3 open-ended questions. Descriptive data will be examined, with a paired samples t-test and measure of magnitude to determine differences in knowledge before and after each training. During this phase, the students and their families will be interviewed about the courses in which they are enrolled. Their teachers (a maximum of 2 per region) and the Pre-ETS counselors (a maximum of 2 per region) will provide feedback as well. We will use constant comparative analysis for the qualitative piece.

3 Results

3.1 Phase 1

In Phase 1, we conducted interviews to hone in on ways to address barriers and enhance facilitators, as well as the KT strategies that Pre-ETS counselors will prefer (Lambert et al., 2023). Demographic information was also collected to ensure representation from a variety of disability categories, as well as representation from rural, suburban, and urban areas. We used thematic analysis (Nowell et al., 2017) to examine the qualitative data. Two researchers examined and analyzed the data separately to reduce bias. The study team also held monthly project staff meetings to discuss findings. The team developed some themes based on expertise, as well as findings in the literature prior to data collection. Table 1 provides a breakdown of the number of families, teachers, and VR counselors interviewed in each state.

Table 1

Number of people interviewed in each state by stakeholder group

Stakeholder group	Virginia	Kentucky
Families	10	10
Teachers	12	10
VR counselors	7	10
Total	29	30

Note: Students had the option to participate in the interviews, along with their parents. In Virginia three students opted to participate, and in Kentucky, four students participated.

The following paragraphs provide information on the results most relevant in development of the lessons.

Across VR counselors, educators, and families, participants felt that providing an interest-driven community-based experience was lacking (Lambert et al., 2023). Participants wanted students with significant disabilities to be exposed to those experiences as early as possible in order to have consistency of exposure, while giving a variety of possibilities. This emerged as one of the most prominent themes in helping students with significant disabilities in future planning, including choosing coursework throughout high school to expand on their interests. Crucially, it was pointed out how important these community relationships or social capital were in rural areas when it came to finding employment. For VR counselors and educators, early exposure was primarily beneficial when it came to informed choice and preparing students for work through resource identification. For families, early exposure to Pre-ETS would provide students with more time to practice skills required for employment and better alignments with academics.

Lambert and colleagues (2023) found that other themes emerged, as well. Those identified by interview participants included time constraints, emphasis on things other than work for students at this age, difficulty engaging students and parents, the need for more training on Pre-ETS and instruction (especially for students with significant disabilities), transportation, collaboration between stakeholders, and limited business partners in the community. Because of the time in which these interviews were occurring, COVID was also identified as a barrier to Pre-ETS implementation.

In terms of resources interviewees believed would help with implementing Pre-ETS services, VR counselors requested more training on how to deliver content to younger students and specific resources and activities for students with significant disabilities. Educators also shared a desire that VR counselors had more professional development on how to work with students with significant disabilities and that they were able to spend more time in schools working directly with students. Educators wished that they themselves knew more about VR and potential services to better prepare students and families to work with VR after graduation. Families also expressed a desire for VR counselors to be more knowledgeable about specific disabilities. Families also wanted to know more about VR services and different providers. They wanted more collaboration and communication, especially during the summer, from schools and VR counselors.

Based on these interview findings, research staff created a curriculum of job exploration and workplace readiness lessons, as well as a community-based work experience in the hope of addressing identified challenges. Starting with students as early as eighth and ninth grades will allow planning of their high school curriculum and IEP development based on career interests and strengths. The curriculum is also designed to be used in a virtual environment or in person, in addition to including clear explanations of alternatives for specific disabilities (vision impairment or non-verbal, for example). The process includes providing feedback to be shared with the student’s family, teacher or case manager, and VR counselor.

3.2 Phase 2: Preliminary findings

At the time this practice brief was submitted, Phase 2 instruction in job exploration, workplace readiness, and completion of a community-based work experience have been completed at four sites and three ongoing in Virginia, one completed and two ongoing sites in Kentucky. Observations shared from instructors across these sites are the need to make the information relevant and engaging to younger students, who may still have a very abstract understanding of work and work expectations. For example, early lesson activities to identify personal interests and strengths were not as straightforward as one may expect. Students were often reliant on instructors to help them identify both personal interests and strengths. In an attempt to engage students, lessons touched on topics that were age appropriate and of high interest. For example, to help students better understand the concept of personal strengths, comic book characters were used to identify individual character strengths, and how they differed from other comic book characters. Superman’s ability to fly was contrasted with Spiderman’s ability to climb. Though not directly work related, introducing the concept in this manner allowed students to relate material to previous knowledge and interests. From there, instruction focused on how strengths and interests could be pulled from academic experiences and related to work environments. Students were given time and support to record in their workbooks personal interests and strengths. Future lessons continue to build on how these identified strengths may be relevant to future jobs and careers.

Similarly, rather than have students generate workplace preferences based on an abstraction of work, lesson material gave specific examples of different preferences and had students identify their preference. Preferences were edited to add accompanying visuals and audio when relevant based on instructor feedback. Examples of preference choices presented to students were working inside or outside, quiet or loud environments, sedentary or active tasks, working alone or in groups, and routine or variable tasks. Students were shown examples of each preference and given time to relate these preferences to their experience at school, at home, and in the community.

Early on, it was observed that some lesson content, pre- and post-tests, and workbook documentation needed to be simplified. Each round of instruction has resulted in a streamlining of lesson activities, and fill-in-the-blank responses were removed from all pre- and post-tests. Research staff are in the process of implementing a pre-lesson technology assessment and guide to introduce basic editing skills to students for use in their workbooks. Table 2 shows preliminary results from the Virginia pre- and post-tests only (Kentucky data are not yet available).

Table 2

Pre- and post-test paired samples t-test preliminary results

Variable	n	M	SD	Two-sided p
Job exploration pre-test	13	9.31	2.36	.011*
Job exploration post-test	13	11.62	3.45
Workplace readiness pre-test	11	17.64	5.41	.048*
Workplace readiness post-test	11	21.73	8.20

*p < .05.

Pre- and post-test scores for both Job Exploration and Workplace Readiness were summed in IBM SPSS Statistics for Windows version 28.0 to calculate the t-test. In addition, Cohen’s d was calculated to determine measure of magnitude, which takes out the effect of the sample size (Cohen, 1988). A calculation of .5 to .7 indicates a medium effect, and .8 or higher indicates a large effect. We hypothesized that students would show improvement from the pre- to the post-tests, after instruction. The results of our two-tailed paired samples t-test supported this. The Cohen’s d for Job Exploration was d = 2.78 and d = 6.02 for Workplace Readiness, indicating a large effect for both sets of lessons.

3.3 Instructional tips

In addition to making abstract concepts as concrete as possible, staff created lesson activities that engaged students to encourage participation and learning. Each lesson consisted of an icebreaker to both introduce the topic, make the lessons entertaining, and to avoid a purely lecture atmosphere. Example icebreaker activities include scavenger hunts, Hedbanz© inspired guessing games, and team problem solving activities. Lessons also included opportunities for group discussion and personal reflection. Some of the most engaged lessons incorporated class quizzes where students were able to rate examples of work ethic or use interactive slides to work through problem solving activities. Each lesson had accompanying workbook pages for students to complete to review lecture material, and record personal responses to questions related to lesson content. The goal of the workbook was to create a space where students could document their growing career awareness. Lesson material referred back to content recorded in the workbook from previous lessons so students could move from a basic understanding of employment to more individualized interests, preferences, and career clusters. The end goal is that these workbooks can be used by educators and VR professionals to base further Pre-ETS activities by supplying the basis of a job seeker profile.

The added benefit of focusing on class participation was the ability to assess students who did not communicate verbally or proficiently through written responses. In these cases, instructors documented student participation and expressions of interests in case reports in case these items were not documented in individual student workbooks. All students received case notes that were then shared with educators, case managers, and families to supplement information gleaned in student workbooks.

Though the workbook would use specific jobs as examples throughout the instructional series, emphasis was placed on students learning about and discovering potential careers in career clusters. This decision was made to place the emphasis not on learning specific skills for a specific job, but rather to explore potential career paths within a cluster. The reasoning is that limiting a student to a specific job could not only limit their career development, but may also be preparing them for a position that would not be relevant in our future economy. Too frequently, individuals with disabilities are relegated to positions that are quickly becoming obsolete or replaced by advancing technology (World Economic Forum, 2020).

4 Discussion

Implementation of Pre-ETS lessons for youths aged 14–16 with significant disabilities in Virginia and Kentucky underscores the importance of work-based experiences, starting early, and collaboration—especially multi-directional communication between school systems, VR, and families. Though Phase 2 instruction was designed based on Phase 1 data, additional information was gleaned during implementation of Pre-ETS lessons. Lessons were modified to be more accessible and practical for virtual and in-person instruction.

Additionally, research staff experiences underscored many of the findings from Phase 1. For example, research staff, as a result of the study design, acted as facilitators between parties in order to identify students, confirm class time, space, supports, and resources by bringing all together through regular meetings and providing regular updates and case reports. Though schools and VR counselors were often interested and eager to participate, the researcher as facilitator role proved essential because providing Pre-ETS instruction in schools was often outside the scope of educators’ and VR counselors’ day to day activities and responsibilities. Having an external facilitator would provide the necessary pressure to ensure that steps were carried out. On occasion, plans to conduct Pre-ETS instruction in certain school systems did not come to fruition because students, space, or time were not readily available. Even with research staff’s role as a facilitator, the difficulty of planning and organizing Pre-ETS instruction demonstrates the need for VR and schools to closely align their goals and communicate regularly to implement Pre-ETS instruction during a school day. Schools and assigned district VR counselors should make a plan together to ensure that Pre-ETS instruction is embedded or a planned component of instruction. Otherwise, it is simply too easy to overlook Pre-ETS instruction in order to meet the many other requirements expected of educators and VR counselors.

Collaboration to deliver Pre-ETS instruction should not be limited to schools and VR. It should also include communication with students and families, especially if cultural or institutional barriers to employment are to be overcome (Ruiz & Scott, 2021). For this study, communication with students’ families was built into the research design. In some systems, instructors were able to communicate directly with students and families through weekly emails and share case reports directly. In some school systems, firewalls and IT restrictions prevent direct communication between instructors and families, and instructors were reliant on educators and case managers to share information with the family. Ensuring that communication is maintained with families is vital to successful implementation of Pre-ETS instruction, especially if the goal of raising employment expectations across all stakeholders is to be met.

Though this research study does not track employment outcomes for student participants, it did follow up with families, educators, and case managers about instruction and instruction outcomes. Some families shared that students became more interested in topics surrounding work and showed a genuine enthusiasm for Pre-ETS instruction. However, the difficulty of scheduling post-instruction interviews further underscores the difficulty of collaboration and communication described by Phase 1 participants.

5 Conclusion

The research team collected a great deal of valuable information to develop the lessons, as well as build in a communication process. Students who participate in the lessons must meet the age criterion (14–16) to ensure they are receiving the services early enough to choose coursework based on their skills and interests, as well as have time for practice and repetition throughout high school. All stakeholders receive specific feedback on each student to facilitate additional reinforcement outside of the Pre-ETS lessons and school hours. In response to requests from all three groups interviewed, the lessons will become part of a toolkit for Phase 3 of our research. Once finalized, the toolkit will be available for anyone to use.

The results of Phase 1 interviews and the preliminary findings of Phase 2 demonstrate the value of Pre-ETS instruction to younger students with significant disabilities. The difficulty of finding time, space, and resources to implement Pre-ETS described by many in Phase 1 proved to be accurate in Phase 2. However, that difficulty should not be the reason students are excluded from early intervention that can have a significant impact on their transition and employment outcomes. Hopefully, Phase 3 of this study will address some of the ways in which KT and collaboration can be increased between all stakeholders.