A Delphi Study to Operationalize Evidence-Based Predictors in Secondary Transition

Abstract

Although there are many activities (e.g., transition services), derived from correlational research, that occur while students are in school that increase the likelihood of positive post-school outcomes, many teachers continue to provide services shown to have little to no effect on outcomes of students with disabilities. The purpose of this study was to operationally define the predictors of post-school success for educators to understand what is necessary to develop, implement, and evaluate secondary transition programs based on predictor research. Results conclude experts in the field reached consensus on an operational definition and set of essential program characteristics for each predictor of post-school success to aid practitioners in implementing these practices.

Delivering secondary educational opportunities that keep youth with disabilities in school while preparing them for a productive adulthood is the ultimate challenge facing high schools today (Stormont, Reinke, & Herman, 2011). There are many determinants of in-school and post-school success for students; effective instruction is a primary factor. Although many evidence-based practices and programs are available, many teachers continue to use practices shown to have little to no effect on outcomes of students with disabilities (Cook, Tankersley, & Landrum, 2009). It is important that educators are equipped to choose and implement practices shown to be successful for students.

Research conducted in the past 30 years to identify the most effective practices to teach academic and functional skills (e.g., Test, Fowler, et al., 2009; U.S. Department of Education, 2010, 2012) has focused on measuring student performance and did not directly link the intervention with post-school outcomes such as enrolling in post-secondary education or attaining employment. In response to the need to establish a link between in-school practices and programs and post-school outcomes, Test, Mazzotti, et al. (2009) reviewed correlation research in secondary transition using the quality indicators suggested by Thompson, Diamond, McWilliam, Snyder, and Snyder (2005) to identify variables or predictors correlated with improved post-school outcomes. In the initial review of empirical literature, researchers identified 16 evidence-based, in-school predictors of improved outcomes in education, employment, and/or independent living for students with disabilities (e.g., community experiences, parental involvement). See http://www.nsttac.org/sites/default/files/assets/pdf/pdf/ebps/In-SchoolPredictorsPSO_Jan2013.pdf for the predictors and their corresponding outcome area(s).

This research significantly advanced the field of secondary transition in special education by identifying evidence-based predictors of post-school success from high quality correlation research. The predictors identified were defined directly from the research studies that established them. Unfortunately, the descriptions of the predictors, as published in empirical research, lack the details necessary for educators and practitioners to develop, implement, or evaluate local programs aligned with the synthesized predictor research. For example, previous literature defined work study as (a) students who participated in work study were 2 times more likely to be engaged in full-time post-school employment (Baer et al., 2003), or (b) students in the Bridges School to Work Program who accepted a post-internship job offer and who completed the internship were more likely to engage in post-school employment (Fabian, Lent, & Willis, 1998). Although a state or local education agency may report having work study opportunities for students, without operational definitions of work study components and the other predictors, policymakers, administrators, and teachers are left to guess, or assume, their program is sufficiently aligned with the research literature. To move beyond guesses, assumptions, and speculations, each predictor needed to be operationally defined and essential program characteristics determined. Therefore, the purpose of this study was to add specificity to the existing definitions, and where necessary, operationally define the predictors of post-school success (Test, Mazzotti, et al., 2009) in such a way as to provide educators with criteria to develop, implement, and evaluate secondary transition programs based on predictor research. The objectives of this study were

• to reach consensus on the operational definition of each predictor of post-school success; and

• to identify and reach consensus on the program characteristics for each predictor of post-school success.

Method

Group Processes

A Delphi procedure (Linstone & Turoff, 2002) was used to gain consensus on the operational definitions and program characteristics of the predictors. The Delphi procedure is a nominal group technique used to solicit input from experts to reach consensus about a particular topic. Consensus is gained through an iterative process of soliciting information from, and providing information to, experts via a series of questionnaires. The Delphi procedure provides a structured, systematic approach to collecting data in situations where the only available alternative may be anecdotal or subjective (Linstone & Turoff, 2002). Strengths of the Delphi procedure include a structured method of communication for individuals with appropriate knowledge of the content to express differing perspectives, provide ongoing feedback, and provide opportunity to edit previous contributions throughout the process (Hsu & Sandford, 2007).

Participants

The authors developed a set of inclusion criteria to identify a purposeful sample of experts in the field of secondary transition and/or career technical education (CTE). For the purpose of this study, an individual was deemed an expert if he or she met one or more of the following criteria:

• author/researcher of scholarly, peer-reviewed work relative to one or more of the predictors of post-school success;

• author/researcher of scholarly, peer-reviewed work relative to one or more specific youth populations (e.g., youth who drop out, Native American youth, youth with emotional behavioral disabilities);

• author/researcher of scholarly, peer-reviewed work relative to one or more specific post-school outcome categories (e.g., post-secondary education or training, employment, or independent living);

• practitioner (e.g., teacher, service provider, educational administrator) with 10 or more years of service to secondary-age youth with disabilities (10 or more years was chosen to ensure practitioners had sufficient experience and expertise).

The authors initially generated a list of 48 potential experts. This list was purposefully reduced to ensure a balance of potential respondents across researchers, local and state education administrators and practitioners, and those with special education or CTE expertise. After reducing the list, 32 experts were sent a letter describing the purpose and process of the study and inviting them to participate. Of the 32 experts, 22 responded to the invitation to participate. Of these 22 respondents, 13 were female; 5 were practitioners with 10 or more years of services to secondary-age youth with disabilities, 3 had 10 or more years of service or research experience in CTE, and 14 were authors/researchers of scholarly, peer-reviewed work relative to predictors of post-school success, post-school outcome categories, or one or more specific youth populations.

Response Rate

Dillman’s (2007) survey methods were utilized to maximize response rate (e.g., using pre-notifications and follow-up reminders to complete survey). Overall response rate, calculated as total number of respondents by total number of possible respondents across all rounds of ranking, was 62%. Response rates varied across rounds ranging from 32% to 100%. Individuals representing each inclusion criteria (e.g., practitioners with 10 or more years of services to secondary-age youth with disabilities or CTE, authors/researchers of scholarly, peer-reviewed work relative to predictors) participated in each round of voting.

Procedures

Data were collected in three phases: (a) clarifying initial definitions, (b) soliciting input from experts and voting on definitions, and (c) reviewing final definitions and characteristics to ensure representation for students with disabilities from culturally and linguistically diverse (CLD) backgrounds (Hughes, 2013; Trainor, Lindstrom, Simon-Burroughs, Martin, & Sorrells, 2008). The process used in each phase of the Delphi procedures is described next.

Phase 1

To clarify the definitions of each predictor, authors re-examined the original 22 articles used to identify the original 16 predictors (Test, Mazzotti, et al., 2009). We recorded any explicit definitions or descriptions of characteristics described in the original literature that could assist in operationalizing each predictor. Next, we reviewed the secondary sources cited in the original 22 articles to better understand the original variable and ensure consistency in definitions across studies, again making note of any differences in definitions or characteristics. Last, in Phase 1, authors reviewed other widely disseminated sources (e.g., textbooks) and recorded any additional definitions or characteristics associated with each predictor. All information gathered from Phase 1 was organized into a table (see Table 1 for sample) and distributed to the experts as the basis for Round 1 in Phase 2.

Table 1.

Sample of Phase 1 Data Collection.

Predictor	Description from Test et al. (2009) Literature review of correlational research	Definitions identified from the literature referenced in the original articles used by Test et al. (2009) to establish the predictor	Behavior characteristics identified from the literature referenced in the original articles used by Test et al. (2009) to establish the predictor	Other sources of definitions or behavior characteristics if definitions are missing or ambiguous from the empirical literature
Work Study	Students who participated in work study were 2 times more likely to be engaged in full-time post-school employment (Baer et al., 2003)	None identified	Work study is defined as a paid or unpaid school sponsored work activity; getting school credit for sponsored work activity (Blackorby & Wagner, 1996;National Longitudinal Transition Study 2 Data Dictionary)	Work study program provides integrated academic, social, vocational, curriculum coupled with community work experiences to prepare youth with disabilities for productive community adjustment (Flexer, Baer, Luft, & Simmons, 2008)
	Students in the Bridges School to Work Program who accepted a post-internship job offer and who completed the internship were more likely to engage in post-school employment (Fabian et al., 1998)	None identified	Work study is defined according to whether the student participated in any work study programs according to Educational Manamagment Information System and school records (Baer et al., 2011)	Work study program—An educational plan in which students alternate between paid employment and formal study (free dictionary, http://www.thefreedictionary.com/work-study+program)

Phase 2

Phase 2 consisted of soliciting information from the experts through multiple rounds of questionnaires distributed via Qualtrics, an online survey software program. In all rounds of ranking, scores were calculated for each definition by summing the rankings assigned by experts for each definition. See Table 2 for an example of calculations. In each subsequent round, we calculated the minimum and maximum scores, mean, and standard deviation for each definition, and the highest ranked definitions for each predictor were carried forward for subsequent ranking. We graphed the rankings and used visual analysis (Kennedy, 2005) to determine the natural breaking point of the scores. Tied statements were included in the next round of ranking.

Table 2.

Example of Calculations for Each Round.

Career awareness item number	Number of responses per item	Individual rank	Total vote
1	13	10 + 10 + 10 + 9 + 9 + 9 + 6 + 6 + 5 + 5 + 2 + 2 + 1	84
2	10	10 + 10 + 10 + 10 + 10 + 9 + 9 + 9 + 9 + 8	94
3	1	10	10
4	10	10 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1	19
5	6	10 + 5 + 5 + 5 + 3 + 2	30
6	4	5 + 3 + 2 + 2	12
7	2	2 + 2	4
8	3	3 + 3 + 5	11
9	1	4	4
10	10	10 + 10 + 10 + 10 + 10 + 9 + 9 + 9 + 9 + 8	94
11	3	3 + 3 + 5	11
12	5	5 + 3 + 2 + 2 + 1	13
13	8	10 + 10 + 9 + 8 + 7 + 1 + 1 + 1	47
14	9	8 + 7 + 6 + 5 + 5 + 1 + 1 + 1 + 1	35

Phase 2, Round 1

For Round 1, experts were asked to review descriptions of each predictor extracted from the literature, write an operational definition for each predictor, and/or add missing program characteristics they deemed necessary for implementation of the predictor in a secondary setting. In the analysis, we combined similar definitions and characteristics, revised definitions to include suggested additions from experts, and eliminated duplicates.

Throughout this consensus building process (i.e., Round 2 and beyond), we were cognizant of remaining in the role of researcher and not letting our personal professional experiences influence results. To this end, we used a number of strategies to ensure results reflected experts’ intent and not that of the researchers. Strategies utilized to reduce researcher bias included using brackets (i.e., [ ]) to indicate a word inserted by us, working in pairs or triads when making decisions, checking our decisions with our co-researchers before continuing, and having experts vote on recommended changes before moving forward to the next round of rankings.

Phase 2, Round 2

In Round 2, we inserted verbs and other words that appeared to be missing from each definition and used brackets to denote our changes to what experts submitted in Round 1. We distributed the complete list of 223 suggested predictor definitions (i.e., 10–15 definitions per predictor) and 211 (i.e., 10–12 sets per predictor) suggested sets of characteristics to the experts and asked experts to rank 10 definitions for each predictor from those generated in Round 1 (with 10 being the highest and 1 being the lowest) and add any explanation, argument, or justification for or against the inclusion of the particular definition or characteristic as part of the final operational definition.

Phase 2, Round 3

As we reviewed the experts’ suggested text additions in Round 2 for the top 10 choices, we realized some suggested revisions could change the definitions or characteristics in substantial ways. For example, the definition of Vocational Education identified in Round 1 was, “Students should enroll in occupationally specific course to learn and practice specific vocational skills.” In Round 2, experts suggested adding “This coursework should also include opportunities for work-based experiences, such as work study, competitive employment, job shadows, etc.” To ensure we honored the experts’ intentions, we sought to gain consensus on the text additions before continuing with the rounds to reduce the definitions. Therefore, for Round 3, we distributed an Excel table containing the original suggested definition of the top 10 definitions from Round 2 and the suggested text additions to the experts who responded to Round 2. Experts were asked to vote yes, keep the addition, or no, do not make the change, for each suggested addition. Simple majority (i.e., 51% and above) was set as the decision rule for incorporating additions into the definitions or characteristics. Once additions were incorporated, we determined that separating the operational definitions from the program characteristics was necessary to ensure the program characteristics specifically aligned with the final operational definition as some suggested characteristics were not universal to all potential definitions. Therefore, to expedite consensus gaining, we distributed only the operational definitions to experts in Rounds 4 and 5.

Phase 2, Round 4

In Round 4, prior to sending out the 164 definitions (i.e., 9–10 definitions per predictor), the research team corrected grammatical and spelling errors and reduced redundancy (e.g., word duplications in a definition) to ensure consistency across definitions, being mindful not to alter the intent of the definition. For example, a suggested definition for self-determination was as follows:

• Self-determination, [is] the ability to set goals for oneself, evaluate options, and take initiative to reach them is based on the ability to make choices, solve problems, and accept consequences of one’s own actions.

We re-wrote the definition as follows:

• Self-determination is the ability to make choices, solve problems, set goals, evaluate options, take initiative to reach one’s goals, and accept consequences of one’s actions.

The revised definitions were distributed to experts, and experts were asked to rank their top three definitions, with 3 being the highest ranking and 1 being the lowest ranking.

Phase 2, Round 5

In Round 5, 48 definitions (i.e., 3 per predictor) were distributed and experts were asked to rank the one operational definition they thought most represented the predictor from the top three identified in Round 4.

Phase 2, Round 6

Round 6 focused on identifying the program characteristics aligned with the final operational definition for each predictor. In earlier rounds of ranking, the panel of experts identified from 3 to 12 program characteristics per predictor. Because the impetus for the study was to help educators know what is necessary to develop, implement, and evaluate secondary transition programs based on predictor research, the number of program characteristics needed to be reduced to only those most relevant for program development, implementation, and/or evaluation. The top 10 sets of program characteristics identified in Round 1 were split apart into individual program characteristics yielding 342 program characteristics (i.e., 10–28 characteristics per predictor). To determine which characteristics were essential, ancillary, or irrelevant given the newly defined operational definition, experts were asked to use their professional expertise and judgment to classify each suggested program characteristic as any of the following:

• essential, meaning the characteristic must be evident for the program to be considered as (or consistent with) a predictor. This characteristic is necessary for implementation and evaluation of the program’s effectiveness;

• ancillary, meaning the characteristic complements, or adds to, the program but is not required for implementation or evaluation (i.e., nice addition that expands or diversifies the predictor); or

• not relevant/exclude from list, meaning the characteristic did not align with the operational definition for the predictor and was not required for program implementation and/or evaluation.

Program characteristics identified as essential or ancillary were retained and included in Round 7 of ranking. Program characteristics identified as irrelevant were excluded from the list of program characteristics. To reduce redundancy and ensure consistency in the program characteristics, we themed the characteristics into like categories, and when possible, collapsed similar characteristics into one statement. For example, the following program characteristics were identified by the experts:

• Use a direct instruction curriculum to teach communication and interpersonal skills.

• Use a direct instruction curriculum to teach conversational, negotiation, and conflict skills.

• Use a direct instruction curriculum to teach group skills.

The research team collapsed these three statements into the following program characteristic:

• Use a direct instruction curriculum to teach communication, interpersonal, conversation, negotiation, conflict, and group skills.

Phase 2, Round 7

For Round 7, researchers asked experts to view each predictor’s operational definition and essential program characteristics as a whole and determine whether they were Acceptable or Required Revision. If an expert felt a revision was required, they were asked to re-write the definition or characteristic incorporating the suggested revision.

Phase 3

Because the number of students from racially and ethnically diverse backgrounds is steadily increasing in our schools (Hughes, 2013), it is important to ensure students’, from diverse populations, secondary transition needs are met through effective interventions and programs (Mazzotti, Rowe, Cameto, Test, & Morninstar, 2013; Morningstar, in press; Trainor et al., 2008). The final respondent group of the Delphi procedure was not representative of diverse populations (e.g., most respondents were White males or females). As we reviewed the final definitions and program characteristics identified by the experts, we noted some concerns as to whether definitions and characteristics were inclusive of diverse populations. For example, it seemed many of the program characteristics were targeted at mainstream transition practices and neglected to recognize the possibility of differences among diverse populations of youth (e.g., differences in expected outcomes of youth from diverse backgrounds, cultural nuances in expected social and workplace behaviors). Therefore, in the final phase, we requested the Equity Assistance Center (EAC), funded by the U.S. Department of Education, to provide guidance regarding equity, access, and culturally responsive characteristics for each predictor. To strengthen the final definitions and characteristics, researchers asked the EAC to review the operational definitions and program characteristics for each predictor with a lens toward equity, access, and culturally responsive characteristic descriptions.

Results and Discussion

Final Definitions and Program Characteristics

After seven rounds of voting during Phase 2, experts reached consensus on the operational definitions and essential program characteristics of the 16 predictors. Table 3 presents the final operational definitions and essential program characteristics. For each of the 16 predictors of post-school success identified through the review of correlation research, experts reached consensus on an operational definition and corresponding essential program characteristics.

Table 3.

Operational Definitions and Essential Program Characteristics of the 16 Predictors Identified in Test, Mazzotti, et al. (2009).

Predictor category	Operational definition and essential program characteristics
Career Awareness	Definition: Career awareness is learning about opportunities, education, and skills needed in various occupational pathways to choose a career that matches one’s strengths and interests.Characteristics:• Provide school-wide comprehensive and systematic opportunities to learn about various careers via job shadowing, internships, guest speakers, industry tours, Career Technical Education (CTE) classes, or career fairs.• Identify skills and qualifications required for occupations aligned with core content areas.• Embed career awareness in the general curriculum to teach about occupations related to the core content areas.• Make explicit connections between academic skills and how those skills are used in various careers throughout all general education classes.• Provide systematic, age-appropriate student assessment of career awareness (e.g., interest inventories, aptitude tests) for students to learn about their preferences and aptitudes for various types of career.• Provide instruction in how to obtain a job in chosen career path.
Occupational Courses	Definition: Occupational courses are individual courses that support career awareness, allow or enable students to explore various career pathways, develop occupational specific skills through instruction, and experiences focused on their desired employment goals.Characteristics:• Embed career awareness activities, career planning, and vocational assessments in all occupational courses.• Design curriculum for each course to include technology, 21st century skills, and employability skills for specific career/career cluster content.• Provide hands-on and community-based opportunities to learn occupational specific skills within each occupational course.• Incorporate Universal Design for Learning (UDL) principles in CTE programs including cooperative education programs to provide access to students with disabilities.• Provide course offerings throughout the school day so scheduling conflicts do not restrict student access to occupational courses.• Provide occupational courses that represent a wide variety of occupational clusters to provide students course choices that match their preferences, interests, needs, and strengths.
Paid Employment/Work Experience	Definition: Work experience is any activity that places the student in an authentic workplace and could include work sampling, job shadowing, internships, apprenticeships, and paid employment. Paid employment can include existing standard jobs in a company or organization or customized work assignments negotiated with the employer, but these activities always feature competitive pay (e.g., minimum wage) paid directly to the student by the employer.Characteristics:• Provide opportunities to participate in job shadowing, work study, apprenticeships, or internships. a Consider work study, apprenticeships, and internship environments that are culturally sensitive to students from different cultural backgrounds.• Provide instruction in soft skills (e.g., problem solving, communicating with authority figures, responding to feedback, promptness) and occupational specific skills (e.g., clerical, machine operation).• Provide transportation training, including the use of public transportation and job-site and community safety.• Conduct job performance evaluations by student, school staff, and employer.• Provide instruction in obtaining (e.g., resume development) and maintaining a job.• Develop a process for community-based employment options in integrated settings with a majority of co-workers without disabilities.• Conduct situational vocational assessments to determine appropriate job matches.• Develop a process to enable students to earn high school credit for paid employment work experience.• Link eligible students to appropriate adult services (e.g., Vocational Rehabilitation, Developmental Disabilities Services) prior to exiting school that will support students in work or further education.• Involve appropriate adult services (e.g., Vocational Rehabilitation or job coach when needed) in the provision of community-based work experiences.• Use age-appropriate assessments to ensure jobs are based on students’ strengths, preferences, interest, and needs.• Ensure employment training placements offer opportunities for (a) working 30+ hr/week, (b) making minimum wage or higher, with benefits, and (c) utilizing individualized supports and reasonable accommodations.
Vocational Education	Definition: Vocational education is a sequence of courses that prepares students for a specific job or career at various levels from trade or craft positions to technical, business, or professional careers.Characteristics:• Provide a sequence of entry level and advanced integrated academic and vocational courses designed to improve students’ reasoning and problem-solving skills, academic knowledge, work attitudes, specific occupational and/or technical skills, and general skills needed for employment.• Provide a combination of in-school and community-based academic, competency-based applied, and hands-on learning experiences in the career pathways based on the local labor market.• Provide connection to post-secondary education and/or employment through site visits and connections with support services (e.g., vocational rehabilitation, disability support services).• Provide opportunities to earn certificates in certain career areas (e.g., Certified Nursing Assistant, Welding, Food Handlers Certification).• Develop business partnerships to ensure a relevant curriculum.• Provide career counseling and guidance to assist students in career planning and development aligned with the students’ preferences, interests, needs, and skills.• Provide instruction in career development through volunteer work, job shadowing, work study, apprenticeships, or internships.• Provide accommodation and supports in CTE courses to ensure student access and mastery of content.• Provide instruction in soft skills (e.g., problem solving, communicating with authority figures, responding to feedback, promptness) and occupational specific skills (e.g., clerical, machine operation).• Measure achievement in soft skills (e.g., problem solving, communicating with authority figures, responding to feedback, promptness) and occupational specific skills (e.g., clerical, machine operation).
Work Study	Definition: A work study program is a specified sequence of work skills instruction and experiences designed to develop students’ work attitudes and general work behaviors by providing students with mutually supportive and integrated academic and vocational instruction.Characteristics:• Provide options for paid and non-paid work experiences both on- and off-campus with options for gaining high school credit for completing program requirements in all 16 occupational clusters.• Develop a plan for earning academic credit on the job through an integrated curriculum focused on work-related skills with school personnel, the student, and his or her parents.• Provide supervision and guidance during the development of work behaviors and skills to address problems, concerns, insights, and learning. a Consider culturally responsive behaviors and skills that address cultural concerns of culturally and linguistically diverse (CLD) students• Develop business/school partnerships, by educating employers about the resources of potential employees, to set up training sites. a Provide businesses with culturally responsive strategies to understand the cultural needs, behaviors, and practices of students from CLD backgrounds.• Develop policies to address liability, including student insurance and other Department of Labor issues/concerns.• Develop a process to match student interests with available sites both on- and off-campus. a Increase the number of available sites by recruiting site partners that reflect the cultural backgrounds of students.• Provide experiences in applied real-work settings supported by instruction.• Place students in work settings that match their preferences, interests, needs, and skills. a Consider partnerships with businesses owned by CLD communities • Provide transportation to vocational training sites.• Provide, or partner with adult services to provide, qualified trained staff to job coach as needed. a Staff qualification should include some cultural competencies or training • Provide self-evaluation and monitoring instruction to students.• Provide students school-based opportunities to reflect, discuss, and share their work placement experiences.• Have school personnel and site employees assess and monitor students’ progress by using job duty forms and task analysis for various sites.
Community Experiences	Definition: Community experiences are activities occurring outside the school setting, supported with in-class instruction, where students apply academic, social, and/or general work behaviors and skills.Characteristics:• Allocate sufficient resources to support meaningful community-based experiences.• Conduct ecological assessments to determine skills needed for various community environments.• Provide instruction on skills needed to safely access community environments as identified via ecological assessments.• Conduct transition assessments with students and families to determine appropriate community environments for current and future activities.• Use community-based instruction to teach, assess, and monitor the obtainment of desired academic and/or functional skills.• Observe and document students’ attainment of desired behaviors and skills across diverse environments.• Instruct students to use public transportation.• Provide supervision during community experiences to guide and direct students in the development of appropriate behaviors and skills needed for specific environments.• Involve parent and adult service providers to support student involvement in community experiences.• Cooperate with community partners (e.g., employers, recreation facilities) to develop community experience sites.• Provide supports for parents to arrange community experiences after school hours.• Train teachers and paraprofessionals in necessary safety, health policies, and liability coverage necessary for students to participate in community experiences.
Exit Exam Requirements/High School Diploma Status	Definition: Exit exams are standardized state tests, assessing single content area (e.g., algebra, English) or multiple skill areas, with specified levels of proficiency that students must pass to obtain a high school diploma. Diploma status is achieved by completing the requirements of the state awarding the diploma including the completion of necessary core curriculum credits.Characteristics:• Teach test-taking strategies and study skills instruction.• Assist students to plan for and use appropriate accommodations when taking the test.• Administer standardized practice tests periodically to monitor progress toward benchmarks.• Provide exit exams at the end of targeted courses designated by the state or at the end of a specific grade level (e.g., 11th).• Offer students, meeting criteria, appropriate accommodations, alternate, or alternative assessment procedures.• Provide student remediation assistance if they fail the test.• Provide students with multiple opportunities to take the test as allowed by the school/district for all students.
Inclusion in General Education	Definition: Inclusion in general education requires students with disabilities to have access to general education curriculum and be engaged in regular education classes with peers without disabilities.Characteristics:• Provide administrative support (e.g., professional development for teachers and paraprofessionals, common planning, providing paraprofessionals) to teachers for students with disabilities included in general education classrooms.• Provide specific instruction to support students with disabilities who are included in general education (e.g., differentiate instruction, learning strategies, study skills, organizational skills, personal management skills).• Evaluate the effectiveness of inclusive programming by using formative assessment to identify when adjustments are needed to accommodate all students’ learning differences (e.g., pace, communication skills).• Develop a receptive school atmosphere for including students with disabilities in general education by educating administrators, teachers, other staff, and students about person-first language and disability rights.• Observe and assess integrated environment to identify and provide interventions for needed academic, social, behavior, and communication skills to ensure a conducive learning environment for all students.• Use diverse instructional strategies to meet the learning needs of all students including UDL, technology, and linking instruction to student interests.• Provide professional development for secondary personnel to ensure personnel are qualified to use UDL and evidence-based instructional strategies.• Engage students as active participants in general education instructional processes utilizing multiple models of inclusive learning.
Program of Study	Definition: A program of study is an individualized set of courses, experiences, and curriculum designed to develop students’ academic and functional achievement to support the attainment of students’ desired post-school goals.Characteristics:• Ensure program of study is inclusive, academically rigorous, and supported by UDL principles.• Design multiple pathways in the general curriculum for satisfying standard diploma requirements.• Provide clearly defined graduation requirements leading to a state sanctioned exit document.• Establish planning process to assist students in developing their program of study.• Provide multiple opportunities (e.g., CTE; community-based work, independent living, and community access experiences; school-based enterprises; duel credit through a cooperative agreement) for students to acquire needed credits to achieve standard diploma and ensure a seamless transition to post-secondary education and employment settings.
Self-Determination/Self-Advocacy	Definition: Self-determination is the ability to make choices, solve problems, set goals, evaluate options, take initiative to reach one’s goals, and accept consequences of one’s actions.Characteristics:• Utilize a student driven Individualized Education Plan (IEP) process to allow students to demonstrate self-awareness, goal setting, problem solving, and self-advocacy.• Collaborate with general education teachers to embed choices into the general curriculum and daily lessons and provide opportunities for students to practice self-determination skills.• Teach students to self-monitor self-determination skills (e.g., accommodations and modifications) and provide opportunities for students to practice the self-monitoring strategy. a As you work with students, take into consideration the cultural nuances involved in teaching self-determination skills and providing opportunities to develop self-determination to students from CLD backgrounds.• Ensure all students, including those with significant disabilities, have a functional communication system to engage in choice making, problem solving, goal setting, taking initiative to reach goals, and accepting consequences for one’s actions.• Conduct age-appropriate transition assessments for students to learn about themselves, set goals, solve problems, use information, make decisions, and identify long-range goals.• Provide opportunities for students to develop self-awareness by engaging in honest and respectful discussions with students about their self-determination assessment responses.• Provide direct instruction in self-determination using a structured curriculum or evidence-based instructional strategy, with guided practice in natural school and community-based settings.• Foster the development of students’ leadership skills.• Expect and support students to make many routine choices for themselves through the course of a school day.• Work collaboratively with students to facilitate achievement of their goals by informing them of their options and the potential consequences of their choices.
Self-Care/Independent Living Skills	Definition: Self-care/independent living skills are skills necessary for management of one’s personal self-care and daily independent living, including the personal management skills needed to interact with others, daily living skills, financial management skills, and the self-management of health care/wellness needs.Characteristics:• Provide instruction, as needed based on assessment data, in (a) financial planning, (b) self-help, (c) cooking, (d) housekeeping, (e) home maintenance, (f) using transportation, (g) clothing care, (h) accessing community services, (i) time/organizational management, (j) self-determination, (k) social roles/citizenship, (l) community/peer relationships, or (m) critical thinking and problem solving.• Embed self-care/independent living skills instruction into academic coursework to help students connect academic skills to post-school goals.• Provide instruction in self-care independent living skills in multiple settings including general education, special education, and community.• Provide individual, small group, or whole class instruction in independent living and self-care skills, as appropriate.• Provide students multiple opportunities to practice independent living skills throughout the school day in real-life situations using real-life materials and equipment. a Independent living skills may include skills for functioning in an environment that requires interaction with people from different CLD backgrounds.• Provide transition services (e.g., completing housing application, obtaining Social Security Disability) for students to accomplish post-secondary independent living goals.• Conduct ongoing assessment of self-care/independent living skills to identify and evaluate levels of skill attainment, maintenance, and generalized use of skills in other settings where use of skills are required.• Teach home and community recreation skills that can be done alone or with others in both organized and informal settings. a and in culturally diverse settings
Social Skills	Definition: Social skills are behaviors and attitudes that facilitate communication and cooperation (e.g., social conventions, social problem solving when engaged in a social interaction, body language, speaking, listening, responding, verbal, and written communication).Characteristics:• Integrate social skills instruction across the curriculum (e.g., general education and community).• Use a direct instruction curriculum to teach communication, interpersonal, conversational, negotiation, conflict, and group skills in context.• Provide opportunities for students to practice communication, interpersonal, conversational, negotiation, conflict, and group skills in context.• Assist students to use problem-solving skills when difficult interpersonal situations arise in context.• Provide parent and school staff information and training in supporting age-appropriate social skill development for their child, taking into consideration the family’s cultural standards.• Use augmentative communication (AC) and assistive technology (AT) devices to encourage communication for students who use AC/AT.• Use ecological assessments to identify the social skills students will be expected to perform in each context.• Provide opportunities for students to practice social skills that foster authentic social interactions that foster the development of friendships.• Teach students to self-evaluate their use of social skills in the appropriate context.• Teach students the social expectations for various environments (e.g., church, school, work, recreation).
Interagency Collaboration	Definition: Interagency collaboration is a clear, purposeful, and carefully designed process that promotes cross-agency, cross-program, and cross-disciplinary collaborative efforts leading to tangible transition outcomes for youth.Characteristics:• Develop wide reaching state interagency teams that include disability related and non-disability related agencies (e.g., Developmental Disabilities, Vocational Rehabilitation, Department of Labor, Social Security Administration) with a common interest in transition service delivery.• Develop and implement formal and informal agreements between agencies responsible for the delivery of transition services.• Develop an agreed on vision and mission of transition services and programs.• Develop an organizational structure that includes a process for identifying membership (e.g., criteria for membership), terms of services, procedures for replacing members, orientation for new members, and web-based and print membership directories.• Coordinate the development of policies and procedures for service delivery and sharing of resources by both school and community agencies.• Implement a state-wide plan that (a) addresses gaps, (b) includes strategies for blending and braiding funding of other resources, (c) streamlines the transition process, and (d) eradicates duplication of service delivery.• Conduct asset/resource mapping to identify all community agencies that support youth with disabilities in the area as well as gaps in service delivery.• Clearly define roles and responsibilities of each organization as a part of the interagency agreement.• Schedule regular times for planning, developing, and measuring the progress and effectiveness of implementing a shared transition service delivery system at all levels (e.g., individual student, school, local, region, state, and nation).• Develop procedures for shared problem solving to address needs of students with disabilities and the barriers they may face during transition process.• Develop procedures for school staff to have systematic way to include students, families, community members, and agencies at different levels of the transition process (e.g., when to invite to IEP meetings, when to refer families to meet with agency, when to provide information sheet to family)• Establish multiple methods of communication and information sharing across agencies.• Provide cross-discipline professional development opportunities for all members of interagency council to ensure members are knowledgeable about services and eligibility criteria.
Parental Involvement	Definition: Parent involvement means parents/families/guardians are active and knowledgeable participants in all aspects of transition planning (e.g., decision making, providing support, attending meetings, and advocating for their child).Characteristics:• Provide relevant information about transition planning to parents through a variety of means (e.g., written, face-to-face, community-based trainings such as Autism Society) at each stage of the transition planning process such as transition from middle to high school, age of majority, and graduation.• Link parents with support networks (e.g., networking opportunities with other parents, advocacy groups).• Provide multiple options for involvement (e.g., pre-IEP planning input, flexible IEP meeting times) and alternate ways to obtain input in the transition planning process. a Consider parents’ perceptions in transition planning that may conflict with mainstream professional ideas. Some parents from CLD backgrounds may not be supportive of transition activities if they feel the plans are contrary to their expectations.• Establish a welcoming atmosphere in the school by developing a system of ongoing communication and interaction (e.g., e-mail, notes home, home visits, and regularly scheduled meetings in addition to IEP meetings). a Consider the language and cultural needs of parents from CLD backgrounds• Provide fairs, brochures, or workshops to educate parents about adult services and post-school supports in the community (e.g., vocational rehabilitation, mental health resources, post-secondary education institutions and supports). a Consider developing material in languages accessible to target communities.• Provide staff training on culturally competent transition planning (e.g., recognizing and honoring differences such as ethnic, socioeconomic, and values of the family).• Actively engage parents in interagency transition councils.• Collaborate with families to identify how the school and family/guardian can support the students in achieving their desired post-school goals.• Share transition assessment results with parents so that parents can use the information to provide training for their child in the home and the community and identify natural supports.
Student Support	Definition: Student support is a network of people (e.g., family, friends, educators, and adult service providers) who provide services and resources in multiple environments to prepare students to obtain their annual transition and post-secondary goals aligned with their preferences, interests, and needs.Characteristics:• Develop and implement procedures for cultivating and maintaining school and community networks to assist students in obtaining their post-secondary goals. a Consider networks that are culturally, racially, and ethnically representative to accommodate the needs of CLD students.• Provide students access to rigorous, differentiated academic instruction. a As well as teachers who use culturally responsive teaching strategies• Link students to appropriate individuals who can assist students in obtaining access to assistive technology resources and teach students to use technology to enhance their academic and functional performance.• Link students to appropriate individuals who can provide support for financial planning, navigating the health care system, adult services, or transportation.• Link students to a community mentor and/or school-based mentor/graduation coach.• Provide opportunities for meaningful engagement in the community (e.g., clubs, friends, advocacy groups, sports, etc.).• Ensure teachers and other service personnel provide ongoing transition assessment to assist in planning for needed supports and resources in school and beyond.
Transition Program	Definition: A transition program prepares students to move from secondary settings (e.g., middle school/high school) to adult life, utilizing comprehensive transition planning and education that creates individualized opportunities, services, and supports to help students achieve their post-school goals in education/training, employment, and independent living.Characteristics:• Provide systems level infrastructure (e.g., highly qualified staff and administrators with defined roles and responsibilities, sufficient budget) to monitor and guide students to obtain post-school goals.• Provide integrated instruction in all areas of independent living (e.g., community living, transportation, recreation leisure, self-advocacy, goal setting, decision making) for all students with disabilities.• Provide individualized transition focused curriculum and instruction based on students’ post-secondary goals in post-secondary education, employment, and independent living (e.g., self-determination and financial planning).• Provide instruction and training in natural environments supported by classroom instruction.• Provide individualized transition services based on students’ post-secondary goals in post-secondary education, employment, and independent living (e.g., self-determination and financial planning).• Provide opportunities for engagement with non-disabled peers in the school and community.• Use interagency collaboration with clearly defined roles and responsibilities to provide coordinated transition services (e.g., Vocational Rehabilitation, Mental Health) at multiple levels (i.e., student, school, districts, region, and state) to assist students in meeting their post-secondary goals.• Monitor and assess students’ progress in the domains of academics, daily living, personal and social, and occupational.• Use multiple strength-based assessments across multiple domains at different points in time to assist student and IEP teams in post-school planning.• Provide training and resources to families to involve them in transition planning and connect them to adult agencies and support and information networks.• Conduct program evaluation to assess effectiveness of transition program.

a

A recommendation by the Equity Assistance Center (EAC) to address cultural relevance and competency.

To strengthen the final definitions and characteristics in Phase 3, the EAC reviewed and added language to some characteristics to address equity and/or diversity. Table 3 includes the 14 suggestions made. For example, one characteristic of the predictor Student Support read as follows:

• Develop and implement procedures for cultivating and maintaining school and community networks to assist students in obtaining their post-secondary goal.

EAC added the following language to this characteristic:

• Consider networks that are culturally, racially, and ethnically representative to accommodate the needs of CLD students.

Furthermore, EAC suggested that every predictor and characteristic consider its impact on students from CLD backgrounds; being mindful that use of mainstream value-based approaches may not serve the needs of all students. This recommendation is supported by Trainor et al. (2008) who suggested a need for increasing educators’ cultural competence and recommend that “educators consider student’s culture and communities in transition planning and service delivery” (p. 62).

Limitations

When considering these findings, four limitations need to be taken into account: (a) selection of experts, (b) response rate, (c) online communication, and (d) potential researcher bias. First, a recognized limitation of any study utilizing a Delphi procedure is the selection of participants identified to lend their experience and expertise to the research (Linstone & Turoff, 2002; Schmidt, Lyytinen, Keil, & Cule, 2001; Welty, 1972). In spite of adhering to strict inclusion criteria to ensure diversity in content expertise (e.g., CTE, special education), application of knowledge (e.g., researchers/academic and public school educators/practitioners), level of application (state department of education and local school district educators), and those with specific knowledge related to the 16 predictors of post-school success, selection of experts was limited by our personal knowledge of individual experts’ research focus, their years of experience, and professional work. Selection of experts was not based on a broad or comprehensive survey of such professionals (e.g., membership of professional organizations associated with those fields) or analysis of their individual contributions; therefore, we can make no statement about how experts’ opinions would generalize to other experts or practitioners.

Another limitation of this study was the reduced and variable response rates from experts through the rounds of the study. To encourage experts to remain active and complete all rounds of the Delphi study, personal emails were sent inviting individuals to participate. During each consecutive round, we sent multiple emails reminding participants of the goal of the study and the importance of their contribution. Previous literature using Delphi procedures suggests decreasing response rates after each consecutive round of voting is not uncommon (Linstone & Turoff, 2002). We checked respondents after each round to ensure individuals representing each of the inclusion criteria (e.g., practitioners with 10 or more years of services to secondary-age youth with disabilities or CTE, authors/researchers of scholarly, peer-reviewed work relative to predictors of post-school success, post-school outcome categories, or one or more specific youth populations) participated in all phases of the study. To increase response rate from the last round of voting and maintain a steady rate at or above 50% for the remainder of the study, we included all 22 experts in Rounds 5 through 7. Throughout each round of voting, experts could add or revise a characteristic, include an argument for or against including a characteristic, or advocate for specific wording of a characteristic. In this way, experts were afforded the opportunity to have their voice heard even when they had not participated in a previous round of voting. This flexibility also ensured experts had the opportunity to add or elaborate characteristics should they realize something was missed previously.

Third, all interactions with experts were through electronic survey and e-mail; intent of edits could only be articulated in writing. Therefore, meaning expressed through voice or facial expression could not be discerned as neither focus groups nor interviews were conducted.

Finally, although we were mindful to not introduce our personal, professional knowledge into the study by not adding or recommending definitions or characteristics and took steps to be transparent when changing any wording, no research study is completely free of bias. It is likely that other researchers reading the same suggestions might have edited the responses differently. However, all edits were approved by our respondents. In spite of these limitations, there are key implications for practitioners and other researcher to consider based on the findings of this study.

Implications for Practice

Operationally defining the programmatic characteristics of the predictors for post-school success may narrow the research-to-practice gap by giving educators information to align secondary transition programs with high quality research shown to increase the likelihood of positive post-school outcomes for youth with disabilities. The operational definitions and program characteristics can be used to develop and expand secondary transition programs, and/or evaluate existing programs. The findings can assist schools and districts in identifying whether they are implementing practices that have been empirically determined to influence change in the areas identified (e.g., post-school outcomes data, in-school outcome data). Doing this will enable schools to invest in providing transition services that will have the best chance of improving students’ post-school outcomes, therefore ensuring the biggest return for resources (e.g., time, effort, finances) invested. Implementing each of the predictors will require a collaborative effort among state and district administration, teachers, and other school staff. Some predictor program characteristics will require action at a state or district level. For example, when examining the program characteristics of vocational education, the following characteristic clearly requires action at a state/district administrative level and will affect at a systems level:

• Provide opportunities to earn certificates in certain career areas (e.g., Certified Nursing Assistant, Welding, Food Handlers Certification).

Other program characteristics for vocational education only require action on the part of a teacher and impact at a student level. For example,

• Provide accommodation and supports in CTE courses to ensure student access and mastery of content.

Multidisciplinary teams should work together to determine to what extent each predictor and program characteristic is being implemented and who will be responsible for developing an action plan to implement each program characteristic.

In addition, although each characteristic is essential to fully implementing a predictor, a multidisciplinary team should make decisions regarding priorities for action based on student performance data, resource availability, policy alignment, and other data available to a team. Certain characteristics may be of greater importance than others for an individual team. There may also be additional program characteristics not identified in this study given the state/local context.

Recommendations for Further Research

This study moves the field of secondary transition another step forward by establishing operational definitions and program characteristics of the predictors with sufficient detail for educators to develop, implement, and evaluate programs. Nevertheless, more work is needed. Researchers need to conduct long-term experimental research verifying the characteristics identified by the panel of experts are accurate for real classroom settings and for a variety of students. A survey to a broader, more comprehensive group of participants could validate as well as confirm or deny the generalizability of the characteristics identified by the small number of experts participating in this study. Empirical research needs to be conducted to determine the effectiveness of the program characteristics, and, what, if any revisions are needed to distinguish between essential and ancillary characteristics. Ongoing review of the literature found evidence to support a 17th predictor (i.e., parent expectations) of post-school success. As new predictors are identified, this procedure will need to be repeated to determine their corresponding operational definition and program characteristics.

In conclusion, the purpose of this study was to clarify the existing definitions of the 16 predictors of post-school success (Test, Mazzotti, et al., 2009), and where necessary, operationally define them in such a way that local educators know what is necessary to develop, implement, and evaluate secondary transition programs based on predictor research. Results indicated experts in the field of secondary transition reached consensus on an operational definition and a set of essential program characteristics for each of the 16 predictors. Policymakers, administrators, and practitioners now have information to assist them in providing evidence-based programs and services to youth with disabilities to prepare them for the transition from school to post-school life.