Conceptualizing and Developing the Secondary Transition Fidelity Assessment to Support Transition Programming and Practice

Abstract

In this article, we describe the conceptualization and development of the Secondary Transition Fidelity Assessment (STFA), a measure intended for use by high school leadership teams to examine secondary programs and practices demonstrated by research to lead to meaningful outcomes for secondary students with and without disabilities. We describe our conceptualization and development efforts, which resulted in a measure with six critical features: adolescent engagement, district-level capacity, school-level capacity, community engagement, family engagement, and professional capacity. Based on stakeholder feedback, we revised the STFA and developed an instruction manual and action planning guide to support the self-evaluation of secondary programs and practices using data from the STFA. Recommendations for the field related to research, policy, and practice are provided.

The transition from high school to adult life is an important time for all youth. There is a pressing need to provide effective secondary supports and services with fidelity to ensure students achieve positive and valued adult outcomes (e.g., employment, postsecondary education, community participation, quality of life). Multitiered systems of supports (MTSS) has evolved in secondary schools with considerations given to academic, behavioral, and transition outcomes for all students, inclusive of youth with and without disabilities (Bohanon et al., 2016; Morningstar et al., 2018). However, the broader education field has historically associated “secondary transition” strategies as specific to youth with disabilities; when in actuality, secondary transition is reflected in the work of college and career readiness for all youth (Lombardi et al., 2018; Trainor et al., 2020). Parallel tracks of secondary school reform result in a dichotomy among educators, with college and career readiness efforts typically deemed only for youth without disabilities, while secondary transition services are often designated for students with disabilities (Morningstar et al., 2018). In response, secondary transition researchers have emphasized the importance of focusing interventions for youth with and without disabilities by merging secondary transition and college and career readiness through frameworks associated with academic, behavioral, and transition skills (Lombardi et al., 2018, 2020; Morningstar et al., 2017).

To further emphasize this point, the Individuals with Disabilities Education Improvement Act (2004) mandated transition services and supports for youth with disabilities to help ensure these students experience positive in-school and postschool outcomes. Eleven years later, the Every Student Succeeds Act (2015) reflected the need for all students to have a “firm foundation” to help ensure all youth are college and career ready upon high school graduation (U. S. Department of Education [U.S. DOE], n.d.). With this in mind, it is important to understand that silos continue to exist between general education and special education. Therefore, it is critical that districts, schools, and secondary education professionals work together to improve policies, programs, and practices for all students (Trainor et al., 2020). High schools across the nation are continually being asked to consider their curricula goals, methods of instruction, and relationships between teachers and students to help ensure all students are prepared to be contributing members of their communities (National Conference of State Legislatures, 2022). The idea that high schools can “do better” for youth with disabilities and other marginalized groups can reduce inequity within our education system and promote more equitable outcomes for all students (Braun et al., 2021).

However, challenges persist when designing, implementing, and evaluating secondary school programs. This ultimately impacts outcomes for all youth related to being prepared with the skills and experiences required to successfully transition into adult life (Mazzotti et al., 2021; Rowe et al., 2015). In particular, program fidelity is a critical issue required to ensure evidence-based secondary programs, initiatives, and practices are implemented as intended and are effective. It has been acknowledged that program fidelity is essential in supporting multiple stakeholders to design and provide educational interventions (Hill & Erickson, 2019; Rowe, Collier-Meek, et al., 2021). Furthermore, program fidelity is necessary for effectively implementing research-based secondary programs and practices within an MTSS framework (Fixsen et al., 2014; Wayne RESA, 2018). While research-based college and career readiness models exist in the literature, the framework described herein is unique in its inclusion of youth with and without disabilities, thereby ensuring all students are empowered with opportunities to learn academic, behavioral, and transition skills necessary for success after high school.

Need for a Secondary Transition Fidelity Assessment

To comprehensively and effectively utilize research-based secondary interventions, implementation science researchers stress the importance of focusing on (a) documenting whether programs and practices are implemented as intended (i.e., implementation fidelity); (b) using assessment data to operationalize factors influencing implementation; and (c) accounting for implementation influences through planning and analyzing the efficacy of the impact on outcomes (Booth et al., 2007; Cook & Odom, 2013; Fixsen et al., 2010). Over the last several decades, there has been an enhanced focus on implementing high-quality and evidence-based educational interventions; therefore, understanding and prioritizing research-based features, elements, and actions that reflect secondary transition programming is a necessary first step (Harn et al., 2013; Mazzotti et al., 2013). The field is replete with recommendations for ensuring youth with and without disabilities are college and career ready. Unfortunately, there has yet to emerge a fidelity measure to evaluate secondary school-level programs and practices that can be used by educational stakeholders to design, implement, and evaluate such programs (Hill & Erickson, 2019; Mazzotti et al., 2013). This means that there are shortcomings in the field because there are currently no existing secondary program-level fidelity measures that reflect guidelines for implementing comprehensive secondary programs and practices for youth with and without disabilities (Jaakkola, 2020; Mazzotti et al., 2021). Additionally, as high schools conduct needs assessments of their school-wide programs, a secondary program-level fidelity measure can help support this work by providing a comprehensive tool to examine current school-wide programs and practices (e.g., use of effective, evidence-based practices; career technical education; counseling services; response to intervention; U.S. DOE, 2016). A secondary program-level fidelity measure can also support high school reform efforts as schools across the nation are being (a) asked to evaluate and assess curricula, (b) review methods used to develop and implement programs and practices, and (c) identify new ways in which students can contribute in a meaningful way to the community at large (National Conference of State Legislatures, 2022). Therefore, it is imperative educators and practitioners have tools to identify critical features and elements of secondary programs and practices that they can operationalize and implement in schools to enable positive outcomes for all youth, inclusive of youth with and without disabilities (Mazzotti et al., 2021; Trainor et al., 2020).

Conceptualizing and Developing the Secondary Transition Fidelity Assessment

The focus of this article is to describe the process used in conceptualizing and developing the Secondary Transition Fidelity Assessment (STFA) and to share recommendations on how the STFA can be used in research, policy, and practice to improve secondary programs and practices. To date, the field of secondary education and transition does not have a measure that adequately assesses school-wide implementation of secondary programs and practices that are designed and utilized to support postschool outcomes for youth with and without disabilities. While fidelity measures exist that support decision-making related to tiers of support students receive within an MTSS framework (e.g., SWIFT Fidelity of Implementation Tool, SWPBIS Tiered Fidelity Inventory; Algozzine et al., 2014, 2017), the focus of the STFA is specific to programs and practices at the secondary level related to Tier 1 or universal supports that could be used to ensure all transition-age students are college and career ready. The STFA was developed in collaboration with the National Technical Assistance Center on Transition (NTACT), now NTACT: the Collaborative (NTACT: C), and the Council for Exceptional Children’s Division on Career Development and Transition (CEC-DCDT). Preliminary validity evidence for the STFA was established as part of this work (Lombardi et al., 2021). Our work included conceptualizing, developing, and validating the STFA to provide high school leadership teams (e.g., general and special educators, guidance counselors, administrators) with a self-assessment to examine secondary transition programs and practices demonstrated from research as leading to meaningful college and career outcomes for all students. Our process for developing the STFA was conducted over 4 years (2016–2020). As described subsequently, the STFA (i.e., critical features, descriptive elements, actionable items) was developed based on existing empirical research, revised based on stakeholder feedback, and finally, validated by users during the last phase (Hirschheim, 2008).

Using an iterative process, we collected feedback from special and general education stakeholders (i.e., researchers, practitioners, administrators) who reviewed drafts of the STFA and provided feedback on the conceptualized critical features, descriptive elements, and actionable items. The goal of this process was twofold: (a) to create a comprehensive measure to assess fidelity of secondary programs and practices for school teams to evaluate current practices, and thereby, prioritize and enhance their efforts toward successful outcomes for secondary students; and (b) to develop a fidelity measure for use within secondary education and transition that extends beyond special education and incorporates school-wide programs and services provided as Tier 1 or universal supports. In this way, the STFA has the potential to inform the entirety of secondary settings and, thus, impact services for all students, inclusive of students with and without disabilities. The STFA development process comprised four phases: (a) conceptualizing the STFA; (b) gathering special and general education stakeholder feedback related to the STFA critical features, descriptive elements, and actionable items; (c) pilot testing the usability of the STFA; and (d) conducting STFA item analysis. Each phase is described below. Our conceptualization and development process used a student-centered approach across each critical feature, descriptive element, and actionable item.

Phase 1: Conceptualizing the Secondary Transition Fidelity Assessment

The initial phase of STFA conceptualization was to convene a subcommittee from the CEC-DCDT Research Committee. Several members of the subcommittee team were affiliated with NTACT (now NTACT: C). Subcommittee membership also included current and past presidents of CEC-DCDT and a nationally representative group of researchers, all of whom previously worked as practitioners (special and general education [e.g., school inclusion specialists], rehabilitation, related services) supporting students with and without disabilities. There were approximately 30 subcommittee members over the course of the 4 years of the project. Members were from a variety of geographic locations, including Arizona, California, Connecticut, Delaware, Kansas, Kentucky, North Carolina, Oklahoma, Oregon, South Carolina, Tennessee, Texas, Virginia, Washington, and Wisconsin. The subcommittee convened in June 2016 to begin the STFA conceptualization process focusing on five steps:

• 1. identify additional team members integral in supporting the measurement development;

• 2. review existing school-level fidelity measures to inform the development process (i.e., SWIFT Fidelity of Implementation Tool, SWPBIS Tiered Fidelity Inventory; Algozzine et al., 2014, 2017);

• 3. connect with experts from the Center on Positive Behavior Interventions and Supports (PBIS; https://www.pbis.org/) and the SWIFT Education Center (https://swiftschools.org/), both of whom had developed exemplar fidelity measures associated with school-wide fidelity within an MTSS context;

• 4. examine specific components of existing school-level fidelity measures for organizational structure and intended purpose and scope (i.e., SWIFT Fidelity of Implementation Tool, SWPBIS Tiered Fidelity Inventory; Algozzine et al., 2014, 2017); and

• 5. review extant research to support development of the STFA critical features, descriptive elements, and actionable items (see Supplemental Table S1: Secondary Transition Fidelity Assessment Item Reference Table).

The five-step process resulted in a measure organized by six critical features, including adolescent engagement and development, district-level capacity, school-level capacity, community engagement, family engagement, and professional capacity. These critical features represented broad domains of secondary education known to impact student outcomes. Each critical feature was supported by descriptive elements associated with actionable items. The critical features reflect the highest order of the framework for considering those critical areas of secondary education and transition to support schools and are used as domain-level organizational concepts for evaluating programs, services, and practices for secondary youth. Within each critical feature, we identified descriptive elements targeting categories of supports and services to be utilized by school personnel when determining improvements. Then, for each descriptive element, we identified actionable items that detail how and what is necessary to operationalize change. For example, educational planning and student skill development were two descriptive elements identified within adolescent engagement. Within each descriptive element, each subcommittee team identified specific education practices considered as item-level descriptors. The critical feature—descriptive element—actionable item approach emulated existing school-level fidelity measures (Algozzine et al., 2014, 2017); current research on evidenced-based predictors of secondary education and transition (Mazzotti et al., 2016; Test et al., 2009); and emerging areas of focus for college and career readiness for youth with and without disabilities (Morningstar et al., 2017).

Additionally, our team recognized the importance of developing a rubric to operationalize implementation to support high school teams in completing the STFA and prioritizing areas for growth. Therefore, we developed scoring metrics that focused on levels of implementation for each item ranging from 0 to 2 (i.e., 0 = not implemented, 1 = partially implemented, 2 = fully implemented). The scoring metrics were aligned with prior exemplar fidelity measures and utilized a three-point Likert-type scale to enable secondary school personnel to consider the extent to which secondary education and transition programs, services, and practices are implemented. We also developed a list of possible school-wide data sources for which stakeholders could refer to when evaluating each critical feature, descriptive element, and actionable item. These data sources were designed to support high school teams in evaluating and scoring actionable items.

The aim of this process was to conceptualize a secondary school-level fidelity measure, the STFA, by identifying critical features of school-wide practices and corresponding relevant descriptive elements and actionable items. Our goal was to develop a measure that could be used by secondary educational stakeholders to effectively and efficiently evaluate secondary programs and practices that could potentially lead to postschool success. As mentioned, the STFA includes six critical features: adolescent engagement and development, district-level capacity, school-level capacity, community engagement, family engagement, and professional capacity, along with corresponding descriptive elements and actionable items. The STFA includes rubric scoring metrics and confirming data sources. Next, we describe our conceptualized STFA, including each critical feature, descriptive element, and actionable item (see Supplemental Table S1 and Lombardi et al., 2021).

Critical Feature 1: Adolescent Engagement and Development

The adolescent engagement and development critical feature is operationalized as, “adolescents are actively engaged with and supported by schools, families, and communities to develop their college and career readiness through affective, cognitive, and behavioral skills in school.” This critical feature includes two descriptive elements (i.e., educational planning; college and career ready skills) and a total of 24 actionable items associated with adolescent engagement and development.

Element 1.0: Educational Planning

Within this descriptive element, two subelements were conceptualized and developed: (a) meaningful participation in educational planning and (b) providing instruction, counseling, support, and guidance in secondary school experiences. Element 1.0 includes 10 actionable items. Examples of actionable items within adolescents meaningfully participate in education planning include adolescents (a) have opportunities to participate in meetings in which decisions are made concerning student in-school and postschool plans and (b) develop individual plans of study that reflect and support postschool career, educational, and adult life goals. Examples of items within adolescents are provided instruction, counseling, support, and guidance include adolescents (a) develop plans based on their interests, abilities, and college and career goals guiding course enrollment and school and community experiences (e.g., career internships, extracurricular activities); (b) explore a broad range of postsecondary education options; and (c) understand how diploma options may affect postsecondary goals (e.g., differences in rigor of courses and college options for students based on the course of study/diploma requirements).

Element 2.0: College- and Career-Ready Skills

Within this element, two descriptive elements were identified focusing on provision of services for adolescents and included 14 actionable items: (a) inclusive instruction and support and (b) instruction and support to prepare for adult roles and responsibilities. Examples of actionable items within adolescents are provided inclusive instruction and supports include adolescents (a) use strategies leading to the development of appropriate academic behaviors and critical thinking skills (e.g., learning strategies, study skills, test-taking skills, self-management, participation, organizational skills); (b) use technology to support college and career readiness (e.g., laptops/computers, mobile applications, calendars, digital tools, digital literacy); and (c) engage in communication and social-emotional development across school, work, and home (e.g., social media, online, email communication, employers, friends, family). Examples of actionable items within adolescents are provided instruction and support to prepare for adult roles and responsibilities include providing instruction in (a) financial literacy (e.g., budgeting, financial decision-making, financial management, assessing credit and debt, interpreting credit reports, managing checking and savings accounts, financial aid for college); (b) adult personal living skills (e.g., personal hygiene, dressing appropriately); and (c) health and wellness (e.g., exercise, personal habits, sex education, mental health, medical care managing medication, health insurance).

Critical Feature 2: District-Level Capacity

The district-level capacity critical feature is operationalized as, “the underlying district infrastructure and policies needed to support schools in the quality coordination, implementation and sustainability of evidence-based college and career ready programs and services that support postschool outcomes for all students.” This critical feature includes three descriptive elements (i.e., knowledgeable and engaged educational leadership, collaboration and community engagement, and evaluation/data-based decision-making) and nine actionable items (three per element) associated with the district’s capacity to implement and sustain secondary education programs and practices.

Element 1.0: Knowledgeable and Engaged Educational Leadership

This descriptive element includes three actionable items: (a) District leadership is knowledgeable of programming and services that support postschool outcomes for all students; (b) district leadership supports changes in programming and services that develop, implement, and coordinate inclusive educational experiences for all students; and (c) district leadership facilitates access to resources (e.g., financial resources, curriculum resources, building resources) to ensure inclusive educational experiences for all students.

Element 2.0: Collaboration and Community Engagement

This descriptive element also includes three actionable items: (a) District has established partnerships with institutions of higher education (i.e., 4-year colleges, community colleges, vocational schools, technical schools); (b) district has established partnerships with community employers to offer community-based learning experiences (i.e., work sampling, job shadowing, internships, paid work experiences); and (c) district has established partnerships with community-based employment organizations (i.e., vocational rehabilitation, workforce centers, adult service providers, recruitment offices).

Element 3.0: Evaluation and Data-Based Decision-Making

This descriptive element includes three actionable items: (a) District has developed and utilizes systems to collect and monitor data that will be used to make decisions about services and supports, including scaling up practices across schools, districts, and the state (program evaluation); these data should be aligned across initiatives relative to secondary education and transition (e.g., dropout, special education, Workforce Innovation Opportunity Act [2014], others relative to local context); (b) district has developed and utilizes a data-based decision-making model for using these data to identify which evidence-based practices and predictors are needed for implementation for improving current secondary education and transition services across policy initiatives; and (c) district identifies, reviews, aligns, and implements federal, state, district, and school policies related to secondary education and transition that support the implementation of accessible and culturally responsive secondary programs and services that support postschool outcomes for young adults and young adults with disabilities (e.g., school completion, career technical education, Workforce Innovation and Opportunity Act, special education, and others related to local context).

Critical Feature 3: School-Level Capacity

The school-level capacity critical feature is operationalized as, “the supports in place within a school that promote and enhance student-centered experiences and preparation for the transition to adult life for all students, which can include tiered systems of support in academics, behavior, social-emotional learning, and college and career readiness.” This critical feature includes two descriptive elements (i.e., leadership and collaboration among school systems, inclusive and ambitious instruction) with a total of 23 actionable items associated with promoting and enhancing student-centered experiences and preparing for the transition to adult life.

Element 1: Leadership and Collaboration Among School Systems

This descriptive element included six actionable items including, but not limited to, (a) the school leadership supports inclusive, safe, and positive schools; (b) school leadership supports positive and safe school culture that focuses on full access for all adolescents, engagement in learning, shared responsibilities, and success for all; and (c) a collaborative (school personnel and adolescents) comprehensive assessment of non-academic skills is conducted, and data are used to make informed decisions.

Element 2: Inclusive and Ambitious Instruction

Seventeen actionable items were identified for this descriptive element. Several focused on tiered interventions aligned to college and career readiness instruction. For example, (a) Tier 1 interventions and instruction (e.g., Universal Design for Learning, PBIS) accessible to all adolescents; (b) instruction across all three tiers (i.e., Tiers 1, 2, 3) includes critical thinking as it relates to non-academic skills and experiences relevant to college and careers; (c) instruction across all three tiers includes interpersonal engagement as it relates to non-academic skills and experiences relevant to college and careers; (d) adolescents are provided access to inclusive academic content (e.g., language arts, math, science, social studies); and (e) adolescents are provided multiple inclusive work-based learning experiences (paid and unpaid) based on their interests, preferences, needs, and strengths.

Critical Feature 4: Community Engagement

The community engagement critical feature is operationalized as, “the collaboration between K-12 institutions and community stakeholders (e.g., employers, adult service providers, parents, students, other community agencies) for the purpose of improving in-school and postschool outcomes for students.” This critical feature resulted in two descriptive elements (i.e., school-community relationships and school-community communication) with eight items between the two descriptive elements.

Element 1: School-Community Relationships

Five items were identified for this descriptive element and included (a) schools and community stakeholders, including parents and students, have a shared vision and common goals; (b) build multiple partnerships across various community stakeholders; (c) schools and community stakeholders provide resources to support community partnerships (e.g., space, technology); (d) engage community stakeholders as key decision-makers in the development and implementation of secondary education and transition programs; and (e) develop the capacity of school staff (e.g., administration, teachers, paraprofessionals) to engage in partnerships with community stakeholders.

Element 2: School-Community Communication

Three actionable items were identified for this descriptive element. The items included (a) schools engage in ongoing two-way communication with community stakeholders to ensure agreed upon vision and goals are being met; (b) schools use virtual communications (e.g., emails, blog posts) and traditional methods (e.g., mailing, flyers, newsletters) to inform the community and listen to their feedback; and (c) schools respond to community stakeholder questions and concerns via conversations, public forums, surveys, and focus groups.

Critical Feature 5: Family Engagement

The family engagement critical feature is operationalized as, “interactive and persistent family behaviors committed to supporting students’ learning and development and supportive of collaborative school efforts, including familial roles as agents and advocates on behalf of their children, capable of resisting barriers, imbalances, and exclusionary actions toward students and families from non-dominant communities.” This critical feature incorporated two descriptive elements (i.e., knowledge and capacity of school staff and support to families) with a total of 13 actionable items across the two.

Element 1: Knowledge and Capacity of School Staff

This descriptive element includes four actionable items: (a) School develops the transition knowledge and skills of stakeholders (e.g., school faculty, staff, related service providers) to engage with families; (b) school adopts policies and procedures for supporting families of students with disabilities from all backgrounds; (c) accountability procedures are in place to encourage school staff to treat families as assets and equal partners in transition planning and decision-making; and (d) staff build trusting and respectful relationships by engaging in two-way communication (e.g., verbal, written).

Element 2: Support to Families

Eight actionable items are associated with this descriptive element. These include, but are not limited to, (a) school provides support to families in the development of their different roles in secondary schools and transition (e.g., instructors, evaluators, advocates, collaborators, decision-makers); (b) staff acknowledge and value the diverse ways families engage in planning and preparing for their youth’s transition from high school; and (c) school builds family efficacy by providing knowledge and tools for families to use to build their confidence in supporting their student’s emotional, physical, and learning needs.

Critical Feature 6: Professional Capacity

Professional capacity is operationalized as, “professional development (PD) and technical assistance (TA) systems in place to ensure all secondary personnel have the capacity to effectively implement research-based secondary policies, programs, and practices to ensure all students are college, career, and community ready.” This critical feature includes three descriptive elements (i.e., capacity to provide quality TA and PD, data-driven PD, quality staff) across 15 actionable items.

Element 1: Capacity to Provide Quality TA and PD

This descriptive element includes nine actionable items: (a) process in place to provide PD and TA to school personnel (i.e., career technical education [CTE], special educators, general educators, transition specialists) on current secondary transition policies and practices (e.g., secondary evidence-based practices and predictors focused on academic, behavioral, transition skills); (b) process in place to provide PD and TA to school personnel (e.g., CTE, special educators, general educators, transition specialists) on data-based decision-making (e.g. analyze and make changes based on data collected on student achievement); and (c) a structure exists that is dedicated to ensuring new content is implemented with fidelity (e.g., follow-up, coaching, and mentoring).

Element 2: Data-Driven PD

This descriptive element includes three actionable items: (a) State-level and district-level data are used quarterly/annually to identify gaps and areas of PD and TA need-related secondary policies, programs, and practices; (b) school-level data are analyzed quarterly/annually to determine specific focus for PD and TA; and (c) school-level data are used quarterly/annually to determine specific focus for PD and TA.

Element 3: Quality Staff

The third descriptive element includes three actionable items: (a) Staff are using knowledge and skills gained from PD and TA to embed and implement secondary evidence-based secondary programs and practices (EBPPs) consistently with students across environments, (b) staff are using data monthly to make decisions relative to providing EBPPs to students, and (c) process in place for teacher evaluation that includes multiple evaluation measures of teachers', administrators', and staff's competence and performance (e.g., student outcome data, observation, coaching notes, survey, interview).

Once the critical features, descriptive elements, actionable items, and scoring metrics were conceptualized based on expert input, examination of existing school-level fidelity measures, and extant review of the research, our next step in our STFA development process was to gain feedback from stakeholders. This included conducting feedback sessions followed by pilot testing in Phase 3.

Phase 2: Gathering Stakeholder Feedback

We held feedback sessions with secondary education stakeholders (e.g., researchers, general and special educators, guidance counselors, school administrators). Stakeholders worked in a variety of settings including higher education, state and local education agencies, public and charter middle/high schools, and community transition programs. Stakeholders were invited to participate in discussion groups through a variety of ways including at national and state conferences, within professional networks, and via personal contacts. The subcommittee team identified participants who stated an interest in the development of the STFA.

Feedback sessions began in fall 2017 and continued through Spring 2019. First, we conducted in-person feedback sessions with secondary special education and transition educators during two national conference presentations. These included two sessions (Boston, MA) and 2018 (Tampa, FL) conferences, as well as CEC-DCDT’s 2017 International Conference in Milwaukee, WI. We also conducted feedback sessions at state conferences, including the 2018 and 2019 Oregon Transition Conference (Hood River and Eugene, OR), as well as the Arkansas Transition Services Cadre (Little Rock, AR; February 2018). To gain input outside of secondary special education and transition, we conducted a similar stakeholder session at the Northeast PBIS Conference in May 2018. This conference allowed us to gain information from general education stakeholders and included secondary general and special educators. To extend our stakeholder groups, we conducted two targeted feedback sessions with school counselors (primarily serving students with and without disabilities), secondary general and special education teachers, and secondary administrators. These occurred with school counselors in Arkansas and secondary general and special education teachers and administrators at an alternative high school in Kentucky.

During all feedback sessions, we first provided an overview of the purpose and goals of the STFA and discussed initial conceptualization and development procedures and the need for stakeholder input. Then, we assigned session participants to small groups to provide input on (a) the research-informed descriptions for each critical feature, its corresponding descriptive elements, and associated actionable items; (b) the scoring rubric associated with the STFA; (c) possible school-level data sources teams might use to support their score; and (d) overall usability of the measure. Table 1 includes guiding questions used during these feedback sessions. Our subcommittee team assigned a facilitator who guided the discussion and a notetaker who recorded each group’s input. We also used a “ticket out the door” survey to gain additional information from participants related to demographics (i.e., title, work setting, age group of students served); overall feedback on usability (i.e., Would you use it? If no, why not?); and interest in participating in subsequent STFA field tests.

Table 1.

STFA Stakeholder Questions for Feedback Sessions.

Definitions and descriptive elements

1. Does the definition sufficiently describe the critical feature? 2. Is each element relevant to the critical feature?

Actionable items

1. Does each item align clearly with the related element? 2. Are there items that are redundant or unnecessary?

Rubric

1. Are the items measurable based on the rubric scale? 2. How hard will it be to evaluate each item?

Possible data sources

1. Are the data sources generalizable across students? 2. Are there other sources of data that should be included?

Other questions

1. Is the content generalizable across students and in alignment with Tier 1? 2. What is confusing? Missing? Needs to be changed?

STFA = Secondary Transition Fidelity Assessment.

During these sessions, we received an abundance of feedback from stakeholders on our critical features’ operational definitions and scoring metrics. After each discussion, all notes were combined into a single document, organized by input across each of the critical features and their corresponding descriptive elements and actionable items. Then, all the information was analyzed and synthesized to inform refinement of the STFA, the scoring metrics, and possible data sources. Prior to the next session held, drafts of the STFA were updated based on prior input, and resulting iterations of the measure were shared with subsequent groups.

Development of the STFA Instructions

During our feedback sessions in Phase 2, stakeholders recommended we provide supporting resources, including instructions and an action planning guide for using the STFA (see online Supplemental Figure S1: Secondary Transition Fidelity Assessment Instructions and Action Planning). This was due largely to stakeholders noting the importance of clear guidance to ensure the STFA was scored appropriately, operationalized, and implemented with fidelity. Feedback from stakeholders indicated school teams would need clear instructions to complete the STFA as well as support to develop an action plan based on areas for growth. Such guidance would better ensure teams could effectively use STFA data to improve programs, practices, and services for youth across the six critical features. Therefore, our subcommittee team developed instructions and an action planning guide to support implementation. The instructions include information about the process a school leadership team (e.g., administrators, secondary general and special educators, guidance counselors, students, family members, community partners) would use to complete the STFA. It also includes information related to four major steps in this process: (a) identifying team members, (b) establishing a location and time for the meeting, (c) preparing for the meeting, and (d) the process for conducting the meeting with the high school leadership team (Bovend’Eerdt et al., 2009; Day & Tosey, 2011; Kiresuk et al., 1995; Mazzotti et al., 2018; National Post-School Outcomes Center, 2013).

Development of the STFA Action Planning Guide

The action planning guide provides stakeholder teams with instructions for how to use the results of the STFA to develop an action plan to address areas for growth related to secondary transition programs and practices, including (a) determining additional key individuals (beyond the secondary school leadership team) who should be involved in planning to successfully influence school change; (b) developing the action plan composed of steps addressing proposed changes; (c) identifying steps for remedying the problem with a direct link to the data-based decision-making; (d) developing goals that are specific, observable, and measurable; (e) reviewing the completed action plan; (f) ensuring follow through; and (g) evaluating the team’s efforts (Bovend’Eerdt et al., 2009; Day & Tosey, 2011; Kiresuk et al., 1995; Mazzotti et al., 2018; National Post-School Outcomes Center, 2013). See online Supplemental Figure S1: Secondary Transition Fidelity Assessment Instructions and Action Planning.

Phase 3: Testing the Usability of the STFA

Usability testing began in fall 2018 and consisted of convening high school teams to complete the iteratively revised version of the STFA and provide user feedback. The first usability test was conducted during two 1-hour sessions with stakeholders in a midwestern, suburban public high school. A subcommittee team member recruited stakeholders who worked with students with and without disabilities and supported students engaged in the transition from high school to adulthood including a school psychologist, general education teacher, special education teacher, school counselor, assistant principal, and transition coordinator. Two research team members completed the usability session, with one facilitating the school team through the process of completing and subsequently discussing the STFA while the other took detailed notes. The first session was structured to reflect the intended process and procedures a school team would use when completing the STFA. The team discussed each critical feature, descriptive element, and actionable item and agreed upon a team rubric score based on possible school data (e.g., school graduation rates, postschool employment rates, school climate data). A second session was conducted specifically with the team’s transition coordinator because she was unable to join the team meeting. During this second session, one research team member met with the transition coordinator to gain additional feedback of usability.

A second team user test occurred with stakeholders in another midwestern state. This included regional transition consultants meeting with special education teachers to discuss the STFA. The regional transition consultants were trained by a subcommittee team member (lead author) on the purpose and parts of the STFA to facilitate gathering input from their special education teachers. Five regional transition consultants met with a special education teacher in their region, and together, they completed the STFA for their school during which the five pairs discussed the STFA. These sessions varied in length from 30 to 45 minutes and did not include other school team members. The regional transition consultants provided written feedback of the results of these sessions to the subcommittee team. This included specific guidance regarding the STFA items and implementation procedures. Based on user testing, minor revisions were made to the STFA and supporting resources.

Phase 4: Conducting the STFA Item Analysis

This phase began in fall 2019 and focused on completing an item analysis for the STFA, which was completed in spring 2020. We were interested in statistically testing the STFA to gain a better understanding of which items were closely related, as well as investigate the dimensionality of the STFA (Lombardi et al., 2021). We obtained institutional review board approval and recruited over 1,500 participants to complete the STFA via an online survey. Our target participants were administrators, special education personnel, secondary general education personnel, guidance counselors, as well as other stakeholders whom the school deemed relevant. Our recruitment included emails to school administrators, posts on social media (e.g., Twitter), and listserv recruitment (e.g., NTACT, Zarrow Center, DCDT, Transition Coalition). All recruitment efforts included information about the STFA, our purpose for the survey, and online links to the survey and informed consent. For the purpose of this paper, we focus only on our initial development. Results of the item analysis are reported in the study by Lombardi et al. (2021). Based on our conceptualization and development of the STFA, the feedback from stakeholders, the usability testing, and the final item analysis, we were able to make iterative revisions with confidence in the finalized STFA.

Recommendations for Research, Policy, and Practice

The purpose of the STFA is to provide secondary school leadership teams (e.g., special and general educators, guidance counselors, administrators) with a self-assessment to examine their high school’s implementation of secondary programs and practices demonstrated by research to lead to meaningful college and career outcomes for all students. Specifically, the focus of this article was to describe the conceptualization and development of the STFA and make recommendations to the field to guide research, policy, and practice. Our process resulted in identifying critical features, descriptive elements, and actionable items relevant and necessary to ensure stakeholders effectively and efficiently evaluate secondary programs and practices. The process concluded with the STFA; a measure ready for item analysis. The final measure included six critical features (i.e., adolescent engagement, district-level capacity, school-level capacity, community engagement, family engagement, professional capacity; see Supplemental Table S1 and Lombardi et al., 2021). Based on stakeholder feedback, we also developed instructions and an action planning guide to support use and implementation of the STFA, thereby ensuring high school leadership teams could use the STFA with fidelity. We provide recommendations for research, policy, and practice as they relate to the field of secondary education and transition.

Recommendations for Research

We recognize further research is needed to refine and validate the STFA. This may include comparing state-level data to determine the efficacy of the measure throughout a cycle of program evaluation. Given the initial development of the STFA was limited to certain regions constrained by where conferences were held (Lombardi et al., 2021), it would be essential to expand further development to include a broader reach of stakeholders. It is evident that the majority of our stakeholders who provided input into development were special educators; therefore, further development should include greater input from general education teachers and other school personnel who work with students with and without disabilities to ensure the STFA is “truly” a comprehensive measure to assess fidelity of secondary programs and practices. We also recognize the measure may not sufficiently reflect unique cultural contexts relevant in schools today given the constraints associated with a clear focus on recruiting participants from diverse communities and schools. The convenience sample used during the stakeholder feedback and user testing phases limited our capacity to ensure diverse representation of experiences and context. Clearly, future research is needed to investigate how this measure supports or limits cultural responsiveness of school communities, families, and students.

Our preliminary evidence supports using the STFA as a validated measure with the potential to evaluate secondary education programs and practices. It is important to note that the STFA was validated for use at the school level by practitioners in the field of secondary education and transition. Future research is needed to determine if the STFA can be utilized as intended—that is with teams of school personnel in diverse contexts. Therefore, an essential next step is to design a development study using the STFA in practice with diverse school teams that include both special and general educators. This study should focus on testing the capacity of the STFA to be used broadly by general and special education teachers, schools, and districts to evaluate the practices, services, and supports being provided to secondary students with and without disabilities. A careful analysis of the reliability and validity of team scores is needed to ensure the STFA is used with fidelity. In addition, it will be important to ensure the STFA is validated for implementation with local and state education agencies. This would allow for the instrument to be used to support high school reform and school-improvement efforts more globally at the state and district levels.

Finally, it will be important to examine the effectiveness of using the STFA in practice. This will include evaluating how teachers, schools, and districts facilitate stakeholder teams in prioritizing the results of the STFA to implement changes in programs and practices impacting students, thereby improving outcomes for all youth. This will allow stakeholders to truly understand the two main aims of the STFA: (a) how results from the STFA can inform multitiered interventions and programs at the Tier 1 level; and (b) how the results from the STFA can be analyzed by school personnel to identify and utilize effective supplemental (Tier 2) and intensive (Tier 3) interventions to improve college and career readiness for all students. It will be especially important to examine the relationship between STFA scores and student outcomes. Student academic and behavioral data routinely collected by schools could be one data source to consider (e.g., attendance, course grades), as are student outcomes after school. Measures of student perceptions that are appropriate for youth with and without disabilities are also important to consider, such as school-wide college and career readiness (Lombardi et al., 2021) and school climate (Rifenbark et al., 2021).

Recommendations for Policy

It is possible for local and state education agencies to utilize the STFA to aggregate data across schools and districts to identify the fidelity of implementation and impact of secondary programs and practices that are research-driven and associated with meaningful college and career outcomes. Adopting the STFA at state and district levels may provide a consistent school-wide measure to help ensure all secondary schools are evaluating, developing, and/or improving programs and practices for all students. However, it is important to keep in mind that adopting the STFA should be done systematically to confirm scalable implementation across schools and districts. The guidance manuals accompanying the STFA can be used as a part of school-wide PD. In addition, state and district leaders can pinpoint research-informed PD in areas where gaps are identified by the STFA (Rowe, Mazzotti, et al., 2021). For example, if the school-level capacity critical feature is identified as a focus area with Element 2: inclusive and ambitious instruction identified as not being implemented, school leaders can support PD on effective inclusive instruction and guide educators to effectively implement instruction and support in inclusive settings rather than segregated ones. By examining aggregated data, operationalizing effective PD, and tracking improvement via the STFA, this measure may serve as a cornerstone to school improvement and strategic planning efforts at the student, classroom, school, and/or district levels.

We intentionally identified critical audiences and participants (e.g., secondary special and general educators, school counselors) to provide feedback throughout the development of the STFA. Our work was generative in nature, focusing on both verbal and written feedback from the wide array of stakeholders in a variety of ways. This process may have contributed to missing information if certain stakeholder groups were not present. We recognize that the majority of our input came from a special education lens, and moving forward, it will be important to gain input from general education teachers and other school personnel (e.g., administrators, guidance counselors) about the utility of the STFA. However, the phases of our work were intentional, in that we first operationally defined critical features using research-informed information and the evidence base. We then gathered feedback and input from potential users and, finally, completed a statistical analysis of the items to confirm the overall structure and dimensions of the measure. Therefore, the STFA may be considered a potentially powerful fidelity tool that can support and encourage systematic program evaluation and planning to support systems change. If the STFA is broadly used to measure secondary school programs and practices across special and general education programs, it is our expectation that it has the potential to support high schools in facilitating college and career readiness for all students.

Recommendations for Practice

As mentioned previously, MTSS has emerged and is prevalent in secondary settings and should focus on the entirety of the student body (Bohanon et al., 2016; Morningstar et al., 2018). Moving beyond a siloed approach to secondary transition for students with disabilities, it is imperative districts and schools consider transition for all students. Development of the STFA may offer the secondary education community with a valid measure to evaluate Tier 1 secondary programs and practices more likely to lead to (a) effective program fidelity, (b) implementation of evidence- and research-based practices, and (c) access to necessary programs for all youth. This is an essential consideration when using an MTSS framework (Fixsen et al., 2014; Wayne RESA, 2018).

The STFA is a measure that secondary educators and school-improvement teams may consider as a capacity-building tool (Lombardi et al., 2021). As a part of our development process, two school teams pilot tested the measure and found the information to be efficacious for evaluating their secondary programs. Using the instructions and action planning guide offers school teams necessary tools to not only evaluate secondary programs but also to make a plan for actionable change, thereby helping ensure youth with and without disabilities are provided evidence-based and research-based programs and practices.

As secondary schools become more ethnically and racially diverse, it is critically important to evaluate programs based on the cultural context of the school and ensure secondary program teams incorporate a focused lens on diversity, equity, and inclusion (Trainor et al., 2020). It will be necessary for stakeholder teams to consider reviewing the results of the STFA as well as student-level data to identify where inequities exist across student demographics, grade levels, and classes. Understanding these gaps will support teams in developing and implementing program goals and improvements that will positively impact student outcomes. SMARTIE (Strategic, Measurable, Ambitious, Realistic, Timebound, Inclusive, and Equitable) goals could be used to support this process and will help ensure program goals reflect diversity, equity, and inclusion (The Management Center, 2022). In addition to school-based, student-level data, other measures can be used to support the evidence that teams access when scoring the STFA. For example, new measures such as the college and career readiness for transition can provide needed information about student readiness for college and careers and can be disaggregated by race/ethnicity, disability, and gender (Lombardi et al., 2022).

Conclusion

The STFA is grounded in evidence-based and research-informed literature describing secondary programs and practices. It is intended to support educators to consider fidelity of implementation of programs, an important tenet of MTSS. Research suggests that evaluating secondary programs resulting in postschool success is not a widely held practice. Additionally, secondary transition programs are typically siloed when it comes to including students with disabilities in this process (Lombardi et al., 2018, 2020; Morningstar et al., 2017). Therefore, having a reliable and valid assessment, like the STFA, to use as a program evaluation tool may help to identify gaps and areas of weakness with regard to supporting all high school students. Furthermore, states and districts might endorse the STFA in a similar vein. Regardless of school, district, or state level, the STFA could be part of strategic planning efforts to improve college and career readiness opportunities for all secondary youth.

Supplemental Material

sj-docx-1-cde-10.1177_21651434231163498 – Supplemental material for Conceptualizing and Developing the Secondary Transition Fidelity Assessment to Support Transition Programming and Practice

Supplemental material, sj-docx-1-cde-10.1177_21651434231163498 for Conceptualizing and Developing the Secondary Transition Fidelity Assessment to Support Transition Programming and Practice by Valerie L. Mazzotti, Mary E. Morningstar, Allison Lombardi, Sheida K. Raley, Dawn A. Rowe, Marcus Poppen, Kyle Reardon, Sarah R. Carlson, Deanne Unruh, Jessica Monahan, Jessica G. Rousey, Janie N. Vicchio and David W. Test in Career Development and Transition for Exceptional Individuals

Supplemental Material

sj-docx-2-cde-10.1177_21651434231163498 – Supplemental material for Conceptualizing and Developing the Secondary Transition Fidelity Assessment to Support Transition Programming and Practice

Supplemental material, sj-docx-2-cde-10.1177_21651434231163498 for Conceptualizing and Developing the Secondary Transition Fidelity Assessment to Support Transition Programming and Practice by Valerie L. Mazzotti, Mary E. Morningstar, Allison Lombardi, Sheida K. Raley, Dawn A. Rowe, Marcus Poppen, Kyle Reardon, Sarah R. Carlson, Deanne Unruh, Jessica Monahan, Jessica G. Rousey, Janie N. Vicchio and David W. Test in Career Development and Transition for Exceptional Individuals