Abstract
Transition services, particularly those that feature work-based learning experiences, often require designated staff to spend the majority of their time in the field. They also require that staff have the skills and supports to properly and effectively deliver these services. Training and monitoring these staff is critically important to ensure youth with disabilities experience strong employment outcomes. Maryland PROMISE was a statewide experimental, multicomponent, and community-based transition project that served 997 youth receiving Supplemental Security Income and their families. To ensure that staff effectively delivered services, Maryland PROMISE provided carefully designed professional development activities that included training, field-based coaching, and performance monitoring. This article reports on the Maryland PROMISE capacity building activities for project staff, provides data on the results of that process, and offers implications for the effective delivery of community-based transition services.
Keywords
Preparation of students with disabilities for the transition from school to adult employment is increasingly focused on services and activities external to the classroom. This is particularly the case for those services that include learning experiences occurring at worksites in the community. The body of evidence continues to grow that shows work-based learning experiences while in high school strongly predict post-school employment success for students with disabilities (e.g., Carter et al., 2010; Luecking et al., 2018; Wehman et al., 2014). These findings are reinforced by the Workforce Innovation and Opportunity Act (WIOA) of 2014 (Pub. L. 113–128) which requires state vocational rehabilitation agencies to prioritize provisions of pre-employment transition services for students before exiting high school. These services include job exploration counseling, work-based learning experiences, work readiness training, counseling on transition or post-secondary education, and instruction on self-advocacy (Workforce Innovation and Opportunity Act [WIOA], 2014). Collaboration between schools and vocational rehabilitation is encouraged to facilitate and carry-out work-based learning experiences.
By definition, the development and delivery of work-based learning experiences require transition service staff to spend the majority of their time in the field identifying, negotiating, and supporting work-based learning experiences with and on behalf of students. The implementation of work-based learning experiences also requires that staff have the knowledge, skills, and support to deliver these community-based services properly and effectively. For that reason, training and monitoring staff is critically important to ensure strong employment outcomes for students and youth with disabilities. Studies have found, however, that transition staff preparation and training is significantly lacking. For example, Mazzotti and Plotner (2016) found that over one half of transition service staff surveyed from the fields of education, vocational rehabilitation, and workforce development seldom or never received training regarding evidence-based employment services. Similarly, Neubert et al. (2018) found that vocational rehabilitation counselors serving transitioning youth identified work experience as an important service but considered themselves unprepared to deliver those services. Overall, studies continue to identify inconsistent transition training quality and content (e.g., Holzberg et al., 2018; Morningstar & Mazzotti, 2014).
Based on the need for staff to work autonomously in the community, monitoring the performance of field-based staff has been identified as an ongoing challenge in transition programs, community employment services, and any human service endeavor. For example, a study of job development practices found that staff autonomy often resulted in considerable variation in both staff activity and client outcomes (Migliore et al., 2010). Similarly, a study by Fabian et al. (2012) found job placements were weakest among those job developers who lacked or did not seek assistance or support from professional networks. Not only have other studies identified similar challenges in vocational rehabilitation programs (e.g., Oertle et al., 2017; Plotner & Simonsen, 2018), the relationship of staff autonomy to accountability and performance is consistently raised as an issue in other types of public sector and human service programs (Arnaboldi et al., 2015; Gruman & Saks, 2011; Kuipers et al., 2014).
These challenges can potentially adversely affect the fidelity and effectiveness of field-based transition interventions. Efforts to address these challenges have been reported in the implementation of large-scale transition models. For example, Fraker et al. (2018) reported promoting implementation fidelity through the combination of self-reported quantified staff activity, targeted training activities, and direct technical assistance (TA) to field staff in the implementation of the National Youth Transition Demonstration (YTD). The authors reported that these strategies were instrumental in ensuring that work-based learning experiences, the key service components of YTD, were consistently facilitated for participating youth. The resulting impact analysis of the project showed a significant positive difference in employment outcomes between youth who received the intervention and youth who did not.
These fidelity practice factors were considered in the development and implementation of Maryland Promoting Readiness of Minors in Social Security Income (PROMISE). Maryland PROMISE was a statewide large-scale experimental, multicomponent, and community-based transition project which served 997 transition-aged youth receiving Supplemental Security Income (SSI) and their families. Maryland PROMISE sought to increase participant employment and financial outcomes as a result of the intervention and included four service components: (a) assertive case management, (b) unpaid and paid work-based learning experiences, (c) benefits counseling and financial literacy services, and (d) family training and information.
We defined work-based learning experience as any activity that placed youth into community-based employment settings, where they learned about careers and developed appropriate worksite behaviors. Work-based learning experience activities included brief and episodic opportunities, such as worksite tours, job shadowing, and informational interviews, to more intensive opportunities, such as volunteer work, service learning, and internships. Paid employment, including customized employment, was defined as full or part-time positions where youth were hired directly by the employer, earned at least minimum wage, and were offered benefits similar to those for employees without disabilities. A team of professionals collaborated to deliver these services with individual staff assigned to particular components. Maryland PROMISE employment specialists were tasked with identifying, negotiating, and supporting work-based learning experiences in local businesses in the community.
To ensure staff effectively delivered work experience services to participating youth, Maryland PROMISE provided carefully designed professional development activities that included training and field-based coaching, established service and outcome expectations for staff, as well as an associated performance monitoring process which featured an intervention report, a fidelity report, and a performance matrix. Therefore, the purpose of this article is to (a) describe the performance improvement methods and strategies implemented to build Maryland PROMISE staff capacity to deliver services that yield work-based learning experiences and paid employment outcomes for participating youth and (b) present preliminary data on the effect of these strategies on improving staff performance in the delivery of employment services in the community. To determine the effect of Maryland PROMISE capacity-building strategies, we asked the following research questions:
Method
Sample
The Maryland PROMISE intervention was organized through a team of professionals including an employment specialist, case manager, and benefits specialist in collaboration with school personnel. These individuals worked cooperatively and intensively to deliver Maryland PROMISE services to the 997 participating youth and their families. There were 27 intervention teams working in five geographic regions across Maryland (i.e., Baltimore City, Eastern Shore, Northern, Southern, and Western). Each intervention team managed an average caseload of 44 youth. The primary role of the employment specialist was to facilitate work-based learning experiences with input from other team members about the youth’s interests, preferences, identified skill levels, and family circumstances that might inform work experience development. For this non-experimental study, researchers examined the service delivery profiles and outcomes of Maryland PROMISE employment specialists employed by the project for at least 6 months. In particular, we compared the outcomes of 30 employment specialists who participated in targeted job development training and field-based coaching to 25 employment specialists who did not receive the same level of capacity-building strategies.
TA coaches were assigned to each team to manage and support the delivery of project services. These staff were highly experienced and well versed in the evidence-based practices related to employment service delivery and how to train and support staff in the field. It was their responsibility to support the intervention teams with all aspects of implementing the Maryland PROMISE core components. For example, the TA coaches provided one-to-one, field-based coaching and modeling on how to negotiate with employers and facilitate work-based learning experiences for youth.
Intervention
Maryland PROMISE established measurable weekly targets to guide project activities and ensure services were consistently delivered across the state. As highlighted in Table 1, employment specialists were expected to make eight to 10 contacts with employers each week and three to five appointments with employers. These weekly targets prompted employment specialists to focus on employers and develop work-based learning experiences and job placements. In addition, employment specialists were expected to make eight to 10 contacts with PROMISE youth weekly. This target ensured continual youth engagement in important employment-related services. These expectations were communicated to staff regularly at weekly team meetings, via email, and during 1:1 field-based coaching. Moreover, the expectations helped to define the roles, responsibilities, and delivery of activities for employment specialists to meet program goals. They also served as a measure for staff accountability and allowed for data-driven supervision and TA to support the field-based delivery of career-related services. To promote staff capacity and accountability, Maryland PROMISE monitored staff performance and provided training and field-based coaching to employment specialists. As cited by professional development research, staff who receive on-going coaching and continual feedback regarding their performance have a sharper focus on providing quality services (Snyder et al., 2015). The strategies used by Maryland PROMISE are described next.
Maryland PROMISE Employment Specialist Weekly Targets.
Note. PROMISE = Promoting Readiness of Minors in Social Security Income.
Performance monitoring and support
Management staff monitored employment specialists’ service delivery and outcomes using three instruments: (a) an Intervention Report, (b) a Fidelity Report, and (c) a Performance Matrix. The Intervention Report allowed Maryland PROMISE to track key intervention components by individual employment specialists. Each time an employment specialist provided a service to a Maryland PROMISE youth, they recorded it into eVolv, the electronic management information system used by the project. These data allowed for real-time tracking of service delivery and monitoring of progress toward outcomes for youth on their caseload. A Fidelity Report provided data on elements of performance that spanned the scope of services and outcomes that comprise the Maryland PROMISE intervention, including work experience and paid employment outcomes. This report was used by Maryland PROMISE leadership staff to ensure the delivery of services was consistent with the project purpose. The Performance Matrix, generated monthly, reported progress on performance measures and other aspects of Maryland PROMISE services and outcomes. These three reports provided real-time information of services delivered and the outcomes of those services and consequently provided information on overall staff activity and performance. The performance management tools also helped to inform the need for targeted training and field-based coaching of employment specialists.
Training and field-based TA
To reinforce intervention fidelity, a key feature of Maryland PROMISE was a continuum of TA provided through a TA coach and designed to support the delivery of practices that increase the quantity and quality of work-based learning experiences and paid employment. Overall, the investment in a train and coach approach contributed to a high level of fidelity to the intended intervention (Luecking et al., 2019). Table 2 provides examples of the training and coaching topics offered to Maryland PROMISE staff and methods used to deliver new knowledge and skills. Aligned with key features for quality professional development (Lyon et al., 2011), methods used to deliver training and coaching included interactive instruction, goal identification, small group discussion, problem-solving, self-reflection, peer collaboration, direct feedback, and follow-along coaching.
Maryland PROMISE Capacity Building Strategies.
Note. PROMISE = Promoting Readiness of Minors in Social Security Income; ACRE = Association of Community Rehabilitation Educators.
Small group discussions to allow staff to share job leads and review individual PROMISE participant needs. bAdditional training and coaching offered to employment specialists.
Using the performance monitoring reports and processes, management identified those employment specialists who needed additional training and coaching to meet service delivery expectations and project outcomes. In addition to these employment specialists, some staff self-identified as wanting to gain additional competencies in work experience and job development. A total of 30 staff completed an additional, highly structured, intensive 40-hr training program with on-going field-based coaching designed to build their skills and knowledge to deliver career-related services to participating PROMISE youth.
Maryland PROMISE TA coaches delivered these trainings to 30 employment specialists. Two cohorts received training during three separate 6-hr training sessions. A third cohort received training using a hybrid approach which included 12-hr face-to-face training and 28-hr asynchronous training using an online learning platform with instructor-led webinar course of instruction. All three cohorts were provided on-going mentoring and field-based coaching. The training was based on the standardized, competency-based training endorsed by the Association of Community Rehabilitation Educators (2019). It covered four domains: (a) core values and principles of employment service practice, (b) planning for individualized work-based learning experiences and paid employment, (c) identifying and negotiating with prospective employers, and (d) workplace and related supports. Each of the employment specialists who completed the intensive training also needed to complete a minimum of 10-hr supplementary field-based assignments with coaching to apply the techniques presented in the training. For example, TA coaches worked one-on-one with employment specialists in the field modeling, practicing, and giving feedback on employer outreach strategies including conducting informational interviews, developing employment proposals, and establishing work-based learning experiences.
Data Analysis
To examine the study’s research questions, we explored the impact of the intensive training and field-based coaching on Maryland PROMISE employment specialists’ individual performance on work experience and paid employment outcomes. In particular, we wanted to know if the employment specialists who received targeted training and field-based coaching placed more youth into work-based learning experiences and paid employment compared to employment specialists who did not participate in these capacity-building opportunities.
Using data from the Maryland PROMISE electronic case management information system (i.e., eVolv), we compared the performance of employment specialists who completed the intensive training (n = 30, aggregated across the three training cohorts) to employment specialists who did not receive training (n = 25) on two criteria: (a) number of work-based experience placements per month obtained by each employment specialist aggregated across each youth on his or her caseload and (b) number of competitive jobs per month obtained by youth with help from each employment specialist aggregated across each youth on his or her caseload. A Shapiro–Wilk test of normality was conducted to determine if an independent samples t-test was appropriate for comparing the number of placements attained by employment specialists who had training and those who had not. We used a non-parametric test (Mann–Whitney U test) to compare outcomes. This two-stage procedure is acceptable when comparing the means of two independent samples (Rochon et al., 2012).
Results
Results showed that the sample deviated significantly from a normal distribution for both work-based learning experiences and paid employment. As illustrated in Table 3, employment specialists who received additional intensive training and supplementary field-based coaching achieved a greater number of work experience and paid employment outcomes for youth on their caseloads. Our analysis found employment specialists who received additional training and field-based coaching secured a median of 1.65 work-based learning experience placements for youth per month. This moderately exceeds the median of 0.37 placements per month secured by employment specialists who did not receive additional training.
Comparison of Employment Specialists’ Attainment of Work-based Learning Experience Placements for Youth Based on Receipt of Additional Training and Field-Based Coaching.
Note: The median number of placements were per month. A Whitney–Mann U test was conducted in place of an independent samples t-test due to violations in assumptions of normality.
As illustrated in Table 4, employment specialists who received additional training and field-based coaching helped youth obtain a median of 0.38 paid employment per month. This slightly exceeded a median of 0.12 paid employment per month achieved through the efforts of employment specialists who did not receive the additional training and coaching.
Comparison of Employment Specialists’ Attainment of Competitive Jobs for Youth Based on Receipt of Additional Training and Field-Based Coaching.
Note. The median number of placements were per month. A Whitney–Mann U test was conducted in place of an independent samples t-test due to violations in assumptions of normality.
Discussion
Throughout the implementation of Maryland PROMISE, there was a considerable investment in training and monitoring staff performance to ensure high fidelity of the intervention to its model standards. Overall, efforts to monitor staff progress toward project outcomes and provide additional targeted training and field-based coaching contributed to improved performance of employment specialists on key employment-related intervention outcomes. That is, students on their caseloads achieved more work-based learning experiences and paid employment placements. It is clear that there were benefits to targeted training and field-based coaching designed to promote staff knowledge and skills in facilitating meaningful work-based learning experiences and paid employment. These findings suggest two circumstances that relate to transition research and transition practices.
First, in large-scale research studies such as Maryland PROMISE, the fidelity of service implementation was critical to evaluate the project’s impacts accurately. That is, when the intervention is delivered as intended, a more definitive interpretation of the results is possible. In Maryland PROMISE, the efforts described here contributed to the achievement of a high degree of fidelity of all intervention services across the state (Luecking et al., 2019). As a result, the subsequent impact analysis of two key aspects of the Maryland PROMISE intervention, work-based learning experiences and paid employment, can be conducted with confidence when comparing the outcomes between participants who received the intervention with those who did not. Other large-scale random assignment research studies, such as the National Youth Transition Demonstration, also found that differential outcomes occurred when a higher dose of employment service intervention was delivered to treatment group youth. Specifically, those youth who received employment-related services from staff who received more intensive training followed by coaching had higher rates of post-project employment and earnings compared to the control group (Fraker et al., 2018). Thus, staff monitoring, training, and support are critical to intervention fidelity and impact analysis.
Second, and perhaps more consequential in the context of evolving transition services, focused staff monitoring, training, and support makes a difference in the ability of students and youth with disabilities to access work-based learning experiences that have been proven to be highly correlated with post-school adult employment (e.g., Luecking et al., 2019; Wehman et al., 2014). With federal requirements in WIOA for delivering work-based learning experiences to transition-age students with disabilities, as well as the shift from providing services in segregated settings to the community, state and local policymakers are seeking ways to build staff capacity to meet these demands. Maryland PROMISE was faced with the challenge of staff capacity building as most of the direct staff at the outset of the intervention demonstrated a lack of competencies necessary to achieve the program’s employment benchmarks. Therefore, project management implemented strategies to build staff knowledge and skills to effectively deliver important employment-related activities and track their efforts to achieve the desired project outcomes. These capacity-building strategies can be adopted in everyday practice in any transition service program that endeavors to expose students to quality community-based work-based learning experiences.
Limitations
The present descriptive study utilized data provided by employment specialists who enter case information in an electronic case management database (i.e., eVolv) at various points of the service process. The data retrieved from the case management database were cross-checked with other records such as staff case notes and TA coaches’ case review logs, subjective recall bias is possible during data entry. Also, the comparisons between the two groups of employment specialists are not purely representative of a carefully matched group according to potentially comparative characteristics such as length of job tenure, previous similar professional job experience, or previous training. This method of analysis therefore yields relational comparisons, but not causal results of the training and support intervention.
Implications for Practice
Building the capacity of staff skills and knowledge can be instrumental in ensuring the delivery of transition services that lead to the intended outcomes. Maryland PROMISE applied direct efforts to build the capacity of staff to understand and implement core interventions with a high degree of fidelity (Luecking et al., 2019). To ensure fidelity was achieved consistently across the regions of the state, Maryland PROMISE staff were offered training opportunities on a wide variety of topics, including an intensive, competency-based training coupled with field-based coaching to better equip employment specialists in planning for and facilitating work-based learning experiences and paid employment outcomes. Staff who took advantage of targeted training were able to learn new skills and develop competencies related to the content. The field-based coaching that followed training supported the implementation of the new knowledge and skills with fidelity. Training coupled with ongoing coaching has implications for promoting quality transition practices. By providing training followed with coaching, secondary transition professionals can benefit from modeling, direct support, and reinforcement of skills needed to promote best practices in transition service delivery.
Implications for Future Research
The capacity-building strategies used by Maryland PROMISE support the model of teach and coach professional development to promote the use of evidence-based practices that facilitate quality work-based learning experiences and paid employment outcomes for transition-aged youth with disabilities. More importantly, this study contributes to the literature on professional development for secondary transition professionals. It offers direction for future research on incorporating formal tests of staff training interventions to build the skills and knowledge of staff tasked with facilitating work-based learning experiences for transition-aged youth within the community to impact positive post-school employment outcomes.
Implications for Policy
With evolving federal requirements for delivering work-based learning experiences to transition-aged youth with disabilities (e.g., WIOA, IDEA), many state and local policymakers are seeking ways to build staff service capacity to meet these requirements. Given that most of the direct service staff lacked advanced employment-related competencies for achieving the Maryland PROMISE program’s employment outcomes for youth, significant investment in professional training was necessary to elevate competencies toward the state-of-the-art. Yet, such training that Maryland PROMISE’s staff required is common to programs serving transition-aged youth nationwide. For example, Mazzotti and Plotner (2016) recently conducted a five-state survey of 592 transition service providers from fields of education, vocational rehabilitation, and workforce development and found that just over half (52%) seldom or never received training for delivering evidence-based employment services. This alarming finding calls for more federal, state, and local funding to provide transition service professionals on-going training in evidence-based employment practices, especially in placing youth with disabilities into work-based learning experiences (e.g., Boardman et al., 2005; Morningstar & Benitez, 2013).
Conclusion
It is necessary for secondary education professionals in education, vocational rehabilitation, and community employment agencies to possess the competencies necessary to prepare students for adult roles, including long-term employment. While it has been substantiated that implementing field-based and career-related services during secondary education for students with disabilities is associated with improved post-school employment outcomes, the research to practice gap has yet to be effectively bridged (e.g., Carter et al., 2010; Luecking et al., 2018; Mazzotti & Plotner, 2016; Wehman et al., 2014). Transition professionals are not fully equipped with the skill set to move transition-age youth from school to work (Benitez et al., 2009; Morningstar & Liss, 2008). Professional development opportunities for secondary transition professionals often are disjointed or not accessible to those who need it most (Morningstar & Liss, 2008). The findings of this study provide some insight as to how employment specialists can be supported to obtain the knowledge, skills, and tools necessary to move youth toward their career goals. Despite the limitations of this study, the findings merit the building of more rigorous evaluations of staff training and performance monitoring of transition professionals who are tasked with facilitating student access to work-based learning experiences. Such work-based learning experiences are becoming more prominent features of transition services and are recognized as noted features that impact positive post-school employment outcomes for transition-aged youth with disabilities.
Footnotes
Acknowledgements
The authors recognize the support from Jade Gingerich and the Maryland Department of Disabilities for the development of this manuscript.
Disclosures
The authors certify that IRB approval was obtained for this study from the University of Maryland, College Park.
Disclaimer
Data for these analyses were collected under a cooperative agreement with the U.S. Department of Education, Office of Special Education Programs associated with PR Award # H418P1400##. The views expressed herein do not necessarily represent the positions or policies of the Department of Education or its federal partners. No official endorsement by the U.S. Department of Education of any product, commodity, service, or enterprise mentioned in this publication is intended or should be inferred.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by Office of Special Education Programs, Office of Special Education and Rehabilitative Services, grant ID H418P1400.