Abstract
BACKGROUND:
One of the goals of PROMISE is to improve employment outcomes resulting in life-span reductions in youth reliance on SSI. Initial and prior employment is associated with long-term career trajectories for youth with disabilities. More research is needed to develop evidence-based practices that improve early employment outcomes for youth with disabilities receiving SSI benefits.
OBJECTIVE:
We seek to provide information about the demographic and intervention factors associated with the successful completion of 200-hour, interest-based summer work experiences.
METHODS:
Outcome data for 126 youth who participated in interest-based summer work experiences were collected. Regression analyses were conducted to explore associations between the number of hours worked and various discrete intervention variables.
RESULTS:
Interest-based job placement, case management, and job readiness training were significant predictors of success in the summer work experience.
CONCLUSION:
Particular aspects of service delivery, such as job readiness training, interest-based job placement, and personalized introductions to local workforce centers by case managers can significantly improve employment outcomes for youth with disabilities. State and federal agencies tasked with improving employment outcome for youth SSI recipients are provided substantiated justification to allocate resources that align with this evidence.
Introduction
Youth recipients of Supplemental Security Income (SSI) encounter significant barriers to competitive employment as they transition into the world of work. These barriers are not primarily related to the impingements of their disabilities but to the existing infrastructure of social supports that are often inadequate to promote a career trajectory. Commensurately, employment outcomes for youth SSI recipients are significantly poorer for youth without disabilities.
Promoting the Readiness of Minors in Supplemental Security Income (PROMISE) is a joint federal research demonstration of the U.S. Departments of Education (USDOE), Health and Human Services, and Labor, with evaluation support for the demonstration from the Social Security Administration. As the lead federal partner, the USDOE funded six model demonstration projects to address barriers and obstacles to economic independence and promote successful education and employment post-school outcomes for youth who receive SSI. Youth SSI recipients between the ages of 14 to 16 were eligible to enroll in the PROMISE demonstration. The USDOE provided approximately $230 million to the following demonstration projects over a five to six year period which commenced in October of 2013: Arkansas; California; Maryland; New York; Wisconsin; and a six-state consortium which included Arizona, Colorado, Montana, North Dakota, South Dakota and Utah.
Several studies have shown that paid work experience for transition age youth leads to higher rates of employment after high school. Among youth with disabilities between ages 16 and 22, Fabian (2007) found that early work experiences were a significant factor in predicting employment, showing youth with prior experience to be 35% more likely to secure employment. Analyses of The National Longitudinal Transition Study – 2 (NLTS-2) shows that early work experience is associated with post-secondary employment (Carter, Austin, & Trainor, 2012). The Broadened Horizons, Brighter Futures (BHBF) project evaluation found that interventions that include work experiences for transition age youth with disabilities significantly predicted later outcomes, with the employment rate of the treatment group exceeding that of the control group (Fraker et al., 2018). Various supported employment services in early work experiences for youth with disabilities have been examined in previous studies, but more research is needed to further examine their effectiveness.
A customized employment project study provided work experiences based on the individual interests of transition-aged youth with disabilities. This approach showed a 62% employment rate for those enrolled in the project (Rogers, Lavin, Tran, Gantenbein, & Sharpe, 2008). Instruction in basic job skills and positive workplace behaviors (“soft skills”) is thought to be an important component of early work experiences. According to the National Technical Assistance Center on Transitions (NTACT), such instruction is an essential characteristic of early work experiences as a predictor to improve independentliving outcomes (NTACT, 2018). Müller and VanGilder (2013) found that interns who were offered jobs at the end of one year in a Project Search program that included skills instruction scored higher on a job readiness assessment than their peers who were not offered jobs. In a national Delphi study, 35 national experts in Vocational Rehabilitation (VR) reached a consensus on a list of the top 26 promising or evidence-based VR practices. Soft skills and Job Club ranked 12th and 14th on that list, respectively (Leahy et al., 2018).
Supported employment services have also been recommended to help youth with disabilities acclimate to the work environment. A study looking at data from the Bridges program operated by the Marriott Foundation for People with Disabilities found that job specific, onsite support provided by program staff predicted higher employment retention for youth with disabilities (Garcia-Iriarte, Balcazar, & Taylor-Ritzler, 2007). Wehman, Chan, Ditchman, and Kang (2014) reported that “supported employment was found to increase the employment rates” (p. 296) of youth with a range of disability types. Specific to Social Security beneficiaries, the authors found that 58% of those who received supported employment obtained successful employment compared to a 37% employment rate for those who did not receive supported employment services.
Transportation or lack of access to reliable transportation is problematic for youth with disabilities in accessing paid work experiences. A 2016 study reported that 36% of participants identified transportation problems as an issue for employment. Transportation problems included not having a driver’s license, limited public transportation hours and days of operation, lack of taxi services, and families not owning a car (Noel, Oulvey, Drake, & Bond, 2016). Lindsay (2010) found that adolescents and young adults reported worrying about accessible and reliable transportation as a barrier to employment.
PROMISE federal initiative
In a response to the low employment rates for youth receiving SSI, the federal initiative known as PROMISE was created. In FY2013, ED funded six standard double-blind model demonstration projects to improve education and employment outcomes of youth SSI recipients and their families. ED awarded cooperative agreements across 11 states to implement the projects, while SSA contracted with Mathematica Policy Research (MPR) to conduct the national evaluation. PROMISE asked the projects to identify innovative methods of providing services and supports to improve outcomes for youth SSI recipients and their families.
AR PROMISE Model Demonstration Project
The Arkansas PROMISE Model Demonstration Project (AR PROMISE) recruited and enrolled 2000 participants in Arkansas between September 2014 and April 2016. At the time of enrollment, youth were 14 to 16 years old and currently receiving SSI benefits from SSA. A random assignment system created and operated by MPR assigned 1027 youth to the program (treatment) group that received the interventions and 973 youth to the usual services (control) group that only received a list of non-AR PROMISE services available to them.
AR PROMISE utilized three primary interventions to improve educational and employment outcomes: intensive case management, supplemental training, and paid work experiences. The focus of this study was the paid work experience intervention specifically, though elements of intensive case management and supplemental training are discussed as well.
AR PROMISE provided at least two work experiences to each program group participant. A work experience (a) took place in an integrated community setting, (b) paid participants at local prevailing wages and (c) consisted of up to 200 hours. The vast majority of these experiences were completed in the summer between 2015 and 2018. All program group youth were scheduled to work in cohorts (400–500) in at least two of the four summers. Over four years, 779 youth recipients of SSI participated in this intervention.
Transition Specialists employed by Arkansas Rehabilitation Services created interest and vocational profiles and assessed the need for supported employment services. The Arkansas Department of Workforce Services and local workforce investment boards provided job readiness training, job development, job placement, and worksite monitoring. Job readiness training consisted of 10–20 hours of instruction in a classroom setting, utilizing a standardized curriculum provided by the local workforce investment boards. AR PROMISE secured employment supports (personal care, job coaching, accommodations, technology, and transportation) utilizing established providers of rehabilitation services and Sources (a Center for Independent Living). AR PROMISE Connectors (case managers) provided coordination and support before, during, and after the summer work experience. Transportation was only offered in summer 2018.
In order to better understand the factors that influence early work experiences for transition-age youth with disabilities, the present study investigated the effects of various employment supports, practices, and youth perceptions on early work experience outcomes for transition-age youth recipients of SSI enrolled in the AR PROMISE model demonstration project.
Method
Participants and procedures
The data for this study was extracted from the PROMISE Management Information System (MIS) and a survey administered at the end of the summer paid job work experience. The PROMISE MIS contained information on duration of employment (hours worked), hourly wages, attendance at job readiness training, and employer information. Staff of the local workforce investment boards entered the data into the MIS at the end of each summer work experience. Additionally, the self-report survey assessed participant perception of the PROMISE work experience including job satisfaction, job coaching, and transportation related experiences.
The AR PROMISE formative evaluation staff administered the survey to participants at 11 sites in August 2018 during scheduled end-of-summer recognition events hosted by local workforce investment boards. Two sites in each region were chosen. Upon entering the site and prior to the start each event, participants were given a paper copy of the survey and a pencil and were instructed to complete the survey prior to the start of the event. Evaluators were on hand to assist with reading comprehension as needed. Participants were asked to answer honestly and given privacy to complete the survey. Surveys were collected from all sites and the data were combined for analysis.
Overall, there were 291 participants enrolled in the AR PROMISE in 2018. However, only 126 participants (43.3%) completed the work experience survey. Other inclusive criteria included participants who (a) enrolled in the research project between August 2014 and April 2016, (b) were 14 to 16 years old at the time of enrollment, (c) were currently receiving SSI benefits at the time of enrollment, (d) participated in the summer work experience intervention in 2018, and (e) completed the work experience survey in August 2018.
Of the total 126 participants who had a paid work experience in the summer of 2018 and took the summer work experience survey, five of them (3.97%) did not respond honestly (i.e., marked the same extreme choices for the entire survey) and 30 (23.81%) had missing value(s) on one or more items. The final sample was comprised of 91 participants (32 females and 59 males) from six geographical regions in Arkansas: central (15.4%), eastern (40.7%), northwest (18.7%), Pulaski (7.7%) and southern (17.6%). At the time of the 2018 summer work experience, the age of the participants ranged between 16 and 20 years with a mean age of 17.75 (SD = 1.05). The demographic information is summarized inTable 1.
Demographic data by three summer work hours categories (N = 91)
Demographic data by three summer work hours categories (N = 91)
The outcome variable in this study was the level of hours worked. AR PROMISE set a goal for participants to work at least 200 hours during each summer work experience. Although the working hours form a continuous variable, the general cutting scores were applied due to project’s presetting goal and existing heterogeneous variances. Participants’ working hours were divided into three categories: “1 = 0 – 99 hours” (below expected), “2 = 100 – 199 hours” (approaching), and “3 = 200 and more hours” (met expectations).
Three sets of predictor variables were used in this study. Available data for the 2018 work experience were examined to identify appropriate indicators of job preparation and training, job experiences, and transportation related items. First, there were three items on the dichotomous scale (1 = Yes and 0 = No) related to job preparation and training. Participants were asked to respond to the statements “My Connector assisted me with my first face to face visit with the local work force office”, “I attended a ‘work readiness’ or ‘job club’ class before I started working”, and “I received additional support on the job for my disability”.
Second, two items asked about overall perceptions of the PROMISE work experience (i.e., “My PROMISE job was related to my interests and preferences” and “My PROMISE job gave me experience(s) that will help me in the future”). Originally, both items were on a 5-point Likert-type scale. Data from these items were converted into a dichotomous scale (1 = agree and 0 = neutral or disagree) for consistency with the previous three items.
Finally, to assess participant perception of transportation services, the following items were included in the survey: “In the future, I would be MORE willing to work if transportation was provided”, “I enjoyed working MORE this year because transportation was provided for me”, “Provided transportation was easier than finding my own transportation”, and “Having transportation provided made my working easier on my family”. Respondents indicated the degree to which they agreed with those items on a 5-point Likert-type scale from 1 = strongly disagree to 5 = strongly agree. All items were summed to obtain an overall transportation perception score with possible values ranging from 4 to 20.
Data analyses
SAS version 9.4 was used for all statistical analyses. Given that the outcome of working hours was divided into three ordinal categories (i.e., below expected, approaching, and met expectations), ordinal logistic regression was applied to model the association between summer working hours and potential influence factors. Following Hosmer, Lemeshow, and Sturdivant’s (2013) recommended procedures, first, we conducted univariate logistic regression analyses to examine whether each given indicator was associated with working hours. In addition, alpha level was set to.25 which recommended by Hosmer et al. (2013) in order to account for important variables at the model building stage. A cumulative logit link function was used to model cumulative probabilities of working hours based on the dividing points. To conduct this analysis properly, a score test was used to examine whether all variables met the proportional odds assumption, that is, if the relationship between each pair of outcome groups (i.e., “below expected” vs. “approaching and met expectations combined” and “below expected and approaching combined” vs. “met expectations”) remained the same. Second, multiple logistic regression analysis (i.e., including two or more predictors simultaneously in a single model) was applied to examine effects of each indicator while controlling for other variables in the model. Odds ratios were computed to aid in interpreting relationships between each predictor and the level of working hours.
Results
Univariate associations with summer working hours
Table 2 presents the univariate associations between each individual predictor and the level of summer working hours. Although only one predictor was statistically significant at. 05 level, we still considered including all predictors for the multivariate analysis because the traditional alpha level (i.e.,. 05) might be too conservative and fail to identify important variables (Bendel & Afifi, 1977; Mickey & Greenland, 1989). All individual predictors have met the proportional odds assumption examined by score test. Overall, the level of working hours was positively associated with “connector assistance”, “attended work readiness”, and “PROMISE job was related to my interests”. In contrast, the level of working hours was negatively associated with “received additional support for my disability”, “job gave me experience(s) that will help me in the future”, and perception of transportation.
Univariate statistics of ordinal logistic regression results for individual predictors
Univariate statistics of ordinal logistic regression results for individual predictors
Odds ratios are also reported to aid in interpreting relationships between each predictor and the level of working hours. Using the item asked about “connector assistance” (i.e., Yes/No) as an example, an odds ratio (e b ) can be described as the probability of “met expectations” versus “approaching or below expected” for those who received assistance (Yes) compared to those who received no assistance (No). Moreover, as constrained by the proportional odds assumption, this estimated odds ratio is also the same for “met expectations or approaching” versus “below expected”.
An odds ratio of one indicates that either dummy coding in the predictor does not yield any change in the likelihood of the level of working hours. Odds ratios above one indicate a positive relationship between the predictor and the level of working hours, while odds ratios below one indicate a negative relationship. For example, the odds ratio for the connector assistance indicator is 2.84, indicating that participants assisted by their connector in the first face to face visit with the local work force office are 2.84 times as likely to reach the 200-hour goal of summer work hours as are participants not assisted by their connector. This is true for both for “met expectations” versus “approaching or below expected” and “met expectations or approaching” versus “below expected”. Additionally, it is important to note that the odds ratios presented in Table 2 are univariate effects and do not control for other predictors in the model.
An ordinal logistic regression model was then estimated that included all the potential predictors of summer working hours. Parameter estimates, standard errors, significance test results, and odds ratios for the multivariate model are displayed in Table 3. Most predictors were statistically significant except perception of transportation and “My PROMISE job gave me experience(s) that will help me in the future.” Compared with the univariate analysis, the multivariate analysis does control for other predictors in the model. For example, the indicator of “My PROMISE job was related to my interests and preferences” has an odds ratio of 3.54. As one moves from Neutral/Disagree to Agree in the equation, the odds of “met expectations” versus the combined effect of “approaching or below expected” is 254% higher given all the other predictors are held constant. Similarly, as one moves from Agree to Neutral/Disagree, the odds of the combined effect of “met expectations or approaching” versus “below expected” is also 254% higher given all the other predictors are held constant.
Multivariate statistics of ordinal logistic regression results
Multivariate statistics of ordinal logistic regression results
Note: Max-rescaled R-Square = .21 aThe estimated log odds for “met expectations” versus “approaching or below expected” when the predictors are evaluated at zero. bThe estimated log odds for “met expectations or approaching” versus “below expected” when the predictors are evaluated at zero. *p<.05
The data for the predictor variables in this study were taken from surveys completed by participants. While evaluators were available to answer questions, confidence in each respondent’s comprehension for each item is unknown. Rather than using participant recall and perception, administrative and financial records would provide a more thorough and continuous look at utilization of job coaching and transportation services. Overall, there were 779 youth participated in the work experience intervention between 2015 and 2018.
Limitations
This study focused on transportation services, which were only provided in 2018, this narrowed the subject pool down to 291. After examining the data quality, the usable data decreased to 91 participants, which limited statistical power. As the project was stipulated by the funding agency as a standard double-blind research model, AR PROMISE project did not gather data on those participants in the control group. As such, no comparative data was available for this use.
Given these limitations, findings from this study indicate that in-person, initial meetings with service providers by case managers, attendance at work readiness training, and the perception that a job placement was interest-based significantly predicted employment outcomes. Most notably, youth who perceived their job placement as related to their interest and preference were 3.54 times as likely to meet the 200-hour goal as those who were not. Youth whose Connector assisted them with their first face-to-face visit with the local workforce office were 3.73 times as likely to meet the 200-hour goal as youth who did not feel receiving assistance. In addition, youth who ranked higher score on attended a “work readiness” or “job club” class before starting work were 3.3 times as likely to meet the 200-hour goal as who ranked it in lower scores.
Unexpectedly, youth who received additional support on the job for their disability were less likely to meet the 200-hour goal. While included in the results, a youth’s perception that their job gave them experiences that would help them in the future was not a significant predictor. Also, the youth’s perception of transportation services was expected to predict employment success but was not significant in the final model.
Interestingly, while lack of access to reliable transportation was reported by participants in the project, as well as noted in the Introduction from previous research, providing those services did not show to predict better outcomes in the summer work experience. While additional on-the-job support was a negative predictor in the study, further studies using financial and administrative documents would provide continuous variables of job coaching support. This data would provide a larger sample over several years to further explore this variable and its relationship to the summer work experience outcomes.
Despite these lingering questions, these finding do underscore the effectiveness of some services. An initial face-to-face meeting with the local workforce office, participant, and the AR PROMISE Connector increases the likelihood that a participant will successfully complete the work experience intervention. As part of the intensive case management, AR PROMSIE Connector assisted youth in navigating disparate service agencies and coordinating activities.
The individualized job placement chosen collaboratively and informed by vocational assessments and personal goals also increases the likelihood that a participant will successfully complete the work experience. This underscores earlier findings of Customized Employment (Certo & Luecking, 2006; Rogers et al, 2008) and importance of youth involvement in job development and job placement. Finally, work readiness training was shown to increase the likelihood that a participant will reach the 200-hour goal.
Conclusion
The results from this analysis corroborate the existing literature demonstrating that paid work experiences, interest-based placements, and supported employment promote significantly better employment outcomes for youth with disabilities. Within the AR PROMISE Model Demonstration Project intervention, the consideration of the youth’s interests and aspects of supported employment, such as in-person, initial meetings with service providers facilitated by Connectors, and attendance at work readiness training predicted positive employment outcomes. Conversely, other aspects of supported employment, additional support on the job, and transportation were not correlated with better employment outcomes. Additional analysis and subsequent research will be required to determine if those youth who required additional support on the job and/or transportation represent a subpopulation of participants whose overall needs may have imposed additional barriers to employment.
State and Federal agencies tasked with improving employment outcomes for youth SSI recipients should consider the specific interventions identified in this manuscript. These findings reinforce incorporating more aspects of customized employment into endeavors to place youth with disabilities in competitive employment. Likewise, these findings provide further evidence that intensive and individualized interventions result in significantly better employment outcomes as well as substantiated justification for state and federal programs to allocate resources that align with this evidence.
Conflict of interest
The authors declare that they have no conflict of interest.
Footnotes
Acknowledgements
The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Office of Special Education Programs, associated with PROMISE Award #H418P140002. Corinne Weidenthal served as the project officer. The views expressed herein do not necessarily represent the positions or policies of the Department of Education or its federal partners. No official endorsement by the U.S. Department of Education of any product, commodity, service or enterprise mentioned in this publication is intended or should be inferred. Research involving human subjects is conducted in accord with the ethical standards of the Institutional Review Board of the University of Arkansas.