Abstract
BACKGROUND:
Structured work support needs assessment could facilitate professionals and increase assessment consistency.
OBJECTIVES:
Evaluating usability of the Work Support Needs Assessment Tool and test if professionals’ (labour experts, coaches) findings become more consistent after a tool training. The tool includes a 21 item checklist for assessing work support needs of people with disabilities.
METHODS:
Usability was explored through 28 interviews with professionals. Consistency was evaluated in an experimental pre-post study design, in which thirty-nine other professionals assessed work support needs of standardized clients before and after a protocolized training. Quantitative content analysis was conducted. Consistency of findings between professionals covered three categories: type (client-focused coaching), focus (topics to be addressed) and duration of support. An increase in consistency was defined as a decrease in the total number of different sub-categories of findings in each category.
RESULTS:
Nineteen professionals indicated that the tool was useful, as they gained relevant information and insights. Regarding consistency, the number of findings differed pre- and post-training for type of support (8 vs 9) and focus of support (18 vs 15 and 18 vs 17).
CONCLUSIONS:
Participants had positive experiences with the tool. Increased consistency in findings of professionals after the training was not demonstrated with the current study design.
Introduction
Disability affects many people, with the global average prevalence rate being estimated at 16% of the adult population [1, 2]. These prevalence rates are combined with high levels of unemployment among people with disabilities leading to high social security costs for society [3]. Employment can have beneficial effects on health, including feeling more autonomous and having an increased sense of wellbeing [4]. Moreover, especially for people with a disability, having a job is associated with a higher level of functioning and a better quality of life [5, 6]. Therefore, it is important that people with disabilities are given the opportunity to enter and remain in the labour market.
People with disabilities encounter problems in finding and holding onto a job [7]. However, when they receive support, these problem can be alleviated [8, 9]. Frequently, vocational rehabilitation professionals are required to assess the type and intensity of support that a person with a disability will need in order to function at work. The type of support can vary from job search assistance (support in finding suitable jobs, writing resumes or job interviews) to job coaching (support when people start work), job training (teaching job-related tasks) or workplace adaptation [2, 11] In order for a professional to make a decision about the necessary support during work, referred to below as ‘work support needs’, the professional must assess what kind of support will be needed in which situations.
Work support needs are often assessed before the person with a disability has a job, because the required amount and type of support plays an important role in the process of matching the right person with a suitable job and workplace. Therefore it is often not feasible to search for a job when work support needs are not known. This makes the assessment of work support needs complex, as the level of support needed is partly dependent upon the actual workplace and the specific job tasks. Additionally, the assessments are often made for young people with little or no work experience. In such cases the professional has to look for practical examples from a person’s daily life to determine the areas in which support is needed, translating these into work situations. A tool that facilitates these work support needs assessments by providing the professional with more structure and guidance could therefore be useful. Furthermore, an universal structured approach in assessing work support needs might increase transparency and consistency in assessments between professionals. To the best of our knowledge, currently no universal tool exists for assessing work support needs of people with disabilities when the future workplace is unknown.
Therefore, we developed the Work Support Needs Assessment Tool. This tool facilitates vocational rehabilitation professionals (labour experts, job coaches) in the process of assessing work support needs of people with disabilities when the future workplace is still unknown. The 21 items included in the tool cover four categories: pre-conditions for work; skills; personal characteristics; and cognitive and executive functions. The tool guides the professional in deciding which items are deemed relevant for the work support needs assessment and in making an overall decision on the work support needs. It is important to evaluate whether professionals feel the tool can be used to assess work support needs in practice (i.e. does it provide the professional with relevant information) and whether the tool can be incorporated in the work process of a professional. These views of professionals concerning the usability of the Work Support Needs Assessment Tool are still unclear. Furthermore, it is not known if professionals’ assessment of work support needs becomes more consistent after using the tool.
The first phase of the study explored the usability of the Work Support Needs Assessment Tool, while the second phase evaluated the consistency of work support needs assessments. Training programmes have been found to be effective in enhancing knowledge and skills as well as guideline adherence of professionals [12, 13]. It is likely that training in the use of our tool will lead to better knowledge and skills in using the tool. Therefore, participants in our study completed a training programme on how to use the Work Support Needs Assessment Tool. Subsequently, we evaluated the influence of this training programme on the consistency of the professional findings generated by using the work support needs tool. The following research questions were addressed: What are the views of professionals regarding usability of the Work Support Needs Assessment Tool?; Do the findings of professionals become more consistent after completing a training programme on the Work Support Needs Assessment Tool?
Methods
Design and procedures
The first phase of the study on the usability of the tool had a qualitative design, with Dutch vocational rehabilitation professionals participating in semi-structured interviews. Prior to the interviews, participants were asked to use the tool to assess work support needs with at least two clients from their caseload.
The consistency of findings on work support needs was evaluated in an experimental pre-post study design using two client vignettes played by an actor. Participants assessed the work support needs of one client before the start of the training programme and without using the tool. The participants then completed the training programme before assessing the work support needs of a second client with use of the tool. The order of the client vignettes alternated between the four training groups.
Ethics
The research was conducted in accordance with the Helsinki declaration. The research proposals were submitted to and approved by the Medical Ethical Committee of the Academic Medical Center, which judged that a comprehensive evaluation was not required since these studies were not subject to the Medical Research Involving Human Subjects Act (reference numbers W17_235#17.273 and W17_ 298 # 17.351).
Participants
Various types of vocational rehabilitation professionals (labour experts, consultants, job coaches) participated in the study. In the first phase, professionals were eligible to participate if the assessment of future work support needs of clients with disabilities was part of their job, and if clients in their caseload did not yet have a job. For the second phase of the study, professionals were eligible to participate if they had not been involved in prior studies in which the tool was developed and if they were experienced in assessing future work support needs of clients with disabilities.
Participants were recruited through flyers that were distributed on professional networks, websites, through a newsletter and by approaching professionals and organizations from the researchers’ network. The recruitment procedure was aimed at various types of vocational rehabilitation professionals from different organizations. All participants gave their written informed consent prior to study participation.
The work support needs assessment tool
The tool development started in 2015, as part of another study (study not published). The aim of the study was to develop a checklist with items that were deemed relevant for assessing work support needs of people with disabilities, when the future workplace is still unknown. A concept mapping approach was used to collect, integrate and conceptualize views and appraisals of the consulted experts (i.e. vocational rehabilitation professionals) [14]. The checklist development followed three stages: (1) literature search (conducted by researchers), (2) a focus group and (3) prioritizing and organizing of items. The literature search resulted in a list of 66 items relevant for assessing work support needs. In the second stage 15 vocational rehabilitation professionals participated in a focus group. Prior to the focus group, participants could online submit relevant items, this resulted in a list of 55 items. All items from the literature search and the online submittal were discussed in the focus group and consensus was reached on a list with 50 items. These items were organized in clusters and prioritized by relevancy online, on a 0–5 scale (0 = not very important; 5 = very important). Items with a mean score of 3.5 or higher were included in the final list. This resulted in a checklist of 21 items. In a second focus group with the same professionals, consensus was reached about the definitions of these 21 items (study not published).
The checklist was further developed into the Work Support Needs Assessment Tool in 2017. The core of the tool is the 21-item checklist (see Table 1). The first step of the tool is to systematically assess every item by answering several questions. These recurring questions guide the professional in deciding whether the item is relevant to the assessment. The second step of the tool facilitates the professional in the process of making an overall work support needs judgement by obtaining answers to several concluding questions (see Table 1). Additionally, professionals can consult an appendix, which provides an overview of the items, corresponding definitions and suggestions on how to question clients on a certain item. See supplement material 1 and 2 for the paper-based (translated into English) and print screens of the digital version of the tool (in Dutch).
21 items and the questions included in work support needs assessment tool
21 items and the questions included in work support needs assessment tool
The interviews were audio-recorded and conducted by the first author (MS) and another member of the research team (SV). Interviews were conducted with the use of a topic list developed by the research team. The topic list included open-ended questions regarding demographic information, how the tool was used, usability of the tool (i.e. whether the tool assisted in obtaining the right information and generated sufficient insight regarding the support needs of the client) and whether the tool could be implemented in the assessment routines of the participant.
Box 1. Description of client vignettes
Phase two: Consistency of the findings regarding work support
Training programme
The consistency of the findings of professionals on work support needs were evaluated after completing a protocoled training programme. The learning objectives of the training programme were: 1) the participant understands the 21 items and related definitions 2) the participant is able to formulate suitable questions to assess relevant items, 3) the participant can use the tool during work support needs assessment. The three-hour training consisted of the following components: 1) general introduction (stating aims and learning objectives), 2) short specific introduction on the tool development process, 3) interactive exercise in pairs (aimed at interiorizing several items and definitions), 4) general discussion about interactive exercise, 5) demonstration of how to use the tool. The training was supervised by the second author (KN), a professionally qualified and experienced teacher.
Client vignettes
The two client vignettes were played by an actor (see Box 1 for short vignettes description). During the session, participants were given the opportunity, as a group, to ask the actor questions, so that they would be able to assess the work support needs. In order to minimize the possibility of differences in the roles played, the actor used a fixed script. After questioning the actor, the participants individually recorded their findings regarding the work support needs of the client.
Outcome measure
The outcome measure was operationalized as the findings of the work support needs assessment and was measured with the following question: ‘Describe the necessary support for the client: include in your assessment the type of support, areas on which the support should focus and the level of support needed’.
Analysis
Phase one: Usability of the work support needs assessment tool
The interviews were transcribed by MS and SV and were randomly checked for inconsistencies by SA. Transcriptions were categorized according to views about usability and implementation in the work processes and summarized for each category by SV. The categorizing and summarizing of the data were checked and discussed by MS, KN and SA until inconsistencies were resolved and adjustments made accordingly. The final analysis was then discussed by the whole research team.
Phase two: Consistency of the findings regarding work support needs
Assessors involved in analysing the work support needs assessments were blinded for group allocation and whether the work support finding was pre- or post-training. To analyse the differences in the consistency of the findings, a quantitative content analysis was conducted in three steps [15]. In Step 1, one assessor (MS) coded all of the findings independently, according to the predetermined codes (type, focus and duration of the support). If necessary, new codes were added. A second assessor (KN) independently coded 10% of the findings. Differences in coding and newly emerging codes were compared and discussed by both assessors until consensus was reached. This resulted in a final coding scheme that was discussed with the other authors (SA, HW, MD). Step 2 involved organizing the codes according to the main and sub-categories by MS and KN (for instance, coaching client or coaching colleagues/manager as sub-categories of the main category ‘type of support’). Lastly, the codes from the anonymous findings on work support needs were connected to the timing of measurement and to client A or B.
Step 3 comprised counting of the different type of findings within the three main categories: type of support (e.g. coaching client or coaching colleagues/manager), the focus of the support (e.g. support aimed at teaching certain skills or adjustments in task instruction) and the duration of the support (e.g. certain number of hours of coaching during a specific period). Within these three categories, the total number of different sub-categories of findings were counted. The total number of different sub-categories of findings in each of the three main categories were then compared across pre- and post-training measurement.
Greater consistency of findings on work support needs was defined as a decrease in the total number of different sub-categories of findings in the post-training measurement, as this would indicate fewer differences in findings with less variation among the participants. An increase in the total number of different findings would indicate the findings were becoming less consistent and more incongruous among the participants.
Results
Participants
Phase one: Usability of the work support needs assessment tool
A total of 28 participants were interviewed between July-October 2017. Of these 28 participants, 13 were male and 14 were working as labour experts. Six participants were job coaches and eight worked as vocational rehabilitation consultants or coaches. The mean interview duration was 45 minutes.
Phase 2: Consistency of the findings regarding work support needs
A total of 41 participants completed the training programme in September 2017. Two participants did not provide a pre- or post-training measurement, therefore only 39 participants were included in the analysis. Of these 39 participants, 17 were male and 19 were labour experts, 15 were vocational rehabilitation consultants and five were job coaches.
Phase one: Usability of the Work Support Needs Assessment Tool
A small majority of the participants (
Participants mentioned that certain items from the tool were very important to address, something they were made aware of or reminded of by using the tool.
Two participants did not consider the tool to be useful because it did not provide them with information they needed. They considered it overly focused on work support, or found that certain items which they normally would consider were lacking in the tool. One participant stated that building trust and developing a bond between the professional and the client was also an important element in their work and that the tool did not facilitate this. Another participant stated that the tool did not sufficiently focus on the living conditions or the nature of the disability of the client.
The majority (
A small majority of the participants (
Phase two: Consistency of the findings on work support needs
The results on consistency in the findings on work support needs are presented below according to three main categories: 1) type of support, 2) focus of support and 3) the duration of the support. The results regarding the second category are presented separately for Client A and Client B due to differences in the content of the vignettes.
Work support findings regarding type of support
Before the training programme, the participants produced eight different sub-categories of findings regarding the type of work support, while after the programme, the participants provided findings that covered nine different sub-categories. This indicates that consistency in findings about the type of support did not improve after completing the programme. See Table 2 for an overview of the findings on the type of support.
Different sub-categories of findings on the type of support
*Percentage of the total number of findings on the type of support (pre-training: N = 70; post-training: N = 73). The categories are not mutually exclusive.
Different sub-categories of findings on the type of support
*Percentage of the total number of findings on the type of support (pre-training: N = 70; post-training: N = 73). The categories are not mutually exclusive.
In the pre-training measurement for Client A, there were a total of eighteen different sub-categories of findings regarding the focus of the support. In the post-training measurement, this number decreased to fifteen. This indicates a small increase in the consistency of the findings regarding the focus of support for Client A after the programme (see Table 3 for an overview).
Client A: Different sub-categories of findings on the focus of support
*Percentage of the total amount of findings on the focus of support (pre-training: N = 102; post-training: N = 76). The categories are not mutually exclusive.
Client A: Different sub-categories of findings on the focus of support
*Percentage of the total amount of findings on the focus of support (pre-training: N = 102; post-training: N = 76). The categories are not mutually exclusive.
Before the training programme, eighteen different sub-categories of findings concerning the focus of support were offered for Client B. This number decreased to seventeen after the programme. Since the difference is very small, an improvement in consistency in findings regarding focus of support for Client B cannot be assumed. See Table 4 for an overview of the findings for the focus of support for Client B.
Client B: Different sub-categories of findings on the focus of support
*Percentage of the total number of findings on the focus of support (pre-training: N = 99; post-training: N = 96). The categories are not mutually exclusive.
Client B: Different sub-categories of findings on the focus of support
*Percentage of the total number of findings on the focus of support (pre-training: N = 99; post-training: N = 96). The categories are not mutually exclusive.
The participants described the duration of support in various ways. Due to the many different kinds of responses given on the duration of support, it was not possible to analyse and codify the majority of the findings and thus it was not possible to draw conclusions regarding the consistency of these findings.
Discussion
This study aimed to explore professionals’ views about the usability of the Work Support Needs Assessment Tool. Most of the participants considered the tool to be useful for assessing the work support needs of clients with disabilities. By using the tool, they obtained useful insights and in-depth information to assist in their assessment of work support needs. The tool also gave them structure and guidance during consultations with clients. A few participants did not find the tool useful, due to its standardized and analytical format or because of missing items. The majority of the participants believed that the tool could be incorporated into their assessment routines.
An overall significant increase in consistency in the professionals’ work support needs judgements could not be demonstrated in this study. The results indicate that findings about the type of support did not become more consistent after the training programme, as the total number of sub-categories of findings on types of support increased. In relation to the findings on the focus of support, a very small increase in consistency was found.
Another similar work-related support needs assessment instrument is the Work-ability Support Scale. This tool has demonstrated acceptable levels of accuracy and reliability and is considered adequate for use in clinical practice [16]. The Work-ability Support Scale can be used in the assessment of vocational rehabilitation needs when a client already has a job or is expected to start in a job. This tool addresses specific items that are also included in our tool, for instance dealing with managers, clients and colleagues, motivation to work, personal support, planning and organizing work tasks and having realistic expectations regarding work or abilities. Differences between both tools relate to the timing of using the tool in the vocational rehabilitation process. Our Work Support Needs Assessment Tool is developed to be used at the start of the guiding process, when the future workplace of the client is still unknown. It therefore produces an initial and generic work support needs judgement that can be used to guide the job searching and matching process. When there is a potential workplace in sight, the initial support needs judgment should be specified and adjusted to the actual workplace. This revised support needs judgement can be conducted by using a more specific tool, such as the Work-ability Support Scale, in which the job tasks and workplace are taken into consideration [16]. Although the majority of consulted professionals in our study had positive views about the Work Support Needs Assessment Tool, caution should be exercised when using the tool as more research is required to increase the psychometric properties of the tool.
Limitations
The Work Support Needs Assessment Tool seemed acceptable for assessing work support needs according to the majority of the professionals in our study. However, our study did not demonstrate an improvement in the consistency of findings regarding work support needs after a training programme. Explanations for this finding might be found in the study design and the way data were analysed. First, the training programme was focused on participants internalizing the items and related definitions included in the tool, along with a method of assessing these items during a client interview. For practical reasons, it was not feasible for participants to individually interview the two clients. This meant it was not possible to allow every participant to complete an individual work support needs assessment overview, as for which the tool was designed. A training programme that focused on judging the work support needs through such an individually generated overview of work support needs assessment might have had other effects on the consistency of the findings regarding work support needs. Second, consistency in work support findings was assessed through quantitative content analysis [15]. Although this type of analysis has a quantitative element (i.e. counting the times a certain word or phrase is used), in our study the sub-categories for type of support and focus of support were based on a qualitative analysis preceding the quantitative analysis. Another method of measuring differences in assessments might produce different results, for instance letting the participants tick fixed multiple choice answer categories and assessing inter-rater reliability through statistical analysis. Another limitation is that due to the small amount of findings in each sub-category of the assessments, it was not possible to conduct stratified analyses based on the type of professional. It is possible that a job coach could have assessed work support needs different from a labour expert.
One strength of this study is the large number of participants included in the entire study, and the variation in type of professions such as labour experts, job coaches or consultants. The participants appeared to be a representative sample of the different type of professionals who are involved in assessing work support needs. Furthermore, using an actor to play the role of the client in two vignettes was also a strength, as this strategy has been associated with the initiation of active learning processes, allowing trainees to experience and reflect through assignments with the actor. Active learning has been found to be an effective method for obtaining new skills or changing behaviour [17]. In addition, vocational rehabilitation professionals use all kinds of sources when collecting information about the functioning of a client. An interview with the client is the most important source of information, as it gives the professional the opportunity to ask questions and to observe, and sometimes test the clients’ functioning, responses and behaviour. A written case description of a client or a video of them would not make it possible for participants to ask questions and would therefore not be a realistic simulation of the use of the tool in practice.
Future research should focus on exploring other suitable and quantitative methods of assessing consistency of findings that follow from using the Work Support Needs Assessment Tool. An increase in consistency of work support needs findings would contribute to the reliability of work support needs assessments. Moreover, further development of the tool should also be considered; for example, the tool might be embedded in a mobile app. An app could possibly offer users more flexibility in relation to their needs and preferences and might also reduce the time it takes to fill out the questions included in the tool. This could further increase the usability of the tool.
Conclusion
The Work Support Needs Assessment Tool offers valuable assistance in the structural assessment of work support needs of clients with disabilities at the start of the guiding process, when the future workplace is still unknown. The tool provides professionals with structure and guidance during client consultations. However, in the current study design we were not able to demonstrate an increase in the consistency of professionals’ findings regarding work support needs after a training programme in the use of the tool.
Conflict of interest
The authors declare that they have no conflict of interest.
Data availability
The datasets generated during and/or analysed during the current study are not publicly available as these include Dutch text data, but are available from the corresponding author on reasonable request.
Funding
This study has been funded by a SUWI grant from ‘The Dutch Social Security Institute: the Institute for Employee Benefits Schemes (UWV)’ Grant number: K1617a.
Footnotes
Acknowledgments
We would like to thank Stefan van der Veen for his contribution to the study.