Abstract
Background
Delayed return-to-work is common after serious injury. Vocational rehabilitation (VR) supports people to return to/remain in work. We developed training for occupational therapists (OTs) and clinical psychologists (CPs) to deliver a VR intervention (Return to Work After Trauma-ROWTATE) to UK trauma patients within a large-scale randomised controlled trial.
Objectives
Evaluate the training, OT/CP competence and confidence to deliver ROWTATE, attitudes/confidence in evidence-based practice (EBP).
Methods
Competency was assessed through Team Objective Structured Clinical Examination and two tasks (management plan, employer letter). Intraclass correlation coefficients (ICCs) estimated inter-rater reliability. Questionnaires measured EBP attitudes (EBPAS-36) and confidence (EPIC) pre and post-training and confidence in ROWTATE delivery post-training. Semi-structured interviews captured training experiences and confidence.
Results
47 OTs and 16 CPs completed training. Competency scores were high (OT range=27.5–37.5; CP range=28.5–37; maximum=38). Inter-rater reliability was moderate (OT ICC=0.56, 95%CI 0.18,0.77; CP ICC=0.73, 95%CI −0.22,0.93). EBPAS-36 scores were high pre and post-training with no significant increase post-training. OT confidence in applying evidence (P = 0.01) and evaluating effect of their actions (P = 0.04) increased significantly post-training. OTs and CPs had high confidence (80–100% across questions) in delivering ROWTATE.
Conclusions
VR training resulted in high competency scores, improvements in EBP confidence and high confidence in delivering ROWTATE.
Keywords
Introduction
Injuries are a major cause of death and disability accounting for 8% of deaths and 11% of years of life lost worldwide (GBD, 2019 Mental Disorders Collaborators, 2022). Injuries can have long-lasting negative effects on physical and psychological function (Kendrick et al., 2017a), quality of life (Black et al., 2011), return to work (Kendrick et al., 2012), work productivity (Kellezi et al., 2022), financial health (Scott, 2022) and health service use (Kellezi et al., 2016). Recent studies report one third of injured people have not returned to work 12 months after injury (Kendrick et al., 2017b).
Both physical and psychological function influence return to work (Zatzick et al., 2008), so addressing both must be at the centre of rehabilitation programmes to support return to work. Although evidence on the effectiveness of vocational rehabilitation (VR) (Donker-Cools et al., 2016) and psychological rehabilitation (Roels et al., 2016; Salazar et al., 2000; Saltychev et al., 2013) is growing, there is limited evidence related to diverse trauma populations (Donker-Cools et al., 2016) and of interventions combining VR and psychological rehabilitation. To address this gap, we developed the Return to Work after Trauma (ROWTATE) VR intervention (Radford et al., 2025a) which is delivered by occupational therapists (OTs) and, where needed, by clinical psychologists (CPs). The key principles of the ROWTATE intervention include: early intervention, multidisciplinary working, identifying injury impact, understanding injury impact on work, individual tailoring, work preparation, collaboration across locations, accommodating injury at work, case coordination, employer engagement, responsiveness/monitoring, timely psychological support, vocational goal setting and review, identifying work alternatives and integrated treatment (full details can be found in paper describing the development of the intervention (Radford et al., 2025a). The ROWTATE intervention is currently being tested in a randomised controlled trial (ISRCTN 43115471), in which therapists were provided with an initial training package to deliver the intervention, refresher training and ongoing learning support provided through experience of intervention delivery and mentoring.
Evaluating therapist training requires assessment of competency as well as important therapist attributes such as attitudes towards evidence-based practice (EBP) and confidence in delivering an intervention. Therapist competency has been defined as, “the extent to which a therapist has the knowledge and skill required to deliver a treatment to the standard needed for it to achieve its expected effects” (Fairburn & Cooper, 2011). Global competence refers to therapists having a broad range of skills to deal with issues presented by patients and limited-domain competence refers to activities related to a specific intervention (Barber et al., 2007). Competence in delivering trial interventions may impact on patient (and hence trial) outcomes, affect fidelity and quality of intervention delivery (Bond et al., 2022), and scalability of trial results. Attitudes towards EBP may impact upon adoption of EBP (Aarons, 2004) and greater confidence in applying/using evidence in practice may be associated with undertaking more EBP (Clyde et al., 2016). Confidence to deliver specific interventions has also been shown to predict implementation of those interventions (Shapiro et al., 2021).
The aims of the analyses presented in this paper were to determine therapists’ a) competency to deliver the intervention after initial training, b) attitudes towards and confidence in EBP before and after initial training, c) confidence to deliver the ROWTATE intervention immediately after initial training, d) perceptions of the usefulness of training and, e) views of the training and ongoing learning support throughout the trial.
Methods
ROWTATE training
The aim of the ROWTATE training package was to prepare therapists to deliver the ROWTATE intervention (Radford et al., 2025a) by increasing their knowledge of the intervention, processes to deliver and document the intervention and confidence to deliver the intervention. Thereafter the training package aimed to promote further skill acquisition and confidence in delivering the intervention.
Details of the ROWTATE training package can be seen in Supplementary Table 1. Adult learning theory (Taylor & Hamdy, 2013) provided the overarching framework. Additionally, social cognitive theory (Torre & Durning, 2015) framed their collective learning as a group of OTs and CPs, on an ongoing basis from the initial training and throughout continued mentoring and interdisciplinary collaboration.
Evidence-Based Practice Attitude Scale Scores (EBPAS) for OTs and CPs pre and Post-Training.
IQR: Interquartile range. [Missing values]. Possible score range for each domain and for total scale score = 0-4. * Lower scores indicate a more positive attitude towards evidence based practice. Scores have been reversed when calculating the total scale score.
Training comprised pre-workshop orientation reading and tasks (answering case study questions, watching pre-recorded presentations, manual), online remote ROWTATE preparation sessions, 2-day workshop and refresher training approximately six-months later (Supplementary Table 1). Workshop content was delivered via a 2-day in person event, based centrally at the university. However, remote delivery was also designed into the delivery plan, given the ongoing covid pandemic. In addition to this core training, three additional sessions were organised in response to challenges OTs encountered during intervention delivery. These related to administering psychological screening measures, collaborative working between the OT and CP, engaging employers and data entry.
Taught sessions introduced therapists to the ROWTATE Intervention process and components, study procedures and documentation using a case study format. Content also included a trial overview, research contamination, how to deliver ROWTATE via telerehabilitation, psychological trauma following serious accidents and experiential group learning on conducting a worksite assessment. Wherever feasible, therapists from each study site were trained together engendering understanding of each other's roles and team-building. Access to all training materials and additional resources was provided via a personal log-in to the study website. All therapists received monthly, small-group online mentoring during intervention delivery, from OTs or CPs trained in the intervention and experienced in VR and research.
The training evaluation comprised quantitative and qualitative components. The quantitative component included assessment of competency, attitudes towards and confidence in EBP, confidence to deliver the intervention and usefulness of training. The qualitative component comprised interviews undertaken between March 2023 and January 2024. The intervention delivery period runs from November 2021 to March 2025, so all therapists were experienced in delivering the intervention at the time of interview.
Competency Assessment
The competency assessment was initially informed by the Return to work after stroke (RETAKE) trial (Powers et al., 2023). However, the ROWTATE competency assessment was developed to reflect the two professions (OTs and CPs) working together, whilst at the same time acknowledging the individual professional roles. Thus, the ROWTATE competency assessment included both individual and team assessment, creating a standardised case study to reflect the trauma population and a team objective structured clinical examination (TOSCE) using a standardised role play using trained actors to represent the “patient” in the case, as recommended by Fairburn and Cooper (Fairburn & Cooper, 2011). Based on learning from Return to work after stroke trial (RETAKE) and the resource burden, we chose to assess competency once.
Competency was assessed immediately following the initial training workshop and was video-recorded. It comprised three components: a TOSCE, a team written task, and an individual written task. Objective Structure Clinical Examinations (OSCEs) were initially developed for structured and objective assessment of medical student competence. Group or team OSCEs were later developed to assess teams of students or professionals completing group tasks (Elliot et al., 1994), including inter-professional teams. The TOSCE comprised a standardised role play using a professional actor with whom the OTs and CPs interact to complete tasks (outlined shortly). The roleplay (i.e., interaction where therapists asked questions) centred around ‘Jake’, who fell from scaffolding at work and sustained multiple fractures, undergoing surgery and initial rehabilitation at the local hospital. The TOSCE focus was Jake's initial clinical interview by the therapists. The team written task involved producing a clinical summary and formulation (a rehabilitation plan) for the proposed intervention based on the role play. The individual written task was to write a letter to Jake's employer explaining the rehabilitation plan. Where possible therapists were placed in teams for the competency assessment comprising OTs and CPs but due to fewer CPs, this was not always possible.
Scoring was based on The Team Formulation Intervention Quality Rating Scale, previously shown to have good face and content validity, high internal consistency and moderate inter-rater reliability. A score sheet (Supplementary box 1) was completed by two independent assessors (one OT, one psychologist) for each therapist. Scores were assigned for eight TOSCE, five formulation and six letter elements (Bucci et al., 2021). Scores were summed across elements to create a score for each assessment component and a total score. Independent assessors (from a range of different professions which include OTs, CPs, and psychology researchers) were trained in undertaking the competency assessment and followed a standard operating procedure. Scoring was piloted on the first six team assessments and discussed, with assessors adjusting scoring where they had interpreted assessment tasks differently. The standard operating procedure was then amended and used to score remaining teams.
Competency scores were assessed for normality and described using medians, interquartile ranges and ranges. TOSCE and formulation scores were described and analysed at a team level and letter and total scores were described and analysed at individual therapist level. Inter-rater reliability was assessed using intraclass correlation coefficients and 95% confidence intervals based on a mean-rating (k = 2), absolute agreement and a 2-way random effects model. Scores were averaged between the two raters to produce mean scores.
Self-Reported Questionnaires
Questionnaires were administered pre- and post-initial training. Pre-training questionnaires collected data on therapists’ work role and qualifications, experience caring for trauma patients, experience in VR, experience in trauma relevant psychological therapies (trauma-focused Cognitive Behavioural Therapy, eye movement desensitisation and reprocessing attitudes to EBP and confidence to implement EBP. Post-training data were collected on attitudes to EBP, confidence to implement EBP, confidence to deliver the ROWTATE intervention, usefulness and organisation of the training (5-point scale: strongly disagree/very poor to strongly agree/very good) and free text space to suggest improvements.
Attitudes towards EBP were measured using the Evidence Based Practice Attitudes and Beliefs Scale (EBPAS-36) (Rye et al., 2017). This scale measures 12 domains of evidence-based practice (see Table 1). Each domain comprises 3 items scored on a 5-point Likert scale (0 “Not at All” to 4 “To a Very Great Extent”). Subscale scores were computed for each domain as the mean of the 3 items. For calculating the total score (but not subscale scores), items from domains 4 (Divergence), 5 (Limitations), 7 (Monitoring), 8 (Balance) and 9 (Burden) were reverse scored and the subscale score was recalculated. The total score was calculated as the mean of the subscale scores. Higher scores indicate more positive attitude towards EBP. Where data was missing, means were computed allowing for one fewer item than make up the scale with missing items imputed as the mean of non-missing cases for that item.The EBPAS has been shown to have a good internal consistency for the total score and adequate to excellent consistency for subscale scores when used with mental health professionals, including psychologists.
Evidence-Based Practice Confidence Scale Scores for OTs and CPs pre and Post-Training.
IQR: Interquartile range. [Missing values]. Scores range from 0% (not all confident) to 100% (completely confident)
Continuous data were assessed for normality and described using medians and interquartile ranges (IQRs). Means and standard deviations were also presented for the EBPAS to allow comparison with previous publications. Categorical data were described using frequencies and percentages. Comparisons between pre- and post-training scores were made using Wilcoxon signed rank tests with a p-value of <0.05 taken as statistically significant. Open-ended questions asked about the usefulness and administration of training as well as suggestions for future training. Data was analysed using content analysis.
Therapists’ Reflections
Therapists’ experiences of training and on-going learning support were ascertained from semi-structured interviews with 12 OTs and 3 CPs exploring acceptability (12 OTs, 3 CPs) and implementation (10 of the 12 OTs, 3 CPs) of the ROWTATE intervention. We also interviewed 4 mentors (two OT and two CPs), two of which also delivered the intervention and were interviewed as therapists (1 OT and 1 CP). Interview topics guides were informed by the findings of the RETAKE study (Powers et al., 2023), a systematic review conducted as part of developing the ROWTATE intervention (Radford et al., 2025a) and the Consolidated Framework for Implementation Research (Damschroder et al., 2009). Interviews asked about how well the ROWTATE training prepared therapists to deliver the intervention, confidence in delivering the intervention and views on the training, mentoring, ongoing learning support, the intervention and intervention manual. Interviews were conducted remotely (online or by phone) with therapists providing verbal and/or written consent to be interviewed. Interviews were transcribed verbatim and analysed using thematic analysis (Braun & Clarke, 2021). The thematic analysis aimed to identify key patterns in the data that addressed the research aims which were to identify the role of the training and continuous support received in developing the therapists’ confidence and competence. The 6 steps proposed by Braun and Clarke (Braun & Clarke, 2021) were followed including data familiarisation (the researcher BK, read and listened to the transcripts multiple times), coding (part of the dataset that was directly relevant to the study aims was coded by BK), theme identification through coding (an initial theme structure was identified by BK), theme refinement took place through a discussion with the wider research team. Once themes were agreed the team refined the thematic map and completed write-up using illustrative examples.
Results
Seventy-one therapists (54 OTs, 17 CPs) commenced training and reported completing the background reading and tasks (see Figure 1 for a diagram). Five live online teaching sessions were delivered between August and December 2021. Further Covid outbreaks, train strikes and staff sickness made scheduling live sessions unfeasible and subsequently recordings of the live taught sessions were sent to therapists. All therapists reported having watched these before attending the workshop. The 2-day initial workshop took place across 11 dates between September 2021 and March 2023, with 63 therapists (47 OTs, 16 CPs) completing the workshop. The 2-day workshop was delivered in-person for seven out of 11 workshops and delivered online for the remaining four as per the reasons above.
Flow of participants through the study.
Characteristics of OTs and CPs Completing Baseline and Follow-up Questionnaires.
[Missing] * CBT (Cognitive Behavioural Therapy) * * EMDR (Eye Movement Desensitisation and Reprocessing)
Competency data are available for 56 therapists comprising 23 therapist teams. Twelve were single profession teams (11 OT-only teams and one CP-only team) and 11 were multiprofessional. Thirteen teams had two and 10 teams had three participants.
Seven therapists were not assessed for competency because they had previously completed similar (but not identical) competency assessments following VR training in the ROWTATE feasibility study (Radford et al., 2025b) or the RETAKE trial (Powers et al., 2023). One therapist had missing data on the letter component of the competency assessment. The flow of therapists though the study is shown in Figure 1.
Competency Assessment
Competency scores are shown in Table 4. Scores varied between the two raters, with the second rater scoring consistently slightly lower than the first. Inter-rater reliability varied from poor for the TOSCE and formulation scores to moderate for the total scores and good for the letter scores. Confidence intervals were wide around most of the ICCs. Mean scores (averaged across the two raters) were high for each component and for the total scores (Table 4, Figure 2), with greater variation in letter scores than for other components. The lowest total score for OTs was 27.5 (72% of maximum score) and for CPs was 28.0 (75% of maximum score). Based on these high scores, all therapists were judged to be competent to deliver the ROWTATE intervention.
Inter-Rater Reliability for Competency Component Scores and Total Scores and Scores Averaged Across the two Raters.
IQR: Interquartile range. [Missing values]
*ICC = intraclass correlation coefficient. ICC <0.5 = poor reliability; 0.5 ≥ ICC<0.75 = moderate reliability; 0.75 ≥ ICC≤0.90 = good reliability, >0.90 = excellent reliability
Maximum possible scores: TOSCE = 16, formulation = 10, letter = 12, total score = 38
Box plots of scores for competency assessment components and for the total score for ots and cps averaged across two raters. OT: Occupational therapist. CP: Clinical psychologist.
Competency Assessment Reflections
Supplementary Table 2 describes characteristics of interviewed therapists. Therapists felt they had acquired adequate skills to deliver the intervention to support patients to return to work/education. They recognised the intervention needed tailoring to each patient and reflected this more global experience of competency: So, I think ROWTATE really helped me gain experience and skills in that [serious injury/VR] community, (…) real life issues with real life questions, and yeah, it really helped me develop I think as an OT. (Therapist 13)
Therapists recognised developing competence would require self-directed learning, time and practice but they identified some specific learning needs (e.g., job retention) that could have benefited from additional training: The only thing is maybe the job retention. I don’t think I’ve really got in there with anybody from a job retention point of view (…). I’ve had one occupational health joint meeting, but I think there could be more to be done around (…) job retention, and does this population want it? (Therapist 7)
When delivering the intervention, some therapists identified unexpected patient-specific challenges. Despite this, they used the tools provided in training (i.e., the intervention manual) to find a successful solution: In the beginning, I had a patient who was suicidal. And I referred to the manual to find out what the actual ROWTATE process of that would be. Whether I needed to be pulling in a ROWTATE CP into that situation, because at the time that person didn’t have psychology input. (Therapist 6)
Evidence-Based Practice
EBPAS scores (Table 1) indicate therapists held positive attitudes about delivering EBP interventions. Total scale scores (maximum of 4) were high at both time points, being 3.1 for OTs and 3.2 for CPs without significant differences pre and post-training. The domains scoring most positively for OTs pre-training were EPB's fit within the values and needs of client and clinician (domain 6) and time and administrative burden with learning EBPs (domain 9), and post-training were positive perceptions of receiving feedback (domain 12) and domain 9 (burden). The domains scoring most positively for CPs pre and post-training were limitations of EBPs (domain 5) and domain 9 (burden). Having a balance between perceptions of clinical skills and science as important in service provision (domain 8) and the perception of job security related to expertise in EBP (domain 10) scored most negatively for OTs and CPs pre- and post-training.
Reflections on Evidence-Based Practice
Therapists were positive about EBP and being part of the trial: there's the whole research side of ROWTATE, which I’d never done any clinical research before, so that was really interesting, kind of getting the updates from the ROWTATE trial regularly was really interesting, keeping up to date with that, seeing what was going on behind the scenes. (Therapist 13)
The therapists were also positive about engaging with the key components of ROWTATE that are informed by evidence (e.g., psychological screening and collaboration between OT and CP). It's a really valuable research project and this is the first time I’ve ever been part of a randomised controlled trial, so that's been quite exciting to be part of that (…). So it was the OT that was liaising with the member of the research team and then she was feeding back to me, and myself and the OT have been in touch quite a lot and do share summaries of our sessions just to make sure that we are always on the same – yeah, we’re kind of on the same page with things and we understand the direction that we’re going in. So, yeah, again the kind of MDT working is really helpful. (…) It is different working on a research project. We had a couple of issues coming up with patients that were kind of not sure whether to keep working with them or to discharge them. It is clear who can support you with those sort of questions so I think that's really helpful. (Therapist 4)
Confidence
EPIC scores (Table 2) indicate that OTs and CPs were confident to deliver EBP. Both professions were least confident in determining whether evidence applied to their patient, and this significantly improved post-training for OTs. Both professions were most confident in asking about patient needs, values and treatment preferences. OTs confidence in evaluating the effect of their actions increased significantly post-training.
Post-training, OTs and CPs indicated they were confident to deliver the ROWTATE intervention (Supplementary Table 3), that they found the training useful and suggested improvements to the training (Supplementary Table 4) which informed refresher training and mentoring.
Reflections on Confidence
The overwhelming response was that the therapists felt confident; either present from the start or developing with time. Initially not very confident at all, and I think, yeah, it's improved as I’ve progressed and I really think that the mentoring has been excellent. (Therapist 14)
Confidence was considered essential given the nature of the work, which involved patients’ psychological wellbeing: My confidence has grown as time has gone on, but I think, you know, you’re working with people's livelihoods, their careers, their jobs, and you have to be sure and confident of what you’re suggesting to people. (…) [mentor] has supported me (…). So I think that's been a huge factor in my confidence, having a mentor that is really responsive and gives really good, experienced advice. (Therapist 6)
Building therapists’ confidence was a specific strategy used by mentors: I think my role as a mentor is to give the OTs confidence that they can do it (…) giving them confidence, knowledge and the space to explore how they do it and ask questions. (Mentor 1)
Discussion
Main Findings
We developed a novel approach to competency assessment, entailing a TOSCE involving standardised role play, team formulation and individual letter writing tasks. The OTs and CPs engaged well in the assessment process and had high scores following VR training. However, inter-rater reliability varied between components of the assessment, being good for both OTs and CPs for the letter writing task and poor for team scores for the TOSCE and formulation. Inter-rater reliability was moderate for both OTs and CPs for the total score.
OTs and CPs held positive attitudes towards evidence-based practice and were confident in delivering evidence-based practice, both before and after the training. OTs’ confidence in determining whether evidence applied to their patient/client and in continually evaluating the effect of their actions increased significantly after training. OTs and CPs were highly confident in delivering the ROWTATE intervention at the end of training.
The interview analyses supported the key findings from the quantitative analysis. The therapists reported feeling competent to deliver the intervention and adapting to the unique needs of each patient. This ability to adapt, together with positive attitudes towards the ROWTATE intervention and research, are likely to reflect the therapists’ willingness to engage with evidence-based practice. Finally, the therapists also felt their confidence developed over time, especially with the support of the mentor and other peers.
Comparisons with the Literature
Our OTs and CPs demonstrated higher levels of competency compared to the RETAKE trial, where the OTs were assessed on knowledge of the intervention process, clinical reasoning and written communication based on a clinical scenario. Only 5% of RETAKE OTs were judged to be highly competent (scored at least 70%) and 72% were judged to be competent (scored 50–69%) (Powers et al., 2023).This may reflect the use of different methods of assessment (team vs individual and standardised role play versus written questions) or potential lack of sensitivity to issues in ROWTATE training which might prove problematic in implementation.
Inter-rater reliability of TOSCEs has previously been found to be low initially, but can improve after further development, including refinement of marking schedules and use of examiner misinterpretations in examiner training (Singleton et al., 1999). Some elements of OSCEs are more reliable than others, with a systematic review reporting assessment of communication skills is less reliable than assessment of clinical skills, probably because the former is more open to judgement than the latter (Brannick et al., 2011). A recent scoping review evaluating tools for assessing therapist competency to deliver psychological interventions reported inter-rater reliability (ICC) for 15 assessment tools ranging from 0.53 to 0.96, consistent with ICCs for the total scores for OTs and CPs in our study. However, none of the competency assessments reporting ICCs in the review used standardised role-play (Ottman et al., 2020) so our findings may differ due to variation in assessment methods. In addition, our TOSCE designed task was much more complex than (OSCE) used in previous studies which tend to be brief and closely structured around identifying and treating a specific clinical problem.
Findings from the interview analysis support previous research indicating that where there is greater confidence (Powers et al., 2023) there is also more willingness to engage with the ROWTATE key components (such as collaboration between OT and CP). The findings also support previous research highlighting the value of mentoring in building skills, positive attitudes (Schoen et al., 2021) and confidence to deliver the intervention (Holmes et al., 2016) as well as improving outcomes of interventions (Clyde et al., 2016).
Strengths and Limitations
To our knowledge, this is the first study to evaluate VR team training for both OTs and CPs using a TOSCE. Our evaluation of training used mixed-methods and was comprehensive, encompassing testing knowledge and skills (global and limited-domain competence) and exploring attitudes, confidence and therapist views. We followed the recommendation of Fairburn and Cooper (Fairburn & Cooper, 2011) to use standardised role-play. Assessments were undertaken by two independent assessors, including both occupational therapy and psychology experience and inter-rater reliability was assessed. This was time and resource intensive (each competency assessment took approximately two hours, plus an additional two hours per assessor) and more complex than many competency assessments (Barber et al., 2007; Bond et al., 2022; Fairburn & Cooper, 2011). However, it provided evidence that therapists were equipped to deliver the trial intervention, the therapists valued the role-playing, and it helped identified areas where individual therapists needed further mentoring support (De Dios Pérez et al., 2024).
All OTs and CPs completing the training completed pre- and post-training questionnaires, so non-response bias did not occur in our study. We used standardised scales, previously used with allied health professionals with acceptable validity and reliability to measure attitudes towards and confidence in delivering EBP. It is possible that an alternative measure of EBP attitudes may have been more discriminating and less subject to a ceiling effect (Landsverk et al., 2023).
The number of OTs and CPs in our study was determined by the number needed to deliver the intervention for the ROWTATE trial. Consequently, the relatively small number of therapists resulted in wide confidence intervals around most ICCs and limited power to detect significant changes between pre and post-training scores for the EBPAS and EPIC. High baseline scores on the EBPAS and EPIC will also have resulted in a ceiling effect, with little scope for improvement. High baseline EBPAS and EPIC scores may have arisen due to therapists with greater interest in research, more positive attitudes towards and more confidence in EBP agreeing to participate in the trial.
The two raters for the competency assessment were trained in completing the assessment, provided with a written standard operating procedure and piloted their scoring on six assessments, with comparison and discussion of scores. Despite this, inter-rater reliability was poor for the TOSCE and for the formulation. It is possible that more extensive piloting or more detailed standard operating procedures guidance may have improved the inter-rater reliability. Inter-rater reliability was good for the letter writing task possibly because assessing whether or not the letter contained the required elements was less open to judgement than assessing TOSCE elements such as active listening or interpersonal effectiveness or assessing the extent to which goals and values were addressed in the formulation. Regardless of the variation in scores between assessors, all OTs and CPs received a high total score, suggesting competence to deliver the ROWTATE intervention.
The therapists interviewed had a range of experiences and came from different locations. However, we were only able to interview 3 CPs, and it was not possible to interview therapists that only delivered the intervention for a short period of time (all therapists interviewed had been trained at least 16 months before the interview and were still part of the study). It is possible that the therapists who left the study earlier might have had different experiences of the intervention and training.
Implications for Research and Practice
Evaluating competence to deliver trial interventions using standardised role play and TOSCEs is time and resource intensive, but demonstrating competence is vital if interventions are to be delivered with high quality and fidelity. The competency assessment for our VR training requires further refinement to increase inter-rater reliability, particularly for the TOSCE and formulation task. This may include more extensive training for assessors and refinement of the scoring guidance. Future research should also explore the use of alternative measures of EBP attitudes to evaluate VR training. Although most therapists were very positive about the training, suggested improvements included increasing content as well as providing case studies and examples during the training. Our findings highlight the importance of providing ongoing learning through refresher training and group mentoring for therapists delivering VR interventions. Our findings also highlight the importance of mentoring in enhancing competence and confidence.
Implications for Vocational Rehabilitation and Professionals
The intervention can also have implications for other rehabilitation professionals. ROWTATE is a rehabilitation intervention that works in three main ways - through
The role of other stakeholders involved in supporting an individual's RTW following serious injury within ROWTATE would depend on the local intervention delivery context, expertise, training and qualifications of the stakeholders and professionals involved, along with the timing and nature of their interventions. In the UK, where there are no rehabilitation counsellors and some employment support professionals only offer short term interventions. These stakeholders would form part of the ‘wider VR team ‘ (see Figure 3) with whom the OT in their case coordination role engages, to ensure seamless delivery. Adaptations of the model to other countries and contexts for delivery by other professionals would require further research.
Stakeholders in the delivery of the ROWTATE intervention. Adapted from RETAKE study [36].
Conclusion
This mixed-methods evaluation demonstrates that a structured, theory-informed training programme can effectively prepare occupational therapists and clinical psychologists to deliver a complex vocational rehabilitation (VR) intervention for people recovering from major trauma. Therapists achieved high competency scores across assessment components, and both professions reported strong confidence in delivering ROWTATE immediately following training. Although attitudes toward evidence-based practice were already positive at baseline, training contributed to meaningful improvements in specific aspects of evidence-based practice confidence among OTs, particularly in applying evidence to individual cases and evaluating clinical actions. Qualitative findings further highlighted therapists’ perceived growth in competence, adaptability, and professional confidence, underscoring the importance of experiential learning, interdisciplinary collaboration, and responsive mentoring.
Despite the overall success of the training, variability in inter-rater reliability (especially for the TOSCE and formulation tasks), suggests the need for further refinement of assessment procedures, including enhanced assessor preparation and clearer scoring guidance. Therapists also identified opportunities to strengthen the training by incorporating additional case examples, more role-specific content, and expanded opportunities for interaction and peer learning. These enhancements, alongside continued mentoring, have the potential to support ongoing skill development and fidelity throughout intervention delivery.
Findings from this study have important implications for VR practice. High-quality, standardised training that combines structured teaching, supported practice, and ongoing mentorship can equip multidisciplinary teams to deliver complex VR interventions with confidence. As vocational rehabilitation pathways evolve to meet the needs of increasingly diverse trauma populations, investment in comprehensive training and competency assessment will remain essential to ensuring consistent, evidence-based, and person-centred care. Future research should explore how refined assessment tools, alternative measures of evidence-based practice attitudes, and enhanced training components may further optimise therapist preparedness and intervention fidelity across settings.
Supplemental Material
sj-docx-1-jvr-10.1177_10522263261430878 - Supplemental material for Evaluating Training for Occupational Therapists and Clinical Psychologists to Deliver Vocational Rehabilitation Within a Clinical Trial: Mixed-Methods Study
Supplemental material, sj-docx-1-jvr-10.1177_10522263261430878 for Evaluating Training for Occupational Therapists and Clinical Psychologists to Deliver Vocational Rehabilitation Within a Clinical Trial: Mixed-Methods Study by Blerina Kellezi, Jain Holmes, Kate Radford, Roshan das Nair, Rebecca Lindley, Claire Mann, Isabel ndrews, Stephen Fallon, Denise Kendrick and On behalf of the ROWTATE study team in Journal of Vocational Rehabilitation
Footnotes
Acknowledgements
The authors would like to thank therapists who delivered the intervention and patient participants who took part in the study. The authors would also like to thank Lena Palmer (mentor for clinical psychologists and trainer), Nima Moghaddam (mentor for clinical psychologists), Susan Hughes (trainer) and those who assisted with the development of the competency assessment and testing the competency assessment (Ruth Tyreman and Abigail Sallis).
Ethics Approval
Ethical approval was granted by the North of Scotland Research Ethics Service (Ref 19/NS/0130) on 20/8/2019.
Informed Consent Statement
Informed consent was obtained from all patient and therapist participants.
Statement of Contributions
DK, KR, BK, TJ, and RdN, designed the study and obtained funding as part of the ROWTATE team. KR, JK, RdN, and BK designed the initial and adapted training as part of the ROWTATE team. The training was delivered by JH, DK, and BK. JH, BK, DK, KR and RdN designed the competency assessment. RL, BK, CM recruited and collected data from patient and therapists participants. BK, JH and DK undertook analyses. SF and IA provided PPI input to the study. BK, JH, and DK wrote the manuscript. All authors critically reviewed the paper and approved the final manuscript.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This paper presents independent research funded by the National Institute for Health and Care Research (NIHR) under its Programme Grants for Applied Research Programme (Reference number RP-PG-0617-20001). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability
Data are available upon reasonable request. Data supporting this work are available on reasonable request. Requests to access data should be made to Denise.Kendrick@nottingham.ac.uk in the first instance.
Patient and Public Involvement
The ROWTATE PPI team has been involved in all stages of the research providing input on the study design and training development and delivery. The two PPIs co-authoring this manuscript provided input also on the interview analysis and write up.
Supplemental Material
Supplemental material for this article is available online.