Abstract
Summary
Work-based assessments (WBAs) were only recently introduced into medicine. However, since their introduction, they have rapidly grown in popularity. WBAs are now a routine part of medical training. As WBAs are being implemented, the practical difficulties with their use have come to light. A major problem is failure of the trainees, trainers and training programmes to adequately engage with them. In this review, the reasons for this and how these can be overcome are discussed.
Introduction
Medical training in the UK has undergone major changes recently, with the introduction of work-based assessments (WBAs) being one of the most significant changes. 1 Since their introduction, the use of the WBAs has become more widespread. WBAs, which were initially trialled with foundation doctors, are now a routine part of training for medical trainees of all grades and all specialities in the UK.
The popularity of WBAs stems from the fact that they are tests of the highest level of clinical competence. Miller 2 proposed a pyramid for assessing clinical competence. The successive levels of this pyramid from the lowest level are knowledge (knows), competence (knows how), performance (shows how) and action (does). WBAs are tests of the highest level of clinical competence. For example, the multi-source feedback (MSF) assessment tests the performance of doctors in their normal practice, i.e. what a doctor ‘does’. Another reason for the popularity of WBAs is that they also assess non-technical skills such as professionalism, decision making and time keeping. 3
How can engagement with work-based assessments be improved.
Methods
A pragmatic review of WBAs was undertaken. The review was limited to WBAs that are routinely used in medicine: Direct Observation of Procedural Skills (DOPs), Mini-Clinical Evaluation eXercises (mini-CEXs), Case-Based Discussions (CBDs) and MSFs. The relevant literature pertaining to WBAs in the medical setting was identified using the online databases: MEDLINE®, Google Scholar and Web of Knowledge.
Time
WBAs are time consuming. A study on three commonly used WBAs discovered that the mean total time required for the completion of a mini-CEX, MSF and DOP, respectively, were 25 min, 6.8 min and the duration of the procedure plus a further third of this time for feedback. 4 These are likely to be underestimates of the true time required for the completion of WBAs, as they only took the time required for the assessment and feedback into consideration and not the time required for locating the assessor or identifying suitable cases. As WBAs are time consuming and there is no dedicated time in clinical practice when they can be undertaken, it is unsurprising then that the lack of time is a major reason that prevents the completion of WBAs. 4,5
A potential solution is to have ‘time written into the job plans of clinical teachers’. 5 Another solution for the lack of assessors’ time is to encourage trainees ‘to use as many different assessors as possible, as this … spreads the workload’. 6 Training assessors in WBAs is required to increase the pool of the assessors, as a lack of training in WBAs was one of the common reasons cited by assessors who refused to undertake WBAs. 7 To further increase the pool of assessors, non-medical assessors may be used. Unfortunately, early experience of using non-medical assessors was that they are even less familiar with WBAs than their medical counterparts. Only 6% of non-medical assessors were rated by trainees as having sufficient knowledge of WBAs. 1 If the practice of using non-medical assessors is to be continued, then targeted training of the non-medical assessors is vital. Such a move is likely to receive the support of non-medical assessors because in a recent survey of non-medical assessors, 92% felt that training would be a help. 8
In addition to addressing the lack of assessors’ time, the lack of time for trainees to undertake WBAs also needs to be addressed. In a survey of psychiatric trainees, 70% of respondents reported that WBAs had a negative impact on the time available for their clinical duties. 1 This finding is surprising because in theory WBAs are observations of routine clinical practice and not an extra event. Therefore, careful planning is required so that this training need, WBAs, is met without compromising clinical care. The most favoured solution by trainees was to have regular protected time with their assessor that is free of clinical commitments, when they could undertake WBAs. 1,9 Including time for WBAs in job plans of assessors and trainees will be costly, ‘which in the current financial climate may be difficult to achieve’. 10 Bindal et al. 7 suggest pre-planning of WBAs into clinical practice as a way of creating time for WBAs but without the financial cost associated with including time in assessors’ and trainees’ job plans for WBAs.
A radical solution to the lack of time problem is to reduce the number of each type of WBAs that a trainee needs to undertake. Such a strategy should be pursued with caution. Owing to inter-assessor variations in marking, WBA ‘needs to be repeated on several occasions for it to be a reliable measure’. 3 It then seems logical that training assessors in using WBAs will improve marking consistency and its educational value, which will then make it possible to reduce the number of each type of WBAs that a trainee needs to undertake. The effect of assessor training on performance was assessed by Cook et al., 11 who evaluated the effect of an assessor training workshop on inter-assessor reliability and accuracy of mini-CEX scores. Surprisingly, they discovered there was no difference in inter-assessor reliability of mini-CEX scores between the assessors who had attended workshop and the control group. Possible explanations for this observation offered by Cook et al. 11 include a small size (31), which means that the study lacked the power to detect small changes, and an ineffective workshop format. A more worrying explanation is that ‘physician raters are impervious to training’. 12 A great deal of effort is being directed to determine the barriers to training assessors and how these can be overcome, with the ultimate aim of reducing the minimum number of WBAs that needs to be undertaken. Such an approach is fundamentally flawed. It fails to recognize these multiple encounters with assessors where different issues are discussed as beneficial. These multiple assessments act as multiple sources of feedback for the trainees, allowing them to truly identify their strengths, the areas they need to work on and how they should make the changes. Therefore, the key is not to reduce the minimum number of WBAs but to make them easier to use.
Administrative burden
Another possible reason for the failure of trainees to engage with WBAs is the bureaucratic burden of the assessments. Trainees’ experience of the foundation programme learning portfolio is that it involves excessive paperwork, a view strongly agreed by 84% and 77% of trainees from the 2004 and the 2005 cohort, respectively. 13 While WBAs are an important part of the foundation programme learning portfolio, there are also several other components. Therefore, from the work by Hrisos et al., 13 it is not possible to make conclusive statements about the bureaucratic burden of WBAs. However, in a survey of medical specialist registrars (SpRs), a number of trainees reported that WBAs created an excessive administrative workload. 4
The fact that WBAs are not easy to use 1 and are not user-friendly 14 is felt by experts to be the main reason for the excessive administrative workload. WBAs need to be simplified ‘so as to make them more user-friendly in busy clinical settings where patient care is the first priority and trainee assessment of less importance’. 15 A fear with simplification of WBAs is that this may dilute their educational value. However, Driessen et al. 16 developed and implemented a lean non-bureaucratic assessment programme that had a high educational value, was robust and was well liked by trainees. Hence, we urge that the current medical assessment programmes are critically reviewed and modified so that they incorporate Driessen’s principles.
Another contributory factor to the administrative burden of the WBAs is the lack of access to computers. Only 11% of respondents to a survey of psychiatric trainees’ views of WBA agreed with the statement that they had easy access to a computer. 1 A solution to the lack of easy access to computers is to embrace the use of the smartphone. Smartphone sales have grown rapidly over the last five years, so much so that in 2011, more smartphones were sold than personal computers. 17 Smartphones can be used to complete WBAs even when there is no access to a computer, and this should increase the completion rates of WBAs. 18
A further reason for the administrative burden of WBAs is the lack of sufficient training in the assessment methodology. While it is tempting to speculate that training the trainees will increase participation, Pelgrim et al. 19 acknowledge the absence of any evidence to support this claim. Assessors also need to be trained in WBAs to further reduce the administrative burden. The need for assessor training in WBAs is supported by the fact that 22% of assessors who responded to a survey on WBAs reported that they had received no training and yet they routinely undertake WBAs. 20 Of the assessors who had training, only 55% felt confident or very confident in undertaking WBAs, 20 which suggests the need for further training in WBAs. The form of training most favoured by assessors for this purpose was workshops/seminars and practical hands-on marking. 20 The popularity of workshops stems from the fact that they are flexible, promote active learning and a variety of teaching methods are employed within this format, including interactive lectures, small-group discussions and exercises, role-plays and simulations. 21
Educational value
A recurring theme is that trainees are unconvinced of the education value of WBAs. A survey of Foundation Year 2 trainees (FY2) demonstrated that 60.9%, 56.1% and 39.1% of trainees felt that DOPs, mini-CEXs and CBDs, respectively, were useless or very useless. 22 A similar result was observed on a survey of GP specialist trainees (GPSTs). Only 55% of GPSTs viewed WBAs as a useful educational tool. 23 In fact, some trainees view WBAs as a hindrance. In a survey of British trainee surgeons, 63.6% of respondents reported that the completion of WBAs had an adverse impact on their training opportunities. 14
One explanation for trainees’ pessimistic view of the education value of WBAs is the lack of effective feedback. For the majority of trainees (94% for verbal and 74% for written), feedback is a useful element of WBAs but a number of assessors (27.5% for verbal and 61% for written) are unable to give effective feedback. 23 Trainees’ perception that assessors are unable to give effective feedback appears to be an accurate reflection of reality. Ali et al. 24 retrospectively analysed the feedback that had been recorded in the WBAs undertaken by urology trainees and discovered that there was a lack of feedback and when there was feedback it was of poor quality. Holmboe et al. 25 evaluated the feedback that was given after a mini-CEX and found it to be poor. Only in 34% of encounters did the assessor ask for self-assessment from the trainee and only in 8% of encounters was an action plan formulated.
Both a lack of appreciation by assessors of the importance of feedback and the fact that they may not be skilled in the process of providing high-quality feedback may explain why assessors fail to give effective feedback. 15 However, there is conflicting evidence in the literature as to the impact of assessor training on the quality of feedback. Salerno et al. 26 discovered that the rate of specific feedback increased from 22% to 38% after faculty development seminars. In contrast, Holmboe et al. 27 discovered that the rate of specific feedback remained unchanged after intervention. A plausible explanation for this observation is that intensive interventions are required to change assessors’ behaviour and improve the quality of their feedback. Holmboe et al.’s intervention 27 was a 20-min didactic teaching session on feedback, while Salerno et al.’s intervention 26 consisted of three 90-min interactive seminars scheduled one week apart. Therefore, educating assessors about feedback principles and techniques needs to be intensive if it is to improve the quality of their feedback and to increase the educational value of WBAs.
The seniority of the assessor influences the quality of feedback. According to trainees, the more senior the assessors, the more useful was feedback that they received. 9 However, FY2 trainees found that the most senior of these assessors (consultants) were reluctant to undertake WBAs. Only 13% of respondents in a survey of a FY2 trainees reported that their consultants were keen to complete WBAs. 9 Apathy may be a factor in poor consultant contribution. However, work by Finall 28 suggests that this may not be the case, as she demonstrated that consultants valued WBAs. However, the conclusion drawn from Finall’s work should be tempered with the knowledge that her survey only captured the view of seven consultant histopathologists from a single hospital. Work by Menon et al. 1 potentially offers an explanation for the poor consultant contribution reported by FY2 trainees. In his survey of STs, 91% rated consultant assessors as willing to complete WBAs. 1 This finding suggests that poor consultant contribution to WBAs may be limited to more junior trainees. A possible explanation for this is that there is less interaction between consultants and junior trainees as compared to senior trainees owing to the different working relationships of these grades with their consultants and the shorter placements of junior trainees. Therefore, consultants should specially be encouraged to contribute more to WBAs of their more junior trainees.
Further reasons for trainees’ pessimistic view of the education value of WBAs are a perceived lack of validity, reliability and evidence base for these instruments, 1 and this may account for failure to engage with them. To address this, (1) threats to the validity and reliability in the workplace should be identified and addressed and (2) trainees need to be educated about the evidence base for these instruments.
Schuwirth and van der Vleuten 29 highlighted a more generic reason for the limited educational value of WBAs. They argue that in medicine ‘assessment is synonymous with punitive examinations whose sole purpose is to pass or fail candidates’ and this ‘has made us fear assessment’. 29 For WBA to be effective, it must be conducted as formative assessments with emphasis on trainees to engage with the feedback to improve themselves. This is unlikely to take place in a culture where assessments are feared. At present, trainees are likely to fear WBAs as they contribute to their annual review of competences, i.e. WBAs are being used in summative assessments. Hence, a cultural shift is required in medicine such that WBAs are viewed in positive light and used solely as formative assessments to improve performance. One way to ensure that WBAs are solely used as formative assessment is to remove all links to ratings from them. In the absence of ratings, it is impossible to use WBAs in summative assessments, and this would encourage a more honest engagement with WBAs. In fact, this has already happened with the foundation programme WBAs, where ratings have been removed. They are now used solely as supervised learning events with candidate feedback.
Assessors’ enthusiasm
Assessors’ enthusiasm for WBAs is likely to influence trainees’ engagement. Nisar and Scott. 30 demonstrated that core trainees valued trainers’ enthusiasm as a key trainer attribute. Trainees have reported a lack of enthusiasm from trainers’ for WBAs. 1 Some assessors view WBAs as tick-box exercises while others have genuine passion for these assessments. A two-pronged strategy is required. Assessors who have low enthusiasm for WBAs should be identified and reasons for their low enthusiasm should be explored. Trainers with high enthusiasm for WBAs should be rewarded so that their enthusiasm is maintained.
Discussion
This review has its limitations. The intention of this review was that it was a pragmatic review of literature, rather than a systemic review of the literature. Hence, there is a possibility that not all relevant studies have been identified in spite of searching multiple databases and manually searching the reference lists. Furthermore, the quality of this review is limited by the quality of the studies. The majority of the studies reviewed were cross-sectional surveys of trainees’ and assessors’ view of WBAs. A concern with the conclusions drawn from such studies is whether the views expressed in the surveys are truly representative of the views of the sample population. To overcome this, a number of surveys have used free text spaces, which allows respondents to express their view over and above which that was specifically asked. It is likely that as the surveys were anonymous the participants expressed their views freely. Furthermore, the unusually high response rates for these surveys suggest that the views expressed by the participants of the surveys are likely to be representative of the sample population. A further concern with drawing conclusions from trainees’ view of WBAs is that their perception of the issues may be different from the reality. However, there is some evidence to suggest that the trainees’ view of WBAs is an accurate reflection of reality. Work by Wilkinson et al. 4 has shown WBAs to be time consuming and this supports the claim that lack of time is a major reason that prevents the completion of WBAs. Similarly, work by Holmboe et al. 25 offers support to trainees’ claim that the quality of the feedback they receive is poor. Even if trainees’ perceptions of WBAs are different from the reality it is nevertheless important to be aware of them. Unless these misconceptions are corrected, trainees are unlikely to engage effectively with WBAs.
At present, there are a number of issues that prevent trainees, trainers and training programmes from adequately engaging with WBAs. Failure to address these issues will limit the long-term value, usefulness and success of WBAs. The importance and relevance of this review is that it highlights these barriers as well as the strategies that can be employed to overcome them. A range of strategies have been considered along with their merits and pitfalls, which allows individuals and institutions to select the strategies that suit their specific needs.
Conclusion
The application of WBAs in medicine has seen a rapid growth in the last few years. However, trainees, trainers and training programmes are not engaging wholeheartedly with them, which limits the educational value of these instruments. Fortunately, the reasons for this are beginning to emerge, along with potential solutions.