Prompting Candidates in Oral Assessment Contexts: A Taxonomy and Guiding Principles

Abstract

Prompting is an aspect of oral assessment that deserves more attention. There appears to be considerable variation in how practitioners conceptualise prompting and how it is deployed in practice. In order to unpack the term and promote the validity of its use in performance assessments, we present a taxonomy of prompting as a continuum of types, namely: presenting the task; repeating information; clarifying questions; probing questions; and finally, leading questions. We offer general principles for consideration when using prompting in oral assessment: neutrality; consistency; transparency; and reflexivity. Whenever oral assessment is planned, assessors should be appropriately trained in the type and degree of prompting required, and candidates suitably briefed to know what to expect. Overall, we aim to raise awareness that quite different behaviours tend to be subsumed under the general term ‘prompting’. This paper provides concrete guidelines for implementing the defensible and effective use of prompting in oral examinations, applicable to a wide range of assessment contexts.

Background

In high-stakes examinations in both medical school and post-graduate medical education, examiners often feel uneasy about the oral assessment format, particularly the notion of examiner prompting of the examinee. Despite long-standing concerns about the reliability and validity of unstructured oral exams,^1-3 and the emergence of the OSCE format founded upon examiner objectivity and consistency, the Viva survives (and arguably thrives) in many clinical assessment contexts as a valued method for assessing higher order competencies such as clinical reasoning and professional behaviour. It does so due to its perceived authenticity, flexibility and interactive format.^4-6 Yet its value can be compromised through misunderstanding of the role and appropriate use of examiner questioning, prompting and/or probing of candidate performance.

In medical schools and specialist training colleges, we have noticed considerable confusion amongst examiners on this issue. Even when examiners are well-trained, there remains uncertainty about what kind of prompting practices are permissible, or whether it is permissible at all. Undoubtedly, examiners want to do the right thing. We suggest that the problem may arise due to a lack of precision around what prompting means in the oral assessment format, and/or in different assessment contexts. Although we focus mainly on the relationship between prompting and oral examinations, we argue that prompting is an important consideration in a wider range of examinations and, indeed, any assessment method that involves an interaction between assessor and candidate. For instance, prompting is a common component of many other assessment types where there is verbal interaction between examiner and candidate, including the Mini-Clinical Evaluation Exercise (min-CEX), Objective Structured Clinical Examinations (OSCEs,) Direction Observation of Procedural Skills (DOPS), Multiple Mini Interviews (MMIs) and Standardised Case-Based Discussions.

Prompting, we argue, is an aspect of oral assessment that is crucial to the validity of its implementation, and deserves more attention in the scholarly literature on the format. In his in-depth analysis of the oral assessment, ⁵ Joughin acknowledges, under the dimension of ‘interaction’ and using the term ‘probing’, that this feature contributes to the unique advantages of the format, as well as constituting a major challenge from a validity perspective. Other authors have noted the potential impact that different prompting practices can have on candidates during oral assessment.^7,8 Beyond this, however, there is relatively little discussion of the nature and forms of prompting in the literature. For example, in one comprehensive guide to the OSCE, prompting receives scant and somewhat ambivalent mention: ‘Opinions will differ as to whether it is appropriate to prompt candidates in an OSCE, but all examiners should know the agreed policy with regard to prompting in advance in order that all candidates are given the same experience’. ⁹ Even a paper explicitly exploring the validity of oral assessment and offering specific guidelines for practice has little to say about the issue of prompting. ²

Our experience is that there is considerable variation in how practitioners conceptualise prompting and how it is deployed in practice. Clarity around prompting policies and techniques is essential. Our aim in this paper therefore is to provide a clearer guide to the different forms of prompting available to examiners, their potential effects on candidates, and to suggest guiding principles for practice.

A taxonomy of prompting

In order to unpack the term and associated practices, we present a taxonomy of prompting. We think of this taxonomy as a continuum of types, rather than as discrete categories. As we see it, the term ‘prompting’ may refer to any of the 5 following activities: presenting the task; repeating information; clarifying questions; probing questions; and finally, leading questions. The first 4 types of prompting may be useful, depending on the assessment context. The final type of prompting poses a significant threat to validity. It is important to stress that when the term is deployed, what it means in context first needs to be elucidated.

Specific wording on a script

In its simplest form, a ‘prompt’ simply refers to the specific wording on an examination or assessment question which presents the task to candidates. It represents a question or information that examiners should provide all candidates during the examination. This class of prompting represents the minimum level of interaction from the examiner and the most neutral form of prompting.

Repeating information

A second type of prompting is simply repeating the information. Here, examiners intend to remind the candidate to think about information they have been provided, and appear to have forgotten. ‘Remember that this is an 80-year-old. . .’ or ‘Is that still the case in light of the previous history?’ This form of prompting may take the form of re-phrasing the original prompt, or if a candidate is taking the content in an unintended direction, of re-directing the candidate back to the original prompt. Such intervention is usually best expressed in a way that only incidentally cues the candidate that their previous responses were off-track; that is, it should not be accompanied by obvious expressions of disapproval or frustration by the examiner. This form of prompting is simply aimed at giving the candidate the opportunity to correct themselves when it seems clear that their response is a result of misremembering or misunderstanding the question prompt.

Clarifying questions

Third, prompting may go further to indicate clarifying questions, such as ‘Can you be more specific?’ or ‘What do you mean by “X”?’ Questions of clarification are commonly used, but some examiners worry that they may be inappropriate in formal assessment contexts. This will depend on the purpose and context of the examination. For example, a question which aims to give the candidate the opportunity to clarify their response would seem appropriate in most oral assessment contexts; it is, after all, one of the fundamental aims of assessment, to find out what the learner knows and understands. Consistency by examiners is key, so that all candidates get similar opportunities to clarify their meaning. It is the examiner’s responsibility to convey that their purpose is wholly to clarify the response, not surreptitiously cue the candidate. In contrast, a question that searches for an alternative response is better categorised as probing or leading; for example, ‘What type exactly?’, ‘Can you phrase that in a different way?’ Candidates in high-stakes assessment are usually highly attuned to such clues.

Probing questions

Fourth, examiners may be permitted to ask probing questions. This is more difficult to standardise across cases and examiners. Depending on how the candidate responds, an examiner prompts by probing deeper to ascertain how well the candidate understands the specific piece of knowledge, or its significance in a broader clinical context, for example, ‘What might be some implications of that approach?’, ‘Under what circumstances would that be appropriate?’ Some forms of structured oral assessment specifically call for this form of prompting in order to assess the candidate’s clinical reasoning ability.^10,11 This form of probing is illustrated in the paper by Simpson and Ballard, ¹² as an example of assessing the extent of candidates’ clinical decision-making skills. We also endorse the guidance provided by Pylman and Ward ¹³ in their 12 tips article on questioning (in formative contexts), especially the importance of distinguishing ‘probing’ from ‘prodding’, and attempting to create a climate of psychological safety, even while conducting a summative examination.

In this from of interrogative prompting, the concept of equivalence seems a more helpful principle than consistency, because the content of the examiner’s probing is likely to vary between candidates depending on their particular knowledge and responses. The examiner must ensure that the nature of the probing is as equitable as possible, even while different specific questions, or different points of the exam, are used for probing. Another risk of probing is that examiners may focus on their particular ‘hobby horses’. This needs to be recognised as a significant source of unfairness and threat to the blueprint alignment and content validity of the examination, and should therefore be specifically addressed during examiner training. Such an approach from examiners may indicate conflicting understanding of the purpose of oral examinations; some may see these assessments as teaching opportunities rather than the observation-focussed and evaluative approach which most high-stakes assessments require. This makes the clarity of examiner briefing, training and the selection process itself, crucial.

Leading and vague questions

Finally, examiners sometimes enact prompting by asking leading questions. This represents the most ‘intrusive’ form of prompting and is rightly discouraged in most high-stakes assessment contexts. Typical examples of leading prompts include: ‘You mean type II, don’t you?’, and ‘It sounds like you would. . .’ Less helpfully to the candidates’ performance, it can also take the form of very vague prompts such as ‘What else?’, which frequently ends in a guessing game that frustrates both candidate and examiner. Although examiners may have good intentions, such prompting makes the examiner complicit in the candidate’s performance. Even if done consistently for all candidates, it threatens the validity of the assessment result. Unfortunately, this type of prompting often occurs in practice, whether intentional or otherwise.

Guiding principles for practice

When considering the appropriateness of different forms of prompting, it is essential to pause and ensure that the purpose of the specific assessment is clear and well understood by all stakeholders. For example, if the assessment context is a high-stakes mastery examination, prompting beyond mere repeating information or clarifying information may be outlawed by assessment leaders. If the assessment context is lower-stakes and primarily used as a learning opportunity for candidates, more probing questions may be highly useful. ¹³ With prompting, as arguably in all assessment, there can be no hard and fast rules. Although we present these general principles for consideration when using prompting in oral assessment, the defensibility of using different prompts will depend on the assessment context and specific purpose.

Strive to be neutral in interactions with the candidate

Whenever examiners prompt, they should try to do so in a way which neither discourages nor reassures the candidate. Candidates should be alerted to this principle of neutrality, and encouraged not to seek affirmation or censure in examiners’ utterances or body language. Positive comments such as ‘Good job’ or ‘Doing well. . .’, or corrections such as ‘Well, it was actually condition y’. – or worse, the dreaded eye-roll – can have significant impact on candidates’ state of mind and subsequent performance. In our experience, developing an appropriate examination ‘poker face’, offering neither affirming nor disapproving clues to candidates’ performance, can be a significant challenge for many examiners. Where certain examiners excel, they can provide helpful role models through strategic pairing of examiners, where appropriate.

Use prompting in a consistent way for all candidates

Unfairness arises when candidates have variable opportunities to display their knowledge and understanding. Examiners should try to be consistent in their approach, especially when probing candidate responses, although as noted, such consistency may need to be more in the manner and degree of probing as in the content itself. A further issue is how the degree of prompting should impact the candidate’s result. Typically, a greater need for prompting will translate into a lower score,^8,14 but this will depend on the assessment context and criteria, and shouldn’t be assumed to be a universal principle. Respecting the candidate’s thinking processes is also important. Examiner impatience should not be a cue for hasty prompting. A well-trained and reflective examiner will combine assessment protocol with considered judgement to determine if and when a prompt is appropriate.

Be clear and transparent about the required forms of prompting

Confusion typically emerges when stakeholders have different conceptions of what prompting means, and what forms of prompting are permissible. Transparency is an important principle for practice. Firstly, assessment leaders need to be well-versed in the different forms of prompting and have clear and justifiable rationales for which forms are required in the specific assessment context. Written guidelines are a helpful way of being transparent and a good starting point for the development of assessor training.

Ensure assessors are adequately trained in prompting techniques

Assessors need to be adequately trained in acceptable and unacceptable behaviours for the specific context. Helping assessors appreciate the taxonomy of prompting is a good place to start, and it makes different forms of prompting explicit and allows for discussions and deliberations to occur with precise language and a shared understanding of meaning. Accessible guidelines for reference with relevant examples are helpful. Training material may include video footage of examinations or mock examinations, with specific ‘dos’ and ‘don’ts’. Seeing prompting executed in practice (and in context) is usually more impactful than a written document that describes the same issue.

Ensure candidates are adequately briefed on prompting expectations

Candidates also need to know whether they can expect prompting from examiners, and what shape this can take. Anxiety about prompting is a potential compounding factor in assessment that is ancillary to the assessment of knowledge, aptitude or skill. Being clear and transparent with candidates helps alleviate such anxiety. ⁴ An important part of this process is also ensuring that there is a clear understanding of who is responsible for time management during the assessment. For example, in an oral examination it should be clearly stated whether it is the candidate’s responsibility to get through the questions in the allocated time, or whether the examiners are expected to ‘move the candidate along’ to get through in the allocated time.

Encourage ongoing reflection in prompting practice

Reflexivity is also required of assessors (not to mention assessment leads, medical schools and specialist training colleges). Different prompting approaches may enhance utility in different contexts, depending on the purpose of the assessment. Case-by-case review of practice will be required in the early stages of developing an assessment task. Finally, in the spirit of continuous improvement, periodic review of prompting guidance, implementation issues and potential impact on candidates will require ongoing monitoring and evaluation.

Conclusion

Despite historical misgivings about its lack of reliability and the challenges of standardising examiner practices, the oral exam remains an important assessment tool in both medical school and postgraduate contexts. In our experience, better understanding the role of prompting and the different ways of implementing it, is a fundamental consideration in all contexts. Our aim in this paper has not been to prescribe any particular form of prompting, but rather to raise awareness of the different behaviours which tend to be subsumed under the term, while offering some general principles for practice. We would welcome further empirical research to validate the taxonomy presented. Regardless of the assessment context, it is crucial that whenever oral assessment is planned, assessors are appropriately trained in the type and degree of prompting required, and candidates are suitably briefed to know what to expect. We hope this taxonomy and guiding principles are helpful in elucidating the varied forms of prompting practices at examiners’ disposal, and allow assessment leads to determine what works best for their context.