Use of Mini Clinical Evaluation Exercise as a Tool to Assess the Orthodontic Postgraduate Students

Abstract

Abstract

Introduction: The mini CEX (Clinical Evaluation Exercise) is a workplace-based assessment tool intended to assess the clinical performance and a range of other skills such as communication skills, clinical judgment, professionalism, efficiency, and the overall clinical care while directly observing a real patient encounter. The clinical sessions are followed by structured feedback sessions between the assessors and the trainees. Mini CEX focuses on the actual performance and not just knowledge. This leads to better learning as well as helps to develop and strengthen the doctor–patient bond.

Materials and Methods: 8 staff members and 9 orthodontic postgraduate students volunteered to participate in this study after the orientation session was taken to sensitize them about the same. An informed consent was taken from all of them. All the students had 2 sessions with all the assessors. As per the protocol of Mini CEX, immediate feedback about their performance was given to the students after each session. Feedback was also obtained from the assessors and the students after the completion of the exercise. This feedback allowed the assessors and the students to give their views on the exercise.

Result: Mean scores of all the students significantly improved in the second session as compared to the first encounter. This was attributed to the feedback which they obtained from the assessors immediately after the session, which helped them improvise on various aspects of patient care. The students themselves felt that the exercise helped them learn better.

Conclusion: Mini CEX can hence be considered as a valid formative assessment tool which when routinely used could enhance the learning experience, improve the communication skills of the students, as well as improve the doctor–patient bond.

Introduction

The assessment of the students based on the number of marks obtained in examinations does not give a true picture of the competence of the students in dealing with the patients. It probably gives an indication of how well read the student is or how good or bad the end result is, without hinting at how it was obtained. Till date, there is no “gold standard” to assess the performance of the students while they interact with the patients during “case history recording” and “clinical examination.” It is agreed upon that any clinical performance has many aspects to it like communication skills, professionalism, clinical judgment, counseling, and the overall competence.^{1, 2} These skills are rarely assessed in the traditional means of assessment. Also, only the final outcome is assessed and not the stepwise procedures followed to reach the final result.

Norcini has summarized the primary weak areas of the traditional assessment method, namely lack of assessment of a large variety of cases, examination by a limited number of examiners, and testing of only a few competencies. These all lead to a lack of reproducibility of the scores.^{3, 4}

However, the assessment can be perceived as “threatening” to students and may alter how they perform, and also change the nature of the collegial relationship between the teacher and the student by strongly emphasizing the “assessor” role of the teacher.^{3, 5}

In Miller’s framework for assessing clinical competence, mini CEX falls in the highest level of the pyramid and collects information about a student’s performance in his/her everyday practice. ⁶ It was originally developed in the USA for the evaluation of internal medicine residents’ clinical skills. The exercise involves direct observation of real patient encounters and immediate structured feedback to the learner after the encounter.^{3, 7, 8}

Mini CEX has found a good acceptance in medical education but not as much in dental education. Barring a few reported incidences where mini CEX was administered to the dental students, the educational research in this area is minimal. A search of databases in February 2017 did not reveal a single incidence of this exercise being carried out to assess students pursuing orthodontics.

In view of this, mini CEX was introduced to the orthodontic postgraduate students to check its feasibility as an assessment tool and to improve learning amongst the students. It also aimed at improving other skills required to acquire clinical competence and to become a good clinician.

Materials and Methods

The study design was quasi experimental and was carried out in the Department of Orthodontics and Dentofacial Orthopedics in a dental college during May–June, 2017. It was approved by the Scientific Advisory Committee and exempted from ethical review. An orientation session was first conducted and a video demonstration was given to sensitize and introduce the orthodontic faculty and postgraduate students to mini CEX. Any doubts expressed were addressed and clarified. A mini CEX checklist was obtained for judging the students on their performance (Annexure 1). Formal training sessions for the faculty were not conducted as it did not seem necessary keeping in mind the study done by Cook in 2009 which demonstrated that assessor training does not necessarily improve interassessor reliability or accuracy of mini CEX scores. ⁹

All the 9 orthodontic postgraduate students volunteered to participate in the study. An informed consent was taken from all of them (Annexure 2). The faculty members of the Department of Orthodontics were requested to assess the students based on the checklist prepared. 8 staff members agreed to assess the students as a part of the study.

All the students had 2 sessions with all the assessors. So, in all, there were 144 sessions conducted over a period of 2 months in the department. All efforts were made to make sure the cases selected were of equal complexity and difficulty level to make the assessment as standardized as possible. Each mini CEX session lasted for about 10 minutes. Each student was rated on his/her ability to take complete patient history, physical examination, communication skills, clinical judgment, professionalism, organization/efficiency, and the overall clinical competence and care. As per the protocol of mini CEX, immediate feedback about their performance was given to the students after each session. These feedback sessions lasted for about 5 minutes and started with appreciating the students on the things that were done well, and hence encouraging them, followed by pointing out to them their mistakes and the things which were not done well and finally explaining them the ways they could improve on the things wrongly done.

At the end, the students’ and the teachers’ perception of the exercise and the students’ performance were obtained by means of structured questionnaires. This feedback allowed them to give their views on the exercise (Annexures 3 and 4, respectively). It had a combination of closed and open-ended questions. Another feedback took opinion of the assessors on the students’ performance (Annexure 5). Statistical analysis of the various checklists obtained was done using paired t test on SPSS software.

Results

As stated earlier, feedback from different sources were obtained to analyze the merits and demerits of the assessment method under investigation. In addition, the students were assessed based on the checklist during their interaction with the patient.

I. Students’ assessment based on the checklist (Annexure 1) and feedback on the students’ performance by the assessors (Annexure 5).

As seen in Table 1 and Figure 1, the mean scores of all the students significantly improved in the second session as compared to the first encounter. The P value of the means between the 2 sessions was found to be less than .05 indicating the difference was statistically significant as seen in Table 2. This can be attributed to the feedback which they obtained from the assessors immediately after the session, which helped them improvise on various skills mentioned earlier.

Table 1.

Mean Mini CEX Value at 2 Different Sessions

	Mean	Std. Deviation
First Session	8.50	1.565
Second Session	14.46	3.552

Table 2.

Comparison of Mean Mini CEX Value at Both the Sessions

First Session–Second Session	Mean Differences	Difference in Std. Deviation	P value
	–5.958	4.012	.000*

Note. *P value < .05 statistically significant.

Figure 1.

Comparison of the Scores Between the First and Second Session of Mini CEX

II. Feedback on the Assessment Method by the Assessors (Annexure 3)

All the assessors felt that this exercise if routinely carried out would improve the doctor–patient bond in addition to the improved communication skills of the doctor. They also agreed to the possibility of a decrease in the litigations against the doctors probably because of the improved bond and rapport with the patient.

Barring 2 assessors, all felt that it would help reduce the dropout rates amongst the patients. All the assessors felt that the exercise would improve learning among the students and could be used as a formative assessment method for the students.

III. Feedback on the Assessment Method by the Students (Annexure 4)

All the students felt that this exercise helped them take the case history and carry out the clinical examination in a detailed manner. They also thought that the whole exercise helped them learn better. Although almost all the students felt that the presence of an assessor was intimidating, they reasoned out that this being the first time, they felt nervous, and if carried out routinely, they would get accustomed to the presence of staff while they dealt with the patients.

The students appreciated the feedback sessions the most. They felt that the constructive feedback helped them identify their weak areas. They thought that with the help of the immediate feedback sessions, they could correlate better and, hence, learn better as it gave them an opportunity to clarify their doubts, if any, with the assessors.

Kappa analysis was also performed to check the interobserver reliability of the checklist made and hence the exercise. The interobserver agreement ranged from 42.9% to 100% and the Kappa coefficient value was found to lie between 0.462 and 1. This showed the exercise was reliable.

Discussion

The mini CEX, since its inception, is considered as a positive tool for the formative assessment of residents in the clinical departments. ¹⁰ Since it is directly observed, it evaluates the actual skill while taking case history, carrying out the examination, and planning the treatment for the patients. Its focus is more on the actual performance and not mere knowledge. Although it can be used both for summative and formative assessment, its more suitable for the formative assessment as it will promote learning amongst the students. ¹¹

Because of its objective approach, this technique is both reproducible and reliable ¹² which was well demonstrated by means of the kappa scores. This was in accordance with Norcini et al who in their study to evaluate the examiner differences in mini CEX found that this exercise produced comparable scores over examiners and settings. This makes it a worthwhile method for evaluation at the local level. ¹³

The fundamental need for any formative assessment is to facilitate learning. Any evaluation exercise can be called successful only if it encourages students to look for more information. This can happen if evaluation is followed by feedback, which is prompt, specific, and actionable. ¹⁰ Since performance-based workplace assessments are immensely important in clinical training, the provision of external feedback to the students about their clinical skills in mini CEX is necessary and plays a significant role in the development of expertise.^{14, 15}

Archer ¹⁶ stated that feedback should not be exclusively trainer-driven but a 2-way process in which trainers provide comments and at the same time encourage trainees to self-reflect on their performance. Archer’s model for effective feedback includes self-monitoring (reflection on action) supported by external feedback and linkage with personal goals (action plan) in a coherent process rather than a series of unrelated events.^{17, 18} The self-reflection and introspection provided to the students after the feedback helped them to take action and follow-up on it, which in turn improved their learning process, like in the study done by Sudarso et al in the year 2016. ¹⁹ It was also found that feedback is more effectively used by learners if provided in a formative assessment than a summative one. ²⁰ Hence, mini CEX can be more effectively used as a tool for formative assessment.

It was observed that mini CEX provided useful feedback to the students to improvise on all the aspects of patient care. This was demonstrated by the improvement in the scores obtained by the students. Also, the students themselves felt that the exercise helped them learn better as indicated by a few comments such as: “It helped us to look at minute details carefully which we otherwise miss to record and pay attention to,” “It has greatly helped in looking at and dealing with patients in a more comprehensive way,” “We became aware about the mistakes or incorrect things we were doing subconsciously,” and “The exercise made me realize that we often tend to overlook small details which matter a lot, and it helped me better those.”

All the students and assessors felt that this exercise if routinely carried out would improve the doctor–patient bond. They also agreed to the possibility of a decrease in the litigations against the doctors probably because of the improved bond and rapport with the patient. Most of them also felt that it will help reduce the dropout rates amongst the patients.

It was even thought before conducting the exercise that the presence of the staff members observing them while they recorded the case histories will be stressful for the students. In the study, though all the students felt that the presence of a staff member was intimidating while they examined the patient, the very fact that all the students agreed that the exercise helped them learn better is a reason enough to use this method routinely in the clinics.

Conclusion

Mini CEX is a practically suited assessment tool in situations involving patient–doctor interactions specially focusing on professionalism and communication skills.

Mini CEX can be considered as a valid and reliable formative assessment tool which when routinely used can enhance the learning experience of the students. It is extremely useful in situations involving patient–doctor interactions as it focuses on professionalism and communication skills apart from other soft skills which will in the long run help to improve the doctor–patient bond. The best part of the method is its feasibility as the exercise can be carried out in the routine clinical settings without any need for additional infrastructure.

This study suggests that mini CEX can be used as a reliable tool to assess postgraduate students. It also suggests that implementation of this exercise routinely in clinics will improve the doctor–patient relationship by enhancing all the qualities required to be a good practitioner.

Learning Experience

A lot of planning was required to carry out the exercise especially because this was the first such attempt in our department and orthodontic specialty, and also, one of the very few in dentistry. It did require a lot of support from the colleagues and the volunteers. But after the orientation sessions held with the participants, they seemed to have liked the concept, and then their enthusiastic participation helped me in smoothly conducting the exercise. Overall it was a very good learning experience for all involved and a very satisfying one for the author.

Further Studies

We should try and obtain a multisource feedback. The feedback obtained from the patients will be a real indicator of whether this exercise can help or not in improving the doctor–patient bond and whether or not it will reduce the dropout rates amongst the orthodontic patients.