The Effect of Uncertainty Training on the Improvement of Diagnostic Ability in Chinese Medical Students

Abstract

Objective

To evaluate the effect of the uncertainty training on improvement of students’ diagnostic ability.

Methods

Data were collected on 70 fifth-year medical students enrolled in the Case Discussion courses on Obstetrics and Gynecology in the spring of 2020. Of these students, 36 were in the uncertainty training group and 34 in the control group. The effect of training was evaluated by cognitively diagnostic assessment which mapped exam questions to 4 attributes assessing clinical reasoning and basic science knowledge.

Results

Uncertainty training was able to improve students’ ability to use basic science concepts for inference and problem solving, and the ability to integrate complex clinical information to arrive at a diagnosis. But it could not improve students’ ability on the basic recall of foundational concepts and the ability to use basic science concepts in clinical reasoning. Medical students could do well in integrating complex clinical information although they didn’t recall basic science knowledge well.

Conclusion

Uncertainty training could be used as an effective teaching method in Case Discussion course on Obstetrics and Gynecology. However, students still need to improve their basic knowledge besides the training.

Introduction

Diagnosis is a complex processing of knowledge and experience using both critical thinking skills and intuition while recognizing uncertainty. Sources of uncertainty include the complexity of clinical information; the probability of particular outcomes; and individual clinician characteristics, such as tolerance for ambiguity ¹ or an individual's ability to cope with complexity, risk, and uncertainty. Uncertainty ² is more of an issue today because of the development of new technologies and the consequent physicians’ insecure attitude toward adopting these new technologies, as well as patients’ involvement in clinical practice. ³

There have been some articles about decision making under ambiguous or uncertain conditions. Politi et al ⁴ presented a communication model to help better understand quality medical decision making, and how patient-centered, collaborative communication enhances the decision-making process. They researched on shared mind and cognitive and communicative skills to highlight how they could facilitate the management of uncertainty during the interactive process involved in medical decision making. They provided simple examples about how to frame messages to achieve shared mind and foster uncertainty tolerance. The results demonstrated strategies such as providing clear explanations, checking for understanding, eliciting the patient's values, concerns, needs, finding common ground, reaching consensus on a treatment plan, and establishing a mutually acceptable follow-up plan could facilitate collaborative decision making. Mumford et al ⁵ developed an ethics training course based on a proposed sensemaking model. The model assumed that a variety of situational considerations would influence a scientist's initial appraisal of the problem situation, including professional codes of conduct, perceived causes of the situation, personal and professional goals, and perceived requirements for attaining these goals. They showed this training led to sizable gains in ethical decision making and these gains were maintained over time. Cristancho et al ⁶ explored a “Reconciliation Cycle” as core element of an intraoperative decision-making model of how experienced surgeons assess and respond to challenges, to better train for flexibility under uncertainty. They analyzed semistructured interviews using constructivist grounded theory and follow-up interviews about the most challenging cases. The Reconciliation Cycle constituted an iterative process of “gaining” and “transforming information.” The cyclical nature of surgeons’ decision making suggested that transforming information require a higher degree of awareness, not yet accounted by current conceptualizations of situation awareness. These articles suggest the medical education community now recognizes the need to formally address uncertainty as part of training. The training helps better understand the role of uncertainty in decision making and develop framework for managing uncertainty.

Facing the uncertainty of clinical thinking, doctors’ responsibility is to use scientific methods to reduce the impact of uncertainty on clinical practice and avoid damage to patients, and strive for better clinical effect. ⁷ Helou et al ⁸ reviewed articles related to uncertainty, and developed a framework to guide medical decisions in various uncertain situations. Their framework depicted the interplay between 6 main themes: recognition of uncertainty, classification of uncertainty, stakeholder perspectives, knowledge acquisition, decision-making approach, and evaluation of the decision-making process. This process originated in a clinician's ability to recognize uncertainty. It asked the decision maker to continually increase her or his understanding of the uncertainty before ultimately arriving at that decision through a purposefully chosen approach or by effectively reducing the uncertainty in the situation. However, the framework hasn’t been validated in teaching diagnosis yet, especially in Chinese students. In China, managing uncertainty is often taught indirectly through observation or informal clinical experiences. While these informal experiences may be beneficial, it is recommended to formally address uncertainty as part of training. ⁹ Therefore, we chose the framework developed by Helou et al as a formal way of training in Chinese medical students to see whether it could improve their ability of diagnosis under uncertain conditions.

To glean relevant diagnostic information about teaching effectiveness, we chose cognitively diagnostic assessment reported by Bangeranye et al. ¹⁰ This assessment measured instructional quality that didn’t rely on students’ ratings of the ability of the instructor, the content of a course, or other measures. The exam questions were mapped to 4 attributes and analyzed the students’ overall mastery of the content tested and the percentage of students mastering each attribute. We furthered our understanding cognitive integration by using a Likert scale to score the students’ answer. ¹¹ It could determine the degree to which the teaching goals have been achieved by the students. We implemented this cognitively diagnostic assessment in an undergraduate Obstetrics and Gynecology course to evaluate course effectiveness from in-class exams.

Methods

Participants

This is a prospective study carried out in the Second Xiangya Hospital of Central South University. The participants were 70 fifth-year medical students enrolled in the Case Discussion courses on Obstetrics and Gynecology from March to June 2020. The fifth-year medical students study clinical classes and observe clinical practice on schedule. Students were randomly assigned to uncertainty training group or control group. The study was approved by the Medical Ethics Committee of the Second Xiangya Hospital (approval number 2020-584, January 14, 2020). Written informed consent was obtained from the subjects prior to study initiation. All students were provided with an information sheet detailing the purpose of the study and the ways in which they could opt out of participation.

Online schooling procedure

Because of the national spread of corona virus disease 2019 (COVID-19)， online schooling was applied to provide education during isolation. Lessons were taught online by Tecent Class or Tecent Meeting (special teaching or conference software supplied by TecentCorp). We chose the Case Discussion course for study. Case Discussion is a course taught during the fifth year of medical school after theory course had been finished. Its goal is to apply knowledge that students had learned on theory course. It presented cases from clinical practice to train students to diagnose. The courses were taught in small groups using a hybrid case-based/problem-based curriculum. Four cases were discussed each course, including common diseases on Obstetrics and Gynecology. There was only one diagnosis for each case. Cases were the same for both groups. In the uncertainty training group, students were trained to diagnose by the procedure of “uncertainty training” below. The training was one time intervention. In the control group, students were taught by the classic way: teachers analyzed the clues from the case and guided students to arrive at diagnosis. The teachers were the same for both groups. They had previous training to conduct the procedure of uncertainty training. Time restriction on the course was 45 min for both groups.

Uncertainty training

Managing uncertainty was trained by using the framework presented by Helou et al (Figure 1). ⁸ First, students would be trained to recognize uncertainty in situation of the case. Next, students would attempt to understand what made the situation uncertain: Was it the ambiguity of a wide probability range or insufficient evidence? Once the uncertainty was identified and classified, students needed to seek more knowledge to clarify the decision. At any or all of these steps, students must assess the situation, synthesize the information, and reflect on the potential outcomes. The culmination of these steps might reduce the uncertainty in the situation to a point where one option becomes clearly better than the other, and there was no longer uncertainty. On the other hand, if uncertainty persist, students would consider various approaches to help, including the use of tools, such as decision aides, decision analysis calculations, or decision trees to illustrate the probability of various outcomes, or the use of team-based or shared decision-making strategies.

Figure 1.

Framework for making decisions under uncertain conditions. Four themes are listed as strategies to reduce uncertainty. The decision maker uses these strategies along with an ongoing evaluation process of assessing, synthesizing, and reflecting on information obtained. ⁸

Cognitively diagnostic assessments

The effect of training was evaluated by cognitively diagnostic assessments. ¹⁰ Both the uncertainty training group and control group received the same tests after course, and the test should be completed within 24 h after being received. The tests consisted of 6 new clinical cases with diagnoses different from the ones studied during training. Each case included 4 questions assessing clinical reasoning or basic science knowledge. The content of the questions was mapped into 4 attributes:

attribute α1 concerned the basic recall of foundational concepts;

attribute α2 concerned the ability to use basic science concepts for inference and problem solving;

attribute α3 concerned the ability to use basic science concepts in clinical reasoning; and

attribute α4 concerned the ability to integrate complex clinical information to arrive at a diagnosis.

The example below illustrated how to identify the set of attributes required for a particular item:

A 21-year-old female primigravida is brought to the emergency department 10 min after giving birth to an infant at home. She received no prenatal care. Her medical history is insignificant. Her blood pressure is 150/95 mm Hg, and heart rate is 100/min. Physical examination reveals an enlarged, tender liver and 2 + symmetrical edema of the lower extremities. The platelet count is 50,000 cells/mL. Schistocytes are present on the peripheral blood smear. Liver enzymes are elevated. The urine is positive for 2 + proteins.

α1 What is the basic pathophysiology of her hypertension?

α2 How could we differentiate her hypertension from other types?

α3 What is the reason for Schistocyte?

α4 What is the best next step in the management of this patient?

The students’ answer to each question was scored by content experts based on a Likert scale (Table 1). ¹¹ Each of the attribute was recorded as “1” if the answer's score was 4 or more, or “0” if the answer's score was blow 4. The 4 binary attributes could be used to construct 16 different attribute profiles (24 = 16). For example, an attribute profile consisting entirely of zeros, (0000), showed failure to master all 4 attributes. An attribute profile of all ones, (1111), indicated mastery of all 4 attributes. The profile (0101) showed that attributes α2 and α4 were mastered by the student but not attributes α1 and α3.

Table 1.

Likert scale used to score the answer.

1	2	3	4	5	6	7
Identifies incorrect points	Identifies 1 correct point but no rationale. May also identify incorrect points	Identifies 1 correct point and provides a correct rationale for the point	Identifies 1 KEY point but no rationale. Also, identifies incorrect points	Identifies 1 KEY point but no rationale	Identifies 1 KEY point and provides a correct rationale for the point	Identifies >1 KEY point and provides a correct rationale for the points

Data collection

Before lessons began, students were invited to investigation of study basis, including interest on Obstetrics and Gynecology, preparation for lessons, study motivation, etc, assessed by a 5-point Likert-type scale (1 = not at all, 5 = extremely). After lessons finished, students were invited to investigation of study effect, including satisfaction with teaching, study efficiency, etc, assessed by a 5-point Likert-type scale. The investigation was put on a special software for investigation named “Investigation Star.” The questionnaires used in this study were validated.^12,13

Statistical analysis

Statistical analyses were performed with STATA10.0 (StataCorp, College Station, TX, USA). Descriptive statistics included means and standard deviation for continuous variables and frequency distributions for categorical variables. Comparisons between categorical variables were tested by Chi-square test. Comparisons between normally distributed continuous variables were performed using Student’s t test or analysis of variance. P value < .05 was considered statistical significant.

Results

Table 2 presented the means and standard deviations of students’ study basis. Of these students, 36 were in the uncertainty training group and 34 in the control group. Scores for interest on Obstetrics and Gynecology, preparation for lessons, study motivation, interest on questioning, reaction, self-study ability, comprehension ability, analysis ability, and applied ability were not significantly different between the 2 groups.

Table 2.

Students’ study basis as assessed by a 5-point Likert-type scale (1 = not at all, 5 = extremely).

N	UNCERTAINTY TRAINING		CONTROL
	36		34
	MEAN	SD	MEAN	SD
Interest on Obstetrics and Gynecology	4.02	0.61	3.85	0.7
Study motivation	3.42	0.69	3.35	0.65
Preparation for lessons	2.75	0.84	2.53	0.19
Interest on questioning	2.39	0.9	2.56	1.08
Reaction	3.17	0.7	3.09	0.83
Self-study ability	3.47	0.65	3.18	0.8
Comprehension ability	3.58	0.6	3.35	0.81
Analysis ability	3.44	0.73	3.26	0.75
Applied ability	3.14	0.59	3.26	0.71

Figure 2 summarized the percentage of mastery for each individual attribute. In this study, the students that had mastered attribute α2 or α4 were significantly higher in uncertainty training group comparing to control. The students who had mastered attribute α1 or α3 were not significantly different between the 2 groups. At a glance, fewer students seemed to have mastered α1 or α2 both in the uncertainty training and control group.

Figure 2.

Group attribute mastery profiles. *p < .05 when comparing the percentage between uncertainty training and control group.

The observed frequencies of the 16 different attribute profiles were reported in Figure 3. Attribute profiles of 1101, 0111, 0101, 0001 were significantly higher in uncertainty group comparing to control. Attribute profiles of 0000 and 0010 were significantly lower in uncertainty training group comparing to control. Attribute profiles of 1111 were not significantly different between the 2 groups.

Figure 3.

Distribution of student profiles across attribute mastery levels. *p < .05 when comparing the percentage between uncertainty training and control group.

Table 3 showed the means and standard deviations of the students’ study effect. The students in uncertainty training group scored higher on satisfaction with teaching, saving time on preparation or review, study efficiency, easy to understand the online lessons. There was no difference between the 2 groups on the scores of mastery on lesson, following others’ questioning and involvement on discussion.

Table 3.

Students’ study effect as assessed by a 5-point Likert-type scale (1 = not at all, 5 = extremely).

N	UNCERTAINTY TRAINING		CONTROL		p
	36		34
	MEAN	SD	MEAN	SD
Mastery on lesson	3.63	0.61	3.61	0.5
Satisfaction on teaching	4.15	0.63	3.63	0.9	<.05
Saving time on preparation or review	3.13	1.02	2.76	0.5	<.05
Following others to ask questions	3.67	0.79	3.42	0.83
Study efficiency	4.04	0.63	3.48	0.71	<.05
Lessons are easy to understand	4.36	0.61	3.84	0.71	<.05
Involvement on discussion	3.67	0.67	3.63	0.65

Discussion

In this study, we found uncertainty training group score comparatively higher than the control group on diagnostic ability. The improvement was mainly on using basic science concepts for problem solving and integrates complex clinical information to arrive at a diagnosis. However, the uncertainty training group didn’t score higher on basic knowledge, or their reasoning ability.

Diagnosis is process of arriving at the true disease by coping with uncertainty. ¹⁴ Data used for diagnosis may be conflicting, ambiguous, or scarce. Understanding the uncertain and unaware quadrant of “unknown unknowns” can have drastic implications for diagnosis. ¹⁵ Recognizing uncertainty could initiate information gathering or as an impetus to select a specific decision approach. ¹⁶ While failure to recognize uncertainty could lead to failure to identify the correct problem and asking the wrong question. ¹⁷

We applied Helou's ⁸ framework in our study, and found the uncertainty training group mainly improve on students’ ability to apply the knowledge they had learned, but not how much the students remembered of the basic knowledge, or their reasoning ability. It accorded with the nature of the framework as guidance of reasoning. It taught students a method to deduce, but did not change their knowledge base. In attribute α2 for application, students were expected to use their basic science knowledge, together with pertinent information from the patient presentation, to deduce diagnosis and its mechanism. In attribute α3 for reasoning, students were expected to make a direct link between the mechanistic explanation of diagnosis to different aspects of the patient's presentation, using the provided information. Rather than expecting medical students to memorize a long list of concepts and “facts,” educators should spend more time emphasizing the importance of critical inquiry and discussing strategies for weighing the risks, benefits, and alternatives to various diagnosis. ¹⁸ As attribute α4—the ability to arrive at a correct diagnosis is the target of the course of Case Discussion, uncertainty training turned out to be effective and could be used as a teaching method of the course.

Cognitively diagnostic assessment was aimed to provide specific diagnostic information on students’ knowledge and processing skills. It evaluated the extent to which the initially set teaching goals for a course were actually achieved. It could also be used to provide students with direct feedback on their learning progress—specifically which skills they had mastered and which require more work on their part. ¹⁰ Teaching medical trainees to examine their cognitive biases could contribute to reducing diagnostic errors. ¹⁹

According to the 4 attributes in our study, students could do well in integrating complex clinical information although they didn’t recall basic science knowledge well. The results were similar to that from Bangeranye et al. ¹⁰ It might be explained that students on clinical medicine tended to diagnose by clinical symptoms and signs. Although uncertainty training could not change much on students’ knowledge basis, it increased the diagnostic accuracy from 67.9% to 85.7%. The Likert scale scoring could indirectly capture the use of learners’ integrated basic science knowledge in clinical diagnosis. ¹¹ The distribution of student profiles across attribute mastery levels also showed students did better in integrating complex clinical information than recalling basic science knowledge. Uncertainty training increased the accuracy of attribute α2, making 0111 the most frequent in the uncertainty training group. The training also reduced the percentage of 0000, increasing the percentage of students that at least diagnosed right (0001). In addition, investigation of study effect showed uncertainty training increased students’ efficiency and understanding. Seen from all above, uncertainty training could improve students’ diagnostic ability, but not basic knowledge.

Limitation

A limitation of this study was that participants were recruited from a single institution on Obstetrics and Gynecology course. It limited the generalizability of the findings. The result might differ from other schools or courses. An additional limitation was the self-reporting survey methods to collect data from students on study basis and effect. Another comparison limitation was that the effectiveness of training was examined only in the fifth-year medical students. It was not sure whether the training would be effective for people who have greater professional expertise, such as residents.

Conclusion

In this study, we validated Helou's framework in teaching diagnosis in Chinese students, and evaluated teaching effectiveness by cognitively diagnostic assessment. The result showed uncertainty training could be used as an effective teaching method in Case Discussion course on Obstetrics and Gynecology. However, students still need to improve their basic knowledge besides the training. Further studies are needed to validate our result in other courses and schools and people with greater professional expertise.

Supplemental Material

sj-doc-1-mde-10.1177_23821205241226818 - Supplemental material for The Effect of Uncertainty Training on the Improvement of Diagnostic Ability in Chinese Medical Students

Supplemental material, sj-doc-1-mde-10.1177_23821205241226818 for The Effect of Uncertainty Training on the Improvement of Diagnostic Ability in Chinese Medical Students by Xiaomeng Xia, Yi Zeng, Pixiang Pi, Xianqing Wu, Xiaoling Fang, Jianlin Chen and Yan Zhong in Journal of Medical Education and Curricular Development