Abstract
The rapid advancement in artificial intelligence has been subjected to much discussion and interest. Chat Generative Pre-Trained Transformer (ChatGPT), a powerful large language model with advanced reasoning ability, has shown promise in medical education and assessment. It has been successful in navigating through not only standard medical licensing examinations1,2 but also in different specialties3 –8 across multiple countries and languages.9,10
In anaesthesia, evidence examining the effectiveness of ChatGPT in tackling multiple-choice questions (MCQs) is emerging. Three sets of MCQs have been analysed—two on the examination overseen by the Fellowship of Royal College of Anaesthetists (FRCA) and two by the American Board of Anesthesiology (ABA). Of the FRCA questions, one assessed 3705 MCQs from a question bank (AnaesthesiaUK Primary FRCA) 11 and the other examined 27 sample MCQs from the Royal College of Anaesthetists website. 12 Both found that ChatGPT-3.5 had results within the thresholds of passing; however, the latter did note a significant improvement in ChatGPT-4. Echoing this, in the assessment of 1231 questions from an ABA preparation book, 13 ChatGPT-3.5 just fell short of a pass but ChatGPT-4 was successful in achieving a passing score. 14
For trainees undergoing specialist training with the Australian and New Zealand College of Anaesthetists (ANZCA), two examinations are compulsory. The first, titled ‘Primary Examination’, assesses candidates in the subjects of applied physiology, pharmacology, anatomy, measurement and equipment. This encompasses both a written and an oral (viva) component. The written aspect comprises 150 MCQs and 15 short answer questions (SAQs). To progress to the viva component, the trainee must pass the MCQs as well as score a minimum of 40% in the SAQs. To achieve a pass in the entire exam, an average mark of 50% or more is required across both the written SAQs and viva components.
The MCQs grading system, in 2017, transitioned to a pass/fail model. The average score required to pass the MCQ barrier was 60.2% 15 for the years 2017 to 2022. Overall, the proportion of candidates passing the entire primary examination stood at 60.5% during the same period.
Previous investigations of ChatGPT-4’s capability to correctly answer anaesthesia-related MCQs have used question sets that originated from an examination preparation book, an unofficial question bank or a very small set of official questions. In this paper, we appraise ChatGPT-4’s capability in MCQs in official questions only but with a considerably expanded set. To closely emulate the ANZCA examination, only officially published questions from the college were used. Unlike the SAQs, where all past papers were published, only one publicly available primary examination paper exists. The questions in this ‘practice’ paper do not arise from one sitting; rather, it is a composite of 120 questions from the two sittings of 2018. 16 The official MCQs of other sittings are unavailable, though some exist in personal websites or question banks set up privately by previous candidates.
In July 2023, ChatGPT-4 was tasked with addressing each question in accordance with ANZCA examination instructions. The same prompt, as seen in the examination, was used for all questions: ‘Please answer the following questions. Each question has five answers. Choose the ONE best answer to each question.’ The answers were then marked by the first author with reference to ANZCA’s recommended reading list of textbooks. 17 In instances where the answers were difficult to discern or information was conflicting, the latest journal articles were utilised. In the rare times where there was still significant doubt, expert opinion was sought. Of note, there were nine questions in the MCQs that involved graphical input, which was beyond the capability of ChatGPT-4 at the time of evaluation.
Overall, ChatGPT-4 achieved a score of 73% with 87 correct answers out of 120 questions (Table 1). If graphical questions were excluded, the mark improved to 78% (87/111 questions).
ChatGPT 4 score by subject, with and without graphic questions.
Among the five subjects examined, pharmacology accounted for the highest number of questions, with 49% of the total (59 out of 120). Physiology questions followed closely behind, accounting for 39% (47/120). Other subjects, namely clinical measurement/equipment (4% or 5/120), statistics (2% or 2/120) and anatomy (2% or 2/120), featured considerably fewer questions.
In the key categories of pharmacology and physiology, ChatGPT-4 performed well. In physiology, it accurately answered 75% of the questions (35/47), which rose to 85% (35/41) if graphic questions were excluded. For pharmacology, the model achieved a mark of 76% (45/59). Unlike physiology, there were no graphic questions in pharmacology.
In the remaining topics, the performance varied. In clinical measurement and equipment, 50% of the questions were answered correctly (5/10), but when excluding graphic questions, the score improved to 63% (5/8). Statistics questions were answered correctly for all two questions. Conversely, both of two anatomy questions were incorrectly answered.
The questions posed here were drawn from the two examination sittings held in 2018. To overcome the MCQ barrier in the first sitting, candidates required a score of 53% (80/150), while in the second sitting, they required 57% (86/150). In 2018, 69% and 84% of actual examinees, respectively, achieved this mark. The specific details regarding ad hoc standardisation and scaling were not publicly disclosed. ChatGPT-4 easily cleared the passing threshold of these MCQ examinations, demonstrating good performance in the core categories of physiology and pharmacology. When graphic questions were excluded, its performance was consistent across all categories, particularly in statistics, clinical measurement and equipment.
The answers generated not only selected the best option but often offered detailed explanations of the rationale behind the selection, and why other options were deemed unsuitable. Not infrequently, the generated answers appeared to display a level of nuance and depth of thought (Figure 1).
Two example answers generated by ChatGPT-4 to multiple-choice questions, appearing to display higher ordered thinking.
ChatGPT-4 has exhibited considerable aptitude, approaching or attaining the passing standard in anaesthesia exams in both the UK and the US.11 –14 In the FRCA question sets, ChatGPT-3.5 accurately answered 69.7% of 3705 questions from the question bank, 11 while ChatGPT-4 answered 63.6% correctly from the 27 sample questions from the Royal College of Anaesthetists. 12 In the ABA exam preparation book, ChatGPT-3.5 achieved a score of 56.2%, 13 while ChatGPT-4 outperformed with a correct rate of 72.1% 14 from a pool of just over 1300 questions. The question sets sourced have been mostly through an unofficial channel such as a preparation book, with the only official source being 27 questions only from the Royal College of Anaesthetists. In our appraisal of ChatGPT-4, utilising an expanded pool of official questions, it not only passed this section of the examination comfortably but compared favourably with the performance of past candidates.
We tried to replicate the exam experience as closely as possible in both the testing and grading aspects. Nevertheless, testing a chatbot in a simulated examination setting is a relatively recent concept, prompting a need to discuss its limitations.
A total of 120 questions were presented to ChatGPT-4 as it was the sole source of official questions. This number is fewer than the 150 questions that are administered in each examination sitting and substantially fewer than the 3705 and 1319 questions assessed in the FRCA11,13 and ABA13,14 examinations. The absence of a larger collection of questions may result in a sample of questions that is not fully representative, reducing validity.
Since the release of the official questions from 2018, the format of the ANZCA Primary examination has been modified. 16 Key changes made include: a reduction from a five-option single best response to four options, a more refined standardisation scheme and an enhanced metareview process. These substantive modifications might mean the that performance in this paper might not necessarily carry through to the new format.
Since this evaluation of MCQs, ChatGPT-4 has seen significant improvement. Whereas it was not capable of interfacing with graphic content in its base model previously, it is now able to both interpret and generate images to questions. 18 Whereas ChatGPT-4 scored zero for visual questions in this study, now it has the engineering capability to address them.
Our study illustrates the ability of a large language model such as ChatGPT-4 to comfortably clear the MCQ threshold of the ANZCA Primary exam. It would be prudent for candidates, however, not to rely solely on ChatGPT-4. It not infrequently generates factual errors and has a tendency to deliver answers confidently, even when 27% of the questions were answered incorrectly. Despite its shortcomings, this easily accessible chatbot could offer advantages to primary examination candidates as an alternative study aid, providing immediate, personalised, problem-oriented solutions at any time of the day or night. Improvements are expected with the evolution of artificial intelligence models. Furthermore, the notion that an untrained, general-purpose chatbot was able to pass the MCQs might inspire further research, leading to the development of innovative alternative approaches to anaesthesia learning and assessment.
Footnotes
Author Contribution(s)
Acknowledgement
Declaration of generative AI and AI-assisted technologies in the writing process: in reviewing this work, the authors used ChatGPT to enhance readability and language. The authors reviewed and edited the content and assume full responsibility for the content of the publication.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
