Abstract
Introduction
Competency-Based Medical Education [CBME] is a framework for organizing medical education programs that emphasize ongoing assessments to monitor learners’ progress. Implemented by the erstwhile Medical Council of India since the 2019–2020 academic year, this approach has influenced undergraduate summative assessments for Forensic Medicine. To assess the implementation, effectiveness, and alignment of the new curriculum with the goals of medical education, it is necessary to compare summative assessment question papers of undergraduate MBBS students from 2017 to 2020.
Methodology
Researchers evaluated 32 summative examination question papers from eight medical universities across India, categorizing them into structured, non-structured, and action verb-based categories, according to Bloom's revised taxonomy's cognitive domain (remember, understand, apply, analyze, evaluate, and create). They compared the data with the pre- and post-implementation benchmarks of the CBME curriculum of the National Medical Commission (NMC) from the 2019 batch MBBS program.
Results
Most universities used unstructured questions and most marks were allocated to remember the levels of the cognitive domain. It was also found that marks were skewed towards specific topics in all universities, leading to inappropriate sampling and coverage of the competencies. The study also revealed that core areas, such as medical jurisprudence and toxicology, were inappropriately assessed.
Conclusion
The quality of the summative exam questions in Forensic Medicine and Toxicology subjects was poor and not aligned with the CBME, highlighting the need to assess the clarity and utility of blueprints currently employed by universities.
Introduction
Competency-based education is an outcome-oriented methodology for designing, implementing, assessing, and evaluating medical education programs using an organizational competency framework. This approach is more than just a different teaching style; it requires a different assessment perspective. It emphasizes the need for ongoing and longitudinal assessments, allowing educators to diagnose learners’ stages of development and determine whether they require additional or alternative learning opportunities to acquire competency. Evaluating a competency-based curriculum is essential for its effective implementation. 1 Recognizing the significance of assessment validity for educators is therefore essential. This is because it is essential to guarantee that we are drawing logical conclusions about our pupils’ academic progress to aid in guiding their future educational endeavors. Assessment plays a pivotal role in effective teaching-learning processes. To impart knowledge effectively, it is imperative to assess it consistently and accurately. 2 The evaluation of students’ proficiency in Forensic Medicine has changed significantly within the competency-based medical education paradigm. The main goal of this assessment was to gauge the students’ understanding of medical jurisprudence, toxicology, postmortem techniques, and forensic pathology. However, with the new CBME curriculum, evaluation goes beyond simple factual memory and emphasizes the application of knowledge to real-world clinical situations. 3 The redesigned written summative assessment included clinical case vignettes and simulated scenarios that required students to integrate their knowledge, clinical skills, and ethical principles to solve complex forensic medicine cases. This shift aligns with the core principles of CBME, which prioritize the development of competencies essential for the practice of medicine, including critical thinking, decision-making, effective communication, and ethical reasoning. 4 Considering this, the National Medical Council of India implemented the CBME curriculum in the nation's medical colleges beginning in the 2019–2020 academic year. 5 This change in medical education has unavoidably influenced the approaches employed in undergraduate summative assessment papers for Forensic Medicine. 6 The National Medical Commission of our country divided Forensic Medicine & Toxicology into various subtopics, such as general information, forensic pathology, clinical forensic medicine, medical jurisprudence, forensic psychiatry, forensic laboratory investigation in medical-legal practice, emerging technologies in forensic medicine, general toxicology, chemical toxicology, pharmaceutical toxicology, and biotoxicology. These subtopics were subdivided into various specific learning objectives. 5 As the transition to CBME continues to shape medical education in India, we compared the summative assessment papers of undergraduate MBBS students in the 2017–2020 batches in Forensic Medicine & Toxicology. It is essential to compare and document the changes in assessments to align them with the overarching goals of medical education.
Methodology
This retrospective, cross-sectional, observational study (STROBE) was conducted at our college between February and March 2024 with approval from the Institutional Ethics Committee of our college (IEC/BU/2024/Ex. 20/63/2024). We analyzed 32 summative examination question papers from eight medical universities across various states of India. The question papers were obtained from the public domain, and researchers collected undergraduate examination question papers in Forensic Medicine and Toxicology from universities’ websites and the college records of constituent colleges after conducting a thorough search for the last four years (2017–20 Batch). The universities’ identities were concealed during the analysis, and explicit consent was not required as all data were sourced from public domains. To enhance the sampling methodology, the country was divided into four geographical zones: east, west, north, and south. Two medical universities were selected from each zone with the criterion that their summative assessment papers were publicly accessible on their respective websites. The questions were categorized based on type into structured questions, non-structured questions, and the presence or absence of action verbs. The action verbs in summative assessment focus on measuring students’ overall competence and understanding at the end of the learning period. These verbs were chosen to align with the assessment's purpose of evaluating learning outcomes and making high-stakes decisions about student performance. 6 Additionally, the questions were categorized based on the cognitive domain of learning according to Bloom's revised taxonomy (remember, understand, apply, analyze, evaluate, and create). 7 Inter-rater reliability was not assessed, as all authors adhered to the Revised Bloom Taxonomy and Guidelines provided by the National Medical Commission, India.
Undergraduate training in Forensic Medicine & Toxicology encompasses Medical Jurisdiction, Forensic Pathology, and Toxicology as primary areas of study. These are divided into six subcategories, including Legal Procedure, Medical Jurisprudence, Forensic Pathology, Forensic Psychiatry, Laboratory Techniques, Emerging Trends, and Toxicology, as specified by the Medical Council of India in 1997. The allocation of marks for each topic was evaluated for each examination. Forensic Pathology is subdivided into several subtopics, including identification, postmortem changes, mechanical injuries, mechanical asphyxia, thermal deaths, sexual offences, and medicolegal issues related to pregnancy, delivery, and abortion. Toxicology is divided into four subtopics: general toxicology, chemical toxicology, drug pharmacy, substance abuse, and biotoxicology.5,8 The percentage allocation of marks in each question paper was analyzed for each subtopic. Each university's assessment paper was divided into two groups: Set A and Set B. Set A comprised old scheme papers from the 2017 and 2018 MBBS batches, while Set B consisted of CBME scheme papers from the 2019 and 2020 MBBS batches. The collected data were organized into Microsoft Excel spreadsheets. A coding system was employed to identify each university and batch of undergraduate students’ assessment papers during the analysis. The universities included in the study were designated U1 to U8, along with the year of the summative assessment batch (U1 B2017, U1 B2018, etc.). The percentage distribution of marks across structured and unstructured questions, marks allotted to questions with action verbs, and marks to assess the level of the cognitive domain. The weights assigned to different subtopics were recorded and compared. The number of structured and unstructured questions, the presence or absence of action verbs as per the revised Bloom's taxonomy, and the focus of the learning domain in the questions were also evaluated to determine the validity of the assessment. The collected data were examined and compared with the pre- and post-implementation benchmarks of the Competency-Based Medical Education (CBME) curriculum of the National Medical Commission (NMC) from the 2019 batch MBBS program. All researchers completed their Revised Basic Course Workshop from the Nodal Centre of the National Medical Commission, India, as well as the Basic Course in Biomedical Research.
Results
Chart 1 (clustered column chart) and Table 1 compare the total number of structured and unstructured questions and the presence and absence of action verbs among the eight universities’ summative assessment papers before and after implementing the CBME curriculum. Before the CBME was adopted, only two institutions (U6 and U7) incorporated more than half of their assessment questions in a structured format. In contrast, the summative evaluation papers for the U1 2017, U2 2017 & 2018, U3 2018, U4 2018, U5 2017 & 2018, and U8 2018 cohorts were composed entirely of unstructured questions. In the evaluation process, two institutions (U2 from 2018 and U5 from 2017) utilized action verbs, representing 50% of the sample. Conversely, the summative assessment papers from the U2 2017, U3 2017 & 2018, U4 2017, and U8 2018 cohorts exhibited a high proportion of questions (over 80%) devoid of action verbs. Following the implementation of Competency-Based Medical Education (CBME), numerous institutions opted to incorporate more than 50% unstructured questions in their 2019 and 2020 batch examinations, except for University (U) 2's 2020 batch, U3's 2019 and 2020 batches, U4's 2020 batch, U5's 2019 batch, U7's 2019 batch, and U8's 2019 and 2020 batches, which employed more than 50% structured questions. Additionally, a limited number of universities from the 2020 and 2019 batches, specifically U2, U4, U7, and U8, utilized more than 50% of action verbs in their assessment paper.
Comparing the Number of Structured & Unstructured Questions, Presence & Absence of Action Verb
Chart 2 (clustered column) and Table 2 illustrate the marks allocated to the Revised Bloom's taxonomy components before and after implementing CBME. We found that five of the eight universities awarded 100% of the marks for the remember component, whereas one university (U6-2018 batch) allocated only 7.5% of the marks for the create component. Additionally, one university (U1-2017) allocated 7.5% of the marks to the analysis component and U5 (2017 batch) distributed 23.5% of the marks to the application component. From batch 2019 onwards, the graph unequivocally demonstrates that most marks were allocated to the “remember” domain, followed by the “understanding” domain. Only U4 (2019 and 2020 batches) opted to assign 24% of the marks to the “applied” component and 8% to the “create” component, respectively. Furthermore, three universities (2019 and 2020 batches) allocated 16.5%, 15.5%, and 5% of the marks to the “create” component, respectively. It is worth noting that no universities allocated marks for the “evaluate” and “analyse” components.
Comparing the Level of the Cognitive Domain as per Revised Bloom's Taxonomy
Chart 3 (clustered column) and Table 3 illustrate the allocation of marks for various Forensic Medicine & Toxicology topics across eight institutions’ summative assessment papers, both before and after the implementation of CBME. Prior to the CBME, our analysis revealed that most universities assigned over 50% of the marks to Forensic Pathology, with toxicology receiving the second-highest allocation. Exceptions to this pattern were observed in the summative assessment papers of the U4 2018, U5 2018, U6 2018 and 2019, and U8 2018 batches. Notably, the introduction of CBME did not alter the distribution of marks among topics. Even under the new curriculum, Forensic Pathology and Toxicology continue to receive the majority of marks. In contrast, subjects such as laboratory investigations and emerging trends, medical jurisprudence, and forensic psychiatry were allocated fewer marks despite their significance in the daily practice of registered medical practitioners. Chart 4 (area diagram) and Table 4 show the distribution of marks allocated to the toxicology subtopics before and after the implementation of the new assessment scheme. The data indicated an uneven distribution of marks among the subtopics of toxicology. For instance, in the 2019 batch of U8, 100% of the marks were allocated to drug pharmacy and substance abuse, whereas no marks were allocated to the same subtopic in the 2017 batch. The distribution of marks for various subtopics in Forensic Pathology before and after the implementation of the CBME is illustrated in Chart 5 (clustered column) and Table 5. An uneven distribution of marks persisted even after the implementation of CBME. The majority of marks were allocated to mechanical injuries, post-mortem changes, and asphyxia, whereas minimal marks were assigned to topics such as identification, sexual offenses, and thermal death.
Distribution of Marks (Percentage) Based on Topics
Distribution of Marks (Percentage) Among Subtopics of Toxicology
Distribution of Marks (Percentage) Among Subtopics of Forensic Pathology
We also found that the lack of uniformity in the number of questions and marks/scores allotted for each type of question in summative assessment papers is a common issue in universities before and after implementing CBME. (Table 6)
Distribution of the total number of questions, marks/scores allotted for different types of questions among different universities before & after implementation of CBME
The data that support the findings of this study are openly available in Figshare at https://figshare.com/s/fd49686d6b04a252dd02.
Discussion
The Graduate Medical Education Regulations, which govern the undergraduate medical education curriculum GMR 1997, have been framed by the Medical Council of India over the last two decades. The erstwhile MCI and National Medical Commission attempted to identify lacunae in the old curriculum in their reports and documents. 8 In 2019, the MCI adopted a competency-based training model to revamp the medical education in India. The newer GMER 2019 provides a clear framework for undergraduate curriculum. It lays down the guidelines for implementation standards. The curriculum framework is designed such that there is significant flexibility for colleges and universities to plan and implement as per their local context.5,9 However, this is not always true. Previous studies have found that universities and colleges sometimes fall short of expectations.10,11 Earlier research aimed at assessing students’ opinions on incorporating Forensic Medicine and Toxicology into their curricula has caused considerable concern among academics. According to the Medical Council of India guidelines (GMER-1997), a range of essays and short-answer questions are permitted. However, objective questions, including very short-answer questions and multiple-choice questions, were restricted to a maximum of 20%. Summative assessments aim to evaluate students’ knowledge and cognitive ability. The quality of the question papers may have been affected by limiting the questions to recall. The application of knowledge is crucial in subjects such as medicine and forensics. To ensure that the assessment is valid, it is essential to test higher-order cognition. Therefore, questions were categorized into recall, comprehension, and application types to evaluate different levels of cognition. 9 According to Kautilya et al., approximately 10 of 30 papers demonstrated a recall score of over 75%. Only 7 of the 30 papers accounted for more than 50% of the marks allocated to the comprehension section. In one study, the application component was assessed to an extent of 12.5%. The authors examined questions in Forensic Medicine and toxicology from 2016 to 2020. 11 Similar results have been observed in summative assessments of anatomy, physiology, Pharmacology, and Microbiology.12–16 The results of this study were consistent with those of the aforementioned studies. The research conducted by Lehl et al. on summative assessment papers of General Medicine between 2009–2017 across three different universities revealed that the percentage of Structured Essay questions fluctuated between 10.2% and 17.6% among the institutions. Only private universities modified essay questions and short notes, accounting for 10.5% and 4.7%, respectively. The study also revealed that the cognitive domain predominantly focused on knowledge across all universities, with 80–96.8% of instances falling within this category. 17 In this study, there was no consistent or standardized distribution of the number of items or questions across universities before and after implementing the Competency-Based Medical Education (CBME) curriculum. Most universities used unstructured questions both before and after the implementation of CBME. Few universities significantly increased the number of structured questions after the implementation of CBME. Moreover, it was observed that action verbs were lacking in most questions before implementing CBME; however, universities such as 2 and 8 increased the number of action verbs in their assessment papers after implementing CBME. The research findings cast doubt on the construct validity of the written assessments, as even after implementing the CBME curriculum, the assessments failed to measure students’ knowledge of most competencies directed by the National Medical Commission. As an application-based course, Forensic Medicine and Toxicology require the assessment of higher-order cognitive skills. Unfortunately, current summative assessment papers have failed to test the appropriate competency domains adequately.
Content validity refers to the extent to which a test or measurement method accurately captures all the aspects of the subject that it aims to measure. To produce valid results, the content of the test or survey must cover all relevant parts of the assessed construct. The validity of the test may be compromised if important aspects are missing or irrelevant. 2
The allocation of marks for various subtopics affects students’ approach to a topic as it is widely acknowledged that students primarily learn what is being evaluated. Consequently, how learning is assessed significantly influences this approach. To ensure that students adopt a more comprehensive approach to universal course content, it is recommended that standardized question papers be developed. This can be achieved through the process of blueprinting and moderation of theory papers. A study conducted by Aneja et al. aimed to compare summative assessment papers on anatomy before and after implementing CBME in ten different universities in North India. The results showed a less uniform distribution of region-wise marks, with some universities excluding specific topics from their papers. Despite the implementation of CBME, the scenario remained largely unchanged. In the new scheme, greater emphasis was placed on the upper limb and head and neck regions, while five universities did not assess general anatomy in the old scheme and only one failed to do so in the new scheme. However, these two universities did not include histological sections in their old schemes. 18 A study conducted by Patke et al. on biochemistry summative assessment (old scheme) papers revealed a significant issue regarding inconsistent allocation of marks to different subtopics. 19 A previous study carried out by Tiwari et al. examined summative assessment papers in Community Medicine from five different universities and discovered that the majority of the marks were allocated to the epidemiology section, whereas other topics, such as recent advances in Community Medicine, International health, health planning and management, mental health, hospital waste management, and geriatric health, were inadequately assessed. 20 A study conducted by Kautilya et al. on summative assessment papers on Forensic Medicine from 2016 to 2020 across five Indian universities revealed that an average of 60% of marks were allocated to forensic pathology. Certain sections of the curriculum are valued more than equally. Notably, core areas such as Toxicology and Medical Jurisprudence, which are clinically more relevant to undergraduate students in their role as first-contact physicians, have been overlooked and neglected. 11 The current study similarly revealed comparable results, with a more significant part of the marks granted to forensic pathology. A skewed distribution of marks was observed in the toxicology section, with a greater allocation towards chemical and general toxicology. In contrast, bio-environmental toxicology, drugs, pharmacies, and substance abuse received significantly lower scores. Additionally, there was an uneven distribution of marks in the subtopics of toxicology across all universities.
The unchecked growth of a particular segment or component of the curriculum characterizes this pattern of distribution marks. This growth was almost undetectable during the early stages. The growth becomes almost uncontrollable, resulting in an unbalanced curriculum, which has been referred to as the “Carcinoma of curriculum." 21
This suggests that successful reform must address formal curricula, institutional policies, evaluation methods, and resource allocation decisions that shape students’ experiences and learning outcomes. 22 This study also demonstrates the comparison of all the universities’ forensic medicine summative assessment papers separately on the basis of structured and unstructured questions, presence and absence of action verbs, distribution of marks on the basis of Bloom's Revised Taxonomy, and distribution of marks as per subtopics of forensic medicine and toxicology. (Table 7). The findings of this study will assist policymakers in developing more effective strategies to implement summative assessments according to the CBME curriculum throughout the nation, involving all stakeholders in the medical education system.
Comparison of Summative Assessment Paper Before and After CBME
However, significant disparities exist in the implementation of CBME in India compared with other nations. Developed countries have generally adopted CBME more comprehensively, integrating clinical training during the initial educational phases and emphasizing skill acquisition through adaptable learning-oriented programs rather than time-based curricula. By contrast, India is still in the nascent stages of CBME adoption, grappling with obstacles such as the requirement for systemic and curricular modifications, acclimating educators and students, and embracing innovative training and evaluation techniques. 23
Limitation & Future Directives
The scope of this study is restricted to Forensic Medicine & Toxicology and Indian Medical Universities. Examining summative assessment papers from other disciplines would provide a more comprehensive understanding of the state of summative assessments in a country's medical education system. Furthermore, although we selected eight medical universities from various regions of India (north, south, east, and west), a more extensive study incorporating summative assessment papers from all medical universities would yield more impactful results.
Conclusion
Transformation is often characterized as a course of action that alters the trajectory of history or progress and may impact an entity's structure or operation. Although some modifications may be planned, they may not always be successful or acceptable to all faculty members. The present investigation revealed that institutions do not follow the assessment advisory put forth by the National Medical Commission despite having implemented a Competency-Based Medical Education (CBME) curriculum for the 2019 MBBS batch.
Footnotes
Abbreviation for Tables and Charts
Acknowledgments
Dr. Dinesh Kumar, Professor, Department of Community Medicine, Pramukhswamy Medical College & Sri Krishna Hospital, for guiding and supporting us.
Ethical Considerations
Approval from the ethics committee was obtained from the Institutional Ethics Committee.
Ethical clearance number IEC/BU/2024/Ex. 20/63/2024 dated 05/03/2024 from the Institutional Ethics Committee bearing DHR registration number Ec/NEW/INST/2021/592 and OHRP Registration number IRB00014178.
Consent for Publication
Taken from IEC.
Author Contributions
Dr Arijit Datta was very keen and focused on conceptualizing ideas in the study design, interpreting the data, and reviewing the manuscript.
Dr Preeti Tiwari contributed to data analysis, drafting, and reviewing manuscript portions.
Dr Dhara Goswami was instrumental in data collection, data analysis and reviewing the manuscript.
Dr. Darshan Galoria contributed to the study framework, data collection, and manuscript review.
Dr Prashant Verma was instrumental in data collection, manuscript drafting, and reviewing the manuscript & revised it critically for important intellectual content.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
Appendix 1.
Total number of questions and examples of structured and unstructured questions.