Sage Journals: Discover world-class research

Abstract

Introduction

Competency-Based Medical Education [CBME] is a framework for organizing medical education programs that emphasize ongoing assessments to monitor learners’ progress. Implemented by the erstwhile Medical Council of India since the 2019–2020 academic year, this approach has influenced undergraduate summative assessments for Forensic Medicine. To assess the implementation, effectiveness, and alignment of the new curriculum with the goals of medical education, it is necessary to compare summative assessment question papers of undergraduate MBBS students from 2017 to 2020.

Methodology

Researchers evaluated 32 summative examination question papers from eight medical universities across India, categorizing them into structured, non-structured, and action verb-based categories, according to Bloom's revised taxonomy's cognitive domain (remember, understand, apply, analyze, evaluate, and create). They compared the data with the pre- and post-implementation benchmarks of the CBME curriculum of the National Medical Commission (NMC) from the 2019 batch MBBS program.

Results

Most universities used unstructured questions and most marks were allocated to remember the levels of the cognitive domain. It was also found that marks were skewed towards specific topics in all universities, leading to inappropriate sampling and coverage of the competencies. The study also revealed that core areas, such as medical jurisprudence and toxicology, were inappropriately assessed.

Conclusion

The quality of the summative exam questions in Forensic Medicine and Toxicology subjects was poor and not aligned with the CBME, highlighting the need to assess the clarity and utility of blueprints currently employed by universities.

Keywords

content validity summative assessment structure questions action verb cognitive domain

Introduction

Competency-based education is an outcome-oriented methodology for designing, implementing, assessing, and evaluating medical education programs using an organizational competency framework. This approach is more than just a different teaching style; it requires a different assessment perspective. It emphasizes the need for ongoing and longitudinal assessments, allowing educators to diagnose learners’ stages of development and determine whether they require additional or alternative learning opportunities to acquire competency. Evaluating a competency-based curriculum is essential for its effective implementation.¹ Recognizing the significance of assessment validity for educators is therefore essential. This is because it is essential to guarantee that we are drawing logical conclusions about our pupils’ academic progress to aid in guiding their future educational endeavors. Assessment plays a pivotal role in effective teaching-learning processes. To impart knowledge effectively, it is imperative to assess it consistently and accurately.² The evaluation of students’ proficiency in Forensic Medicine has changed significantly within the competency-based medical education paradigm. The main goal of this assessment was to gauge the students’ understanding of medical jurisprudence, toxicology, postmortem techniques, and forensic pathology. However, with the new CBME curriculum, evaluation goes beyond simple factual memory and emphasizes the application of knowledge to real-world clinical situations.³ The redesigned written summative assessment included clinical case vignettes and simulated scenarios that required students to integrate their knowledge, clinical skills, and ethical principles to solve complex forensic medicine cases. This shift aligns with the core principles of CBME, which prioritize the development of competencies essential for the practice of medicine, including critical thinking, decision-making, effective communication, and ethical reasoning.⁴ Considering this, the National Medical Council of India implemented the CBME curriculum in the nation's medical colleges beginning in the 2019–2020 academic year.⁵ This change in medical education has unavoidably influenced the approaches employed in undergraduate summative assessment papers for Forensic Medicine.⁶ The National Medical Commission of our country divided Forensic Medicine & Toxicology into various subtopics, such as general information, forensic pathology, clinical forensic medicine, medical jurisprudence, forensic psychiatry, forensic laboratory investigation in medical-legal practice, emerging technologies in forensic medicine, general toxicology, chemical toxicology, pharmaceutical toxicology, and biotoxicology. These subtopics were subdivided into various specific learning objectives.⁵ As the transition to CBME continues to shape medical education in India, we compared the summative assessment papers of undergraduate MBBS students in the 2017–2020 batches in Forensic Medicine & Toxicology. It is essential to compare and document the changes in assessments to align them with the overarching goals of medical education.

Methodology

This retrospective, cross-sectional, observational study (STROBE) was conducted at our college between February and March 2024 with approval from the Institutional Ethics Committee of our college (IEC/BU/2024/Ex. 20/63/2024). We analyzed 32 summative examination question papers from eight medical universities across various states of India. The question papers were obtained from the public domain, and researchers collected undergraduate examination question papers in Forensic Medicine and Toxicology from universities’ websites and the college records of constituent colleges after conducting a thorough search for the last four years (2017–20 Batch). The universities’ identities were concealed during the analysis, and explicit consent was not required as all data were sourced from public domains. To enhance the sampling methodology, the country was divided into four geographical zones: east, west, north, and south. Two medical universities were selected from each zone with the criterion that their summative assessment papers were publicly accessible on their respective websites. The questions were categorized based on type into structured questions, non-structured questions, and the presence or absence of action verbs. The action verbs in summative assessment focus on measuring students’ overall competence and understanding at the end of the learning period. These verbs were chosen to align with the assessment's purpose of evaluating learning outcomes and making high-stakes decisions about student performance.⁶ Additionally, the questions were categorized based on the cognitive domain of learning according to Bloom's revised taxonomy (remember, understand, apply, analyze, evaluate, and create).⁷ Inter-rater reliability was not assessed, as all authors adhered to the Revised Bloom Taxonomy and Guidelines provided by the National Medical Commission, India.

Undergraduate training in Forensic Medicine & Toxicology encompasses Medical Jurisdiction, Forensic Pathology, and Toxicology as primary areas of study. These are divided into six subcategories, including Legal Procedure, Medical Jurisprudence, Forensic Pathology, Forensic Psychiatry, Laboratory Techniques, Emerging Trends, and Toxicology, as specified by the Medical Council of India in 1997. The allocation of marks for each topic was evaluated for each examination. Forensic Pathology is subdivided into several subtopics, including identification, postmortem changes, mechanical injuries, mechanical asphyxia, thermal deaths, sexual offences, and medicolegal issues related to pregnancy, delivery, and abortion. Toxicology is divided into four subtopics: general toxicology, chemical toxicology, drug pharmacy, substance abuse, and biotoxicology.^5,8 The percentage allocation of marks in each question paper was analyzed for each subtopic. Each university's assessment paper was divided into two groups: Set A and Set B. Set A comprised old scheme papers from the 2017 and 2018 MBBS batches, while Set B consisted of CBME scheme papers from the 2019 and 2020 MBBS batches. The collected data were organized into Microsoft Excel spreadsheets. A coding system was employed to identify each university and batch of undergraduate students’ assessment papers during the analysis. The universities included in the study were designated U1 to U8, along with the year of the summative assessment batch (U1 B2017, U1 B2018, etc.). The percentage distribution of marks across structured and unstructured questions, marks allotted to questions with action verbs, and marks to assess the level of the cognitive domain. The weights assigned to different subtopics were recorded and compared. The number of structured and unstructured questions, the presence or absence of action verbs as per the revised Bloom's taxonomy, and the focus of the learning domain in the questions were also evaluated to determine the validity of the assessment. The collected data were examined and compared with the pre- and post-implementation benchmarks of the Competency-Based Medical Education (CBME) curriculum of the National Medical Commission (NMC) from the 2019 batch MBBS program. All researchers completed their Revised Basic Course Workshop from the Nodal Centre of the National Medical Commission, India, as well as the Basic Course in Biomedical Research.

Results

Chart 1 (clustered column chart) and Table 1 compare the total number of structured and unstructured questions and the presence and absence of action verbs among the eight universities’ summative assessment papers before and after implementing the CBME curriculum. Before the CBME was adopted, only two institutions (U6 and U7) incorporated more than half of their assessment questions in a structured format. In contrast, the summative evaluation papers for the U1 2017, U2 2017 & 2018, U3 2018, U4 2018, U5 2017 & 2018, and U8 2018 cohorts were composed entirely of unstructured questions. In the evaluation process, two institutions (U2 from 2018 and U5 from 2017) utilized action verbs, representing 50% of the sample. Conversely, the summative assessment papers from the U2 2017, U3 2017 & 2018, U4 2017, and U8 2018 cohorts exhibited a high proportion of questions (over 80%) devoid of action verbs. Following the implementation of Competency-Based Medical Education (CBME), numerous institutions opted to incorporate more than 50% unstructured questions in their 2019 and 2020 batch examinations, except for University (U) 2's 2020 batch, U3's 2019 and 2020 batches, U4's 2020 batch, U5's 2019 batch, U7's 2019 batch, and U8's 2019 and 2020 batches, which employed more than 50% structured questions. Additionally, a limited number of universities from the 2020 and 2019 batches, specifically U2, U4, U7, and U8, utilized more than 50% of action verbs in their assessment paper.

Table 1.

Comparing the Number of Structured & Unstructured Questions, Presence & Absence of Action Verb

University	Batch	Structured	Unstructured	Action verb present	Action verb absent
U1	B2017	26.7	73.3	26.7	73.3
U1	B2018	0	100	33.3	66.7
U1	B2019	30.4	69.6	39.1	60.9
U1	B2020	33.3	66.7	44.4	55.6
U2	B2017	0	100	0	100
U2	B2018	0	100	50	50
U2	B2019	33.3	66.7	26.6	73.4
U2	B2020	65.3	34.7	84.6	15.4
U3	B2017	22.22	77.78	5.55	94.45
U3	B2018	0	100	6.66	93.34
U3	B2019	77.4	22.6	19.35	80.65
U3	B2020	64.3	35.7	28.5	71.5
U4	B2017	13.3	86.7	33.3	66.7
U4	B2018	0	100	14.3	85.7
U4	B2019	31	69	51.72	48.28
U4	B2020	51.42	48.58	28.57	71.43
U5	B2017	0	100	50	50
U5	B2018	0	100	25	75
U5	B2019	53.33	46.67	33.33	66.67
U5	B2020	34.48	65.52	41.37	58.63
U6	B2017	47.6	52.4	47.6	52.4
U6	B2018	50	50	40	60
U6	B2019	39.5	60.5	34.88	65.12
U6	B2020	15.15	84.85	39.39	60.61
U7	B2017	0	100	25	75
U7	B2018	0	100	25	75
U7	B2019	59.46	40.54	32.43	67.57
U7	B2020	38.46	61.54	50	50
U8	B2017	0	100	0	100
U8	B2018	7.14	92.86	21.42	78.58
U8	B2019	68.19	31.81	68.19	31.81
U8	B2020	80	20	40	60

Chart 2 (clustered column) and Table 2 illustrate the marks allocated to the Revised Bloom's taxonomy components before and after implementing CBME. We found that five of the eight universities awarded 100% of the marks for the remember component, whereas one university (U6-2018 batch) allocated only 7.5% of the marks for the create component. Additionally, one university (U1-2017) allocated 7.5% of the marks to the analysis component and U5 (2017 batch) distributed 23.5% of the marks to the application component. From batch 2019 onwards, the graph unequivocally demonstrates that most marks were allocated to the “remember” domain, followed by the “understanding” domain. Only U4 (2019 and 2020 batches) opted to assign 24% of the marks to the “applied” component and 8% to the “create” component, respectively. Furthermore, three universities (2019 and 2020 batches) allocated 16.5%, 15.5%, and 5% of the marks to the “create” component, respectively. It is worth noting that no universities allocated marks for the “evaluate” and “analyse” components.

Table 2.

Comparing the Level of the Cognitive Domain as per Revised Bloom's Taxonomy

Bloom's Revised Taxonomy
University	Batch	Action verb absent	Remember	Understand	Apply	Analyze	Evaluate	Create
U1	B2017	73.3	80	12.5	0	7.5	0	0
U1	B2018	66.7	90	10	0	0	0	0
U1	B2019	60.9	92.5	7.5	0	0	0	0
U1	B2020	55.6	75	25	0	0	0	0
U2	B2017	100	100	0	0	0	0	0
U2	B2018	50	69.5	30.5	0	0	0	0
U2	B2019	73.4	100	0	0	0	0	0
U2	B2020	15.4	81.25	18.75	0	0	0	0
U3	B2017	94.45	100	0	0	0	0	0
U3	B2018	93.34	100	0	0	0	0	0
U3	B2019	80.65	92	8	0	0	0	0
U3	B2020	71.5	89	11	0	0	0	0
U4	B2017	66.7	95	5	0	0	0	0
U4	B2018	85.7	100	0	0	0	0	0
U4	B2019	48.28	70	6	24	0	0	0
U4	B2020	71.43	85	7	0	0	0	8
U5	B2017	50	47.05	23.52	23.53	0	0	0
U5	B2018	75	100	0	0	0	0	0
U5	B2019	66.67	82	0	0	0	0	18
U5	B2020	58.63	79	6	0	0	0	15
U6	B2017	52.4	82.5	17.5	0	0	0	0
U6	B2018	60	87.5	5	0	0	0	7.5
U6	B2019	65.12	96	4	0	0	0	0
U6	B2020	60.61	72	28	0	0	0	0
U7	B2017	75	88.23	11.77	0	0	0	0
U7	B2018	75	73.52	26.48	0	0	0	0
U7	B2019	67.57	66	18	0	0	0	16
U7	B2020	50	79	6	0	0	0	15
U8	B2017	100	100	0	0	0	0	0
U8	B2018	78.58	87.5	12.5	0	0	0	0
U8	B2019	31.81	62.5	33.75	0	0	0	3.75
U8	B2020	60	67.5	25	0	0	0	7.5

Chart 3 (clustered column) and Table 3 illustrate the allocation of marks for various Forensic Medicine & Toxicology topics across eight institutions’ summative assessment papers, both before and after the implementation of CBME. Prior to the CBME, our analysis revealed that most universities assigned over 50% of the marks to Forensic Pathology, with toxicology receiving the second-highest allocation. Exceptions to this pattern were observed in the summative assessment papers of the U4 2018, U5 2018, U6 2018 and 2019, and U8 2018 batches. Notably, the introduction of CBME did not alter the distribution of marks among topics. Even under the new curriculum, Forensic Pathology and Toxicology continue to receive the majority of marks. In contrast, subjects such as laboratory investigations and emerging trends, medical jurisprudence, and forensic psychiatry were allocated fewer marks despite their significance in the daily practice of registered medical practitioners. Chart 4 (area diagram) and Table 4 show the distribution of marks allocated to the toxicology subtopics before and after the implementation of the new assessment scheme. The data indicated an uneven distribution of marks among the subtopics of toxicology. For instance, in the 2019 batch of U8, 100% of the marks were allocated to drug pharmacy and substance abuse, whereas no marks were allocated to the same subtopic in the 2017 batch. The distribution of marks for various subtopics in Forensic Pathology before and after the implementation of the CBME is illustrated in Chart 5 (clustered column) and Table 5. An uneven distribution of marks persisted even after the implementation of CBME. The majority of marks were allocated to mechanical injuries, post-mortem changes, and asphyxia, whereas minimal marks were assigned to topics such as identification, sexual offenses, and thermal death.

Table 3.

Distribution of Marks (Percentage) Based on Topics

% mark distribution based on Topic
University	Batch	General Info & Legal Procedure	Forensic Pathology	Medical Jurisprudence	Psychiatry	Lab & Emerging trends	Toxicology
U1	B2017	5	57.5	5	5	12.5	15
U1	B2018	5	55	5	5	0	30
U1	B2019	8.75	53.75	12.5	6.25	0	18.75
U1	B2020	6.25	37.5	12.5	6.25	6.25	31.25
U2	B2017	5.55	50	0	5.55	0	38.9
U2	B2018	0	50	0	11.11	0	38.89
U2	B2019	15	35	7.5	0	0	42.5
U2	B2020	3.75	46.25	20	0	0	30
U3	B2017	5	60	12.5	5	0	17.5
U3	B2018	0	90	0	0	0	10
U3	B2019	1	53	5	12	0	29
U3	B2020	9	69	5	1	0	16
U4	B2017	5	65	0	5	0	25
U4	B2018	0	40	12.5	10	0	37.5
U4	B2019	4	40	14	2	6	34
U4	B2020	2	50	12	2	0	34
U5	B2017	11.76	53	11.76	0	0	23.48
U5	B2018	11.76	41.17	11.76	11.76	0	23.55
U5	B2019	6	46	18	6	0	24
U5	B2020	14	38	4	8	0	36
U6	B2017	0	45	10	5	0	40
U6	B2018	5	37.5	17.5	0	0	40
U6	B2019	9	29	23	7	3	29
U6	B2020	0	42	14	7	10	27
U7	B2017	0	61.76	14.72	11.76	0	11.76
U7	B2018	0	64.7	11.77	0	0	23.53
U7	B2019	8	44	14	6	6	22
U7	B2020	12	40	4	8	2	34
U8	B2017	0	50	12.5	12.5	0	25
U8	B2018	0	37.5	12.5	25	0	25
U8	B2019	25	43.75	12.5	6.25	0	12.5
U8	B2020	6.25	62.5	6.25	6.25	6.25	12.5

Table 4.

Distribution of Marks (Percentage) Among Subtopics of Toxicology

% mark distribution in Toxicology
University	Batch	General Toxicology	Chemical toxicology	Drug, Pharmacy & substance abuse	Bio & Environmental toxicology
U1	B2017	0	33.3	0	66.7
U1	B2018	83.3	0	0	16.7
U1	B2019	40	60	0	0
U1	B2020	0	0	40	60
U2	B2017	42.8	42.8	0	14.4
U2	B2018	14.3	50	0	35.7
U2	B2019	29.4	35.3	0	35.3
U2	B2020	12.5	41.6	20.9	25
U3	B2017	0	0	0	100
U3	B2018	0	50	0	50
U3	B2019	0	82.75	17.25	0
U3	B2020	0	100	0	0
U4	B2017	60	0	0	40
U4	B2018	53.3	0	33.3	13.4
U4	B2019	0	47.06	17.65	35.39
U4	B2020	17.6	53	5.9	23.5
U5	B2017	0	0	100	0
U5	B2018	50	50	0	0
U5	B2019	50	50	0	0
U5	B2020	0	28.22	27.78	50
U6	B2017	12.5	43.75	12.5	31.25
U6	B2018	0	25	12.5	62.5
U6	B2019	6.8	31.03	38.03	24.14
U6	B2020	0	62.96	14.82	22.22
U7	B2017	0	100	0	0
U7	B2018	0	0	100	0
U7	B2019	0	27.27	0	72.73
U7	B2020	0	23.54	29.41	47.05
U8	B2017	0	100	0	0
U8	B2018	20	0	40	40
U8	B2019	0	100	0	0
U8	B2020	50	0	0	50

Table 5.

Distribution of Marks (Percentage) Among Subtopics of Forensic Pathology

% mark distribution in Forensic Pathology
University	Batch	Identification	PM Changes & Autopsy	Mechanical Injuries	Asphyxia	Thermal Death	Sexual Offences	Pregnancy, abortion and Infant death
U1	B2017	17.4	8.7	0	0	43.5	30.4	0
U1	B2018	9.09	9.09	0	45.44	9.09	9.09	18.2
U1	B2019	0	11.64	11.64	11.62	18.6	11.62	34.88
U1	B2020	0	16.67	0	33.33	16.67	16.67	16.67
U2	B2017	11.11	11.11	33.3	11.11	11.11	11.11	11.11
U2	B2018	11.11	22.22	11.11	11.12	16.66	16.66	11.12
U2	B2019	0	21.5	0	0	0	35.7	42.8
U2	B2020	16.2	8.8	16	16	27	16	0
U3	B2017	0	37.5	12.5	12.5	16.7	20.8	0
U3	B2018	22.23	0	44.45	8.33	5.66	13.88	5.65
U3	B2019	26.4	11.3	16.9	0	28.3	15.2	1.9
U3	B2020	0	17.4	33.4	40.5	1.45	0	7.25
U4	B2017	7.6	11.5	30.7	19.2	11.5	0	11.5
U4	B2018	18.8	37.5	25	18.75	0	0	0
U4	B2019	0	30	20	30	0	20	0
U4	B2020	0	36	8	12	12	16	16
U5	B2017	27.78	27.78	22.22	0	0	22.22	0
U5	B2018	35.7	0	35.7	0	0	28.6	0
U5	B2019	4.34	21.73	39.1	4.3	17.39	0	13.04
U5	B2020	5.26	42.11	21.05	0	15.79	0	15.79
U6	B2017	11.11	11.11	11.11	0	38.89	16.67	11.11
U6	B2018	13.4	13.4	20	13.4	13.4	13	13.4
U6	B2019	6.9	13.7	20.69	0	34.49	0	24.13
U6	B2020	9.52	33.34	26.2	16.66	4.76	0	9.52
U7	B2017	23.81	38.1	19.04	0	0	0	19.04
U7	B2018	18.19	0	22.72	22.71	0	18.19	18.19
U7	B2019	9.1	4.54	45.45	13.64	0	27.27	0
U7	B2020	5	45	20	0	15	15	0
U8	B2017	0	50	25	25	0	0	0
U8	B2018	26.67	46.67	13.33	13.33	0	0	0
U8	B2019	0	14.28	42.86	0	0	14.28	28.58
U8	B2020	0	10	30	20	0	20	20

We also found that the lack of uniformity in the number of questions and marks/scores allotted for each type of question in summative assessment papers is a common issue in universities before and after implementing CBME. (Table 6)

Table 6.

Distribution of the total number of questions, marks/scores allotted for different types of questions among different universities before & after implementation of CBME

% marks distribution based on question types
University	Batch	Total Number of Items/Questions	Total Marks	% Marks LAQ	% Marks SAQ	% Marks VSAQ/ MCQ
U1	B2017	15	100	50	50	0
U1	B2018	12	100	50	50	0
U1	B2019	23	100	25	68.75	6.25
U1	B2020	18	100	25	75	0
U2	B2017	16	100	0	100	0
U2	B2018	16	100	0	100	0
U2	B2019	15	100	20	80	0
U2	B2020	26	100	25	75	0
U3	B2017	18	100	17.5	82.5	0
U3	B2018	15	100	17.5	82.5	0
U3	B2019	31	100	30	60	10
U3	B2020	28	100	30	60	10
U4	B2017	15	100	30	70	0
U4	B2018	14	100	30	70	0
U4	B2019	29	100	20	80	0
U4	B2020	35	100	20	80	0
U5	B2017	8	100	0	100	0
U5	B2018	8	100	0	100	0
U5	B2019	28	100	20	80	0
U5	B2020	29	100	20	80	0
U6	B2017	21	100	35	65	0
U6	B2018	20	100	35	65	0
U6	B2019	43	100	20	80	0
U6	B2020	33	100	20	80	0
U7	B2017	8	100	0	100	0
U7	B2018	8	100	0	100	0
U7	B2019	37	100	20	80	0
U7	B2020	26	100	20	80	0
U8	B2017	7	100	25	75	0
U8	B2018	14	100	0	100	0
U8	B2019	22	100	37.5	62.5	0
U8	B2020	25	100	37.5	62.5	0

The data that support the findings of this study are openly available in Figshare at https://figshare.com/s/fd49686d6b04a252dd02.

Discussion

The Graduate Medical Education Regulations, which govern the undergraduate medical education curriculum GMR 1997, have been framed by the Medical Council of India over the last two decades. The erstwhile MCI and National Medical Commission attempted to identify lacunae in the old curriculum in their reports and documents.⁸ In 2019, the MCI adopted a competency-based training model to revamp the medical education in India. The newer GMER 2019 provides a clear framework for undergraduate curriculum. It lays down the guidelines for implementation standards. The curriculum framework is designed such that there is significant flexibility for colleges and universities to plan and implement as per their local context.^5,9 However, this is not always true. Previous studies have found that universities and colleges sometimes fall short of expectations.^10,11 Earlier research aimed at assessing students’ opinions on incorporating Forensic Medicine and Toxicology into their curricula has caused considerable concern among academics. According to the Medical Council of India guidelines (GMER-1997), a range of essays and short-answer questions are permitted. However, objective questions, including very short-answer questions and multiple-choice questions, were restricted to a maximum of 20%. Summative assessments aim to evaluate students’ knowledge and cognitive ability. The quality of the question papers may have been affected by limiting the questions to recall. The application of knowledge is crucial in subjects such as medicine and forensics. To ensure that the assessment is valid, it is essential to test higher-order cognition. Therefore, questions were categorized into recall, comprehension, and application types to evaluate different levels of cognition.⁹ According to Kautilya et al., approximately 10 of 30 papers demonstrated a recall score of over 75%. Only 7 of the 30 papers accounted for more than 50% of the marks allocated to the comprehension section. In one study, the application component was assessed to an extent of 12.5%. The authors examined questions in Forensic Medicine and toxicology from 2016 to 2020.¹¹ Similar results have been observed in summative assessments of anatomy, physiology, Pharmacology, and Microbiology.^12–16 The results of this study were consistent with those of the aforementioned studies. The research conducted by Lehl et al. on summative assessment papers of General Medicine between 2009–2017 across three different universities revealed that the percentage of Structured Essay questions fluctuated between 10.2% and 17.6% among the institutions. Only private universities modified essay questions and short notes, accounting for 10.5% and 4.7%, respectively. The study also revealed that the cognitive domain predominantly focused on knowledge across all universities, with 80–96.8% of instances falling within this category.¹⁷ In this study, there was no consistent or standardized distribution of the number of items or questions across universities before and after implementing the Competency-Based Medical Education (CBME) curriculum. Most universities used unstructured questions both before and after the implementation of CBME. Few universities significantly increased the number of structured questions after the implementation of CBME. Moreover, it was observed that action verbs were lacking in most questions before implementing CBME; however, universities such as 2 and 8 increased the number of action verbs in their assessment papers after implementing CBME. The research findings cast doubt on the construct validity of the written assessments, as even after implementing the CBME curriculum, the assessments failed to measure students’ knowledge of most competencies directed by the National Medical Commission. As an application-based course, Forensic Medicine and Toxicology require the assessment of higher-order cognitive skills. Unfortunately, current summative assessment papers have failed to test the appropriate competency domains adequately.

Content validity refers to the extent to which a test or measurement method accurately captures all the aspects of the subject that it aims to measure. To produce valid results, the content of the test or survey must cover all relevant parts of the assessed construct. The validity of the test may be compromised if important aspects are missing or irrelevant.²

The allocation of marks for various subtopics affects students’ approach to a topic as it is widely acknowledged that students primarily learn what is being evaluated. Consequently, how learning is assessed significantly influences this approach. To ensure that students adopt a more comprehensive approach to universal course content, it is recommended that standardized question papers be developed. This can be achieved through the process of blueprinting and moderation of theory papers. A study conducted by Aneja et al. aimed to compare summative assessment papers on anatomy before and after implementing CBME in ten different universities in North India. The results showed a less uniform distribution of region-wise marks, with some universities excluding specific topics from their papers. Despite the implementation of CBME, the scenario remained largely unchanged. In the new scheme, greater emphasis was placed on the upper limb and head and neck regions, while five universities did not assess general anatomy in the old scheme and only one failed to do so in the new scheme. However, these two universities did not include histological sections in their old schemes.¹⁸ A study conducted by Patke et al. on biochemistry summative assessment (old scheme) papers revealed a significant issue regarding inconsistent allocation of marks to different subtopics.¹⁹ A previous study carried out by Tiwari et al. examined summative assessment papers in Community Medicine from five different universities and discovered that the majority of the marks were allocated to the epidemiology section, whereas other topics, such as recent advances in Community Medicine, International health, health planning and management, mental health, hospital waste management, and geriatric health, were inadequately assessed.²⁰ A study conducted by Kautilya et al. on summative assessment papers on Forensic Medicine from 2016 to 2020 across five Indian universities revealed that an average of 60% of marks were allocated to forensic pathology. Certain sections of the curriculum are valued more than equally. Notably, core areas such as Toxicology and Medical Jurisprudence, which are clinically more relevant to undergraduate students in their role as first-contact physicians, have been overlooked and neglected.¹¹ The current study similarly revealed comparable results, with a more significant part of the marks granted to forensic pathology. A skewed distribution of marks was observed in the toxicology section, with a greater allocation towards chemical and general toxicology. In contrast, bio-environmental toxicology, drugs, pharmacies, and substance abuse received significantly lower scores. Additionally, there was an uneven distribution of marks in the subtopics of toxicology across all universities.

The unchecked growth of a particular segment or component of the curriculum characterizes this pattern of distribution marks. This growth was almost undetectable during the early stages. The growth becomes almost uncontrollable, resulting in an unbalanced curriculum, which has been referred to as the “Carcinoma of curriculum."²¹

This suggests that successful reform must address formal curricula, institutional policies, evaluation methods, and resource allocation decisions that shape students’ experiences and learning outcomes.²² This study also demonstrates the comparison of all the universities’ forensic medicine summative assessment papers separately on the basis of structured and unstructured questions, presence and absence of action verbs, distribution of marks on the basis of Bloom's Revised Taxonomy, and distribution of marks as per subtopics of forensic medicine and toxicology. (Table 7). The findings of this study will assist policymakers in developing more effective strategies to implement summative assessments according to the CBME curriculum throughout the nation, involving all stakeholders in the medical education system.

Table 7.

Comparison of Summative Assessment Paper Before and After CBME

	U1			U2			U3			U4			U5			U6			U7			U8
	1	2	3	1	2	3	1	2	3	1	2	3	1	2	3	1	2	3	1	2	3	1	2	3
Structured Questions	13.35	31.85	18.5	0	49.3	−49.3	11.11	70.85	−59.74	6.65	41.21	−34.56	0	43.9	−43.9	48.8	27.32	21.48	0	48.96	−48.96	3.57	74.09	−70.52
Unstructured Questions	86.65	68.15	18.5	100	50.7	49.3	88.89	29.15	59.74	93.35	58.79	34.56	100	56.1	43.9	51.2	72.68	−21.48	100	51.04	48.96	96.43	25.9	70.53
Presence of Action Verb	30	41.75	11.75	25	55.6	−30.6	6.105	23.83	−17.72	23.8	40.14	−16.34	37.5	37.35	0.15	43.8	37.13	6.67	25	41.21	−16.21	10.71	54.09	−43.38
Absence of Action Verb	73.15	58.25	14.9	75	44.4	30.6	93.89	76.17	17.72	76.2	59.8	16.4	62.5	62.65	−0.15	56.2	62.86	−6.66	75	58.78	16.22	89.29	45.9	43.39
Remember	85	83.75	1.25	84.75	90.62	−5.87	100	90.5	9.5	97.5	77.5	20	73.52	80.5	−6.98	85	84	1	80.87	72.5	8.37	93.75	65	28.75
Understand	11.25	16.25	5	15.25	9.37	5.88	0	9.5	−9.5	2.5	6.5	−4	11.76	3	8.76	11.25	16	−4.75	19.12	12	7.12	6.25	29.37	−23.12
Apply	0	0	0	0	0	0	0	0	0	0	12	−12	11.76	0	11.76	0	0	0	0	0	0	0	0	0
Analyse	3.25	0	3.25	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Evaluate	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Create	0	0	0	0	0	0	0	0	0	0	4	−4	0	16.5	−16.5	3.75	0	3.75	0	15.5	−15.5	0	5.62	−5.62
Marks for General Info and Legal Procedure	5	7.5	−2.5	2.77	9.37	−6.6	2.5	5	−2.5	2.5	3	−0.5	11.76	10	1.76	2.5	4.5	−2	0	10	−10	0	15.62	−15.62
Marks for Forensic pathology	56.25	45.66	10.59	50	40.62	9.38	75	61	14	52.5	45	7.5	47.08	42	5.08	41.25	35	6.25	63.23	42	21.23	43.75	53.12	−9.37
Marks for Medical Jurisprudence	5	12.5	−7.5	0	13.75	−13.75	6.25	5	1.25	6.25	13	−6.75	11.76	11	0.76	13.75	18.5	−4.75	13.24	9	4.24	12.5	9.37	3.13
Marks for Psychiatry	5	6.25	−1.25	8.33	0	8.33	2.5	6.5	−4	7.5	2	5.5	5.88	7	−1.12	2.5	7	−4.5	5.88	7	−1.12	18.75	6.25	12.5
Marks for Lab and Emerging Trends	6.25	3.125	3.125	0	0	0	0	0	0	0	3	−3	0	0	0	0	6.5	−6.5	0	4	−4	0	3.12	−3.12
Marks for Toxicology	22.5	25	2.5	38.89	36.25	2.64	13.75	22.5	−8.75	31.25	34	−2.75	23.51	30	−6.49	40	28	12	17.64	28	−10.36	25	12.5	12.5

Total number of questions analyzed: 663
Structured questions	Unstructured questions
Classify thermal injuries. Describe the degree of burn, cause of death & postmortem findings in case of death due to burns. (2 + 2 + 1 + 2 Difference between privacy and confidentiality. What are the justified instances in patient care for breach of confidentiality? (2 + 3)	PCPNDT Act & its amendment.⁶ Define drug dependence. Write signs, symptoms, treatments & medicolegal importance of Opium poisoning.¹⁰
All summative assessment papers have 100 marks; however, the allocation of marks for long-answer, short-answer, and very short-answer questions varies among universities. Also, the number of total questions varies in different universities. However, sometimes, questions are repeated among different university assessment papers.

However, significant disparities exist in the implementation of CBME in India compared with other nations. Developed countries have generally adopted CBME more comprehensively, integrating clinical training during the initial educational phases and emphasizing skill acquisition through adaptable learning-oriented programs rather than time-based curricula. By contrast, India is still in the nascent stages of CBME adoption, grappling with obstacles such as the requirement for systemic and curricular modifications, acclimating educators and students, and embracing innovative training and evaluation techniques.²³

Limitation & Future Directives

The scope of this study is restricted to Forensic Medicine & Toxicology and Indian Medical Universities. Examining summative assessment papers from other disciplines would provide a more comprehensive understanding of the state of summative assessments in a country's medical education system. Furthermore, although we selected eight medical universities from various regions of India (north, south, east, and west), a more extensive study incorporating summative assessment papers from all medical universities would yield more impactful results.

Conclusion

Transformation is often characterized as a course of action that alters the trajectory of history or progress and may impact an entity's structure or operation. Although some modifications may be planned, they may not always be successful or acceptable to all faculty members. The present investigation revealed that institutions do not follow the assessment advisory put forth by the National Medical Commission despite having implemented a Competency-Based Medical Education (CBME) curriculum for the 2019 MBBS batch.

Footnotes

Abbreviation for Tables and Charts

Acknowledgments

Dr. Dinesh Kumar, Professor, Department of Community Medicine, Pramukhswamy Medical College & Sri Krishna Hospital, for guiding and supporting us.

ORCID iDs

Arijit Datta

Preeti Tiwari

Ethical Considerations

Approval from the ethics committee was obtained from the Institutional Ethics Committee.

Ethical clearance number IEC/BU/2024/Ex. 20/63/2024 dated 05/03/2024 from the Institutional Ethics Committee bearing DHR registration number Ec/NEW/INST/2021/592 and OHRP Registration number IRB00014178.

Consent for Publication

Taken from IEC.

Author Contributions

Dr Arijit Datta was very keen and focused on conceptualizing ideas in the study design, interpreting the data, and reviewing the manuscript.

Dr Preeti Tiwari contributed to data analysis, drafting, and reviewing manuscript portions.

Dr Dhara Goswami was instrumental in data collection, data analysis and reviewing the manuscript.

Dr. Darshan Galoria contributed to the study framework, data collection, and manuscript review.

Dr Prashant Verma was instrumental in data collection, manuscript drafting, and reviewing the manuscript & revised it critically for important intellectual content.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

Data are provided in the supplementary information file. Datasets generated and/or analyzed during the current study are available from the following DOI.

Appendix 1.

Total number of questions and examples of structured and unstructured questions.

References

Frank

Snell

Ten Cate

, et al. Competency-based medical education: Theory to practice. Med Teach. 2010;32(7):638–645. doi:https://doi.org/10.3109/0142159X.2010.501190

Obilor

Miwari

. Content validity in educational assessment. International Journal of Innovative Education Research. 2022;10(2):57–69.

Shah

Desai

Jorwekar

Badyal

Singh

. Competency-based medical education: An overview and application in pharmacology. Indian J Pharmacol. 2016;48(Suppl 1):S5–S9. https://doi.org/10.4103/0253-7613.193312

Rege

. Assessment in competency-based medical education: A paradigm shift. J Postgrad Med. 2020;66(4):184–186. https://doi.org/10.4103/jpgm.JPGM_1182_20

Medical Council of India. 2019. Competency-Based Undergraduate Curriculum. Retrieved from https://www.nmc.org.in/wp-content/uploads/2020/01/UG-Curriculum-Vol-I.pdf.

Oermann

Kardong-Edgren

Rizzolo

. Summative Simulated-Based Assessment in Nursing Programs. Journal of Nursing Education. 2016;55:323–328. doi:https://doi.org/10.3928/01484834-20160516-04

Utica University. Revised Bloom’s Taxonomy. Utica University; n.d. Accessed 2014 July 5. Available from: https://www.utica.edu/academic/Assessment/new/Bloom%20tx%20revised%20combined.pdf.

Medical Council of India. 1997. Regulations on Graduate Medical Education, 1997. Retrieved from https://www.nmc.org.in/wpcontent/uploads/2017/10/GME_REGULATIONS-1.pdf.

Medical Council of India. 2019. Assessment Module for Undergraduate Medical Education Training Program. Retrieved from https://www.nmc.org.in/wp-content/uploads/2020/01/Module_Competence_based_02.09.2019.pdf.

10.

Sharma

Harish

Chavali

. Teaching, training and practice of forensic medicine in India-An overview. Journal of Indian Academy of Forensic Medicine. 2005;27(4):247–251.

11.

Dayanidhi

Datta

Hegde

Tiwari

. Evaluating the content validity of the undergraduate summative exam question papers of Forensic Medicine & Toxicology from 6 medical universities in India. Asia Pacific Scholar. 2023;8(2):57. doi:https://doi.org/10.29060/TAPS.2023-8-2/OA2778

12.

Aggarwal

Agarwal

. Analysis of undergraduate pharmacology annual written examination papers at Pt. BD Sharma University of Health Sciences Rohtak. National Journal of Physiology. Pharmacy, and Pharmacology. 2017;7(5):509.

13.

Bhattacharya

Wagh

Malgaonkar

Kartikeyan

. Analysis of content of theory question papers in preliminary examinations and marks obtained by first-year MBBS students in physiology. International Journal of Physiology. Nutrition and Physical Education. 2017;2(2):856–868.

14.

Choudhary

Chawla

Choudhary

. Content validity of first MBBS Physiology examinations and its comparison with teaching hours devoted for different sub-divisions of physiology. Journal of Physiology and Pathophysiology. 2012;3(1):8–11.

15.

Chowdhury

Saha

Talukder

, et al. Evaluation of pharmacology written question papers of MBBS professional examinations. Bangladesh Journal of Medical Education. 2017;8(2):12–17. https://doi.org/10.3329/bjme.v8i2.33331

16.

Mehta

Kikani

. Descriptive analysis of II-MBBS university question papers of microbiology subject. Journal of Education Technology in Health Sciences. 2019;6(2):44–47. https://doi.org/10.18231/j.jeths.2019.011

17.

Lehl

Gupta

Hande

. The long and short of summative assessment in competency-based medical education: Time to raise the bar? Journal of Association of Physicians of India. 2022;70(2):11–12.

18.

Aneja

Saha

Garg

Bang

Bansal

Aulakh

. Summative assessment in competency-based medical education - A critical analysis. Bulletin of Environmental Pharmacology and Life Sciences. 2022;4(Special Issue):159–167.

19.

Patke

Dahake

Kuyare

. Evaluation of quality of MBBS Biochemistry theory question papers of medical institutions in Maharashtra. International Journal of Research in Medical Sciences. 2017;5(10):4336–4343. doi:https://doi.org/10.18203/2320-6012.ijrms20174114

20.

Tiwari

Priya

Datta

. Level of cognitive domains and weightage of various topics in the undergraduate summative examination question paper of community medicine: A cross-sectional study. J Clin Diagn Res. 2024;18(2).

21.

Abrahamson

. Diseases of the curriculum. J Med Educ. 1978;53(12):951–957. https://doi.org/10.1097/00001888-197812000-00001

22.

Hawick

Cleland

Kitto

. Getting off the carousel: Exploring the wicked problem of curriculum reform. Perspectives on Medical Education. 2022;6:337–343. doi:https://doi.org/10.1007/S40037-017-0371-Z

23.

Gupta

Menon

. Psychiatry training for medical students: A global perspective and implications for India’s competency-based medical education curriculum. Indian J Psychiatry. 2022;64:240–251. doi:https://doi.org/10.4103/indianjpsychiatry.indianjpsychiatry_187_22

A Comparative Study on Assessing the Summative Assessments Before and After Implementing Competency-Based Medical Education in India

Abstract

Introduction

Methodology

Results

Conclusion

Keywords

Introduction

Methodology

Results

Discussion

Limitation & Future Directives

Conclusion

Footnotes

Abbreviation for Tables and Charts

Acknowledgments

ORCID iDs

Ethical Considerations

Consent for Publication

Author Contributions

Funding

Declaration of Conflicting Interests

Data Availability Statement

Appendix 1.

References