Abstract
Introduction
Residency program directors are still exploring means to teach and evaluate ACGME core competencies. Finding simple means to reconcile ACGME core competencies with daily encounters offers a unique opportunity. Reflective writing through narratives may be the answer to simple and affordable means to achieve such goals.
Methods
To investigate the educational value of reflective writing, we conducted a prospective, randomized, cross-over pilot study among family practice residency program residents. The intervention group was introduced to the educational intervention. During the intervention, each narrative was analyzed by the authors, sentence by sentence, helping the learners spot ACGME core competencies. A week later, both groups were given five preselected narratives (test narratives) to analyze and identify what ACGME core competencies were reflected. A week later, the control group was subjected to the same intervention to comply with the cross-over design of the study. Data were then collected and a statistical analysis was completed.
Results
Nine learners were randomized into the control group and ten were randomized into the intervention group. Each learner analyzed ten sentences within the five test-narratives. The mean score for each learner across the ten sentences was calculated. The grand mean score for each group was calculated. The grand mean score for the control group was 58.75 (SD 13.4). The grand mean score for the intervention group was 69.90 (SD 15.8). Our one-tailed t test analysis showed no significant difference between the two groups (t = 1.647, df = 17, p = 0.057).
Conclusion
Our pilot study failed to show any statically significant improvement in the learners ability to reconcile the ACGME core competencies with their daily encounters using reflective writing. We identified several possibilities for the negative outcome. Sample size seems to be a major contributor. Further prospective randomized studies using larger sample sizes would be worthwhile to answer our research question.
Introduction
Since 2001, the Accreditation Council for Graduate Medical Education (ACGME) mandated all residency programs in the United States to teach residents and evaluate their performance in six domains of medical education: medical knowledge, patient care, system-based practice, practice-based learning and improvement, interpersonal and communication skills, and professionalism.1,2
More than a decade later, residency programs are still exploring means to teach and evaluate competencies in the required domains. Some domains such as medical knowledge and patient care are relatively easy to fulfill. Addressing more advanced domains remains a challenge.3–6 Large residency programs can afford advanced and high-fidelity pedagogies to fulfill their requirements.7–10 Small programs may not be so lucky. Finding simple means to introduce ACGME core competencies into daily encounters offers a unique opportunity to teach the learners, and may be, evaluate their performance in one or more of ACGME core competencies, particularly when such means are affordable, effective, and suitable for implementation in more than one setting.
Reflective writing through narratives may be the answer to simple and affordable means of introducing ACGME core competencies into daily encounters, thus serving as a framework for teaching and may be assessment as well.11–14 Recent studies showed that current tools of assessment cannot evaluate the competencies independent of one another. 15 Using reflective writing may provide the opportunity to evaluate more than one core competency simultaneously.
Over the past three years, the first author encouraged his learners to reflect on their experiences, positive or negative, at any of the locations they have rotated through without disclosing the type of rotation, the location, or the people involved. The reflections were analyzed by the first author. The embedded overt and/or covert core competencies were identified and inserted within the text of each narrative. At a later date, the analyzed narrative was discussed with the learner who wrote it, one-on-one, focusing on the identified ACGME core competencies and the lessons learned from the narrative. Preliminary data show that this educational intervention of analyzing and discussing the narratives with the learners left a perceived positive impact on their gained knowledge and skills. 16 Our learners thought that when the ACGME core competencies were identified in their narratives, they enhanced their ability to become competent in those areas, and recommended the use of such intervention to teach ACGME core competencies outside structured classes or workshops.16,17 We believe that such intervention may further aid faculty in assessing whether such assumption can be validated.
To further investigate the educational value of such intervention, we conducted a prospective, randomized, cross-over pilot study among residents at Kaiser Fontana family practice residency program. We could not include general surgery residents, the only other residency besides family practice, due to the fact that they were exposed to this methodology and their participation would have led to an inherent bias. The authors secured an IRB approval. No consent was required. The study was locally funded.
Methods
We are not aware of any published literature that specifically used this model to introduce the ACGME core competencies in the learning environment. Therefore, we could not conduct power calculation and we did not know the number of learners needed in each randomized arm of the study. We chose to invite all residents in the family practice residency program since they were naïve to this model. Our convenience sample had a total of 27 residents throughout the training period of three years.
Once the randomization was completed, the control group of residents was excluded. The intervention group was introduced to the educational intervention. Sixty to 90 minutes were allocated for the intervention; however, the objectives were achieved in 45 minutes. The intervention consisted of a small group discussion facilitated by the authors. The authors introduced five randomly selected narratives from a pool of reflective writings. During the intervention, each narrative was analyzed by the authors, sentence by sentence, helping the learners to identify the pertinent ACGME core competencies and inserting them in the text when applicable.
A week after the intervention, both groups were given five preselected narratives (test narratives) to analyze and identify what ACGME core competencies were reflected by the narrative. The learners were asked to analyze only two highlighted sentences in each test narrative (Fig. 1). The highlighted sentences of the test narratives were pre-analyzed by six educational experts to minimize the inter-rater variability and to improve internal validity. The input of each expert was taken into consideration. The authors calculated adjusted score for each core competency related to each highlighted sentence of the test narratives. For example, if the six experts identified a total of 16 core competencies in sentence #1 of narrative C, the score is distributed as 0 for medical knowledge, 4 for patient care, 6 for interpersonal and communication skills, 5 for professionalism, 1 for system-based practice, and 0 for practice-based learning and improvement, we would calculate adjusted score for each core competency by dividing the frequency of that competency by the total number of competencies, and then multiplying the result by hundred (Table 1). As such, the adjusted score for medical knowledge would be zero; 25.00 for patient care, 37.50 for interpersonal and communication skills, 31.25 for professionalism, 6.25 for system-based practice, and 0 for practice-based learning and improvement. The learners test narratives will be scored accordingly, that is if the learner identifies patient care only, he would receive 25.00 out of the total score of 100. If he identifies patient care and interpersonal and communication skills, he would receive 25.00 + 37.50 = 62.50 out of the total score of 100, and so on.
A sample Narrative.
ACGME core competencies identified by experts for the sample narrative with calculation of the adjusted score for each core competency.
A week later, the control group was subjected to the same intervention to comply with the cross-over design of the study. Data were then collected and a statistical analysis was completed.
Results
Nineteen of the 27 residents were able to participate in the study. Nine learners were randomized into the control group and 10 were randomized into the intervention group. The breakdown by postgraduate year (PGY) level is depicted in Table 2. Each learner analyzed 10 sentences within the five test narratives. The mean score for each learner across the 10 sentences and the grand mean score for each group were calculated. The grand mean score for the control group was 58.75 (SD 13.4) (Table 3) and that for the intervention group was 69.90 (SD 15.8) (Table 4). Since our theory was that the intervention can only benefit the learners, the two mean scores were compared using one-tailed t-test. Our one-tailed t-test analysis showed no significant difference between the two groups (t = 1.647, df = 17, P = 0.057).
Breakdown of the learners by years of training.
Control group scores for each of ten sentences out of 100 points for each sentence, means and grand mean for the group.
Intervention group scores for each of ten sentences out of 100 points for each sentence, means and grand mean for the group.
Discussion
Our educational framework assumed that helping the learners with analysis and identification of ACGME core competencies within any reflective writing will improve their knowledge and skills, as well as their ability to incorporate the ACGME competencies in their daily encounters, hence serving as useful means for teaching and may be evaluation of learners' performance in the stated domains of medical education. This notion was based on prior retrospective data that we alluded to in the introduction section. However, our results from this prospective, randomized pilot study failed to show any statistical significance. This may, in part, be related to some of the limitations of this study. We did not perform a priori power calculation due to the fact that the population we can draw from was limited (convenience sample).
We propose several possibilities for the lack of statistical significance. One possibility could be that our results represent type II error. Post priori analysis showed that our study was underpowered because of the small convenience sample that we could draw from. The effect size was only 0.43. If we were to seek an effect size of 0.7, with an alpha level of 0.05 and a study power of 80%, we needed 26 learners in each randomized arm. Another possibility could be the fact that there was wide variability among the educational experts in their evaluation of the test narratives for which reaching a consensus was difficult. A third possibility could be the fact that some of the advanced learners could have learned how to identify the ACGME competencies in their daily encounters through their clinical training over the years. And finally, we must entertain the possibility that our intervention was not effective in the first place.
Conclusion
Our pilot study failed to show any statically significant improvement in the learners' ability to identify the ACGME core competencies using narratives from their daily encounters. Post priori analysis showed our study to be underpowered; however, we were able to calculate the number of learners we needed for each randomized arm. Further prospective studies using larger sample sizes are worthwhile to determine if this simple, reproducible, affordable educational tool is an effective pedagogy to teach the ACGME core competencies using the learners' daily encounters.
Author Contributions
Conceived and designed the experiments: SJ and RM. Analyzed the data: SJ. Wrote the first draft of the manuscript: SJ. Contributed to the writing of the manuscript: SJ. Agree with manuscript results and conclusions: SJ and RM. Jointly developed the structure and arguments for the paper: SJ. Made critical revisions and approved final version: SJ and RM. Both authors reviewed and approved of the final manuscript.