Mindfulness-Based Teaching Competency Assessment: Comparing Self-Assessment With Expert Evaluation in the Veterans Administration – Compassionate Awareness Learning Module Program

Abstract

Background

Meditation, including Mindfulness-Based Interventions (MBI), is a required Complementary and Integrative Health intervention at the US Department of Veterans Affairs (VA). Training VA clinicians to provide MBI at scale must address fidelity concerns and the assessment of clinician competency.

Objective

The psychometric properties of the Mindfulness-Based Intervention: Teaching Assessment Criteria (MBI:TAC), a widely used tool for assessing facilitator competence, continue to be explored. To support the dissemination of MBI, the utility of using the MBI:TAC for self-assessment for clinicians in a national training program was evaluated.

Methods

In a training cohort of VA clinicians (n = 39), participant self evaluations on 2 domains of the MBI:TAC are compared to the competency scores of 2 expert evaluators as based on the observations of a 10-minute exercise. Additionally, the inter-rater reliability between the 2 experts was explored.

Results

Intraclass Correlation for the 2 expert evaluators for Guiding Practice was significant (ρ = .83, P = .003), but was not significant for Embodiment of Mindfulness (ρ = .34, P = .186). Self-evaluation scores were not significantly correlated to expert rater scores such that participants rate their level of competence higher than expert scores.

Conclusion

The MBI:TAC, while an essential tool in teacher training, may not produce accurate scores when used for self-assessment. Instruction from a senior teacher is needed for accurate scoring. Interrater reliability may be improved with enhanced domain operationalization and training. Implications for MBI training are explored.

The demand for Mindfulness-Based Interventions (MBIs) in clinical settings and the research into these practices has expanded world-wide in recent years. ¹ This expansion is also taking place at the Veterans Health Administration (VA), the largest healthcare system in the USA with more than 380,000 employees and 1300 medical centers/outpatient clinics. ² Consistent with its history of innovation, the VA continues to undergo a large-scale transformation to a Whole Health system of care that empowers and equips people to take charge of their health and well-being and to live their life to the fullest. This transformation includes the provision of complementary and integrative health (CIH) approaches as a standard component of care.^3,4 Mindfulness is at the heart of the Whole Health system and meditation, including MBIs, is a required intervention throughout the VA. Preliminary findings suggest that Veterans engaged in Whole Health care, when compared to those in standard models of care, report greater improvements in health and well-being outcomes, as well as a 3-fold reduction in opioid usage. ⁵ Employees engaged in delivering Whole Health had lower levels of burnout and higher retention. Due to the positive outcomes and success at the Whole Health flagship sites, there is a mandate for the integration of Whole Health into Primary Care and Mental Health settings across the VA healthcare system. ⁶ Additionally, VA policy requires various evidence based CIH approaches, including meditation, be made available for Veterans on site, via telehealth, or through care in the community if deemed appropriate by the care team as a part of their comprehensive health and well-being plan. ⁷ A significant barrier to making MBIs available at every VA facility was the paucity of high-quality training to meet the needs of a large, national clinical workforce at scale.

VA CALM (Veterans Administration – Compassionate Awareness Learning Module) was developed to address the need for healthcare professionals to teach high quality MBIs on a national scale. This year-long training, described in detail elsewhere, ⁸ produced significant and large magnitude increases in competence and self-efficacy in teaching MBIs. The training program also confers additional benefits on the participants including significant and large magnitude improvements in mindfulness, self-compassion, burnout, and stress. The VA CALM program was developed across 6 iterations with 23 clinicians in the first cohort in 2014, scaling up to 295 clinicians currently in training expected to graduate in 2023. To date, nearly 500 clinicians have been trained. These clinicians reported a mean age of 45 years of age (SD = 10.4) and 8.7 years (SD = 6.7) of VA employment and from diverse disciplines including psychologists (48%), social workers (33%), physicians (10%), nurses (5%), and other clinicians including dentists, pharmacists, and occupational therapists (3%). ⁸ Participants in the VA CALM program are trained to teach the curriculum as described in The Clinician’s Guide to Teaching Mindfulness. ⁹ This 90-minute, 8 session course teaches a variety of mindfulness and compassion practices (eg, body scan, breath, sound, loving kindness, mindful walking, compassionate breathing) in a group format.

There are specific benefits and challenges in training highly experienced, mid-career professionals. Such professionals have established expertise in the assessment and treatment of medical and psychiatric disorders. Mental health clinicians also have years of training and experience with the complex skills of psychotherapy and group relations. Teaching an MBI, however, requires an integration of disparate skills when Western empiricism meets ancient contemplative practices. ¹⁰ This integration of skills, some of which are novel for seasoned clinicians, requires a careful roadmap of shared competency domains.

The incorporation of the Mindfulness-Based Intervention: Teaching Assessment Criteria (MBI:TAC) ¹¹ competence assessment tool into the VA CALM training addresses various program goals. First, the competency tool was used to demonstrate the effectiveness of the VA CALM program in increasing participant competence, ⁸ a key outcome of the parent study. Second, use of the MBI:TAC “maps the territory” of the skills needed to teach an MBI and introduces a shared language. ¹² Third, inclusion of the MBI:TAC in training boosts teacher effectiveness, as suggested by past research in other settings that inclusion of this assessment of competence enhanced teacher development. ¹³ Fourth, use of the MBI:TAC addresses past concerns about the variability of diverse training programs around the world, summarized as the “thorny issue of clinician training” (p. 605). ¹⁴ As an assessment tool and general framework of training, the MBI:TAC brings consistent competency metrics to the field. And fifth, the MBI:TAC is a guide for both fellow teacher trainees and faculty to offer domain specific constructive feedback during training. In a practical way, the domains of the MBI:TAC provide a concrete framework through which to assess a facilitator’s teachings; offering clarity when the diverse qualities of a skilled teacher can be seen as ephemeral.

Questions persist about the risks of introducing a competency measure that could result in reductionism and the loss of the qualities MBI training seeks to develop; namely curiosity and trust emergence becoming supplanted by judgement and performative striving. ¹⁵ The psychometric properties of the tool continue to be explored. The parent VA CALM study introduced a new way of using the MBI:TAC in teacher training. Participants self-assessed their own competence at 3 time points; before the training began, at the 6-month point, and at the end of the training at the 12-month point. Prior to this study, the concordance between trainee self-evaluation and expert observation using the MBI:TAC was not known. The present study explored the relationship at the completion of the training between how the participants self-assessed their teaching competencies and the evaluations from the VA CALM faculty.

We posed 2 research questions for the present study: (1) What is the correlation between participant self-evaluation and expert evaluation and (2) is there high inter-rater reliability between the expert evaluators?

Method

Participants

In the parent study, participants (N = 74) were divided into 2 groups to facilitate in-person instruction in smaller, more manageable cohorts. ⁸ The present study was conducted with the second cohort only (n = 39), delivered at VA Greater Los Angeles Healthcare System in November 2019. The study was approved by the Institutional Review Board and the participants of the second cohort also signed consents to be video recorded for this subgroup study.

Measure

The MBI:TAC ¹¹ assesses mindfulness-based teaching competencies and comprises 6 domains including: (1) Coverage/Organization of Curriculum, (2) Relational Skills, (3) Embodiment, (4) Guiding Practices, (5) Course Themes, and (6) Group Process. A six-point Likert-type scale is used on each domain with the qualifiers incompetent, beginner, advanced beginner, competent, proficient, and advanced. In this study, the qualifier incompetent was replaced with the less pejorative term no experience. The MBI:TAC was adapted for the parent study by asking respondents for a self-assessment on the measure, rather than an expert assessment. Specifically, participants were asked to assess their mindfulness instructional skills in group settings for the 6 domains. For this study, only the domains of Embodiment and Guiding Practices were used.

Procedures

The final, 2-day, in-person training was the culmination of a year-long training but presented some significant logistics challenges for 2 faculty members to review all cohort participants (n = 39) as they led a 10-minute guided practice in a dyad. Given the time constraints, 35 of the participants received a faculty rating. The faculty selected 2 of the domains for assessment: (1) Embodiment, and (2) Guiding Practices. A comprehensive assessment of all domains would require observations of more extended teaching and was not possible in the time available. Participants were introduced to the MBI:TAC at the onset of the training as roadmap of competency but were not given specific training in the use of the measure as an evaluation tool. Participants had weekly peer practice and offered feedback to 1 another, but it was not in the full MBI:TAC format, thus participant skill in using the MBI:TAC as an assessment tool was limited.

Both faculty members had received prior training in the use of the MBI:TAC, including direct instruction at a workshop from the instrument’s creators. The observed practices included mindfulness of breath, body scans, and loving kindness practice. To enhance inter-rater reliability, the 2 faculty first observed 1 dyad practice together, then scored the 2 domains separately, then they compared their scores and through discussion, reached a consensus on an agreed-upon approach to scoring. They repeated this approach for second dyad, scoring the 2 domains separately and then through discussion reached a consensus score. During the 2-day training, there were 6 instances were 1 faculty member watched and scored a session live and the other faculty could watch the exact same session on video. Therefore, these 6 live and recorded pairs of observations of the exact same encounter were included in the inter-rater reliability assessment along with the 2 initial dyads completed at the start of the observations. In total there were n = 8 observations that were focused on the same encounter and were included in the inter-rater reliability assessment.

After the inter-rater procedures described above, the faculty continued to assess as many participants as time allowed (n = 27; Rater 1 n = 14; Rater 2 n = 13). Each of these additional measures had just 1 faculty member rating. Cumulatively, with n = 8 participants evaluated by both faculty for the inter-rater assessment, and an additional n = 27 participants rated by just 1 faculty member, a total of n = 35 participants were given a competency score.

Faculty ratings were compared to participant ratings on Embodiment and Guiding Practices. For the 8 dyads observed by both faculty, the single consensus score was used in the analysis. Of the 35 participants with faculty scores, we narrowed our analysis to the 25 instances where we had both a faculty and a participant score.

Data Analyses

Several analyses were conducted to analyze the reliability of these data when considering (a) the ratings of the 2 expert evaluators on 8 sessions that were rated by both faculty and (b) the self-report ratings of participants and ratings by the expert evaluators for 25 sessions where both data points were available. Analyses consisted of examining differences using paired t-tests. A two-way random effects model (single measure and consistency) was used to calculate the intraclass correlation coefficient [or ICC (3, 1)]. ¹⁶ This model treats raters and observations as random effects and partitions the variance into variability within raters, between observations, and error. The ICC (or ρ) is a more conservative estimate of the similarity of assessments provided by raters.

Results

The participants had an average age of 46 years (range 25 to 69), worked at the VA an average of 8 years, and were female (75%) with 18% also identifying as Veterans. Participants self-identified as White (74%), Black (11%), Asian American (5%), Hispanic (8%), and Native American (4%) as well as declined to identify (8%; total greater than 100% due to multiracial responses.) Occupations of the participants included Social Worker (41%), Psychologist (32%), Nurse Practitioner/Nursing (9%), Physician (8%), and other clinicians (10%). Most participants (72%) already had a regular, personal mindfulness practice prior to the training with a frequency of 1-2 times per week (27%), 3-4 times per week (39%), and 5 or more times per week (34%) with the most frequent length of practice in the 10-20 minute range (55%).

The analyses of the 2 faculty raters, conducted prior to arriving at mutually agreed upon scores, suggests there is no evidence of significant difference using paired t-tests between the Rater 1 and Rater 2 scores on Embodiment and Guiding Practices (d = −.15, P = .685 and d = .54, P = .170, respectively, Table 1). There was no evidence to suggest significant differences between raters using a paired t test; however, it is likely this is due to our small sample size. Moreover, a difference of .54 on the Guiding Practices domain represents a substantively meaningful difference. We also examined the Intraclass Correlation (ICC), a more conservative estimate of inter-rater agreement. The ICC found that rater scoring for Guiding Practice was significant (ρ = .83, P = .003), but the ICC for Embodiment of Mindfulness was not significant (ρ = .34, P = .186).

Table 1.

Inter-Rater Agreement and Differences in Session (n = 8) Rated by Both Observers.

	Embody	Guide
Rater 1 mean	3.63	3.88
Rater 1 SD	.92	.83
Rater 2 mean	3.75	3.63
Rater 2 SD	.46	.74
Difference
t	−.42	1.53
p	.685	.170
d	−.15	.54
Intraclass correlation
ρ	.34	.83
F	2.03	10.67
p	.186	.003

Bold values are statistically significant.

To determine the accuracy of self-assessment, the results of expert rater observation of Embodiment (M = 4.04, SD = .98) and Guiding Practices (M = 4.04, SD = 1.10) were compared with the self-report scores for Embodiment (M = 4.76, SD = .97) and Guiding Practices (M = 4.64, SD = .95; Table 2). Rater and self-report scores for these 2 domains were found to be significantly different from 1 another using paired t-tests (d = −.56, P = .009; d = −.49, P = .022, respectively). There was little evidence to suggest consistency in ratings between self-reports and expert evaluators. ICCs findings were not statistically significant or impressive in magnitude for Embodiment and Guiding Practices and were not significant for either domain (ICC: ρ = .14, P = .244, and ρ = .29, P = .075, respectively).

Table 2.

Consistency and Differences Between Observer Ratings and Self-Reports (n = 25).

	Embody	Guide
Observation mean	4.04	4.04
Observation SD	.98	1.10
Self-report mean	4.76	4.64
Self-report SD	.97	.95
Difference
t	−2.82	−2.45
p	.009	.022
d	−.56	−.49
Intraclass correlation
ρ	.14	.29
F	1.33	1.82
p	.244	.075

Bold values are statistically significant.

Discussion

This study examined the inter-rater reliability of 2 expert evaluators for the MBI:TAC. The analysis of similarities, however, suggests there are inter-rater reliability differences between the domains. For Guiding Practices, all analyses indicate the raters’ scores were highly correlated and significant. The findings for the Embodiment domain, however, indicate the rater’s scores lack a high-level of Inter-rater reliability. This analysis suggests that expert raters provided very similar ratings on Guiding Practices, but more work is needed to carefully hone a similar operational definition of Embodiment. It is also likely that raters need additional training in the use of the MBI:TAC on the Embodiment domain to enhance assessment reliability.

Three prior studies have reported on the reliability of scores between raters using the MBI:TAC. The first 2 studies suggested the Embodiment domain may have lower inter-rater reliability than other domains.^17,18 This study supports these findings. Additional work was done for a more recent study to enhance the descriptors of the embodiment domain and apply additional training to raters resulting in improvements in the Embodiment reliability score. ¹⁹ Notably, the outcomes from the present study took place before the Embodiment descriptor enhancement and additional training recommendation occurred. Few studies have reported inter-rater reliability when assessing MBI:TAC domains, and no prior study reports reliability findings in a sample of mid-career healthcare professionals making these findings an important contribution to the field.

This study also explored the correlation between mindfulness teacher trainee self-evaluation ratings with expert ratings. Analyses suggest low inter-rater reliability between self-evaluation and expert evaluation on the domains of Embodiment and Guiding Practices. Expert raters gave consistently lower scores than the participants gave themselves indicating the trainees over-estimate their level of competence. Reports from diverse fields of practice suggest self-evaluations may be inaccurate in clinical settings. For example, medical residents tasked with self-rating their interpersonal skills with patients had a low level of agreement with patient ratings, yet faculty observers and patient scores were strongly, significantly correlated. ²⁰ In a study comparing expert evaluators and self-ratings of psychotherapists on cognitive therapy skills, there was a significant correlation, but the therapists significantly over-rated their competence relative to the expert rater. ²¹ Notably, the less competent therapists over-rated their skills relative to the therapist who met competency criteria. There are many blind spots to self-knowledge, or one’s beliefs about how they typically think, feel, and behave. And these blind spots are often associated with negative consequences. ²² It is striking that mindfulness training has been proposed as a path to self-knowledge and reduction of these blind spots, ²³ yet the findings here suggest this gap remains. One possible explanation may be that engaging in hundreds of hours of mindfulness training primes the participants to see themselves as “proficient,” the qualitative rating that most closely matches the participant mean. The expert evaluator mean most closely matches the qualifier “competent,” the level below proficient. Another possible explanation is related to the lack of participant training in the use of the MBI:TAC. Participants were given a reference page summarizing the various levels of competence for each domain rather than formal training in the tools use. This may have introduced scoring error.

Despite the lack of concordance between trainees and faculty for competency assessment, we recommend the field introduce self-assessment into training models. The field has been reluctant to encourage self-evaluation, perhaps due to the core mindfulness skill of noticing experience in a nonelaborative and nonevaluative manner. Mindfulness training reduces reactivity and defensiveness to ego-threatening content, ²⁴ and facilitator self-evaluations, particularly for the early stages of training, may enhance reactivity and defensive states. Yet MBI facilitators, who invest considerable time and effort in turning towards experience and seeing clearly, could benefit when their own level of competence is also more keenly observed. The use of the MBI:TAC for self-assessment also is a critical tool for a rapidly growing field struggling with scaling training models to meet consumer needs for well-trained facilitators.

After the present study was completed, the Mindfulness-Based Interventions: Teaching and Learning Companion (MBI:TLC) was introduced. ²⁵ The MBI:TLC was specifically developed based on feedback from various training organizations that trainees needed a reflection tool to support their development and enhance accurate self-evaluation. MBIs and related trainings are a rapidly advancing field. The findings here reveal a concern that the field has already begun to address by the development of the MBI:TLC. Our team has also seen this need and begun to incorporate the MBI:TLC into our training model. Yet the developers of the MBI:TLC report, “The tendency to negatively self-judge one’s development and skills as an MBP (Mindfulness Based Practice) teacher is highly prevalent” (p. 2). ²⁵ While this also is consistent with our own experience as teacher trainers, the present study suggests it is a positivity bias, or overestimating one’s competence, that may be problematic in teacher training, at least in our sample of diverse, mid-career, healthcare professionals.

Limitations

Limitations should be considered when interpreting the results of the present study. There were only 2 expert raters who evaluated 8 trainees resulting in a limited number of data points for the inter-rater reliability analyses. The results of the present study, however, suggest concerns regarding the operationalization and training related to the Embodiment domain. This issue has been addressed in recently published work. ¹⁹ The MBI:TAC was created to assess competencies during actual teaching sessions. In the present study, raters observed 10-minute guided practices with a peer rather than an actual teaching session. A possible limitation is the short duration of the observed practice rather than the entirety of the intervention. Self-evaluations were made on a more generalized assessment of teaching competencies in one-on-one clinical encounters rather than practice dyads. Additionally, the presence of a senior teacher observing a dyad for the task of evaluation, clipboard in hand, introduces a potential stressor that may impact competency assessment. The 2 expert evaluators in the study generated a single score for each domain rather than a composite score of each of the domain’s key learning features. For example, the Embodiment domain consists of key features or sub-components including Focus, Responsiveness, Steadiness, Allowing, and Natural Presence. ¹¹ Given the time limitations of the observations in the present study, we were not able to assess each of these key features. Future studies comparing expert evaluation with self-assessment would be enhanced by rating these sub-components. Outcomes of such an approach could guide instructional approaches with specific attention to the key features needed to close any discrepancy between expert and self-evaluation.

Conclusion

The MBI:TAC is an important tool for training experienced clinicians to become mindfulness facilitators. This tool offers a framework, shared language, and competency benchmarks essential for training. The present study explores possible strengths and limitations of the inter-rater reliability of expert raters. After the conclusion of this study, subsequent enhancements to the operational definitions of the Embodiment domain and the creation of the MBI:TLC for trainee support are welcome additions and suggest the field, in our view, is on the right track. However, trainees using the MBI:TAC to self-assess their competencies may overestimate their skills. This supports the essential role of the senior teacher in the training and assessment of mindfulness facilitators. In both clinical and community settings, those who attend an MBI rely solely on the teacher’s understanding and embodiment of mindfulness to deliver a high-quality class. Senior faculty must ensure MBI teachers see their own strengths, needs, and capacities clearly. An additional time commitment could be added into the training model to ensure teachers understand the competency domains and key learning features needed for an optimal developmental transformation.