Sage Journals: Discover world-class research

Abstract

Background

The current gold standard for measuring fidelity (specifically, adherence) to cognitive behavioral therapy (CBT) is direct observation, a costly, resource-intensive practice that is not feasible for many community organizations to implement regularly. Recent research indicates that behavioral rehearsal (i.e., role-play between clinician and individual with regard to session delivery) and chart-stimulated recall (i.e., brief structured interview between clinician and individual about what they did in session; clinicians use the client chart to prompt memory) may provide accurate and affordable alternatives for measuring adherence to CBT in such settings, with behavioral rehearsal yielding greater correspondence with direct observation.

Methods

Drawing on established causal theories from social psychology and leading implementation science frameworks, this study evaluates stakeholders’ intention to use behavioral rehearsal and chart-stimulated recall. Specifically, we measured attitudes, self-efficacy, and subjective norms toward using each, and compared these factors across the two methods. We also examined the relationship between attitudes, self-efficacy, subjective norms, and intention to use each method. Finally, using an integrated approach we asked stakeholders to discuss their perception of contextual factors that may influence beliefs about using each method. These data were collected from community-based supervisors (n = 17) and clinicians (n = 66).

Results

Quantitative analyses suggest moderately strong intention to use both methods across stakeholders. There were no differences in supervisors’ or clinicians’ attitudes, self-efficacy, subjective norms, or intention across methods. More positive attitudes and greater reported subjective norms were associated with greater reported intention to use either measure. Qualitative analyses identified participants’ specific beliefs about using each fidelity measure in their organization, and results were organized using the Consolidated Framework for Implementation Research.

Conclusions

Strategies are warranted to overcome or minimize potential barriers to using fidelity measurement methods and to further increase the strength of intention to use them.

Plain Language Summary: The best way to measure fidelity, or how closely a clinician follows the protocol, to Cognitive Behavioral Therapy (CBT) is watching the session. This is an expensive practice that is not feasible for many community organizations to do regularly. Recent research indicates that behavioral rehearsal, or a role-play between the clinician and individual with regard to session delivery, and chart-stimulated recall, or a brief discussion between an individual and the clinician about what they did in session with the clinician having access to the chart to help them remember, may provide accurate and affordable alternatives for measuring fidelity to CBT. We just completed a study demonstrating that both methods are promising, with behavioral rehearsal offering scores that are the most similar to watching the session. Drawing on established theories from social psychology and leading implementation science frameworks, this study evaluates future supervisor and clinician motivation to use these fidelity measurement methods. Specifically, we measured supervisor (n = 17) and clinician (n = 66) attitudes, norms, self-efficacy, intentions, and anticipated barriers and facilitators to using each of these fidelity measurement tools. Quantitative and qualitative analyses suggest similar intention to use both methods, and concerns about barriers to using each method. Further research is warranted to minimize the burden associated with implementing fidelity measurement methods and deploying strategies to increase use.

Keywords

community mental health fidelity measurement behavioral intention mixed methods

Implementation research and quality improvement efforts require accurate measurement of the fidelity of to psychosocial interventions delivered in community settings (England et al., 2015). Fidelity is (1) often hypothesized as the mechanism through which client outcomes improve; (2) the most proximal indicator of treatment quality (McLeod et al., 2013); and (3) the primary implementation outcome of interest (Proctor et al., 2011). Until recently, accurate and low-cost methods to measure fidelity, which include both adherence to the components of an intervention and the skillfulness with which the intervention is executed (i.e., competence), did not exist (Cox et al., 2019; Perepletchikova et al., 2007). Direct observation, the gold standard for assessing fidelity of psychosocial interventions, is resource-intensive and nearly impossible to implement in community settings.

A recently conducted randomized controlled trial (RCT) tested novel methods to assess fidelity (specifically, adherence, or how extensively a clinician follows a protocol as intended) to cognitive-behavioral therapy (CBT) delivered to pediatric populations compared with direct observation. Two promising methods emerged: behavioral rehearsal (i.e., role-play between clinician and individual) (Beidas et al., 2014) and chart-stimulated recall (CSR) (i.e., brief structured interview between an individual and the clinician about what they did in session; clinicians use the chart to prompt their memory) (Cunnington et al., 1997; Goulet et al., 2007; Jennett & Affleck, 1998; Miller et al., 2010; Salvatori et al., 2008). Behavioral rehearsal (BR) was most accurate, but CSR also performed well. These findings suggest the potential for these two methods to be used in lieu of direct observation in implementation research and practice, as they require fewer resources (Becker-Haimes et al., 2020; Becker-Haimes et al., 2022). Economic evaluation of the two methods is forthcoming. The current study was designed to identify whether one has a greater chance of being used in community mental health settings and why.

Causal models from social psychology, such as the Theory of Planned Behavior (TPB), can help predict how likely one is to use a method and explain why. In these models, behavioral intention is the most proximal determinant of behavior (Fishbein & Ajzen, 2010; Sheeran, 2002) and can therefore help explain and predict the implementation of evidence-based practices (Fishman et al., 2020). In the present study, behavioral intention refers to the amount of effort one is willing to expend to implement BR or CSR. To use either method, supervisors and clinicians must have a relatively strong intention to do so. The strength of intention to use a fidelity measurement method may depend on the method, the people using it, and the context in which they work (Fishman et al., 2018; Wolk et al., 2019). Given that supervisors and clinicians typically are not required to use BR or CSR, relatively strong intention to use a method will increase the likelihood it is used (Eccles et al., 2007; Webb & Sheeran, 2006).

This study examines factors explicitly posited in the TPB. Behavioral intention is a function of one’s attitudes, perceived behavioral control, (also referred to as self-efficacy), and subjective norms (Fishbein & Ajzen, 2010; Sheeran, 2002; Webb & Sheeran, 2006). Attitudes are defined as the perceived advantages or disadvantages of using BR or CSR. Self-efficacy is a construct representing one’s perceived ability to perform a behavior, such as routinely using BR or CSR. Subjective norms include the perception of whether important others expect one to use BR or CSR and also the perception of whether other clinicians are using these methods. These three factors are malleable and can be targeted by strategies when intention to engage in a specific behavior (in this case, use of BR or CSR) is relatively weak (Fishbein & Ajzen, 2010).

In addition to examining psychological factors per the TPB, this study also examines contextual factors often studied and prioritized in implementation science frameworks (e.g., organizational culture and climate). Contextual factors, drawn from the Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2009), are important to consider because stakeholders’ perceptions of contextual factors can influence their beliefs about using a method and also moderate the relationship between intentions and behavior (Fishman et al., 2018, 2020). Even for those who are motivated to use a fidelity measurement method, their context must make it possible (and ideally easy) to follow through. Policies, resources, and other organizational characteristics can influence the likelihood that one is able to use BR or CSR when motivated to do so (Fishman et al., 2018, 2020). For example, a clinician who is highly motivated to use a method regularly but lacks the needed materials is unlikely to use that method frequently.

In this mixed-methods study, we endeavored to address the following aims: (1) Estimate the strength of intention to use BR and CSR in a sample of community mental health clinicians experienced with the methods (2) Measure attitudes, self-efficacy and subjective norms, which explain the strength of intention; and (3) Identify contextual factors from the CFIR, such as organizational culture and climate, that may underly these attitudes, self-efficacy, and subjective norms. We hypothesized that supervisors and clinicians would have stronger intention and perceive fewer contextual factors that might serve to impede the use of CSR compared with BR because the former most closely resembles existing supervisory processes in community organizations (Dorsey et al., 2018) (Beidas et al., 2014) and may be perceived as less anxiety provoking (Edmunds et al., 2013).

Methods

The City of Philadelphia and the University of Pennsylvania Institutional Review Boards approved this study. Informed written or verbal consent or assent was obtained from all participants.

Study Setting and Agency Participation

Using a list of organizations accepting public mental health insurance within a 50-mile radius of Philadelphia County, we contacted agency leadership via phone and email to invite them to participate in the RCT (Becker-Haimes et al., 2022). Outpatient mental health agencies or residential treatment facilities were eligible if they provided care to youth clients (ages 7–24) and employed clinicians who used at least one of 12 CBT techniques measured for the study (e.g. operant strategies targeted at youth). Supervisor, clinician, and client participants (youth and, in the case of minors, their guardians) were recruited from 27 community-based agencies providing mental health services to youth in the Philadelphia tri-state area.

Leadership from each agency invited clinicians to attend an informational meeting conducted at each agency. Research staff introduced the study to clinicians at these meetings and enrolled interested parties. Clinicians were eligible if they endorsed having at least three youth (7–24) clients on their caseload with whom they planned to use at least one CBT technique in the next month. Participants completed quantitative and qualitative evaluations. No supplementary relationships were established or preexisted between interviewers and participants. No specific information about interviewers was formally disclosed aside from affiliation with the University of Pennsylvania and standard informed consent requirements (e.g., mandated reporting).

Procedures and Sampling

Qualitative Interview Procedures

Qualitative interviews were conducted by trained, bachelor- or master-level research coordinators (including authors Hoffacker, Fugo, and Lieberman). Pilot interviews were conducted by PhD-level research team members (Drs. Becker-Haimes and Beidas). All interviewers were women. Interviewers were trained by reviewing previous interviews, shadowing trained interviewers, conducting mock interviews, and conducting two supervised interviews. Participants completed interviews in a private space at their place of work. Qualitative interviews lasted roughly one hour. No repeat interviews were conducted, transcripts were not returned to participants for comment or correction, and participants did not provide feedback on findings. Interview themes converged and reached saturation, and additional interviews were not needed.

Supervisors (n = 17)

One supervisor per agency was invited to complete a structured questionnaire that measured strength of attitudes, self-efficacy, subjective norms, and intention to use the measurement methods, and to participate in a qualitative semi-structured interview. Ten supervisors declined to participate in either portion of the study due to schedules. Fourteen supervisors completed the questionnaire and 17 completed the interview. Due to time constraints, three supervisors who completed the interview did not complete the quantitative questionnaire. Supervisors were compensated $50 per hour.

Clinicians (n = 66)

Of 218 eligible clinicians invited to participate in the larger study, 84 declined to participate, eight listed reasons other than preference not to participate and ineligibility (e.g., leadership requested that interns not participate), yielding 126 clinicians randomly assigned to study conditions. Seven clinicians withdrew, five discontinued due to failure to enroll any clients, and eight were unable to complete study activities due to interruption caused by coronavirus disease 2019 (COVID-19). Two clinicians who completed all study activities virtually due to COVID-19 were excluded from analyses, as was one clinician who completed the BR condition during piloting (after which the model of the measure changed meaningfully), yielding 103 clinicians who were included in quantitative analysis. Importantly, only clinicians assigned to BR (n = 32) and CSR (n = 34) conditions are included in the present study, as these conditions of the RCT were most promising with regard to their accuracy (Becker-Haimes et al., 2020; Becker-Haimes et al., 2022).

Clinicians completed a structured questionnaire that measured attitudes, self-efficacy, subjective norms (referred to as norms going forward), and intention after completing the measurement method to which they were assigned in the RCT. Thirty-six randomly selected participating clinicians (roughly one from each condition per agency) were invited to participate in an additional qualitative semi-structured interview upon completion of their measurement condition to assess attitudes, self-efficacy, norms, and contextual factors. One clinician in the BR condition was excluded from qualitative analyses as all study activities were conducted virtually due to COVID-19. The dataset reported here consists of only clinicians randomized to the BR (n = 17) and CSR (n = 18) conditions of the RCT. All clinicians invited to complete the additional interview agreed. Clinicians were compensated $50 per hour.

Qualitative interviews were recorded using a digital audio recorder, inputted into REDCap (Harris et al., 2019), transcribed, and coded using NVivo12 software and Microsoft Excel.

Measures

TPB Constructs

Consistent with our first and second aim, we examined TPB constructs in two ways. First, among supervisors and clinicians, we used gold standard established quantitative methods to measure attitudes, norms, self-efficacy, and intention to use each of the two methods (Fishman et al., 2020). Scales were modeled from items developed by Fishbein & Azjen (2010), typically yielding alphas >0.80 and test-retest reliability of >0.70. Examples of items from the clinician version are below; parallel items were developed for supervisors. Supervisor questionnaires inquired about each method included in the study while clinician questionnaires inquired only about the condition to which they were randomized.

Second, we followed qualitative belief elicitation interview procedures associated with the TPB and similar causal models. Where quantitative measures offered a measure of the directionality of constructs underlying intention (e.g., how positive or negative attitudes are), qualitative measures assess the specific beliefs underlying positive or negative constructs (Erbe et al., 2020; Lederer & Middlestadt, 2014).

TPB Construct Quantitative Questionnaires

Attitudes. Attitudes towards each method were measured by six items asking respondents to use scales anchored by opposite adjective pairs to evaluate their use of this method as relatively “burdensome” versus “manageable,” “foolish” versus “wise,” “bad” versus “good,” “useless” versus “effective,” “confusing” versus “clear,” “unpleasant” versus “pleasant.” The response options, which ranged from 0 to 10, were averaged across these items.

Self-efficacy. Self-efficacy was measured with two items (response range 1 “strongly disagree” to 5 “strongly agree”): “It is totally up to me whether I participate in [BR/CSR],” (to measure perceived behavioral control) and “Participating in [BR/CSR] is something I could learn how to do.” The overall score represents an average of these two items.

Subjective Norms. Norms were measured with two items (response range 1 “strongly disagree” to 5 “strongly agree): “Most clinicians would approve of me participating in [BR/CSR],” “Most clinicians with a job like mine would participate in [BR/CSR].” The overall score represents an average of these two items.

Intention. Intention was measured using one item (response range 1 “very unlikely” to 7 “very likely”): “Imagine that your supervisor uses behavioral rehearsal/chart-stimulated recall about once a month with you to evaluate your fidelity to CBT. How likely are you to participate in this type of supervision?”

Demographics

Demographics included age, gender, race/ethnicity, licensure status, education, role within the agency, human services work history, weekly hours spent working at the present agency, and average time allotted for supervision (Table 1). Clinicians were also asked about the typical caseload, year they initiated CBT in their work, and participation in city-wide training initiatives in evidence-based practices.

Table 1.
Supervisor and clinician demographics.

Variable Supervisor Clinician

n = 17 BR n = 32 CSR n = 34

n % n % n %

Female gender 10 58.8 28 87.5 27 79.4

Hispanic/Latinx 3 17.6 4 12.5 2 5.9

Racial identity^a

White/Caucasian 14 82.4 22 68.8 29 85.3

Asian or Pacific Islander 0 0 3 9.4 1 2.9

Black or African American 2 11.8 5 15.6 4 11.8

Other 1 5.9 1 3.1 0 0

Age (M, SD) 40.18 (10.48) – 36.35 (12.62) – 37.71 (12.43) –

Education

Bachelor’s degree – – 1 3.1 1 2.9

Master’s degree 14 82.4 29 90.6 33 97.1

Doctorate 2 11.8 2 6.3 1 2.9

Type of employee

Salaried – – 17 53.1 17 50

Fee-for-service – – 13 40.6 17 50

M (SD) M (SD) M (SD)

Years of experience

In human services 15.73 11.39 9.28 10.95 10.24 9.99

As a supervisor 6.22 3.92 – – – –

Hours of supervision per week

Individual supervision – – 0.83 0.48 0.88 0.50

Group supervision – – 0.81 0.65 0.74 0.58

Note. BR = behavioral rehearsal; CSR = chart-stimulated recall.

^a
Reflects the number and percentage of participants endorsing this identity.

Qualitative Interviews to Elicit Beliefs Informing Attitudes, Self-Efficacy, and Norms and Contextual Factors

The qualitative interviews had two goals: (1) to identify specific beliefs informing attitudes, self-efficacy, and subjective norms; and (2) to understand contextual factors that may underly these beliefs or moderate the relationship between intention and behavior. Figure 1 depicts the relationships between these constructs via three pathways: Pathway A, the influence of contextual factors on the TPB constructs of attitudes, self-efficacy, and subjective norms; Pathway B, TPB constructs informing behavioral intention; and Pathway C, moderating effect of contextual factors on the intention-behavior gap. In the interviews, we focused on Pathway A because these methods have not yet been adopted in routine care in community mental health. Semi-structured individual interviews with clinicians were conducted using 10 open-ended questions to identify salient “tip of the tongue” beliefs that spontaneously come to mind when thinking about the behavior of interest (Sutton et al., 2003). Rather than relying on preconceived lists of statements identified by researchers, this method identifies beliefs commonly reported by the population of interest.

Figure 1.
Causal model predicting EBP use.

To identify beliefs underlying attitudes, we asked respondents to share the perceived advantages and disadvantages of using BR and/or CSR. To identify beliefs underlying self-efficacy, we asked what would make it difficult or easier to routinely use BR and/or CSR. To identify beliefs underlying norms, we asked participants who would approve or disapprove of them routinely using BR and/or CSR. The interview guide was rehearsed with members of the research team, and three pilot interviews were conducted at the first participating agency. The guide was subsequently adjusted to focus on the content and preserve the approximate one-hour time limit.

Data Analysis

Quantitative Analyses of Questionnaire Data. First, for each method, and consistent with Aim 1, we report on descriptive analyses for the four TPB variables: attitudes, self-efficacy, norms, and intention. Second, we conducted paired samples and independent samples t-tests to examine if supervisor or clinician attitudes, self-efficacy, norms, and intention toward BR and CSR differed. Third, and consistent with Aim 2, to ensure that these constructs explained variance in intention as expected, we conducted a linear regression using bootstrapped p-values (Austin & Tu, 2004) to test if attitudes, self-efficacy, norms, and condition predicted intention in the clinician sample. The sample size was too small for the supervisor sample to conduct a similar analysis.

Qualitative Analysis of Belief Elicitation Interview Data. In service of our second aim, we identified which beliefs were most commonly reported when asked each of the elicitation questions. Using a content analysis approach, coders summed the frequency for which specific beliefs were reported (Middlestadt et al., 2011; Middlestadt et al., 2013). For example, when analyzing self-efficacy beliefs, the coders tracked how often respondents reported that “time” would make it difficult to use BR and/or CSR routinely. As another example, when analyzing normative beliefs, the coders tracked how often supervisors were mentioned when asked who would approve or disapprove of them using BR and/or CSR routinely. Two individuals separately coded responses and demonstrated high intercoder reliability. Coders had an average of 96% agreement (range = 92%−98%). We report the beliefs most commonly endorsed (i.e., mentioned by at least 15% of participants).

In service of our third aim, we employed an integrated approach to organizing the belief elicitation interview data using the CFIR to identify contextual factors not captured by TPB-focused analyses. Four members of the research team reviewed five individual transcripts, identifying themes independently, and convened to compare preliminary codes. The final codebook (available upon request) contained 15 unique, neutral codes which were mapped onto the five CFIR domains (Intervention Characteristics, Outer Setting, Inner Setting, Individuals Involved, Process). The same codebook was used for supervisor and clinician interviews. Once the codebook was finalized, supervisor and clinician transcripts were coded by four coders. The coding team met weekly to review questions and reach consensus. Four transcripts were coded independently and reviewed as a group to achieve reliability (reliability rate ≥ 0.80); transcripts then were coded independently with a subset of 12 transcripts overlapping between coders to assess final reliability across codes (reliability rate ≥ 0.80).

Results

TPB Quantitative Survey Findings

See Table 2 for descriptive statistics for attitudes, self-efficacy, norms, and intention.

Table 2.
t-tests testing attitudes, self-efficacy, norms, and intentions in supervisors and clinicians.

BR CSR t(14) p

n M SD n M SD

Supervisors

Attitudes 15 6.46 1.87 – 6.30 1.91 −0.320 .744

Self-efficacy 15 4.17 0.65 – 4.23 0.68 0.695 .479

Norms 15 3.30 0.77 – 3.77 0.78 2.168 .057

Intention 14 5.07 1.49 – 5.43 0.65 0.751 .468

t(64)

Clinicians

Attitudes 32 6.29 2.34 34 6.60 2.66 0.512 .618

Self-efficacy 32 3.86 0.61 34 4.01 0.53 1.105 .291

Norms 32 3.70 0.67 34 3.76 0.80 0.338 .748

Intention 32 5.59 1.34 34 5.74 1.42 0.416 .688

Note. BR = behavioral rehearsal; CSR = chart-stimulated recall.

Intentions. There were no significant differences between supervisors’ intention to use BR or CSR. There were no differences in clinicians’ intention to use BR or CSR. For both types of participants, intentions were fairly high.

Attitudes, Self-efficacy, and Norms. Differences between supervisor attitudes, self-efficacy, and norms also yielded no significant differences between BR and CSR (all ps > .05; Table 2). Clinician attitudes, self-efficacy and norms also showed no significant difference between BR and CSR (all ps > .05; Table 2).

Relationships Between the Variables. Pearson’s correlation showed significant correlations between clinician intention and attitudes (r = .465, p < .01), intention and norms (r = .429, p < .01), and attitudes and norms (r = .375, p < .01; Table 3).

Table 3.
Paerson correlations between attitudes, self-efficacy, norms, and intention in clinicians.

Attitudes Self-efficacy Norms Intention

Attitudes – 0.099 0.375* 0.465*

Self-efficacy 0.099 – 0.135 0.033

Norms 0.375* 0.135 – 0.429*

Intention 0.465* 0.033 0.429* –

p < .01

Linear regression investigating the relationships between attitudes, self-efficacy, norms, condition, and intention to use either BR or CSR in clinicians was statistically significant (F [4, 65]* = 6.35, p < .001), with more positive attitudes and supportive norms related to stronger intention (see Table 4).

Table 4.
Clinician attitudes, self-efficacy, norms, and condition predicting intention.

Variable B SE B p

Attitudes 0.196 0.064 0.355* .036

Self-efficacy −0.112 0.263 −0.046 .706

Norms 0.563 0.218 0.301* .031

Condition −0.062 0.297 −0.023 .843

Note. R² = 0.294; B = unstandardized coefficient; B = standardized coefficient.

p < .05.

Qualitative Belief Elicitation Interview Results

Themes that emerged from TPB coding and CFIR coding overlapped, such that beliefs coded within attitude, self-efficacy, and normative constructs (Pathway B, Figure 1) were captured within the CFIR’s coding of contextual factors (Pathway A, Figure 1). Because CFIR coding sought to categorize the responses in a manner consistent with implementation science terminology, and because the TPB method prioritizes lists of factors, findings from CFIR coding elaborated rich contextual information informing the specific beliefs identified by TPB coding. Due to this overlap, CFIR findings are presented within the TPB constructs they underly (Table 5).

Table 5.
Theory of planned behavior and consolidated framework for implementation research integration of results.

CFIR domains TPB constructs

Beliefs underlying attitudes Normative referents Self-efficacy beliefs

Dislikes/disadvantages Likes/advantages

Inner setting Either fidelity method Supervision time may become less helpful than it is now.^a n/a n/a There is already a shortage of time to complete important tasks in supervision (e.g. administrative tasks, clinical skill building, etc.)^aFor client crisis sessions, fidelity measurement may not be perceived as a good use of time.^bFrequent disruptions (walk-in clients) may complicate consistent execution of a method.^bClinicians are already overburdened by current demands and may not have the capacity to incorporate additional supervision activities.^b

BR specific n/a n/a n/a n/a

CSR specific n/a n/a n/a Additional effort is required to ensure that session notes are prepared for supervisor to review in supervision.^b

Intervention characteristicsEither fidelitymethod Measure may not give an accurate portrayal of what transpired in session due to memory lapse, inability to communicate complexity of the session.^a Fidelity method may make clinicians more aware of when they use CBT in session^b n/a n/a

BR specific Roleplaying with a supervisor can be awkward^b Additional opportunity to reflect on what happened in session may be a benefit. ^b Clinicians felt clients may disapprove if they knew BR was implemented.^b n/a

CSR specific Perceived “interrogative” nature of CSR format may cause clinicians discomfort. (5XY004)^a CSR may make clinicians more mindful of why CBT is useful.^b Clinicians identified that other clinicians may dislike CSR due to tension, “interrogative” nature. (5XY004)^b n/a

Individuals InvolvedEither fidelitymethod Measures may be overly structured, causing supervision to feel “rote” or “robotic” (4SN004, 4QU004)^aAssessment may detract from modeling and mentorship currently taking place in supervision^aPressure on clinicians to perform or a sense of constant critical evaluation, resulting in anxiety and detriment to supervisor rapport.^cPerceived infringement on clinician autonomy in session^a Measure may benefit supervision time by lending structure^aIncorporating a method may reinforce clinician knowledge about how to use CBT techniques (or help clinicians learn CBT techniques)^aOpportunity for rich feedback^a Clinic leadership would approve of implementing a fidelity method as it would promote EBP uptake^a Using a fidelity measurement may not be appropriate for all clinicians:Seasoned clinicians may benefit less than newer clinicians who are still familiarizing themselves with CBT. However, newer clinicians may benefit less than seasoned clinicians as they need supervision time to develop general clinical skills, learn proper documentation procedures, etc. ^aMay not be appropriate for traumatized or highly complex client sessions as clinician may not be able to give sufficient context for fidelity method to be adequately executed.^a

BR specific BR may not give an accurate portrayal of what transpired in session due to supervisor inability to adequately act as client.^a Value role play as a training tool despite discomfort^a n/a n/a

CSR specific Memory lapse may lead to inaccuracy.^a Ability to use session note mitigates risk of memory lapse.^a n/a n/a

Outer SettingEither fidelitymethod n/a n/a n/a Fee-for-service clinicians may be unable to execute this kind of task given that they are not paid for their time outside of session and their schedules may be unwieldy.^b

BR specific n/a n/a n/a n/a

CSR specific n/a n/a n/a n/a

Note*. Response’s orientation within CFIR coding indicated by row, TPB orientation indicated by column.

Qualitative findings indicate the relationship between CFIR-identified contextual factors and TPB constructs as represented by Pathway A (Figure 1). CBT = cognitive behavioral therapy; TPB = theory of planned behavior; BR = behavioral rehearsal; CSR = chart-stimulated recall; CFIR = Consolidated Framework for Implementation Research.

^a
Expressed by both supervisors and clinicians.

^b
Expressed by clinicians.

^c
Expressed by supervisors.

Beliefs Underlying Attitudes

Commonly Reported Dislikes/Disadvantages

TPB-guided coding highlighted dislikes/disadvantages of implementing either fidelity measurement method that spanned various CFIR categories. In the intervention characteristics domain, supervisors and clinicians expressed concern that BR may not give an accurate portrayal of what happened in the session. For example, clinicians reported that their memory may worsen if BR occurs too long after the session, that role-plays may not adequately capture client or clinician behavior, and supervisors, as well as clinicians, worried that clinicians may be unable to give sufficient contextualization or background about a specific client or session for the supervisor to adequately act as the client. Concern about memory lapse was also present for CSR, but supervisors and clinicians agreed that having session notes mitigated this concern. In the domain of Inner Setting factors, clinicians worried that their supervision time would be less helpful if these tools were used.

Pertaining to the CFIR domain Individuals Involved, for both fidelity measurement methods supervisors and clinicians agreed that implementing these tools into their supervision could lend more structure, though opinions around the value of that addition varied. Most supervisors and clinicians thought that integrating such a measure would feel “robotic,” “rote,” or overly structured (4SN004; 4QU004), and that it could detract from modeling and mentorship that takes place in less structured supervision. Supervisors and clinicians agreed that adding either method may increase pressure on clinicians in two ways. First, using a fidelity check may apply pressure to change the behavior of clinicians in session with their clients. Supervisors and clinicians viewed this as an infringement on clinician autonomy, as evidenced in the following quote from a supervisor speaking about CSR, “So I think if I had a list of did you do this or that, it would take away from the clinician’s feeling like they had some autonomy to do what they felt was appropriate in the session at the time” (4WH004). Second, they felt that introducing these methods may create a threatening sense of constant evaluation or “looking for problems rather than…growth opportunities”(4QJ004), although some added that they valued role-play experiences for the learning opportunities they provide despite this discomfort. One supervisor pointed out that performance reviews currently depend upon professionalism and general good standing as an employee, and expressed concern that using CBT fidelity assessment may begin to factor into performance appraisals (which they viewed unfavorably). Clinicians also identified this tension as a concern, although discomfort around the “interrogative” (5XY004) nature of a method was only voiced regarding CSR, while anxiety in BR seemed to refer to awkwardness of role-playing with a supervisor.

Commonly Reported Likes/Advantages

Participants also identified the advantages of using the methods. In the CFIR domain of Individuals Involved, supervisors and clinicians hypothesized that incorporating either method may reinforce clinician knowledge about how to use CBT techniques (or help clinicians learn to use CBT techniques) and provide an augmented opportunity for rich feedback in supervision. Specific to BR, clinicians identified the benefit of additional opportunities to reflect on what happened in session with their supervisor, while clinicians liked that CSR would make them more mindful of why CBT is useful. In the domain of Intervention Characteristics, clinicians liked that using these methods routinely would make them more aware of the CBT that they do use in the session.

Commonly Reported Self-Efficacy Beliefs

Supervisors and clinicians expressed concerns about their ability to execute either method for reasons that primarily mapped onto the CFIR domains of Inner Setting, Outer Setting, and Individuals Involved. Regarding Inner Setting factors, supervisors and clinicians reported that supervision time is a precious commodity, and this time is frequently dedicated to reviewing clinical concerns, other modalities of therapy, and administrative issues/agency procedures. Clinicians stated that they already lack time in supervision to discuss necessary topics, particularly for clients in crisis. Using a new measure that would take time from these priorities caused clinicians concern that they may “waste” their supervision time (5AM004), as this intervention may “get in [the] way of what [they] actually need from supervision,” (5AM004). Between the expectations around session time and documentation, clinicians often feel they do not have time to complete certain exercises that are currently mandated by the clinic, stating that “we already don’t do the mandated stuff … I think adding one more thing, I just don’t think anyone would do it well,” (5AM004). Specific to CSR, clinicians expressed concern related to the need for session notes to be prepared, available, and ready for supervisor review to execute the method. These tasks constituted “just one more thing we have to do” in an already busy schedule (5JQ006).

Relatedly, in the CFIR domain of Outer Setting, clinicians expressed concern about either method in terms of how they may impact fee-for-service clinicians. Because these clinicians are often not paid for time that is not spent with clients, every additional task on their plate constitutes unpaid work. Adding time to supervision that may not be perceived as contributing directly to their growth as a clinician (as time spent focusing on development of clinical skills or necessary administrative topics may) and for which they are not paid constituted an understandable burden. Additionally, fee-for-service clinicians often work at multiple sites, making the logistics of scheduling time to execute this kind of supervision task difficult.

In the CFIR domain of Individuals Involved, supervisors and clinicians felt that implementing these methods may not be equally appropriate for all clinicians. Some felt that more seasoned clinicians would benefit less from these methods than newer clinicians who are still familiarizing themselves with CBT, while others believed that it would be less appropriate to use these methods with a new clinician who should spend supervision time learning more basic elements of therapy (e.g., rapport-building, proper documentation). Supervisors and clinicians thought that using these methods would be more difficult or less appropriate for complex or highly traumatized clients, as they would not be able to give sufficient context to communicate a comprehensive idea of the session.

Commonly Reported Normative Referents

Supervisors and clinicians thought that clinic leadership (e.g., upper management, clinical leads, supervisors) would approve of implementing BR or CSR as it would promote CBT uptake in their organization, a perceived benefit. Clinicians expressed concern that their clients may disapprove if they knew that BR was being implemented, as they feared their clients may dislike the concept of someone acting as them during supervision. Clinicians believed that other clinicians may dislike using CSR in supervision, citing the same reasons expressed as dislikes surrounding this method (e.g., tension, perceived interrogative nature of CSR). As these comments pertained to the opinions or evaluations of others, all of these observations coded by TPB as normative referents were mapped onto the CFIR domain of Individuals Involved.

Discussion

We examined intention and determinants of intention to use two promising methods to index adherence to CBT in community mental health. We did so using approaches that include a causal theory (TPB) and a leading implementation science determinant framework (CFIR) with hopes of establishing a richer understanding of contextual factors that underlie beliefs. We hypothesized that stakeholders would have a stronger intention and perceive fewer barriers to using CSR compared to BR. Quantitative measurement suggests that intentions to use both methods were moderately strong and not significantly different from each other or across supervisors and clinicians. Attitudes and norms were related to the clinician intention to use the methods, suggesting potential targets for future strategies to strengthen intentions toward using these methods. Qualitative data highlights the specific beliefs and the contextual factors informing them that should be influenced to change attitudes and norms. These insights suggest that there are many multi-level factors influencing individual attitudes, self-efficacy, and norms that must be addressed prior to implementation of any method to measure components of fidelity, including adherence (see Pathway A, Figure 1).

Given that clinician intentions to use BR and/or CSR were related to attitudes and norms (Pathway B, Figure 1), effective and tailored implementation strategies to increase clinicians’ use of BR and/or CSR could target these specific beliefs. For example, an implementation strategy targeting attitudes could include messages to change beliefs about perceived advantages and disadvantages to using such methods. The field of social psychology has developed messages that can change attitudes, which thus strengthens intention and changes behavior (Fishbein & Ajzen, 2010). Strategies can target norms with communication-based interventions to leverage message effects (Fishbein & Yzer, 2003; Hornik & Woolf, 1999). For example, if norms explain the bulk of variance in intention, messages that change one’s perception of the norms have been effective in various contexts (Cialdini et al., 2006).

Understanding the contextual factors that influence implementation is a core activity of implementation science. First, although clinicians and supervisors implicitly described how the outer setting, or broader sociopolitical context, impacted their ability to use these fidelity measurement methods, they did not explicitly or routinely mention these factors in the belief elicitation interview with the exception of financial burden imposed by the payment structure of fee-for-service clinicians. Given the overburdened nature of community mental health, it is critical to note that these fidelity methods will likely not be implemented unless they are financially incentivized, mandated by payers, and/or incorporated into standard workflow. Had we included organizational directors or other higher-level policy makers (county or state mental health leaders), these themes might have emerged more consistently.

Second, clinicians and supervisors raised concerns about the methods themselves, with a particular focus on whether either could accurately portray what happened in session and the potential loss of focus in supervision that may accompany implementation of a tool focused on CBT (in lieu of other modalities or approaches). Likely, this concern relates to attitudinal perceptions of the acceptability and the feasibility of these methods (Proctor et al., 2011). One implementation strategy to address these perceptions includes training to enhance supervisor and clinician understanding of how these methods can and should be used (Beidas et al., 2012). Another strategy would be to partner with community stakeholders to redesign approaches to be more amenable to the realities of community mental health and ensure their acceptability and feasibility (Lyon & Bruns, 2019). Although we endeavored to do so given our longstanding experience in community mental health, co-creation of the measures may have resulted in more acceptable approaches (Metz, 2015). To address these concerns, implementation efforts may need to target buy-in around the accuracy of BR and CSR and opportunities created when using these methods for growing clinicians’ skills in CBT. It may also be helpful to describe client benefits of delivering CBT with fidelity. One strategy that may broadly address these targets could include education of key clinic stakeholders around the accuracy of BR and CSR and the known benefit of CBT for youth.

Third, concerns were raised about the characteristics of the inner setting, or the clinics in which community mental health is delivered, particularly related to time and the chaotic nature of community mental health delivery (Stewart et al., 2021). Much of these inner setting challenges are likely due to systemic outer setting factors such as poor reimbursement. Community mental health workers are often overburdened and undercompensated, resulting in a highly stressful working environment (Beidas et al., 2016a). While incorporating BR or CSR into regular practice may increase the quality of treatment, addition of new demands could increase stress surrounding burden and potentially lead to burnout. This risk may be particularly poignant for fee-for-service model employees as they typically are not compensated for such activities that fall outside of clinical work (Beidas et al., 2016b).

Fourth, with regard to individuals involved, a number of factors related to clinician autonomy and characteristics came up, with some interesting suggestions that these types of methods are not appropriate for more seasoned clinicians or complex sessions. These themes are corroborated by the literature on implementation of evidence-based practices (McHugh & Barlow, 2010), and specifically CBT. Participants expressed that these methods do not adequately capture the complexity of what is needed in community mental health. While it is understandable that one evidence-based practice like CBT may not be sufficient to meet the complex needs of community mental health care delivery, we argue that it is imperative from an ethics perspective to ensure that we are delivering the highest quality care in these settings, which currently includes CBT approaches.

In this work, we endeavored to bring together an established causal theory from social psychology, the TPB, and an established framework from implementation science, the CFIR, consistent with recent calls to grow our understanding of how theories and frameworks operate rather than reifying existing approaches (Kislov et al., 2019). Interestingly, there was strong convergence between the two approaches, with the TPB guiding our methods for eliciting beliefs about using each method, and the CFIR offering an organizing structure that allowed for more meaningful insights and richer contextual understanding to inform future implementation strategy selection (Waltz et al., 2019). Aligning terminology and approaches was not without challenges, most notable of which was the orientation of CFIR findings within TPB results to avoid redundancy. Given the conceptual convergence in coding results, decisions about which context-rich CFIR findings presented a novel, necessary context and which findings were sufficiently represented in reported TPB results presented a particularly complex challenge that could only be navigated through multiple iterations of integration and critical feedback from investigators with varying expertise. Bringing together these approaches required collaboration between scientists from different disciplines (Falk–Krzesinski et al., 2010), and allows for a merging of approaches that advance our understanding.

Limitations of the current work include the following. First, with interviews and surveys, there is potential for social desirability bias. This appears to have been limited given that clinicians reported both negative and positive beliefs about each fidelity measurement method. Second, supervisors did not have direct experience with the methods that they rated, which may have influenced their response style. Third, this work is hypothetical given that these methods are not currently mandated by systems. Finally, as we did not examine actual supervisor and clinician use of these methods, we can only speak directly to factors informing intention (Pathways A and B, Figure 1). Future research should examine the potential moderation effect of contextual factors on the intention-behavior gap (Pathway C, Figure 1).

In conclusion, if these methods are to achieve their promise, they will need to be implemented in community mental health settings. This work elucidates key targets and implementation strategies that will need to be deployed to take these methods to scale in community mental health, and illustrates a process that can be used more broadly in implementation science merging the TPB and CFIR approaches.

Variable	Supervisor	Clinician
Female gender	10	58.8	28	87.5	27	79.4
Hispanic/Latinx	3	17.6	4	12.5	2	5.9
Racial identity^a
White/Caucasian	14	82.4	22	68.8	29	85.3
Asian or Pacific Islander	0	0	3	9.4	1	2.9
Black or African American	2	11.8	5	15.6	4	11.8
Other	1	5.9	1	3.1	0	0
Age (M, SD)	40.18 (10.48)	–	36.35 (12.62)	–	37.71 (12.43)	–
Education
Bachelor’s degree	–	–	1	3.1	1	2.9
Master’s degree	14	82.4	29	90.6	33	97.1
Doctorate	2	11.8	2	6.3	1	2.9
Type of employee
Salaried	–	–	17	53.1	17	50
Fee-for-service	–	–	13	40.6	17	50
	M	(SD)	M	(SD)	M	(SD)
Years of experience
In human services	15.73	11.39	9.28	10.95	10.24	9.99
As a supervisor	6.22	3.92	–	–	–	–
Hours of supervision per week
Individual supervision	–	–	0.83	0.48	0.88	0.50
Group supervision	–	–	0.81	0.65	0.74	0.58

		BR		CSR	t(14)	p
Supervisors
Attitudes	15	6.46	1.87	–	6.30	1.91	−0.320	.744
Self-efficacy	15	4.17	0.65	–	4.23	0.68	0.695	.479
Norms	15	3.30	0.77	–	3.77	0.78	2.168	.057
Intention	14	5.07	1.49	–	5.43	0.65	0.751	.468
							t(64)
Clinicians
Attitudes	32	6.29	2.34	34	6.60	2.66	0.512	.618
Self-efficacy	32	3.86	0.61	34	4.01	0.53	1.105	.291
Norms	32	3.70	0.67	34	3.76	0.80	0.338	.748
Intention	32	5.59	1.34	34	5.74	1.42	0.416	.688

	Attitudes	Self-efficacy	Norms	Intention
Attitudes	–	0.099	0.375*	0.465*
Self-efficacy	0.099	–	0.135	0.033
Norms	0.375*	0.135	–	0.429*
Intention	0.465*	0.033	0.429*	–

Variable	B	SE	B	p
Attitudes	0.196	0.064	0.355*	.036
Self-efficacy	−0.112	0.263	−0.046	.706
Norms	0.563	0.218	0.301*	.031
Condition	−0.062	0.297	−0.023	.843

CFIR domains	TPB constructs
Inner setting Either fidelity method	Supervision time may become less helpful than it is now.^a	n/a	n/a	There is already a shortage of time to complete important tasks in supervision (e.g. administrative tasks, clinical skill building, etc.)^aFor client crisis sessions, fidelity measurement may not be perceived as a good use of time.^bFrequent disruptions (walk-in clients) may complicate consistent execution of a method.^bClinicians are already overburdened by current demands and may not have the capacity to incorporate additional supervision activities.^b
BR specific	n/a	n/a	n/a	n/a
CSR specific	n/a	n/a	n/a	Additional effort is required to ensure that session notes are prepared for supervisor to review in supervision.^b
Intervention characteristicsEither fidelitymethod	Measure may not give an accurate portrayal of what transpired in session due to memory lapse, inability to communicate complexity of the session.^a	Fidelity method may make clinicians more aware of when they use CBT in session^b	n/a	n/a
BR specific	Roleplaying with a supervisor can be awkward^b	Additional opportunity to reflect on what happened in session may be a benefit. ^b	Clinicians felt clients may disapprove if they knew BR was implemented.^b	n/a
CSR specific	Perceived “interrogative” nature of CSR format may cause clinicians discomfort. (5XY004)^a	CSR may make clinicians more mindful of why CBT is useful.^b	Clinicians identified that other clinicians may dislike CSR due to tension, “interrogative” nature. (5XY004)^b	n/a
Individuals InvolvedEither fidelitymethod	Measures may be overly structured, causing supervision to feel “rote” or “robotic” (4SN004, 4QU004)^aAssessment may detract from modeling and mentorship currently taking place in supervision^aPressure on clinicians to perform or a sense of constant critical evaluation, resulting in anxiety and detriment to supervisor rapport.^cPerceived infringement on clinician autonomy in session^a	Measure may benefit supervision time by lending structure^aIncorporating a method may reinforce clinician knowledge about how to use CBT techniques (or help clinicians learn CBT techniques)^aOpportunity for rich feedback^a	Clinic leadership would approve of implementing a fidelity method as it would promote EBP uptake^a	Using a fidelity measurement may not be appropriate for all clinicians:Seasoned clinicians may benefit less than newer clinicians who are still familiarizing themselves with CBT. However, newer clinicians may benefit less than seasoned clinicians as they need supervision time to develop general clinical skills, learn proper documentation procedures, etc. ^aMay not be appropriate for traumatized or highly complex client sessions as clinician may not be able to give sufficient context for fidelity method to be adequately executed.^a
BR specific	BR may not give an accurate portrayal of what transpired in session due to supervisor inability to adequately act as client.^a	Value role play as a training tool despite discomfort^a	n/a	n/a
CSR specific	Memory lapse may lead to inaccuracy.^a	Ability to use session note mitigates risk of memory lapse.^a	n/a	n/a
Outer SettingEither fidelitymethod	n/a	n/a	n/a	Fee-for-service clinicians may be unable to execute this kind of task given that they are not paid for their time outside of session and their schedules may be unwieldy.^b
BR specific	n/a	n/a	n/a	n/a
CSR specific	n/a	n/a	n/a	n/a

Footnotes

Acknowledgments

The authors are grateful for the support that the Department of Behavioral Health and Intellectual Disability Services has provided us to conduct this work within their system, for the Evidence Based Practice and Innovation (EPIC) group, and for the partnership provided to us by participating agencies, clinicians, and clients. The authors thank Kelly Zentgraf, MS, Courtney Gregor, MsED, and Mary Phan, BA, for their roles as research coordinators on this project over the study period, as well as Steven Marcus, PhD, and Eliza Maceal, M.Sc. for their assistance with statistical analyses.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Drs Sonja Schoenwald and Rinad Beidas both serve as co-authors on this paper, and have editorial affiliations with the Journal. Dr. Schoenwald is Co-Founding Editor in Chief of Implementation Research and Practice. Dr. Beidas was previously an Associate Editor; all decisions on this manuscript were made by another editor. Dr. Beidas is principal at Implementation Science & Practice, LLC. She receives royalties from Oxford University Press, consulting fees from United Behavioral Health and OptumLabs, and serves on the advisory boards for Optum Behavioral Health, AIM Youth Mental Health Foundation, and the Klingenstein Third Generation Foundation outside of the submitted work.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institute of Mental Health, (grant number R01 MH108551).

ORCID iDs

Carlin Hoffacker

Sonja K. Schoenwald

Shannon Dorsey

Trial Registration

This study was registered with ClinicalTrials.gov (Identifier NCT02820623).

References

Becker-Haimes

E. M.

Marcus

S. C.

Klein

Schoenwald

S. K.

Dorsey

Mandell

D.S.

McLeod

B. D.

Fugo

P. B.

Beidas

R. S

. (2020, December 15-17). A randomized trial to identify accurate and cost-effective fidelity measurement methods for cognitive-behavioral therapy: Project FACTS preliminary findings. [Conference Presentation], 13th Annual Conference on the Science of Dissemination and Implementation in Health.

Becker-Haimes

E. M.

Marcus

S. C.

Klein

M. R.

Schoenwald

S. K.

Fugo

P. B.

McLeod

B. D.

Dorsey

Williams

N. J.

Mandell

D. S.

Beidas

R. S

. (2022). A randomized trial to identify accurate measurement methods for adherence to cognitive-behavioral therapy. Behavior Therapy, Advance online publication. https://doi.org/10.1016/j.beth.2022.06.001

Austin

P. C.

J. V

. (2004). Bootstrap methods for developing predictive models. The American Statistician, 58(2), 131–137. https://doi.org/10.1198/0003130043277

Beidas

R. S.

Stewart

R. E.

Adams

D. R.

Fernandez

Lustbader

Powell

B. J.

Aarons

G. A.

Hoagwood

K. E.

Evans

A. C.

Hurford

M. O.

Rubin

Hadley

Mandell

D. S.

Barg

F. K

. (2016a). A multi-level examination of stakeholder perspectives of implementation of evidence-based practices in a large urban publicly-funded mental health system. Administration And Policy In Mental Health, 43(6), 893–908. https://doi.org/10.1007/s10488-015-0705-2

Beidas

R. S.

Cross

Dorsey

. (2014). Show me, don't tell me: behavioral rehearsal as a training and analogue fidelity tool. Cognitive and Behavioral Practice, 21(1), 1–11. https://doi.org/10.1016/j.cbpra.2013.04.002

Beidas

R. S.

Edmunds

J. M.

Marcus

S. C.

Kendall

P. C

. (2012). Training and consultation to promote implementation of an empirically supported treatment: A randomized trial. Psychiatric Services, 63(7), 660–665. https://doi.org/10.1176/appi.ps.201100401

Beidas

R. S.

Stewart

R. E.

Benjamin Wolk

Adams

D. R.

Marcus

S. C.

Evans

A. C.

Jr Jackson

Neimark

Hurford

M. O.

Erney

Rubin

Hadley

T. R.

Barg

F. K.

Mandell

D. S

. (2016b). Independent contractors in public mental health clinics: implications for use of evidence-based practices. Psychiatric Services (Washington, DC), 67(7), 710–717. https://doi.org/10.1176/appi.ps.201500234

Cialdini

R. B.

Demaine

L. J.

Sagarin

B. J.

Barrett

D. W.

Rhoads

Winter

P. L

. (2006). Managing social norms for persuasive impact. Social Influence, 1(1), 3–15. https://doi.org/10.1080/15534510500181459

Cox

J. R.

Martinez

R. G.

Southam-Gerow

M. A

. (2019). Treatment integrity in psychotherapy research and implications for the delivery of quality mental health services. Journal of Consulting and Clinical Psychology, 87(3), 221. https://doi.org/10.1037/ccp0000370

10.

Cunnington

Hanna

Turnhbull

Kaigas

T. B.

Norman

G. R

. (1997). Defensible assessment of the competency of the practicing physician. Journal of the Association of American Medical Colleges, 72(1), 9–12

11.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C

. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 1–15. https://doi.org/10.1186/1748-5908-4-50

12.

Dorsey

Kerns

S. E.

Lucid

Pullmann

M. D.

Harrison

J. P.

Berliner

Thompson

Deblinger

. (2018). Objective coding of content and techniques in workplace-based supervision of an EBT in public mental health. Implementation Science, 13(1), 19. https://doi.org/10.1186/s13012-017-0708-3

13.

Eccles

M. P.

Grimshaw

J. M.

Johnston

Steen

Pitts

N. B.

Thomas

Glidewell

Maclennan

Bonetti

& & Walker

. (2007). Applying psychological theories to evidence-based clinical practice: identifying factors predictive of managing upper respiratory tract infections without antibiotics. Implementation Science, 2(1), 1–14. https://doi.org/10.1186/1748-5908-1-1

14.

Edmunds

J. M.

Kendall

P. C.

Ringle

V. A.

Read

K. L.

Brodman

D. M.

Pimentel

S. S.

Beidas

R. S

. (2013). An examination of behavioral rehearsal during consultation as a predictor of training outcomes. Administration and Policy in Mental Health and Mental Health Services Research, 40(6), 456–466. https://doi.org/10.1007/s10488-013-0490-8

15.

England

M. J

., Butler

A. S

., & Gonzalez

M. L

. (2015). Psychosocial interventions for mental and substance use disorders: A framework for establishing evidence-based standards. National Academy Press.

16.

Erbe

R.G.

Middlestadt

S.E.

Lohrmann

D.K.

Beckmeyer

J.J

. (2020). A salient belief elicitation examining adolescents’ meditation beliefs using the reasoned action approach. Health Promotion Practice. 21(4), 633–6641. https://doi.org/10.1177/1524839918811803

17.

Falk–Krzesinski

H. J.

Börner

Contractor

Fiore

S. M.

Hall

K. L.

Keyton

J. J.

Spring

Stokols

Trochim

Uzzi

. (2010). Advancing the science of team science. Clinical and Translational Science, 3(5), 263–266. https://doi.org/10.1111/j.1752-8062.2010.00223.x

18.

Fishbein

., & Ajzen

. (2010). Predicting and changing behavior: the reasoned action approach. Psychology Press Taylor & Francis Group LLC. https://doi.org/10.4324/9780203838020

19.

Fishbein

Yzer

. (2003). Using theory to design effective health behavior interventions. Communication Theory, 13(2), 164–183. https://doi.org/10.1111/j.1468-2885.2003.tb00287.x

20.

Fishman

Beidas

Reisinger

Mandell

D.S

. (2018). The utility of measuring intentions to use best practices: A longitudinal study among teachers supporting students with autism. Journal of School Health, 88(5), 388–395. https://doi.org/10.1111/josh.12618

21.

Fishman

Lushin

Mandell

D.S

. (2020). Predicting implementation: comparing validated measures of intention and assessing the role of motivation when designing behavioral interventions. Implementation Science Communications, 1(1), 81. https://doi.org/10.1186/s43058-020-00050-4

22.

Goulet

Jacques

Gagnon

Racette

Sieber

. (2007). Assessment of family physicians’ performance using patient charts: interrater reliability and concordance with chart-stimulated recall interview. Evaluation & the Health Professions, 30(4), 376–392. https://doi.org/10.1177/0163278707307924

23.

Harris

P.A.

Taylor

Minor

B.L.

Elliott

Fernandez

O'Neal

McLeod

Delacqua

., Kirby

., Duda

S. N

., & REDCap Consortium (2019). The REDCap consortium: building an international community of software platform partners. Journal of Biomedical Informatics, 95, 103208. https://doi.org/10.1016/j.jbi.2019.103208

24.

Hornik

Woolf

K. D

. (1999). Using cross-sectional surveys to plan message strategies. Social Marketing Quarterly, 5(2), 34–41. https://doi.org/10.1080/15245004.1999.9961044

25.

Jennett

Affleck

. (1998). Chart audit and chart stimulated recall as methods of needs assessment in continuing professional health education. Journal of Continuing Education in the Health Professions, 18(3), 163–171. https://doi.org/10.1002/chp.1340180306

26.

Kislov

Pope

Martin

G. P.

Wilson

P. M

. (2019). Harnessing the power of theorising in implementation science. Implementation Science, 14(1), 1–8. https://doi.org/10.1186/s13012-019-0957-4

27.

Lyon

A. R.

Bruns

E. J

. (2019). User-centered redesign of evidence-based psychosocial interventions to enhance implementation—hospitable soil or better seeds? JAMA Psychiatry, 76(1), 3–4. https://doi.org/10.1001/jamapsychiatry.2018.3060

28.

Lederer

A. M.

Middlestadt

S. E

. (2014). Beliefs about meditating among university students, faculty, and staff: A theory-based salient belief elicitation. Journal of American College Health, 62(6):360–3609. https://doi.org/10.1080/07448481.2014.907296

29.

McHugh

R. K.

Barlow

D. H

. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65(2), 73–84. https://doi.org/10.1037/a0018121

30.

McLeod

B. D.

Southam–Gerow

M. A

., Tully

C. B

., Rodríguez

., & Smith

M. M

. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20(1), 14–32. https://doi.org/10.1111/cpsp.12020

31.

Metz

. (2015). Implementation brief: the potential of co-creation in implementation science. National Implementation Research Network. Retrieved from https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-Metz-ImplementationBreif-CoCreation.pdf

32.

Miller

P. A.

Nayer

Eva

K. W

. (2010). Psychometric properties of a peer-assessment program to assess continuing competence in physical therapy. Physical Therapy, 90(7), 1026–1038. https://doi.org/10.2522/ptj.20080137

33.

Middlestadt

S. E

., Lederer

A. M

., Smith

N. K

., Doss

., Hung

C. L

., Stevenson

L. D.

Fly

A. D

. (2013). Determinants of middle-school students asking parents for fruits and vegetables: A theory-based salient belief elicitation. Public Health Nutrition, 16(11):1971–8. Epub 2012 Oct 22. PMID: 23089302. https://doi.org/10.1017/S1368980012004557

34.

Middlestadt

S. E

., Stevenson

L. D

., Hung

C. L

., Roditis

M. L

., Fly

A. D

., & Sheats

J. L

. (2011). Beliefs underlying the decision to eat breakfast: the role of theory-based behavioral analysis in the development of policy, communication and educational interventions for healthy eating. Food Studies, 1(4), 45–54. PMID: 24089658

35.

Perepletchikova

Treat

T.A.

Kazdin

A.E

. (2007). Treatment integrity in psychotherapy research: analysis of the studies and examination of the associated factors. Journal of consulting and clinical psychology, 75(6), 829–841. https://doi.org/10.1037/0022-006X.75.6.829

36.

Proctor

E. K

., Silmere

., Raghavan

., Hovmand

., Aarons

G. A

., Bunger

., Griffey

., & Hensley

. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7

37.

Salvatori

., Simonavicius

., Moore

., Rimmer

., & Patterson

. (2008). Meeting the challenge of assessing clinical competence of occupational therapists within a program management environment. The Canadian Journal of Occupational Therapy, 75(1), 51–560. https://doi.org/10.1177/000841740807500109

38.

Sheeran

. (2002). Intention—behavior relations: A conceptual and empirical review. European Review of Social Psychology, 12(1), 1–36. https://doi.org/10.1080/14792772143000003

39.

Stewart

R.E.

Mandell

D.S.

Beidas

R.S

. (2021). Lessons from maslow: prioritizing funding to improve the quality of community mental health and substance use services. Psychiatric Services, 72(10), 1219–1221. https://doi.org/10.1176/appi.ps.202000209.

40.

Sutton

French

D.P.

Hennings

S.J.

Mitchell

Wareham

N.J.

Griffin

Hardeman

Kinmonth

A.L

. (2003). Eliciting salient beliefs in research on the theory of planned behaviour: the effect of question wording. Current Psychology, 22(3), 234–251. https://doi.org/10.1007/s12144-003-1019-1

41.

Waltz

T.J.

Powell

B.J.

Fernández

M.E.

Abadie

Damschroder

L.J

. (2019). Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implementation Science, 14(1), 42. https://doi.org/10.1186/s13012-019-0892-4

42.

Webb

T.L.

Sheeran

. (2006). Does changing behavioral intentions engender behavior change? A meta-analysis of the experimental evidence. Psychological Bulletin, 132(2), 249. https://doi.org/10.1037/0033-2909.132.2.249

43.

Wolk

C.B.

Becker-Haimes

E.M.

Fishman

Affrunti

N.W.

Mandell

D.S.

Creed

T.A

. (2019). Variability in clinician intentions to implement specific cognitive-behavioral therapy components. BMC Psychiatry, 19(1), 406. https://doi.org/10.1186/s12888-019-2394-y

Variable	Supervisor		Clinician
	n = 17		BR n = 32		CSR n = 34
	n	%	n	%	n	%
Female gender	10	58.8	28	87.5	27	79.4
Hispanic/Latinx	3	17.6	4	12.5	2	5.9
Racial identity^a
White/Caucasian	14	82.4	22	68.8	29	85.3
Asian or Pacific Islander	0	0	3	9.4	1	2.9
Black or African American	2	11.8	5	15.6	4	11.8
Other	1	5.9	1	3.1	0	0
Age (M, SD)	40.18 (10.48)	–	36.35 (12.62)	–	37.71 (12.43)	–
Education
Bachelor’s degree	–	–	1	3.1	1	2.9
Master’s degree	14	82.4	29	90.6	33	97.1
Doctorate	2	11.8	2	6.3	1	2.9
Type of employee
Salaried	–	–	17	53.1	17	50
Fee-for-service	–	–	13	40.6	17	50
	M	(SD)	M	(SD)	M	(SD)
Years of experience
In human services	15.73	11.39	9.28	10.95	10.24	9.99
As a supervisor	6.22	3.92	–	–	–	–
Hours of supervision per week
Individual supervision	–	–	0.83	0.48	0.88	0.50
Group supervision	–	–	0.81	0.65	0.74	0.58

CFIR domains	TPB constructs
	Beliefs underlying attitudes		Normative referents	Self-efficacy beliefs
	Dislikes/disadvantages	Likes/advantages	Normative referents	Self-efficacy beliefs
Inner setting Either fidelity method	Supervision time may become less helpful than it is now.^a	n/a	n/a	There is already a shortage of time to complete important tasks in supervision (e.g. administrative tasks, clinical skill building, etc.)^aFor client crisis sessions, fidelity measurement may not be perceived as a good use of time.^bFrequent disruptions (walk-in clients) may complicate consistent execution of a method.^bClinicians are already overburdened by current demands and may not have the capacity to incorporate additional supervision activities.^b
BR specific	n/a	n/a	n/a	n/a
CSR specific	n/a	n/a	n/a	Additional effort is required to ensure that session notes are prepared for supervisor to review in supervision.^b
Intervention characteristicsEither fidelitymethod	Measure may not give an accurate portrayal of what transpired in session due to memory lapse, inability to communicate complexity of the session.^a	Fidelity method may make clinicians more aware of when they use CBT in session^b	n/a	n/a
BR specific	Roleplaying with a supervisor can be awkward^b	Additional opportunity to reflect on what happened in session may be a benefit. ^b	Clinicians felt clients may disapprove if they knew BR was implemented.^b	n/a
CSR specific	Perceived “interrogative” nature of CSR format may cause clinicians discomfort. (5XY004)^a	CSR may make clinicians more mindful of why CBT is useful.^b	Clinicians identified that other clinicians may dislike CSR due to tension, “interrogative” nature. (5XY004)^b	n/a
Individuals InvolvedEither fidelitymethod	Measures may be overly structured, causing supervision to feel “rote” or “robotic” (4SN004, 4QU004)^aAssessment may detract from modeling and mentorship currently taking place in supervision^aPressure on clinicians to perform or a sense of constant critical evaluation, resulting in anxiety and detriment to supervisor rapport.^cPerceived infringement on clinician autonomy in session^a	Measure may benefit supervision time by lending structure^aIncorporating a method may reinforce clinician knowledge about how to use CBT techniques (or help clinicians learn CBT techniques)^aOpportunity for rich feedback^a	Clinic leadership would approve of implementing a fidelity method as it would promote EBP uptake^a	Using a fidelity measurement may not be appropriate for all clinicians:Seasoned clinicians may benefit less than newer clinicians who are still familiarizing themselves with CBT. However, newer clinicians may benefit less than seasoned clinicians as they need supervision time to develop general clinical skills, learn proper documentation procedures, etc. ^aMay not be appropriate for traumatized or highly complex client sessions as clinician may not be able to give sufficient context for fidelity method to be adequately executed.^a
BR specific	BR may not give an accurate portrayal of what transpired in session due to supervisor inability to adequately act as client.^a	Value role play as a training tool despite discomfort^a	n/a	n/a
CSR specific	Memory lapse may lead to inaccuracy.^a	Ability to use session note mitigates risk of memory lapse.^a	n/a	n/a
Outer SettingEither fidelitymethod	n/a	n/a	n/a	Fee-for-service clinicians may be unable to execute this kind of task given that they are not paid for their time outside of session and their schedules may be unwieldy.^b
BR specific	n/a	n/a	n/a	n/a
CSR specific	n/a	n/a	n/a	n/a

Stakeholder intention to engage in fidelity measurement methods in community mental health settings: A mixed methods study

Abstract

Background

Methods

Results

Conclusions

Keywords

Methods

Study Setting and Agency Participation

Procedures and Sampling

Qualitative Interview Procedures

Supervisors (n = 17)

Clinicians (n = 66)

Measures

TPB Constructs

TPB Construct Quantitative Questionnaires

Demographics

Qualitative Interviews to Elicit Beliefs Informing Attitudes, Self-Efficacy, and Norms and Contextual Factors

Data Analysis

Results

TPB Quantitative Survey Findings

Qualitative Belief Elicitation Interview Results

Beliefs Underlying Attitudes

Commonly Reported Dislikes/Disadvantages

Commonly Reported Likes/Advantages

Commonly Reported Self-Efficacy Beliefs

Commonly Reported Normative Referents

Discussion

Footnotes

Acknowledgments

Declaration of conflicting interests

Funding

ORCID iDs

Trial Registration

References