A Glimpse into the “Black Box”: Which Elements of Consultation in an EBP are Associated with Client Symptom Change and Therapist Fidelity?

Abstract

Background: Consultation is an implementation strategy that improves delivery and clinical outcomes for Cognitive Processing Therapy (CPT), an evidence-based practice (EBP) for posttraumatic stress disorder (PTSD). However, little is known about the specific components of consultation that influence the fidelity of treatment delivery or clinical outcomes. Methods: The current study examined whether specific activities performed during CPT consultation meetings were associated with better fidelity to the CPT protocol among 60 newly trained therapists or improved clinical outcomes among 135 clients treated by these therapists. Consultation activities that fall under three broad categories (discussion of the application of CPT to individual cases, review/feedback on fidelity, and technical difficulties) were measured by consultant checklists for each consultation session. Treatment fidelity (adherence to the protocol and competence of delivery) was rated by trained observers for a random sample of therapists’ CPT sessions following consultation. The self-reported PTSD Checklist-IV assessed PTSD symptom change. Results: Multilevel regression analyses indicated that higher therapist consultation attendance predicted a greater decrease in their clients’ PTSD symptoms and that attendance was not associated with observer-rated treatment fidelity. Discussion of the application of specific CPT strategies was the only consultation activity that was significantly associated with greater improvement in PTSD symptoms. Lastly, no consultation activities were significantly associated with treatment fidelity. Conclusions: Our findings suggest that specific consultation strategies such as emphasizing the discussion of the application of specific CPT strategies to individual cases during consultation meetings may be effective in improving the clinical outcomes of CPT.

Keywords

Consultation treatment fidelity cognitive processing therapy PTSD

With the growing emphasis on Evidence-based Practices (EBPs) for Posttraumatic Stress Disorder (PTSD; American Psychological Association [APA], 2018; Schnyder & Cloitre, 2015), public and private mental health organizations are investing substantial time and resources to train their therapists to provide these treatments (Edmunds et al., 2013). In addition to these investments, national and state-level policy directives have been introduced to train therapists to deliver EBPs such as Cognitive Processing Therapy (CPT; Resick et al., 2017) and Prolonged Exposure (PE; Foa et al., 2007) Therapy (Clark, 2011; Eftekhari et al., 2013; McHugh & Barlow, 2010). However, research has demonstrated that initial, didactic training is not sufficient to promote sustained, high-quality treatment delivery. Several studies have shown that some form of follow-up support (i.e., consultation) appears to be necessary after workshops (Herschell et al., 2010; Schoenwald et al., 2004; Sheridan et al., 2009; Webster-Stratton et al., 2014).

Consultation is defined as ongoing, focused interaction with a specialist in an effort to increase competence in the area of the specialist's expertise (Edmunds et al., 2013). Continued support through consultation is believed to facilitate further learning and mastery, and may increase the likelihood of skilled, ongoing delivery of EBPs (Nadeem et al., 2013). This form of follow-up support has been associated with improved treatment fidelity; both adherence (delivery of essential elements of the protocol) and competence (skill of delivery) are higher among therapists who receive consultation (Beidas & Kendall, 2010; Edmunds et al., 2017; Herschell et al., 2010; Sholomskas et al., 2005). Previous research has demonstrated that higher fidelity can lead to improved client outcomes (Farmer et al., 2017; Mandell et al., 2013; Marques et al., 2019; Strunk et al., 2010), and that fidelity may serve as a mediator between implementation of an EBP and client outcomes (Henggeler et al., 1997). However, little research has been conducted to identify whether specific aspects of consultation might lead to higher fidelity or improved client outcomes.

To make training as efficient and successful as possible, an understanding of the strategies that enhance the effectiveness of consultation is essential (Monson et al., 2018; Stirman et al., 2013). Nadeem and colleagues (2013) discussed the importance of “unpacking the black box” of consultation and identified a need for research focused on identifying strategies and mechanisms that drive consultation's impact on implementation outcomes. Their review of the literature distinguished several key aspects of consultation, including continued training, problem solving implementation barriers, case support, and accountability (Nadeem et al., 2013). Furthermore, they suggested that through consultation, therapists are able to share and reflect on their experiences of delivering the treatment, receive feedback, and identify strategies to address challenges that may arise during their therapy sessions. Other researchers have found that factors such as consultation attendance and caseload are associated with fidelity (Edmunds et al., 2017; Jackson et al., 2017). Finally, some research has suggested that clients experience more improvement if they are not the first client that their therapist treats when being trained in a new intervention, which suggests that therapists may be more effective in providing new treatments with practice and consultation (Johnson et al., 2019).

Several studies suggest activities that may contribute to the relationship between aspects of consultation and therapist adherence and competence. Studies have shown that behavioral rehearsal and modeling can increase the use of EBP strategies (Bearman et al., 2013; Beidas et al., 2014; Edmunds et al., 2013). Additionally, the process of instruction, case review, self-evaluation, and feedback can help cultivate skill development in therapists (Brosan et al., 2008; Edmunds et al., 2013; Stirman et al., 2017; Tracey et al., 2014). Following workshop training, guidance on CPT and exposure to case discussion during consultation allow therapists to implement consultants’ suggestions in practice and reflect upon their previous work (Moore et al., 2009), which may ultimately benefit clients. Timely feedback allows therapists to correct and deepen their delivery and understanding of the treatment while receiving simultaneous support from supervisors or consultants. Whether case review and feedback need to include some form of observation has not been empirically determined. Researchers have considered observation and feedback to be the gold standard for fidelity monitoring (Beidas & Kendall, 2010; Monson et al., 2018). However, there is some evidence that observation may not be necessary in all circumstances. For example, Ward and colleagues (2013) determined that, for at least some aspects of EBP delivery, therapists may be able to self-report accurately, suggesting that observation may not be a necessary component of consultation.

The parent study of this manuscript compared the standard form of CPT consultation without any observation and immediate feedback on session recordings to consultation that included review of audio segments and a no consultation comparison group (Monson et al., 2018). Therapists in the standard consultation condition experienced significantly greater changes in PTSD symptoms than those who did not receive any consultation, with no significant differences detected between standard consultation and consultation including review of session audio (Monson et al., 2018). In this form of consultation, which has been used to train thousands of therapists over the past two decades (Chard et al., 2012), therapists attend a group consultation meeting with 4–6 therapists. Each therapist discusses their most recent sessions with their training cases, describing their efforts to provide the prescribed elements of each session, the “stuck points” or automatic thoughts that they focused on, and brings up any challenges they encountered. Agendas are set to prioritize cases where challenging or emergent issues arise, but time is allocated for each therapist to have time to receive consultation on their cases. The consultant asks questions about the case and the session to assess whether key activities in the session occurred, if the therapist doesn’t describe them, and provides guidance on how to address the barriers that were encountered while still adhering to the protocol. Additionally, the consultant provides feedback and guidance about key stuck points to address, what to cover in the next session and how to use CPT strategies based on the working conceptualization of the case. They elicit questions, comments, and suggestions from other group members as well.

The authors speculated that the findings regarding the effectiveness of this approach may be due to the highly structured nature of the CPT protocol, the manual for which contains session agendas and detailed session-by-session descriptions (Monson et al., 2018). With protocols of this nature, it may be important in consultation to focus on conceptualizing and understanding how to apply CPT strategies to each individual case (e.g., what beliefs and cognitions are maintaining the PTSD symptoms for each individual) rather than providing feedback on short segments of audio. Alternatively, technical difficulties in playing audio may have led to a less efficient use of time in the condition that included audio review of sessions.

Collectively, research to date has improved our understanding of the potential active ingredients of consultation, but understanding of the relationship between clinical (e.g., symptom change) or implementation (e.g., fidelity) outcomes and the nature, dose, and exposure to specific consultation activities remains incomplete. A close examination of both the specific elements of consultation and the amount of exposure to each element can elucidate why and how consultation leads to improvements in key clinical and implementation outcomes, and can inform the development of more efficient and effective consultation strategies. Using data from the parent study, comparing post-workshop support and consultation strategies (Monson et al., 2018), we examined consultation sessions that occurred over the 6 months following a workshop on CPT.

To capture real-world applications of CPT and understand what occurs in consultation as usual, we attempted to unpack the “black box” of consultation as routinely provided in national training and consultation programs. Based on the findings from the parent study, our primary hypotheses were that: 1) the dose or exposure to specific consultation activities would be more strongly associated with fidelity and client symptom change than consultation attendance alone, 1a) the cumulative amount of feedback on fidelity to which a therapist was exposed during consultation would be associated with improvements in therapist fidelity (adherence and competence), but that 1b) cumulative exposure to specific elements of consultation (discussion of the application of CPT to individual cases) would be associated with greater changes in PTSD symptoms, and 2) distracting factors such as technical difficulties would be associated with less improvement in fidelity and client symptom change. We also sought to replicate and extend findings from previous research suggesting that consultation attendance was associated with better outcomes (Jackson et al., 2017) and that therapists’ later clients would experience greater improvement (Johnson et al., 2019).

Method

Participants

Therapists

Therapist participants (n = 134) were recruited from Veterans Affairs Canada Operational Stress Injury clinics, Canadian Forces mental health services, and the broader Canadian community. All participants provided voluntary informed consent after a clear description of the Research Ethics Board-approved study procedures. Mental health therapists were eligible to participate in the study if they: attended a standardized CPT workshop, were licensed mental health therapists with psychotherapy in their scope of practice, were currently providing psychotherapy to individuals who were therapist-assessed to have PTSD, consented to be randomized to one of the study conditions, and were willing to provide audio recordings of therapy sessions and measures of PTSD symptoms from consenting clients. In this study, we included the subset of therapists (n = 60) who were randomized into a condition that included consultation and provided session and client materials. Therapist participant characteristics can be found in Table 1. Full details from the parent study can be found in Monson et al., 2018.

Table 1.

Therapist demographic and background characteristics (n = 60).

Variable	Consultation Including Audio Review (n = 30)	Standard (n = 30)
Age (M, SD)	47.21, 10.19	48.86, 10.19
Gender
Male	30%	21%
Female	70%	79%
Education
PhD/PsyD	28%	35%
Master's (counseling, social work)	56%	35%
Bachelor's	7%	11%
MD	7%	5%
Years of Practice
5 or fewer	7%	7%
6–10	17%	29%
11–20	50%	29%
20 or more	27%	36%
CBT experience
No prior training or experience	0%	4%
Basic knowledge, no formal training	7%	11%
Coursework or training, no supervision	43%	36%
Coursework or training with supervision	50%	50%
Caseload Size (M, SD)	22.00, 13.65	28.96, 15.24
PTSD Caseload (% of clients)
<25%	47%	57%
25–50%	30%	21%
50–75%	7%	7%
>75%	17%	14%
Practice Setting
Provincial Health Agency	27%	21%
Private Practice	30%	29%
Operational Stress Injury Clinic	20%	18%
Canadian Forces or other Federal Health Service	18%	17%
Other	3%	14%

Note. Sample characteristics are provided fir the therapists who provided client data. Means (M) and standard deviations (SD) reported for continuous variables. Sample sizes (n for total, I per condition; and n for a portion of the sample) and corresponding percentages (%) are reported for categorical variables.

Clients

Participating therapists facilitated recruitment by identifying eligible new or existing clients who provided informed consent for all study procedures. To participate, clients were required to be diagnosed with PTSD by their therapist based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV; American Psychiatric Association, 2000) criteria. Client participants (n = 135) also consented to participate in CPT and have their sessions audio-recorded and reviewed by independent fidelity raters, as well as potentially listened to by other therapists during consultation. Clients were permitted to continue other psychotherapeutic interventions if they were not trauma-focused. Consistent with research-informed standards, clients were ineligible if they had current uncontrolled psychosis or bipolar disorder, substance dependence (abuse was permitted), imminent suicidality or homicidality, or significant cognitive impairment (mild to moderate traumatic brain injury was permitted). Further details on the client participants’ characteristics can be found in Table 2.

Table 2.

Client demographics and baseline clinical characteristics (n = 135).

Variable	Consultation Including Audio Review (n = 74), M (SD)/n(%)	Standard (n = 61), M (SD)/n(%)
Gender
Male	37 (55.2%)	30 (51.7%)
Female	30 (44.8%)	28 (48.3%)
Race/Ethnicity
White	62 (92.5%)	52 (89.7%)
Black	2 (3.0%)	0 (0.0%)
Hispanic	0 (0.0%)	1 (1.7%)
Asian/Pacific Islander	1 (1.5%)	1 (1.7%)
Indigenous	1 (1.5%)	1 (1.7%)
Other	1 (1.5%)	3 (5.2%)
Relationship Status
Committed Relationship	44 (65.7%)	32 (57.1%)
Single/Widowed/Divorced	23 (34.3%)	24 (42.9%)
Military Status
Veteran	12 (18.5%)	13 (22.8%)
Active Duty	18 (27.7%)	10 (17.5%)
Non-Military	35 (53.8%)	34 (59.6%)
Age (Years)	41.94 (11.85)	38.00 (11.27)
Education (Years)	13.28 (2.33)	12.52 (2.03)
Pre-Treatment PTSD (PCL)	61.68 (10.89)	61.04 (11.49)
# of Sessions Attended	9.09 (4.40)	9.61 (4.01)
# of Comorbid Axis-I Diagnoses	1.23 (1.11)	1.11 (1.01)
Axis-II Comorbidity
Absent	49 (81.7%)	34 (73.9%)
Present	11 (18.3%)	12 (26.1%)

Note. Means (M) and standard deviations (SD) reported for continuous variables. Sample sizes (n for total, I per condition) and corresponding percentages (%) are reported for categorical variables.

Procedures

Therapists were invited to participate prior to, or at the standardized CPT workshops. After completing the workshop, in the parent study (Monson et al., 2018) therapists who provided informed consent were randomized to one of the three post-workshop consultation conditions: Standard Consultation and Standard Consultation + Audio Review. The third condition was No Consultation and was not analyzed in this manuscript. Full participation in the study required therapists to provide audio recordings of all CPT sessions with at least two clients to be used during consultation and for fidelity ratings, and to collect client symptom and functioning measures. To incentivize participation, therapist participants who uploaded the required number of session recordings and client symptom measures were eligible to become “Quality-rated” CPT Providers and placed on a roster of CPT providers (see www.cptforptsd.com) if they met a minimum threshold for competence. After the 6-month post-workshop phase was completed, all therapists received written feedback on their fidelity in two CPT sessions. Written feedback on their fidelity throughout the consultation period is not part of routine consultation and would not be feasible in routine care. Thus, it was delayed until the end of the study to provide a benefit for therapists in all conditions without introducing a confound. Therapists also received compensation for completing study assessments.

Post-workshop consultation conditions

After randomization in the parent study, therapists joined consultation groups within their assigned conditions based on the meeting times that were most feasible for their schedules. On average, therapists enrolled 2 new or established clients into consultation. Cases were enrolled as therapists were able to identify clients who were interested in enrollment in the study. Welch's t-test indicated that there were no significant differences in the number of clients per therapist between the two consultation conditions, t(10.95) = 1.05, p = .315.

Standard consultation

Therapists in the Standard condition attended a standard 2-day CPT workshop, received the CPT manual and related materials, and had access to resources available through the free CPT-web online training (https://cpt.musc.edu/index). Participants in the Standard consultation condition received 6 months of weekly 1-h group web- and telephone-based consultation with a CPT expert consultant. Four to six therapists were on each call. As described in the introduction, consultation calls included discussions of how to apply the CPT protocol to specific cases, challenging cases, treatment obstacles, and specific issues raised by participants within each group.

Standard consultation with audio review

Therapists in the Standard + Audio condition received all of the elements of the Standard consultation condition, in addition to receiving feedback from other group members as well as from the expert on a presentation of segments of their audio recordings (typically 5–10 min) from a recent session. Participants were asked to play segments that reflected their use of a key skill from that session or to play a segment in which they struggled to deliver CPT. In later sessions, consultants were encouraged to ask them to play random segments of their session if therapists did not identify specific parts that were challenging for them. Therapists typically presented a recording every other call, and those who did not present session recordings in a given week had the opportunity to provide a brief check-in about the progress of their current cases and receive input and feedback as in standard consultation. Four to six therapists attended each call. While this condition included all elements of standard consultation, because a portion of the consultation session was spent listening to the audio of CPT cases, and thus there may have been less time for the standard consultation elements. Audio review may have allowed for more specific feedback and guidance and for greater learning by hearing other therapists’ sessions.

In both conditions, therapists received written feedback on fidelity for randomly selected sessions after the consultation phase was complete.

Consultants

Expert CPT consultants (n = 5; four PhD level psychologists and one master's level social worker), who had at least five years of experience providing CPT consultation served as consultants in both consultation conditions to control for possible consultant-related effects. They were trained by the principal investigators (PIs) in study consultation procedures for each condition by study team members prior to beginning consultation calls. The prescribed and proscribed activities for each consultation condition are described in manuals (Chard, 2009; Stirman & Monson, 2011) and the PIs reviewed self-reported consultation checklists and recordings of consultant calls monthly to ensure that consultant fidelity to condition (80% or more of the prescribed elements) was maintained. Consultants met every four to six weeks during the first wave of consultation, and subsequently received support and feedback as needed from the principal investigators throughout the study. Consultants completed a checklist of consultation activities after each session, which was also completed for a subsample of sessions by independent observers who reviewed randomly selected recordings of consultation sessions.

Measures

Consultation activity checklist

Upon completion of each consultation session, consultants filled out a checklist indicating activities that were completed during the consultation session as well as which clinicians were in attendance. A subset (15%) of these facilitator checklists were then rated by observers to assess agreement between observers and consultants (scale: 0–1, observer-therapist agreement k = .97; the two observers’ agreement k = .93). Consultation activities from the checklist comprised three broad categories: (1) discussion of the application of CPT to individual cases, comprising the following activities: discussing cases without audio review, case selection/appropriateness for CPT, discussing the use of specific CPT strategies for individual cases, guidance on overcoming clinical challenges, and prioritizing case discussion during agenda setting), (2) reviewing and providing feedback on fidelity, comprising: discussion of how to adhere to the protocol, playing audio sessions, receiving feedback on their fidelity based on the segment, and reviewing copies of work material such as CPT worksheets or stuck point logs), and (3) distractions and technical difficulties, comprising off-topic discussion and technical difficulties. Off-topic discussion was very minimal, so this category essentially reflects technical challenges. For scoring, to determine the amount of therapist exposure to each element, a cumulative score for each category was created for each individual, each consultation week, based on the sum of the activities in each broad category that occurred for each consultation meeting at which the therapist was present. Thus, at the individual therapist level, this variable provided a cumulative count of how many times therapists were exposed to each element over the course of consultation. The mean accumulation scores of consultation activities across the consultation phase are presented in Table 3.

Table 3.

Accumulation of consultation activities over consultation sessions.

Consultation Week	Consultation Condition	Discussion of the Application of CPT to Individual Cases		Reviewing/Feed-back on Fidelity		Technical Difficulties
Consultation Week	Consultation Condition	M	SD	M	SD	M	SD
2	Standard	3.38	2.46	0	0	0.30	0.46
2	Including Audio Review	3.73	2.90	1.73	1.80	0.45	0.74
14	Standard	28.56	11.82	0	0	1.19	1.04
14	Including Audio Review	27.07	10.22	16.43	6.93	2.75	2.65
26	Standard	44.44	17.13	0	0	2.15	1.70
26	Including Audio Review	48.09	10.33	30.73	9.01	3.82	3.00

Note. M = mean, SD = standard deviation.

Therapist fidelity

Therapist CPT fidelity was assessed using a modified version of the CPT fidelity measure, the Therapist Adherence and Competence Scale (TAC; Resick et al., 2008). The TAC measures both adherence to specific unique and essential CPT elements (scale: 0–3, ICC = 0.87), and competence in delivering these interventions (scale: 0–6; ICC = 0.79). A randomly selected subset of CPT sessions from early, mid- and late protocol (290 sessions, 19% of the total number of session recordings and approximately 3 per therapist who provided recordings) were coded by trained observers who met regularly to discuss ratings and facilitate rater agreement. Mean scores for adherence and competence were calculated for analyses.

Trauma assessment

PTSD diagnosis was determined by the client's treatment providers. For study purposes, we also administered the Posttraumatic Stress Disorder Checklist, DSM-IV version (PCL-IV; Weathers et al., 1993). The PCL-IV is a 17-item, self-report questionnaire of the severity of distress related to PTSD symptoms. It is highly correlated with Structured Clinician-Administered PTSD Assessments, such as the DSM Structured Clinical Interview (Cronbach's alpha = 0.94). Therapists asked clients to complete a PCL-IV pre-treatment (to determine eligibility), before the first session, and prior to every CPT session.

Analytic plan

Multilevel regression analyses (i.e., mixed-effects modeling) were conducted to: (1) analyze the relationship between the accumulation of consultation attendance and activities and observer-rated fidelity (adherence and competence) to the CPT protocol at subsequent therapy sessions, and (2) test whether the accumulation of these activities predicted lower levels of PTSD in subsequent sessions of the therapists’ clients. A multilevel modeling approach allows for detection of and accounting for patterns of fidelity over time, since previous work has suggested that fidelity may decrease over the course of the protocol (Boswell et al., 2013), and also accounts for the degree to which variation in fidelity and clinical change can be attributed to the therapist and/or client level. Moreover, multilevel models permit the analysis of repeated measures nested within clients and therapists and all models were run using the lme4 (Bates et al., 2015) and lmerTest (Kuznetsova et al., 2017) packages in R Statistical Software (R Core Team, 2018). The lme4 package defaults to maximum likelihood to account for missing data.

Adherence and competence

To test our hypotheses regarding the relationship between the consultation attendance and activities and the therapist's adherence and competence, we evaluated two (one for adherence and one for competence) 2-level models for each of the four predictors (attendance, discussion of the application of CPT to individual cases, reviewing for fidelity, and technical difficulties). Our dependent variable, repeated, observer-rated fidelity in randomly selected sessions (adherence and competence; Level-1) was nested within clinicians (Level-2). Session number was also included as a covariate at the time level based on findings by Boswell and colleagues (2013), which suggested that adherence and competence to a CBT protocol decrease over client sessions; we examined our data to determine whether a similar pattern existed, with a plan to include session number as a covariate at the time level if that pattern was identified. Because we found that therapist degree level was associated with baseline competence in another study (Sijercic et al., 2020) that analyzed data from the same parent study, we included it as a covariate in our competence analyses. Before running a predictor model, we ran an unconditional model, excluding the predictor variable and covariates, to examine the variance across each level. Partial correlation coefficients (pr) were reported as measures of effect sizes of our predictors (Kirk, 1996).

Client PTSD symptoms (PCL)

To test our hypotheses regarding the association between consultation attendance and activities and the client's PTSD symptom change, we conducted a 2-level model for each predictor (attendance, discussion of the application of CPT to individual cases, reviewing for fidelity, and technical difficulties). Our dependent variable, the repeated, change score of client PTSD symptoms from baseline (PCL; Level-1), was nested within clients (Level-2). Accumulation of consultation attendance and consultation activities, in separate models, were Level-1 time-varying variables, nested within clients (Level-2). To examine whether therapists’ later CPT clients experienced better outcomes, a separate 2-level model included a predictor variable indicating whether the client was the therapist's first or a later client, as determined by the start date of therapy for each client. The start date was defined as the date the client began CPT. Before conducting a predictor model, we first evaluated an unconditional model, excluding the predictor variable and covariates, to examine the variance across each level. Partial correlation coefficients (pr) were reported as measures of effect sizes of our predictors (Kirk, 1996).

Results

Demographics

Therapists

There were more Female therapists than Male therapists between the Standard Consultation Plus Audio group (Male 30%; Female 70%) and Standard Consultation group (Male 21%); Female 79%). The majority of therapists in both groups had a Masters degree (counseling or social work) or higher, the Standard Consultation Plus Audio group (Masters 56%; PhD/PsyD 28%); the Standard Consultation Group (Masters 35%; PhD/PsyD 35%). Additionally, therapists in both groups had similar caseload sizes, the Standard Consultation Plus Audio group (M = 22; SD = 13.65) and Standard Consultation group (M = 28.96; SD = 15.24). Of those caseloads, the percent of clients that were being treated for PTSD varied, but the majority between the two groups indicated that less than 25% of their caseload consisted of clients being treated for PTSD. On average, therapists treated 2 clients in the study. Further details on the client participants’ characteristics can be found in Table 1.

Clients

Males and Females were nearly evenly distributed between the Standard Consultation Plus Audio group (Male 55.2%; Female 44.8%) and Standard Consultation group (Male 51.7%; Female 48.3%) (n = 135). There were no significant differences between the two groups (Monson et al., 2018). The majority of clients in both groups were white (92.5% in the Standard Consultation Plus Audio group (92.5%); 89.7% inthe Standard Consultation Group. Military status varied among the two groups, the Standard Consultation Plus Audio group (Veterans 18.5%, Active Duty 27.7%, and Non-Military-affiliated 53.8%); Standard Consultation Group (Veterans 22.8%, Active Duty 17.5%, and Non-Military-affiliated 59.6%). The average pre-treatment PTSD PCL score was similar between the two groups as well, the Standard Consultation Plus Audio group (M = 61.68); the Standard Consultation Group (M = 61.04). Lastly, the average number of CPT sessions attended by clients in both groups was about nine. Further details on the client participants’ characteristics can be found in Table 2.

Adherence and competence

The variance decomposition of the unconditional model revealed that almost one hundred percent of the variances of adherence and competence were attributed to within-therapist (Level-1) and almost none to between-therapists (Level-2).

We first examined the association between the therapist’ consultation attendance and their observer-rated adherence and competence to the CPT protocol. Other studies (e.g., Boswell et al., 2013) have shown that fidelity decreased over the course of the protocol. We observed a similar pattern of decrease in the current study, although it did not reach statistical significance, Adherence: b = −0.02, t(49.45) = −1.34, p = .09, pr = −.13; Competence: b = −0.01, t(177) = −0.54, p = .59, pr = −.04. We therefore controlled for session number in our model to evaluate the impact of consultation attendance and activities on fidelity while keeping session number constant. We found no significant main effects of consultation attendance for adherence, pr = −.09, and competence, pr = −.10. Additionally, therapist degree level, which was added as a covariate due to this therapist characteristic's association with early therapist competence (Sijercic et al., 2020), was not significantly associated with changes in competence in any of the analyses, all ps > .06, all prs < .14. We then examined whether three consultation activity categories predict the therapist's observer-rated fidelity. Again, we found no significant main effects of discussion of the application of CPT to individual case activities, pr = −.09; pr = −.13, reviewing fidelity, pr = −.09; pr = −.10, or technical difficulties, pr = −.02; pr = −.08, for adherence and competence, respectively. The results indicate that the accumulations of consultation attendance and categories of consultation activities leading up to the therapy session are not associated with therapist adherence and competence. The fixed effects of our models for therapist fidelity are presented in Table 5.

Table 4.

Descriptive statistics for number of days since baseline and PCL-IV (N = 134).

Session	n	Days Since Baseline		PCL-IV
Session	n	M	SD	M	SD
0	87	0.00	0.00	60.78	11.18
1	119	12.97	17.37	59.64	12.24
2	119	22.08	18.94	58.47	13.09
3	108	35.27	42.60	57.83	12.53
4	100	44.77	44.40	57.45	14.14
5	93	53.30	44.52	55.16	14.57
6	93	62.80	46.83	53.05	15.58
7	90	74.54	50.33	50.82	14.46
8	90	83.26	52.07	49.45	15.60
9	86	96.37	55.01	47.67	16.00
10	85	109.80	62.05	46.74	14.77
11	80	117.79	62.60	46.44	14.63
12	80	118.05	41.50	43.21	15.79

Note. PCL-IV = Posttraumatic Stress Disorder Checklist, M = mean, SD = standard deviation.

Table 5.

Fixed effects for three level model of consultation attendance and categories of activities predicting adherence and competence.

Fixed effects	Adherence					Competence
Fixed effects	Estimate	SE	df	t	p	Estimate	SE	df	t	p
Attendance
Intercept	2.48	0.08	165.86	29.54	<.001	2.72	0.22	176.00	12.29	<.001
Attendance	−0.01	0.01	134.48	−1.10	.275	−0.02	0.02	176.00	−1.36	.176
Session Number	−0.01	0.01	180.66	−0.97	.336	0.00	0.02	176.00	0.19	.850
Discussion of the application of CPT to individual cases
Intercept	2.48	0.08	160.21	29.95	<.001	2.73	0.22	176.00	12.50	<.001
Discuss CPT Strategies	0.00	0.00	123.40	−1.17	.246	−0.01	0.00	176.00	−1.78	.077
Session Number	−0.01	0.01	183.46	−0.95	.341	0.01	0.02	176.00	0.35	.726
Review fidelity
Intercept	2.47	0.08	161.02	30.61	<.001	2.67	0.12	176.00	12.41	<.001
Review fidelity	0.00	0.00	96.64	−0.95	.347	−0.01	0.01	176.00	−1.39	.165
Session Number	−0.02	0.01	184.89	−1.51	.133	−0.01	0.02	176.00	−0.33	.744
Technical difficulties
Intercept	2.45	0.08	156.49	30.45	<.001	2.65	.22	176.00	12.31	<.001
Technical difficulties	0.00	0.02	117.55	−0.26	.798	−0.03	.03	176.00	−1.06	.292
Session Number	−0.02	0.01	184.86	−1.56	.120	0.00	.02	176.00	−0.26	.796

Note. SE = standard error, df = degrees of freedom, t = t statistic, p = p-value.

PCL

Clients’ session-to-session PCL scores are presented in Table 4. Variance decomposition of the unconditional model indicated that about two-thirds (63.2%) of the variance in PCL change scores was attributed to within-client variability (Level-1), and the remaining 36.8% of the variance was attributed to between-clients (Level-2).

While controlling for baseline PTSD symptom severity and the number of days that the client had been in therapy, higher therapist consultation attendance predicted greater change in clients’ PTSD symptom severity. The effect size was medium, pr = .20. We also observed that the therapists’ later clients improved in their PTSD symptoms at a faster rate than their earlier counterparts, b = −0.02, t(1,145) = -3.50, p < .001, pr = −.10. The effect size of this association was small.

The same analysis was repeated for each of the three categories of consultation activities. Again, the analysis controlled for baseline symptom severity and the number of days since the clients’ baseline sessions. Accumulation of therapists’ discussion of the application of CPT to individual cases was associated with greater change in the levels of PTSD symptoms, pr = .19, but this association was not statistically significant, p = .056. Because we observed a small-to-medium effect size, we performed exploratory post-hoc analyses of individual activities within this category to inform further research on consultation strategies. Discussing application of specific strategies of CPT to individual cases, b = 0.35, t(948) = 2.09, p = .039, pr = .21, was the only consultation activity in this broader category associated with greater decrease in the clients’ PTSD symptoms. The other two consultation activity categories, reviewing fidelity, pr = −.11, and technical difficulties, pr = 0.08, were not significantly associated with client progress. The fixed effects of our models for client symptom change are presented in Table 6.

Table 6.

Fixed effects for Two level model of consultation attendance and categories of activities predicting client PTSD symptom changes.

Predictor	PCL
Predictor	Estimate	SE	df	t	p
Attendance
Baseline PCL	0.00	0.01	968.68	−0.58	.560
Attendance	0.33	0.15	126.06	2.28	.024
Days since client baseline	0.10	0.01	993.69	11.91	<.001
Discussion of the application of CPT to individual cases
Baseline PCL	0.00	0.00	965.60	−0.08	.935
Discuss CPT Strategies	0.10	0.05	96.87	1.94	.056
Days since client baseline	0.10	0.01	998.00	12.65	<.001
Review fidelity
Baseline PCL	0.03	0.01	986.31	4.06	<.001
Review fidelity	−0.06	0.09	34.83	−0.66	.515
Days since client baseline	0.10	0.01	996.32	12.91	<.001
Technical difficulties
Baseline PCL	0.00	0.00	956.70	2.12	.034
Technical difficulties	0.39	0.57	74.12	0.68	.498
Days since client baseline	0.10	0.00	992.10	13.74	<.001

Note. PCL = Posttraumatic Stress Disorder Checklist, SE = standard error, df = degrees of freedom, t = t statistic, p = p-value.

Discussion

In this study of Canadian therapists who were newly trained in CPT, we found that higher consultation attendance predicted a greater decrease in their clients’ PTSD symptoms. The amount of discussion of application of CPT to individual cases during consultation was marginally associated with improvements in PTSD symptoms. Within that broad category of consultation activities, discussing the application of specific CPT strategies was uniquely and significantly related to improvements in PTSD. The remaining consultation activity categories examined – reviewing and providing feedback on fidelity to the protocol and experiencing technical difficulties – were not associated with observer-rated CPT adherence or competence, nor client symptom change.

This work expands on the EBP consultation literature by examining whether engagement in consultation meetings, or in specific consultation activities are associated with clinical or implementation outcomes. The lack of an association between specific consultation activities and CPT treatment fidelity is surprising, as previous literature has identified an association between consultation and fidelity (Beidas & Kendall, 2010; Edmunds et al., 2017; Herschell et al., 2010; Jackson et al., 2017; Sholomskas et al., 2005). While adherence was on average relatively high in this study, there was room for improvement in competence (Monson et al., 2018). Moving forward, it would be important to look more at the mechanisms affecting therapist competence (McLeod et al., 2018). For example, the therapists might be taking time and reflecting more on the cases following consultation and that may have impacted more subtle aspects of the therapy process that were not explicitly identified in the fidelity measures, such as targeting the most central stuck points or improving the focus of their Socratic questions (Pace et al., 2020; Sijercic et al., in 2021). It may be important to base feedback on fidelity on full session ratings and use formal fidelity assessment tools as the basis for feedback, rather than general guidance based on the therapist's observation of a short segment of the session, to see improvement (Caron & Dozier, 2019; Lu et al., 2012). Before concluding that consultation activities or review of session recordings is not necessary to promote better fidelity, it is important to replicate the parent study (Monson et al., 2018) with larger samples, with less structured EBPs, and to examine other approaches to providing feedback on fidelity using session recordings. It remains possible that audio may be more necessary when there is greater latitude in selecting interventions based on case conceptualization to assist the therapist in making decisions about how best to work with individual cases based on available clinical information and interactions that occurred in the previous session.

Client PTSD symptoms were significantly associated with therapists’ overall consultation attendance, supporting previous research that participating in consultation does result in benefits for clients (Edmunds et al., 2017; Jackson et al., 2017). These findings were anticipated, since in the parent study, clients of therapists who were randomized into consultation experienced better outcomes than those of therapists who did not receive consultation (Monson et al., 2018). Consultation attendance may also explain why, in our study, as well as in another recent study, the second and subsequent clients who were treated while their therapists were receiving consultation had better symptom outcomes than the first client seen during consultation (Johnson et al., 2019). Though an alternative explanation is that therapists simply gain more experience over time, our additional analyses suggest that what occurs during consultation may account for some of the differences in symptom improvement. While the three main categories that we examined did not reach statistical significance, we did find evidence of small effects. Our post-hoc analysis suggests that discussion and selection and application of specific CPT strategies to individual clients during consultation is associated with improvements in PTSD symptoms. As these activities allowed for guidance regarding the types of trauma-related beliefs and Socratic questions that might be particularly important, they may have been more beneficial than more technical feedback on CPT fidelity. Allowing more time for discussion of the application of CPT to individual cases and planning next steps in terms of delivering interventions may be a better use of consultation time if the ultimate goal is achieving better client outcomes.

Our analyses did not suggest, though, that the improvements in client outcomes were due to specific consultation activities that led to improvements in fidelity. While there have been mixed findings regarding fidelity as a mechanism of treatment outcome (Webb et al., 2010), some studies have indicated a relationship between fidelity and symptom change for CPT (Farmer et al., 2018; Holder et al., 2017; Keefe et al., under review). Thus, our findings are about adherence and competence are somewhat surprising, in light of other findings to the contrary (Herschell et al., 2010). However, one potential reason for differences in our findings was that our sampling strategy differed somewhat from studies that have explicitly examined links between fidelity and outcomes, as we randomly sampled single sessions from different phases of consultation, rather than sessions across protocols for each individual client. It is possible that the way we measured fidelity and session activities did not capture CPT elements that are necessary to produce outcomes. Case discussion and guidance on CPT strategies may lead to a greater focus on the specific beliefs that contribute to and maintain PTSD symptoms for individual clients and more guidance on using Socratic dialogue in upcoming sessions. Competence in both of these activities have been associated with greater symptom change, but were not specifically captured for every session on the standard CPT fidelity measure (Farmer et al., 2017). Additionally, as we saw small effects for competence, it is possible that research with a larger sample may be needed to determine whether there are notable associations between consultation activities and competence.

Additional research is needed to better understand whether and how consultation leads to improved client-level outcomes. Consultation may have had an impact on self-efficacy or may have mitigated potential impacts of lower self-efficacy in providing CPT (Pace et al., 2020). In fact, our findings associating attendance in consultation and better outcomes for therapists’ later clients suggest that factors such as the therapist's increased experience and confidence in the treatment or their ability to provide it may have an impact on client progress. Something about the consultation process may have also improved therapeutic alliance, a factor associated with symptom improvement and retention, or it may have reduced therapist burnout (Sijercic et al., 2021; Keefe et al., under review). Additionally, consultation may impact more complex interactions between factors such as self-efficacy and fidelity, fidelity and alliance (Keefe et al., under review), or clinician and client skill (Stirman et al., 2021), which may contribute to symptom change.

Our results also suggest that technical difficulties were not significantly associated with PTSD symptoms or fidelity. In light of challenges that can still occur in lower-resource settings, such as low internet bandwidth and delays in streaming session audio over the web, it is encouraging that such challenges do not necessarily result in poorer outcomes. Though overall rates of technical difficulties were fairly low in this study, the association between technical difficulties and subsequent PTSD symptoms approached statistical significance. It may be important to anticipate and proactively address potential challenges to efficient internet and phone-based consultation meetings, especially in contexts where sampling or reviewing of session recordings is determined to be necessary.

Although other studies have found that live video observation results in better clinical outcomes as compared to telephone consultation (Funderburk et al., 2015), this is the first study, to our knowledge, to assess the associations between distinct activities within consultation and clinical outcomes. Furthermore, this study focused on an EBP for PTSD, whereas previous studies that examined consultation have focused on interventions for substance use disorders (Miller et al., 2004; Sholomskas et al., 2005), depression, and anxiety (Beidas & Kendall, 2010; Rakovshik et al., 2016). Our findings suggest that “unpacking the black box” to examine consultation strategies may result in useful guidance about the types of activities to emphasize during consultation meetings (Nadeem et al., 2013).

Despite these strengths, some limitations must be considered. First, these findings are based on consultation for a specific, structured treatment for a single mental health disorder. Consultation for trauma-focused treatment entails the discussion of methods for treating PTSD, and thus findings regarding this form of consultation may not extend to treatments for other problems – particularly those treatments that provide more latitude for selecting which interventions to use in each session. However, CPT is a cognitive behavioral therapy, which is a theoretical foundation for many evidence-based treatments that address a variety of disorders (e.g. depression, anxiety, insomnia), and thus some aspects of our results may generalize. Therapists who participated in this study had a wide range of training backgrounds, years of experience, prior CBT training, and attitudes towards CPT. While therapist and organizational characteristics were largely shown not to be associated with competence in the therapists’ earliest CPT sessions, and their degree type was included as a covariate in our analyses of competence based on recent findings (Sijercic et al., 2020), it is possible that other factors influenced trajectories of improvement. Although we would ideally be able to control for more therapist-level variables, there are also problems with oversaturating models with covariates (Westfall & Yarkoni, 2016). Thus we focused only on what had been shown in our previous research with this sample to be associated with fidelity. Still, because we specified random intercepts in our multilevel models, we were able to control to some extent for therapist factors that might have influenced the baseline levels of clinician skills.

An additional limitation is that we did not manipulate different aspects of consultation other than review of session audio, which would be necessary to more conclusively identify whether specific elements can produce better outcomes. Finally, we did not examine some elements of consultation that have been discussed in previous literature, such as behavioral rehearsal (Beidas et al., 2014; Edmunds et al., 2013) and modeling (Bearman et al., 2013). Studies on these components were published after the parent study was underway, and these elements of consultation were not integrated into the standard consultation strategy, as the parent study was designed as a randomized, controlled comparison of CPT consultation as usual to a version that included audio review and fidelity feedback (Monson et al., 2018). While consultants sometimes modeled interventions such as asking Socratic questions or emphasizing and reinforcing homework completion, this was captured under “discussing the application of CPT to individual cases”, as modeling was not included on the checklist. Behavioral rehearsal is not commonly used in standard CPT consultation and thus we did not assess it on session checklists.

Future directions in research on psychotherapy consultation should include understanding mechanisms that might underlie the significant relationships found in the present study. Further work is underway to better understand patterns of fidelity, as well as other factors such as therapeutic alliance, and their relationships to clinical outcomes (Keefe et al., under review). Additionally, a recent model proposed by McLeod and colleagues (2018) proposes that self-efficacy and trainee/consultant alliance are important variables in the pathway from consultation to training outcomes. Self-efficacy could influence one's desire to attend consultation and apply CPT to their individual case activities. Furthermore, the degree to which a therapist feels aligned with the consultant could also influence the discussion and conceptualization of cases and receptivity to feedback and guidance. Further exploration of how consultation activities interact with these factors may be fruitful directions to better understanding how and why consultation works.

Footnotes

Author note

Mr. Song is now at the University of California, Berkeley. Dr. Suvak passed away in June 2020 and was instrumental in guiding analyses in this paper.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This work was supported by the Canadian Institutes for Health Research grant [259353; Monson & Stirman]. Additionally, resources and support were provided by the National Center for PTSD and the VA Palo Alto Healthcare System. Opinions expressed in this article are the authors’ own and do not necessarily represent those of the Department of Veterans Affairs or the Royal Canadian Mounted Police.

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Canadian Institutes of Health Research (grant number 259353).

ORCID iDs

Jiyoung Song

Shannon Wiltsey Stirman

Appendix A: Weekly Consultation Checklist-Standard Consultation ( Stirman & Monson,2011 )

Consultant:

Time:

Did you engage in the following CORE activities as a part of consultation as usual?

____ Record the call

____Set an agenda to prioritize discussion

____Discuss clinicians’ cases

____Discuss interventions to use in upcoming sessions

____Discuss ways to overcome challenges

____Discuss selection of CPT cases

____Discuss how to deliver/application of CPT interventions for specific cases

____Encourage clinicians to stick to CPT protocol

____Discuss how to handle emergent situations

Did you engage in any PROSCRIBED activities in today's call?

____Play audio of case material

____Review copies of the clinicians’ worksheets or stuck point logs

____Discuss fidelity ratings

____Play clips of “textbook” examples of CPT interventions

____Circulate materials that are not available to clinicians in all study conditions

____Encourage clinicians to review their own session audio

____Off-topic (non-CPT/non-emergency) discussion (more than 3–5 min)

If so, please explain:

Did you encounter any technical difficulties today?

Appendix B: Weekly Consultation Checklist-Standard Consultation ± Audio ( Stirman & Monson,2011 )

Consultant:

Time:

Did you engage in the following CORE activities as a part of consultation as usual (note that bold items should happen in every call)?

____Record the call

____Set an agenda to prioritize discussion

____Play audio of case material

check one:___1 case __2 cases __>2 cases

____Review copies of the clinicians’ worksheets or stuck point logs

____Discuss fidelity ratings

____Discuss the cases of clinicians who did not present audio

____Discuss upcoming sessions

____Discuss ways to overcome challenges

____Discuss selection of CPT cases

____Discuss how to deliver/application of CPT interventions for specific cases

____Encourage clinicians to stick to CPT protocol

____Discuss how to handle emergent situations

____Encourage Clinicians to identify timestamps for audio clips to play next week

Did you engage in any of the following optional elements?

___Encourage clinicians to review their own session audio

___Play clips of “textbook” examples of CPT interventions

Did you engage in any of the following PROSCRIBED activities?

___Circulate materials that are not available to clinicians in all study conditions

___Skipped audio replay during this call

___Off-topic (non-CPT/non-emergency) discussion (more than 3–5 min)

If so, please explain:

Did you encounter any technical difficulties? Please describe:

References

American Psychiatric Association (2000). Diagnostic and statistical manual of mental disorders (DSM-IV TR) (4th ed.). Author.

American Psychological Association (2018). Clinical practice guideline for the treatment of posttraumatic stress disorder (PTSD) in adults. Author.

Bates

D. M.

Mäechler

Bolker

Walker

(2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01

Bearman, S. K., Weisz, J. R., Chorpita, B. F., Hoagwood, K., Ward A., Ugueto, A. M., & Bernstein, A. (2013). Research Network on Youth Mental Health. More practice, less preach? the role of supervision processes and therapist characteristics in EBP implementation. Adm Policy Ment Health, 40(6), 518–529. https://doi.org/10.1007/s10488-013-0485-5

Beidas

R. S.

Kendall

P. C.

(2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice, 17(1), 1–30. https://doi.org/10.1111/j.1468-2850.2009.01187.x

Beidas

R. S.

Cross

Dorsey

(2014). Show me, don’t tell me: Behavioral rehearsal as a training and analogue fidelity tool. Cognitive and Behavioral Practice, 21(1), 1–11. https://doi.org/10.1016/j.cbpra.2013.04.002

Boswell

J. F.

Gallagher

M. W.

Sauer-Zavala

S. E.

Bullis

Gorman

J. M.

Shear

M. K.

Woods

Barlow

D. H.

(2013). Patient characteristics and variability in adherence and competence in cognitive-behavioral therapy for panic disorder. Journal of Consulting and Clinical Psychology, 81(3), 443. https://doi.org/10.1037/a0031437

Brosan

Reynolds

Moore

R. G.

(2008). Self-evaluation of cognitive therapy performance: Do therapists know how competent they are? Behavioural and Cognitive Psychotherapy, 36(5), 581–587. https://doi.org/10.1017/S1352465808004438

Caron

E. B.

Dozier

(2019). Effects of fidelity-focused consultation on Clinicians’ implementation: An exploratory multiple baseline design. Administration and Policy in Mental Health and Mental Health Services Research, 46(4), 445–457. https://doi.org/10.1007/s10488-019-00924-3

10.

Chard

(2009). Cognitive processing therapy consultation manual. (Unpublished manuscript).

11.

Chard

K. M.

Ricksecker

E. G.

Healy

E. T.

Karlin

B. E.

Resick

P. A.

(2012). Dissemination and experience with cognitive processing therapy. Journal of Rehabilitation Research & Development, 49(5), 667–678. https://doi.org/10.1682/JRRD.2011.10.0198

12.

Clark

D. M.

(2011). Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: The IAPT experience. International Review of Psychiatry, 23(4), 318–327. https://doi.org/10.3109/09540261.2011.606803

13.

Edmunds

J. M.

Beidas

R. S.

Kendall

P. C.

(2013). Dissemination and implementation of evidence–based practices: Training and consultation as implementation strategies. Clinical Psychology: Science and Practice, 20(2), 152–165. https://doi.org/10.1111/cpsp.12031

14.

Edmunds

J. M.

Brodman

D. M.

Ringle

V. A.

Read

K. L.

Kendall

P. C.

Beidas

R. S.

(2017). Examining adherence to components of cognitive-behavioral therapy for youth anxiety after training and consultation. Professional Psychology: Research and Practice, 48(1), 54. https://doi.org/10.1037/pro0000100

15.

Eftekhari

Ruzek

J. I.

Crowley

J. J.

Rosen

C. S.

Greenbaum

M. A.

Karlin

B. E.

(2013). Effectiveness of national implementation of prolonged exposure therapy in veterans affairs care. JAMA Psychiatry, 70(9), 949. https://doi.org/10.1001/jamapsychiatry.2013.36

16.

Farmer

C. C.

Mitchell

K. S.

Parker-Guilbert

Galovski

T. E.

(2017). Fidelity to the cognitive processing therapy protocol: Evaluation of critical elements. Behavior Therapy, 48(2), 195–206. https://doi.org/10.1016/j.beth.2016.02.009

17.

Foa

Hembree

Rothbaum

B. O.

(2007). Prolonged exposure therapy for PTSD: Emotional processing of traumatic experiences therapist guide. Oxford University Press.

18.

Funderburk

Chaffin

Bard

Shanley

Bard

Berliner

(2015). Comparing client outcomes for two evidence-based treatment consultation strategies. Journal of Clinical Child & Adolescent Psychology, 44(5), 730–741. https://doi.org/10.1080/15374416.2014.910790

19.

Henggeler

S. W.

Melton

G. B.

Brondino

M. J.

Scherer

D. G.

Hanley

J. H.

(1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65(5), 821. https://doi.org/10.1037/0022-006X.65.5.821

20.

Herschell

A. D.

Kolko

D. J.

Baumann

B. L.

Davis

A. C.

(2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30(4), 448–466. https://doi.org/10.1016/j.cpr.2010.02.005

21.

Holder

Holliday

Williams

Mullen

Surís

(2018). A preliminary examination of the role of psychotherapist fidelity on outcomes of cognitive processing therapy during an RCT for military sexual trauma-related PTSD. Cognitive Behaviour Therapy, 47(1), 76–89. https://doi.org/10.1080/16506073.2017.1357750

22.

Jackson

C. B.

Herschell

A. D.

Schaffner

K. F.

Turiano

N. A.

McNeil

C. B.

(2017). Training community-based clinicians in parent-child interaction therapy: The interaction between expert consultation and caseload. Professional Psychology: Research and Practice, 48(6), 481–489. https://doi.org/10.1037/pro0000149

23.

Johnson

J. J.

Zlotnick

Cerbo

L. A.

Andrade

J. T.

Nargiso

Bonner

Wiltsey-Stirman

(2019). Randomized cost-effectiveness trial of group interpersonal psychotherapy (IPT) for prisoners with major depression. Journal of Consulting and Clinical Psychology, 87(4), 392–406. https://doi.org/10.1037/ccp0000379

24.

Keefe

Hernandez

Johanek

Landy

M. S. H.

Sijercic

Shnaider

Wagner

Lane

J. E. M.

Monson

Stirman

S. W.

(under review). Competence in delivering Cognitive Processing Therapy and the therapeutic alliance both predict PTSD symptom outcomes.

25.

Kirk

R. E.

(1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56(5), 746–759. https://doi.org/10.1177%2F0013164496056005002

26.

Kuznetsova

Brockhoff

P. B.

Christensen

R. H. B.

(2017). Lmertest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–26. https://doi.org/10.18637/jss.v082.i13

27.

Yanos

P. T.

Gottlieb

J. D.

Duva

S. M.

Silverstein

S. M.

Xie

Rosenberg

S. D.

Mueser

K. T.

(2012). Use of fidelity assessments to train clinicians in the CBT for PTSD program for clients with serious mental illness. Psychiatric Services, 63(8), 785–792. https://doi.org/10.1176/appi.ps.201000458

28.

Mandell

D. S.

Xie

Reisinger

Marcus

S. C.

(2013). The role of treatment fidelity on outcomes during a randomized field trial of an autism intervention. Autism, 17(3), 281–295. https://doi.org/10.1177/1362361312473666

29.

Marques

Valentine

S. E.

Kaysen

Mackintosh

M. A.

Dixon De Silva

L. E.

Ahles

E. M.

Youn

S. J.

Shtasel

D. L.

Simon

N. M.

Stirman

S. W.

(2019). Provider fidelity and modifications to cognitive processing therapy in a diverse community health clinic: Associations with clinical change. Journal of Consulting and Clinical Psychology, 87(4), 357–369. https://doi.org/10.1037/ccp0000384

30.

McHugh

R. K.

Barlow

D. H.

(2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65(2), 73. https://doi.org/10.1037/a0018121

31.

McLeod

B. D.

Cox

J. R.

Jensen-Doss

Herschell

Ehrenreich-May

Wood

J. J.

(2018). Proposing a mechanistic model of clinician training and consultation. Clinical Psychology: Science and Practice, 25(3), e12260. https://doi.org/10.1111/cpsp.12260

32.

Miller

W. R.

Yahne

C. E.

Moyers

T. B.

Martinez

Pirritano

(2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology, 72(6), 1050. https://doi.org/10.1037/0022-006X.72.6.1050

33.

Monson

C. M.

Shields

Suvak

M. K.

Lane

J. E.

Shnaider

Landy

M. S.

Wagner

A. C.

Sijercic

Masina

Wanklyn

S. G.

Stirman

S. W.

(2018). A randomized controlled effectiveness trial of training strategies in cognitive processing therapy for posttraumatic stress disorder: Impact on patient outcomes. Behaviour Research and Therapy, 110, 31–40. https://doi.org/10.1016/j.brat.2018.08.007

34.

Moore

D. E.

Jr Green

J. S.

Gallis

H. A.

(2009). Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. Journal of Continuing Education in the Health Professions, 29(1), 1–15. https://doi.org/10.1002/chp.20001

35.

Nadeem

Gleacher

Beidas

R. S.

(2013). Consultation as an implementation strategy for evidence-based practices across multiple contexts: Unpacking the black box. Administration and Policy in Mental Health and Mental Health Services Research, 40(6), 439–450. https://doi.org/10.1007/s10488-013-0502-8

36.

Pace

B. T.

Song

Suvak

M. K.

Shields

Monson

C. M.

Wiltsey Stirman

(2020). Therapist self-efficacy in delivering cognitive processing therapy. Cognitive and Behavioral Practice, 28 (3), 327–335. https://doi.org/10.1177/26334895211030220

37.

Rakovshik

S. G.

McManus

Vazquez-Montes

Muse

Ougrin

(2016). Is supervision necessary? Examining the effects of internet-based CBT training with and without supervision. Journal of Consulting and Clinical Psychology, 84(3), 191. https://doi.org/10.1037/ccp0000079

38.

R Core Team (2018). R: A Language and Environment for Statistical Computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/.

39.

Resick

P. A.

Galovski

T. E.

Uhlmansiek

M. O.

Scher

C. D.

Clum

G. A.

Young-Xu

(2008). A randomized clinical trial to dismantle components of cognitive processing therapy for posttraumatic stress disorder in female victims of interpersonal violence. Journal of Consulting and Clinical Psychology, 76(2), 243–258. https://doi.org/10.1037/0022-006X.76.2.243

40.

Resick

P. A.

Monson

C. M.

Chard

K. M.

(2017). Cognitive processing therapy for PTSD. A Comprehensive Manual.

41.

Schnyder

Ulrich

Cloitre

Marylene

. (2015). Schnyder U, Cloitre M (eds.) (2015) Evidence based treatments for trauma-related psychological disorders: A practical guide for clinicians. Springer International Publishing Switzerland. https://doi.org/10.1007/978-3-319-07109-1_1

42.

Schoenwald

S. K.

Sheidow

A. J.

Letourneau

E. J.

(2004). Toward effective quality assurance in evidence-based practice: Links between expert consultation, therapist fidelity, and child outcomes. Journal of Clinical Child and Adolescent Psychology, 33(1), 94–104. https://doi.org/10.1207/S15374424JCCP3301_10

43.

Sheridan

S. M.

Swanger-Gagné

Welch

G. W.

Kwon

Garbacz

S. A.

(2009). Fidelity measurement in consultation: Psychometric issues and preliminary examination. School Psychology Review, 38(4).

44.

Sholomskas

D. E.

Syracuse-Siewert

Rounsaville

B. J.

Ball

S. A.

Nuro

K. F.

Carroll

K. M.

(2005). We don’t train in vain: A dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology, 73(1), 106. https://doi.org/10.1037/0022-006X.73.1.106

45.

Sijercic

Lane

J. E.

Gutner

C. A.

Monson

C. M.

Stirman

S. W.

(2020). The association between clinician and perceived organizational factors with early fidelity to cognitive processing therapy for posttraumatic stress disorder in a randomized controlled implementation trial. Administration and Policy in Mental Health and Mental Health Services Research, 47(1), 8–18. https://doi.org/10.1007/s10488-019-00966-7

46.

Sijercic, I., Liebman, R. E., Stirman, S. W., & Monson, C. M. (2021). The Effect of Therapeutic Alliance on Dropout in Cognitive Processing Therapy for Posttraumatic Stress Disorder. Journal of traumatic stress, 34(4), 819–828. https://doi.org/10.1002/jts.22676

47.

Stirman

S. W.

Gutiérrez-Colina

Toder

Esposito

Barg

Castro

Beck

A. T.

Crits-Christoph

(2013). Clinicians’ perspectives on cognitive therapy in community mental health settings: Implications for training and implementation. Administration and Policy in Mental Health and Mental Health Services Research, 40(4), 274–285. https://doi.org/10.1007/s10488-012-0418-8

48.

Stirman, S. W., Gutner, C. A., Gamarra, J., Suvak, M. K., Vogt, D., Johnson, C., Wachen, J. S., Dondanville, K. A., Yarvis, J. S., Mintz, J., Peterson, A. L., Young-McCaughan, S., Resick, P. A., & STRONG STAR Consortium (2021). A Novel Approach to the Assessment of Fidelity to a Cognitive Behavioral Therapy for PTSD Using Clinical Worksheets: A Proof of Concept With Cognitive Processing Therapy. Behavior therapy, 52(3), 656–672. https://doi.org/10.1016/j.beth.2020.08.005

49.

Stirman

S. W.

Monson

C. M.

(2011). Audio-enhanced CPT consultation manual. (Unpublished Manuscript. Available from the authors upon request).

50.

Stirman

S. W.

Pontoski

Creed

Xhezo

Evans

A. C.

Beck

A. T.

Crits-Christoph

(2017). A non-randomized comparison of strategies for consultation in a community-academic training program to implement an evidence-based psychotherapy. Administration and Policy in Mental Health and Mental Health Services Research, 44(1), 55–66. https://doi.org/10.1007/s10488-015-0700-7

51.

Strunk

D. R.

Brotman

M. A.

DeRubeis

R. J.

Hollon

S. D.

(2010). Therapist competence in cognitive therapy for depression: Predicting subsequent symptom change. Journal of Consulting and Clinical Psychology, 78(3), 429–437. https://doi.org/10.1037/a0019631

52.

Tracey

T. J.

Wampold

B. E.

Lichtenberg

J. W.

Goodyear

R. K.

(2014). Expertise in psychotherapy: An elusive goal? American Psychologist, 69(3), 218. https://doi.org/10.1037/a003509

53.

Ward

A. M.

Regan

Chorpita

B. F.

Starace

Rodriguez

Okamura

, & Research Network on Youth Mental Health (2013). Tracking evidence based practice with youth: Validity of the MATCH and standard manual consultation records. Journal of Clinical Child & Adolescent Psychology, 42(1), 44–55. https://doi.org/10.1080/15374416.2012.700505

54.

Weathers

F. W.

Litz

B. T.

Herman

D. S.

Huska

J. A.

Keane

T. M.

(1993). The PTSD checklist (PCL): Reliability, validity, and diagnostic utility. Paper presented at the 9th annual convention of the international society for traumatic stress studies, San Antonio, TX.

55.

Webb

C. A.

DeRubeis

R. J.

Barber

J. P.

(2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78(2), 200. https://doi.org/10.1037/a0018912

56.

Webster-Stratton

C. H.

Reid

M. J.

Marsenich

(2014). Improving therapist fidelity during implementation of evidence-based practices: Incredible years program. Psychiatric Services, 65(6), 789–795. https://doi.org/10.1176/appi.ps.201200177

57.

Westfall

Yarkoni

(2016). Statistically controlling for confounding constructs is harder than you think. PloS one, 11(3), e0152719.