Abstract
With the growing emphasis on Evidence-based Practices (EBPs) for Posttraumatic Stress Disorder (PTSD; American Psychological Association [APA], 2018; Schnyder & Cloitre, 2015), public and private mental health organizations are investing substantial time and resources to train their therapists to provide these treatments (Edmunds et al., 2013). In addition to these investments, national and state-level policy directives have been introduced to train therapists to deliver EBPs such as Cognitive Processing Therapy (CPT; Resick et al., 2017) and Prolonged Exposure (PE; Foa et al., 2007) Therapy (Clark, 2011; Eftekhari et al., 2013; McHugh & Barlow, 2010). However, research has demonstrated that initial, didactic training is not sufficient to promote sustained, high-quality treatment delivery. Several studies have shown that some form of follow-up support (i.e., consultation) appears to be necessary after workshops (Herschell et al., 2010; Schoenwald et al., 2004; Sheridan et al., 2009; Webster-Stratton et al., 2014).
Consultation is defined as ongoing, focused interaction with a specialist in an effort to increase competence in the area of the specialist's expertise (Edmunds et al., 2013). Continued support through consultation is believed to facilitate further learning and mastery, and may increase the likelihood of skilled, ongoing delivery of EBPs (Nadeem et al., 2013). This form of follow-up support has been associated with improved treatment fidelity; both adherence (delivery of essential elements of the protocol) and competence (skill of delivery) are higher among therapists who receive consultation (Beidas & Kendall, 2010; Edmunds et al., 2017; Herschell et al., 2010; Sholomskas et al., 2005). Previous research has demonstrated that higher fidelity can lead to improved client outcomes (Farmer et al., 2017; Mandell et al., 2013; Marques et al., 2019; Strunk et al., 2010), and that fidelity may serve as a mediator between implementation of an EBP and client outcomes (Henggeler et al., 1997). However, little research has been conducted to identify whether specific aspects of consultation might lead to higher fidelity or improved client outcomes.
To make training as efficient and successful as possible, an understanding of the strategies that enhance the effectiveness of consultation is essential (Monson et al., 2018; Stirman et al., 2013). Nadeem and colleagues (2013) discussed the importance of “unpacking the black box” of consultation and identified a need for research focused on identifying strategies and mechanisms that drive consultation's impact on implementation outcomes. Their review of the literature distinguished several key aspects of consultation, including continued training, problem solving implementation barriers, case support, and accountability (Nadeem et al., 2013). Furthermore, they suggested that through consultation, therapists are able to share and reflect on their experiences of delivering the treatment, receive feedback, and identify strategies to address challenges that may arise during their therapy sessions. Other researchers have found that factors such as consultation attendance and caseload are associated with fidelity (Edmunds et al., 2017; Jackson et al., 2017). Finally, some research has suggested that clients experience more improvement if they are not the first client that their therapist treats when being trained in a new intervention, which suggests that therapists may be more effective in providing new treatments with practice and consultation (Johnson et al., 2019).
Several studies suggest activities that may contribute to the relationship between aspects of consultation and therapist adherence and competence. Studies have shown that behavioral rehearsal and modeling can increase the use of EBP strategies (Bearman et al., 2013; Beidas et al., 2014; Edmunds et al., 2013). Additionally, the process of instruction, case review, self-evaluation, and feedback can help cultivate skill development in therapists (Brosan et al., 2008; Edmunds et al., 2013; Stirman et al., 2017; Tracey et al., 2014). Following workshop training, guidance on CPT and exposure to case discussion during consultation allow therapists to implement consultants’ suggestions in practice and reflect upon their previous work (Moore et al., 2009), which may ultimately benefit clients. Timely feedback allows therapists to correct and deepen their delivery and understanding of the treatment while receiving simultaneous support from supervisors or consultants. Whether case review and feedback need to include some form of observation has not been empirically determined. Researchers have considered observation and feedback to be the gold standard for fidelity monitoring (Beidas & Kendall, 2010; Monson et al., 2018). However, there is some evidence that observation may not be necessary in all circumstances. For example, Ward and colleagues (2013) determined that, for at least some aspects of EBP delivery, therapists may be able to self-report accurately, suggesting that observation may not be a necessary component of consultation.
The parent study of this manuscript compared the standard form of CPT consultation without any observation and immediate feedback on session recordings to consultation that included review of audio segments and a no consultation comparison group (Monson et al., 2018). Therapists in the standard consultation condition experienced significantly greater changes in PTSD symptoms than those who did not receive any consultation, with no significant differences detected between standard consultation and consultation including review of session audio (Monson et al., 2018). In this form of consultation, which has been used to train thousands of therapists over the past two decades (Chard et al., 2012), therapists attend a group consultation meeting with 4–6 therapists. Each therapist discusses their most recent sessions with their training cases, describing their efforts to provide the prescribed elements of each session, the “stuck points” or automatic thoughts that they focused on, and brings up any challenges they encountered. Agendas are set to prioritize cases where challenging or emergent issues arise, but time is allocated for each therapist to have time to receive consultation on their cases. The consultant asks questions about the case and the session to assess whether key activities in the session occurred, if the therapist doesn’t describe them, and provides guidance on how to address the barriers that were encountered while still adhering to the protocol. Additionally, the consultant provides feedback and guidance about key stuck points to address, what to cover in the next session and how to use CPT strategies based on the working conceptualization of the case. They elicit questions, comments, and suggestions from other group members as well.
The authors speculated that the findings regarding the effectiveness of this approach may be due to the highly structured nature of the CPT protocol, the manual for which contains session agendas and detailed session-by-session descriptions (Monson et al., 2018). With protocols of this nature, it may be important in consultation to focus on conceptualizing and understanding how to apply CPT strategies to each individual case (e.g., what beliefs and cognitions are maintaining the PTSD symptoms for each individual) rather than providing feedback on short segments of audio. Alternatively, technical difficulties in playing audio may have led to a less efficient use of time in the condition that included audio review of sessions.
Collectively, research to date has improved our understanding of the potential active ingredients of consultation, but understanding of the relationship between clinical (e.g., symptom change) or implementation (e.g., fidelity) outcomes and the nature, dose, and exposure to specific consultation activities remains incomplete. A close examination of both the specific elements of consultation and the amount of exposure to each element can elucidate why and how consultation leads to improvements in key clinical and implementation outcomes, and can inform the development of more efficient and effective consultation strategies. Using data from the parent study, comparing post-workshop support and consultation strategies (Monson et al., 2018), we examined consultation sessions that occurred over the 6 months following a workshop on CPT.
To capture real-world applications of CPT and understand what occurs in consultation as usual, we attempted to unpack the “black box” of consultation as routinely provided in national training and consultation programs. Based on the findings from the parent study, our primary hypotheses were that: 1) the dose or exposure to specific consultation activities would be more strongly associated with fidelity and client symptom change than consultation attendance alone, 1a) the cumulative amount of feedback on fidelity to which a therapist was exposed during consultation would be associated with improvements in therapist fidelity (adherence and competence), but that 1b) cumulative exposure to specific elements of consultation (discussion of the application of CPT to individual cases) would be associated with greater changes in PTSD symptoms, and 2) distracting factors such as technical difficulties would be associated with less improvement in fidelity and client symptom change. We also sought to replicate and extend findings from previous research suggesting that consultation attendance was associated with better outcomes (Jackson et al., 2017) and that therapists’ later clients would experience greater improvement (Johnson et al., 2019).
Method
Participants
Therapists
Therapist participants (n = 134) were recruited from Veterans Affairs Canada Operational Stress Injury clinics, Canadian Forces mental health services, and the broader Canadian community. All participants provided voluntary informed consent after a clear description of the Research Ethics Board-approved study procedures. Mental health therapists were eligible to participate in the study if they: attended a standardized CPT workshop, were licensed mental health therapists with psychotherapy in their scope of practice, were currently providing psychotherapy to individuals who were therapist-assessed to have PTSD, consented to be randomized to one of the study conditions, and were willing to provide audio recordings of therapy sessions and measures of PTSD symptoms from consenting clients. In this study, we included the subset of therapists (n = 60) who were randomized into a condition that included consultation and provided session and client materials. Therapist participant characteristics can be found in Table 1. Full details from the parent study can be found in Monson et al., 2018.
Therapist demographic and background characteristics (n = 60).
Note. Sample characteristics are provided fir the therapists who provided client data. Means (M) and standard deviations (SD) reported for continuous variables. Sample sizes (n for total, I per condition; and n for a portion of the sample) and corresponding percentages (%) are reported for categorical variables.
Clients
Participating therapists facilitated recruitment by identifying eligible new or existing clients who provided informed consent for all study procedures. To participate, clients were required to be diagnosed with PTSD by their therapist based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV; American Psychiatric Association, 2000) criteria. Client participants (n = 135) also consented to participate in CPT and have their sessions audio-recorded and reviewed by independent fidelity raters, as well as potentially listened to by other therapists during consultation. Clients were permitted to continue other psychotherapeutic interventions if they were not trauma-focused. Consistent with research-informed standards, clients were ineligible if they had current uncontrolled psychosis or bipolar disorder, substance dependence (abuse was permitted), imminent suicidality or homicidality, or significant cognitive impairment (mild to moderate traumatic brain injury was permitted). Further details on the client participants’ characteristics can be found in Table 2.
Client demographics and baseline clinical characteristics (n = 135).
Note. Means (M) and standard deviations (SD) reported for continuous variables. Sample sizes (n for total, I per condition) and corresponding percentages (%) are reported for categorical variables.
Procedures
Therapists were invited to participate prior to, or at the standardized CPT workshops. After completing the workshop, in the parent study (Monson et al., 2018) therapists who provided informed consent were randomized to one of the three post-workshop consultation conditions: Standard Consultation and Standard Consultation + Audio Review. The third condition was No Consultation and was not analyzed in this manuscript. Full participation in the study required therapists to provide audio recordings of all CPT sessions with at least two clients to be used during consultation and for fidelity ratings, and to collect client symptom and functioning measures. To incentivize participation, therapist participants who uploaded the required number of session recordings and client symptom measures were eligible to become “Quality-rated” CPT Providers and placed on a roster of CPT providers (see www.cptforptsd.com) if they met a minimum threshold for competence. After the 6-month post-workshop phase was completed, all therapists received written feedback on their fidelity in two CPT sessions. Written feedback on their fidelity throughout the consultation period is not part of routine consultation and would not be feasible in routine care. Thus, it was delayed until the end of the study to provide a benefit for therapists in all conditions without introducing a confound. Therapists also received compensation for completing study assessments.
Post-workshop consultation conditions
After randomization in the parent study, therapists joined consultation groups within their assigned conditions based on the meeting times that were most feasible for their schedules. On average, therapists enrolled 2 new or established clients into consultation. Cases were enrolled as therapists were able to identify clients who were interested in enrollment in the study. Welch's t-test indicated that there were no significant differences in the number of clients per therapist between the two consultation conditions, t(10.95) = 1.05, p = .315.
Standard consultation
Therapists in the Standard condition attended a standard 2-day CPT workshop, received the CPT manual and related materials, and had access to resources available through the free CPT-web online training (https://cpt.musc.edu/index). Participants in the Standard consultation condition received 6 months of weekly 1-h group web- and telephone-based consultation with a CPT expert consultant. Four to six therapists were on each call. As described in the introduction, consultation calls included discussions of how to apply the CPT protocol to specific cases, challenging cases, treatment obstacles, and specific issues raised by participants within each group.
Standard consultation with audio review
Therapists in the Standard + Audio condition received all of the elements of the Standard consultation condition, in addition to receiving feedback from other group members as well as from the expert on a presentation of segments of their audio recordings (typically 5–10 min) from a recent session. Participants were asked to play segments that reflected their use of a key skill from that session or to play a segment in which they struggled to deliver CPT. In later sessions, consultants were encouraged to ask them to play random segments of their session if therapists did not identify specific parts that were challenging for them. Therapists typically presented a recording every other call, and those who did not present session recordings in a given week had the opportunity to provide a brief check-in about the progress of their current cases and receive input and feedback as in standard consultation. Four to six therapists attended each call. While this condition included all elements of standard consultation, because a portion of the consultation session was spent listening to the audio of CPT cases, and thus there may have been less time for the standard consultation elements
In both conditions, therapists received written feedback on fidelity for randomly selected sessions after the consultation phase was complete.
Consultants
Expert CPT consultants (n = 5; four PhD level psychologists and one master's level social worker), who had at least five years of experience providing CPT consultation served as consultants in both consultation conditions to control for possible consultant-related effects. They were trained by the principal investigators (PIs) in study consultation procedures for each condition by study team members prior to beginning consultation calls. The prescribed and proscribed activities for each consultation condition are described in manuals (Chard, 2009; Stirman & Monson, 2011) and the PIs reviewed self-reported consultation checklists and recordings of consultant calls monthly to ensure that consultant fidelity to condition (80% or more of the prescribed elements) was maintained. Consultants met every four to six weeks during the first wave of consultation, and subsequently received support and feedback as needed from the principal investigators throughout the study. Consultants completed a checklist of consultation activities after each session, which was also completed for a subsample of sessions by independent observers who reviewed randomly selected recordings of consultation sessions.
Measures
Consultation activity checklist
Upon completion of each consultation session, consultants filled out a checklist indicating activities that were completed during the consultation session as well as which clinicians were in attendance. A subset (15%) of these facilitator checklists were then rated by observers to assess agreement between observers and consultants (scale: 0–1, observer-therapist agreement k = .97; the two observers’ agreement k = .93). Consultation activities from the checklist comprised three broad categories: (1) discussion of the application of CPT to individual cases, comprising the following activities: discussing cases without audio review, case selection/appropriateness for CPT, discussing the use of specific CPT strategies for individual cases, guidance on overcoming clinical challenges, and prioritizing case discussion during agenda setting), (2) reviewing and providing feedback on fidelity, comprising: discussion of how to adhere to the protocol, playing audio sessions, receiving feedback on their fidelity based on the segment, and reviewing copies of work material such as CPT worksheets or stuck point logs), and (3) distractions and technical difficulties, comprising off-topic discussion and technical difficulties. Off-topic discussion was very minimal, so this category essentially reflects technical challenges. For scoring, to determine the amount of therapist exposure to each element, a cumulative score for each category was created for each individual, each consultation week, based on the sum of the activities in each broad category that occurred for each consultation meeting at which the therapist was present. Thus, at the individual therapist level, this variable provided a cumulative count of how many times therapists were exposed to each element over the course of consultation. The mean accumulation scores of consultation activities across the consultation phase are presented in Table 3.
Accumulation of consultation activities over consultation sessions.
Note. M = mean, SD = standard deviation.
Therapist fidelity
Therapist CPT fidelity was assessed using a modified version of the CPT fidelity measure, the Therapist Adherence and Competence Scale (TAC; Resick et al., 2008). The TAC measures both adherence to specific unique and essential CPT elements (scale: 0–3, ICC = 0.87), and competence in delivering these interventions (scale: 0–6; ICC = 0.79). A randomly selected subset of CPT sessions from early, mid- and late protocol (290 sessions, 19% of the total number of session recordings and approximately 3 per therapist who provided recordings) were coded by trained observers who met regularly to discuss ratings and facilitate rater agreement. Mean scores for adherence and competence were calculated for analyses.
Trauma assessment
PTSD diagnosis was determined by the client's treatment providers. For study purposes, we also administered the Posttraumatic Stress Disorder Checklist, DSM-IV version (PCL-IV; Weathers et al., 1993). The PCL-IV is a 17-item, self-report questionnaire of the severity of distress related to PTSD symptoms. It is highly correlated with Structured Clinician-Administered PTSD Assessments, such as the DSM Structured Clinical Interview (Cronbach's alpha = 0.94). Therapists asked clients to complete a PCL-IV pre-treatment (to determine eligibility), before the first session, and prior to every CPT session.
Analytic plan
Multilevel regression analyses (i.e., mixed-effects modeling) were conducted to: (1) analyze the relationship between the accumulation of consultation attendance and activities and observer-rated fidelity (adherence and competence) to the CPT protocol at subsequent therapy sessions, and (2) test whether the accumulation of these activities predicted lower levels of PTSD in subsequent sessions of the therapists’ clients. A multilevel modeling approach allows for detection of and accounting for patterns of fidelity over time, since previous work has suggested that fidelity may decrease over the course of the protocol (Boswell et al., 2013), and also accounts for the degree to which variation in fidelity and clinical change can be attributed to the therapist and/or client level. Moreover, multilevel models permit the analysis of repeated measures nested within clients and therapists and all models were run using the lme4 (Bates et al., 2015) and lmerTest (Kuznetsova et al., 2017) packages in R Statistical Software (R Core Team, 2018). The lme4 package defaults to maximum likelihood to account for missing data.
Adherence and competence
To test our hypotheses regarding the relationship between the consultation attendance and activities and the therapist's adherence and competence, we evaluated two (one for adherence and one for competence) 2-level models for each of the four predictors (attendance, discussion of the application of CPT to individual cases, reviewing for fidelity, and technical difficulties). Our dependent variable, repeated, observer-rated fidelity in randomly selected sessions (adherence and competence; Level-1) was nested within clinicians (Level-2). Session number was also included as a covariate at the time level based on findings by Boswell and colleagues (2013), which suggested that adherence and competence to a CBT protocol decrease over client sessions; we examined our data to determine whether a similar pattern existed, with a plan to include session number as a covariate at the time level if that pattern was identified. Because we found that therapist degree level was associated with baseline competence in another study (Sijercic et al., 2020) that analyzed data from the same parent study, we included it as a covariate in our competence analyses. Before running a predictor model, we ran an unconditional model, excluding the predictor variable and covariates, to examine the variance across each level. Partial correlation coefficients (pr) were reported as measures of effect sizes of our predictors (Kirk, 1996).
Client PTSD symptoms (PCL)
To test our hypotheses regarding the association between consultation attendance and activities and the client's PTSD symptom change, we conducted a 2-level model for each predictor (attendance, discussion of the application of CPT to individual cases, reviewing for fidelity, and technical difficulties). Our dependent variable, the repeated, change score of client PTSD symptoms from baseline (PCL; Level-1), was nested within clients (Level-2). Accumulation of consultation attendance and consultation activities, in separate models, were Level-1 time-varying variables, nested within clients (Level-2). To examine whether therapists’ later CPT clients experienced better outcomes, a separate 2-level model included a predictor variable indicating whether the client was the therapist's first or a later client, as determined by the start date of therapy for each client. The start date was defined as the date the client began CPT. Before conducting a predictor model, we first evaluated an unconditional model, excluding the predictor variable and covariates, to examine the variance across each level. Partial correlation coefficients (pr) were reported as measures of effect sizes of our predictors (Kirk, 1996).
Results
Demographics
Therapists
There were more Female therapists than Male therapists between the Standard Consultation Plus Audio group (Male 30%; Female 70%) and Standard Consultation group (Male 21%); Female 79%). The majority of therapists in both groups had a Masters degree (counseling or social work) or higher, the Standard Consultation Plus Audio group (Masters 56%; PhD/PsyD 28%); the Standard Consultation Group (Masters 35%; PhD/PsyD 35%). Additionally, therapists in both groups had similar caseload sizes, the Standard Consultation Plus Audio group (M = 22; SD = 13.65) and Standard Consultation group (M = 28.96; SD = 15.24). Of those caseloads, the percent of clients that were being treated for PTSD varied, but the majority between the two groups indicated that less than 25% of their caseload consisted of clients being treated for PTSD. On average, therapists treated 2 clients in the study. Further details on the client participants’ characteristics can be found in Table 1.
Clients
Males and Females were nearly evenly distributed between the Standard Consultation Plus Audio group (Male 55.2%; Female 44.8%) and Standard Consultation group (Male 51.7%; Female 48.3%) (n = 135). There were no significant differences between the two groups (Monson et al., 2018). The majority of clients in both groups were white (92.5% in the Standard Consultation Plus Audio group (92.5%); 89.7% inthe Standard Consultation Group. Military status varied among the two groups, the Standard Consultation Plus Audio group (Veterans 18.5%, Active Duty 27.7%, and Non-Military-affiliated 53.8%); Standard Consultation Group (Veterans 22.8%, Active Duty 17.5%, and Non-Military-affiliated 59.6%). The average pre-treatment PTSD PCL score was similar between the two groups as well, the Standard Consultation Plus Audio group (M = 61.68); the Standard Consultation Group (M = 61.04). Lastly, the average number of CPT sessions attended by clients in both groups was about nine. Further details on the client participants’ characteristics can be found in Table 2.
Adherence and competence
The variance decomposition of the unconditional model revealed that almost one hundred percent of the variances of adherence and competence were attributed to within-therapist (Level-1) and almost none to between-therapists (Level-2).
We first examined the association between the therapist’ consultation attendance and their observer-rated adherence and competence to the CPT protocol. Other studies (e.g., Boswell et al., 2013) have shown that fidelity decreased over the course of the protocol. We observed a similar pattern of decrease in the current study, although it did not reach statistical significance, Adherence: b = −0.02, t(49.45) = −1.34, p = .09, pr = −.13; Competence: b = −0.01, t(177) = −0.54, p = .59, pr = −.04. We therefore controlled for session number in our model to evaluate the impact of consultation attendance and activities on fidelity while keeping session number constant. We found no significant main effects of consultation attendance for adherence, pr = −.09, and competence, pr = −.10. Additionally, therapist degree level, which was added as a covariate due to this therapist characteristic's association with early therapist competence (Sijercic et al., 2020), was not significantly associated with changes in competence in any of the analyses, all ps > .06, all prs < .14. We then examined whether three consultation activity categories predict the therapist's observer-rated fidelity. Again, we found no significant main effects of discussion of the application of CPT to individual case activities, pr = −.09; pr = −.13, reviewing fidelity, pr = −.09; pr = −.10, or technical difficulties, pr = −.02; pr = −.08, for adherence and competence, respectively. The results indicate that the accumulations of consultation attendance and categories of consultation activities leading up to the therapy session are not associated with therapist adherence and competence. The fixed effects of our models for therapist fidelity are presented in Table 5.
Descriptive statistics for number of days since baseline and PCL-IV (N = 134).
Note. PCL-IV = Posttraumatic Stress Disorder Checklist, M = mean, SD = standard deviation.
Fixed effects for three level model of consultation attendance and categories of activities predicting adherence and competence.
Note. SE = standard error, df = degrees of freedom, t = t statistic, p = p-value.
PCL
Clients’ session-to-session PCL scores are presented in Table 4. Variance decomposition of the unconditional model indicated that about two-thirds (63.2%) of the variance in PCL change scores was attributed to within-client variability (Level-1), and the remaining 36.8% of the variance was attributed to between-clients (Level-2).
While controlling for baseline PTSD symptom severity and the number of days that the client had been in therapy, higher therapist consultation attendance predicted greater change in clients’ PTSD symptom severity. The effect size was medium, pr = .20. We also observed that the therapists’ later clients improved in their PTSD symptoms at a faster rate than their earlier counterparts, b = −0.02, t(1,145) = -3.50, p < .001, pr = −.10. The effect size of this association was small.
The same analysis was repeated for each of the three categories of consultation activities. Again, the analysis controlled for baseline symptom severity and the number of days since the clients’ baseline sessions. Accumulation of therapists’ discussion of the application of CPT to individual cases was associated with greater change in the levels of PTSD symptoms, pr = .19, but this association was not statistically significant, p = .056. Because we observed a small-to-medium effect size, we performed exploratory post-hoc analyses of individual activities within this category to inform further research on consultation strategies. Discussing application of specific strategies of CPT to individual cases, b = 0.35, t(948) = 2.09, p = .039, pr = .21, was the only consultation activity in this broader category associated with greater decrease in the clients’ PTSD symptoms. The other two consultation activity categories, reviewing fidelity, pr = −.11, and technical difficulties, pr = 0.08, were not significantly associated with client progress. The fixed effects of our models for client symptom change are presented in Table 6.
Fixed effects for Two level model of consultation attendance and categories of activities predicting client PTSD symptom changes.
Note. PCL = Posttraumatic Stress Disorder Checklist, SE = standard error, df = degrees of freedom, t = t statistic, p = p-value.
Discussion
In this study of Canadian therapists who were newly trained in CPT, we found that higher consultation attendance predicted a greater decrease in their clients’ PTSD symptoms. The amount of discussion of application of CPT to individual cases during consultation was marginally associated with improvements in PTSD symptoms. Within that broad category of consultation activities, discussing the application of specific CPT strategies was uniquely and significantly related to improvements in PTSD. The remaining consultation activity categories examined – reviewing and providing feedback on fidelity to the protocol and experiencing technical difficulties – were not associated with observer-rated CPT adherence or competence, nor client symptom change.
This work expands on the EBP consultation literature by examining whether engagement in consultation meetings, or in specific consultation activities are associated with clinical or implementation outcomes. The lack of an association between specific consultation activities and CPT treatment fidelity is surprising, as previous literature has identified an association between consultation and fidelity (Beidas & Kendall, 2010; Edmunds et al., 2017; Herschell et al., 2010; Jackson et al., 2017; Sholomskas et al., 2005). While adherence was on average relatively high in this study, there was room for improvement in competence (Monson et al., 2018). Moving forward, it would be important to look more at the mechanisms affecting therapist competence (McLeod et al., 2018). For example, the therapists might be taking time and reflecting more on the cases following consultation and that may have impacted more subtle aspects of the therapy process that were not explicitly identified in the fidelity measures, such as targeting the most central stuck points or improving the focus of their Socratic questions (Pace et al., 2020; Sijercic et al., in 2021). It may be important to base feedback on fidelity on full session ratings and use formal fidelity assessment tools as the basis for feedback, rather than general guidance based on the therapist's observation of a short segment of the session, to see improvement (Caron & Dozier, 2019; Lu et al., 2012). Before concluding that consultation activities or review of session recordings is not necessary to promote better fidelity, it is important to replicate the parent study (Monson et al., 2018) with larger samples, with less structured EBPs, and to examine other approaches to providing feedback on fidelity using session recordings. It remains possible that audio may be more necessary when there is greater latitude in selecting interventions based on case conceptualization to assist the therapist in making decisions about how best to work with individual cases based on available clinical information and interactions that occurred in the previous session.
Client PTSD symptoms were significantly associated with therapists’ overall consultation attendance, supporting previous research that participating in consultation does result in benefits for clients (Edmunds et al., 2017; Jackson et al., 2017). These findings were anticipated, since in the parent study, clients of therapists who were randomized into consultation experienced better outcomes than those of therapists who did not receive consultation (Monson et al., 2018). Consultation attendance may also explain why, in our study, as well as in another recent study, the second and subsequent clients who were treated while their therapists were receiving consultation had better symptom outcomes than the first client seen during consultation (Johnson et al., 2019). Though an alternative explanation is that therapists simply gain more experience over time, our additional analyses suggest that what occurs during consultation may account for some of the differences in symptom improvement. While the three main categories that we examined did not reach statistical significance, we did find evidence of small effects. Our post-hoc analysis suggests that discussion and selection and application of specific CPT strategies to individual clients during consultation is associated with improvements in PTSD symptoms. As these activities allowed for guidance regarding the types of trauma-related beliefs and Socratic questions that might be particularly important, they may have been more beneficial than more technical feedback on CPT fidelity. Allowing more time for discussion of the application of CPT to individual cases and planning next steps in terms of delivering interventions may be a better use of consultation time if the ultimate goal is achieving better client outcomes.
Our analyses did not suggest, though, that the improvements in client outcomes were due to specific consultation activities that led to improvements in fidelity. While there have been mixed findings regarding fidelity as a mechanism of treatment outcome (Webb et al., 2010), some studies have indicated a relationship between fidelity and symptom change for CPT (Farmer et al., 2018; Holder et al., 2017; Keefe et al., under review). Thus, our findings are about adherence and competence are somewhat surprising, in light of other findings to the contrary (Herschell et al., 2010). However, one potential reason for differences in our findings was that our sampling strategy differed somewhat from studies that have explicitly examined links between fidelity and outcomes, as we randomly sampled single sessions from different phases of consultation, rather than sessions across protocols for each individual client. It is possible that the way we measured fidelity and session activities did not capture CPT elements that are necessary to produce outcomes. Case discussion and guidance on CPT strategies may lead to a greater focus on the specific beliefs that contribute to and maintain PTSD symptoms for individual clients and more guidance on using Socratic dialogue in upcoming sessions. Competence in both of these activities have been associated with greater symptom change, but were not specifically captured for every session on the standard CPT fidelity measure (Farmer et al., 2017). Additionally, as we saw small effects for competence, it is possible that research with a larger sample may be needed to determine whether there are notable associations between consultation activities and competence.
Additional research is needed to better understand whether and how consultation leads to improved client-level outcomes. Consultation may have had an impact on self-efficacy or may have mitigated potential impacts of lower self-efficacy in providing CPT (Pace et al., 2020). In fact, our findings associating attendance in consultation and better outcomes for therapists’ later clients suggest that factors such as the therapist's increased experience and confidence in the treatment or their ability to provide it may have an impact on client progress. Something about the consultation process may have also improved therapeutic alliance, a factor associated with symptom improvement and retention, or it may have reduced therapist burnout (Sijercic et al., 2021; Keefe et al., under review). Additionally, consultation may impact more complex interactions between factors such as self-efficacy and fidelity, fidelity and alliance (Keefe et al., under review), or clinician and client skill (Stirman et al., 2021), which may contribute to symptom change.
Our results also suggest that technical difficulties were not significantly associated with PTSD symptoms or fidelity. In light of challenges that can still occur in lower-resource settings, such as low internet bandwidth and delays in streaming session audio over the web, it is encouraging that such challenges do not necessarily result in poorer outcomes. Though overall rates of technical difficulties were fairly low in this study, the association between technical difficulties and subsequent PTSD symptoms approached statistical significance. It may be important to anticipate and proactively address potential challenges to efficient internet and phone-based consultation meetings, especially in contexts where sampling or reviewing of session recordings is determined to be necessary.
Although other studies have found that live video observation results in better clinical outcomes as compared to telephone consultation (Funderburk et al., 2015), this is the first study, to our knowledge, to assess the associations between distinct activities within consultation and clinical outcomes. Furthermore, this study focused on an EBP for PTSD, whereas previous studies that examined consultation have focused on interventions for substance use disorders (Miller et al., 2004; Sholomskas et al., 2005), depression, and anxiety (Beidas & Kendall, 2010; Rakovshik et al., 2016). Our findings suggest that “unpacking the black box” to examine consultation strategies may result in useful guidance about the types of activities to emphasize during consultation meetings (Nadeem et al., 2013).
Despite these strengths, some limitations must be considered. First, these findings are based on consultation for a specific, structured treatment for a single mental health disorder. Consultation for trauma-focused treatment entails the discussion of methods for treating PTSD, and thus findings regarding this form of consultation may not extend to treatments for other problems – particularly those treatments that provide more latitude for selecting which interventions to use in each session. However, CPT is a cognitive behavioral therapy, which is a theoretical foundation for many evidence-based treatments that address a variety of disorders (e.g. depression, anxiety, insomnia), and thus some aspects of our results may generalize. Therapists who participated in this study had a wide range of training backgrounds, years of experience, prior CBT training, and attitudes towards CPT. While therapist and organizational characteristics were largely shown not to be associated with competence in the therapists’ earliest CPT sessions, and their degree type was included as a covariate in our analyses of competence based on recent findings (Sijercic et al., 2020), it is possible that other factors influenced trajectories of improvement. Although we would ideally be able to control for more therapist-level variables, there are also problems with oversaturating models with covariates (Westfall & Yarkoni, 2016). Thus we focused only on what had been shown in our previous research with this sample to be associated with fidelity. Still, because we specified random intercepts in our multilevel models, we were able to control to some extent for therapist factors that might have influenced the baseline levels of clinician skills.
An additional limitation is that we did not manipulate different aspects of consultation other than review of session audio, which would be necessary to more conclusively identify whether specific elements can produce better outcomes. Finally, we did not examine some elements of consultation that have been discussed in previous literature, such as behavioral rehearsal (Beidas et al., 2014; Edmunds et al., 2013) and modeling (Bearman et al., 2013). Studies on these components were published after the parent study was underway, and these elements of consultation were not integrated into the standard consultation strategy, as the parent study was designed as a randomized, controlled comparison of CPT consultation as usual to a version that included audio review and fidelity feedback (Monson et al., 2018). While consultants sometimes modeled interventions such as asking Socratic questions or emphasizing and reinforcing homework completion, this was captured under “discussing the application of CPT to individual cases”, as modeling was not included on the checklist. Behavioral rehearsal is not commonly used in standard CPT consultation and thus we did not assess it on session checklists.
Future directions in research on psychotherapy consultation should include understanding mechanisms that might underlie the significant relationships found in the present study. Further work is underway to better understand patterns of fidelity, as well as other factors such as therapeutic alliance, and their relationships to clinical outcomes (Keefe et al., under review). Additionally, a recent model proposed by McLeod and colleagues (2018) proposes that self-efficacy and trainee/consultant alliance are important variables in the pathway from consultation to training outcomes. Self-efficacy could influence one's desire to attend consultation and apply CPT to their individual case activities. Furthermore, the degree to which a therapist feels aligned with the consultant could also influence the discussion and conceptualization of cases and receptivity to feedback and guidance. Further exploration of how consultation activities interact with these factors may be fruitful directions to better understanding how and why consultation works.
Footnotes
Author note
Mr. Song is now at the University of California, Berkeley. Dr. Suvak passed away in June 2020 and was instrumental in guiding analyses in this paper.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
This work was supported by the Canadian Institutes for Health Research grant [259353; Monson & Stirman]. Additionally, resources and support were provided by the National Center for PTSD and the VA Palo Alto Healthcare System. Opinions expressed in this article are the authors’ own and do not necessarily represent those of the Department of Veterans Affairs or the Royal Canadian Mounted Police.
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Canadian Institutes of Health Research (grant number 259353).
Appendix A: Weekly Consultation Checklist-Standard Consultation ( Stirman & Monson,2011 )
Consultant:
Time:
Did you engage in the following
____ Record the call
____Set an agenda to prioritize discussion
____Discuss clinicians’ cases
____Discuss interventions to use in upcoming sessions
____Discuss ways to overcome challenges
____Discuss selection of CPT cases
____Discuss how to deliver/application of CPT interventions for specific cases
____Encourage clinicians to stick to CPT protocol
____Discuss how to handle emergent situations
Did you engage in any
____Play audio of case material
____Review copies of the clinicians’ worksheets or stuck point logs
____Discuss fidelity ratings
____Play clips of “textbook” examples of CPT interventions
____Circulate materials that are not available to clinicians in all study conditions
____Encourage clinicians to review their own session audio
____Off-topic (non-CPT/non-emergency) discussion (more than 3–5 min)
If so, please explain:
Did you encounter any technical difficulties today?
Appendix B: Weekly Consultation Checklist-Standard Consultation ± Audio ( Stirman & Monson,2011 )
Consultant:
Time:
Did you engage in the following
____Record the call
____Set an agenda to prioritize discussion
____Play audio of case material
check one:___1 case __2 cases __>2 cases
____Review copies of the clinicians’ worksheets or stuck point logs
____Discuss fidelity ratings
____Discuss the cases of clinicians who did not present audio
____Discuss upcoming sessions
____Discuss ways to overcome challenges
____Discuss selection of CPT cases
____Discuss how to deliver/application of CPT interventions for specific cases
____Encourage clinicians to stick to CPT protocol
____Discuss how to handle emergent situations
____Encourage Clinicians to identify timestamps for audio clips to play next week
Did you engage in any of the following optional elements?
___Encourage clinicians to review their own session audio
___Play clips of “textbook” examples of CPT interventions
Did you engage in any of the following PROSCRIBED activities?
___Circulate materials that are not available to clinicians in all study conditions
___Skipped audio replay during this call
___Off-topic (non-CPT/non-emergency) discussion (more than 3–5 min)
If so, please explain:
Did you encounter any technical difficulties? Please describe: