Abstract
Keywords
Treatment fidelity refers to the methodological strategies used to examine and enhance the reliability and validity of cognitive behavioural interventions [1]. Treatment fidelity is necessary to maintain internal and external validity and to ensure a fair comparison between interventions [2]. In the absence of fidelity evaluation, ineffective techniques may be implemented and disseminated at a high cost to the community and the individual [3]. For example, if the level of fidelity is not monitored, one cannot ascertain whether a significant treatment effect is due to the intervention being tested or to non-specific treatment factors [4]. Conversely, if non-significant results are found and fidelity was not considered, it is not possible to discern whether the absence of treatment effects is due to an ineffective intervention or to poor treatment fidelity [1]. As a result, potentially beneficial treatments for specific patient populations may be prematurely disregarded.
Establishing treatment fidelity is essential in determining what aspects of the intervention are being implemented. In addition, research on treatment fidelity can contribute to our understanding of the mechanisms of therapy [5]. Carefully designed measures that include subscales or different components or facets of therapy have been argued to be potentially useful to determine those components related to therapeutic change [5], [6].
There is emerging evidence for the effectiveness of cognitive behavioural therapy (CBT) for first-episode psychosis (FEP) patients [7]. Individual CBT may be effective in reducing psychotic symptoms, assisting FEP patients in adjusting to their illness and improving quality of life [7–9]. No previous FEP psychotherapy trials, however, have empirically assessed treatment fidelity and described the intervention being tested by using specifically designed measures. This limits the capacity of these trials to determine what therapeutic techniques are being implemented as well as the specific components of therapy related to therapeutic change.
The aim of the present study was to develop and evaluate a measure specifically designed to assess fidelity to the different therapeutic components (i.e. therapy phases) of the individual intervention of a psychotherapy clinical trial (the EPISODE II trial). The relapse prevention therapy (RPT) is a formulation-based, cognitive behavioural intervention aimed to prevent psychotic relapse in a cohort of symptomatically remitted FEP patients. The aims of the present study included description of the fidelity procedures of the study, the psychometric properties of the measure as well as characterization of therapy implementation across different phases of intervention.
Method
EPISODE II trial of relapse prevention in early psychosis
The EPISODE II trial is a prospective, assessor blinded, randomized controlled trial with a 30 month follow up. Eligible patients were randomly allocated to one of two treatments: RPT or treatment as usual (TAU). The intervention trial was designed to determine whether relapse rates can be reduced through a multi-component therapeutic intervention compared to TAU within two specialists FEP programmes. A detailed description of the EPISODE II rationale and sample has been provided elsewhere [10], [11].
Study participants and recruitment
Patients from the Early Psychosis Prevention and Intervention Centre (EPPIC) in Melbourne and from Barwon Health, Victoria Australia, were recruited between November 2003 and May 2005. Inclusion criteria for entry to the trial were as follows: age between 15 and 25 years; meeting DSM-IV criteria for a first episode of a psychotic disorder (American Psychiatric Association) [12]; <6 months of prior treatment with antipsychotic medication; remission of positive psychotic symptoms, defined as ≥ 4 weeks of scores of ≤ 3 (mild) on the subscale items hallucinations, unusual thought disorder, conceptual disorganization and suspiciousness of the expanded version of the Brief Psychiatric Rating Scale [13].
The study was approved by the Northwestern Mental Health Research and Ethics Committee and by Barwon Health Research and Ethics Committee. All potential participants who met inclusion criteria were given the opportunity to participate. Participants received detailed information about the study and the two interventions were clearly explained by the research assistant. For participants younger than 18 years, consent was also obtained from a parent or guardian.
Interventions
Patients randomized to TAU continued with their routine treatment via the EPPIC programme or Barwon Health. TAU was manualized and coordinated by an outpatient case manager and a consultant psychiatrist with access to specialized continuing care teams including home treatment and a group programme [14], [15]. All case managers were orientated to early psychosis treatment guidelines and were provided with a range of additional therapy manuals and standardized audiovisual and print-based materials. Fidelity was monitored through fortnightly one-to-one supervision for all case managers with a senior clinician and via weekly multi-disciplinary case review meetings.
Participants allocated to RPT were treated by an EPISODE II research therapist who additionally adopted the role of outpatient case manager for the duration of their treatment at EPPIC. In addition to the individual therapy, participants assigned to RPT were offered family therapy. Individual RPT was provided by two clinical psychologists (TP and DG). Both research therapists had numerous years of clinical experience within EPPIC outpatient services.
The manualized individual therapy consisted of six phases of therapy (each of one corresponding to a component of therapy) underpinned by a cognitive behavioural framework and informed by previous psychosocial trials conducted at EPPIC [16–18]. The treatment manual included detailed guidelines regarding how the therapy was to be conducted. This included both explanations of the therapeutic techniques to be delivered in each therapy phase as well as those non-specific therapists behaviours that were expected to be performed throughout the therapy. Individual therapy consisted of 12–14 sessions provided approximately fortnightly within a 7 month therapy window.
The following therapy phases were implemented: (i) engagement and individual assessment; (ii) case formulation and therapeutic agenda; (iii) psychoeducation; (iv) early warning signs (EWS); (v) CBT to address proximal risk factors; and (vi) review and termination. A detailed description of the techniques included in each therapy phase is provided in Table 1. Therapists were required to engage in interventions from the corresponding therapy phase, but any intervention could be implemented when clinically indicated.
Relapse prevention therapy treatment modules
CBT, cognitive behavioural therapy; EWS, early warning signs.
Fidelity to individual therapy was ensured via the following procedures: (i) treatment manual detailing intervention techniques throughout therapy; (ii) weekly clinical supervision focused upon relapse providing regular feedback with team clinical supervisors (JG and DW); and (iii) audiotaping of individual therapy sessions for all participants who provided additional consent.
Development of the fidelity measure
The Relapse Prevention Therapy–Fidelity Scale (RPT-FS) was designed to provide relevant information regarding how therapy phases (or components of therapy) and general therapeutic factors were implemented. It contains 45 items organized into seven subscales. The six first subscales correspond to each therapy phase/component, namely assessment/engagement, agenda, psychoeducation, EWS, CBT to address proximal risk factors, and review/termination. The seventh subscale consists of general therapeutic factors that are expected to be delivered throughout the RPT such as setting an agenda for the session, setting and/or reviewing homework, utilizing Socratic questioning, summarizing the content of the session and additional non-specific therapeutic factors (i.e. reflective listening, empathic and genuine attitude, eliciting feedback).
The RPT-FS was developed via a consensus process that involved the EPISODE II research team as well as consultation with other psychosocial research teams at EPPIC. First, two clinical psychologists from the research team extracted the most relevant therapeutic components included in the intervention manual and classified them into either therapeutic phases or general therapeutic interventions. Second, a review panel including the clinical supervisors, authors of therapy manuals, and research therapists assessed all the items in terms of their relevance to the intervention, representativeness to the phase of therapy, and specificity of the therapeutic behaviour. Finally, successive review meetings were undertaken in order to select and rewrite items in accordance with the following principles: the items should be representative of the key therapeutic elements of the therapy phase and the general therapeutic factors thought to be necessary throughout therapy. Furthermore a rater manual was developed to be used in conjunction with the RPT-FS. This included rating rules, definitions and examples of each item. Copies of the RPT-FS and rating manual are available on request.
Estimation of psychometric properties of the RPT-FS
Sample for fidelity ratings
Of the 41 participants initially randomized to the RPT group, four (9.7%) refused consent to be taped. A total of 258 RPT sessions were undertaken with the remaining 37 participants. From these, 197 audiotapings (76.3%) for which consent was obtained were available (phase 1, engagement and assessment, n = 41 tapes; phase 2, agenda, n = 23 tapes; phase 3, psychoeducation, n = 37 tapes; phase 4, EWS, n = 40 tapes; phase 5, CBT, n = 47 tapes; phase 6, review and termination, n = 9 tapes). Subsequently, following recommendations from the Treatment Fidelity Workgroup of Health Behaviour Change Consortium (BCC) [1], [5], [21], a representative sample of audiotapes were randomly selected by the study statistician (SC), stratified by phase of therapy. As a result, 46 session tapes were selected.
Randomly selected sessions from each therapy phase were rated by an independent rater (MA-J). Sessions were randomly ordered and coded by the study statistician in order to keep the rater blind to phase allocation. Items were rated on the basis of the presence/absence of therapeutic behaviour. After each session the rater allocated scores for therapy phases (a function of the number of items carried out in each subscale) and general therapeutic factors (the number of general therapeutic factors performed during the session). Finally, the blind rater made a global assessment of which phase of treatment was being undertaken.
Inter-reliability of the RPT-FS
Twelve of these sessions, two from each therapy phase, were randomly selected, ordered and coded for inter-reliability purposes. A second rater (DW) made the inter-reliability ratings. Inter-rater reliability was assessed using a two-way random effects model intraclass correlation coefficient (ICC2,1) with absolute agreement for each RPT-FS subscale. Values of ≥ 0.75 are considered indicative of adequate consistency and agreement between raters. The internal reliability of all seven subscales of the RPT-FS was assessed using Cronbach's alpha [22].
Treatment fidelity and assessment of therapy components
Treatment fidelity was evaluated in two ways. First, fidelity was assessed in terms of the percentage of correct true positive global ratings, that is, when the rater correctly identified the phase of therapy. Sensitivity, specificity, positive predictive values (PPV) and negative predictive values (NPV) were further estimated for each RPT-FS subscale [23], [24].
Second, minimum cut-off scores for both therapy phases and general therapeutic factors were set in advance to ensure treatment fidelity. Given that the items from each subscale were considered key therapeutic elements of the therapy phase, the therapist was required to perform at least 50% of the elements of the phase being implemented in each session. Similarly, 50% of the general items were required in all therapy sessions. Each session was analysed according to the proportion of fulfilled items for each subscale. For example, if half of the items for the subscale engagement and assessment were carried out in a session, the engagement subscale score was 50. Means, standard deviations and ranges were calculated for each subscale.
Results
Estimation of psychometric properties of the RPT-FS
Estimates of inter-rater reliability ranged from 0.6 (General factors subscale) to 1.0 (Phase 2 total score), which is indicative of perfect agreement. All subscales showed good internal reliability, with alpha coefficients above 0.8 (Table 2).
Inter-rater reliability and internal consistency of RPT-FS subscales
CBT, cognitive behavioural therapy; CI, confidence interval; EWS, early warning signs; ICC, intraclass correlation coefficient; RPT-FS, Relapse Prevention Therapy–Fidelity Scale.; †Engagement and Assessment, n = 12; Agenda, n = 6; Psychoeducation, n = 9; EWS, n = 8; CBT, n = 7; Review and Termination, n = 4.; ‡No variance due to perfect agreement between raters, thus the confidence interval cannot be calculated.; §Cronbach's alpha; ¶F test with true value 0.
Treatment fidelity and assessment of therapy components
Sessions were correctly allocated in 43 of the 46 audiotapes rated. Thus, the global percentage of accurate therapy phase discrimination was 93.4%. Estimates of sensitivity for each therapy phase ranged from 85.7% (CBT subscale) to 100.0% (Psychoeducation, EWS and Review and Termination Subscales). Furthermore, estimates of specificity, PPV and NPV for all RPT-FS subscales ranged from 97.1% to 100.0%, 91.6% to 100.0% and 97.1% to 100.0% respectively (Table 3).
RPT-FS subscales mean scores, percentage scores and discriminative values
CBT, cognitive behavioural therapy; EWS, early warning signs; NPV, negative predictive value; PPV, positive predictive value; RPT-FS, Relapse Prevention Therapy–Fidelity Scale.; †Engagement and Assessment, n = 12; Agenda, n = 6; Psychoeducation, n = 9; EWS, n = 8; CBT, n = 7; Review and Termination, n = 4.
The average proportion of items carried out within RPT phases ranged from 35.7 (in the random selection of tapes from the CBT therapy phase) to 83.3 (in the randomly selected tapes from the setting a therapeutic agenda phase). With the exception of the CBT phase, all therapy phases scored above 50 (Table 3).
Figure 1 depicts the average proportion of items that were fulfilled in each therapy phase. Visual analysis of therapeutic performance across phases showed that research therapists primarily engaged in interventions from the matching therapy phase. Conversely, general therapeutic factors were undertaken throughout RPT phases, ranging from 52.3 (in the random selection of tapes from the engagement and assessment therapy phase) to 71.4 (in the random selection of tapes from the CBT therapy phase) with no statistically significant difference across phases of therapy (F = 1.13, p = 0.36).
Therapeutic techniques implemented in each phase of therapy. Percentage scores were estimated by categorizing tapes according to therapy phase. CBT, cognitive behavioural therapy; EWS, early warning signs.
Discussion
The present study sought to develop a measure to both assess treatment fidelity throughout different therapy phases and to determine those therapeutic components delivered during the experimental intervention of the Episode II trial. Each subscale was specifically designed to include a different component of therapy and its major therapeutic ingredients.
Psychometric properties of the RPT-FS
First, we aimed to evaluate whether the subscales of the RPT-FS could be rated reliability and were internally consistent. Ratings of sessions of a single treatment modality require sensitivity to variation between sessions of the same therapeutic modality [25], [26]. Conversely, when sessions from two different therapeutic modalities are considered together, acceptable inter-rater reliability can be obtained by merely identifying the intended treatment modality (i.e. binary categorization), as opposed to more complex description and discrimination of the therapists’ behaviours implemented throughout the therapy. Despite the inherent difficulty the inter-rater reliabilities of the RPT-FS were high and significantly different from zero across all subscales. Internal consistency was also high for all RPT-FS subscales. Furthermore, the RPT-FS discriminated, almost perfectly, between phases of therapy. These findings suggest that the new fidelity measure was adequate to reliably investigate the therapeutic components of the intervention.
Therapeutic interventions measured by the RPT-FS
This study illustrated the nature and frequency of the interventions implemented through the RPT. Although therapists primarily engaged in interventions from the appropriate therapeutic phase, flexibility in therapy was evident. Some techniques, such as agenda formulation and EWS, were mostly phase-specific, whereas others such as engagement, CBT and psychoeducation were delivered to some degree across different phases of therapy. Furthermore, the general treatment factors were delivered to a similar degree throughout the therapy.
There has been considerable debate on the use of treatment protocols in outcome research versus theory-driven individual case formulation. Supporters of the latter have cautioned that the use of treatment manuals may undermine clinical creativity [27]. Others have argued that rigid treatment protocols may overlook the central role of assessment and case formulation [28]. As mentioned here, the EPISODE II intervention manual provided specific guidelines regarding how specific therapeutic techniques and non-specific therapist behaviours were to be implemented.
The present study provides further evidence in support of the use of treatment protocols. The present findings show that the implementation of therapeutic elements for developing a therapeutic relationship and engaging patients were not undermined by the treatment protocol; rather, such techniques were implemented throughout the intervention. Furthermore, therapists performed a diverse range of techniques in each stage of therapy. This suggests that, while adhering to the RPT, therapists remained flexible and responsive to the participant's changing emotional needs.
Interestingly, the current study found the frequency of implementation of CBT techniques to be lower in relation to other therapeutic techniques. There are several possible factors that may account for these results. First, some FEP patients may have poorer level of insight, which reflects a psychological process of minimization of symptoms and the impact of psychosis [29]. McGlashan et al. referred to this coping strategy as ‘sealing over’, implying that patients who utilize this recovery style are less likely to attempt to understand their symptoms or to place ‘their psychotic experiences in perspective with their lives’ [30]. Given the encouragement that CBT provides to individuals to integrate and understand their symptoms and illness, it may be plausible that those with poorer insight experienced the therapy as somewhat more threatening, which may have affected their engagement with CBT techniques. Alternatively, it is possible that therapists judged that participants required additional strategies to aid their engagement in therapy and engaged in supportive and empathic approaches less directive than structured CBT. This is supported by the finding that the implementation of general therapeutic strategies was particularly high, although not significantly different, during the CBT phase. Likewise, it is plausible that therapists judged that some participants, consistent with their remitted clinical status, did not require structured CBT. We intend to examine these aspects including the relationship between participants’ clinical characteristics, level of insight and therapy implementation in further studies.
Limitations
This study had some limitations. First, we provided data on a representative sample of sessions from each therapy phase. Although this enabled us to assess the psychometric properties of the fidelity measure, we could draw no conclusions on the relationship between patient characteristics, therapy implementation and outcome measures. These research questions will be addressed in forthcoming studies. Second, we assessed therapeutic performance in terms of the timing and frequency of phase-specific techniques as well as general therapeutic factors. As a result, we could not investigate therapeutic competence at the individual level, considering either patient characteristics or contextual factors of therapy. Third, given the distinctive characteristics of the RPT, which was designed as a multi-modal cognitive behavioural intervention for relapse prevention, the present findings can be generalized only to remitted FEP patients who receive this form of therapy. Finally, we did not have available data on the control group therapy. Although the present study was not designed to address treatment differentiation issues, we could not fully establish the differences between both treatment modalities.
Conclusions
In summary, the present study describes the development of a brief, reliable and internally consistent measure to determine both treatment fidelity and the therapy components implemented throughout the experimental intervention. The relationship between specific treatment components and participant outcomes will be analysed via objective ratings of all therapy sessions using this methodology. In the meantime, the present study represents the first step towards the improvement of treatment fidelity and internal validity of FEP psychotherapy research.
Footnotes
Acknowledgements
The EPISODE II trial was funded by an independent research grant from Eli Lilly via the Lilly Melbourne Academic Psychiatric Consortium. The present study was further funded by a grant from the Marqués de Valdecilla Public Foundation–Marqués de Valdecilla Research Institute (FMV-IFIMAV), Santander, Spain. The authors report no additional financial or other affiliation relevant to the subject of this article. The authors acknowledge Dr Cesar Gonzalez-Blanch of University Hospital Marques de Valdecilla, Santander, Spain, and Sara Gook, Joe Buckby, Dr Sarah Bendall and Dr Sarah Hetrick of ORYGEN Research Centre/University of Melbourne for helpful comments on an earlier draft of this article.