Abstract
Improving Access to Psychological Therapies is a UK government-funded initiative to widen access to the psychological treatment of depression and anxiety disorders. The author has had the opportunity to independently assess 90 Improving Access to Psychological Therapies clients, using a standardised semi-structured interview, the Structured Clinical Diagnostic Interview for DSM Disorders (SCID) and to listen to their account of interaction with the service. The results suggest that only the tip of the iceberg fully recovers from their disorder (9.2%) whether or not they were treated before or after a personal injury claim. There is a pressing need to re-examine the modus operandi of the service.
Keywords
Introduction
The PACE (pacing, graded activity, and cognitive behaviour therapy: a randomised evaluation) trial of the effectiveness of cognitive behaviour therapy (CBT) and graded exercise for chronic fatigue syndrome came in for fierce criticism in this journal on the grounds that when objective measures of outcome were used the effectiveness of CBT disappeared (Geraghty, 2017; Vink, 2017). The authors of the PACE trial relied on subjective self-report measures to ‘promote’ the cognitive behaviour therapy and graded exercise therapy protocols that they themselves had developed. This cast into doubt the wisdom of spending £5 million of the taxpayer’s money on the trial (Marks, 2017). However, the UK Government’s Improving Access to Psychological Therapies (IAPT) programme has similarly relied on subjective outcome measures (Layard and Clark, 2014) offering little by way of accountability for the £1 billion pound spent on IAPT since its inception. Furthermore, there has been a complete absence of published reports of independent evidence on the effectiveness of IAPT. In this study, the author, an Independent Expert Witness to the Court, has had the opportunity to audit the mental health trajectory of a sample of IAPT clients, both before and after the event that triggered their personal injury claim, such events ranged from slips to serious road traffic accidents.
Method
The prime duty of the Expert Witness is to the Court, but without reliable knowledge of the effectiveness of treatment in routine practise, the Expert will have difficulty in advising the Court on treatment options. The medico-legal context provides a window through which the effectiveness of interventions delivered by providers, such as IAPT, can be viewed. Unfortunately, the UK government prevented funds from the IAPT programme from being used for research. The author’s medico-legal protocol follows the format detailed by Scott and Sembi (2002) involving an open-ended, semi-structured interview Structured Clinical Diagnostic Interview for DSM Disorders (SCID); First et al., 1997) using the DSMIV TR criteria (American Psychiatric Association, 2000), a screen for malingering and review of records.
Assessment using multiple sources of information is held to be the most reliable form of assessment for both medico-legal and clinical purposes; yet strangely IAPT (in evaluating itself) has relied entirely on just two sources of information, an open-ended interview and psychometric test results. IAPT clinicians rely on a non-standardised open-ended interview to chart client’s treatment voyage. The administration of psychometric tests does not make this type of interview reliable. It is not that the psychometric tests per se are invalid, but client scores only have meaning in a carefully defined context. Clinical diagnoses in open-ended contexts have imperfect validity with kappas (levels of agreement) of 0.1–0.3 between routine open-ended interviews and the ‘gold standard’ research diagnoses achieved with a semi-structured standardised diagnostic interview (Rettew et al., 2009) such as the SCID (First et al., 1997). The SCID begins with an open-ended interview in which the client is encouraged to tell their ‘story’ but then questions are asked about each of the symptoms that comprise a diagnostic set and on the basis of the client’s response and all the information available, including records, a judgement is made about whether a particular symptom can be regarded as present at a clinically significant level with regard to published guidelines, or to put it technically information and criterion variance are controlled for. No such controls are in place for routine open-ended interviews. Thus, the methodology employed in this study incorporated use of the SCID (First et al., 1997).
In the 90 cases considered in this article, there was no evidence of malingering and no challenge from opposing experts on the veracity of the clients’ reports. The sample is described in Table 1.
Description of sample (n = 90).
Of the sample (n = 90), 58 per cent had one disorder, 31 per cent two disorders, 8.9 per cent three disorders, 1.1 per cent four disorders and 1 per cent five disorders. In a study of 2300 psychiatric outpatients, Zimmerman et al. (2008) found an average number of disorders per client of 1.9. The study covered IAPT services in the North West of England. The range of disorders is shown in Table 2.
Range of disorders (n = 90).
From Table 2, it can be seen that if clinicians were adept at screening the four most prevalent disorders and accurately confirming diagnoses, then they would be on-target with 84.3 per cent of disorders. While population profiles may vary from location to location, Pareto’s ‘Law of the Vital Few’ likely operates with 20 per cent of disorders contributing 80 per cent of the workload.
Ethical approval
This study did not fall under the UK National Patient Safety and National Research Ethics Service (NRES) definition of research (National Patient Safety Agency, 2010) as clinicians’ treatment decisions were not manipulated nor were any experimental interventions used in this audit.
Results
The results of IAPT treatment are presented below, first in terms of IAPT’s sole metric of success, psychometric test results administered in the context of an open-ended interview, and then in relation to the author’s independent assessment using a standardised diagnostic interview.
Psychometric test results
In the current sample, IAPT furnished General Practitioners (GPs) with before and after psychometric test data on 29 (32.2%) of the sample. These were people who had had treatment in terms of IAPT’s definition of attending at least two treatment sessions. Given that 26 people attended less than two sessions and there was missing data on session completion for two people, data could have been furnished on 62, so that IAPT is only furnishing data on 29/62 (46.8%) of those for whom they have data. IAPT claims treatment data completeness for 96.8 per cent of people who received treatment. There appears no accountability to GPs.
IAPT’s chief outcome measures are the PHQ-9 and GAD-7 (Kroenke et al., 2001). Clinically reliable improvement is defined by IAPT as a reduction by six or more on the former and four or more on the latter. Recovery is defined as a reduction to below a score of 10 on the Patient Health Questionnaire (PHQ-9) (10 or more is regarded as a ‘case’) and a reduction to a score below 8 on the Generalised Anxiety Disorder (GAD-7). Three of the 29 were below the threshold for a ‘case’ of depression; excluding these from the analysis, the recovery rate on PHQ-9 alone was 6/26 (21.3%) and reliable improvement 3/26 (11.5%). On the GAD-7, there two initial missing values and one case below ‘caseness’; excluding these from the analysis, the recovery rate was 12/26 (46.0%) and 3/26 (11.5%) reliably improved. Looking at recovery on both the PHQ-9 and GAD-7, two cases are excluded because of missing data and two because they were below ‘casenes’ leaving 25 cases for consideration and a recovery rate on both of 6/25 (24.0%). There were no cases (0.0%) of reliable improvement on both. The rate of recovery in this sample of 24.0 per cent is substantially less than the recovery rate of 40.0 per cent of clients that is claimed by IAPT (Gyani et al., 2013).
Results of independent assessment using a standardised semi-structured interview
Table 3 shows how clients fared following CBT treatment, as assessed using the SCID interview (First et al., 1997).
Recovery following IAPT treatment (n = 90).
The overall mean recovery rate across all disorders was 9.2 per cent. Three people were excluded from the post-traumatic stress disorder (PTSD) analysis in Table 3 as PTSD treatment was incomplete at time of assessment. One person was excluded from the ‘Disorders excluding PTSD and depression’ category in Table 3 as treatment was incomplete at time of assessment. One quarter of the sample (23 people; 25.6%) had IAPT treatment before litigation and 67 people (74.4%) had IAPT treatment after the commencement of litigation (Table 4).
Recovery rates for people treated before and after beginning of litigation.
Table 4 indicates that litigation makes no difference to recovery rates. It may be anticipated that the recovery rate post personal injury claim would be less as litigants may have a vested interest in exaggerating debility but inspection of Table 4 does not support this. Further it could be argued that litigants would minimise their distress prior to the personal injury claim and if they had had treatment before exaggerating how useful it had been i.e. that recovery rates would be higher before than after but Table 4 does not support this. IAPT’s ability to engage and sustain client involvement is summarised in Table 5.
IAPT’s engagement and retention of clients (n = 90).
IAPT: Improving Access to Psychological Therapies.
From Table 5, it can be seen that one in four clients fell at the first organisational hurdle. The number of sessions attended did not differ significantly between those who recovered from all disorders and those who did not. However, there was a significant difference (p < 0.05) between the mean number of sessions attended before personal injury 3.5 (4.0) and the mean number of sessions 6.2 (6.1) attended after personal injury.
Forty-eight of the 69 clients (missing value on number of sessions attended by one client) who attended one or more treatment sessions (69.6%) had a sub-therapeutic dose of treatment (defined as attending less than eight sessions – Layard and Clark (2014) state that clients need to receive an average of at least eight sessions). A total of 68 of the 89 clients (76.4%; missing value in one case) had either a sub-therapeutic dose of treatment or did not engage in the treatment process.
Discussion
Inspection of the psychometric test data that was supplied for this study via GPs revealed a recovery rate of 23.0 per cent. This is consistent with the University of Chester’s Centre for Psychological Therapies almost identical figure of 22.0 per cent ‘moving to recovery’ for those who started therapy (Griffiths and Steen, 2013) whereas IAPT’s claim of 44.0 per cent ‘moving to recovery’ was for those completing treatment. Griffiths and Steen (2013) observed that, when all patients ‘referred’ to the IAPT programme are considered, the comparable figure is 12.0 per cent.
IAPT routinely administers the PHQ-9 and GAD-7 at weekly treatment sessions but scores on measures can decrease due to repeated test administration (Longwell and Truax, 2015), it is very doubtful whether these measures are suitable for weekly administration, casting into doubt IAPT’s method of assessing its own effectiveness. Psychometric tests are a measure of the severity of a disorder, so applying tests to a patient’s disorder that is unknown, is problematic and using it as a yardstick for recovery hazardous. Psychometric tests can reasonably be used as a surrogate outcome measure after it is first demonstrated that an intervention (in this case the IAPT service) is effective in routine practice using a ‘gold standard assessment’, but not before. There is a distinction between hard outcome measures, for example, how many people die following a cardiovascular intervention, and surrogate measures, for example, lowering of cholesterol. The danger is that the marketing of products/services is based on the surrogate measures that have a loose association with the hard measure. Surrogate measures ease the research burden but can be very misleading
Clients typically present to a service at their worst and there will necessarily be some regression to the mean on any psychometric test, and distinguishing such changes from the impact of a service is inherently problematic. It is possible to employ measures such as how many patients show clinically significant reliable change (Jacobson and Truax, 1991), but such metrics were developed for use in randomised controlled trials where the diagnostic status of the client is very precisely determined. Applying such metrics to a heterogeneous client population is highly questionable.
IAPT clients routinely complete the PHQ-9, GAD-7 and devices to measure the extent of phobic avoidance. However, such devices can be very misleading, for example, one client with a phobia of travelling by car was discharged on the basis that her Specific Phobia score had gone down to 0 but she still met DSM criteria for a phobia, and while she was not avoiding travelling by car, she was still highly anxious in it.
Layard and Clark (2014) assert that each client receives a professional assessment. However, notes from IAPT Services often arrive at the Expert Witnesses desk with a disclaimer that they do not make a diagnosis and their findings cannot be relied on for medico-legal purposes. The author knows of no other body that makes such a disclaimer.
IAPT therapists are encouraged to make ‘provisional diagnoses’ but do not employ a standardised semi-structured interview in making their ‘professional assessments’. It is therefore unsurprising that there is a gap between IAPT assessments and the results of a SCID interview. To illustrate the point one person was identified by the IAPT therapist as suffering from PTSD following a car accident but the therapist missed that he was also suffering from depression and discharged him from treatment on the grounds that his flashbacks were not as disturbing. However, a comprehensive SCID interview revealed that he was still suffering from PTSD. This case example shows how comorbidity has been missed by the arbitrary nature of criteria used for discharge, whereas the SCID interview has a built-in criterion threshold above which a symptom can be considered present.
Unstructured interviews miss comorbidity (Zimmerman and Mattia, 1999) and IAPT assessments fall into this category. This will result in failure to treat additional disorders while clients wish for treatment for all the disorders for which they present (Zimmerman and Mattia, 2000). A standardised interview such as the SCID ensures that a clinician is looking for disorders so that nothing is overlooked. By contrast, in an unstructured interview, the clinician is likely to stop at the first disorder that they come across.
NICE recommendations are diagnosis specific. Given the vagaries of IAPT’s diagnostic procedures, it is not possible to assess whether the IAPT therapists are National Institute for Health and Clinical Excellence compliant (NICE-compliant) (Gyani et al., 2013). This means that it is not possible to determine whether an appropriate evidence-based treatment protocol has been used, raising an important accountability issue. Confronted with a client, the IAPT clinician literally does not know what he is dealing with and can make no prediction as to the likely best pathway.
IAPT does not employ any measure of treatment fidelity. Fidelity has two components: (a) a measure of adherence, the extent of a focus on identified targets for a disorder and the matching treatment strategy; (b) a measure of competence – how skilfully treatment is conducted. Competence without adherence is meaningless (Scott, 2013).
The IAPT goal has been to have weekly supervision, with 70 hours of supervision a year, far more supervision than is found anywhere else in routine practice. However, the outcome results for IAPT suggest that although supervision may be a necessary condition for the delivery of an evidence-based treatment, it is not sufficient. Close attention has also to be paid to fidelity in supervision. Supervisors are responsible for ensuring that the therapeutic processes followed by the supervisee produce a real-world difference in client’s lives. In this, they have clearly failed. The prime function of a supervisor should be to ensure the translation of an evidence-based treatment into routine practice.
In the absence of a standardised diagnostic interview, it is impossible for IAPT to accurately determine its treatment population. Clients could easily have a personality disorder for which their clinicians are neither trained to identify nor treat. Current IAPT assessments typically involve a 30-minute telephone assessment, which is not fit for reliable diagnosis and should be replaced by the administration of a comprehensive standardised semi-structured interview. Clients would be directed to the appropriate clinician and the initial assessor would be tasked with tracking how the client fared in treatment and monitoring whether there had been fidelity to treatment.
Group intervention and computer-assisted CBT is a rarity. Given that the avowed intention of IAPT is to provide treatment for as many clients as possible, it is surprising that only one of the 90 participants in this audit underwent a group intervention within IAPT, with two others having been offered group CBT but declined. Group interventions for depression and some anxiety disorders appear as effective as individual therapy (Scott, 2009) and it is surprising that consideration appears not to have been given to at least some judicious combination of individual and group intervention. Furthermore, none of the 90 participants in this audit had undergone computer-assisted CBT, although one was offered it and declined.
Client testimonies
The recorded testimonies of clients are contained in Appendix 1. They testify to difficulties with the ‘opt in’ system, problems with the telephone assessment, with little evidence that treatment made a real-world difference to their problems, albeit that a small proportion of clients found the interventions ‘helpful’.
Friends and family test
The UK Government (2013) have suggested that National Health Service (NHS) users are asked ‘How likely are you to recommend our (service) to friends and family if they needed similar treatment?’ with six possible responses: 1: extremely likely; 2: likely; 3: neither likely nor unlikely; 4: unlikely; 5: extremely unlikely; 6: do not know. It would be interesting to know the answers given to this question not only by clients but also those given by IAPT staff.
Limitations
This audit is limited to a sample of IAPT clients whose difficulties have been triggered or exacerbated by some trauma, and as such, the sample may not be representative of all IAPT clients. An independent study using a standardised semi-structured interview of consecutive attenders at IAPT would be needed to confirm the present findings. In the absence of such a study, there are serious doubts as to whether the ‘Emperor has any clothes’. Clinical Commissioning Groups should see IAPT as just one model of service delivery and for which evidence of effectiveness is not proven.
Conclusion
From the present findings, it must be concluded that only the tip of the iceberg of IAPT clients recover when assessed independently using a ‘gold standard’ diagnostic interview. IAPT claims that almost half recover (based on the administration of weekly repeated psychometric tests) appears inflated, and even using their own metric the more likely figure is a quarter. However, care has to be taken in interpreting the psychometric test results in this study as GPs were furnished with only half the test data. IAPT needs to address major issues of independent assessment and accountability. Three-quarters of IAPT clients either had a sub-therapeutic dose of treatment or did not engage in treatment.
IAPT employs the most junior members of staff at the front-end of assessment and pays lip service to diagnosis, which is an integral part of predicting the best treatment for a particular client. There should be a shift to employing more senior staff at the front-end, skilled at least to assess reliably the 20 per cent of disorders that probably make up 80 per cent of the workload. The existing 30-minute telephone consultation is wholly inadequate for predicting the best course of treatment for an individual. Senior staff, including Supervisors, should be involved in monitoring the prescribed treatment to ensure fidelity to an evidence-based treatment protocol, with appropriate targets and matching treatment strategies. In the current regimen, it is unsurprising to find that 68.6 per cent of low intensity workers (responsible for seeing 70% of referrals) and 50 per cent of high intensity therapists are reported to be suffering from burnout (Westwood et al., 2017). There is likely to be poor job satisfaction if therapists do not feel that they are making a real-world difference in client’s lives.
Both for the sake of its staff and its clients, IAPT has to undergo radical reform. Given the high levels of stress in IAPT, it is doubtful as to whether the present Service would pass a ‘Friends and Families Test’. The IAPT Service has been very top-down with little evidence of really listening to either therapists or clients at the coalface. There is a pressing need for both top-down and bottom-up processing.
Given that for depression and many anxiety disorders, group CBT appears as effective as individual CBT, it was surprising to find it was virtually non-existent in this study. IAPT has to move beyond providing occasional workshops for its staff on group CBT and capitalise on the improved access that group work would facilitate. Likewise, there was virtually no evidence of the use of technology (e.g. CBT software) to help enhance outcomes, which needs reappraisal. This author wholeheartedly supports the concept of IAPT, but the implementation of evidence-based treatment has gone badly awry under its auspices.
Footnotes
Appendix 1
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship and/or publication of this article.
