Abstract
Randomised control trials (RCTs) provide the most rigorous form of evidence for the efficacy of any new therapy, including psychotherapy [1]. It is not surprising that the advocates of intensive dynamic psychotherapy (IDP) should be challenged [2],[3] to provide evidence of its efficacy by this method. However, it has been argued that RCTs may be an insufficient or misleading basis for evaluating IDP [1,4–7]. Such arguments include: (i) RCTs are not usually representative of clinical practice [1],[4]; (ii) they are uninformative and too costly [1],[4]; (iii) random allocation is very difficult [1]; (iv) blindness to the allocation process may not be possible [5],[6]; and (v) RCTs are premature in this early phase of psychotherapy research [1] and, in practice, such trials may not be feasible [1],[7]. This paper particularly addresses the last issue.
There are no published reports of an authentic RCT of IDP [8], However, of relevance to the present project were the publications by Stevenson and Meares [9] and by Linehan et al. [10] concerning the treatment of Cluster B personality disorder. The former [9] provided no control group. The latter [10] reported on a group-controlled prospective trial, but the experimental therapy was not based on psycho-dynamic principles. The above trials were less than 2 years, which Denman [1] defined as the minimum duration of long-term therapy. The Menninger study, reported by Wallerstein [11], examined therapies lasting more than 2 years but was retrospective.
The only published report of an attempt at an authentic RCT of IDP, albeit of less than 2 years' duration, was that by Candy et al. [7] in 1972. They described a pilot study for an ambitious RCT in which it was intended to compare the outcomes following time-limited (18 months) psychotherapy with outcomes following two control procedures. Subjects were to be excluded if suffering from significant degrees of personality disorder. The project foundered principally because, in the first 17 months of the pilot study, only eight of 113 referred patients met the intake criteria and, of these, two withdrew before commencing therapy. Based on the experience of the pilot study, Candy et al. concluded that their project, as originally defined, was not feasible. Moreover, they reported that ‘in prevailing circumstances, a comparative trial of this type is not a productive procedure for assessing this form of psychotherapy.’ [7]
The background to the New South Wales Section of Psychotherapy outcomes project
Circumstances in Australia have, until recently, favoured IDP. One of these has been a significant Government (Medicare) rebate towards fees paid for the services of psychiatrists in private practice, which has brought IDP within the financial reach of most Australians. Most of the psychiatrists who provide IDP in New South Wales work in Sydney, and the great majority of these are members of the Section of Psychotherapy of the Royal Australian and New Zealand College of Psychiatrists. In 1994 an internal survey within the Section, which has about 150 members, indicated that there were probably more than 40 psychiatrists who were accredited by various training institutes to conduct IDP.
From its creation in 1989, the Section of Psychotherapy had a major concern that the Federal Government would restrict the Medicare rebate for IDP. Medicare officials had repeatedly asked the College for evidence favouring the efficacy of IDP, including local outcomes data. In April 1994, in response to this pressure, and as part of its ongoing scientific program, the Branch Committee of the New South Wales Section of Psychotherapy held a scientific meeting titled Outcomes in Psychotherapy. At this meeting there was a call for a collaborative effort engaging IDP practitioners, general and academic psychiatrists in the execution of a controlled trial of IDP. There was agreement that the experimental procedure should resemble IDP as it was being locally practised. Additionally, it was agreed that the trial should employ clinically meaningful outcome measures and monitoring of process phenomena. Towards the end of this meeting, 19 IDP practitioners offered to act as therapists for the experimental procedure.
Brief history of the psychotherapy outcome project (April 1994 to February 1997)
Following this first open meeting, a research subcommittee was asked to devise and oversee the proposed trial. During the ensuing 34 months, five complete draft protocols were circulated for discussion, amendment and ratification. Three further open meetings (22 March 1995, 30 August 1995 and 28 February 1996) encouraged the fullest participation of the members of the Section in the project.
By mid-1996 it seemed that the only remaining barrier to the commencement of the trial was uncertainty regarding the number of volunteer therapists. The project had failed to attract more than the original 19 IDP practitioners and about 10 other psychiatrists offering to perform the control procedure. The research subcommittee felt that limited therapist numbers would, at best, assure the treatment of 25 subjects by each procedure. The subcommittee was therefore confronted with the following question: would 25 subjects in each cell provide sufficient power to answer essential questions regarding outcomes? In June 1996 the subcommittee sought an answer to this specific question by consulting a data analyst. The data analyst made projections based on outcomes data from the Stevenson and Meares [9] study (n = 30). He projected various sample sizes needed to detect significant differences in effect sizes between subject groups. He concluded: ‘Clearly the proposed n = 25 (cohort = 50) is too small unless you believe that group differences (experimental vs control) would be quite substantial.’
The subcommittee decided, nevertheless, to commence the admission of trial subjects in the hope that more therapists would volunteer as the project got under way. However, in August 1996 the new Federal Government made an unheralded announcement that it would limit the availability of IDP by imposing new limits on the Medicare rebate. As a group, IDP practitioners immediately focused attention on the threat to their professional practices and were distracted from the outcomes project. In February 1997 the Section of Psychotherapy Branch Committee minuted that the project had been discontinued.
The final research protocol (August 1996)
Before the decision to terminate the project was made, the subcommittee had developed the final research protocol. Subjects would be referred to the study from two of Sydney's tertiary mood disorders clinics. A screening psychiatrist would admit subjects according to the following inclusion criteria: age 22–55, DSM-IV diagnosis of major depression and/or dysthymia associated with one or more Cluster B or C personality disorders, failure to remit from depression following a course of cognitive-behaviour therapy and a month of antidepressant therapy, and willingness to accept random allocation and other conditions of the research protocol. Exclusion criteria would be gross factors that would render subjects unsuitable for psychotherapy such as psychosis, severe substance abuse or severe antisocial traits. Subjects who were not accepted for the study, or who withdrew from the study, would return to the referring tertiary mood disorders clinic for further treatment or triage. The experimental or control therapists could initiate psychoactive drug treatments, inpatient admissions or second opinions. Informed consent would be mandatory.
The experimental procedure involved twice-weekly IDP. The control procedure involved 16 weeks of interpersonal therapy (IPT) followed by good clinical care. These procedures would terminate at 2 years. However, psychotherapy could commence anew after the completion of follow up (36 months). There would be no standardisation of the experimental procedure but therapists would be encouraged to participate in peer-review groups. Therapy processes would be audiotaped for later process analysis. Assessment measures would include a DSM-IV diagnosis [12] by a screening psychiatrist and various measures by a research psychologist. The latter would record: (i) personality disorder diagnosis by the criteria of the Personality Diagnostic Questionnaire – Revised form (PDQ-R) [13]; (ii) symptom measure such as the Symptom Checklist: 90-R (SCL 90-R) [14]; (iii) mood by the Beck Depression Inventory [15]; (iv) social competence by the Interview Schedule for Social Intervention [16]; and (v), for later analysis, any clinical contingencies or deviations from the protocol.
The research psychologist would record outcome measures at 3, 6, 9, 12, 15, 18, 24 and 36 months. These would include target symptom self-ratings, behavioural measures similar to those employed in the Stevenson and Meares [9] study and checklists maintained by therapists regarding health resource utilisation. Repeated measures analyses would be used to explore differences between the scores.
Method
To obtain the opinions of the members of the research subcommittee, a questionnaire was completed by each of them in late-1998. The questionnaire had three sections.
Perceived obstacle to the successful completion of naturalistic RCT of IDP questionnaire responses: item evaluation (n = 8)
Second, the questionnaire solicited additional comments regarding obstacles to the completion of a RCT of IDP and publication of meaningful results in a peer-reviewed journal.
Third, the questionnaire asked: ‘If the Federal government had not restricted benefits in August 96, do you think the Project would have proceeded to the publication of meaningful results in a peer-reviewed journal?’.
Results
Responses to 17 listed items according to degree of obstacle
Table 1 records the responses of the subcommittee (n = 8) to the 17 items in the first section of the questionnaire. The table lists items according to the number of respondents who considered that the nominated obstacle posed a significant threat, that is, raw scores of 1, 2 or 3. The table does not display the responses signifying ‘little threat’ or ‘no obstacle’.
The concordance between the respondents to the questionnaire (n = 8) was calculated to provide a Kendall coefficient of concordance (W). This proved to be very low (W = 0.042) indicating little agreement in their rating across the 17 ranks of items. However, at face value there was considerable agreement between the raters regarding items at the top and bottom of the scale.
Volunteered comments regarding major obstacles
There were 19 volunteered comments from the eight respondents. These concerned the following issues: (i) obtaining and maintaining the full cooperation of sufficient trial therapists, and attracting additional trial therapists (5); (ii) obtaining sufficient volunteer trial subjects (3); (iii) overcoming difficulties with standardisation of treatment and inter-rater reliability (3); (iv) problems regarding lack of funding (2).
In addition there was one response relating to each of the six following issues: the choice of the experimental procedure, choice of study population, leadership, finding suitable outcome measures, difficulties in obtaining follow-up data, difficulties in the phase of statistical analysis and write up.
Was the project feasible?
In response to the question regarding project feasibility, six of the eight subcommittee members (75%%) recorded no. One of the two who answered yes annotated ‘more likely than not’.
Discussion
The most dramatic finding of the survey was the response by seven of the eight subcommittee members that, even if funding had not been restricted, the project would probably not have been successful. This is a reversal of a presumed earlier belief that there was a prospect of success.
Review of the questionnaire item responses and the volunteered comments showed that the most vexatious issues identified by a majority of the subcommittee fell into six main groups.
Issues related to the choice of the experimental procedure and its standardisation
In response to the questionnaire, the item relating to the standardisation of procedures was accorded highest priority (Table 1). This priority was supported by three volunteered narrative comments. Also very high on the item choice list were choice of experimental procedure, choice of study population and naturalistic goal. In the context of the main goal of the project, the authors consider all the above to be closely related issues.
From its inception, the goal of the project was intended to be an examination of IDP as it was locally practised. However, Sydney's IDP practitioners work in independent private practice and employ techniques according to their prior experience and training. Any attempt to reformulate or standardise their activities would be inimical to their concepts of therapy. Moreover, there are several schools of IDP in Sydney, each with differing attitudes toward the use of key techniques, for example, the use of transference interpretations.
Several of the academically oriented subcommittee members argued that the experimental therapy could be neither properly studied nor subject to replication trials, unless its procedures were accurately defined and followed. After months of discussion, the subcommittee members arrived at a decision that there should be no attempt to standardise, but that therapy sessions would be audiotaped for later process analysis in a semi-independent study.
In conformity with the naturalistic goal, it was initially argued that the subjects should be drawn from the usual referral base of IDP therapists. However, because this might interfere with the referral procedures and practices of IDP therapists, this option was abandoned. The next suggestion was to recruit from the clientele of tertiary referral clinics. Sydney has such clinics for patients with eating disorders, anxiety disorders and mood disorders. The subcommittee sought to recruit subjects with disorders that commonly led them to attend IDP therapists or to present at tertiary clinics. Patients with chronic depression associated with certain character disorders best met this criterion. The Directors of two of Sydney's mood disorders clinics offered to assist the project by providing a referral base. It seemed both clinically reasonable and ethically laudable to study subjects who, because of disturbed character traits, seemed resistant to standard treatments for depression. The research subcommittee asked the Board of Research of the College for their approval of the project. This approval had not been received at the time the project halted.
Obtaining and maintaining the cooperation of trial IDP therapists
Five of the eight members volunteered (narrative) concerns that the project might not attract and maintain the cooperation of sufficient therapists to achieve the statistical goals of the trial. Some comments relating to ‘continuing cooperation from therapists’ and ‘maintaining enthusiasm and momentum for project among therapists’. The questionnaire had not included an item regarding continuing cooperation of the trial therapists, but four ‘item’ respondents had indicated concern about ‘obtaining sufficient experimental therapists’ and two of these judged that this problem ‘might prove insurmountable’ (see Table 1).
Lack of commitment by IDP therapists was much discussed by the subcommittee. The subcommittee failed to attract additional therapists by advertisements in College newsletters and by mailed invitations to all Section members. The subcommittee considered asking the volunteer IDP therapists to concurrently treat several subjects but decided that this would cause excessive disruption to their clinical routines.
Termination of therapy at 24 months
This issue was accorded second highest priority as obstacle in the ‘item choice’ list. Representatives of the IDP practitioners stressed on several occasions that they did not work towards assigned termination dates and that therapy usually lasted well beyond 2 years. At the open meetings, various IDP practitioners had expressed lack of enthusiasm for the project if it would require the termination of therapy at 2 years. The more academically orientated members of the subcommittee insisted that outcome measures would be meaningless unless the trial procedures had actually ceased. One of these added that the trial would cause an intolerable burden on professional resources if it should exceed 2 years. Those proposing a fixed termination date at 2 years prevailed.
Obtaining funding
When questioned about problems obtaining funding, four of the members (75%%) chose the item affirming that this obstacle ‘might eventually prove insurmountable’, but none responded that it posed a ‘definitely insurmountable’ obstacle. There were two additional volunteered comments indicating concern regarding funding. However, seven members asserted that the project would probably have failed in spite of assured funding.
Immediately after the first open meetings, the Section had sought and received confirmation of sufficient College funding to cover ongoing expenses for 2 years. This would allow funds to employ a research assistant (i.e. a trained research psychologist) and funding for minor expenses. However, the major cost of the project would be for the time of the participating therapists. Medicare officials informally advised that study subjects would be entitled to rebates if these services were standard medical treatments. The subcommittee assumed that this resource was secure but, in August 1996, the Government unexpectedly restricted funding. The research subcommittee then considered seeking direct funding for an outcomes trial by the Health Insurance Commission or by application for a research grant from the National Health and Medical Research Council. However, the attention of the volunteer IDP practitioners was then so distracted that the project stalled.
Choice of control procedure and engaging sufficient control therapists
Responses to the item choice segment indicated (see Table 1) that five members thought that the choice of a control procedure would pose some degree of obstacle and five thought that there might be insufficient control therapists. There were additional comments regarding the enlistment of trial therapists, but it was not clear whether these referred to the experimental or control groups.
In search of a control procedure, the subcommittee first canvassed the use of a protocol of ‘good clinical care’ [3] designed to provide a frequency of attention approximating that provided for the experimental group. However, some members argued that a greater degree of standardisation, preferably employing a manual, was desirable. In early 1996 interpersonal therapy (IPT) was proposed as the control therapy. This therapy had, to a considerable extent, been standardised and is easily taught [17]. In favour, both clinically and ethically, of its employment, it had been shown that IPT was efficacious in trials involving patients suffering from chronic depression [18]. There were three arguments against IPT. First, there were probably too few trained IPT therapists in Sydney. Second, although patients with personality trait disturbance [18] had been among those successfully treated with IPT, it had not been designed for this purpose. Third, the apparent efficacy of IPT as a treatment for cases of major depression and dys-thymia might limit, if not negate, its usefulness as a control procedure.
The advice from the statistical consultant highlighted this particular issue: ‘The more the IPT group changes, the smaller the difference in effect size units between the two groups, and hence the larger the sample size required’, and: ‘It is possible that the study subjects, all suffering from a DSM-IV personality disorder, might have responded better to IDP than IPT, but there does not appear to be any empirical basis for such a prediction.’
In spite of these disadvantages, the subcommittee chose IPT as the control procedure. In early 1996, when IPT was finally designated the control procedure, several of the original 10 volunteer control therapists agreed to accept training and supervision in IPT. The research subcommittee thought it would be reasonable to ask control therapists to concurrently treat several subjects, and hoped that additional IPT therapists might be attracted or trained as the project gained momentum.
Obtaining sufficient trial subjects
Because the project did not proceed to an intake of subjects, the questionnaire did not include an item directly concerning this issue. However, three members volunteered their concern that there might be too few volunteer subjects to sustain the project. There was little, if any, basis for predicting the frequency at which subjects might volunteer for this trial. Moreover, high attrition rates are usually reported in psychotherapy trials [19].
Future options
Six obstacles have been listed in approximate order of perceived difficulty. The authors offer three amendments to the protocol that might allow a successful trial.
The original aim, to conduct a naturalistic study of IDP, was later contradicted by inclusion in the final protocol of the requirement that therapies would terminate at 2 years. In the current paper, we provide evidence that this caused loss of enthusiasm for the project among IDP practitioners and contributed significantly to its failure. We therefore now advocate the acceptance of a protocol permitting therapy to proceed for as long as the therapists judge it to be clinically desirable. Follow up would commence from the date of termination. The pursuit of this option would lead to a study of long duration, and might be arduous for some researchers. However, the resulting trial should be more acceptable to the IDP practitioners and would have greater external validity than that described in the August 1996 protocol. The feasibility of this option is supported by the existence of a multicentre study in Norway [Harvik OE, personal communication, 1999] which is comparing the process and outcomes of various psychotherapies without fixed limits to their duration.
Our second recommendation relates to the choice of the control therapy and the need to attract and retain a larger number of control therapists. For several reasons, we advocate returning to an earlier considered option to employ ‘good clinical care’ [3] as the control therapy. First, the abandonment of IPT might improve the opportunity to demonstrate statistically significant outcomes benefit following the IDP procedure. The research team would then be forced to choose to either invent, or adopt, a standardised protocol of ‘good clinical care’, or to continue to eschew standardisation, but to analyse the processes of the control therapy alongside those associated with the experimental procedure. Second, because the control therapists would require little additional training, general psychiatrists might be more easily enlisted into the trial.
The authors' third recommendation relates to extending the trial to Melbourne and, possibly, to other Australian capitals. Although new organisational and ethical issues would result [20], it is reasonable to predict that by doubling the population base one might double the potential numbers of therapists and trial subjects. This should enhance the capacity of the trial to demonstrate a statistically significant outcomes difference between the procedures.
The experience of this project might also lead one to a different conclusion: that it is premature, at this stage in the history of psychotherapy research, to attempt to conduct a RCT of IDP. Outcomes researchers might turn their attention towards an alternate form of evidence for the efficacy of IDP. For example, a large series of single-case design studies [21] might provide outcome evidence at a level of plausibility that is acceptable to many critics. However, the researchers would still have to contend with problems regarding the standardisation of treatment procedures and of the provision of appropriate control procedures.
Footnotes
Acknowledgements
The authors wish to acknowledge the other members of the Research Subcommittee of the New South Wales Section of Psychotherapy: Russell Meares, Kevin Vaughan, Eng-Kong Tan, Ron Spielman, Peter Hoey and Kay Wilhelm. We also thank Dusan Hadzi-Pavlovic for data-analytic advice.