When Research Meets Reality–-Lessons Learned from a Pragmatic Multisite Group-Randomized Clinical Trial on Psychosocial Interventions in the Psychiatric and Addiction Field

Abstract

Research on treatments for patients with co-occurring psychiatric and substance use disorders is of core importance and at the same time highly challenging as it includes patients that are normally excluded from clinical studies. Such research may require methodological adaptations which in turn create new challenges. However, the challenges that arise in such studies are insufficiently discussed in the literature. The aim of this methodology paper is, firstly, to discuss the methodological adaptations that may be required in such research; secondly, to describe how such adaptations created new challenges in a group-randomized clinical trial on Integrated Treatment amongst patients with co-occurring psychiatric and substance use disorders. We also discuss how these challenges might be understood and highlight lessons for future research in this field. Trial registration: NCT00447733.

Introduction

Research on treatments for patients with co-occurring psychiatric and substance use disorders is of core importance. Continuing emphasis on evidence based medicine calls for empirically validated treatments based upon rigorous treatment research methods.^1,2 The randomized controlled trial (RCT) is considered the best clinical research method mainly due to its ability to maximize internal validity which enables studies to attribute differential outcomes to the experimental manipulation rather than other causes.^1,3 However, critical views have emerged in recent years. Whilst the traditional RCT is an excellent method in pharmaceutical trials on homogenous groups of patients, it is not as feasible in trials with complex psychosocial interventions on heterogeneous groups of patients.^4,5 The study of complex interventions in non-selected clinical populations is difficult and challenging. This is especially true for studies in the psychiatric and addiction field which include patients that are normally excluded from clinical studies.^6–8 Such research may require methodological adaptations which in turn create new challenges. This makes a systematic use of such research methodology important and evaluation reports on projects based on these methodologies imperative.

The aim of this methodology paper is, firstly, to discuss the methodological adaptations that may be required in clinical research on complex interventions in non-selected clinical populations; secondly, to describe how such adaptations created new challenges in a randomized clinical trial on Integrated Treatment. We also discuss how these challenges might be understood and highlight the lessons important for future research in this field.

Clinical Research Methods and Adaptations

The Randomized Controlled trial

One main feature of the RCT is that it provides strong internal validity through strict inclusion and strict exclusion criteria. This strong internal validity, however, comes at the expense of poorer external validity which means reduced generalizability of the results to the heterogeneous groups of patients and settings in everyday clinical practice.^6,9,10

Another prerequisite for establishing strong internal validity is a strict manualized treatment conducted by highly trained and specialized therapists. In everyday clinical practice, however, clinicians are facing a wider range of disorders which requires a more eclectic approach tailored to meet the differential needs of the individual patient. Additionally, the treatment is typically provided by health-care professionals with different experiences and training.^9–11 It has often proved difficult to implement manualized treatment approaches as this presupposes basic skills, time consuming training and highly motivated therapists. ¹²

Another main feature of the RCT design is that it intends to establish equal groups in regard to known and unknown confounding factors by randomization. However, this presupposes a sample size that is large enough compared to the number of variables, ⁴ which might not be the case in many randomized clinical trials as difficulties in recruitment are more the rule than the exception.^13–15 It is estimated that 19% of all worldwide pharmaceutical RCTs fail to enroll any patients, and that 90% fail to enroll the target number of patients within the target amount of time. ¹⁶ There is no reason to believe this would be different for studies in the psychiatric and addiction fields as these studies include patients that are normally excluded from most RCTs.^6–8

Another feature of rigorous RCTs is blinding of patients, therapists and researchers to treatment allocation. However, in studies with complex interventions blinding is often difficult or even impossible to accomplish. This means that patients might withdraw from the study when allocated to a less preferred treatment, and by this pose a threat to the between-group equivalence.^1,17 Additionally, in clinical practice the choice of treatment is usually a result of collaboration between the patient and the therapist. ⁹

The Pragmatic RCT

These discrepancies between the “ideal research” and the “real-world” settings have motivated a distinction between efficacy and effectiveness studies along with a search for complementary research methods. Efficacy studies investigate the effect of specific treatment factors under ideal conditions, like strict RCTs with a strong emphasis on internal validity. Effectiveness studies, on the other hand, with a stronger emphasis on external validity, investigate the effect of treatments under “real-world” conditions.^18,19 When it comes to complementary research methods, these often involve modifications of the traditional methods. All such modifications, however, create new challenges.

One of these complementary research methods is the pragmatic RCT. ⁹ The key aim of the pragmatic RCT is to reflect the heterogeneity of patients in clinical practice by using broader inclusion criteria and keeping exclusion criteria to a minimum, thereby enhancing the generalizability of the RCT to patients and settings in everyday clinical practice. Consequently, the pragmatic RCT can be regarded as lying in between efficacy- and effectiveness studies. The pragmatic RCT is also more flexible in defining the intervention and tends not to use a treatment manual. Usually the concern of the traditional RCT is to evaluate specific active ingredients in the treatments provided. The pragmatic RCT, investigating more complex interventions, is more interested in examining each treatment as an entity or a “black box”. ⁹ Such interventions might be impossible or even unethical to compare with placebo. An alternative would be to compare the experimental treatment to “treatment as usual” (TAU). This means that the patients in the control group receive the treatment they would normally have received. However, TAU is a difficult term to define because it depends greatly on the preference, skills, knowledge and resources of the therapists delivering it. ⁹

Multi-Center trials

A well-known challenge in conducting trials evaluating complex psychosocial interventions is that the expected effect-sizes tend to be small, especially when the control condition is TAU or another active treatment which will also constitute a treatment effect. Frequently, such studies require large sample sizes to ensure sufficient statistical power. Additionally, inclusion, treatment adherence and follow-up is often challenging when studying patients with multiple disorders. This renders studies vulnerable to type 2 statistical error. One way of meeting this challenge is to use a multi-center design. If no single center has a sufficient case load, trials including patients from several centers might provide an adequate study population. In multi-center studies, the randomization of participants might be done on an individual or center-level, ie, the individually randomized group treatment trial (IRGT) or the group-randomized clinical trial (GRCT). ²⁰ Individual randomization is usually applied when the intention is to study effects at the individual level. The group-randomized clinical trial is useful in examining the context in which the therapy is given or the differences of interventions in specific groups (schools, hospitals, etc.). ²⁰ Additionally, GRCTs may reduce the risk of contamination between treatment conditions. In treatment research on psychosocial methods it is difficult to hide or mask the different methods that are to be randomized. This creates a risk of contamination of knowledge between the therapists and patients in the intervention and control groups. In the GRCT the different centers are allocated to the different treatment conditions as a whole. All the therapists in one center will therefore belong to the same treatment condition, minimizing the risk of contamination. ²¹

However, multi-center trials represent organizational, administrative, economic, treatment fidelity and statistical challenges. In order to secure inclusion, fidelity, protocol adherence and data quality, it is essential to have a trial administrator at each center managing and monitoring the trial.^1,22 Additionally, one has to consider clustering of participants in both types of multi-center trials. In the IRGT clustering may develop during the course of the trial, whilst one has to consider clustering from the start of the trial in the GRCT. This clustering threatens the statistical assumption of independent observations and should be accounted for by hierarchical statistical analyses.^23–25 Further, such intra-class correlations may reduce statistical power and should be compensated for in protocol sample size calculations.^26–28

Methodological Choices Made in a Trial on Integrated Treatment

This study was designed to investigate the effectiveness of Integrated Treatment (IT) amongst patients with anxiety and/or depression in addition to substance use disorders (SUD) in psychiatric outpatient clinics of Community Mental Health Centers (CMHCs) in Norway. The control condition was treatment as usual (TAU). IT is a combined treatment for both the psychiatric and the substance use problems of the patient. The main components of IT are Motivational Interviewing and Cognitive Behavioral Therapy along with a more active and comprehensive treatment approach. ²⁹ The treatment was not manualized, but a descriptive clinical guideline manual was provided and used in staff training. Three to five therapists at each clinic in the Intervention Group received training in IT. The clinicians in the control group were promised teaching of the experimental intervention when the study was over.

To obtain external validity, we chose a pragmatic RCT design. In order to acquire the calculated sample-size, we chose to run a multi-center study. As contamination of knowledge between therapists and patients between groups was an obvious risk, we decided to randomize on center-level. Blinding was judged impossible and therefore the allocation was open at inclusion. In order to secure inclusion, treatment fidelity, protocol adherence, and data quality, we chose to train and pay one therapist at each center in a 10% position as a local trial administrator.

The CMHCs were randomized by draw at the center-level. The groups were stratified with respect to urban or rural catchment areas. Five centers were drawn to the intervention group and four centers to the control group.

All new referrals to the psychiatric outpatient clinics during the inclusion period were to be screened with the Alcohol Use Disorder Identification Test (AUDIT) ³⁰ and the Drug Use Disorder Identification Test (DUDIT). ³¹ Those who scored above the cut-off levels associated with abuse or dependence were subsequently referred to the local trial administrator for assessment of the inclusion criteria. The diagnostic inclusion criteria were assessed with the Structured Clinical Interview for the DSM-IV diagnoses, ie, the SCID 1³² and the SCID 2. ³³

The local trial administrators had 3 days of training in how to run the project and in using the evaluation instruments including the SCID and the Addiction Severity Index (EuropASI), chapter E.^34,35 The training on the research instruments consisted of lectures, clinical examples and practicing on case vignettes.

The included patients were assessed at baseline, 6 and 12 months. According to the local trial administrators, the baseline interviews took between 2–5 hours, and the follow-up interviews took between 1–3 hours each.

According to power calculations 108 patients were needed. As we expected between 20 and 30 percent to drop out from treatment and assessments, we aimed to include a total of 150 patients in the study (ie, N = 75 in each group). Thus, each of the 9 CMHCs were expected to include between 15 and 20 patients. The study was approved by the Norwegian Regional Ethics Committee.

Challenges Experienced

Center recruitment

Considerable time was spent on recruiting CMHCs into the trial. Thirty five CMHCs from 3 out of 5 Regional Health Trusts were invited. More information was provided through meetings at the CMHCs that responded positively to the invitation. Most of the CMHCs were positive with respect to gaining knowledge about the effects of the intervention, but reluctant in regard to the workload demanded by such studies. In the end, 10 CMHCs agreed to participate in the study. However, one center withdrew just before randomization, leaving us with a total of 9 centers. The recruitment process significantly delayed the initiation of the project and put the trial behind schedule from the beginning.

The contracts between the participating CMHCs and the project management would have to be worded differently between the experimental and the control group. We therefore decided to make the written agreements with each participating center after treatment allocation. This turned out to be a mistake as one of the centers allocated to the control group withdrew 2 months into the project. They stated this was due to workload and staffing problems. During the course of the trial, another center in the control group did not manage to include any patients. As failing to include the target number of patients would mean risking type two statistical error, several attempts were made to include additional centers to the control group even though it would have compromised the RCT design. Towards the end of the study period, we managed to include two more centers which, however, did not manage to include any patients. This left us with 5 centers in the Intervention Group (IG) and two centers in the Control Group (CG) (Fig. 1).

Figure 1

Flowchart of CMHCs on centre level.

Patient recruitment

A total of 5323 patients were referred to the CMHCs during the inclusion period (Fig. 2). Only 35% of these patients were actually screened with the AUDIT and the DUDIT (CG: 50%, IG: 27%) with a range from 17%–64% between CMHCs (data not shown). Of these 1867 patients, 88 (5%) handed in a blank form (CG: 7%, IG: 3%). All of the blank forms from the CG came from the CMHC that did not manage to include any patients and constitutes 20% of the screened forms from that CMHC.

Figure 2

Flowchart of patients in the intervention group and the control group.

Eighteen per cent of the screened patients scored above the cut-off level of the AUDIT and/or the DUDIT (CG: 10%, IG: 27%), ranging from 5%–34% between centers. Only 31% of those that scored above the cut-off level were actually referred to the local trial administrator for the baseline evaluation (CG: 33%, IG: 30%), ranging from 0%–50% between centers. The main reason for not being referred was “no oral consent to participate in the study” (CG: 32%, IG: 59%). Another reason was that the clinicians deemed the patient's motivation for substance use treatment as being too low (IG: 0.04%) or that the patient was not in need of such treatment (CG: 14%). The latter reason was only stated by the CMHC that did not manage to include any patients.

In the end, only 81 patients were included in the study. This was 4% of the total number of screened patients and 24% of the patients that scored above the cut-off level of the AUDIT and/or the DUDIT (CG: 2%, 25%, IG: 6%, 23%), ranging from 0%–7% and 15%–40% respectively, between centers. After inclusion, 2 patients in the CG and 3 patients in the IG withdrew their consent, leaving us with 21 patients in the CG and 55 patients in the IG. This means that the number of participants available for the intention to treat analyses were 76, ranging from 6–16 participants between centers.

After inclusion, two patients in the IG never met for treatment, 11 patients received less than 5 sessions (CG: 3, IG: 8) and 14 patients never returned for follow-up interviews (CG: 3, IG: 11) (Data not shown). This left us with 56 completers, defined as receiving 5 sessions or more and meeting for at least one follow-up interview (CG: 17, IG: 39), ranging from 4–14 between centers. Only 48 patients had at least 5 sessions and met for both follow-up interviews (CG: 15, IG: 33) (data not shown). There was no significant difference in the number of completers between the IG and the CG (χ ² = 0, 80; df = 1; P = 0, 374).

There were three main differences in the referral policy of patients with SUD across centers. At one extreme, the centers referred all patients with SUD to the addiction outpatient unit except patients with co-occurring schizophrenic spectrum disorders. Other CMHCs treated patients with co-occurring SUD at the psychiatric outpatient clinic as long as the SUD was not severe (harmful use and abuse only). At the other extreme, one CMHC treated all patients with mental illness at the psychiatric outpatient clinic regardless of co-morbid SUD. We found that the CMHC that failed to include any patients was amongst the four CMHCs that referred all patients with co-morbid SUD to the addiction outpatient unit. The one CMHC that did not refer patients with SUD elsewhere was the only CMHC that managed to include the target number of patients during the initial inclusion period. Otherwise, we did not find any relationship between the center referral policy and the number of people that scored above the threshold level of the screening instruments (data not shown).

The inclusion period proved to be insufficient. After the planned inclusion period of 9 months only one CMHC had managed to recruit the target number of patients and only 55 patients were included in total (CG: 14, IG: 41). The remaining CMHCs and their coordinators were willing to continue recruiting for another 3 months. Nevertheless, after those 3 months only 74 patients were included (CG: 18, IG: 56) (data not shown). The remaining CMHCs were asked to continue recruiting until the target number of patients was included or until the end of the year (another 9 months). All CMHCs from the CG and one CMHC from the IG accepted this. Still, patient enrolment was slow and one CMHC (CG) did not manage to recruit any patients at all by the end of the additional year.

In the CMHC that managed to include the target number of patients during the initial recruiting period, another problem arose. They had so many eligible patients that it exceeded the treatment capacity of the clinicians that were trained in the experimental intervention. They therefore stopped recruiting when the minimum number of required patients was included, 2 months ahead of schedule. To address this problem we chose to train two additional therapists at that clinic.

Follow-up interviews

The next challenge was to get patients to return for the follow-up assessments. One CMHC (IG) managed to complete follow-up interviews for all patients at 6 and 12 months. At the other extreme, one CMHC (IG) lost 6 out of 14 patients at both follow-up sessions. The other CMHCs lost 1–2 patients each from both follow-up sessions (data not shown). In order to reduce drop-out from the follow-up assessments, the patients were offered €25 for each follow-up session as a compensation to cover their travel expenses. The Regional Medical Ethics Committee did not accept a higher monetary incentive, to prevent it from becoming an incentive for participation on its own.

Discussion

This study illustrates the difficulties experienced in a multi-site group-randomized clinical trial with complex interventions in a non-selected clinical population. In our study the problems arose early while recruiting centers. Further problems arose while recruiting patients and encouraging them to return for their follow-up assessments.

Recruiting centers

The first time-consuming challenge was recruiting centers. Thirty five CMHCs were asked to participate and only 9 accepted. The main reasons for declining participation were challenges in the clinics with respect to time and financial demands. Other reasons were shortage of specialists, lack of a local supportive infrastructure, inadequate research training, and data collection challenges. It seems that many clinics value research and want to contribute to it, but few have the resources needed to follow it through. This was clearly demonstrated when one of the participating CMHCs withdrew 2 months into the project.

Researchers have a formidable challenge in both motivating centers to contribute to research and to keep the burden of participation at an acceptable level. If the burden of participation is too high, the study will run the risk of collecting incomplete and missing data. To accomplish good motivation, adherence and implementation of research methods in the clinic, it is important to involve the clinics in the project at an early stage and to let them contribute actively in the planning and execution phases of the project.^36,37 This will both enhance the centers’ feeling of “ownership” towards the study, and also contribute to a design that is in accordance with the clinics’ needs and workload. However, making contracts with the clinics at an early stage might be premature due to high application refusal rates.

Some of the clinic leaders stated that they would not want to participate in the study if they were allocated to the control group, as it would not be worth the effort for them. This highlights further the importance of including the centers at an early stage of the study planning process and to use strong acknowledgement and reinforcement strategies. Strong reinforcements are essential to make the effort of participation worthwhile regardless of allocation. One reinforcement strategy could be to give the co-workers at the centers a role that fulfills the Vancouver requirements for co-authorship. ³⁶ In our study the control group centers were promised ownership of all data from their own center along with treatment manuals and teaching in the experimental intervention after the study.

The local trial administrators have an important role in the study and it is essential to select appropriate people for these positions. However, we did not have the luxury of selecting between several candidates for the job. At most CMHCs it was difficult to find even one single interested person. Thus, the local trial administrators were a mix of people with varying education, training, interests, motivation and time available for the trial. Furthermore, the more centers that are involved in a multi-site trial and the larger the geographical area they are spread over, the greater the responsibility and project work-load for the local trial administrators. This could create differences in how the data is collected between centers.

Recruiting patients

The therapists at the CMHCs were supposed to screen all new referrals with the AUDIT and the DUDIT. However, only 35% were actually screened. According to the local trial administrators this was due to several factors. Some therapists felt the screening was a new burden on their already busy schedule, some felt insecure addressing the issue of alcohol and drug use and that it was particularly awkward to introduce this subject during the first session and some were concerned it would threaten the therapeutic alliance to address such a delicate issue. In one center the leader had resigned shortly after the project started. His successor had little interest in the project and did not encourage or support the staff to continue screening. Most local trial administrators, however, commented that they found the screening relevant and some of the clinics even wanted to keep it as part of their routine after the project was over.

Our understanding is that these problems were insufficiently considered. Therapists are often uncomfortable with addressing the issue of alcohol and drug use with the patient and need extra coaching.^38–42 Unless the importance of screening is strongly emphasized by the local trial administrator and/or the research group and followed by strong support from the clinic management, such new routines are easily forgotten. Most clinical research projects require the establishment of new routines to gather the required data for the project. Implementation of new routines and treatment methods has long shown to be a challenge in most settings and require formidable resources on many levels.^43,44 Ideally, such routines should be well established before the start of the study. ³⁶

An unforeseen problem was the very low prevalence of patients with problematic alcohol and drug use identified by the AUDIT and the DUDIT in some of the CMHCs. From previous prevalence findings of SUD in psychiatric outpatient clinics, one would expect this number to be higher.^45,46 This might be explained by different referral policies between centers. During the planning phase of the project new national mental health services guidelines were published. These guidelines were interpreted differently by the centers. One interpretation was to establish separate addiction outpatient units as part of the CMHC, rendering the psychiatric outpatient clinic of the CMHC almost deplete of patients comorbid of mental and SUD.

The therapists were asked to refer all patients that scored above the cut-off level of the AUDIT and the DUDIT to the local trial administrator for baseline evaluations. However, only about a third were actually referred. The main reason for this was that the patients did not want to participate in the study. In the Intervention Group, participation in the study could imply a change of therapist to one that had learned the intervention. This, however, does not explain why the patients in the Control Group refused to participate. This could be due to how the therapists presented the project to the patients, perhaps reflecting their own attitudes towards the project.^44,47 At one CMHC the clinicians deemed 13 patients as not needing treatment although they scored above the threshold level of the screening instruments. Considering this was the CMHC that did not manage to include any patients, this could mean that the project was not well enough implemented in that clinic. Several studies have stated that a prerequisite for implementing a trial into a clinical setting is a supportive clinic management that motivates the clinicians to participate.^43,44

We soon realized that inclusion was happening too slowly, so we contacted the CMHCs and urged their management to make sure that the psychiatric outpatient clinics would take responsibility for their share of patients with co-occurring SUD instead of referring them elsewhere. This created several dilemmas. In the Control Group it would be unethical to keep the patients with co-occurring SUD at the psychiatric outpatient clinics if they did not feel they had the competency to treat them there. On the other hand, different policies in handling referrals in regard to co-occurring SUD between the IG and the CG would bias our sample. Additionally, our Intention to Treat (ITT) design would keep the patient data in the group they were allocated to even though the patients were referred to another treatment facility (eg, addiction treatment) after inclusion in the study. This means that if the CG would refer their patients to an addiction treatment facility, it would make it difficult to conclude whether there were any real differences in the interventions given between the IG and the CG.

To address the slow inclusion rate, we extended the inclusion period by one year. Extending the inclusion period is problematic in several ways. It exceeds the original funding and increases the risk of having to deal with therapist turn-over.^48,49 During the first year of the project, we had to train 4 additional therapists. We experienced that the extended inclusion period did not increase inclusion numbers to a great extent. During the initial inclusion period we included about 37% of the calculated sample size needed. After the first 3 months of the extended inclusion period we had included 49% of the calculated sample size, and we ended up with only 51% (ITT) of the calculated sample size after an additional year of recruiting. This means that the cost of extending the inclusion period is greater than the benefit.

Follow-up assessments

Not unexpectedly, follow-up proved challenging. As the first CMHC to finish including patients experienced great difficulties in reaching the patients for follow-up assessments, we applied for additional funding to pay the participants a symbolic amount for their participation in the study. This necessitated additional approval from the Regional Medical Ethics Committee. We are unsure if this payment had an effect on follow-up rates. The additional strategies employed seem more important. Many of the local trial administrators used a lot of creativity in reaching the patients for their follow-up assessments, such as interviewing patients that had moved to other parts of the country by phone and mail and making appointments for interviews at the patient's workplace when the patient was unable to visit the clinic. Our experience is that the local trial administrators who put more effort into the task than was expected of them and who where the most creative, managed to reach the most patients for their follow-up assessments. Our interpretation is that this type of research puts heavy demands on the participating centers and local trial administrators, and that these demands and costs are often underestimated. The result might be under-funding and the use of inadequate strategies which could easily endanger multi-site studies. Securing adherence is a complex issue that requires multiple and comprehensive strategies at different stages of the trial.^50,51 This requires the researchers to be directly involved and integrated into the research work of each center, thereby improving data quality and adherence.^43,44

Ethical considerations

Commonly, clinical trials encounter problems in recruitment, follow-up and adherence which may reduce the studies’ probability of demonstrating an effect (type 2 statistical error). This is ethically problematic because one risks wasting financial and human resources. The Medical Ethics Committees therefore make an assessment of the realism in each project prior to approval. When encountering such challenges during the course of the trial the researchers are faced with the choice of aborting the study or reinforcing the measures. To be able to make these choices at the appropriate time the researchers need to monitor the project continuously and to have created strategies on how to handle such difficulties in advance. In our case, we chose to reinforce the measures by training additional therapists, extending the inclusion period and providing monetary incentives. The latter two reinforcement strategies necessitated additional approval from the Regional Ethics Committee. In either case, whether the researchers choose to abort the study or to reinforce the measures, the researchers have an ethical obligation to report these challenges in a way that prevents other researchers from encountering the same problems.

Lessons Learned

This study illustrates common challenges in conducting pragmatic multi-site group-randomized studies on heterogeneous groups of patients with multiple problems and complex intervention methods. The study shows that although it is difficult and challenging, it is possible to conduct such a study. In communicating with clinicians and patients, the study has been regarded as interesting and worthwhile. This points towards possibilities and potentials.

The first challenge we met was difficulties in recruiting centers and motivating their participation regardless of which treatment condition they would be allocated to. The lesson is to involve the participating centers at an early stage of the planning phase of the study. This will both enhance the centers’ feeling of “ownership” towards the study, and also contribute to a design that is in accordance with the clinics’ needs and workload.

Our second challenge was that the local trial administrators were a mix of people with different education, training, interests, motivation and time available for the trial. The lesson is to visit each center regularly, at least once a month, to ensure data-quality at each center and uniformity in data collection between centers.

Our third challenge was that the therapists did not comply with the screening routines of the study. This might be explained by heavy work-load, difficulties in thematizing substance misuse and lack of support from the clinics’ management. The lesson is to make sure that the new routines are well implemented before the start of the trial.

Our fourth challenge was different referral policies between centers and that these policies were changed during the planning phase of the trial due to new national guidelines. This affected the number of patients eligible for the trial. The lesson is to be aware of policies (formal and informal) and organizational changes and to take them into consideration when planning the study.

Our fifth challenge was that a low percentage of the patients that screened positive for substance use disorders were referred for the inclusion assessments of the trial. The main reason was lack of consent from the patients. This might be explained by how the study was presented to the patients by the therapists. Again, the lesson is to involve the centers at an early stage of the planning phase of the project, ensuring a feeling of “ownership” towards the study, and to make sure that the center management is supportive and encouraging towards the therapists in complying with the demands of the study.

Our sixth challenge was the slow and, ultimately, low inclusion rate. Our experience is that extending the inclusion period did not increase the inclusion number to a great extent. The lesson is to focus all efforts and resources on the original recruitment period. This is when all the participants of the study are still fully focused on this one task, and one minimizes the risk of therapist turn-over and other additional costs.

Our seventh and last challenge was to encourage patients to return for the follow-up assessments. Our experience is that the local trial administrators who put more effort into the task than was expected of them and who where the most creative, managed to reach the most patients for their follow-up assessments. The lesson is that this type of research puts heavy demands on the participating centers and local trial administrators, and that these demands and costs are often underestimated.

Summing up

Studies of complex interventions in unselected clinical populations are essential in the development of evidence based treatments in the psychiatric and addiction field. The methodological problems are considerable but it is possible to overcome the challenges by thorough planning and addressing the obstacles at an early stage.

Authors’ Contributions

All the authors fulfill the Vancouver requirements for authorship. RG and HW have been involved in the conception and design of the study. RG and LW have been involved in the acquisition of the data. HW and LW have been involved in interpreting the data. LW has drafted the manuscript. All authors have been involved in revising the manuscript critically for important intellectual content and approved the version to be published.

Authors’ Information

LW is a psychiatrist and PhD-fellow at the Norwegian Center for Addiction Research at the Institute of Clinical Medicine, University of Oslo. HW is a psychiatrist and professor emeritus at the Norwegian Center for Addiction Research at the Institute of Clinical Medicine, University of Oslo. RG is a psychologist, Head of the Department of Research and Development at the Alcohol and Drug Treatment Health Trust in Central Norway and Associate professor at the Norwegian Center for Addiction Research at the Institute of Clinical Medicine, University of Oslo.

Funding

The study was funded by the Research Council of Norway, the Regional Center for Co-occurring Disorders of Substance Abuse and Mental Health and the Center of Competence for Addiction Issues–-Region East

Competing Interests

Author(s) disclose no potential conflicts of interest.

Disclosures

As a requirement of publication author(s) have provided to the publisher signed confirmation of compliance with legal and ethical obligations including but not limited to the following: authorship and contributorship, conflicts of interest, privacy and confidentiality and (where applicable) protection of human and animal research subjects. The authors have read and confirmed their agreement with the ICMJE authorship and conflict of interest criteria. The authors have also confirmed that this article is unique and not under consideration or published in any other publication, and that they have permission from rights holders to reproduce any copyrighted material. Any disclosures are made in this section. The external blind peer reviewers report no conflicts of interest.