Systematic Review and Meta-Analyses of Psychotherapies for Adolescents with Subclinical and Borderline Personality Disorder: A Reply to the Commentary by Jørgensen,Storebø,and Simonsen

We thank Dr. Jørgensen, Dr. Storebø, and Dr. Simonsen for their feedback ¹ and the opportunity to discuss our article’s findings further. ² Our reply to the main points (1 to 5 in the following; order changed) listed in the commentary by Jorgensen et al.

First, we have not been able to retrieve any published protocol, which makes it difficult to determine whether the review could be prone to selective reporting bias. This is an important question that was addressed by a recent systematic review that explored trends in the publication of protocols of systematic reviews. ³ While there is an increase in the publication of protocols on the whole, publication of a protocol can also create a significant delay in a project’s time line, and this can hamper knowledge synthesis research in rapidly advancing fields such as child and adolescent psychiatry. We note, for example, that since the publication of this review, an additional randomized controlled trial (RCT) has been published. ⁴ While systematic reviews with published protocols often report their methodology more comprehensively, we welcome specific questions about any aspect of the systematic review or meta-analysis strategy to improve transparency. Although we understand that a published protocol would provide evidence for a rigorous, methodological approach to systematic reviews, our study did adhere to the Preferred Reporting for Systematic reviews and Meta-analyses guidelines in hopes of identifying and critically appraising relevant data from our included studies. ⁵

The authors only searched seven electronic databases for relevant articles and used an English language restriction. This may have caused for some RCTs to be left out, for example, in the studies of Santisteban et al., Salzer et al., and a pilot study by Gleeson et al.

We can certainly see that a limitation is that our search is limited by our English language restriction. However, looking back at our literature search, we identified all three of these articles, but they were excluded for different reasons. Salzer et al. ⁶ was excluded as the study was not written in English, other than the abstract. Gleeson et al. ⁷ was excluded as this study included a large number of adults. Although they recruited from age 15 to 25, their mean age was 18.4 (SD = 2.9). We felt that adolescents were not the primary patient population and thus did not comply with our eligibility criteria. Furthermore, the sample’s objective was to report treatment outcomes for first episode psychosis with co-occurring borderline personality disorder (BPD). Finally, Santisteban et al. ⁸ was excluded at the level of the meta-analysis: The authors reported the profiles of clinical change and a dimensional score of BPD functionality, which precluded pooling of this type of data with the other effect size estimates across studies.

The authors conclude that “the studies were rated as being of very high quality” (p. 5), but it is unclear on what ground this quality assessment was made. However, the ratings on several domains may be too optimistic.

With respect to the risk of bias, as described in our article, we used the Cochrane risk-of-bias tool to assess the risk of bias in each included RCT, which rates seven domains of potential bias. ⁹ This tool, however, does not address all forms of bias, and we appreciate your concerns regarding additional domains of potential bias not considered by our article. For example, when you discussed allegiance bias, this draws to mind the study by Rossouw and Fonagy that involved mentalization-based therapy. ¹⁰ In a recent meta-analysis, the summary relative odds ratio of effect sizes in allegiant trials were compared with nonallegiant using random and fixed models, which was 1.31 ([1.03 to 1.66], p = 0.30, I² = 53%), indicating larger effects when allegiance exists. ¹¹ Allegiance is particularly applicable in psychotherapy research and may threaten to the validity of conclusions from comparative outcome studies. ¹² However, in retrospect, it was difficult to assess for allegiance bias quantitively in our meta-analysis as there were few studies for any given modality; this will be the objective of future reviews that identify more studies for any single modality.

Additionally, the review is prone to some methodological issues concerning clinical heterogeneity. The authors pooled experimental therapies and control therapies into meta-analyses not taking into consideration the heterogeneity of the experimental interventions, control treatments, type of outcome assessors, and time points.

As we mentioned in our article, there were certainly limitations given the extensive clinical and methodological heterogeneity across the included studies. However, we attempted to account for heterogeneity in a few ways, such as by comparing effect sizes from the completion of treatment to those in extended follow-up, which suggests that the significance of treatment was not retained over time. As the overall study yield was small, this precluded a more extensive subgroup analysis to look at particular psychotherapeutic modalities (e.g., dialectical behavioral therapy vs. cognitive behavioral therapy), control treatments (e.g., treatment as usual vs. good clinical care), or specific time points. Thus, it was necessary to standardize the results of the studies to a uniform scale before they could be combined by computing a standardized mean difference (SMD). The SMD is frequently used as a summary statistic in meta-analysis when the studies all assess the same outcome but measure it in a variety of ways, and it expresses the size of the intervention effect in each study relative to the variability observed in that study. ^13,14