Lessons learned from Restoring Study 329: Transparent reporting,open databases and network meta-analyses as the way forward

Study 329 is a poster-child for the so-called crisis in evidence-based medicine. Published in 2001 by the Journal for the American Academy of Adolescent Psychiatry (JAACAP), this double-blind, randomized controlled trial sponsored by GlaxoSmithKline compared paroxetine, imipramine and placebo in the treatment of adolescent major depression. The authors concluded that ‘paroxetine is generally well tolerated and effective for major depression in adolescents’. A nebulous statement, but apparently supported by a significant improvement in a variety of widely used and validated scores for depression. According to the results from this study, psychiatrists who prescribe paroxetine to a 15-year-old with major depression are practising evidence-based medicine. Or at least they were, until a major re-analysis in 2015 found no difference between paroxetine and placebo when only outcome measures pre-specified in the original study protocol were considered (Le Noury et al., 2015). The statistically significant findings in Study 329 original article, it emerged, relied on four new outcome measures, introduced post hoc by the sponsor. It is known that pharmaceutical industry–funded studies are more likely to show favourable efficacy results for investigational drugs than independent trials. Study 329 demonstrates how outcome-switching and selective reporting of results can be used to manipulate results. This was the topic of a recent commentary in Australian & New Zealand Journal of Psychiatry (ANZJP) (Mulder et al., 2016), which also highlighted the mute response from the JAACAP to the results and implications of the 2015 re-analysis. Should the 2001 paper be withdrawn from publication (www.elsevier.com/about/company-information/policies/article-withdrawal)? This is an important issue, which is at the heart of the debate in the scientific literature about evidence-based medicine and, therefore, evidence-based practice. Ben Goldacre and colleagues at the Centre for Evidence-Based Medicine at the University of Oxford have been systematically investigating the issue of outcome-switching in trials published in top medical journals and sharing their results on the website COMPare-trials.org. They have found that poor trial reporting often masks inadequate design, and studies such as 329 are just examples of a widespread practice in all areas of medical research, not only psychiatry. This is despite the endorsement by prominent medical journals of the Consolidated Standards of Reporting Trials (CONSORT) Statement, a guideline for authors to prepare reports of trial findings, facilitating their complete and transparent reporting and aiding their critical appraisal and interpretation (www.consort-statement.org/). Of course, good quality trials may be poorly reported. However, transparent reporting on inadequate trials will reveal deficiencies in the design if they exist, as in the case in Study 329. Among other things, CONSORT requires there to be ‘completely defined pre-specified primary and secondary outcome measures’ and clear reporting of ‘any changes to trial outcomes after the trial commenced, with reasons’. Both are absent in Study 329. Goldacre and his colleagues are now challenging the editors of scientific journals, and, alongside, other pro-transparency initiatives are contributing to a mounting pressure that will hopefully lead to a change in culture in medical publishing and prevent inadequate studies passing through the publication process (Goldacre, 2016).

While we fully support the efforts of these groups, we wish to address the needs of practicing clinicians, who want to know which is the most reliable estimate of paroxetine in young people with major depressive disorder. Individual trials and also pair-wise meta-analyses are not enough. Network meta-analysis has the great advantage over standard meta-analysis of comparing all treatments against each other (even if there is no trial data comparing the interventions directly) and ranking them according to their relative efficacy or tolerability with a precise degree of confidence (Mavridis et al., 2015). Another important advantage of network meta-analysis is that the effects of sponsorship bias are distributed (and therefore diluted) across the network. The case of Study 329 is a clear demonstration of this added value. A recent network meta-analysis has compared all antidepressants for major depression in children and adolescents (Cipriani et al., 2016). The search strategy, though, was completed in May 2015, a few months before the publication of the re-analysis of study 329. As a result, Study 329 was included with the original 2001 data biased towards paroxetine; however, the overall results from the network meta-analysis showed that paroxetine was not statistically significantly different from placebo. This is because by combining direct with indirect evidence, the bias in studies that favour the investigational drug over placebo is mitigated by the findings of other indirect comparisons in the network. Interestingly, the final results of the network meta-analysis showed that, among all antidepressants, only fluoxetine was significantly more effective than placebo. This is probably a robust result, because it is consistent with the direct and indirect evidence comparing fluoxetine with all the other compounds in the network.

Notwithstanding their own limitations, network meta-analyses are considered the best methodology for evidence synthesis and treatment guidelines. However, development of such statistical methods should not prevent researchers from striving for complete transparency in clinical psychopharmacology. Open access to raw data from trials would allow for validation of results (Goldacre, 2016) and more sophisticated analysis, such as individual patient data meta-analysis, which can enable researchers to identify predictors of response and tailor treatment to the individual. Accessing patient-level data, a serious problem, requires concerted and effective action by the pharmaceutical industry and research community worldwide. The right statements are now being made; a number of large pharmaceutical companies have committed to sharing anonymized patient-level data from clinical trials (see clinicalstudydatarequest.com), and major medical journals are already on board. Next, we need to see words turn to action.