Advancing the quality and safety of behavioural interventions in mental health research: A how-to guide from the MAGNET Clinical Trial Network

Abstract

This is the first in a series of Position Papers from the Mental Health Australia General Clinical Trials Network (MAGNET) intended to promote the standard of mental health research in Australia. This paper focuses on improving the quality and safety of non-pharmacological trials with a mental health focus, which for the purpose of this paper, are those testing ‘complex’ behavioural interventions (including lifestyle or psychotherapy interventions) with clinical populations. This is timely after last year’s update of the National Statement for Ethical Conduct in Human Research which is intended to provide extended guidance on assessing, mitigating and managing risk and the introduction of the Australian Commission on Safety & Quality in Healthcare’s National Clinical Trials Governance Framework. However, what the implementation of these research policies means for behavioural trials in mental health, given their many nuances, is only being realised. This paper outlines historical issues in the conduct of behavioural trials in mental health (lack of consensus on the concept of harm; lack of governance and inconsistent data collection and/or trial procedures around harms). Next, we detail the methods for developing recommendations to aid triallists’ monitoring and assessing safety during the conduct of behavioural mental health trials that evaluate lifestyle or psychotherapy interventions in clinical populations. Finally, we present a decision-making algorithm to support implementation. Ultimately, we intend to promote quality and safety of behavioural interventions in mental health, to better understand the risk/benefit profile of these treatments and to minimise unnecessary risk to participants and triallists.

Background

Over the past decade, the number of clinical trials in mental health has grown comparably to other areas of medicine (Wortzel et al., 2020). Investigator-led trials have increased faster than industry-sponsored trials as have the number of non-pharmacological trials (Wortzel et al., 2020). In mental health, non-pharmacological trials can include (but are not limited to) those testing Transcranial Magnetic Stimulation, brain training or psychoanalysis, for example, but for the purpose of this paper, we focus on those defined as ‘complex’ interventions’(Craig et al., 2013), which typically have multiple, interacting behavioural components that require tailoring by interventionists for participants and have numerous, variable outcomes (Craig et al., 2008) (henceforth referred to as ‘behavioural’ interventions, treatments or trials). Evaluating the direct and indirect risks and benefits of complex, behavioural treatments in a mental health clinical trial setting is challenging. Incomplete safety monitoring and reporting of treatment harms has been extensively discussed (Berk and Parker, 2009; Herzog et al., 2019; Horigian et al., 2010; Mejía-Castrejón et al., 2024) and described as an ‘under-recognised problem’ (Abdel Shaheed et al., 2022) that creates biased clinical decision-making. While this problem is relevant to other fields evaluating complex behavioural interventions (Abdel Shaheed et al., 2022), there are unique methodological challenges in mental health research. Lack of objective outcomes and diagnostic procedures and heterogeneous symptomatology and aetiologies across psychiatric disorders adds complexity to adjudication.

There remains pressing need to calibrate and enhance safety around behavioural mental health trials in accordance with clinical practice and clinical education settings and the domestic regulatory environment. In Australia, there is a spotlight on clinical trials after the recent update of the National Statement for Ethical Conduct in Human Research (‘the National Statement’) (National Health and Medical Research Council and Council and Universities Australia, 2023) and the introduction of the Australian Commission on Safety & Quality in Healthcare’s National Clinical Trials Governance Framework (‘the National Framework’) (Australian Commission on Safety and Quality in Health Care, 2022) to improve the quality and safety of Australian clinical trials. Currently, however, the National Health and Medical Research Council (NHMRC) clinical trial guidance documents pertain to non-behavioural trials (National Health and Medical Research Council, 2018) and a standardised, risk-assessed safety approach for behavioural trials is lacking. Specific considerations for behavioural mental health trials are thus required.

To advance the quality and safety of our trials and equip the next generation of Principal Investigators (PIs) and triallists, we cannot assume behavioural interventions to be innocuous (as has been the case historically), bound by fewer safeguards than other investigational products. This is the first in a series of Position Papers written on behalf of the Mental Health Australia General Clinical Trials Network (MAGNET) (MAGNET Operating Committee et al., 2023) that focus on maximising the quality of a specific area of mental health clinical trials. Building on the work of others (Mejía-Castrejón et al., 2024), we detail the methods for the development of a series of recommendations and proposed decision-making algorithm for assessing safety and monitoring harms in behavioural trials of clinical mental health populations. Our objective is to improve trial quality and advance our understanding of risk-benefits around behavioural mental health treatments thereby providing balanced information for clinical decision-makers to optimise outcomes for those with mental health conditions.

Historical issues in behavioural mental health trials

The National Statement (National Health and Medical Research Council and Council and Universities Australia, 2023) defined harms as physical, psychological, cultural, legal or economic harm or the devaluation of personal worth. While not exclusive to behavioural mental health trials (Palmer et al., 2020), harms associated with their conduct can include clinical deterioration (6% of participants in psychotherapy trials) (Cuijpers et al., 2023; Rozental et al., 2017); null effects (50% of participants in psychotherapy trials) (Boettcher et al., 2014; Cuijpers et al., 2021); simultaneous iatrogenic and non-iatrogenic effects (e.g. symptom improvement on one measure while deterioration on another); malpractice, unethical conduct, low therapeutic alliance, incorrect or inappropriate application of otherwise appropriate treatments, contraindications, treatment dissatisfaction, increased dependency, abnormal illness behaviour (Berk et al., 2013) and perceived adverse effects on participants, their family or friends (Duggan et al., 2014). Unfortunately, only two-thirds of studies report harms which are often incomplete, ill-defined and inconsistent (Papaioannou et al., 2021). While these concepts are extensively discussed in the psychotherapy literature (Berk and Parker, 2009; Herzog et al., 2019; Horigian et al., 2010; Mejía-Castrejón et al., 2024), it is less common in lifestyle psychiatry.

Incomplete safety and harm reporting may come from behavioural trials having fewer compulsory reporting processes compared to drug trials, which have clear procedural and oversight mechanisms for harm detection (required by the Therapeutic Goods Administration). Variation in reporting requirements also exists between Institutional sponsors and human research ethics committees around investigator-initiated trials. Data Safety Monitoring Boards (DSMBs) which oversee trial safety are seldom used for behavioural mental health trials (Wortzel et al., 2020). Lack of governance and procedural streamlining contributes to a dearth of harms data which perpetuates the cycle of unreliable and generally under-appraised estimates.

Another issue is the lack of clarity around key terms underpinning trial procedures to monitor harms (Berk and Parker, 2009; Herzog et al., 2019; Horigian et al., 2010; Mejía-Castrejón et al., 2024). Notably, event parameters such as (a) frequency, (b) severity, (c) reaction term, (d) expectedness, (e) seriousness and (f) relatedness are ill-defined and poorly operationalised in the literature. Trials reporting harms only partially report procedures (Papaioannou et al., 2021) and Adverse Events are often incorrectly used to refer to any safety event that occurs. Assessing ‘relatedness’ is complex, as behavioural interventions are multi-factorial, use cognitive or health behaviours enacted every day, with far-reaching social and emotional consequences (Mejía-Castrejón et al., 2024). Furthermore, journals, funders and trial registries do not routinely mandate harms reporting.

Frameworks and policy to promote clinical trial quality

In addition to the National Statement (National Health and Medical Research Council and Council and Universities Australia, 2023) and National Framework (Australian Commission on Safety and Quality in Health Care, 2022), international resources, such as the 33-item SPIRIT statement (Chan et al., 2013), designed to provide an international standard for trial protocol development and reporting tools, such as the 16-item CONSORT-Harms (Junqueira et al., 2023) checklist for integration with the CONSORT-Outcomes (Butcher et al., 2022) or CONSORT-Social & Psychological Interventions (Montgomery et al., 2013), all promote the inclusion of harms data. Yet uptake is inconsistent (Junqueira et al., 2021).

Methodology for developing the proposed recommendations and new decision-making algorithm for behavioural mental health trials

On 28 November, 2023, the second annual MAGNET Clinical Trial Network Annual Scientific Meeting (ASM) was held in Scarborough, Western Australia. This meeting saw more than 90 mental health and psychiatry researchers, academics, clinicians, and lived experience representatives convene with two key objectives. First, to review and endorse prospective mental health clinical trials proposed by Australian researchers to encourage collaboration, innovation and advancement. Second, to identify the areas of interest and respond to key gaps in the field that had been raised by the MAGNET membership in the preceding year.

One key priority area identified was the absence of guidance within the scientific literature or clinical trials training around the conduct of behavioural trials in mental health, specifically around the monitoring, adjudication and reporting of harms (of which adverse events are a key component). The ASM therefore dedicated a 1 hour session within the programme which was chaired by an expert in behavioural mental health trials (A.T.) who presented the following key issues in this field to a panel of experts in the field for discussion (including S.L.R., A.G. and M.L.C.): (1) that the definition of a clinical trial is broad; (2) by definition, harms do not have to be related to the intervention; (3) that behavioural trials should not be assumed to be innocuous; (4) the need to consider Good Clinical Practice (GCP) and ethical guidelines (highlighting international and national as well as local procedures); (5) that these guidelines are heavily weighted to therapeutic goods (medication) trials and (6) that there are different steps involved in harms monitoring. This led to a broader discussion among the ASM attendees, themes of which were documented by the lead author (A.O.).

Following this workshop in February 2024, the lead author (A.O.) circulated an invitation to the MAGNET membership inviting those with expertise in behavioural mental health trials to join a working group to develop a set of recommendations as the basis for MAGNET Position Paper entitled ‘Advancing the quality and safety of behavioural interventions in mental health research: A how-to guide from the MAGNET Clinical Trial Network’.

From this, 18 experts in the field opted into the working group with representation from Victoria, New South Wales, Tasmania and Australian Capital Territory. However, one-third were Early Career Researchers, Mid-Career Researchers and Senior Researchers, respectively. Areas of expertise covered included mood disorders, eating disorders, psychosis, substance use, and psychosocial, psychological and lifestyle interventions using various delivery modes (face-to-face, online, telehealth).

After the themes identified from the workshop were consolidated, a list of 15 recommendations based on the discussion at the ASM were drafted by the first author in consultation with the writing group which was provided a 14-day consultation period. Based on this feedback process, the recommendations were then refined from 14 to 7 and a Supplementary algorithm was designed by A.O. and T.J. to support implementation. The final version of the recommendation and algorithm was then endorsed in July 2024 and subsequently presented to the National Mental Health Research Community at the ASM of the Society of Mental Health Research conference on 5 November 2024 in Sydney for feedback from a broader group of stakeholders.

Using this approach and building on others’ attempts (Mejía-Castrejón et al., 2024), we developed a series of recommendations (presented below) and a clinical trial decision-making algorithm to assess safety and monitor harms in behavioural mental health trials (Figure 1). This is designed to complement the National Statement (National Health and Medical Research Council and Council and Universities Australia, 2023), National Framework (Australian Commission on Safety and Quality in Health Care, 2022) and International Council of Harmonisation (ICH)-Good Clinical Practice (International Council for Harmonisation Guideline for Good Clinical Practice, 2016).

Figure 1.

Suggested algorithm for assessing, reporting and mitigating harms in behavioural mental health trials (adapted from Mejía-Castrejón et al. (2024)).

Seven recommendations to support algorithm implementation

We recommend that high-quality clinical trials of a behavioural mental health intervention consider the following:

1. Event parameters are operationalised a priori where possible and consistently adjudicated throughout the trial;

2. There is opportunity for adequate, comprehensive event collection throughout the trial which considers harm typologies (Duggan et al., 2014);

3. An Escalation of Care protocol is developed at the trial outset;

4. A DSMB (or alternative) is established and operated using NHMRC Guidelines where appropriate;

5. Participant eligibility for trial is dynamically assessed;

6. Events are appropriately resolved;

7. Events data are appropriately reported.

PIs comply with the overarching institutional body’s reporting requirements when reporting SAEs to the Human Research Ethics Committees and DSMB (typically 24–48 hours from notification). In publications, triallists will state if no harms were reported and/or publish aggregate events data (per CONSORT-Harms [Junqueira et al., 2023]) and by treatment allocation and exposure level. Predictors of harm such as demographic, biological, psychosocial, intervention or trial status (e.g. lost to follow-up, completed) are presented to inform clinical practice and other triallists’ future attempts to operationalise expectedness and relatedness. Exploratory analyses could include prevalence of participants who deteriorated, did not respond/improve (using reliable change index, dose-response associations, effect modifiers [therapeutic alliance, treatment expectancy], number needed to treat [NNT] and the number needed to harm [NNH]). In pharmaceutical trials, a major reason for failure to progress an agent is adverse events balancing the NNT vs NNH ratio. Even agents with robust efficacy are shelved because of adverse events and risks. It is recommended that a comparable balancing exercise be conducted for behavioural interventions. We recommend contributing trial data to publicly available data repositories (e.g. the Health Studies Australian National Data Asset, HeSANDA [Australian Research Data Commons, 2021]) for well-powered studies of treatment harms and benefits.

Future considerations for improving conduct of behavioural mental health trials

Event reporting may be daunting for triallists in often resource-poor trial settings, but it is necessary. While covering other areas of clinical trial risk assessment (e.g. Risk Assessment Plans: a dynamic quality assurance document reviewed and updated throughout a trial) (National Health and Medical Research Council, 2018; National Institute of Health and Care Research, 2024) was beyond the scope of this paper, the algorithm presented here should align with such documents. Promisingly, the costs of risk mitigation and trial monitoring (e.g. external or internal adjudication/monitoring, training) are becoming a commonplace within trial budgets which is critical to support uptake.

Practical tips to promote high-quality behavioural mental health trials include seeking out trial coordinators with skillsets that are not just relevant to the trial population or intervention of interest but to support the implementation of harm detection and mitigation to avoid the need to outsource tasks and maximise cost recovery. For example, trial coordinators who are registered mental health clinicians (psychologists, mental health nurses) can be involved in escalation of care duties. It is also advisable to train staff at trial initiation about harm detection, classification, resolution and procedures rather than on-the-job learning to promote efficiency, continuity and to minimise inaccuracies in reporting. It may also be mutually beneficial to initiate collaborative arrangements with other triallists whereby PIs and trial staff who establish DSMBs for a specific project (who act on a voluntary basis) reciprocate that arrangement for their collaborator’s trial.

Several limitations of our approach are noted. We focussed on complex behavioural interventions specifically those psychological or lifestyle focussed in the nature for clinical populations but acknowledge nuances of other non-pharmacological interventions (e.g. digital interventions, cognitive remediation, social/wellbeing interventions) and other mental health populations (e.g. prevention trials). Second, this piece is a written form of PI’s perspectives, with contributions from Lived Experience personnel, to guide other clinical triallists not from trial participants’ perspectives. What PIs perceive as harmful or beneficial may differ from participants, highlighting the role of Lived Experience personnel on trial teams and DSMBs. In addition, this paper recommends using structured instruments to detect events; however, many are imperfect (Papaioannou et al., 2021). The field awaits development of appropriately designed instruments tailored to specific intervention modalities. Finally, while we believe our proposed algorithm and accompanying recommendations fill a gap in this field of mental health, we have yet to evaluate its implementation, which we intend to do and report on in subsequent case studies. We expect that this process may lead to refinement of the algorithm.

How can MAGNET help?

MAGNET is Australia’s first clinical trial network focused on adult mental health (MAGNET Operating Committee et al., 2023). This is a national collaboration of researchers, academics, carers, lived experience partners, and First Nations peoples from 40 institutions, health services, industry bodies, consumer groups and other stakeholders who are collectively transforming the ways mental health treatments are developed. Key activities include cross-cutting platforms designed to support day-to-day conduct and quality of mental health clinical trials: biostatistics; health economics; partnerships with lived experience and First Nations people; psychiatric assessments; and trial governance. Shared resources, including examples of trial documents to support this algorithm, are free to members (magnetctn.com). The network has implemented four signature trials to demonstrate network utility and answer critical research questions in the field, including one behavioural mental health trial of a clinical mental health population that guided the development of this algorithm (Davis et al., 2024).

While we consider that this can act as a standalone document which provides sufficient information to be used by triallists to guide their trial activity, less-experienced triallists or those new to behavioural intervention trials may require additional support. For example, those who are not familiar with the validated adverse event tools can gain access to them via the MAGNET website’s searchable function on its assessment platform by becoming a MAGNET member (otherwise accessible from the literature). We also intend to deliver online training workshops which will be free to MAGNET members (noting membership is free to national and international mental health triallists) to support uptake and gain feedback should subsequent revisions be necessary.

Finally, we note that the regulatory and governance landscapes around non-pharmacological trials are rapidly changing in Australia such that it is likely that their independent auditing will become a requirement regardless of the clinical trial setting (university, hospital). Administrators auditing compliance activities and regulatory bodies which establish auditing frameworks for behavioural mental health trials may find this algorithm helpful for this purpose.

Conclusion

This paper provides seven recommendations and a proposed decision-making algorithm designed to promote quality and safety of behavioural mental health trials. This should be considered in conjunction with the SPIRIT (Chan et al., 2013) and CONSORT-Harms (Junqueira et al., 2023) reporting frameworks and key domestic research policies of the contemporary clinical trial regulatory environment in Australia.

Supplemental Material

sj-docx-1-anp-10.1177_00048674251319680 – Supplemental material for Advancing the quality and safety of behavioural interventions in mental health research: A how-to guide from the MAGNET Clinical Trial Network

Supplemental material, sj-docx-1-anp-10.1177_00048674251319680 for Advancing the quality and safety of behavioural interventions in mental health research: A how-to guide from the MAGNET Clinical Trial Network by Adrienne O’Neil, Tayla John, Alyna Turner, Philip J Batterham, Ayla Barutchu, Rachel Fiddes, Josephine Chambers, Susan L Rossell, Christopher Davey, Sean Carruthers, Madeleine L Connolly, Katherine L Mills, Amelia Gulliver, Orli Schwartz, Erica Neill, Jessica A Davis, Jessica Roydhouse and Michael Berk in Australian & New Zealand Journal of Psychiatry