Abstract
Objectives
While internet-based interventions have been used for more than 10 years in Dutch forensic psychiatric outpatient care, it is not clear if, why and for whom they work. The objective of this study was to investigate whether the addition of an internet-based intervention to forensic psychiatric outpatient treatment as usual (TAU) resulted in better treatment outcomes in regulatory emotional self-efficacy, treatment readiness and aggression.
Methods
A multicentre mixed-methods pilot randomized controlled trial was used to compare TAU to TAU with the addition of the internet-based intervention ‘Dealing with Aggression’. Thirty forensic psychiatric outpatients, treated in four Dutch organizations for aggression regulation problems, were included. Three validated questionnaires were administered at baseline, mid-treatment, post-treatment and follow-up. Semi-structured interviews with patients and therapists from the experimental condition were conducted to gain in-depth insight into the quantitative results.
Results
No differences between groups on the outcome measures were found. Often-mentioned reasons for ineffectiveness were suboptimal integration in treatment and the requirement of cognitive and reflective skills. However, several patients reported having experienced benefits from the intervention, mostly because treatment was repeated, combined with new insights in their behaviour.
Conclusions
Although the trial was underpowered due to issues with recruitment, no proof for effectiveness of the internet-based intervention was found. Moreover, mostly highly motivated and relative cognitive skilled patients participated, indicating that internet-based interventions might fit only a select number of patients. New insights into why and for whom internet-based interventions might work were identified, providing points of improvement for intervention design and implementation in practice.
Keywords
Background
Forensic mental healthcare focuses on treatment of patients who show delinquent behaviour, are on the verge of or have committed a crime and have one or more concurrent psychiatric disorders. 1 Forensic care can be delivered in both inpatient and outpatient settings. In outpatient settings, patients receive justice-system-linked mental healthcare without hospitalization. One of the main objectives of forensic treatment is to decrease the chances of recidivism. This is partly achieved by targeting predictors of aggression, such as emotion regulation in treatment. Cognitive behavioural therapy (CBT) is commonly used in forensic mental healthcare because it helps patients develop skills to better manage their emotions and in turn reduce aggression. However, treating aggression remains a complex endeavour. For example, effect sizes for CBT targeting aggression are often smaller than those for anxiety or mood disorders.2,3 In order to further improve forensic treatment without requiring valuable time from often already overworked staff, there is an increasing interest in internet-based interventions.4–6 These types of interventions are based on evidence-based treatment models such as CBT and present the patient with knowledge and skills via multiple lessons that contain, e.g. written reflective assignments or multimedia applications such as videos. 7 CBT-based internet-based interventions can be used outside treatment sessions by the patient, allowing them to acquire knowledge and skills individually. This possibility to work on treatment independently might result not just in more efficient treatment but also in increased self-efficacy. 8 These advantages are especially relevant for forensic outpatients, since they often receive only one treatment session per week. They could potentially benefit from more intensive treatment and more tools to generalize skills acquired in treatment to their daily lives. Because of their potential for forensic outpatient care, internet-based interventions have been used for over 10 years in Dutch forensic psychiatric facilities. 4 However, despite their increasing use, rigorous and ecologically valid research on their effectiveness remains limited.
Although internet-based interventions have shown promising results in general psychiatric populations, their effectiveness in forensic mental healthcare – particularly in forensic outpatient populations – remain unclear.6,9 Research on internet-based interventions in non-forensic populations shows positive results. To illustrate, meta-analyses consistently show that internet-based interventions are effective for a broad range of psychiatric disorders, such as depression or post-traumatic stress disorders, often resulting in outcomes similar to face-to-face interventions.10–15 However, because of the unique characteristics of the forensic population, these positive conclusions cannot simply be generalized to forensic mental healthcare. 9 These characteristics include low motivation for their often obligatory treatment, generally lower education levels, the presence of complex psychosocial problems and high levels of co-morbidity.16–18 Reviews on the use of technology in treatment of offenders have shown that there is not enough (high-quality) research on internet-based interventions to draw conclusions on their effectiveness.6,9 Most studies examined internet-based interventions within prison or inpatient settings, whereas forensic outpatient care is underrepresented. 8 Furthermore, internet-based interventions are often introduced only as part of a study and were not used in forensic practice before that. This means that results often have limited ecologically validity, amongst other things, because researchers provide much support for implementation and adherence to these interventions during the study, which would not be the case if these interventions would be implemented in regular practice. Moreover, these interventions are often not integrated in standard care – which is often the case in practice 19 – but used as a stand-alone addition. This highlights the need for more evaluation studies that focus on interventions that are already used in clinical practice.
Additionally, most research on internet-based interventions in forensic settings is primarily focused on effectiveness, overlooking important factors related to their implementation and usage in routine care. Previous studies have shown that low adherence to internet-based interventions occur regularly in forensic psychiatry.8,20 Adherence refers to the extent to which patients use the intervention as intended by the developers, which can be measured through, for example, log-in frequency and completion of lessons.21–23 Beyond adherence, engagement represents a broader concept that encompasses not only behavioural usage but also affective and cognitive components. 24 Adherence and engagement are both associated with the effectiveness of internet-based interventions.23,25–27 Furthermore, not much is known about how to best integrate these interventions in forensic practice, highlighting the need for more research into their implementation. 8 Consequently, understanding patient and therapist perspectives on the use of these interventions is important to optimize the design, promote the adherence and engagement and ultimately improve implementation in practice. 28
This study aims to evaluate the added value of the existing Dutch internet-based intervention ‘Dealing with Aggression’ (in Dutch: Omgaan met Agressie) in four forensic outpatient centres. This specific intervention has not been evaluated in earlier research, both inside and outside forensic psychiatry. Furthermore, to our knowledge, no randomized controlled trials (RCTs) of other Minddistrict interventions have been or are being conducted in forensic psychiatry. The main goal of this study is to investigate whether the addition of the internet-based intervention ‘Dealing with Aggression’ to forensic psychiatric outpatient treatment as usual (TAU) results in better treatment outcomes (aggression, regulatory emotional self-efficacy and treatment readiness), compared to patients that receive only in-person TAU. The second objective is to investigate if adherence to and engagement with the intervention are predictors of effectiveness on the outcome measures. The third objective is to gain insight into the patients’ perceived effectiveness of the intervention. The fourth objective is to gain insights into reasons for (in)effectiveness of the internet-based intervention according to participating patients and their therapists.
Methods
Study design
This study was a multisite, mixed-methods RCT, conducted within four Dutch forensic organizations that offer outpatient care: Transfore (four outpatient clinics), Kairos (two outpatient clinics), De Woenselse Poort (one outpatient clinic) and GGZ Noord-Holland-Noord (three outpatient clinics). This study was approved by the medical ethical board Oost-Nederland of the Radboud University Medical Centre (NL80846.091.22) and was conducted in accordance with the principles of the Declaration of Helsinki. The study is registered at ClinicalTrials.gov (NCT05711342). The study protocol has been previously published in Digital Health. 29
Deviations from the study protocol
Multiple changes were made to the study protocol to ensure feasibility. Initially, a power analysis was conducted with G*Power, with an effect size of 0.5, a β-power of 0.8, a p-value of 5% and an independent two-sided t-test as method of analysis, showing the need of including 154 participants in total (accounting for a dropout of 20%). However, due to very low usage of the internet-based intervention, recruitment of the intended number of participants was challenging. Although the organizations intended to let all therapists employ the intervention with their patients, most therapists did not use the intervention anymore or did not start using it at all and were also not motivated to start using it for this study. In order to still gain insight into potential effectiveness, the initial RCT was changed to a mixed-methods pilot RCT, which is considered a reliable method to conduct valid and feasible evaluation research.30,31 For a pilot RCT, a sample size of 10 to 25 per condition is suggested.32,33 The change to a pilot RCT meant that the intended number of included participants for this study was modified to 30 patients (15 per condition). The second change to the study protocol was the removal of the 8-week waiting period before including participants that started treatment. During the study, therapists stated that they often introduced the study to the patients that had already started treatment instead of patients that they saw during intake. Since there are no clear guidelines or best practices for when to introduce these interventions in forensic psychiatry, we decided to base our decisions on the preferences of the participating therapists. Third, in most participating outpatient clinics, patients received group therapy, unless contraindicated. For this reason, most aggression regulation therapy was delivered in a group format, which limited possibilities to include patients in individual therapy for this study. To expand the pool of eligible patients, both patients that received individual treatment and patients from group treatments were recruited and included in the study. Fourth, the pre- and post-measurement of risk factors using the FARE were removed, as its inclusion would require numerous additional actions from already hard-to-involve therapists, including a change in the existing timing of administering the Forensisch Ambulante Risico Evaluatie (FARE). Moreover, at the time of the study, there was no consensus amongst the participating forensic psychiatric organizations regarding which version of the FARE should be used for the risk assessment. Because two different versions were used, analysis would not have been possible. Fifth, instead of asking responsible therapists for the exact number of sessions during the data collection period, both the therapists and the patients were asked if the intervention led to a higher treatment efficiency. This was decided because an overview of all treatment sessions appeared to be difficult to provide – partly since treatment in forensic psychiatry can take a very long time and is provided by different care providers. To summarize, the study design was created in cooperation with participating organizations; the situation in practice proved to be more capricious than anticipated. To account for these adaptations, semi-structured interviews were given a more prominent place within the mixed-methods design, putting more emphasis on barriers to the use of the internet-based intervention in practice.
TAU and the intervention
TAU consisted of standard treatment for aggression regulation. There are minor differences in the way of delivery of these types of treatment in the participating outpatient clinics, and they are all based on the same underlying principles and models. All treatment programmes in forensic psychiatry target dynamic risk factors that have been identified by means of risk assessment instruments, based on the Risk–Need–Responsivity (RNR) model. 34 Treatment is shaped by CBT, often combined with techniques from occupational therapies such as art, drama or psychomotor therapy, in which patients learn practical skills to improve daily functioning and emotion regulation in real-life situations.
The existing internet-based intervention ‘Dealing with Aggression’ was developed by Minddistrict in collaboration with GGZ Noord-Holland Noord. Minddistrict is an eHealth platform that facilitates online diaries and internet-based interventions to support patients and therapists in their treatment. The content of ‘Dealing with Aggression’ is based on both relevant literature on aggression regulation as well as the knowledge and experience of therapists and patients. The intervention was first introduced in the Netherlands over 10 years ago and was updated in 2022 by reducing the number of lessons and adding more multimedia regarding experiences of experts and other patients. For this study, the intervention was added to TAU in the experimental group and was intended to be used in a blended way, i.e. integrated in face-to-face treatment. The intervention is focused on three main objectives: (1) increasing the motivation to change, (2) acquiring skills for dealing with conflict and (3) breaking the cycle of aggression by providing knowledge on situational, emotional, cognitive and physical triggers. In Appendix A, a more elaborate description of the content of the 10 lessons is provided, and screenshots can be found in Appendix B. Each lesson consists of multiple components, i.e. written text, assignments and explanatory videos with therapists and/or experts by experience. When a patient completes a lesson, the therapist is expected to send written feedback on the assignments. Because ‘Dealing with Aggression’ is used in a blended way, each lesson is intended to be briefly discussed in an in-person treatment session, which typically takes place on a weekly basis. However, if appointments were missed, patients were encouraged to independently work on the intervention, and in the next treatment session, both lessons were discussed.
Participants
All participants of this study were treated in a forensic psychiatric outpatient clinic, with their main treatment goals being focused on aggression regulation problems. In order to be eligible for participation, a patient had to be 18 years or older, a patient had to receive in-person one-on-one and/or group treatment focused on aggression and the responsible therapist had to indicate that participating was not expected to result in any harm. Patients were excluded if they were unable to read the Dutch language, currently resided in any psychiatric inpatient clinic, were in current psychiatric crisis and did not have access to a device on which they could access the intervention or if the responsible therapist indicated that the patient's psychosocial problems would interfere with participation.
Therapists screened patients on suitability when their cases were first discussed in a ‘multidisciplinary consultation’, which takes place before treatment starts. To increase feasibility of sampling, they were also able to discuss participation of participants that had already started treatment. If a patient was deemed eligible, the therapist then introduced the study to the patient and asked them consent for being contacted by the researcher. If the patient agreed, the researcher informed them about participation verbally and via an information folder. It was made clear to patients that if they are assigned to the control condition, they would still be able to use any internet-based intervention after data collection had been completed. Furthermore, the researcher clearly explained that the decision to participate or drop out of the study will not affect their treatment process in any way. If, after a week, the patient decided to participate, they were asked to sign an informed consent.
Materials and procedure
Participant flow
The flow of the participants throughout the RCT is visualized in Figure 1. One week after signing the informed consent, participants were randomized by the principal researcher (L.K.H.) via https://sealedenvelope.com/. They would then receive an e-mail with a link to the baseline questionnaires (T0) and were randomly assigned to the experimental or control condition. Participants were assessed four times: at baseline (T0); at mid-treatment assessment (T1) when they are supposed to be halfway throughout the internet-based intervention, i.e., 6 weeks after baseline; after completing the intervention (T2) – 14 weeks after baseline to account for often-occurring delays in treatment; and at follow-up (T3), 3 months after completing the intervention. Participants in both conditions were compensated for their invested time with two €10 vouchers at T1 and T3.
Participant flow through the mixed-methods randomized controlled trial (RCT) of the internet-based intervention ‘Dealing with Aggression’.
Research materials
This study employed quantitative methods to investigate aggression, regulatory emotional self-efficacy and treatment readiness at four time points. Moreover, engagement and adherence were measured using log data and quantitative methods at three time points. Additionally, qualitative methods were used to gain insight into the experiences of therapists and patients regarding the added value of the internet-based intervention. An overview of all materials used in this study is provided in Table 1.
Overview of variables, measurement instruments and timing of measurements.
Quantitative data
Aggression
Self-reported aggression was measured using the Aggression Questionnaire (Aangepaste Versie van de Agressie Vragenlijst (AVL-AV)).35,39 The 12 items of the questionnaire are divided into four subscales: physical aggression, verbal aggression, anger and hostility. The AVL-AV demonstrated good reliability and validity in previous research. 35 Internal consistency in the current sample was good, with a Cronbach's alpha of α = 0.82
Regulatory emotional self-efficacy
Regulatory emotional self-efficacy was measured using the 12-item Regulatory Emotional Self-Efficacy (RESE) scale. 36 The RESE scale assesses self-efficacy in managing negative emotions (eight items) and in expressing positive emotions (four items). Negative emotional self-efficacy refers to the capability to ameliorate negative emotional states and avoid being overcome by negative emotions such as anger or irritation. Positive self-efficacy is the perceived capability to express positive emotions such as joy or pride. Previous research has demonstrated the validity and reliability of the RESE scale.36,40 In this study, the internal consistency of the RESE was excellent with a Cronbach's alpha of α = 0.90.
Treatment readiness
Treatment readiness was assessed by the Corrections Victoria Treatment Readiness Questionnaire (CVTRQ), a self-report questionnaire that was validated in earlier research. 37 The total scale consists of 20 items, divided into four subscales. The subscale attitude and motivation (AM) measures attitudes and beliefs about treatment programmes and the desire to change, emotional reactions (ER) measures emotional responses to the individual's offending behaviour, offending beliefs (OB) refers to beliefs about personal responsibility for offending behaviour and efficacy (EF) measures perceived ability to participate in treatment programmes. The Cronbach’s alpha in this sample (α = 0.75) indicates an acceptable internal consistency.
Adherence
Adherence was defined as completing all lessons of the internet-based intervention. To assess the degree of adherence for every participant, usage log data was used to record the number of lessons completed. 41 Usage log data were automatically collected by the Minddistrict system and included a participant identifier, a record of activities available to participants, completion status of each activity and corresponding time stamps when completed.
Engagement
Engagement was measured with the TWente Engagement with Ehealth and Technologies Scale (TWEETS). 38 The questionnaire was incorporated in the intervention itself after lessons 1, 5 and 10, to ensure that the participants completed the TWEETS at the right point in time of the intervention. The scale employs a definition of engagement that incorporates behaviour, cognition and affect and has been shown to have a good validity and reliability. 38 The TWEETS contains nine items and has three slightly different versions that can be used at different points in time: one for expected engagement – to be used when someone starts using an intervention; one for current engagement – to be used during the use of an intervention; and one for past engagement – to be used when a user has completed or stopped using an intervention. In this study, a good internal consistency was found, with a Cronbach's alpha of α = 0.81.
Qualitative data
All semi-structured interviews were conducted by L.K.H., who had no therapeutic or personal relationship with the participants. The interviewer approached the interview from a pragmatic perspective, focusing on the reasons for experienced (in)effectiveness as well as barriers and facilitators for implementation and did not ask about individual treatment objectives or offenses. The interviewer is experienced in interviewing forensic psychiatric patients.
Patient interviews
All patients who were allocated to the experimental condition were invited to participate in a semi-structured interview after completing quantitative data collection. An interview scheme was used to identify perceived benefits, barriers and points of improvement of the internet-based intervention. During the interview, patients were able to share their experiences with the internet-based intervention and how it was integrated in their treatment. They also were able to reflect on whether the intervention had an effect on their aggression, self-efficacy and treatment readiness in order to explain the outcomes of the RCT. Finally, they could provide ways to improve the content and design of the intervention. The interview guide used for patient interviews is provided in Appendix C.
Therapist interviews
At the end of the data collection period, all therapists who participated in the study were invited for a 30-min interview. The same categories as for the patient interview were used for this interview scheme, asking therapist to reflect on possible improvement on aggression, self-efficacy and treatment readiness. They were also asked to provide a broader perspective on why internet-based interventions might be (in)effective for their patients. Finally, therapists were asked to provide points of improvement and other perceived benefits for the design and implementation of internet-based interventions in their current practice. The interview guide used for therapist interviews can be found in Appendix D.
Statistical analyses
Quantitative data were analysed using the IBM SPSS software (version 24.0) on an intention-to-treat (ITT) basis. Significance was accepted at 0.05 or lower. To determine whether there is more improvement on aggression, regulatory emotional self-efficacy and treatment readiness in the experimental compared to the control group, multilevel repeated-measures linear mixed models were used, with time and group as fixed effects, and group * time as an interaction effect. Participant IDs were included as a random factor. To investigate whether adherence and engagement predicted effectiveness, three linear regression analyses were run with the three outcome measures, in which the score on the TWEETS on T0 and on T1 and the number of completed lessons were used as independent variables and scores on T0, T1 and T2 for all outcome measures were used as dependent variables.
Data collected through the interviews with patients and therapists were analysed by L.K.H. and H.K. First, to explain quantitative results, a deductive approach was used, in which the outcome measures aggression, regulatory emotional self-efficacy and treatment readiness served as the main codes. Fragments that were related to experienced effectiveness but did not fit with any of the predetermined constructs were inductively coded. Second, the implementation barriers and points of improvement were analysed using an inductive approach, using the method of constant comparison. On the basis of these fragments, an initial coding scheme with main codes and subcodes was established by both researchers. This coding scheme was then applied to the interviews by the first coder (L.K.H.), which then was sent to the second coder (H.K.). An agreement rate of 84% was found, which is seen as acceptable. 42
As a last step, quantitative and qualitative data were triangulated in an explanatory mixed-methods design, in which qualitative results were used to illustrate the findings of quantitative analyses. 43
Results
Descriptive statistics
Data were collected between January 2022 and June 2024. In total, nine therapists and 30 patients participated in this study. An overview of descriptive data of the participants is provided in Table 2. The majority of participants are male (n = 27; 90%), are aged between 26 and 50 years and are undergoing some form of group therapy focused on reducing aggression. Furthermore, half of the participants have been diagnosed primarily with intermittent explosive disorder (n = 15; 50%), and a substantial majority (n = 24; 80%) are receiving treatment without any judicial measures in place.
Descriptive statistics of the participants (N = 30; intervention group n = 15; control group n = 15).
Quantitative results
The mean scores of the intervention and control group on aggression, regulatory self-efficacy and treatment readiness on the different measurement moments are provided in Table 3.
Means and standard deviation of the AVL-AV scores of the intervention group and control group (N = 30).
AVL-AV: Aangepaste Versie van de Agressie Vragenlijst; RESE: Regulatory Emotional Self-Efficacy; CVTRQ: Corrections Victoria Treatment Readiness Questionnaire.
Aggression (AVL-AV)
First, no significant fixed effect for group on aggression was found (b = −1.16 (SE = 4.08), F = 1.15, p = 0.293). Moreover, no fixed effect of time on aggression was found (T1: b = 4.78, T2: b = 4.72, T3: b = 3.09, F = 1.36, p = 0.269). Finally, no significant interaction between group and time was found, suggesting that the change in aggression over time was similar in both groups (T1: b = −2.79, T2: b = −4.37, T3 = −2.7, F = 0.69, p = 0.565).
Regulatory emotional self-efficacy (RESE)
Considering regulatory emotional self-efficacy, no significant fixed group effect was found for regulatory emotional self-efficacy (b = 2.12 (SE = 3.95), F = 0.413, p = 0.526). Moreover, a significant fixed effect of time was found (F = 3.70, p = 0.016), indicating that regulatory emotional self-efficacy scores differed across time points. However, none of the individual time phases (T1: b = −.05, T2: b = −1.36, T3: b = .231) reached statistical significance when examining the parameter estimates (all were p > 0.05), suggesting that the overall time effect for both groups was driven by small, non-significant changes across multiple time points. Finally, no significant interaction effect between group and time was found (T1: b = 0.51, T2: b = −1.03, T3: b = 0.66, F = 0.49, p = 0.691).
Treatment readiness (CVTRQ)
Regarding treatment readiness, no fixed effect for group was found (b = −1.28 (SE = 2.80), F = 0.63, p = 0.435). Additionally, no fixed time effect was found (T1: b = −1.76, T2: b = −3.75, T3: b = −2.64, F = 1.90, p = 0.138). Finally, no significant interaction effect between group and time was found for treatment readiness (T1: b = −2.65, T2: b = 1.85, T3: b = 1.02, F = 1.18, p = 0.325).
Engagement and adherence
Additionally, we examined whether the level of engagement is predictive of the number of completed lessons in the ‘Dealing with Aggression’ intervention, using a linear regression analysis. Table 4 shows the completion score for each participant as well as the mean scores of the participants from Group 1 on engagement after the first lesson (T0), the fifth lesson (T1) and the last lesson (T2), measured by the TWEETS questionnaire.
Means and standard deviation of the TWEETS a scores of the intervention group (n = 15).
TWEETS: TWente Engagement with Ehealth and Technologies Scale
The minimum score of the TWEETS is 9, and the maximum score is 45.
Of the 15 participants, nine completed at least 50% of the intervention, and five completed 100%. Linear regression analyses were run to investigate whether the scores on the TWEETS at T0, T1 and T2 could predict outcomes. For aggression, no significant predictive effects were found of engagement on T0 (B = −4.23, SE = 3.32, t(14) = −1.27, p = 0.424), T1 (B = 8.85, SE = 7.02, t(8) = 1.26, p = 0.427) and T2 (B = .23, SE = 0.80, t(4) = 0.29, p = 0.821). With regard to regulatory emotional self-efficacy, no significant results were found either on T0 (B = −3.62, SE = 5.53, t(14) = −0.65, p = 0.632), T1 (B = 10.92, SE = 11.69, t(8) = 0.93, p = 0.522) and T2 (B = −0.39, SE = 1.33, t(4) = −0.29, p = 0.82). Finally, for treatment readiness, no significant changes were found on T0 (B = 1.78, SE = 1.48, t(14) = 1.20, p = 0.443), T1 (B = −3.49, SE = 3.12, t(8) = −1.12, p = 0.464) and T2 (B = 0.90, SE = 0.36, t(4) = 2.53, p = 0.240).
Additionally, another linear regression was conducted to investigate whether engagement predicts the number of completed lessons of the intervention. Considering the TWEETS scores after lesson 1, the regression coefficient was not significant (B = −0.17, SE = 0.33, t(14) = −0.54, p = 0.599), indicating that early engagement was not a significant predictor for completing the whole intervention. With regard to the participants who completed at least five lessons, no significant relationship between engagement and completion rate was found either (B = −0.04, SE = 0.36, t(8) = −0.10, p = 0.924).
Qualitative results: experienced effectiveness
Of the 15 participants in the intervention group, 11 were interviewed. Four patients were not interviewed because they had left treatment before the interview could be scheduled. Additionally, nine therapists were interviewed. Table 5 shows the codes, subcodes and their corresponding definitions regarding the experienced outcomes.
Experienced effect of internet-based intervention ‘Dealing with Aggression’ by patients (n = 11) and therapists (n = 9).
The total number of times a code was mentioned in all focus groups and interviews.
The number of patients that mentioned a code, and (#) the total number of times the code was found in all focus groups.
The number of therapists that mentioned a code, and (#) the number of times the code was found in all interviews with patients.
Aggression
Eight out of 11 patients indicated that the intervention contributed positively to their aggression regulation. The intervention was perceived as useful for gaining more knowledge and insight into their aggressive behavior. One patient (P8) commented on this as follows: It did help me. To think about certain things, like the consequences of actions and how they affect others. It also helped me see how important it is to set boundaries. I can’t do that yet, but I’ve become aware of it. It has been useful for me. (Quote P8)
Amongst therapists, opinions were divided. Two out of the nine therapists observed that their patients had benefited from the intervention in terms of regulating their aggression. However, three other therapists reported no experienced benefits, partly because patients struggled to understand the lessons. Four therapists indicated that effectiveness varied per patient and that for some patients with more persistent patterns of aggression, the intervention was not effective at all. This was illustrated by a therapist who stated the following: It is hard to answer. I think some patients can reduce their aggression with the intervention, but most show really persistent aggressive patterns so changing that asks a lot. (Quote T2)
Regulatory emotional self-efficacy
Seven out of 11 patients believed that the intervention helped them gain more confidence in managing negative emotions in a healthy way. One patient described this as follows: Yes, in the sense that it's [completing the intervention] like a pat on the back I can give myself. I often hear, ‘Hey [name], we can see that you’re doing well, that you have it under control.’ And occasionally there are moments when I feel tension and aggression building up, but I can say, ‘Congratulations: you’re aware of it. Great! Now just take a step back, and you’ll be fine afterwards. Count to 10, smoke a cigarette, and then it's gone. (Quote P2) No, not really. I’m doing the intervention, but I can’t yet apply it in practice. And the intervention doesn’t take place in a real situation, so I understand it and can put it into words and even make a kind of plan of how I should handle it, but that hasn’t always worked out. It has improved somewhat, but I haven’t really felt good about it yet. (Quote P7)
Treatment readiness and treatment motivation
In the interviews, participants were told that treatment readiness refers to recognizing the need for treatment, while treatment motivation is the willingness to engage in it. However, most patients and therapists used these terms interchangeably. The four patients who did address treatment readiness did not identify a direct effect and mostly stated that their willingness to participate in treatment was already quite high. One patient (P11) explained this as follows:
Not really. I already have a good attitude, also within the group. That has not changed or improved due to the intervention. I think I do my best to gain as much as I can from my treatment. That has not changed. So no, I don’t think the intervention has helped in that part.
The level of treatment readiness was also mentioned less frequently by therapists. On the occasions when this concept was discussed, two therapists were positive about it: Of course it works best if it is already high, but it can further increase. When you start in the treatment room patients often say they don’t have a problem, but the people around them have a problem. And by using conversations and how the intervention can give insight in daily situation someone can start to think: ok maybe I do have to address my behavior. (Quote T4) I’m not sure whether motivation really increases. I think the main advantage is that you can get started right away, and it requires relatively little time from the therapist to help someone with a specific request for help. But that request or goal does need to be there already. (Quote T5)
Treatment efficiency
Therapists and patients mentioned that the use of the internet-based intervention did not result in fewer treatment sessions. All patients attended the regular (group or individual) sessions in parallel with the intervention. Three therapists did mention that the topics covered in the intervention aligned well with their own (group) therapy, leading them to notice that some themes resonated more quickly with patients who completed the intervention. They mentioned that this could potentially result in fewer treatment sessions in the long term – especially for motivated patients who were a bit further along in treatment. However, this remained a potential as opposed to an actual observed benefit.
Synthesis of effectiveness outcomes
In Table 6, an overview of the quantitative and qualitative outcomes is provided on a patient level – only for those who participated in the interviews. If participants indicated that the internet-based intervention was helpful in any way in addressing one of the study outcomes, their response was categorized as ‘added value’ in this table. We did not differentiate between major or minor perceived benefits, as doing so would require quantifying qualitative data and involve excessive interpretation by the researchers. If participants expressed uncertainty about whether the intervention improved any of the outcomes, their response was categorized as ‘unsure’. Finally, if participants reported experiencing no effects from the intervention, their response was categorized as ‘no added value’. In general, participants from the intervention group were mostly experiencing benefits from the intervention on aggression, self-efficacy and treatment readiness. Although no significant effect on aggression (AVL-AV) was found, most participants from the intervention group did show a decline in scores on aggression and found the intervention to be helpful in decreasing aggression. More variation in effect scores was found on self-efficacy (RESE). Two of the four participants who reported no added value or were unsure also scored lower on self-efficacy over time. Finally, variation in effect scores was also found on treatment readiness (CVTRQ). Overall, participants were reporting benefits of the intervention on treatment readiness. The two participants that reported no added value of the intervention on treatment readiness also showed a decline in effect scores.
Individual effectiveness of internet-based intervention ‘Dealing with Aggression’ based on the quantitative and qualitative data (n = 11).
AVL-AV: Aangepaste Versie van de Agressie Vragenlijst; RESE: Regulatory Emotional Self-Efficacy; CVTRQ: Corrections Victoria Treatment Readiness Questionnaire.
Qualitative results: reasons for (in)effectiveness
In the interviews, additional attention was paid to reasons for why the intervention was or was not effective due to the aforementioned issues with inclusion. In Table 7, the identified codes related to reasons for (in)effectiveness are provided, including a definition and one or two example quotes from therapists (T) and patients (P). For overview purposes, the codes are categorized into three categories: characteristics related to the intervention, the patient and the treatment context.
Reasons for ineffectiveness of intervention, according to patients (n = 11) and therapists (n = 10), with the total number of participants that mentioned a code and the total number of quotes.
The total number of times a code was mentioned in all focus groups and interviews.
The number of patients that mentioned a code, and (#) the total number of times the code was found in all focus groups.
The number of therapists that mentioned a code, and (#) the number of times the code was found in all interviews with patients.
In Table 8, reasons for effectiveness of the intervention are provided, again categorized into characteristics of the intervention, patient and context. Quotes from both therapists and patients are provided.
Reasons for effectiveness of intervention, according to patients (n = 11) and therapists (n = 10).
The total number of times a code was mentioned in all focus groups and interviews.
The number of patients that mentioned a code, and (#) the total number of times the code was found in all focus groups.
The number of therapists that mentioned a code, and (#) the number of times the code was found in all interviews with patients.
Discussion
This mixed-methods study examined whether adding the internet-based intervention ‘Dealing with Aggression’ to forensic psychiatric outpatient treatment improved aggression, emotional self-efficacy and treatment readiness. We also assessed whether participant engagement predicted adherence and, using a mixed-methods design, conducted interviews to explore perceived effectiveness and reasons for (in)effectiveness. Due to very low usage in practice, therapists’ low motivation to start using the intervention with their patients and a low number of patients who were willing to use the intervention, the intervention design was changed from an RCT to a pilot RCT. While the number of participants fits this design, the study does not have enough power to detect differences between both groups. In line with experiences of therapists, no changes between groups on the outcome measures were found. Only a significant change in regulatory emotional self-efficacy between time points was found for both groups. This result can be explained by the treatment itself. Several patients reported that they experienced benefits, mostly because parts of what they learned during treatment were repeated in the intervention, combined with new insights, resulting in an increased learning effect. Much-mentioned reasons for ineffectiveness by both patients and therapists were suboptimal integration in face-to-face treatment; a high level of required reading, writing and reflective skills; and low motivation of patients and therapists. While conclusions have to be drawn with care due to the low number of participants, no convincing proof for effectiveness of the internet-based intervention in forensic psychiatry was found based on the quantitative analyses. While most participants were relatively positive about the intervention, the interviews indicated that the intervention is generally not of added value for the forensic psychiatric population. However, new insights into why, for whom and for which treatment objectives these types of interventions do (not) work were identified, resulting in points of improvement for intervention design and implementation in practice.
For whom do internet-based interventions work?
In general, there is a need to gain insight into why internet-based interventions work for some, but not all, patients. This is especially relevant for forensic psychiatry – not just because very little is known yet about technology in treatment but also due to the high level of heterogeneity and difficulty in including these patients in (internet-based) treatment in general.8,44 This study pointed into the direction of two factors that might play a role in solving the ‘what works for whom’ puzzle: treatment motivation and engagement. First, during inclusion, only patients who were motivated and often further along in their treatment process were willing to participate. To illustrate, the scores on treatment readiness – a different but somewhat related construct – in this study were around 65, while those in another Dutch study in forensic patients were about 5 points lower. 45 The interviews showed that the main reason for the overrepresentation of motivated patients seems to be that they are more willing to take every opportunity to further change their behaviour and put in extra time and effort, which in their case included the internet-based intervention. Since treatment motivation is generally quite low in forensic populations, 16 this finding shows that internet-based interventions might not be feasible and suitable for a large share of forensic psychiatric patients – especially those who receive mandatory treatment and are at the beginning of their treatment. A study on an online cognitive behavioural relaxation programme found that motivation was indeed related to adherence and treatment outcomes, but more research is necessary to further investigate these relationships – especially in populations with generally low treatment motivation, such as forensic psychiatric patients. 46 Second, engagement did not seem predictive of the outcomes on aggression, self-efficacy or treatment readiness, nor did it predict adherence. While other research has shown that engagement can predict effectiveness, not much is known about the relationship between engagement and adherence.27,47 Because research shows that engagement seems to be a useful predictor of effectiveness and the interviews in this study also showed the importance of being engaged with a technology, future research on the role of engagement in internet-based – and other types – of interventions in forensic psychiatry is warranted.
The importance of implementation
Despite the participating organizations’ intention to use the internet-based intervention for all patients, there were many problems with inclusion. In line with this, earlier log data research has shown that internet-based interventions are not widely used in a forensic organization, despite many implementation efforts. 8 Furthermore, log data from the four participating organizations collected during this study showed the same patterns: usage of internet-based interventions did not increase over time, even during and after the Covid-19 pandemic. 48 While this study and previous research identified several factors that contributed to low uptake in practice, one of the most important reasons seems to be that many therapists do not see the added value for most of their patients.8,28 The main explanation for this is that the intervention was deemed to be too complicated for most forensic patients. Another explanation was that the intervention was almost never discussed during treatment sessions – this often happened only when a patient initiated the topic themselves. Indeed, earlier research has shown that a blended approach in which the therapist actively integrates the intervention in treatment seems to be an important precondition for implementation and effectiveness.49,50 In line with this, therapists indicated that integration is challenging, amongst other things due to limited time, suboptimal skills and lack of perceived support by management. Consequently, in line with implementation models such as the nonadoption, abandonment, scale-up, spread and sustainability (NASSS) framework, this study highlights the importance of multilevel implementation programmes, including better integration into treatment protocols, adequate therapist skills and a more active role for management.8,51
Another specific recommendation is to reassess the place of internet-based intervention in patient journeys. Currently, the underlying idea in forensic practice is to start using these interventions as soon as possible, possibly even to bridge the waiting list time. 4 This is contradictory to the findings in this study, which shows that most patients who were open to using the intervention were further along in their treatment trajectory. Therapists indicated that the intervention was less suitable for patients with less insights and treatment motivation, which is often the case before and at the beginning of treatment. An advantage of placing these interventions later in the treatment process is that these interventions can be used as a ‘refresher’ or summary of what was learned during treatment. In that way, they could even be used as a stand-alone tool, requiring very little involvement of therapists and contributing to the efficiency of care. A potential limitation is that removing therapist support could result in lower engagement and adherence rates. 50 Yet it does overcome the barrier that therapists often do not have time to discuss the intervention. More research on the potential of internet-based interventions to speed up the final stage of treatment is required.
Implications for intervention design
The interviews and earlier research both show that an important limitation is that internet-based interventions require too much reading, writing and reflective skills from patients – which many do not fully possess.8,44,52 To illustrate, therapists indicated that many questions from the intervention were too difficult and abstract for patients. This is not just an issue for forensic psychiatry, but for many psychiatric patients in general, since it is estimated that about 40% of people with a severe mental disorder also have an intellectual disability. 53 This raises the question of whether these types of language-based interventions are the best fit for vulnerable, hard-to-involve patient populations. Consequently, a recommendation for practice is to first discuss with a patient whether it is feasible and desirable to start working on an internet-based intervention. In this study, some therapists indicated the importance of shared decision-making to ensure that the patient gets the opportunity to provide their opinion and prevent top-down decision-making by the therapist when discussing internet-based interventions. 19 Second, it seems to be worthwhile to explore other, less language-based possibilities to deliver treatment to patients. The use of more experiential technologies such as wearables and virtual reality might be more suitable for most forensic patients. 52 Indeed, a study in the Netherlands showed that therapists and patients found these types of interventions most promising – partly because they were expected to result in more treatment motivation. 28 Furthermore, another recommendation that some participants gave was that interventions should be shorter and focused on simple topics, preferably ones for which there was not enough time during face-to-face treatment sessions. Examples were lifestyle themes such as sleep, physical activity or simple relaxation exercises. The focus on single-topic, short interventions is in line with earlier research, which recommended to create mental health apps that are more in line with commercial apps, which are often designed for one single purpose with not many sub-topics. 54 Consequently, a conclusion of this study is that a good fit between the design of a technology and the skills and preferences of a patient is a necessary precondition for usage and effectiveness. In order to overcome these design-related barriers, co-creation in which patients are involved from the start is recommended. 55
Strengths and limitations
The main strength of this study was its mixed-methods design. The addition of in-depth interviews with patients resulted in a more nuanced explanation of (in)effectiveness and implementation factors. Furthermore, the addition of the therapist perspective yielded more overarching insights in the perceived effectiveness and use of internet-based interventions in general. An important limitation of this study was the limited sample size: because of the aforementioned problems with inclusion due to low usage of the intervention and resistance of therapists and patients, we changed the design into a pilot RCT, despite a very active inclusion strategy. This means that this study is underpowered to show differences between conditions, and no robust conclusions can be drawn based on the quantitative data. Moreover, because of the heterogeneity of the included patients in terms of diagnosis, form of received treatment and earlier treatment, generalization to Dutch forensic outpatient care in general is not possible. Furthermore, due to major differences of forensic care between countries in the way forensic treatment is shaped, 56 conclusions are also not generalizable to an international setting. Regardless, this study still yielded valuable initial insights and starting points for optimizing intervention design and implementation.
Additionally, a selection bias was present in this study. Most patients who wanted to participate were motivated for their treatment, while it was more challenging to include unmotivated patients with less progress in their treatment. Because therapists were responsible for including patients, there is no clear overview of how many patients were invited to participate, what percentage declined and what their reasons were. However, the interviews with the therapists did provide some insight into why patients did not participate. Consequently, this sample is not representative of the forensic psychiatric patient population, resulting in, for example, more positive experiences in the included population.
A possible limitation of the interviews is that participants could have contributed effects to the intervention, while these effects were actually partly or fully caused by the in-person treatment. Because the intervention was used as an integrated part of treatment, separating intervention and treatment effects might have been difficult, meaning that conclusions about experienced effectiveness of the intervention should be drawn with care – especially because the quantitative part of study was underpowered. Another limitation concerns the inclusion of participants receiving group-based aggression regulation therapy. In group settings, therapists reported to have less opportunity to discuss each lesson during face-to-face sessions, which may have influenced how these patients experienced the integration of the online component of the intervention. Finally, measuring aggression and related concepts with self-reported questionnaires in a valid and reliable way is very challenging. This can be illustrated by discrepancies in quantitative and qualitative effects: did a patient experience an effect that was not there, or were questionnaires unable to capture them?
Conclusion
This mixed-methods pilot RCT showed no effects of an internet-based intervention ‘Dealing with Aggression’ on any outcome measure. Qualitative data showed that some, but not all, patients experienced benefits. The main reason for this was that they viewed the intervention as a refresher of topics that were already discussed in treatment. Treatment motivation is a potential predictor for adherence and effectiveness and should be further investigated. Difficulties with inclusion and interviews with therapists showed that usage of internet-based interventions in clinical practice is very low – despite many active efforts by organizations. A reason for this is that the language-based approach does not fit a large share of the often unmotivated, low-literate patient population. While conclusions have to be drawn with much care, this study indicates that the internet-based intervention ‘Dealing with Aggression’ does not seem to have much added value for most forensic psychiatric patients. This not only became clear from quantitative and qualitative findings but also by the low usage of this intervention in practice – which might be explained by low perceived added value by therapists.
Supplemental Material
sj-docx-1-dhj-10.1177_20552076261415917 - Supplemental material for The added value of an internet-based intervention for treatment of aggression in forensic psychiatric outpatients: A mixed-methods pilot randomized controlled trial
Supplemental material, sj-docx-1-dhj-10.1177_20552076261415917 for The added value of an internet-based intervention for treatment of aggression in forensic psychiatric outpatients: A mixed-methods pilot randomized controlled trial by Hanneke Kip, Lisa Klein Haneveld, Iris ten Klooster and Saskia M Kelders in DIGITAL HEALTH
Supplemental Material
sj-pdf-2-dhj-10.1177_20552076261415917 - Supplemental material for The added value of an internet-based intervention for treatment of aggression in forensic psychiatric outpatients: A mixed-methods pilot randomized controlled trial
Supplemental material, sj-pdf-2-dhj-10.1177_20552076261415917 for The added value of an internet-based intervention for treatment of aggression in forensic psychiatric outpatients: A mixed-methods pilot randomized controlled trial by Hanneke Kip, Lisa Klein Haneveld, Iris ten Klooster and Saskia M Kelders in DIGITAL HEALTH
Supplemental Material
sj-docx-3-dhj-10.1177_20552076261415917 - Supplemental material for The added value of an internet-based intervention for treatment of aggression in forensic psychiatric outpatients: A mixed-methods pilot randomized controlled trial
Supplemental material, sj-docx-3-dhj-10.1177_20552076261415917 for The added value of an internet-based intervention for treatment of aggression in forensic psychiatric outpatients: A mixed-methods pilot randomized controlled trial by Hanneke Kip, Lisa Klein Haneveld, Iris ten Klooster and Saskia M Kelders in DIGITAL HEALTH
Footnotes
Acknowledgements
The authors would like to thank all patients and therapists for participating in this study. They would also like to acknowledge and give thanks to all project team members for their contribution in setting up the study. Finally, the authors would like to thank Minddistrict for providing their log data and the TWEETS output.
Ethical approval
This study was approved by the medical ethical board Oost-Nederland of the Radboud University Medical Centre (NL80846.091.22) and was conducted in accordance with the principles of the Declaration of Helsinki. The study is registered at ClinicalTrials.gov (NCT05711342).
Contributorship
Hanneke Kip: funding acquisition, conceptualisations, methodology, formal analysis qualitative data, writing original draft, supervision, project administration. Lisa Klein Haneveld: conceptualisations, methodology, data collection, formal analysis qualitative data, formal analysis quantitative data, writing original draft, project administration. Iris ten Klooster: methodology, formal analysis quantitative data, writing – review & editing. Saskia M Kelders: methodology, supervision, writing – review & editing.
Funding
The authors disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was supported by Kwaliteit Forensische Zorg (grant number 2021-157).
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Guarantor
Hanneke Kip and Lisa Klein Haneveld are both guarantors of this study.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
