Abstract
There is a need to improve a) identification and monitoring of people with mental illness on probation and b) understanding of the impact of interventions on mental health outcomes for the probation population. If data were routinely collected using validated screening tools and shared between agencies, this could inform practice and commissioning decisions, and ultimately it could improve health outcomes for people under supervision. The literature was reviewed to identify brief screening tools and outcome measures that have been used in prevalence and outcome studies conducted with adults on probation in Europe. This paper shares findings from the UK-based studies in which 20 brief screening tools and measures were identified. Recommendations are made based on this literature regarding suitable tools for use in probation to routinely identify a need for contact with mental health and/or substance misuse services and to measure change in mental health outcomes.
Keywords
Introduction
Contact with a probation service can provide an opportunity for practitioners in the health and justice field to monitor and potentially help to improve the mental health of people that are often marginalised and are unlikely to access support until they are at crisis point. However, in the UK, a recent joint thematic inspection has highlighted numerous difficulties in relation to supporting people with mental health needs and disorders in the criminal justice system. These include failure to identify people with mental health needs throughout the criminal justice pathway, a need for a memorandum of understanding to improve data-sharing between agencies, a shortage of mental health services in England and Wales and long waiting lists for the services that are available (HM Inspectorate of Probation et al., 2021). Research into the prevalence of mental health needs and the efficacy of mental health interventions within the probation population and improvements to routine data collection around mental health needs in probation practice could begin to address these difficulties including through routine data and research findings being used to inform commissioning and service delivery decisions (Public Health England, 2020), ensuring that practice is based on the latest evidence-base and demonstrating the need for additional investment in mental health provision.
People on probation with mental illness are likely to also experience drug and/or alcohol misuse, unstable accommodation or homelessness, difficult family relationships, low levels of literacy and health literacy and a lack of access to and/or understanding of technology (Power, 2020; Revolving Doors Agency, 2017; Sirdifield et al., 2019). This complexity of need together with challenges like poor past experiences of service access and a lack of GP registration can form barriers to service access. Continuity of care as people progress through the criminal justice pathway is also problematic (HM Inspectorate of Probatiodn et al., 2021; HMIP, 2016). If probation staff understand an individual’s mental health needs and how they may relate to offending behaviour, they can affect positive change by signposting to relevant provision where it is available, including via Community Sentence Treatment Requirements if appropriate, and supporting continuity of care for people released from prison (HMPPS and NPS, 2019: p. 6).
Conversely, a lack of understanding of the mental health needs of those on probation, barriers to access and a lack of services that meet needs can result in poor health outcomes for this population and avoidable use of crisis care (Brooker et al., 2009; Public Health England, 2020; Revolving Doors Agency, 2017).
However, there are numerous ethical and practical challenges in improving our understanding of this population’s needs. ‘Gold standard’ screening tools for mental illness can be time-consuming to complete, and screening may need to be undertaken by a qualified clinician. Such resources are not always available to probation staff or researchers, so various methodological approaches have been employed to date in practice, research studies and evaluations.
Currently, data on mental health needs collected by probation staff in England and Wales are often based on simple self-report rather than validated screening tools. More work is required to ensure that data are collected consistently and in a research-informed way that is helpful to those commissioning and providing services (HM Inspectorate of Probation et al., 2021; Public Health England, 2020). By ensuring that the tools used to measure and assess mental health are valid and reliable, probation services can have greater confidence that the support they offer is effective and supported by accurate data.
In this paper, we report findings from a literature review that aimed to identify brief screening tools and outcome measures that have been used with adult probation populations to establish the prevalence of mental illness or symptoms of mental illness, or to investigate the effectiveness of interventions aimed at improving mental health outcomes for this population. The search encompassed studies conducted in European countries, but due to the volume of the literature and variations in the role of probation services in relation to mental health, our focus in this paper is primarily on UK studies. Those reported from other parts of Europe are briefly summarised for information. We report key findings from the UK studies to show how the measures have been used in practice and research and consider the meaning of scores on these measures and any information reported on the benefits or challenges of their use with probation populations for health and justice practitioners and for researchers.
Materials and methods
Although this is not a full systematic review, our approach was guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We searched PsycINFO, MEDLINE, IBSS, CINAHL, AMED, ASSIS and Scopus (November 2021). To review as many potentially useful tools and measures as possible, the scope of the search was broad. No limit was set on the date of the included studies. The search strategy for PsycINFO is in Supplement Appendix 1 and was translated for the remaining databases. We also drew upon our existing knowledge of the field and a recent systematic review that aimed to identify the literature about approaches to improving mental health outcomes for adults on probation and/or the health needs of this group (Brooker et al., 2020).
To be included in the review, studies had to have used a brief screening tool or outcome measure to assess a common or serious mental illness/disorder (including substance use disorders and personality disorders) and to have applied the screening tool or outcome measure to adults on the caseload of probation services (including people on parole). Additionally, studies had to be published in English and conducted in a European country. Studies were excluded if they involved participants under the age of 18, were focussed on other settings/elements of the criminal justice system (e.g. prison), were not empirical studies or did not directly administer mental health screening tools.
Upon completing the search, the studies were further filtered by location. This paper presents findings from UK-based studies. Details of studies reported from other parts of Europe are in Supplement Appendix 2.
A list of brief screening tools and outcome measures was then compiled, and data were extracted from the identified studies together with associated publications around the validity of the measures into a bespoke data extraction form.
Results
A total of 15 UK-based papers or reports were included in the review, which employed 20 different brief screening tools and measures in total (see Figure 1 and Tables 1 and 2). This included four measures of substance misuse (AUDIT, CAGE, DAST and UNCOPE), 9 measures of mental state/symptom severity (GHQ-12, SF-36, geriatric depression scale [short form], CORE-34/OM, GAD-7, K6, K10 and PHQ-9), two tools for identifying ‘likely cases’ of mental illness (MINI and SAPAS) and individual measures of needs (CANFOR-S), service use (CSSRI-EU), self-efficacy (GSES), problem-solving ability (SPSI-R) and functional impairment (WSAS). PRISMA chart. Measurement in mental health and probation studies. Overview of included papers and measures used.
Prevalence and descriptive studies
We identified six prevalence studies of substance misuse and/or mental illness in probation samples (Brooker et al., 2011, 2012; Fitton et al., 2018; Hatfield et al., 2004; Newbury-Birch et al., 2009; Pluck and Brooker, 2014) together with three papers focussing more on the use of particular measures (Pluck et al., 2012, 2015; Shaw et al., 2012) (see Table 2), a health needs assessment (Brooker et al., 2009) and a study providing insight into the prevalence of suicidal ideation and attempts in a probation population (Brooker et al., 2021). It is difficult to directly compare findings from these studies due to variation in the settings, approaches to sampling and screening tools/measures employed.
Alcohol and drug use
Four studies used AUDIT to identify hazardous and harmful alcohol use and possible dependence on alcohol (Table 2). This ‘gold standard’ tool is quick to complete, and the studies reported prevalence rates of harmful, hazardous or possibly dependent drinking of 56% (Brooker et al., 2012), 31% (Fitton et al., 2018) and 69% (males) and 53% (females) (Newbury-Birch et al., 2009). The latter study compared findings from AUDIT with those from the Offender Assessment System (OASys – the main assessment tool used by probation staff in England and Wales to identify risks and needs). The use of AUDIT in probation practice was recommended as OASys under-estimated the prevalence of alcohol use disorders.
In terms of drug misuse, several measures were identified, all of which are straightforward to use and have minimal administration times (see Table 1 for details of the number of items in each measure). Brooker et al. (2012) reported that 12.1% of participants scored above the cutoff score for problematic drug use on the DAST-20, whilst Fitton et al. (2018) reported that 3% of participants scored above equivalent cutoff score for the DAST-10. Finally, scores from UNCOPE and CAGE in a health needs assessment in two UK regions indicated that ‘almost half of the sample were identified at risk of alcohol abuse or dependence while 39 per cent of the sample was at risk of substance misuse’ (Brooker et al., 2009: p. 49).
Mental health
It is also possible to conclude that there is a high prevalence of mental illness and dual diagnosis within the probation population from these studies – those using the MINI reported overall prevalence rates for current mental illness of 39% (with 72% of these cases also having a drug or alcohol problem) (Brooker et al., 2012) and 47% (Fitton et al., 2018). In another linked study based on data from the MINI, Pluck and Brooker (2014) report on the lifetime and 1 month prevalence of deliberate self-harm, stating that 32.4% (95% CI = 25.6–40.0) of the overall study sample had attempted suicide in their lifetime (Pluck and Brooker, 2014 360), with 5.2% reporting deliberate self-harm within the past month, and 14.5% reporting having ideas around self-harm during that month. Overall, 8.7% (95% CI = 9.4–25.0) of participants were classified as ‘high’ suicidality risk.
Brooker et al. (2012) also explored ‘likely caseness’ of personality disorder using Standardised Assessment of Personality – Abbreviated Scale (SAPAS), reporting a prevalence of 47% based on scoring 3+ on this measure. The validation of this measure with a probation population and the characteristics of those with probable personality disorder are expanded upon in other papers produced from this study (Pluck et al., 2012, 2015), showing that although SAPAS is unable to distinguish individual types of personality disorder, it has potential as a brief screen for probable cases in probation practice. The optimal cutoff score for identifying cases of personality disorder using the SAPAS was established as 3 – with a sensitivity of 0.73 and specificity of 0.9 (Pluck et al., 2012). The use of SAPAS was also explored when compared to the OASys PD screen by Shaw et al. (2012). Here, 40.3% of the sample was classified as likely PD cases using SAPAS, whereas 15.1% screened positive using the OASys PD screen. SAPAS is recommended alongside the OASys PD screen as it is viewed as a useful tool but may have limitations when it comes to identifying the emotionally unstable antisocial personality disorder subgroup (Shaw et al., 2012: p. 164).
A health needs assessment conducted in two regions of the UK using various measures including version two of the short form 36 (SF-36) showed that the mental and physical health and functioning of people on probation ‘was significantly poorer than that of both the general population and manual social classes using comparative standardised data derived from the Third Oxford Healthy Life Survey’ (Brooker et al., 2009: p. 49).
Brooker et al. (2021) analysed data from 274 people on probation in a Community Rehabilitation Company who received a Kessler 6 score of 12+ indicating a need for psychological intervention during the study period to investigate variables that could be associated with suicidality. The sample was divided into a current suicidal ideation group (41%), a group with previous suicide attempts (36%) and a ‘no history’ group (23%). Those that had attempted suicide and those with suicidal thoughts were significantly more likely to have a higher score than the ‘no history’ group for psychological distress (K6 scores of 25+ for 38% of the attempt group, 36% of the ideation group and 15% of the no history group), generalised anxiety (GAD-7 scores of 15+ for 60% of the attempt group, 52% of the ideation group and 37% of the no history group), low mood (PHQ-9 scores of 20+ for 38% of the attempt group, 35% of the ideation group and 12% of the no history group) and functional impairment (WSAS score above 10 for 54% of the attempt group and 34% of those with no history).
Finally, in Hatfield et al.‘s study (2004), staff recorded 25.1% of approved premises (AP) residents as having a known psychiatric diagnosis. The GHQ-12 was used to measure psychological distress, and 29.9% of residents that were not receiving mental health support scored 5+ on this measure – indicating psychological distress.
Needs and service receipt
Findings from Brooker et al. (2011) also suggest that those with a current mental illness are likely to have a higher level of need (a mean score of 10.53 on the CANFOR-S) than those without (a mean score of 4.59). Results from the CSSRI-EU indicate that they are unlikely to access mental health services, with 60% of those with a current mood disorder, 59% of those with a current anxiety disorder, half of those with a current psychotic disorder and 55% of ‘likely cases’ of personality disorder not reporting accessing any mental health service (Brooker et al., 2011).
Outcome studies
In addition, we identified four outcome studies (Callender, 2021; Fowler et al., 2020; Long et al., 2018; Newbury-Birch et al., 2014). Three of these papers relate to the use of Mental Health Treatment Requirements (MHTRs), which can be recommended by the courts, but despite the high prevalence of mental illness in the probation population have rarely been used.
First, Long et al. (2018) piloted an approach to improving uptake of MHTRs in one probation area. Consecutive probation clients were screened for psychological distress using the K10, and eligibility was restricted to those scoring 20+ with ‘primary care-level’ mental health needs. Those with more severe needs were referred elsewhere if they were not already in contact with services. This was a small uncontrolled study, with just 48 people completing treatment during the study period, but these clients showed improvement across a range of mental health measures between baseline and 6-month follow-up, including significant reductions in those scoring above the threshold on measures of psychological distress, depression and anxiety (Table 2).
Second, Fowler et al. (2020) conducted a follow-up to this pilot after a full service had been commissioned. The K6 replaced the K10 as the initial screening tool, and the referral route was extended so that Probation Officers could directly refer those scoring 13+ to the service, rather than needing an MHTR to be recommended by the court. Those with symptoms of psychosis or high risk of suicide were not eligible for the service, which was delivered by a third sector provider based within probation. This study illustrates the difficulty in engaging people with this type of intervention – 569 people were referred to the service within the study timeframe, but just 75 completed all 12 one-to-one sessions, which were based on a cognitive-behavioural therapy manual that was tested in the pilot. Again, statistically and clinically significant reductions in scores were recorded across a range of measures (Table 2) with the numbers receiving a clinically significant score for general distress, anxiety, low mood and difficulties with social adjustability reducing by approximately 45%, 40%, 55% and 39%, respectively. Thirdly, Callender (2021) investigated changes in aspects of mental health after people completed an MHTR and similarly found reductions in average measures of global distress, anxiety and depression.
Newbury-Birch et al. (2014) reported findings from a randomised controlled trial comparing a control group that received feedback on screening for harmful or hazardous alcohol use and an information leaflet with a) a group receiving this plus 5 minutes of structured brief advice and b) a group receiving the feedback, leaflet, structured brief advice and 20 min of brief lifestyle counselling for reducing hazardous or harmful drinking. As there were no significant differences between groups in terms of the proportion scoring less than eight on AUDIT at six or 12 months, the study concludes that the addition of brief advice or lifestyle counselling did not produce any extra benefit in reducing harmful or hazardous drinking (Newbury-Birch et al., 2014: p. 540).
Discussion
Our interest in this topic arose when considering which measures to employ in a (currently ongoing) study aiming to introduce a research-informed approach to identifying health and social care needs amongst people on probation and increase understanding of the needs of the probation population and the extent to which they are being met by current service provision. When designing the study, we needed to consider which measures would be practical for use by probation staff and could potentially provide meaningful data for service commissioners and providers.
This review of mental health outcome measures used in the United Kingdom to assess the mental health of probationers is the first ever undertaken. We identified only 20 such measures which, in essence, measured substance use, mental health symptoms, likely mental illness/disorder and diagnosis, receipt of services and individual measures of needs. This small group of 20 outcome measures/measurement tools, laid out in Table 1, has been reported in 15 studies since 2004. This is an indication of how little research has been undertaken in the field of probation and mental health world-wide as these findings chime with the results of three systematic reviews of mental health, suicide and substance misuse recently reported (Brooker et al., 2020; Sirdifield et al., 2020a, 2020b).
The breakdown of the type of research where outcome measures are reported shows that just four studies examined outcomes as the result of an intervention; 10 reported prevalence and one report concerned a health needs assessment. Three of the four outcome studies used simple pre- and post-scores to measure the impact of referral for an MHTR and were uncontrolled. These studies would be classified as quasi-experimental pre- and post-test designs and are generally considered to be weak. Campbell et al. (1963) have classically described the problems associated with such studies as history, maturation effects, testing itself, regression to the mean and the loss of subjects over time. The fourth outcome study employed a randomised controlled trial (Newbury-Birch et al., 2014), a much stronger design, to examine the impact of providing advice to problem drinkers on probation. Although a high proportion (68%) of the sample was retained at 6-month follow-up, there were no significant changes in mean AUDIT scores following the intervention. Overall, it is important to note too that none of those four UK studies provided an intervention for people with a serious mental illness. This is in stark contrast to the literature in the United States where the role of the probation officer with seriously mentally ill people has been extensively reported (Epperson et al., 2014). This represents a significant and important gap in the UK literature.
Varying estimates of the frequency of serious mental illnesses were obtained in the prevalence studies, for example, Brooker et al. (2012) found that 11% experienced a psychosis and 14.5% a major depressive illness. Fitton et al. (2018), in a smaller sample of 33 older probationers (aged 50 and over), reported that 3% had a diagnosis of a psychosis although this figure rose to 19% when psychosis was considered over a lifetime. Both studies used the MINI to diagnose psychosis.
In a recent review of outcome measures used in the forensic mental health services, the primary objective was to identify ‘instruments which are clinically relevant, to increase applicability to real world settings’ (Ryland et al., 2021: p. 4). The review addressed all key criteria outlined by the COnsensus-based Standards for health Measurement INstruments (COSMIN) group which has developed taxonomy to define the various qualities of outcome measures (Mokkink et al., 2010). One key element of the COSMIN criteria is content validity which seeks to establish if the instrument measures what it is intended to measure. Furthermore, this should reflect those outcomes that are most important for stakeholders, including service users and relatives. Judged by this yardstick, the only measure that shows content validity of all those we review above in probation mental health studies is the CANFOR-S. This was tested with 60 forensic mental health service users, and all items were found to be at least moderately relevant (Thomas et al., 2008).
So, what does the review teach us about measures that probation staff might use routinely to assess need for mental health services in their populations? There are a small number of validated measures that have been successfully used in probation settings in the UK. It is important to note that these measures may not have been validated specifically with probation populations. This could potentially be an area for future research. However, the results of testing with other populations, including those within other areas of the criminal justice system are encouraging (see references in Table 1). Whilst our conclusions are based on a limited number of studies, these studies show the value of using these types of measures in probation practice and research. For the screening of primary care mental health, the K6 and K10 are obvious candidates to use in routine assessment, and maybe SAPAS if personality disorder is important. Indeed, SAPAS has been recommended for use alongside OASys in order ‘to enhance sensitivity to antisocial cases and incorporate information about risk’ (Shaw et al., 2012: p. 156).
However, better measures of the specific severity of depression are the PHQ-9 (and maybe the GDS-15 for older people) and for anxiety the GAD-7. In the case of these two latter measures, one would need to ensure that clinical levels of anxiety and depression are being reached. For the seriousness of substance misuse, the short forms of DAST and AUDIT would be useful. Newbury-Birch et al. (2009) found that OASys alone seriously underestimated the number of people on probation with an alcohol-related need.
To obtain a sense of what needs clients had in relation to their mental health the CANFOR-S, which has been service-user assessed, would be most suitable. This leaves open the question of measures that might be employed to screen for/assess serious mental illness. Here, one option would be a systematic review of probation research conducted in the United States. Another option might be to use the Psychosis Screening Questionnaire (PSQ) (Bebbington and Nayani, 1995). It consists of five items, and Coid and Ulrich (2011) demonstrated its use in a sample of remand and sentenced prisoners drawn from 131 prisons. Using cutoff points of 3+ for men and 2+ for women, it was established that the PSQ was a valid tool when compared with diagnostic interviews. However, further use of the tool in probation would need to be rigorously examined.
Conclusion
It is an unavoidable conclusion of the recent thematic review of mental health in the criminal justice system that more should be done to help probation staff recognise mental illness. We have reviewed the UK (and European) literature to identify measures that have been used in previous prevalence/descriptive and outcome studies. We recommend a suite of measures which probation staff could easily be trained to use to identify conditions which, in some cases, OASys under-reports. These include, inter alia, brief measures for mental health, psychosis and personality disorder screening, depression, anxiety and alcohol and drug consumption.
Supplemental Material
Supplemental Material – Identifying mental illness and monitoring mental health in probation service settings
Supplemental Material for Identifying mental illness and monitoring mental health in probation service settings by Charlie Brooker, Coral Sirdifield and Thomas Parkhouse in the European Journal of Probation.
Supplemental Material
Supplemental Material – Identifying mental illness and monitoring mental health in probation service settings
Supplemental Material for Identifying mental illness and monitoring mental health in probation service settings by Charlie Brooker, Coral Sirdifield and Thomas Parkhouse in the European Journal of Probation.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This study/project is funded by the National Institute for Health Research (NIHR) [Research for Patient Benefit Programme (NIHR 201091). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.
Supplemental Material
Supplementary material for this article is available online.
Author biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
