Abstract
As government administrative data sets are increasingly made available for new (non-administrative) purposes, there is a need to improve access to such resources for voluntary and community organizations, social enterprises and private businesses for statistical analysis and evaluation purposes. The Justice Data Lab set up by the Ministry of Justice in the UK presents an innovative case of how administrative data can be linked to other data held by organizations delivering public services. The establishment of a unit within a secure setting holding evaluation and statistical expertise has enabled providers of programmes aimed at reducing re-offending to obtain evidence on how the impact of their interventions differs from that of a matched comparison group. This article explores the development of the Justice Data Lab, the methodological and other challenges faced, and the experiences of user organizations. The article draws out implications for future development of Data Labs and the use of administrative data for the evaluation of public services.
Introduction
Administrative data has long been routinely collected and held centrally by the public sector, although there has been relatively little progress in exploiting opportunities for its application for statistical analysis and evaluation purposes (Harper and Mayhew, 2012). This article explores the origin, operation and challenges faced by the Justice Data Lab (JDL), a pioneering initiative that has provided 117 (as of September 2014) analyses to charities, social enterprises, public and private sector organizations in England and Wales. The Ministry of Justice (MoJ), a ministerial department of the UK government, has established the JDL as an internal unit able to provide secure access to personal data. Datasets containing information on aggregate re-offending measures were provided to organizations to help them assess the effect of their programmes related to criminal justice. The service supplies measures of re-offending for cohorts of individuals provided by organizations working in criminal justice, alongside re-offending measures for a matched comparison group of offenders selected through propensity score matching. The resulting measures of re-offending are compared in order to provide evidence on the extent to which there has been a statistically significant change in re-offending amongst the target group.
In its linking of data held by smaller providers of public services with relevant government held data, the JDL represents an important innovation in evaluation practice. While there have been previous examples of the use of administrative data in evaluations (Riippa et al., 2014; Roos et al., 1979; Saunders and Heflinger, 2004), this is the first attempt to provide such an evaluation space at scale and to facilitate access to publicly-held administrative data for smaller organizations. The JDL therefore provides a valuable case study for the exploration of the various challenges encountered when using such sources for the evaluation of specific interventions.
This article addresses three main research questions: RQ1: How was the JDL established and developed? RQ2: How has it been used by service delivery organizations? RQ3: What are the methodological and organizational challenges and how have these been addressed? We draw on a range of published project documents, interviews with eight voluntary sector users of the service, five non-users (i.e. other voluntary sector organizations delivering similar services), four key informants involved in the design and use of the JDL, a feedback survey of users, and group discussions held at three meetings with users, non-users and data providers. Rather than being an evaluation of the JDL, the article explores how it has been set up and the challenges faced in the process. We show how emerging evaluation opportunities are dependent on the ability of key actors to span the boundaries between professional evaluation services, policy makers, public sector statistical services and academia.
The article specifically reflects on how those involved in implementing the JDL have had to negotiate three areas of methodological tension and policy/organizational challenge: conceptual challenges related to the use of re-offending as an indicator of the rehabilitation of offenders; the problem of small samples sizes and a related lack of statistical significance in some of the reports; and challenges experienced in matching and developing comparison groups. There are also challenges related to the motivations of public service delivery organizations to participate in the initiative and policy challenges related to the willingness of government departments to share their data more widely. The article concludes by drawing out implications for the future use of administrative data and effective approaches to providing such data in a safe and affordable manner. The experience of the JDL therefore has wider lessons for the development of affordable quasi-experimental evaluations in other important social policy areas, including education, crime, health and substance misuse. Such wider application would have significant implications for many non-profit organizations, social enterprises and private businesses involved in delivering public services as well as for public sector providers.
Context
The development of the Justice Data Lab needs to be understood in relation to trends in public policy since the early 1990s (and particularly so in the UK) to increase competition and organizational diversity in the delivery of public services (e.g. Dunleavy and Hood, 1994; Walsh, 1995). The development of such ‘quasi-markets’ for public contracts has resulted in the growth of private, non-profit and social enterprise involvement in service delivery alongside public sector provision. Hence the Transforming Rehabilitation programme of the Ministry of Justice (MoJ) has the stated aim of ‘opening up the market to a diverse range of new rehabilitation providers to get the best out of the public, voluntary and private sectors’ (MoJ, 2014b). Accompanying this national policy trend has been a policy emphasis on measuring the outcomes of public services with a view to improving the evidence base on ‘what works’ (Cabinet Office, 2012). There has also been a growing appetite amongst different types of service provider for new ways of evidencing their outcomes (Arvidson and Lyon, 2013; NPC, 2012) and a similar shift in the requirements of philanthropic foundations and individuals who increasingly expect to see measurable impacts resulting from their donations (Van Vliet et al., 2013). However, while organizations may want to increase their use of evaluation, many in the UK report that they lack specialist evaluation knowledge and capabilities (NPC, 2012). Similar issues have been raised internationally (e.g. Cousins et al., 2014, in the case of Canada).
The second contextual factor shaping the JDL has been the development of the administrative data agenda and the growing interest in ‘Big Data’ within the public sector (ADRN, 2012; Eversley and Mayhew, 2011; Harper and Mayhew, 2012). Policy makers are increasingly of the persuasion that the wider application of existing administrative data made possible by improved analytical tools represents a low cost and robust way to inform policy development and practice. The UK government has set out its intention to provide more open data to help increase the transparency of public service delivery and outcomes (Cabinet Office, 2012), an aim that is supported by the Government Statistical Service Data Strategy (GSS, 2013). This is resulting in the release of large amounts of data which is freely available at no cost, as long as legal requirements relating to privacy and confidentiality issues are complied with. Although some data remains too sensitive for such release, recent innovations may allow its use in policy without compromising individual privacy. In this respect there has been a growing use of administrative data in many countries to understand trends in populations (i.e. both people and businesses) and a related interest in applications for policy evaluation, particularly where there is a need for more robust counterfactuals and comparison groups (Morris and Herrmann, 2013).
The take-up of new sources of data and related innovations in data analytics by voluntary and community organizations around the world has been hindered by a number of factors, including ongoing difficulties in accessing relevant administrative data which remains safeguarded by Government (De Souza and Smith, 2014). Improving access to data faces a number of challenges, most notably with respect to the need to ensure anonymization and minimize the risk of any unintended data disclosures (Information Commissioner’s Office, 2012). Access also has to be combined with the expert statistical skills needed to both analyse the data and present it in a way that is comprehensible to non-statisticians.
Establishing and operating the Justice Data lab
The Ministry of Justice has a statutory mandate to protect the public and reduce re-offending, including through rehabilitating offenders and helping them overcome the social and life disadvantages that they may face. The one year proven re-offending rate 1 for adult offenders released from custody during October 2011 to September 2012 was 42.5 percent, and the equivalent figure for offenders starting a court order was 33.6 percent (MoJ, 2014c). There is rich administrative data held by the MoJ which records individuals’ interactions with the Criminal Justice System, including instances of re-offending.
A key factor in the genesis of the Justice Data Lab was a growing demand from voluntary and community sector (VCS) organizations for support to help them measure the impacts of their offender rehabilitation efforts. This was needed to both ensure that their interventions were having the desired effect and to identify any scope for improvement. There are particular difficulties with making re-offending data available widely; information about individuals, their criminal history and re-offending behaviour is classified as sensitive personal data by the Information Commissioner. In practice, this requires additional safeguards around the data and other enforceable measures, including ensuring that any information sharing with third parties is legal and proportionate.
Prior to the establishment of the JDL, VCS organizations were largely dependent on their own attempts to collect re-offending data from sources such as local and national police and probation services. However, such data tended to be variable in quality and piecemeal, making it difficult (if not impossible) to apply for comparative purposes (i.e. across agencies or across jurisdictions). The disparate and variable nature of such data also rendered it expensive to collect, with further risks arising with respect to confidentiality and privacy. The lack of access to high quality information was found to be limiting organizations’ ability to demonstrate impact to funders and public sector commissioners of services, as well as restricting potential for learning and improvements to services (MoJ, 2014a).
There was therefore a perceived need to further explore the potential avenues for improving access to MoJ administrative data. In 2011, in response to research that had highlighted how VCS organizations were struggling to evaluate their effectiveness in reducing re-offending, New Philanthropy Capital (NPC) – a think tank focused on helping organizations to measure their impact – advocated a new service that could respond to such issues, with the MoJ subsequently receiving ministerial approval to embark on a feasibility study. In 2012, the MoJ consulted with a range of potential data users in order to better understand how sensitive data could be shared in a way which ensured its protection while also maximizing its potential. This successful engagement led to a pilot project being initiated in 2013 and an extension of this into 2015. The JDL has been able to provide organizations with re-offending data free of charge since its establishment.
How does the Justice Data Lab work?
Participating organizations supply the JDL with details of the offenders they have worked with. The JDL requires names and dates of birth, and there is an option to provide conviction dates, intervention start dates and intervention end dates, Police National Computer (PNC) identifier or Prison number. Organizations also have to provide information on the nature of the intervention and service, referral processes, where the intervention took place and the timing.
For the analysis presented here, interventions are broadly categorized according to the main form of support offered, although it is important to note that some may be seeking to address the multiple issues often faced by offenders by combining different types of support. Figure 1 shows the range of services examined, with employment support standing out as by far the most common type of intervention (with 81 published requests), followed by accommodation services (13 requests) but with other types of support, such as restorative justice and health (one request each), being only minimally represented.
Published findings from the Justice Data Lab according to primary type and statistical significance.
The JDL team matches these individuals to the re-offending datasets held within the MoJ and uses statistical modelling techniques to generate a matched comparison group of offenders with very similar characteristics, including demographic details, criminal history and employment and benefit history. As standard, the JDL supplies an aggregate one-year proven re-offending rate and frequency of re-offending for the target group and its matched comparison group as well as the average time to the first offence within a year of release from custody or start of probation for those who went on to re-offend in both groups. The measures for both groups are compared using statistical testing to assess the impact of the organization’s work to reduce re-offending. The results are then returned to the organization before being published on the gov.uk website as Official Statistics, alongside a summary of the findings to date, thus promoting transparency and ensuring that findings produced through this service can be used by others. In these reports, the findings are explained along with limitations of the analysis. A cautious approach is adopted with respect to the statistical significance of the findings, with a clear statement as to what can and cannot be concluded from the results. 2
The JDL is an example of a ‘Data Lab’ that operates as a tabulation unit, building on a high quality data set and with a considerable investment having been made in automating the evaluation effort. It is designed to respond to the needs of smaller organizations that lack the skills or resources needed to collect and process outcome data and are often unable to afford expensive consultancy services. All organizations who wish to look at their impact or outcome data must, nevertheless, have a degree of in-house understanding of data collection and its use. Locating the Data Lab within the statistics profession helps to provide the structure and culture needed to manage risks around the use of sensitive personal data, also ensuring that it avoids interference on the part of Government Ministers (i.e. the Government Statistical Service is an independent body within the UK Government). The JDL is therefore different from other similar initiatives across government that aim to allow access for approved researchers (i.e. academics) to administrative data in a secure setting. 3
What has the Justice Data Lab shown so far?
In its first year of operation, the JDL published 59 reports on the effectiveness of re-offending programmes, rising to 117 published reports by 11 September 2014. The spread of different intervention types and the number of statistically significant increases or decreases for the latest findings is shown in Figure 1.
These findings relate to those organizations who have sought to determine their effectiveness through the JDL and, as such, should not be taken as definitive guidance on which types of programmes are most likely to reduce re-offending. Where an inconclusive result has been observed, this does not mean that the programme does not impact on re-offending. In all cases where an inconclusive result has been observed, organizations were recommended to submit further data over a longer period of time once available, in order to more precisely identify the impact of the service or programme on re-offending (MoJ, 2014a). Figure 1 shows how the statistical significance of the results varies between types of intervention, with significant results exhibited by four out of five education related interventions, six out of 13 accommodation interventions and 22 out of 81 employment interventions. Of the other types (18 cases), only one was significant – a youth intervention.
Figure 2 presents the published findings from each of the sectors that have requested information: private sector (32 reports), VCS organizations (37 reports) public sector (40 reports), and also educational institutions (eight reports). It is notable that, although the JDL was initially designed in response to the needs of VCS organizations, the largest number of published findings pertains to interventions or services provided by the public sector. The VCS interventions exhibit the highest number of statistically significant reductions in the one year proven re-offending measure, although there were also three cases of increased reoffending. The private sector interventions exhibit a number of cases where the rate of re-offending is higher than the control group. Further research is needed to examine whether this reflects the quality of provision or reflects programmes designed to increase employment amongst some of the ‘hardest to help’ offenders, where organizations may have been encouraged by public service commissioners to experiment with new ways of delivering services.
Types of organizations using the JDL.
Tackling methodological challenges
As an innovative development in the application of administrative data to evaluation, the JDL has had to overcome a number of methodological challenges, many of which are encountered in other quasi-experimental designs for evaluating public services. The approach taken has been shaped by the nature of the particular issues relating to the administrative data set being used, the data provided by service delivery organizations, and the legal framework in the UK that regulates data sharing and use. Challenges arise in relation to three main problem areas: the key indicators used in the reports (the dependent variables), the quality of the comparison group and related independent variables, and issues relating to sample size and confidence limits.
A fundamental challenge has been around the derivation of variables for statistical analysis from the available administrative data. The decision to focus on the one year proven re-offending measure reflects the limitations of existing administrative data and the difficulties experienced in accessing information on the drivers of changes in offender behaviour. Commenting on the methodology, key informants emphasized the difference between re-offending (committing a crime) and re-conviction (being found guilty). There are also demands for additional and complementary evaluations to ‘go beyond’ the data provided to assess the impact of any interventions on the behaviour change of individuals who had previously been convicted (McNeill et al., 2012). Many organizations working within the criminal justice system do not see the reduction of re-offending rates as their sole objective, but rather focus on desistance from crime as part of a personal journey. Relapses can and are expected to happen and so there is a need to examine progress against the frequency of re-offending and the severity of the offence. The chief executive of a charity supporting prisoners expressed the need for exploration of ‘more detailed aspects of the pattern of re-offending, such as the volume of pre- to post-programme offending, the severity of re-offences, penalties for re-offences and differences in volume of re-offending between different sub-cohorts. We believe that it is important to analyse these outcomes in detail to thoroughly evaluate the impact of an intervention’. Following this and other such feedback, the JDL is now working with users to extend the range of outcome indicators available, thus allowing further conclusions to be drawn on how an intervention or service has affected re-offending.
The second set of methodological challenges relates to the matching process and use of propensity score matching (PSM). Ravallion (2008) reviews a number of different approaches, including PSM, and their application to anti-poverty programmes. Different approaches suit different conditions, but with adequate data, propensity score matching combined with some single difference modelling can overcome some of the selection biases associated with non-experimental approaches to evaluation.
The use of PSM requires a strict set of assumptions to be made, without which the modelling will not hold; as with all evaluation designs of this kind there is a risk of unobserved ‘independent’ variables having an influence on parts of the comparison group. For example, although drug problems, mental health conditions, quality of housing and home-life, and employment opportunities are all known to influence re-offending, there is little in the way of robust administrative data which can truly reflect each individual’s circumstances. However, the very large comparison groups available through the JDL allow assumptions to be made in order to reduce this problem. The experience of the JDL has been accompanied by a deepening of understanding of how PSM can be applied for different sentence and intervention types. Particular care is taken with respect to quality assuring each report, with the JDL team of statisticians also routinely undertaking sensitivity analyses to assess how different parameters or variables impact on the results.
The documentation accompanying reports therefore clearly sets out any limitations and where it is not possible to provide evaluative results because the statistical matching approach is unlikely to be appropriate. This includes people who are part of gangs; have committed terrorist offences; are aged 14 or under; are vulnerable young people/adults (e.g. with mental health or learning difficulties); are substance misusers; and people who have committed sexual or domestic violence offences. This can exclude many criminal justice interventions targeted at vulnerable people and substance misusers, given that the currently available administrative data are unlikely to capture individuals’ salient characteristics.
The third area of challenge relates to sample sizes and confidence limits of analyses. Evidence of outcomes is highly dependent on sample sizes, with larger samples allowing more precise estimates of the impact of interventions. The JDL has identified 26 programmes which demonstrate a significant decrease in re-offending and seven with a significant increase. Figure 3 shows the relationship between the size of the matched treatment group and the range (confidence interval) presented around the change in re-offending for four types of provider – public, private, VCS and educational institutions. For a large proportion of requests involving smaller sample sizes, results have been inconclusive. On the whole, the larger the cohort, the greater confidence that the analysis is representative of offenders, or those with the specific characteristics the organization is seeking to address. Of the 16 reports with samples of over 500 individuals, half had significant results. For the remaining 83 percent of cases with samples of less than 500 people, just over a quarter of the reports are statistically significant. This proportion does not appear to vary with size. Many VCS users of the JDL are smaller organizations that do not aim to work with large numbers of individuals over any given period of time and are therefore often unable to provide large samples. For some organizations that have not recorded the personal identifiers of their users, collecting such data at a later date from prisons and referral agencies has been a particular challenge. Hence it is often difficult for such organizations to establish with confidence that their interventions have led to genuine changes in re-offending behaviour, irrespective of the quantitative technique used.
Sample size and confidence intervals of JDL reports in prepared for organizations in the public sector, private sector, voluntary/community sector and educational institutions.
Response of delivery organizations and their use of the Justice Data Lab
The fact that service providers have made use of the JDL, with 117 published analyses (as of September 2014) demonstrates that there is a demand for the service, with initial feedback supporting that the information was found to be useful for demonstrating the impact of their services both internally and externally (MoJ, 2014a). Despite the JDL service having been scoped and developed in collaboration with VCS organizations, it is notable that only 37 of the 117 findings published were for that sector, with much of the interest emanating from parts of the public sector and other private sector providers wanting to demonstrate the impacts of their interventions.
There is some evidence of these experiences of the JDL contributing to the development of an ‘evaluation culture’ amongst users, particularly smaller VCS organizations, many of whom had little prior experience of evaluation. The interview evidence also supports that users have gained a better understanding of statistics related to impact and evaluation following their engagement with the JDL and the explanations provided in the published reports. There is also evidence of organizations using the JDL to complement other forms of evaluation. 4 However, there is a perceived risk that the use of such a service may lead to an overemphasis on one type of (quantitative) evidence. One user organization, for instance, referred to the JDL as supporting a ‘mixed economy of data with some soft and some hard . . . with the marketization of services there is greater demand to quantify outcomes and there is a more valuing of hard at the expense of soft’ (CEO of charity supporting men in prison).
The JDL has paid particular attention to ensuring that the results published are interpreted correctly and are not open to misinterpretation, including through explicit statements that reports should not be compared against each other. In part, this is in response to concerns from some organizations that the reports could be used without considering other forms of evaluation, and without a clear understanding of how to interpret the results. For instance, the CEO of one VCS organization was concerned about the danger of overly simplistic conclusions being drawn by some readers of the reports due to a lack of understanding of the limitations of the data and without being able to refer to other evaluation evidence. There are particular challenges around how to report the large proportion of insignificant results without damaging the reputation of the service involved.
The JDL therefore explicitly states that reports should not be compared against each other. The importance of this point was reiterated by some user interviewees, one of whom was concerned that the JDL’s cautionary advice on how to interpret results was not being followed and warning of the ‘danger of league tabling and comparing to others on a measure you might not have been set up to cover’. Interviews with non-users similarly identified the risk of negative or insignificant published results becoming a major barrier to the wider uptake of the JDL. There is a concern that the methodological limitations may provide results that could lead to a loss of future funding. This was considered a particular problem in light of the recent major policy reforms and reorganization of service delivery affecting the English criminal justice system (MoJ, 2014b). Although policy makers interviewed expected the JDL to have an increasing impact on policy making through the provision of new evidence on ‘what works’, they also pointed out that recent and ongoing significant changes to offender rehabilitation policy in the UK are likely to complicate attempts to isolate the contribution of the JDL initiative.
Users of the JDL have been particularly keen to address the confidentiality and data protection issues raised by the use of sensitive personal data. The JDL provides a legal gateway to access this data in a safe and secure system; as such its establishment as a specific data lab residing behind the firewall of a government department appears to have been key. While some providers of services are capturing consent during interventions, some organizations argue that requesting consent on an entry form may have led to a decrease in participants (although it has been hard to identify if other changes in their intervention may also have affected participation). Gaining consent retrospectively is also very difficult. The MoJ was considered to have adopted a pragmatic interpretation which allowed organizations to share data for this specific purpose and which is in accord with the Offender Management Act and the Data Protection Act, whereby data sharing between providers is legal when there is a lawful and proportionate reason to share data for a particular purpose. The Privacy Impact Assessment produced by the MoJ sets out the legislation which enables the sharing of data where consent has not been collected, and makes assurances that the purpose of the JDL is ‘likely’ to satisfy conditions for data sharing. However the onus is on charities ‘as Data Controllers, to satisfy themselves that the sharing of the data with the MoJ complies with their legal obligations’ (MoJ, 2013: 9). The report further states that ‘Organisations should obtain their own legal advice about these issues if it is considered necessary’ (MoJ, 2013: 9). However, some organizations have reported that this guidance is too vague, requiring them to seek further legal clarification of their position to enable data sharing, which has led to them delaying their application to the JDL.
Discussion
Within the context of England and Wales, the Justice Data Lab has been instrumental in helping smaller organizations access administrative data and evidence of the outcomes from their work with offenders. Its development has been made possible through a combination of political will and the ambition of the Statistics Unit in the MoJ to be transparent and work with its customers. After only 15 months of operation, the JDL was awarded the Royal Statistical Society Award for Excellence in Official Statistics. The Royal Statistical Society judging panel commented that the award was being given ‘For the use of statistical techniques to assess success (or failure) in a critical area, and for the exceptionally close way MoJ statisticians have worked with their users, mainly non-statisticians, to provide the most useful possible service, and for the way feedback was both encouraged and acted upon’. In November 2014 the Justice Data Lab was also awarded the Government Finance Insight Award. The judges were impressed by the use of data in an innovative way for key decision makers, presenting complex data in a way that is understood by all and the way data systems have been used to support the work of the wider departmental family.
There is clearly a demand for the service amongst delivery organizations, although also a degree of caution and hesitancy on the part of some. It is of particular interest to note that a programme developed primarily with VCS organizations in mind is also being used by a significant proportion of private and public sector bodies. For all these organizations, the JDL is free at the point of use, and therefore provides a low cost source of evidence to complement other sources. This presents significant savings to organizations that would otherwise need to contract out such work to other evaluators at considerable cost. The cost per report for the MoJ is estimated to be a fraction of that of other survey work or independent data collection. Value for money in service delivery is also expected to be demonstrated in the future as organizations adjust their services on the back of learning derived from the analysis conducted by JDL.
The size of the samples of offenders provided for analysis is crucial, with larger samples (over 500 people) being more likely to provide significant results. However, smaller organizations and programmes often lack such scale, or may have to wait many years to build up the number of users needed to develop a large enough sample. This may deter some small and innovative programmes from using the JDL. Further research on the JDL over time will explore these issues in more depth.
The JDL’s requirement that reports must be placed in the public domain encourages a culture of transparency and appreciation of impact which, to date, has arguably been lacking in public service provision. The service provides high quality data on reoffending rates, although organizations are also sensitive to the limitations of the results. The methodological challenges mean that care is needed in the interpretation of the results. Both the MoJ and user organizations have been engaged in efforts to promote greater awareness around the use of the results and to reduce the risk of misunderstanding leading to a backlash against evaluations more generally. The competitive environment within which service providers are operating presents a particular challenge, given the risk that negative evidence or insignificant results could result in the loss of contracts from the public sector and loss of other funding. Nevertheless, while there are risks associated with transparency, organizations using the service appear to be keen to demonstrate their good practice and to explore ways of improving the quality of the samples provided, including safeguards to ensure that organizations requesting data are discouraged from supplying selective samples with a view to biasing the results in in a positive direction. Service delivery organizations are also working with the MoJ to support the development of a greater range of indicators that can demonstrate desistence and changes in behaviour.
The experience of the JDL is now being shared widely across the UK government and has also attracted international interest, with non-profit organizations in Australia and the USA currently developing plans with their respective governments. In the UK, the interest in increasing evaluations and providing evidence of effectiveness of public spending has resulted in feasibility studies being conducted related to education, employment and health data. This is a significant investment for Government, particularly given the need to commit resources into ensuring that personal data is successfully developed into safe products which can be shared securely and robustly. There are significant opportunities for Government departments to work together and develop a common strategy, including through sharing administrative data to improve the underlying information about individuals, and the potential to expand the range of outcome measures. For example, drug treatment charities have been particularly concerned with measuring the re-offending and employment outcomes of their services, rather than focusing solely on measures of substance misuse outcomes.
Data sharing across government and public sector departments has increasingly moved up the political agenda with the development of programmes such as Troubled Families which aim to support families facing a myriad of difficulties and needing to engage with multiple services. Better data sharing between departments has potential to improve resource efficiency by reducing duplication and ensure that services are better designed and targeted. Increased data sharing is still in its infancy and faces some opposition from civil liberties campaigners concerned to protect the right of individuals to anonymity. However models similar to the JDL which provide aggregated analysis can, in time, make a significant contribution to the evidence base on effective interventions and increase opportunities to further develop services that are able to provide multiple social justice outcomes.
Conclusion
This article has explored the recent development and experience of the Justice Data Lab, an innovative approach that allows access to sensitive personal data for evaluation purposes in ways that have not been attempted before. The volume of output from such a small service is particularly significant in the context of its relatively short period of operation, demonstrating that Government Departments, working in partnership with other actors (in this case VCS organizations, social enterprises and private businesses) can be well placed to contribute to the evidence base and show real engagement with users. As a pilot project, the JDL has wider implications for policy making and the provision of data on the outcomes of public service interventions. As lessons are identified, new datasets developed and methodologies adapted, other Data Labs are expected to emerge and also further innovations in design with relevance to a range of service areas. There has been a growing interest from other government departments in the UK and internationally in response to the important role that the JDL is playing in demonstrating how the wider use of administrative data can help a range of organizations involved in service delivery.
The JDL has to be seen in its wider context of both a ‘Big Data’ agenda and the growing interest in forms of public service evaluation that can support learning and innovation. As more opportunities for data analysis become available, there is a need for innovative platforms and a cadre of data curators and analysts (DeSouza and Smith, 2014). The case of the JDL demonstrates the value of relatively inexpensive quasi-experimental designs and also potential for such approaches in random control trials in the future. New opportunities for linking disparate administrative data sets need to be accompanied by both political will and the further development of safe ways of ensuring anonymity and confidentiality.
This article presents findings from the initial pilot phases of the JDL, with further research needed to assess its longer-term contributions and wider influence. The future impact of the initiative itself will depend to a large extent on how delivery organizations continue to make use of the results to complement other evaluations and improve their services. These evaluations have the potential to support improved service delivery by organizations and to shape the decisions of both policy makers and public service commissioners. In the first instance, we may be able to see the legacy of the JDL through scaling up of services shown to be effective as well as adaptations to services shown to be less effective but with potential for improvement. Secondly, the impact will be seen from changes to the decision making of policy makers, both at a local level (as commissioners of services decide which providers to contract with through better provision of evidence) and at a national level.
Footnotes
Acknowledgements
The authors would like to thank the organizations who shared their views on the JDL at a number of events, those who have commented on this article and the reviewers who have helped develop the article. The work of Georgina Eaton, Sarah French and Ben Kogan. The funding from the Oak Foundation has supported the work of authors. All views expressed are those of the authors who write in a personal capacity alone.