Documentary Analysis Within a Realist Evaluation: Recommendations From a Study of Sexual Assault Referral Centres

Abstract

Realist approaches are increasingly used in studies of complex health interventions/evaluations to understand how programmes work, for whom and under what circumstances. Mixed-method data sources can be used to generate, refine and test realist programme theories, which explore causal links about the contexts that affect the mechanisms of an intervention and lead to the production of different outcomes. The realist approach provides a framework for a detailed understanding of how a programme functions, aiding with the implementation, refinement or adaptation of interventions. Documentary analysis provides an overview of the theoretical and practical functioning of a service and the ways it is structured to provide interventions. Data are often collected early in the evaluation and can include service specifications, organisational policies and procedures and routine audit data. This paper describes a two-stage process of documentary analysis, where data extraction forms and journey maps are used to explore how Sexual Assault Referral Centres (SARCs) in England respond to the mental health and substance use needs of users. Using documentary analysis as part of a sequential data collection process can be valuable in informing subsequent data sources (e.g qualitative interview schedules can be used to further test and refine theories from a documentary analysis). Considerations for researchers in applying documentary analysis include the value of keeping initial searches broad, to capture documents from a range of sources; the need for clarity about the prioritisation of data sources in the selection process; the benefit in establishing a standardised extraction form that incorporates the wider context within which the intervention functions; taking steps to ensure face validity and transferability during interpretation of data sources; the benefits of transforming information from the data extraction form into a visual journey map.

Introduction

Realist Evaluation

Realist evaluation is a theory-driven approach commonly used to assess complex systems/interventions and asks ‘what works, for whom, under what circumstances, and how’ (Dalkin et al., 2015; Wong et al., 2016). Developed in 1997 by Pawson and Tilley, this approach is underpinned by a realist philosophy which holds that to evaluate a programme, it is necessary to understand how it brings about change (Pawson, 2013). One of the tenets of the realist approach is that interventions are active and reach their outcomes by making changes to the views and behaviours of the stakeholders (Dalkin, 2015; Pawson, 2006). The triggering of changes (or mechanisms) is mediated by who the individuals are and the context in which the intervention operates (Wong et al., 2016). Indeed, interventions function within complex and dynamic social systems – systems which are consistently subject to negotiation, resistance and adaptation. The realist emphasis on understanding contexts and mechanisms is, therefore, valuable in improving complex health services and when attempting to implement, upscale or adapt existing interventions (Bertotti et al., 2018; Nurjono et al., 2018). Similarly, Medical Research Council guidance outlines that an understanding of how an intervention functions is essential for effective implementation (Craig et al., 2008).

The realist approach has been increasingly used in evaluations of complex health systems/interventions (Shearn et al., 2017). The complexity of both the intervention(s) and the organisational contexts/structures in which they are implemented means that outcomes inevitably vary. Traditional methods of measuring intervention effectiveness, therefore, generate conflicting results with little information about why an outcome has occurred. (Jagosh, 2019).

Realist evaluations create a conceptual framework in which to explore the broader functioning of how the intervention works (Pawson & Tilley, 1997). Pawson (2006) states that there are contextual layers that make up the implementation of complex interventions. These layers can function at a macro-level (i.e the external context which shapes the function of a service), meso-level (i.e the functioning and structure of individual services/teams) or micro-level (i.e individual stakeholder behaviours and beliefs within a service) (Lacouture et al., 2015). Mixed-methods data collection has been identified as a useful way to capture the in-depth data needed to effectively evaluate these layers within complex service interventions (Noyes et al., 2019), providing a more comprehensive understanding of large multi-faceted systems (Bazeley, 2018).

In order to explore what works, for whom and under what circumstances, a realist evaluation identifies an intervention’s underlying generative mechanisms (M), surrounding contextual factors that mediate the mechanisms (C) and the resulting pattern of outcomes (O) (Pawson & Tilley, 1997). These context-mechanism-outcome (CMO) configurations provide the framework for conducting analyses within a realist evaluation (Pawson et al., 2004; Pawson 2006). A realist enquiry has a recommended set of phases. First, initial programme theories (IPTs) are developed from existing research, for example, through systematic or realist reviews. Data is then collected via quantitative or qualitative means, such as documentary analysis or interviews, and is used to test and refine the IPTs. These data are analysed in order to identify CMO configurations, which are then synthesised into the IPTs. (Pawson & Tilley, 1997). This is an iterative process and earlier phases can be returned to at any point (Gilmore et al., 2019). The overall findings are then synthesised, a list of final CMOs are produced, and recommendations for service improvement across different contexts can be made (Wong et al., 2016).

Documentary Analysis

There has been an increase in the use of organisational and institutional documents as a data source in qualitative research (Bowen, 2009), and this can be a key part of realist evaluations (Pawson & Tilley, 2004). The process of reviewing and evaluating these documents is known as documentary analysis. Documents can be collected from a variety of sources, including public records (such as annual reports and policy documents), personal documents (such as emails and duty logs) and physical evidence (documents found at the study site such as leaflets and posters) (O’Leary, 2014). Documentary analysis has multiple functions as part of a wider realist study. It is able to help researchers understand the context within which individual stakeholders in a service function (i.e the micro-level) by increasing researchers’ knowledge of the external context which shapes the operation of a service (i.e macro-level), through examination of national policies, service specifications and guidelines, and researchers’ knowledge of the functioning and structure of a service (i.e meso-level), through examination of data sources on how individual organisations and teams operate. Additionally, it can highlight questions that will inform the next stage of research, and finally, the results of the analysis can be compared with other forms of data (Bowen, 2009).

Documentary Analysis in a Mixed-Methods Realist Evaluation

This paper describes how a two-stage process of documentary analysis was incorporated into a mixed-methods realist evaluation, as part of a National Institute for Health Research programme on the effectiveness of Sexual Assault Referral Centres (SARCs) in England with regard to mental health and substance use needs (the MiMOS study NIHR 16/117/03). SARCs are commissioned by NHS England in conjunction with a range of partners such as Local Authorities and Criminal justice systems such as Police services and Police and Crime Commissioners (NHS England, 2016). SARCs are multi-disciplinary centres that provide holistic forensic and health care services to people who have experienced a sexual assault. This includes a forensic medical examination (FME), health care, safeguarding and crisis care.

Given the nature of the service, it is not surprising that there are high levels of mental health and substance use needs identified in individuals attending SARCs. However, how SARCs respond to these needs has been shown to vary across the country (Brooker & Durmaz, 2015; Brooker et al., 2018). The NHS Service Specification for SARCs (NHS England and NHS Improvement 2019) recognises that mental health needs should be identified and addressed, but there is currently limited detail on how this issue should be approached in practice.

The MIMOS study is a mixed-methods programme of research funded by the National Institute for Health Research, which aims to explore how effective SARCs’ responses are, not just in the identification and assessment of mental health and substance use need but in the onward referral for support. The study consists of several interconnected work packages (further details can be found on the study website here: https://mimosstudy.org.uk). Initially, systematic (Stefanidou et al., 2020) and then realist reviews (Stefanidou et al., 2021) were conducted to gain an understanding of the current evidence base (Ariss et al., 2020).

One of these work packages consisted of mixed-methods case studies of six sites. Case study analyses included documentary analysis, focus groups with SARC staff and partner agencies and interviews with service users. The documentary analysis, reported in this paper, was conducted first in this process in order to build a picture of how mental health and substance use needs were identified and responded to by the six case study sites.

This paper describes the documentary analysis component of the MiMOS study and explores how it was incorporated into the wider realist evaluation. The paper attempts to provide transparency and guidance for other researchers around this aspect of the analytical process within a realist evaluation. We describe in detail the methods used in the documentary analysis and draw out key learning points and recommendations based on our findings.

Methodology

The documentary analysis was a two-stage process. Firstly, extraction forms were created, based on national service specifications (Health and Justice, 2018; NHS England, 2016), and these were used to identify data that captured the broader context of the functioning of the SARCs. Secondly, data were synthesised from the extraction form in order to form ‘journey maps’.

Journey mapping is a method that has a tradition in market research, where it is associated with understanding the relationship between the service-user and provider. However, a recent review discovered a lack of common understanding of the term, heterogeneous perspectives and diverse reference literature (Følstad & Kvale, 2018). A wide range of data collection approaches have been documented, including co-creation of visual maps with service-users, observations, questionnaires, facilitated workshops with teams of service providers etc.

Recent uses of journey mapping in health services have aligned the method with process mapping, which has its origins in industrial quality improvement or quality management approaches, such as ‘Lean’ or ‘Six Sigma’ (Trebble et al., 2010), and is therefore focused on directly informing interventions to improve efficiency. However, despite the various approaches and origins, journey mapping can usefully be employed in healthcare research to portray the intervention experience from the perspective of the service user (McCarthy et al., 2016; Trebble et al., 2010).

In this study, journey maps did not directly inform a quality improvement process. Instead, they functioned as theory development tools to inform a wider programme of Realist Evaluation (Pawson, 2013), which aimed to develop, refine and test hypotheses about approaches to substance use and mental health issues in SARCs. The journey maps provided a chronological depiction of the interventions, allowing an accessible way to identify points in the service-user journey that could influence mental health or substance use outcomes.

In a departure from some mixed-methods approaches to journey mapping, data for this aspect of the research programme were collected solely from organisational documents. As part of a wider, theory-driven research programme, the documentary analysis provided an understanding of formal, organisational structures and processes. Combined with a journey mapping approach, this provided a framework against which to compare and contrast findings from other data sources and perspectives. The research team comprises members from multiple disciplines with a range of experiences. This has similarities to approaches that use facilitated workshops with multi-disciplinary teams (e.g McCarthy et al., 2016). However, for this project the research team members were responsible for data analysis and had regular meetings to collaborate on the development of the maps; people with lived-experience were consulted as part of this process.

Sexual Assault Referral Centres provide complex interventions in complex contextual environments; therefore, a consideration of complexity is paramount in order to produce findings that can be usefully applied. Whilst Pawson’s ‘VICTORE’ complexity checklist (Pawson, 2013 p. 43–44) is constructed from the perspective of policy implementation, it has transferrable categories, which can help to construct important elements of theoretical complexity, from the perspective of the user of the intervention and which can be gleaned from organisational documentation. For instance, these sources contain valuable information regarding the ‘choice architecture’, ‘implementation chains’, ‘context’, ‘temporal mapping’, ‘monitoring systems’ and ‘long-term adaptations’, which can be incorporated into candidate theories.

Study Stages

The documentary analysis incorporated four key processes: (1) documents were collected from the six SARC case study sites, (2) a data extraction form was developed to extract evidence from the documents, (3) the extracted data were converted into a chronological ‘journey map’ and (4) the journey maps were used to develop and refine initial program theories and inform the subsequent project activities within the case study analysis (i.e staff and service user interviews). This study analysed data using a critical realist lens. Critical realism accepts the concept of objective realities but argues that positivist reasoning alone cannot be used to understand the world. Instead, subjective experiences are seen as equally valid. This approach combines explanation and interpretation, recognising that social contexts and social conditioning also influence how we describe and experience the world (Archer et al., 2016)

Recruitment

A national audit of SARC services was conducted, and using data from the responses, a cluster analysis was undertaken based on their approaches to mental health and substance use; three clear clusters of types emerged. Six SARC case study sites were then selected that represented examples from the three cluster types of SARCs within England, based on the diversity of their service models. These were used to explore in-depth the contexts, mechanisms and outcomes that operate within the SARCs, and to test and refine the IPTs from the systematic and realist reviews.

Data Collection

Data were collected from a variety of sources. Researchers approached the six SARC study sites and requested any relevant documentation including policy documents, service specifications and proformas (e.g assessment/intake forms, feedback forms, staff training documents and any published research). Further data were gathered from sources such as commissioner reports and the SARCs’ websites. An internet search using Google was also conducted using the names of the SARC sites in order to include any further relevant reports or studies which had not already been identified.

Development of Data Extraction Form

A data extraction form was developed to capture the macro- and meso-level contexts within SARCs, as well as elements relating to what works for whom and in what context in relation to mental health and substance use support within SARCs.

National SARC and sexual assault policies/service specifications (Health and Justice, 2018; NHS England, 2016) were used to inform both the macro-and meso-level elements of the extraction form. A web-search was conducted to find this documentation, and two documents met the criteria: Strategic Direction for Sexual Assault and Abuse Services: Lifelong care for victims and survivors 2018–2023 (Health and Justice, 2018) and Service Specification No. 30 Sexual Assault Referral Centres (NHS England, 2016). These documents included many recommendations and specifications for SARCs, and we extracted all elements that focused on the identification and treatment of mental health and substance misuse. We also extracted all core principles from the Strategic Direction specification (Health and Justice, 2018), as they represented nationally applicable macro-level elements, and used these to create six overarching categories within the data extraction form, for example, ‘strengthening the approach to prevention’, ‘promoting safeguarding and the safety, protection and welfare of victims and survivors’ and ‘introducing consistent quality standards’ (see Supplementary File S1 for full details). We then generated a list of key indicators relating to mental health or substance use pathways within individual SARCs, and these represented meso-level elements, which indicated how the core principles were implemented at each site. These indicators included examples such as ‘recognition that reducing the risk of future re-victimisation is central in aiding service users recovery and ongoing safety’ and ‘acknowledgement that safeguarding vulnerable individuals is a priority for the service’. The meso-level elements were ordered under the relevant six core macro-level national principles to facilitate data ‘selection’ rather than ‘collection’, thereby, filtering out data which does not fit the conceptual framework of the research question (Bowen, 2009).

For example, the meso-level key indicator ‘evidence of consistency in care regardless of a person’s demographics and complexity of needs’ was assigned under the macro-level core principle ‘introducing consistent quality standards’. In order to be included in the extraction form, the key indicators did not need explicitly to mention mental health and substance use needs as long as they were related to these issues. For example, the key indicator ‘increase awareness of the services provided by SARCs, particularly through the lesbian, gay, bisexual, transgender, queer/questioning and intersex (LGBTQI) communities, Black and minority ethnic communities and vulnerable women’s centres’ was added to the extraction form because minority groups and people from LGBTQI+ communities are vulnerable to experiencing mental health disorders (Grey et al., 2013) and people with complex mental health/substance use needs face barriers to accessing health services (Ross et al., 2015). Therefore, awareness raising would be important amongst these groups.

The data extraction form, including the core principles and key indicators, can be seen in Supplementary Information file.

Data Extraction

The first stage of analysis involved a process of data immersion, whereby the researcher (HP) conducted an initial superficial examination of the documents for each site, followed by data familiarisation through reading and re-reading of the documents (Bowen, 2009). The results of the systematic and realist review allowed a greater understanding and sensitivity to the vocabulary and the context in which the documents were produced (Altheide et al., 2008) and allowed the researcher to home in on evidence that was related to mental health and substance use. An initial extraction form was developed by the researcher (HP) in Microsoft Excel, and this was then reviewed and refined by other researchers on the study team (JD, KT). Once the extraction form was finalised, all documents were reviewed for evidence of the macro- and meso-level elements. Details were then extracted into the form, using mostly exact phrasing. A separate but identical extraction form was used for each site.

Two processes were employed to ensure the intersubjective verifiability of decisions made during data extraction and ensure that key concepts could be readily communicated and understood. Regular team meetings were continued throughout this phase in order to discuss any refinements of the extraction form that were needed, as data extraction proceeded, to ensure all appropriate data were extracted. To further ensure relevant data were extracted and identify areas of ambiguity regarding assigning data to indicators within the extraction template, a second researcher (JD) also reviewed a piloted data extraction form for the first case study site, made her reflections and identified any areas for refinement. The main validation process was a meeting between researchers (JD) and (HP) where each indicator and its extracted data were discussed, and any areas of disagreement or ambiguity explored. Some double coding was completed on the pilot data extraction form; however, this was not formally recorded. A high level of agreement was found between researchers. and following the meeting, any agreed refinements were made and a final data extraction template was created. This was used by the lead rater (HP) to extract data from the six sites.

The two researchers who conducted data extraction are both female with experience of working with survivors of sexual assault. The first rater (HP) holds an MSc in forensic psychology and has specialised in research on violence against women and gendered service provision. The second rater (JD) is a clinical psychologist who specialises in perinatal and child mental health and researches in the field of parental mental health and violence and abuse. This prior knowledge of domestic and sexual violence, mental health issues and clinical service provision allowed for a greater sensitivity to the organisational processes which might impact service users with mental health and substance use issues, such as different assessment techniques or trauma-informed approaches. Experience in clinical service provision also meant there was a familiarity with the types of documents used within these services, such as the Trauma Screening Questionnaire or a standardised risk assessment.

The need for researchers to have an awareness of any preconceptions they hold has been noted to be particularly important in emotive, sensitive subject matter (Cowles, 1988) such as sexual offences. It is for this reason that there was great utility in ensuring researchers used a critical, self-reflective approach to the extraction. This meant researchers examining their assumptions around the subject matter during the analytic process, considering topics such as rape myths (e.g that rape always involves physical force) or the social stigmas surrounding certain groups of survivors, such as sex workers or males. This approach was aided by reflecting on the process in regular meetings with the wider research team. These meetings allowed for an open discussion of these potential preconceptions and for the team to reflect on how these topics might be useful to consider when analysing the data – for instance, reflecting on data where sexual assault is referenced as an act of physical violence.

Refinement of Data Extraction Form

Following the data extraction verification meeting, it became evident that the form did not contain adequate fields that were specific to the SARCs’ mental health and substance misuse (MH/SU) pathways. This was because the policies/service specifications used to inform the extraction form (i.e Health and Justice, 2018; NHS England, 2016) contained few recommendations or indicators, which related to mental health or substance use, highlighting a lack of guidance around these issues at the macro-level. To address this, two new core principles and associated indicators were developed by examining and drawing on the results of our systematic review (Stefanidou et al., 2020) around effective mental health and substance use pathways within SARCs. The additional key principles were: (1) ‘Identification of mental health and substance use needs’ and (2) ‘How the service addresses mental health and substance use needs’.

Journey Maps

After all data were extracted from the six SARC case study sites, it was used to create a bespoke ‘journey map’ (see Supplementary File S2) for each SARC, a process that is also described in the work of McCarthy et al. (2016). The purpose of this stage of documentary analysis was to provide a condensed and chronological representation of how SARC pathways identify and respond to MH/SU needs, as well as how they might be experienced by service users. This streamlined representation of the data aimed to provide a more accessible picture of the SARCs functioning, for use by the study researchers to inform other aspects of the MiMOS programme, and to allow for gaps in knowledge to be easily identified. The chronological depiction of the data facilitated an understanding of the SARCs’ interventions, and where potential blockages may occur. This was achieved by selecting relevant information from the data extraction forms and creating a ‘map’ of the service user pathway for each individual SARC.

The pathway was separated into the following stages: (1) before contact with SARC, (2) initial contact with SARC and (3) further contact and onward referral. The data extraction form and original documents were re-reviewed to gain an insight into the chronology of each SARC’s pathway, and then relevant data were input into the appropriate journey map stage. Sub-headings were added to the journey map to reflect the different elements of each SARC’s pathway. Particular attention was paid to detailing places where there was an explicit mention of the mental health and substance use assessment and treatment pathway. These points were colour coded in order to give a visual sense of the pathway.

Assessing the Value of the Documents

Whilst synthesising and condensing the data, it was necessary as part of ‘data selection’ to assess the value each document had to the journey map and overall research question. As the main function of an SARC is to provide forensic and physical health services, many documents related only to these aspects. Because of this, it was essential to interpret the purpose of the document and filter out the ones that were not relevant to the mental health and substance misuse pathway. This process was aided by a clear research question, the effectiveness of SARCs with regard to mental health and substance use.

Another aspect of data selection was considering not just the specific focus of the document but also how it functions in its wider context. Atkinson & Coffey, 2011 state that organisational documents are designed to appear like a ‘true’ representation of the service, but in fact, they are not transparent and construct their own representations or ‘documentary realities’. It is suggested that to address this researchers need to maintain a critical approach in their analysis and consider why a document was made and what its role is in the broader organisational context. The documents in the current study were examined regarding their ‘documentary reality’ and how this would influence their input into the journey map and resulting program theories. For example, an SARC’s website was useful when considering someone’s first interaction with the service or examining how the SARCs themselves describe their pathways. When exploring how this pathway functioned on a day-to-day level, however, higher priority was placed on data from the commissioner’s reports – as their purpose was to understand and assess the ‘true’ picture of the SARC’s functioning. Having a range of sources, produced for different purposes, was a useful way to understand the strength and nature of the evidence provided by a certain document. As stated by Atkinson and Coffey 1997, ‘We have to approach them for what they are and what they are used to accomplish’ (p. 47). The conjunction of data from various sources also provided greater confidence in the ‘trustworthiness (credibility) of the findings’ (Bowen, 2009, p. 30).

Program Theory Development

Following completion, the maps were reviewed by the research team and annotated with relevant questions or thoughts, highlighting the gaps in knowledge or uncertainties around the SARCs’ mental health and substance use pathways. This included reviewing IPTs developed from our systematic and realist reviews and exploring how they might be refined by the data from the documentary analysis, as well as how they might be further tested in the next stages of the evaluation through staff and service user interviews. With respect to the next stages of the programme, data from the journey maps were transformed into questions for interviews with staff and service users in three distinct ways. First, clarification questions were developed, which would help to fill in gaps in the journey map. These ensured a thorough understanding of service users’ journey through the SARC. For example, a question for staff might be ‘what is the procedure if mental health needs are identified during the phone referral?’

Secondly, IPTs from earlier stages of data collection (i.e the systematic and realist reviews) were cross-referenced with the journey maps to identify places where specific theories could be tested. For example, the SARC process can take several hours, meaning a person prescribed opiate substitution therapy may miss a dose or more and start to experience significant discomfort (Clinical, 2017; Independent Expert Working Group, 2017). When considering how examinations are scheduled for intoxicated or withdrawing service users, a relevant theory from the realist review was identified: ‘If SARCs don't stock medication for substance use, then service users with withdrawal symptoms can't be managed within the SARC and may have to go to A&E/rearrange Forensic Medical Examination’ (Stefanidou et al., 2020). In order to test this theory, a number of questions were generated to be asked during staff interviews, for example, ‘do you stock medication for substance use in the SARC? What procedures do you have in place to deal with withdrawal? What are the implications of this for the Forensic Medical Examination?’

Thirdly, new programme theories were generated from the journey maps, and questions were developed to further test and refine these theories. For instance, there was variation across SARCs in the language used to describe sexual assault on their websites – such as use of the term ‘sexual violence’ as opposed to ‘sexual assault’. It was suggested that this choice of wording may act as a barrier for survivors who – despite their experience being very violating and traumatising – consider their assault as non-violent due to a lack of obvious physical injuries. To test this, a question was developed to be asked during service user interviews: ‘how did the materials you read prior to accessing the SARC affect how you felt about attending?’

Lived Experience Advisory Group

The research team worked in conjunction with a lived experience advisory group (LEAG), who were consulted through all stages of the study, from planning to the final synthesis of data. During documentary analysis, a condensed version of one of the journey maps was sent to the LEAG. The journey map was discussed in a meeting with the LEAG members, and new questions and potential theories were added. This was an important element in the iterative process, giving a fuller picture of the pathway experience and potential ways in which the context of the SARC may affect the micro-level behavioural mechanisms of the service user.

Learning Points

Data Collection

There were variations between SARCs regarding the amount of documentation available for the documentary analysis. Some of the sites were able to provide many documents from a variety of sources. This appeared to be for a number of reasons: that a service pathway was particularly complex and required more documentation; that a site was larger or had been established for longer, or simply that some services used a higher number of standardised pathways and proformas compared to others; the latter services utilising emails and online portals predominantly. The type of documentation received from the SARCs was also varied, including site-specific proformas, standardised assessments, documents from training days, referral forms, feedback forms, published research and materials given to service users. An internet search using the name of the SARC site was able to add documents such as commissioner reports and research, which had not been already provided by the SARC.

Key Recommendations:

Keep initial searches broad and aim to use documents from a range of sources.

Data Extraction

Following the process of piloting and refinement, the extraction form proved a useful way of synthesising what was for some SARCs a large amount of detailed documentation. The standardised extraction form provided valuable insights into the processes operating at the macro- and meso-levels within SARCs, whilst also highlighting any gaps in knowledge requiring further investigation. One SARC’s data extraction form, for instance, had very little information extracted under the core macro-level principle ‘Ensuring an appropriately trained workforce’, indicating that further information needed to be collected around staff training during the subsequent qualitative research interviews with SARC staff at this site. The forms also acted as a useful point of comparison between the SARCs’ individual policies, as it was possible to choose an indicator of interest and then use the extraction forms to view the variation between the sites.

It was apparent during the extraction process that a level of interpretation was required by researchers due to a lack of specificity in the documents and the subjective nature of some indicators. For example, evidencing the indicator ‘recognition that reducing the risk of future re-victimisation is central in aiding service users’ recovery and ongoing safety’ involved an interpretation of what procedures might reduce future risk of re-victimisation: ranging from organising a multi-agency meeting for repeat-attendance service users, to simply ensuring that service users had somewhere safe to go to after the appointment. It was also clear that prior knowledge of the topic greatly increased sensitivity to the data and allowed an increased precision when completing the extraction. For instance, extracting data for the indicator ‘evidence of a trauma-informed approach to care, linking trauma and mental health by recognising its effects and human response’ required background knowledge on trauma-informed care and how the approach could be identified within service policies.

Bowen (2009) states that when selecting and analysing data a researcher has a responsibility to create an objective and fair representation of the documents, whilst using sensitivity to identify and respond to the more implicit underlying meanings. Researchers who led the analysis in this study were supported in applying sensitivity and underlying meanings as a result of their involvement in the systematic and rapid realist review projects from the MiMOS programme. The knowledge that the researchers had gleaned from the results of these reviews, in relation to both the theoretical and practical nature of the functioning of SARCs, facilitated this process. In addition, any potential bias arising from the interpretative nature of the extraction was reduced by using two researchers to complete the extraction (HP and JD) and having regular review meetings with the wider study team to discuss the process and results.

Key Recommendations:

Develop a structure for the extraction form that is based on documents that help understand the wider context within which the intervention functions, for instance, service specifications or national guidance. Identify gaps in the data extraction form as topics for further investigation

Be aware of the level of interpretation needed in data extraction and find ways to reduce bias, for example, having regular research meetings and using double extraction

It is valuable for the researcher to have prior knowledge of the topic and identify and address knowledge gaps as they arise.

Developing the Journey Maps

The journey maps focused on the identification and response to mental health and substance use needs within the SARCs. To ensure this focus was maintained when collating and condensing the information from the data extraction forms to include in the journey maps, a priority was placed on data directly related to the mental health and substance use pathway, such as the standardised assessments. Where possible, the use of exact wording gives the reader of the journey map an accurate idea of what the pathway may have felt like for a service user. When selecting data from the extraction forms, researchers ensured that evidence was included from a range of the original data sources – for example, ensuring that evidence from commissioner reports was included as well as evidence from proformas and internal documentation at the SARC. This aimed to give an overview of a variety of causal mechanisms that could be operating in the SARCs beyond the organisational processes, such as low staffing levels, which may cause stress within the team, and use of service user feedback, which could influence ongoing service developments.

Variability in the quantity of data available for specific themes presented some difficulties. When large amounts of data had been extracted, the more challenging was the process of synthesising and condensing the information into the journey map. In these cases, the emphasis lays on the process of data selection and how to maintain clarity when presenting a complex pathway. Where there was a lack of data, the challenge lays in creating an accurate and chronological picture of the service user pathway.

The result was six journey maps of varying complexity. Those with richer data provided a detailed and holistic view of the contexts within which the interventions were operating. The maps were helpful to simplify some very complex pathways, meaning a member of the research team could gain a good understanding without having to refer to original documentation. The high level of detail also allowed for a narrower focus when considering which areas may require further investigation in the subsequent qualitative interviews with staff and service users. The journey maps with less detail were not able to provide such focus; however, they proved important in highlighting when basic information was needed to gain an understanding of the pathway.

Key Recommendations:

Conduct a two-stage documentary analysis: Begin with a detailed standardised extraction form and using this create a journey map, which facilitates an accessible visualisation of the data, highlighting key data relevant to the research question

Ensure clarity about which data should be prioritised in the selection process

Include data from a range of sources

• Journey maps are useful even in cases when data are lacking; to highlight areas where further investigation is required.

Identifying Mechanisms

As well as providing contextual knowledge of the service, the documentary analysis illuminated some of the underlying mechanisms that lead to different outcome patterns in different contexts. Identifying these generative mechanisms is a key part of a realist evaluation. Dalkin et al. (2015) highlights that to understand a system you cannot just consider what the outcome is of an intervention, it is also essential to consider what has led to it and why. The journey maps provide an early step in identifying some of these generative mechanisms, and the annotation of the maps allows for a list of potential mechanisms to be developed and fed back into the IPTs. For example, when the journey map detailed the language used on an SARC website, a potential mechanism was identified whereby a service user may feel that they should not attend because their experience does not fit with the service’s idea of an assault, as described in the definitions used on the SARC’s website. The journey maps also highlighted numerous mechanisms and resources that were not site-specific and might be operating across the SARCS. For example, the existence of clear, formal referral pathways to other supports services.

The journey maps additionally allowed for an examination of the effectiveness of the services’ implementation chain (the sequence of events that lead to successful outcomes). Pawson (2006) states that for an intervention to succeed it depends on the cumulative success of this sequence and that the integrity of the implementation chain (for instance, a patient pathway) should be explored. He recommends that studies establish which elements of the chain are required for a particular outcome to occur. This includes looking at where the parts of the chain are blocked and will prevent the desired outcome. By creating the chronological journey maps from the extracted data, it was possible to gain a clearer representation of the chain of implementation. This meant that when annotating the maps researchers were able to identify points of ‘blockage’ in the chain and could explore these further in the qualitative interviews. For example, the data indicated that appointments are rearranged for service users who are intoxicated/withdrawing from substances. This point in the chain may be a ‘blockage’ for service users with these needs (i.e they may not come back for an appointment – the desired outcome), and therefore was identified as an area that required further exploration.

Key Recommendations:

Annotating journey maps is a useful way to uncover potential generative mechanisms

Journey maps can be analysed for points of ‘blockage’ or necessary linkages in the implementation chain

Use documentary analysis as part of sequential data collection. Documentary analysis can be valuable to inform qualitative interviews so that interview schedules can be used to further test and refine theories.

Input From the Lived Experience Advisory Group

Previous realist studies have noted the value of using the views of a lived experience group as a part of the iterative process (Pearson et al., 2015). Indeed for this study there was a great utility in this. Due to the large amount of raw data in the data extraction form it was agreed by the research team that the data should undergo initial interpretation before consultation with the LEAG group. The research team were able to use sensitivity and experience to create the journey maps (N.B. these maps represented a more digestible form of the data), which allowed for the creation and refinement of programme theories in collaboration with the LEAG group. A new point of view was gained by asking the lived experience advisory group (LEAG) to suggest program theories based on the completed journey maps. Some competing theories were also highlighted in this process. For example, literature identified in the realist review had stated that standardised psychological assessments are beneficial within health services as compared to clinical judgement (Stefanidou et al., 2020). This was disputed by the LEAG and their feedback suggested that standardised assessments around mental health or substance use early on in the appointment might cause negative reactions and disengagement from the process. These insights were used to refine the IPTs and were tested in the qualitative interviews. The journey maps proved to be useful as boundary objects, to translate the research team’s evolving understanding of complex implementation chains and gain valuable insights from people with lived experience.

Key Recommendations:

Input from a lived experience advisory group can uncover new and competing theories to be tested

Journey maps are a useful way to make the data representative but still accessible to individuals outside the research team.

Discussion

A defining feature of a realist evaluation is the process of making the implicit program theories explicit, and then using an iterative process of evidentially refining, supporting or contesting these theories (Jagosh, 2015). The documentary analysis plays a key part in the iterative process of theory construction. Guidance within reporting standards recommends beginning a study with initial rough programme theories to provide a framework with which to synthesise evidence (Wong et al., 2016). These are then updated and refined into programme theories as literature, and emerging evidence is collected (Pawson, 2006). In the current study, numerous IPTs were developed from our systematic and rapid realist reviews. The documentary analysis then led to the generation of valuable new and refined program theories, giving some of the existing theories a narrower and more specific focus. The information from the journey maps also highlighted which of the IPTs may indeed be operating at ground level within United Kingdom SARCs and which required further investigation.

Early Knowledge of a Complex Pathway

It is an established approach that documentary analysis is conducted at the beginning of a research project (e.g Caulley, 1983), when researchers are trying to understand the make-up of the phenomenon. This proved to be valuable in the current study, where the documentary analysis added to the knowledge generated from the systematic and rapid realist review by providing early, localised understanding of the services. However, Bowen (2009) warns about the limitations of documentary analysis within research, outlining that its value lies in the process of combining it with other methodologies and data sources. Atkinson and Coffey 1997 similarly state that organisational records cannot be the sole source to understand the real functioning of a service, but that if researchers remain aware of the document’s purpose and broader contextual functioning, then they still play an essential role in building a picture of an organisation. For this reason, the documentary analysis in this study did not aim to create a complete and fully accurate view of the service but instead sought to improve researchers’ knowledge of the relevant pathways, as well as generating and refining theories that could be tested in subsequent qualitative interviews.

Indeed, conducting the analysis prior to interviewing staff and service users had numerous advantages. Researchers developed an initial understanding of the specific structure of interventions within each of the six SARC sites, which due to the complexity of the pathways and the range of data sources would have been challenging to gain without the documentary analysis. The increased knowledge at this stage of the process allowed for a greater sensitivity to underlying programme theories when conducting the subsequent qualitative interviews. Establishing a prior awareness of the surrounding organisational contexts meant that the subsequent qualitative interviews could be focussed on identifying and exploring underlying generative mechanisms, rather than discussing the structure of the service. Performing the documentary analysis additionally meant that researchers were aware of whether the data provided in the interviews corroborated or refuted the documentary analysis, allowing the opportunity to probe into any discrepancies (Yanow, 2007).

Researcher Sensitivity During Extraction

During data extraction, it was recognised that sensitivity to the topic aided the ability of the analysts to quickly recognise relevant data and to make data categorisation decisions, which concurred with other members of the team. However, this may imply that the researcher will hold a particular stance, which could influence how the data are interpreted. This potential for bias has been noted as a feature of qualitative research (Chenail, 2011). However, it is also suggested that knowledge of the topic is integral to the qualitative process and that the important consideration is transparency and self-reflection on the part of the researcher (Galdas, 2017).

Local Context

In order to evaluate complex services and understand the varying impacts of interventions Pawson et al. (2005) recommend contextualising the differing outcomes, thinking about variables such as organisational culture, staffing levels, leadership and resource allocation. The RAMESES II reporting standards (Wong et al., 2016) state that due to the complexity of capturing underlying CMOs, it is beneficial in a realist evaluation to collect a broad range of data. Indeed, the addition of the documentary analysis and the resulting increase in site-specific knowledge was integral to this study, in terms of contributing to the overall programme theories within this study’s realist evaluation. This process was greatly helped by the variety of sources and documents collected, meaning that a broader range of these contextual variables could be identified for further analysis. The service websites, for example, gave an idea of the culture within the SARC (e.g the language used and organisational ethos), whilst commissioner reports provided information around how service specifications were met (e.g staffing levels) and how resources were used.

One challenge presented by the process of contextualising each SARC was the variation in the quantity and richness of data for each site. It was important to consider what this variation may imply. For instance, a variation in the content of documents, such as the standardised proformas, was an initial indication of a variation of site-specific practice. A variation in the number of documents, however, was not interpreted as that SARC to be doing ‘less’ but instead that different information would be required to create a full picture of the service, for example, through interviews with staff and service users. The documentary analysis was approached with a curious rather than judgemental stance, seeing it as a first set of data which could be expanded upon and tested in the qualitative interviews.

Historical Context

Bowen (2009) states that documentary analysis can be useful to illuminate the historical origin of a phenomenon, which results in another useful contextual variable within the realist approach. Clarke (2013) highlights from a realist perspective that complex interventions are made up of a multitude of parts interacting in contexts, including the historical. Realist literature has also identified the need to include a historical perspective to capture the forces that have led to the present state (Connelly, 2007). The documentary analysis provided an element of this historical context. Examples of this were a Care Quality Commissioning report that detailed how a service had been without a manager for several months and a published study that described the implementation of a new mental health pathway.

Wider National Context

A useful output of the documentary analysis, as well as the localised contextual knowledge, was an understanding of the macro-level infrastructural complex systems that the interventions lay within. By using national service specifications and guidance to structure the data extraction form, researchers were able to examine the policies that SARCs were trying to adhere to. The process also identified the areas where these policies lacked specificity. The realisation that new core principles for the management of mental health and substance misuse were required, in order to extract the relevant data, was an important step in understanding the broader national context. When discussing documentary analysis, Bowen (2009) states that incompleteness of information in the documents should be examined, as it might indicate that particular issues have not been considered, or voices have not been heard. In this case, it allowed researchers to consider that potentially the needs of those with mental health or substance issues had not been comprehensively addressed within the wider national context of sexual assault services.

When considering the issue of incomplete data, however, it is important to consider the broader context of this documentary analysis. Although our research question focussed on mental health and substance misuse pathways, traditionally an SARC’s primary function is for forensic and physical health. The necessity for trauma-informed care within United Kingdom health services, however, is now widely acknowledged (Sweeney et al., 2018), especially in services for sexual assault. A key part of this approach is the manner in which services address vulnerabilities such as mental health and substance misuse needs (Substance Abuse and Mental Health Services Administration, 2014).

Links to Past Research

Documentary analysis is one of the most commonly used tools in health policy research (Dalglish et al., 2020). The benefits of the approach outlined in this study have been echoed in past research, for example, that documentary analysis acts as a valuable supplementary data source to provide context and coherence to other forms of data collection (Shaw et al., 2004), in particular when combined with qualitative interviews (Bardach & Patashnik, 2019). Previous literature has also highlighted how documents are not transparent representations of organisational processes and present their own ‘documentary reality’ (Atkinson and Coffey 1997). To account for this potential superficiality within the documentary analysis, it has been suggested that an interpretative approach with clearly defined research strategies can be used in replacement of purely positivist stance (Shaw et al., 2004). This paper provides support for this approach and demonstrates how documentary analysis can be conducted through a lens of critical realism to address potential limitations of this data source.

The approach that we adopted, to use documentary analysis within a mixed-methods Realist Evaluation, has been used successfully in previous studies. For instance, ‘to support the development of the programme theories and the contextualisation of data collected in the field’ (Rycroft-Malone et al., 2015, p. 15). More recently documentary analysis has been used in Realist studies to specifically inform IPTs (Gilmore et al., 2019). This has included a specific focus on the ‘comprehensive understanding of the process and the overall outcomes in relation to pathway development and implementation’ (Banks et al., 2017, p. 3). Banks et al. (2017) used documentary analysis to generate ‘accounts of the pathway development in each locality, which fed into a comparative matrix table including data on both pathway content and timing’ (Banks et al., 2017). However, our study utilised a journey mapping approach to create a theoretical framework of the service from the hypothesised perspective of service users.

The two-stage documentary analysis (i.e transforming the extracted data into a journey map) was an integral addition to the wider realist evaluation. Although it can be a labour-intensive process (Bowen, 2009), the large gains to researcher’s knowledge base and the numerous emergent theories allowed for a more in-depth and evidence-based understanding of the complex functioning of SARCs. A two-stage analysis allows for a high-level of detail in capturing the macro- and meso-level layers of large-scale interventions, whilst the distilled and accessible representation of the data in the journey maps facilitates the exploration of knowledge gaps and the service’s implementation chain. This two-stage iterative data collection technique is recommended to researchers approaching complex health service evaluations from a realist perspective.

Supplemental Material

sj-pdf-1-ijq-10.1177_16094069211047818 – Supplemental Material for Documentary Analysis Within a Realist Evaluation: Recommendations From a Study of Sexual Assault Referral Centres

Supplemental Material, sj-pdf-1-ijq-10.1177_16094069211047818 for Documentary Analysis Within a Realist Evaluation: Recommendations From a Study of Sexual Assault Referral Centres by Holly Price, Jill Domoney, Steven Ariss, Elizabeth Hughes and Kylee Trevillion in International Journal of Qualitative Methods