Sage Journals: Discover world-class research

Abstract

Background: Outcomes for children who have been received into state care due to maltreatment or neglect are very poor in comparison to the general population. A mental health focussed intervention is being trialled in the UK which seeks to improve outcomes for such children. Although the main study is concerned with outcome evaluation (child mental health measures), a robust process evaluation adds nuance to the findings by examining the relationship between the interventions, the participants and the context-thereby determining what works, for whom and in what context. This paper sets out the protocol for that process evaluation. Methods: The process evaluation is embedded within a Randomised Control Trial and uses Realist Evaluation as its theoretical framework. It sets out the mechanisms of change that are used to effect positive outcomes and outlines the various aspects of the context (including service provision and the legal system that provides the statutory basis for involvement with families). Data collection is primarily qualitative and takes place with a large group of stakeholders. The analysis of the interplay between context, mechanisms and outcomes will provide a richer understanding of the main trial outcomes. Discussion: The nature of the interaction between the social, legal and practice context is complex. The use of a variety of methods including case studies, focus groups and analysis of routine data are justified and it is argued that they will provide for greater understanding of the nature of the interactions within such a challenging context.

Keywords

Foster Care Maltreatment Children Process Evaluation Realist Randomised Controlled Trial

Background

The prevalence of mental health difficulties in children who have been placed in out-of-home care is significantly higher than that of children who have never been looked after(Meltzer et al., 2003; Schoemaker et al., 2020; Vasileva & Petermann, 2017). The impact on care experienced children is not only a result of maltreatment but may also be influenced by the way the support and care they receive is delivered. Factors such as the number of placements children experience or the length of time it takes to be placed can influence their wellbeing(Fawley-King et al., 2017; Perry & Price, 2017; Tarren-Sweeney, 2008). This can have significant lifelong costs to the children and to society: children who are looked after or who have been abused or neglected have poorer overall health(Halfon et al., 1995; Harden, 2004; Turney & Wildeman, 2016), poorer mental health (Engler et al., 2020; Vasileva & Petermann, 2017), are less ready for school (Geiger & Beltran, 2017; Lynch et al., 2017), poorer educational outcomes(Stapleton & Chen, 2020; Tarren-Sweeney & Goemans, 2019) and lower rates of employment(Cassarino-Perez et al., 2018; Kim et al., 2019).

The Best Services Trial (BeST^?) is a Randomised Controlled Trial (RCT) that seeks to evaluate the clinical effectiveness and cost effectiveness of the New Orleans Intervention (NIM), a mental health intervention for children under the age of five entering care due to maltreatment (i.e. abuse and/or neglect)(Minnis et al., 2010). NIM has had a previous evaluation that compared routine data before and after(C. H. Zeanah et al., 2001), but has never been evaluated in a RCT. For this study, clinical effectiveness is defined using mental health outcomes (primarily the Strengths and Difficulties Questionnaire) and a range of relationship, psychiatric diagnostic and cognitive measures over a 2.5-year period. A health economic analysis aims to determine which intervention provides value for money(Deidda et al., 2018). The trial has two sites, Glasgow and London: two local authorities (Glasgow and Renfrewshire) feed into the Glasgow site and five Local authorities (Barking and Dagenham, Bromley, Croydon, Sutton and Tower Hamlets feed into the London site. Every child under five placed in local authority care due to abuse or neglect is eligible for inclusion in the trial (with the exception of those whose parents are unavailable to take part).

Consented families are randomised to one of two trial arms: Services as Usual (SAU- the Social Work process that children and families normally go through to assess parenting capacity) or NIM.

Process evaluation

Although the evaluation of interventions through RCTs is considered the ‘gold standard’ in scientific research, RCTs evaluating complex interventions require a detailed process evaluation to further explain results (Moore et al., 2015). The RCT outcome evaluation on its own does not allow for an analysis of the context(s) within which the interventions were delivered, nor of why interventions are, or are not, effective. A robust process evaluation can furnish policy-makers with additional information to help them decide (1) If an intervention is effective in one context, can we be confident that (a) another organisation (or set of professionals) will deliver it in the same way? And (b) if they do, will it produce the same outcomes in new contexts? (2) If an intervention is ineffective overall in one context, can we be confident that (a) the failure is attributable to the intervention itself, rather than to poor implementation? (b) The intervention does not benefit any of the target population? (c) If it was delivered in a different context, would it be equally ineffective?

BeST^? compares two ‘complex interventions’, defined as ‘interventions with several interacting components’ or where the implementation or interaction with the surrounding context is viewed as ‘complex’(Moore et al., 2015). ‘Realist’ process evaluation methods are well suited for this type of intervention(Fletcher et al., 2016; Pawson & Tilley, 1997).

The Realist Evaluation framework (Pawson & Tilley, 1997) provides a theoretical framework within which to understand the relationships between context, mechanisms and outcomes. It posits that the ‘world is an open system, or a constellation of structures, mechanisms and contexts’(Kazi, 2003), that ‘persons are complex particulars and the events of interest-for example, programme outcomes associated with human services-are the result of complex transactions of many different kinds and cannot be explained simply in terms of a causal link between events at the surface’(Kazi, 2003). Figure 1 gives an overview of the component parts of the realist method.

Figure 1.

Overview of realist evaluation adapted from (Pawson & Tilley, 1997).

The quantitative findings of BeST^? will be described alongside the Context-Mechanism-Outome Pattern Configurations (CMOCs ) in order to identify the configuration of elements necessary to replicate the results of the trial(Pawson & Tilley, 1997). This will be achieved through qualitative examination of the way contextual factors shape, and are shaped by, intervention mechanisms and outcomes, and how causal mechanisms within the context might sustain the status quo or enhance intervention effects. CMOCs will allow us to identify which elements of the interventions and context are necessary for the desired outcomes-describing what works, for whom and why.

Interventions used in BeST^?

Irrespective of whether the family is randomised to NIM or SAU, the child will always have an allocated social worker within the local authority who will oversee the management of the child’s care. The outcome of the assessments/interventions undertaken by SAU or NIM will be reported to the allocated social worker and it is their responsibility to progress the child’s care plan.

NIM

NIM is aimed at ‘improving outcomes in young maltreated children in foster care’ through provision of ‘assessments and intervention to children…who are placed in foster care for abuse or neglect, and to their birth and foster families’(C. H. Zeanah et al., 2001). One of the aims of BeST^? is to establish whether NIM effects better and more timely permanent placement decisions by the legal system: the more intensive scrutiny afforded to parents by NIM might allow the child welfare and legal systems to make more timely and more robust placement decisions (Turner-Halliday et al., 2017; C. H. Zeanah et al., 2001), thereby reducing placement instability(Boyd et al., 2015). Prior to the beginning of the study two thirds of children under five in foster care in Glasgow returned to their birth parents within a year and, of those, two thirds had a further period in care(Minnis et al., 2010). Preliminary evidence from a routine data exploration of the service in New Orleans was that following intervention from NIM, the rate of re-accommodation was much lower than the rates prior to the implementation of NIM and the rates of maltreatment for subsequent siblings were also much lower(C. H. Zeanah et al., 2001).

In BeST^?, NIM is delivered by two teams: one in each trial site, comprising both healthcare (a Consultant Psychiatrist, Consultant Psychologist, Clinical Psychologist and Psychology assistant) and social care staff. Participants randomised to NIM are offered a structured attachment-based assessment involving each actual and potential caregiver. It uses structured interviews, self-report measures and observations (C. H. Zeanah et al., 2001) including The Working Model of The Child Interview (C. Zeanah & Benoit, 1995), Crowell Procedure (Crowell et al., 1988), Child Behaviour Checklist(Achenbach & Rescorla, 2000) and Ages and Stages Questionnaire (Squires et al., 1997). An intervention is then tailored for every family, drawing on a small range of relationship-based therapeutic techniques, including Circle of Security (Powell et al., 2009), Child Parent Psychotherapy (Lieberman et al., 2006) and Video Interactive Guidance (Silhánová et al., 2011), all of which comply with recommendations of a meta-analysis that examined ways of improving parental sensitivity(Bakermans-Kranenburg et al., 2003). Parents are also referred as required to other agencies for help with substance misuse, mental health issues or intrafamilial violence. At the end of the process there is a recommendation of either rehabilitation to birth family or adoption(C. H. Zeanah et al., 2001). Figure 2 outlines the NIM process.

Figure 2.

NIM Logic Model. N.B. These processes are not necessarily linear. Often the parent/child assessment and the foster carer/child assessment occur simultaneously. Assessment/intervention activity is detailed in the pink boxes (top) and the mechanism of change is detailed in the green (bottom).

SERVICES-AS-USUAL

Families not randomised to NIM will receive SAU, which is the assessment and intervention that social services and others normally implement when children are removed from parental care. These SAU are delivered by local authorities who have a statutory duty to safeguard children and provide services (National Guidance for Child Protection, 2014; Great Britain. Department for Education and Skills, 2007). Social workers assess the family and help to engage them with support/clinical services. How SAU is operationalised varies across the UK (Figure 3).

Figure 3.

Services-as-usual Logic Model. Assessment/intervention activity is detailed in the pink boxes (top) and the mechanism of change is detailed in the green (bottom).

The Glasgow and Renfrewshire versions of SAU are an ideal control for NIM because they also offer a relationship-based assessment of the parental capacity to care for the child, but have a social work ethos, including more naturalistic observations of the family and unstructured assessments of case files. They do not contain an infant mental health treatment component. The timelines for both Glasgow and Renfrewshire are necessarily shorter as there is no formal intervention component. Currently SAU in London are relatively sophisticated compared to many areas of the UK because of their close relationship to South London and Maudsley NHS Trust which is a partner in this collaboration. All of the London local authorities adopt one of three different methods for assessing families. The first is an assessment delivered by a specialist team who will offer an assessment of parenting capacity (as with the Glasgow and Renfrewshire models). The second is an assessment carried out by the allocated social worker. The third is a specialist assessment carried out by an independent social worker/psychologist/psychiatrist. All three methods of assessment must be undertaken within 26 weeks as per the expectations set out by the judiciary. In contrast with NIM, the systems of care that comprise SAU have developed iteratively, borne out of local authority contexts and processes and have never been formally evaluated. Further exploration of the detailed nature of SAU is a key objective of this process evaluation. The heterogenous nature of services as usual is seen as a strength in that it will allow for a more detailed exploration of what works, for whom and why.

The key differences between the arms of the trial are summarised in Table 1 below:

Table 1.

Differences between NIM, SAU in Glasgow and SAU in London.

NIM	SAU (Glasgow)	SAU (London)
• Multi-disciplinary (psychology/Social work/Psychiatry) • 3-month assessment • Standardised psychological assessment focus (detailed focus on attachment relationships) • 6–9 months in-house treatment if indicated • Recommendation made re re-unification or adoption – usually after 1 year	• Uni-disciplinary – Social work • 3-month assessment • Detailed review of case notes • Observational assessment at contact (naturalistic focus on quality of parent-child relationship) • No ‘official’ treatment component but can refer to external services (e.g. psychology, psychiatry) if required • Recommendation made re re-unification or adoption – usually after 3 months	One of the following: • Specialist Social Work Assessment (as per Glasgow Model) • Case-holding Social Worker’s assessment (using the methodology of the Glasgow model) • Independent expert assessment (usually a social worker, psychiatrist or psychologist)

Contexts

There are many commonalities between the trial sites and New Orleans, where NIM originates, including the nature of family demographics, dysfunction and challenges (Minnis et al., 2010). There are also several differences in context both within the UK and between the UK and New Orleans.

The UK has a National Health Service, free at the point of contact, and a fully-developed welfare state that includes a preventative social work system. These are not available in New Orleans. The preventative social work approach in the UK might mean that, for those families where children experience maltreatment and enter into foster care, family problems may be more entrenched as those families with less entrenched difficulties may already have received support and change may have been affected earlier. In contrast, in New Orleans, families receiving NIM are more likely to be previously unknown to services, so may have a greater range of severity of difficulties, some of which may respond better to intervention(Minnis et al., 2010).

The use of ‘dual registered’ foster carers, primed to adopt the child if rehabilitation to birth family is not possible, is the norm in New Orleans but is relatively rare in the UK. The prevailing model in the UK involves children being placed with ‘temporary’ foster carers whose role is to care for and nurture children while their birth parents undergo assessment. A decision is then taken by social services as to whether those children should return to the care of their parents or should be placed in more permanent alternative care (of which adoption is the primary route for children in BeST^?). This necessitates disruption in the attachment that has been formed with the foster carer in the hope that a strong relationship can be forged with birth parents (in the case of rehabilitation) or with alternative permanent carers. In New Orleans, ‘dual-registered’ foster carers can assume care of the children when they are removed from their parents and, if children are unable to be returned to their birth parents, can go on to become the child’s permanent carers, necessitating no disruption in the child-foster carer relationship and no additional trauma or loss. The NIM intervention is undertaken with birth parents and the foster carers caring for the child at that time. In New Orleans, this means that any work done on promoting the mental health of the child is likely to be done with a carer that will ultimately assume permanent care of the child(C. H. Zeanah et al., 2001). In The UK, should the child not return to the care of their birth parents, the carers who assume permanent care of the child may not have undertaken work with NIM in respect of the child. This is a very important contextual difference between the UK and the US which does not affect the delivery of NIM but could have a profound impact on its effects.

There are also fundamental differences in the way the legal systems of Glasgow, London and New Orleans make decisions about children in foster care. In London and New Orleans the legal systems are judge led proceedings in which timescales are defined by statute, and a judge has oversight of the decisions that are taken in respect of children in foster care. In New Orleans, the Adoption and Safe Families Act decrees that if a child is in foster care for 15 of the previous 22–months, the state must move to terminate parental rights and that permanence hearings for those children must be held at least every 12 months. In London, the target for making a long-term placement decision is 26 weeks (Children and Families Act 2014 s.14 (2)(ii)), although these timescales can be relaxed up to 40 weeks if a child is randomised to NIM. In contrast, Glasgow has a lay tribunal system in which most children in foster care come under the jurisdiction of the Children’s Hearing System: specially trained members of the public (Members of the Children’s Panel) sit in Children’s Hearings and make decisions about all aspects of a child’s life-from where that child lives, the duration of frequency of family contact, to whether that contact should be supervised and beyond (Norrie, 2013). The judiciary in Glasgow, known as Sheriffs, have little oversight of the day-to-day decision-making in respect of children in foster care other than establishing the facts of the circumstances that led to the reception into care and hearing any appeals of decisions made at Children’s Hearings (Norrie, 2013). Unlike in New Orleans and in London, there are no statutory timescales applied in Glasgow to long term decision-making. These differences may influence how successfully or otherwise the NIM model may be applied in each of the UK sites.

Key process evaluation questions

1) What are the key components of NIM and SAU in the Glasgow and London sites?

2) Are both Interventions in both sites adhered to or do they change during the study?

3) What is impact of NIM and SAU on wider systems in both sites?

4) How does the wider system impact on the functioning of NIM and SAU in both sites?

Methods and data collection

The relationship between the key process evaluation questions and an overview of the data to be collected, in two phases, is summarised below and in Tables 2 and 3. Phase one ran from 2016 to 2019 and is pertinent to the bedding in of the interventions and of the trial itself. The second phase (2019–2024) will capture perceptions of the interventions and how they operate. Once the interventions and the trial are well established within their contexts.

Table 2.

Relationship between Process Evaluation Questions and data collection methods.

	Key Process Evaluation Questions	What Are the Key Components of NIM and SAU in the London and Glasgow Sites?	Are Both Interventions in Both Sites Adhered to or do They Change during the Study?	What is Impact of NIM and SAU on Wider Systems in Both Sites?	How Does the Wider System Impact on the Functioning of NIM and SAU in Both Sites?
Data Collection Method	Birth parents			x	x
	Foster carers/Adoptive parents			x	x
	NIM and SAU practitioners	x	x	x	x
	NIM and SAU Managers	x	x	x	x
	Local Authority Social Workers	x	x	x	x
	Local Authority Social Work Managers	x	x	x	x
	Legal personnel			x	x
	Case studies	x	x	x	x
	Monitoring data/routine data			x	x
	Emergent issues		x	x	x

Table 3.

Data collection methods, participants and frequencies.

Participants	Method of Data Collection	Number/Frequency	Site	Trial Arm
Birth parents	Interview	3 interviews per arm, per site, per phase	Both	Both
Foster carers/Adoptive parents	Interview	3 interviews per arm, per site, per phase	Both	Both
NIM and SAU practitioners	Focus group	1 per group, annually	Both	Both
NIM and SAU Managers	Interview	1 per group, annually	Both	Both
Local Authority Social Workers	Focus group	1 group per site per phase	Both	Both
Local Authority Social Work Managers	Interview	2 Interviews per site per phase	Both	Both
Legal personnel	Focus groups	9 focus groups	Both	Both
Case studies	Case study	2 per arm per site	Both	Both
Monitoring data/routine data	Case record analysis Service use questionnaire	Data gathered for every participant	Both	Both
Emergent issues	As required	As required	Both	Both

Questions 1 and 2

Over both phases of data collection, a purposive sample of parents/foster carers, social workers, intervention team members, managers and birth parents are invited to take part in interviews and focus groups to identify the perceived key components of NIM and SAU in both sites and to capture any changes in practice or delivery during the lifetime of the study. The frequency and type of data collected are outlined in Table 3.

Quantitative data on every participant are collected regarding the use of health and social services both within and out with NIM and SAU. In addition, Professors Zeanah and Larrieu of Tullane University (where NIM originated) have agreed to rate 10 randomly selected reports from the NIM service in Glasgow to ensure model fidelity using a checklist prepared for the study. The NIM service in Glasgow will then assist with ensuring model fidelity in London with the London NIM service. At the start of phase two, a focus group with Professor Zeanah and the managers from the Glasgow and London NIM services took place to ensure adherence to the NIM intervention.

Questions 3 and 4

Because of the importance, for the success of the interventions, of positive interactions with the social work and legal systems ( synchronicity between the outputs of NIM and various legal proceedings, having NIM reports tabled at court etc.), we will need detailed understanding of these processes. A skilled social work researcher extracts routine quantitative data on the children and families coming through the trial relating their journeys through the interventions and the social work and legal systems and qualitative information from social workers and legal professionals regarding these wider systemic processes.

Qualitative methods need to be responsive to the needs and circumstances of the different participant groups: focus groups allow for both exploration of the area under discussion and observation of the group dynamics which highlight commonalities and differences between participants. Birth parents who have had children removed from their care and foster or adoptive carers warrant individual one-to-one interviews given the sensitivity of the subject matter and the need to be flexible with the participants. Given the limited number of senior managers within SAU and NIM, individual interviews may facilitate in depth discussions and more granular understanding of services and strategic perspectives.

Birth parents

Semi structured interviews have been conducted with a sample of birth parents involved each service (intervention and control) in both sites centring around their motivation to take part in the research, their experience and understanding of the randomisation process, their experience of involvement with the assessments and (if offered) any treatment and of their views of the social work and legal systems.

Foster carers/adoptive parents

A purposive sample of foster carers involved SAU and NIM across both sites are invited to take part in semi structured interviews focusing on their perceptions of their roles and the impact they have on children’s longer-term outcomes. Further interviews will also take place with dual registered foster carers.

All Foster Carers are asked to fill out a Use of Services Questionnaire, allowing fuller assessment of model fidelity and health economic analysis.

NIM and services as usual practitioners

Approximately annual focus groups are conducted with each of the teams delivering NIM and SAU across both sites focusing on the running of each service, and interactions with social work and health services. Time is also devoted to issues that each service feels is important, including the interface each service has with the legal system, specific legal challenges and opportunities identified by practitioners.

NIM and services as usual managers

Approximately annual semi structured interviews take place with the managers of the SAU and NIM services including discussion of operational challenges and opportunities.

Local authority social workers

Groups of case managing social workers are invited to take part in focus groups (one per phase per site), examining social worker’s experiences of the care system and of working with SAU or NIM and views of the strengths and challenges of legal system for participating children There are separate focus groups for Glasgow and London social workers.

Local authority social work managers

As with social work practitioners, managers are asked to take part in semi structured interviews (one per phase per site) looking at the interface between the trial and both SAU and NIM (across both sites). Managers are asked to discuss issues brought to them by staff and also operational/strategic issues that they have identified.

Interviews with senior managers and first line managers within each of the authorities, along with scrutiny of local policies and procedures, are used to draw up models of services-as-usual across the sites.

Legal personnel

Interviews and focus groups are used to describe and evaluate the legal systems in which the trial operates. In Glasgow, separate focus groups are undertaken with legal practitioners representing birth families, adoptive families and local authorities to examine perceptions of the legal system, experiences of SAU and of NIM and views on how the legal system influences outcomes for children. Focus groups are also undertaken with those legal personnel who make decisions about children in care – specifically Sheriffs and members of the Children’s Panel – to examine views of SAU and NIM, the interface of NIM and SAU with the legal system and how the system influences outcomes.

In London, focus groups are conducted with solicitors representing birth families, adopters and local authorities – focussing on the same topics as the corresponding groups in Glasgow. Focus groups with judges in London examine their interaction with SAU and NIM and more general views on how the legal system influences outcomes. A focus group with staff and managers from the Children and Families Court Advisory Service (CAFCAS) examines their interface with the trial and with SAU and NIM. CAFCAS staff are also invited to discuss their views of the system and its relationship to children’s outcomes.

Case studies

Four case studies have been conducted in each site concerning participants, their social workers and/or therapists around the time of initial involvement with either arm of the trial and a later exercise (during Phase 2) following up all participants involved in the earlier case studies. Sampling is purposive; SAU and NIM identify a number of cases where there were indications, during the process of assessment, about the nature of the final recommendation. Two cases from each service are selected; one case from each service where indications are suggestive of the child/ren being rehabilitated home, and one case from each service where it looks likely that a recommendation will be made for adoption. Participants are invited to take part in case study interviews based on their involvement in the case and their ability to provide a view on the child from their specific perspective; that is, from the angle of the parent, the foster carer, the area team social worker and the key workers involved in the case from either SAU or NIM.

Monitoring data/routine data

Social work case records are used to quantify the length of time taken for children to enter into a permanent placement and will also be used to map children’s legal journey’s over the time they are accommodated. These data will allow for analysis of the impact of the legal system and the variety of decisions made within legal fora on children’s outcomes. Health records are used to identify adverse events and will also allow researchers to quantify and describe children’s health service use.

We collate data from NIM and SAU teams at each site, including monthly returns from staff regarding training and supervision received in the team and on appointments offered to families and the proportion of these attended. In addition, Use of Services diaries will be completed by foster carers and brought to the clinic at each research assessment to allow accurate completion of the Use of Services Questionnaire. These data will contribute to both the assessment of model fidelity and to the health economic analysis.

Emergent issues

During the lifetime of the trial, the process evaluation team may conduct further interviews and focus groups if required to describe and evaluate any emergent issues or unforeseen difficulties that have been identified either by our existing focus groups or through trial management activity. This could potentially involve other professional groups or stakeholders if the issues that are identified require information that is held by groups other than those we have included here. This part of the process evaluation will be iterative by definition and will seek to log and describe the interplay of these emergent issues with the services and the context.

Analysis

All the interviews and focus groups are transcribed verbatim, then is read by at least two members of the research team. Thematic Analysis (Braun & Clarke, 2006)provides the framework for this process. An initial coding of the transcripts will identify key categories in line with the realist analysis, and to answer the four key questions, then we will produce a thematic analytical framework used to analyse the remaining interviews. This is an iterative process and is continually reviewed and agreed upon as the research progresses to ensure that it continues to meet our research needs.

Discussion

Using a realist evaluation model, the mechanisms of the interventions (SAU and NIM), aspects of the wider context, and the context-mechanism-outcome patterns will be detailed. The quantitative outcomes measured by BeST^? are concerned with child mental health, the time taken for children to be placed in a permanent placement, relationship functioning and cost effectiveness. The process evaluation will allow for a richer understanding of each of these outcomes.

SAU and NIM have a number of constituent parts. Although other process evaluations have detailed the theory of change underpinning trial interventions (e.g. Arthur et al. (2020); Moran and Gutman (2020); O’Brien et al. (2019)), we are unaware of other process evaluations which seek to outline which of the component parts of the interventions are considered ‘key’ to the delivery of the intervention and the effecting of change.

Despite conceptual frameworks existing for Process Evaluation (e.g. Moore et al. (2015); Pawson and Tilley (1997)), few published protocols of realist evaluations embedded within RCTs exist(French et al., 2020; Gilissen et al., 2020; O’Brien et al., 2019). The core of the realist method is understanding the influence of the context and the intervention mechanisms on the outcomes(Moore et al., 2015; Pawson & Tilley, 1997). Often, descriptions of the services being compared are not sufficient to unpick the complex interactions of context, mechanism and outcomes(French et al., 2020; Murdoch, 2016; Wells et al., 2012). Case studies are a useful element within a range of methods that ‘capture the dynamic and complex relationship between intervention and context during implementation’ (Grant et al. (2020). The frequency and timing of our data collection is intended to capture change over time (Gilissen et al., 2020; Maddison et al., 2020).

There are distinct legal and social work systems in operation over both the sites in which the trial operates. The interventions are also embedded within multi agency contexts. Using the methodology set out in this paper, the impact of both SAU and NIM on the wider context, and the impact of the context on both SAU and NIM will be interrogated. Although previous process evaluations have documented the interactions in such complex and challenging contexts (e.g. Bridges et al. (2017); Kirkpatrick et al. (2018); Segrott et al. (2017)) we are unaware of any evaluations that have sought to interrogate how the legal system and decisions made by legal tribunals interact with the interventions being tested or influence the outcomes. In this regard, BeST^? is offering a novel contribution to the literature.

Regardless of our quantitative findings, our process evaluation will offer useful information to researchers and policy-makers who may wish to test or roll out NIM in different social and legal contexts in future and will offer insight into what works, in what context and for whom.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This clinical trial is funded by National Institute for Health Research, Public Health Research Unit (12/211/54). Prof Ruth Dundas is supported by Medical Research Council grant MC_UU_00022/2, and Chief Scientist Office Grant SPHSU17

Ethical apporval

The study received favourable ethical opinion from the West of Scotland Ethics Committee

ORCID iDs

Gary Kainth

Helen Minnis

References

Achenbach

T. M.

Rescorla

L. A.

(2000). Manual for the ASEBA preschool forms and profiles (Vol. 30). University of Vermont, Research center for children.

Arthur

Y. A.

Boardman

G. H.

McCann

T. V.

(2020). Qualitative process evaluation of a problem‐solving and story‐bridge based mental health literacy program with community leaders in ghana. International Journal of Mental Health Nursing, 30(3), 683–693. https://doi.org/10.1111/inm.12832

Bakermans-Kranenburg

M. J.

Van Ijzendoorn

M. H.

Juffer

(2003). Less is more: meta-analyses of sensitivity and attachment interventions in early childhood. Psychological Bulletin, 129(2), 195–215. https://doi.org/10.1037/0033-2909.129.2.195

Boyd

K. A.

Balogun

M. O.

Minnis

(2015). Development of a radical foster care intervention in Glasgow, Scotland. Health Promotion International, 31(3), 665–673. https://doi.org/10.1093/heapro/dav041

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Bridges

May

Fuller

Griffiths

Wigley

Gould

Libberton

(2017). Optimising impact and sustainability: A qualitative process evaluation of a complex intervention targeted at compassionate care. BMJ Quality & Safety, 26(12), 970–977. https://doi.org/10.1136/bmjqs-2017-006702

Cassarino-Perez

Crous

Goemans

Montserrat

Sarriera

J. C.

(2018). From care to education and employment: A meta-analysis. Children and Youth Services Review, 95(2018), 407–416. https://doi.org/10.1016/j.childyouth.2018.08.025

Crowell

J. A.

Feldman

S. S.

Ginsberg

(1988). Assessment of mother-child interaction in preschoolers with behavior problems. Journal of the American Academy of Child and Adolescent Psychiatry, 27(3), 303–311. https://doi.org/10.1097/00004583-198805000-00007

Deidda

Boyd

K. A.

Minnis

Donaldson

Brown

Boyer

N. R.

McIntosh

(2018). Protocol for the economic evaluation of a complex intervention to improve the mental health of maltreated infants and children in foster care in the UK (The BeST? services trial). BMJ open, 8(3), Article e020066. https://doi.org/10.1136/bmjopen-2017-020066

10.

Engler

A. D.

Sarpong

K. O.

Van Horne

B. S.

Greeley

C. S.

Keefe

R. J.

(2020). A systematic review of mental health disorders of children in foster care. Violence, & Abuse. 23(1). 255–264. https://doi.org/10.1177/1524838020941197

11.

Fawley-King

Trask

E. V.

Zhang

Aarons

G. A.

(2017). The impact of changing neighborhoods, switching schools, and experiencing relationship disruption on children’s adjustment to a new placement in foster care. Child Abuse & Neglect, 63(Jan 2017), 141–150. https://doi.org/10.1016/j.chiabu.2016.11.016.

12.

Fletcher

Jamal

Moore

Evans

R. E.

Murphy

Bonell

(2016). Realist complex intervention science: Applying realist principles across all phases of the medical research council framework for developing and evaluating complex interventions. Evaluation, 22(3), 286–303. https://doi.org/10.1177/1356389016652743

13.

French

Pinnock

Forbes

Skene

Taylor

S. J.

(2020). Process evaluation within pragmatic randomised controlled trials: what is it, why is it done, and can we find it?—a systematic review. Trials, 21(1), 1–16. https://doi.org/10.1186/s13063-020-04762-9

14.

Geiger

J. M.

Beltran

S. J.

(2017). Readiness, access, preparation, and support for foster care alumni in higher education: A review of the literature. Journal of Public Child Welfare, 11(4–5), 487–515. https://doi.org/10.1080/15548732.2017.1354795

15.

Gilissen

Pivodic

Wendrich-van Dael

Gastmans

Vander Stichele

Engels

Van den Block

(2020). Implementing the theory-based advance care planning ACP+ programme for nursing homes: Study protocol for a cluster randomised controlled trial and process evaluation. BMC Palliative Care, 19(1), 1–18. https://doi.org/10.1186/s12904-019-0505-7

16.

Great Britain. Department for Education and Skills (2007) Statutory guidance on making arrangements to safeguard and promote the welfare of children under section 11 of the Children Act 2004. TSO.

17.

Grant

Bugge

Wells

(2020). Designing process evaluations using case study to explore the context of complex interventions evaluated in trials. Trials, 21(1), 1–10. https://doi.org/10.1186/s13063-020-04880-4

18.

Halfon

Mendonca

Berkowitz

(1995). Health status of children in foster care: The experience of the center for the vulnerable child. Archives of Pediatrics & Adolescent Medicine, 149(4), 386–392. https://doi.org/10.1001/archpedi.1995.02170160040006

19.

Harden

B. J.

(2004). Safety and stability for foster children: A developmental perspective. The Future of Children, 14(1). 31–47. https://doi.org/10.2307/1602753

20.

Kazi

(2003). Realist evaluation for practice. British Journal of Social Work, 33(6), 803–818. https://doi.org/10.1093/bjsw/33.6.803

21.

Kim

Rosenberg

Farmer

E. B. M.

(2019). Estimating the effects of independent living services on educational attainment and employment of foster care youth. Children and Youth Services Review, 96(3), 294–301. https://doi.org/10.1016/j.childyouth.2018.11.048

22.

Kirkpatrick

Lennox

Taylor

Anderson

Maguire

Haddad

Stirzaker

(2018). Evaluation of a complex intervention (Engager) for prisoners with common mental health problems, near to and after release: Study protocol for a randomised controlled trial. BMJ open, 8(2). Article e017931. https://doi.org/10.1136/bmjopen-2017-017931

23.

Lieberman

A. F.

Ippen

C. G.

Van Horn

(2006). Child-parent psychotherapy: 6-month follow-up of a randomized controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 45(8), 913–918. https://doi.org/10.1097/01.chi.0000222784.03735.92

24.

Lynch

Ashcraft

Paul-Ward

Tekell

Salamat

Schefkind

(2017). Occupational therapy’s role in mental health promotion, prevention, & intervention with children & youth: Foster care (pp. 1–3). School Mental HealthAmerican Occupational Therapy Association. [Information sheet].

25.

Maddison

Hargreaves

E. A.

Jiang

Calder

A. J.

Wyke

Gray

C. M.

Draper

(2020). Rugby Fans in Training New Zealand (RUFIT-NZ): protocol for a randomized controlled trial to assess the effectiveness and cost-effectiveness of a healthy lifestyle program for overweight men delivered through professional rugby clubs in New Zealand. Trials, 21(1), 139. https://doi.org/10.1186/s13063-019-4038-4

26.

Meltzer

Gatward

Corbin

Goodman

Ford

(2003). The mental health of young people looked after by local authorities in England. The Stationery Office.

27.

Minnis

Bryce

Phin

Wilson

(2010). The "Spirit of New Orleans": Translating a model of intervention with maltreated children and their families for the Glasgow context. Clinical Child Psychology and Psychiatry, 15(4), 497–509. https://doi.org/10.1177/1359104510376124

28.

Moore

G. F.

Audrey

Barker

Bond

Bonell

Hardeman

Wight

(2015). Process evaluation of complex interventions (p. 350). Medical Research Council guidance. BMJ.

29.

Moran

Gutman

L. M.

(2020). Mental health training to improve communication with children and adolescents: A process evaluation. Journal of Clinical Nursing, 30(3–4). 415–432. https://doi.org/10.1111/jocn.15551

30.

Murdoch

(2016). Process evaluation for complex interventions in health services research: analysing context, text trajectories and disruptions. BMC Health Services Research, 16(1), 407. https://doi.org/10.1186/s12913-016-1651-8

31.

National Guidance for Child Protection (2014). Scotland: Consultation. APS Group Scotland.

32.

Norrie

K. M.

(2013). Children’s hearings in Scotland (3rd ed.). Sweet & Maxwell/Thomson Reuters.

33.

O’Brien

Buston

Wight

McGee

White

Henderson

(2019). A realist process evaluation of Enhanced Triple P for Baby and Mellow Bumps, within a Trial of Healthy Relationship Initiatives for the Very Early years (THRIVE): study protocol for a randomized controlled trial. Trials, 20(1), 351. https://doi.org/10.1186/s13063-019-3395-3

34.

Pawson

Tilley

(1997). Realistic evaluation. sage.

35.

Perry

K. J.

Price

J. M.

(2017). The role of placement history and current family environment in children’s agpgression in foster care. Journal of Child and Family Studies, 26(4), 1135–1150. https://doi.org/10.1007/s10826-016-0642-z

36.

Powell

Cooper

Hoffman

Marvin

R.S.

(2009). Circle of Security. In Zeanah

C.H.

(Ed.), Handbook of Infant Mental Health (pp. 450–467). The Guildford Press.

37.

Schoemaker

N. K.

Wentholt

W. G.

Goemans

Vermeer

H. J.

Juffer

Alink

L. R.

(2020). A meta-analytic review of parenting interventions in foster care and adoption. Development and Psychopathology, 32(3), 1149–1172. https://doi.org/10.1017/S0954579419000798

38.

Segrott

Murphy

Rothwell

Scourfield

Foxcroft

Gillespie

Morgan-Trimmer

(2017). An application of extended normalisation process theory in a randomised controlled trial of a complex social intervention: Process evaluation of the strengthening families Programme (10–14) in Wales, UK. SSM Population Health, 3(Dec 2017), 255–265. https://doi.org/10.1016/j.ssmph.2017.01.002.

39.

Silhánová

Cave

Fukkink

Sancho

Doria

Bristow

Trevarthen

(2011). Video interaction guidance: A relationship-based intervention to promote attunement, empathy and wellbeing. Jessica Kingsley Publishers.

40.

Squires

Bricker

Potter

L. J. J. o. p. p.

(1997). Revision of a parent-completed developmental screening tool: Ages and Stages Questionnaires. Journal of Pediatric Psychology, 22(3). 313–328. https://doi.org/10.1093/jpepsy/22.3.313

41.

Stapleton

D. H.

Chen

R. K.

(2020). Better outcomes for children in treatment foster care through improved stakeholder training and increased parent-school collaboration. Children and Youth Services Review, 114(July 2020), 105010. https://doi.org/10.1016/j.childyouth.2020.105010.

42.

Tarren-Sweeney

(2008). Retrospective and concurrent predictors of the mental health of children in care. Children and Youth Services Review, 30(1), 1–25. https://doi.org/10.1016/j.childyouth.2007.05.014

43.

Tarren-Sweeney

Goemans

(2019). A narrative review of stability and change in the mental health of children who grow up in family-based out-of-home care. Developmental Child Welfare, 1(3), 273–294. https://doi.org/10.1177/2516103219874810

44.

Turner-Halliday

Kainth

Young-Southward

Cotmore

Watson

McMahon

Minnis

(2017). Clout or doubt? Perspectives on an infant mental health service for young children placed in foster care due to abuse and neglect. Child Abuse & Neglect, 72(October 2017), 184–195. https://doi.org/10.1016/j.chiabu.2017.07.012.

45.

Turney

Wildeman

(2016). Mental and physical health of children in foster care. Pediatrics, 138(5). Article e20161118. https://doi.org/10.1542/peds.2016-1118

46.

Vasileva

Petermann

(2017). Posttraumatic stress symptoms in preschool children in foster care: The influence of placement and foster family environment. Journal of Traumatic Stress, 30(5), 472–481. https://doi.org/10.1002/jts.22217

47.

Wells

Williams

Treweek

Coyle

Taylor

(2012). Intervention description is not enough: Evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions. Trials, 13(1), 95. https://doi.org/10.1186/1745-6215-13-95

48.

Zeanah

Benoit

(1995). Clinical applications of a parent perception interview in infant mental health. Child and Adolescent Psychiatric Clinics of North America, 4(3), 539–554. https://doi.org/10.1016/S1056-4993(18)30418-8

49.

Zeanah

C. H.

Larrieu

J. A.

Heller

S. S.

Valliere

Hinshaw-Fuselier

Aoki

Drilling

(2001). Evaluation of a preventive intervention for maltreated infants and toddlers in foster care. Journal of the American Academy of Child and Adolescent Psychiatry, 40(2), 214–221. https://doi.org/10.1097/00004583-200102000-00016

Process evaluation protocol for the BeST ? Services trial

Abstract

Keywords

Background

Process evaluation

Interventions used in BeST?

NIM

SERVICES-AS-USUAL

Contexts

Key process evaluation questions

Methods and data collection

Questions 1 and 2

Questions 3 and 4

Birth parents

Foster carers/adoptive parents

NIM and services as usual practitioners

NIM and services as usual managers

Local authority social workers

Local authority social work managers

Legal personnel

Case studies

Monitoring data/routine data

Emergent issues

Analysis

Discussion

Footnotes

Declaration of conflicting interests

Funding

Ethical apporval

ORCID iDs

References

Interventions used in BeST^?