Abstract
Objectives
Better integrated health and social or long-term care is high on government policy agendas in many countries. In England, successive pilot programmes, with related national evaluations, have been introduced to better integrate care to meet the needs of people requiring multi-agency help. However, researchers evaluating such programmes both in England and internationally face a daunting number of challenges produced by service delivery and research regulatory systems. This paper analyses the challenges encountered in seeking to undertake a prospective quasi-experimental evaluation of the impacts of community based multi-disciplinary teams (MDTs) on patient experience and outcomes, as part of a wider evaluation of the Integrated Care and Support Pioneers programme. The paper also identifies a number of general lessons for research commissioners, study site participants, and those tasked with undertaking such evaluative research.
Methods
We reviewed our research activities and timelines from the start of the evaluation. We created a narrative history - using reports to the funder, applications to research and ethics regulatory bodies and correspondence with Pioneer sites, regulatory bodies and data providers - to describe the challenges faced and our approaches to attempting to mitigate them.
Results
We experienced four key challenges: (1) unrealistic commissioner research specifications; (2) negotiating with and recruiting multiple organisations and services at potential study sites; (3) navigating research ethics and governance systems; and (4) recruiting participants for primary data collection and obtaining (with their consent) their linked routine service use data. The first two challenges resulted from the lack of shared understanding of evaluation feasibility and constraints between local health and care system actors and national level commissioners of evaluation, plus no clear incentive for local sites to participate. The third and fourth challenges were the product of multiple, protracted, and unnecessarily risk-averse research approval processes which affected both the nature and quantity of the data we could collect.
Conclusions
We recommend that major changes are made to the regulation of policy research to enable more robust evaluation to take place and that disproportionately high levels of risk aversion in approval processes for non-interventional, low-risk studies are addressed. In addition, the evaluation commissioning process needs to be far better informed at an early stage about which elements in programmes can feasibly be evaluated before research specifications are advertised.
Introduction
The importance of evaluating health and care policy programmes involving innovative service changes, complex novel interventions and new ways of working, along with system-wide changes designed to improve care, is widely acknowledged.1,2 The English National Health Service (NHS), as a universal publicly financed health care system, might be considered a conducive research environment: patients have a unique NHS number and there are high quality data on hospital use and a strong clinical trials infrastructure. The evaluation of complex interventions, however, is itself both complex and challenging,3,4 particularly when it involves multiple health and care organisations outside the acute hospital sector.
As in many countries, better integrated health and social care is high on the government’s policy agenda in England.5,6 Since 2008, three major integration pilot programmes have been initiated, with associated evaluations (Integrated Care Pilots (2009-2011), Integrated Care and Support Pioneers (2013–2018) and New Care Model ‘Vanguards’ (2015–2018)). The pilots have had mixed results. 7 As interventions, integrated care programmes are complex and multi-faceted, and evaluating their outcomes is known to be problematic. 8 Where participating sites are encouraged to innovate ‘bottom-up’ to meet local population needs, researchers may, for example, struggle to identify a relatively standard integration initiative of strategic significance that can feasibly be evaluated and that is sufficiently distinct from previous approaches to integration to offer scope for comparison.
Such challenges are not entirely unexpected. What may not be anticipated is quite the extent to which the multi-layered research governance system in England can compound ‘routine’ research challenges when seeking to evaluate complex integration initiatives designed to improve patient experiences and outcomes. The aims of this paper are therefore: (a) to describe how a number of different types of challenges combined to significantly impede our attempt to evaluate a specific integration initiative - community-based health and social care integrated multi-disciplinary teams. These challenges arose despite the evaluation being part of a wider evaluation of a very high-profile national pilot programme (the Integrated Care and Support Pioneers), and having the benefit of Ministerial support and local goodwill; and (b) to outline ways in which evaluative research on complex integrated care initiatives might be better commissioned, governed, and facilitated locally, so that evaluations of this type of initiative, designed as they are to benefit both individuals and health and care systems, do not become infeasible.
The 25 Pioneers, selected by the then Department of Health (DH), volunteered to encompass whole system integration, make services more patient-centred, improve patient experience, outcomes and quality of life, and reduce budget pressures. 9 Focussing on horizontal integration between NHS and social care services, they had access to expertise (e.g. from NHS England) and information-sharing conferences, but very limited additional funding for early stage project management and none for service delivery. Each encompassed at least one NHS Clinical Commissioning Group (CCG) area and one Local Authority (LA). When the evaluation was being undertaken, CCGs were responsible for the local planning and commissioning of NHS hospital and community health services. Elected LAs are statutorily responsible for publicly subsidised provision of adult social care (i.e. long-term care) in their areas, separate from the NHS. Most Pioneers deployed similar initiatives that built on their previous activities, often targeted at older people with multiple chronic conditions and/or at risk of hospital admission.10,11
This paper draws on our experiences of evaluating community-based, health and social care integrated multi-disciplinary teams (MDTs) as part of a broader national evaluation (2015–2022) of the Pioneer Programme. MDTs, one of the most widely reported integration initiatives by the Pioneers,10,12 and widespread in England, are an evolution of existing approaches to care co-ordination and interprofessional information sharing, and are neither entirely novel nor a risk to patients as the focus of research. Our evaluation sought to provide stronger evidence regarding the costs and benefits of integration than previous studies, the limitations of which include a paucity of rigorous comparative research on outcomes for those receiving more integrated care and similar patients not receiving such care (as well as the lack of data on the impact of integrated care initiatives on patients’ use of health and care services).
Studies that consider the role of social workers in integration are rare. 13 We therefore designed a prospective, multiple-method, quasi-experimental evaluation to compare the outcomes, care experiences and costs of care for patients aged 55 years and older with multiple chronic conditions who had been added to the caseloads of community-based MDTs that included both health and social care staff, with those of matched patients not on MDT caseloads. The aim was to determine if such MDTs improve care delivery and outcomes over time and reduce costs, and, using qualitative research, to explain any observed differences in the effectiveness and/or cost-effectiveness of such teams in different local system contexts.
Original protocol and revisions
We planned to recruit patients aged 55 and over with multiple chronic conditions added to the caseloads of at least two MDTs in each of at least three Pioneers, with help from MDT administrative staff, and matched non-MDT patients from the same geographical areas through general practices, with help from the local NHS Commissioning Support Unit (CSU). Local administrative staff would invite eligible patients to complete a postal questionnaire to be returned to us. We would send consenting participants follow-up surveys and interview a purposively selected sub-sample about care experiences twice during an 18-month follow-up period. Those involved in recruiting MDT and matched patients would record all eligible patients sent a survey, which for ethical reasons would not be shared with us. We would therefore only have consenting patients’ personal details. Likewise, we would not identify participants to NHS or other staff, thereby avoiding any risk of their care being affected by study participation.
Where patients nominated informal carers, we would send them a similar postal questionnaire, with participating carers receiving an 18-month follow-up questionnaire, and a sub-sample being interviewed twice. Purposively selected NHS, CCG and LA managers would be interviewed about the place of MDTs in their local system. A pro forma describing each MDT would be completed, and frontline MDT staff surveyed online and a sub-sample interviewed twice, about MDT working.
To understand ‘usual care’, similar non-MDT professionals attached to other local general practices would be surveyed and interviewed about how they integrate care for patients similar to those receiving MDT support. Each MDT’s meetings would be observed three times to understand how participants work to coordinate patient care. Where participating patients provided consent, their survey data would be linked with their routine service use data (Hospital Episode Statistics (HES), primary care, and social care).
Original protocol and revisions of MDT evaluation.
aAll changes were approved by the relevant research governance and ethics bodies.
bThere were delays in conducting some interviews and follow-up surveys due to the COVID-19 pandemic.
Methods
MDT evaluation time frame of key events, activities and decisions.
Notes. CAG = Confidentiality Advisory Group; CSU = Commissioning Support Unit; DARS = Data Access Request Service; HES = Hospital Episode Statistics; HRA = Health Research Authority; IRAS = Integrated Research Application System; NHSD = NHS Digital; NHS REC = National Health Service Research Ethics Committee; R&D = Research and Development; WP = Work Package.
Results
We encountered four main challenges in our research. Each is discussed below.
Challenge 1: Commissioner/funder research specification
Requirement to undertake comparative, counter-factual research
The research specification required an evaluation of the Pioneers’ impact on ‘users’ and carers’ experience and outcomes, on the overall quality of care and support, on the care provider market, and on the wider health and care system’.14(p13) The programme and its initiatives were to be subject to outcome and cost-effectiveness evaluation, involving a ‘comparative (or controlled) research design’14(p15) requiring a ‘highly selective and pragmatic approach.’14(p15) The research team was initially to work with DH to map the 25 Pioneers’ initiatives by building on Erens et al’s
10
qualitative evaluation of the first 14 Pioneers, scope methodological challenges, assess the feasibility of delivering an outcomes evaluation, and subsequently produce a more detailed evaluation proposal than originally submitted. The DH acknowledged that robust outcome evaluation would require: ‘a clearly defined intervention, target population group, and success criteria, as well as a sufficiently large cohort or sample of subjects to enable measurement of impact, and the availability of good controls to allow measurement of the ‘counterfactual’ (i.e. the impact of no intervention)'14(p15)
The DH also acknowledged that this would be challenging given the Pioneers’ diverse interventions, target populations, success criteria, and limited availability of suitable comparator sites. Pioneers were expected to co-operate with the national evaluation. 14
Arguing that it was neither feasible nor meaningful to undertake cost-effectiveness analysis at Pioneer level, we proposed conducting a number of cost-effectiveness analyses, with integral qualitative research, of specific initiatives targeted by Pioneers at sub-areas and/or sub-populations. It was clear to us at the outset that there was no appetite among the Pioneers for randomised controlled trials, and that some form of quasi-experimental design would be required. The other key consideration was the extent to which similar interventions could be evaluated across several Pioneers to produce generalisable learning.
Identifying a clearly defined intervention to evaluate
Having won the contract, our first challenge, therefore, was to identify a relatively clearly defined integration-related intervention, involving both health and social care staff, and of strategic and systemic significance, that could feasibly be studied using a comparative design involving a matched group of non-intervention patients. Scoping activities included interviewing 53 key individuals in 24 Pioneers (autumn 2015 to spring 2016) about their current integration plans and activities. A number of discrete integration activities were considered. Community-based MDTs were chosen because they were one of the Pioneers’ most widely reported integration initiatives,10,12 similar to other English integration pilots. 7 Their caseloads often include older people with chronic conditions, a priority for most Pioneers. 10 They bring together professionals from a range of health, care, and community and voluntary sector organisations to co-ordinate, and, in some cases, deliver care.
Our scoping work also suggested the existence of geographical areas within individual Pioneers that were not served by MDTs, allowing for the recruitment of matched local non-MDT patients. As MDTs can vary operationally and/or structurally, this suggested scope for also comparing outcomes and costs between different MDT models, subject to statistical power.
In proposing a mixed-method quasi-experimental design, we aimed to quantitatively assess the impact of MDT care on patient experience, health-related outcomes, service use and costs, while qualitative methods would allow for an in-depth understanding of patient, informal carer and staff experiences. We hoped that exploring contextual differences between Pioneers and their MDTs might explain any observed difference in quantitative outcomes.
Our proposal, submitted to DH (July 2016) following a year’s intensive scoping work, was approved. However, in reality, our proposed design proved extremely difficult to implement.
Challenge 2: Negotiating with, and recruiting, research sites
Recruiting sites to a multi-site evaluation generates many challenges, especially where multiple organisations’ and professional groups’ participation is required. These include: ascertaining that the intervention as implemented locally meets the national evaluation’s inclusion criteria; establishing capacity and willingness to participate in a manner that satisfies requirements to undertake independent, methodologically robust research (including helping recruit NHS patients where research teams cannot approach them directly); and understanding who the key stakeholders are, including ascertaining who has the authority to commit each organisation to participation.
Recruiting Pioneer sites
Recruiting, and negotiating with, individual sites proved time-consuming and resource-intensive, especially from autumn 2016 to late spring 2017. The evaluation was promoted at Pioneer events, evaluation workshops and multiple meetings with senior managers in Pioneer sites. Patient representatives had opportunities to feed into the design and methods. We specified a need to recruit Pioneers that had several community-based MDTs involving both health and social care professionals, and where it would be feasible to recruit a non-MDT matched patient group and obtain routine service use data to link with questionnaire data.
Senior managers were asked to facilitate access to: MDT staff, and their help in identifying and recruiting participants; GPs to help identify and recruit matched non-MDT patients; and managers with responsibility for data sharing arrangements on behalf of their organisations. A local principal investigator was required for research governance purposes.
Difficulties encountered negotiating with multiple stakeholders
Ten Pioneers expressed interest. However, multiple meetings revealed that most were ineligible. Their MDTs had not yet been implemented, only had short-term funding, included no social care staff and lacked administrative capacity to support research, or senior managers wanted our design and data collection adapted to suit local evaluation needs, making it impossible to compare findings across sites. Three Pioneers’ MDTs eventually met our inclusion criteria, had senior managers’ support and appeared to have capacity to participate. In those Pioneers, identifying the range of organisations involved in commissioning, funding or providing MDT care then proved challenging. CCGs, LAs and multiple NHS providers, data managers, MDT operational managers and frontline staff and, in some cases, community and voluntary sector organisations, were all required to ‘sign up’ to the evaluation.
Despite CCG directors at two Pioneers being willing to act as local principal investigators, they could not assume formal responsibility for agreeing to participation by all relevant services and organisations. Furthermore, CCG and LA managers’ views about local MDTs’ capacity to support the evaluation or about caseload numbers and throughput were not necessarily echoed by MDT operational managers, whose estimates tended to be considerably more conservative, making predicting patient recruitment rates difficult. Even within organisations, contradictory views emerged about which types of service use data could be accessed (with patient consent) and how datasets might be extracted. Well-intentioned MDT staff proposed helping frail patients to complete questionnaires, which would have broken ethical commitments not to identify participating patients to staff and would have compromised data quality.
Delays were incurred for many months in one Pioneer while senior managers decided which MDTs could participate, only for the chosen MDTs’ administrative support funding to be withdrawn just as we started negotiating with operational managers. With no alternatives proposed, we were only able to work with two Pioneers, increasing the number of MDTs studied in one to broaden the scope of the evaluation to a limited degree.
Challenge 3: Navigating research ethics and governance approvals systems
As our research involved NHS patients and staff, it required approval from a NHS Research Ethics Committee (REC) and the Health Research Authority (HRA) (through the Integrated Research Application System), our university, and local Research & Development (R&D) offices, which are often based in acute hospitals. R&D offices ascertain local organisations’ ‘capability and capacity’ to undertake the proposed research, including the research-related funds and resources available to support site participation (e.g. those generated by patient accruals). They scrutinise study documentation, including the specific activities to be undertaken locally by NHS staff, either in supporting research activities or as participants, and can request changes. Multi-site research is scrutinised by multiple R&D offices.
The multiple layers of bureaucracy involved in gaining approvals are challenging when working to tight time frames, with fixed budgets and in multiple sites. In addition, we encountered what might be considered unexpectedly high levels of risk aversion, especially surrounding patient confidentiality. For instance, despite services agreeing to take part in our study, they would not grant us on-site access to the minimum amount of information necessary to recruit patients directly. Likewise, a CSU with a data-sharing agreement with the local general practice federation had to obtain permission from individual GPs before contacting patients to invite them to participate in the study.
Navigating research ethics requirements regarding patient recruitment
As academic researchers, we could not access contact details for, or directly approach and recruit, NHS patients. Negotiating with sites on the basis of their capacity to send out baseline questionnaires was proving problematic. Therefore, in February 2017, we applied to the Confidentiality Advisory Group (CAG) (which advises the HRA about whether researchers can access patient identifiable information without patients’ prior consent in specific circumstances) for permission to obtain from MDTs and patients’ GPs the names, addresses and NHS numbers of MDT patients and potential matched patients. We proposed having honorary contracts with NHS organisations to allow access to data on site to recruit patients and obtain their consent to follow up and to link their survey data with their routine service use data. Our rationale was to ensure consistency across sites in recruitment processes and not to burden busy frontline staff. Our request was limited to the minimum information necessary to accurately identify the cohort members, prior to consent being obtained. However, our request was rejected on the grounds that honorary contracts would not make us part of the care team and that ‘Section 251 support under the COPI [Control of Patient Information] Regulations would be required in order to give a legal basis for this access to patient data’ (advice dated 21 February 2017). Asking care staff to send out surveys on our behalf was deemed a ‘practicable alternative’. It is not clear on what basis CAG made this judgement on behalf of individual Pioneers and their MDTs.
We could, therefore, only include sites where MDT administrative staff had capacity, on top of their usual work, to support the evaluation. Their role, following study-specific training, included mailing study invitations and baseline questionnaires to eligible patients. As they could not know for ethical reasons who had returned completed questionnaires to us, they had to send two reminders to each, increasing their workload and research costs, and risking annoying participants who had completed the questionnaire. MDT support staff also helped with contact details for the MDT staff survey and interviews as well as arranging MDT meeting observations. We developed excellent working relationships with these staff, but were conscious that the evaluation’s fate depended on their goodwill, with research inevitably having a lower priority than their other responsibilities.
Unable to go through CAG, we submitted our Integrated Research Application System application (February 2017). This application described our approach to recruiting MDT patients with the help of MDT staff and matched non-MDT patients from other local general practices, and how we would work with CSUs to access routine service use data to link with questionnaire data (with patient consent and pseudonymised service use data). Having clarified minor points for the HRA and NHS REC, we received formal NHS ethics approval (August 2017).
Requirement to establish the legality of our plans
However, the HRA posed two queries that effectively required us to establish both the legality of our plans for routine service use data access, extraction and linkage (despite having described the patient consent model) and that of the CSUs to access identifiable patient data to help us recruit matched non-MDT patients from general practices. We spent months working with the three Pioneers (prior to the loss of the third) to establish how best to access and extract individual-level general practice and social care data and link these in pseudonymised form with our patient survey data.
Local organisations were cautious about committing. Pioneer 1’s CCG and its local CSU worked to establish how the latter could support data linkage processes and whether it had the legal right to identify potential matched patients in their routine service data and recruit them on our behalf. Working with patient consent as the legal basis for data linkage, this CSU agreed with us, after some months, what they believed to be an appropriate form of wording for the ‘consent to link data’ patient information and consent sheet.
The legal basis for access to personal, identifiable data and health records by the CSU at Pioneer 1 to identify and recruit controls would be through a formal data sharing and processing agreement between the CSU and the local general practice federation. A similar data sharing arrangement would apply for social care data and the CSU would support us in applying to NHSD for HES data. Once matched non-MDT patients had completed consent forms and survey questionnaires, and returned them to the research team, their data linkage arrangements would be the same as for MDT patients. Although, ideally, all the participating sites would use the same process locally (necessitating multiple data sharing agreements), we acknowledged that other arrangements might need to be put in place in Pioneer 2. We received formal HRA approval in May 2018, over a year after submitting our original application.
Obtaining site-specific research governance approvals
We contacted the Pioneer 1 and two Pioneer 2 R&D offices and local Clinical Research Networks prior to receiving HRA approval, hoping to expedite local reviews. However, following HRA approval, an R&D manager requested a more formal version of our evaluation protocol before we could get local R&D approval. We also had to negotiate the issue of study accruals, defined by us as the number of participants successfully retained at 18 months follow up, but by the R&D offices as the number of patients consenting to participate at baseline.
Concern was expressed about local capacity to recruit the patient numbers required and in the planned time frame, and about the staff time costs. It was unclear to us who had the final say on local approval – MDT staff or R&D offices. In Pioneer 2, we negotiated financial support from the local Clinical Research Network to compensate MDTs for time spent helping with recruitment. Obtaining R&D approvals thus took several months. MDT patient recruitment started in Pioneer 1’s MDTs in October 2018, and in Pioneer 2 between May and August 2019. MDT administrative staff were given extensive training, processes piloted and study materials supplied. Having initially begun seeking formal approvals in February 2017, it was some 20 months before recruitment started. New data protection regulations (General Data Protection Regulation) were also introduced in the UK during the protracted period of seeking study approvals, which required submission of an amendment to the HRA, covering changes to our participant information sheets and study website.
Implications of the time spent recruiting sites and obtaining governance approvals
Even before recruitment started in Pioneer 2, and with an evaluation completion date of June 2020, the impacts of the time spent recruiting sites and obtaining formal approvals meant re-considering the feasibility of the planned volume and timing of fieldwork, including the number of follow-up points. We therefore submitted (March 2019) the first of three substantive amendments to NHS REC and the HRA, while at the same time requesting the first of a series of no-cost extensions to the study’s completion date. While the NHS REC and the HRA approved both substantive and more minor amendments quickly, each amendment also had to be notified to and approved by the local R&D offices, with further time implications.
Impact of COVID-19
When the COVID-19 pandemic-related national lockdown started in England in March 2020 non-COVID-19 research involving NHS staff was paused. However, with fieldwork underway, we were well placed to explore the impacts of COVID-19 on how MDTs were operating and on study participants. Our amendment request to the HRA was approved quickly. We modified methods and data collection tools to explore COVID-19’s impacts.
In May 2020, NIHR published a framework for restarting paused research, involving obtaining funder, research sponsor and site R&D approval, though R&D offices were especially busy fast-tracking COVID-related research. We obtained permission to re-start in Pioneer 2 in early July, but could not restart in Pioneer 1 until mid-September 2020 because of concerns about local capacity and the possibility of causing distress to relatives who might receive follow-up questionnaires sent to deceased participants. This added to the interval between baseline and follow-up surveys for some participants.
Challenge 4: Recruiting MDT and non-MDT patients and obtaining their routine service use date
Recruiting and retaining participants is a common challenge in prospective studies. We planned to recruit MDT patients for 6 months at each site, based on information originally received about MDT caseloads and throughput. However, we had to seek permission several times to extend the recruitment phase, and recruited fewer MDT patients than planned. The study time frame eventually did not allow for further recruitment.
Difficulties faced attempting to recruit matched non-MDT patients
Our major challenge was in recruiting matched non-MDT patients. To match at an individual level, and on a number of pre-defined characteristics, we needed to recruit and retain approximately three times as many non-MDT as MDT patients. We had worked with Pioneer 1’s CSU from the second half of 2017 to establish how non-MDT patients might be recruited through general practices. The CSU, through its access to general practice data and a data sharing agreement with the local general practice federation, and using data from us regarding the characteristics of our recruited MDT sample, would identify a pool of potential matches and send them baseline questionnaires, with those wishing to participate returning completed questionnaires to us.
After many meetings concerning data protection impact assessments, and data sharing and processing agreements, our university signed a Memorandum of Understanding with the CSU in 2019. This detailed the CSU’s role in recruiting non-MDT caseload patients, extracting general practice and social services use data for MDT and non-MDT patients and securely delivering these data to us in pseudonymised form, and in helping us to apply to NHSD for HES data through a Data Access Request Service application. However, by early 2020, it became apparent that unforeseen governance requirements meant that for each potential non-MDT patient they identified in GP records, the CSU would have to obtain the individual patient’s GP’s permission before inviting him/her to participate in the evaluation. The CSU had planned to mail out study invitations and baseline survey questionnaires to potential matches in large batches over a limited time period. Having to wait for individual GP approvals made this and recruiting matched non-MDT patients within the study time frame impossible, and would potentially have affected data quality, as GPs could choose which patients were invited to participate.
It also became extremely unlikely that we would be able to obtain social services or general practice records of service use for participating MDT patients within the study time frame. Reluctantly, we decided to construct a matched group, based on HES data, and to confine our service use-related outcomes to emergency and other hospital admissions contained in HES, with the cost-effectiveness analysis also being confined to costing hospital-related service use.
Obtaining service use data
Linking service use data to primary self-report questionnaire data was essential to the evaluation, both for measuring routine service use and comparing the costs of MDT and non-MDT care. However, primary, hospital and social care service use data are controlled by different organisations. Despite an ongoing drive in England to integrate individuals’ health and care records,15,16 linked records were not available at our sites.
As described, much effort went into negotiating the type of service use data that might be extracted from organisations’ databases and shared for research purposes. It was eventually agreed that Pioneer 1’s CSU would, through data sharing agreements, provide us with individual level pseudonymised general practice and adult social care data for study participants (based on participant consent) through a secure link to be stored on our institutional secure server, which met NHSD security standards, and linked by us with our survey data (using participant study ID codes). When, as described above, this became impossible, we focused on obtaining HES data. The CSU could not receive HES data for research purposes on our behalf or undertake data linkage for us, but assumed that it could send the NHS numbers and study ID codes of consenting MDT participants to NHSD so that their HES data might be extracted by NHSD and sent to us in pseudonymised form to link with our survey data.
As we were not permitted to have participants’ NHS numbers, we planned to send consenting patients’ study ID, name and date of birth to the CSU which would also receive from the MDT administrators (from both Pioneers) databases containing the study IDs, names, dates of birth and NHS numbers for all caseload patients sent a baseline questionnaire (one NHS organisation sending information to another) so that only the NHS number and study IDs of consenting patients would be sent to NHSD for data extraction purposes. However, in March 2021, we were informed that the CSU could not, in fact, pass NHS numbers of consented patients to NHSD through its Data Services for Commissioners Regional Office.
We then began the Data Access Request Service application process, outlining an approach by which the MDT administrators would send directly to NHSD the unique study number, NHS number and year of birth for all eligible MDT patients who had joined the caseload during the study recruitment period and who had been sent a study invitation baseline questionnaire. We would provide NHSD with the study numbers for those MDT patients who took part in the study and who had consented to data linkage. However, we were informed by email a few months into the application process that NHSD, for General Data Protection Regulation reasons, could not receive data from the MDT administrators which included personal information about unconsented individuals, but that we could make a CAG application to obtain it. We were asked to contact a sample of MDT patients to confirm that they would find it acceptable for us to send information about them directly to NHSD rather than to the CSU (as in the original consent form) which we did. Those contacted had no objection. We then investigated the possibility of sending names and dates of birth and unique study numbers for consenting MDT participants to NHSD in the hope that they could successfully be linked to NHS numbers and through these to their HES data, and submitted an application in March 2021. HES data were eventually received in May 2023 (by which time the research team had been disbanded), including data which would allow us to construct a comparison group.
Discussion
The National Audit Office 17 has signalled the importance of evaluating public policy initiatives in England. However, evaluating complex health and care systemic integration pilots and their constituent patient level interventions is never likely to be easy, not least because of the multiple stakeholders, differing local contexts, populations and interventions involved. NHS research that goes beyond clinical trials is very exposed to a health and care system under huge pressure, as was the case throughout the Pioneer evaluation even before the pandemic. Thus, helping researchers conduct an ambitious ‘national’ evaluation is likely to be a low priority for local managers and frontline staff, who may question whether it will generate local learning sufficient to offset the time required to assist, even where the site is voluntarily participating in a pilot programme.
We were fortunate in experiencing enthusiasm from senior managers at Pioneer sites, MDT staff who went out of their way to help and staff working for research regulatory bodies, who were individually helpful. Although unable to conduct the evaluation as originally planned, by pragmatically addressing the challenges faced, we successfully collected some data on patient outcomes and patient, informal carer and staff experiences, including during COVID-19. We were also able to observe MDT meetings. 18 Our attempt to evaluate MDTs was, though, significantly negatively affected by the mismatch between research commissioner expectations and what could be achieved in practice at local study sites, and by the layers of research and data governance regulation, which affected our ability to recruit participants and access their service use data. However, our experiences also provide a solid basis from which to argue for the types of changes needed to ensure the continued feasibility of complex evaluations of integrated care initiatives.
Setting realistic expectations for comparative research design
There exists a major mismatch between, on the one hand, the ambitions of evaluation commissioners (and researchers) to provide policy-relevant research evidence to improve system-wide health and social services integration to benefit patients, and, on the other, the suitability of the far from integrated local processes for securing engagement in research and of the multi-layered research and information governance systems to enable such research. Arguably this mismatch is particularly stark where the research specification, like that for the Pioneer evaluation, requires a comparative research design, but researchers face a lack of clarity over who can commit local sites to participating, who controls and can share routine individual level patient data necessary for outcome measurement (even with patient consent), and which regulatory organisation holds ultimate responsibility for approving an evaluation. This is in a context where, in principle, all the sites have committed to cooperating with a national evaluation.
One question arising is whether, while research governance and other arrangements remain as they are, it is realistic for evaluation briefs to stipulate that some form of comparative evaluation involving primary and secondary data is undertaken. Furthermore, there is no incentive for comparator areas or staff to take part. From a research feasibility and policy relevance perspective, it may be more important to ascertain whether over time local health and care systems are displaying the attributes associated with more ‘integrated’ care and/or to identify which approaches to horizontal integration are progressing better than others, rather than trying to treat the evaluation of a sector-wide policy direction as if it can be moulded into the shape of an intervention trial.
The case for closer working relationships
Kumpunen et al. 8 argue that integrated care evaluation commissioners need to be more realistic in terms of their expectations and evaluation timelines, and may benefit from working with those who have experience of evaluating integrated care when writing their specifications.
Notwithstanding the need for independent evaluation, closer working between research commissioners, the successful research team and local health and care system managers during the local initiation and implementation of large, complex evaluations would both initially inform and then keep commissioners and researchers abreast of what is feasible locally. It might also encourage more meaningful commitment to evaluation by successful bidders for national pilot programmes, as might the linking of any programme funding to a requirement to participate in research. This would not only ensure that all stakeholders share realistic expectations of what is achievable, but would potentially save researcher effort, time and resources at the start of complex evaluations, especially where initiatives that cross organisational boundaries are being evaluated and where, as in our evaluation, success in executing an evaluation depends to a large extent on the support of already stretched frontline NHS and local authority staff.
Early discussions involving researchers, pilot site stakeholders (including frontline staff), and, importantly, pilot programme commissioners and research commissioners (if different) might usefully confirm which of the pilot programme’s components can feasibly be evaluated across the sites, which organisations have the capacity to participate and the authority to commit others to participating, what routine data can be made available to researchers (with patient consent, on a ‘public interest’ basis, or anonymously), and who, apart from the locally designated principal investigator, holds local responsibility and accountability for the success of the evaluation and the site resources needed to participate fully.
Given the commissioning brief, the Pioneers evaluation was always going to be challenging. However, it was only as work began with sites to elucidate what was feasible that the full scale of the practical challenges emerged. Many of these could not have been foreseen and were beyond the research teams’ and local sites’ capacity to control or resolve. Our difficulties in simply trying to identify key stakeholders and negotiate participation with multiple organisations suggest a need for both the inclusion of fully funded scoping and feasibility stages for evaluations similar to ours, and for the review and revision to the relevant legal and administrative frameworks shaping local contexts for the conduct of this kind of research.
Streamlining research governance
While participant welfare, ethical, and research and information governance regulations are necessary, we encountered multiple, repetitive layers of bureaucracy. This meant it was some 20 months between our first applying for approvals and patient data collection starting. The longest delay was occasioned by having to establish the legality of our plans for extracting and linking routine service use data with our primary data, and recruiting matched patients, despite having had NHS Research Ethics Committee approval. It should not be the responsibility of researchers or participating organisations to take the legal risk. Ideally, comprehensive legal advice should be available from the national research regulatory body.
Delays occasioned by lengthy and arguably overly risk-averse approaches that prevent researchers themselves recruiting patients to non-interventional research may deter them from collecting data from patients, which would be detrimental to understanding the impact of integration efforts. Patient confidentiality is essential. But the privacy restrictions we encountered, mentioned above, seemed unduly cautious and over-regulated.
Multi-layered processes and risk aversion on the part of research regulators have implications for research resources, time frames and ultimately for whether research designed to benefit the NHS and patients can be conducted in a manner and a time frame that allows those benefits to be realised, keeps pace with policy change and informs policy development.
Such challenges are not unique to the MDT evaluation. Furthermore, they are ongoing. Our experiences align with the findings of a more recent online consultation survey 19 of UK health services researchers’ experiences of NHS research ethics, and research and information governance processes, which suggests that the challenges we faced have not been resolved. Respondents viewed the NHS research ethics system as disproportionate to the risks posed by individual projects, particularly in the case of non-interventional and health services research studies. They reported experiencing ‘fragmentation, duplication and inconsistency’19(p7) from R&D offices, which were viewed as ‘highly risk averse’,19(p7) frequently going beyond their remit by questioning research designs and methods that already had HRA approval. Respondents called for reduced duplication and form-filling.
The government itself has acknowledged concerns about increasing bureaucracy ‘which became particularly apparent for the R&D system during the pandemic, much of which has added limited value or in some cases led to negative behaviours or consequences.’20(p5) This situation is not unique to the UK. Fragmentation and gaps in research ethics and governance systems and processes have been reported in Ireland, 21 while researchers comparing research ethics and governance processes for non-interventional research across European countries report arbitrary differences in what is required. 22 This suggests possibilities for mutual learning for research governance bodies in different countries about what is essential to determining research-related risks. Our experiences, and those of others, suggest that further streamlining of ethical and governance bodies and processes is still needed.
Easier access to patients and their service use data
Our biggest challenges related to recruiting patients and obtaining (with consent) their routine service use data, raising questions about who should control access to patients and who controls their data. We encountered questions about the legality of our approaches to recruitment and data linkage, and were unable to recruit a matched group through general practices because of the stringent GP approvals involved.
The study involved protracted discussions at each site about which organisations held which types of data, and how data might be accessed, extracted and shared with us. We received conflicting information from the CSU and NHSD about which data the MDT administrators could send to NHSD to identify MDT patients and suitable control patients. Concerns about information governance have also been expressed by those health services researchers in the UK surveyed by Snooks et al., 19 including the bureaucracy, cost, delays and stress to researchers in obtaining data sets from NHSD, information governance teams in local organisations not understanding non-interventional research, and varying data sharing requirements.
Patient-centred care lies at the heart of integrated care programmes. It is crucial that evaluations explore not just staff perspectives on integrating care but also assess the impacts on patients and their informal carers, through analysis of both self-report and routine service use data, ideally linked. If researchers experience too many impediments, the risk is that the patient voice gets lost and policy becomes informed only by professionals.
Even the principal investigator of the major, high profile UK Biobank study, involving half a million UK citizens, had to resort to asking participants in 2021 not to opt out of the government’s General Practice Data for Planning and Research initiative to add general practice records to those held by NHSD. 23 Despite having their consent when entering the Biobank, the study had not been able to access participants’ records (the exception being for COVID-related research, based on UK COVID-19 emergency legislation) because of GPs’ control over and reluctance to release the data. Similar problems are faced by locally initiated evaluations. Wilson et al, 24 reviewing the evidence on the quality of the locally commissioned evaluations of the New Care Models Vanguards, noted that, despite evaluators’ best efforts, data-sharing agreements and information governance procedures were difficult to negotiate, and significant barriers existed to the conduct of outcome analyses. This resulted in an inability to conduct planned quantitative analyses.
This issue urgently needs to be addressed by central government. If efforts to better coordinate care and evaluate them are to be efficient and effective, greater priority must be attached to investment in linked patient records and data sets, and easier access for research purposes (particularly where patient consent is being used as the legal basis for accessing and linking datasets).
Some of the challenges we encountered regarding access (with consent) to patient routine data mirror system weaknesses the Pioneers were designed to address. Our observations of MDT meetings, 18 for example, revealed the challenges faced when staff do not have access to shared patient records but, instead, are working from different systems. As such, they might be seen as empirical evidence of the barriers to integrated care as well as to research on integrated care. More positively, Goldacre and Morley 25 have set out recommendations about how health data might be accessed and analysed securely. Moreover, the Department of Health and Social 26 has revised its policy on health and care data, following the Goldacre review and the generally very positive experience of data sharing under special measures during the COVID-19 pandemic. But changes have yet to be implemented.
Limitation
There is one main limitation with our research: the difficulties we encountered may not be fully generalisable. The challenges we faced were especially complex as we were studying an integration programme where sites were developing their own integration initiatives. Moreover, we were recruiting and studying a specific integration initiative, MDTs, which involved multiple organisations/services, staff, patients and carers. Researchers conducting evaluations in less dynamic and complex environments may not experience the same high degree of difficulty.
Conclusions
Seasoned researchers might expect some negotiation with commissioners and funders regarding what can realistically be achieved in evaluations of complex health and social care interventions and the likelihood of experiencing a number of challenges in recruiting study sites and individual patients. Yet, our experiences suggest that changes are needed both to how the research-preparedness of participating study sites is facilitated, including through greater awareness and involvement by research commissioners in establishing what is feasibly evaluable at local sites, and to the regulation of policy research to enable more robust evaluations. Current arrangements involving multiple layers of approvals, including clarifying which agency ultimately holds final approving powers, require re-thinking, as do disproportionately high levels of risk aversion in approval processes for non-interventional, low-risk studies.
While much of the regulatory machinery operates at the level of individual organisations and professions, increasingly the research required to improve integration has to be undertaken across organisations and/or at local and regional system levels. This underscores the need for national, standardised and clear guidance, for example, on data sharing. There also needs to be streamlining of the various regulatory bodies involved and the practical processes required in obtaining approvals for research, both at local sites from local services and organisations, and from national and local research governance bodies.
Footnotes
Acknowledgments
We wish to thank all of those - patients, informal carers and staff - who participated in the MDT evaluation, and staff at the Pioneer sites who helped to facilitate its implementation.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is an independent evaluation funded by the NIHR Policy Research Programme (Evaluation of the Integrated Care and Support Pioneers Programme in the context of new funding arrangements for integrated care in England (2015–2022), PR-R10 -1014–25001). NIHR played no role in the conduct of the research or preparation of this manuscript. The views expressed are those of the authors and are not necessarily those of the NIHR or the Department of Health and Social Care.
Ethical statement
Data availability statement
The datasets generated and analysed during the current study are not publicly available because they contain information that would identify the research sites, individuals, and/or case-material and it would not be feasible to redact or otherwise anonymise them. The data custodian is Professor Nicholas Mays, London School of Hygiene & Tropical Medicine.