Evaluation of public involvement in research: time for a major re-think?

Abstract

The way that public involvement in research has been evaluated as a complex intervention has derailed the development of an evidence base. Two alternative approaches are available for constructing and evaluating patient involvement, each of which requires us to revisit the purposes and values that underpin it in each stage of the research process.

Background

Public involvement (PI) in research has been defined as ‘research being carried out with or by members of the public rather than to, about or for them as mere research participants’. ¹ PI in research often refers to consultation or collaboration with lay people on activities such as choosing which outcomes to measure, designing recruitment materials and presenting findings.

Increasingly, PI is mandated by many institutions involved in health and social care research, dovetailing with movements towards PI in health care and the patient as consumer. ² Research funders in the UK commonly require applicants to detail how PI has, and will continue to, inform their proposed research.^3,4

Difficulties in evaluating the benefits of public involvement

Considerable guidance is available^5–7 on how to undertake PI, largely based on grey literature and qualitative research. This has identified important issues around tokenism and the well-being of lay people in research environments. ⁸ More problematic has been the development of an evidence base of the impact of PI on research, fuelled by rhetoric such as:‘ … active involvement of the public in the research process leads to research that is more relevant to people and is more likely to be used’. ⁹

Such statements present a consequentialist rationale for PI, ¹⁰ one in which PI is endorsed as morally right based on its beneficial consequences in improving the quality and relevance of research. This rationale has been presented as instrumental, ¹¹ pragmatic, ¹² and the methodological argument.^2,13,14

Reviews of empirical work do suggest PI can improve the quality, relevance and ethical conduct of research.^6,15–17 Nonetheless, the quest for evidence of such benefit has been difficult, prompting suggestions that academics have failed in this regard. ¹⁸

We propose four key reasons for these difficulties. Firstly, the entire consequentialist rationale for PI in research is contested by those proposing a deontological rationale (i.e. giving rightful voice and power to those at the heart of health care). This rationale has been termed ideological, ¹² moralistic ¹⁹ and the moral and ethical perspective,^2,13 its position summed up as: ‘defining consumer involvement outcomes solely in terms of research quality ignores the rights of those being researched or likely to benefit from the research’. ¹³

Secondly, there is contention about what types of impact such as improved study recruitment rates are worth investigating for whom, when and why. ²⁰ Similarly, disagreement persists about which indicators and methods can and should be used, with concerns about weak study designs. ²¹ But most importantly, underpinning these disagreements is a trend towards evaluating PI in the same manner as one would evaluate therapeutic interventions such as surgical procedures, drugs or screening programmes.

Evaluating public involvement as if it were a therapeutic intervention

Much clinical and health services research concerns the evaluation of therapeutic interventions and employs a variety of research methods and processes to do so. PI is a component of the research process, i.e. the activities of research funding, design, conduct, analysis and dissemination. Yet instead of evaluating PI as part of the research process it is commonly evaluated as if it were itself a therapeutic intervention. This approach, which tends towards defining and evaluating PI as a complex intervention^22,23 or complex social intervention, ⁷ is the reason why evaluation has proved difficult.

Complex interventions comprise two or more active components (e.g. medication plus adherence counselling). Their evaluation often includes process evaluation, comparison of outcomes between those receiving and not receiving the intervention, and realist evaluation. The latter emphasizes the context of delivery and has been recommended and used for PI in studies in the UK funded by the National Institute of Health Research with INVOLVE^13,24 and the Medical Research Council.^25–27

This trend has arisen because of the complexity and importance of context regarding PI in research. ¹³ However, this has also created a misconstruction of PI in research, using unsuitable methods such as realist evaluation. ²⁴ This has derailed the development of a meaningful and robust evidence base. In place of evaluating (and hence implicitly constructing) PI as if it were a therapeutic intervention, alternative constructions of PI are needed from which approaches to evaluation will then follow.

Alternative constructions of public involvement in research

If we reject the construction of PI as an intervention and recognize it as part of the research process, what alternative constructions exist? From a deontological perspective we might construct PI as a contribution of expertise and advocacy, equitable to the contribution of clinicians, statisticians or others. Therefore, we might focus evaluation on the processes by which: the purposes of PI in a given scenario can be made explicit; lay knowledge can be brought forth and integrated with other expert knowledge; and advocacy and accountability for and by all parties can best be achieved. (The notion of advocacy also lies at the heart of controversy about training lay people in research methods, ²⁸ in which the implicit concern is whether one can advocate for lay people having adopted the interests of the researcher through training.)

Alternatively, we might adopt a consequentialist rationale, constructing PI in research as a methodological activity to improve research quality and drawing inspiration from evaluation of research methods. Such evaluation usually comprises critique of their congruence with stated purpose and theory. Crucially, research methods are not tested against pre-determined outcomes in the manner that we would evaluate a complex intervention, and in which PI is commonly evaluated.

Conclusions

Theorization of PI has tended towards an internal focus, ²⁹ categorizing levels of power and responsibility. However, we also need to theorize the nature and purposes of PI in relation to the overarching aims and processes of health research. Different purposes and constructions of PI may suit different stages of the research process. For example, the purpose of lay representation on a generic research funding panel may be public accountability – best served by a deontological construction of PI as public advocacy. In contrast, the purpose of lay involvement in study design may be to improve the quality of the research regarding such aspects as recruitment, attrition and patient-centred outcomes. This may be best served by a consequentialist construction of PI as a methodological activity.

PI in research cannot – and should not – be evaluated as an intervention but rather as part of the research process. Alternative constructions of PI, that acknowledge this, are possible. Chiming with recent calls for the development of common goals for PI in research,^18,29 we must revisit its values and purposes. This would provide the foundations for identifying constructions that allow meaningful evaluation of those intentions and by appropriate means.