Abstract
Co-produced research is said to create new knowledge through including the perspectives of those traditionally excluded from knowledge production, which in turn is expected to enhance research quality and impact. This article critically examines academic and UK voluntary sector literature concerning participatory and co-produced approaches to explore how quality is currently understood in co-produced research. Drawing on early career researchers’ experiences of a programme of co-produced research, the authors illustrate how theory and practice of co-production can differ, and the implications for conceptualising ‘research quality’ within co-produced research. Drawing on debates within qualitative research, community work and policy studies, the article outlines a potential framework for raising questions of ‘quality’, co-produced by research partners as part of the research process. Key dimensions of this framework are process, outcomes and autonomy.
Keywords
Introduction
Co-produced research involves collaboration between partners in all stages of a research process (Facer and Enright, 2016), consciously situating individuals and communities as partners rather than as research subjects (Armstrong and Alsop, 2010; Pearce, 2008). Beyond this, there is less consensus about what co-production should look like. Such ambiguity is challenging because the concept of co-production can be subject to diversion, distraction and ‘conceptual stretching’ (Flinders et al., 2015: 263; Innes et al., 2019: 15), continually re-imagined to meet the beliefs, needs and wants of those involved in individual research projects (Thomas-Hughes, 2018). Hence, co-production becomes a ‘dirty’ concept, in moving from idea to practice becomes tarnished and distanced from its ‘pure’ conceptual origin (Innes et al., 2019: 16). Although the ideal of co-production as a tool for democracy and equality has been well-articulated (Bell and Pahl, 2018), there has been less emphasis on methodology and how this relates to the concept of ‘quality’ in co-produced research. This is important because there are multiple understandings of what co-production is or should be, informed by different disciplinary and theoretical influences and varying participatory, community-based and collaborative research practices (Bell and Pahl, 2018; Facer and Enright, 2016; Horner, 2016), and challenging because co-production is dependent on, and produced through, human relationships (Campbell et al., 2016).
Early career perspectives
Drawing on our reflections as early career researchers (ECRs) and contemporary UK literature on collaborative research, this article examines how ‘quality’ is conceptualised in co-produced research.
ECRs often have unique insights into co-produced research, from undertaking most of the practical, administrative and relational aspects of co-production (Oliver et al., 2019), and generally being ‘closer to the lived experience of collaborative and interdisciplinary research’ with more intensive involvement with community partners and participants than senior academics (Enright and Facer, 2017).
The impetus for this article is the shared experience of the authors as two of the six ECRs working on the co-produced ‘Productive Margins: Regulating for Engagement’ (PM) research programme. Funded by the UK Research Councils’ Connected Communities programme for interdisciplinary and collaborative research between academics and different ‘communities’, PM was a partnership between multi-disciplinary academics from the universities of Bristol and Cardiff and seven diverse neighbourhood-based, identity-based and faith-based community organisations, exploring how co-production might enable re-imagining of regulatory practices (McDermont et al., 2020).
The six PM ECRs worked at a programme and project level, some employed by the community organisations on behalf of the programme, others by a university. One ECR worked on both a project and across the programme; another worked solely on the programme; the others worked on one project each. With previous experience including academic research, community work, campaigns and government, the experience of working on PM generated shared reflections about the quality of co-produced research.
Data used in this article are from individual ECR interviews undertaken by the two programme ECRs and two ECR focus groups. Quotations used are from the individual ECR interviews.
In this article, we consider how research quality could be conceptualised in co-produced research, drawing on the authors’ reflections and ECRs’ perspectives. In the next section, we explore the complexity of co-production and how it is often reported; we then discuss the relationship between co-production and notions of research ‘quality’, before examining the ECRs’ perspectives on specific themes relating to quality. The final sections outline a speculative framework that future co-productive research teams might use to co-design mechanisms for assessing what ‘research quality’ might mean in the context of their project.
The complexity of co-production and a focus on relationships
‘Co-production’ can describe research between practitioners, policy makers and service users (aiming to improve public services), and research between academics, community organisations and residents (aiming to create new knowledge). Both are claimed to generate academic insights and public benefit (Campbell et al., 2016). Co-producing public services is argued to produce higher quality, more effective services (Boyle et al., 2010, Boyle and Harris, 2009; Pestoff, 2012). Co-producing research between universities and community partners is considered to produce more socially relevant findings through gaining access to ‘authentic’ community knowledge (Goodson and Phillimore, 2012: 8).
Recent years have seen a renewed interest in co-productive methodologies in the UK (Bell and Pahl, 2018; Durose and Richardson, 2016), as part of a wider move towards participatory practices in areas such as the arts, industry and government (Facer and Enright, 2016). This is driven by the potential for aligning new knowledge production with the expectations, needs and values of a diverse society (Xavier Grau et al., 2017), more representation of marginalised groups in research with increased opportunities for local empowerment (Guta et al., 2013), and a response to the demands for research ‘with, not on’ (Bell and Pahl, 2018: 107).
Greater democratisation of research has also been advocated in feminist, de-colonising and indigenous research (summarised in Edwards and Brannelly, 2017); collaborative community-university partnerships (Tremblay and Hall, 2014) and participatory evaluation (involving stakeholders in the evaluation process (Guijt, 2014), including co-produced research (Eisenstadt, 2015)).
A related challenge for co-production is that it can also be a ‘buzzword’ for neoliberalism (Horner, 2016), invoking the language of transformation and social change whilst obscuring power inequalities (Bell and Pahl, 2018: 107/8; Brigstocke and Noorani, n.d.; Horner, 2016). Research outputs can also be co-opted to serve the political agendas of more powerful groups (Oliver et al., 2019). Hence, co-production and other participatory approaches ‘cannot be as straightforwardly positive as much of the literature would like us to think’ (Kara, 2017: 299).
Reporting on co-production
Claims of co-production’s effectiveness and impact cannot be readily substantiated (Flinders et al., 2016: 262). In public services, reporting has tended to emphasise the quality of relationships in co-production rather than knowledge production or public-service-related outcomes (Durose et al., 2014); evaluations tend to focus more on process than impact (Social Care Institute for Excellence, 2013). For example, a systematic review of co-production in public services found that it was impossible to say if or how co-production had contributed to improvements, with meeting research outcomes or objectives rarely discussed in the literature because ‘co-creation/co-production is perceived as a value in itself’ (Voorberg et al., 2015: 1349/50). Similarly, collaborative and co-produced research reports are often noted for their lack of critical engagement in the appropriateness, effectiveness and impact of the methodologies (Martikke et al., 2015; Nind, 2011), instead taking a celebratory tone, emphasising (assumed) benefits (Flinders et al., 2016; Nind, 2014) while under-reporting conflict (Nind, 2014). Whether co-production increases empowerment is an empirical question, not something intrinsic to co-production (Farr, 2018).
Reflexive accounting is often missing from reports, despite reflexivity being an important foundation for co-production (Banks and Armstrong, 2014; Durose et al., 2012) and essential to cycles of action and reflection (Brydon-Miller et al., 2003) in action-research. Reflexivity has many meanings but involves researchers questioning their assumptions and relationships with participants (Orr and Bennett, 2009). Researchers’ positionality can affect knowledge generation, whether as ‘insiders or outsiders’ in relation to participants (Berger, 2015) or as ECRs (Enright and Facer, 2017). Collaborative reflexivity is a process that – is argued – can generate new meanings (Popa et al., 2015) and ensure that the standpoints and biases of co-producers are considered when analysing data and drawing conclusions (Goodson and Phillimore, 2012: 12). Hence, reflexivity should be embedded within the research process (Caretta, 2015).
The closeness of researchers to community partners, the emotional and relational nature of co-production, and challenges of time and money, may make it difficult to make time for, and to reflect critically but empathically on, the effectiveness and impact of co-production as a research process (Flinders et al., 2016; Heron, 1996). This makes it more difficult for there to be debate and dialogue about the principles, practice and quality of such research.
Notions of research quality in co-production
Generally, research ‘quality’ has been conceptualised in various ways, often relating to different research paradigms (Denzin and Lincoln, 2013a). This has generated decades of debate about what constitutes ‘good quality’ research (such as whether criteria for quantitative research could, or should, also apply to qualitative work (e.g. Denzin and Lincoln, 2003) or whether alternative principles should apply (Spencer et al., 2003; Seale, 1999).
Whilst different notions of quality criteria have generated much debate in qualitative research, these are less discussed in co-produced research, as co-production is often assumed to produce better ‘quality’ research. Co-production is said to be better quality because: it is more likely to present an authentic picture of the lived experience of community members (Devotta et al., 2016: 664); it can introduce issues and themes that may have been overlooked by professional researchers without that lived experience, thereby increasing the validity of research (Goodson and Phillimore, 2012: 11) and resulting in relevant, impactful research (Armstrong and Alsop, 2010; Durose et al., 2012). In medicine and public health, community-based co-produced approaches have been said to improve the quality of research data (Leung et al., 2004; Voorberg et al., 2015). Yet there is little evidence of how co-production changes research, policy or practice, or compares to alternatives (Oliver et al., 2019).
Furthermore, ‘quality’ in relation to co-produced research tends to be framed as the ‘quality of relationships between academic and non-academic co-producers’ rather than specifics such as research questions (Bell and Pahl, 2018: 109; Campbell et al., 2016: 38), so is described as ‘unavoidabl[y] morally charged’ (Nind et al., 2012: 651).
Literature addressing quality in co-produced research implies that indicators of quality relate to transparency in decision-making; perceived usefulness by community partners; potential for promoting social change; and consistency of new knowledge across project partners.
Facer et al. (2016) note that a good indicator of quality is that a collaborative project can reflexively locate itself within its contextual landscape and identify its choices in relation to accountability, decision-making, goals and methods. Quality in action-research has also been seen as transparency about choices at each stage of the inquiry (Reason, 2006). Others have referred to ‘the credibility / validity of knowledge derived by the process according to whether the resulting action solves problems for the people involved and increases community self-determination’ (Kindon et al., 2010: 14), suggesting that involvement in co-produced research that addresses their concerns can generate a greater sense of community control. This is echoed in the view that ‘criteria’ for good quality research (such as validity and credibility) can be measured by the willingness of local stakeholders to act on its findings (Denzin and Lincoln, 2013b), validity referring to what the particular community regards as ‘real’ and ‘useful’, arising from agreement between partners through reasoning and dialogue (Lincoln et al., 2013).
Other writers refer to ‘quality’ as the extent to which co-production is an ideological practice with the potential to inspire transformative social change in communities (Banks and Armstrong, 2014; Heron and Reason, 2008). Through bringing forward ‘previously hidden or marginalised perspectives and experiences’ (Beebeejaun et al., 2015), research is said to unlock key (but currently unknown) aspects of knowledge crucial to effecting social change (Bell and Pahl, 2018; Facer and Enright, 2016).
Finally, some writers argue that quality indicators relate to the knowledge produced. For example, criteria such as validity can be conceptualised as the extent to which researchers’ claims about knowledge correspond to participants’ (Cho and Trent, 2006); the extent to which findings are recognisable to the people involved (Nind and Vinha, 2014); incorporate ‘insider/outsider’ knowledge; and are consistent with other knowledges, reflecting a synergy between different types of knowing (Ostrom, 1996). Another dilemma is that community partners may dislike research outcomes, raising questions about who owns the data.
Having outlined the complexity of co-production and the dilemmas in conceptualising what good quality co-produced research might look like, we now turn to the perspectives of ECRs.
Early career researchers in co-production
ECRs are often uniquely immersed in the ‘messy realities’ and ‘lived experience’ of collaborative, co-produced and participatory research (Enright and Facer, 2017: 621) and, as such, have a perspective that can offer distinct methodological insights.
In this article, we use data from four reflective interviews and two focus groups with the six PM ECRs, including the authors, to illustrate how ‘quality’ was understood and enacted in the co-produced context. 1 ECRs were accountable to the university, the community organisation they worked in, and individual community participants.
Each project was allocated the equivalent of one full-time ECR. Reflecting the academic ‘culture of temporary project-based employment’ (McAlpine, 2010), with ECRs ‘at the forefront of the economics of austerity within the academy’ (Enright and Facer, 2017: 621), contracts were fixed-term. The uncertain nature of this precarious employment contradicts building the ‘authentic’ (Cook, 2009: 283), long-term, relational, emotionally-laboured (and sometimes painful) relationships (Facer and Enright, 2016: 72 and 147/8) at the heart of co-productive partnerships (Campbell et al., 2016). As one ECR stated: one-year projects ethically are quite tenuous in those settings because you have to work so hard to get people involved and to say really it’s going to be worth you being involved in this project and at the same time to really manage expectations and say “also funding stops in like six months and then there’s gonna be nothing, potentially”.
ECRs in co-produced research are ‘often the only people with a significant proportion of their workload dedicated to the research project’ (Enright and Facer, 2017: 630), required to ‘perform at a high level of intellectual autonomy’ in areas in which no other member of the research team had any expertise’ (Enright and Facer, 2017: 626), while feeling responsible for (metaphorically) ‘steer[ing] the ship’ (Enright and Facer, 2017: 630). Many regularly worked significantly beyond their contracted hours to meet ‘dual academic and community expectations’ (Enright and Facer, 2017: 630), consequently experiencing acute professional and emotional risks (Enright and Facer, 2017; Felt et al., 2013; Oliver et al., 2019).
PM ECRs reported feeling that they were ‘shouldering the project’, and although having their own perspectives on co-production, often had little influence over decision-making: ‘I felt like a huge amount of pressure to you know deliver this project in a sense without having decision-making powers, which I didn’t’.
Overlapping quality debates in research co-production and community development
Reflections from the PM ECRs highlighted similarities in claims for, and critiques of, co-produced research and community work. Both community development and community research have a commitment to working with communities (Mayo et al., 2013). Community development literature highlights difficulties in attributing changes to community interventions, as it often involves working with other partners, dealing with complex, fluctuating circumstances of community members (Gilchrist and Taylor, 2016). An overview of voluntary-sector impact studies highlighted that community development work and intervention can assume impacts, and that third sector involvement is always beneficial, arguing that more rigorous studies are needed before firm conclusions can be drawn about the impacts of specific interventions (Kamerade, 2015). In common with co-produced research reports, some community project accounts may tell a story of ‘how we did this empowering project and how much participants liked it’ not ‘how people in this locality or network became more empowered’ (Chanan, 2009: 4). This is despite the well-used argument that, to be taken seriously, community development work ‘must be able to show what difference it makes for communities and how this delivers desired policy and social outcomes’ (Gilchrist and Taylor, 2016: 149). One ECR stated, I guess one of the things is whether quality is what’s demonstrated through having outputs. . . . So is it implicit that if you have something that looks nice, that people can engage with, that therefore the process has led you to produce something, so it must be good because that’s the output.
It has been argued that research with communities differs from traditional research (where universities undertake research on communities, rather than with them: Campbell et al., 2016). Some contend that co-produced research is not primarily research at all: ‘It is a community organising and/or development project of which the research is only one piece’ (Stoecker, 1999: 845). Some ECRs believed that they were undertaking community development, rather than research, as their project was not framed as ‘research’: . .[x] say(s) it’s been a research project but . . that word research, I wasn’t allowed to use it . . . Months and months and then suddenly, and I said, “Oh of course people don’t feel like they’re on a research project, we are not allowed to use the word research”.
Hence if a project is community development rather than research, notions of ‘research quality criteria’ are likely to be absent. Some ECRs felt that while there were overlaps between community development and research, each could be relevant at distinct stages of a project: you were building a group and you needed the local community knowledge. . .community development stuff. . . whereas when you are kind of up and running you are then starting to look at OK, this is more research-type knowledge that I need. . .You know, we are now talking about ethics, or we are talking about research. . . .that needs to come from the University. . . .I think the knowledge that is being drawn upon is really dependent upon the particular part of the programme you are at, and the questions that people have of you as well.
Other co-produced research has found that different phases of a project may have more or less co-production; for instance, ‘analytic’ stages (such as data analysis and writing-up) remain the remit of the academics (Bain and Payne, 2016), despite being central to knowledge construction (Nind, 2011: 354).
Power, expertise and quality in co-production
As noted above, Facer et al. (2016) suggest that being able to articulate choices in relation to decision-making and accountability can be an indicator of research quality. In some projects, ECRs found that power dynamics between project partners did not appear to have been sufficiently worked through for a shared vision and objectives to emerge. Some ECRs reflected that there was a tendency for some academics to take an intellectual ‘step back’ in group discussion, perhaps for fear of reinforcing traditional power imbalances between university and community. Therefore, in some projects, ‘significant opportunities’ were created for ‘dominant interests to interfere with and shape both research projects and the knowledge they produce’ (Thomas-Hughes, 2018: 232), like Freeman’s observations of the ‘tyranny of structurelessness’ (1973). One ECR noted that there were few overt disagreements within their project’s working group (i.e. where academics and community organisation representatives decide on project activities), but this was because one or two powerful individuals took decisions outside of the group, which were then presented back to meetings as a ‘done deal’; different views about co-production, raised outside of meetings, were often not expressed in the group, creating confusion and tension. Hence, some of the basics of promoting research quality did not seem to be in place. Several ECRs reported an ongoing lack of cohesion around the ‘big picture’ of what co-production meant within projects, some feeling that there had been a concentration on the minutiae of research activities to avoid more difficult methodological and ideological discussions.
Some ECRs reflected that adopting certain ways of working (such as sharing around tasks like chairing, taking notes) could give the appearance of co-production but often served to ‘perform’ co-production rather than shifting power. ‘Performing’ co-production could hinder discussions about what quality, success or indeed failure might look like in individual projects: ‘when you’re using resources and there’s. . .objectives you’re not going to go along and say “Look, this was a total let down”.’
Attempts by individuals or organisations to exercise power were perceived to relate directly to conflicts within projects and misunderstandings about project’s purposes, aims and foci, and could overshadow debates about project or research quality. Balancing a coherent project vision and the needs and wishes of individuals and organisations within a project could be challenging. ECRs were also acutely aware of the environment of austerity in which community organisations were operating and often felt that ‘power plays’ were understandable as part of wider drives to secure funding for core organisational services or gain recognition of organisational expertise.
Some ECRs experienced powerful community organisation ‘gatekeepers’ selecting project participants, often appearing to rely on ‘usual suspects’ and so potentially excluding others: ‘Partly as a result of historically people working or being involved in different organisations, but also egos, definitely egos within the community and who . . .gets heard’. This was felt to have implications for research quality, as deliberately selecting one or more individuals as representative of a group assumes there is a single homogeneous viewpoint (Devotta et al., 2016), implying shared identities and common experiences. It risks privileging one perspective or experience over another (Atfield et al., 2012) and masking pre-existing proximity (including emotional ties and relationships) between researchers and community partners (Flinders et al., 2016).
From these experiences, we have reflected on the nature of the ‘better quality’ of co-produced research and considered how quality might be conceptualised to be useful to future co-productive research projects. As noted above, a key question is whether the priority in co-production is the quality of the participation or the quality of the research (Freeman and Mathison, 2009, also Nind, 2011: 351), some commentators suggesting that there could be trade-offs between involvement and validity (Brown et al., 2012). We would argue that these do not need to be in competition, but, like the ‘Achieving Better Community Development (ABCD)’ framework (for understanding, planning, evaluating and learning from community development, rooted in community work principles of participation (Barr and Hashagen, 2002) and participatory evaluation), can seek to meaningfully integrate the two (see also Clark et al., 2012).
The need for a ‘quality framework’
Though co-production is inherently complex and ‘messy’ (Thomas-Hughes, 2018) and a ‘traditional’ emphasis on specifying research questions and methods is less important in co-produced research (Campbell et al., 2016), we argue that, as well as the relational quality of the process, conceptualising quality in the practice, findings and outputs of co-produced research can help to create frameworks that enable ‘co-productive arrangements to be sustained’ (Innes et al., 2019: 20). Thus far, little has been done on developing theories and methods for understanding and assessing the value and quality of co-produced research, and whether it delivers on its claims for equality, democracy and equity (Facer and Pahl, 2017).
In relation to co-operative inquiry, four kinds of knowledge have been identified: experiential (gained through direct encounters with people, places or things); practical (gained through practice and knowing ‘how to’ do something); propositional (knowing about something, as expressed in statements and theories); and presentational (informing how experiential knowledge is ordered: Heron, 1996; Heron and Reason, 2006). Participatory research findings can thus be ‘valid’ (or of ‘good’ quality) when the four knowledge types are coherent and consistent with each other, and co-producers broadly agree. Each knowledge type could be regarded as valid according to its own internal criteria – for example, practical knowledge might refer to a particular skill and knowing how to perform it in different contexts. Procedures that can apply these validity criteria include challenging uncritical subjectivity, making time for reflection, and reflection and action cycles. Co-operative inquiry cycles between four phases of reflection and action, beginning with propositional knowing, then practical knowledge, followed by experiential knowing, and finally propositional knowing (Reason, 1994). These phases can be repeated several times so that ideas from the early stages can be checked and developed. This framework, also applied in more recent research (Rajagopalan and Midgley, 2015), gives a helpful starting point for considering how to conceptualise quality in co-production and, importantly, includes the idea of ‘reflection and action’ cycles within the research process. We believe that co-productive methodology can benefit from greater consideration of validity and reflexivity in this way.
Drawing on our reflections as ECRs and ideas from different bodies of literature, we have developed a framework of questions for co-produced research projects to utilise when considering research quality. These questions cover three different dimensions (process, outcomes, autonomy), following Alkire et al. (2009). These dimensions reflect important elements of co-production; as noted above, process is often well covered in reports of co-production, outcomes less so (and unintended outcomes are also important, as by closer involvement and deploying different kinds of knowledge, co-production can generate surprises). Consistent with the literature on community development and evaluation, we argue that any quality framework should be appropriate to co-production, with evaluation and quality standards developed with all partners and built into projects from the outset (Clark et al., 2012; Morse et al., 2002). This is in line with Facer and Pahl’s (2017) suggestion that co-produced research projects need to identify a ‘theory of change’ that can be articulated as a long-term goal, with intermediate steps to achieve it (e.g. Rogers, 2014). We hope that this article also contributes to theoretical discussions through building on such literature.
Recommendations for developing a ‘quality framework’
Approaches from different fields offer ideas for developing a framework. From community development, the ABCD approach covers process, outcome, and community empowerment (Barr and Hashagen, 2002); evaluation toolkits (Taylor et al., 2005; UNESCO, 2009), recommending participative monitoring and evaluation as integral to participatory development (Estrella, 2000). From policy studies, McConnell (2010) developed a ‘policy success’ approach focusing on three dimensions – process, programme and political. The Equality and Human Rights Commission (EHRC) equalities measurement framework (Alkire et al., 2009), developed from Sen’s (2001) capabilities approach and international human rights frameworks, includes autonomy (empowerment, choice and control). As empowerment and equality are values reflected in community development and co-produced research, we believe there may be potential to learn from these approaches.
Distinguishing between process and outcome is important in considering the quality of co-produced research (Voorberg et al., 2015); both are important but neither alone is enough. As Campbell et al. (2016) discuss, co-production is a qualitatively different form of research and, as such, quality criteria should differ too. They suggest that features to be considered when assessing funding proposals for collaborative and co-produced research should include: a two-phased approach (to allow for learning prior to a full bid); a focus on partnerships not projects; clarity of what is at stake; recognition of ‘translation’ capabilities (so that insights can support change); facilitation; understanding of managing flexibility; evidence of reflective learning; sensitivity to ethical concerns; appropriate research team and skills; clarity of governance framework and identification of intermediaries. As well as guiding funders, we believe that an assessment of the quality of co-productive methodology could have wider usage. Primarily, as noted above, it could be embedded in the design and delivery of co-produced projects, so useful to partners in reflecting on their experiences; it could enable those interested in co-production to assess reports of projects and their achievements, including peer reviewers.
Drawing on our data and literature discussed above, we propose the following framework to assess quality in co-produced research. In Table 1, we suggest criteria and questions that could be used throughout the design, delivery and reflection stage of co-produced research. The framework draws on the ABCD community evaluation and other approaches outlined in this section. As noted earlier, we identify three different domains: process, outcomes and autonomy. Process because the research activities need to be participatory and include inputs from all partners; outcomes because these refer to the extent to which project aims have been met and a weakness in existing reports (whether such outcomes are intended or unintended); autonomy is identified separately to highlight whether individuals and ‘community’ have become more empowered or built capacity as a result of the project.
Framework for quality assessment.
For example, regarding questions 10–11 (ethics), one PM project identified dilemmas where a new project relating to a long-standing campaign for a local supermarket could ‘open old wounds’ and create tensions between the new project and managing expectations that it would result in a new supermarket locally.
Rather than simple yes/no answers to these questions, we suggest a continuum of responses. McConnell (2010) conceptualised ‘success’ along a spectrum from success, conflicted success (where the policy is heavily contested), precarious success (on the edge of failure). McConnell applied this spectrum across different policy dimensions (process, programme and politics) where the type of success in one dimension may differ from success in other dimensions. This framework includes both interpretation and fact, as policy can be evaluated by both supporters and opponents, so success is to some extent ‘constructed’ – one person’s success is another person’s failure. For co-production, terms such as ‘success’ or ‘failure’ can be value-laden, though there can be learning from projects that turn out differently than expected (Smith and Delamont, 2019). We believe too that exploring such questions and allowing for a spectrum of responses could enable different interpretations within a project team and the development of a more nuanced assessment. Responses could be grouped under broad categories (e.g. high, medium, low),’ high’ denoting that the answers indicate high quality; ‘medium’ as meeting the minimum across several questions; and ‘low’ as being below expectations.
Table 2 is presented as an example of using a spectrum of responses to two example questions from the framework described in Table 1.
Sample answers across a spectrum.
Concluding note
Co-production carries multiple meanings and expectation that vary across contexts and that can be used for different political agendas. The promise that co-productive methodologies can generate ‘better’ quality research can, currently, be understood as a statement of belief. Given the celebratory tone attached to much reporting of co-production and limited evidence about its outcomes, more debate is needed about how to sustain the flexibility and innovation that co-production can bring, whilst taking seriously the challenges of co-production.
Drawing on data from ECRs and literature from across academic, community-development and service-delivery sources, we argue that the claims that are central to co-production – that it enhances impact and better-quality research – need to be clearly conceptualised. If there is no agreed way to assess the quality of individual co-produced research studies, merely stating that a methodology inevitably results in better quality amounts to belief, not judgement. We are concerned that, if there is insufficient rigour, then studies may be judged inadequate and findings ignored. Although co-production can be emergent and messy, the notion of quality can and should be integral to its ethos and practice. We acknowledge that this can be time-consuming and so context-specific that comparisons between them are limited. Nonetheless we believe that this quality framework could give communities, academics and funders alike an assurance that the time, effort and emotional labour they commit to a project is worthwhile in its contribution to wider knowledge production and/or social change, alongside providing assurance to funders that research funding invested in this methodology is money well spent.
The framework has been drawn from a synthesis of studies and literature from the community development, research, policy and equality studies referred to earlier. This is an early, experimental and speculative framework that we would urge future co-produced teams to develop further as an integral part of their own co-productive processes.
In summary, we contend that first, there is no reason not to assess co-production against research standards; secondly, criteria such as validity are relevant, though require adapting to the particular co-produced research context; thirdly, that any such evaluation should be built into the process of co-production itself; so therefore, fourthly, measures of success or good quality should themselves be co-produced as an integral part of co-produced design and delivery. We recommend that community-academic research partners try out this framework within the design and delivery of their projects, to see how such a framework might apply in practice and what refinements may be needed. In effect, we are suggesting greater ‘regulation’ of co-production to enhance both process and outcomes. This could include how a project would be evaluated, so that both project teams and external ‘consumers’ of co-produced knowledge could assess the strengths and limitations of the research.
Footnotes
Acknowledgements
The authors would like to thank colleagues working on the Productive Margins: Regulating for Engagement programme who helpfully commented on early drafts of this article. Particular thanks to Dr Angela Piccini, Dr Jenny Barke and Professor Morag McDermont.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research on which this article is based was supported by the Economic and Social Research Council (Grant reference No. ES/K002716/1).
