Abstract
Research impact is high on the agendas of governments and funders all over the world. As participatory action-oriented research (PAR) intends to have an impact – positive changes in policy and practice – interest in PAR is rising. But although impact may sound clear, it is susceptible to various interpretations. Hence assessing the impact of PAR is a challenge.
In this article we describe a PAR project with multi-stakeholder participation at its core, and focus on the effectiveness (co-impact) of this project. We elaborate on two interpretations of impact: findings-based and process-based impact. Although interest in research impact tends to prioritise findings-based impact which focuses on substantive results and shows little or no attention for the effects of the participation process (process-based impact), we acknowledge both types of impact. We equally value the process-based impact, as it does justice to the potential richness of participation.
In presenting our research approach and findings, we provide insight in how both findings-based (substantive results) and process-based (less tangible effects) impact can be generated and assessed.
We experienced that strict vigilance was crucial for detecting process related effects, without comprising the findings-based ones. Our research approach, with its consciously chosen sequence of actions, a variety of data gathering methods, and informed selection of stakeholder groups, appears to pay off in generating co-impact. Additionally, the operationalisation of a conceptual impact framework in an applicable impact assessment tool, proved to be useful for discovering less tangible and even hidden effects at the individual and group levels of stakeholders both within and outside the PAR setting. We believe that focusing on substantive results only undervalues the richness of the process of participation and its related effects.
Keywords
Introduction
High on the agendas of governments and funders across the globe is the concern to ensure that, research has impact beyond the academic realm, brings positive change or benefits to various domains such as public policy or services, health, economy, and the environment (Banks et al., 2017; Morton, 2015). In response to the need for more effective approaches to realize impact, also referred to as changes in policy and practice, interest in participatory action-oriented research (PAR) is rising (Abma et al., 2019; Banks et al., 2017; Garretsen, van de Goor, & van de Meehn, 2021; Jacobs, 2010; Natland & Hansen, 2016).
Typical for all PAR is that research should contribute to both (a) practically relevant and scientific knowledge building, and (b) to real improvement or change in a problematic situation of a specific population in a specific context (Abma & Widdershoven, 2005; Fals-Borda & Rahman, 1991; Migchelbrink, 2016; Reason & Bradbury, 2001; Van Regenmortel et al., 2016; VanderPlaat, 1999). Van Regenmortel et al. (2016, p.10) summarizes this typical double objective of PAR as ‘to prove’ and ‘to improve’. To realize this double objective, collaboration between researcher and participants is necessary (Abma et al., 2011; Banks et al., 2017; Bergold & Thomas, 2012; Heron & Reason, 2001; Kindon, Pain, & Kesby, 2017). Due to collaboration changes, varying from the individual to the collective level, occur (Banks et al., 2017). When PAR is implemented, impact is co-produced and therefore labelled as ‘co-impact’ (Banks et al., 2017).
In PAR, all those whose interests are touched upon by the research should actively be given a voice (Abma et al., 2011). In particular, it should express the interests of the so-called silenced voices; vulnerable and marginalized populations whose participation in research is not always evident and whose voices are not usually heard (Abma et al., 2011, 2019; Abma et al., 2009; Siesling & Garretsen, 2014; Van Regenmortel et al., 2016). In addition to the double objective of PAR, its participatory character may bring about other effects, such as social equality and justice, ownership and control over knowledge and change. In combination with dialogue, it also promotes balances in power (Abma et al., 2011; Abma & Widdershoven, 2005; Baum et al., 2006; Brydon-Miller, 1997; Cargo & Mercer, 2008; Fals-Borda, 2001; Healy, 2001; Jason et al., 2003; Kim, 2016; Park, 1993; Reason, 2006; Van Regenmortel et al., 2016).
Research Context and Aim
With the aim of reaching the double research goal ‘to prove’ and ‘to improve’, we conducted a PAR project between July 2016 and June 2022 (see Figure 1), in which we deliberately chose for the participation of multiple stakeholders, i.e. people with a stake in the issue under study and whose contribution is important for the research (Abma et al., 2011). Stakeholders in our research project were considered as partners in research and the emphasis was on shared responsibility and control over ideas, processes and outcomes (Abma & Broerse, 2007; Frankham, 2009; Numans et al., 2019). We therefore define our research approach as ‘partnership research’ (Numans et al., 2019). This fits under the umbrella of PAR, which covers a variety of study approaches from participatory to action-oriented research (Banks, et al., 2013; Kim, 2016; Kindon et al., 2017). Roadmap PAR project.
The central theme of our PAR project was societal vulnerability, referring to the structurally vulnerable position of specific individuals or groups in society, as well as to their negative experiences when interacting with social policy practitioners, often leading to a continuation or increase of their vulnerable state (Numans, Boog, Van Regenmortel, & Schalk, 2020; Vettenburg, 1998). The ultimate goal of the research was to obtain a better understanding of the concept of societal vulnerability and its underlying mechanisms as experienced by vulnerable persons (‘to prove’), and to disclose their suggestions on possible reforms to reduce their perceived vulnerability (‘to improve’).
Throughout the PAR project we made an informed selection of groups of participants (stakeholders) who were invited to participate in certain phases of the research project, assuming that they could contribute significantly to realize the double research objective.
Three groups of stakeholders participated in this project. First, the stakeholder group of ‘vulnerable persons’, which concerns persons who do not enjoy full physical, psychological and/or social well-being (Bruggeman et al., 2018; Jehoel-Gijsbers, 2004; Metz, 2009; Provinciale Raad voor de Volksgezondheid en Maatschappelijke Zorg in Noord-Brabant [PRVMZ], 2010), cannot or insufficiently meet social policy’s standards of self-reliance and social participation, and therefore are dependent on government support (care and social welfare services) in order to participate in society (Numans et al., 2023; Numans et al., 2020). In the research project, priority was given to the lived experiences of vulnerable persons and to relevant improvements for vulnerable populations. Knowledge building (‘to prove’) preceded the phase ‘to improve’. Vulnerable persons are the leading carriers of information on what it is like to be (allegedly) vulnerable, as well as on which kinds of change or impact in social policy practice they would acknowledge as relevant for people from vulnerable populations. We therefore considered vulnerable persons our primary stakeholders from the outset as they played a central role in both phases of our PAR project.
Secondly we involved co-researchers. Four persons from vulnerable populations and four social workers who, together with the first and second authors, operated in a mixed research team. The stakeholder group of co-researchers was established to contribute to ‘to prove’: to bring out the voice of vulnerable persons by conducting two to three in-depth interviews with the ‘vulnerable persons’ group of stakeholders.
In the third place, we included representatives of social work organizations, i.e. social policy practitioners, who are in touch with vulnerable persons. We assembled them in the ‘advisory board group’, assuming that this stakeholder group would be in the position to implement changes and willing to propagate the voice of vulnerable populations towards other relevant parties. Although the advisory board group was particularly established to contribute to ‘to improve’, we consciously involved these stakeholders in the whole research project. By means of presenting and discussing preliminary findings of the ‘to prove’ phase, they indirectly were in contact with the stakeholder group ‘vulnerable persons’ and gradually became prepared to reflect on the actual needs of the vulnerable population.
The ‘to prove’ phase resulted in a thorough insight into the negative experiences of vulnerable persons in their daily lives, especially when interacting with social policy practitioners. These experiences formed the basis of the last – ‘to improve’ - phase of our PAR project (August 2020 and July 2022).
With the stakeholders groups of vulnerable persons and advisory board group we embarked on a journey to co-construct an agenda aimed at improving social policy practice. Concurrently, we remained vigilant to spot other effects arising from the application of research procedures, data gathering methods and utilization of a tool to assess our data. In short, our research approach could potentially provoke a range of effects all of which, like the improvement agenda (the substantive results), should equally be subject to systematic examination.
In this paper, we investigate the effectiveness of participation of multiple stakeholders in PAR. We focus on the ‘to improve phase’, and explore whether our research approach proved to be effective, i.e. led to the intended impact, as well as possibly caused unintended (less tangible) effects. For this, we needed more clarity about how to understand the concept of ‘impact’, and whether and how impact can be assessed.
The Concept of ‘Impact’
The concept of ‘impact’ covers a wide range of definitions. This influenced the development of different models and frameworks aimed at assessing research impact (Banks et al., 2017; Morton, 2015). As the concept of ‘impact’ is subject to multiple interpretations, assessing the impact of PAR is difficult or at least challenging (Abma et al., 2019; Morton, 2015).
According to Abma et al. (2019, p.221–222), “in its simplest form, impact can be seen as the difference the research has made to the community of practice, the legacy it leaves behind”. Such a definition of ‘impact’ - preferred by governments and funders of research across the world - tends towards a narrowly defined explicit, concrete notion of impact. It is understood as an identifiable event that can be pre-determined, observed, measured, and evaluated (Abma et al., 2019; Banks et al., 2017; Morton, 2015; Pain, 2014). In addition, this interpretation of ‘impact’ is based on a traditional, positivist, linear model of the research process, with impact created at the end as a result of the findings (Banks et al., 2017; Pain et al., 2015). In other words, in this notion ‘impact’ is ‘findings-based’. According to Pain (2014, p.21) impact here is ‘striking a blow’: a single significant blow, with a linear temporal track from incident to effect.
However, research impact can be seen not just as a handoff of research findings (substantive results), but as a process of engagement with and between research participants (stakeholders) around various stages of the research (Nutley et al., 2007). Precisely this engagement, in the form of collaboration between researchers and participants, is characteristic for PAR. Consequently, in PAR impact is shaped in co-impact as it is co-produced (Banks et al., 2017; Pain et al., 2015).
The notion of ‘co-impact’ moves away from the linear research model with its focus on ‘findings-based impact’ (substantive research findings). It recognises and values the ‘process-based impact’ equally, i.e. the changes or effects that occur throughout the research process, as a result of the participatory action of stakeholders (Abma et al., 2019; Banks et al., 2017; Morton, 2015; Pain, 2015).
Preferring the broader interpretation of ‘impact’ which appreciates the richness of participation, we found inspiration in the conceptual framework of co-impact developed by Banks et al. (2017). We translated this framework into an impact assessment tool. In this framework Banks et al. (2017, p.543) distinguishes three types of co-impact, which entail ‘participatory impact’, ‘collaborative impact’ and ‘collective impact’, respectively referring to (1) the change in thinking, emotions, and practice of research partners, as a result of their involvement in the process of conducting PAR; (2) the take-up and use of the research findings to influence and change practice and/or policy; and (3) a deliberate strategy of the research partners to achieve a specific, targeted change in practice and/or policy.
We assumed that with this impact assessment tool, we would be sufficiently armed to discover and excavate the effects of our PAR project, certainly the non-substantive ones. To be perceptive to other than substantive results, an instrument to appraise hidden effects associated with participation in research, is more than welcome. In fact, our method was to categorise the ‘to improve’ phase data in the three different types of impact mentioned above.
In this paper we present the substantive (tangible) results of the ‘to improve’ phase of our PAR project – the co-construction of an improvement agenda – as well as the effects (less tangible results) of the process of participation. Besides attention to findings-based impact (substantive results) we present a profound insight in the individual and collective effects of the process of intensive participation in research (process-based impact). Effects observed, both in- and outside the research context, as well as the different process-based effects for the different stakeholders involved, are brought forward. Central questions to be answered in this paper, thereby exploring whether our PAR project proves to be effective, are: (1) Did the participation of stakeholders lead to achieving the research goal, i.e. a tangible agenda for improvement of social policy practice?; (2) Did participation lead to other less tangible effects at both group and individual level of stakeholders?
Method
Research Methodology
In order to realize the ‘to improve’ objective we followed a responsive methodology, structuring the participation in the research process and constantly keeping an eye on the needs of vulnerable persons.
Responsive methodology is related to participatory action-oriented research. Abma and Widdershoven (2006, p.29) state that “in a responsive approach, research is regarded as a joint search process with stakeholders for the value and meanings of their practice. It aims to increase mutual understanding between stakeholders as a vehicle for practice improvement”.
Introduced by Robert Stake, it has its origins in the 70’s in the field of educational evaluation (Abma & Stake, 2002; Abma & Widdershoven, 2006; Guba & Lincoln, 1989; Nierse & Abma, 2011; Stake, 1975, 2004). Over the years, the original ideas about responsive methodology further evolved (Abma & Stake, 2002; Stake, 2004) and interactive versions were developed aimed more specifically at involving different stakeholders in exchanging their perspectives, thereby enhancing personal and mutual understanding and learning (Abma, 2005a; Abma, 2005b; Abma & Broerse, 2007; Abma & Stake, 2002; Nierse & Abma, 2011; Nierse et al., 2011). As an interactive methodology it acknowledges plurality of perspectives and clarifies the issues of as many stakeholders as possible (Abma & Stake, 2002; Guba & Lincoln, 1989; Nierse & Abma, 2011; Stake, 1975, 2004). Specific attention is paid to the inclusion of marginalized groups (Nierse & Abma, 2011).
Key instrument in responsive methodology is dialogue. Dialogue within and between stakeholder groups challenges active participation of all involved (Abma et al., 2009; Nierse & Abma, 2011). Dialogue can contribute to less tangible effects such as (more) social equality, social justice, ownership and empowerment of participants. As such these represent the opposite of substantive results, which are for example written recommendations for health or well-being improvement. Responsive methodology fits perfectly in PAR with respect to potential other effects of participation as mentioned in the Introduction.
Dialogue is used to facilitate collaboration between different stakeholder groups (Nierse et al., 2011) and to reach a shared understanding or consensus regarding a specific goal (Abma et al., 2009, 2019; Snoeren et al., 2011). In the context of asymmetrical power relations dialogue can function as a vehicle for balancing power (Abma et al., 2019).
In responsive methodology, power imbalances are ideally handled by working in steps (Guba & Lincoln, 1989; Nierse & Abma, 2011). We followed this approach. First, we organised homogeneously composed groups of both stakeholder groups: vulnerable persons and advisory board group. Within each group a focus group method was applied in order to promote dialogue, aimed at developing a shared understanding of the issues at stake. This was followed by a heterogeneously composed focus group, attended by all participants of both stakeholder groups, aimed at reaching a shared understanding.
Responsive methodology often integrates multiple research methods (Nierse et al., 2011). In addition to the focus groups already mentioned, we asked all stakeholders to reflect on their participation in the project. For this purpose we developed an impact assessment tool in the form of a short questionnaire, based on the conceptual framework of co-impact of Banks et al. (2017).
Design and Procedures
The participatory journey towards the co-construction of an improvement agenda demanded a general plan outlining the overall research strategy, including a set of actions in a specific sequence allowing ample space for participatory sessions.
During the last part of this PAR project aimed at ‘to improve’ we continued with the participation of the following stakeholders: vulnerable persons, advisory board group members and co-researchers, although the latter had a limited role. The responsive methodology, dialogue, and reflection on outcomes of dialogue sessions were the common factor in the final research actions. The sequence of actions was designed in such a way that the vulnerable persons’ expressed needs for improvement of social policy practice would remain the thematic core throughout all sessions organised for one stakeholder group or a mix of stakeholder groups (see Figure 1.). As such, outcomes of sessions were continuously linked together, to increase insights and understanding, and to reflect on previous sessions.
Following the planning we executed several steps. We started with renewed contact with respondents (Aug-Sep 2020) who participated in the in-depth interviews and had agreed to be contacted again (written informed consent). Renewed contact was established by the stakeholder group of co-researchers. They had had in-depth contact with respondents and were therefore the appropriate stakeholders to renew this contact in the ‘to improve’ phase. Of the 16 respondents, 12 respondents had given permission to be approached again. 3 out of 12 respondents remained involved in the journey towards the improvement agenda. Due to the co-researchers’ modest role in this phase, the fact that they did not participate in the focus group sessions, they were not invited to answer the short questionnaire.
The second step consisted of organizing 3 homogeneously composed focus groups of ‘vulnerable persons’ (FG1, respondents) (Nov 2020 – Jun 2021). The first focus group session concentrated on checking the obtained data on criticisms of social policy practice (member check). The second session entailed the fine-tuning of outcomes based on an analysis of expressed criticisms. The last session aimed to rank criticisms in a vertical order.
The third step entailed the organization of 2 homogeneously composed focus groups of advisory board group members (FG2, n = 10) (Jul 2021). Point of departure for these focus group sessions were the outcomes of the focus group sessions of the stakeholder group ‘vulnerable persons’. In this manner we continued to give vulnerable persons the lead as main carriers of information of the issue(s) at stake. The aim of the first focus group session was to inform the advisory board group members of the process of ranking of criticisms by the stakeholder group ‘vulnerable persons’, to familiarise these members with the (priority) list of points of criticism, and to reflect on those. In the second focus group session the members explored possibilities for improvement.
In the fourth step we mixed the two stakeholder groups into one heterogeneous stakeholder group aimed at finding common ground and eventually establishing a co-constructed agenda for improvement of social policy practice (Feb – Jul 2022). To this purpose we organized two focus group sessions. The combined group agreed on the establishment of a working group composed of representatives of both stakeholder groups. The working group was to elaborate the potential practical modalities of the improvement agenda. To this end, we organized two working group meetings.
All focus group sessions lasted approximately three hours and the working group meetings two hours. The first and second authors prepared and facilitated all meetings. The duration of the project was strongly influenced by the Covid pandemic. The process was delayed, because planned meetings had to be rescheduled. In addition, due to the Covid pandemic and measures taken by the government, such as a required physical distance between participants during meetings, it was not possible to use audio records. The first author made written reports of all meetings, in which dialogues, as a key instrument, were recorded. These written reports were the source of information for the observable tangible and less tangible group level effects caused by the participation process. All reports were returned to the stakeholders for validation (member check). The first and second authors summarized all input under the categories ‘substantive results of the co-constructive process’ and ‘less tangible effects of participation at the group level’. No analysis programme was used.
The short questionnaires used to gain insight in the perceived changes at the individual level, were filled in by 10 of the 13 stakeholders in total (vulnerable persons: 2, advisory board group members: 8). No analysis programme was used. The first and second authors categorised the answers in line with the questions on the questionnaire. They processed all input anonymously.
All stakeholders gave written informed consent to participating in the journey for improvement. The research protocol for this part of the research project was approved by the Ethics Review Board of Author’s University (RP267).
Findings
In the ‘to improve’ journey the participation of stakeholders remained a key activity. Realizing that all effects are intertwisted in the dynamics of stakeholders’ participation, we intentionally focused separately on the two aforementioned research questions. In the following we make a distinction between substantive results (research question 1) and less tangible effects of the process of participation at both group and individual level (research question 2).
First, we present the less tangible effects at the group level, accompanied by a presentation of the substantive results. Both followed from the focus group sessions organised for the co-construction of the improvement agenda. Both types of effects go through recursive cycles. Less tangible effects of focus group sessions in one way or another influence the substantive results of the co-construction of the improvement agenda. The substantive results in turn colour the next focus group session. In short, the chain of group sessions shows a mutual interference in the interplay of effects of participation. As mentioned under Methods, written reports of all group sessions formed the basis of the improvement-related effects, including the effects at group level.
Second, we present other less tangible effects of the process of participation at individual level. As mentioned before, evidence of these effects is based on individual responses to a short questionnaire, including indicators for change derived from the conceptual framework of Banks et al. (2017). The quotes of stakeholders cited are translated from Dutch by the first author.
Outcomes of the co-construction of an improvement agenda for social policy practice
In the last heterogeneous focus group session, participants decided that the working group should elaborate the outline, thereby developing a ‘friendly service’ guideline for practitioners and/or organizations; the improvement agenda. The working group originally consisted of 3 members of the advisory board group and 3 members of the other stakeholder group, supported by the first and second authors. Unfortunately, the working group downsized to the three members of the stakeholder group of vulnerable persons. Currently, the development of the guideline is still a work in progress (see Figure 1).
Less Tangible Effects of the Process of Participation for the Individual Stakeholders
Participatory Impact
Participatory impact is described by Banks et al. (2017, p.543) as “the change in thinking, emotions, and practice of researchers and core partner organizations, which happens as a result of their involvement in the process of conducting PAR”.
For participatory impact we distinguished the following four indicators for change: change in (1) thinking, (2) emotions, (3) attitude, and (4) acting.
With respect to the indicator (1) change in thinking, stakeholders’ expressions ranged from no change (S10_F_FG1, S1_M_FG2, S7_F_FG2), to reinforced conviction (S2_M_FG2, S8_F_FG2, S4_F_FG2), gained insight (S2_M_FG2, S4_F_FG2, S6_F_FG2, S8_F_FG2, S9_F_FG1), and raised consciousness (S3_F_FG2).
In total one stakeholder of the focus group ‘vulnerable persons’, and five stakeholders of the advisory board group mentioned changes in thinking. For example, a stakeholder of the group ‘vulnerable persons’ stated: “Instead of fighting for my right, I am now convinced that it ís my right to ask for professional help and support” (S9_F_FG1). Stakeholders of the advisory board group mentioned for example a reinforced conviction of the need to involve vulnerable persons in both practice and research because of the additional value of their input (S2_M_FG2, S4_F_FG2, S8_F_FG2). They also mentioned that they developed the insight that participation in research and practice yields outcomes, e.g. personal and mutual learning. They added that participation is not a simple and organic activity, but that it needs proper guidance and monitoring (S2_M_FG2, S4_F_FG2, S6_F_FG2, S8_F_FG2).
With respect to the indicator (2) change in emotions: One stakeholder of the focus group ‘vulnerable persons’ mentioned feeling respected, feeling a full human being, and feeling empowered (S9_F_FG1). “I feel myself included in the research process as a full member. My input is actually listened to and considered seriously. I am feeling stronger due to this process. ” (S9_F_FG1).
Regarding the indicator (3) change in attitude: Two stakeholders of the advisory board group mentioned becoming more passionate about vulnerable persons. One of them stated: “Every story of people with experiences in vulnerability teaches me how people can get into trouble, but also how creative and resourceful people can be. I take this with me, it encourages me to stop being judgemental” (R5_M_FG2). The other stakeholder mentioned now being more inclined to involve vulnerable persons in practice (S2_M_FG2).
With regard to the last indicator, (4) change in acting: Two stakeholders expressed changes. One stakeholder of the stakeholder group ‘vulnerable persons’ stated: “I don’t only indicate what goes wrong, the points of criticism, but now I am thinking of the ideal ‘social shop’ and highlighting its characteristics. First I was in a fighting mode, now I am in an explanation mode. I also set my boundaries better” (S9_F_FG1). One stakeholder of the advisory board group mentioned that, despite her vulnerability, she presents herself more as a powerful, intelligent person (S3_F_FG2).
Overall, all stakeholders, except one from the group ‘vulnerable persons’ (S10_F_FG1) and two from the advisory board group (S1_M_FG2, S7_F_FG2), mentioned indicators related to participatory impact. Although it was the first time for most stakeholders to participate in participatory action-oriented research, all stakeholders expressed that they experienced their participation as an enrichment, and that they have the impression that their input has been seriously taken into account.
Collaborative Impact
According to Banks et al. (2017, p.543), collaborative impact is based on the take-up and use of findings of collaborative research by individuals and organizations to change practice and/or policy and influence attitudes and culture.
For collaborative impact we distinguished the following three indicators: (1) research uptake, (2) research use, and (3) research impact. According to Banks et al. (2017, p. 550), referring to Morton and Flemming (2013), research uptake (1) entails that people are interested in the research, read it, talk about it, come to a presentation, etc. Research use (2) means that people do something with the research, change their views, pass it on to someone else, apply it to practice or policy. And research impact (3) refers to a contribution to change as a result of research use.
With regard to (1) research uptake, none of the stakeholders explicitly referred to the indicator of research uptake. However, all stakeholders attended the focus group meetings where findings were presented and discussed.
In contrast, (2) research use was referred to in a variety of ways by all but one stakeholder of the group ‘vulnerable persons’ (S10_F_FG1) and two stakeholders of the advisory board group (S1_M_FG2, S4_F_FG2). For example, a stakeholder of the group ‘vulnerable persons’ mentioned: Sharing research findings with other vulnerable persons (peers) (S9_F_FG1). Examples mentioned by stakeholders of the advisory board group are: dissemination and discussing research findings with colleagues and social policy makers aimed at sharing knowledge and raising awareness (S6_FG2, S7_F_FG2); convincing relevant representatives in the social domain (fellow directors, social policy makers) to integrate the criticisms of vulnerable persons in social policy and its implementation, and processing research findings in policy documents of the organization (S2_M_FG2); continuation of participatory work as this research has confirmed the importance of this way of working (S8_F_FG2), and inviting persons with experiential knowledge (vulnerable persons) to participate in two recent new projects (R5_M_FG2); raising awareness during courses among practitioners and volunteers about the mechanism that vulnerable persons tend to accept what is available instead of requiring responses to their real needs (S3_F_FG2). This latter stakeholder also referred to (3) research impact: the recognition of this mechanism among practitioners and volunteers in their own practices.
In sum, all stakeholders except for one from the stakeholder group ‘vulnerable persons’ (S10_F_FG1) and two from the advisory board group (S1_M_FG2, S4_F_FG2), mentioned an indicator related to collaborative impact.
Collective Impact
Banks et al. (2017, p.543) defines collective impact as “a deliberate strategy of the research partners to achieve a specific, targeted change in practice and/or policy based on issues highlighted via the research”.
Following this definition the indicators of collective impact are a deliberate strategy, and a specific, targeted change in both practice and policy.
Outcomes Heterogeneous Focus Group Sessions Both Stakeholder Groups.
In answering this question, stakeholders’ expressions ranged from no belief in the potential for collective impact (S1_M_FG2, S4_F_FG2), to little belief because of the dependency on a wide variety of factors such as time and budget (S6_F_FG2), and to belief (S9_F_FG1, S10_F_FG1, S2_M_FG2, S3_F_FG2, S5_M_FG2, S7_F_FG2, S8_F_FG2).
Some of the stakeholders who have confidence in the potential of collective impact, mentioned conditions under which collective impact could be reached. For example: research findings need to be operationalised (S8_F_FG2, S10_F_FG1); advisory board group members should stay committed – in thinking and acting - to contributing to the implementation of a tool or end product to achieve change (S9_F_FG1, S10_F_FG1); and, real improvement of societal vulnerability needs continuous interaction between social policy practitioners and vulnerable persons (S9_F_FG1).
A hindering factor mentioned by most stakeholders was the Covid pandemic. Focus group sessions were frequently postponed. Stakeholders perceived this as a serious obstacle to recalling content and process outcomes of the previous meeting, as well as to mutual cohesion between stakeholders.
Discussion
This paper takes a focused look at the effectiveness of the last phase of our PAR project. This phase, aimed at improvement, was guided by the findings of a previous ‘to prove’ phase. Its results unveiled that vulnerable persons (our respondents) perceived insufficiencies in social policy practice, which is supposed to adequately respond to the actual needs of vulnerable persons. Aimed at bringing about change of or impact on these perceived insufficiencies, we continued with the last – ‘to improve’- phase of our PAR project. Stakeholders remained involved and through a responsive methodology and dialogue we, as research partners, worked on the accomplishment of an improvement agenda for social policy practice and its implementers. In the interactive process of homogeneous and heterogeneous focus group sessions it became evident that active participation continuously impacted on the co-construction of the intended improvement agenda, the substantive results.
Outcomes Homogeneous Focus Group Sessions ‘Vulnerable Persons’.
We assumed that, due to participatory action, change or impact at individual level appears prior or simultaneously to changes at group level. Moreover, all individual stakeholders have networks outside the research context and they may or may not share research experiences with other individuals or groups not involved in this particular research endeavour. In other words, less tangible effects of PAR occur within and outside the research context. Whereas the changes at the focus group level can be observed and recorded by the researchers, all effects at individual level, as well as group level effects outside the research context, take place beyond researchers’ perspective. Therefore we were more than eager to gain insight in the dynamics of PAR and the complexity of its effects.
Outcomes Homogeneous Focus Group Sessions ‘Advisory Board Group Members’.
A broad interpretation of impact assisted in gaining a nuanced understanding of what participation could bring about besides the substantive and observable research effects at the focus group level within the PAR setting.
To bring all effects of participation, including the hidden ones, of our research into view, the conceptual framework of co-impact (Banks et al., 2017) was processed in a short questionnaire aimed at all stakeholders. The indicators conceptualised under the three different types of co-impact were mirrored in three groups of questions respectively related to ‘participatory’, ‘collaborative’, and ‘collective’ impact. Most stakeholders recognised the different fields of changes or impact and voiced their individual perceptions in writing. Hence, the framework proved to be useful as an assessment tool for a process-based impact, as well as for potential impact outside the research context. It is noticeable that stakeholders’ accounts related to ‘participatory impact’ – the less tangible effects at the individual level - were richly articulated. Although ‘findings-based impact’ instead of ‘process-based impact’ appears to be more valued than changes experienced at individual level (Banks et al., 2017, p.543), the completed questionnaires gave another impression. Most stakeholders detected changes in e.g. their thinking, attitudes and emotions, and these were experienced as an enrichment.
With respect to changes within the research context, impact occurred at both focus group level and individual level as mentioned above. With this acquired knowledge our second research question has been answered: participation leads to other less tangible effects at both group and individual level of stakeholders.
In addition, by applying the assessment tool inspired by Banks et al. (2017) the occurrence of changes outside the research context became evident. Following the stakeholders’ answers to the short questionnaire, some of them selectively passed research process related experiences on to members of their networks. Some pointed to the value of participation itself. Others stressed the importance of involving experts by experience (vulnerable persons) in (research) practices, or explained how mechanisms of vulnerable persons work when they are confronted with social policy practitioners.
Stakeholders enrich the research process and increase its impact, as the effects of their participation resonate beyond the research setting. It is exactly this double ‘collaborative impact’ – research uptake and use within and outside the research context – that can be considered the value of working with stakeholders. Individual stakeholders associate with others, and in this way research impact may fan out much wider. We therefore subscribe to the following: “Action [participation in research] is like a pebble that is thrown into a pond, the effects ripple out to the edge and even beyond” (Abma et al., 2019, p.100).
However, the reach of effects, including the research findings, depends on the social environment of each stakeholder. With due respect for the stakeholders ‘vulnerable persons’, we assume that stakeholders of the advisory board group, representatives of social work organisations, will bring their research experiences further within their professional networks. Therefore, a well thought through selection of professional stakeholders is important when embarking on a PAR. In this respect we agree with Reed et al. (2018, p.9) who argues in favour of a strategic rather than a complete selection of stakeholders.
With regard to our first research question – whether participation led to a tangible agenda for improvement of social policy practice – the stakeholders co-constructed an outline for a joint improvement agenda. Currently, a ‘friendly service’ guideline for social policy practice is waiting further elaboration. In short, a tangible improvement agenda has not yet been realised.
However the outline provides evidence of ‘collaborative impact’ according to the definition of Banks et al. (2017). Uptake and use of research findings took place in the last ‘to improve’ phase of our PAR project. The findings from the first ‘to prove’ phase served as point of departure, and were woven into the whole structure, design and procedures of the last improvement phase. The responsive methodology kept the exchange of different perspectives going without losing the focus on the actual needs of vulnerable persons and the perceived insufficiencies of social policy practices.
Once the ‘friendly service’ guideline has materialised, and stakeholders agree on a deliberate strategy aimed at a specific, targeted change in practice and/or policy based on issues highlighted through the research (Banks et al., 2017), a third type of co-impact, labelled as ‘collective impact’, can be applied. According to Pain (2014), only then “striking a blow” happens. However, a long “walking together” (Pain, 2014) is needed before and after the accomplishment of the outline of the improvement agenda. During the process of “walking together” influential effects at individual and group levels may occur, maybe even more impactful than the intended substantive results.
It is worth mentioning that most stakeholders believed in an ultimate ‘collective impact’ (“striking a blow” according to Pain, 2014). For this to happen, some stakeholders emphasised the requirement of sustained engagement of all stakeholders, a lasting partnership fuelled by a sense of ownership. After all, partnership automatically includes participation, but participation does not automatically include partnership (Numans et al., 2019). If the elaboration of the guideline is only undertaken by a very limited number of stakeholders ‘vulnerable persons’, the existence of a partnership is rather questionable.
The level of engagement of stakeholders, let alone the development or continuation of a partnership, are to a large extent out of the researchers’ reach. This is a limitation of PAR. Certainly, researchers can influence factors supporting engagement, such as the choice of research design, including two-way dialogue and the handling of power balances, but context factors also exist that researchers have no control over (Morton, 2015; Reed et al., 2018). These may include available time of the stakeholders, change of working environment, and unforeseen circumstances such as Covid. In fact, if Covid had not intervened, the engagement of stakeholders could have been stronger, and hence ‘collective impact’ might have been approached or even achieved.
As mentioned in the Introduction and under Method, PAR as a research approach and dialogue as an instrument may bring about even more effects, such as social equality and justice, ownership, power balances, and empowerment (Abma et al., 2009, 2011; Abma & Widdershoven, 2005; Baum et al., 2006; Brydon-Miller, 1997; Cargo & Mercer, 2008; Fals-Borda, 2001; Healy, 2001; Jason et al., 2003; Kim, 2016; Park, 1993; Reason, 2006; Van Regenmortel et al., 2016). It is evident that these concepts that are rich in content would need accurate and precise theoretical reflection. Unfortunately this is beyond our research goal and the scope of this paper. However, our observations as well as stakeholders’ perceptions point to the occurrence of such side effects. The voices and concerns of vulnerable persons were leading in respectively the first and last phases of our PAR project. Talking ‘with’ instead of ‘about’ them was embodied in the entire research. Participating vulnerable persons shared with us their particular experiences: they appreciated having been respondents, telling their stories which were seriously taken into consideration, but their contribution during the improvement phase was perceived as more important. They were more in control and could direct the improvement agenda.
They felt in the position to illustrate in detail to stakeholders of the advisory board group how social policy practice could operate differently. At the same time, advisory board group members reflected seriously on the list of criticisms on social policy practice, presented by the participating vulnerable persons. In fact, these vulnerable persons experienced first-hand some of the principles of fairness, equality, equity, and even ownership and empowerment (see Findings, under Participatory impact). So far, a precarious balance in power took shape.
But reflection on the needs of vulnerable persons has to be translated into action. Here we ran into an imbalance of power. Stakeholders of the advisory board group, expected to have the power to bring the improvement agenda further and to share their power with the stakeholder group ‘vulnerable persons’, withdrew from the elaboration task, leaving the stakeholders of vulnerable persons without proper assistance. It seemed as if the empowerment of vulnerable persons, the ultimate beneficiaries of the implementation of an improvement agenda, was only temporary. Due to withdrawal of advisory board group members from the elaboration task, we observed that the vulnerable persons fell back, more or less, into their previous state of dependency. The vulnerable persons expressed a certain disappointment (Table 2).
To conclude, in this paper we explored whether our PAR project with multistakeholder participation proved to be effective, i.e. led to (intended) impact. If we had applied a narrow interpretation of ‘impact’, regarding the substantive results, the answer is “yes” (see Table 3.). However, all stakeholders agreed that the achieved outline of the joint improvement agenda is still in need of further elaboration. Work remains to be done before vulnerable persons can benefit from actual improved social policy practices, i.e. developing and implementing a strategy to achieve a specific targeted change in practice and/or policy (Banks et al., 2017). From a broader interpretation of ‘impact’, in that case we arrive at ‘collective impact’. Until now, in this broader view the outline points to ‘collaborative impact’ (Banks et al., 2017).
Thanks to the application of the impact assessment tool inspired by Banks et al. (2017), we untangled the interplay of even more observable and hidden effects at both group and individual levels, both within and outside the research setting.
Like many others (e.g. Abma et al., 2019; Banks et al., 2017; Morton, 2015; Nierse et al., 2011; Pain, 2015) we highly value the ‘process-based’ impact of the research without ignoring the importance of the research’s substantive findings. Nevertheless, contrary to what we hoped for, with our PAR project we did not “strike a blow” (Pain, 2014), arriving at ‘collective impact’. Moreover, it remains extremely difficult, if not impossible to attribute our research effects to PAR. At most, the concept of ‘contribution’ provides an adequate label as other external factors than PAR context related ones are at play (Morton, 2015). However, we tend to conclude that “walking together” (Pain, 2014) is very rewarding, provided that it follows a well thought through planning, including procedures, methods and impact assessment instruments. After all, remaining vigilant for effects does not automatically mean that researchers are guaranteed to capture all effects. Nevertheless, the undertaking of an energy demanding PAR is worth the effort.
Footnotes
Acknowledgments
The empirical data on which this paper was based is gathered and analysed as part of Tranzo/Academic Collaborative Center Social Work of Tilburg University, commissioned by ContourdeTwern, a social work organization in the city of Tilburg in the Netherlands. The authors would like to thank respondents, co-researchers and other stakeholders for their contributions to the design and conduct of this study.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
