Abstract
This study introduces a mixed-method model for the realistic evaluation of programmes promoting the experience of social inclusion of people in disadvantaged positions. It combines qualitative and quantitative methods for exploring the context-mechanism-outcome- configurations of four cases consisting of development projects. Qualitative analyses depict the context-mechanism-outcome-configurations using participants’ interviews and small success stories as data. Quantitative analyses of a longitudinal survey including the Experiences of Social Inclusion Scale examine the context-mechanism-outcome-configurations in a larger group of participants and re-test the qualitative findings. Thus, they help to overcome the positive selection bias of the small success stories. The mixed-method approach is fruitful especially because the qualitative and the quantitative analyses amend each other’s shortcomings. In the promotion of social inclusion, it is important to help people to see themselves as active agents and allow them to connect to larger social domains.
Keywords
Introduction
Social inclusion has been on the policy agenda in the European Union (EU) and elsewhere for decades, and recently, it has become a targeted outcome of health and welfare programmes (Leemann et al., 2022a; Steward, 2000). In Finland, social inclusion is a central policy goal at the national, regional, and municipal levels. Yet, it is often understood ambiguously (Nousiainen, 2021).
In Finland, one of the largest programmes to promote social inclusion has been the Programme for Sustainable Growth and Jobs 2014–2020. As part of it, the European Social Fund (ESF) and its thematic priority of social inclusion has targeted people in the most disadvantaged positions and at risk of poverty and exclusion, for example, ethnic minorities, immigrants, long-term unemployed and disabled persons. Its objective was to get them on ‘a path supporting working abilities’ and to strengthen social inclusion (www.rakennerahastot.fi, 2014). However, the thematic priority has not specified any concrete programme theory, that is a testable assumption about how, why, for whom and under which circumstances (see Ravn, 2019) the promotion of social inclusion works. Instead, some 500 projects with their own more or less explicitly defined programme theories have been funded.
The objective of this article is to introduce a tangible mixed-method model for the realistic evaluation of interventions aiming at the promotion of social inclusion of disadvantaged people. We seek to show the following: (1) how to use programme participants’ qualitative interviews and their small success stories to depict the context-mechanism-outcome (CMO)-configurations in programmes promoting the experience of social inclusion and (2) how to use the Experiences of Social Inclusion Scale (ESIS) in a longitudinal survey in order to examine outcomes and to test the qualitative findings. We illustrate the operation of the approach with an evaluation of four cases promoting social inclusion within five Finnish development projects funded by the ESF.
The need for mixed methods using both quantitative and qualitative tools for realistic evaluations has been recognized in previous studies (Nousiainen, 2021; Ravn, 2019). While qualitative methods are important for understanding how mechanisms work, statistical intragroup comparisons allow assessing the extent of their impact and the strength of their activation, thus supporting the argument on causal mechanisms (Ravn, 2019). While longitudinal surveys with baseline and follow-up measurements are sometimes used in realistic evaluations (e.g. Kazi et al., 2011), there are only few examples on how these are combined with qualitative analysis (e.g. Martin and Tannenbaum, 2017). This article presents a detailed illustration on how to use a qualitative method to understand how mechanism work, which, according to Bonell et al. (2022), are still relatively few. Furthermore, there are only few realistic evaluations focusing on programmes of social inclusion (e.g. Van der Veken et al., 2020) and more understanding of how they work in different social and cultural contexts is needed.
The experience of social inclusion
We base our understanding of social inclusion on a critical framework emphasizing agency and developed particularly with regard to people in disadvantaged positions (Isola et al., 2017; Leemann et al., 2022a; Steward, 2000). Its theoretical backbone is the capability approach, and the framework emphasizes both macro-level social structures and meso-level social interaction for social inclusion at the individual level, particularly the subjective domain of the experience of social inclusion (Isola et al., 2017; Leemann et al., 2022a). Hence, when we mention social inclusion, we specifically refer to the experience of social inclusion.
Even in welfare states like Finland, prolonged hardships may reduce people’s perceived opportunities, as sociological and psychological studies have shown. Poverty may turn objectives from the long-term to the short-term and frame life with negative aspiration, thus turning it into a day-to-day struggle (Isola et al., 2021). Poverty does not only refer to material deprivation and fewer possibilities but also to lower levels of social contacts and support (Leemann et al., 2022a). It may also prevent people from seizing opportunities (e.g. labour markets) or seeking help (Nousiainen, 2021).
This indicates how disheartening experiences can undermine the psychological and social prerequisites of agency. First, experiences of not being able to control one’s life make life devoid of meaning, thus, weakening the sense of coherence (Antonovsky, 1987: 92). Second, if basic psychological needs for relatedness, competence, and autonomy are thwarted, people may become helpless or unmotivated (Deci and Ryan, 2000: 252). Third, repeated experiences of failure are likely to lower one’s sense of mastery and self-efficacy (Bandura, 1978: 143). This way, the experience of social inclusion refers to conditions for self-determination, motivation, and efficacy: perceived possibilities for action, a sense of competence, autonomy, and meaningfulness as well as supportive relations with others. Therefore, we see the experience of inclusion as crucial for being an active agent in one’s own life and in society.
Previous research also shows that the experience of social inclusion is associated with socioeconomic status and health. In a study among people in vulnerable positions, social inclusion was associated with poverty, various measures of mental health, and wellbeing (Leemann et al., 2022a). A study among the general Finnish population found that respondents with lower education, disadvantaged labour market positions, impaired health, functional or work capacity had a weaker experience of social inclusion and a higher risk for a very weak experience of social inclusion (Leemann et al., 2022b). This illustrates how macro-level structures, such as societal inequality, are reflected at the micro-level experience of inclusion.
Our theoretical framework provides a basis for programme theories for social inclusion, too. While providing opportunities for social and political participation is a predominant element of many social inclusion policies (Silver, 2010), programmes should also focus on the subjective prerequisites of agency. An obvious way to tackle experiences that undermine agency is to provide situations in which people feel themselves meaningful, related, autonomous, and competent. Previous studies have shown that the experience of inclusion may indeed be strengthened by trying new and self-confidence enhancing activities and by caring interaction and recognition, which, in turn, can help to spot new opportunities (Mäntylä, 2020; Nousiainen, 2021). A realistic evaluation of sport-for-development programmes found that meeting with peers and taking responsibility were the key mechanisms for strengthening social inclusion among socially vulnerable groups, whereas the experience and skill of the coaches were important contextual factors (Van der Veken et al., 2020).
In the ESF social inclusion priority, many projects have adopted programme theories that focus on providing people in disadvantaged positions with experiences that strengthen their sense of inclusion. We tested our mixed-method approach with four cases of interventions applying this type of programme theory in five ESF projects. The cases were as follows:
Step-by-step training for working ability in two projects in a Western Finnish town and in a Southern Finnish village. It sought to increase coping skills and self-efficacy for the unemployed through workshops, training and participation in various activities: renovating an old public building, exercising, cooking and outdoor trips. These were supposed to provide participants with experiences of success and gaining a more positive outlook on opportunities regarding employment. The participants were supposed to proceed towards more demanding skills and finally to training to increase their employability.
Civic participation in developing the environment of a disadvantaged suburb in a Northern Finnish city. The case was designed to provide suburb-dwellers with experiences of mastery by being part of decision making and hands-on development work. The case consisted of a project redesigning a widely avoided ‘Troll Forest’ into a ‘Troll Park’ for use by all inhabitants.
The volunteering case was a Lutheran congregation’s project in an Eastern Finnish town that sought to strengthen the volunteers’ experience of social inclusion in the following three ways: (1) providing experiences of meaningfulness through opportunities to influence the work of the congregation, (2) encouraging participants to try new or challenging activities thus providing experiences of mastery, and (3) providing contacts with peers, consequently increasing the sense of relatedness. These were promoted with several volunteer activities, for example, organizing lunch and outdoor events, a young influencers’ group, and creative social media posts.
Arts-based methods were exercised in workshops at a youth guidance centre in a Western Finnish City. The 10-week workshops combined arts exercises and mindfulness. They were intended to increase the participants’ awareness of their way of thinking, facilitate creativity, and encourage them to try new activities. In addition, meeting and interacting with peers was intended to advance social skills and reduce loneliness.
The cases in our study were chosen on the basis of project documents. Once the cases were chosen, the programme theories were specified in discussions with the project staff. (The initial programme theories were rather intuitive sometimes.) The reason for studying several cases was to consider different ways of promoting social inclusion through agency-strengthening experiences. Cases 1 and 4 emphasized providing the participants with experiences of efficacy and mastery in their own lives whereas cases 2 and 3 focused more on providing them with opportunities to be active agents in their local communities.
Since the programme theories of social inclusion rely on subjective experiences, there is no certainty that the participants will respond in the intended way. In our cases, the unpaid development or construction work might be seen as exploitation, the art exercises and group activities as irrelevant tinkering detached from everyday life, or interaction among peers might discourage rather than inspire. Our mixed-method approach examines whether such interventions actually work, for whom and in what circumstances.
The realistic perspective
A premise of realistic evaluation is that every policy (implicitly or explicitly) assumes that in a suitable context, activating certain mechanisms will produce outcomes (Sheaff et al., 2021). Therefore, exploring and understanding the CMO-configurations is central to realistic evaluation. The realistic perspective prioritizes generative views on causality, for example, understanding the transformative potential of phenomena, over successionist views describing constant associations between events or variables (Greenhalgh et al., 2017; Pawson and Tilley, 1997: 32–34).
Outcomes can simply be defined as the changes a programme seeks to produce (Pawson and Tilley, 1997: 74). In our study, the key outcomes were the increases in the experience of social inclusion that resulted from participating in the interventions. Because outcomes are highly dependent on the underlying mechanisms, we can only see those changes in the experience of inclusion as outcomes for which we understand the respective mechanisms and contexts.
Mechanisms are complex conjunctions of action, processes and generative structures that make participants respond accordingly (Sheaff et al., 2021). In other words, mechanisms trigger human responses to interventions and the economic, informational or other resources they provide. These responses involve changes in cognition, emotions, or interaction, thus, altering behaviour (Bonell et al., 2022). Dalkin et al. (2015) define mechanisms as the combinations of resources offered by programmes and the stakeholders’ reasoning in response that will provide desired outcomes. In our study, the project activities are seen as resources that provide participants with possibilities to experience something new. These experiences are assumed to change participants’ reasoning concerning themselves thus strengthening social inclusion.
Contexts are conditions which increase or decrease the outcomes that mechanisms produce, therefore, amplifying or restricting their capacity to work. Contexts may be social, physical or structural and they effect the mechanisms and, thus, differ from the mere settings (e.g. times and places) in which a programme is implemented (Sheaff et al., 2021). Contexts have different layers, and on the individual level the background of the participants, their life-histories or previous experiences can be seen as important contextual factors (Korteniemi, 2005: 20; Ravn, 2019).
In our mixed-method approach, we combined an empathetic understanding of meanings with before-and-after measurements, following Pawson and Tilley’s (1997: xvi) methodological pluralism. We used participants’ semi-structured interviews and the small success stories they told as qualitative data to depict the CMO-configurations of social inclusion. Our quantitative data consisted of a two-round longitudinal survey including the ESIS measuring the experience of social inclusion. The survey allowed us to examine the entity of positive and negative outcomes and to re-test the results of the qualitative analyses. Using alternative sources of data to re-test inferences is an inherent part of realist research practice (Manzano, 2016).
Depicting CMO-configurations with small success stories
The small success stories in our approach are firsthand depictions of singular perceived impacts of social inclusion interventions (see Nousiainen, 2021). As the mechanisms and outcomes in social inclusion programmes are related to subjective experiences they cannot be observed directly. The small success stories make such experiences visible as depicted by the participants. They are narratives on lived experience of programme participation and often elucidate contexts, too.
According to narrative psychology, sense and meaning in life are best grasped through narratives (Hänninen and Koski-Jännes, 1999). In stories and their temporally unfolding plotlines, singular events get their meaning. Stories illuminate human experience and agency by making past action understandable, thus creating conditions for future action (Abma, 1999: 5; Van Wessel, 2018). Stories, therefore, function as cognitive resources through which the environment may be more understandable and predictable (du Preez, 2013: 108).
Moreover, stories articulate interlinking spheres of meaning: emotional, causal, and ethical. Stories have explanatory potential based on their causal meaning, that is, the way individual events are seen to be linked together as chains of causes and effects (Hänninen and Koski-Jännes, 1999). Therefore, stories can be used to detect causal mechanisms. Even if they do not prove the mechanisms with certainty, telling stories reveal how programmes did or did not make an impact according to the lived experience of programme actors (Van Wessel, 2018).
Because narratives are resources for action and have explanatory potential, small success stories can be a valuable method for exploring the CMO-configuration. Small success stories are based on Georgakopoulou’s (2015; Bamberg and Georgakopoulou, 2008) small narratives approach which extends the traditional narrative analysis for examining less elaborate and nonlinear plots telling of mundane or every-day events. Unlike conventional success stories (e.g. Lavinghouze et al., 2007), small success stories focus on the individual and mundane impacts experienced by a programme participant. They form meanings in interaction with the audience and seldom have eloquent and fluently advancing plotlines. Thus, they allow even less elaborate stories to be used as material for narrative analysis.
In our study, we gathered small success stories with 26 semi-structured thematic interviews from all four cases. The interviewees were active project participants who were likely to have positive experiences to share. Therefore, they were thought to be able to describe functioning mechanisms. They were selected based on the project staffs’ recommendations. All interviews were recorded, transcribed and anonymized.
When the interviewees are chosen based on project staff’s recommendations, careful attention should be paid to minimizing social desirability effects. Our interviews did not contain straight-forward questions about programme theories. Instead, we adopted a more indirect approach (see Bonell et al., 2022) to hear firsthand stories and to prevent the interviewees from feeling an urge to support the programme theory in their speech. We did not explicitly ask the interviewees to tell success stories but requested to hear about their experiences in the projects. Furthermore, the interviewers were not involved in the projects and no project workers were present in the interviews. The interviewees were told that the information gathered will be used anonymously for scientific purposes. 1
As typical for the realist interviews, our interview structure was meant to capture the participants stories about the programme (see Greenhalgh et al., 2017). It had five general discussion themes: the participant’s background, entry to the project, personal feelings on participation, and perceived effects. Finally, the interviewees were presented one or two preliminary findings from the survey and asked to explain what might have caused them. The interviews were informal conversations where the participants could talk rather freely about their experiences. This allowed them to take up problems, too. One interview was conducted in English and 22 in Finnish. Three interviews were conducted in Somali by a native Somali-speaking research assistant who also transcribed and translated those interviews into Finnish.
The identification of the small success stories in the data has three stages. First, all fragments containing a depiction of a positive outcome are identified. Clearly, hypothetical or fictitious stories should be omitted because the focus is on firsthand experiences. Then, the stories describing the strengthening of the experience of social inclusion are categorized into outcome patterns. In our study, we found three patterns: 69 meaningfulness stories, 49 relatedness stories, and 79 self-efficacy stories. Finally, causes for the changes are identified. We found altogether 157 stories that mentioned a reason for outcome. Following Manzano (2016), we considered these to depict real activities and processes affecting social inclusion.
The small success stories are used to depict the CMO-configurations of social inclusion interventions. As for the outcomes, the stories depict the impacts the interventions caused in participants and their experience of social inclusion. In our study, we identified three types of outcomes: meaningfulness, self-efficacy and relatedness. The stories of meaningfulness depicted processes that strengthened the participants’ experience of social inclusion through feeling themselves to be important and needed. The stories of self-efficacy described processes where the participants gained more self-confidence helping them to adopt new activities. The stories of relatedness depicted outcomes that provided the participants with new capabilities to cope in social situations and helped them to feel more connected to others. In these stories, the interviewees found a more positive relation to themselves or others which, respectively, enhanced their subjective prerequisites for agency. Sometimes, they described also new activities which were adopted as a result, such as new ways of making art, plans for the future, a part-time job, new hobbies or seeking help (Interviews 12, 21, 23, 24 and 25).
The stories also portrayed the interplay between resources and reasoning which Dalkin et al. (2015) have defined as mechanisms in complex social programmes. The projects provided the participants with possibilities to be part of something important (with the respective autonomy), to see oneself as an active and a respected member of the community, help others, learn new things, feel connected to others in new ways or other means to build supportive relations to others. Experiencing these things facilitated the participants to think more positively about themselves or their future opportunities. Especially experiences of interaction with peers, staff or the surrounding community were present in the stories of all three outcome types. Even if the strengthening of the experience of inclusion occurred at the level of the individual cognition or emotions, it was usually rooted in a larger interpersonal or social domain. Similarly, meeting peers and taking responsibility, have been noticed pivotal in an earlier evaluation of social inclusion among people in vulnerable positions (Van der Veken et al., 2020).
At least two context factors were present in most of the success stories. First, local communities or peer groups emerged as appropriate social or cultural contexts enabling the mechanisms to work (see Pawson and Tilley, 1997: 57; Sheaff et al., 2021). Communities that were willing to accept people in disadvantaged positions as active agents (e.g. volunteers in a local elderly home) were prerequisites for allowing the participants to experience themselves useful or important. Also, groups of peers were frequently mentioned as the contexts of empowering social interaction. Second, an important context factor was the participants’ individual background. The positive changes were often contrasted by challenging personal life situations prior to the interventions, particularly problems of social interaction or feelings of worthlessness.
At their best, small success stories can illustrate all three aspects of a CMO-configuration in a short fragment. We illustrate this with an excerpt of a relatedness story from an arts-based methods case participant:
Quote 1. A: Even if there are different people, and even if in the beginning I thought I would not be with them, that they are in a completely different ‘scene’, I got along with everybody and had fun with everybody. And we were all sincere [–] . . . I just got along with people whom I first thought I would not. I liked people who I thought I would not like because they were so different. Somehow my prejudices were dismantled there when there were strangers who spoke confidentially and profoundly, so somehow it reduced my social anxiety enormously after those visits, because it was like a form of rehabilitation to be with people and to be able to talk and express things and to be sensitive with others [–] Q: Was it easy for you to meet other people earlier? Or difficult? A: No . . . Not at all. I have had a diagnosed social anxiety since primary school, so I did not . . . In fact, previously I did not want to be involved in any of these kinds of things, and I was totally ready . . . to be alone. [–] It is strange, when I started to visit here, suddenly my social anxiety has decreased and I feel like going to places and trying different activities. (Interview 24)
In this story, the depicted outcome meant easing of the social anxiety the participant had suffered for years. During the intervention, the whole prospect of her life changed: before the project she was prepared to live alone, and after it, she found the will to participate in activities involving social interaction. The outcome seems even more dramatic as the interviewee compared the10-week arts-workshops to years of fruitless therapy elsewhere in her interview.
The causal mechanism depicted in the story was the unexpected experience of sincere and profound discussions with participants who the interviewee initially deemed different. This experience dismantled her prejudices towards others and allowed her to feel comfortable in social situations. A crucial part of this experience was being a member in a group making art instead of being an object of the ‘cold’ and ‘academic’ gaze of medical profession, as she also mentioned in her interview. While the change happened in her subjective feelings and thoughts, the project provided the resources, the occasion, the activity, and the place, in which the experience of positive social interaction was realized.
The story also presented at least two contextual factors that appeared as prerequisites for the mechanism. The first was the personal background, the social fears the interviewee has been suffering for years. A participant with different difficulties might have experienced the workshop differently. The other context factor (planned or coincidental) was the presence of peers she thought were different from her. In a more homogeneous group, the experienced effect might not have been so strong.
Paying attention to the details in stories not only helps to illustrate CMO-configurations but also enhances credibility. In Quote 1, for example, the participant depicted the outcome with before and after comparisons. Elsewhere in the interview, she gave a lot of details: how she was now able to walk on the street without anxiety, the health centre where she sought help before and the dance group she was going to attend, for example (Interview 24). Such mentions facilitate a more profound understanding of the occurred impacts and their meaning to the participant.
Obviously, all small success stories are not as detailed and coherent as Quote 1. Most of them are short accounts of an outcome or a mechanism and cannot, therefore, be used to describe the CMO-configurations as such. However, even less illustrative stories are worth noticing as they make it possible to identify patterns of outcomes and mechanisms. In our study, the interviewees brought up many experiences which could be identified as mechanisms strengthening social inclusion. These included the following:
The diversity of mechanisms illustrates the highly subjective nature of the promotion of the experience social inclusion. The interventions work by providing their participants with opportunities to experience something new, but the participants’ reactions always depend on contexts, most clearly on their individual backgrounds. The other apparent context factor is the presence of open local communities or other participants which enabled the participants to see themselves as meaningful parts of a whole, for example, through working for a common purpose. It is also noteworthy, that only one of the identified mechanisms was clearly specific to a certain intervention.
While the participants’ small success stories can evidently depict the diverse underlying mechanisms and offer generative explanations to causality, they also have a ‘positive selection bias’. They do not explain programme failures or produce comprehensive views of positive and negative outcomes. These problems must be tackled with other methods, especially with before-and-after measurements among larger cohorts of programme participants as Pawson and Manzano-Santaella (2012) have suggested.
Examining outcomes and re-testing inferences with a longitudinal survey
The qualitative small success stories require an alternative set of data to shed light on the overall successfulness of the interventions. In addition, the generative argument on causality will greatly benefit of statical associations between phenomena, at least when they point to the same conclusion (Ravn, 2019). In our study, we had the ESIS at our disposal. The ESIS operationalizes the theoretical framework on the experience of inclusion into a survey instrument (Isola et al., 2017; Leemann et al., 2022a). We used it to measure the changes in the experience of social inclusion to analyse how common and how strong the outcomes were. Quantitative data also allow comparing between cases and respondent groups in order to re-test the mechanisms and contexts observed in qualitative analyses.
The ESIS consists of 10 statements on perceived possibilities for action, meaningfulness and relatedness evaluated on a 5-point Likert-type scale (Figure 1). Previous studies have found it to be associated with several vulnerabilities such as unemployment, poverty, ill health and loneliness (Leemann et al., 2022a, 2022b).
The Experiences of Social Inclusion Scale (ESIS).
We targeted our survey at all participants of each case. The first round was conducted at the beginning (baseline data) and the second at the end of a person’s time in the intervention (follow-up data). The surveys were administered by the project staff.
Besides the ESIS, the survey questionnaire should include questions on the participants’ background, such as labour market status (see Table 2). These can be used to group the respondents to examine participants with positive and negative outcomes, that is, the programmes’ ‘winners’ and ‘losers’. Our questionnaire also included seven statements on possible experiences of participation or experienced benefits (see Table 3) which we used to examine if they were statistically associated to outcomes and could work as mechanisms in a larger group of participants, too. The statements were formulated based on theoretical assumptions on possible mechanisms to strengthen inclusion (Isola et al., 2017). In this manner, we could use our longitudinal survey to re-test the inferences of the qualitative analysis.
The data collection was conducted in Finnish and Somali. The data were anonymized, but the respondents could be identified with a unique numerical identifier, so that solitary baseline answers could be deleted and changes observed. Altogether, 118 participants answered the baseline questionnaire and 78 answered our follow-up questionnaire. To assess the overall experience of social inclusion, the sum index was calculated of the 10 items of the ESIS with a range between 0 and 100 points (Leemann et al., 2022a). We allowed one missing value on the ESIS. Missing values were interpolated with the mean of the other nine answers. The variable depicting our main outcome, the change in the experience of inclusion, was calculated through subtracting the respondents’ baseline ESIS score from the follow-up score. The resulting variable had a possible range between –100 and +100. We tested the statistical significance of our observations with
Concerning outcomes, the picture arising from the quantitative analyses was not as positive as the qualitative findings suggested. The participants’ mean ESIS score increased by 2.2 points from 74.4 points at baseline to 76.6 points in the follow-up survey. However, the difference was not statistically significant (
We found altogether 40 participants with an increase and 26 with a decrease in their ESIS scores. Moreover, there was a remarkable variation in the changes of ESIS scores. The strongest increase was 55 points and the strongest decrease 37.5 points. For 16 participants, the increase was substantial, 12.5 points or more, while 12 participants had a substantial decrease. All in all, these findings suggested somewhat positive outcomes as a majority reported a positive change. They also indicated noteworthy negative outcomes, a third of the participants reported a worsened situation, which were not observed in the qualitative analysis. This highlights how necessary it is to supplement success stories with quantitative data.
Comparisons between our cases’ outcomes showed only a few differences (Table 1). All the cases were able to produce a slight improvement in the mean experience of social inclusion, but the changes or the differences between cases were nowhere near statistical significance. Nevertheless, these findings support our qualitative observations suggesting that the cases were rather similar as similar mechanisms often worked in all of them.
Comparison of cases, changes in ESIS scores.
ESIS: experiences of social inclusion scale.
The survey data also allowed re-testing individual-level contexts by comparing respondent groups and examining whether participants’ backgrounds were associated with outcomes. A key background variable in our analysis was labour market status at baseline (see Table 2).
Changes in ESIS score according to labour market status.
ESIS: experiences of social inclusion scale.
The employed participants experienced the most negative outcomes, and they were the only group in which a negative change was found for a majority. The most positive outcomes were found for students and participants in activation measures. Labour market status seemed to be a relevant individual-level context factor since the differences between participant groups approached statistical significance (
The most evident differences were found when examining the outcomes according to the participants’ baseline level of ESIS as a context factor. The outcomes were clearly more positive among participants with baseline ESIS scores below the general adult population in Finland, that is, less than 75 points (
Furthermore, the participants for whom a decline in the ESIS score was found (
These findings confirmed our qualitative finding that the individual background was an important context factor. The interventions clearly were working for participants who had a weaker experience of inclusion at the baseline.
In the final step we used our quantitative data to re-test the observations concerning mechanisms. If the experiences of participating in the interventions functioned as mechanisms strengthening inclusion, the changes in ESIS scores should be associated with more positive opinions about the activities. For this, we compared outcomes among participant groups with positive and negative experiences of participating in the interventions.
The comparisons of respondent groups (Table 3) showed that participants with positive experiences of participation also reported more positive changes in their ESIS scores. Therefore, such experiences may, indeed, be causes of the outcomes. The most unambiguous finding concerned the possibility to help others that was statistically significant and was identified as an explanation for outcomes in the small success stories. Similar findings concerned having one’s views considered, being able to affect the living environment, and having meaningful things to do. However, the disagreeing groups were often disproportionally small leaving these and some other rather substantial differences statistically non-significant.
Change in mean ESIS scores according to the perceived benefits of participation.
ESIS: experiences of social inclusion scale.
Independent samples Mann–Whitney U tests.
Another noteworthy finding in Table 3 is the statistically significant association between free or affordable food and change in ESIS score. Perhaps, free food helps participants financially or eating together may have positive social effects. However, this remains speculative since receiving food was not discussed as a cause of positive outcomes in the interviews.
Explaining failures
Explaining programme failures is another aim of realistic evaluation (Greenhalgh et al., 2017). Our quantitative analyses indicated a weakening of the experience of social inclusion for a third of the participants.
Explaining negative outcomes poses challenges to the small success stories approach. However, allowing interviewees to freely speak about their experiences, instead of explicitly focussing on successes leaves room for discussing problems and failures, too. In our interview data, such accounts were few, apparently because of our sampling criteria. Although some failures were discussed, our study would have benefitted from an additional set of interviews focussing primarily on failure stories.
Like successes, also interventions’ failures were linked to participation experiences. In our interviews, practical difficulties in carrying out the tasks caused negative feelings. A participant of the civic participation case brought up the bureaucratic mentality and ‘lack of common sense’ of city planning officials as a reason for disappointment (Interview 16). Another disappointment was a small number of participants (Interviews 2, 4 and 6).
While constructive interaction with other participants was seen to build the experience of inclusion, conflicts might undermine it. One of our interviewees mentioned that participants with mental illnesses were difficult to get along with (Interview 5). We also had an open-ended question in our questionnaire in which one respondent reported feelings of resentment because of a conflict among participants. Even if interaction among different kind of people was a strengthening experience for participants with mental health troubles (as in Quote 1), it might be experienced differently by others.
The survey data are also useful in explaining failures. The findings reported in Table 3 suggest that if a participant’s opinions are being ignored or when her position in the intervention is restricted solely to being helped, for example, this might increase the likelihood of negative outcomes. Experiencing activities is a highly subjective matter and no mechanism can be expected to work similarly among all participants.
A sound explanation for failures lies in contexts. The relatively high average baseline ESIS score indicated that many participants were not in a disadvantaged position, at least regarding the experience of social inclusion. People with a strong experience of social inclusion at the beginning had more negative outcomes. Moreover, the results from Table 2 suggest that the outcomes were negative for participants in paid employment or pensioners. Hence, one explanation for failure is the projects’ inability to particularly recruit more participants with a weakened experience of social inclusion.
Conclusion
Our mixed-method model of realistic evaluation of social inclusion turned out to be fruitful, especially because the qualitative and the quantitative analyses were able to amend each other’s shortcomings. Qualitative interviews and small success stories depicted the CMO-configurations of social inclusion as experienced by the participants and helped to identify patterns of outcomes and mechanisms. Longitudinal surveys with the ESIS examined the outcomes, contexts and mechanisms among larger groups of participants and helped to overcome the positive selection bias of the small success stories.
The participants’ small success stories offer generative explanations on causality. They often illustrate the interplay between resources and reasoning, that is, mechanisms (see Dalkin et al., 2015) strengthening the experience of social inclusion. These mechanisms work through subjective experiences that provide the participants with reasons to think more positively about themselves and their future opportunities. In our study, the success stories also revealed three patterns of outcomes, that is, ways in which the experience of inclusion was strengthened: the enhancement of self-efficiency, meaningfulness and relatedness. The stories also indicated the most important contexts factors: participants’ individual backgrounds and communities or peer groups. Central to promoting social inclusion is helping people to see themselves as active agents and allowing them to connect to larger social domains, as also Van der Veken et al. (2020) have noted.
When collecting small success stories, the interviewees are chosen among participants who are most likely to have positive experiences to share. They are those stakeholders who can inform evaluators about underlying mechanisms making programmes work. Success stories make big and small successes visible which may, also, encourage the programme practitioners to take part in the evaluation even if they would not have any formal obligation to participate. However, such a positive selection bias is also the biggest pitfall of the small success stories. Interviewees chosen in this way seldom explain programme failures or illuminate negative outcomes. Tackling this problem requires supplementary methods, for example an additional sample of interviews focusing explicitly on failure stories. Furthermore, it is useful to encourage interviewees to bring up also programme failures or problems in interviews.
Another limitation of the small success stories is that they focus on micro-level successes and changes in individual experience – although these are sometimes brought forth by community-level accomplishments. Even if the experience of social inclusion always reflects certain societal structures, participants’ stories are, evidently, less suited for evaluating programmes with macro-level goals.
Quantitative data and changes in ESIS scores provide a comprehensive view of both positive and negative outcomes. According to our survey, a majority of the participants experienced an increase in their experience of social inclusion. Yet, we also found a significant minority (one third) of participants whose ESIS score declined. Our analyses showed that individual backgrounds provided a sound explanation for the interventions’ failures since the participants’ mean ESIS score was rather high already at baseline. Thus, these quantitative findings support our qualitative analyses and confirm that the cases were somewhat successful and that the individual context factors can explain failures also among larger groups of participants.
Quantitative data can also be used to ‘re-test’ the qualitative inferences (see Manzano, 2016). Based on the theoretical understanding concerning the experience of social inclusion (Isola et al., 2017; Leemann et al., 2022a), we were able to include questions on possible mechanisms in our questionnaire. The findings confirmed that positive experiences of participation were sometimes associated with more positive outcomes and could, therefore, work as mechanisms as our qualitative analysis suggested. Although the main point of realistic evaluations lies in the generative understanding of causal tendencies, associations between variables can support or disprove inferences on causality.
A requirement for mixed-method realistic evaluations is that the qualitative and quantitative methods work in concert. Both data should examine the same interventions, participants and outcomes, and be also theoretically compatible. For instance, our argument on the causality would have been even stronger had we been able to link the interviews straightforwardly to the measured changes in ESIS scores on the individual level. This would have enabled us to target the interviews to ‘winners’ and ‘losers’ and compare interviews according to ESIS scores. We did not, however, collect data with personal identifiers due to ethical and data management concerns.
Footnotes
Acknowledgements
The authors would like to thank Dr Anna-Maria Isola for support and Mrs Hodan Mohamed for conducting interviews in Somali.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the European Social Fund as part of the Project to co-ordinate the promotion of social inclusion – Sokra (S20220).
