Abstract
In recent years, there have been increasing demands for public policy design to be guided by empirical evidence, especially in programmes addressing social inclusion. In this scenario, qualitative approaches are often subordinated to the macro results obtained from quantitative instruments. Based on the results of a mixed-methods RCT research design, this article explores how qualitative research tools might provide qualitative significance – information as valuable as numerical data obtained using standardised scales. This proposal is based on the results of a research project addressing problems associated with barriers to employment and mental health among its participants. The impact of gender-sensitive issues – i.e., the unequal distribution of care responsibilities – or the development of mental health problems such as anxiety are often underestimated in studies based purely on statistical data. Our approach underscores the value of the testimonies and narratives obtained through qualitative tools – e.g., semi-structured interviews and focus groups – and how these can provide first-class evidence for the design and impact assessment of public policies tackling social inclusion.
Introduction
A notable effort has been made in recent years to ensure that public policy programmes and interventions are rooted in evidence, in a trend that has come to be known as “evidence-based policy” (Cairney, 2016; Newcomer & Hart, 2021; Sanderson, 2002; Yang et al., 2008). This approach, a development of the 1980s “evidence-based medicine” (Sackett, 1997), argues that decision-makers should base the design and shaping of public policy on rigorously and objectively generated evidence (Cartwirght & Hardie, 2012). This, in turn, required implementing impact assessments of public-funded programmes. Perhaps unsurprisingly, these assessments were based on a paradigm prioritising the generation of quantitative data to support the results obtained, especially in the application and distribution of social welfare policies (Creswell & Poth, 2017; Hendren et al., 2022; Pearce & Raman, 2014).
However, despite this growing interest in including more and better evidence in the design of public policies, there have been significant limitations in the Spanish context – and neighbouring countries – when this viewpoint has been applied to policies aiming to tackle social exclusion (Graziano & Hartlapp, 2019). The evaluations assessing this type of policies have incorporated clinical trials in their design (Plano Clark et al., 2013), focusing on several measurements through the use of numerical coefficients and standardised scales – thus demonstrating the dominance of quantitative approaches in the field (Hendren et al., 2022). However, notwithstanding the importance of macro data, social reality poses a challenge when trying to generate evidence that adequately grasps the different levels of meaning underpinning social actors’ practices in context. The positivist approach upon which evidence-based medicine relies may oversimplify complex social phenomena, neglecting qualitative insights and ethical considerations in evidence-based policy-making.
Indeed, for interventions in the field of social welfare, the procedures proposed within Randomised Controlled Trials (RCTs) would benefit from using methodological tools centred on the analysis of their practices and life experiences (Creswell & Plano Clark, 2011). Qualitative tools have proven effective in reducing bias and providing other types of evidence (Green, 2001; Greene et al., 1989; Mason, 2006; Pluye & Hong, 2014). For instance, empirical materials obtained from qualitative techniques and tools could offer more varied and nuanced textures than quantitative data, allowing a richer, more profound understanding of aspects such as employment inclusion factors, perceived barriers to employment, or narratives around mental health. Therefore, examining the opportunities offered by qualitative tools in RCTs is essential for studies aiming to create evidence in social exclusion research contexts.
This paper aims to reflect on the use of qualitative tools within an RCT (Allen-Perkins et al., 2022), focusing on the Spanish “Plan Vital de Inclusión Social” [Vital Plan for Social Inclusion] (PVIS) 1 This protocol stresses the importance of “qualitative significance” – i.e., the valorisation of qualitative data as first-class evidence in evidence-based studies addressing social exclusion. The aim of this protocol was to design a methodology capable of generating qualitative significance from quantitative evidence. In this text, we defend the potential of qualitative data for impact assessment, beyond a rhetoric based purely on quantification.
Methods
Research Design and Sample
The study was conducted in the Autonomous Community of Extremadura, located in Western Spain and bordering with Portugal. This was one of the regions selected by the Spanish government to implement the PVIS, based on a number of socio-demographic factors – this region has one of the country’s highest rates for indicators such as AROPE 2 (the share of the total population at risk of poverty or social exclusion), which was 38.7% in 2022, and BITH (in Spanish, Low Work Intensity Indicator) 3 (9.9% in 2022). The study was conducted in four of the eight healthcare management areas in which Extremadura is subdivided (Cáceres, Navalmoral, Badajoz, and Zafra-Llerena), a choice that ensured data collection from both rural and urban areas.
The study was conducted in three stages between 01 June 2022 and 30 November 2023: (1) review of relevant literature; (2) data collection – RCT and application of qualitative instruments; and (3) analysis of empirical materials. Although the study protocol initially planned to collect empirical material at three key points (initial/baseline assessment in March, midline assessment in June, and final assessment in October 2023), for reasons beyond the control of the research team, only two quantitative data collections were conducted – between June 2023 and November 2023.
The protocol was based on a mixed-methods research design (Creswell & Plano Clark, 2011), which integrated, on equal terms, measurements of life (collected through qualitative techniques) and measurements of impact (collected through quantitative techniques). The quantitative analysis was based on an RTC with a sample size of n = 707 participants, divided as follows: (G1) social and employment assistance group, n = 171; (G2) healthcare assistance group, n = 185; (G3) PVIS group, n = 185, and (GC) control group, n = 166. Each intervention was assessed using validated, standardised scales. Qualitative data collection methods were also used with 22 participants.
The selection of participants was conducted between 1 and 30 June 2023. Criteria for inclusion were as follows: (1) females residing in the region of Extremadura, aged between 30–40 years, receiving assistance to prevent the risk of poverty and social exclusion; (2) those who voluntarily agreed to participate after signing an informed consent form. The final selection of participants was randomly drawn from those meeting both eligibility criteria. Participation in the study was voluntary.
Sample Size Calculation
After initial screening, a total of 2752 women were identified as eligible to participate in the study. Anticipating a possible small size effect (Cohen’s f = 0.15, with β = 0.80 and α = 0.05), a minimum sample size of 492 women (123 in each group) was required. Due to the possibility of loss during the follow-up phase, the sample was increased by 10%, thus requiring a minimum of 540 women (135 in each group). Finally, 707 women agreed to participate and started the study.
Instruments and Data Collection
Quantitative Tools
Although the initial research protocol included thirteen questionnaires to quantify the impact of interventions in the socio-occupational and healthcare spheres – two scales designed by the research team and eleven standardised and validated scales – in the end, only eight questionnaires were applied: the two ad hoc scales developed by the research team and six standardised and validated scales. The decision not to use all the standardised scales initially considered was based on assessment criteria derived from the indicators considered. The instruments finally applied in the research dealt with aspects related to residential exclusion (author’s own scale), self-perception of barriers to employment (author’s own scale), employability factors (Employability Factors Scale, EFE) (Martínez-Rueda & Galarreta, 2021), expectations during job search (Perceived Control Expectations in Job Search Scale, ECPBE) (Piqueras Gómez et al., 2016), quality of life (Medical Outcomes Study, MOS, 36-Item Health Survey) (McHorney et al., 1993), anxiety levels (Hamilton Anxiety Rating Scale, HAM-A) (Hamilton, 1959), levels of physical activity (International Physical Activity Questionnaire, IPAQ) (Craig et al., 2003), and suicide risk (Plutchik Suicide Risk Scale) (Guirao Goris & Gallud, 2006). The questionnaires were administered by an external multidisciplinary team, using mobile devices. They were not part of the research team but were hired for this purpose and received specific training for correct data collection. A member of the research team monitored the data management process. All the questionnaires and scales were administered in Spanish.
Qualitative Tools
Fieldwork was conducted by members of the research team, all of whom had extensive experience in qualitative research. The qualitative instruments used were semi-structured interviews, a focus group, informal conversations, participant observation, and a field diary (Díaz de Rada, 2011). These techniques allowed us to gather thick descriptions (Geertz, 1973) of employability barriers – structural and perceived – and the impact of long-term unemployment on the participants’ mental health.
Semi-structured interviews (n = 22) examined the study participants’ experiences, perceptions, attitudes, and feelings regarding employment inclusion and job search. The interviews were conducted in spaces where the participants felt comfortable and at ease – in most cases, their own homes. All interviews were conducted in Spanish. The focus group (2 groups, 16 participants) was also led by members of the research team. As with the semi-structured interviews, it tried to include a balanced number of participants from urban and rural contexts. The focus group was conducted in a space provided by the team that carried out the quantitative data collection, where the participants had attended other activities in the past. Participant observation was primarily conducted in the participants’ own homes. During the initial stages of fieldwork, we also conducted informal conversations with participants in this research project and with workers in the healthcare and employment departments of the Junta de Extremadura. These conversations allowed us to identify and adjust the categories of analysis and refine the observation and interview protocols (Swain & King, 2022). Finally, all the empirical material obtained was noted down in a field diary.
Data Collection
Quantitative data was collected in two stages – although the initial protocol established three key points for data collection, these had to be modified for reasons outside of our control. The first data collection phase took place between 1 July–10 August 2023, and consisted of the administration and completion of all questionnaires by the women participating in the survey. The data collected during this phase served as a baseline from which to assess the impact of the actions carried out after the initial data collection phase. The second data collection phase took place between 15–30 November 2023. This second phase began once the intervention phase was completed. The empirical material obtained during this phase was compared with the data obtained during the first phase to analyse the impact of the intervention. The qualitative data were collected during the intervention, between 26 September and 1 Novembre 2023.
Quantitative data were collected using a web application designed for this purpose and developed by the Computer Engineering Department of the University of Extremadura, complying with all the necessary security and connectivity standards. Data collection was carried out by technicians hired by the project for this purpose. A total of 37 people participated in the process. The technicians underwent an intensive training programme developed by the research team. Qualitative data were collected by two members of the research team with expertise in qualitative methodology. Data collection, both quantitative and qualitative, followed the same procedure. The person in charge of administering the questionnaires or conducting the interviews contacted the research participant by telephone to arrange a date. Data collection was carried out in the participant’s home to facilitate their participation and to create a calm and trusting atmosphere.
Data Analysis
Data were triangulated by analysing and comparing all the empirical material obtained. Collected quantitative data were analysed with IBM SPSS statistical software. Qualitative data were analysed diachronically to examine the actual impact of interventions in the participants’ testimonies, and their possible evolution. Data collection and analysis followed current legal frameworks for personal data handling, to ensure protection and confidentiality.
The semi-structured interviews and the focus group discussion were audio-recorded and transcribed. Transcriptions were analysed with ATLAS.ti qualitative analysis software, following an approach based on Grounded Theory (White & Cooper, 2022) that included two levels of analysis: 1) an initial iterative process of adjusting and open coding of categories of analysis based on the emic data gathered during fieldwork, and (2) a second stage of axial coding, grouping, and interpretation of analytical categories. The adjustment during the second stage took into account the main categories of analysis identified in scientific literature and the observation and interview guides developed by the research team. Four research team members analysed the materials following the coding process described above. During the first stage, each researcher coded the empirical material collected. During the second stage, this empirical material was cross-checked by the rest of the research team, generating new interpretations of categories. In case of discrepancies in the analysis, a fifth member of the research team arbitrated the disagreements.
Ethical Aspects
The PVIS has been approved by the Ethics Committee associated with the inclusion programmes implemented by the Spanish Government (Order ISM/208/2022, 10th of March). The study was conducted in accordance with the Declaration of Helsinki and the Belmont Report. Confidentiality of personal data was guaranteed throughout the research process. Data management followed the FAIR guiding principles (Findable, Accessible, Interoperable, Reusable).
The protection of the participants’ data has been a central element of this research. Data management was in line with current Spanish legislation (Organic Law 3/2018, 5th of December, on the Protection of Personal Data and Guarantee of Digital Rights). None of the data collected contained information that could led to the identification of the participants in the study. All records were anonymised. The data collected were uploaded directly to a virtual repository to which only the researchers had access. Audio recordings of interviews were destroyed after transcription. Documents containing information were only accessible to members of the research team and were always stored in secure facilities.
Results
Quantitative Results
Participants’ Socio-Demographic Profile.
Data are expressed as frequencies (percentages), mean±standard deviation, and median (interquartile range).
Abbreviations: IMV, minimum vital income; REG, regional benefit from junta de Extremadura.
Statistical differences between groups (p < .05): ns = no difference between any of the groups; a = differences between control group versus social/employment group; b = differences between control group versus healthcare group; c = differences between control group versus PVIS group; d = differences between social/employment group versus healthcare group; e = differences between social/employment group versus PVIS group; f = differences between healthcare group versus PVIS group.
Among the study participants, 87.4% had completed some level of education. The most common was primary education (38.0%), followed by secondary education (24.2). Only 6.5% of the participants had completed a university degree. A total of 129 participants (18.2%) were currently studying, with secondary education being the most common (14.0%). More than half of the participants reported that they were taking a skills training course (53.5%), and 62.8% of the participants had finished a training course before the start of the research study.
Residential exclusion was analysed using the research team’s own questionnaire. The mean score was 22.76 ± 10.06 out of a 40-point score, where the higher the score, the higher the perceived residential exclusion. No significant differences were observed between the different groups. Only one participant reported being homeless, and another had nowhere to live. In addition, 1.8% of the participants indicated that their housing conditions were unsafe, and 3.7% that they were inadequate. Finally, the mean number of cohabitants with reported substance abuse was 0.11 ± 0.40.
Participants’ Employment and Employability Data
More than half of the study participants were unemployed (58.3%), and 17.1% were engaged in unpaid domestic work. In addition, 28 participants (4.0%) were studying, with significant differences between the social/employment and the PVIS group (1.3% vs 6.5%; p < .05). On the other hand, 27.0% of the study participants stated that they were not available for work. The main reason indicated was caring for children or ill or disabled adults (67.5%), followed by inability to work due to their own illness or disability (13.1%). No statistically significant differences were found between groups.
The mean number of employment offers that participants applied for in the last year was 3.47 ± 12.17 with a median (interquartile range) of 2.00 (1.00–2.00). On the other hand, most participants (96.6%) had attended fewer than five job interviews during the last year. No significant differences between groups were observed regarding the variables described above. Most participants (89.0%) devoted fewer than 10 hours weekly to employment search. The most common method for job search was via the internet or social media (37.2%), followed by public employment services (24.5%) and informal networks (acquaintances) (18.7%). Interestingly, 10.2% of the participants declared having rejected an employment offer during the last year. Finally, almost 29% of the participants were willing to accept an employment offer outside of their place of residence.
Regarding the participants’ employability results, the mean score of self-perceived barriers to employment was 38.96 ± 14.96 out of a maximum of 80 points – where the higher the score, the higher the self-perceived barriers to employment. No significant differences were observed between the different groups. The mean score on the EFE questionnaire was 61.73 ± 21.15. After categorising the scores into degrees of employability, 57.9% of the participants were found to be highly or very highly employable. In addition, statistically significant differences were found in the percentage of participants with medium employability between the PVIS group and those in the control and social/employment groups (28.6% vs 23.5% and 28.6% vs 24.6%; p < .01, respectively), and in participants with very high employability between the control and PVIS groups (25.3% vs 15.7%; p < .05). Finally, expectations of perceived job search control were analysed. No statistically significant differences were found regarding any of the indicators examined – job search self-efficacy, job search success, internal locus of control, and professional experience – or in the total ECPBE score.
Participants’ Healthcare Data
The mean scores for quality of life (and for each of its dimensions) reveal no significant differences between the study groups. The dimensions with the highest mean scores were physical function (87.80 ± 19.87) and physical role (78.33 ± 24.40). In contrast, the dimensions with the lowest scores were declared health evolution (51.17 ± 24.14) and mental health (63.80 ± 23.24). Notably, the mean overall health assessment score was 73.50 ± 17.55 out of 100.
At the same time, 5.8% of the participants reported a change in their dependency status during the last year. Regarding the use of antidepressants and anxiolytics, 13.6% of participants were taking an antidepressant, and 15.6% were taking an anxiolytic. In addition, the consumption of antidepressants was significantly higher in the healthcare group than in the control and social/employment groups (17.3% vs 10,8% and 17.3% vs 9.9%; p < .05, respectively).
The HAM-A scale was used to examine possible anxiety or depression. No significant differences were found in the HAM-A total scores or its subscales. However, almost half of the participants obtained scores compatible with the assessment of the presence of anxiety or symptoms compatible with anxiety. In addition, 22.4% of participants obtained scores compatible with the assessment of the presence of depression or symptoms compatible with depression (Figures 1 and 2). Statistically significant differences were observed between the different groups, with no anxiety identified in the social/employment and healthcare groups (56.1% vs 45.4%; p < .05). HAM-A scores for anxiety and depression in each group. Plutchik suicide risk scale scores.
Finally, Plutchik scale was used to assess suicide risk. The mean score among the study participants was 3.80 ± 3.43 out of a 15-point score, where the higher the score, the higher the suicide risk. As shown in Figure 2, 28.4% of participants presented scores compatible with suicide risk, with those in the control group being significantly lower than those in the social/employment and PVIS groups (21.8% vs 26.9% and 21.8% vs 33.2%; p < .05, respectively).
Qualitative Results
The thematic analysis of the empirical materials collected during fieldwork revealed two main categories regarding the participants’ situation of unemployment and insecurity, which define the construct of barriers for change. The first one is what we have defined as barriers to employment – which could be described as the structural and contextual barriers that hinder the participants’ ability to get or maintain a job, but also those that are self-perceived. The second category is associated with the participants’ mental health and narratives around illness.
Socio-Demographic Characteristics of the Qualitative Research Participants.
Barriers to Employment
The scenario of barriers to employment was shaped by the participants’ material reality, their specific situation in the social structure, and self-perceived barriers. The importance of the first factor was apparent in the analysis of the interview contexts. Many appointments were difficult to set up, with participants describing tight schedules despite being unemployed. On some occasions, the participants arrived in the company of their husbands – a figure exerting both “protection” and “control” – with one participant suggesting that “he could speak better”: “I have shared custody and I am alone, I have to rely on others to look after the child, I cannot move beyond a very close radius because, in the end, with joint custody, I cannot go somewhere else. As long as my son is still young, I can’t, I am very limited in that sense. I work part-time due to my work-life balance.”(#22)
Many of the employment biographies and perceived limitations revolved around the need to care for others, which appeared as a major barrier in different accounts – justified as a problem in accessing employment or even attending job interviews: “I have to be with my baby [...] looking for a job, everything to find a job as soon as I got my residency permit. Well, I looked everywhere and, I don’t know, nothing came out, and in a slip up I got pregnant. Well, those personal things, and I was ready to start doing interviews, and then it’s like, for all I had been rowing, I got stuck again. That’s why I am getting the benefit, because it is one thing to be on your own, but it is different when it is a baby and you; so basically, that is why I did it. But with a child now, it is difficult to find a job.” (#7)
The gender barrier emerged clearly in those contexts where inequalities in the distribution of caregiving roles were limiting access to employment. Some of the barriers in the more complex situations could be due to a combination of different kinds of exclusion – e.g., histories of squatting or serious illness, among others – which created insurmountable hurdles: the possibilities of finding a job were reduced because time and efforts were directed towards caring – for a partner, their children, a house, or their finances. Looking after a dependent partner was described in the following excerpt as a limiting factor – not only to seek employment but even to access programmes that could eventually help in the participant’s employability, such as upskilling or reskilling training courses: “Mi partner has schizophrenia. I cannot leave the house. If I leave the house, it can be much worse because he becomes violent, and then it is impossible to leave the house like this... And then, I cannot find any kind of job.” (#13)
In these testimonies, training appeared as a necessary precondition for access to or improvement of their job situation. In many cases, the participants seemed to assume that they did not have enough training and that their employability would improve if they could receive additional training. Those narratives often dwelt on bad decisions made in the past, sometimes while the participants were still young – decisions that were perceived as having shaped their employment journeys, limiting their current opportunities. In these cases, the lack of educational qualifications was perceived as a barrier that was too difficult to overcome. “I haven’t been able to complete many courses in life, with my situation. It all went very wrong in the transition from school to high school; that was the key. If I could go back, I would study and change many things.” (#19) “I think the most important thing is education. I have not been able to receive any training in my life, and I think that is the most important thing, that has been my problem. What I want is to work, and if I receive the right training, I think it might be easier to find a job doing what I like.” (#1)
As shown in the following excerpt, overqualification could also become a challenge when trying to maintain custody of a child. Again, the role of primary caregivers and the need to look after others appeared as a limiting factor hindering access to employment: “My studies are really limiting because, in the end, in a job, and I do not mind sweeping streets, well, if you have qualifications, they reject you. Sometimes I think it would have been better if I had not studied anything at all. A competitive examination [for public service jobs]? Because you will be thinking: ‘Girl, go for an examination!’ But that is just the same: you need a really good score, and they might send you to a faraway town. And then I would have to give up custody of my son, because I cannot go. That is my barrier.” (#3)
On the other hand, some of the study participants had migrated from different countries and found that their qualifications were not officially recognised in Spain, thus needing to start all over again. Complaints about verification processes were recurrent – the participants had completed their training, but without accreditation, they found it difficult to get employment in their area of expertise. This situation sometimes led them to enrol in training courses without aiming to acquire new skills – the only goal was “to get a piece of paper”: “I had my own apartment, my independence, my job: everything. But well, I came here, and I had to start from zero in everything, except for my partner. It isn’t easy, but when I arrived, I did not have the money to verify my qualifications. I completed my degree, but getting it verified here is money and a lot of bureaucracy, a lot, it takes a very long time. For some people it takes many years to get that, and anyway, it’s not even the time, it’s the money.” (#14)
For migrants, situations of insecurity due to lack of employment were, on occasion, complicated further due to the need to care for younger siblings or other relatives – sometimes while undergoing bureaucratic processes to regularise their situation. In addition, some study participants described racist experiences that limited the range of employment offered, the working conditions – e.g., informal, precarious, or even illegal arrangements – and, sometimes, the withdrawal of job offers. “The problem is when some cases or situations are discriminatory, because yes I have experienced ethnic discrimination. You notice it sometimes, the question of why do I not have the papers if I am qualified – even though I have the skills, because they have seen me doing the stuff, everyone is aware that I have the skills. But they don’t give me the chance, and if I don’t get a chance, how am I going to show them? They haven’t even given me a chance for an interview, not even that.” (#8)
Many of the constructed barriers were perceived as unavoidable. For many participants, the aim of finding employment clashed with the reality of being unable to do it due to barriers limiting their employability and access to new opportunities. Although some participants indicated their willingness to find employment so they could afford certain goods and services, in other cases, their situation of marginality seemed to be accepted as inevitable – the barriers perceived as insurmountable hurdles that blocked their access to employment. “I see on Facebook people saying: ‘Just go and work.’ But not everyone is able to get out and work, I wish. I would love it if my daughter, instead of eating lentils two days in a row, could have some sirloin steak.” (#11)
Mental Health and Employability
Mental health issues, the construction of problems, and their impact on employability emerged in the participants’ testimonies as part of a broad category – suggesting results that often complemented but sometimes contradicted the quantitative scores. However, the context of illness eventually emerged, providing a better understanding of narratives of insecurity and uncertainty. In some cases, our study participants focused the mental health problem on a relative – e.g., someone with addiction problems or severe mental illnesses within the schizophrenia spectrum. “I have been living ten, twelve years with this situation at home, I have tried to find employment. I have looked after elderly women, but I really couldn’t. I tried, but I had to come back to look after him because I did not know how to handle the situation.” (#7)
In this case, the results obtained in the quantitative tests measuring mental health (quality of life scale, Plutchik scale, and HAM-A scale) and the literature on the subject coincide with the participants’ narratives and discourses. For the participants in our study, their mental health was, at the very least, fragile – although most of them would not admit it due to fear of stigmatisation or were not even able to identify what they were experiencing. However, the narratives around the subject of anxiety or the need for medication to cope with their everyday life eventually revealed a stark reality. “I went to see a doctor recently because I was feeling a bit of anxiety and having very bad thoughts. They gave me some pills, like they couldn’t care less. They haven’t taken any notice. The doctor doesn’t take any notice, they don’t look into it, they give me pills, pills, something like Alprazolam, but I cannot remember. They gave me the pill, and it was like saying ‘go home, leave me in peace here.’ They never did a follow-up of anything. To be honest, they make you feel like you are a nuisance. In many occasions they treat you, in my personal experience, badly: you go to the doctor, and they kick you out.” (#13)
More than narratives of resilience, these discourses are a way of verbalising that the participants have limited scope for dealing with these problems, thus ignoring or invisibilising them. Often, the role of primary caregiver made it impossible to focus on self-care – the need to care for others was perceived as more important than the participants’ mental health, which was minimised or seen as a problem that could not be dealt with. “I don’t have time to think about it, because the impact would be great, but you cannot let it impact you. A mother doesn’t have the luxury of falling ill with a cold or something being wrong with your head, of course. And yes, there are times when you are not able to sleep. It doesn’t happen to me anymore because I’ve become numb, but yes, it happens, because suddenly you are on the street and you think, I have a boy [interviewee becoming emotional and holding back tears] That is what I find harder... because that’s it, I have a boy and I find myself on the street and that... But you become numb, and that’s that. It’s normal.” (#2)
When enquiring about coping strategies, some participants mentioned medication but never as a long-term solution – only as a palliative to help cope with a particular situation. However, since the insecurity and precariousness also remain unsolved, the mental health problems did not disappear. This led to narratives built on the rejection of medication, sometimes due to fears of becoming “hooked”: “I think I needed them, but no, I do not like taking those, because I think afterwards the body, a person’s blood, adapts to it. I don’t know, I’m not fond of pills if you can do it naturally. Teach your body to control it, whatever, it’s better to exercise. Of course, if it’s something really serious, then you have to look for – but a natural way of avoiding it is better, because it is very hard.” (#5)
Indeed, some participants focused on the link between both issues, noting that it was difficult not to “end up in a bad way” or remain unaffected. That was particularly true for those participants who acted as primary caregivers for their relatives but also did this professionally – as one of the few options available in the labour market. “A girl I met a year ago has problems and was seeing a psychologist, because it is hard. When I was working as live-in carer the stress was huge, because the person was not well, the person you are looking after, but you have to treat them nicely and if affects you to see them. Looking after a person with Alzheimer is difficult. You end up in a bad way.” (#15)
Many conversations started with a denial of reality, underscoring their resilience and capacity to carry on instead. Eventually, however, different narratives emerge where this ability disappears. Conversations about suicide ideation are challenging, but some testimonies addressed this problem. “Yes, yes... I value life a lot, that is what I have been taught, but I know people... I know a person who took his own life. It was a friend we made when we were looking for jobs. He was also getting this benefit [...] He was Spanish, young, around 35 or 38 years old, we swapped telephone numbers – a friend to chat and all that, he would phone when he was sad, and he would say ‘With this life I won’t be able to... lots of problems, I am going to kill myself.’ At first I got scared and I started saying ‘oh please, don’t say that, you shouldn’t think that way, this is just a tough time, but you’ll get over it, all this will pass, you have to be strong, think differently, think positive, God loves you, everything will work out, just be patient.’ And then he would phone me and ask for money for food or whatever, I would give him whatever I had and tried to talk to see if he would calm down. He didn’t listen to me, he jumped out of a window... He killed himself. And it is a very tough experience, I felt awful, even today, here – the guilty conscience is killing me. Could I have done anything better to avoid that? But I couldn’t, I couldn’t, like he didn’t listen to me and he killed himself, a 38-year-old, 35, with his whole life ahead and he killed himself... And I have something here – what if I had helped him.” (#21)
Discussion
One of the most important perceived barriers emerging from the analysis was lack of time due to the participants’ roles as primary caregivers – looking after children, dependent adults, or partners who were unemployed or had a history of addiction (Heitink et al., 2017; Kimmel, 1998). These structural barriers were clearly identified and described as challenges that had, in the past, hindered employment opportunities. A large proportion of the participants associated difficulties in finding a job with their work-life balance (Matias & Fontaine, 2014; Wattis et al., 2013). Although publications addressing barriers to employment suggest gender as one of the main challenges (Barsoum, 2019; Dolan & Stancanelli, 2021; Ferragina, 2020; Koolwal, 2021; van der Lippe & van Dijk, 2002), this factor was not verbalised as such in the empirical materials obtained. The perception of a causal link between lack of training and unemployment is also a frequent occurrence in the participants’ accounts (Augustine, 2014). Conversely, in some cases, being educated could also be perceived as a problem – in competitive contexts where the candidates might be considered overqualified (Herrera Cuesta, 2017; Shershneva & Fernández Aragón, 2018). Finally, ethnic background is one of the most important barriers to employment mentioned in the literature (Agudelo-Suárez et al., 2009; Midtbøen, 2015).
When enquiring about mental health in a research context, sometimes a certain degree of denial can be observed – perhaps due to the stigmatisation associated with mental illness (Caltaux, 2003; Krupa et al., 2009; Milot, 2019). The need to care for a person in a situation of dependency is again a clear challenge (Crew & Davis, 2006; Gutman et al., 2003; Marwaha & Johnson, 2004; Rosenheck et al., 2006). In other cases, health or mental health problems are described as luxuries that the participants cannot really afford to dwell in (Carolan & de Visser, 2018; Mohr et al., 2010). Unemployment or job insecurity is one of the leading causes of mental health issues (Netto et al., 2016). However, the existence of mental health problems associated with stress or depression is not always verbalised in interviews (Danziger & Seefeldt, 2003; Sano et al., 2011).
The preliminary results of our research offered interesting findings on the subject of data collection. Firstly, the testimonies obtained with qualitative instruments painted a different picture from quantitative data on certain issues. The participants’ testimonies, obtained with discursive tools, provided a more nuanced focus on the particularities of the subjects, thus expanding and highlighting issues that quantitative data can hardly reveal (Green, 2001; Pluye & Hong, 2014).
One example would be the contextualisation of barriers to employment. For instance, the participants’ narratives showed how structural barriers and individual backgrounds affected the daily experience and perceptions associated with the search for employment – e.g., work-life balance or overqualification as an issue rather than a positive factor (Matias & Fontaine, 2014; Shershneva & Fernández Aragón, 2018; Wattis et al., 2013). Notably, qualitative significance emerged in the narratives addressing the participants’ self-perceived employability and expectations of finding a job. Data obtained with the EFE questionnaire suggested that most participants had high or very high employability. However, when asked about the number of job interviews they had attended or the number of hours they spent looking for employment, the participants tended to answer what they assumed was expected from their profiles as active jobseekers – what they thought interviewers, administrative staff, researchers, or policy-makers would want to hear. The number of CVs sent or job interviews attended declared in the questionnaires seemed too high compared to the narratives emerging during the individual interviews or the focus group. These indicated limited access to interviews, often due to the participants having to look after children or other dependents – an issue that was reflected in quantitative data. Although quantitative data showed that expectations of perceived control around job search – e.g., indicators of job search self-efficacy, job search success, internal locus of control, and professional experience – were consistent, variations regarding job search commitment and rejection of employment offers suggested it might be important to consider different individual strategies and decisions as modulators during the search for employment (Bergnehr, 2016; Caliendo et al., 2015; Lindsay, 2010; Pohlan, 2019).
The factors shaping the participants’ strategies and agency included personal preferences and family considerations. However, the testimonies offered about not being available for work were often structured around caring for children, caring for ill, disabled, or elderly adults, or the participants’ own illnesses or disabilities (Crew & Davis, 2006; Heitink et al., 2017; Kimmel, 1998). In contrast to the consistency of the numerical values associated with expectations of perceived control, the interviews suggested a variety of conditioning factors limiting the participants’ ability to schedule interviews and engage more actively in the search for employment. On the one hand, the lack of time and resources linked to their roles as primary caregivers suggest that this is critical in hindering access to employment opportunities – which underscores a shared perception that the labour market gave little consideration for employees’ life-work balance needs. On the other hand, several participants noted the importance of training as an essential factor in improving employability (Augustine, 2014; Barsoum, 2019). However, the responsibilities derived from their caregiving roles again limited their ability to undertake professional development activities, such as training or further education courses.
The narratives constructed around the idea of care underlined the importance of a variable, gender, which did not appear in the quantitative scales but became clear when using research tools prioritising the discursive analysis of the participants’ testimonies. The accounts examined were built around an incapacity to access paid employment, often due to an unequal share of care responsibilities. Thus, gender was expressed as a barrier in the construction of the idea of care – as a role often undertaken by women, which in turn hinders their access to the labour market (Dolan & Stancanelli, 2021; Ferragina, 2020; Koolwal, 2021; van der Lippe & van Dijk, 2002). As noted above, this presented a marked contrast with the results yielded by the scales measuring employability – with almost 57.9% of participants having high or very high employability. As we suggested, this might be due to the participants’ perception that they should provide the answers that were expected from them.
Another issue revealed by the comparison between the quantitative data and the discourses obtained with qualitative tools was the participants’ socio-residential exclusion. Data obtained with the quantitative instruments indicated a very low possibility of the participants having to deal with some residential exclusion or housing problems. Only 3.7% of participants considered that their housing conditions were inadequate, and 1.8% that they were unsafe – contrasting with the Spanish average (9.9% and 4.5%, respectively) (EAPN, 2022, p. 10). The results yielded by these scales did not indicate the existence of participants living under threat of eviction or violence. However, the testimonies offered during the interviews and focus group revealed a number of problems regarding housing that were not reflected in quantitative data. This could be due to different reasons. For instance, it could be that, when answering the questionnaires, the participants took for granted or normalised that some of the housing issues they were experiencing were logical and expected in their context. Therefore, they would not address these problems unless explicitly asked about them in the semi-structured interviews or if they emerged during the focus group discussions. On the other hand, it could be due to a better rapport with the person conducting the interviews than the one administering the questionnaires – something that might happen due to the intimacy created during semi-structured interviews. Finally, it might also be possible that the instrument used for this type of quantitative measurement – the socio-residential scale – does not meet the needs for which it was designed and cannot comprehensively measure these types of problems.
On the other hand, it is essential to emphasise the results obtained regarding mental health. The quantitative results obtained suggested a prevalence of anxiety disorders and depression among the study participants. As noted above, almost half of the participants obtained scores consistent with an assessment of the presence of anxiety or symptoms consistent with anxiety, while 22.4% obtained scores consistent with an assessment of the presence of depression or symptoms consistent with depression. These data contrast with the national averages – 10.4% of the Spanish population show signs or symptoms of anxiety, and 5.9% of Spanish women show signs of depressive disorder. These differences might suggest that there is a social gradient in mental health problems, with a higher prevalence among the unemployed than among the working population (Subdirección General de Información Sanitaria, 2021). However, the review of the participants’ testimonies suggests that their mental health issues were not only a measurable score, but a subjectively perceived experience that affected their search for employment and professional decisions. Some of the participants tended to deny or minimise their mental health problems, possibly due to the associated social stigma (Caltaux, 2003; Krupa et al., 2009; Milot, 2019). At the same time, the qualitative instruments revealed how unemployment and situations of residential exclusion could trigger or aggravate mental health problems (Caliendo et al., 2015; Lindsay, 2010; Thapa & Kumar, 2015). In addition, some of the participants seemly perceived that worrying about their mental health was a “luxury” that they could not indulge in, particularly in situations of economic insecurity or because of their role as primary caregivers (Carolan & de Visser, 2018). Indeed, some of the participants’ testimonies associated mental health problems with the toll of looking after relatives struggling with addictions or severe mental illnesses (Orford et al., 2013). These narratives helped provide qualitative significance to the quantitative scores, highlighting that mental health challenges were closely associated with complex family and personal dynamics.
On the other hand, the data examined also suggested that quantitative methods might provide better insight than qualitative ones on the subject of suicide ideation. The participants’ scores on the questionnaires addressing this issue showed a greater prevalence of suicide ideation among the study participants compared to those who actually verbalised it through the qualitative tools. This avenue is worth exploring further in the future because it shows the potential of quantitative instruments to assess suicide ideation – a subject that, a priori, might appear more suited for a qualitative approach. However, this is a challenging subject to broach in the initial semi-structured interviews and focus groups, at least until a good rapport between researchers and subjects is established (Abbe & Brandon, 2014). Nevertheless, once mutual trust has been established, qualitative instruments could help identify specific triggers and coping strategies – thus contributing to better-designed interventions and psychosocial support programmes.
Limitations
The implementation of different strategies and incentives to encourage participation and adherence to the project faced several limitations. Among these, we can highlight the compulsory nature of participation among the beneficiaries of REG and the absence of compensation. These limitations had a significant impact on the participation and retention of participants during the data collection phase.
Conclusions
Evidence generated with qualitative instruments is very different from quantitative data. Instead of precise statistical values, qualitative instruments provide intersubjectivity and rich, descriptive narratives. In the case of the mixed-methods research protocol examined here, the inductive analysis approach allowed us to review data and establish categories without preconceptions – offering an emic view of the impact of self-perceived barriers on employment search, and the narratives intertwining mental health and (un)employability. The analysis of these two categories revealed a scenario of denial of employment opportunities and refusal to look for work, which the participants justified by structural situations that sometimes appeared as insurmountable barriers and, at other times, were assumed to be normal. In this scenario, according to the participants’ narratives, the impact on their mental health was also inevitable – a barrier that increased, changed, and adapted depending on myriad strategies and personal conditioning factors, thus becoming a major constraint to seeking better employment opportunities.
The results presented here allow us to argue the importance of including qualitative data in impact assessments of public policies – to enrich and provide an essential context for the findings obtained with numerical instruments. In light of the outcomes observed in our research, future interventions developed within the “Plan Vital de Inclusión Social” should consider new categories in their design and implementation. These include perceptions regarding housing, biographical trajectories, the centrality of care, and the impact of quality of life on mental health. Using the participants’ testimonies to guide the design and development of our research protocols will allow us to reach beyond those approaches that understand the qualitative dimension only as a “complement” to the numerical results. Drawing on recent evidence, this study underscores the value of working with mixed-methods RCT models, offering qualitative instruments a central role in the production of evidence and impact assessments and, thus, in the design of public policies.
Supplemental Material
Supplemental Material - Qualitative Significance as First-Class Evidence in the Design and Assessment of Public Policies: The Vital Plan for Social Inclusion in Extremadura, Spain
Supplemental Material for Qualitative Significance as First-Class Evidence in the Design and Assessment of Public Policies: The Vital Plan for Social Inclusion in Extremadura, Spain by Diego Allen-Perkins, Borja Rivero Jiménez, David Conde Caballero, Sergio Rico Martín, and Lorenzo Mariano Juárez in International Journal of Qualitative Methods.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: “Proyecto de estudio inicial de detección de necesidades y diseño e implementación de ivnestigación pre-postest con el grupo de control y de tratamiento del Plan Vital de Inclusión Social en el marco del Plan de Recuperación, Transformación y Resiliencia” is financed by the European Union NextGenerationEU fund.
ORCID iDs
Supplemental Material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
