Sage Journals: Discover world-class research

Abstract

Despite the advice given in evidence-based policy guidelines, intervention programmes incorporating effectiveness assessments are still rare in the Spanish context. To emphasise the potential of qualitative data for impact assessment, this study presents a mixed-methods study protocol evaluating the impact of European Union intervention programmes aimed at improving employability for women at risk of poverty and social exclusion. The protocol for impact assessment is based on a randomised clinical trial of three intervention groups – women with assistance in the social and employment sphere; with healthcare assistance; or with a personalised plan combining both – and a control group, applying validated, standardised scales at three key points during a 12-month period. The protocol also incorporates qualitative techniques to gather thick descriptions of the participants experiences. This proposal underscores the importance of producing qualitative empirical materials – not just as a complementary feature or to illustrate statistical results, but because they allow breaking traditional paradigms and incorporating “qualitative significance” as a tool for effectiveness assessment. The impact assessment of interventions is presented in terms of numerical correlations as well as findings derived from the processes of understanding embedded in the participants’ life narratives.

Keywords

mixed methods qualitative evaluation social justice ethnography whole-systems change

Contributions to Current Understanding

Scientific literature has emphasised the importance of an evidence-based approach to policy-making and impact assessment of public policies (Hendren et al., 2018; 2022; Newcomer & Hart, 2021; Yang et al., 2008). The increasing importance of evidence-based policy-making (Cairney, 2016; Sanderson, 2002) is underpinned by evidence hierarchy proposals developed within biomedical research (Cartwright & Hardie, 2012). Increasingly, public policy assessment design relies on systematic reviews (SR) or projects methodologically conceived as clinical trials (Flemming et al., 2008; Hall & Howard, 2008). This approach has also fostered the development of mixed-methods research designs (Caracelli & Greene, 1997; Greene et al., 1989). It has been suggested that incorporating qualitative data might help improve the validity of the research and reduce bias (Boivin, 2021; Creswell & Plano Clark, 2011; Creswell et al., 2009; Greene et al., 1989; Johnson & Onwuegbuzie, 2004; Mason, 2006; Pluye & Hong, 2014). However, a review of relevant literature on mixed-methods research methods suggests that qualitative approaches appear almost exclusively “embedded” or “nested” within the prevailing framework of Randomised Control Trials (RCTs) (Pearce & Raman, 2014; see also Creswell & Poth, 2017; Hendren et al., 2018; Ospina et al., 2018). Quantitative approaches have long enjoyed an almost absolute, dominant position for the generation of evidence when assessing the effectiveness of intervention programmes. From this viewpoint, qualitative data have only played a secondary role – to provide nuance and help refine the interpretation of quantitative data (Boruch, 1997; Spillane et al., 2010). However, an alternative to the orthodoxy of this evidentialist paradigm, anchored in numbers and statistics, is also possible – assuming that data produced through qualitative research could and should be valued as able to generate first-class evidence on the impact of the actions implemented. Indeed, it is possible to consider that the narratives of personal experiences might provide “qualitative significance” within intervention impact assessments.

This article presents a research protocol design that integrates qualitative and quantitative data with equal value. This research is associated with a Spanish-wide project examining public policies for employment inclusion – the “Plan Vital de Inclusión Social” [Vital Plan for Social Inclusion] (PVIS). The PVIS is a pilot project from the Spanish Ministry of Inclusion, Social Security and Migrations, part of the national Recovery, Transformation and Resilience Plan – a strategy to foster economic growth and create jobs following the COVID-19 crisis whose execution is financed by the Next Generation EU Programme through its Recovery and Resilience Facility. The PVIS aims to improve the employability of women receiving assistance to prevent the risk of them falling into poverty and social exclusion. The novel aspect of the PVIS is that it allows customising interventions, following an individual approach based on a preliminary screening and assessment of the participants’ needs (Table 1). To assess its effectiveness, the PVIS also allows the comparison between four different groups: (G1) group receiving assistance only in the social and employment sphere; (G2) group receiving assistance only in the healthcare sphere; (G3) group benefiting from the PVIS – customised interventions in both spheres; and (GC), a control group.

Table 1.

Interventions in each sphere.

Social and Employment Sphere	Healthcare Sphere
Social assistance	Psychological therapy sessions
Emotional assistance	Help to quit smoking and other substance uses
Customised plans for employment inclusion	Anxiety control
Assistance during job searches	Working on assertiveness and self-esteem
Advice on self-employment and entrepreneurship	Physiotherapy
Intermediation	Postural hygiene
Work–life balance measures	Pilates sessions
Soft-skilling and reskilling	Yoga sessions
Active employability training	Nutrition and lifestyle habits
Other interventions in the social and employment sphere	Nutrition and healthy eating sessions
Help with travelling costs	Oral/dental hygiene sessions
Other training	Hiking sessions
Employment opportunities	Relaxation techniques

The research protocol suggested is based on a mixed-methods, quantitative-qualitative protocol that integrates, with equal value, measurements of life (collected through qualitative techniques) and measurements of impact (collected through quantitative techniques) (Figure 1). The effect of interventions for each one of the four groups considered within the PVIS will be assessed using different data acquisition and analysis methods. The quantitative analysis is based on an RCT with a sample size of 2000 participants, divided as follows: (G1) n = 350 participants; (G2) n = 350 participants; (G3) n = 950 participants; (GC) n = 350 participants. Each intervention will be assessed using validated, standardised scales. Quantitative data acquisition will be conducted at three different points within a 12-month period: an initial measurement at the start of fieldwork (baseline), halfway through it (midline), and once the fieldwork is completed (endline). At the same time, different qualitative techniques will be used to gather data from 10 participants in each of those groups, to obtain thick descriptions of their practices and dynamics during active employment search (n = 40 participants). Techniques will include, among others, informal conversations, note-taking in a field diary, participant observation, and in-depth interviews.

Figure 1.

Research protocol.

All data will be triangulated through the analysis and comparison of the empirical material obtained through both the RCT and the qualitative techniques. Qualitative data will be analysed diachronically to examine the real impact of interventions in the narratives and life experiences of the study participants, and their possible evolution. Data collection and analysis will follow the normative ethical framework for personal data handling, ensuring their protection and confidentiality.

Background

A reduction of the income gap and poverty rates is one of the main priorities for European public policy agendas (Objective 4 of the EU Cohesion Policy for 2021–2027). This has become an even more pressing issue as a result of the COVID-19 pandemic, which has caused widening inequalities in those countries already underperforming (Eurostat, 2022). In the Spanish context, inequalities have historically affected a large percentage of the population due to the limited impact of instruments aimed at economic redistribution, tackling poverty, and supporting vulnerable groups (Homs et al., 2017). Indeed, although the AROPE rate (at-risk-of-poverty or social exclusion) steadily decreased between 2014 and 2019 (from 30.2 to 26.2), in the aftermath of the COVID-19 pandemic this rate increased again (27.8 in 2021) (INE, 2021).

The example of Spain and other neighbouring countries highlights the limitations of social protection measures to tackle poverty and social exclusion (Graziano & Hartlapp, 2019). A wide range of programmes and policies have been advanced to counter this situation, which have been heterogeneous both in their design and targets (Rhodes, 2015). Common elements are fostering employability plans and, more recently, advocating for implementing minimum income programmes (Ståhl & MacEachen, 2021; van Parijs, 2004). In general lines, these plans tend to target social groups disproportionally impacted by social and employment exclusion, such as the young (Dhakal et al., 2018; Kluve et al., 2017), older adults (Anderson et al., 2013), people with disabilities (Lysaght et al., 2012; Morwane et al., 2021; Vornholt et al., 2018), people with mental health problems (Alfozan & Alkahtani, 2021; Charette-Dussault & Corbière, 2019; Gmitroski et al., 2018; Lettieri & Villoria, 2017), people with addictions to drugs and other substances (Sutton et al., 2004) and, significantly, women (Barsoum, 2019; Dolan & Stancanelli, 2021; Ferragina, 2020; Madero-Cabib et al., 2019; Koolwal, 2019; 2021; van der Lippe & van Dijk, 2002). Approaches to problems range from the analysis of social and employment inclusion factors (Dhakal et al., 2018; Wilson, 2006; Shore et al., 2011) to barriers to employment (Anderson et al., 2013; Morwane et al., 2021; Charette-Dussault & Corbière, 2019; Gmitroski et al., 2018; Harmuth et al., 2018) or employability (Lettieri & Villoria, 2017; Alfozan & Alkahtani, 2021; Lysaght et al., 2012; Williams et al., 2016; Vornholt et al., 2018; van der Lippe & van Dijk, 2002).

Impact assessment of these programmes is usually based on comparing the numerical value yielded by a number of indicators – largely macroeconomic statistics – measured before and after the interventions (Innes, 2002). In addition, sometimes assessment protocols incorporate also clinical trials within the design of interventions (Plano Clark et al., 2013). This reliance on measurable entities – numerical coefficients and standardised scales – underscores the dominance of the quantitative approach in research in this field (Hendren et al., 2022).

However, it is clear that processes of randomisation, control, and comparison cannot always grasp the nuances of social reality (Plano Clark et al., 2013; Spillane et al., 2010). A number of research studies have suggested incorporating alternative approaches beyond the quantification of intervention impacts to obtain assessments that are closer to real life and practices (Creswell et al., 2009; Flemming et al., 2008; Kennedy-Martin et al., 2015; Nastasi & Schensul, 2005; Sandelowski, 1996; Schumacher et al., 2005; Song et al., 2010). The protocol presented in this article aims to show how a combination of methods, tools, and approaches can help achieve a more detailed understanding of the complexity of the social processes shaping social and employment inclusion for people participating in these intervention programmes.

Objectives

The main aim of this research protocol is to assess the impact of the different interventions considered within the design of the PVIS programme. In addition, the protocol aims to highlight the potential of qualitative methods and techniques to create qualitative significance within impact assessment procedures generally.

Specific Goals Could Also Include

1. Describing different job search routes for people at risk of poverty and social exclusion.

2. Producing qualitative and quantitative empirical materials on the impact of interventions on specific aspects such as employability skills or the participants’ mental and physical health.

3. Comparing data gathered through different quantitative and qualitative instruments and methods and describing the scope and limitations of approaches focused on the notion of Verstehen as an indicator of effectiveness and efficiency.

Explanation and Justification of the Method

This study protocol is based on a mixed-method research design (Creswell & Plano Clark, 2011). Table 2 summarises the research techniques and empirical materials generated through each approach. The quantitative approach focuses on applying 13 standardised and validated scales, which will be used to quantify the impact of interventions in different spheres. The qualitative approach aims to explore employment inclusion dynamics through the personal experiences and accounts of the participants. This approach includes several techniques, including semi-structured interviews (Kallio et al., 2016), focus groups (Hennink, 2014), and participant observation (DeWalt & DeWalt, 2011) within different observational units – i.e., participants’ homes, institutional spaces, and during job searches. Data will be analysed with attention to the most salient themes emerging from the quantitative scales, informal conversations, observations noted in the field diary, and recordings of interviews and focus groups. This will allow us to identify the main categories articulating the experiences and problems associated with employment inclusion.

Table 2.

Research techniques and resulting empirical materials.

Tool	Empirical Material Obtained
Standardised scales	Results of the administration of each scale
Informal conversations	Verbatim record of informal conversations in the field diary. These will help design the observation and interview guides and adjust the final analytical categories
Participant observation of social and employment practices	Recording observations of the social and physical environment in which interactions are conducted between different agents participating in processes of social and employment inclusion – i.e., appointments with social services, professionals engaged in different inclusion routes, employment services, and possible job interviews
In-depth, semi-structured interviews	Personal responses to experiences, perceptions, attitudes, and meanings emerging in narratives about employment inclusion and job search, following a guide designed by the research team
Field diary	Notes taken down on a field diary. Analysis based on the reflection and interpretation of notes taken during fieldwork – based on observations, informal conversations, and interviews, but with an added element of reflection, contextualisation, and interpretation of the phenomena recorded
Photographic and audiovisual records	Photographs and audiovisual material of the spaces where interactions took place, consumer items, or records of the participants’ physical transformations
Focus group	Recording the participants’ narratives around unemployment and its consequences
Document analysis	Formal document analysis of “narrative service records” to complement personal accounts and narratives of employment

Materials and Methods

Study Period

The study will be conducted between 1 January 2023 and 15 December 2023. During this period, we will review relevant literature on the topic, conduct fieldwork (RCT and application of qualitative instruments), and analyse the empirical materials obtained. Quantitative data will be gathered at three key points: at the start of fieldwork (March 2023), halfway (June 2023), and once it is completed (October 2023). Qualitative data will be collected between March 2023 and October 2023. Two months after completing the fieldwork (December 2023), a report will be submitted, including data, final conclusions, recommendations, and scientific findings.

Scope of the Study

The study will be conducted in the autonomous community of Extremadura, located in Western Spain, along the border with Portugal. This region was selected by the Spanish government as one of the regions for PVIS implementation due to the social and demographic profile of its population. The study will be conducted within four of the eight different healthcare management areas in which the region is subdivided. These have been selected to ensure data collection from both rural and urban areas. The research population will be women between 30 and 45 years of age that receive assistance to prevent the risk of poverty and social exclusion. The sample will include a total of 2000 participants, randomly selected from a total population of 4419 women.

Measuring Instruments

Table 2 summarises the research techniques included in the protocol and the empirical materials generated through each of them.

Quantitative Tools

The protocol includes thirteen questionnaires – two scales designed by the research team and eleven standardised, validated scales – to quantify the impact of interventions in the social/employment and healthcare spheres. These instruments measure levels of residential exclusion (authors’ own scale), self-perception of barriers to employment (authors’ own scale), employability factors (Employability Factors Scale, EFE) (Martínez-Rueda & Galarreta, 2021), expectations during job search (Perceived Control Expectations in Job Search Scale, ECPBE) (Gómez et al., 2016), quality of life (Medical Outcomes Study, MOS, 36-Item Health Survey) (McHorney et al., 1993), anxiety levels (Hamilton Anxiety Rating Scale, HAM-A) (Hamilton, 1959), nicotine dependence (Fagerström Test for Nicotine Dependence, FTND) (Heatherton et al., 1991), motivation for smoking cessation (Richmond test) (Richmond et al., 1993), alcoholism (CAGE questionnaire) (Corrao et al., 1999), severity of alcohol dependence (Severity of Alcohol Dependence Scale, SADQ) (Stockwell et al., 1979), levels of physical activity (International Physical Activity Questionnaire, IPAQ) (Craig et al., 2003), oral/dental health (Oral Health Impact Profile-14) (Slade, 1997), and suicide risk (Plutchik Suicide Risk Scale) (Guirao Goris & Gallud, 2006). A team of social workers and occupational therapists will administer these questionnaires to the 2000 participants, using mobile devices. These professionals are not part of the research team and will receive specific training for correct data gathering. A member of the research team, experienced in statistical data management, will monitor the data management process. The IT system guarantees participants’ anonymity during all stages of the analysis and interpretation process.

Qualitative Tools

The qualitative instruments are those commonly used for ethnographic fieldwork (Díaz de Rada, 2011) and include different tools – i.e., informal conversations, participant observation, in-depth interviews, and notes taken in a field diary. Complementary techniques include photographic and audiovisual recordings, focus groups, and analysis of egodocuments (autobiographical writings). These techniques will allow us to gather thick descriptions (Geertz, 1973) of the barriers (structural and perceived) to access employment, and describe the impact of long-term unemployment on the participants’ mental health. Moreover, we will be able to examine the relationships established between the participants and the institutional environment that shapes their opportunities, and to recognise the strategies of vital support existing within the informal or grey economy – including the solidarity provided by peer, community, and family support networks.

During the initial stages of fieldwork, we will conduct informal conversations with participants in this research project and with workers in the healthcare and employment departments of the Junta de Extremadura. These conversations will allow us to identify and adjust the categories of analysis and refine the observation and interview protocols (Swain & King, 2022).

The observation protocol aims to record interactions between actors in different contexts, from daily life interactions and socialisation in public spaces – i.e., parks, public squares, or leisure environments, among others – to appointments with employment services, attending job interviews, or meetings with social services and other professionals engaged in different inclusion routes. Participant observation will allow us to identify social interaction strategies and practices between the various actors and examine the consequences of the observed social behaviours.

A record of the empirical material obtained during fieldwork will be kept in the field diary. In this diary, we will note informal conversations, observations, and interviews, to describe and help interpret the actions observed from an emic perspective (Díaz de Rada, 2011). Each member of the research team will have their own field diary, and we will also have a shared diary to facilitate interpretation at later stages.

Semi-structured interviews will examine experiences, perceptions, attitudes, and feelings regarding employment inclusion and job searches among the study participants. Interviews will be a conversational dialogue based on an interview guide prepared by the research team. This guide will be adjusted based on the categories of analysis emerging from the review of relevant literature and other empirical materials gathered during our research. A total of 54 interviews with 18 participants will be conducted. The selection of participants for interviews will ensure an equal balance of individuals from both urban and rural contexts (five urban and five rural participants in each group). Before the interviews, each participant will be informed of the study goals and will need to provide written consent. We will aim for interviews to be conducted in places where the participants feel comfortable. All interviews will be audio-recorded. Video recording will only take place if the participants provide specific consent.

Another qualitative tool that will be used for gathering empirical materials is the focus group – one to be conducted at an advanced stage during fieldwork with eight participants (two from each intervention group). With a semi-structured and relatively open debate (Hennink, 2014), the focus group aims to examine participants’ experiences of unemployment and its consequences. We will also explore the participants’ CVs and former employment contracts to complement their narratives. This will allow us to create “narrative service records” describing their previous employment experiences. We will also analyse cover and motivation letters for new job applications, and other types of communications associated with the assistance received. Whenever possible, we will try to obtain photographic and audiovisual recordings during all the research stages in which qualitative techniques are used. The images obtained from the ethnographic fieldwork will be a very useful source of information on the materiality and sociocultural dynamics of the research participants (Brisset Martin, 2004). Audiovisual records will also be used to examine the spaces in which the participants conduct their daily activities – i.e., homes, home furnishings and furniture, or community spaces, among others.

Fieldwork will be conducted by four members of the research team, extensively experienced in qualitative research. A fifth team member will supervise and coordinate the work conducted.

Contributions of Qualitative Methods to RCT

This study protocol aims to illustrate how the use of what we have called “qualitative significance” – all the empirical materials obtained by applying qualitative instruments – can help broaden and enrich the information yielded by quantitative data. We argue that qualitative data in mixed-method research designs are not only valuable as a complement to quantitative data. On the contrary, the variety of available tools and the rich and diverse texture of the empirical materials they help produce means we can use them as first-class evidence.

Qualitative research allows us to examine crucial aspects of employment inclusion and job search – individual experiences, perceptions, attitudes, and attributed meanings – which cannot be fully comprehended through quantitative research. The use of qualitative methods provides “thick descriptions” (Geertz, 1973) of the structural and perceived barriers that hinder access to employment. Moreover, they allow us to describe in detail other issues, such as the consequences of long-term unemployment on the research participants’ mental health. They also help us examine the relationships established between the participants and the institutional environment that plays a crucial role in shaping their opportunities, as well as the support strategies prevailing within informal economy frameworks – in particular, the solidarity provided by peer, neighbour, and family support networks.

Sampling/Recruitment

Participants

Selection of participants will be carried out between 1 January 2023 and 1 March 2023. Criteria for inclusion will be: (1) women residing in the region of Extremadura, aged between 30–40 years, receiving assistance to prevent the risk of poverty and social exclusion; (2) participation is voluntary, after signing the informed consent form.

Recruitment Strategy

National and regional government bodies will be responsible for selecting participants for this research project. The selection will be made on the basis of socio-demographic data managed by government agencies at both levels. The final selection of participants will be randomly drawn from women meeting both eligibility criteria.

Choice of Sample Size

The study population sample for quantitative techniques will include 2000 participants. Qualitative analysis will include a total of 40 participants. We will conduct 54 interviews with 18 participants at three different stages – initial, halfway, and final. It is estimated that this number of interviews will be sufficient to attain thematic saturation – the stage where further interviews reveal no new themes or categories of analysis (Lewis-Beck et al., 2004). We will also conduct a focus group discussion with eight participants. Finally, the analysis of egodocuments will include five participants.

Data Analysis

A mobile device application has been developed to automatise quantitative data collection. This application offers each participant a catalogue of standardised scales that they need to fill in, indicating which ones have already been completed and the dates on which new ones ought to be completed. Once quantitative data collection has been conducted, the results will be analysed with the IBM SPSS statistical software.

Semi-structured interviews and the focus group will be audio-recorded and transcribed. Recordings of the application of other qualitative techniques will also be transcribed. Transcriptions will be analysed with ATLAS.ti qualitative analysis software. Qualitative data analysis will follow an ethnographic approach based on grounded theory (White & Cooper, 2022). This approach will consider two levels of analysis: (1) an initial iterative process of adjusting and open coding of categories of analysis, based on the emic data gathered during fieldwork, and (2) a second stage of axial coding, grouping, and interpretation of analytical categories. For this adjustment, we will consider the main categories of analysis identified in scientific literature and the observation and interview guides designed by the research team. Coding and analysis will include all the empirical material obtained.

Four members of the research team will be responsible for analysing the empirical materials. Following the two-level process described above, during an initial stage each researcher will code the empirical material collected. Then, during the second stage, the rest of the research team will examine this empirical material – suggesting, if necessary, new interpretations of categories. In case of disagreements at this second stage, a fifth member of the research team will arbitrate and help reach a consensus.

Impact assessment for each methodological proposal will follow established standards for analysing and interpreting quantitative and qualitative variables. This protocol aims to offer diverse and complementary ways of conceptualising the impact of interventions. In addition, we have also included a process of continuous exchange between the two types of empirical materials. The indicational review of the quantitative results will allow us to reflect on and adjust the qualitative instruments, improve the understanding of the numerical results, and refine the categories of qualitative analysis. The two processes of analysis are based on the diachronic character of the data, while the validation hypothesis is based on the modification of the results obtained during the generation of empirical materials. By combining these two approaches, the qualitative significance will add accounts and narratives of experiences to the different categories in which the results of the scales would have contemplated only numerical results, thus broadening the fields of meaning for the participants and enriching the analyses proposed.

Rigour

The protocol follows strict criteria of credibility, precision, and replicability of research procedures to guarantee rigour (Fàbregues & Molina-Azorín, 2017).

Credibility

Different triangulation methods will be applied to ensure data quality, precision, and validity. Empirical material will be collected by using different methods and will be analysed by different researchers. Results obtained through the midline RCT will be used to adjust the qualitative instruments. This triangulation will allow us to refine the final analytical categories. Five researchers will participate in this process – four will be responsible for coding and creating categories, and a fifth will act as coordinator and arbiter.

Precision

Due to the application of different data collection techniques, the guarantee of data precision is imperative. All empirical materials obtained will be used in the design of the interview and focus group guides. These materials will allow us to validate the correspondence between categories emerging from coding, theoretical categories guiding the analysis, and data obtained through different methods – including quantitative methods.

Replicability

This protocol is a pilot design focused on a specific Spanish region. To ensure it could be replicated in other scenarios and regions, the methodology described above will be explained in detail in further reports. The protocol follows the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist for reporting results and describing the data collection methods, research design, and analytical procedures developed (Tong et al., 2007). In addition, the research team will work toward ensuring open access to raw data once confidentiality issues are sorted to ensure that participants cannot be identified – complying with transparency and openness criteria for scientific research.

Limitations of the Study

Qualitative assessment of the impact of intervention programmes is not common. This approach values the importance of understanding phenomena, in contrast to the traditional assessment approach based solely on statistics. This requires establishing a viewpoint outside the dominating paradigm; consequently, the possibilities of comparing this study with previously published research are limited.

Other limitations of the study are its temporal limitations, which are determined by funding bodies. Indeed, the timing of measurements might affect the assessment of the programme’s effectiveness – since the impact of some of the interventions might not be appreciable for months or even years into the future. As a result, it might be necessary to consider a further assessment at a later date. On the other hand, given the large size of the study population, a certain level of heterogeneity in the impact of interventions could be expected – e.g. participants engaging with different psychologists or social workers. However, the design of the study and analysis procedures is intended to minimise the influence of different professionals being involved in the intervention programme.

Implications for Research and Dissemination

It has been suggested that the potential of mixed-methods studies might be limited due to the low level of interaction between different methods and the dominance of quantitative approaches (Creswell & Plano Clark, 2011). This study aims to underscore that the importance of qualitative data is not limited to illustrating specific points or statistical results. On the contrary, the inclusion and analysis of accounts of personal experiences can help produce more nuanced impact assessments. This is a novel approach in the field of employment policy impact assessment – where, to date, the contribution of ethnographic studies has been limited. The protocol introduced here, and the research study that will follow, could contribute to shaping the design of future public policies on social and employment inclusion in other regions and contexts. The results obtained could be disseminated in meetings with policy-makers, social workers, and healthcare professionals, both at academic meetings and conferences and those addressed to the general public.

Data Confidentiality

Protection of the participants’ personal data is a central concern in this study protocol. Data collected during fieldwork will be treated in line with the Spanish Organic Law 3/2018 of 5 December on Protection of Personal Data and Guarantee of Digital Rights. Data collected, including via mobile device applications, will not include information that could allow the study participants to be identified. Besides, all data written down in field diaries will be anonymised, including location data. Data collected through mobile applications will not be stored in the devices themselves but cloud-based and will only be accessible to members of the research team. Audiovisual records will be destroyed following transcription. Written documents containing personal information will only be accessible to members of the research team and will be stored in secure locations.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Programa Operativo FEDER Extremadura (2014–2020) y Fondo Europeo Desarrollo Regional (FEDER), grant number: GR21153. The “Plan Vital de Inclusión Social” is financed by the European Union NextGenerationEU fund.

Ethical Approval

The PVIS has been approved by the Ethics Committee associated with the Spanish government inclusion programmes (Order ISM/208/2022, of 10 March). The study will be conducted in accordance with the Declaration of Helsinki and the Belmont Report. Confidentiality of personal data will be guaranteed throughout the investigative process. Data management will follow the FAIR guiding principles (Findable, Accessible, Interoperable, Reusable) for scientific data management.

Informed Consent

All subjects must sign an informed consent form before their inclusion in the study. They will be informed of the characteristics of their participation, research goals, channels for dissemination of research results, and the possibility of expanding on the information provided or ceasing their involvement altogether at any time without fear of consequences. The informed consent form will follow a standardised model from the Ministry of Inclusion, Social Security and Migrations, and will be collected by the research team and professionals involved in administering the instruments. The consent form to be completed will be available in the data collection application. In addition, participants in interviews and the focus group will need to provide oral informed consent at the start of each session. The recording will only start once consent has been indicated, with the footage including the participants providing oral consent.

ORCID iDs

Diego Allen-Perkins

David Conde-Caballero

Borja Rivero Jiménez

Lorenzo Mariano Juárez

References

Alfozan

S. K.

Alkahtani

M. A.

(2021). Barriers to employment for individuals with intellectual disabilities: A systematic review. Journal of Critical Reviews, 8(1), 1016–1028. https://doi.org/10.1002/jid.3659

Anderson

K. A.

Richardson

V. E.

Fields

N. L.

Harootyan

R. A.

(2013). Inclusion or exclusion? Exploring barriers to employment for low-income older adults. Journal of Gerontological Social Work, 56(4), 318–334. https://doi.org/10.1080/01634372.2013.777006

Baron

(2018). A brief history of evidence-based policy. Annals of the American Academy of Political and Social Science, 678(1), 40–50. https://doi.org/10.1177/0002716218763128

Barsoum

(2019). ‘Women, work and family’: Educated women’s employment decisions and social policies in Egypt. Gender, Work and Organization, 26(7), 895–914. https://doi.org/10.1111/gwao.12285

Boivin

J. A.

(2021). The importance of qualitative methods in mixed methods studies. International Journal of Adult Education and Technology, 12(2), 1–17. https://doi.org/10.4018/IJAET.2021040101

Boruch

(1997). Randomized experiments for planning and evaluation: A practical guide. Sage Publications, Inc.

Brisset Martín

D. E.

(2004). Visual anthropology and photographic analysis. Gazeta de Antropología, 20, 1.

Cairney

(2016). The politics of evidence-based policy making. Palgrave Macmillan.

Caracelli

V. J.

Greene

J. C.

(1997). Crafting mixed-method evaluation designs. In Greene

J. C.

Caracelli

(Eds.), Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms (pp. 19–32). Jossey-Bass.

10.

Cartwright

Hardie

(2012). Evidence-based policy. Oxford University Press.

11.

Charette-Dussault

É.

Corbière

(2019). An integrative review of the barriers to job acquisition for people with severe mental illnesses. The Journal of Nervous and Mental Disease, 207(7), 523–537. https://doi.org/10.1097/NMD.0000000000001013

12.

Corrao

Bagnardi

Zambon

Arico

(1999). Exploring the dose-response relationship between alcohol consumption and the risk of several alcohol-related conditions: A meta-analysis. Addiction, 94(10), 1551–1573. https://doi.org/10.1046/j.1360-0443.1999.9410155111.x

13.

Craig

C. L.

Marshall

A. L.

Sjöström

Bauman

A. E.

Booth

M. L.

Ainsworth

B. E.

Pratt

Ekelund

Yngve

Sallis

J. F.

Oja

(2003). International physical activity questionnaire: 12-Country reliability and validity. Medicine and Science in Sports and Exercise, 35(8), 1381–1395. https://doi.org/10.1249/01.MSS.0000078924.61453.FB

14.

Creswell

J. W.

Fetters

M. D.

Plano Clark

V. L.

Morales

(2009). Mixed methods intervention trials. In Mixed methods research for nursing and the health sciences (pp. 161–180). Blackwell Publishing.

15.

Creswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed methods research (2nd ed.). Sage.

16.

Creswell

J. W.

Poth

C. N.

(2017). Qualitative inquiry and research design. Choosing among five approaches. Sage.

17.

DeWalt

K. M.

DeWalt

B. R.

(2011). Participant observation. A guide for fieldworkers. Altamira Press.

18.

Dhakal

S. P.

Connell

Burgess

(2018). Inclusion and work: Addressing the global challenges for youth employment. Equality, Diversity and Inclusion, 37(2), 110–120. https://doi.org/10.1108/EDI-12-2017-0290

19.

Díaz de Rada

(2011). El taller del etnógrafo. Materiales y herramientas de investigación en etnografía. UNED.

20.

Dolan

Stancanelli

(2021). Women’s employment, wages, and the household. Journal of Family and Economic Issues, 42(S1), 101–106. https://doi.org/10.1007/s10834-020-09744-2

21.

Eurostat (2018). Estadísticas sobre distribución de la renta. https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Archive:Income_poverty_statistics/es&oldid=400584#Tasa_y_umbral_de_riesgo_de_pobreza

22.

Eurostat (2022). Persons at risk of poverty or social exclusion by age and sex - EU 2020 strategy. https://ec.europa.eu/eurostat/databrowser/view/ilc_peps01/default/table?lang=en

23.

Fàbregues

Molina-Azorín

J. F.

(2017). Addressing quality in mixed methods research: A review and recommendations for a future agenda. Quality and Quantity, 51(6), 2847–2863. https://doi.org/10.1007/s11135-016-0449-4

24.

Ferragina

(2020). Family policy and women’s employment outcomes in 45 high-income countries: A systematic qualitative review of 238 comparative and national studies. Social Policy and Administration, 54(7), 1016–1066. https://doi.org/10.1111/spol.12584

25.

Flemming

Adamson

Atkin

(2008). Improving the effectiveness of interventions in palliative care: The potential role of qualitative research in enhancing evidence from randomized controlled trials. Palliative Medicine, 22(2), 123–131. https://doi.org/10.1177/0269216307087319

26.

Geertz

(1973). The interpretation of cultures. Basic Books.

27.

Gmitroski

Bradley

Heinemann

Liu

Blanchard

Beck

Mathias

Leon

Barbic

S. P.

(2018). Barriers and facilitators to employment for young adults with mental illness: A scoping review. BMJ Open, 8(12), Article e024487. https://doi.org/10.1136/bmjopen-2018-024487

28.

Gómez

R. P.

Morejón

A. R.

Palenzuela

D. L.

(2016). Validación de una escala de control personal: Una medida específica de las expectativas de control percibido de búsqueda de empleo. Revista de Psicología Del Trabajo y de Las Organizaciones, 32(3), 153–161. https://doi.org/10.1016/j.rpto.2016.06.001

29.

Graziano

Hartlapp

(2019). The end of social Europe? Understanding EU social policy change. Journal of European Public Policy, 26(10), 1484–1501. https://doi.org/10.1080/13501763.2018.1531911

30.

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy, 11(3), 255–274. https://doi.org/10.3102/01623737011003255

31.

Guirao Goris

J. A.

Gallud

(2006). Escala de Riesgo suicida de Plutchik. In Gallud

Guirao Goris

J. A.

(Eds.), Escalas e instrumentos para la valoración en atención domiciliaria (pp. 113–115).

32.

Hall

Howard

(2008). A synergistic approach: Conducting mixed methods research with typological and systemic design considerations. Journal of Mixed Methods Research, 2(3), 248–269. https://doi.org/10.1177/1558689808314622

33.

Hamilton

(1959). The assessment of anxiety states by rating. British Journal of Medical Psychology, 32(1), 50–55. https://doi.org/10.1111/j.2044-8341.1959.tb00467.x

34.

Harmuth

Silletta

Bailey

Adams

Beck

Barbic

S. P.

(2018). Barriers and facilitators to employment for adults with autism: A scoping review. Annals of International Occupational Therapy, 1(1), 31–40. https://doi.org/10.3928/24761222-20180212-01

35.

Heatherton

T. D.

Kozlowski

L. T.

Frecker

R. C.

Fagerstrom

K.-O.

(1991). The Fagerström test for nicotine dependence: A revision of the Fagerström tolerance questionnaire. British Journal of Addiction, 86(9), 1119–1127. https://doi.org/10.1111/j.1360-0443.1991.tb01879.x

36.

Hendren

Luo

Q. E.

Pandey

S. K.

(2018). The state of mixed methods research in public administration and public policy. Public Administration Review, 78(6), 904–916. https://doi.org/10.1111/puar.12981

37.

Hendren

Newcomer

Pandey

S. K.

Smith

Sumner

(2022). How qualitative research methods can be leveraged to strengthen mixed methods research in public policy and public administration? Public administration review. https://doi.org/10.1111/puar.13528

38.

Hennink

M. M.

(2014). Focus groups discussions. In Understanding qualitative research. Oxford University Press.

39.

Homs

Schulz

Vernacotola

(2017). Social and employment policies in Spain -depth analysis.

40.

INE (2021). Tasa de riesgo de pobreza por comunidades autónomas. https://www.ine.es/jaxiT3/Tabla.htm?t=9963

41.

Innes

J. E.

(2002). Knowledge and public policy. The search for meaningful indicators (2nd ed.). Transaction Publishers.

42.

Johnson

R. B.

Onwuegbuzie

A. J.

(2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26. https://doi.org/10.3102/0013189X033007014

43.

Kallio

Pietilä

A. M.

Johnson

Kangasniemi

(2016). Systematic methodological review: Developing a framework for a qualitative semi-structured interview guide, Journal of Advanced Nursing, 72, 2954–2965. Blackwell Publishing Ltd. https://doi.org/10.1111/jan.13031

44.

Kennedy-Martin

Curtis

Faries

Robinson

Johnston

(2015). A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. [Trials Electronic Resource], 16(1), 495. https://doi.org/10.1186/s13063-015-1023-4

45.

Kluve

Puerto

Robalino

Romero

J. M.

Rother

Stöterau

Weidenkaff

Witte

(2017). Interventions to improve the labour market outcomes of youth: A systematic review of training, entrepreneurship promotion, employment services and subsidized employment interventions. Campbell Systematic Reviews, 13(1), 1–288. https://doi.org/10.4073/csr.2017.12

46.

Koolwal

(2019). Improving the measurement of rural women’s employment global momentum and Survey research priorities. http://www.worldbank.org/prwp

47.

Koolwal

(2021). Improving the measurement of rural women’s employment: Global momentum and survey priorities. World Development, 147(1), 105627. https://doi.org/10.1016/j.worlddev.2021.105627

48.

Lettieri

Villoria

E. D.

(2017). A systematization of the international evidence related to labor inclusion barriers and facilitators for people with mental illness. Sociologica, 11(3), 515. https://doi.org/10.2383/89515

49.

Lewis-Beck

M. S.

Bryman

Futing Liao

(2004). Theoretical saturation. In Lewis-Beck

M. S.

Bryman

Futing Liao

(Eds.), The SAGE encyclopedia of social science research methods (pp. 1123–11123). Sage.

50.

Lysaght

Cobigo

Hamilton

(2012). Inclusion as a focus of employment-related research in intellectual disability from 2000 to 2010: A scoping review. Disability and Rehabilitation, 34(16), 1339–1350. https://doi.org/10.3109/09638288.2011.644023

51.

Madero-Cabib

Undurraga

Valenzuela

(2019). How have women’s employment patterns during young adulthood changed in Chile? A cohort study. Longitudinal and Life Course Studies, 10(3), 375–397. https://doi.org/10.1332/175795919X15628474680736

52.

Martínez-Rueda

Galarreta

(2021). La Evaluación de la Empleabilidad en las Empresas de Inserción. Escala de Factores de Empleabilidad (EFE) Gizatea. Manual Profesional.

53.

Mason

(2006). Mixing methods in a qualitatively driven way. Qualitative Research, 6(1), 9–25. https://doi.org/10.1177/1468794106058866

54.

McHorney

C. A.

Ware

J. E.

Raczek

A. E.

(1993). The MOS 36-item short-form health survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Medical Care, 31(3), 247–263. https://doi.org/10.1097/00005650-199303000-00006

55.

Morwane

R. E.

Dada

Bornman

(2021). Barriers to and facilitators of employment of persons with disabilities in low- and middleincome countries: A scoping review. African Journal of Disability, 10, 1–12. https://doi.org/10.4102/AJOD.V10I0.833

56.

Murad

Asi

Alsawas

Alahdab

(2016). New evidence pyramid. Evidence-based Medicine, 21(4), 125–127. https://doi.org/10.1136/ebmed

57.

Nastasi

B. K.

Schensul

S. L.

(2005). Contributions of qualitative research to the validity of intervention research. Journal of School Psychology, 43(3), 177–195. https://doi.org/10.1016/j.jsp.2005.04.003

58.

Newcomer

Hart

(2021). Evidence-building and evaluation in government. Sage.

59.

Ospina

S. M.

Esteve

Lee

(2018). Assessing qualitative studies in public administration research. Public Administration Review, 78(4), 593–605. https://doi.org/10.1111/puar.12837

60.

Pearce

Raman

(2014). The new randomised controlled trials (RCT) movement in public policy: Challenges of epistemic governance. Policy Sciences, 47(4), 387–402. https://doi.org/10.1007/s11077-014-9208-3

61.

Plano Clark

V. L.

Schumacher

West

Edrington

Dunn

L. B.

Harzstark

Melisko

Rabow

M. W.

Swift

P. S.

Miaskowski

(2013). Practices for embedding an interpretive qualitative approach within a randomized clinical trial. Journal of Mixed Methods Research, 7(3), 219–242. https://doi.org/10.1177/1558689812474372

62.

Pluye

Hong

Q. N.

(2014). Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews. Annual Review of Public Health, 35(1), 29–45. https://doi.org/10.1146/annurev-publhealth-032013-182440

63.

Rhodes

(2015). Employment policy. Between Efficacy and Experimentation. In Wallace

Pollack

M. A.

Young

(Eds.), Policy-making in the European union (pp. 294–318). Oxford University Press.

64.

Richmond

R. L.

Kehoe

L. A.

Webster

I. W.

(1993). Multivariate models for predicting abstention following intervention to stop smoking by general practitioners. Addiction, 88(8), 1127–1135. https://doi.org/10.1111/j.1360-0443.1993.tb02132.x

65.

Sandelowski

(1996). Focus on qualitative methods using qualitative methods in intervention studies. Research in Nursing & Health, 19(4), 359–364. https://doi.org/10.1002/(sici)1098-240x(199608)19:4<359::aid-nur9>3.0.co;2-h

66.

Sanderson

(2002). Evaluation, policy learning and evidence-based policy making. Public Administration, 80(1), 1–22. https://doi.org/10.1111/1467-9299.00292

67.

Shore

L. M.

Randel

A. E.

Chung

B. G.

Dean

M. A.

Ehrhart

K. H.

Singh

(2011). Inclusion and diversity in work groups: A review and model for future research. Journal of Management, 37(4), 1262–1289. https://doi.org/10.1177/0149206310385943

68.

Slade

G. D.

(1997). Derivation and validation of a short-form oral health impact profile. Community Dentistry and Oral Epidemiology, 25(4), 284–290. https://doi.org/10.1111/j.1600-0528.1997.tb00941.x

69.

Song

M.-K.

Sandelowski

Happ

M. B.

(2010). Current practices and emerging trends in conducting mixed methods intervention studies in the health sciences. In Tashakkori

Teddlie

(Eds.), SAGE handbook of mixed methods in social & behavioral research (pp. 725–748). Sage. https://doi.org/10.4135/9781506335193.n28

70.

Spillane

J. P.

Pareja

A. S.

Dorner

Barnes

May

Huff

Camburn

(2010). Mixing methods in randomized controlled trials (RCTs): Validation, contextualization, triangulation, and control. Educational Assessment, Evaluation and Accountability, 22(1), 5–28. https://doi.org/10.1007/s11092-009-9089-8

71.

Ståhl

MacEachen

(2021). Universal basic income as a policy response to COVID-19 and precarious employment: Potential impacts on rehabilitation and return-to-work. Journal of Occupational Rehabilitation, 31(1), 3–6. https://doi.org/10.1007/s10926-020-09923-w

72.

Stockwell

Hodgson

Edwards

Taylor

Rankin

(1979). The development of a questionnaire to measure severity of alcohol dependence. British Journal of Addiction to Alcohol & Other Drugs, 74(1), 79–87. https://doi.org/10.1111/j.1360-0443.1979.tb02415.x

73.

Sutton

Heaver

Cebulla

Smith

(2004). Drug and alcohol use as barriers to employment: A review of the literature. Loughborough University.

74.

Swain

King

(2022). Using informal conversations in qualitative research. International Journal of Qualitative Methods, 21, 160940692210850. https://doi.org/10.1177/16094069221085056

75.

Tong

Sainsbury

Craig

(2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19(6), 349–357. https://doi.org/10.1093/intqhc/mzm042

76.

van Parijs

(2004). Basic income: A simple and powerful idea for the twenty-first century. Politics and Society, 32(1), 7–39. https://doi.org/10.1177/0032329203261095

77.

van der Lippe

van Dijk

(2002). Comparative research on women’s employment. Annual Review of Sociology, 28(1), 221–241. https://doi.org/10.1146/annurev.soc.28.110601.140833

78.

Vornholt

Villotti

Muschalla

Bauer

Colella

Zijlstra

van Ruitenbeek

Uitdewilligen

Corbière

(2018). Disability and employment–overview and highlights. European Journal of Work and Organizational Psychology, 27(1), 40–55. https://doi.org/10.1080/1359432X.2017.1387536

79.

White

R. E.

Cooper

(2022). Grounded theory. In White

R. E.

Cooper

(Eds.), Qualitative research in the post-modern era (pp. 339–385). Springer. https://doi.org/10.1007/978-3-030-85124-8_9.

80.

Williams

Dodd

L. J.

Steele

Randall

(2016). A systematic review of current understandings of employability. Journal of Education and Work, 29(8), 877–901. https://doi.org/10.1080/13639080.2015.1102210

81.

Wilson

(2006). Developing a model for the measurement of social inclusion and social capital in regional Australia. Social Indicators Research, 75(3), 335–360. https://doi.org/10.1007/s11205-004-2527-6

82.

Yang

Zhang

Holzer

(2008). Dealing with multiple paradigms in public administration research. In Yang

Miller

(Eds.), Handbook of research methods in public administration (pp. 25–44). CRC Press.

Employment Inclusion Policies and Evidence-Based Policy: Incorporating “Qualitative Significance” Into a Mixed-Methods Study Protocol

Abstract

Keywords

Contributions to Current Understanding

Background

Objectives

Specific Goals Could Also Include

Explanation and Justification of the Method

Materials and Methods

Study Period

Scope of the Study

Measuring Instruments

Quantitative Tools

Qualitative Tools

Contributions of Qualitative Methods to RCT

Sampling/Recruitment

Participants

Recruitment Strategy

Choice of Sample Size

Data Analysis

Rigour

Credibility

Precision

Replicability

Limitations of the Study

Implications for Research and Dissemination

Data Confidentiality

Footnotes

Declaration of Conflicting Interests

Funding

Ethical Approval

Informed Consent

ORCID iDs

References