Policy Feedback & Research Methods: How Qualitative Research Designs With Marginalized Groups Inform Theory

Abstract

As a body of literature that informs and analyzes the effects of policy, policy feedback research will benefit from more qualitative research designs with marginalized populations. Qualitative research designs allow for theoretical and conceptual development, as well as ensure multiple perspectives are fully captured. This article argues that qualitative methods with different populations will expand applications of policy feedback theory and lead to critical insights for its development. Three examples of potential areas of theory development are outlined including informing the variable of participation, examining more nuanced effects of specific policy elements, and understanding the interactions of multiple policies. The article concludes with the suggestion of more mixed-method designs utilizing qualitative approaches involving participants to test existing hypotheses and inform new relationships. Policy feedback theory raises vital questions of democratic legitimacy and policy effectiveness that benefit from research conducted with, and informed by, marginalized populations.

Introduction

Research designs are vital to theory development. They determine the findings of studies and can be structured to generate, test, and further develop theories. Research designs and methodological approaches, therefore, require specific investigation to determine the utility of existing inquiries, and where and how they can be diversified for theoretical development.

This article argues that policy feedback theory examining political participation stands to gain important theoretical insights from qualitative inquiry that includes more direct involvement with marginalized populations. Although the benefits of mixed method studies, including longitudinal case studies and statistical analysis, are evident, the theory requires perspectives best captured through additional qualitative methods with different groups. Such applications serve to modify and develop the theory and can inform the understanding of the relationship between policy and engagement.

The methods outlined in this article include qualitative methods that allow for lived experiences of populations to be more expansively included. There is no attempt, therefore, to offer a single definition of qualitative methods. Rather, the article considers the utility of qualitative data collection with marginalized populations. Methods that allow for the inclusion of more perspectives from the bottom-up – with their lived experiences directly included in the research – are of particular focus (Michener et al., 2020). Such methods include interviews, focus groups, ethnography, participant observation, community engaged research, participatory action research, among others. ¹ It is argued that qualitative data provides leverage to theory development and can inform the definition of variables and refining of relationships and hypotheses. Drawing not only from theory but also from the data ensures the lived experiences of populations are included in the variables and relationships of interest (Timmermans & Tavory, 2012).

The purpose of this article is to consider how policy feedback can benefit from research designs with different populations through qualitative methods. Traditionally, the focus on the feedback effects on political engagement and participation has included government elites, interest groups, and mass publics (see for example Campbell, 2003; Pierson, 1993). Although there are various strands of policy feedback research that include different levels of analysis, this article focuses on studies of mass publics. That is, the study of everyday people and beneficiaries of policies (SoRelle & Michener, 2022). Research examining mass publics often considers the meaning of citizenship and the power of groups, with examinations of the effects of benefits on political participation (SoRelle & Michener, 2022). This article argues that studies of mass publics – and those examining political participation in particular – stand to benefit from studies with vulnerable populations that include more qualitative research designs. There is a need to expand beyond the traditional quantitative mass public investigations and case study historical approaches to better incorporate perceptions and experiences of different groups.

To begin, the article introduces existing policy feedback literature to exhibit the need for more qualitative research with marginalized populations. The discussion then turns to recent calls for further theory development and how qualitative inquiries serve future research. Here, the focus turns to three main threads of theory development: defining participation, considering multiple policy elements, and disentangling complex effects. Qualitative investigation with multiple populations allows for the consideration of nuances in experiences with policies and different forms of participation, in addition to the complexity of effects. The article concludes with a call for more qualitative and, where possible, mixed-method research on policy feedback that include lived experiences with the welfare state.

Qualitative inquiries are important to understanding meaning, context, and processes, contributing nuance and research for public policy (Maxwell, 2020). In addition to the increasing calls for qualitative inquiry in public administration research (see for example Ospina et al., 2017; Zhu et al., 2019), more qualitative approaches have been identified as beneficial to understanding policy problems and relationships (Maxwell, 2020). As a theory that examines elements of democracy that contribute to discussions of public policy and political science – and has vital implications for policymakers and public servants – examining the methodological approaches utilized by policy feedback scholars is imperative. Engaging in these discussions ensures the theory continues to develop and has far reaching applications with more nuanced understandings.

Existing Policy Feedback Literature

Policy feedback literature has evolved and has come to include different threads, levels of analysis, and variables of interest. Early policy feedback literature included the examination of institutional processes over time, and examinations of state capacities, interest groups, and lock-in effects (Béland et al., 2022). The historical institutionalist origins of policy feedback theory point to the benefits of deep case studies and longitudinal investigations, as well as associated limitations (see for example Skocpol, 1992). Case study methods offer thick descriptions of causal mechanisms and important narrative accounts to descriptive statistics (Byrne et al., 2012). They allow for the consideration of historical cases and rare events, as well as omitted variables and interaction effects within cases (Bennet & Elman, 2006). Policy feedback scholars have utilized case study approaches to trace the effects of social welfare changes on politics through mass publics (see for example Campbell & Morgan, 2005). Often, case studies include quantitative survey analysis. This has usually included a mix of cross-sectional and longitudinal surveys, utilizing existing mass data from multiple surveys. In some cases, scholars have created their own surveys (see for example Gidengil, 2020; Mettler, 2005).

These research designs allow for the testing of the effects of policy designs on democratic participation. They contribute to our understanding of the effects of policies and their influence on social stratification, civic engagement, political mobilization, institutional trust, and perceptions of the state, as well as individual civic and political capacities (see for example Bruch et al., 2010; Kumlin, 2004; Watson, 2015). Policies are in fact seen as influencing politics, shaping how citizenship is conceived, government capacity, the power of groups, policy problem definition, and political agendas (Mettler & SoRelle, 2014; Pierson, 2000).

Existing research considers how policies provide resources and affect ideas that then shape the political process and involvement (Campbell, 2012). There are important relationships between policies and participation. Not only do policies distribute material goods, but they also have resource and interpretive effects. Policies dictate the resources one receives that influence one’s ability and willingness to participate politically, and also send messages regarding one’s place in society (Campbell, 2003; Pierson, 1993). They can, in turn, increase engagement from their target populations, which can lead to policy change (Campbell, 2003; Mettler, 2005). At other times, interactions with certain policies and institutions lead to certain populations disengaging from politics altogether (Weaver & Lerman, 2010). Many feedback studies examining participation include case study approaches with quantitative methodologies, although in some studies additional qualitative methods have also been used.

Mass public literature offers some helpful conditions of when feedback occurs (see for example Soss & Schram, 2007), but there is a need to consider feedback effects with different populations. Michener et al. (2020) present a bottom-up approach, which they argue includes inquiries not only from the perspective of policymakers and elite institutions but also from “…those at the proverbial “bottom,” who face substantial disadvantages within contemporary social structures and navigate a distinct constellation of public policies as they attempt to secure their welfare” (p. 155). Populations that are relatively easy to access are often those included in traditional surveys and existing research. The underrepresented – those often excluded from politics – are also often excluded from research (Gallegos et al., 2023). There is a need therefore, to interrogate research design choices and their associated limitations to consider the extent to which some populations may be excluded from research. In fact, existing mass public studies have pointed to limitations of research designs and called for closer examinations with more marginalized groups, along with more qualitative inquiry.

In her study of social security, Andrea Campbell (2003) examines the participation of senior citizens in the US. Campbell shows that even when programs for other groups – namely, the poor – were cut, senior citizens were able to guard certain programs from retrenchment. Campbell shows these effects using aggregated large N surveys. However, as Campbell herself argues in the conclusion, seniors receive more from the government than other groups, particularly those that participate less (Campbell, 2003, p. 146). This points to the need for examinations with more marginalized groups and the consideration of how such populations may or may not be represented in existing data.

Similarly, in her investigation of the effects of the training and education provisions of the G.I. Bill on veteran democratic participation, Suzanne Mettler (2005), also utilizes survey data. Her analysis shows positive resource and cognitive effects of policy design, where the experiences of veterans as “first-class” citizens through the program led to increased rates in civic participation (Mettler, 2005, p. 10). Like Campbell (2003)’s study, Mettler adds vital considerations of resource and interpretive effects of policy designs. Mettler, however, supplemented the historical analysis and survey data with interviews. The interviews allowed Mettler to examine veteran’s experiences with the Bill in a more in-depth manner and within the context of their lives (p.177). Although the study includes some qualitative methods, Mettler makes a note of the limits associated with the subgroups examined, and in particular, “second-class” citizens, illustrating once more the limitations associated with capturing the experiences of marginalized groups (Mettler, 2005).

Policy feedback scholars that examine marginalized groups, furthermore, point to the limitations associated with data availability, which directly speaks to the need for more qualitative research with marginalized populations. Weaver and Lerman (2010) examine the relationship between criminal justice institutions and democratic participation among a marginalized population. They utilize surveys to add punitive activity of the state to the institutions considered by policy feedback scholars. They find that contact with the criminal justice system decreased political participation. Weaver and Lerman (2010) conclude by pointing to important limitations of surveys themselves, arguing that “… the potential to pursue this type of analysis will be limited until surveys of American politics begin incorporating items that query respondents about their contact with the criminal justice system” (Weaver & Lerman, 2010, p. 831). They go on to suggest that future research needs to consider community level dimensions, the shaping of narratives, and differences associated with some types of criminal justice contacts and political behavior (Weaver & Lerman, 2010). Qualitative inquiry is well positioned to expand upon the work done by Weaver and Lerman and further explore the effects of the criminal justice system on participation, particularly in the ways the authors themselves highlight. More community level considerations and the inclusion of narratives and heterogeneity will benefit from ethnographic and more in-depth interview and observatory methods.

As Weaver and Lerman (2010) highlight, the experiences of marginalized groups may be excluded from existing data. Different contexts, furthermore, may also have limitations in data available which can further limit comparative studies and examinations in different contexts. Some contexts, for example, may have different forms of data collection on specific political elements of interest, including how policies are administered (SoRelle & Michener, 2022). In the US context – the context most often studied in policy feedback – there may be an abundance of data available to test policy feedback mechanisms. This may not be the case in other countries. There may be even greater variation in relation to marginalized groups. There has been, for example, a call for the inclusion of race and ethnicity in studies of the effects of policies (Béland et al., 2022; Michener, 2019). For some countries, such as Canada and Australia, however, there is a lack of race-based data available for such examinations using survey and statistical analyses (see for example Owusu-Bempah & Luscombe, 2021; Renzaho, 2023). This in turn, limits comparative inter-country quantitative analyses, which are not commonly found in policy feedback literature. ² Advances in technology, data storage, and survey analyses across countries will undoubtedly lead to the proliferation of comparative longitudinal policy feedback studies some have called for (Bussi et al., 2022).

More qualitative research designs will contribute to the proliferation of comparative studies. Data collection using various qualitative methods, such as ethnography, archival research, interviews, and observation, can develop novel hypotheses with new variables (Mahoney, 2007). Comparative qualitative designs allow for theory testing and the testing of multiple implications, and the consideration of intervening mechanisms and homogeneity, and new hypotheses (Mahoney, 2007). Comparisons can also help sharpen measurement, allow for the exploration of the prevalence of causal capacity, and provide raw material that allow for the construction of theories (Seawright, 2018). Bloemrad (2006) uses in-depth interviews with immigrants and refugees along with quantitative methods, to examine the integration and political participation of immigrants in Canada and the United States. Interviews allowed for the tracing of how immigrants learned about naturalization processes in more detail along with their feelings and knowledge of citizenship (Bloemraad, 2006).

In addition to capturing more experiences and increasing comparative research, qualitative data may also provide necessary nuance and understanding to the study of feedback effects. Gidengil (2020) provides one of few studies of feedback in Canada using survey data to examine the effects of social assistance policies on political engagement. In doing so, Gidengil points to the limitations of the study design illustrating the lack of inclusion of marginalized populations once more: “We cannot rule out the possibility that the survey underrepresents the truly marginalized and disempowered” (p. 188–189). Gidengil (2020) goes on to argue for more qualitative approaches to “enrich our understanding of feedback effects” (p. 192). The need for qualitative approaches, Gidengil emphasizes, can assist in our full understanding of the effects from the perspective of those interacting with street-level bureaucrats and how these experiences relate to their perceptions of politics. It can also ensure the inclusion of different groups. Focus groups, semi-structured interviews, and ethnographic and participatory methods can enrich our understanding of feedback effects (Gidengil, 2020).

At times, qualitative data can better capture frustrations with government and various services (Soss, 2002). Although surveys capture responses to questions, they do not allow for more open-ended considerations of experiences participating or experiences with policies that capture specific effects. In a recent examination of the effects of policies on lone mothers’ political engagement, for example, Shore (2020) concludes by citing the lack of lived experiences captured in the survey data and calls on more qualitative studies with lone mothers.

Qualitative methods can offer thick descriptions that inform relationships. In his examination of the political dimensions of welfare participation through SSDI and AFDC, Soss (2002) conducts ethnography, participant observation, and interviews (as well as quantitative survey data analysis). In justifying his methodological choices, Soss argues that interviews allowed for more depth and validity in findings, a bottom-up approach that captured participant experiences, various data sources including interactions within and outside offices, and direct contact with day-to-day welfare participation (Soss, 2002). Soss (2002) also outlines the difficulty of obtaining survey data regarding welfare participation. Qualitative inquiry, therefore, is necessary to capture thick descriptions of experiences with policies, and in particular, the experiences of marginalized groups. In a study that not only includes qualitative methods (among others), but also examines the perspective of marginalized populations – namely poor Black and Latino groups – Michener (2018), considers the role of Medicaid design and implementation influenced how these populations interpreted government and participated. Using qualitative interviews, Michener (2018) provides a close investigation of policy advocacy. Studies utilizing more qualitative methods are often examining marginalized populations, showing the need for more of such studies and highlighting their contributions.

Understanding how feedback effects occur among multiple different populations can expand our understanding of effects and their translation across groups and contexts. It can also strengthen the contributions of policy feedback to democracy literature. Marginalized populations may experience different effects and participate in different ways not already captured in existing research. By understanding how different groups participate because of the policies they access, there can be additional consideration given to how policies maintain inequalities and therefore influence the realization of democratic ideals, connecting theory to practice. Existing literature provides the necessary impetus for more qualitative designs with marginalized populations. Such investigations, furthermore, stand to develop the theory further and offer important areas of future study.

The Theoretical Leverage Gained From Qualitative Inquiry

Qualitative inquiry with populations – such as interviews, focus groups, and ethnography – will add the perspective of populations that can inform policy feedback theory development. Although there has been discussion around selection biases and confidence in causal inferences with policy feedback applications using quantitative data (see for example Campbell, 2012; Gidengil, 2020), there is a need to also consider how qualitative data can contribute and overcome limitations commonly associated with quantitative analyses.

Current literature has called for the application of the theory across contexts, the need for more nuanced considerations of particular aspects, as well as the expansion and diversification of research designs (Béland & Schlager, 2019; Bussi et al., 2022; Rosenthal, 2021). Examinations beyond historical institutionalist approaches have also been discussed, with the need to understand how policy characteristics may have multiple consequences and how they relate and interact with the government (Béland et al., 2022). Qualitative research designs with more participant-involved methods, furthermore, can inform the definitions and conceptual boundaries of variables, in addition to the careful consideration of specific policy elements, and the disentangling of multiple and interacting effects. These three threads of policy feedback research are utilized below to illustrate the benefits of qualitative research designs for policy feedback theory development.

Theory development requires an interrogation of the variables of interest – and in particular who they ignore – as well as the effects and relationships examined and hypothesized. In relation to marginalized groups that have often been ignored or excluded from data available for investigations of feedback, there is a need to understand how the variables and relationships exist, or do not, among different groups. Qualitative methods can lead to more nuanced understandings of participation and policy, as well as the complexity of effects.

Defining Participation

Political engagement is often the dependent variable of study in studies of mass policy feedback effects (Larsen, 2019). Such studies benefit from inductive qualitative investigations with the inclusion of different forms of engagement and experiences whilst engaging. To understand the relationship between policy and participation, there is a need to contextualize participation according to the population under study. Many marginalized groups furthermore, may not participate in traditional ways, limiting our understanding of not only how they might participate beyond traditional forms, but also why they do so. Participation is multifaceted. Additional expressions of political agency require consideration and help identify more nuanced effects of policies. If policies influence multiple forms of participation – including where individuals participate – these effects contribute to the understanding of the effects of policy on engagement.

Qualitative methods are well positioned to contribute to more expansive considerations of participation. Open-ended questions around how a group participates that do not include researcher biases through pre-determined options may yield new insights. Qualitative inquiry, furthermore, can allow for researchers to engage in discussions around why individuals participate in the way they do, and provide a thicker description of how a population expresses their political agency and their experiences when doing so. How that relates to experiences with policies furthermore requires first-hand accounts from participants that cannot be fully captured through other methodologies.

Existing policy feedback literature examines various forms of participation that will benefit from additional qualitative inquiry with different groups to better understand the relationship between policy and participation. Scholars have often considered registering to vote, voting, contributing to campaigns, participating in rallies or protests, and involvement in civic or community organizations, involvement with unions, for example (see for example Bruch et al., 2010; Campbell, 2003; Mettler, 2011; Schneider & Ingram, 1993; Soss, 2002; Weaver & Lerman, 2010). More recently, in Take a Number, Gidengil (2020) recognizes there are many ways to be politically active and utilizes a definition of participation that includes formal activities often examined in existing policy feedback literature and informal activity. Boycotting products, demonstrating, working with others in the community to solve a problem, making a complaint, volunteering, and donating to a charity are characterized as informal activities (Gidengil, 2020).

Additional and newer forms of participation – often coined ‘citizen participation’ or ‘political engagement’ – have been on the rise with more deliberative and participatory forums in which the public can participate and add to policy feedback investigations (Buss et al., 2006; Cain et al., 2003; Chambers, 2003). Such forms of participation expand the venues where democracy is practiced, including institutionalized non-elected and even non-institutionalized venues in society (Warren, 2003). They can also inform the relationship between policy and engagement, particularly in relation to marginalized groups.

Recent literature has considered the role of different spaces and venues of engagement utilized by marginalized populations, such as tenant organizations (Michener & SoRelle, 2022) and homeless shelters (Pue & Kopec, 2023). Michener (2020) examines the role of local organizations and their membership in the development of urban policy within the civil legal domain. Organization – a form of participation – of nonelites is rarely examined, and Michener argues, is a major limitation to existing studies (Michener, 2020). White (2022) reviews the literature on the carceral state and political engagement, whilst arguing that there is a need for qualitative data. White argues that this should include different forms of participation and nonvoting participation in particular (White, 2022). The growth of different forms of participation and the civil society’s role in cultivating capacity and providing opportunities to negotiate power requires a more comprehensive picture of the participatory opportunities available (Han & Kim, 2022). Including these venues and forms of participation to studies of feedback through qualitative inquiry, will inform the knowledge of policy effects on multiple forms of participation, many of which are particularly valuable to vulnerable groups.

Such forms of participation create opportunities for various populations to influence policy. They also inform our understanding of the effects of policies, which can influence not only who participates in these ways, but also who has access to such venues. Some forms of participation can be more tokenistic and serve as surface level consultations rather than meaningful engagement, with inclusion in such spaces itself insufficient (Arnstein, 1969; Olson, 2002). These experiences captured through more qualitative inquiry can speak to important elements related to participatory models. Participatory experiences are vital to consider, since many forms of political engagement include venues that perpetuate and maintain inequality, limiting access to some populations (Levac & Weibe, 2020). Such inquiries can also highlight the latter half of the feedback loop: influence on subsequent policy. With more qualitative inquiries that include participant experiences, policy feedback scholars can learn more about the multiple forms of participation. This can include considerations of the lack of feedback; where even given venues of engagement, policies do not reflect certain interests (Michener, 2019). Such studies are important not only to understanding the dynamics of when feedback does or does not occur, but also to consider the power, inclusion, and experiences of marginalized groups.

Nuamah (2021) examines school closure policies and community meeting participation and finds a mobilization fatigue (“collective participatory debt”) among predominantly poor and black individuals. Due to a lack of democratic transparency and responsiveness during the process of participating, however, their future participation and trust in government was negatively impacted. This study not only utilized observations of the meetings, but also interviews, to examine the participation of a vulnerable group. Nuamah finds negative and positive experiences with the policy process itself: relationships that would not have been found through other methods. Experiences with the policy process led to a fatigue in participation that influenced future engagement (Nuamah, 2021). These experiences are key findings for policy feedback, they explain the effects of not only policy on engagement, but also experiences with engagement that influences future participation, and therefore, influence on subsequent policy.

Variables can be defined by populations under investigation, rather than apply existing definitions that limit investigations (Harding & Seefeldt, 2013). There may be ways in which certain groups experience policies and participate, which can inform variables and, therefore, subsequent quantitative investigations. As traditional forms of political participation have several barriers for marginalized groups, and we know that marginalized groups are in fact still participating albeit in different ways, it is important for us to include these acts to understand how policy experiences may contribute to inequities in participation, and where and how experiences with specific policies may in fact lead to some forms of participation. Marginalized populations may also identify specific elements of policies that have effects on engagement, particularly since there may be different policies targeting different groups that require more inductive investigation. This added nuance allows for the consideration of the effects of different policy elements – another important area of theory development.

Considering Policy Elements

Different mechanisms of policy feedback include different policy design elements and their effects, illustrating the complex and important relationships between policy designs and feedback (Béland et al., 2022). Adding investigations among different populations stands to also benefit considerations of on-the-ground experiences and allow for the close examination of different policy design characteristics and their effects. Several design features combine and bring nuance to the effects of single features, with some elements triggering different or counteracting effects (Béland et al., 2022; Bussi et al., 2022; Mettler, 2011; Soss, 2002). Different factors and characteristics constitute effective policy design (Fernández-I-Marín et al., 2021). Policy design includes different tools and instruments that interact to influence engagement. Little is known about such nuanced interactions and effects. Policy designs are not simply abstract concepts, but procedural aspects individuals interact with (Bruch & Soss, 2018). They therefore include not only the text of policies, but also the practices and consequences associated (Schneider & Ingram, 1997). Beyond rules, this includes the benefits and burdens, implementation structures, and social constructions (Pierce et al., 2014).

Different policy characteristics and their effects have been examined in existing literature. Qualitative approaches may inform these elements and ensure all relevant policy design characteristics and tools are captured in policy feedback investigations. They can also add to existing examinations. Careful considerations of some aspects of policies may be better captured through on-the-ground research with different populations. Scholars have considered how policies are distributed and compared the effects of means-tested and universal policies (Campbell, 2003; Mettler, 2005; Skocpol, 1992). Means-tested policies can send messages to individuals that they are treated with suspicion (Schneider & Ingram, 1993). Bruch et al. (2010) take the discussion further to consider authority structures, which include paternalistic designs that enforce obligations and promote civic marginality. Visibility of policies has also been identified as important, sending messages regarding the importance of a given population or social issues to governments (see for example Pierson, 1993; Rosenthal, 2023; SoRelle & Shanks, 2023). And yet, understanding the messages that policies send may be better captured through thick descriptions. For example, Rosenthal (2023), argues that visibility of the state has different impacts on the political engagement of white populations compared to black and coloured populations in the US through extensive ethnographic research.

Experiences with policies through on the ground interactions with service providers and street-level bureaucrats can also be better captured through qualitative methods. Feedback scholars have identified implementation and administrative mechanisms of policies as important to understanding feedback effects (see for example Bussi et al., 2022; Bruch & Soss, 2018). Afterall, administrative translation of policies and implementation are political forces, reorganizing power relations in society and generating political interests (Brodkin, 1997; Moynihan & Soss, 2014). How programs are administered can influence experiences with policy, and therefore the messages sent, and constituencies created, by policy (Barnes et al., 2023; Campbell, 2012). Literature has established the effects of front-line services and administrators, with implementation playing a role in reorganizing power relations and therefore also generating political interests (Moynihan & Soss, 2014). These administrative encounters are important; experiences in these settings bring meaning (Soss & Schram, 2007). As the street-level bureaucrat literature and studies of administrative burdens exhibit, there can be variations in these encounters (Brodkin, 2016; Herd & Moynihan, 2018; Lipsky, 1980). Such heterogeneity in experiences can be captured through qualitative methods (Small & Calarco, 2020). In doing so, studies can contribute to policy feedback literature and its’ limited examination of administrative themes and implementation (Moynihan & Soss, 2014).

As Barnes and Henly (2018) consider in their qualitative study of administrative burdens, experiences with policies and services have important effects on political participation and inclusion, further informing our understanding of the relationship between policy and politics. The literature on administrative burdens, front-line services, and street-level bureaucracy utilizes qualitative methods with groups that access varying services – including those from marginalized groups – and therefore highlight the contribution of such research designs to considerations of the delivery of policies. To increase studies that examine the relationship between implementation and engagement, but also how implementation interacts with policy design elements, qualitative inquiry is particularly helpful.

In addition to closer examinations of specific policy elements, qualitative inquiry with marginalized groups will also add considerations of how these elements interact, where there may be multiple levels of exclusion, and how some elements may have more positive or inclusive effects on different groups. If a policy is means-tested but is visible, or perhaps another is universally distributed but inaccessible, how do these internal elements interact to influence political engagement? Breaking these elements down with the help of thick descriptions of experiences with policies can tease out these varying effects and their interactions (see for example Bussi et al., 2022; Watson, 2015).

Disentangling Effects

As policy feedback literature has identified, there is an inherent complexity of policy effects and interactions. For marginalized populations, this can include additional complexities associated with multiple levels of marginalization. Given different social constructions and messages sent, in addition to accessing multiple policies, more cross-cutting effects can be observed among marginalized populations (Béland et al., 2022; Schneider & Ingram, 1993). This includes interactions within one policy from how it is written to its implementation, that is, how different characteristics within a given policy interact with one another, and interactions between different policies.

In-depth qualitative methods can better capture these various complex effects through a close investigation of the effects on a population and, therefore, contribute not only to policy feedback theory but also to policy design, street-level bureaucratic and administrative burden, and political behaviour literature. Such investigations can capture the diversity of experiences from the perspective of those accessing services and uncover the effects of different policies and elements. They also lead to more hypotheses regarding how the relationships between policy design elements, political engagement, and policy change are understood.

Recent feedback literature has called for the disentangling of different effects. This includes the examination of design characteristics and their varying effects on engagement as well as separating resource and interpretive effects and adding considerations of multiple policies and their effects on engagement (Bruch et al., 2010; Bussi et al., 2022; Rosenthal, 2021). Such complexities will benefit from thick descriptions that include lived expertise of marginalized groups.

In relation to different effects within a particular policy, for example, Jacobs et al. (2022) consider political efficacy more closely in their disentangling of resource and interpretive effects using survey data. They examine panel data, offering long term effects, and present a mechanism of political efficacy as an interpretive effect that can have indirect effects on participation. Their measure of political efficacy includes a standardized factor index of seven survey items that asked agreement regarding different statements, such as, if public officials care about what the respondent thinks, if respondents care about the affordability of heath care, if respondents consider themselves qualified to participate in politics, and elements of policy learning regarding the ACAs impact. Resource effects include measures regarding the status of respondent’s health insurance, and usage of ACA features (Jacobs et al., 2022). Qualitative research stands to benefit such investigations of resource and interpretive effects. In fact, considerations of heterogeneity of experiences, as well as in-depth data on certain experiences can be particularly helpful. The knowledge of different experiences is facilitated by exposure of a researcher with a field site (or with a population) and can capture diversity that will inform theory and subsequent hypotheses (Small & Calarco, 2022). This is relevant for studies with marginalized groups, many of whom face multiple levels of exclusion. For example, a study of poor racialized groups can consider different experiences based on race as well as gender and other identity factors informed by direct involvement with those groups – some of whom may not be fully captured in surveys, and others that will provide more in-depth understandings. Open-ended interviews can provide additional information regarding the efficacy of participants, in addition to specific experiences that add depth to their responses. Understanding different experiences with the ACA, through in-depth interviews, for example, can supplement survey data and add to future survey instruments, informing questions and variables of interest.

There is also added complexity in considering not only interactions and effects within a policy, but also between them. Complex problems require mixes in policy tools and integrated approaches to policymaking (Howlett et al., 2017). As vulnerable groups often experience multiple levels of marginalization and exclusion from and by multiple state apparatuses, research with them is particularly important in relation to multiple policies and their interactions. For example, populations experiencing extreme poverty may be interacting with multiple state apparatuses and policies at once, each with their own complex effects.

Policy integration requires the increased advocacy of coherent and cooperative policy formulation and implementation across policy sectors (Domorenok et al., 2021; Tosun & Lang, 2017). Many individuals interact with multiple policies. As Rosenthal (2021) argues, in the US case, nearly 30% of those receiving any universal program have also claimed at least one means-tested benefit during their lifetime. We need to better understand how different policies and their, at times, counteracting effects influence political engagement. The mobilizing effects of universal policies can be canceled out by the demobilizing effects of means-tested policies, for example (Rosenthal, 2021). Qualitative methods can assist this disentangling, by considering how different experiences with policies influence political engagement. The complexity of these interactions necessitates careful on-the-ground investigations of experiences with policy and government, along with individual and group perceptions of government and politics. Individuals can point to experiences with different policies that inform the understanding of their interactions.

Qualitative research designs with marginalized groups will therefore further how scholars define and examine participation and policy elements, as well as add nuance to the understanding of complex effects. These three areas of theory development illustrate just some examples where qualitative inquiry stands to benefit policy feedback theory.

Discussion: The Benefits of More Qualitative Research Designs

Qualitative research can complement existing statistical techniques and increase our understanding of not only policies, but also their various elements and the experiences associated, from populations that may be missing from existing analyses. Research that includes more direct involvement of different populations can add to our understanding of measurements and variables, as well as heterogeneity in effects (Harding & Seefeldt, 2013). This is particularly helpful in policy feedback, where existing mass public investigations are limited in their consideration of the heterogeneity of effects across different subpopulations.

As a theory that explains a relationship, but also includes subjective experiences among different target populations, studies of feedback includes both quantitative and qualitative questions. Oftentimes, qualitative study questions focus on the how and why, articulating intentions and perspectives of social interactions and processes (Agee, 2009). Such questions are important to policy feedback theory and stand to fill important gaps in the application and development of the theory in particular. How do policies influence democratic participation? This adds nuance to existing investigations and the consideration of interactions between different effects as well as multiple forms of participation. Similarly considering: “Why are some groups participating in different ways?”, allows us to highlight the influence of policies on how certain groups are participating.

Qualitative inquiries can add nuance to feedback effects, teasing out and considering how they interact, add considerations to the variables of study, and most importantly, consider the perspectives of populations missing in existing research. The benefits of quantitative approaches, however, can also contribute to qualitative inquiry. In a body of research that considers important relationships between people and the state, as well as macro mass public level data, qualitative and quantitative approaches can be complementary.

Existing mixed-method research, primarily in public administration, often includes a dominance of quantitative methods, where qualitative methods often take a “secondary status” (Hendren et al., 2018). There are a variety of mixed method designs, ranging from fully to partially mixed quantitative and qualitative methods, when they are used in reference to one another (related to timing and sequence) as well as which method is emphasized (Belardinelli & Mele, 2020; Creswell, 2021; Leech & Onwuegbuzie, 2009). Purposes can include triangulation, complementarity, development, imitation, and expansion (Greene et al., 1980; Hendren et al., 2018). Integrative multi-method designs are particularly helpful where there is a relationship between the different methods, with one informing the other (Seawright, 2016). Qualitative methods therefore, need to be considered more carefully in mixed-method designs, and can play a more dominant role depending on the question and availability of data.

There are, however, challenges associated with mixed-method designs. They require diverse and collaborative research teams with researchers with different skills as well as resources (Belardinelli & Mele, 2020; Creswell, 2021). As Gehlbach (2015) argues, “…the use of multiple methods should be a goal for [a] subfield, not for any particular scholar” (12). Although Gehlbach refers to comparative politics, the sentiment transfers to the discussion of policy feedback. The call for more mixed methods in feedback, therefore, does not need to only include within studies, but also encourage collaboration between projects and institutions, where studies can build off one another and contribute to theory development together. For example, adding studies with marginalized populations can consider where qualitative data adds to the stories of mass publics. Or, using qualitative methods can inform variables that are then further explored in quantitative surveys and analyses.

Existing research will benefit from more qualitative research that includes more lived expertise within a given study to supplement case study approaches and quantitative research designs (Campbell, 2015; Cartwright, 2019; Michener, 2018). Mixed methods can benefit not only the investigation of relationships but also causal analyses. There is a strength added to mixed-method designs, particularly in relation to causal considerations (Seawright, 2016). Qualitative methods consider causal mechanisms as processes allowing for careful considerations of what causal mechanisms are and making them helpful to multiple policy process and public policy theories (Starke, 2013). Although quantitative and experimental methods can assist with issues of endogeneity within causal inferences, there is a need to include the lived experiences of those on the margins. Qualitative methods, such as interviews and ethnography, furthermore, not only ensure such perspectives are included, but can also generate and test hypotheses, as well as trace causal processes (SoRelle & Michener, 2022).

Future research needs to incorporate more qualitative inquiries that better capture experiences with policies and resulting engagement from the bottom-up to inform the hypotheses and relationships. This can also include collaborative and community engaged methods, such as participatory action research (Lake & Wendland, 2018; MacDonald, 2012).

It is worth noting, however, that research designs with different groups require important ethical considerations. Qualitative research can include extensive fieldwork within communities, as well as building relationships with different groups. There is a need, therefore, for careful considerations related to the approach including potential benefits to the community, as well as harm to participants and the researcher (Dickson-Swift et al., 2009; Ngozwana, 2018; Traianou, 2020). Scholars need to be aware of the emotion work of qualitative research and consider how their research benefits those they work alongside (Dickson-Swift et al., 2009). Many qualitative researchers have discussed these difficulties as well as different approaches to interviewing and other qualitative methods that place participants and their well-being at the centre, particularly when working with marginalized populations (see for example Campbell et al., 2021; Fujii, 2018; Yanow & Schwartz-Shea, 2014).

Conclusion

Qualitative inquiry with different populations in policy feedback will allow for more in-depth investigations that add nuance to the existing applications and generate new theoretical insights. Understanding experiences of marginalized groups with polices and how they engage deepens our understanding of policy feedback, where it occurs, why it may not always occur, and the nuanced mechanisms that contribute to policy change.

The use of qualitative inquiry, furthermore, does not need to come at the loss of quantitative research. In fact, mixed-method designs can be particularly helpful, whether within one project or in relation to projects of one methodological paradigm building from another. They can then add to the theory with new hypotheses, relationships, and examinations that will ensure policy feedback considers multiple populations, and the perspective of vulnerable populations in particular.

More welfare state investigations that include the lived experiences of those most marginalized are needed. Public policy research benefits from working alongside participants to reveal the complexity of factors and multi-level contexts that influence decision-making (Merrick, 2022). The influence of institutional structures on decision-making benefits from integration of different methodological and theoretical approaches (Merrick, 2022). Policy feedback considers important relationships between policies and those they benefit (or do not), making research designs themselves a vital consideration to the future development of the theory. By increasing methodological diversity, the theory can begin to hypothesize the relationships between policy and participation among multiple target populations. This will, in turn, inform discussions of social mobilization and policy change, as well as political agency, inclusion, and inequality.