Online focus groups as a tool to study policy professionals

Abstract

Scholars interested in political elites and policy professionals only sporadically rely on focus group methods. In this article, we argue why this is a missed opportunity. Based on our own recent research experience, we suggest three innovations to the focus group method that should make it appealing for political scientists who study professionals active in the policy process. First, focus groups allow for the study of the interactions between political elites in ways other methods cannot. Second, the method can be used for hypothesis testing when focus groups are designed accordingly. Third, the option to conduct focus groups online has major advantages in terms of participant costs, ease of recruitment, and, potentially, research equity. We hope that our suggestions are useful for researchers wanting to build on any one—or all—of these innovations.

Introduction

“It turns out the hypothesis was rejected!” says Fay Miller, strategist for a consumer research company in the TV series Mad Men, after having collected evidence from a focus group of young, unmarried secretaries. She thus suggests a new framing for the campaign on the client’s product. Yet, Don Draper, the director of the advertisement company in the series, is not convinced: “You go in there, and you stick your finger in people’s brains, and they go on talking—bla bla bla—just to be heard,” he says, and dismisses the new route (Mad Men 2010).

In the real world, the use of focus groups has spread widely in marketing studies since the 1970s (e.g., Cox et al. 1976). Interestingly, however, they are infrequently used in political science; we rarely “stick our fingers” in people’s brains in this way to study political actors and processes. This is surprising because a wide variety of research questions in our field addresses the behavior and interactions between such actors, which mirror positions, ideas, and power relationships that political scientists are interested in. Focus group interviews offer a way to observe these interactions in a purposefully designed setting, varying, for instance, the topics, cases, or scenarios to be discussed, the composition of the focus groups, or the role of the moderator (Cyr 2016).

In this article, we make the case for a more frequent use of (online) focus groups with selected elite representatives in political science research (see also Stanley 2016). Building on existing contributions on the use of focus groups in the social sciences (e.g., Cyr, 2019), we add three innovations when employing focus groups, which we hope will make this methodology more appealing to political scientists.

First, while the focus group methodology is increasingly used in several fields of social science, it is rarely applied to the study of political elites, such as members of political parties, bureaucracies, or interest groups (Kullberg (1994) presents one of such rare studies). We argue that this is a missed opportunity because focus groups with political elites are uniquely placed to combine the advantages of other methods to study policy professionals. Second, we suggest that the use of focus groups can be innovated by using them to empirically test hypotheses (Junk et al., 2023) rather than, as dominantly noted (e.g., Morgan, 1996), only being used for explorative research questions aimed at building theoretical propositions. Third, we propose that new opportunities to conduct focus groups online through video conferencing, rather than face to face, make them a feasible and affordable strategy for studying policy professionals. We hope that our suggestions are useful for researchers wanting to work with either one or several of these innovations. We illustrate each of them based on our experiences with implementing online focus groups with around 50 participants in three countries.

Innovation I: Focus groups with elite participants

A key quest in political science is understanding the behavior of political elites, that is, small groups of people “who exercise disproportionate power and influence” on political debates and/or decision-making (Higley, 2016). These groups include, most intuitively, elected officials, but also civil servants, consultants, and organizational representatives from businesses, trade unions, or non-governmental organizations (NGOs) in their capacity as stakeholders in the political process. While even “influencers” on social media platforms such as Instagram or TikTok can constitute “political elites” in this sense, we focus specifically on (elite) policy professionals who hold positions in policy-relevant organizations such as parliaments, bureaucracies, or interest groups.

Political scientists have employed a set of diverse methods to study such political elites and their relationships within and between the organizations they represent or are active in. One can broadly distinguish between the following methods:

(1) Those that rely on the coding and analysis of text and other publicly available data (e.g., Coen and Vannoni, 2020; LaPira and Thomas, 2017);

(2) Those that rely on information requested from organizations and persons of interest, commonly through surveys of or interviews with organizational representatives (e.g., Beyers et al. 2020; Harvey, 2011; Peabody et al., 1990; Heaney 2014; Leech, 2002);

(3) Those that entail the observation of behaviors of elites during their professional activities by means of ethnography-type “shadowing” (e.g., Adler-Nissen and Eggeling 2022; Gravante and Poma 2022) or with field experiments (e.g., Grose et al., 2022); and

(4) Those that (artificially) manipulate the situations elites must navigate, such as in survey experiments and audit studies (e.g., Aizenberg 2023; Kalla and Broockman 2016; Rasmussen and Reher 2023).

Focus group interviews with elite participants constitute a data collection method that can combine some of the benefits of these methods. First, like interviews and surveys, they request information from participants, typically in a way that is related to a concrete topic or case. Second, unlike traditional interviews, they do so in an interactive, group-based setting that allows for observing actual exchanges between participants. This observational approach to group interactions bears some resemblance to ethnographic studies, although an important difference is that the focus group setting is an artificial one, designed and partly controlled by the researcher. Third, this partly controlled context provides opportunities to purposefully design and vary the settings in which elites interact, which means they can be designed in ways that parallel the logic of designing experiments (see also our Innovation II).

With this unique combination of traits in mind, we think that important political science research questions can be answered with the help of focus groups to acquire data from selected actors active in public policy and public administration. As discussed by Cyr (2019: 12–14), focus group interviews can extend beyond mere group-based interviews when they explicitly identify the interaction as a unit of analysis, key mechanism, or phenomenon studied. This focus on interactions makes focus groups well suited to address research questions addressing relations between actors within or across organizations, especially in areas where aspects of socialization, group dynamics, and responses are considered critical for a proper understanding of the political process. For such questions, focus group participants are selected based on their organizational or professional roles.

In our project, for instance, we were interested in knowledge sharing between interest organizations because information and “advocacy intelligence” are crucial resources for organizations that seek to influence policy (Chalmers 2013; Hall and Deardorff 2006). Nevertheless, the factors that facilitate and hinder knowledge sharing between lobbyists have rarely been researched, arguably because traditional methods are not well suited to observe them. This is why we employed focus group interviews that allowed us to observe how organizational representatives share insights with each other in varying constellations (Junk et al., 2023).

The policy professionals we studied were, therefore, leaders of interest organizations such as business associations, labor unions, and NGOs. An underlying assumption is that their interaction is (partly) structured by their professional roles and positions. Studies of bureaucratic politics recurringly find support for “Miles’ Law,” that “Where you stand depends on where you sit” (Miles, 1978: 401), meaning that officials reflect their organization’s viewpoints and preferences. This assumption partly applies to survey and experimental research on elites as well: researchers collect data from individuals to learn about the organizations they represent. In the case of focus groups, it is similarly paramount to design the research setting in a way that addresses policy professionals as organizational representatives rather than private individuals. Of course, there is likely to be some “noise” when treating individuals as embodiments of organizations, given variation within organizations in perspectives, levels of knowledge, personality traits, and so on. Both when selecting participants for focus groups and when interpreting the findings, it is important to consider such alternative explanations (cf. Junk et al., 2023). At the same time, focus groups are arguably especially useful when researchers want to open the “black box” of unitary organizations (e.g., interest groups, parties, parliamentary committees, etc.) and assess variation or relationships within them by conducting focus group interviews with a larger number of representatives from the same organization.

An obvious and central difference between focus group interviews with elites and ordinary citizens is that elite participants are likely to know each other beforehand. As we see it, this can be an advantage when applying the focus group method to political elites, because the group setting partly mirrors normal professional interactions. Policy professionals are accustomed to group meetings in office buildings or in video conferences, as their work consists of formal policy negotiations or political debates with other well-known or less familiar stakeholders in the political system. This makes it relatively easy for researchers to create an ecologically valid context compared to the extensive precautions needed for other types of participants (e.g., Gamson, 1992). ¹

Another potential benefit of conducting focus groups is that they set a relatively low bar for entry for the researcher. When used for exploratory purposes, for instance, in the early phase of a research project, a relatively low number of focus group interviews can already span a wide range of perspectives on a topic and help generate hypotheses. Compared to “traditional” interviews covering the same number of actors, focus groups are, in this sense, more efficient. They provide much more “bang for the buck,” so to speak. Moreover, the novel format of exchanges on a topic with fellow professionals may provide an incentive to participate. This is relevant here, especially given that some actors experience many interview and/or survey requests. In our project, the ease of recruitment for the focus group interviews was (surprisingly) high, varying between 47% (Ireland) and 84% (the Netherlands) of the contacted interest group representatives who agreed to participate. Compared to typical response rates in surveys among interest groups of around 20–40% (Marchetti, 2015), this is very high.

Of course, limitations to the use of focus groups with political elites also exist. Compared to one-on-one, academically confidential interviews, the shared format might disincentivize participants from talking openly and freely about (all of) their activities. Furthermore, social dynamics create opportunities for stronger personalities within the focus group to dominate the conversation, effectively reducing the participation time for others. For a particular set of research questions, these limitations may, however, entail advantages as well. A “lack of openness” or a “dominance” on the part of particular participants on particular topics is usually not coincidental but reflective of some professional practice or mode of interaction. The scope of and conditions for information sharing in the group setting can hence be meaningful subjects of study in themselves rather than only being a methodological nuisance. We suggest that, when carefully designed, questions about the interactions between policy professionals (on a specific topic or case) might fruitfully be studied using focus groups.

Of course, the extent to which focus groups are a useful tool—or not—depends on many factors, including their precise purpose and design. Important design choices include the role of the moderator and the design of the moderator memo (see our example in Appendix A), as well as the selection of participants for the focus groups. In the next section, we suggest that this can be systematically varied across focus groups when researchers wish to employ them to test hypotheses.

Innovation II: Focus groups for theory testing

Although Fay Miller, our Mad Men character, talked about hypothesis testing, the currently dominant approach situates focus groups as more appropriate for theory building (see Morgan, 1996; Cyr, 2019) or as a pretest before surveys (Cyr, 2018), or sometimes as a means to speak to relatively open a priori theoretical expectations (e.g., Saunders and Klandermans, 2019). For instance, building on the seminal citizen-oriented, focus group-based study of Gamson (1992), focus groups have effectively been employed in social constructivist research to study everyday narratives among professionals, most notably in world politics (Stanley, 2016). However, like Fay Miller, we argue that online focus groups can also be used for theory testing.

To do so, we suggest that researchers ask what observable implications existing theories about elite behavior have for interactions in (specifically designed) focus groups and assess how actually observed patterns in the focus groups support these expectations. Our understanding of theory testing is, therefore, not necessarily probabilistic. For instance, a large number of focus groups and (stratified) random sampling from an underlying population would be necessary to facilitate probabilistic hypothesis tests. However, even with moderate numbers of participants, focus groups allow observing whether (and under which conditions) elite participants actually behave in line with predefined expectations. “Theory testing” in this latter (non-probabilistic) understanding entails probing the validity of a claim that links a variable to a potential outcome (Mahoney, 2008). ²

In what follows, we describe key considerations in a theory-testing approach to focus groups, illustrated with the research design of our project. As noted above, while our application of focus groups was used for non-probabilistic hypothesis testing (Junk et al., 2023), we think that our reflections are useful for researchers who consider using focus groups for diverse purposes.

An approach to focus groups that is geared at theory testing requires considerable preparation around the composition of focus groups to align them with predefined (and potentially pre-registered) hypotheses. In our case, the aim was to examine whether knowledge exchange between interest group representatives was hindered by organizational competition. This aim drove the design of the focus groups to test hypotheses formulated around two types of competition and knowledge exchange (see Junk et al., 2023).

To test these, we constructed four types of focus groups with four selected organizational participants each that systematically varied the level of presumed competition between participants. More specifically, we expected competition on two dimensions—policy influence and member recruitment (cf. Berkhout 2013)—which we reasoned depended on organizations’ policy areas of activity and membership types. Table 1 summarizes this logic, which determined the composition of organizations in the four types of focus groups in our project. Based on this, we developed a sampling strategy to select organizations from the population of active interest groups, proxied by a pool of survey respondents in the project (see also Junk et al., 2022). We followed an identical sampling protocol (see Appendix B) in three countries (Denmark, Ireland, and the Netherlands) and implemented these with an identical moderator memo (see Appendix A).

Table 1.

Selection of focus group participants and group composition.

Source: Junk et al. (2023).

Although our total of 12 focus groups (including 48 organizations), which were distributed equally over the three countries, is low compared to survey-based studies, we see our implementation as a successful systematic trial of using the focus group method to test hypotheses. For those wishing to apply a similar strategy, we list a number of research design choices that are important to consider.

First, focus groups should be constructed varying specific characteristics of interest. Like in (survey) experiments, it is important to have a narrow research focus, considering that the number of required focus groups and participants is positively related to the breadth of the research question (i.e., the number of variables of interest). Researchers can identify units of observation at multiple levels, such as the group level, participant level, or based on the number and kinds of interactions (nodes) between specific “dyads” of participants. Observations below the focus group level will be more numerous and therefore provide more statistical leverage in case the aim is a quantitative analysis (and potentially a probabilistic approach to hypothesis testing). ³

Second, besides the key independent variables of interest, focus group design should try to hold as much theoretically relevant variation constant as possible. In our case, potential “confounders” could have been the individual lobbyist’s job experience, position in the organization, or ties to other organizations. If it is not possible to control for these in advance, researchers can attempt to include the pre-measurement of relevant participant characteristics to be able to “control” for confounding factors (e.g., employing a concise list of survey questions before the interview).

Third, when it comes to the role of the moderator in theory-testing designs, many of the considerations can parallel the moderator role in more traditional focus group approaches (Cyr, 2019). As in all interview methods, the relational aspect between interviewees and interviewer will play a significant role, and researchers may choose to select a moderator who will make it more likely for participants to “open up.” Moreover, a decision must be made as to whether the moderator should take a minimally intrusive role, to allow the conversation to go in any direction the participants want, or actively aim to draw all participants into the discussion and “correct” for potential tangents that the conversation might take. It may in fact be the researcher’s intention to capture whether interactions between participants develop naturally, independent from the moderator’s presence. This is important for the use of this methodology with political elites, as natural conversations in the different types of focus groups will partly depend on the ecological validity of the group composition. In some constellations, the provision of a platform, in a more or less equal manner, to participants in a focus group may be at odds with the roles that actors usually take in the policy process (i.e., reflecting some variable of interest, such as power relationships). Interviewing less and more prominent actors in the same focus group may cause an unnatural situation, which the researcher might choose to avoid or employ intentionally, depending on the research aims. In any case, moderators need adequate preparation in anticipation of the quite varying situations they will encounter to ensure that they do not compensate for the potential differences between the focus groups. To ensure all focus groups are implemented as equally as possible, moderator interventions should therefore be consistent across focus groups.

Fourth, the formulation of moderator questions must also be carefully prepared. These can be narrowly formulated questions, geared toward the mechanism of interest and allowing comparison across groups. Alternatively, if the unit of analysis is the interaction between participants, relatively broad questions may also be helpful. In our project, we asked participants quite broadly to share best practices in their advocacy during the COVID-19 crisis and in different categories of their work. This allowed us to identify how much knowledge was shared in the differently composed focus groups. Another possible design is to employ “vignettes” in the moderator questions, which are similar to the design of a survey experiment. This means the focus group moderator might choose to present theoretically grounded scenarios to participants and either vary the scenarios between several equally composed focus group meetings or use the same scenario in differently composed focus groups to test the effects on group-level interactions. The advantage of working with such scenarios in focus groups (rather than survey experiments) is that they allow the observation of the resulting social processes in response to the scenario based on the real-world roles and functions of the focus group participants. Like in traditional survey experiments, questions regarding the credibility and ecological validity of the scenario should also be evaluated explicitly at the end of the focus group.

Finally, after the focus groups have been conducted, the transcriptions must be systematized, for instance, through manual or automated coding based on the categories of interest of the researcher. In our project, we coded the number of direct, unsolicited interactions between participants, the number of instances in which concrete information about advocacy practices was shared, as well as the type of this information. Researchers with a focus on elite interaction could develop similar coding schemes depending on the types of exchanges between participants they are interested in.

Table 2 summarizes this section by listing our considerations about the most important research design choices when using focus groups for theory testing.

Table 2.

Focus group design in the theory-testing approach.

	Design considerations
Research goal	Test of (a small number of) hypothesized relationships or assessment of observable implications of a theory
Selection of participants and group composition	Theoretically deduced, with maximum control over theoretically relevant differences among participants and potentially (random) variation of group composition
Moderation and intervention by researcher	Must be as uniform across groups as possible Option: “Treatment”-like interventions by moderator/researcher
Time points in data collection	Pre- and post-treatment observations of participants or group dynamics possible
Mode of analysis	Typically coding of transcriptions to enable replicable and (partly) quantitative analysis
Unit of analysis	For instance, the group, participant, or dyad level, depending on the research question
Other important considerations	Comparability between focus groups Bias through non-response to invitations to participate Ecological validity of setting and group composition Pre-registration of hypotheses Ethics approval and participant consent requirements (see Footnote 1)

While we hope that focus groups will be used more often to test predefined hypotheses, it is important to stress that this method is not suitable for all research questions. Questions that require some form of anonymity toward peers, such as highly sensitive issues, are unlikely to be suited for focus groups. The fact that peers are present limits what participants will want to convey. As our example illustrates, focus groups might be most useful when this is a feature, not a bug. Theories related to the practices of professionals, networking, learning, or other social mechanisms that emerge when professionals interact are likely to be those best suited to testing through focus groups. This is an important benefit, as it is often difficult to tap into such dynamics via other means.

Innovation III: Online focus groups

Our third and final contribution relates to the opportunities in terms of validity, practicality, and efficiency offered by conducting focus groups online (which seems increasingly common; see Stewart et al., 2009: 611; Gaiser, 2008). Regarding ecological validity, the COVID-19-related lockdowns in 2020 and 2021 have meant that large proportions of the professional world have become accustomed to having work meetings via videoconferencing (Vargo et al., 2021). The normalized nature of this mode of working makes it likely that potential participants will behave similarly during online focus groups compared to their routine professional interactions. This applies to a broad range of potential participants even beyond the political elites studied in our project, such as individuals with management roles in public and private organizations, journalists, or street-level bureaucrats.

Furthermore, there are practical research design benefits. As long as potential participants speak a common language, videoconferencing allows for grouping elites in ways that were previously unfeasible or very difficult, most notably from different organizations, without imposing travel time on participants. This is likely to increase recruitment success. The duration of the focus group is the effective amount of time participants offer to a research project. Moreover, conducting interviews online entails a relief for project budgets, considering that no travel is required and no location needs be booked. In this sense, online focus groups may have the potential to increase research equity because they can be performed on small budgets.

Our project was conducted against this backdrop in 2021. Given that face-to-face meetings were not feasible, we conducted all focus group interviews online. This proved unproblematic, since participants had already made the transition to online activities as part of their daily work practices at this point. Perhaps because the focus groups were relatively small (four participants), each of the participants was able to express their experiences and respond to other participants in our 45–60 minute-long meetings. ⁴ The online format meant that organizational representatives, sometimes based in different locations, were able to participate at a relatively low cost, without having to take “time off” or reschedule other activities and appointments for the day. This might explain our high recruitment success rates (between 47% and 84% in all countries).

Of course, there are also potential downsides to conducting focus group interviews online. Videoconferences introduce structure and formality to communication by virtue of the medium. They are less personal and potentially less open and interactive outside the realm of the conversation. A downside is that this can reduce the “emic” character of focus groups, as it becomes more difficult to allow “for a participant-led conversation to organically flow, with only minimal interruption by the moderator” (Cyr, 2019: 11). We see this problem as less pronounced for focus groups with policy professionals because they are used to work-related conversations on online platforms. Still, some topics and interactions will be deemed unfit for the online medium, especially topics that participants are hesitant to discuss in recorded videos.

Conclusion

We suggest three innovations for researchers wanting to employ focus groups. First, focus groups are an appealing way to study political elites because they combine the advantages of existing methods in a highly fruitful way. They allow gathering topic- or case-specific information directly from elites, while also observing phenomena related to professionalism, socialization, and interaction between elite actors in a (partly) controlled setting. Second, we suggest that focus groups be used to test theories. This can be achieved by systematically varying the selection of participants and the construction of focus groups or by introducing treatment-like interventions in the focus groups. Although focus groups only allow for a limited set of expectations to be assessed, we see them as a great way to add to the methodological toolkit of scholars interested in theory testing. While this article mainly focuses on the benefits of focus groups for non-probabilistic hypothesis testing, we expect our approach to be extendable to probabilistic hypothesis-testing designs as well. In such a case, two additional steps should be considered. To start, researchers must randomly select a large enough group of participants from a full population and consider potential patterns in opt-outs very carefully. In addition, more reflection on variation and comparability in the focus group design and moderator interventions will be necessary. While it is likely to remain a challenge to generalize from data that is generated with such a high degree of qualitative and situational researcher intervention, we think that it is worthwhile exploring in the future. Third, we think that the current normalization of online communication tools for meetings provides an excellent opportunity to start conducting focus groups with elite participants with high levels of ecological validity and even on a large scale. Online focus groups reduce the costs for both participants and researchers, which means they are a relatively equitable research method.

Overall, this article seeks to stress that online focus groups have the potential to gather valuable, novel data in a relatively cheap manner. Focus group methods help us better understand organizational and personal political interactions, bridge quantitative and qualitative methodological gaps, and are feasible for researchers across the academic hierarchy. We wrote this article because we are convinced by the promise of this method for broader use in our discipline and hope that, unlike Don Draper, so will some of the readers.

Supplemental Material

Supplemental Material - Online focus groups as a tool to study policy professionals

Supplemental Material for Online focus groups as a tool to study policy professionals by Joost Berkhout, Michele Crepaz, Marcel Hanegraaff and Wiebke Marie Junk in Research & Politics. Please find additional material at: https://doi.org/10.7910/DVN/WLKIB6