Qualitative Thematic Analysis of Transcripts in Social Change Research: Reflections on Common Misconceptions and Recommendations for Reporting Results

Abstract

This paper, on qualitative thematic analysis (QTA) in social change research, falls somewhere between a reflective piece and a how-to guide. Using two examples from my own previous research, I discuss why QTA in the field of social change or social justice, which often analyzes the words of vulnerable, marginalized, or underserved populations, is so fraught, so contested, and so often dismissed. Qualitative thematic analysis of interviews or focus groups is a common research tool used in the field, but the guidelines, scope, and practices of this tool are varied and ill-defined. I have witnessed in my students’ papers and in peer reviewing for journals that there are a handful of assumptions and misconceptions that appear repeatedly, for example around intercoder reliability and frequency counting, that reduce the quality of analysis. This paper focuses on how to conduct QTAs that address social change: complex social problems faced by underserved populations, such as those dealing with poverty and inequality. By discussing the methods used in two of my own social change research projects, this paper offers a balanced method for both promoting rigor and understanding the limits and strengths of this method.

In my own experience teaching, peer reviewing, and submitting journal articles, I have encountered many challenges faced by those (including myself) attempting to research the messy, complex topics of social change and social justice. The challenge I will focus on lies in presenting persuasive evidence regarding the state of complex social problems when the method of data analysis includes qualitative thematic analysis of the spoken or printed words of underserved populations, such as might be gathered through interviews or focus groups. I have faced skepticism on both sides of this equation, with others distrusting my own findings and, at times, finding myself unconvinced of the results of others. This is an unfortunate situation if one believes, as do I, that the stories and perspectives of underserved populations hold enormous importance in helping us better understand barriers to social change and social justice, and paths toward solutions. In other words, demonstrating that the words of others can reveal some hidden “truth” about a complex social problem is rightly regarded skeptically, but we nonetheless must find a way to move forward so that such invaluable data can contribute to evidence-building, theory-building, and ultimately social change.

Qualitative thematic analysis (QTA) is a common method of analyzing the spoken or written word. There are many excellent pieces written on how to conduct QTA – many of which will be covered here. My aim is not to reinvent the methodological wheel. However, there remain areas of confusion I believe need to be clarified. Four items have received insufficient attention:

1. The unique skepticism that arises in working with marginalized and underserved populations, especially the common assumptions that (1) our subjects are being dishonest and (2) our findings are cherry-picked;

2. Clear guidelines for conducting QTAs of interview and focus group transcripts with such groups, especially with regard to the need for tests of intercoder reliability;

3. Clear guidelines for how to present QTAs in ways that other scholars (some of whom may be inherently skeptical of qualitative research) will find to have sufficient ‘rigor’; and

4. A clear defense of QTA’s potential for building theory and general understanding of complex social problems, rather than seeing the method as only representing a specific case with no generalizability.

This paper aims to achieve this and to provide clear guidance on some thorny issues for scholars embarking on this type of research. The paper will review some of the most oft-cited scholarship on how to conduct QTAs and will describe what I feel is missing from these efforts, or where I think there is conflicting advice. The second section of the paper illustrates some of this guidance through one of my own previous QTAs. In the final section, I make recommendations for reporting the findings of QTAs. Ultimately, the six recommendations I make for writing up results are as follows: (1) sufficiently explain the method; (2) quantify the frequency of theme appearance; (3) use many examples; (4) when possible, publish anonymized versions of raw transcripts online; (5) when possible, verify findings with affected populations; and (6) if warranted, bring in (and build) theory to promote generalizability.

This paper briefly describes why QTA is such a challenging method and then discusses the above recommendations and why they are important for furthering this type of research. Throughout the paper I will use two examples of QTAs I have conducted in the recent past – one from a project analyzing interviews of survivors of intimate partner violence (IPV) and one from a project analyzing interviews from individuals living in high poverty, low-vaccination neighborhoods about reasons for not receiving a COVID-19 vaccination. I do not use these examples to illustrate findings from the research, but rather to illustrate some of the flawed assumptions behind the ‘right way’ to analyze, interpret, and present qualitative data.

I will note here that this paper focuses primarily on transcripts of interviews and focus groups. This is because it is the type of research I do, because it is particularly challenging to try to represent the essence of another’s experience, and because doing so elicits much skepticism from outsiders, in part because the source material is less accessible. However, much of this paper, particularly regarding how to present findings, should also be useful to scholars focusing on critical analysis of media texts such as press coverage, or scholars using thematic analysis of interviews in conjunction with textual analysis of related media (e.g., Orbe & Kinefuchi, 2008). In reading and reviewing papers that use discourse analysis, framing analysis, or thematic analysis to qualitatively analyze press coverage or other media texts, all the problems I address here arise, particularly skepticism regarding perceived a priori agendas and assumed cherry-picking to fit those agendas.

Literature Review

Qualitative Thematic Analysis

My impression of the extant literature is that most of the writing on QTA has focused on the fields of psychology and healthcare. To my knowledge, there is little explicitly addressing the field of communication, my own research area, and so this review attempts to address the usefulness of QTA across a variety of social science fields, including communication.

Within many social science fields, there is a tension between quantitative, more positivistic approaches to studying human behavior and qualitative and more constructivist or critical epistemologies (Goertz & Mahoney, 2012). This tension has manifested in confusing guidance for scholars. Within communication studies, even though many conduct qualitative analysis, the field is often associated more with positivist perspectives and traditional quantitative content analysis methods (Graneheim & Lundman, 2004; Vaismoradi et al., 2013). To be clear, this paper focuses on those scholars approaching research from a constructivist, interpretive, or critical perspective, asking questions that cannot be easily quantified. That said, content analysis exists along an epistemological continuum, and it is possible for multiple approaches to be combined (Braun & Clarke, 2021).

Thematic analysis and qualitative thematic analysis are terms that are often used in papers but can mean many things. It is clear there remains much confusion over what it is, and perhaps more importantly, what it is not. Thematic analysis is often used to mean content analysis, and can therefore be qualitative, quantitative, or a combination of the two (Braun & Clarke, 2021). Qualitative thematic analysis (QTA), in its most basic form, is content analysis or thematic analysis that is qualitative. While several authors have created specific guidelines for how to conduct QTAs, most notably Braun and Clarke (2006), it remains a “flexible” tool, which is part of why so many are drawn to it (Braun & Clarke, 2006; Vaismoradi et al., 2013; Braun and Clarke’s 2006 oft-cited article usefully lays out a step-by-step guide for conducting QTAs while still maintaining flexibility in approach, which I will not fully review here, but their recommendations include creating initial codes for the data and then sorting those codes into themes. As codes or topics (more descriptive) become themes (more interpretive), and themes are potentially combined to create larger themes, the analysis takes on increasingly greater levels of interpretation and abstraction.

Given how broad QTA is, it is perhaps more useful to clarify what QTA is not. It is not coding using a priori hypotheses or codebooks (Braun & Clarke, 2021; Vaismoradi et al., 2016). It is not (typically) analysis of institutional publications or media such as newspapers, television shows, or government documents. It does not require multiple coders and measures of intercoder reliability (Braun & Clarke, 2021; Morse, 1997; Vaismoradi et al., 2013).

The issue of intercoder reliability is contentious, and others would disagree. Some suggest that more coders are better and/or that intercoder reliability is a desirable feature of QTA (Castleberry & Nolen, 2018; Maguire & Delahun, 2017). Some conduct studies that combine deductive coding (with multiple coders, based on theoretical frameworks, hypotheses, and pre-defined codebooks) and inductive coding (developing themes based on the text without a priori ideas) and so conduct QTA in tandem with deductive approaches that may benefit from intercoder reliability (Fereday & Muir-Cochrane, 2006; Xu & Zammit, 2020). This may cause some confusion around whether QTA requires multiple coders, but to study messy, complex social phenomena, we must study real people and their real words. We need to take the data as they stand without trying to fit them into preconceived hypotheses or theories (though theories can be used in the analysis, as I will discuss further on). This means there is enough interpretation involved that expecting multiple individuals to read the data the same way is not usually possible.

QTA is not the same as critical discourse analysis (CDA), even though both techniques can be used with a critical lens and may address some of the same societal concerns. They are, in a sense, two sides of a coin, with QTA typically (though not always) analyzing the voices of those with lived experiences of social phenomena and CDA typically (though not always) analyzing media texts produced by powerful elites (e.g., newspapers or Hollywood films).

Part of the confusion around QTA stems from different epistemologies of researchers and different beliefs regarding what kinds of questions QTA can or should answer. This paper addresses researchers who use QTA to understand why particular social phenomena occur. In the social scientific paradigm, causal hypotheses for explaining human behavior are typically defined, tested, and refined. On the other hand, researchers with more interpretive or constructivist starting points ask broader research questions, without formal causal hypotheses, and with the understanding that interpretation of findings is subjective, and thus value laden. Researchers grounded in more interpretive or critical epistemologies are drawn to questions that produce messy, complex answers about why humans behave as they do – questions for which it is difficult or impossible to create quantitative variables upon which to conduct statistical analyses, and more difficult still to develop coding instructions that would pass intercoder reliability tests. Scholars working within interpretive or critical paradigms are likely to have conscious or unconscious hypotheses or assumptions going into research, but do as best as they can to keep those assumptions in check and to remain open to findings that do not fit within expectations. Acknowledging one’s epistemological starting point explicitly when reporting findings is thus helpful for establishing expectations regarding how the analysis should proceed and what conclusions can be drawn.

Skepticism About the Rigor of QTA

There is widespread acknowledgement among those who work with this method that it faces skepticism (Castleberry & Nolen, 2018; Graneheim & Lundman, 2004; Lawless & Chen, 2019; Nowell et al., 2017). In my view, this is even more true when the research claims to represent the voices of marginalized and underserved populations. This is the case for several reasons, namely: (1) underserved groups are infrequently heard from in the public sphere; (2) outsider academics are not seen as legitimate representatives of marginalized groups; (3) QTAs stemming from research on marginalized populations often lack measures of intercoder reliability; (4) findings from small communities, especially ones that feel most unfamiliar, are not considered to be generalizable; and (5) findings often appear cherry-picked.

Underserved Groups Are Not Heard from Much, and Therefore Distrusted

The news media are less likely to quote individuals from underserved communities or publish their perspectives (Dekavalla & Jelen-Sanchez, 2017; Gans, 1979). For some topics, their behaviors and attitudes are seen as mysterious and confusing. For example, during the COVID-19 pandemic in the United States, many underserved populations were not getting vaccinated. The media narrative that arose to explain this phenomenon was that these groups were either uninformed or filled with conspiracy theorists (Kogen et al., 2022). This messaging was so engrained into public discourse that many were skeptical when I presented my findings that this narrative was missing major reasons why underserved populations were not vaccinated. Many assumed my subjects were lying to me. Similarly, when I presented my research on IPV to a group of scholars, one member of the audience asked how I knew the women were not lying to me about their stories of abuse.

Academic Outsiders Are Not Seen as Legitimate Representatives of Marginalized Communities

Secondly, many rightly question how academics who are not members of a researched community can accurately reflect its various perspectives. What could academics in their ivory towers possibly understand about the lives of the marginalized? Sometimes academics who attempt to do so are critiqued for being naïve, manipulated, manipulative, or flat out lying. For example, Alice Goffman’s 2014 book On the Run was an ethnography about a disadvantaged neighborhood in Philadelphia and what it was like for a group of young, Black men in that community to live under the constant watch of police. The book was heavily critiqued, primarily by saying that Goffman lied, or was being lied to, or took advantage of her informants, or, if she did get real insights from her informants, that it must have been because she was sleeping with them (Singal, 2015).

Lack of intercoder reliability

Audiences are understandably skeptical of conclusions drawn by a single author, on a subjective dataset that no one else has access to. Intercoder reliability is sometimes cited as necessary to ensure that the analysis is rigorous and legitimate. However, as discussed above, the extant writing on this topic squarely comes down on the side of intercoder reliability not being required for QTA. Those who do argue for some measure of intercoder reliability (e.g., Castleberry & Nolen, 2018; Maguire & Delahun, 2017), likely have in mind analyses that address manifest, rather than latent, content. When the codes being generated address latent content, it is interpretive, subjective, and biased. It is nearly impossible to have two people interpret data the same way across an entire dataset. An example of manifest, descriptive content would be something like coding “whether a respondent lives in the city or the suburbs.” These are comments which are relatively easy to code.

On the other hand, when attempting to glimpse “the essence” of individuals’ lived experiences, interpretation is necessary. As Braun and Clarke (2021) state, themes “draw together data that on the surface appear disparate” (p. 341). It will, unconsciously or not, be influenced by individual beliefs. This is a limitation of the method, but it should not be seen as a strike against QTA – it is an element of the method that ought to be plainly acknowledged.

I will take a moment here to drive home this defense of the single analyst. There is an attitude within certain sectors of academia that statistical analysis of quantitative data is more legitimate than qualitative analysis. It is, however, naïve to believe that a single statistical data analyst alone in a room with their dataset is anymore immune to bias, exaggeration, or bending data to tell a story (even if unintentionally) than the lone qualitative analyst. This is evidenced by books like Darrell Huff’s How to Lie with Statistics, the scandals surrounding the replication crisis (Pashler & Wagenmakers, 2012), and websites such as aspredicted.org which demand that quantitative scholars state their hypotheses before engaging in analysis to ensure they are not fishing for statistically significant relationships. And yet, we do not ask for the equivalent of intercoder reliability in quantitative studies – that two scholars, given the same research question, come up with the same exact analysis of the dataset.

Findings are not generalizable

Fourth, the method is critiqued for producing findings that are specific to a very narrow case, and therefore not generalizable, or that if the findings apply elsewhere that it is incumbent upon others, not the original researcher, to figure this out. This belief that findings are not generalizable is also held by qualitative scholars (e.g., Castleberry & Nolen, 2018; Guba & Lincoln, 1989; Herzog & Kelly, 2023). Guba and Lincoln (1989) state that it is the responsibility of the reader of an evaluation to decide whether the findings can translate to other contexts. Likewise, Castleberry and Nolen (2018) state: “conclusions from qualitative research are not usually generalizable” because they “can often not be replicated” (p. 812) and thus that the onus is on the reader “to identify differences and similarities between the research context and their situations in order to determine relevance and applicability of study findings” (p. 813). Even in the textbook I currently use to teach research methods to graduate students, which I think is excellent, I need to insert caveats about the assumptions of the authors when they say things like: “For interpretive, critical, and rhetorical researchers, generalization is not an important issue to consider, as these scholars focus more on subjectivity” (Croucher & Cronn-Mills, 2019, Chapter 5).

On the other hand, many scholars understand that qualitative data can build or improve generalizable theories (e.g., Fereday & Muir-Cochrane, 2006; McLeod, 2001; Simons, 2015). Indeed, there is a case to be made that generalized theory about human behavior can only come from an intimate understanding of complex phenomena. Simons (2015, p. 181) discusses the paradox of the “universal in the particular,” meaning that, for particular types of phenomena, “if we study the singular case in sufficient depth, and are able to capture its essence” that this can lead us to uncover something of “universal significance.” Similarly, McLeod (2001, Chapter 3) discusses how producing “an exhaustive description of the phenomena of everyday experience” can help us “[arrive] at an understanding of the essential structures of the ‘thing itself.’”

Deeply interrogating a localized phenomenon makes inherent sense for complex cultural experiences that are difficult to control through experiments or capture in surveys. It is difficult to understand through a survey, for example, community experiences of violence. This is not to say that every individual’s experience of community violence is the same, but we can better understand phenomena such as the psychological toll of violence if we do more listening and engaging. Insights about how humans interact with the world can only be achieved by digging deep into lived experiences.

Findings appear cherry-picked

Finally, what I see as the thorniest problem is that many are skeptical of this work because the way data are presented is, frankly, often difficult to believe. I have peer-reviewed and read many papers in which authors make broad claims and use a single quote to illustrate the claim and justify their conclusion. Even to someone who is supportive of this type of research, this comes across as cherry-picking. I will return to this issue in the final section.

Undergirding all the above is the rarely articulated truth that hearing why some members of underserved populations act as they do can make us uncomfortable. Those of us working in the field of social change understand that many outsiders’ assumptions about why some individuals engage in certain behaviors are often wrong, and indeed often guided by elitist assumptions about how individuals ought to behave, or by beliefs that behaviors depend only on personal decisions and little on environmental and societal structures that constrain choices. Analyses addressing issues around social change often attempt to bring those structures to light. Therefore, trying to convince outsiders to question their own assumptions and acknowledge a reality which might place part of the onus of change on societal structures, will be an uphill battle. Believing that behaviors are all about making bad individual choices is a convenient way to disregard those structures. This means those of us working in the field need to make it difficult for audiences to dismiss our findings.

Not all dismissals of qualitative research are nefarious. Being skeptical is a good quality, and one that we try to instill in young scholars. When I read something in the news or in academic journals that does not sound quite right, that does not pass the ‘sniff test,’ and the evidence presented is thin or otherwise questionable, I will often brush the conclusion aside. When I conduct peer reviews for journals that use QTA, and one example is used to legitimate a broad claim, I am often unconvinced by it. If this is the response of someone supportive of the method, it is unsurprising that those wary of the method are even more skeptical.

Conducting Qualitative Thematic Analyses

There is much excellent writing on how to conduct QTAs, so I will not dwell upon it here other than to summarize some of the most important principles. For those new to the method, I would recommend Braun and Clarke’s (2006) foundational article on QTA and their follow-up 2021 piece, in which they refine the method and describe it as “reflexive thematic analysis.” There are many other good pieces written on the topic (see, for example, Castleberry & Nolen, 2018; Vaismoradi et al., 2013; Vaismoradi et al., 2016; and Lawless & Chen (2019) for a piece that focuses on the field of communication).

Themes, Topics, Categories, and Codes

There are many ways to define and categorize themes, sub-themes, topics, categories, and codes (among other terms used). Vaismoradi et al. (2016) and Braun and Clarke (2021) usefully outline distinctions between codes and topics, which tend to be explicit, descriptive, and aligned with manifest content (e.g., ‘comments about length of wait times at clinics’), and themes, which are more interpretive, latent, and get at “the essence of what the interviewees are saying” and their experiences (Vaismoradi et al., 2016, p. 102). These pieces are useful for trying to understand the variety of terms used to code and categorize data in QTA.

While approaches may differ, they hold in common the suggestion that scholars need to approach the data many times, re-reading, recoding, refining codes, and comparing codes until things fall into place and the researcher feels they can tell some kind of narrative about what is happening in the data. My own process is to (very) roughly code some large chunk of the text (usually between 50% and 100%) until I feel I am starting to see patterns emerge or reaching saturation with regard to at least one research question. I then like to start drawing – creating mind maps, often referred to as thematic maps, concept maps, hierarchies, or visualized axial coding. (See Vaismoradi et al. (2016) for a good discussion of different levels of abstraction.) This aids in clarification, but others have different strategies. I also like to draw mine on paper, although there are a variety of digital tools to assist with this process. Figure 1 represents the final concept map used during my research on COVID-19 vaccine hesitancy (though this was probably the 50th iteration of it). This process is conducted in tandem with coding new data, recoding old data multiple times, and/or refining codes, themes, and sub-themes until things seem to feel reasonably well settled. This process of constant comparison between new and old data (Glaser & Strauss, 1967) is time consuming, but necessary to ensure the researcher’s interpretations are consistent throughout the dataset and reflect what is occurring in the data.

Figure 1.

Example of axial coding from a qualitative thematic analysis.

Using data analysis software such as NVivo or Atlas.ti greatly facilitates the coding process, since in QTA a single sentence may fall under more than one code, category, or theme, making this difficult to track without such software. (Some scholars advocate mutually exclusive categories or codes, meaning that the same sentence cannot fall into two categories or codes, but the general consensus is that this is only the case if you intend to conduct statistical analyses (Insch et al., 1997).

Intercoder Reliability

We have already discussed above that intercoder reliability need not be a goal of QTA. An example from our COVID-19 project illustrates how the same data can result in two entirely different conclusions. During our research on COVID-19 vaccination hesitancy, a pattern I saw in the transcripts was that pop-up clinics that successfully persuaded individuals to vaccinate sometimes used messages that emphasized convenience, along the lines of: “You could do this right here and now and be done with it, so if you were thinking about it but there hasn’t been a good time, now is a good time.” Individuals could decide in the moment, without an appointment, at no cost, with no identification or insurance needed. This concept, vaguely formed, framed many of my codes, as one of the themes that emerged from the project was that many members of the underserved communities we were working with, many of whom had already had Covid (for a variety of structural reasons related to housing and to who were considered essential workers), faced so many other challenges in their daily lives that getting vaccinated was simply not a priority. But this was my own reading, and was admittedly influenced by my previous research.

Others might have interpreted the data differently. A comment within the transcripts like “people don’t want to leave their neighborhood to go to a pharmacy 10 minutes away where you have to make an appointment and have to show insurance,” which to me lent credence to the idea that inconvenience and time constraints were impediments to vaccination, could have been interpreted in almost the opposite way – that convenience (a pharmacy only 10 minutes away, for which you could easily make an appointment online ahead of time, and was free as long as you had insurance) seemed to not be enough to promote vaccination. ¹ Indeed, some people I spoke with who disagreed with my findings stated this: the fact that some were loath to travel 10 minutes to a pharmacy suggested that inconvenience was not a barrier, and thus that interviewees likely had more political reasons for not getting vaccinated, or believed in conspiracy theories, and did not wish to tell me. If inconvenience was even suggested as a reason for vaccination delay, skeptics felt, it was probably an excuse made to appease the interviewer.

In other words, different researchers may have different takeaways from the same dataset. If intercoder reliability cannot serve as a proxy for rigorous research, the individual researcher must make the case that their personal, perhaps even biased, read of the data is legitimate. This is the topic of the next section.

Recommendations for Reporting the Results of QTA

What methods can one use to make research findings more legitimate, and therefore more believable? This is where I feel scholars, especially young scholars, need more guidance. Unless scholars do a better job of presenting findings in a rigorous manner, we will continue to face extreme skepticism regarding representing the voices of marginalized and underserved populations.

One requirement, of course, is that findings actually be legitimate. As stated previously, even I am skeptical of some findings I read in others’ manuscripts when certain methodological protocols are not followed. It is likely the case that some scholars are, at the very least, gently bending their findings to conform to some shape they consider worthwhile. This may be unconscious. The methods outlined in this section will benefit both the analysts who have important findings to share but have unconvincing explanations and the analysts who need some basic protocols in place to ensure they are not unconsciously reaching too far from the evidence to claim a conclusion they want or expect to see.

Here, I make six recommendations I believe would do much to strengthen the credibility of QTA. Briefly, those recommendations are to (1) clearly explain and defend the method; (2) count and report the frequency of themes; (3) provide many examples for each theme; (4) publish anonymized transcripts online; (5) verify findings with affected communities; and (6) if warranted, use findings to build theory. I acknowledge it is not feasible to follow all of these for every project, but argue that researchers should try to follow as many of the suggestions as possible.

Recommendation 1: Explain the Method and Defend Not Including Multiple Coders

Many papers say that the method of “qualitative thematic analysis” was employed, or that they “analyzed” transcripts of interviews or focus groups, without describing what they mean, or using only vague descriptions (Castleberry & Nolen, 2018; Vaismoradi et al., 2013). This immediately raises red flags, suggesting the analyst read the transcripts once or twice, followed their gut, and that was the end of it. Authors should include, at the very least, a name for the method they used to analyze the data (e.g., qualitative thematic analysis, reflexive thematic analysis), a description of the dataset analyzed (e.g., transcripts of 25 in-depth interviews), a brief description of how the method was carried out (e.g., whether themes were developed from initial descriptive coding of all the data, whether techniques such as constant comparison or axial coding were used), which software program (if any) they used to analyze the data, how many coders were used, and why they considered this a legitimate method for the research questions at hand. While scholars typically include a method section in their papers, the method often focuses on protocols followed for data collection rather than data analysis, suggesting an explanation of data analysis protocols is unnecessary.

The single qualitative thematic analyst need not shy away from using a single coder, but should plainly acknowledge and defend it. A reader who notices a lack of explanation for having only a single coder will likely have their antennae go up, especially if they are accustomed to deductive or quantitative methods. Make it clear that having a single coder is an acceptable, often necessary, way to conduct QTA.

Recommendation 2: Count Frequencies

As discussed above, cherry-picking is a real phenomenon, and can lead to outsiders dismissing findings. Saying that a phenomenon exists in the world and finding one quote to illustrate it does not read as rigorous research. You can say it emerged as a theme, as if it were lying there waiting for someone to unearth it, but readers are unlikely to believe it without a preponderance of evidence.

Unfortunately, I have found there is an extreme discomfort regarding quantifying frequencies in QTA. Some of the most-referenced articles on QTA specify that counting or quantifying themes should not be done. They argue that counting should not “stand as a proxy for significance” (Vaismoradi et al., 2013, p. 404), or that if the research aim is “to build conceptual understanding, then there is little need for quantifying qualitative data” (Thompson, 2022, p. 1416). I have received pushback from scholars when making the suggestion on paper drafts. For some scholars, there seems to be a sense that the research is not as pure or meaningful when it is reduced to mere numbers, that it is reductive, that participants are somehow discounted, or that scholars are pretending to be within the social scientific paradigm when they should not have to pretend.

I strongly disagree with this position and put forth that arguments made regarding themes are much more convincing when numbers are attached. The number of times a theme is repeated by unique individuals is suggestive of its importance (as is how strongly it is emphasized by a particular speaker or speakers). This is similar to what Owen (1984) refers to as recurrence (how often a theme arises), though Owen does not explicitly advocate for considering recurrence of a theme across unique individuals, nor for reporting the number of recurrences.

Recurrence is an indicator of the strength and salience of a theme. For example, if I state that a theme in my interviews regarding healthcare access is that patients of color experience racism during hospital visits, and I include a single quote of an interviewee who said this, this finding is almost certain to be dismissed by skeptics, who will likely believe I entered the research with an a priori agenda. But if I state that five of 20 interviewees experienced this, the finding instantly becomes harder to brush aside. This may seem like a simple point, but I have been surprised by not only how infrequently scholars report such numbers, but also at how resistant they are to reporting numbers when requested to do so. I do not mean to imply that a resistance to counting suggests fraud, but rather that there is a perceived and unhelpful epistemological clash between quantification and qualitative, inductive methods.

This recommendation also applies to media analyses, such as thematic analyses or framing analyses of press coverage. Too often I see papers that state a particular type of news event, for example, is framed in a certain way. One or two examples are included to ‘prove’ the point, but no information is given as to how common this framing was in the data set. Without this, I am quite skeptical as to whether this was a common trend.

Counting how many times a code appeared is not meant to ‘prove’ a point, nor to make the research ‘appear’ more scientific. The act of counting in itself does not make something positivistic or ‘truthier.’ All it does is provide an honest picture of what is happening within the data, so that readers can decide for themselves how much credence they give to the findings. It may be the case that a theme only appears once, but because of context the analyst still thinks it is important or provides nuance that is useful. Once again, the point is not to provide rules for which findings should count, but to be as honest as possible. While Owen advocates counting recurrences within a single interview, I recommend counting and reporting the number of unique individuals mentioning a theme, as one interviewee mentioning something a dozen times is quite different than a theme recurring in 12 of 20 interviews.

Recommendation 3: Provide Many Examples

Provide examples of themes by quoting a variety of unique respondents verbatim. Likewise, for discourse analyses or framing analyses of media, use examples and provide references for the media sources that are being excerpted, if possible. This suggestion is not revolutionary; Braun and Clarke (2006, p. 95) note a lack of examples as one of the five major pitfalls of QTA. However, their warning remains often unheeded. I still see many QTAs and framing analyses in which a single quotation is given and the audience is expected to believe that the one quotation is symbolic of the lot. Skeptics are likely to view single quotations, or quotations coming from one source, as cherry-picked. In my own experience of others reading my papers, I have also often been told that this is the most interesting and engaging piece of the research! It is unsurprising that the real words of vulnerable populations are more interesting than the verbiage of an academic, but it is worth remembering, as it is unfortunate that so much of their own perspective is eschewed in favor of that of the verbose scholar.

Of course, word limits imposed by journals pose a challenge for including lengthy interview excerpts. Some scholars advocate for journals allowing greater word counts for QTAs (Braun & Clarke, 2021; Levitt et al., 2017) for this reason, especially now that most journals are read online. Another option would be to provide a file of more extensive excerpts on the journal’s or authors’ websites.

Recommendation 4: When Possible, Publish Raw Transcripts

This is a newer idea, and I admit that I have not yet done this (but intend to for future projects). It only occurred to me during my last research project, and it was too late to try this as it would have violated the consent agreements. While it is increasingly common, and in some cases even required, to publish datasets online (see, for example, the European Commission’s open data policy (“European Commission,” nd)), most of this data is quantitative. Publishing qualitative research data online is not totally novel, but it remains rare. The Qualitative Data Repository was created in 2015 (housed at Syracuse University in the United States) to, in part, make “the process and products of qualitative research more transparent” (“About the Qualitative Data Repository,” para. 4). The UK Data Service (2023) provides a list of international qualitative data archives.

This is also akin to the concept of auditing to increase transparency (Guba & Lincoln, 1989; Nowell et al., 2017). Auditing requires providing a paper trail that discusses discussion-making processes and other crucial points in the analysis process. Similarly, publishing field notes or journaling thought processes have also been suggested as ways to promote transparency (Guba & Lincoln, 1989, p. 242; Nowell et al., 2017; Vaismoradi et al., 2013). However, I have my doubts as to whether more than a very small handful of people ever look at these – the very thought of trying to read someone else’s field notes gives me a headache. I am also skeptical that simply saying a log of decisions was audited would provide me comfort if I did not find the findings believable to begin with.

I would argue that concepts around auditing are outdated in the age of increasing digital access. Auditing got its start as a way to promote transparency in qualitative research before the internet was widespread. Guba and Lincoln (1989, p. 243) describe the audit as a process in which these notes are literally handed over to an auditor who “attests to the quality and appropriateness of the inquiry process.” At that time, of course, publishing transcripts whole hog on the internet, so that anyone could perform their own ‘audit’ of the material, was not a possibility.

Now, however, transcripts that would previously be ignored by most could be used by others interested in the topic to conduct their own original analysis of the data. This serves the double benefit of making findings more transparent (and therefore more rigorous) while also increasing access to the voices of the marginalized, which could increase research relevant to these groups and amplify their voices.

Privacy Concerns

Researchers may hesitate to put raw transcripts of interviews with marginalized populations online, for fear of loss of privacy, fear of further stigmatization, or fear that it might scare away potential interviewees. These are all legitimate concerns, but ones that can be addressed. Online publishing of transcripts could be opt-in (though this would likely make publication of focus group transcripts difficult if only some members opted in), and consent forms can make it clear that transcripts will have any identifying information removed. The European Commission now even requires funded researchers conducting interviews and focus groups to post anonymized and redacted transcripts online whenever feasible. If researchers want an even higher level of privacy protection for their subjects, the Commission recommends a “restricted route of data access,” such as requiring a researcher interested in the dataset to seek specific permission and outline how they intend to use the data.

For a previous project (Kogen, 2022), I used raw excerpts of de-identified interviews to conduct a participatory QTA with IPV survivors. The raw interview excerpts included interviews with the women who were in the room conducting the analysis. Although I feared the women would be uncomfortable seeing their own words used as data or would be concerned that others would figure out their identity, a debrief with the interviewees at the end of the project revealed they felt very comfortable with the transcripts I used and did not feel they could be identified through them.

Recommendation 5: When Possible, Verify Findings and Themes With Affected Communities

If feasible, verify themes with your interviewees or affected communities. They will be able to tell you whether you are hitting the mark (Castleberry & Nolen, 2018; Guba & Lincoln, 1989). Guba and Lincoln (1989, pp. 238–239) refer to this as “member checks.” They state, “If the evaluator wants to establish that the multiple realities he or she presents are those that stakeholders have provided, the most certain test is verifying those multiple constructions with those who provided them” (p. 239). This will likely not appease those critics of QTA who believe respondents lie, but it will help appease those who question the researcher’s subjective interpretation of findings.

Better yet, include members of the affected community in the analysis. It is increasingly recognized that affected communities have insights that are invaluable for building knowledge (Kogen, 2022; Liebenberg et al., 2020) and furthermore that projects that include affected communities in knowledge building have greater societal impact than those that do not (van den Besselaar et al., 2018, p. 57).

Recommendation 6: If Warranted, Build Theory

This recommendation addresses the perceived generalizability problem inherent in qualitative analysis, as discussed in the first part of this paper. While many scholars, even those who conduct QTAs, suggest they are not useful for building theory, my position and that of many others (e.g., Ayres et al., 2003; Braun & Clarke, 2021; Garvey & Jones, 2021; Simons, 2015), is that qualitative research helps us understand the complex, messy, essence of social phenomena, and that this is fundamental for building theory about human behavior or about societal structures. The very point of a theory is that it is at least somewhat generalizable under certain circumstances.

For example, in my research with IPV survivors, which focused on clients of one specific organization, one finding was that internal communication, such as through private journaling, helped survivors feel more empowered. It could, of course, be the case that journaling for survivors who were clients of a different organization might not work to the same degree. But when the researcher really digs in to understand why journaling and other forms of internal communication are helpful, it suddenly becomes much clearer how such an intervention could be integrated into different organizations operating in different locations or contexts. In that study (Kogen, 2022), I offered a theoretical model regarding the role of journaling and other forms of internal communication in healing, empowerment, and ultimately collective action.

In this case, as with grounded theory (Glaser & Strauss, 1967), themes garnered from QTA may be taken to increasingly greater levels of abstraction until the researcher feels they have gotten to the essence of the phenomenon, or at least a portion of that essence, and arrived at something resembling a theory, or an added nuance to existing theory. While it will not be the case that every dataset holds the potential to build theory, a researcher can and should be open to that possibility.

Braun and Clarke (2006) are admittedly skeptical of using QTA to build theory, and refer to others using QTA in this way as conducting “grounded theory ‘lite’” because they do not commit to the full grounded theory process as described by Glaser and Strauss (1967). Braun and Clarke do not explicitly state that QTA cannot be used to build theory, but assert that researchers using QTA “need not subscribe to… implicit theoretical commitments” (p. 81). I would argue that QTA can be used to build theory when constant comparison is used and there is sufficient data to build a case.

Conclusion

The tension between quantitative and qualitative approaches to research is old and is unlikely to dissipate any time soon. However, it is still worth reminding audiences that both approaches are valuable and can even enhance each other. They need not be competing, with scholars eschewing beneficial aspects of the opposing epistemology.

When it comes to research trying to better understand human behavior in complex social circumstances, the need for qualitative research is obvious. Researchers who only include numbers, such as health data indicators, will have difficulty explaining why an intervention did or did not work, or what solutions may be effective, without making many assumptions. The only way to understand how social problems affect populations and what can be done to help address problems is to engage with and talk to affected populations.

At the same time, however, qualitative research findings have the handicap of coming across as anecdotal evidence, and therefore less believable. This is certainly the case for QTAs. The six recommendations presented here are designed to address this skepticism for researchers conducting QTAs. To summarize, these recommendations are to (1) explain and defend the method; (2) count and report the frequency of themes; (3) provide many examples for each theme; (4) publish anonymized transcripts online; (5) verify findings with affected communities; and (6) if warranted, use findings to build theory. While not all recommendations apply to every project, I believe a more consistent application of them across QTAs would do much to build credibility around the method. Qualitative research in the social science fields needs better methods to balance the presumed rigor found in quantitative research and the nuance necessary for social change research. While the ideas presented in this paper are neither methodologically complicated nor revolutionary, I believe greater adherence to them would lead to greater uptake of findings. In order to persuade decision makers and advance policy that leads to social change, more skeptic-resistant methods are required.