Abstract
Big qualitative data (Big Qual), or research involving large qualitative data sets, has introduced many newly evolving conventions that have begun to change the fundamental nature of some qualitative research. In this methodological essay, we first distinguish big data from big qual. We define big qual as data sets containing either primary or secondary qualitative data from at least 100 participants analyzed by teams of researchers, often funded by a government agency or private foundation, conducted either as a stand-alone project or in conjunction with a large quantitative study. We then present a broad debate about the extent to which big qual may be transforming some forms of qualitative inquiry. We present three questions, which examine the extent to which large qualitative data sets offer both constraints and opportunities for innovation related to funded research, sampling strategies, team-based analysis, and computer-assisted qualitative data analysis software (CAQDAS). The debate is framed by four related trends to which we attribute the rise of big qual: the rise of big quantitative data, the growing legitimacy of qualitative and mixed methods work in the research community, technological advances in CAQDAS, and the willingness of government and private foundations to fund large qualitative projects.
The term “big qualitative data (big qual),” or research involving large qualitative data sets, was likely borrowed from the term “big data.” Big Data typically refers to the large quantitative data sets increasingly used by academic researchers, government and nonprofit agencies, the private sector, and nonacademic political researchers. Big quantitative data has been the subject of a vigorous public debate related to individual privacy rights and the appropriate analysis and interpretation of such data (e.g., Brooks, 2013; Ohm, 2012; Shah, Horne, & Capellá, 2012). The original conception of big data tended to assume that data constituted numbers not words and images (Diebold, 2003, 2012). After big quantitative data emerged, however, many newly introduced conventions for big qual were also developed that began to change the fundamental nature of some qualitative research. The purpose of this methodological essay is to distinguish big qual from big data and then to present a broad debate about the extent to which big qual may be transforming some forms of qualitative inquiry.
Before the term big qual emerged, one of the earliest examples of large-scale mixed methods research projects was the Framington Heart Study, a medical study begun in 1948 with 5,209 participants that has continued to the present day with several new cohorts of participants including many of the children and grandchildren of the original participants (Levy & Brink, 2005). Over 1,000 medical papers have been published from the Framington data (Mahmood, Levy, Vasan, & Wang, 2014). While the Framington study was primarily quantitative, interviews were also conducted to measure psychosocial health factors such as anxiety, depression, social support, and hostility (K. Davidson, MacGregor, Stuhr, Dixon, & MacLean, 2000). Many large-scale mixed methods medical studies followed the Framington Heart Study (Plano Clark, 2010) including the present-day Precision Medicine Initiative (PMI), a mixed methods medical study conducted by the National Institutes of Health (NIH), whose goal is to enroll 1 million participants to study the role of genetics and lifestyle in health outcomes (Collins & Varmus, 2015). Though the PMI is a mixed methods study, it is as yet unclear the degree to which the research design will incorporate both primary and secondary qualitative data as the project evolves.
While a comprehensive big qual literature review is beyond the scope of this methodological essay, our initial review of large-scale qualitative and mixed methods studies conducted in the last 5 years uncovered research in a broad range of disciplines including agriculture (e.g., Charatsari & Papadaki-Klavdianou, 2017), business (e.g., St-Hilaire, Gilbert, & Lefebvre, 2018), environmental protection (e.g., Lynn, 2017), health and medicine (e.g., Hurst et al., 2016; Jenkins, Slemon, Haines-Saah, & Oliffe, 2018; Mayberry, 2016), public safety (e.g., Kerrison, Cobbina, & Bender, 2018), sociology and anthropology (e.g., Knight, Cottrell, Pickering, Bohren, & Bright, 2017; Manning & Greenwood, 2018; Reed, Strzyzykowski, Chiaramonte, & Miller, 2018), and education (e.g., Brower et al., 2017; Rutledge, Cohen-Vogel, & Osborne-Lampkin, 2012; Calma, 2013; Eta, Kallo, & Rinne, 2018; LaPointe-McEwan, DeLuca, & Klinger, 2017). Our initial review also showed that fewer than half of big qual studies involved primary data collection in the field. Many secondary data big qual studies in our review involved data downloaded from social media sources such as Facebook or Twitter (e.g., Greene, Choudhry, Kilabuk, & Shrank, 2011), qualitative data drawn from open-ended comment box questions on quantitative surveys (e.g., Elsesser & Lever, 2011), consumer research conducted by the private sector or political research conducted by nonacademic researchers (e.g., Clow & James, 2010), and content analysis conducted through computerized text-mining techniques (e.g., Guest & MacQueen, 2008).
Big qual can also be aligned with “rapid qualitative inquiry” (Beebe, 2014) and “multi-sited ethnography” (Coleman & von Hellermann, 2011), though we include in our definition both “slow” and “rapid” methods and qualitative traditions beyond ethnography including, for instance, case study, narrative inquiry, and grounded theory. Our review revealed that the most common research tradition for big qual was the case study (e.g., Brower et al., 2017; Calma, 2013).
Based on our review, we define big qual as data sets containing either secondary qualitative data or primary data with at least 100 participants, analyzed by teams of researchers, often funded by a government agency or private foundation, and conducted either as a stand-alone project or in conjunction with a large quantitative study. 1
Saldaña (2013) observed that “a metacognition of method, even in an emergent, intuitive, inductive-oriented, and socially conscious enterprise such as qualitative inquiry, is vitally important” (p. 40). This methodological essay is intended to ask metacognitive questions about big qual by making issues explicit that have thus far remained largely implicit based on our review of the qualitative research methods literature. Not unlike case study research, we have found big qual designs to be very flexible in terms of how they can be combined with other qualitative traditions. Nonetheless, currently big qual is a collection of methods that lacks the rich philosophical history and broader application we find in qualitative traditions such as phenomenology or grounded theory. Therefore, we hope to initiate a debate about whether big qual might someday be grounded in a deeper philosophy. Before presenting our discussion, we first must acknowledge how our work with large qualitative data sets contextualizes the debate presented here.
Research Contexts
The research context for this methodological essay was an ongoing 5-year mixed methods research project of a major policy shift in the delivery of developmental education (DE or remediation) in the 28 state colleges in Florida (Brower, Bertrand Jones, Hu, & Park-Gaghan, in press; Brower et al., 2017; Mokher, Spencer, Park, & Hu, 2019; Nix, Bertrand Jones, Brower, & Hu, in press; Park-Gaghan et al., in press; Park, Woods, Hu, Bertrand Jones, & Tandberg, 2018; Woods, Hu, Bertrand Jones, & Tandberg, 2018, 2019). The qualitative research methods for the DE project were informed by the work of researchers from two previous K-12 projects. One project was part of a multiyear research and reform effort focused on identifying the combination of essential components and programs, practices, processes, and policies that make some high schools in large urban districts particularly effective with students from traditionally low-performing subgroups (Rutledge, Cohen-Vogel, & Osborne-Lampkin, 2012). The second explored the implementation of district programs used to train and certify school leaders in Florida’s 67 districts (Rutledge, Cohen-Vogel, Osborne-Lampkin, & Roberts, 2015). Our perspective on big qual is grounded in our work on these projects and supported by our individual and collective interrogation of our data collection and analysis processes. We present a summary of our process for the DE project, specifically, to provide context for our debate.
As a mixed methods research project, the qualitative research team on the DE project collaborated with the quantitative research team through frequent meetings and informal discussions. Research findings from the quantitative team informed questions that were asked in successive iterations of the qualitative interview protocols and qualitative findings informed questions asked on annual surveys administered by the quantitative research team.
Over the course of 5 years, the overarching qualitative research question shifted from policy implementation processes to promising institutional practices in community colleges to organizational change and transformation. Our qualitative sampling strategy was a maximum variation sample at both the institution level and the individual level, which involved “purposely picking a wide range of cases to get variation on dimensions of interest” (Patton, 2015, p. 267). The 21 institutions in our 5-year sample represented the majority of institutions in the Florida College System. Institutions were located in every region of the state and differed by enrollment size, location (i.e., rural, suburban, and urban), and average performance on student outcome measures from the quantitative data set. At the individual level, we sampled different types of campus personnel including presidents, administrators, faculty, advisors, and support staff as well as students reflecting the diversity of community college student populations (i.e., students of color, veterans, English-language learners, immigrant students, undocumented students, parents and working adults, students with disabilities, first-generation college students, economically disadvantaged students, LGBTQ students, homeless students, and formerly incarcerated students). We also conducted interviews with state legislators and external policy stakeholders.
Our data for the DE project consisted of field notes, verbatim transcripts from focus groups and individual interviews, and institutional documents collected on site visits to 21 state colleges in Florida (with some repeat visits) over a 5-year period. To date, we have conducted 42 site visits and 166 focus groups comprising over 1,100 total participants.
Given the volume of data, the project consisted of a team of researchers (ranging from five to six) who conducted data analysis. The research team met weekly to share findings and discuss the analysis process. Our data analysis process employed pattern coding to identify central concepts and properties in the data (Corbin & Strauss, 2008; Miles, Huberman, & Saldaña, 2014). To begin the process, one researcher read the verbatim transcripts to ascertain where the subject of the participants’ response had changed and a new paragraph should begin. This process was, in effect, a form of precoding. We then read through the field notes, institutional documents, and focus group data to synopsize the chronology of institutional processes.
To establish inter-coder reliability, a number of specific analytic processes were necessary. Each of the codes in our coding frameworks needed consistent definitions in a codebook that researchers could refer to frequently. Our codebook de-emphasized codes reflecting highly theoretical or abstract concepts due to the likelihood that codes would be interpreted differently by different researchers. After we coded a subset of the data, we ran the Cohen’s κ coefficient function in NVivo, a computer-assisted qualitative data analysis software (CAQDAS) program. In the first round of reliability testing, we evaluated two pairs of researchers, comparing the coding of an individual researcher with the coding of another individual researcher. In the second round of reliability testing, we followed the same process by comparing two sets of researchers.
We then initiated pattern coding. In this process, we developed an a priori coding framework with codes at three levels (i.e., parent, child, and grandchild nodes) based on our initial reading of the data. During this process, we identified additional emergent themes not captured under existing codes, resulting in additional codes.
Our data collection and analysis efforts were iterative. During the subsequent years of the developmental education project, we refined our field note observation forms and focus group protocols based on themes that emerged from the first round of data collection. We used our coding frameworks from the first year of data analysis as a starting point for the codebook in the second year. To do this, we subdivided some of the most frequently used codes into child and grandchild codes and collapsed some infrequently used codes into the parent codes. We also added codes for emergent themes that had not been coded in the first round of coding. For instance, an a priori parent code included “students.” This code was subdivided into “students, general” and “student populations.” However, infrequently used grandchild codes “students, off-campus work” and “students, on-campus work” were collapsed into “student work.”
This process was repeated each year to encompass emergent themes and reflect the overall project’s shifting research question. Our discussions of the process as outlined above provided the foundation for the questions we identify below.
Debating Big Qual
The debate presented here is framed by four related trends to which we attribute the rise of big qual: the rise of big quantitative data, the growing legitimacy of qualitative and mixed methods work in the research community (Creswell & Clark, 2011), technological advances in CAQDAS (Bazeley & Jackson, 2013; J. Davidson, Paulus, & Jackson, 2016), and the willingness of government and private foundations to fund large qualitative projects (Cheek, 2008; Plano Clark, 2010). For each question below, we present the opportunities for innovation as well as the constraints and challenges in research designs for large qualitative data sets that have emerged from our work.
What Opportunities and Constraints Are Presented by Funded Research Involving Big Qual?
Because big qual can be costly to conduct, many, though not all research projects involving large qualitative data sets have been funded by a government agency or private foundation (Cheek, 2008; Plano Clark, 2010). This can be a potential constraint in creating iterative research designs. Miles, Huberman, and Saldaña (2014) have contrasted the linear and sequential nature of quantitative research methods with the iterative and cyclical nature of qualitative inquiry. Indeed, some qualitative traditions such as grounded theory (Charmaz, 2014; Corbin & Strauss, 2008; Denzin & Lincoln, 2017) employ theoretical sampling methods, which study emergent constructs and social phenomena through alternating cycles of data collection and data analysis.
Funding, induction, and iteration
Patton (2015) has commented on the uncertainty inherent in funded qualitative research due to balancing the funders’ need for an intentional, structured, and systematic research plan with the flexibility necessary to explore emergent themes that invariably arise during the research process: How will they [funders] know what will result from the inquiry if the design is only partially specified? The answer is: They won’t know with any certainty. All they can do is look at the results of similar qualitative inquiries, inspect the reasonableness of the overall strategies in the proposed design, and consider the capacity of the researcher to fruitfully undertake the proposed study. (p. 44)
The qualitative story line
Another constraint in funded big qual projects is related to capacity issues. Because funding enables researchers to collect more data, funded projects can generate such a large quantity of data that data reduction techniques become essential. With this volume of data, it can be challenging to separate the “noise” from the main “story line” in the data, making it difficult to answer the question: What are the data actually saying? In some instances, this required us to employ counting techniques in the DE project and then to provide the rationales and methods of counting identified by other qualitative researchers (Hannah & Lautsch, 2011).
In addition, big qual can present challenges for reporting theories and findings in a cohesive narrative when drawn from so much data. Collectively, we have learned to report the theory and research findings generated from big qual in a variety of ways, including single case studies with sections linking individuals’ lived experiences within broad institution-level or system-level patterns (e.g., Brower, Bertrand Jones, & Hu, 2018; Nix et al., in press; Rutledge et al., 2015), and multiple case studies with vignettes illustrating findings within cases coupled with figures and/or tables summarizing patterns across all cases (e.g., Brower et al., in press; Brower, Mokher, Bertrand Jones, Cox, & Hu, 2019; Cohen-Vogel, Rutledge, & Osborne-Lampkin, 2011; Rutledge et al., 2015), and individual examples nested within composite institutions (e.g., Arnault, 2002; Brower et al., 2017; Conant, 2014). Composite institutions (or individuals) involve presenting data in vignettes that contain data from more than one institution or individual. Composite institutions or individuals can be used as a means of consolidating and summarizing large quantities of data as well as highlighting similar patterns across several examples from the data such as in a multiple case study when several cases share important characteristics.
Mixed methods and Big Qual
A factor that can offer either opportunities or constraints in big qual is whether the project is mixed methods. In some instances, the quantitative and qualitative research designs are truly integrated and, in other instances, the qualitative research design is merely an “add-on” to a large quantitative project. Truly integrated designs may increase between-methods or mixed methods triangulation (Burke Johnson, Onwuegbuzie, & Turner, 2007; Denzin, 1978), while quantitative and qualitative research designs can run parallel with little integration when qualitative research is seen merely as an add-on to a quantitative project.
In addition, while it might be ideal for big qual research teams to be comprised entirely of researchers who subscribe to a pragmatist research paradigm, it may be unrealistic in real-world research settings for all members of a team to subscribe to this perspective. Instead, it may be more likely that, taken as a whole, the mixed methods team will subscribe to “dialectical pluralism,” which Creamer (2018) defines as “a paradigm that reflects what some consider to be the overarching logic of mixed methods: the deliberate engagement with different points of view and ways of achieving knowledge” (p. 245). Nonetheless, the dialectical pluralism perspective with its mixture of positivist, realist, social constructionist, and interpretivist research epistemologies can lead to either significant misunderstandings on mixed methods projects or research designs that are more collaborative, triangulated, and ultimately more rigorous.
What Opportunities and Constraints Are Presented by Sampling Strategies Used in Big Qual Research Designs?
We contrast the sampling strategies in quantitative and qualitative research as the difference between probability sampling that seeks to establish generalizability by generalizing from a sample to a population and purposeful sampling that seeks to establish transferability by selecting information-rich cases for an in-depth understanding of a phenomenon (Lincoln & Guba, 1985; Patton, 2015; Peshkin, 2001). From both a methodological and practical standpoint, important reasons remain for conducting deep single-site analyses within narrowly bounded, small n qualitative studies (Patton, 2015).
Extending opportunities for generalization
Nonetheless, we propose that large qualitative data sets may be gradually moving qualitative data away from purposeful sampling for transferability in the direction of sampling for generalization (Maxwell & Chmiel, 2014; Polit & Beck, 2010). Big qual can employ a variety of qualitative sampling strategies, including single significant case sampling; comparison-focused sampling; group characteristics sampling; theory-focused and concept sampling; instrumental-use multiple case sampling; sequential and emergence-driven sampling; analytically focused sampling; and mixed, stratified, or nested sampling (Patton, 2015). Though not all sampling strategies in big qual are alike, the authors’ research projects often employed maximum variation sampling. Though distinct from quantitative random sampling, maximum variation sampling does share a concern with the variability and representativeness of the sample.
We argue that sampling for big qual affords significant opportunities for innovation. Many qualitative researchers have pointed to the capacity of qualitative research for generating theory (e.g., Charmaz, 2014; Corbin & Strauss, 2008; Denzin & Lincoln, 2017; Patton, 2015). We agree. In our experience, conducting big qual research suggests that large qualitative data sets and their sampling strategies can contribute to theory-building in unique ways. Qualitative research has traditionally had the luxury of looking at small bounded systems (e.g., individuals, small groups, organizations). We often delimit qualitative research questions by stating that phenomena are beyond the scope of our study. However, large n studies may decrease the artificial boundaries we create for practical reasons around bounded systems.
Adding depth and breadth to the analysis
Sampling in large n studies may lend itself to bounding cases more broadly by place and/or time (e.g., enactment of a federal education policy over a range of years in different states or student learning processes from preschool through graduate school). Because big qual allows us to conduct longitudinal qualitative research (E. Davidson & Weller, 2016), we can examine stages and cycles of social phenomena because “patterned periodicities provide us with a short- and long-term understanding of how the rhythms of life and work may proceed” (Miles, Huberman, & Saldaña, 2014, p. 211) rather than focusing more narrowly on phenomena from a cross-sectional perspective.
In the DE project, for instance, the central finding of a conference paper based on a single year of cross-sectional big qual data shifted when it was further developed into a journal article based on multiple years of data. Specifically, the cross-sectional conference paper found that, from the perspective of administrators and staff, the open-access mission of community colleges in Florida was compromised by sweeping state-level legislation focused on efficiency (Nix et al., 2016). However, the eventual journal article based on longitudinal data found that some campus personnel became more accepting of reform efforts over the course of four years when they saw that the impact of Senate Bill 1720 on equality was not as negative as they initially feared (Nix et al., in press).
In addition, large n studies with many research participants at multiple research sites can adopt sampling strategies that help to explore a phenomenon nested within multiple units of analysis (e.g., how policy implementation unfolds at the state, institution, group, and individual levels or educational equity at the individual, classroom, school, district, and state levels). To some degree, qualitative research has always allowed us to study social phenomena at multiple units of analysis (Creamer, 2018; Yin, 2013). However, like binoculars that can focus near or far, large qualitative data sets enhance this function by allowing us to dial down to closely examine phenomena at the micro or individual level and then dial out to view phenomena at the macro or societal level. In this way, researchers can begin to discover large-scale patterns linking each level of analysis to a coherent whole. Thus, with the significant variation that is now possible in large qualitative data sets, coupled with the human pattern-identification capability, it may be possible to examine social phenomena outside narrowly bounded systems to generate more elaborate, big picture theories from data.
For instance, a manuscript about students with stigmatized and minoritized identities linked student experiences to efforts of staff to assist these students in managing their stigma to persist and succeed in community college. This article also linked student and staff interactions at the individual unit of analysis to a broader institutional ethic of care as well as the open-access community college mission at the institutional unit of analysis (Brower et al., 2018).
In addition, big qual that are part of mixed methods projects can improve theory by initially theorizing about large systems or hypothesizing about emerging constructs and associations among constructs, which quantitative research can then verify, extend, or disprove.
An example of theory-building from the DE reform project was a qualitatively derived typology of four broad policy implementation patterns (oppositional, circumventing, satisficing, and facilitative implementation) developed in conjunction with 14 specific behaviors such as improvising and leaving the institution (Brower et al., 2017). An empirically grounded typology, or classification of a social phenomenon according to type, can be derived from qualitative data using methods such as those developed by Kluge (2000). Qualitative typologies can be either “indigenous typologies,” derived from research participants’ in vivo classifications of their own cultural settings, “analyst-constructed typologies,” derived from researchers’ identification of patterns in qualitative data or a mixture of both (Patton, 2015). While the case was specific to an individual higher education policy in a particular state context, the breadth and depth of the data likely increased the researchers’ ability to identify the four broad patterns as well as the potential transferability of the theory to other education policies and policy domains. Thus, the big qual sampling strategy helped to ensure that the typology was both exhaustive and comprehensive, increasing the likelihood that all implementation patterns and behaviors had been identified. In addition, the breadth of the data allowed researchers to distinguish between widespread and infrequent implementation behaviors. A small n qualitative study with data from only three institutions would likely have identified a frequently coded behavior like fundamental rule change or improvisation but might have failed entirely to identify leaving the organization as an implementation behavior because it appeared in the data only 3 times across all institutions.
Contextual details in Big Qual
Despite the strengths of big qual research designs, our experience with large n qualitative studies is that with so many participants in the sample, the researchers who collect the data can be more likely to remember contextual details from the most vivid interview participants and research environments. Therefore, even with thorough field notes, some of the contextual richness of field work can be lost in large qualitative studies. As qualitative researchers, we have not abandoned the language of transferability. However, we suggest it may be time to begin posing the following question: If big qual is less focused on examining unique and information-rich cases, is the sampling logic slowly moving away from transferability in the direction of generalization?
What Opportunities and Constraints Are Presented by Team-Based Data Analysis Processes Facilitated by Technology in Big Qual?
Inter-coder reliability is a way to ensure coding consistency and agreement among a team of researchers (Bazeley & Jackson, 2013; J. Davidson et al., 2016). We argue that one drawback of establishing inter-coder reliability in a large qualitative project is that coding for abstract concepts or sensitizing concepts (Patton, 2015), which can be essential in generating theory, may be de-emphasized. For instance, it may never be possible for a team to reach agreement on the definition of abstract yet essential terms such as “metacognition” or “policy entrepreneur.” We recognize that while new technological features of CAQDAS greatly facilitate coding (Bazeley & Jackson, 2013; J. Davidson et al., 2016), they can also constrain coding by making it normative for teams of researchers to use features such as the Cohen’s κ coefficient function to establish reliability among team members. Burla et al. (2008) and Everitt (1996) have reported that κ ranges of .41–.60 represent moderate intercoder reliability, values greater than .60 indicate satisfactory reliability, and values greater than .80 represent nearly perfect reliability. However, κ coefficients are not the only quality criterion, and Hai-Jew (2017) has pointed out that κ scores tend to decrease with a large number of codes and a large number of researchers. Therefore, we suggest that while Cohen’s κ coefficient ranges above .60 may be ideal for big qual projects, the real value in calculating κ coefficients lies in the discussions that necessarily take place among researchers related to their differing understandings of codes, the definitions of codes, and how these definitions apply to the data.
Nevertheless, precise definitions for codes, which are necessary in establishing reliability and consistency in coding among teams of researchers, may contribute to greater clarity with respect to the concepts present in the data. Thus, while the de-emphasis on abstraction can detract from theory-building processes, the greater precision, clarity, and creativity of group processes may be beneficial to identifying patterns in qualitative data and ultimately to theory-building.
Induction, teams, and technology
Some aspects of team-based data analysis in big qual can make inductive research designs challenging. Inductive methods begin, with specific observations and builds toward general patterns.…The strategy of inductive designs is to allow the important analysis dimensions to emerge from patterns found in the cases under study without presupposing in advance what the important dimensions will be. (Patton, 1990, p. 56). more than just a paraphrasing.…It is more than just noting concepts in the margins of the field notes or making a list of codes as in a computer program. Identifying patterns involves interacting with the data using techniques such as asking questions about the data, making comparisons between data, and then developing those concepts in terms of their properties and dimensions. (Corbin & Strauss, 2008, p. 66)
We argue that certain systematic aspects of data analysis with a research team may move the pattern identification process of qualitative research away from emergent data analysis. Technical difficulties can arise in the software file when a coding framework evolves significantly over time with a team of researchers. This relative lack of flexibility in the coding framework can make it more daunting to identify high-level patterns in the data. Due to these difficulties, an individual researcher working alone may have a greater ability to employ emergent coding by changing the coding structure as the project progresses (Saldaña, 2013).
The underlying “reality” in the data
Perhaps more importantly, the necessity of assigning codes to text with precise definitions tends to assume there is one underlying reality in the data instead of many possible realities. As Stake (1995) observed, “most qualitative researchers not only believe that there are multiple perspectives or views that need to be represented, but that there is no way to establish, beyond contention, the best view” (p. 108).
Another challenge that arises with team-based data analysis is related to interpreting data from the etic (cultural outsider perspective) versus the emic (indigenous or cultural insider perspective) (Denzin & Lincoln, 2017; Pike, 1967) and the co-construction of meaning between participant and researcher. In some instances, a subset of the research team in big qual projects will go into the field to collect data, but a larger team of researchers will analyze that data. However comprehensive the interview transcripts and/or field notes might be, researchers who did not collect data in the field lack the benefit of entering into the participants’ cultural setting and do not have the benefit of recalling contextual details (e.g., body language, facial expressions, details in the research environment). Because the number of researchers who did not go into the field can sometimes outnumber the researchers who collected the data on a big qual project, the co-construction of meaning with the participant can also decrease in the analysis process when the etic perspective is emphasized.
In addition, depending on the roles that researchers have played on a big qual project in terms of the amount of data they have collected and analyzed and their number of years working on a longitudinal project, some researchers on the team may have an advantaged position. Specifically, some of the researchers will have a better “bird’s eye view” of the totality of the data, including the themes that cut across years of the project, institutions, and participant groups and how those themes have evolved over time.
Interpreting data through consensus
Despite these constraints, we suggest that group coding may also result in a more collaborative, inclusive, and creative process than individual coding. Weston et al. (2001), for instance, suggest that “a research team builds codes and coding builds a team through the creation of shared interpretation and understanding of the phenomenon being studied” (p. 382). We have found that arriving at a “best interpretation” of data with a team can initially be a time-consuming process. However, our experience conducting data analysis with multiple researchers eventually results in a consensual interpretation that is more comprehensive and nuanced than the interpretation of a single researcher.
Implications of the Debate
Recent technological advances in CAQDAS (Bazeley & Jackson, 2013; J. Davidson et al., 2016) and the increasing willingness of government and private foundations to fund large qualitative projects (Cheek, 2008; Plano Clark, 2010) make this interdisciplinary discussion increasingly essential across academic disciplines. Already, we have seen new research methods for big qual diffuse to diverse fields such as health sciences and medicine, business, education, environmental science, sociology, social work, anthropology, agriculture, and information science (e.g., Armstrong, Riemenschneider, Nelms, & Reid, 2012; Guest & MacQueen, 2008).
We present this debate about large qualitative data sets as a meta-cognitive process intended to spark a broader discussion about whether big qual methods could someday become a qualitative tradition grounded in philosophical underpinnings. In sparking this discussion, we argue that the benefits of big qual include a more inclusive, collaborative analytic process and increased transferability, breadth, depth, and theory-building potential. The challenges of big qual include a de-emphasis on abstract concepts in the coding framework, a possible decrease in the contextual depth in the data set, and potentially an emphasis on the etic researcher perspective.
These benefits and challenges require us to educate students about the traditional aims of qualitative inquiry and to initiate discussions with other researchers about how these aims may be evolving with the introduction of new methods and technological advances in data analysis software. We contend that in order to do this, we must strive to make our methods and their underlying assumptions as explicit as possible. Moreover, it requires us to think deeply about the “whys” and “hows” as we continue to engage in discussions around big qualitative inquiry.
Future Research Directions
Osborne-Lampkin, Cohen-Vogel, Feng, and Wilson (2018) comment on the use of theory in guiding scientific inquiry: Theory provides the ideas researchers use about how phenomenon operate in the world. Empirical studies then tests those theories, using findings from them to modify or refine the theory.…Frameworks help researchers set forth predictions about their study outcomes, shape the study design, and once data are collected, are used as a “mirror to check whether the findings agree with the framework or whether there are discrepancies.” (p. 189)
Despite the recent focus on understanding the ways in which practitioners and policy makers define, acquire, interpret, and ultimately use research in education, an increased understanding of how practitioners and policy makers make sense of and use research continues to be an area ripe for scientific study (Tseng, 2012). We contend that collaboration between practitioners and researchers in developing frameworks and processes for research designs that use large qualitative data sets can enhance or support the development of innovative methodological designs and approaches as well as support efforts to ensure that the “supply-side” attempts of researchers to address the “demand-side” needs of the “end user” are met (Tseng, 2012). Moreover, illuminating frameworks and designs, and developing tool kits that outline methodological procedures can also increase the rigor of research being conducted and the use by practitioners, policy makers, and researchers, alike.
Our questions about the considerations and opportunities for big qual data emerged through our work as researchers engaged in large-scale qualitative and mixed methods studies. Yet less is known about the enabling structures and supports that facilitate this work. Additional research around the organizational structures such as the methodological teams, frameworks, and procedures for carrying out this work deserve additional examination.
In moving forward, we must ask ourselves: Are there opportunities to not only adjust theory and methodological approaches to better-fit big qual designs but also build upon theoretical knowledge? Additional research around the intersections and divergences of big qual in purely qualitative studies and varying types of mixed methods (e.g., qual-quant, quant-qual) studies is still needed. Also, how do we apply that knowledge to develop research questions, design studies, and analytical approaches to enhance our ability to conceptualize and carry out research that better informs policy and practice, either for multiple purposes or for narrowly tailored research questions? Future studies will also need to determine the academic disciplines most likely to employ big qual methods and to explore the emerging research conventions of each by studying the organizational structures, theoretical frameworks, and most common procedures used by researchers.
Conclusion
Engaging in large-scale qualitative or mixed methods data collection and analysis is nothing new. We propose that many newly introduced and still evolving conventions for large qualitative data sets may be changing the fundamental nature of some qualitative research. Thus, we advocate for a frank discussion of research methods to preserve the traditional nature of constructivist qualitative inquiry while remaining open to opportunities for innovation. We acknowledge that our questions may have varying saliency depending on where researchers situate themselves on the continuum of perspectives on qualitative research and hope that these questions spark further debate regarding the nature of qualitative inquiry.
Footnotes
Acknowledgments
We would like to thank Dr. Lora Cohen-Vogel, University of North Carolina at Chapel Hill, and Dr. Stacey Rutledge, Florida State University, for their work conducted with the National Center on Scaling up Effective Schools. We also thank Dr. Carolyn Herrington, Florida State University, and Dr. Jessica Sidler Folsom, Iowa State University, for their work conducted with Dr. La’Tara Osborne-Lampkin at Florida State University. We have built upon the methodological approaches employed in studies and projects conducted by these researchers with Dr. La’Tara Osborne-Lampkin. Finally, we thank Dr. Ralph S. Brower, Florida State University, for his thoughtful feedback on this article.
Authors’ Note
The opinions expressed are those of the authors and do not represent views of the Institute of Education Sciences, the U.S. Department of Education or the Bill & Melinda Gates Foundation.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported in this article was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A160166 to Florida State University, and in part by a grant from the Bill & Melinda Gates Foundation.
