Abstract
Quantitative education research is often perceived to be “objective” or “neutral.” However, quantitative research has been and continues to be used to perpetuate inequities; these inequities arise as both intended effects and unintended side effects of traditional quantitative research. In this review of the literature, we synthesize how quantitative researchers have attempted to use critical paradigms to address questions of equity in education research published over the past 15 years. We identify and describe three main tensions that critical and quantitative researchers navigated: (a) creating and analyzing social group categories, (b) trying to describe commonalities within group experiences without erasing heterogeneity of experience within the group, and (c) determining what is a “significant” result when conducting critical and quantitative research.
Quantitative research continues to receive outsized attention in education research, policy, and practice (Garcia et al., 2018). This is due, in part, to the perceived “objectivity” and “neutrality” of numbers (Bonilla-Silva & Zuberi, 2008; Covarrubias & Vélez, 2013; Gillborn et al., 2018). However, critical researchers have highlighted how quantitative research is not neutral and has been used to perpetuate inequities (e.g., Arrellano, 2022; Zuberi, 2001) even as it enables researchers to identify trends across large data sets and make causal claims around people and constructs related to education. Tabron and Thomas (2023), among others, argued that we must specifically acknowledge, understand, and disrupt the “eugenic origins of quantitative research” (p. 760). 1 In education research, statistical analyses have been used to rank and sort students, constructing students whose bodies and behaviors do not fit the statistical “norm” as “abnormal” (Baglieri et al., 2011) or “nonstandard” (Davis, 1997) and deficient.
As a result, even quantitative research in education that attempts to address equity-related issues has led to damaging outcomes and negative side effects for minoritized groups (e.g., Keenan, 2022; Sablan, 2019; Viano & Baker, 2020). 2 In the United States, this research has been used to justify excluding or marginalizing students with disabilities, students who are not cisgender or heterosexual, students who are not White, students from religious minority groups, students whose home language is not English, and students who were born or whose family immigrated from outside of the United States (Baynton, 2001). In other words, quantitative research has often been used in ways that are in conflict with critical paradigms because of broader positivist and postpositivist paradigms underlying most quantitative research (Kivunja & Kuyini, 2017) and the resulting harms caused (Wofford & Winkler, 2022).
However, critical scholars have also argued that quantitative research has the potential to shed light on historical and enduring inequities in educational opportunities and education “debts” (Ladson-Billings, 2006) that have accumulated over time and offer counterstories of and with minoritized communities (e.g., Sablan, 2019). Over the past 15 years, researchers have called for a paradigm 3 shift away from positivism and postpositivism in quantitative education research toward “critical quantitative inquiry” (e.g., Tabron & Thomas, 2023, p. 760). To support this shift, researchers have developed frameworks 4 such as quantitative criticalism (Stage, 2007; Stage & Wells, 2014), critical race quantitative intersectionality (Covarrubias & Vélez, 2013), and quantitative critical race theory (QuantCrit; e.g., Garcia et al., 2018; Gillborn et al, 2018). Using these frameworks, researchers attempt to maximize the benefits of quantitative research, such as demonstrating large-scale social inequities, while minimizing the negative side effects, such as failing to acknowledge heterogeneity within groups in attempts to get large sample sizes.
Many critical and quantitative researchers who draw on and apply these frameworks acknowledge that eugenics was originally an intended effect of quantitative research, not merely an unwanted side effect (Tabron & Thomas, 2023); quantitative researchers, then, must strategically and intentionally disrupt those eugenic origins. Furthermore, scholars who engage in critical quantitative inquiry aim to problematize social constructions and group categorizations, shed light on inequities, and highlight communities’ strengths and assets to disrupt inequities and advance social justice (e.g., Gillborn et al., 2018). Researchers have also adapted conceptual and theoretical frameworks traditionally used for critical and qualitative research to better understand the assets and experiences of specific groups or the effects of particular types of categorization, including TribalCrit (Sabzalian et al., 2021), LatCrit (Covarrubias & Lara, 2014), DisCrit (Cruz et al., 2021), and queer theory (Curley, 2019b).
Recently, scholars have synthesized the growing body of education research that applies critical approaches to quantitative methods, unpacking the epistemological, ontological, and axiological roots (Tabron & Thomas, 2023) and the outlets in which quantitative and critical education research has been published (Tabron & Thomas, 2023; Wofford & Winkler, 2022). In this review, we take a different approach where we focus on empirical research and synthesize how researchers have attempted to minimize the side effects or (un)intended negative consequences of quantitative education research (e.g., Zhao, 2017). 5 Through this review, we shed light on the tensions that arise when trying to maximize the benefits and minimize the negative side effects of applying frameworks, such as quantitative criticalism, critical race quantitative intersectionality, and QuantCrit, that are in line with critical quantitative inquiry. Specifically, the following questions guide this review:
Research Question 1: How do researchers apply quantitative criticalism, critical race quantitative intersectionality, and QuantCrit to minimize the side effects of quantitative research?
Research Question 2: What new tensions arise when researchers apply quantitative criticalism, critical race quantitative intersectionality, and QuantCrit to quantitative methods in education research?
Research Question 3: How do researchers navigate the tensions of engaging in critical and quantitative research?
Separating quantitative research from its eugenic origins under positivist and postpositivist paradigms and creating new frameworks and methods under critical paradigms is a complex process that can lead to new tensions and further side effects. With the growing body of research in this area, it is important to interrogate the effects, side effects, and tensions of conducting critical and quantitative education research.
Definitions and Guiding Frameworks
In this review, we explicitly focus on empirical education research studies that name and apply quantitative criticalism, critical race quantitative intersectionality, and QuantCrit as frameworks. These studies acknowledge that numbers do not simply “speak for themselves” (Gillborn et al., 2018) and that quantitative research is shaped by researchers’ and others’ biases as they make decisions about what data to collect and/or access, how to analyze that data, and how to report the results of their analyses to inform decision-making (Garcia et al., 2018; Stage, 2007).
In this section, we trace the emergence and evolution of critical quantitative inquiry over the past 15 years. However, it is also important to note that education and social science researchers have been engaged in critical and quantitative research for more than a century (Garcia et al., 2018). This scholarship builds on traditions and research of Black scholars and scholars of color who have applied quantitative methods for liberation, transformation, and social justice (e.g., Tabron & Thomas, 2023; Zuberi, 2001; Zuberi & Bonilla-Silva, 2008).
In 2007, Stage edited a special issue of New Directions for Institutional Research, coining the term “quantitative criticalist” to describe a group of higher education researchers (Bensimon et al., 2004; Carter, 1999; Carter & Hurtado, 2007; Drew, 1996; Hurtado & Carter, 1997) who were committed to (a) questioning postpositivist quantitative paradigms, models, and assumption and (b) using large data sets and quantitative methods to shed light on educational inequities. Stage and Wells (2014; Wells & Stage, 2015) then edited two additional special issues of that journal, extending this conceptual framework with calls to conduct “culturally relevant research” in context (Stage & Wells, 2014, p. 3). Stage’s (2007) and Stage and Wells’s (Wells & Stage, 2015) conceptions of quantitative criticalism aligned with the work of critical theorists (Kincheloe & McClaren, 1994; Lather, 1992), assuming the social construction and mediation of knowledge generation, research as a value-laden activity, the role of power and privilege in research, and the assumption that the status quo in research perpetuates oppression (Kincheloe & McClaren, 1994). According to Stage, quantitative criticalist research “asks questions that seek to challenge” the status quo (p. 8) with the goals of equity in and from research.
Along somewhat different lines, guided by tenets of critical race theory, Covarrubias and Vélez (2013) proposed a “critical race quantitative intersectionality” (CRQI) framework, defined as “an exploratory framework and methodological approach . . . that utilizes quantitative methods to account for the material impact of race and racism and its intersection with other forms of subordination” (p. 276). Through CRQI, Covarrubias and Vélez aimed to shed light on social constructions and how they affect people of color through “critical manipulation” and “contextualization of data” that are represented in numerical terms.
Extending this work, Garcia et al. (2018) edited a special issue of Race, Ethnicity, and Education on critical race theory and quantitative research, in which Gillborn et al. (2018) adopted the term “QuantCrit,” applying the tenets of critical race theory to quantitative methods. Gillborn and colleagues offered five principles for conducting quantitative and critical research, including acknowledging:
The centrality of racism
Numbers are not neutral
Categories and groups are neither “natural” nor given: for “race” read “racism”
Voice and insight: data cannot “speak for itself”
Social justice/equity orientation. (p. 169)
Gillborn and colleagues stressed the importance of using quantitative methods to advance social justice, mirroring calls within qualitative research to consider “transformational” and “transactional” approaches to validity that consider who benefits from research in addition to technical questions about how research should be conducted (e.g., Cho & Trent, 2006).
As Tabron and Thomas (2023) argued, the different names are more than just “wordplay,” although researchers sometimes use the names of these frameworks interchangeably. Rather, they represent different foci, schools of thought, assumptions, and goals. In this review, we synthesize empirical education research that has applied these frameworks to uncover the ways in which researchers navigate the side effects of quantitative research and the tensions that arise as researchers attempt to maximize the benefits and minimize the harmful side effects of quantitative research using these frameworks.
Researcher Positionality
In writing this review, many aspects of our identities and experiences informed how we read and analyzed the existing research. Because the articles reviewed addressed a wide range of topics, identities, and experiences, we had different relationships with each article.
Some aspects of the first author’s positionality that most frequently came to the forefront while writing this article were my Jewish identity and experiences as an able-bodied “critical special educator (Connor, 2013) from a “hyper-diverse” city (Malsbary, 2016). As a Jew, I see the lingering effects of eugenics on my own community and am aware that other communities continue to be affected as well. I do not “own” disability, but I have a responsibility to push back on ways in which statistics have been used to construct differences as deficits in schools by using numbers to sort and rank human diversity. The regular conversations I have about the complexities, pleasures, and challenges of sorting humans into groups and communities have also shaped my understanding of some of the nuances of labeling and categorizing.
As a U.S.-born White woman, daughter of immigrants, mother, former middle school teacher in a large mid-Atlantic city, and current university professor, the second author reflects on my positionalities and how they shape research and practice. In my research on social-justice-oriented teacher education policy and practice, I have drawn on and applied critical and quantitative frameworks to address questions related to (in)justice and (in)equity in teacher education and teaching. Like many of the scholars included in this review, I view research as a political act and raise questions about whose interests are served, whose perspectives are represented, and whose voices are included or excluded.
Together, we believe that quantitative research, like all research, is not neutral but, rather, is subject to the assumptions of the researcher and the context of the research. Quantitative research has caused great harm but also has the potential to be used to provide counternarratives and to better understand and disrupt social inequities. As researchers, we wrestle with the challenges of representing the experiences and identities of communities and groups who are (and are not) different from us while also recognizing our responsibility to not only conduct research about communities with our shared identities and experiences. We understand the importance of incorporating and citing the expertise of members of any community we are writing about (Tabron & Thomas, 2023) while also recognizing that nobody can speak for specific communities. In this review, we drew on our multiple intersectional identities and our experiences as quantitative researchers to synthesize the literature and raise questions about critical quantitative inquiry, social group categories and labeling, and statistical methods. We interrogated the ways in which we have been captured and (mis)represented in large data sets, drew on our cultural intuitions to interpret findings, and acknowledged how our prior quantitative research is subject to similar critiques and tensions that we raise in the following.
Methods
To identify literature for this review, we searched the Educational Resource Information Center (ERIC), Google Scholar, and ProQuest Education databases using the terms “critical quantitative” or “quantitative critical*” or “quantcrit” and “education,” filtered by “peer-reviewed.” We focused on the years between 2008 and 2022, looking for articles that came after the special issue of New Directions for Institutional Research in 2007. This search process initially yielded 143 publications.
We also conducted electronic searches of journals that published special issues on quantitative and critical perspectives (i.e., New Directions for Institutional Research, Race Ethnicity and Education), tracing research that cited major conceptual publications on quantitative and critical perspectives. For example, we cross-checked citations from the most cited articles on critical and quantitative research in education (e.g., Gillborn et al., 2018; Sablan, 2019; Stage, 2007) and references from two recent reviews of the literature on critical and quantitative research (i.e., Tabron & Thomas, 2023; Wofford & Winkler, 2022).
Based on these initial searches, we read abstracts to identify articles that were empirical, peer reviewed, and explicitly named and used critical approaches to quantitative inquiry. We then read full texts of articles and looked for studies that clearly documented the purpose of the study, theoretical and conceptual frameworks, participants, data sources, quantitative analyses, and findings. We independently identified studies that fit the criteria and met to discuss studies that one of us identified as not meeting the criteria. We excluded studies that did not explicitly name a specific critical and quantitative approach (e.g., Garces, 2012; Young & Young, 2018) and those that employed qualitative or mixed methods (e.g., Covarrubias et al., 2018; Russell et al., 2022; Sabzalian et al., 2021).
After identifying the studies for this review, we carefully read each study. We created a shared Google spreadsheet where we described each study, including the broader educational topic, research purpose and questions, theoretical and conceptual frameworks applied (along with authors cited), sample/participants, data sources, analyses, key findings, tensions addressed, general observations, and journals of publication. Over a period of 9 months, we met biweekly to discuss specific studies and explore areas of convergence and divergence across studies. As we met, we identified and took notes on the framework named and used, how researchers navigated or addressed specific side effects of quantitative research, and the broader tensions that arise in each study as researchers attempted to maximize the benefits and minimize the side effects of quantitative research. Following these meetings, the first author analyzed across the literature to clarify tensions and develop categories and emergent themes. The second author reviewed initial analyses, and the two authors discussed themes and tensions, pointing to consistencies and inconsistencies across the literature. Together, the two authors constructed the three themes outlined in the following, going back to notes and the literature for review.
Trends Across the Critical and Quantitative Research
Across the literature, we identified 58 empirical studies that met our criteria, noting the increase in the number of publications over the past 5 years (see Figure 1). The reviewed articles were published between 2012 and 2022, ranging from zero publications (2012 and 2016) to 15 publications in a year (2021 and 2022). In particular, we noted the increased number of empirical studies applying critical paradigms to quantitative methods following special issues on quantitative criticalism (Stage & Wells, 2014; Wells & Stage, 2015) and the Race Ethnicity and Education special issue on critical race theory and quantitative methods (see Garcia et al., 2018; see Figure 1).
Number of Studies Using Critical and Quantitative Frameworks by Year
Approximately half of the studies (n = 28, 48%) cited Stage (2007), Stage and Wells (2014), and Wells and Stage (2015), applying “quantitative criticalist” and “critical quantitative perspectives”; half of the studies applied QuantCrit (n = 29, 50%), citing Gillborn et al. (2018) and Garcia et al. (2018); and a smaller fraction (n = 10, 17%) applied CRQI, citing Covarrubias and Vélez (2013) and Solórzano and colleagues (Solórzano & Ornelas, 2002; Solórzano & Villalpando, 1998; Solórzano & Yosso, 2001). Eight studies (14%) applied a combination of conceptual frameworks including critical quantitative inquiry (i.e., citing Stage, 2007) and QuantCrit (i.e., citing Gillborn et al., 2018 Garcia et al., 2018) or CRQI (i.e., citing Covarrubias & Vélez, 2013).
The empirical research was published in general education research journals (e.g., American Educational Research Journal, Education Policy, Teachers College Record); higher education journals (e.g., International Journal of Higher Education, Journal of Applied Research in Community College, Journal of College Student Development); critical education journals that focus explicitly on race, in/equity, and oppressive systems and structures in education (e.g., Equity & Excellence in Education, Journal of Underrepresented and Minority Progress, Journal of Negro Education); educational administration journals (e.g., Educational Administration Quarterly); subject-specific journals (e.g., Investigations in Mathematics Learning, Journal of the American Chemical Society, Physical Review Physics Education Research); and urban education journals (e.g., Urban Education, The Urban Review; see Table 1).
Journals and Number of Studies Using Critical and Quantitative Frameworks
We found that researchers aimed to amplify the experiences, perspectives, and assets of minoritized groups and highlight the effects of oppressive structures in and on education systems across studies. Although the authors varied in terms of how they labeled and categorized groups, researchers considered participants’ ethnoracial and cultural identities (n = 53, 91%), gender and sexual identities (n = 20, 35%), socioeconomic status (n = 7, 12%), college generation status (n = 2, 4%), citizenship or migrant status (n = 2, 4%), home languages (n = 1, 2%), religion (n = 1, 2%), and disability status (n = 1, 2%). Approximately one-third of studies (n = 20, 35%) attempted to consider intersectional identities of the focal samples and participants (e.g., by addressing race, gender, or dis/ability).
Slightly more than half of the studies in this review focused on higher education (n = 31, 54%), and the remainder focused on elementary and secondary education (n = 26, 46%). A handful of studies focused on the educational pipeline and attainment from elementary through postsecondary employment (i.e., Alemán et al., 2022; Covarrubias & Lara, 2014; Covarrubias & Liou, 2014). Most studies explicitly focused on the experiences, opportunities, and outcomes of students (n = 46, 79%), whereas a smaller fraction of the studies focused on educators and educational policies, including administrators, teachers, faculty, teacher evaluation, and curriculum (n = 12, 21%).
Researchers focused on a variety of constructs related to educational outcomes, including student achievement, persistence, and graduation/completion rates across grade levels and disciplines (Pérez Huber et al., 2018; Suárez et al., 2021; Van Dusen & Nissen, 2020; Wronowksi et al., 2022); historical patterns of exclusion and opportunity gaps (e.g., Wilson & Urick, 2022); dispositions (e.g., Young & Cunningham, 2021; Zerquera & Gross, 2017); institutional and caring agents (e.g., Xiong, 2021); microaggressions (e.g., Powell et al., 2024); campus climate; and sense of belonging (e.g., Oxendine & Taub, 2021).
Navigating Tensions and Side Effects of Critical and Quantitative Research
As researchers engaged in critical and quantitative research, they grappled with tensions that arose as they attempted to maximize benefits and minimize the harmful side effects of quantitative education research. We identified three main tensions: (a) tensions related to creating and analyzing social group categories, (b) tensions related to trying to describe commonalities within group experiences (e.g., anti-Black racism as distinct from other forms of oppression) without erasing heterogeneity of experience within the group, and (c) tensions in determining what is a “significant” result when conducting critical and quantitative research (see Figure 2).
Navigating Tensions Across Critical and Quantitative Research
Tension 1: Social Group Construction and Categorization
Determining which groups to create and who belongs in each group is a complex and political decision rather than a neutral process of sorting and labeling (Gillborn et al., 2018; Strunk & Hoover, 2019). Sorting and classifying research participants can lead to benefits, such as identifying patterns of discrimination, and negative side effects, such as essentializing groups. Many systems of categorization, like race, are contextual and have changed—and likely will continue to change—over time and to serve different purposes. 6 For example, the category “Hispanic” or “Latinx” is often considered an ethnicity, not race, on many official surveys. Participants, then, can select “Hispanic or Latinx” and also a racial category (e.g., White, Black). However, researchers often use “Latinx” as a racial category for analysis to compare with other racial categories (e.g., Jang, 2019, compared the experiences of Latinx and White students while coding for “national origin”). Although participants may have the option to choose both “White” and “Latinx,” researchers often do not include those participants in the same category as others who identify as White and not Latinx (e.g., Alemán et al., 2022).
Other systems for categorizing and classifying people also represent political acts based on a long history of oppression and marginalization in the United States. For example, many surveys ask participants to select between only two options (i.e., male or female) for identifying gender. Strunk and Hoover (2019) argued that “asking participants to select identity categories with which they do not identify can, in and of itself, be oppressive” (p. 195). Several researchers (e.g., Van Dusen & Nissen, 2020) posited that missing data related to gender identity should be treated as an intentional decision to opt out of the gender binary rather than as missing data. Researchers (e.g., Curley, 2019b) have recommended offering more expansive categories of gender identity or allowing participants to write their own preferred identification.
Across studies, researchers attempted to minimize the side effects of social group construction and categorization while using preexisting categories from secondary data sets (e.g., Allen & Wolniak, 2019; Bielby et al., 2014; Cormier et al., 2021; Jang, 2019; Maffini & Dillard, 2022). When researchers did not make the original decisions about how to categorize participants, they gave explicit descriptions of the benefits and side effects of the existing system of categorization. Sometimes, they renamed categories or reclassified participants. For example, while applying the CRQI framework (Covarrubias et al., 2018), Alemán and colleagues (2022) created the category “Latina/o and Chicana/o” from census data to describe participants who indicated they had “Hispanic origin” that was Mexican.
In other studies, researchers tried to minimize the side effects of using secondary data by collecting their own data that addressed the intersectional identities, including racialized and gendered identities, of participants. Applying the QuantCrit framework, these studies included explicit descriptions of those decisions rather than defaulting to commonly collected identity markers (e.g., Powell et al., 2024; Van Dusen et al., 2021; Wronowski et al., 2022).
As we elaborate in the following, decisions around social group categorization shape the ways in which researchers shed light on inequities and take into account and draw on the assets of the multiple intersectional identities of individuals and groups. Additionally, social group categorization guides the types of statistical analyses researchers conduct, how researchers interpret data, and ultimately, how researchers determine the importance and relevance of the research.
Tension 2: Understanding Commonalities in Groups’ Experiences Without Erasing Heterogeneity
One benefit of quantitative research is its ability to identify and describe broad patterns clearly and concisely, which can make inequitable opportunities across groups apparent. However, one side effect of focusing on broad patterns is that diversity within groups may become flattened or erased. Another side effect of sorting and categorizing people is that it can suggest that there are essential differences across groups, particularly when the same groups are used regardless of the topic being studied. Critical and quantitative researchers tried to navigate this tension by making strategic decisions to focus on a single group, create a binary, or add additional groups. However, each of these decisions involved navigating different benefits and side effects, described in the following.
Focusing on a Single Group
Nineteen studies focused on the experiences and perceptions of one group of participants, such as Black math teachers (Frank et al., 2021), Native American 7 undergraduates (Oxendine et al., 2020), or southeast Asian community college students (Xiong, 2021). These authors explained that they applied CQRI and QuantCrit frameworks to avoid “gap gazing” (Young & Cunningham, 2021)—deficit-based comparisons between a dominant reference group and everyone else. Instead of making comparisons, these studies focused on understanding the trajectories of a single group. For example, applying the QuantCrit framework (Gillborn et al., 2018; N. Lòpez et al., 2018), Young and Cunningham (2021) examined Black girls’ dispositions and growth in math to offer a counternarrative of specific group experiences. Although each of these studies focused on a single group, some of these studies focused on diversity within the group being studied to disrupt essentialization of that group, as when Covarrubias and Liou (2014) attempted to better understand diversity within the category “Asian American” by disaggregating by family, gender, socioeconomic status, and citizenship.
However, one side effect of focusing on a single group is that the larger context that members of these groups exist within, including inequitable opportunities afforded to different groups, could not be explored or addressed. For example, Howe and colleagues (2022) focused exclusively on the experiences of women graduate students, but their findings that women graduate students expressed uncertainty about how to balance academic careers with family life did not address differential expectations faced by mothers versus fathers in academia (much less the unique pressures faced by nonbinary parents).
Creating a Binary to Highlight Inequitable Opportunities
Alternatively, 11 researchers sorted participants into categorical binaries to highlight inequitable opportunities offered to dominant versus nondominant groups or to recenter the experiences of groups that have traditionally been constructed as “other.” Creating a binary can illuminate privilege experienced by a dominant group. For example, following the principles of critical quantitative perspectives (Stage, 2007) and applying QuantCrit (Gillborn et al., 2018), Abrica & Hatch-Tocaimaza (2019) compared the experiences of White and “underrepresented minority” college students. In other studies, Campbell (2023) compared the experiences between White and Black women teachers, and Maffini and Dillard (2022) compared the perceptions of White and Black college students. Across these studies, the researchers applied principles of QuantCrit to make explicit that the experiences and perceptions of White participants represent the experiences and perceptions of a group that has enjoyed unique privilege in the United States rather than a group that is meant to be the reference “norm” or ideal that all groups are supposed to emulate.
However, one side effect of creating binaries is that sorting people into only two categories radically simplifies and potentially oversimplifies the complexity and diversity of human experience. For example, Fong and colleagues (2019) compared the perspectives of Indigenous and non-Indigenous students, suggesting that there was an essential sameness within each group’s beliefs (e.g., that the beliefs of Black, White, Asian American, and Latinx students are basically the same and interchangeable) rather than differences across how groups experienced U.S. power structures. In another study, Zerquera and Gross (2017) focused on Latina/o students in higher education but compared their experiences with faculty of color versus White faculty, suggesting that all faculty of color may automatically know how to support Latinx students.
Some binaries also did not adequately untangle the relationships among privileged and marginalized aspects of individuals’ identities. For example, N. López and colleagues (2018) used “dynamic centering” to make White women the reference group in their study because White women were the group with the highest educational attainment in their sample. They acknowledged that White men would be the “center” or reference group in a study of income inequality but did not critically examine the difference between whiteness, which affords many advantages in the United States, and female gender identity, which does not. 8
Similarly, Wronowski and colleagues (2022) compared the experience of college students who identified as practicing any religion with students who identified as atheist, agnostic, or “none.” However, this comparison ignored the ways in which Christianity, like whiteness, is often not acknowledged, allowing students who come from Christian backgrounds to identify as “none,” whereas members of religious minority groups, such as Muslims or Hindus, are more likely to recognize the cultural dimensions of religion, identifying, for example, as a “secular Jew” rather than as a neutral “none” (Edwards, 2021). In other words, those who identify as religiously “none” are likely to still experience some Christian privilege, for example, if they wish to spend Protestant/Catholic Christmas with their family and can count on paid time off work, whereas those who come from other religious traditions would be more likely to need to use vacation time to spend holidays such as Diwali, Eid, or Rosh Hashanah with family.
Pérez Huber et al. (2018) compared the educational attainment of Latinas/os with “whites and Asians” in places, suggesting that the experiences of Asian American students are the same as those of White students. However, Covarrubias and Liou (2014) explicitly critiqued quantitative researchers’ tendency to create a binary with Asian American and White participants on one side and Black and Latinx participants on the other side, explaining, “This often reinforces the model minority stereotype of Asian Americans that presumes their school success can be conflated with that of White Americans” (p. 2). The model minority stereotype minimizes or erases the discrimination and racism that Asian Americans continue to face in education and the larger U.S. society (Goodwin, 2010). The model minority stereotype that all Asian American students are successful also conceals a great deal of diversity within the category “Asian American” (Jang, 2018).
Increasing the Number of Groups and Subgroups
Applying QuantCrit frameworks, three researchers who collected their own data expanded the number of racialized and gendered categories participants could select (Van Dusen et al., 2021; Wronowski et al., 2022) or allowed participants to describe their identities in an open-response text box (Knowles & Hawkman, 2020) so participants could identify with a broader range of categories or even create new categories. However, this approach had the side effect of creating many very small groups, which were challenging to work with using inferential statistical methods that rely on large sample sizes. Furthermore, attempts to understand diversity, heterogeneity, and individuality sometimes obscured group trends. As researchers divide participants into more groups, the size of each group decreases, which can make it harder to have enough statistical power to find statistically significant results (Schudde, 2018). In each study, researchers wound up combining, consolidating, or removing small groups to conduct inferential statistical analyses.
Tension 3: Competing Understandings of “Significance”
Traditionally, quantitative researchers have treated the “significance” of their results as a primarily technical question that can be answered with large sample sizes and close adherence to established statistical methods and analytic procedures. However, critical and quantitative researchers have pushed back on the idea that the experiences of small groups are insignificant or that fidelity to a particular method is sufficient for producing a significant result. Levin (2006) summarized concerns about traditional significance testing, including that the term “significant is terribly misleading,” suggesting importance rather than probability (p. 531). One advantage of using large samples is that they can illuminate the ways in which larger social systems work. Forms of oppression, such as racism, are institutionalized scripts in U.S. schools (I. F. H. López, 2000), which can lead to predictable outcomes that are legible in large-scale data sets. For example, large-scale quantitative research can show ways in which Black students face exclusionary discipline practices more frequently and more severely than White students for the same types of behavior (e.g., Rocque & Paternoster, 2011); these large-scale patterns provide evidence that racism, not individual student characteristics or behaviors, is the causal factor leading to discrepancies in treatment.
To minimize the side effects of using traditional quantitative methodologies to identify and understand significant effects, differences, and relationships, researchers (a) explored alternatives to using the p value or alpha coefficient as the sole determiner of significance, (b) used and problematized latent class analysis and exploratory factor analysis as tools to better understand the structure of data within and across groups, and (c) wrote positionality statements to make the role of the researcher explicit in determining what counts as significant results.
Alternatives to p Value or Alpha Coefficient
Traditional determiners of statistical significance, such as p values, rely on large sample sizes to try to rule out the likelihood that relationships among variables occurred by chance. One side effect of using statistical methods that require large samples is that quantitative researchers often omit members of small groups (e.g., trans students) or aggregate small groups into larger groups (e.g., Indigenous students may be aggregated into a group of “students of color” even though Indigenous students may identify as members of a particular nation rather than as a racial group). Calls to “abandon” or “demote” the p value are appearing in mainstream statistics journals, with recommendations to, for example, use the p value as a continuous measure to help interpret results rather than as a strict cutoff (e.g., McShane et al., 2019).
Three studies explicitly addressed the role of p value in research conducted from critical and quantitative frameworks. Two took steps to reduce the likelihood that the experiences of small groups would be considered insignificant by increasing the p value threshold or abandoning the p value entirely. Mahatmya and colleagues (2022) chose to set the p value at .15 in their study of BIPOC (Black, Indigenous, and other people of color) women teachers in Iowa. Alternatively, Nissen and colleagues (2021) chose not to use p values at all, explaining, “p-values depend on sample size and lead to selective reporting and selective attention that can ignore injustices borne by the most underrepresented and marginalized groups of students.”
On the other hand, Stewart (2013) used p < .01 in a study of the experiences of “racially minoritized” college students, explaining that using a lower p value was feasible due to “the overall large sample size” (p. 189) of respondents to the College Senior Survey. However, the authors excluded students who identified as “American Indian” or “other” due to small sample sizes of those groups, demonstrating the tension between increasing statistical likelihood and inclusivity to small groups.
Using Quantitative Methods to Understand Underlying Structures and Patterns in Data
Seven articles used quantitative methods as tools to try to better understand the relationships among variables in their data. Specifically, five studies used factor analysis, and two used latent class analysis. Wilson and Urick (2022) found significant differences across six latent classes based on the interactions between race/ethnicity (Black, Latinx, or White) and socioeconomic status (more or less affluent). Although this finding certainly contributes to the literature suggesting that racism and economic inequality continue to affect educational opportunities, it would be hard to interpret results if the authors had found that these classes were not significant—that would not necessarily mean that racism and economic inequality no longer mattered. These latent classes were also “predominantly,” but not exclusively, made up of, for example, Black versus White students, suggesting that race/ethnicity mattered in this sample but was not the only relevant factor in determining latent classes.
Framed by QuantCrit, Powell and colleagues (2024) used exploratory factor analysis and item response theory to examine the psychometric properties of a survey of the teachers’ experiences with racial microaggressions (TERM) scale. They found that “no items on the . . . scale were being viewed differently because of the gender of the teacher” and concluded that “this scale is able to capture the shared racialized experiences of Black teachers, male or female.” On the one hand, this finding suggests that the authors have found something essential and shared about the experience of anti-Black racism; on the other hand, as the authors noted, most other research discusses “the nuances and differences at the intersection of race and gender identity,” acknowledging that anti-Black racism may be experienced somewhat differently by male, female, and nonbinary teachers.
One side effect of using factor analysis is that outliers must be removed (Beavers et al., 2013). However, outliers are not just data points; they represent the experiences of particular participants whose experiences may differ from the norm in ways that are important to understand. Curley (2019b) argued that “framing outliers as central in quantitative studies challenges the harmful expectations and norms of quantitative research” (p. 2); we discuss the implications of explicitly studying, rather than discarding, outliers in the discussion section.
Being
Explicit About the Role of Researcher Positionality and Bias in Quantitative Research
A key tenet of critical and quantitative research is that statistical analyses are not any more objective than other research methods because quantitative research is still conducted by humans, who make subjective decisions about what data to collect and how to analyze, interpret, and report that data. Accordingly, 11 articles we reviewed included positionality statements to be more explicit about how the researcher’s beliefs, biases, and lived experiences affect the research. However, a side effect of positionality statements is that they risk becoming a laundry list of census-driven categories, social positions, and identities or an act of performative allyship (de los Ríos & Patel, 2023). In eight of those articles, researchers did not follow up or make connections to that statement elsewhere in the article to describe how the authors’ positionalities impacted various methodological decisions. Van Dusen and Nissen (2020d, 2022) stated that they had asked for an “equity audit” by outsiders but did not mention what the audit found or how they addressed the results of those audits.
However, in three of the articles, researchers were explicit about how they drew on their “cultural intuition” (Garcia et al., 2022; Pérez Huber et al., 2018) or “experiential knowledge” (Harmon et al., 2023) throughout the research process to guide their methodological decisions, particularly their analyses. Harmon and colleagues (2023) explained that “there were specific instances where the statistical results, when interpreted in isolation, might contradict the lived dynamics between a Black father and son” (p. 471) in their study, and so they used their own experiences as Black fathers of Black sons to resolve those contradictions and interpret results. For example, their study found that the sons of “well educated and well to do” Black fathers reported feeling less welcome and supported at school. The authors interpreted these results to reflect Black fathers teaching their sons how to recognize, understand, and navigate racism in school rather than a causal relationship between higher levels of education and income and lower levels of sense of belonging.
Discussion and Implications
Across articles, we found examples of researchers trying to mitigate common side effects of quantitative research related to creating labels and categories, trying to identify commonalities of experiences without erasing heterogeneity, and determining the significance of results. In the studies we reviewed, researchers used statistics to understand enduring inequities but also to provide assets-based counternarratives that illuminated the strengths and funds of knowledge of minoritized groups. Yet researchers’ attempts often created new side effects, leaving researchers to navigate tensions that arose as they attempted to maximize the benefits and mitigate the harmful side effects of their methodological decisions. In this section, we offer suggestions for using quantitative research to promote equity rather than further marginalizing individuals and groups who have already been historically marginalized by traditional quantitative education research.
Research conducted from a critical and quantitative perspective must contend with the fact that categories such as race, gender, or dis/ability are social constructions with real consequences. Applying the principles of QuantCrit, one common way that researchers have addressed this tension is in the analysis of their results, for example, recognizing that the effects of “race” are actually often the effects of racism; race, then, may simply be the proxy variable for this form of oppression in many data sets. Existing categories may fail to fully capture the complexity of individuals’ identities or how their identities and group memberships change across time, space, and place (Kaplan & Garner, 2020). However, many categories still have explanatory power, so researchers should use them cautiously, acknowledging the complexity and limitations of these labels.
Most of the articles we reviewed used secondary data; using secondary data is quicker, easier, and cheaper than collecting new data. However, one side effect of using secondary data is that the researcher has less control over what data are collected, which categories are used, and how participants are sorted into categories. Researchers must be explicit, then, about the strengths and limitations of the data they are using, as when Van Dusen and Nissen (2020) explained that participants who chose not to select one of two options for gender may have reflected an intentional choice rather than an error.
Future research using critical and quantitative perspectives can explicitly examine processes of categorization to better understand how participants think about their own identities and group memberships. Researchers might use text boxes to allow participants to describe their own identities rather than giving participants a list of options to select from (e.g., Curley, 2019b). If researchers decide to combine categories for quantitative analysis, they can engage in member checks to see if the aggregate category continues to represent participants’ understanding of their own identity and group membership (Godwin, 2020). Viano and Baker (2020) allowed participants to choose multiple options and then asked “which of these identities best represents” them as a way to balance allowing participants to self-identify with the need for creating larger groups for analysis.
Critical and quantitative research can also explicitly engage with how categorization changes over time. Goldstein (2005) argued that “the construction of identity” is not “a stable process generated only from within” but “involves a constant negotiation with the shifting social and cultural circumstances” (p. 79). Some of these changes are formal and bureaucratic, as when the U.S. census first gave respondents the option to choose to identify multiple racial classifications in 2000, formalizing the official recognition of identities that many people already claimed in other contexts. Some changes may reflect changing circumstances, as when an able-bodied person becomes disabled or when someone moves to a new country and becomes an immigrant. Other changes may reflect changes in how people construct their own identities and which groups they align with over time. Researchers must contend with the ways in which individuals in their samples and they themselves have shifting identities over time.
Future research can also explicitly examine differences between how individuals may self-identify and how others may categorize those individuals. Campbell-Montalvo (2019) addressed this explicitly, comparing differences in how individuals were racialized across different documents in two elementary schools. N. López and colleagues (2017) distinguished among “street race,” “socially assigned race,” and “self-perceived race,” acknowledging that people may be racialized differently by different people (including themselves) across different settings and for different purposes.
Most articles we reviewed focused on race and racism as units of analysis, often including other, intersectional aspects of participants’ identities as well (e.g., gender). Sen and Wasow (2016) argued that quantitative researchers should conceptualize race as a “bundle of sticks” or composite variable rather than an essential and unitary category. This framing allows quantitative researchers to further investigate heterogeneity of experiences and perspectives within a racial or ethnic group by disaggregating those “sticks” and examining them separately. Disaggregating the variables that make up “race” is similar to intersectionality in that it acknowledges diversity within particular groups (e.g., that students from the same racial or ethnic group may have different experiences) but also different in that it allows researchers to understand commonalities across groups (e.g., ways in which students from different ethnoracial groups may also share common experiences with the same “stick”).
Quantitative research can explicitly address how the researcher’s values and positionality affect the decisions that researcher makes throughout the research process rather than assuming that data can “speak for itself” (Gillborn et al., 2018, p. 169). Any individual researcher, or even research team, is necessarily limited in perspective; these limitations become particularly worrisome when members of dominant groups are conducting research into the lives and experiences of members of minoritized groups. Curley (2019b) gave examples of how someone who is not adequately familiar with the diversity of how people understand and describe their gender and gender identities may misinterpret some responses as errors or mischievous responses and “inadvertently describe someone’s identity as ‘wrong’” (p. 5). Tabron and Thomas (2023) advocated drawing on the expertise of individuals and groups outside of those represented by the research team through reading and citing research from members of focal groups in the research.
Quantitative education research should also acknowledge the full humanity of its participants, for example, through “person-centered analyses” (Godwin, 2020), rather than giving researchers full and total authority over every decision. Person-centered quantitative analyses might include processes such as member checking to allow participants to weigh in on researchers’ decisions, for example, around consolidating categories for analysis, and to make sure that researchers’ analyses and findings remain aligned with the experiences and perspectives of groups being studied.
Outliers and small groups have traditionally been viewed as threats to the significance and validity of many types of statistical analyses; however, Curley (2019b) argued that outliers can form the basis of case studies rather than being discarded because outliers represent heterogeneity in people’s experiences and perspectives in social science research. Godwin (2020), similarly, encouraged researchers to understand individual responses and data points in addition to larger trends. Schudde (2018) offered several suggestions for understanding heterogeneous effects, including ways in which members of small groups may be affected differently by the same treatments: (a) using larger p values to reduce the likelihood of Type II errors when studying small groups, (b) using power analysis to explore useful sample sizes, and (c) using theory along with cultural intuition to support models.
Quantitative researchers must also understand ways in which dominance and oppression are and are not related to numerical size in particular places and contexts. Not all minoritized groups are numerical minorities. In some school districts, racial and ethnic “minority” groups may actually make up the majority of students, yet their experiences are not necessarily adequately or accurately represented by education research. On the other hand, some groups are genuinely very small in many educational settings: Trans students, d/Deaf or hard of hearing students, Indigenous students, members of religious minority groups, and others may represent 1% to 2% (or less) of any random sample of students or educators across the United States. In some districts or regions, particular ethnoracial groups may be exceptionally small as well. Researchers who want to study very small groups face unique methodological considerations when conducting quantitative research about these groups because many types of statistical analyses require large sample sizes.
Rather than allowing the effects on those small groups to “wash out” (Curley, 2019b) into the larger population’s average findings, researchers can conduct separate analyses on small groups, including descriptive statistics if the samples are too small for other types of analyses. Schudde (2018) noted that administrative data may be a useful source of data about small groups because they include the entire population of, for example, students in a district rather than just a sample. Scholars can also explore and develop new quantitative research methods that are appropriate for small groups or exploring diversity within groups.
Researchers can also draw on multiple and mixed methods using critical frameworks; as Smith (2006) argued, “reality is complex and contingent, then so research must be” (p. 471). Covarrubias and colleagues (2018) explored how critical and quantitative research can also include testimonios that are “intersectional and transformational” to better understand complex quantitative data and use it to promote equity.
Limitations
Our review synthesized empirical research that explicitly named specific frameworks (i.e., quantitative criticalism, CRQI, and QuantCrit). This limited our review in scope and time. We did not include other research that is consistent with a critical quantitative paradigm that named or applied other frameworks (e.g., critical race theory) to shed light on oppressive systems and structures or the assets of minoritized groups through quantitative approaches (e.g., Young & Young, 2018). Furthermore, this review omits critical and quantitative empirical research that predates the publication of these frameworks and, in particular, does not fully incorporate the long history of Black scholars and scholars of color who have been engaged in critical and quantitative research for more than a century (see e.g., Zuberi, 2001).
Additionally, our study was limited to peer-reviewed empirical research that was published in academic journals. We did not include book chapters or dissertations. These publication sources also offer valuable contributions and applications and new ways of thinking about critical and quantitative approaches in education research (see e.g., Curley, 2019a; Saunders, 2015).
Furthermore, our review explicitly excluded research that applies critical and quantitative frameworks but employs mixed methods (e.g., Garcia et al., 2022; Sabzalian et al., 2021). This decision allowed us to focus on the effects and side effects of quantitative research. However, this decision also failed to highlight the power and potential of mixed methods to mitigate the harmful side effects of traditional approaches to quantitative research, which we explore in the discussion section.
Conclusion
Although quantitative research in education has historically been (and continues to be) used to cause harm, through this synthesis, we argue that research methods are, ultimately, tools that can be used for a variety of purposes, including to promote equity. A paradigm shift to new frameworks such as quantitative criticalism and QuantCrit may help mitigate some of the side effects of dominant postpositivist approaches, yet more work needs to be done. The majority of research we reviewed focused on race and ethnicity, but critical and quantitative research methods can be used to better understand how religion, dis/ability, home language(s), gender identity, national origin, and other aspects of students’ and educators’ multiple identities and experiences affect education as well, using an assets-based lens to better understand communities’ strengths and ways in which oppression continues to impact specific communities in different ways. Critical and quantitative research can continue to develop tools like microaggression scales (e.g., TERM; Powell et al., 2024) to better understand perspectives and lived experiences rather than to sort and rank people. Researchers can also contextualize data within place, space, and history rather than trying to generalize to broader populations. Finally, researchers should continue to advance quantitative and critical frameworks while also problematizing and advancing paradigms, frameworks, and statistical methods as they work toward disrupting inequities and advancing justice in education and education research.
Footnotes
Acknowledgements
We thank Joni Kolman and our anonymous reviewers for their helpful feedback to improve our article.