I Can’t Even Think Count Straight: A Queer Revolution in Survey Research?

Abstract

In this essay, I consider how queer methods can inform survey research on sex, gender, sexuality, and other dimensions of social difference and inequality. Critical attention to methodological assumptions, disciplinary norms, and inherited tools can afford sexualities scholars opportunities to reconsider established practices and help chart a more thoroughly transformative era in the sociology of sexualities and related fields. I review a number of recent methodological advances that can facilitate the integration of queer methods across registers of survey research—from study design and data collection to analysis and reporting. I conclude with several recommendations, including that scholars: (1) treat surveys as representational tools, not as neutral mechanisms of revelation; (2) resist statistical fetishism and the tacit notion that statistical complexity equals conceptual sophistication; and (3) recognize that data created by all kinds of surveys can be used as tools for social justice or domination, regardless of their intended consequences.

I will admit to feeling a bit uneasy when I hear concerns among sexuality scholars and other critical methodologists about the need to refine survey methods to better capture human experiences of sex and sexuality. I’ve spent much of my career thinking about the relationships between survey tools and the ways we use them to produce knowledge about sex, gender, race, and sexuality (Grzanka 2016; Grzanka, Zeiders, and Miles 2016). There is certainly room for improvement, and survey researchers must be sensitive to shifts in the ways that, for example, individuals conceptualize and report their sexualities and gender. Indeed, transformations and cross-cultural differences in the terms and categories people use to define and label their sexual orientation and gender identities present important challenges to researchers seeking to do the things most survey methodologists do with data: describe observable patterns (see Silva [2025], this issue). And it is crucial that sexualities scholars stay vigilant about the kinds of questions social scientists and policymakers are increasingly asking to determine who’s queer and gender expansive and how many of us there are (Wang 2023).

I think the ongoing work to more accurately and justly identify and describe diverse populations and social problems is vital, but critiques of existing tools and practices in survey research (including those from the Left) often frame such concerns in epistemic terms that are quite consistent with traditional, masculinist, and positivist perspectives. These are the very frameworks that queer methods, feminisms, critical race theories, and other post-structural and anti-positivist movements have worked to deconstruct (Compton, Meadow, and Schilt 2018; Ghaziani and Brim 2019; Pfeffer 2018). Thus, we should ask ourselves: when we talk about improving survey methods so that we can better assess (e.g., count and account for) people as they really are, are we falling into the positivist trap that seductively suggests we can capture reality if we just figured out how to ask the “perfect” question (Bowleg 2008)? Even when dressed up in the language of intersectionality, for example, the colonial mindset that undergirds the settler heritage of Western social science is a powerful force, and it can smuggle into otherwise critical projects the insidious beliefs that: (a) tools can and should be perfected, (b) only sophisticated statistical analyses can reveal what’s otherwise obscured by existing approaches, and (c) these data can “save us,” as sociologist of science Anthony Ryan Hatch (2022) put it, from oppressive social forces.

I think that questions about critiquing and improving methods must begin with a serious interrogation of our methodologies, or the ideas and values that undergird our use of specific research methods (Harding 1987). This work is not quick, but it is often dirty insomuch as it necessitates the messy feelings and daunting self-reflexivity that come from looking at our cherished, multigenerational, inherited ideas and wondering which of them must be changed, which should be kept, and which should be thrown out (McClelland 2017). One of the bad habits learned in disciplinary settings—hardly exclusive to sociology!—is the failure to even name the epistemic values and goals that motivate a given research project or area of study (Grzanka and Cole 2021). But this is a requisite first step in better understanding our typically tacit investments in what kinds of knowledge are possibly produced by our research. Put simply: what do we think we can know, and why do we want to know it?

Assumptions/Attachments/Disinvestments

To illustrate this point, it’s helpful to consider the similarities and differences in how social scientists in various disciplinary contexts generally conduct survey research. In American sociology, the historical gold standard of survey research has been the nationally representative, probability sample (Luker 2008). One advantage of representative datasets that are collected annually is that this allows for the study of change over time; we can look to the General Social Survey, for example, for evidence of how abortion attitudes have shifted (or not) across several decades. These kinds of datasets are rarely produced by individual researchers or small labs. They are instead the product of massive research operations at major research centers, such as NORC or the U.S. Census Bureau. In this research context, individual researchers who may want to probe these datasets to test hypotheses and/or ask questions about sexualities (or gender, or race) are beholden to data they did not create and questions they did not ask. The sloppy and often crude ways gender identity and sexual orientation—not to mention race and other relevant identity variables—are asked on these national surveys artificially limit what can be learned about the population and arbitrarily constrain, in particular, how nonnormative sexual and gender identities are represented (Baumle 2018). Survey methodologists informed by queer and trans perspectives have made many important scholarly interventions in the past several decades (Westbrook and Saperstein 2015). But the impact of these developments in queer and trans survey research is limited by the extent to which these insights are incorporated into the datasets most commonly used by sociologists, demographers, and other social scientists who are concerned with data that represents demographic trends in the broader population.

In psychology, whose disciplinary norms are less organized around representative datasets, researchers are far more accustomed to creating their own survey research instruments even if they use existing and standardized measurement tools. ¹ In other words, because psychologists often rely on convenience (i.e., nonprobability) samples of undergraduate students or samples collected using online surveys distributed via social media or collected via crowdsourcing networks (e.g., Amazon’s Mechanical Turk [MTurk], Prolific, etc.), they often exercise far more power over the type of data they collect. This does not mean their surveys are invariably or even typically higher quality, but it does mean that they can nimbly tailor questions to suit specific research aims. One advantage of this approach is that surveys can be adjusted to address contemporary and highly dynamic concerns, such as the emergence of a term (e.g., “pansexual”), or its decline in use or cultural salience (e.g., “heteroflexible”). Previously validated scales, such as measures of internalized heterosexism, can also be easily tweaked to function in different populations or replaced to help answer new questions. For example, an item such as “I feel discriminated against as a gay man” used in one study could become “I feel discriminated against as a bisexual man” in a follow-up study. A drawback, however, is that it is difficult to make meaningful comparisons across nonequivalent surveys, which hinders the capacity to confidently assess changes over time for items or questions that are inconsistently asked. Even slight differences in the format (e.g., “Check all that apply” versus “Please choose one”) or content (e.g., “Homosexual” versus “Gay”) in a demographic item can make comparisons spurious or impossible. Though convenience samples are not statistically representative of the population of interest, that may not be the goal of the study, and therefore not a fundamental limitation. Instead, the findings may be framed as logically or analytically representative (Luker 2008) insomuch as they illuminate the substance and contours of a dynamic or construct under investigation (akin to the goals and strengths of qualitative inquiry).

So, on the occasion of the inaugural volume of Sex & Sexualities, an important moment in the history of the sociology of sexualities, I suggest that our commitment to “better” methods would benefit tremendously from sexualities scholars marking and naming our tacit investments (Pfeffer 2018) and disciplinary assumptions. These are practices queer theorists have taught us for generations (Stein and Plummer 1994), but they can be hard to hold onto in the epistemic currents of our disciplines, which sometimes function like a riptide (Grzanka and Cole 2021). Employing relatively straightforward statistics, two recent examples from quantitative sociology (Mittleman 2023; West et al. 2024) suggest the dividends of such a move, especially in a time of increased enthusiasm for the inclusion of sexuality and gender variables in national and international datasets (Wang 2023).

Though statistics are regularly positioned as more objective tools than qualitative methods (Luker 2008), the angle at which one approaches quantitative data can obscure or illuminate what’s actually going on statistically. To illustrate this point, Mittleman (2023) quantitatively explored a theoretical axiom of queer studies that is under-investigated in mainstream population research: sexual fluidity, or the extent to which individuals’ sexual identities and attractions change over time (Diamond 2009). Using five waves of annual data collected since 2013 among a nonclinical population of U.S. adults (N = 17,712 from the Population Assessment of Tobacco and Health [PATH]), Mittleman explored the extent to which respondents’ reported identities and attractions changed over time. He found that among the over 2,500 total identity changes reported in the data, the vast majority (86%) of changes in identity were between three categories: straight, bisexual, and “something else.” Conversely, he found far fewer identity changes among those who identified as gay/lesbian, which was a more “stable” category.

Mittleman (2023) then tested whether accounting for sexual fluidity might change our conclusions about observed health outcomes for sexual minority adults; it did not. However, sexually fluid adults reported significantly worse outcomes than those who consistently reported that they were straight. Sexually fluid respondents were far more like stable sexual minorities than those who were stably straight. Moreover, the failure to account for sexual fluidity grossly underestimated the overall size of the population of sexual minorities in the data, which Mittleman notes is a prime goal of demographic research. So, while accounting for sexual fluidity did not drastically change the conclusions drawn from the PATH survey regarding health inequities, measuring sexual fluidity did unsettle assumptions about what demographic knowledge can be gleaned from cross-sectional (i.e., single timepoint) assessments of who counts as LGBQ+. Mittleman concludes that no single survey administration can accurately quantify how many people “are” queer. Instead, he argued demographers should dedicate attention to the dynamics of social complexity that are already well-established in the sexualities literature. So, the problem for Mittleman isn’t in how we ask about sexuality per se, but how we cut the data and the way we seek knowledge from it.

Relatedly, West et al. (2024) found that national survey data may misdirect our conclusions about sexual minority health outcomes, particularly if measures arbitrarily constrain options for sexual orientation identity. West and colleagues used five consecutive years of the National Survey of Family Growth (2015–2019) to test whether differences in how sexual identity was asked would significantly affect tests of subgroup differences (straight vs. everyone else) on various health outcomes, including substance use and reproductive health. This national survey was randomly distributed with two different versions of the sexual identity question: one 3-question option (gay, straight, bisexual) and one with four choices, including “something else.” They found that, for both cisgender women and men, there were statistically different conclusions on 15% to 16% of the outcome measures between the three- and four-category versions of the survey questionnaire. When using the four-category version, they found significant differences in a variety of dependent variables (three each for women and men), including wanting a/another child, cigarette and marijuana use, and having ever contracted a sexually transmitted infection. Whereas the exclusive straight/gay/bisexual triad reflects heteronormative ideas that “bisexual” captures anyone who is not monosexual in their attraction or identity, the simple addition of a fourth “something else” category allowed West and colleagues to draw more nuanced and statistically meaningful conclusions from the data. Echoing Mittleman (2023), West et al. (2024) do not suggest the four-category sexual identity question is a fix-all solution for measuring sexual identities. They instead underscore the need for all survey researchers to consider “fluidity and complexity when it comes to the construct of sexual orientation identity” (p. 123).

Mittleman (2023) and West et al. (2024) point toward the wily issue of construct validity—or the confidence we have that we’re measuring what we think we’re measuring—when it comes to sexual identity and related constructs. While perhaps a bit “unsexy” as a research topic, construct validity is of profound consequence for the knowledge we create about sex and sexuality with our survey instruments. A perennial problem across the social sciences is that virtually all commonly used survey instruments are developed with and normed upon majority- or dominant-group populations. Take the case of the three-category (gay, straight, and bisexual) sexual identity item West et al. (2024) examined: ask any person under age 25 if that would be sufficient, and you are likely to receive an eye roll. Survey instruments have historically been developed by (older) experts who are historically included in exclusionary institutions. The epistemic exclusion (Settles et al. 2021) of people of color and other marginalized social groups from knowledge production in social science has meant that the social life we’re assessing with our science is largely based on measures reflecting White, cis, straight, male, able-bodied, older, educated, and American perspectives. How might this systematic exclusion undermine our aspirations to better understand both normative and nonnormative sex and sexualities?

In feminist psychology, Galupo et al. (2014) asked sexual minorities (N = 285) what they thought of the Kinsey Scale and the Klein Sexual Orientation Grid, two of the most widely used survey measurement tools to assess sexual orientation/identity. Their respondents offered an extensive qualitative critique of both tools, especially the bipolar Kinsey Scale, along a number of structural and content-based dimensions. They found the scales (1) lacked comprehensiveness in terms of capturing relevant information about sexuality (including sexual fluidity), (2) were dichotomously organized in ways that effaced sexual experiences outside of binaries, and (3) failed to accommodate the ways sexual desires, identities, and behaviors are motivated by things besides sex and gender (such as personality and other ephemeral characteristics). These findings suggest that, at least in this convenience sample, both Kinsey and Klein’s measures lacked construct validity.

Similarly, critical methodologist McClelland (2017) demonstrated how survey tools can (mis)lead researchers into a false sense of measurement validity. Consistent with Cantril’s self-anchored ladder approach popularized in the 1960s in his nationwide survey of American well-being, ² McClelland asked respondents to rate themselves on a 10-point scale of “overall sexual satisfaction” and to define the anchors, as well as the midpoint. In her purposively derived sample of racially and sexually diverse undergraduate students (N = 34), McClelland found some degree of consistency in sexual satisfaction ratings. For example, many respondents used the term “mind-blowing” to describe the best sexual satisfaction, and respondents commonly organized the ladder in terms of emotions and sexual outcomes. However, she also observed meaningful differences in how respondents conceptualized and structured the anchors and midpoint of the scale. Some respondents rated their sexual satisfaction in terms of partner type, time, and who was satisfied; others stressed orgasm (e.g., present or not) or the kind of sex (e.g., oral and anal). For some, a lack of sexual satisfaction involved pain, whereas for others it was the absence of pleasure. McClelland’s findings suggest that while a standard sexual satisfaction rating scale may be reliable, it may be invalid to the extent that respondents mean very different things when they rate their level of satisfaction. By ceding power to respondents, she suggests the self-anchored ladder encourages “conceptual disruption”: a skepticism toward extant research tools and taken-for-granted practices, as well as the embrace of disruptive methods that displace normative standards in each exchange for tools that foreground those perspectives and experiences that have been historically marginalized—in social science, as well as in society.

Galupo and McClelland’s respective projects underscore the extent to which survey research tools can give us false impressions of both (a) the ideological distance between us, as researchers, and our participants, as well as (b) the homogeneity of perspectives among our participants. In other words, if a measurement tool is initially found to be reliable and valid, it can lead us to think we share an understanding of a term or construct—such as sexual pleasure—with most or even all our respondents. However, our respondents (i.e., sample of the population of interest) may actually define this construct in ways that are quite different from how we have conceptualized it. There also may be critical distinctions among our participants such that “sexual pleasure” to one subset or respondents is completely orthogonal to another subgroup’s understanding of the term. This is not the same as measuring actual variance in responses, such as different rates at which individuals report experiencing sexual pleasure, which is something we want to know if it exists. If we think we’re measuring “sexual pleasure” but we’re really measuring something else entirely, this is error, whereby we claim to know something we do not know.

Another way to mitigate these threats to construct validity is to develop survey instruments that are rooted in the lived experiences and actual language of stigmatized minority groups, who are experts in their own lives. In my own research on sexual orientation beliefs—or what everyday people believe sexual orientation is, where it comes from, and the extent to which it can change—my colleagues and I inverted the typical way that prejudice and discrimination measures are developed. Rather than base our Sexual Orientation Beliefs Scale (SOBS; Arseneau et al. 2013) on the worldviews of straight and cis people, we began developing and validating the scale using ideas from queer and feminist theory and the thoughts and perspectives of queer and trans people. This meant that the items we proposed for the scale incorporated social constructionist themes—such as the notion that sexual orientation is culturally and historically contingent—with more depth and frequency than they appear in heteronormative discourse about sexuality. We also used exclusively queer and trans respondents to conduct the initial validation of the instrument and then moved on to study how the measure might perform psychometrically with straight respondents. Every aspect of the SOBS’s development was designed to buck the commonplace, yet methodologically suspect, ways most psychometric work on prejudice is conducted. We wanted to start with the marginalized groups who are the targets of discrimination and anchor the instrument in their subjectivities, rather than those of the dominant group (Grzanka 2016). Ultimately, this allowed us to develop an instrument that moves beyond a reductionist “born this way”-or-not binary and to capture elements of essentialist and social constructionist beliefs that are not exclusively about how straight people perceive or discriminate against sexual minorities.

From Queer Methods to Queer Data?

Though none of the empirical projects I described in the previous section explicitly invoke the term “queer method,” they circulate in a time where increased scholarly attention has been paid to the potential for queer theory to catalyze consequential methodological innovations and contestations, as well as to destabilize the divide between “theory” and “method.” Four celebrated volumes by Browne and Nash (2010), Ghaziani and Brim (2016, 2019), and Compton, Meadow, and Schilt (2018) represent the gestalt of the queer methods movement, which has importantly been led by several key figures in the sociology of sexualities. These contributions have challenged social scientists and humanists to reconsider the role of empiricism in the study of all sexualities (including heterosexualities; see Silva 2017), as well as to deepen the field’s engagement with other critical movements, including intersectionality and anticolonial theories. Though the queer methods literature certainly implicates questions of measurement and the refinement of survey tools, the thrust of this work has more commonly been to address the epistemic contradictions of subjecting the messiness of all that is “queer” to the disciplinarity and order of “method” (Ward 2016). Ghaziani and Brim (2019), for example, echoed sociologists Stein and Plummer (1994) in their call for a queering of method as opposed to a method for studying queer (people, things). Plainly, they write, “Methods are queered when we use the tenets of queer theory to tweak or explore what is possible with our existing procedures” (Ghaziani and Brim 2019:15; emphasis original). The literature on queer methods is littered with familiar rhetoric of queer studies, including terms such as displacing, disrupting, rethinking, interrogating, and resisting—all of these typically directed at violent disciplinary norms and social institutions. But it would be a strong mischaracterization of the queer methods discourse to suggest that the thesis of such a project is the rejection of method, particularly social scientific methods, as fundamentally at odds with the transdisciplinary project of queer studies. As literary scholar Love (2019) notes, “To see one’s practices as beyond method and utterly undisciplined is a failure to reckon with queer scholars’ position in the university; it fails to recognize the violence of all scholarly research—even its most insurgent and intimate forms” (p. 30). Instead, I read and suggest using queer methods as a means to cautiously and even optimistically approach knowledge production with an informed skepticism—to return to McClelland’s (2017) use of the term—toward inherited tools and practices.

Queer data studies is an exciting complement to the queer methods movement, drawing on critical science and technology studies approaches to consider the travels and consequences of knowledge about queer lives. Similar to queer methods, Keilty (2023) notes in the introduction to Queer Data Studies that scholars working in this emerging field are not pro- or anti- data, but instead take data as a site of potential that is always mediated throughout relations of power (see also Budnick, Pao, and Velasco [2025] in this issue). Wang’s (2023) contribution is helpful for considering how sociologists of sex and sexualities might navigate the international zeitgeist for counting up queer bodies in various forms of surveillance, including governmental and activist projects. Rejecting the notion of a data “gap” as a colonial, extractive metaphor for describing what we do not know about queer people and their lives, Wang suggests “situated indications” as an alternative analytic: “data and narrative practices that bounce between situations in an uneven topography to elevate positioned LGBTI experiences and find common contexts of struggles” (2023: 176). Wang evades the question of “good” or “bad” data—and avoids data nihilism or what he calls “scholarly narcissism” (p. 168)—and reframes the stakes of “global datafication” in terms of a dynamic process of indications: “What’s queer is not to be found within the apparatuses and tools (for enumeration or interpretation) but in their multiple, unintended, perverse and thus queer uses” (p. 175). For Wang, provocatively, the question for queer data studies is not so much “What do we think we (can) know?”, but “What do we hope to do with this knowledge?”

I began this brief meditation by naming my skepticism toward scholarly talk about refining survey tools and building better instruments, even though, admittedly, that is a lot of what occupies my time and thoughts when it comes to the sociology of sexualities. Queer methods and, now, queer data studies offer critical resources for avoiding the twin trapdoors of enlightened positivism or universal pessimism. In conclusion, rather than desire the perfect tool, analyze ourselves into statistical oblivion, or leverage our political futures with data we can never fully control, I think a methodological calculus informed by queer methods can reframe our investments thusly:

1. We should humbly and modestly consider how our survey tools and ancillary methods function as representational heuristics—ways of thinking about the empirical world that offer clues to patterns, anomalies, differences, and similarities in experiences, which should be (but are often not) a hallmark of intersectional inquiry (Cole 2008). Our tools can then be thought of as just that: assistants in our deeply human and therefore particular and subjective pursuit of knowledge rather than objective and external measures of phenomena that exist outside of the research endeavor, waiting to be discovered.

2. Statistics, like literally any other mode of analysis, is a language for claims-making and knowledge creation. Complex statistics are not fundamentally better than more straightforward approaches, and statistical abstraction is sometimes helpful and sometimes harmful. We should use statistics when appropriate to the research aims and pursue complexity when necessary, but never as a maxim and never without justification. Further, we should consider statistics to be productive of knowledge, rather than revelatory.

3. Political work in survey research, like the kind many sexualities scholars pursue in the interest of social justice for members of marginalized and oppressed social groups, has the potential to buttress efforts for collective, expansive liberation. But data are never in and of themselves liberatory: data are spoken for and about by political actors, and they exist within discourses that are impossible to control and extremely challenging even to influence (Clare, Grzanka, and Wuest 2023). The same data we think will minimize violence or promote reparation can be used to expand theaters of violence and resist efforts at restitution (see Shuster and Campos-Castillo [2025], this issue). We should question the hope(s) we invest in our data, not because we should be hopeless, but because data—including survey data—are among many tools that can be used to fight structural violence.