Theory before method: The case for methodological pluralism in political science

Abstract

Political science is undergoing a pronounced methodological shift toward causal identification and design-based inference, increasingly marginalizing qualitative and observational approaches. We argue that this shift rests on the flawed premise that methods can be ranked in the abstract, independently of the research question and the theories at stake. Drawing on a Popperian understanding of scientific progress, we develop a framework of comparative theory testing in which method choice is derived from the competing theories themselves. When rival theories generate observationally non-equivalent implications—causal, correlational, or descriptive—any form of evidence capable of adjudicating between them has epistemic standing. This grounds methodological pluralism not in a normative appeal for inclusivity, but in the internal logic of rigorous theory testing. We illustrate the framework through canonical examples, propose criteria for evaluating the rigor of comparative theory tests, and show that design-based methods earn their place within—not above—this framework. The result is a unified perspective that preserves our capacity to engage big-picture questions while maintaining the standards of falsifiability and theoretical precision that scientific inquiry requires.

Introduction

Comparative theory testing has its roots in a Popperian understanding of science, wherein progress comes through the empirical evaluation of competing explanatory theories by focusing on their observationally non-equivalent testable implications (Popper, 2002 (1959): 9, 10). In this tradition, comparing causal theories about why a phenomenon occurs is at the heart of scientific progress. Scholars put forth falsifiable causal claims, with empirical implications about ‘who, what, when, where, how, or in what order’ type of questions. In political science, testing alternative theoretical arguments and their different implications has typically called for descriptive evidence (e.g., behavior/beliefs of actors and timing/sequence of events) as well as evidence on the direction and size of causal relations. This tradition continues to be visible in highly acclaimed books about political regimes, ¹ state formation, ² nation building, ³ and mass political violence ⁴ among others. Τhese works employ theory-based research designs, selecting the methods and evidence to test their main arguments against alternatives. In line with this tradition, we argue that choices regarding methodology and evidence should be guided by the research question and relevant theories, rather than some abstract methodological hierarchy.

Comparative theory testing diverges from a dominant methodological tradition in political science that can be traced back to classic works such as Lijphart (1971) or King, Keohane, and Verba (KKV) (1994) and which more recently has been refined by certain causal identification approaches (Gerber and Green, 2012; Green, 2005). ⁵ In this tradition, scholars articulate strictly correlational or causal claims (based on randomization). The priority is to establish causal connections between variables over focusing on whether and how these connections fit within a broader theoretical framework, or the validity of a general theory. As a result, observational evidence—particularly of a qualitative kind—is treated as a flawed approximation to experiments. By contrast, theory-based research is agnostic between different types of evidence and methods used in theory testing so long as they conform to a set of rigorous criteria.

What causal identification approaches often miss, we contend, is the deeper theoretical relevance and interpretability of findings, including the mechanisms driving effects, their durability over time, and their generalizability beyond narrow contexts. For example, a landmark field experiment by Gerber and Green (2000) established a causal link between get-out-the-vote interventions and voter turnout, finding that face-to-face canvassing significantly increased participation while phone calls had no effect. While this rigorously identifies an average treatment effect between mobilization tactics and turnout, it provides limited insight into the underlying causal arguments—such as the psychological or social mechanisms at play, why certain messages resonate more, or whether these effects should persist across elections, demographics, or regions without additional theoretical scaffolding. Note that this is not an argument against the Gerber and Green experiment as such or experiments in general. Our point is that this approach represents one application of the theory-testing logic where a randomized experiment is the appropriate instrument to test one type of implication—for example, the causal effect of mobilization on turnout—not a universal template for empirical political science.

In juxtaposition, let’s look at a different example from the comparative theory testing approach that does not involve an experiment. Ziblatt (2006) asks why some states adopt federal state institutions whereas others develop a unitary state structure. Ziblatt tests his argument on infrastructural capacity against the existing ideational or power-based arguments through a comparison of two instances of state formation: the unification of Germany led by Prussia (federal outcome) and the unification of Italy led by Piedmont (unitary outcome). The book compiles a wide array of evidence ranging from historical statistics on state capacity and military might for all pre-unification German and Italian states to descriptive evidence focusing on the two main actors that led unification in Italy and Germany, Piedmont and Prussia. The latter part of the analysis relies on works by historians and archival material such as private correspondence to test expectations on leaders’ ideological leanings and sequence of events. From our perspective, Gerber and Green (2000) and Ziblatt (2006) are equally rigorous works.

We advocate for a renewed emphasis on comparative theory testing, where methodological choices are tethered to the substance of falsifiable competing theories rather than a rigid hierarchy favoring experimental designs, thus fostering methodological pluralism that values evidence of all types in so far as they help us adjudicate between observationally non-equivalent testable implications of theories. This perspective does not penalize scholars who tackle questions that defy randomization—such as historical processes of democratization or institutional emergence. Instead of forcing phenomena into experimental molds, the research design should emerge from the research question, at times necessitating qualitative depth, in other cases calling for quantitative or experimental approaches. This approach counters the narrowing of inquiry toward ‘tractable’ questions, preserves the discipline’s ability to engage big-picture puzzles, and elevates works that might otherwise be marginalized in high-impact outlets.

Evidence of this de-prioritization of methodological pluralism is striking. The APSR’s editors’ reports document this trend, showing that articles using case study/Small N constituted only 4.3% of the works published 2020–2024 (Gerring et al., 2025). ⁶ This trend has also been echoed by editors of journals like the AJPS when they write: ‘A leading trend in recent years has been growing concern with causal inference, and constructing empirical tests that convincingly demonstrate causal processes outlined in theories’ (Reiter and Berinsky, 2025). And in the same paragraph, they caution that ‘an exclusive focus on causal inference risks narrowing the field, in the sense that inevitably some areas of great scholarly significance experience limits regarding the degree to which causal inference can be established within plausible empirical designs’ (Reiter and Berinsky, 2025). But this trend is also increasingly discussed in recent works across political science subfields (Ashworth et al., 2021; Binder, 2020; Clark and Golder, 2015). Despite surface-level pluralism in the discipline overall, design-based methods have come to dominate the venues that matter most for careers: they now lead in the top 20 journals, are concentrated among authors at elite institutions, and command a growing citation premium—all per Torreblanca et al.’s (2026) analysis of over 91,000 articles published in 156 political science journals from 2003 to 2023.

Our argument builds on existing discussions in the field but also diverges in important ways. On the one hand, debates sparked by the so-called Perestroika movement and critiques of KKV’s Designing Social Inquiry (King et al., 1994) challenged the growing dominance of formal modeling and quantitative methods in the early 2000s, advocating for greater methodological diversity and the inclusion of qualitative and interpretive approaches. While the Perestroika critique is compatible with our argument, our point is not about equity amongst different methodological approaches, but rather that methodological choices should be linked to the research question and the testable implications of the competing theories. On the other hand, post-KKV discussions have refined how we use qualitative methods while questioning the potential for randomization and, by extension (quasi) experimental designs, in subfields in which data are by and large produced by history (Brady and Collier, 2004; Przeworski, 2007). While we concur with these points, we argue that the underlying logic of comparative theory testing does not justify a priori treatment of any methodological approach or type of empirical evidence as preferable over others.

What distinguishes our contribution from these earlier critiques is the specific theoretical ground on which we defend methodological pluralism. Perestroika argued for equity among approaches, while post-KKV critics argued against overrelying on quantitative methods and underutilizing qualitative ones. We ground our case for methodological pluralism in the internal logic of theory comparison itself. Once we accept that causal theories generate multiple types of observationally non-equivalent implications—causal, correlational, or descriptive—then our methodological choices must follow those implications rather than an external hierarchy. This moves the discussion from a normative appeal for fairness or feasibility concerns about specific methodologies to an epistemological argument about what rigorous theory testing actually requires. The closest precursors to our argument do not make the case. Goertz and Mahoney’s (2012) two-cultures argument is essentially taxonomic. They describe two existing research traditions and argue for their mutual respect, but their argument has no mechanism for deciding which methods a new research question should use, because method choice is determined by which culture the researcher already inhabits. Our framework answers that question directly by deriving method choice from the theories under examination. Seawright’s integrative approach similarly begins from existing method types and asks how they complement each other, rather than starting from the theory and asking what it requires (Seawright, 2016). Neither framework adjudicates method choice from first principles. Ours does.

In what follows, we first present our understanding of theory comparison and its implications for the role of evidence in research design, applicability of process tracing as a basis for multi-method research, and principles of case selection. We then discuss potential limits of our approach before concluding.

Comparing theories and research design

The fundamental building blocks of theory comparison are competing falsifiable theories, each generating logically consistent and empirically distinguishable propositions that relate to a phenomenon of interest. Scholars testing alternative theories cast a wide net and consider multiple plausible explanations before trying to adjudicate. We argue that all methods that produce evidence that differentiates between competing theories explaining a specific phenomenon are valuable, thus moving beyond abstract methodological debates detached from specific research questions and theories.

By theory, we mean a simplified version of the world that offers an internally consistent causal story about how a specific phenomenon occurs. Arguments that are too broad or unfalsifiable, that is, lacking mechanisms or clear testable implications, are not considered theories in our framework. Theory comparison requires taking two steps. The first step is logical, and it involves laying out the observable implications of the relevant arguments. This step also allows the researcher to identify the dimensions (such as timing, sequence, location, and features of actors) on which the various theories have divergent testable implications. The second step is empirical, and it refers to the practice of explicitly considering the relative validity of alternative theories based on evidence that would tilt the balance.

Based on this understanding, theory-comparison encompasses and goes beyond restrictive understandings of causal identification. When comparing theories, the type of empirical evidence that is useful depends on the substance of the theories to be tested. Competing theories by definition disagree on some of their testable implications (hence are not observationally equivalent), yet they might have a subset of implications sharing the same expectations. The key to comparing theories is selecting the type of evidence that increases our confidence in one argument while reducing our confidence in a competing argument or, at the very least, leaving it the same.

To illustrate our point concerning how different testable implications flowing from various theories can be adjudicated using different types of evidence, we consider competing arguments on the historical emergence of representative institutions. Table 1 summarizes simplified versions of two competing theories within this research area as recently examined by Deborah Boucoyannis (2021).

Table 1.

Emergence of representative institutions.

Dimension of expectation	Type of expectation	Explanatory theory: War-taxation	Explanatory theory: Justice-provision
1: War	Causal	War proneness increases the likelihood of representative institutions.	Strong institutions increase the likelihood of war.
2: Domestic power balance	Correlational/Causal	Contexts in which rulers are more powerful than their subjects are less likely to develop representative institutions.	Contexts in which rulers are more powerful than their subjects are more likely to develop representative institutions.
3: Function of representative institutions	Descriptive	Early parliaments focus on taxation and war.	Early parliaments focus on resolving conflicts between domestic actors.

A strand of bellicist theory, which for brevity we call War-Taxation Theory, argues that the underlying impetus for the emergence of these institutions was the pressure to wage war, which required the ruler to raise taxes (Tilly, 1990). ⁷ In this theory, the ruler would ideally prefer to raise taxes without yielding any control to the subjects, but representative institutions that meet regularly and allow elites to supervise the ruler’s decisions emerge if the latter lacks the military or financial power to wage war without making domestic concessions. Boucoyannis shows, however, that taxation was a very infrequent concern of early assemblies, so it cannot possibly account for their emergence or consolidation. Instead, for her theory, which we call Justice-Provision Theory, the main impetus for representative institutions is the predictable and standardized resolution of conflicts among the main actors or groups in a society. In this theory, the ruler’s main function is to provide justice, that is, resolve conflicts before they escalate in ways that are costly for the parties involved, and representative institutions help her fulfill this function (Boucoyannis, 2021).

Table 1 shows that these two theories yield testable implications along several dimensions. The first relates to the role of war: The War-Taxation Theory expects contexts that are more war-prone to be more likely to develop representative institutions; by contrast, the Justice-Provision Theory suggests that strong representative institutions increase capacity for waging wars, but war proneness does not result in representative institutions. In other words, while both theories expect a positive association between incidence of war and existence of representative institutions, their causal arrows face opposite directions.

The second dimension relates to the relative power of the ruler vis-à-vis her leading subjects: War-Taxation Theory expects contexts in which the ruler has more relative military and financial power compared with her strongest subjects to be less likely to develop representative institutions; whereas Justice-Provision Theory expects the opposite, that is, contexts in which the ruler has more relative power (i.e., the means to enforce her decisions) would be more likely to develop representative institutions.

Finally, the two theories offer divergent expectations on the primary functions of representative institutions once they emerge. According to the War-Taxation Theory, the main function should relate to decisions on taxation and war-making. According to the Justice-Provision Theory, the primary function should be focusing on resolving economic and social conflicts among domestic actors (Boucoyannis, 2021: 87–105).

This example illustrates our point about testable implications and the different types of evidence they call for. The mere observation that war proneness and representative institutions are positively correlated cannot adjudicate between the two theories, since both agree on this empirical expectation. Additional research is necessary to determine the direction of causation between war and representative institutions. Importantly, the two theories have different expectations on whether the relationship between domestic balance of power and emergence of parliaments is positive or negative, which makes correlational evidence sufficient for this expectation. Last but not least, a key testable implication that empirically distinguishes between the theories concerns the functions of parliaments once they emerge, which requires descriptive evidence on the debates and decisions of early parliaments.

As the discussion above makes clear, given the multiple types of empirical implications that can be derived from causal theories, there is no basis for embracing a generic ranking of methods or types of evidence independently of the research question. All methods that produce evidence capable of differentiating between competing theories are valuable. To return to the brief illustration above, studies that produce one of the following three types of evidence would equally serve the purpose of testing the two theories against each other: (a) evidence that establishes the direction of the causal arrow between wars and representative institutions—for example, advanced statistical analysis using instrumental variables and/or process tracing based on historical evidence that establishes sequence in cases; (b) large-N historical data to provide correlational evidence on the relationship between representative institutions and the (financial and military) power of kings compared with the aristocracy; (c) archival data and/or secondary historical sources to study the type of issues that were discussed in medieval parliaments. Since all three types of evidence speak to key testable implications that distinguish between the competing theories, there is no a priori basis for arguing that one is more desirable than another, though, of course, compiling more types of evidence would test the theories more thoroughly than relying on one type of relevant evidence.

Stating that methods are valuable in so far as they produce evidence capable of differentiating between competing theories is a necessary but not sufficient standard for rigorous testing. Any form of evidence can be used well or poorly within this framework. We therefore propose the following criteria for rigorous comparative theory testing. First, the competing theories must generate genuinely non-equivalent testable implications—if they agree on all observable dimensions, no evidence can adjudicate between them. Second, the evidence brought to bear should target the specific dimensions on which the theories disagree, not merely confirm the predictions of the favored theory. Third, the selection of evidence must be transparent and replicable. Sources should be identified clearly, their potential biases acknowledged, and cross-checking against sources with differing biases is preferable to reliance on a single archive or dataset. Fourth, the credibility of a verdict increases with the number and variety of non-equivalent implications tested. A single confirmatory finding carries less weight than a pattern of converging evidence from multiple distinct implications. These criteria apply equally to qualitative, quantitative, and experimental evidence—which is itself part of our point. The question is never which method is used, but whether it is used in a way that genuinely tests competing theories against each other.

Comparative theory testing and its methodological implications

Our understanding of comparative theory testing has methodological implications in three areas: the role of descriptive evidence, process tracing and multi-method research, and practices within case studies.

Role of descriptive evidence

From a theory-comparison perspective, the apparent dichotomy between descriptive and causal evidence, currently reified in political science, breaks down. KKV explicitly refer to ‘dual goals of describing and explaining’ and equate explanation with linking a cause and an effect (King et al., 1994: 34). In this framework, descriptive inference sequentially comes prior to causal inference and is valuable to the extent that it contributes to the latter. ⁸ Other scholars sustain the duality but go beyond the KKV understanding. Gerring (2012), for instance, defines descriptive arguments as those that answer what, when, who, and how type of questions, whereas causal arguments respond to why questions. Thus, while description is a necessary step toward causal analysis, it should also be valued independently when description focuses on analytically important phenomena and contributes to more efficient data collection (Gerring, 2012).

We agree with Gerring’s justifications for granting independent status to description, but we argue that descriptive evidence can also be key to answering why questions. The answers to what, where, who, and how questions are an integral part of the scientific process not only because they offer definitions of important phenomena or result in efficient data collection, but also because they help us adjudicate between competing theories about why a phenomenon occurs. The answer to the question of ‘what did medieval parliaments do?’ is valuable not only because this question is important in its own right, but also because the answer helps distinguish between two competing theoretical arguments about why representative institutions emerged: one that is driven by a logic of war and taxation, the other by a logic of justice-seeking. Thus, descriptive and causal evidence are equally valuable, not only because they can fulfill complementary purposes, but also because they can serve the same purpose while testing implications of competing causal theories.

This point suggests a more prominent role for qualitative methods in today’s political science. Descriptive evidence is typically produced by the detailed study of a small number of cases based on historical material (archives and/or secondary sources) or interviews, focus groups, or ethnographic material. An approach that focuses on theory testing treats descriptive qualitative evidence and causal identification practices as equally relevant insofar as they help adjudicate between testable implications.

Yet our broader point is also applicable to testing causal arguments using descriptive evidence in the form of large-N summary statistics. Consider, for example, a recent article that tests the empirical implications of a theory labeled ‘fickle prosociality’ (Roman and Thompson, 2025). The argument puts forward two empirical expectations. First, when members of a marginalized group, such as the LGBTQ+ group, face substantial violence from civilians, the initial societal response is an increase in positive attitudes toward the group in question. Second, this shift decays and reverts to initial levels of prejudice as time passes and the event loses salience. The authors use surveys conducted before, right after, and significantly after two salient events in which members of the LGBTQ+ group were targeted to evaluate these expectations. When testing the first expectation, which links civilian targeting of marginalized groups to improvement in societal attitudes, the article uses correlational evidence showing that respondents interviewed post-event were more likely to display support for LGBTQ+ individuals. The second expectation concerns when and how trends in attitudes toward marginalized groups vary over time. Thus, when testing this theoretical expectation, the article uses descriptive statistics on temporal trends in attitudes to show that the initial post-event hike in support decreases and disappears over time.

Process tracing and multi-method research

Among the methods used by political scientists, process tracing broadly understood is the one that most closely embodies the logic of the practice we outline (Collier, 2011; Mahoney, 2015; Mylonas and Shelef, 2017). Process tracing starts with the formulation of internally consistent arguments about how a certain outcome occurs and then derives multiple testable implications from these arguments. Such empirical expectations can be derived from the main causal mechanism that links initial conditions to an outcome of interest or take the form of auxiliary implications that follow from the internal logic of the argument (Collier, 2011). ⁹ In addition, process tracing generally considers testable implications that allow the researcher to empirically compare different theories (Bennett and Checkel, 2014). Recently, scholars have made this analytical underpinning of process tracing even more explicit through the application of the Bayesian approach (Bennett, 2014; Fairfield and Charman, 2017). ¹⁰ As Ashworth et al. (2021) put it, empirical findings gain credibility when they update our theoretical priors in surprising ways, rather than merely confirming isolated causal links without broader context.

Scholars who put forth theoretical causal claims, generating testable implications about ‘who, what, when, where, how, or in what order’ type of questions, use process tracing to render their research analytically transparent and relevant to testing causal arguments. We posit that the underlying logic of process tracing, that is, comparing competing causal theories, should be emulated by scholars irrespective of the types of methods they use to bring about evidence: qualitative, (observational) quantitative, (quasi) experimental. ¹¹

This approach is also broadly compatible with understandings of multi-method research that envision different methodological approaches as playing complementary or ‘integrative’ roles in explaining social scientific phenomena (Ingram and Harbers, 2020; Lieberman, 2005; Seawright, 2016). Lieberman’s nested-analysis, for instance, suggests that we should test whether the main causal mechanism maps onto how events unfold in the cases-on-the-regression-line. This increases our confidence that there is a causal link, not just a correlation. Seawright’s ‘integrative’ approach underlines the importance of qualitative evidence in testing assumptions that experimental and/or observational large-N analyses take for granted (Seawright, 2016, 2021). Steps in our causal mechanisms and assumptions we make about initial conditions or shared conceptual understandings can all be understood as testable implications of competing theories. Therefore, the multi-method research tradition fits comfortably within the broader framework of comparative theory testing we describe here.

Process tracing, as a broad way of comparing theories that encompasses different types of methods, can also help reconcile the ‘two cultures’ of qualitative and quantitative approaches (Goertz and Mahoney, 2012; Mahoney and Goertz, 2006). In their influential article, Mahoney and Goertz identify key distinctions between qualitative and quantitative approaches in terms of values, beliefs, and norms. They argue that the two cultures differ in their approach to explanation, which for qualitative works is about accounting for an outcome in a single case (‘causes of effects’ approach) and for quantitative works about estimating the average effect of independent variables (‘effects of causes’ approach). ¹² By extension, they argue that the two methods differ in their understanding of causality, with qualitative works utilizing the logic of necessary and sufficient causes, and quantitative ones adopting a probabilistic approach. In our framework, we bring the two cultures together in two ways: First, comparative theory testing is by and large compatible with a ‘causes of effects’ approach in that the main goal is explaining a given phenomenon. Yet, we argue that this approach is applicable to both abstract questions about social phenomena (e.g., why do representative institutions emerge?) and case-specific ones (e.g., why did representative institutions emerge in England?). Second, we highlight that theories often yield many types of implications. Some may be formulated as necessary and/or sufficient conditions, while others might entail probabilistic expectations about general trends in a population of units (e.g., individuals, subnational units, organizations, or states). What matters is whether repeated and different types of tests, utilizing qualitative or quantitative evidence, tilt the balance in favor of some theories over others.

Implications for case selection

Theory comparison also carries implications for case selection in small-N studies. A classic answer to the question of how to select cases, dating back to John Stuart Mill, is that we should choose cases to approximate the experimental setup. Thus, ideally all relevant factors except for the one of interest should be controlled for to isolate the impact of the main explanatory factor. In this vein, an ideal small-N comparative case analysis requires researchers to match countries, or other units, on as many relevant factors as possible, and then observe whether the presence or absence of a potential explanatory factor coexists with the presence or absence of the outcome of interest. Cases should therefore be selected to ensure variation on the dependent variable.

Yet it has long been obvious to researchers using case analysis to test theories that this understanding of controlled comparison does not accurately capture what it achieves in practice (Slater and Ziblatt, 2013). The main function of controlled case comparison is not taking a snapshot of the presence or absence of an independent and a dependent variable. In fact, within the framework of testing competing theories, the expectation that the independent variable coexists with (and precedes) the dependent variable is neither the only nor, necessarily, the most important observable implication. Competing arguments can generate several empirically distinct implications that go beyond the coexistence and sequence between an independent and a dependent variable. Furthermore, competing theories might in fact agree on the main explanatory factor but disagree on the causal logic that connects it to the outcome of interest. Therefore, depending on the nature of the research topic, one could potentially test competing theories with a variety of empirical evidence from cases where the dependent variable takes only a positive or high value, provided the scholar focuses on implications which go beyond the simple coexistence and sequence of independent and dependent variables. ¹³

Indeed, it is possible to find compelling studies with no or limited variation on the dependent variable. One example is Amartya Sen’s classic book, Poverty and Famines. Sen (1982) compares theories on famines that focus on crop failure and food shortage with his argument on exchange entitlement. The case selection in the book includes countries in South Asia and Africa during periods of time in which they all experienced famines. Yet the analysis is still meaningful as it tests the different expectations of the two theories against several descriptive features of famine: trends in the price of food, movement of food out of famine areas, and location/social status of victims (Varshney, 2008).

Another compelling example with limited variation on the dependent variable comes from Adria Lawrence’s work on insurgencies in colonial contexts. Lawrence (2010) tests the role of anti-colonial grievances in explaining violent mobilization, against her own argument about the role of internal competition within the nationalist elite. The empirical analysis primarily comes from one case in which a nationalist insurgency took place, Morocco in the 1950s, and hence does not display variation in the dependent variable. ¹⁴ Yet, the analysis is cogent as it juxtaposes the divergent testable implications of the two opposing theories along several observable aspects of the conflict: fracturing in the nationalist movement, targeting by the insurgents, and continuation of violence after the French retreat.

Limitations of comparative theory testing

We now turn to the potential limitations of our framework, addressing questions of external validity, reliability, and its epistemological foundations.

External validity and methodological pluralism

A central critique pertains to external validity. Testing causal claims through in-depth observational evidence from a particular case may yield high internal validity, but raises questions about how far the findings travel. The obvious solution would be to conduct more rather than fewer studies that rely on in-depth analysis to evaluate rival theories (also see Goertz and Haggard, 2023). Such studies may be carried out either by individual researchers who share a common thematic focus but specialize in different temporal or geographical contexts, or through a more centralized model in which a team of scholars undertakes a large-scale project conducted across multiple settings, drawing on comparable evidence to test the same competing arguments.

When assessing external validity, qualitative evidence must be judged against the performance of (quasi) experimental or observational Large-N analysis on this same dimension. Like process tracing, field and survey experiments generate high precision evidence within a specific context (Pepinsky, 2019), and like historical process tracing, they offer no guarantee that results would replicate elsewhere (Cartwright, 2011; Findley et al., 2021; Pepinsky, 2019). Large-N observational research faces analogous challenges. Examples include issues of ecological fallacy and critiques of the assumption that regression coefficients are representative across units (Aronow and Samii, 2016: 255).

Reliability and methodological pluralism

A further objection is that methods relying on qualitative accounts are often subjective and difficult to replicate. Yet, this problem is not unique to qualitative work but it applies equally to quantitative research, where scholars rely on historical sources to compile their datasets, and even to randomized controlled trials. For instance, surveys designed around specific events—like pre- and post-event studies—capture unique temporal contexts that cannot be precisely reproduced. Additionally, subjective choices in survey design–such as question framing (e.g., loss vs. gain domains), the use of images versus text, or the choice between fictitious and realistic vignettes–introduce variability that can influence results. These decisions reflect researchers’ judgments about what will yield meaningful data, yet they undermine the notion that quantitative methods are inherently more objective than qualitative ones. While randomized controlled trials mitigate some of these issues, they share with qualitative research a reliance on subjective methodological choices, pointing to a broader challenge for replicability across the social sciences.

These problems are not completely insurmountable in qualitative research either. The concerns regarding subjectivity and reliability undersell the knowledge that historians produce. While historians disagree about the broad conditions that lead to specific events or even on whether such conditions can be established, they tend to agree on facts-on-the-ground such as whether and when events occurred, the order in which they occurred, where they occurred, and the socio-political position of the actors involved. Returning to theory comparison, if our testable implications bear on any of these dimensions, such information can serve as a reliable basis to adjudicate between competing theories. And so long as we provide precise references to the archives and secondary historical accounts documenting dates, actors, and locations, then this research process remains replicable.

Historical accounts and/or archival sources can also generate reliable information on more complex topics, such as the composition of a group or organization, long-term socio-economic indicators, or functions of institutions. Where testable implications concern such topics rather than uncontroversial facts, replicability and reliability require a more robust defense (Kreuzer, 2019; Lustick, 1996; Trachtenberg, 2006). Several strategies are available. Scholars can exclusively utilize information on which historians of different schools agree (Lustick, 1996). Similarly, when they utilize a diverse set of sources with different biases (for example, archives of rival states or organizations), they can rely on the facts these sources agree upon (Kreuzer, 2019; Lustick, 1996). Furthermore, they can do all of the above while being transparent about which sources were not consulted and why (Lustick, 1996).

Nor are experimental and quasi-experimental methods immune to their own forms of bias and interpretive challenges. Key sources of bias include non-compliance among participants, which can undermine treatment integrity; attrition, where subjects drop out over time and introduce selection effects; and spillover, where effects leak across spatial or social boundaries, contaminating control groups. Beyond these operational issues, theoretical concerns persist, such as the durability of observed effects—whether they persist beyond the study period—and the validity of the treatment as a proxy for the underlying concept, which can obscure the broader meaning of results (Deaton, 2010; Hansen and Tummers, 2020). ¹⁵

Thus, while it is true that scholars should be attentive to and transparent about potential sources of bias and subjectivity in their research, these problems are shared across methodological traditions.

Falsifiability

The third critique relates to the epistemological underpinnings of comparative theory testing. Our approach takes falsification as the bedrock of scientific progress. Therefore, it is subject to the critiques that have been mounted against it.

The comparative theory-testing approach does not provide criteria for conclusive falsification, that is, it leaves unclear when a theory should be considered falsified and discarded. Some of the key critiques against this understanding of science question whether falsification is even possible. The more extreme version, based on Kuhn’s (1962) notion of incommensurability, argues that scientists who adhere to different paradigms differ so much in their methodology, perceptions, and conceptual vocabulary that their theories are often not comparable at all. The softer version, based on Lakatos’ notions of ‘scientific research programs,’ suggests that falsification is often a much slower and messier process than Popper acknowledged. According to Lakatos (1970, 1978), scientists typically resist abandoning theories until their core assumptions and expectations have been consistently and repeatedly refuted.

These philosophical debates cannot be resolved here. Nonetheless, much of political science rests on shared conceptual foundations, epistemological and ontological premises, and core research questions (for evidence, see Kuehn and Rohlfing, 2024). Our methodological argument about the role of description in theory testing is therefore orthogonal to Kuhn’s concerns and consistent with Lakatos’ perspective. So long as a sufficient number of political scientists agree that the discipline’s central task is to answer why questions about significant political phenomena using empirical evidence, our argument remains relevant.

To be sure, theories are rarely discarded outright in the social sciences, though there are examples of research programs that have ‘degenerated’ in the face of consistent empirical rebuttal, to use Lakatos’ term (e.g., primordialism in nationalism studies). Comparing theories is best conceived as a ‘quest’ to uncover the causes of important socio-political phenomena. While we may not often end up with one theory that survives all empirical tests, there are ways to identify which theories are ‘progressing’ or ‘degenerating’ in Lakatos’ terms. One path would be a state-of-the-field type of article that evaluates theories based on accumulated empirical evidence—without additional empirical testing. Such works would outline competing theories and their testable implications on a given topic, systematically assemble findings from existing works that help distinguish between rival theories, and offer a conclusion about which are progressing and which are degenerating. They would go beyond a synthetic review by providing an inclusive map of rival theories alongside the evidence that speaks to them.

Conclusion

The apparent narrowing of methodological choices rewarded in the profession has profound implications for the questions we ask and our ability to compare theories. The notion that methodological value derives from proximity to experimental conditions is not new (see, for example, Lijphart, 1971). KKV similarly portray the central task of social science as overcoming the fundamental problem of causal inference—determining the direction and size of the relationship between a limited number of variables (King et al., 1994: 79). Therefore, the merit of a method lies in how closely it replicates the logic of experimentation. Consequently, randomized controlled trials occupy the top of this hierarchy, followed by survey experiments and other experimental or quasi-experimental approaches (Barrett and Carter, 2010; Gerber and Green, 2012; Green, 2005; Humphreys and Weinstein, 2009). The so-called ‘causal identification revolution’ in political science can thus be seen as the culmination of a trajectory initiated roughly half a century ago.

We, by contrast, have suggested that significant theoretical progress can also be achieved through an eclectic use of methods that assess the non-equivalent observational implications of competing theories and examine which withstand efforts at falsification. This approach calls for methodological pluralism. No method should be regarded as intrinsically superior. Rather, the choice of method should stem from the observable implications of the theories under consideration and from their points of disagreement, not from methodological preferences independent of these arguments.

Our framework is most directly productive when competing theories are internally consistent, falsifiable, and observationally non-equivalent. But our framework’s logic remains useful even where theory is less developed. Asking what would distinguish one’s argument from alternatives, and what evidence would speak to that, disciplines the research process at the theory-generating stage as well.

Turning to causal identification, our framework does not treat this as an exception or a competing standard. When the primary theoretical disagreement between competing arguments concerns the existence, direction, or magnitude of a causal effect, then design-based inference can be the appropriate method for testing that specific implication. Design-based methods earn their place not because they occupy the top of an external hierarchy, but because they are the right instrument for a particular type of theoretically derived implication.

The broader understanding of our research endeavor, as comparing theories rather than solely establishing causal relations between variables, has significant implications for how our discipline is organized and the type of work that is considered valuable. It is relatively uncontroversial to suggest that the top-ranked discipline-wide political science journals are skewed toward works that put a premium on causal inference, with survey experiments accounting for much of this trend (Torreblanca et al., 2026). Scholars have raised the asymmetric distribution of methods in leading journals and argued that lack of methodological diversity narrows the scope and novelty of research questions political scientists ask (Gerring, 2017; Grossman et al. 2026; Kreuzer, 2019). ¹⁶

We go further by arguing that descriptive evidence is as essential for theory testing as it is for concept formation and theory-building. Descriptive information obtained from historical sources or qualitative fieldwork on when, where, or how events transpire has the potential to empirically distinguish between theories just as effectively as experimental work. Method choice follows from what theories require, not from a priori methodological rankings. Theory comes before method—not because pluralism is equitable, but because it is what rigorous inquiry demands.