Comment on: “Case-to-Factor Ratios and Model Specification in Qualitative Comparative Analysis (QCA)”

Abstract

The article by Thiem and Mkrtchyan appears in a context of rising interest about the number of cases needed to perform a QCA analysis. They question the benchmark tables proposed by Marx and Dușa (2011) and conclude the ratios from those tables are “misplaced.” We believe this conclusion deviates from the scientific method and focus on two essential aspects: (1) their approach to the case to factor ratios and (2) omitted variable bias. We conclude our tables hold until a better alternative is proposed and encourage QCA researchers to continue using them.

Introduction

The analysis by Thiem and Mkrtchyan (2022; from here on, TM) is a useful discussion in the context of a rising interest about the number of cases that are required in QCA. We thank TM for seriously engaging with our work and critically assessing it.

TM question the benchmark tables proposed by Marx and Dușa (2011, from here on MD), and claim: “there is nothing in the algorithmic machinery of QCA that puts an upper limit on the number of exogenous factors given a certain number of cases.”

We believe this statement should be taken with a grain of salt. Space does not permit us to reiterate the starting points for and aims of developing the benchmark tables we propose. Interested readers are referred to the original articles (Marx 2010; Marx and Dușa 2011). We focus on two aspects of the TM article: (1) their approach to the case-to-factor ratios and (2) omitted variable bias.

Case-to-Factor Ratios

While they acknowledge that case-to-factor ratios do incur some costs, their article does not present any replacement solution over MD, nor do they improve on MD. Instead, they claim unfavorable case-to-factor ratios are misplaced and present an example to uncover causal structures in extreme situations (two cases for 30 causal conditions).

That statement shifts away from the scientific method. Every science has at least a rule of thumb between the number of variables and the (required) number of cases, otherwise running into the problem of overdetermination: The outcome can be explained by too many sufficient causes.

“Anything goes” in QCA if TM are right, and that is problematic. But not everything goes, nor should it. With enough random sampling, it is obvious that any dataset can be replicated through artificial data. That is perfectly possible in statistics, and the more so in crisp sets QCA with only two values to sample from.

The problem of overdetermination is serious; with strong statistical recommendations, there should be enough cases for a given number of independent variables. QCA makes no exception, despite TMs’ conclusions.

Their Table 1, showing how QCA can draw causal inferences with 30 causal conditions and only two cases, is the cornerpiece of their article. No scientific domain would accept such a situation, so where do they deviate from the accepted scientific method?

First, all causal conditions except one are constant. The same kind of table could be constructed for a regression, but no one would include 29 constant variables in a model. That table can be reduced to just two columns and two cases, a ratio in perfect accordance with the MD benchmarks. The 29 other constant ones could be considered scoping conditions on which the two cases were selected for further investigation.

A second and more serious problem arises from the current debate over the QCA solution models (Baumgartner and Thiem 2020; Dușa 2022; Ragin 2008; Thiem 2022). If the parsimonious solution is the (only) correct one, this can only happen by employing all possible remainders as if they were contributing to the occurrence of the outcome.

For 30 causal conditions where only two cases are observed, there are more than a billion remainders to be employed, essentially drawing conclusions based on practically 100% hypothetical evidence. That is anything but science.

QCA does not recommend an uncritical use of all possible remainders, but a conscious process of eliminating anything that can be proven as not conducing to the occurrence of the outcome.

In promoting Table 1 as “evidence” that there should be no worries about the case-to-factor ratios, TM closely follow the principle of material implication exposed by Dușa (2022) as a problematic alternative to the classical, Boolean minimization method.

Omitted Variable Bias

TM aim to show the application of MD is prone to “omitted variable bias” (i.e., the analysis excludes relevant explanatory factors) and try to make the point via a replication of an ASPR article by adding two of these “omitted” variables in their analysis.

TM claim: “[w]ith the two omitted exogenous factors PD and PP added to the analysis, QCA returns 22 rival models that fit the data equally well.”

It is not clear how they arrive at 22 rival models and how these models compare to the one(s) of Muriaas et al. (2022). If one adds two explanatory conditions to the existing six of the original analysis one gets one model with eight conditions which is further analyzed, not 22. Nor is it clear what they mean with “rival models that fit the data equally well.” How do they assess that rival models fit the data equally well? Does this mean each model has the same consistency and/or coverage? We would be surprised.

More significantly, Muriaas et al. did carefully consider the two explanatory conditions TM discuss and include in their replication, but they consciously decided not to make use of them:

Two major conditions were initially coded and considered for QCA: GEF target (candidate or party) and GEF as a party penalty rather than a payment. When the GEF target was included in a six condition QCA model, the limit for a population of 34 or less, extreme model ambiguity and failure to generate parsimonious explanations without contradicting simplifying assumptions resulted. In addition, the GEF target is more or less covered by the state-driven condition given that most of the candidate targeted instruments were nonstate driven and the party targets were state-driven. Whether GEF penalizes or provides payment as a condition is omitted given that less than one third of the cases have the presence of a party penalty, thus, not enough variation. (Muriaas et al. 2022:506; note target corresponds to PD above and penalty to PP above)

The authors do not include the two explanatory conditions after carefully exploring their possible contribution. Not because they were never considered, as TM seem to suggest in citing the authors. This is not an example of omitted variable bias but an illustration of a process on how to make sense of the data based on existing knowledge, concepts, and data. It is precisely this type of dialog between cases, concepts (conditions), and explanations that we want to foster through our benchmark tables.

TM do not elaborate how their results challenge the findings of Muriaas et al. They just note that how “this result is to be interpreted substantively must be left to the experts in the field of political representation.” Well, the experts (authors) did interpret the relevance of the two variables, and expert reviewers do not seem to support the conclusion by TM on the relevance of these two conditions.

More generally, the omitted variable bias is a passe-partout argument that can be used all the time for any research. One can never exclude omitted variable bias. Any phenomenon of interest can have between a few and more than 30 explanatory conditions/variables. Even if previous research was fully used to employ variables, there might still be relevant variables left out, which might emerge out of in-depth case-studies.

Ignoring our proposed benchmark tables would excuse researchers from going back to the cases for further investigation and miss spotting relevant variables. The application of our benchmark tables should inform researchers they are indeed missing a relevant explanation and force them resolve this “contradiction.” This is further elaborated in our article.

Conclusion

We do not claim the approach presented in MD is perfect. Any method or approach is subject to improvement, and the one from MD makes no exception. But it is a very long way from there to claiming there should be no case-to-factor ratios whatsoever.

TM do recognize that low case-to-factor ratios are problematic, yet they provide no solution to the problem. Quite the contrary, they promote the idea that worries about these ratios are misplaced, thus divorcing from any accepted scientific method.

We acknowledge that the application of the benchmark tables can result in equally relevant models to be analyzed in QCA, but this is not related to the benchmark tables as such but with model selection in a world of many possible explanatory factors. That is another issue already flagged in other publications.

We fully agree with TM’s last sentence that to advance knowledge, researchers are better off with “relying on the current state of knowledge in their respective field.” We would also add that researchers should rely on their understanding of the cases they analyze. That is precisely what we aim to achieve through our benchmark tables. Their application in the APSR paper, which TM critically discuss, is an illustration of benchmark tables’ added value and how their employment can contribute to increasing the substantive knowledge in a field. But it does not support the claims by TM on omitted variable bias.

We do not think TM provide sufficient grounds to dismiss the benchmark tables, and their alternative approach of “anything goes” does not really offer an alternative. We encourage future QCA researchers to use them until better alternatives are proposed.