How do researchers choose their goals of inference? A survey experiment on the effects of the state of research and method preferences on the choice between research goals

Abstract

In empirical research, scholars can choose between an exploratory causes-of-effects analysis, a confirmatory effects-of-causes approach, or a mechanism-of-effects analysis that can be either exploratory or confirmatory. Understanding the choice between the approaches is important for two reasons. First, the added value of each approach depends on how much is known about the phenomenon of interest at the time of the analysis. Second, because of the specializations of methods, there are benefits to a division of labor between researchers who have expertise in the application of a given method. In this preregistered study, we test two hypotheses that follow from these arguments. We theorize that exploratory research is chosen when little is known about a phenomenon and a confirmatory approach is taken when more knowledge is available. A complementary hypothesis is that quantitative researchers opt for confirmatory designs and qualitative researchers for exploration because of their academic socialization. We test the hypotheses with a survey experiment of more than 900 political scientists from the United States and Europe. The results indicate that the state of knowledge has a significant and sizeable effect on the choice of the approach. In contrast, the evidence about the effect of methods expertise is more ambivalent.

Introduction

In empirical research, scholars can pursue a number of goals of inference in the analysis of a phenomenon. First, one can take an exploratory causes-of-effects approach, asking for the causes of an outcome (“Why are some populist parties successful and others not?”). Second, in a deductive, hypothesis-testing perspective, one can ask about the effects of one or multiple causes on an outcome (“What is the effect of an economic crisis on the electoral success of populist parties?”). Third, one can take a “mechanism-of-effects” and make inferences about one or multiple mechanisms that connect a cause to the outcome.

There are two main reasons that it is important to know the conditions under which researchers choose one perspective over the others. First, the value of an approach and the efficiency of spending research resources (attention, money, time) depend on the state of knowledge about an outcome. When little is known about a new phenomenon, an exploratory causes-of-effects promises the greatest benefits. The more knowledge that has been accumulated in exploratory research, the larger the value of effects-of-causes and mechanisms-of-effects research that tests the exploratorily derived hypotheses on causal effects and causal mechanisms. If researchers chose a goal of inference that creates a mismatch with the state of knowledge, then the allocation of resources to research projects would not be optimal.

Second, with an increasing specialization of qualitative and quantitative research methods, there are increasing benefits to a division of labor between researchers who have expertise in quantitative, qualitative, and mixed-methods approaches. In the context of a broader argument about method preferences and research cultures, ¹ it has been argued that qualitative researchers primarily take causes-of-effects and mechanisms-of-effects perspectives and that quantitative researchers follow a confirmatory effects-of-causes approach (Goertz and Mahoney 2012, chapter 2). The explanation for their preferences is that researchers are socialized into the use of different research methods and choose the goal of inference that fits their preferred method. Qualitative and quantitative researchers realize research designs, regardless of the current state of knowledge, which would probably be a waste of research resources.

In this paper, we use the two lines of argument to formulate and test hypotheses about the choice between the three goals of inferences. This contributes to a better understanding of whether the state of knowledge or the preferred method influence the choice of a goal of inference. Our study can further indicate how efficiently researchers allocate resources to empirical projects and whether and what kind of division of labor is present in empirical political research.

We conducted a preregistered survey experiment with more than 900 political scientists from Europe and the United States. Participants were randomly assigned to two versions of a vignette describing an abstract research topic (see Figure A1 in the appendix). The vignettes varied with regard to the maturity of the state-of-the-art for the research topic. After reading the vignette, participants were asked to choose between three research designs corresponding to the three different goals of inference described above. In addition to the experimental manipulation, we also collected data on the method preferences of respondents, which allow us to explore how qualitative, quantitative, and mixed-method researchers chose their goals of inferences.

The results indicate that the experimentally manipulated state of research substantially influenced how scholars specify their goals of inference. Being confronted with an understudied research topic, participants largely chose an exploratory causes-of-effects approach. In contrast, when facing a mature state of research, scholars were more likely to opt for a confirmatory effects-of-causes or mechanism-oriented approach. The respective effect sizes are substantial and statistically significant. The results for the effect of method preferences on goals of inference are mixed. While we find evidence that quantitative and mixed-method are more likely to prefer confirmatory hypothesis-testing research relative to qualitative researchers, our results are mostly inconclusive with regard to the effect of method preferences on choosing exploratory or mechanism-oriented goals of inferences. We conclude that these findings generally indicate the efficiency of spending research resources within the field of political science. However, academic socializations with regard to method preferences seem to impede the strict division of labor.

Goals of inference and hypotheses on the choice among them

We define a “goal of inference” along the classic distinction between a causes-of-effects perspective (or approach) on a phenomenon and an effects-of-causes perspective (e.g., Morgan and Winship 2014, 53–56). Following the established usage, we use “effect” synonymously with “outcome” or simply Y. A study focusing on the causes of an effect asks for the causes of an outcome. When we take the electoral success of populist parties as the phenomenon of interest, a causes-of-effects perspective asks: What are the causes of the electoral success of populist parties? This question implies that causes-of-effects research is exploratory because the goal is to find potential causes for an outcome. An effects-of-causes approach reverses that perspective and asks for the effects of potential causes on the outcome. From an effects-of-causes viewpoint, one would choose a specific variable such as “economic crisis” and ask a confirmatory, hypothesis-testing question: What is the effect of an economic crisis on the electoral success of populist parties?

The distinction between causes-of-effects and effects-of-causes is significant because it distinguishes two equally important and complementary goals of inference. However, it is incomplete because it does not capture the rise of mechanism-centered research since the mid-1990s (Mahoney 2010). We take this development into account and add a mechanism-of-effects perspective to the original two-fold distinction. We define “mechanism-of-effects” as a goal of inference that aims to study the mechanisms that connect a cause to the outcome.

Based on the three-fold distinction of goals of inference, we test two hypotheses about the choice among them. The first hypothesis takes the choice of a goal of inference as dependent on the state of knowledge about the outcome and the marginal added value that the pursuit of a goal of inference promises. When an empirical phenomenon is new and has not yet been studied (a global pandemic caused by a new virus; the transition of economies to sustainable energy production etc.), exploratory causes-of-effects research is the most valuable, allowing one to produce initial insights into possible causes and mechanisms producing the outcome.

The more knowledge that has been collected in causes-of-effects studies, the larger the marginal added value of confirmatory effects-of-causes and mechanisms-of-effects research becomes. One gets a better understanding about possible causes and mechanisms of an outcome on which resources could be focused in a hypothesis-testing study. This implies that mechanism-oriented research can be exploratory or confirmatory, but that we see bigger added value when it is confirmatory and builds on evidence for a causal effect for a relationship of interest.

The marginal added value of a goal of inference is linked to the efficient use of research resources. Resources are spent more efficiently on exploratory research when little is known about an outcome and should be gradually shifted to hypothesis-testing research the more knowledge is gained. The same argument holds for the allocation of resources between tests of effects and mechanisms. A complete understanding of an empirical phenomenon requires knowledge about an effect and the underlying mechanism (Dessler 1991). The more insights that have been gained in tests of causal effects, the more resources should be spent on the analysis of mechanisms, and vice versa. We summarize the arguments about the state of knowledge in hypotheses 1 and 2.

H1: Given an understudied research topic, researchers are more likely to prefer an exploratory causes-of-effects approach.

H2: Given a research topic that was already extensively studied in previous research, researchers are more likely to prefer a confirmatory approach of studying the effects-of-causes or a research strategy based on the mechanisms linking a specific cause and effect.

A second explanation for the choice of an approach focuses on the type of method that a researcher usually applies in empirical research. It has been stipulated that qualitative researchers tend to ask causes-of-effects questions and that quantitative researchers prefer effects-of-causes research (Gerring 2012, 333–335; Mahoney 2008). A hypothesis-testing mechanisms-of-effects analysis is primarily chosen by qualitative researchers who use process tracing for the analysis of mechanisms (Goertz and Mahoney 2012, chapter 3). Mixed-method researchers have not been included in this debate and, accordingly, our results for “mixed-methods researcher” should be considered as exploratory evidence. The advantage of pursuing a goal of inference that best fits a researcher’s expertise is that one can benefit from specialization. The standards for the implementation of quantitative and qualitative methods have been consistently rising. This creates an incentive for specializing in one method and the corresponding research approach with the best fit. It has been argued that this specialization socializes scholars into research cultures that are coherently followed in empirical research projects (Goertz and Mahoney 2012, chapter 1). If this argument was correct, empirical researchers would not take the state of knowledge about a phenomenon into account, but instead consistently pursue the goal of inference that fits their own methods specialization and research culture. We summarize this reasoning in hypothesis 3.

H3: Researchers specify their goals of inference based on their predetermined research cultures.

Research design and analysis

Design and sampling scheme

We test these hypotheses using a web-survey experiment with political science researchers in the US and Europe. Participants were presented with two different versions of a vignette, which described an abstract research topic. The experimental manipulation varied the state of knowledge about the research topic, with participants being randomly assigned to research topic that “has already been extensively studied” or “so far has not been extensively studied.” Randomization was blocked by country of residence of the participants. After reading the vignette, participants answered a single-choice question that asked them to choose between different goals of inference. Option one was to conduct a causes-of-effects study (exploratory design); option two was an effects-of-causes study (confirmatory design) and option three, a study about a causal mechanism (process-oriented design). The outcome question was measured as multi-nominal variable consisting of three categories with one for each option.

For the measurement of the preferred research method, at the end of the survey, we asked respondents about the methods and the number of cases with which they usually work in their research. With these measures, we classified respondents as quantitative, qualitative, or mixed-method researchers. In addition, we collected data on the age, gender, and professional status of respondents. The survey also contained both an attention and a manipulation check.

Our experiment was part of a larger web-survey, which was fielded among political scientists affiliated with departments in the US and Europe between June and December 2018. ² Participants were invited via personalized emails. An Amazon gift coupon worth 10 USD, British Pounds or Euros, respectively, was offered to every participant that completed the survey. We report the details of the sampling scheme in appendix section A.1. Figure 1 describes the experimental design in a flow-chart. The experiment was preregistered with the Open Science Framework. ³

Figure 1.

Flow-chart of experimental design and sampling procedure.

A total of 1047 participants completed the survey. The final sample size used in the empirical analysis was 901. Participants were excluded because of failed attention checks, because of a self-declared non-empirical research focus (formal theory, political theory) or otherwise inappropriate method preference. Finally, respondents were excluded if they had missing data for other covariates. Of the 901 respondents, 466 respondents were working in the US and 435 in Europe at the time of the survey.

Empirical analysis

We estimate the effect of the manipulated state of research and self-declared method preference on the choice of a goal of inference with a multinomial logit model with robust standard errors clustered by country of residence. We include as covariates the age in years, gender, and professional status of respondents and an indicator measuring if respondents are working at universities in Europe or US. The appendix section A.1. reports descriptive statistics for all covariates and randomization checks.

For the experimental manipulation of the state of research, we estimate the change in the probability of pursuing a goal of inference induced by the presence of a research topic that has not been extensively studied as opposed to one that has already been studied to a significant degree. For the method preferences, the marginal effect refers to the change in probability of choosing a goal of inference for a quantitative or mixed-method researchers relative to the reference category of “qualitative researcher.” The theoretical arguments that inform hypotheses 3 only distinguish between qualitative and quantitative researchers because they are at the center of the underlying debate. In our analysis, we add “mixed-methods” as a third type. The qualitative-quantitative distinction seemed too coarse-grained for measuring the preferred method of empirical researchers who should have the opportunity to declare themselves as “crossing boundaries” (Goertz and Mahoney 2012, chapter 1).

With regard to the size of treatment effects, we consider a difference of at least 10% in proportions as substantially important. Based on statistical power simulations conducted before the data collection, we estimate that a sample size of at least 800 responses is needed to detect such a substantively important treatment effect. ⁴

Results

We report the results of the multinomial logit regression model in Figure 2, which summarizes the estimated main effects of the experimental treatment condition and research method preferences of respondents. ⁵ The three categories, “exploratory,” “confirmatory,” and “process” in the three panels represent the goals of inference of causes-of-effects, effects-of-causes, and mechanisms-of-effects.

Figure 2.

Marginal effects from multinomial logit models.

For the state of knowledge, Figure 2 shows that when respondents are confronted with an understudied research topic as opposed to one that has been extensively studied, it is about 54% more likely that an exploratory design is chosen. At the same time, the treatment condition of an understudied research topic decreases the probability that a respondent chooses a confirmatory or process-oriented research design by 22% and 32%, respectively. All effects achieve statistical significance and meet the criterion of substantive importance, thus supporting hypotheses 1 and 2.

The effect of methods practices on design choices is more ambiguous. Quantitative and mixed-method researchers are less likely to opt for an exploratory design relative to qualitative researchers, but the effect sizes are at the margin of substantive importance (about 7 and 6%, respectively) and not statistically significant. Quantitative and mixed-method researchers are more likely than qualitative researchers to choose a confirmatory design. Effect sizes are substantial, with an increase of 26% for quantitative researchers and 15% for mixed-method researchers. Regarding the effect of methods preference on the choice of a mechanisms-of-effects approach, quantitative and mixed-method researchers are less likely than qualitative researchers to choose such a design. This effect is significant and substantial for quantitative researchers, who are about 19% less likely to choose a process-oriented research design than qualitative researchers. The effect is much weaker and not statistically significant for mixed-method researchers. We do not find relevant effects for the other covariates in the model. ⁶

We further explored the treatment-by-covariate interaction effects (not preregistered); regarding state of research and the method preference, we find that qualitative, quantitative, and mixed-methods researchers’ responses are substantially similar to the experimental treatment of the manipulated state of research as described above. In the reverse perspective, we do find that the effect of methods preferences is moderated by the experimental treatment condition. Given the treatment condition of an understudied research topic, researcher largely opt for an exploratory research design, regardless of their method preferences. However, when confronted with an extensively studied research topic, quantitative researchers favor a confirmatory research design but, in contrast, qualitative researchers are most likely to choose a process-oriented research design. Detailed results for the interaction effects are described in the appendix section A.3.

Discussion

Hypothesis 1 is supported by the finding that the choice of an exploratory causes-of-effects perspective is more likely for understudied topics. Hypothesis 2 receives support because an effects-of-causes and a mechanisms-of-effects analysis are more likely for phenomena that have been studied extensively. The empirical evidence is mixed for hypothesis 3. Our results indicate that, independent of the experimental manipulation, methods preferences are relevant to determining the goal of inference. In general, quantitative researchers are more likely to favor an effects-of-causes design relative to qualitative scholars and qualitative scholars are more likely to opt for a process-oriented perspective relative to quantitative scholars. However, beyond that, most main effects of method preferences on goals of inference are not statistically significant and/or not substantially important.

Hypotheses 1 and 2 state that one chooses a goal inference based on the marginal added value to the state of knowledge and to emphasize the efficient use of resources. Hypothesis 3 explains the choice of the goal with the socialization and specialization into quantitative or qualitative methods. The two explanations are not mutually exclusive and it is plausible that a researcher pursues a goal of inference that fits their own expertise and the state of knowledge. This would be ideal because the best marginal added value would be produced by researchers who are most qualified to realize a given goal of inference. Our analysis of moderation effects indicates that, to some degree, researchers’ choices reflect this ideal. We find that while scholars from all research cultures largely prefer to conduct an exploratory study, given the condition of an understudied research topic, substantial differences exist in the preferences of qualitative and quantitative researchers given a saturated state of knowledge about a topic. Under these conditions, scholars opt for goals of inference that match their respective method specializations.

The estimates for the category of self-declared mixed-methods researchers yield exploratory insights that are also mixed. We do find that, relative to qualitative researchers, mixed-method scholars are more likely to conduct confirmatory effects-of-causes research, but with regard to the other goals of inference, the effects are small and not statistically significant.

We see the following points as potential limitations of the analysis. First, the responses to the manipulation check were inconclusive. The manipulation check asked respondents if they agree with the statement that “Empirical studies about the causes-of effects (i.e., What causes Y?) are only useful to analyze new phenomena or research topics that have received little attention by previous research (Yes/No).” We expected that the survey experiment would make respondents aware of the relationship between the state of knowledge and goals of inference. However, out of 901 respondents, only 60 did answer affirmatively. Based on the large effect size of the experimental treatment in our main analysis, however, we believe that the state-of-the-art manipulation was perceived as intended and does not undermine the interpretability of the estimates. ⁷

Second, we randomly assigned the state of knowledge to participants to eliminate reverse causation in the test of hypotheses 1 and 2. However, we could not randomize the method preferences of researchers. Accordingly, the results from the respective analysis should be considered to be observational evidence. Nevertheless, the randomization of the state of knowledge ensures that researchers cannot select a goal of inference because they believe, possibly driven by their methods preference (see above), that a certain phenomenon has (or has not) been understudied, therefore addressing the main selection problem of an observational analysis of effects of methods preferences.

Third, with regard to external validity, we have limited knowledge about the representativeness of the sample. For the US, we compared our sample with demographics of the members of the American Political Science Association (APSA). The comparison indicated that our sample broadly matched the demographic characteristics of APSA members in terms of gender but substantial differences existed with regard to age, with APSA members being older on average than the respondents in our survey. However, for our sample of European scholars, we did not have such a reference group available. Moreover, we had very different response rates among European countries, both in absolute and relative terms. Response rates ranged from little more than 6% in Finland to 38.6% in Italy. The European sample is mostly biased towards scholars based in Germany and the UK, which account for more than half of the sample. ⁸

Conclusion

The choice between three complementary goals of inference—causes-of-effects, effects-of-causes, and mechanisms-of-effects—is important to understand because it influences how efficiently researchers spend their resources and use their expertise to contribute to the accumulation of knowledge. We analyzed two explanations of the ways in which researchers allocate their resources towards different goals of inference, namely, the existing state of research and method preferences. While the former indicates the flexibility of scholars in choosing goals of inference based on the feasibility of a study design for a given state of research, the latter highlights the path dependency of being socialized into a research culture that determines preferences for goals of inference. Our findings suggest that scholars are indeed capable of adjusting their goals of inference with regard to the state of research for a phenomenon. We find that choices among goals of inference are largely determined by the maturity of the state of the knowledge for a given topic. When examining a research topic that has not been extensively studied, most scholars chose an exploratory design, regardless of their method preferences. However, when facing a saturated state of knowledge, research cultures, that is, socialized preferences for specific research methods, have more influence on the choice of the goal of inference.

We see two main opportunities for follow-up research. First, additional replication studies of our research with variations in geographic scope and extensions of the participant pool would advance the external validity and robustness of main findings. Second, further studies could put more emphasis on mixed-method research to understand better how researchers, who are versatile in the use of two methods that are good for different purposes, choose between goals of inference.

Supplemental Material

Supplemental Material - How do researchers choose their goals of inference? A survey experiment on the effects of the state of research and method preferences on the choice between research goals

Supplemental Material for How do researchers choose their goals of inference? A survey experiment on the effects of the state of research and method preferences on the choice between research goals by Felix Bethke and Ingo Rohlfing in Research & Politics

Supplemental Material

Supplemental Material - How do researchers choose their goals of inference? A survey experiment on the effects of the state of research and method preferences on the choice between research goals

Supplemental Material for How do researchers choose their goals of inference? A survey experiment on the effects of the state of research and method preferences on the choice between research goals by Felix Bethke and Ingo Rohlfing in Research & Politics