Sage Journals: Discover world-class research

Abstract

Control variables are included in regression analyses to estimate the causal effect of a treatment on an outcome. In this article, we argue that the estimated effect sizes of controls are unlikely to have a causal interpretation themselves, though. This is because even valid controls are possibly endogenous and represent a combination of several different causal mechanisms operating jointly on the outcome, which is hard to interpret theoretically. Therefore, we recommend refraining from interpreting the marginal effects of controls and focusing on the main variables of interest, for which a plausible identification argument can be established. To prevent erroneous managerial or policy implications, coefficients of control variables should be clearly marked as not having a causal interpretation or omitted from regression tables altogether. Moreover, we advise against using control variable estimates for subsequent theory building and meta-analyses.

Keywords

multivariate regression research methodology causal inference control variables reporting

Multivariate regression is an important tool for empirical research in organization studies, management, and economics. Beyond settings in which regression analysis is used to statistically predict a left-hand side variable given a set of explanatory variables, the main purpose of these methods is to control for confounding influence factors between a treatment and an outcome in order to obtain consistent causal effect estimates.¹ However, in practice, scholars often overstate the role of control variables in regressions. In this article, we argue that, while essential for the identification of causal effects, control variables do not necessarily have a causal interpretation themselves. This is because even valid controls are often correlated with other unobserved factors, which render their marginal effects uninterpretable from a causal inference perspective (Westreich & Greenland, 2013; Keele et al., 2020). Consequently, researchers need to be careful with attaching too much meaning to control variables and should consider to ignore them when interpreting the results of their analyses.

Drawing substantive conclusions from control variable estimates is common, however. Authors frequently make use of formulations such as “control variables have expected signs” or “it is worth noting the coefficients² of our control variables”. Below, we present the results of a literature review of papers published in Organization Science and Strategic Management Journal between 2015 and 2020, in which we found that 47% of manuscripts using regression methods also explicitly discussed the estimated effect sizes of controls. This is in line with Carlson & Wu (2012), who identified 48% of papers published in the Academy of Management Journal, Journal of Applied Psychology and Strategic Management Journal in 2007 that interpreted and discussed the effects of controls. Moreover, in our own experience as authors of quantitative research papers, we frequently encountered instances in which reviewers asked us to provide an interpretation of control variable coefficients. The justification that was often given was that, although they were not the main focus of the analysis, controls could still provide valuable information for other researchers in the field who are investigating related research questions.

The methodological literature in organizational research usually highlights that control variables should carry the same importance in an empirical analysis as the main independent variables of interest (Becker, 2005; Spector & Brannick, 2011; Carlson & Wu, 2012; Atinc et al., 2012). To increase rigor and improve the transparency of published research articles, Becker (2005) recommends to report all regression coefficients of control variables as well as their significance levels. Similarly, Spector & Brannick (2011) advocate that controls should be given equal status to the main treatment variable in the analysis. Atinc et al. (2012) consider it to be best practice to provide an ex-ante prediction of the sign of the relationship between the controls and outcome variable based on theory, which should subsequently be checked against the empirical evidence. In a recent paper, Becker et al. (2015) provide a more cautionary recommendation regarding generalizing from control variable estimates if it involves out-of-sample extrapolation, but otherwise consider it to be appropriate. Overall, the general consensus in the organizational literature thus seems to be that interpreting control variable estimates is safe, as it adds to the body of cumulative evidence regarding a particular effect size.

This article builds on the graphical framework to causality (Pearl, 2000; Durand & Vaara, 2009; Hünermund & Bareinboim, 2023). Causal diagrams have already been established as a powerful tool for determining which control variables are relevant to a given regression model (Cinelli et al., 2022; Hünermund et al., 2022). In addition, the approach offers a distinct perspective on the proper interpretation and communication of control variable results, which differs from prior methodological practices found in the organizational and management literature. In the following, we will explicate the view that control variables, while certainly an important ingredient in many causal research designs, do not have the same status as the main variables of interest in an empirical analysis. In particular, we will argue that in many situations valid controls can nonetheless be endogenous. Therefore, interpreting their estimated effect sizes in light of prior theory could lead to potentially misleading conclusions. A valid causal interpretation of control variables rests on strong assumptions and usually requires accounting for all influence factors of the outcome variable under study. Since this is unlikely to be fulfilled in many research contexts, we recommend authors to exercise caution when interpreting control variables and consider omitting estimated coefficients of control variables from regression tables, or relegating them to an appendix. Finally, we discuss what our recommendations imply for the practice of meta-analysis, which has recently gained traction in many fields including organizational research (Aguinis et al., 2010).

Do Researchers Attach Substantive Meaning to Control Variables?

To assess the degree to which researchers interpret control variable estimates in their studies, we conducted a review of all articles published in Organization Science and Strategic Management Journal between January 2015 and December 2020. We chose these two journals because of their high prestige in the management and organization field as well as their reputation for high-quality empirical research. Our sample includes all quantitative articles that employed parametric regression models such as ordinary least squares (OLS), logit, probit, Poisson, etc. This choice was made because effect sizes of control variables can usually not be summarized by a single coefficient (or marginal effect) in non and semiparametric models. The use of such methods is rare in our sample anyway though.

We manually categorized papers according to whether they interpret or draw substantive insights from the coefficients or marginal effect estimates of control variables. Examples of such an interpretation range from “the control variable CEO tenure is positively related to performance” to “the effect sizes of control variables are in line with previous studies.” The latter interpretation is thereby of relevance because authors of future research papers might be tempted to develop a theory based on this seemingly accumulating empirical evidence. The result of our review shows that interpreting control variables was common practice in the analyzed journals during our period of observation. For the Strategic Management Journal, we identified a total of 497 quantitative research articles, of which 233 (47%) proceeded to interpret the effects of control variables. For Organization Science, out of a total of 275 quantitative articles, 131 (47%) provided an interpretation of control variable estimates. Detailed results of the literature review and example quotes demonstrating the different ways in which authors interpret control variable estimates in practice are reported in the supplemental material to this article.

The Causal Interpretation of Control Variables

The relationship between the main explanatory variables and controls in a regression can be complex, therefore it is beneficial to explicitly depict them in a causal diagram (Pearl, 2000). Durand & Vaara (2009) were the first to introduce causal graphs as a tool for management researchers. Cinelli et al. (2022) provide a useful overview of the different functions of control variables in regression analyses, explicitly leveraging the graphical framework. Here, we focus on the causal interpretation and reporting of control variable estimates, which has been a topic of ongoing debate in the organizational literature.

Figure 1a presents a simple model with a treatment variable $X$ and an outcome variable $Y$ . Both variables are connected by an arrow, denoting the direction of causal influence between them. In addition, there are two confounding variables, $Z_{1}$ and $Z_{2}$ , that are affecting the treatment and the outcome. $Z_{1}$ and $Z_{2}$ are correlated, as a result of a common influence factor they share, which is denoted by the dashed bidirected arc in the graph. The fact that $Z_{1}$ and $Z_{2}$ are correlated creates what is known as a backdoor path between the treatment and the outcome (Pearl, 2000). $X$ and $Y$ are not only connected by the direct causal path $X \to Y$ , but also by a second path, $X \leftarrow Z_{1} \leftarrow - - \to Z_{2} \to Y$ , which creates a spurious, noncausal correlation between them.

Figure 1.

(a–d) Examples of causal diagrams with valid control variable $Z_{1}$ .

Backdoor paths are defined as any sequence of arrows connecting the treatment and outcome variable (irrespective of their orientation) that remains if arrows emitted by the treatment are deleted from the graph (Pearl, 2000). Because of the latter requirement, they are easy to spot in the causal diagram. Since all the arrows emitted by $X$ are deleted, backdoor paths have to point into $X$ instead; that is, they enter “through the backdoor,” which is where the name comes from.

Control variables in a multivariate regression model are invoked to block such backdoor paths and obtain a consistent estimate of the causal effect of $X$ on $Y$ , in which case one speaks of an effect to be causally identified. For this purpose, it is sufficient to control for any variable that lies on the open path.³ Thus, in the example of Figure 1a, the researcher has the choice between either controlling for $Z_{1}$ or $Z_{2}$ , since both would allow to identify the causal effect of interest. The choice between different admissible sets of control variables is thereby of high practical relevance. Researchers often have fairly detailed knowledge about the treatment assignment mechanism $Z_{1} \to X$ ; for example, because there are organizational or administrative rules that determine individual treatment status, which can be exploited for identification purposes (Angrist, 1990; Flammer & Bansal, 2017). At the same time, the set of variables $Z_{2}$ that are direct influence factors of $Y$ will likely be large. Thus, in practical applications, it might be much easier to control the treatment assignment mechanism instead of trying to include all variables that have an effect on the outcome in a regression.

Nevertheless, although controlling for $Z_{1}$ is sufficient to obtain a consistent estimate for $X$ , its marginal effect will itself not correspond to any causal effect of $Z_{1}$ on $Y$ . That is because $Z_{1}$ is correlated with $Z_{2}$ and will thus partially pick up an effect of $Z_{2}$ on $Y$ too (Cinelli & Hazlett, 2020). To illustrate this phenomenon quantitatively, we parameterize the causal graph in Figure 1a in the following way:

\begin{aligned} z_{1} & \leftarrow u + ε_{1}, \\ z_{2} & \leftarrow u + ε_{2}, \\ x & \leftarrow z_{1} + ε_{3}, \\ y & \leftarrow x + z_{2} + ε_{4}, \end{aligned}

(1)

with

n = 10, 000

, and

U

ε_{i}

being standard normal. True effect sizes are set equal to one. Note that

U

is assumed to be unobserved and appears in the functions assigning values to

Z_{1}

and

Z_{2}

. This creates an error correlation between the two variables. We then run a regression of

Y

X

and

Z_{1}

, which gives a consistent coefficient estimate for

X

(

{\hat{β}}_{X}

= 1.017), while the effect of

Z_{1}

(

{\hat{β}}_{Z_{1}}

= 0.499) turns out to be biased. By contrast, if we also include

Z_{2}

in the regression, the coefficient of

Z_{1}

drops to zero (

{\hat{β}}_{Z_{1}} = - 0.019

), which corresponds to its actual causal effect on

Y

in this example (since

Z_{1}

does not appear in line 4 of Equation 1). Detailed simulation results together with associated standard errors (bootstrapped with 1,000 replications) are reported in Table 1.

Table 1.

Ordinary Least Squares (OLS) Regressions With Varying Adjustment Sets.

	Figure 1a			Figure 1b	Figure 1c	Figure 1d
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
Treatment Variable
$X$	1.017	1.004	1.015	0.993	1.001	0.991	1.006	1.003
	(0.015)	(0.006)	(0.010)	(0.012)	(0.008)	(0.057)	(0.007)	(0.010)
Control Variables (not to be interpreted causally)
$Z_{1}$	0.499		−0.019	1.503	1.004	4.565		0.004
	(0.018)		(0.013)	(0.014)	(0.014)	(0.069)		(0.016)
$Z_{2}$		0.993	0.997					0.009
		(0.008)	(0.008)					(0.019)
$Z_{3}$							0.994	0.991
							(0.008)	(0.010)
$Z_{4}$							0.991	0.988
							(0.008)	(0.010)
$Z_{5}$							1.011	1.009
							(0.006)	(0.008)

Note: Simulation results (as discussed in the main text) using different backdoor-admissible adjustment sets for the causal models depicted in Figure 1a to d. Bootstrapped standard errors (with $1, 000$ replications and $n = 10, 000$ ) in parentheses. True effect sizes for all variables equal to one.

Figure 1b and c highlight under which conditions effect estimates of control variables can be interpreted causally. In Figure 1b, there are two backdoor paths: $X \leftarrow Z_{1} \to Y$ and $X \leftarrow Z_{1} \leftarrow - - \to Y$ . Both paths can be intercepted by $Z_{1}$ , which is thus a valid control variable. Data simulated according to the following system:

\begin{aligned} z_{1} & \leftarrow u + ε_{1}, \\ x & \leftarrow z_{1} + ε_{2}, \\ y & \leftarrow x + z_{1} + u + ε_{3}, \end{aligned}

(2)

with coefficients again set equal to one, confirm that the causal effect of

X

can be consistently estimated in a regression of

Y

X

and

Z_{1}

(

{\hat{β}}_{X}

= 0.993). However, once again the coefficient estimate for

Z_{1}

(

{\hat{β}}_{Z_{1}}

= 1.503) is biased (Table 1, column 4). Although

Z_{1}

is a valid control variable in Equation 2, it is nonetheless endogenous (Frölich, 2008). Note that the unobserved variable

U

enters the combined error term

ν = u + ε_{3}

in line 3 of Equation 2. At the same time,

U

is an argument of the function assigning values to

Z_{1}

(line 1 of Equation 2), which lets

Z_{1}

become correlated with the error term.⁴

This is different in Figure 1c. Here the two backdoor paths are $X \leftarrow Z_{1} \to Y$ and $X \leftarrow - - \to Z_{1} \to Y$ . When we simulate data according to:

\begin{aligned} z_{1} & \leftarrow u + ε_{1}, \\ x & \leftarrow z_{1} + u + ε_{2}, \\ y & \leftarrow x + z_{1} + ε_{3}, \end{aligned}

(3)

we now find that a regression of

Y

X

and

Z_{1}

provides a consistent estimate for both

X

as well as for

Z_{1}

(

{\hat{β}}_{X}

= 1.001;

{\hat{β}}_{Z_{1}}

= 1.004; Table 1, column 5). In this situation, the regression coefficient for the control variable

Z_{1}

has a causal interpretation. This is because, unlike in the previous situations, we are able to account for all influence factors of

Y

, apart from the exogenous error term

ε_{3}

. In particular, there is no unobserved variable

U

jointly affecting the outcome

Y

and (at least one of) the regressors

(X, Z_{1})

anymore.

Finally, Figure 1d depicts a more complex setting, with several admissible sets of controls, each sufficient to identify the causal effect of $X$ on $Y$ (Textor & Liśkiewicz, 2011). One possibility in this situation is to simply control for $Z_{1}$ , which is the only direct influence factor of $X$ , and thus blocks all paths entering $X$ through the backdoor. To witness, we simulate data from the system:

\begin{aligned} z_{1} & \leftarrow u_{1} + ε_{1}, \\ z_{2} & \leftarrow z_{1} + u_{1} + u_{2} + ε_{2}, \\ z_{3} & \leftarrow z_{2} + ε_{3}, \\ z_{4} & \leftarrow z_{2} + ε_{4}, \end{aligned} \begin{aligned} z_{5} & \leftarrow z_{2} + u_{2} + ε_{5}, \\ x & \leftarrow z_{1} + ε_{6}, \\ y & \leftarrow x + z_{3} + z_{4} + z_{5} + ε_{7}, \end{aligned}

(4)

and regress

Y

X

and

Z_{1}

, which gives a consistent estimate for the effect of

X

(

{\hat{β}}_{X}

= 0.991; Table 1, column 6). Similarly, controlling for the direct influence factors of

Y

(

Z_{3}

Z_{4}

, and

Z_{5}

) also blocks all backdoor paths and leads to a consistent effect estimate for

X

(

{\hat{β}}_{X}

= 1.006; Table 1, column 7). A third alternative is to control for the entire set of covariates (

Z_{1}

Z_{2}

Z_{3}

Z_{4}

, and

Z_{5}

) which also leads to a consistent estimate for

X

(

{\hat{β}}_{X}

= 1.003; Table 1, column 8), although this would be the most data-intensive identification strategy leading to slightly less precise estimates compared to the previous specification, due to fewer degrees of freedom. This example illustrates that the minimally sufficient set of controls (here

Z_{1}

) for identifying the causal effect of

X

is often much smaller than the total number of confounding variables in a model. At the same time, the estimated marginal effects for the control variables only have a causal interpretation if all the direct influence factors of

Y

(here

Z_{3}

Z_{4}

, and

Z_{5}

) are accounted for in the regression. As we argued above, this is unlikely to be the case, since in many real-world settings the number of causal factors determining

Y

might be prohibitively large.⁵

Examples

In the following section, we present practical applications that illustrate our previous theoretical points. To start as simply as possible, we were looking for examples that employ standard regression models instead of more advanced empirical techniques. One challenge in this regard is that papers with simple OLS regressions often refrain from making causal claims and instead resort to alternative formulations to describe effect sizes, such as “association,” “pattern,” or “link” (Hernán, 2018). However, a recent paper by Hoffman & Strezhnev (2023) provides a fitting example. The authors estimate the causal effect of longer travel time on the probability of a default judgement being made in eviction cases as a result of defendants not showing up or being late to court. Using OLS in a sample of more than 200,000 eviction proceedings in the city of Philadelphia between 2005 and 2021, they find that an increase of one hour in estimated travel time raises the likelihood of a default judgement by 3.8%–8.6%. This effect is meaningful because defaults are difficult to reopen and tenants who fail to show up in court cannot benefit from “Civil Gideon” protections offered in major urban areas in the United States.

In their models, Hoffman and Strezhnev control for neighborhood characteristics such as census tract income levels, as well as race and ethnicity. Interestingly, for our discussion, they also control for building characteristics and find a positive and statistically significant effect size of multi-unit apartment buildings (compared to row houses or single family dwelling) on the probability of default judgements. However, as the authors discuss in their paper, this relationship is unlikely to have a causal interpretation, since building characteristics might be correlated with other influence factors such as unfavorable terms in residential leases or the geographical distribution of dwellings within the city. We now turn to an example closer to the organizational research context.

Early Research Exposure and Career Choices

Azoulay et al. (2021) investigate the effect of early career exposure to frontier research on the career trajectory of potential innovators. Their specific empirical setting is the Associate Training Program (ATP) of the National Institutes of Health (NIH) in the United States. The ATP was started in 1953 as a training program for recent MD graduates. Participants were sent to the NIH intramural campus in Bethesda, Maryland, to receive two to three years research training under the supervision of NIH investigators. Since the NIH was originally established within the Marine Hospital Service, participation in the program fulfilled a draftee’s military service obligation. Therefore, applying to the ATP became particularly popular among young physicians during the Vietnam War period (1965–1975).

After a first screening round, applicants were invited to an interview on NIH campus to determine who would eventually be selected to participate in the program. Selection criteria were related to applicants’ prior research activities (which Azoulay et al. measure by their number of pre-ATP publications), their academic achievements (proxied by whether they were elected to the $A Ω A$ Honor Medical Society), experience (i.e., whether they held a PhD at the time of application and the number of internships they had completed), and the reputation of the institutions where applicants received their training (measured by NIH grants for applicants’ medical school and internship hospital). Importantly, Azoulay et al. argue that although the pool of applicants to the ATP was indeed a highly selected group, selection at the (second) interview stage was based entirely on these observable characteristics. Applicants were early in their career and rather homogeneous in their characteristics. It was therefore hard to select them based on their future research potential beyond a few observable markers. This feature of the particular institutional setting allows Azoulay et al. to employ a selection-on-observables design. Based on that they estimate that ATP participants were twice as likely to pursue a research-focused career later on compared to unsuccessful applicants. As a result, trainees accumulated more publications, citations and grant funding over their life-cycle. Furthermore, they were significantly more likely to receive prestigious career awards, including the Nobel Prize, and to become elected members of the National Academy of Sciences.

Figure 2 synthesizes the assumptions leading to the empirical strategy by Azoulay et al. in the form of a causal diagram. Controlling for applicants’ prior research activities, academic achievements, experience and school reputation (the authors incorporate several covariates for each of these dimensions, including medical school and internship hospital fixed effects) is a valid backdoor adjustment set for estimating the causal effect of ATP participation on the choice of pursuing a research career in this graph.⁶ The analysis depends crucially on the assumption that the unobserved (latent) variable research potential does not directly affect program participation (ATP Participation $⧸ ⇠$ Research Potential); that is, interviewers are not able to select applicants based on private information.

Figure 2.

Causal diagram visualizing the empirical strategy by Azoulay et al. (2021).

Azoulay et al. employ an inverse probability weighting estimator (Austin & Stuart, 2015). As such, covariates are only used to estimate the propensity score of receiving treatment and do not appear in an outcome regression. However, in this setting, it would also not be advisable to interpret the effect of control variables such as prior research activities on career choice. The latent node research potential jointly affects an applicant’s prior research activities as well as future career choices. Thus, while prior research activity is a valid control for the effect of ATP participation, it is also endogenous, similar to the situation in Figure 1b. Consequently, even if we were to find a positive correlation between prior research activities and pursuing a research career (which is not reported by Azoulay et al.), it would be premature to conclude that, for example, early publication success during medical school is a significant driver of subsequent career choices, since both of these variables are likely confounded by an applicant’s overall ability. The research design only allows to draw policy conclusions for the treatment variable ATP participation. Researchers should, therefore, be careful not to overinterpret their empirical results, even if that promises to offer interesting additional perspectives on a given research topic.

Analyst Coverage and Innovation

In applied empirical research, it is not uncommon for estimated treatment effects to change significantly when more advanced identification strategies are employed compared to standard OLS regression. For example, Hopp et al. (2020) find that CEO appearance is no longer related to company performance once firm fixed effects are incorporated in the analysis. Using a discrete choice experiment, Mas & Pallais (2017) show that on average workers value flexible work arrangements much less than a simple compensating wage differentials regression would indicate. Furthermore, because of simultaneity bias, policing levels and crime rates are often positively related, while Mello (2019) demonstrates a negative causal effect of policing on crime, exploiting a natural experiment in a difference-and-difference design.

If subsequent research uses the same variables as controls, however, it becomes immediately clear that their estimated regression coefficients should not be interpreted in a causal way. One case in point comes from the literature on analyst coverage and innovation. He & Tian (2013) find a negative relationship between analyst coverage and patenting in a study of U.S. public firms from 1993 to 2005, using difference-and-differences and an instrumental variable approach. It has been theorized that this result arises because external stock market analysts following a firm often exert excessive pressure on executives, which can worsen managerial myopia and impede investment in long-term innovation projects. For these reasons, analyst coverage is used as a control variable in other studies of R&D activities in publicly listed firms. However, since it is not the main variable of interest in these studies, often less stringent identification strategies are employed, possibly leading to unexpected results. For example, Chen et al. (2016) and Huang et al. (2022) consistently find positive effects of analyst coverage in regressions with the natural logarithm of patents as the dependent variable, which seemingly contradicts He & Tian (2013).

Analyst coverage can be a valid control variable even though it is endogenous (if the underlying causal structure, e.g., corresponds to Figure 1b). Nevertheless, interpreting positive regression coefficients as evidence against He & Tian (2013) would be a mistake. To use the analogy of Bayesian updating, control variable estimates from studies such as Chen et al. (2016) and Huang et al. (2022) should not alter the posterior probability of analyst coverage having a negative effect on innovation. Consequently, in another study by Chen et al. (2021), in which analyst coverage is also included as a control variable in a patent count regression, the regression coefficients of the controls are not reported. This is in accordance with the recommendations that we will discuss in the following.

Discussion and Recommendations

Beyond pure prediction tasks, the purpose of regression analysis in organizational research is typically to build and test theories that explain the causal mechanisms underlying a studied phenomenon (Sutton & Staw, 1995). In this article, we argued that attaching substantive meaning to the marginal effects of biased control variables is problematic, however, as researchers could develop false intuitions or draw erroneous managerial and policy conclusions. Therefore, we think it is advisable to not discuss the results obtained for control variables in quantitative papers, unless the researchers can be sure that they have accounted for all relevant influence factors of the outcome in a regression (all-causes regression). Since in many practical settings this is unlikely to be the case, we recommend to treat controls as nuisance parameters, which are included in the analysis for identification purposes (and discussed as such) but their effects are not interpreted (Liang & Zeger, 1995; Meehl, 1971). This corresponds to the way control variables are treated by nonparametric matching estimators (Heckman et al., 1998) and modern machine learning techniques for high-dimensional settings (Chernozhukov et al., 2018; Hünermund et al., 2023). These methods do not report estimation results related to controls, either because there would be simply too many covariates in the analysis (which is the primary use-case for machine learning) or marginal effects of control variables are not returned by the estimation protocol (as in the matching case).

Our recommendations thereby depart from prior literature insofar as control variable should not be promoted to have equal status with the other variables in the study (Spector & Brannick, 2011: p. 297). Research designs based on control variables are employed to estimate the causal effect of a treatment variable on an outcome. As such, the treatment variable cannot be endogenous, otherwise estimates would be biased and other, more suitable research designs (such as instrumental variables, regression discontinuity designs, etc.) should be applied. By contrast, control variables can be endogenous (Frölich, 2008) and, as we argued in the preceding theoretical discussion, will likely be so in practice.⁷ Controls should be chosen to close all backdoor paths between a treatment and outcome, based on a theoretical model of the context under study (Bono & McNamara, 2011). As we have demonstrated previously, it is thereby not necessary to include all causal influence factors of the outcome variable in a regression. Our example (Azoulay et al., 2021) illustrates that in many cases it might actually be easier to control the treatment assignment mechanism instead, if institutional knowledge is richer about what determines treatment take-up compared to the potentially long list of variables that affect the outcome. Moreover, in many situations, researchers have the choice between different valid adjustment sets (see Figure 1d), which highlights their auxiliary nature for the analysis.

Since accounting for all influence factors of the outcome might be unrealistic in many contexts and control variables are therefore likely to be endogenous, interpreting their effect sizes in light of theory is potentially dangerous. Authors could infer wrong conclusions for managerial advice and subsequent studies might be inclined to build a theory based on biased empirical results. To avoid this, we therefore recommend to refrain from interpreting control variables in published papers. Moreover, predicting the sign of control variable estimates ex-ante (Atinc et al., 2012) is difficult if endogenous control variables can pick up the effect of a multitude of other influence factors. Therefore, formulations such as “estimates of control variables have expected signs” should be avoided. As a “nudge” to stir the research community away from overinterpreting control variables, we find it appropriate that authors omit their coefficients entirely from regression tables or relegate them to an appendix.

Table 2 (which is an adapted version of Table 1) illustrates such a regression table format, in which check marks are included to indicate which variables were controlled for. This corresponds to the way how estimation results are presented, for example, in papers using nearest neighbor or propensity score matching. We acknowledge that omitting control variable coefficients in regression tables constitutes a tradeoff with respect to transparency. However, we believe that this suggestion is justified by the lower risk of drawing incorrect theoretical inferences from empirical studies. When authors have important reasons for reporting regression coefficients of control variables, the format of Table 1 constitutes a viable compromise, in our view. Compared to the standard format of regression table, Table 1 clearly separates controls from the treatment variable and includes a note that the control variable estimates should not be interpreted causally.

Table 2.

Regression Table Without Control Variable Coefficients Being Reported.

	Figure 1a			Figure 1b	Figure 1c	Figure 1d
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
$X$	1.017	1.004	1.015	0.993	1.001	0.991	1.006	1.003
	(0.015)	(0.006)	(0.010)	(0.012)	(0.008)	(0.057)	(0.007)	(0.010)
$Z_{1}$	✓ $✓$		✓	✓	✓	✓		✓
$Z_{2}$		✓	✓					✓
$Z_{3}$							✓	✓
$Z_{4}$							✓	✓
$Z_{5}$							✓	✓

Note: Bootstrapped standard errors in parentheses.

We emphasize that we agree with Becker (2005) in that control variables should be carefully discussed and authors need to justify their validity based on prior theory. However, their estimated coefficients are less relevant. It suffices to discuss the rationale for selecting specific control variables in the empirical section and to clearly indicate their inclusion in the table notes. Since the proper justification of a regression design can only come from theory, we caution against deciding about the inclusion of control variables based on their incremental contribution to the $R^{2}$ of the model (Carlson & Wu, 2012). This is the celebrated “no causes in, no causes out” principle (Cartwright, 1989), which states that the validity of causal inferences must ultimately be supported by theoretical considerations external to the data. For example, bad controls (as discussed by Cinelli et al. (2022)) often have a lot of predictive power but nonetheless lead to invalid causal inferences. Therefore, we also do not see a reason why authors should report models with and without control variables and compare the share of explained variance between them (as suggested, e.g., by Atinc et al. (2012) and Becker et al. (2015)).

Our recommendations are in line with Westreich & Greenland (2013) who discuss a similar problem with respect to the interpretation of potentially endogenous controls in epidemiology. Because epidemiological studies usually present the results of multivariate regression analyses right after a table with descriptive statistics of the data, they coined the term table 2 fallacy. Keele et al. (2020) discuss related examples from the field of political science. They emphasize that for estimates of control variables to be given a causal interpretation, their effects need to be themselves causally identified. Since this is only plausible if there are no omitted variables (or the controls are unrelated to the omitted variables), we recommend researchers to focus attention on one causal factor (or a small set) at a time, for which backdoor paths can realistically be enumerated, and treat control variables as nuisance parameters instead.

Finally, we caution against including estimates of potentially biased controls in meta-analyses (Aguinis et al., 2010). Such studies pool the effects of a focal variable on an outcome across several papers. According to Becker (2005), systematic reporting of control variables facilitates cumulative science and knowledge aggregation by significantly increasing the pool of studies from which effect sizes for meta-analyses can be drawn from:

Nonreporting of control variable findings hinder any meta-analyses that would have otherwise included the controls. For instance, in a study of the relationship between employee commitment and organizational citizenship behavior, a researcher might control for extraversion and agreeableness but not report the findings for the controls. As a result, later meta-analyses cannot include these findings in the assessment of connections between personality and organizational citizenship behavior. Becker (2005: p. 285)

This recommendation refers to meta-analyses of partial correlations and marginal effects (“meta-regression”), which are increasingly common in organizational research and economics (Stanley & Doucouliagos, 2012). Compared to zero-order correlations, they have the advantage of being able to filter out other potential confounding influence factors in settings when randomized control trials are not feasible.⁸ However, the quoted passage fails to mention that control variables (here extraversion and agreeableness) are unlikely to have a causal interpretation themselves and, therefore, add little to the evidence base regarding a certain effect size. As their coefficients may represent a combination of several different causal mechanisms jointly operating on the outcome (here citizenship behavior), they do not provide accurate information about a theoretically meaningful quantity. Moreover, coefficients can vary substantially depending on which admissible adjustment sets are used (e.g., compare columns 6–8 in Table 1). Consequently, meta-analyses should be restricted to the main treatment variable(s), for which a plausible identification argument can be established, which highlights once again the unequal status of treatment and control variables in regression analysis.

To conclude, there is no reason to be worried if the estimated coefficients of control variables do not have expected signs, since they are likely to be biased anyway in practical applications. Instead, researchers should rather focus on interpreting the marginal effects of the main variables of interest in their manuscripts. The estimation results obtained for controls, by contrast, have little substantive meaning and can therefore safely be omitted—or relegated to an appendix. This approach will not only prevent researchers from drawing wrong causal conclusions based on endogenous controls, but will furthermore allow to streamline the discussion sections of quantitative research papers and save on valuable manuscript space.

Supplemental Material

sj-pdf-1-orm-10.1177_10944281231219274 - Supplemental material for On the Nuisance of Control Variables in Causal Regression Analysis

Supplemental material, sj-pdf-1-orm-10.1177_10944281231219274 for On the Nuisance of Control Variables in Causal Regression Analysis by Paul Hünermund and Beyers Louw in Organizational Research Methods

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iDs

Paul Hünermund

Beyers Louw

Supplemental Material

Supplemental material for this article is available online.

Notes

Author Biographies

Paul Hünermund is an Assistant Professor of Strategy and Innovation at Copenhagen Business School. He pursued his studies in economics at the University of Mannheim, HEC Lausanne, and NYU Stern School of Business, and holds a PhD in business economics from KU Leuven in Belgium. His research has been published in the Journal of Management Studies, the Econometrics Journal, Research Policy, Journal of Product Innovation Management, International Journal of Industrial Organization, MIT Sloan Management Review, and Harvard Business Review, among others. Dr. Hünermund serves on the editorial board of the Journal of Causal Inference and the executive team of the Technology and Innovation Management (TIM) division at the Academy of Management.

Beyers Louw is a Postdoctoral Researcher in Strategic Management at the University of Groningen. He pursued his studies at Stellenbosch University, University of Gothenburg, Norwegian School of Economics, and Maastricht University. His work on quantitative research methodology has been published in the Journal of Causal Inference.

References

Aguinis

Pierce

C. A.

Bosco

F. A.

Dalton

D. R.

Dalton

C. M.

. (2010). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14(2), 306-331. https://doi.org/10.1177/1094428110375720

Angrist

J. D.

(1990). Lifetime earnings and the Vietnam Era Draft Lottery: Evidence from social security administrative records. The American Economic Review, 80(3), 313-336. https://doi.org/10.3386/w3514

Atinc

Simmering

M. J.

Kroll

M. J.

(2012). Control variable use and reporting in macro and micro management research. Organizational Research Methods, 15(1), 57-74. https://doi.org/10.1177/1094428110397773

Austin

P. C.

Stuart

E. A.

(2015). Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Statistics in Medicine, 34(28), 3661-3679. https://doi.org/10.1002/sim.6607

Azoulay

Greenblatt

W. H.

Heggeness

M. L.

(2021). Long-term effects from early exposure to research: Evidence from the NIH “yellow berets”. Research Policy, 50(9), 104332. https://doi.org/10.1016/j.respol.2021.104332

Becker

T. E.

(2005). Potential problems in the statistical control of variables in organizational research: A qualitative analysis with recommendations. Organizational Research Methods, 8(3), 274-289. https://doi.org/10.1177/1094428105278021

Becker

T. E.

Atinc

Breaugh

J. A.

Carlson

K. D.

Edwards

J. R.

Spector

P. E.

(2015). Statistical control in correlational studies: 10 essential recommendations for organizational researchers. Journal of Organizational Behavior, 37(2), 157-167. https://doi.org/10.1002/job.2053

Bono

J. E.

McNamara

(2011). Publishing in AMJ—part 2: Research design. Academy of Management Journal, 54(4), 657-660. https://doi.org/10.5465/amj.2011.64869103

Carlson

K. D.

(2012). The illusion of statistical control: Control variable practice in management research. Organizational Research Methods, 15(3), 413-435. https://doi.org/10.1177/1094428111428817

10.

Cartwright

(1989). Nature’s Capacities and Their Measurement. Clarendon Press.

11.

Chen

Hsu

P.-H.

Podolski

E. J.

(2016). Be nice to your innovators: Employee treatment and corporate innovation performance. Journal of Corporate Finance, 39, 78-98. https://doi.org/10.1016/j.jcorpfin.2016.06.001

12.

Chen

S.-S.

Kao

W.-C.

Wangc

(2021). Tax policy and innovation performance: Evidence from enactment of the alternative simplified credit. Journal of Banking and Finance, 125, 106081. https://doi.org/10.1016/j.jbankfin.2021.106081

13.

Chernozhukov

Chetverikov

Demirer

Duflo

Hansen

Newey

Robins

(2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1-C68. https://doi.org/10.1111/ectj.12097

14.

Cinelli

Forney

Pearl

(2022). A crash course in good and bad controls. Sociological Methods & Research. https://doi.org/10.1177/0049124122109955

15.

Cinelli

Hazlett

(2020). Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 82(1), 39-67. https://doi.org/10.1111/rssb.12348

16.

DeSimone

J. A.

Brannick

M. T.

Boyle

E. H. O.

Ryu

J. W.

(2021). Recommendations for reviewing meta-analyses in organizational research. Organizational Research Methods, 24(4), 694-717. https://doi.org/10.1177/1094428120967089

17.

Durand

Vaara

(2009). Causation, counterfactuals, and competitive advantage. Strategic Management Journal, 30(12), 1245-1264. https://doi.org/10.1002/smj.793

18.

Flammer

Bansal

(2017). Does a long-term orientation create value? Evidence from a regression discontinuity. Strategic Management Journal, 38(9), 1827-1847. https://doi.org/10.1002/smj.2629

19.

Frölich

(2008). Parametric and nonparametricregression in the presence of endogenous control variables. International Statistical Review, 76(2), 214-227. https://doi.org/10.1111/j.1751-5823.2008.00045.x

20.

J. J.

Tian

(2013). The dark side of analyst coverage: The case of innovation. Journal of Financial Economics, 109(3), 856-878. https://doi.org/10.1016/j.jfineco.2013.04.001

21.

Heckman

J. J.

Ichimura

Todd

(1998). Matching as an econometric evaluation estimator. The Review of Economic Studies, 65(2), 261-294. https://doi.org/10.1111/1467-937X.00044

22.

Hernán

M. A.

(2018). The c-word: Scientific euphemisms do not improve causal inference from observational data. American Journal of Public Health, 108(5), 616-619. https://doi.org/10.2105/AJPH.2018.304337

23.

Hoffman

D. A.

Strezhnev

(2023). Longer trips to court cause evictions. Proceedings of the National Academy of Sciences of the United States of America, 120(2), e2210467120. https://doi.org/doi.org/10.1073/pnas.221046712

24.

Hopp

Wentzel

Rose

(2020). Chief executive officers’ appearance predicts company performance, or does it? A replication study and extension focusing on CEO successions. The Leadership Quarterly, 101437. https://doi.org/10.1016/j.leaqua.2020.101437

25.

Huang

Y.-H.

lih Liang

Truong

Q.-T.

Wang

(2022). No new tricks for old dogs? Old directors and innovation performance. Technological Forecasting & Social Change, 179, 121659. https://doi.org/10.1016/j.techfore.2022.121659

26.

Hünermund

Bareinboim

(2023). Causal inference and data fusion in econometrics. The Econometrics Journal. https://doi.org/10.1093/ectj/utad008

27.

Hünermund

Louw

Caspi

(2023). Double machine learning and automated confounder selection—A cautionary tale. Journal of Causal Inference, 11(1), 20220078. https://doi.org/10.1515/jci-2022-0078

28.

Hünermund

Louw

Rönkkö

(2022). The choice of control variables: How causal graphs can inform the decision. Academy of Management Proceedings, 2022(1). https://doi.org/10.5465/AMBPP.2022.294

29.

Keele

Stevenson

R. T.

Elwert

(2020). The causal interpretation of estimated associations in regression models. Political Science Research and Methods, 8(1), 1-13. https://doi.org/10.1017/psrm.2019.3

30.

Liang

K.-Y.

Zeger

S. L.

(1995). Inference based on estimating functions in the presence of nuisance parameters. Statistical Science, 10(2), 158-173. https://doi.org/10.1214/ss/1177010028

31.

Mas

Pallais

(2017). Valuing alternative work arrangements. American Economic Review, 107(12), 3722-3759. https://doi.org/10.1257/aer.20161500

32.

Meehl

P. E.

(1971). High school yearbooks: A reply to Schwarz. Journal of Abnormal Psychology, 77(2), 143-148. https://doi.org/10.1037/h0030750

33.

Mello

(2019). More cops, less crime. Journal of Public Economics, 172, 174-200. https://doi.org/10.1016/j.leaqua.2020.101437

34.

Pearl

(2000). Causality: Models, Reasoning, and Inference. (1st ed.). Cambridge University Press.

35.

Spector

P. E.

Brannick

M. T.

(2011). Methodological urban legends: The misuse of statistical control variables. Organizational Research Methods, 14(2), 287-305. https://doi.org/10.1177/1094428110369842

36.

Stanley

Doucouliagos

. (2012). Meta-Regression Analysis in Economics and Business. Routledge Advances in Research Methods. Routledge, 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN.

37.

Sutton

R. I.

Staw

B. M.

(1995). What theory is not. Administrative Science Quarterly, 40(3), 371. https://doi.org/10.2307/2393788

38.

Textor

Liśkiewicz

. (2011). Adjustment criteria in causal diagrams: An algorithmic perspective. In Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, pages 681-688. AUAI Press.

39.

Westreich

Greenland

(2013). The table 2 fallacy: Presenting and interpreting confounder and modifier coefficients. American Journal of Epidemiology, 177(4), 292-298. https://doi.org/10.1093/aje/kws412

40.

Wooldridge

J. M.

(2002). Econometric Analysis of Cross Section and Panel Data. The MIT Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.23 MB