Sage Journals: Discover world-class research

Abstract

Understanding the experiences of vulnerable workers is an important scientific pursuit. For example, research interest is often in quantifying the impacts of adverse exposures such as discrimination, exclusion, harassment, or job insecurity, among others. However, routine approaches have only focused on the average treatment effect, which encapsulates the impact of an exposure (e.g., discrimination) applied to the entire study population—including those who were not exposed. In this paper, we propose using a more refined causal quantity uniquely suited to address such causal queries: The effect of treatment on the treated (ETT) from the causal inference literature. We explain why the ETT is a more pertinent causal estimand for investigating the experiences of vulnerable workers by highlighting three appealing features: Better interpretability, greater accuracy, and enhanced robustness to violations of empirically untestable causal assumptions. We further describe how to estimate the ETT by introducing and comparing two estimators. Both estimators are conferred with a so-called doubly robust property. We hope the current proposal empowers organizational scholars in their crucial endeavors dedicated to understanding the vulnerable workforce.

Keywords

belonging discrimination equity,diversity,and inclusion inequality average treatment effect among the treated

Identifying and resolving barriers to equity and inclusion is at the forefront of organizational research (Amis et al., 2021; Avery et al., 2023; Bapuji et al., 2020a; Restubog et al., 2023). This necessitates understanding the unique challenges faced by the vulnerable, minoritized, and marginalized workforce (henceforth termed vulnerable workforce for brevity; Restubog et al. (2021)), such as discrimination¹ , exclusion, sexual harassment, and job insecurity. For example, organizational and management scholars have investigated how sexual harassment increases organizational withdrawal (Cortina & Areguin, 2021); how pregnancy discrimination affects the well-being and health of working mothers and their babies (Hackney et al., 2021); how perceived human immunodeficiency virus (HIV) stigma affects job effectiveness (Ocampo et al., 2023); how job insecurity affects workplace behaviors (Shoss et al., 2023); or how disability discrimination leads to the compensation gap between job candidates with vs. without disabilities (Speach et al., 2023); among many others. Such causal queries are a cornerstone of organizational research because they shed light on the disadvantages and barriers experienced by the understudied and vulnerable workforce, advance organizational theories, inform intervention development, and promote equality and inclusion in organizations and society (Amis et al., 2021, 2020; Bapuji et al., 2020a,b; Restubog et al., 2021, 2023).

While quantifying the effects of adverse exposures (i.e., “treatments”) among vulnerable workers is an essential scientific pursuit, current analytic approaches are inadequate for engaging in this task. Take discrimination against women in the workplace as an example. The percentage of women reporting being discriminated against in equal pay or consideration for promotion is 41% (National Public Radio et al., 2017). Routine analyses ignore this information and seek to estimate the impact of gender discrimination on an outcome of interest, such as well-being, among the entire study population of women employees. The causal quantity targeted here is the so-called average treatment effect (ATE). (An equivalent term is the average causal effect; ACE.) By estimating the ATE, the study population encompasses women who did not experience or perceive discrimination (at the time of investigation² ) and eludes a more focused assessment of how discrimination affects those discriminated against. The consequence of using the ATE is an inability to obtain accurate and meaningful estimates of the effect of discrimination among women who experienced discrimination.

In this article, we explain why the ATE, notwithstanding its widespread adoption, has critical shortcomings that undermine its relevance for causal inference in research on vulnerable populations. To address these limitations, we introduce an alternative, more fine-grained causal quantity: The so-called effect of treatment on the treated (ETT). (An equivalent term is the average treatment effect on the treated; ATT.) The ETT is a well-established causal quantity in the statistics and causal inference literature and has been widely adopted in substantive research areas such as education (see, e.g., (Morgan & Winship, 2015: Section 2.7.1); Paulsen & McCormick (2020); Stephan et al. (2009); Zang et al. (2023)), economics (see, e.g., Heckman et al. (1999)), and public health (see, e.g., Reifeis & Hudgens (2022); Webster (2022)). Here, we put forth a novel application of the ETT for organizational science.

In the remainder of this article, we explain what the ETT is and why targeting the ETT advances organizational science. We then describe two different methods to estimate the ETT: one from the statistics literature and one from the social sciences literature. Both methods have a so-called “doubly robust” property protecting the estimators from biases due to incorrectly assumed regression models. Further, we illustrate the estimation procedure using two real-world studies with publicly available datasets. A glossary of causal inference terms we use throughout is presented in Table 1 for readers unfamiliar with this framework. The R (R Core Team, 2023) scripts to reproduce the example analyses, as well as the Supplemental Online Materials (SOMs), are available online on GitHub (https://github.com/wwloh/ett-org). Through methodology advancement in causal inference, we hope this article contributes to more accurate and inclusive organizational science.

Table 1.

Glossary of Key Causal Inference Terms Used in This Paper.

Term	Description
Estimand	What a researcher is seeking to quantify using an estimation procedure.
Potential outcome	An individual’s outcome had they been exposed, possibly
	counter to fact, to a given level of the focal predictor (i.e., exposure or “treatment”).
Causal estimand	A causal quantity that formalizes a researcher’s
	causal question in terms of potential outcomes;
	this is the target of an estimation procedure.
ATE	A causal estimand contrasting the average potential
	outcomes for the entire study population; also termed ACE.
ETT	A causal estimand contrasting the average potential outcomes
	among only those observed as being exposed or “treated” ( $X = 1$ );
	also termed average treatment effect among/for/in/on the treated (ATT).
Confounder	A covariate that, when unadjusted (or not statistically controlled) for,
	generates a noncausal (i.e., “spurious”) association between exposure and outcome,
	thus inducing nonrandom bias in the effect estimate.
Ignorability	An empirically untestable causal assumption that exposure is independent
	of potential outcomes, possibly conditional on given baseline covariates that suffice to
	eliminate all confounding; also termed no unmeasured confounding.
Propensity score	The probability that an individual would have been exposed conditional
	on their baseline covariates (such as those sufficient to support conditional ignorability).
IPW	A weight allocated to each individual, based on their observed
	exposure and propensity score, so that exposure is independent of the covariates
	(predicting the propensity score) in the weighted pseudo-population.
Doubly robust estimator	An estimator that combines (i) a propensity score model for exposure given
	baseline covariates and (ii) an outcome model given exposure and baseline covariates.
	A doubly robust estimator is protected from biases even when one of these
	models is incorrectly specified as long as the other is correctly specified, without
	having to know which is (in)correct. This can make the estimator more
	convincing in practice than others which rely on only a single
	(either propensity score or outcome) model.

ATE = average treatment effect; ACE = average causal effect; ETT = effect of treatment on the treated; IPW = inverse probability of treatment weight.

Why is the ETT Relevant?

Before proceeding to the technical details of the ETT, we first seek to address the question: How is the ETT relevant to organizational research?

We put forth that the ETT aligns seamlessly with the core research foci on vulnerable populations in contemporary organizational research. As opposed to the ATE, which conceptualizes the average effect of an exposure on the entire study population, the ETT focuses on the exposed subpopulation. For example, in other fields outside organizational science, the ETT has been used to quantify the effect of attending college among college students (Stephan et al., 2009), the effect of a training program for the participants (Falk et al., 2005), and the effect of smoking on those who smoked (Auld, 2005). This appealing feature of the ETT makes it exceptionally well suited for understanding the experiences of the vulnerable workforce. Organizational scholars can use the ETT to address causal questions such as the effects of discrimination among workers who experienced it, the consequences of exclusion among individuals who were marginalized, and the impact of job insecurity among those facing employment uncertainty.

To provide concrete examples of past studies in organizational research where the ETT is readily applicable, we sampled from three leading journals that regularly publish original empirical studies: Journal of Applied Psychology, Journal of Management, and Academy of Management Journal. We searched for articles that used quantitative methods to study the effects of adverse exposures as illustrative examples. Our search was conducted in November 2023. Given the large number of studies, we selected ten articles published most recently (in 2023 or in press). In one such example, scholars studied how Asian employees’ past experiences with racial discrimination at work affected their propensity to combat racism against Black coworkers (Jun et al., 2023). In another example, scholars studied how job insecurity influenced job performance and counterproductive work behaviors among working adults in the United States (Shoss et al., 2023). Please see Table 2 for the complete list of examples.

Table 2.

Examples of Original Research Articles Where the ETT is Applicable.

Article	Journal	Focal exposure	Outcome(s) of interest	Target population (for the ATE)
Gloor et al. (2023)	JM	Perceptions of selective incivility (the frequency of uncivil acts toward female colleagues because of their gender)	Organizational identification	Early career academics
Jun et al. (2023)	JAP	Racial discrimination at work	Propensity to combat racism against Black coworkers	Asian employees
King et al. (2023)	JAP	Racial microaggressions at work	Burnout, job satisfaction, racism-related vigilance at work	Black employees
Lebel et al. (2023)	JAP	Financial precarity	Burnout	University nonfaculty employees during COVID-19
Liao et al. (2023)	AMJ	Being nominated for but not receiving an award (nonwinner nominees vs. non-nominees)	Collaboration responsiveness	Employees who are exposed to employee award programs
Little & Masterson (2023)	AMJ	(A lack of) perceived organizational support	Home stress, family-work conflict, Interpersonal deviance at work	Couples who recently had a baby and wherein the mother was about to return to work after maternity leave
Ocampo et al. (2023)	JAP	Stigma associated with human immunodeficiency virus (HIV)	Job effectiveness (i.e., in-role performance and organizational citizenship behaviors)	Employees with HIV
Qu et al. (2023)	JM	Quantitative and qualitative job insecurity	Radical and incremental creativity	Employees working at companies that do not offer lifetime employment
Sayre (2023)	JAP	Pay volatility	Physical symptoms, insomnia	Tipped workers, gig workers, workers in sales, marketing, and finance
Shoss et al. (2023)	JM	Job insecurity	Job performance, counterproductive work behaviors, Self-presentation ingratiatory behavior, knowledge hiding	Working adults in the United States

The papers were alphabetically ordered by the first author’s last name. AMJ = Academy of Management Journal; JAP = Journal of Applied Psychology; JM = Journal of Management; ETT = effect of treatment on the treated; HIV = human immunodeficiency virus.

Next, to demonstrate the broad relevance of such causal questions in the organizational literature, we carefully reviewed three leading journals publishing integrative reviews of empirical research or theoretical advances in the field: Academy of Management Annals, Academy of Management Review, and Annual Review of Organizational Psychology and Organizational Behavior. A perusal of all issues published between 2020 and 2023 revealed various exposures to which the ETT can be readily applied. These include abusive supervision, bullying, customer mistreatment, discrimination, harassment, income instability, job insecurity, limited access to resources, mental health issues, perceived organizational support, social exclusion or belonging threats, stereotypes or prejudices, stigma, victimization, and workplace incivility. Please see Table 3 for the complete list of exposures.

Table 3.

Examples of Reviews Demonstrating Organizational Relevance of the ETT.

Article	Journal	Research domains	Exposure(s) of interest
Amis et al. (2021)	AMA	Organizational practices reinforce or reproduce inequality	Discrimination; job insecurity; lack of access to resources
Avery et al. (2023)	AROPOB	Race and organizational justice	Bullying; discrimination; harassment; incivility
Bapuji et al. (2020a)	AMA	Economic inequality	Discrimination; income instability; job insecurity
Bowles et al. (2022)	AROPOB	Gender and organizational negotiations	Gender stereotypic norms
Brands et al. (2022)	AMA	Gender in social network research	Discrimination; exclusion
Bruton et al. (2023)	AMA	Racial inequality in entrepreneurship	Discrimination; lack of access to resources
Cortina & Areguin (2021)	AROPOB	Sexual harassment in the workplace	Sexual harassment
Eisenberger et al. (2020)	AROPOB	Perceived organizational support	Perceived organizational support
Hebl et al. (2020)	AROPOB	Modern discrimination in organizations	Discrimination
Kelloway et al. (2023)	AROPOB	Mental health in the workplace	Mental health problems or illnesses
Kreiner et al. (2022)	AROPOB	Stigmatized work and stigmatized workers	Being stigmatized
Mah et al. (2023)	AMA	Women’s underrepresentation in the executive ranks	Discrimination
Phillips et al. (2022)	AMA	Race and gender inequality	Belonging threats; discrimination
Prasad (2023)	AMR	The model minority	Discrimination
Rosado-Solomon et al. (2023)	AMA	Mental health and mental illness in organizations	Mental health problems or illnesses
Salmon et al. (2023)	AMA	Indigenous peoples and contexts	Discrimination; exclusion
Thatcher et al. (2023)	AMA	Intersectionality in the workplace	Discrimination, harassment
Vogel & Bolino (2020)	AMR	Abusive supervision	Abusive supervision
Zhang et al. (2021)	AMA	Stigma	Being stigmatized
Zhong et al. (2023)	AMA	Workplace aggression	Abusive supervision; bullying; customer mistreatment; harassment; incivility; ostracism; victimization

The papers were alphabetically ordered by the first author’s last name. AMA = Academy of Management Annals; AROPOB = Annual Review of Organizational Psychology and Organizational Behavior; AMR = Academy of Management Review; ETT = effect of treatment on the treated.

Notably, these adverse exposures are not confined to an esoteric or narrow research area but rather of core interest to scholars across a wide variety of research domains and organizational contexts. For example, discrimination was an exposure of interest not only within discrimination research (which has deep roots and a history of more than a century in organizational research; Colella et al. (2017)) (Hebl et al., 2020), but consistently appeared in different research domains, such as research on Indigenous Peoples and contexts (Salmon et al., 2023), racial inequality in entrepreneurship (Bruton et al., 2023), women’s underrepresentation in the executive ranks (Mah et al., 2023), intersectionality in the workplace (Thatcher et al., 2023), and inequality in organizations (Amis et al., 2020; Avery et al., 2023; Bapuji et al., 2020b; Phillips et al., 2022), among others.

Given the state of the literature and the prevalence and significance of causal queries pertinent to the experiences of the vulnerable workforce, we put forth that our introduction of the ETT to organizational research has the potential to advance the field in three ways:

Enhanced Research on Diversity and Inequality. Evident from our review of the literature is the increasing interest in research on diversity and inequality (Amis et al., 2021, 2020; Bapuji et al., 2020b; Ezerins et al., 2023; Speach et al., 2023; Thatcher et al., 2023). However, for progress in substantive research and theoretical advancement, researchers must be judicious about the interpretability and usefulness of their analytic choices. In the sequel, we discuss the shortcomings of routine methods for examining the challenges and experiences of vulnerable workers. To address these concerns, we introduce the ETT as an alternative causal inferential tool. Our aim is to enhance the accuracy and robustness of empirical investigations of diversity and inequality in organizational science.

Strengthening Causal Inference. Notwithstanding causal inference being a central pursuit in organizational and management science, the validity of causal claims can be called into question (Cornelissen & Kaandorp, 2023). To contribute to ongoing conversations about bolstering causality in organizational research, our work encourages scholars to think carefully about the causal quantity they are estimating. Defining the target causal quantity before estimation can better align theory with empirical analysis (Hernán, 2015). Currently, the common practice is to (implicitly) target the ATE by default, even when the ATE is unsuitable for the research question at hand. In this article, we critically evaluate this practice and demonstrate the benefits of the ETT as an alternative target causal quantity for organizational science.

Practical Impact on Organizational Practices. Using the ETT to focus on vulnerable segments of the workforce who experienced challenges has the practical impact of guiding scholarship that informs organizational practices and interventions toward achieving equity and inclusion (Lambert et al., 2022). Continuing the example of discrimination in the workplace among Asian employees (Jun et al., 2023), routine approaches targeting the ATE conceptualize the effects of discrimination among the entire study population. In contrast, the ETT allows for a more nuanced and refined focus on the subset who experienced racism. This makes the ETT more pertinent and precise for guiding efforts to address challenges for vulnerable segments of the workforce toward more effective and equitable organizational practices (Restubog et al., 2023).

What is the ETT?

We now define the causal quantities researchers seek to estimate, or estimands, for articulating the causal effects of substantive interest. For expository purposes, we will use a running example of discrimination as the exposure of focal interest. Note, however, that our causal reasonings broadly apply to other adverse exposures faced by vulnerable workers. As stated in the previous section, we provide concrete examples of such exposures in Tables 2 and 3.

We will utilize concepts from the established potential outcomes framework, commonly called the Neyman-Rubin Causal Model (Holland, 1986; Splawa-Neyman et al., 1990; Rubin, 1974). Let $X$ denote whether an individual experienced discrimination ( $X = 1$ ) or not ( $X = 0$ ), and let $Y$ denote the outcome of interest. Let $Y^{1}$ and $Y^{0}$ denote an individual’s potential outcomes had they, possibly different from what was observed, experienced or not experienced discrimination, respectively.³ The ATE (Rubin, 1974) adopted in routine analyses is formally defined as:

ATE \equiv E (Y^{1} - Y^{0}) .

(1)

The ATE conceives the average difference in potential outcomes for the entire study population (precisely, the hypothetical or source population from which the study sample is drawn). Hence, the ATE answers the causal query: What is the impact of discrimination if every individual in the population hypothetically experienced discrimination versus did not experience discrimination, regardless of whether they actually experienced discrimination?

The ETT (Heckman & Robb, 1985) is defined as:

ETT \equiv E (Y^{1} - Y^{0} | X = 1) .

(2)

The ETT is the average difference in potential outcomes among those who experienced discrimination (

X = 1

). Hence, the ETT answers the causal query: Among individuals who experienced discrimination, what is the impact of discrimination had they counterfactually not experienced discrimination? Put differently, among those who were discriminated against, how different would their expected outcomes be had they not experienced discrimination instead?

Why the ETT Enhances Organizational Science

We offer three reasons why the ETT is better suited for investigating the impacts of adverse exposures among vulnerable workers than the ATE: Interpretability, accuracy, and robustness.

Interpretability

The ETT offers a more straightforward and informative interpretation of the causal effects of substantive interest than the ATE. Despite a common misconception that individuals in a vulnerable population “are all alike” (Salmon et al., 2023: p 451), adverse experiences such as discrimination are unlikely to occur to all individuals in a study population; see, for example, National Public Radio et al. (2017) and Restubog et al. (2023). Yet, the ATE quantifies the impact of discrimination among the entire study population. This necessitates conceptualizing the potential outcomes for all individuals under the extreme hypothetical scenario where everyone experienced discrimination, including those who reported not experiencing discrimination within the study context ( $X = 0$ ).

In contrast, the ETT targets the impact of discrimination by focusing on individuals who experienced discrimination within the context of the study ( $X = 1$ ). This bestows the ETT with a more meaningful interpretation than the ATE. This can be especially relevant for informing organizational practices of allocating supporting resources for vulnerable workers. By using the ETT, organizations, policymakers, and stakeholders can better tailor specific efforts or resources to support those exposed to adverse experiences such as discrimination, exclusion, or harassment in the workplace (American Psychological Association, 2021; Hebl et al., 2020).

Accuracy

The ETT assesses the causal effects of interest with more fine-grained accuracy, whereas the ATE is a coarser quantification. To see why, following Equation (2.10) of Morgan & Winship (2015), the ATE can be expressed as a weighted sum of two different causal estimands:

\begin{aligned} ATE & \equiv E (Y^{1} - Y^{0}) \\ = \underset{\equiv ETT}{\underset{⏟}{E (Y^{1} - Y^{0} | X = 1)}} P (X = 1) + \underset{\equiv ETC}{\underset{⏟}{E (Y^{1} - Y^{0} | X = 0)}} P (X = 0) . \end{aligned}

(3)

The effect of exposure on the unexposed group is denoted by ETC and analogously defined as the ETT. The weights are the population proportions that experienced the exposure of interest, such as discrimination,

P (X = 1)

, or did not experience discrimination,

P (X = 0)

. When the impact of discrimination is heterogeneous, so that those who experienced discrimination suffer more severe effects than those who did not, the ATE will inaccurately measure the effect of discrimination among those who experienced discrimination.

As a hypothetical numerical example, suppose that $50 %$ of the study population experienced discrimination while the rest did not. Now, suppose discrimination increases work stress among the exposed group by $1.2$ (on a five-point scale), whereas the impact of discrimination among the unexposed group was $0$ . Under this scenario, scholars targeting the ATE would conclude that the impact of discrimination was $0.6$ , thereby understating the true effect of discrimination among those discriminated against.

Robustness

The ETT is more robust against violations of empirically untestable causal assumptions. That is, the ETT can be consistently estimated under weaker causal conditions than the ATE, thus strengthening the credibility of the conclusions. We elaborate on this in the following section.

Differing Causal assumptions of the ETT vs. the ATE: Ignorability

In this section, we focus on the “ignorability” (also termed “no unmeasured confounding”) assumption that can be relaxed to consistently estimate the ETT but not the ATE. We explain each empirically untestable causal assumption that must be met for either the ATE or the ETT causal estimand to be nonparametrically identified using the observed data.

Adverse experiences, such as discrimination, are not randomized in real-world settings such as the workplace. Therefore, noncausal correlations between discrimination and the outcome of interest can arise due to their mutual associations with confounding variables (Imbens & Rubin, 2015; Morgan & Winship, 2015; Shadish et al., 2002). We will assume that a set of precisely measured baseline covariates $C$ exists that suffices to ensure that discrimination and the potential outcomes are conditionally independent if discrimination does not affect the outcome. Formally, this assumption is termed strong ignorability (Greenland & Robins, 2009) and can be expressed as:

(Y^{1}, Y^{0}) ⫫ X | C .

(4)

This is a sufficient assumption for the ATE to be identified; see, for example, Morgan & Winship (2015: Section 4.3.3).

However, strong ignorability is likely to be violated in practice. Past research shows that individuals aware of the adverse impacts of discrimination seek to avoid and lower their chances of experiencing discrimination in organizations (Bruton et al., 2023). This induces an association between their potential outcome had they experienced discrimination ( $Y^{1}$ ) and their aversion to experiencing discrimination ( $X$ ). The resulting correlation violates strong ignorability in (4) and precludes consistent estimation of the ATE.

Instead, consider the following causal assumption that requires only $Y^{0}$ , the potential outcome in the absence of experienced discrimination, to be conditionally independent of $X$ given $C$ :

Y^{0} ⫫ X | C .

(5)

This assumption in (5) is termed weak ignorability (Greenland & Robins, 2009; Imbens, 2000), because strong ignorability in (4) implies weak ignorability in (5) but not vice versa. Crucially, weak ignorability suffices for the ETT to be identified. We will demonstrate empirically in a simulation study how strong ignorability in (4) is violated so that estimates of the ATE are biased, but weak ignorability in (5) is satisfied so that estimates of the ETT are unbiased. In brief, the ETT is more robust to likely violations of stringent causal—empirically untestable—assumptions than the ATE, making the ETT a more appealing causal estimand for investigating the experiences of vulnerable workers.

How to Estimate the ETT

In this section, we describe the steps to estimate the ETT. Because the estimation procedures are broadly applicable to various (adverse) exposures, we present this material with minimal contexts of research questions. Nonetheless, for illustrative purposes, we will continue to use the exposure of discrimination as a running example.

When assumption (5) can be substantively justified, we can consistently estimate the ETT using the observable data as:

ETT = \sum_{c} {E (Y | X = 1, C = c) - E (Y | X = 0, C = c)} P (C = c | X = 1) .

(6)

For simplicity, we abbreviate the notation

\sum_{c \in C}

, where

C

is the support of

C

, simply as

\sum_{c}

. Proof of this result is provided in SOM section A.⁴ To simplify the presentation of the estimation methods in the sequel, we use a single (scalar) covariate

C

.⁵ However, in practice, researchers should collate a comprehensive compilation of covariates to more defensibly justify the weak ignorability assumption in (5); we return to this in the applied examples and the discussion in later sections.

Why a Linear Regression-based Estimator is Undesirable

Observe from the expression in (6) that if $Y$ truly depends linearly on $X$ and $C$ , especially with only a main effect for $X$ , then the coefficient of $X$ parametrizes the ETT. To see why, suppose that the observed data were generated according to the following true model:

E (Y | X, C) = β_{0} + β_{x} X + β_{c} C .

(7)

Then

E (Y | X = 1, C = c) - E (Y | X = 0, C = c) = β_{x}

for all values of

C = c

, and plugging this into (6) yields

β_{x}

as an estimator of the ETT.

However, when the observed data are generated from a process that differs from the assumed linear model for the conditional expected outcome above, using an incorrectly specified linear model can lead to biased estimates of the ETT.⁶ This vulnerability motivates using estimators that avoid relying solely on a correctly assumed regression model for $Y$ such as (7). In the next section, we describe two different estimators protected from biases even when the assumed regression model for $Y$ is incorrect. We then empirically compare both estimators with several other estimators (each relying on a single statistical model) in the simulation studies in a later section.

Doubly Robust Estimator 1: Augmented Inverse Probability Weighting

We first introduce an estimator recently proposed in the statistics literature by Moodie et al. (2018) and consists of the following steps:

A1.

Fit to the observed data a logistic regression model for $X$ given the baseline covariates in $C$ . The propensity score is the conditional probability that an individual has an observed value of $X = 1$ given $C$ (Rosenbaum & Rubin, 1983). A simple example of a propensity score model with only main effects for $C$ is:

logit {Pr (X = 1 | C)} = α_{0} + α_{1} C .

(8)

The model may be fitted using, for example, maximum likelihood estimation.

A2.

Obtain the predicted propensity score for each individual, denoted by $P$ , using the fitted model in (8); for example, by plugging in coefficient estimates of $α_{0}$ and $α_{1}$ .

A3.

Construct the inverse propensity score weights, or inverse probability of treatment weights (IPWs; Reifeis & Hudgens (2022)) for each individual as:

W = {\begin{matrix} 1, & X = 1; \\ \frac{P}{1 - P}, & X = 0. \end{matrix}

Among those who did not experience discrimination (

X = 0

W

is the predicted odds of an individual experiencing discrimination, conditional on the baseline covariates. We explain the rationale behind using IPW later in this section.

A4.

Among the subgroup who did not experience discrimination ( $X = 0$ ), fit a linear regression model with $Y$ as the response or dependent variable, and $C$ as predictors or independent variables. For example, a linear regression model with main and quadratic effects for the predictors is:

E (Y | C, X = 0) = β_{0} + β_{1} C + β_{2} C^{2} .

(9)

We included a quadratic term for

C

as an expository example of allowing a nonlinear relationship between the outcome

Y

and covariate

C

. In a later section, we explain the motivation for including nonlinear functions of the covariates in the outcome model. Denote the resulting estimated model using, for example, ordinary least squares (OLS), by

\hat{E} (Y | C, X = 0)

. For example, if

({\hat{β}}_{0}, {\hat{β}}_{1}, {\hat{β}}_{2})

are the OLS estimates of the coefficients in (9), then

\hat{E} (Y | C, X = 0)

is obtained by plugging in the estimates for the unknown coefficients.

A5.

Calculate for each individual the difference $D = Y - \hat{E} (Y | C, X = 0)$ . Hence, among those who experienced discrimination ( $X = 1$ ), the prediction $\hat{E} (Y | C, X = 0)$ can be considered an imputed potential outcome had there not been discrimination ( $Y^{0}$ ). The estimator of the ETT is then the average over all individuals:

{\hat{ETT}}_{DR} = {(\sum_{i} X_{i})}^{- 1} \sum_{i} {X_{i} - (1 - X_{i})} W_{i} D_{i} .

(10)

In this paper, an

i

subscript always indexes an individual from

1

N

, where

N

is the sample size and the notation for the sum

\sum_{i = 1}^{N}

is written simply as

\sum_{i}

for brevity.

Nonparametric percentile bootstrap confidence intervals (CIs) (Efron & Tibshirani, 1994) may be constructed by randomly resampling observations with replacement and repeating all the above steps for each bootstrap sample.

The estimator ${\hat{ETT}}_{DR}$ utilizes a propensity score model (8), and the outcome regression model (9). Doing so confers the estimator with doubly robustness (hence the “DR” subscript), whereby the estimator is protected from biases due to incorrectly specifying one model when the other is correctly specified, without requiring knowing which is correct (Glynn & Quinn, 2010; Robins et al., 2007; Schafer & Kang, 2008). We empirically demonstrate the reduction in biases when either the propensity score or outcome regression model is incorrectly specified using a simulation study in a later section. Details on how to construct the estimator in (10), and why it is doubly robust, are shown in SOM section C.

We briefly explain the motivation behind using IPW. IPW eliminates measured confounding by creating a pseudo-population with $W$ copies of each individual from the observed sample. This pseudo-population comprises individuals from the observed $X = 1$ and $X = 0$ groups, but in proportions that differ from the observed sample. For example, an individual in the $X = 1$ group has a weight of $W = 1$ ; they contribute one copy of themself to this pseudo-population. As another example, suppose that a particular individual in the $X = 0$ group has a predicted propensity score of $P = 0.75$ . Then, this individual has a weight of $W = 3$ ; they contribute three copies of themselves to this pseudo-population. This pseudo-population has the crucial property that the covariate distributions are (approximately) similar, or “balanced,” across groups with different levels of the pre-treatment covariates $C$ (West et al., 2014). Therefore, when weak ignorability holds, and the propensity score model is correctly specified, the confounders are not associated with discrimination, and the effect of discrimination is thus not subject to measured confounding.⁷

Doubly Robust Estimator 2: Weighted Least Squares

The estimator introduced above is not the only estimator endowed with double robustness. In the social sciences literature, Morgan & Winship (2015: Section 7.3) describe a weighted regression estimator of the ETT that can be obtained by carrying out the following steps: B1.

Carry out steps A1 - A3 as described above.

B2.

Fit a weighted linear regression model to the entire observed sample, with (i) $Y$ as the response or dependent variable; (ii) $X$ and $C$ as predictors or independent variables; and (iii) $W$ as weights. For example, a linear regression model with main effects for the predictors and quadratic terms for the covariates is:

E (Y | C, X) = β_{0} + β_{1} X + β_{2} C + β_{3} C^{2} .

(11)

The weighted least squares (WLS) estimator of the coefficient

β_{1}

in (11) is an estimator of the ETT, which we denote by

{\hat{ETT}}_{WLS}

Therefore, the difference between the estimator

{\hat{ETT}}_{DR}

in (10) and

{\hat{ETT}}_{WLS}

using (11) centers on the fitted regression models for the outcome in (9) and (11), respectively. The former fits an outcome model to only those who did not experience discrimination (

X = 0

), thereby permitting different covariate-outcome relations specific to this subgroup. This more parsimonious approach allows for interactions between

X

and

C

on the outcome by construction and can be advantageous when estimating conditional effects, as we explain in SOM section D. In contrast, the latter fits an outcome model to the entire sample and parametrizes the effect of discrimination as a single model coefficient (

β_{1}

). Misspecification biases can potentially arise because any heterogeneity of the treatment effect is implicitly averaged over the distribution of the covariates. In principle, effect heterogeneity can be explicitly parameterized using a flexible regression model with meaningful interactions between

X

and (higher-order terms of)

C

. But in practice, conceiving such a model with multiple variables in

C

can be complicated or unfeasible due to the curse of dimensionality. Moreover, when there are continuous covariates in

C

so that there are categorical by continuous interactions in the outcome model, the continuous covariates should be mean-centered, and the categorical predictors (e.g.,

X

) effect-coded, to improve interpretations of the first-order effects (West et al., 1996). Finally, we describe in SOM section D how the ETT causal estimand can be readily used to investigate putative heterogeneity of the impact of discrimination across different pre-defined subgroups.

Simulation Studies

We conducted a series of Monte Carlo simulation studies to empirically probe the operating characteristics of the doubly robust ETT estimators introduced above, alongside several other ETT estimators which are not doubly robust. We present all details in SOM section E; here, we briefly state the points empirically demonstrated in the simulation studies. First, the ATE required more stringent—empirically untestable—causal conditions for unbiased estimation than the ETT. Second, the doubly robust estimators presented above were protected from biases when either the propensity score or outcome model was incorrectly specified. In contrast, other estimators that relied on only a single model (either the propensity score model or the outcome model) were prone to biases when that assumed model was incorrect. All the estimators yielded biases of similar magnitude when both models were incorrectly specified.

Illustrations using Real-world Datasets

In this section, we illustrated the introduced estimators of the ETT by analyzing two publicly available real-world datasets. To guide researchers on implementing the ETT in practice, we provided sample analysis code in R (R Core Team, 2023) for estimating the ETT in both examples.

Example 1: Effect of Unfair Supervision on Work Stress

Between December 2003 and September 2004, researchers from the University of Kansas surveyed working professionals as part of the Professional Worker Career Experience Survey (Rosenbloom & Ash, 2010). Data collected as part of the survey included measures of work-family conflict, job satisfaction, life satisfaction, and work stress. Further details of the study are provided by Rosenbloom & Ash (2010).

For the purpose of illustration, we considered whether experiencing unfair supervision influenced respondents’ overall work stress. Exposure to unfair supervision ( $X$ ) was measured using one item. Participants indicated whether they disagreed with the statement, “My supervisor is unfair to me.” 57.4% of the sample strongly disagreed $(X = 0$ ), and the remaining 42.6% of the sample reported otherwise $(X = 1)$ . The outcome of interest ( $Y$ ) was the mean response to six items measuring their work stress: (i) “I feel frustrated with my work;” (ii) “I feel discouraged about my work;” (iii) “I feel that things are out of control;” (iv) “I feel overwhelmed by my work;” (v) “I feel like giving up on my job;” and (vi) “I feel unable to get out from under my work.” Each item rated how much they agreed or disagreed with the statement on a six-point scale (1 = strongly disagree to 6 = strongly agree). Higher mean scores indicated higher overall work stress. To simplify the analysis for expository purposes, we assumed that the self-reported variables Gender, Current career field, Age, Education level, and Time in current position, collectively denoted by $C$ , sufficed for weak ignorability in (5) to hold. We considered the 537 participants with complete data recorded on the variables $(Y, X, C)$ used for the analysis.

We estimated the effect of unfair supervision on work stress among those who had such an experience $(X = 1)$ . We used both doubly robust estimators, fitting a propensity score model with main effects for $C$ , same as (8), to calculate the IPW. For the ${\hat{ETT}}_{DR}$ estimator in (10), we assumed a linear regression model for the outcome with all main effects for $C$ , and purely for illustration, a two-way interaction between Gender and Current career field and a quadratic term for Age; this was fitted to the subset who did not experience unfair supervision ( $X = 0$ ). The estimated ETT was $0.88$ (95% CI $= (0.68, 1.09)$ ). For the ${\hat{ETT}}_{WLS}$ estimator in (11), we assumed a linear regression model for the outcome with a main effect for $X$ and the same functions of $C$ as in the previous outcome model. The estimated ETT was $0.89$ (95% CI $= (0.69, 1.09)$ ). These results showed that among those who experienced unfair supervision, unfair supervision (vs. the absence of) increased work stress by about $0.89$ on a six-point scale. Put differently, among workers who received unfair supervision, their expected stress levels would be reduced by $0.89$ (on a six-point scale) had unfair supervision been addressed so they had received fair treatment instead.

Box 1

[Sample analysis code to estimate the ETT in Example 1]

How do we estimate the ETT? Here, we provide sample analysis code in R (R Core Team, 2023) for calculating the doubly robust (steps A1–A5) and weighted least squares (steps B1–B2) estimators. For illustration, we considered five baseline covariates: Gender (“GENDER”), Current career field (“Q4ITNON”), Age (“AGE”), Education level (“DEMO4”), and Time in current position (“Q2”). In practice, researchers should use substantive knowledge to procure a richer set of covariates toward justifying the weak ignorability assumption in (5).

We first carried out steps A1–A3 to fit a propensity score model for experiencing unfair supervision (“SUPFAIR”), obtain the predicted propensity scores, and allocate the IPW to each individual. We then carried out step A4 to fit an outcome regression model for work stress (“WRKSTRES”) among only the subset who did not have an unfair supervisor. Finally, we use these fitted models to carry out step A5. To help researchers implement this in practice, we have developed an R function EstimateETT to calculate the ${\hat{ETT}}_{DR}$ estimator in (10): The weighted least squares estimator ${\hat{ETT}}_{WLS}$ estimator in (11) can be calculated as: Nonparametric percentile bootstrap confidence intervals can be readily obtained by randomly sampling observations with replacement and then repeating all the above steps.

Example 2: Effect of Discrimination on Civic Engagement

In 2011, the Stanford University Center on Adolescence initiated a longitudinal study of civic purpose development in adolescence (Damon, 2017). A sample of high school seniors in California completed a survey in November 2011 (time 1). They completed a follow-up survey 21 months later (time 2) after transitioning out of high school. Survey and interview questions covered topics such as civic engagement, perceptions of America, and experiences with discrimination. Further details of the study are provided by Damon (2017).

For the purpose of illustration, we followed the analyses in Ballard (2016) and considered how the frequency of discriminatory experiences at time 1 affected participants’ subsequent fair society beliefs at time 2. Discrimination frequency ( $X$ ) was measured using one item. Participants indicated how often they felt personally discriminated against for any reason. 49.7% of the sample reported experiencing discrimination sometimes or often $(X = 1$ ), and the rest reported that they had not experienced discrimination or experienced it rarely $(X = 0)$ . The outcome of interest ( $Y$ ) was the average response to three items measuring how fair they thought America was: (i) “Basically, people get fair treatment in America, no matter who they are;” (ii) “In America, you have an equal chance no matter where you come from or what race you are;” (iii) “America is a fair society where everyone has an equal chance to get ahead.” Each item rated how much they agreed or disagreed with the statement on a five-point scale (1 = strongly disagree to 5 = strongly agree). To simplify the analysis for expository purposes, we assumed that the self-reported variables Gender and Ethnicity, collectively denoted by $C$ , sufficed for weak ignorability in (5) to hold. We considered the 449 participants with complete data recorded on the variables $(Y, X, C)$ used for the analysis.

We estimated the effect of discrimination on fair society beliefs among those who frequently experienced discrimination $(X = 1)$ . We used both doubly robust estimators, fitting a propensity score model with main effects for $C$ , same as (8), to calculate the IPW. For the ${\hat{ETT}}_{DR}$ estimator in (10), we assumed a linear regression model for the outcome with all main and two-way interaction effects for $C$ ; this was fitted to the subset who reported infrequently experiencing discrimination ( $X = 0$ ). The estimated ETT was $- 0.34$ (95% CI $= (- 0.52, - 0.16)$ ). For the ${\hat{ETT}}_{WLS}$ estimator in (11), we assumed a linear regression model for the outcome with a main effect for $X$ and all main and two-way interaction effects for $C$ . The estimated ETT was $- 0.34$ (95% CI $= (- 0.52, - 0.16)$ ). These results showed that among those who frequently experienced discrimination in adolescence, experiencing discrimination (vs. not) undermined fairness society beliefs by $0.34$ on a five-point scale. Put differently, among those discriminated against, their expected fairness society beliefs would be higher by $0.34$ (on a five-point scale), had they not experienced discrimination instead.

Box 2

[Sample analysis code to estimate the ETT in Example 2]

How do we estimate the ETT? Here, we provide sample analysis code in R (R Core Team, 2023) for calculating the doubly robust (steps A1–A5) and weighted least squares (steps B1–B2) estimators. For illustration, we considered two baseline covariates: Gender (“GENDERT1”) and Ethnicity (“ETH_MCT1”). In practice, researchers should use substantive knowledge to procure a richer set of covariates toward justifying the weak ignorability assumption in (5).

We first carried out steps A1–A3 to fit a propensity score model for discrimination (“DISCRIMT1”), obtain the predicted propensity scores, and allocate the IPW to each individual. We then carried out step A4 to fit an outcome regression model for fairness society beliefs (“FAIR_SB_T2”) among only the subset who did not experience discrimination. Finally, we use these fitted models to carry out step A5. To help researchers implement this in practice, we have developed an R function EstimateETT to calculate the ${\hat{ETT}}_{DR}$ estimator in (10): The weighted least squares estimator ${\hat{ETT}}_{WLS}$ estimator in (11) can be calculated as: Nonparametric percentile bootstrap confidence intervals can be readily obtained by randomly sampling observations with replacement and then repeating all the above steps.

Discussion

In this paper, we introduced the ETT causal estimand as an alternative to the ATE for organizational scholars. It is important to note that we are not suggesting the ETT is uniformly superior to the ATE without context. Instead, we recommend scholars carefully choose a target causal estimand that can best answer their research questions. As we demonstrated in the preceding sections, the ETT is uniquely suited to address causal queries pertinent to the unique experiences of vulnerable populations.

An avenue for future extensions of the estimators presented in this paper regards missing data. To simplify the illustrations, in both example analyses, we included only observations with complete data recorded on the variables $(Y, X, C)$ . This was to ease estimating the propensity score and outcome models using maximum likelihood and OLS, respectively. However, such complete case analyses exclude observations with partially recorded data, which are rife in practical applications. Therefore, a valuable direction for future work is to utilize missing data methods that consider all available information in an incomplete data set after establishing that missingness is at random (Enders, 2022). In the context of this paper, procedures focusing on either propensity score-based estimators (see reviews of different methods by Cham & West (2016), and Choi et al. (2019)) or outcome regression-based estimators (see, for example, Vansteelandt et al. (2010)) may be extended to accommodate missing data when fitting the propensity score or outcome model to construct ETT estimators.⁸ However, separate assumptions must be maintained, and separate imputation model(s) for the missing data mechanism must be specified (Vansteelandt et al., 2010; Zhang et al., 2013). Researchers must thus strive to ensure the distinct models are compatible (or “congenial”; Meng (1994)) with one another to avoid model misspecification biases. Alternatively, an approach that may be simpler to implement is to run a sensitivity analysis to gauge the extent of missing data substantively affecting the effect estimates (Mathur, 2023).

As with all causal inferences of nonrandomized treatments, the ignorability assumption must be met for consistent effect estimation (Imbens & Rubin, 2015; Morgan & Winship, 2015; Shadish et al., 2002). We emphasize that the weak ignorability assumption for the ETT is less stringent than the analogous strong ignorability assumption routinely made (implicitly) when targeting the ATE. Hence, the reduced reliance on empirically untestable assumptions makes the ETT a more appealing estimand than the ATE in practice. Nonetheless, this assumption must be rigorously substantiated using subject matter and domain-specific expertise (Steiner et al., 2010). As a practical recommendation, researchers should prioritize adjusting for covariates strongly (or even only) predictive of the outcome, and avoid nonconfounders associated with treatment only (i.e., unassociated with the outcome) which can increase finite-sample variability (Loh & Ren, 2023; VanderWeele, 2019). While adjusting for observed confounders can reduce the impact of any unmeasured confounders associated with those observed (Stuart, 2010), there remains the possibility of bias due to unmeasured confounding. Therefore, researchers should also consider a sensitivity analysis to judge the impact of hypothetical unmeasured confounding on the conclusions (Hong et al., 2021).

In practice, exactly one of the potential outcomes ( $Y^{1}$ or $Y^{0}$ ) can be observed for each individual. This is commonly termed the “fundamental problem of causal inference” (Holland, 1986). Hence, another assumption that must be met is causal, or counterfactual, consistency (VanderWeele, 2009), which can be formally stated as:

Y = Y^{1} X + Y^{0} (1 - X) .

(12)

This assumption may be violated when so-called “interference” (Hudgens & Halloran, 2008) is present. Using the running example of discrimination as exposure, interference can arise when an individual’s experience of discrimination affects others’ outcomes, such as when the experiences of discrimination are shared. Such peer influence or “spillover” effects are likely to occur when individuals are causally connected in social networks; see, for example, An & VanderWeele (2019); Aronow et al. (2021); Athey et al. (2018); Loh & Ren (2022).

Conclusion

[W]e implore researchers to conduct more studies on understudied, vulnerable populations. Otherwise, organizational research will continue to prioritize the well-understood, safe, and privileged subset of employees. This approach will lead to the formation of general theories based on these insights that are inaccurately applied to all employees, particularly the vulnerable workforce. (Restubog et al., 2023: p 2206)

We included this quote here not just to emphasize its compelling arguments on the importance of studying the work experiences of vulnerable employees, but also to highlight a discrepancy in organizational theories scholars are actively seeking to address: Knowledge obtained from the privileged subset of employees cannot be applied to the vulnerable workforce. In the current article, we have sought to draw scholars’ attention to a similar discrepancy in the methodological realm: Causal quantities such as the ATE are a coarse assessment for the entire study population that cannot be accurately applied to all individuals, especially the subset exposed to adverse experiences. To address this discrepancy, we put forth the ETT as a causal estimand better suited for studying the experiences of the vulnerable workforce. The ETT offers a more intuitive interpretation, enables a more accurate evaluation, and admits a more robust estimator. We hope that our introduction of the ETT to organizational research can contribute to a more accurate and inclusive organizational science toward an equitable society.

Supplemental Material

sj-pdf-1-orm-10.1177_10944281241246772 - Supplemental material for Enhancing Causal Pursuits in Organizational Science: Targeting the Effect of Treatment on the Treated in Research on Vulnerable Populations

Supplemental material, sj-pdf-1-orm-10.1177_10944281241246772 for Enhancing Causal Pursuits in Organizational Science: Targeting the Effect of Treatment on the Treated in Research on Vulnerable Populations by Wen Wei Loh and Dongning Ren in Organizational Research Methods

Footnotes

Acknowledgments

The authors thank the Associate Editor, Dr. Louis Tay, for his constructive suggestions and valuable advice in guiding the manuscript through the review process. We thank both anonymous reviewers for their insightful and helpful comments, which have greatly improved the manuscript. We are grateful to Dr. Cathryn Johnson and Dr. Karen Hegtvedt for engaging in enlightening discussions on mechanisms of organizational inequality that inspired the development of this paper.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship and/or publication of this article.

ORCID iDs

Wen Wei Loh

Dongning Ren

Supplemental material

Supplemental material for this article is available online on GitHub (https://github.com/wwloh/ett-org).

Notes

Author Biographies

Wen Wei Loh is a statistician and methodologist working on causal inference. He is enthusiastic about developing and applying statistical methods in collaboration with applied researchers in social and health sciences to improve health and equity.

Dongning Ren is a social and quantitative psychologist committed to promoting social justice. Leveraging psychological theories, modern causal inference techniques, and her lived experiences, she seeks to identify and address systemic barriers to diversity and inclusion.

References

American Psychological Association. (2021). Equity, diversity, and inclusion framework. https://www.apa.org/about/apa/equity-diversity-inclusion/framework.pdf

Amis

J. M.

Brickson

Haack

Hernandez

(2021). Taking inequality seriously. Academy of Management Review, 46(3), 431-439. https://doi.org/10.5465/amr.2021.0222

Amis

J. M.

Mair

Munir

K. A.

(2020). The organizational reproduction of inequality. Academy of Management Annals, 14(1), 195-230. https://doi.org/10.5465/annals.2017.0033

VanderWeele

T. J.

(2019). Opening the blackbox of treatment interference: Tracing treatment diffusion through network analysis. Sociological Methods & Research, 51(1), 141-164. https://doi.org/10.1177/0049124119852384

Arnold

B. F.

Khush

R. S.

Ramaswamy

London

A. G.

Rajkumar

Ramaprabha

Durairaj

Hubbard

A. E.

Balakrishnan

Colford

J. M.

(2010). Causal inference methods to study nonrandomized, preexisting development interventions. Proceedings of the National Academy of Sciences, 107(52), 22605-22610. https://doi.org/10.1073/pnas.1008944107

Aronow

P. M.

Eckles

Samii

Zonszein

. (2021). Spillover effects in experimental data. In Green

D. P.

Druckman

J. N.

(Eds.), Advances in experimental political science (pp. 289-319). Cambridge University Press. https://doi.org/10.1017/9781108777919.021

Athey

Eckles

Imbens

G. W.

(2018). Exact p-values for network interference. Journal of the American Statistical Association, 113(521), 230-240. https://doi.org/10.1080/01621459.2016.1241178

Auld

M. C.

(2005). Causal effect of early initiation on adolescent smoking patterns. Canadian Journal of Economics/Revue Canadienne D’économique, 38(3), 709-734. https://doi.org/10.1111/j.0008-4085.2005.00299.x

Avery

D. R.

Hall

A. V.

Preston

Ruggs

E. N.

Washington

(2023). Is justice colorblind? a review of workplace racioethnic differences through the lens of organizational justice. Annual Review of Organizational Psychology and Organizational Behavior, 10(1), 389-412. https://doi.org/10.1146/annurev-orgpsych-120920-052627

10.

Ballard

P. J.

(2016). Longitudinal links between discrimination and civic development among Latino and Asian adolescents. Journal of Research on Adolescence, 26(4), 723-737. https://doi.org/10.1111/jora.12221

11.

Bapuji

Ertug

Shaw

J. D.

(2020a). Organizations and societal economic inequality: A review and way forward. Academy of Management Annals, 14(1), 60-91. https://doi.org/10.5465/annals.2018.0029

12.

Bapuji

Patel

Ertug

Allen

D. G.

(2020b). Corona crisis and inequality: Why management research needs a societal turn. Journal of Management, 46(7), 1205-1222. https://doi.org/10.1177/0149206320925881

13.

Bowles

H. R.

Thomason

Macias-Alonso

(2022). When gender matters in organizational negotiations. Annual Review of Organizational Psychology and Organizational Behavior, 9(1), 199-223. https://doi.org/10.1146/annurev-orgpsych-012420-055523

14.

Brands

Ertug

Fonti

Tasselli

(2022). Theorizing gender in social network research: What we do and what we can do differently. Academy of Management Annals, 16(2), 588-620. https://doi.org/10.5465/annals.2020.0370

15.

Bruton

G. D.

Lewis

Cerecedo-Lopez

J. A.

Chapman

(2023). A racialized view of entrepreneurship: A review and proposal for future research. Academy of Management Annals, 17(2), 492-515. https://doi.org/10.5465/annals.2021.0185

16.

Cham

West

S. G.

(2016). Propensity score analysis with missing data. Psychological Methods, 21(3), 427-445. https://doi.org/10.1037/met0000076

17.

Choi

Dekkers

O. M.

le Cessie

(2019). A comparison of different methods to handle missing data in the context of propensity score analysis. European Journal of Epidemiology, 34(1), 23-36. https://doi.org/10.1007/s10654-018-0447-z

18.

Colella

Hebl

King

(2017). One hundred years of discrimination research in the journal of applied psychology: A sobering synopsis. Journal of Applied Psychology, 102(3), 500-513. https://doi.org/10.1037/apl0000084

19.

Cornelissen

Kaandorp

(2023). Towards stronger causal claims in management research: Causal triangulation instead of causal identification. Journal of Management Studies, 60(4), 834-860. https://doi.org/10.1111/joms.12897

20.

Cortina

L. M.

Areguin

M. A.

(2021). Putting people down and pushing them out: Sexual harassment in the workplace. Annual Review of Organizational Psychology and Organizational Behavior, 8(1), 285-309. https://doi.org/10.1146/annurev-orgpsych-012420-055606

21.

Damon

. (2017). Stanford civic purpose project: Longitudinal study of youth civic engagement in California, 2011-2013. https://doi.org/10.3886/ICPSR36561.v1

22.

Efron

Tibshirani

R. J.

(1994). An introduction to the bootstrap. Chapman & Hall/CRC.

23.

Eisenberger

Rhoades Shanock

Wen

(2020). Perceived organizational support: Why caring about employees counts. Annual Review of Organizational Psychology and Organizational Behavior, 7(1), 101-124. https://doi.org/10.1146/annurev-orgpsych-012119-044917

24.

Enders

C. K.

(2022). Applied missing data analysis (2nd ed.). The Guilford Press.

25.

Ezerins

M. E.

Simon

L. S.

Vogus

T. J.

Gabriel

A. S.

Calderwood

Rosen

C. C.

(2023). Autism and employment: A review of the “new frontier” of diversity research. Journal of Management, 50(3), 1102-1144. https://doi.org/10.1177/01492063231193362

26.

Falk

Lalive

Zweimüller

(2005). The success of job applications: A new approach to program evaluation. Labour Economics, 12(6), 739-748. https://doi.org/10.1016/j.labeco.2004.05.002

27.

Gloor

J. L.

Okimoto

T. G.

Gazdag

B. A.

Ryan

M. K.

(2023). How identity impacts bystander responses to workplace mistreatment. Journal of Management. https://doi.org/10.1177/01492063231177976

28.

Glynn

A. N.

Quinn

K. M.

(2010). An introduction to the augmented inverse propensity weighted estimator. Political Analysis, 18(1), 36-56. https://doi.org/10.1093/pan/mpp036

29.

Greenland

Robins

J. M.

(2009). Identifiability, exchangeability and confounding revisited. Epidemiologic Perspectives & Innovations, 6(1), 4. https://doi.org/10.1186/1742-5573-6-4

30.

Hackney

K. J.

Daniels

S. R.

Paustian-Underdahl

S. C.

Perrewé

P. L.

Mandeville

Eaton

A. A

(2021). Examining the effects of perceived pregnancy discrimination on mother and baby health. Journal of Applied Psychology, 106(5), 774-783. https://doi.org/10.1037/apl0000788

31.

Hebl

Cheng

S. K.

L. C.

(2020). Modern discrimination in organizations. Annual Review of Organizational Psychology and Organizational Behavior, 7(1), 257-282. https://doi.org/10.1146/annurev-orgpsych-012119-044948

32.

Heckman

J. J.

Lalonde

R. J.

Smith

J. A.

(1999). Chapter 31 - the economics and econometrics of active labor market programs. In Ashenfelter

O. C.

Card

(Eds.), Handbook of labor economics (pp. 1865-2097). Elsevier. https://doi.org/10.1016/S1573-4463(99)03012-6

33.

Heckman

J. J.

Robb

(1985). Alternative methods for evaluating the impact of interventions. In Heckman

J. J.

Singer

B. S.

(Eds.), Longitudinal analysis of labor market data (pp. 156-246). Cambridge University Press. https://doi.org/10.1017/CCOL0521304539.004

34.

Hernán

M. A.

(2015). Invited commentary: Agent-based models for causal inference-reweighting data and theory in epidemiology. American Journal of Epidemiology, 181(2), 103-105. https://doi.org/10.1093/aje/kwu272

35.

Holland

P. W

. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81 (396), 945-960.

36.

Hong

Yang

Qin

(2021). Did you conduct a sensitivity analysis? A new weighting-based approach for evaluations of the average treatment effect for the treated. Journal of the Royal Statistical Society: Series A (Statistics in Society), 184(1), 227-254. https://doi.org/10.1111/rssa.12621

37.

Hudgens

M. G.

Halloran

M. E.

(2008). Toward causal inference with interference. Journal of the American Statistical Association, 103(482), 832-842. https://doi.org/10.1198/016214508000000292

38.

Imbens

G. W.

(2000). The role of the propensity score in estimating dose-response functions. Biometrika, 87(3), 706-710. https://doi.org/10.1093/biomet/87.3.706

39.

Imbens

G. W.

Rubin

D. B.

(2015). Causal inference in statistics, social, and biomedical sciences. Cambridge University Press.

40.

Jun

Phillips

L. T.

Foster-Gimbel

O. A.

(2023). The missing middle: Asian employees’ experience of workplace discrimination and pro-Black allyship. Journal of Applied Psychology, 108(2), 225-248. https://doi.org/10.1037/apl0001068

41.

Kelloway

E. K.

Dimoff

J. K.

Gilbert

(2023). Mental health in the workplace. Annual Review of Organizational Psychology and Organizational Behavior, 10(1), 363-387. https://doi.org/10.1146/annurev-orgpsych-120920-050527

42.

King

D. D.

Fattoracci

E. S. M.

Hollingsworth

D. W.

Stahr

Nelson

(2023). When thriving requires effortful surviving: Delineating manifestations and resource expenditure outcomes of microaggressions for black employees. Journal of Applied Psychology, 108(2), 183-207. https://doi.org/10.1037/apl0001016

43.

Kreiner

Mihelcic

C. A.

Mikolon

(2022). Stigmatized work and stigmatized workers. Annual Review of Organizational Psychology and Organizational Behavior, 9(1), 95-120. https://doi.org/10.1146/annurev-orgpsych-012420-091423

44.

Lambert

Caza

B. B.

Trinh

Ashford

(2022). Individual-centered interventions: Identifying what, how, and why interventions work in organizational contexts. Academy of Management Annals, 16(2), 508-546. https://doi.org/10.5465/annals.2020.0351

45.

Lebel

R. D.

Yang

Parker

S. K.

Kamran-Morley

(2023). What makes you proactive can burn you out: The downside of proactive skill building motivated by financial precarity and fear. Journal of Applied Psychology, 108(7), 1207-1222. https://doi.org/10.1037/apl0001063

46.

Liao

Feng

Zhu

Guan

O. Z.

(2023). The award goes to…someone else: A natural quasi-experiment examining the impact of performance awards on nominees’ workplace collaboration. Academy of Management Journal, 66(5), 1303-1333. https://doi.org/10.5465/amj.2021.0662

47.

Little

L. M.

Masterson

C. R.

(2023). Mother’s reentry: A relative contribution perspective of dual-earner parents’ roles, resources, and outcomes. Academy of Management Journal, In press. https://doi.org/10.5465/amj.2019.1344

48.

Loh

W. W.

Ren

(2022). Estimating social influence in a social network using potential outcomes. Psychological Methods, 27(5), 841-855. https://doi.org/10.1037/met0000356

49.

Loh

W. W.

Ren

(2023). Data-driven covariate selection for confounding adjustment by focusing on the stability of the effect estimator. Psychological Methods, No Pagination Specified–No Pagination Specified. https://doi.org/10.1037/met0000564

50.

Mah

Kolev

K. D.

McNamara

Pan

Devers

C. E.

(2023). Women in the c-suite: A review of predictors, experiences, and outcomes. Academy of Management Annals, 17(2), 586-625. https://doi.org/10.5465/annals.2021.0042

51.

Mathur

M. B.

(2023). The m-value: A simple sensitivity analysis for bias due to missing data in treatment effect estimates. American Journal of Epidemiology, 192(4), 612-620. https://doi.org/10.1093/aje/kwac207

52.

Meng

X.-L.

(1994). Multiple-imputation inferences with uncongenial sources of input. Statistical Science, 9(4), 538-558. https://doi.org/10.1214/ss/1177010269

53.

Moodie

E. E. M.

Saarela

Stephens

D. A.

(2018). A doubly robust weighting estimator of the average treatment effect on the treated. Stat, 7(1), e205. https://doi.org/10.1002/sta4.205

54.

Morgan

S. L.

Winship

(2015). Counterfactuals and causal inference. Cambridge University Press.

55.

National Public Radio, Robert Wood Johnson Foundation, & Harvard T.H. Chan School of Public Health. (2017). Discrimination in America: Experiences and views on effects of discrimination across major population groups in the United States [January 26 – April 9]. https://www.rwjf.org/en/insights/our-research/2017/10/discrimination-inamerica–experiences-and-views.html

56.

Ocampo

A. C. G.

Chen

Restubog

S. L. D.

Wang

Decoste

(2023). A cross-lagged longitudinal investigation of the relationship between stigma and job effectiveness among employees with HIV. Journal of Applied Psychology, 108(6), 889-904. https://doi.org/10.1037/apl0001051

57.

Paulsen

McCormick

A. C.

(2020). Reassessing disparities in online learner student engagement in higher education. Educational Researcher, 49(1), 20-29. https://doi.org/10.3102/0013189X19898690

58.

Phillips

L. T.

Jun

Shakeri

(2022). Barriers and boosts: Using inequity frames theory to expand understanding of mechanisms of race and gender inequity. Academy of Management Annals, 16(2), 547-587. https://doi.org/10.5465/annals.2020.0314

59.

Pincus

F. L.

(1996). Discrimination comes in many forms: Individual, institutional, and structural. American Behavioral Scientist, 40(2), 186-194. https://doi.org/10.1177/0002764296040002009

60.

Prasad

(2023). The model minority and the limits of workplace inclusion. Academy of Management Review, 48(2), 336-356. https://doi.org/10.5465/amr.2021.0352

61.

Walter

Zhang

(2023). Unpacking the role of job insecurity for employee creativity: A multidimensional perspective. Journal of Management. https://doi.org/10.1177/01492063231213656

62.

R Core Team. (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. https://www.R-project.org/

63.

Reifeis

S. A.

Hudgens

M. G.

(2022). On variance of the treatment effect in the treated when estimated by inverse probability weighting. American Journal of Epidemiology, 191(6), 1092-1097. https://doi.org/10.1093/aje/kwac014

64.

Restubog

S. L. D.

Deen

C. M.

Decoste

(2021). From vocational scholars to social justice advocates: Challenges and opportunities for vocational psychology research on the vulnerable workforce. Journal of Vocational Behavior, 126, 103561. https://doi.org/10.1016/j.jvb.2021.103561

65.

Restubog

S. L. D.

Schilpzand

Lyons

Midel Deen

(2023). The vulnerable workforce: A call for research. Journal of Management, 49(7), 2199-2207. https://doi.org/10.1177/01492063231177446

66.

Robins

J. M.

Sued

Lei-Gomez

Rotnitzky

(2007). Comment: Performance of double-robust estimators when “inverse probability” weights are highly variable. Statistical Science, 22(4), 544-559. https://doi.org/10.1214/07-STS227D

67.

Rosado-Solomon

E. H.

Koopmann

Lee

Cronin

M. A.

(2023). Mental health and mental illness in organizations: A review, comparison, and extension. Academy of Management Annals, 17(2), 751-797. https://doi.org/10.5465/annals.2021.0211

68.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55.

69.

Rosenbloom

J. L.

Ash

R. A.

(2010). Professional worker career experience survey, United States, 2003-2004. https://doi.org/10.3886/ICPSR26782.v1

70.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688-701. https://doi.org/10.1037/h0037350

71.

Rubin

D. B.

(1986). Comment: Which ifs have causal answers. Journal of the American Statistical Association, 81(396), 961-962. https://doi.org/10.2307/2289065

72.

Salmon

Chavez

R. J. F.

Murphy

(2023). New perspectives and critical insights from indigenous peoples’ research: A systematic review of indigenous management and organization literature. Academy of Management Annals, 17(2), 439-491. https://doi.org/10.5465/annals.2021.0132

73.

Sayre

G. M.

(2023). The costs of insecurity: Pay volatility and health outcomes. Journal of Applied Psychology, 108(7), 1223-1243. https://doi.org/10.1037/apl0001062

74.

Schafer

J. L.

Kang

(2008). Average causal effects from nonrandomized studies: A practical guide and simulated example. Psychological Methods, 13(4), 279-313. https://doi.org/10.1037/a0014268

75.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

76.

Shoss

M. K.

Schlotzhauer

A. E.

Carusone

(2023). Working hard or hardly working? an examination of job preservation responses to job insecurity. Journal of Management, 49(7), 2387-2414. https://doi.org/10.1177/01492063221107877

77.

Speach

M. E. P.

Badura

K. L.

Blum

T. C.

(2023). Everything is negotiable, but not for everyone: The role of disability in compensation. Journal of Applied Psychology, 108(4), 571-594. https://doi.org/10.1037/apl0001039

78.

Splawa-Neyman

Dabrowska

D. M.

Speed

T. P.

(1990). On the application of probability theory to agricultural experiments. essay on Principles. Section 9. Statistical Science, 5(4), 465-472. https://doi.org/10.1214/ss/1177012031

79.

Steiner

P. M.

Cook

T. D.

Shadish

W. R.

Clark

M. H.

(2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15(3), 250-267. https://doi.org/10.1037/a0018719

80.

Stephan

J. L.

Rosenbaum

J. E.

Person

A. E.

(2009). Stratification in college entry and completion. Social Science Research, 38(3), 572-593. https://doi.org/10.1016/j.ssresearch.2009.02.001

81.

Stuart

E. A.

(2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1-21. https://doi.org/10.1214/09-STS313

82.

Thatcher

S. M. B.

Hymer

C. B.

Arwine

R. P.

(2023). Pushing back against power: Using a multilevel power lens to understand intersectionality in the workplace. Academy of Management Annals, 17(2), 710-750. https://doi.org/10.5465/annals.2021.0210

83.

VanderWeele

T. J.

(2009). Concerning the consistency assumption in causal inference. Epidemiology, 20(6). 880-883. https://doi.org/10.1097/EDE.0b013e3181bd5638

84.

VanderWeele

T. J.

(2019). Principles of confounder selection. European Journal of Epidemiology, 34(3), 211-219. https://doi.org/10.1007/s10654-019-00494-6

85.

Vansteelandt

Carpenter

Kenward

M. G.

(2010). Analysis of incomplete data using inverse probability weighting and doubly robust estimators. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 6(1), 37-48. https://doi.org/10.1027/1614-2241/a000005

86.

Vogel

R. M.

Bolino

M. C.

(2020). Recurring nightmares and silver linings: Understanding how past abusive supervision may lead to posttraumatic stress and posttraumatic growth. Academy of Management Review, 45(3), 549-569. https://doi.org/10.5465/amr.2017.0350

87.

Webster

A. J.

(2022). Causal attribution fractions, and the attribution of smoking and BMI to the landscape of disease incidence in UK Biobank. Scientific Reports, 12(1), 19678. https://doi.org/10.1038/s41598-022-23877-4

88.

West

S. G.

Aiken

L. S.

Krull

J. L.

(1996). Experimental personality designs: Analyzing categorical by continuous variable interactions. Journal of Personality, 64(1), 1-48. https://doi.org/10.1111/j.1467-6494.1996.tb00813.x

89.

West

S. G.

Cham

Thoemmes

Renneberg

Schulze

Weiler

(2014). Propensity scores as a basis for equating groups: Basic principles and application in clinical treatment outcome research. Journal of Consulting and Clinical Psychology, 82(5), 906-919. https://doi.org/10.1037/a0036387

90.

Zang

Sobel

M. E.

Luo

(2023). The mobility effects hypothesis: Methods and applications. Social Science Research, 110, 102818. https://doi.org/10.1016/j.ssresearch.2022.102818

91.

Zhang

Wang

M. S.

Toubiana

Greenwood

(2021). Stigma beyond levels: Advancing research on stigmatization. Academy of Management Annals, 15(1), 188-222. https://doi.org/10.5465/annals.2019.0031

92.

Zhang

Liu

Zhang

Tang

Zhang

(2013). Causal inference with missing exposure information: Methods and applications to an obstetric study. Statistical Methods in Medical Research, 25(5), 2053-2066. https://doi.org/10.1177/0962280213513758

93.

Zhong

Lian

Hershcovis

M. S.

Robinson

S. L.

(2023). Mitigating or magnifying the harmful influence of workplace aggression: An integrative review. Academy of Management Annals, 17(2), 516-545. https://doi.org/10.5465/annals.2021.0144

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.25 MB