An Updated Guideline for Assessing Discriminant Validity

Abstract

Discriminant validity was originally presented as a set of empirical criteria that can be assessed from multitrait-multimethod (MTMM) matrices. Because datasets used by applied researchers rarely lend themselves to MTMM analysis, the need to assess discriminant validity in empirical research has led to the introduction of numerous techniques, some of which have been introduced in an ad hoc manner and without rigorous methodological support. We review various definitions of and techniques for assessing discriminant validity and provide a generalized definition of discriminant validity based on the correlation between two measures after measurement error has been considered. We then review techniques that have been proposed for discriminant validity assessment, demonstrating some problems and equivalencies of these techniques that have gone unnoticed by prior research. After conducting Monte Carlo simulations that compare the techniques, we present techniques called CI_CFA(sys) and ${\text{χ}}^2$ (sys) that applied researchers can use to assess discriminant validity.

Among various types of validity evidence, organizational researchers are often required to assess the discriminant validity of their measurements (e.g., J. P. Green et al., 2016). However, there are two problems. First, the current applied literature appears to use several different definitions for discriminant validity, making it difficult to determine which procedures are ideal for its assessment. Second, existing guidelines are far from the practices of organizational researchers. As originally presented, “more than one method must be employed in the [discriminant] validation process” (Campbell & Fiske, 1959, p. 81), and consequently, literature on discriminant validation has focused on techniques that require that multiple distinct measurement methods be used (Le et al., 2009; Woehr et al., 2012), which is rare in applied research. To fill this gap, various less-demanding techniques have been proposed, but few of these techniques have been thoroughly scrutinized.

We present a comprehensive analysis of discriminant validity assessment that focuses on the typical case of single-method and one-time measurements, updating or challenging some of the recent recommendations on discriminant validity (Henseler et al., 2015; J. A. Shaffer et al., 2016; Voorhees et al., 2016). We start by reviewing articles in leading organizational research journals and demonstrating that the concept of discriminant validity is understood in at least two different ways; consequently, empirical procedures vary widely. We then assess what researchers mean by discriminant validity and synthesize this meaning as a definition. After defining what discriminant validity means, we provide a detailed discussion of each of the techniques identified in our review. Finally, we compare the techniques in a comprehensive Monte Carlo simulation. The conclusion section is structured as a set of guidelines for applied researchers and presents two techniques. One is CI_CFA(sys), which is based on the confidence intervals (CIs) in confirmatory factor analysis (CFA), and the other is ${\text{χ}}^2\left(\text{sys}\right)$ , a technique based on model comparisons in CFA.

Current Practices of Discriminant Validity Assessment in Organizational Research

To understand what organizational researchers try to accomplish by assessing discriminant validity, we reviewed all articles published between 2013 and 2016 by the Academy of Management Journal (AMJ), the Journal of Applied Psychology (JAP), and Organizational Research Methods (ORM). We included only studies that directly collected data from respondents through multiple-item scales. A total of 97 out of 308 papers in AMJ, 291 out of 369 papers in JAP, and 5 out of 93 articles in ORM were included.

The term “discriminant validity” was typically used without a definition or a citation, giving the impression that there is a well-known and widely accepted definition of the term. However, the few empirical studies that defined the term revealed that it can be understood in two different ways: One group of researchers used discriminant validity as a property of a measure and considered a measure to have discriminant validity if it measured the construct that it was supposed to measure but not any other construct of interest (A in Figure 1). For these researchers, discriminant validity means that “two measures are tapping separate constructs” (R. Krause et al., 2014, p. 102) or that the measured “scores are not (or only weakly) associated with potential confounding factors” (De Vries et al., 2014, p. 1343). Another group of researchers used discriminant validity to refer to whether two constructs were empirically distinguishable (B in Figure 1). For this group of researchers, the term referred to “whether the two variables…are distinct from each other” (Hu & Liden, 2015, p. 1110).

Figure 1.

Two definitions of discriminant validity, as shown in the AMJ and JAP articles. (A) Items measure more than one construct (i.e., cross-loadings). (B) Constructs are not empirically distinct (i.e., high correlation).

These definitions were also implicitly present in other studies through the use of various statistical techniques summarized in Table 1. In the studies that assessed whether measures of two constructs were empirically distinguishable, comparison of CFA models was the most common technique, followed by calculating a correlation that was compared against a cutoff. The CFA comparison was most commonly used to assess whether two factors could be merged, but a range of other comparisons were also presented. In the correlation-based techniques, correlations were often calculated using scale scores; sometimes, correction for attenuation was used, whereas other times, estimated factor correlations were used. These correlations were evaluated by comparing the correlations with the square root of average variance extracted (AVE) or comparing their CIs against cutoffs. Some studies demonstrated that correlations were not significantly different from zero, whereas others showed that correlations were significantly different from one. In the studies that considered discriminant validity as the degree to which each item measured one construct only and not something else, various factor analysis techniques were the most commonly used, typically either evaluating the fit of the model where cross-loadings were constrained to be zero or estimating the cross-loadings and comparing their values against various cutoffs.

Table 1.

Techniques Used to Assess Discriminant Validity in AMJ, JAP, and ORM.

	AMJ (n = 27)		JAP (n = 73)		ORM (n = 5)
Techniques using correlation estimates
Scale score correlation ( ${\text{ρ}}_{\text{SS}}$ )	7	25.9%	8	11.0%	3	60.0%
Factor correlation ( ${\text{ρ}}_{\text{CFA}}$ )	0	0.0%	2	2.7%	1	20.0%
Disattenuated correlation ( ${\text{ρ}}_{\text{DTR}}$ )	0	0.0%	1	1.4%	1	20.0%
Techniques to compare AVE to a certain value
AVECFA vs. Square of ${\text{ρ}}_{\text{CFA}}$ ( $\text{AVE}/{\text{SV}}_{\text{CFA}}$ )	2	7.4%	4	5.5%	1	20.0%
AVECFA vs. Square of ${\text{ρ}}_{\text{SS}}$ ( $\text{AVE}/{\text{SV}}_{\text{SS}}$ )	1	3.7%	1	1.4%	0	0.0%
AVECFA vs. .5	2	7.4%	1	1.4%	0	0.0%
AVEPLS vs. Square of ${\text{ρ}}_{\text{SS}}$	2	7.4%	0	0.0%	0	0.0%
Techniques to show low cross-loadings
CFA (structure coefficients)	2	7.4%	0	0.0%	0	0.0%
Exploratory factor analysis	1	3.7%	0	0.0%	0	0.0%
Techniques using fit indices of CFA models
No comparison (only the proposed model)	3	11.1%	1	1.4%	0	0.0%
Compared with nested models with fewer factors( ${\text{χ}}^2\left(\text{merge}\right))$	8	29.6%	43	58.9%	1	20.0%
Compared with model with fixed correlation of 1 ( ${\text{χ}}^2\left(1\right)$ )	4	14.8%	1	1.4%	2	40.0%
Compared with model with fixed correlation of 1 (CFI(1))	0	0.0%	0	0.0%	1	20.0%
Techniques requiring multiple measurement methods
GCES approach	0	0.0%	0	0.0%	1	20.0%
MTMM approach	0	0.0%	1	1.4%	0	0.0%
Generalizability theory approach	0	0.0%	1	1.4%	0	0.0%
Techniques that are difficult to classify
CFA results not presented in detail	1	3.7%	4	5.5%	0	0.0%
No clear evidence provided	0	0.0%	3	4.1%	0	0.0%
Comparison with existing research results	0	0.0%	2	2.7%	0	0.0%
Experimental results as expected	0	0.0%	2	2.7%	0	0.0%

Note: The sum exceeds 100% because some studies use multiple techniques. AMJ = Academy of Management Journal; JAP = Journal of Applied Psychology; ORM = Organizational Research Methods; CFA = confirmatory factor analysis; AVE = average variance extracted; ${\text{ρ}}_{\text{CFA}}$ = factor correlation obtained from CFA; ${\text{ρ}}_{\text{TR}}$ = disattenuated correlation using tau-equivalent reliability; AVE_CFA = AVE obtained from CFA; AVE_PLS = AVE obtained from partial least squares; GCES = generalized coefficient of equivalence and stability; MTMM = multitrait-multimethod. For a detailed description of the symbols, see Table 4.

Our review also revealed two findings that go beyond cataloging the discriminant validation techniques. First, no study evaluated discriminant validity as something that can exist to a degree but instead used the statistics to answer a yes/no type of question. Second, many techniques were used differently than originally presented. Given the diversity of how discriminant validity is conceptualized, the statistics used in its assessment, and how these statistics are interpreted, there is a clear need for a standard understanding of which technique(s) should be used and how the discriminant validity evidence produced by these techniques should be evaluated. However, to begin the evaluation of the various techniques, we must first establish a definition of discriminant validity.

Defining Discriminant Validity

Most methodological work defines discriminant validity by using a correlation but differs in what specific correlation is used, as shown in Table 2. For example, defining discriminant validity in terms of a (true) correlation between constructs implies that a discriminant validity problem cannot be addressed with better measures. In contrast, defining discriminant validity in terms of measures or estimated correlation ties it directly to particular measurement procedures. Some studies (i.e., categories 3 and 4 in Table 2) used definitions involving both constructs and measures stating that a measure should not correlate with or be affected by an unrelated construct. Moreover, there is no consensus on what kind of attribute discriminant validity is; some studies (Bagozzi & Phillips, 1982; Hamann et al., 2013; Reichardt & Coleman, 1995; J. A. Shaffer et al., 2016) define discriminant validity as a matter of degree, while others (Schmitt & Stults, 1986; Werts & Linn, 1970) define discriminant validity as a dichotomous attribute. These findings raise two important questions: (a) Why is there such diversity in the definitions? and (b) How exactly should discriminant validity be defined?

Table 2.

Definitions of Discriminant Validity in Existing Studies.

Category	Definition/Description of Technique
1: True or estimated correlation between constructsa	“[T]he degree to which the absolute value of the correlation between the two constructs differ from one.” (Reichardt & Coleman, 1995, p. 516)“Evidence of discriminant validity exists if other constructs do not correlate strongly enough with the construct of interest to suggest that they measure the same construct.” (McKenny et al., 2013, p. 156)“Discriminant validation implies that correlation between traits is low. If both traits were identical, the correlation between the trait factors would be near one.” (Kenny, 1976, p. 251)“[D]iscriminant validity exists when estimates of the trait correlations were two or more standard errors below 1.0.” (Schmitt & Stults, 1986, p. 18)“[D]iscriminant validity consists of demonstrating that the true correlation of [two traits] is meaningfully less than unity.” (Werts & Linn, 1970, p. 208)
2: Correlation between measures	“[A] test [should] not correlate too highly with measures from which it is supposed to differ.” (Campbell, 1960, p. 548)“[A test] correlates less well or not all with tests with which theory implies it should not correlate well.” (McDonald, 1985, p. 220)“[T]he extent to which measures of theoretically distinct constructs are unrelated empirically to one another.” (J. A. Shaffer et al., 2016, p. 82)“[I]f two or more concepts are unique, then valid measures of each should not correlate too highly.” (Bagozzi et al., 1991, p. 425)“[T]he degree of divergence among indicators that are designed to measure different constructs.” (Hamann et al., 2013, p. 72)“[T]he degree to which measures of distinct concepts differ.” (Bagozzi & Phillips, 1982, p. 469)“Measures of different attributes should…not correlated to an extremely high degree.” (Nunnally & Bernstein, 1994, p. 93)“[A] measure of a construct is unrelated to indicators of theoretically irrelevant constructs in the same domain.” (Strauss & Smith, 2009, p. 1)
3: Correlation between measure and other construct	“[D]iscriminant validity is shown when each measurement item correlates weakly with all other constructs except for the one to which it is theoretically associated.” (Gefen & Straub, 2005, p. 92)“Discriminant validity is inferred when scores from measures of different constructs do not converge. It thus provides information about whether scores from a measure of a construct are unique rather than contaminated by other constructs.” (Schwab, 2013, p. 33)
4: Combination of categories 1 and 3.	“[T]he item’s…loading on constructs other than the intended one is relevant to discriminant validity…. At the level of the constructs, this correlation tells us about discriminant validity.” (John & Benet-Martínez, 2000, p. 359)Voorhees et al. (2016) classified discriminant validity into the construct-level (i.e., low correlation) and the item-level (i.e., absence of cross-loading).

a. Many articles in this category are ambiguous on whether discriminant validity is a property of a construct or a property of a scale from which construct correlation is estimated.

Origin of the Concept of Discriminant Validity

The term “discriminant validity” was coined by Campbell and Fiske (1959), who presented a validation technique based on the long-standing idea that tests can be invalidated by too high correlations with unrelated tests (Campbell, 1960; Thorndike, 1920). However, the term was introduced without a clear definition of the concept (Reichardt & Coleman, 1995); instead, the article focuses on discriminant validation or how discriminant validity can be shown empirically using multitrait-multimethod (MTMM) matrices. The original criteria, illustrated in Table 3, were as follows: (a) two variables that measure the same trait (T1) with two different methods (M1, M2) should correlate more highly than any two variables that measure two different traits (T1, T2) with different methods (M1, M2); (b) two variables that measure the same trait (T1) with two different methods (M1, M2) should correlate more highly than any two variables that measure two different traits (T1, T2) but use the same method (M1); and (c) the pattern of correlations between variables that measure different traits (T1, T2) should be very similar across different methods (M1, M2) (Campbell & Fiske, 1959).

Table 3.

Multitrait-Multimethod Correlation Matrix and Original Criteria for Discriminant Validity.

		Method M1			Method M2
	Traits	T1	T2	T3	T1	T2	T3
M1	T1	1						Discriminant Validity:All MTHM > HTHMAll MTHM > HTMMHTMM11 ≈ HTMM31 HTMM12 ≈ HTMM32 HTMM13 ≈ HTMM33
	T2	HTMM11	1
	T3	HTMM12	HTMM13	1
M2	T1	MTHM21	HTHM24	HTHM27	1
	T2	HTHM22	MTHM25	HTHM28	HTMM31	1
	T3	HTHM23	HTHM26	MTHM29	HTMM32	HTMM33	1

Note: HTMM = same method and different traits (heterotrait-monomethod); MTHM = different methods and same trait (monotrait-heteromethod); HTHM = different methods and different traits (heterotrait-heteromethod).

Generalizing the concept of discriminant validity outside MTMM matrices is not straightforward. Indeed, the definitions shown in Table 2 show little connections to the original MTMM matrices. A notable exception is Reichardt and Coleman (1995), who criticized the MTMM-based criteria for being dichotomous and declared that a natural (dichotomous) definition of discriminant validity outside the MTMM context would be that two measures x1 and x2 have discriminant validity if and only if x1 measures construct T1 but not T2, x2 measures T2 but not T1, and the two constructs are not perfectly correlated. They then concluded that a preferable continuous definition would be “the degree to which the absolute value of the correlation between the two constructs differs from one” (Reichardt & Coleman, 1995, p. 516). A similar interpretation was reached by McDonald (1985), who noted that two tests have discriminant validity if “the common factors are correlated, but the correlations are low enough for the factors to be regarded as distinct ‘constructs’” (p. 220).

Constructs and Measures

Discriminant validity is sometimes presented as the property of a construct (Reichardt & Coleman, 1995) and other times as the property of its measures or empirical representations constructed from those measures (McDonald, 1985). This ambiguity may stem from the broader confusion over common factors and constructs: The term “construct” refers to the concept or trait being measured, whereas a common factor is part of a statistical model estimated from data (Maraun & Gabriel, 2013). Indeed, Campbell and Fiske (1959) define validity as a feature of a test or measure, not as a property of the trait or construct being measured. In fact, if one takes the realist perspective that constructs exist independently of measurement and can be measured in multiple different ways (Chang & Cartwright, 2008), ¹ it becomes clear that we cannot use an empirical procedure to define a property of a construct.

The Role of the Factor Model

Factor analysis has played a central role in articles on discriminant validation (e.g., McDonald, 1985), but it cannot serve as a basis for a definition of discriminant validity for two reasons. First, validity is a feature of a test or a measure or its interpretation (Campbell & Fiske, 1959), not of any particular statistical analysis. Moreover, discriminant validity is often presented as a property of “an item” (Table 2), implying that the concept should also be applicable in the single-item case, where factor analysis would not be applicable. Second, a factor model where each item loads on only one factor may be too constraining for applied research (Asparouhov et al., 2015; Marsh et al., 2014; Morin et al., 2017; Rodriguez et al., 2016). For example, Marsh et al. (2014) note that in psychology research, the symptoms or characteristics of different disorders commonly overlap, producing nonnegligible cross-loadings in the population. Constraining these cross-loadings to be zero can inflate the estimated factor correlations, which is problematic, particularly for discriminant validity assessment (Marsh et al., 2014). Moreover, a linear model where factors, error terms, and observed variables are all continuous (Bartholomew, 2007) is not always realistic. Indeed, a number of item response theory models have been introduced (Foster et al., 2017; Reise & Revicki, 2014) to address these scenarios and have been applied to assess discriminant validity using MTMM data (Jeon & Rijmen, 2014).

Generalized Definition of Discriminant Validity

We present a definition that does not depend on a particular model and makes it explicit that discriminant validity is a feature of a measure instead of a construct: ² Two measures intended to measure distinct constructs have discriminant validity if the absolute value of the correlation between the measures after correcting for measurement error is low enough for the measures to be regarded as measuring distinct constructs.

This definition encompasses the early idea that even moderately high correlations between distinct measures can invalidate those measures if measurement error is present (Thorndike, 1920), which serves as the basis of discriminant validity (Campbell & Fiske, 1959). The definition can also be applied on both the scale level and the scale-item level. Consider the proposed definition in the context of the common factor model:

$\text{Σ}=\text{Λ Φ}{\text{Λ}}^{\prime }+\text{Θ}$

1

where $\text{Σ}$ is the interitem correlation matrix; $\text{Φ}$ is the factor correlation matrix, where all correlations are assumed to be positive for simplicity; $\text{Λ}$ is a factor pattern (loading) matrix; and $\text{Θ}$ is the item error covariance matrix. Within this context, our definition can be understood in two equivalent ways:

${\text{Σ}}_{i,j}\ll \text{ΛJ}{\text{Λ}}_{\mathrm{i,j}}^{\prime }$

2

$\left({\Lambda }^{\prime }\text{Λ}\right){}^{-1}{\text{Λ}}^{\prime }\left(\text{Σ}-\text{Θ}\right)\text{Λ}({\Lambda }^{\prime }\text{Λ}{)}_{\mathrm{k,l}}^{\text{−1}}\ll 1$

3

where J is a unit matrix (a matrix of ones) and $\ll$ denotes much less than. Equation 2 is an item-level comparison (category 2 in Table 2), where the correlation between items i and j, which are designed to measure different constructs, is compared against the implied correlation when the items depend on perfectly correlated factors but are not perfectly correlated because of measurement error. Equation 3 shows an equivalent scale-level comparison (part of category 1 in Table 2) focusing on two distinct scales k and l. The factor correlations are solved from the interitem correlations by multiplying with left and right inverses of the factor pattern matrix to correct for measurement error and are then compared against a perfect correlation. Generalizing beyond the linear common factor model, Equation 3 can be understood to mean that two scales intended to measure distinct constructs have discriminant validity if the absolute value of the correlation between two latent variables estimated from the scales is low enough for the latent variables to be regarded as representing distinct constructs.

Our definition has several advantages over previous definitions shown in Table 2. First, it clearly states that discriminant validity is a feature of measures and not constructs and that it is not tied to any particular statistical test or cutoff (Schmitt, 1978; Schmitt & Stults, 1986). Second, the definition is compatible with both continuous and dichotomous interpretations, as it suggests the existence of a threshold, that is, a correlation below a certain level has no problem with discriminant validity but does not dictate a specific cutoff, thus also allowing the value of the correlation to be interpreted instead of simply tested. Third, the definition is not tied to any particular measurement process (e.g., single administration) but considers measurement error generally, thus supporting rater, transient, and other errors (Le et al., 2009; Schmidt et al., 2003). Fourth, the definition is not tied to either the individual item level or the multiple item scale level but works across both, thus unifying the category 1 and category 2 definitions of Table 2. Fifth, the definition does not confound the conceptually different questions of whether two measures measure different things (discriminant validity) and whether the items measure what they are supposed to measure and not something else (i.e., lack of cross-loadings in $\text{Λ}$ , factorial validity), ³ which some of the earlier definitions (categories 3 and 4 in Table 2) do.

This definition also supports a broad range of empirical practice: If considered on the scale level, the definition is compatible with the current tests, including the original MTMM approach (Campbell & Fiske, 1959). However, it is not limited to simple linear common factor models where each indicator loads on just one factor but rather supports any statistical technique including more complex factor structures (Asparouhov et al., 2015; Marsh et al., 2014; Morin et al., 2017; Rodriguez et al., 2016) and nonlinear models (Foster et al., 2017; Reise & Revicki, 2014) as long as these techniques can estimate correlations that are properly corrected for measurement error and supports scale-item level evaluations.

Overview of the Techniques for Assessing Discriminant Validity

The techniques for assessing discriminant validity identified in our review can be categorized into (a) techniques that assess correlations and (b) techniques that focus on model fit assessment. The techniques and the symbols that we use for them are summarized in Table 4.

Table 4.

Techniques Included in the Simulation of This Study.

Statistic/Technique	Symbol	Description/Criteria
Correlation estimation methods discussed in this study
Scale score correlation	${\text{ρ}}_{\text{SS}}$	Correlation between unit-weighted sums of scale scores (i.e., most commonly reported)
Factor correlation	${\text{ρ}}_{\text{CFA}}$	Correlation obtained through confirmatory factor analysis (CFA) (i.e., Figure 3)
Disattenuated correlation using parallel reliability	${\text{ρ}}_{\text{DPR}}$	${\text{ρ}}_{\text{SS}}$ is converted to an error-adjusted correlation using parallel reliability (i.e., the standardized alpha), also called the heterotrait-monotrait (HTMT) ratio
Disattenuated correlation using tau-equivalent reliability	${\text{ρ}}_{\text{DTR}}$	${\text{ρ}}_{\text{SS}}$ is converted to an error-adjusted correlation using tau-equivalent reliability (i.e., the coefficient alpha)
Disattenuated correlation using congeneric reliability	${\text{ρ}}_{\text{DCR}}$	${\text{ρ}}_{\text{SS}}$ is converted to an error-adjusted correlation using congeneric reliability (i.e., the composite reliability or omega)
Common notation for expressing correlation-related techniques
The point estimate is less than a cutoff	${\text{ρ}}_{\text{XX}}\left(\text{cut}\right)$	For every factor pair, the point estimate of ${\text{ρ}}_{\text{XX}}$ is less than a cutoff
The confidence interval (CI) is less than 1	CIXX(1)	For every factor pair, the CI of ${\text{ρ}}_{\text{XX}}$ is strictly less than one
The CI is less than a cutoff	$\text{C}{\text{I}}_{\text{XX}}\left(\text{cut}\right)$	For every factor pair, the CI of ${\text{ρ}}_{\text{XX}}$ is strictly less than a cutoff
Other correlation-related techniques
Average variance extracted (AVE) compared with the SV obtained from factor correlations	$\text{AVE}/{\text{SV}}_{\text{CFA}}$	For every factor pair, the AVEs of the two factors are greater than the square of ${\text{ρ}}_{\text{CFA}}$ (i.e., used as originally proposed by Fornell & Larcker, 1981a)
AVE compared with the SV obtained from scale score correlations	$\text{AVE}/{\text{SV}}_{\text{SS}}$	For every factor pair, the AVEs of the two factors are greater than the square of ${\text{ρ}}_{\text{SS}}$ (i.e., common misuse)
Techniques that focus on model fit
A chi-square comparison with a model with a fixed correlation of 1	${\text{χ}}^2\left(1\right)$	For every factor pair, the chi-square difference between the unconstrained model and the constrained model in which ${\text{ρ}}_{\text{CFA}}$ is fixed at one is statistically significant
A chi-square comparison with a model that merges two factors into one	${\text{χ}}^2\left(\text{merge}\right)$	For every factor pair, the chi-square difference between the unconstrained model and the constrained model in which the two factors are merged into one is statistically significant
A comparative fix index (CFI) comparison with a model with a fixed correlation of 1	CFI(1)	For every factor pair, the CFI difference between the unconstrained model and the constrained model in which ${\text{ρ}}_{\text{CFA}}$ is fixed at one is greater than .002
A chi-square comparison with a model with a fixed correlation of a cutoff	${\text{χ}}^2\left(\text{cut}\right)$	For every factor pair, the chi-square difference between the unconstrained model and the constrained model in which ${\text{ρ}}_{\text{CFA}}$ is fixed at a cutoff is statistically significant
A CFI comparison with a model with a fixed correlation of a cutoff	$\text{CFI}\left(\text{cut}\right)$	For every factor pair, the CFI difference between the unconstrained model and the constrained model in which ${\text{ρ}}_{\text{CFA}}$ is fixed at a cutoff is greater than .002

Techniques That Assess Correlations

There are three main ways to calculate a correlation for discriminant validity assessment: a factor analysis, a scale score correlation, and the disattenuated version of the scale score correlation. While item-level correlations or their disattenuated versions could also be applied in principle, we have seen this practice neither recommended nor used. Thus, in practice, the correlation techniques always correspond to the empirical test shown as Equation 3. Regardless of how the correlations are calculated, they are used in just two ways: either by comparing the point estimates against a cutoff (i.e., the square root of the AVE) or by checking whether the absolute value of the CI contains either zero or one. Using the cutoff of zero is clearly inappropriate as requiring that two factors be uncorrelated is not implied by the definition of discriminant validity and would limit discriminant validity assessment to the extremely rare scenario where two constructs are assumed to be (linearly) independent. We will next address the various techniques in more detail.

Factor Analysis Techniques

Factor correlations can be estimated directly either by exploratory factor analysis (EFA) or CFA, but because none of the reviewed guidelines or empirical applications reported EFA correlations, we focus on CFA. The estimation of factor correlations in a CFA is complicated by the fact by default latent variables are scaled by fixing the first indicator loadings, which produces covariances that are not correlations. Correlations (denoted ${\text{ρ}}_{\text{CFA}}$ ) can be estimated by either freeing the factor loadings and scaling the factors by fixing their variances to 1 (i.e., A in Figure 2) or standardizing the factor covariance matrix (i.e., B), for example, by requesting standardized estimates that all SEM software provides. Both techniques produce the same estimate, although the standard errors (and CIs) can be different (Gonzalez & Griffin, 2001).

Figure 2.

Factor correlation estimation. (A) Fixing the variances of factors to unity (i.e., not using the default option). (B) Fixing one of the loadings to unity (i.e., using the default option).

Of the correlation estimation techniques, CFA is the most flexible because it is not tied to a particular model but requires only that the model be correctly specified. Thus, cross-loadings, nonlinear factor loadings or nonnormal error terms can be included because a CFA model can also be used in the context of item response models (Foster et al., 2017), bifactor models (Rodriguez et al., 2016), exploratory SEMs (Marsh et al., 2014) or other more advanced techniques. However, these techniques tend to require larger sample sizes and advanced software and are consequently less commonly used.

Scale Score Correlations and Disattenuated Correlations

The simplest and most common way to estimate a correlation between two scales is by summing or averaging the scale items as scale scores and then taking the correlation (denoted ${\text{ρ}}_{\text{SS}}$ ). ⁴ The problem with this approach is that the scores contain measurement errors, which attenuate the correlation and may cause discriminant validity issues to go undetected. ⁵ To address this issue, the use of disattenuated or error-corrected correlations where the effect of unreliability is removed is often recommended (Edwards, 2003; J. A. Shaffer et al., 2016):

${\text{ρ}}_{12}=\frac{{\text{ρ}}_{XY}}{\sqrt{{\text{ρ}}_X{\text{ρ}}_Y}}$

4

where the disattenuated correlation ( ${\text{ρ}}_{12}$ ) is a function of the scale score correlation ( ${\text{ρ}}_{XY}$ ) and the scale score reliabilities ( ${\text{ρ}}_X, {\text{ρ}}_Y$ ). That is, a disattenuated correlation is the scale score correlation from which the effect of unreliability is removed. ⁶

Reliability can be estimated in different ways, including test-retest reliability, interrater reliability and single-administration reliability, ⁷ which each provide information on different sources of measurement error (Le et al., 2009; Schmidt et al., 2003). Given our focus on single-method and one-time measurements, we address only single-administration reliability, where measurement errors are operationalized by uniqueness estimates, ignoring time and rater effects that are incalculable in these designs. The two most commonly used single-administration reliability coefficients are tau-equivalent reliability, ⁸ often referred to as Cronbach’s alpha, and congeneric reliability, usually called composite reliability by organizational researchers and McDonald’s omega or ω by methodologists. ⁹ As the names indicate, the key difference is whether we assume that the items share the same true score (tau-equivalent, B in Figure 3) or make the less constraining assumption that the items simply depend on the same latent variable but may do so to different extents (congeneric, C in Figure 3). Recent research suggests that the congeneric reliability coefficient is a safer choice because of the less stringent assumption (Cho, 2016; Cho & Kim, 2015; McNeish, 2017).

Figure 3.

The assumptions of parallel, tau-equivalent, and congeneric reliability. (A) parallel, (B) tau-equivalent, (C) congeneric.

The reliability coefficients presented above make a unidimensionality assumption, which may not be realistic in all empirical research. While the disattenuation formula (Equation 4) is often claimed to assumes that the only source of measurement error is random noise or unreliability, the assumption is in fact more general: All variance components in the scale scores that are not due to the construct of interest are independent of the construct and measurement errors of other scale scores. This more general formulation seems to open the option of using hierarchical omega (Cho, 2016; Zinbarg et al., 2005), which assumes that the scale measures one main construct (main factor) but may also contain a number of minor factors that are assumed to be uncorrelated with the main factor. However, using hierarchical omega for disattenuation is problematic because it introduces an additional assumption that the minor factors (e.g., disturbances in the second-order factor model and group factors in the bifactor model) are also uncorrelated between two scales, which is neither applied nor tested when reliability estimates are calculated separately for both scales, as is typically the case. While the basic disattenuation formula has been extended to cases where its assumptions are violated in known ways (Wetcher-Hendricks, 2006; Zimmerman, 2007), the complexities of modeling the same set of violations in both the reliability estimates and the disattenuation equation do not seem appealing given that the factor correlation can be estimated more straightforwardly with a CFA instead.

While simple to use, the disattenuation correction is not without problems. A common criticism is that the correction can produce inadmissible correlations (i.e., greater than 1 or less than –1) (Charles, 2005; Nimon et al., 2012), but this issue is by no means a unique problem because the same can occur with a CFA. However, a CFA has three advantages over the disattenuation equation. First, CFA correlations estimated with maximum likelihood (ML) can be expected to be more efficient than multistep techniques that rely on corrections for attenuation (Charles, 2005; Muchinsky, 1996). Second, the disattenuation equation assumes that the scales are unidimensional and that all measurement errors are uncorrelated, whereas a CFA simply assumes that the model is correctly specified and identified. Third, calculating the CIs for a disattenuated correlation is complicated (Oberski & Satorra, 2013). However, this final concern can be alleviated to some extent through the use of bootstrap CIs (Henseler et al., 2015); in particular, the bias-corrected and accelerated (BCa) technique has been shown to work well for this particular problem (Padilla & Veprinsky, 2012, 2014).

AVE/SV or Fornell-Larcker Criterion

While commonly used, the AVE statistic has been rarely discussed by methodological research and, consequently, is poorly understood. ¹⁰ One source of confusion is the similarity between the formula for AVE and that of congeneric reliability (Fornell & Larcker, 1981a):

$\text{AVE}=\frac{{{\displaystyle \sum }}_{i=1}^k{\text{λ}}_i^2}{{{\displaystyle \sum }}_{i=1}^k{\text{λ}}_i^2+{{\displaystyle \sum }}_{i=1}^k{\text{σ}}_{e_i}^2}$

5

$\text{CR}=\frac{{\left({{\displaystyle \sum }}_{i=1}^k{\text{λ}}_i\right)}^2}{{\left({{\displaystyle \sum }}_{i=1}^k{\text{λ}}_i\right)}^2+{{\displaystyle \sum }}_{i=1}^k{\text{σ}}_{e_i}^2}$

6

The meaning of AVE becomes more apparent if we rewrite the original equation as:

$\text{AVE}=\frac{{{\displaystyle \sum }}_{i=1}^k{\text{σ}}_{x_i}^2{\text{ρ}}_i}{{{\displaystyle \sum }}_{i=1}^k{\text{σ}}_{x_i}^2}$

7

where ${\text{ρ}}_i= \frac{{\text{λ}}_i^2}{{\text{σ}}_{x_i}^2}$ is the reliability and ${\text{σ}}_{x_i}^2= {\text{λ}}_i^2+{\text{σ}}_{e_i}^2$ is the variance of item i. This alternative form shows that AVE is actually an item-variance weighted average of item reliabilities. With standardized estimates, AVE reduces to an average of item reliabilities. Thus, the term “average indicator reliability” might be more informative than “average variance extracted.”

Fornell and Larcker (1981a) presented a decision rule that discriminant validity holds for two scales if the AVEs for both are higher than the squared factor correlation between the scales. We refer to this rule as AVE/SV because the squared correlation quantifies shared variance (SV; Henseler et al., 2015). Because a factor correlation corrects for measurement error, the AVE/SV comparison is similar to comparing the left-hand side of Equation 3 against the right-hand side of Equation 2. Therefore, AVE/SV has a high false positive rate, indicating a discriminant validity problem under conditions where most researchers would not consider one to exist, as indicated by A in Figure 4. This tendency has been taken as evidence that AVE/SV is “a very conservative test” (Voorhees et al., 2016, p. 124), whereas the test is simply severely biased.

Figure 4.

The inappropriateness of the AVE as an index of discriminant validity. (A) despite high discriminant validity, the AVE/SV criterion fails, (B) despite low discriminant validity, the AVE/SV criterion passes.

In applied research, the AVE/SV criterion rarely shows a discriminant validity problem because it is commonly misapplied. The most common misapplication is to compare the AVE values with the square of the scale score correlation, not the square of the factor correlation (Voorhees et al., 2016). Another misuse is to compare AVEs against the .5 rule-of-thumb cutoff, which Fornell and Larcker (1981a) presented as a convergent validity criterion. Other misuses are that only one of the two AVE values or the average of the two AVE values should be greater than the SV (Farrell, 2010; Henseler et al., 2015). The original criterion is that both AVE values must be greater than the SV. Finally, the AVE statistic is sometimes calculated from a partial least squares analysis (AVE_PLS), which overestimates indicator reliabilities and thus cannot detect even the most serious problems (Rönkkö & Evermann, 2013).

Heterotrait-Monotrait Ratio (HTMT)

The heterotrait-monotrait (HTMT) ratio was recently introduced in marketing (Henseler et al., 2015) and is being adopted in other disciplines as well (Kuppelwieser et al., 2019). While Henseler et al. (2015) motivate HTMT based on the original MTMM approach (Campbell & Fiske, 1959), this index is actually neither new nor directly based on the MTMM approach. The original version of the HTMT equation is fairly complex, but to make its meaning more apparent, it can be simplified as follows:

${\text{HTMT}}_{ij}=\frac{\overline{{\text{σ}}_{ij}}}{\sqrt{\overline{{\text{σ}}_i}\overline{{\text{σ}}_j}}}$

8

where $\overline{{\text{σ}}_i}$ and $\overline{{\text{σ}}_j}$ denote the average within scale item correlation and $\overline{{\text{σ}}_{ij}}$ denotes the average between scale item correlation for two scales i and j. This simpler form makes it clear that HTMT is related to the disattenuation formula (Equation 4). In fact, HTMT is equivalent to a disattenuated correlation of unit-weighted composites using parallel reliability (i.e., the standardized alpha, proof in the appendix). Thus, while marketed as a new technique, the HTMT index has actually been used for decades; parallel reliability is the oldest reliability coefficient (Brown, 1910), and disattenuated correlations have been used to assess discriminant validity for decades (Schmitt, 1996). Thus, the term “HTMT” is misleading, giving the false impression that HTMT is related to MTMM and obscuring the fact that it is simply a variant of disattenuated correlation. Thus, we favor a more transparent term “disattenuated correlation using parallel reliability” (denoted ${\text{ρ}}_{DPR}$ ) because this more systematic name tells what the coefficient is (i.e., correlation), how it is obtained (i.e., disattenuated), and under what conditions it can be used (i.e., parallel reliability).

Because the ${\text{ρ}}_{DPR}$ statistic is a disattenuated correlation, it shares all the interpretations, assumptions, and limitations of the techniques that were explained earlier. Compared to the tau-equivalence assumption, this technique makes an even more constraining parallel measurement assumption that the error variances between items are the same (A in Figure 3). Most real-world data deviate from these assumptions, in which case ${\text{ρ}}_{DPR}$ yields inaccurate estimates (Cho, 2016; McNeish, 2017), making this technique an inferior choice.

Model Fit Comparison Techniques

Model comparison techniques involve comparing the original model against a model where a factor correlation is fixed to a value high enough to be considered a discriminant validity problem. Their general idea is that if the two models fit equally well, the model with a discriminant validity problem is plausible, and thus, there is a problem. The most common constraints are that (a) two factors are fixed to be correlated at 1 (i.e., A in Figure 5) or (b) two factors are merged into one factor (i.e., C in Figure 5), thus reducing their number by one.

Figure 5.

(A) constrained model for χ²(1), (B) common misuse of χ²(1), (C) constrained model for χ²(merge), (D) model equivalent to C.

The key advantage of these techniques is that they provide a test statistic and a p-value. However, this also has the disadvantage that it steers a researcher toward making yes/no decisions instead of assessing the degree to which discriminant validity holds in the data. Compared to correlation-based techniques, where a single CFA model provides all the estimates required for discriminant validity assessment, model comparison techniques require more work because a potentially large number of comparisons must be managed. ¹¹ We next assess the various model comparison techniques presented and used in the literature.

${\text{χ}}^2\left(1\right)$

In the ${\text{χ}}^2\left(1\right)$ test, the constrained model has the correlation between two factors fixed to be 1, after which the model is compared against the original one with a nested model ${\text{χ}}^2$ test. While the nested model ${\text{χ}}^2$ test is a standard tool in SEM, there are four issues that require attention when ${\text{χ}}^2\left(1\right)$ is applied for discriminant validity assessment. First, it is easy to specify the constrained model incorrectly. As discussed earlier, SEM models estimate factor covariances, and implementing ${\text{χ}}^2\left(1\right)$ involves constraining one of these covariances to 1. ¹² However, methodological articles commonly fail to explain that the 1 constraint must be accompanied by setting the variances of the latent variables to 1 instead of scaling the latent variables by fixing the first item loadings (J. A. Shaffer et al., 2016; Voorhees et al., 2016). Indeed, our review provided evidence that incorrect application of this test may be fairly common. ¹³ An incorrect scaling of the latent variable (i.e., B in Figure 5) can produce either an inflated false positive or false negative rate, depending on whether the estimated factor variances are greater than 1 or less than 1. We demonstrate this problem in Online Supplement 1.

The second issue is that some articles suggest that the significance level of the ${\text{χ}}^2$ difference test should be adjusted for multiple comparisons (Anderson & Gerbing, 1988; Voorhees et al., 2016). Their logic is that if the Type I error of an individual test equals $\text{α}$ , the probability that a Type I error occurs at least once (familywise Type I error) is much greater than $\text{α}$ when multiple tests are done. Multiple comparison corrections address this issue by adjusting the individual test $\text{α}$ level to keep the familywise Type I error at the intended level. To address this issue, Anderson and Gerbing (1988, n. 2) recommend applying the Šidák correction. Voorhees et al. (2016) further claim that the common omission of the correction is “the most troublesome issue with the [ ${\text{χ}}^2\left(1\right)$ ] approach” (p. 123). These concerns are ill-founded. Šidák and the related Bonferroni corrections make the universal null hypothesis that all individual null hypotheses are true (Hancock & Klockars, 1996; Perneger, 1998; J. P. Shaffer, 1995). Thus, in the context of ${\text{χ}}^2\left(1\right)$ , the universal null hypothesis is that all factors are perfectly correlated. This hypothesis is almost certainly always false, rendering tests that rely on it meaningless. Additionally, if the hypothesis is of interest, fitting a single-factor model to the data (i.e., merging all factors into one) provides a more straightforward test. While multiple comparison techniques can be useful (Castañeda et al., 1993), there are scenarios where they should not be applied (Perneger, 1998), and the literature on discriminant validity has failed to provide compelling reasons to do so.

The third issue is that the ${\text{χ}}^2\left(1\right)$ technique omits constraints that the perfect correlation implies: If the correlation between two factors equals 1, their correlations with all other factors should be equal as well. However, estimated correlations are unlikely to be exactly the same, often producing an inadmissible solution with a nonpositive definite latent variable covariance matrix. While this is not a problem for the ${\text{χ}}^2$ test itself, it produces a warning in the software and may cause unnecessary confusion. ¹⁴ This can be addressed by adding the implied equality constraints, but none of the reviewed works did this. Moreover, it is easier to estimate an equivalent model by simply merging the two factors as one ( ${\text{χ}}^2\left(\text{merge}\right)$ ).

The fourth and final issue is that the ${\text{χ}}^2\left(1\right)$ technique is a very powerful test for detecting whether the factor correlation is exactly 1. Paradoxically, this power to reject the null hypothesis has been interpreted as a lack of power to detect discriminant validity (Voorhees et al., 2016). Even if the latent variable correlation is only slightly different from 1 (e.g., .98), such small differences will be detected as statistically significant if the sample size is sufficiently large. However, in this case, it is difficult to interpret the latent variables as representing distinct concepts. This is a genuine problem with the ${\text{χ}}^2\left(1\right)$ test, and two proposals for addressing it have been presented in the literature.

CFI(1)

To address the issue that the ${\text{χ}}^2\left(1\right)$ test can flag correlations that differ from 1 by trivial amounts as significant, some recent articles (Le et al., 2010; J. A. Shaffer et al., 2016) have suggested comparing models by calculating the difference between the comparative fit indices (CFIs) of two models (ΔCFI), which is compared against the .002 cutoff (CFI(1)). This idea is adapted from Cheung and Rensvold’s (2002) proposal in the measurement invariance literature, and the .002 cutoff is based on the simulation by Meade et al. (2008).

The idea behind using ΔCFI in measurement invariance assessment is that the degrees of freedom of the invariance hypothesis depend on the model complexity, and the CFI index and consequently ΔCFI are less affected by this than the ${\text{χ}}^2$ (Meade et al., 2008). This idea is reasonable in the original context, but it does not apply in the context of CFI(1) comparison where the difference in degrees of freedom is always one, leaving this test without its main justification. Indeed, CFI(1) can be proved (see the appendix) to be equivalent to calculating the $\text{Δ}{\text{χ}}^2$ and comparing this statistic against a cutoff defined based on the fit of the null model ( ${\text{χ}}_B^2)$ and its degrees of freedom ( $d{f}_B)$ ,

$\text{Δ}{\text{χ}}^2>1+.002\left({\text{χ}}_B^2-d{f}_B\right)$

9

instead of using the .95 percentile from the ${\text{χ}}_{\left(1\right)}^2$ distribution, or 3.84, as a cutoff. Because the expected value of ${\text{χ}}_B^2$ increases with sample size in a way similar to $\text{Δ}{\text{χ}}^2$ , this comparison can be regarded as less sensitive to sample size. However, it is unclear whether this alternative cutoff has more or less power (i.e., whether $1+.002\left({\text{χ}}_B^2-d{f}_B\right)$ is greater or less than 3.84) because the effectiveness of CFI(1) has not been studied.

Changing the test statistic—or equivalently the cutoff value—is an ultimately illogical solution because the problem with the ${\text{χ}}^2\left(1\right)$ test is not that its power increases with sample size but that a researcher is ultimately not interested in whether the correlation between two variables differs from exactly 1; rather, a researcher is interested in whether the correlation is sufficiently different from 1. Thus, a more logical approach is to change the null hypothesis instead of adjusting the tests to be less powerful.

${\text{χ}}^2\left(cut\right)$ and $\mathrm{CFI}\left(cut\right)$

The lack of a perfect correlation between two latent variables is ultimately rarely of interest, and thus, it is more logical to use a null hypothesis that covers an interval (e.g., ${\varphi }_{12}$ >.9). This test can be implemented in any SEM software by first fitting a model where ${\varphi }_{12}$ is freely estimated. If the estimate falls outside the interval (e.g., less than .9), then the correlation is constrained to be at the endpoint of the interval, and the model is re-estimated. A significant result from a nested model comparison means that the original interval hypothesis can be rejected. We use the labels ${\text{χ}}^2\left(\text{cut}\right)$ and $\text{CFI}\left(\text{cut}\right)$ to denote tests that depend on whether the comparison is performed based on the ${\text{χ}}^2$ test or by comparing $\text{Δ CFI}$ against the .002 cutoff. While the idea that a number less than 1 can be used as a cutoff was briefly mentioned in John and Benet-Martínez (2000) and J. A. Shaffer et al. (2016), we are unaware of any studies that have applied interval hypothesis tests or tested their effectiveness.

${\text{χ}}^2\left(merge\right)$ and Other Comparisons Against Models with Fewer Factors

Many studies assess discriminant validity by comparing the hypothesized model with a model with fewer factors. This is typically done by merging two factors (i.e., C in Figure 5), and we refer to the associated nested model comparison as ${\text{χ}}^2\left(\text{merge}\right)$ . As mentioned above, this test is a more constrained version of ${\text{χ}}^2\left(1\right)$ where all pairs of correlations with either of the factors and a third factor are constrained to be the same (i.e., D in Figure 5). Because this test imposes more constraints than ${\text{χ}}^2\left(1\right)$ does, it has more statistical power. It also avoids the inadmissible solution issue of ${\text{χ}}^2\left(1\right)$ . Merging two factors will always produce the same ${\text{χ}}^2$ regardless of how the latent variables are scaled, and thus, this test is less likely to be incorrectly applied. However, the test also has a major weakness that, in contrast to ${\text{χ}}^2\left(1\right)$ , this test cannot be extended to other cutoffs (i.e., ${\text{χ}}^2\left(\text{cut}\right)$ ). Thus, while the test can be applied for testing perfect overlap between two latent variables, it cannot answer the question of whether the latent variables are sufficiently distinct.

As with the other techniques, various misconceptions and misuses are found among empirical studies. The most common misuse is to include unnecessary comparisons, for example, by testing alternative models with two or more factors less than the hypothesized model. Another common misuse is to omit some necessary comparisons, for example, by comparing only some alternative models, instead of comparing all possible alternative models with one factor less than the original model.

Single Model Fit Techniques

Discriminant validity has also been assessed by inspecting the fit of a single model without comparing against another model. These techniques fall into two classes: those that inspect the factor loadings and those that assess the overall model fit.

Cross-Loadings

Cross-loadings indicate a relationship between an indicator and a factor other than the main factor on which the indicator loads. Beyond this definition, the term can refer to two distinct concepts, factor pattern coefficients or factor structure coefficients (see Table 5), and has been confusingly used with both meanings in the discriminant validity literature. ¹⁵ Structure coefficients are correlations between items and factors, so their values are constrained to be between –1 and 1. Pattern coefficients, on the other hand, are analogous to (standardized) coefficients in regression analysis and are directional (Thompson & Daniel, 1996). Structure coefficients are not directly estimated as part of a factor analysis; instead, they are calculated based on the pattern coefficients and factor correlations. If the factors are rotated orthogonally (e.g., Varimax) or are otherwise constrained to be uncorrelated, the pattern coefficients and structure coefficients are identical (Henson & Roberts, 2006). However, the use of uncorrelated factors can rarely be justified (Fabrigar et al., 1999), which means that, in most cases, pattern and structure coefficients are not equal.

Table 5.

Factor Pattern Coefficients, Correlations, and Structure Coefficients.

	Definitions				Numerical Example
Items	Factor Pattern Coefficients		Factor Structure Coefficients		Factor Pattern Coefficients		Factor Structure Coefficients
	Factor1	Factor2	Factor1	Factor2	Factor1	Factor2	Factor1	Factor2
x1	${\text{λ}}_{11}$	0	${\text{λ}}_{11}$	${\text{λ}}_{11}\text{ρ}$	.70	0	.70	.35
x2	${\text{λ}}_{21}$	0	${\text{λ}}_{21}$	${\text{λ}}_{21}\text{ρ}$	.60	0	.60	.30
x3	${\text{λ}}_{31}$	0	${\text{λ}}_{31}$	${\text{λ}}_{31}\text{ρ}$	.80	0	.80	.40
x4	0	${\text{λ}}_{42}$	${\text{λ}}_{42}\text{ρ}$	${\text{λ}}_{42}$	0	.90	.45	.90
x5	0	${\text{λ}}_{52}$	${\text{λ}}_{52}\text{ρ}$	${\text{λ}}_{52}$	0	.70	.35	.70
x6	0	${\text{λ}}_{62}$	${\text{λ}}_{62}\text{ρ}$	${\text{λ}}_{62}$	0	.80	.40	.80

Note: Factor correlation $\text{ρ}=.5$ . The factor structure coefficients are the matrix multiplication of the factor pattern coefficients and factor correlations.

While both the empirical criteria shown in Equation 2 and Equation 3 contain pattern coefficients, assessing discriminant validity based on loadings is problematic. First, these comparisons involve assessing a single item or scale at a time, which is incompatible with the idea that discriminant validity is a feature of a measure pair. Second, pattern coefficients do not provide any information on the correlation between two scales, and structure coefficients are an indirect measure of the correlation at best. ¹⁶ Third, while the various guidelines differ in how loadings should be interpreted (Henseler et al., 2015; Straub et al., 2004; Thompson, 1997), they all share the features of relying mostly on authors’ intuition instead of theoretical reasoning or empirical evidence. In summary, these techniques fall into the rules of thumb category and cannot be recommended.

Single-Model fit

The final set of techniques is those that assess the single-model fit of a CFA model. A CFA model can fit the data poorly if there are unmodeled loadings (pattern coefficients), omitted factors, or error correlations in the data, none of which are directly related to discriminant validity. While some of the model fit indices do depend on factor correlations, they do so only weakly and indirectly (Kline, 2011, chap. 8). Thus, while a well-fitting factor model is an important assumption (either implicitly—e.g., the various ${\text{ρ}}_D$ —or explicitly—e.g., CFA techniques), model fit itself will not provide any information on whether either of the two empirical criteria shown in Equation 2 and Equation 3 holds.

Summary: Under What Conditions Is Each Technique Useful?

Our review of the literature provides several conclusions. Overall, ${\text{χ}}^2\left(\text{cut}\right)$ and CI_CFA (cut) can be recommended as general solutions because they meet the definition of discriminant validity, have the flexibility to adapt to various levels of cutoffs, and can be extended to more complex scenarios such as nonlinear measurement models (Foster et al., 2017), scales with minor dimensions (Rodriguez et al., 2016), or cases in which factorial validity is violated because of cross-loadings. Some of the other techniques can be useful for specific purposes. ${\text{χ}}^2\left(\text{merge}\right)$ , ${\text{χ}}^2\left(1\right)$ , and CI_CFA(1) can be used if theory suggests nearly perfect but not absolutely perfect correlations. They can also be useful as a first step in discriminant validity assessment; if any of them indicates a problem, then so will any variant of the techniques that use a cutoff of less than 1. Disattenuated correlations are useful in single-item scenarios, where reliability estimates could come from test-retest or interrater reliability checks or from prior studies. These techniques could also be used in multiple item scenarios, if a researcher does not have access to SEM software, or in some small sample scenarios (Rosseel, 2020). If the effect of measurement error can be assumed to be negligible, even using scale score correlations can be useful as a rough check. The techniques that assess the lack of cross-loadings (pattern coefficients) and model fit provide (factorial) validity information, which is important in establishing the assumptions of the other techniques, but these techniques are of limited use in providing actual discriminant validity evidence.

Monte Carlo Simulations

We will next compare the various discriminant validity assessment techniques in a Monte Carlo simulation with regard to their effectiveness in two common tasks: (a) quantifying the degree to which discriminant validity can be a problem and (b) making a dichotomous decision on whether discriminant validity is a problem in the population. For statistics with meaningful interpretations, we assessed the bias and variance of the statistic and the validity of the CIs. For the dichotomous decision, we set .85, .9, .95, and 1 as cutoffs and estimated the Type I error (i.e., false positive) and Type II error (i.e., false negative) rates for the conclusion that the factor correlation was at least at the cutoff value in the population.

Simulation Design

For simplicity, we followed the design used by Voorhees et al. (2016) and generated data from a three-factor model. We assessed the discriminant validity of the first two factors, varying their correlation as an experimental condition. Voorhees et al. (2016) used only two factor correlation levels, .75 and .9. We wanted a broader range from low levels where discriminant validity is unlikely to be a problem up to perfect correlation, so we used six levels: .5, .6, .7, .8, .9, and 1. The third factor was always correlated at .5 with the first two factors. The number of items was varied at 3, 6, and 9, and factor loadings were set to [0.9,0.8,0.8], [0.8,0.7,0.7], [0.5,0.4,0.4], [0.9,0.6,0.3], and [0.8, 0.8, 0.8]. For the six- and nine-item scenarios, each factor loading value was used multiple times. All factors had unit variances in the population, and we scaled the error variances so that the population variances of the items were one. Cross-loadings (in the pattern coefficients) were either 0, 1, or 2. This condition was implemented following the approach by Voorhees et al. (2016); in the cross-loading condition, either the first or first two indicators of the second latent variable also loaded on the first latent variable. The same value was used for both loadings, and the values were scaled down from their original values so that the factors always explained the same amount of variance in the indicators. In the six- and nine-item conditions, the number of cross-loaded items was scaled up accordingly. Sample size was the final design factor and varied at 50, 100, 250, and 1,000. The full factorial (6 × 3 × 5 × 3 × 4) simulation was implemented with the R statistical programming environment using 1,000 replications for each cell. We estimated the factor models with the lavaan package (Rosseel, 2012) and used semTools to calculate the reliability indices (Jorgensen et al., 2020). The full simulation code is available in Online Supplement 2, and the full set of simulation results at the design level can be found in Online Supplement 3.

Correlation Estimates and Their Confidence Intervals

We first focus on the scenarios where the factor model was correctly specified (i.e., there were no cross-loadings). Because ${\text{ρ}}_{\text{DPR}}$ and HTMT were proven equivalent and always produced identical results, we report only the former. We also omit the two low correlation conditions (i.e., .5, .6) because the false positive rates are already clear in the .7 condition.

Because the pattern of results is very similar for correlations and their confidence intervals, we present both sets of results together. Table 6 shows the correlation estimates by sample size, number of items, and factor loading conditions. Table 7 shows the coverage and the balance of the CIs by sample size and selected values of loading condition, omitting ${\text{ρ}}_{\text{SS}}$ because of its generally poor performance in the correlation results. Ideally, the coverage of a 95% CI should be .95, and the balance should be close to zero. The CIs for ${\text{ρ}}_{\text{CFA}}$ were obtained from the CFAs, and for ${\text{ρ}}_{\text{DPR}}$ , we used bootsrap percentile CIs, following Henseler et al. (2015). For comparison, we also calculated the bootstrap percentile CIs for ${\text{ρ}}_{\text{DTR}}$ and ${\text{ρ}}_{\text{DCR}}$ . All bootstrap analyses were calculated with 1,000 replications.

Table 6.

Correlation Estimates by Sample Size, Number of Items, and Loadings.

Sample Size, Indicators, Loadingsa	Technique	Factor Correlation
		.7		.8		.9		1.0
		M	SD	M	SD	M	SD	M	SD
50	${\text{ρ}}_{\text{CFA}}$	.694	.111	.801	.088	.897	.070	1.000	.049
	${\text{ρ}}_{\text{DPR}}$	.697	.108	.802	.086	.901	.066	1.003	.047
	${\text{ρ}}_{\text{DTR}}$	.698	.108	.804	.086	.903	.066	1.005	.048
	${\text{ρ}}_{\text{DCR}}$	.694	.108	.800	.087	.898	.066	1.000	.047
	${\text{ρ}}_{\text{SS}}$	.583	.095	.672	.080	.753	.065	.838	.045
100	${\text{ρ}}_{\text{CFA}}$	.700	.076	.801	.061	.899	.045	1.000	.032
	${\text{ρ}}_{\text{DPR}}$	.700	.075	.802	.059	.900	.044	1.002	.031
	${\text{ρ}}_{\text{DTR}}$	.701	.075	.803	.060	.901	.044	1.003	.031
	${\text{ρ}}_{\text{DCR}}$	.699	.076	.801	.060	.899	.044	1.000	.031
	${\text{ρ}}_{\text{SS}}$	.588	.068	.673	.057	.755	.042	.841	.030
250	${\text{ρ}}_{\text{CFA}}$	.699	.043	.799	.038	.899	.028	1.001	.019
	${\text{ρ}}_{\text{DPR}}$	.699	.043	.799	.037	.900	.027	1.001	.019
	${\text{ρ}}_{\text{DTR}}$	.700	.043	.800	.037	.900	.027	1.002	.019
	${\text{ρ}}_{\text{DCR}}$	.699	.043	.799	.037	.899	.027	1.001	.019
	${\text{ρ}}_{\text{SS}}$	.588	.039	.672	.035	.757	.027	.842	.018
1,000	${\text{ρ}}_{\text{CFA}}$	.699	.023	.800	.018	.899	.014	1.000	.009
	${\text{ρ}}_{\text{DPR}}$	.699	.023	.800	.018	.899	.014	1.000	.009
	${\text{ρ}}_{\text{DTR}}$	.699	.023	.800	.018	.899	.014	1.000	.009
	${\text{ρ}}_{\text{DCR}}$	.699	.023	.800	.018	.899	.014	1.000	.009
	${\text{ρ}}_{\text{SS}}$	.589	.021	.674	.017	.756	.013	.842	.009
3	${\text{ρ}}_{\text{CFA}}$	.699	.043	.799	.038	.899	.028	1.001	.019
	${\text{ρ}}_{\text{DPR}}$	.699	.043	.799	.037	.900	.027	1.001	.019
	${\text{ρ}}_{\text{DTR}}$	.700	.043	.800	.037	.900	.027	1.002	.019
	${\text{ρ}}_{\text{DCR}}$	.699	.043	.799	.037	.899	.027	1.001	.019
	${\text{ρ}}_{\text{SS}}$	.588	.039	.672	.035	.757	.027	.842	.018
6	${\text{ρ}}_{\text{CFA}}$	.699	.039	.799	.030	.900	.019	1.000	.009
	${\text{ρ}}_{\text{DPR}}$	.699	.039	.799	.030	.900	.019	1.000	.009
	${\text{ρ}}_{\text{DTR}}$	.699	.039	.800	.030	.900	.019	1.000	.009
	${\text{ρ}}_{\text{DCR}}$	.699	.039	.799	.030	.900	.019	1.000	.009
	${\text{ρ}}_{\text{SS}}$	.639	.038	.731	.030	.823	.021	.914	.010
9	${\text{ρ}}_{\text{CFA}}$	.699	.038	.798	.028	.899	.016	1.000	.006
	${\text{ρ}}_{\text{DPR}}$	.699	.038	.798	.028	.899	.016	1.000	.006
	${\text{ρ}}_{\text{DTR}}$	.699	.038	.798	.028	.899	.016	1.001	.006
	${\text{ρ}}_{\text{DCR}}$	.699	.038	.798	.028	.898	.016	1.000	.006
	${\text{ρ}}_{\text{SS}}$	.657	.037	.751	.029	.845	.018	.941	.007
0.5, 0.4, 0.4	${\text{ρ}}_{\text{CFA}}$	.709	.156	.802	.161	.915	.157	1.012	.170
	${\text{ρ}}_{\text{DPR}}$	.727	.153	.819	.158	.934	.157	1.028	.168
	${\text{ρ}}_{\text{DTR}}$	.728	.153	.819	.158	.934	.157	1.029	.168
	${\text{ρ}}_{\text{DCR}}$	.711	.152	.802	.156	.917	.155	1.012	.169
	${\text{ρ}}_{\text{SS}}$	.288	.058	.325	.058	.369	.053	.407	.053
0.8, 0.7, 0.7	${\text{ρ}}_{\text{CFA}}$	.699	.051	.803	.042	.900	.035	1.001	.028
	${\text{ρ}}_{\text{DPR}}$	.701	.052	.805	.043	.903	.036	1.004	.029
	${\text{ρ}}_{\text{DTR}}$	.702	.052	.805	.043	.904	.036	1.004	.029
	${\text{ρ}}_{\text{DCR}}$	.699	.051	.802	.043	.900	.036	1.001	.028
	${\text{ρ}}_{\text{SS}}$	.544	.043	.624	.037	.699	.033	.778	.026
0.8, 0.8, 0.8	${\text{ρ}}_{\text{CFA}}$	.699	.043	.799	.038	.899	.028	1.001	.019
	${\text{ρ}}_{\text{DPR}}$	.699	.043	.799	.037	.900	.027	1.001	.019
	${\text{ρ}}_{\text{DTR}}$	.700	.043	.800	.037	.900	.027	1.002	.019
	${\text{ρ}}_{\text{DCR}}$	.699	.043	.799	.037	.899	.027	1.001	.019
	${\text{ρ}}_{\text{SS}}$	.588	.039	.672	.035	.757	.027	.842	.018
0.9, 0.6, 0.3	${\text{ρ}}_{\text{CFA}}$	.697	.062	.800	.052	.901	.047	1.000	.036
	${\text{ρ}}_{\text{DPR}}$	.767	.083	.879	.073	.989	.071	1.101	.065
	${\text{ρ}}_{\text{DTR}}$	.768	.082	.879	.073	.990	.071	1.102	.066
	${\text{ρ}}_{\text{DCR}}$	.698	.072	.799	.062	.901	.058	1.003	.049
	${\text{ρ}}_{\text{SS}}$	.453	.049	.519	.044	.586	.040	.651	.034
0.9, 0.8, 0.8	${\text{ρ}}_{\text{CFA}}$	.700	.043	.797	.032	.902	.023	1.000	.014
	${\text{ρ}}_{\text{DPR}}$	.701	.043	.798	.033	.904	.025	1.002	.015
	${\text{ρ}}_{\text{DTR}}$	.701	.043	.798	.033	.904	.024	1.002	.015
	${\text{ρ}}_{\text{DCR}}$	.700	.043	.796	.033	.902	.024	1.000	.014
	${\text{ρ}}_{\text{SS}}$	.610	.040	.695	.032	.787	.024	.872	.015

Note: M = mean; SD = standard deviation; Techniques: CFA = confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; SS = summed scale without disattenuation.

a. When the sample size was varied, the number of indicators was 3, and all loadings were .8. When the number of indicators was varied, the sample size was 250, and all loadings were .8. When the loadings were varied, the sample size was 250, and the number of indicators was 3.

Table 7.

Coverage and Balance of 95% Confidence Intervals by Loadings and Sample Size.

Loadings	Sample Size	Technique	Factor Correlation
			.7		.8		.9		1.0
			Coverage	Balance	Coverage	Balance	Coverage	Balance	Coverage	Balance
0.8, 0.8, 0.8	50	${\text{ρ}}_{\text{CFA}}$	.905	–.057	.899	–.085	.912	–.068	.955	–.023
		${\text{ρ}}_{\text{DPR}}$	.921	–.037	.922	–.048	.908	–.056	.927	–.045
		${\text{ρ}}_{\text{DTR}}$	.922	–.040	.915	–.059	.908	–.068	.909	–.067
		${\text{ρ}}_{\text{DCR}}$	.927	–.027	.919	–.049	.916	–.046	.933	–.033
	100	${\text{ρ}}_{\text{CFA}}$	.915	–.051	.925	–.055	.936	–.046	.955	–.025
		${\text{ρ}}_{\text{DPR}}$	.917	–.021	.938	–.022	.944	–.030	.940	–.030
		${\text{ρ}}_{\text{DTR}}$	.922	–.030	.932	–.028	.940	–.042	.938	–.038
		${\text{ρ}}_{\text{DCR}}$	.925	–.019	.932	–.016	.943	–.027	.941	–.021
	250	${\text{ρ}}_{\text{CFA}}$	.966	–.016	.938	–.022	.947	–.025	.949	–.015
		${\text{ρ}}_{\text{DPR}}$	.967	–.011	.947	–.011	.949	–.015	.946	–.024
		${\text{ρ}}_{\text{DTR}}$	.968	–.008	.949	–.013	.944	–.022	.942	–.026
		${\text{ρ}}_{\text{DCR}}$	.967	–.005	.948	–.008	.948	–.012	.945	–.017
	1,000	${\text{ρ}}_{\text{CFA}}$	.949	–.001	.955	–.017	.954	–.008	.950	–.002
		${\text{ρ}}_{\text{DPR}}$	.938	.004	.956	–.012	.947	–.007	.947	–.007
		${\text{ρ}}_{\text{DTR}}$	.947	.003	.956	–.010	.952	–.004	.953	–.009
		${\text{ρ}}_{\text{DCR}}$	.946	.006	.956	–.008	.951	–.003	.951	–.005
0.9, 0.6, 0.3	50	${\text{ρ}}_{\text{CFA}}$	.911	–.034	.919	–.030	.923	–.036	.954	.015
		${\text{ρ}}_{\text{DPR}}$	.851	–.143	.859	–.141	.822	–.176	.806	–.194
		${\text{ρ}}_{\text{DTR}}$	.852	–.142	.848	–.150	.810	–.188	.803	–.197
		${\text{ρ}}_{\text{DCR}}$	.964	–.017	.957	–.023	.942	–.035	.952	–.023
	100	${\text{ρ}}_{\text{CFA}}$	.942	–.016	.939	–.005	.949	–.019	.944	.002
		${\text{ρ}}_{\text{DPR}}$	.859	–.135	.843	–.157	.810	–.190	.756	–.244
		${\text{ρ}}_{\text{DTR}}$	.866	–.128	.842	–.158	.803	–.195	.747	–.253
		${\text{ρ}}_{\text{DCR}}$	.956	–.010	.943	–.007	.945	–.017	.940	–.020
	250	${\text{ρ}}_{\text{CFA}}$	.940	–.006	.943	–.017	.937	–.031	.964	–.002
		${\text{ρ}}_{\text{DPR}}$	.832	–.164	.786	–.214	.694	–.306	.581	–.417
		${\text{ρ}}_{\text{DTR}}$	.826	–.170	.781	–.219	.682	–.318	.568	–.430
		${\text{ρ}}_{\text{DCR}}$	.943	–.007	.949	–.005	.937	–.023	.942	–.012
	1,000	${\text{ρ}}_{\text{CFA}}$	.958	.006	.947	–.023	.950	–.008	.947	–.027
		${\text{ρ}}_{\text{DPR}}$	.621	–.379	.453	–.545	.279	–.721	.117	–.883
		${\text{ρ}}_{\text{DTR}}$	.613	–.387	.446	–.552	.264	–.736	.108	–.892
		${\text{ρ}}_{\text{DCR}}$	.955	–.005	.947	–.003	.951	–.017	.945	–.017

Note: Balance is the difference between frequencies for the population value above and below the confidence interval. The number of indicators was always 3. Techniques: CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability.

The first set of rows in Table 6 shows the effects of sample size. Scale score correlation ${\text{ρ}}_{\text{SS}}$ was always negatively biased due to the well-known attenuation effect. All disattenuation techniques and CFA performed better, and in large samples (250, 1,000), their performance was indistinguishable. This result was expected because all these approaches are consistent and their assumptions hold in this set of conditions. In the smallest sample size (50), CFA was slightly biased to be less efficient than the disattenuation-based techniques, but the differences were in the third digit and thus were inconsequential. The number of indicators, shown in the second set of rows in Table 6, affects the bias of the scale score correlation ${\text{ρ}}_{\text{SS}}$ because increasing the number of indicators increases reliability and, consequently, reduces the attenuation effect. The effect for other techniques was an increase in precision, which was expected because more indicators provide more information from which to estimate the correlation.

The third set of rows in Table 6 demonstrates the effects of varying the factor loadings. When the factor loadings were equal (all at .8), the performance of CFA and all disattenuation techniques was identical, which was expected, as explained above. Table 7 shows that in this condition, the confidence intervals of all techniques performed reasonably well. The balance statistics were all negative, indicating that when the population value was outside the CI, it was generally more frequently below the lower limit of the interval than above the upper limit. In other words, all CIs were slightly positively biased. This effect and the general undercoverage of the CIs were most pronounced in small samples. $\text{C}{\text{I}}_{\text{CFA}}$ performed slightly worse than disattenuated correlations when the sample size was very small and the correlation between the factors was weaker. However, outside the smallest sample sizes, the differences were negligible in the third decimals.

When the loadings varied, ${\text{ρ}}_{\text{DTR}}$ and ${\text{ρ}}_{\text{DPR}}$ became positively biased. The reason for this result is that these estimation techniques assume tau-equivalence or equal reliability of the indicators; when this assumption does not hold, the techniques have been shown to produce negatively biased reliability estimates, thus leading to overcorrection for attenuation and producing positive bias. The same results are mirrored in the second set of rows in Table 7; both $\text{C}{\text{I}}_{\text{DPR}}$ and $\text{C}{\text{I}}_{\text{DTR}}$ produced positively biased CIs with poor coverage and balance.

In summary, Table 6 and Table 7 show that the best-performing correlation estimate was ${\text{ρ}}_{\text{CFA}}$ , followed by ${\text{ρ}}_{\text{DCR}}$ and that the corresponding confidence intervals $\text{C}{\text{I}}_{\text{CFA}}$ and $\text{C}{\text{I}}_{\text{DCR}}$ outperform the others. When the condition of being tau-equivalent was violated (e.g., the loadings were .9, .6, and .3), ${\text{ρ}}_{\text{CFA}}$ and ${\text{ρ}}_{\text{DCR}}$ produced estimates that were more accurate than those produced by other methods, but ${\text{ρ}}_{\text{CFA}}$ was slightly more precise, having smaller standard deviations. Similarly, $\text{C}{\text{I}}_{\text{CFA}}$ and $\text{C}{\text{I}}_{\text{DCR}}$ were largely unaffected and retained their performance from the tau-equivalent condition.

Inference Against a Cutoff

Our main results concern inference against a cutoff and are relevant when a researcher wants to make a yes/no decision about discriminant validity. We start by assessing the performance of the techniques that can be thought of as tests of a perfect correlation or as rules of thumb. Because the differences between ${\text{ρ}}_{\text{DPR}}$ (i.e., HTMT) and ${\text{ρ}}_{\text{DTR}}$ were negligible, only the former is reported. Similarly, $\text{C}{\text{I}}_{\text{DTR}}$ is omitted due to nearly identical performance with $\text{C}{\text{I}}_{\text{DPR}}$ .

Table 8 clearly shows that some of the techniques have either unacceptably low power or a false positive rate that is too high to be considered useful. Techniques that directly compare the point estimates of correlations with a cutoff (i.e., ${\text{ρ}}_{\text{CFA}}\left(1\right)$ , ${\text{ρ}}_{\text{DPR}}\left(1\right)$ , and ${\text{ρ}}_{\text{CR}}\left(1\right)$ ) have very high false negative rates because an unbiased and normal correlation estimate can be expected to be below the population value (here, 1) exactly half the time. In contrast, the AVE/SV technique that uses factor correlation following Fornell and Larcker’s (1981a) original proposal has a very high false positive rate. As expected based on our analysis, the misused version using scale score correlation, $\text{AVE}/{\text{SV}}_{\text{SS}}$ has a smaller false positive rate because the attenuation bias in the scale score correlation worked to offset the high false positive rate of the AVE comparison. Clearly, none of these techniques can be recommended.

The various model comparisons and CIs performed better. Among the three methods of model comparison (CFI(1), ${\text{χ}}^2$ (1), and ${\text{χ}}^2\left(\text{merge}\right)$ ), ${\text{χ}}^2$ (1) was generally the best in terms of both the false positive rate and false negative rate. While the difference was small, it is surprising that ${\text{χ}}^2$ (1) was strictly superior to ${\text{χ}}^2\left(\text{merge}\right)$ , having both more power and a smaller false positive rate. Our explanation for this finding is that although ${\text{χ}}^2\left(\text{merge}\right)$ imposes more constraints on the model, these constraints work differently when the factors are perfectly correlated and when they are not. When the factors are perfectly correlated, imposing more constraints means that the model can be declared to misfit in more ways, thus leading to lower power. In contrast, when the correlation between the factors is less than 1, the additional constraints are somewhat redundant because constraining the focal correlation to 1 will also bias all other correlations involving the focal variables. Thus, the amount of misfit produced by the first constraint is greater than the other constraints that ${\text{χ}}^2\left(\text{merge}\right)$ contributes. This phenomenon leads to a higher false positive rate because while the additional constraints contribute degrees of freedom, they contribute less misfit.

Another interesting finding is that although $\text{CFI}\left(1\right)$ was proposed as an alternative to ${\text{χ}}^2\left(1\right)$ based on the assumption that it had a smaller false positive rate, this assumption does not appear to be true: the false positive rates of these techniques were comparable, and in larger samples (250, 1,000), the false positive rate of $\text{CFI}\left(1\right)$ even exceeded that of ${\text{χ}}^2\left(1\right)$ . Generalizing this finding to larger models requires caution because the CFI comparison depends on the fit of the null model, which depends on model size. Nevertheless, it is clear that the CFI comparison does not generally have a smaller false positive rate than ${\text{χ}}^2\left(1\right).$

The performance of the CIs ( ${\text{CI}}_{\text{CFA}}\left(1\right)$ , ${\text{CI}}_{\text{DPR}}\left(1\right)$ , and ${\text{CI}}_{\text{DCR}}\left(1\right)$ ) was nearly identical in the tau-equivalent condition (i.e., all loadings at .8), but in the congeneric condition (i.e., the loadings at .3, .6, and .9), ${\text{CI}}_{\text{DPR}}\left(1\right)$ had an excessive false positive rate due to the positive bias explained earlier. ${\text{CI}}_{\text{DCR}}\left(1\right)$ had a larger false positive rate than ${\text{CI}}_{\text{CFA}}\left(1\right)$ , particularly in small samples, possibly due to violating the large sample assumption of bootstrapping. In summary, Table 8 supports the use of ${\text{CI}}_{\text{CFA}}\left(1\right)$ and ${\text{χ}}^2\left(1\right)$ . The former had slightly more power but a larger false positive rate than the latter. The performance of these two techniques converged in large samples.

Table 8.

Detection Rates of the Discriminant Validity Problem as a Perfect Correlation by Technique.

Sample Size	Technique	Loadings and Factor Correlation
		0.8, 0.8, 0.8				0.9, 0.6, 0.3
		.7*	.8*	.9*	1.0†	.7*	.8*	.9*	1.0†
50	ρCFA(1)	.000	.001	.016	.513	.007	.016	.063	.503
	ρDPR(1)	.000	.001	.016	.530	.051	.110	.258	.770
	ρDCR(1)	.000	.000	.016	.511	.014	.031	.107	.529
	CICFA(1)	.013	.075	.293	.992	.108	.194	.388	.984
	CIDPR(1)	.016	.072	.273	.986	.410	.616	.898	1.000
	CIDCR(1)	.014	.066	.256	.983	.244	.397	.697	.988
	$\text{AVE}/{\text{SV}}_{\text{CFA}}$	.161	.582	.968	1.000	.733	.941	.992	1.000
	$\text{AVE}/{\text{SV}}_{\text{SS}}$	.014	.141	.692	.975	.178	.457	.729	.871
	CFI(1)	.008	.037	.160	.935	.085	.140	.249	.916
	${\text{χ}}^2$ (1)	.006	.043	.199	.967	.110	.185	.332	.968
	${\text{χ}}^2\left(\text{merge}\right)$	.009	.056	.243	.967	.130	.210	.358	.968
100	ρCFA(1)	.000	.000	.002	.520	.000	.001	.028	.510
	ρDPR(1)	.000	.000	.002	.528	.016	.059	.204	.827
	ρDCR(1)	.000	.000	.002	.517	.001	.006	.051	.525
	CICFA(1)	.000	.004	.106	.990	.022	.088	.241	.985
	CIDPR(1)	.000	.003	.105	.984	.241	.386	.773	.999
	CIDCR(1)	.000	.003	.093	.980	.076	.182	.483	.984
	$\text{AVE}/{\text{SV}}_{\text{CFA}}$	.063	.603	.993	1.000	.822	.980	.999	1.000
	$\text{AVE}/{\text{SV}}_{\text{SS}}$	.000	.071	.690	.990	.097	.438	.728	.878
	CFI(1)	.000	.006	.096	.973	.028	.083	.211	.955
	${\text{χ}}^2$ (1)	.000	.002	.081	.973	.024	.086	.234	.972
	${\text{χ}}^2\left(\text{merge}\right)$	.000	.005	.105	.970	.038	.114	.263	.977
250	ρCFA(1)	.000	.000	.000	.519	.000	.000	.006	.505
	${\text{ρ}}_{\text{DPR}}\left(1\right)$	.000	.000	.000	.527	.001	.016	.148	.920
	ρDCR(1)	.000	.000	.000	.515	.000	.001	.017	.509
	CICFA(1)	.000	.000	.005	.983	.000	.005	.124	.984
	CIDPR(1)	.000	.000	.005	.984	.074	.225	.516	.999
	CIDCR(1)	.000	.000	.005	.982	.002	.030	.204	.980
	$\text{AVE}/{\text{SV}}_{\text{CFA}}$	.009	.584	1.000	1.000	.911	.999	1.000	1.000
	$\text{AVE}/{\text{SV}}_{\text{SS}}$	.000	.010	.671	.999	.028	.426	.697	.886
	CFI(1)	.000	.000	.019	.994	.001	.020	.133	.987
	${\text{χ}}^2$ (1)	.000	.000	.004	.974	.000	.006	.123	.976
	${\text{χ}}^2\left(\text{merge}\right)$	.000	.000	.008	.972	.000	.014	.152	.976
1,000	ρCFA(1)	.000	.000	.000	.491	.000	.000	.000	.510
	ρDPR(1)	.000	.000	.000	.495	.000	.001	.106	.997
	ρDCR(1)	.000	.000	.000	.488	.000	.000	.000	.518
	${\text{CI}}_{\text{CFA}}\left(1\right)$	.000	.000	.000	.983	.000	.000	.001	.985
	CIDPR(1)	.000	.000	.000	.981	.000	.028	.320	1.000
	CIDCR(1)	.000	.000	.000	.980	.000	.000	.015	.979
	$\text{AVE}/{\text{SV}}_{\text{CFA}}$	.000	.583	1.000	1.000	.988	1.000	1.000	1.000
	$\text{AVE}/{\text{SV}}_{\text{SS}}$	.000	.000	.666	1.000	.001	.396	.667	.900
	CFI(1)	.000	.000	.000	1.000	.000	.000	.017	1.000
	${\text{χ}}^2$ (1)	.000	.000	.000	.979	.000	.000	.001	.981
	${\text{χ}}^2\left(\text{merge}\right)$	.000	.000	.000	.978	.000	.000	.003	.979

Note: Averages over all indicator numbers. Techniques: $\text{ρ}$ = comparing the correlations against a cutoff; CI = comparing if a cutoff is included in the 95% confidence interval; $\text{AVE}/\text{SV}$ = comparing the AVE statistics against the squared correlation; CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; CFI = nested model comparison using the CFI rule; ${\text{χ}}^2$ = nested model test; (1) = two factors are constrained to be perfectly correlated; (merge) = two factors are merged into one.

* False positive rate. ^†true positive rate = (1 – false negative rate).

Table 9 considers cutoffs other than 1, using values of .85, .90, and .95 that are sometimes recommended in the literature, showing results that are consistent with those of the previous tables. The techniques that compared the CI against a cutoff (i.e., ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ , ${\text{CI}}_{\text{DPR}}\left(\text{cut}\right)$ and ${\text{CI}}_{\text{DCR}}\left(1\right)$ ) had more power than those that compared an estimate against a cutoff (i.e., ${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$ , ${\text{ρ}}_{\text{DPR}}\left(\text{cut}\right)$ , and ${\text{ρ}}_{\text{DCR}}\left(\text{cut}\right)$ ), especially in small samples, and between the three, ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ showed a lower false positive rate while having power similar to that of ${\text{CI}}_{\text{DPR}}\left(\text{cut}\right)$ and to a smaller extent ${\text{CI}}_{\text{DCR}}\left(\text{cut}\right)$ . Among the techniques that compared model fit (i.e., $\text{CFI}\left(\text{cut}\right)$ and ${\text{χ}}^2\left(\text{cut}\right)$ ), $\text{CFI}\left(\text{cut}\right)$ had slightly more power but also a higher false positive rate in small samples than ${\text{χ}}^2\left(\text{cut}\right)$ . In larger samples, the power of the two techniques was similar, but ${\text{χ}}^2\left(\text{cut}\right)$ generally had the lowest false positive rate. This finding and the sensitivity of the CFI tests to model size, explained earlier, make ${\text{χ}}^2\left(\text{cut}\right)$ the preferred alternative of the two. In summary, ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ and ${\text{χ}}^2\left(\text{cut}\right)$ are generally the best techniques. They have different strengths: ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ has slightly more power, but ${\text{χ}}^2\left(\text{cut}\right)$ enjoys a considerably lower false positive rate.

Table 9.

Detection Rates by Technique Using Alternative Cutoffs.

Sample Size	Technique	Cutoff and Factor Correlation
		.85				.9				.95
		.7*	.8*	.9†	1.0†	.7*	.8*	.9†	1.0†	.7*	.8*	.9*	1.0†
50	${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$	.085	.289	.784	.950	.049	.126	.517	.925	.033	.064	.193	.866
	${\text{ρ}}_{\text{DPR}}\left(\text{cut}\right)$	.121	.348	.818	.968	.075	.182	.576	.946	.051	.101	.270	.892
	${\text{ρ}}_{\text{DCR}}\left(\text{cut}\right)$	.095	.301	.785	.958	.056	.140	.524	.930	.036	.073	.212	.857
	${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$	.694	.945	.992	.996	.413	.788	.982	.995	.216	.439	.895	.992
	${\text{CI}}_{\text{DPR}}\left(\text{cut}\right)$	.695	.940	.999	1.000	.477	.777	.990	1.000	.331	.522	.884	.999
	${\text{CI}}_{\text{DCR}}\left(\text{cut}\right)$	.669	.929	.999	1.000	.441	.746	.988	1.000	.305	.476	.858	.999
	$\text{CFI}\left(\text{cut}\right)$	.416	.781	.971	.992	.227	.509	.915	.987	.146	.259	.649	.977
	${\text{χ}}^2\left(\text{cut}\right)$	.541	.884	.992	.998	.301	.629	.967	.997	.192	.333	.760	.995
100	${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$	.041	.194	.850	.973	.023	.068	.510	.953	.015	.034	.126	.907
	${\text{ρ}}_{\text{DPR}}\left(\text{cut}\right)$	.066	.264	.878	.980	.037	.110	.582	.963	.022	.061	.206	.925
	${\text{ρ}}_{\text{DCR}}\left(\text{cut}\right)$	.044	.211	.846	.975	.022	.072	.518	.952	.014	.036	.141	.897
	${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$	.396	.891	.996	.999	.189	.525	.987	.998	.115	.230	.734	.996
	${\text{CI}}_{\text{DPR}}\left(\text{cut}\right)$	.455	.888	.999	1.000	.279	.577	.989	1.000	.198	.337	.760	1.000
	${\text{CI}}_{\text{DCR}}\left(\text{cut}\right)$	.410	.869	.999	1.000	.236	.528	.986	1.000	.166	.280	.721	.999
	$\text{CFI}\left(\text{cut}\right)$	.317	.793	.987	.998	.156	.417	.952	.996	.096	.188	.617	.990
	${\text{χ}}^2\left(\text{cut}\right)$	.314	.818	.997	1.000	.169	.419	.974	.999	.114	.209	.614	.997
250	${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$	.016	.102	.927	.989	.008	.029	.513	.977	.004	.014	.071	.945
	${\text{ρ}}_{\text{DPR}}\left(\text{cut}\right)$	.027	.169	.942	.991	.013	.060	.589	.980	.006	.025	.143	.956
	${\text{ρ}}_{\text{DCR}}\left(\text{cut}\right)$	.015	.110	.919	.989	.008	.030	.508	.977	.004	.014	.077	.939
	${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$	.145	.704	.998	1.000	.079	.233	.985	1.000	.049	.114	.426	.998
	${\text{CI}}_{\text{DPR}}\left(\text{cut}\right)$	.224	.743	.999	1.000	.139	.336	.987	1.000	.095	.192	.540	.999
	${\text{CI}}_{\text{DCR}}\left(\text{cut}\right)$	.168	.698	.999	1.000	.097	.262	.981	1.000	.064	.137	.468	.999
	$\text{CFI}\left(\text{cut}\right)$	.208	.819	.996	.999	.093	.321	.981	.998	.063	.117	.571	.996
	${\text{χ}}^2\left(\text{cut}\right)$	.134	.635	.998	1.000	.078	.210	.975	1.000	.053	.111	.381	.999
1,000	${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$	.002	.030	.976	.999	.000	.006	.505	.995	.000	.002	.025	.980
	${\text{ρ}}_{\text{DPR}}\left(\text{cut}\right)$	.003	.095	.980	.999	.000	.025	.622	.997	.000	.003	.093	.983
	${\text{ρ}}_{\text{DCR}}\left(\text{cut}\right)$	.002	.033	.975	.999	.000	.006	.507	.995	.000	.002	.026	.979
	${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$	.032	.253	1.000	1.000	.016	.071	.980	1.000	.006	.033	.174	1.000
	${\text{CI}}_{\text{DPR}}\left(\text{cut}\right)$	.069	.378	1.000	1.000	.028	.146	.984	1.000	.011	.074	.309	1.000
	${\text{CI}}_{\text{DCR}}\left(\text{cut}\right)$	.036	.279	1.000	1.000	.020	.081	.979	1.000	.008	.038	.198	1.000
	$\text{CFI}\left(\text{cut}\right)$	.063	.846	.999	1.000	.027	.132	.995	1.000	.013	.060	.400	1.000
	${\text{χ}}^2\left(\text{cut}\right)$	.033	.238	1.000	1.000	.018	.071	.975	1.000	.007	.035	.170	1.000

Note: Averages over all indicator numbers and loading. Techniques: $\text{ρ}$ = comparing correlation against a cutoff; CI = comparing if a cutoff is included in the 95% confidence interval, CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; CFI = nested model comparison using the CFI rule; ${\text{χ}}^2$ = nested model test.

* False positive rate. ^†true positive rate = (1 – false negative rate).

Effects of Model Misspecification

We now turn to the cross-loading conditions to assess the robustness of the techniques when the assumption of no cross-loadings is violated. The results shown in Table 10 show that all estimates become biased toward 1. ${\text{ρ}}_{\text{DCR}}$ was slightly most robust to these misspecifications, but the differences between the techniques were not large. Table 11 presented the detection rates of different techniques using alterative cutoffs and over the cross-loading conditions and showed similar results. All techniques were again affected, and both the power and false positive rates increased across the board when the correlation between the factors was less than one. As before, the effect was stronger for smaller population correlations.

Table 10.

Mean Correlation Estimate Under Model Misspecification.

Cross-Loading Estimation	Technique	Population Cross-Loadings and Factor Correlation
		No Cross-Loadings				1 Cross-Loading				2 Cross-Loadings
		.7	.8	.9	1.0	.7	.8	.9	1.0	.7	.8	.9	1.0
Assumed zero	${\text{ρ}}_{\text{CFA}}$	.699	.800	.901	1.001	.819	.878	.938	1.002	.885	.921	.957	1.000
	${\text{ρ}}_{\text{DPR}}$	.707	.808	.911	1.012	.808	.876	.943	1.011	.879	.922	.962	1.008
	${\text{ρ}}_{\text{DTR}}$	.708	.809	.911	1.013	.808	.876	.943	1.012	.879	.922	.963	1.008
	${\text{ρ}}_{\text{DCR}}$	.700	.800	.901	1.002	.801	.868	.935	1.002	.871	.914	.955	1.000
	${\text{ρ}}_{\text{SS}}$	.560	.640	.720	.801	.633	.692	.749	.801	.700	.740	.777	.805
Estimated	${\text{ρ}}_{\text{CFA}}$					.700	.801	.903	1.009	.694	.790	.894	1.021

Note: Averages the correlations over the conditions shown in Table 6. Techniques: CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; SS = summed scale without disattenuation.

Table 11.

Detection Rates by Technique Using Alternative Cutoffs Under Model Misspecification.

Cross-Loadings and Whether Estimated	Technique	Cutoff and Factor Correlation
		.85				.9				.95
		.7*	.8*	.9†	1.0†	.7*	.8*	.9†	1.0†	.7*	.8*	.9*	1.0†
No cross-loadingsAssumed zero	${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$	.034	.152	.886	.980	.018	.055	.511	.965	.011	.026	.101	.927
	${\text{ρ}}_{\text{DPR}}\left(\text{cut}\right)$	.052	.216	.906	.986	.029	.091	.592	.973	.018	.045	.175	.941
	${\text{ρ}}_{\text{DCR}}\left(\text{cut}\right)$	.037	.161	.883	.982	.019	.060	.514	.966	.012	.029	.111	.920
	${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$	.313	.697	.997	.999	.169	.400	.984	.998	.091	.199	.555	.997
	${\text{CI}}_{\text{DPR}}\left(\text{cut}\right)$	.356	.735	.999	1.000	.225	.455	.988	1.000	.152	.276	.620	1.000
	${\text{CI}}_{\text{DCR}}\left(\text{cut}\right)$	.315	.691	.999	1.000	.192	.400	.983	1.000	.129	.227	.558	.999
	$\text{CFI}\left(\text{cut}\right)$	.248	.810	.989	.998	.121	.341	.961	.996	.075	.151	.557	.991
	${\text{χ}}^2\left(\text{cut}\right)$	.251	.641	.997	.999	.137	.328	.973	.999	.086	.167	.478	.998
1 cross-loadingAssumed zero	${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$	.445	.837	.958	.980	.168	.442	.895	.965	.038	.098	.450	.925
	${\text{ρ}}_{\text{DPR}}\left(\text{cut}\right)$	.253	.757	.962	.986	.109	.297	.883	.974	.051	.115	.368	.938
	${\text{ρ}}_{\text{DCR}}\left(\text{cut}\right)$	.194	.739	.954	.982	.067	.229	.863	.965	.029	.067	.290	.919
	${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$	.900	.996	.999	.999	.621	.917	.997	.999	.311	.541	.940	.997
	${\text{CI}}_{\text{DPR}}\left(\text{cut}\right)$	.732	.996	1.000	1.000	.469	.787	.999	1.000	.288	.462	.867	.999
	${\text{CI}}_{\text{DCR}}\left(\text{cut}\right)$	.727	.996	1.000	1.000	.425	.781	.999	1.000	.237	.408	.859	.999
	$\text{CFI}\left(\text{cut}\right)$	.912	.985	.995	.998	.596	.930	.989	.996	.214	.482	.946	.991
	${\text{χ}}^2\left(\text{cut}\right)$	.880	.995	.999	1.000	.563	.895	.997	.999	.254	.469	.918	.998
1 cross-loadingEstimated	${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$	.058	.191	.848	.962	.037	.088	.518	.938	.025	.051	.160	.883
	${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$	.385	.750	.994	.998	.245	.488	.978	.996	.155	.287	.641	.994
	$\text{CFI}\left(\text{cut}\right)$	.367	.871	.986	.996	.216	.498	.964	.993	.147	.267	.745	.987
	${\text{χ}}^2\left(\text{cut}\right)$	.342	.711	.996	.999	.221	.433	.972	.998	.157	.264	.591	.995
2 cross-loadingsAssumed zero	${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$	.855	.943	.970	.977	.411	.819	.936	.961	.067	.203	.740	.918
	${\text{ρ}}_{\text{DPR}}\left(\text{cut}\right)$	.779	.940	.975	.983	.307	.761	.941	.969	.110	.230	.709	.932
	${\text{ρ}}_{\text{DCR}}\left(\text{cut}\right)$	.760	.930	.970	.979	.235	.733	.929	.961	.057	.143	.663	.914
	${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$	.996	.998	.999	.999	.938	.996	.998	.998	.492	.773	.992	.996
	${\text{CI}}_{\text{DPR}}\left(\text{cut}\right)$	.997	1.000	1.000	1.000	.803	.996	.999	1.000	.459	.693	.995	.999
	${\text{CI}}_{\text{DCR}}\left(\text{cut}\right)$	.996	1.000	1.000	1.000	.793	.995	1.000	1.000	.397	.660	.993	.999
	$\text{CFI}\left(\text{cut}\right)$	.986	.994	.996	.997	.941	.984	.992	.995	.434	.858	.976	.990
	${\text{χ}}^2\left(\text{cut}\right)$	.996	.998	.999	.999	.916	.994	.998	.998	.415	.720	.989	.996
2 cross-loadingsEstimated	${\text{ρ}}_{\text{CFA}}\left(\text{cut}\right)$	.105	.257	.791	.922	.071	.150	.530	.895	.045	.091	.239	.837
	${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$	.510	.814	.987	.993	.384	.614	.973	.990	.291	.444	.752	.987
	$\text{CFI}\left(\text{cut}\right)$	.565	.921	.983	.992	.408	.704	.968	.988	.303	.466	.885	.980
	${\text{χ}}^2\left(\text{cut}\right)$	.482	.790	.992	.997	.361	.573	.971	.995	.276	.408	.709	.990

Note: Averages over all conditions. Techniques: $\text{ρ}$ = comparing the correlations against a cutoff; CI = comparing if a cutoff is included in the 95% confidence interval; CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; CFI = nested model comparison using the CFI rule; ${\text{χ}}^2$ = nested model test.

* False positive rate. ^†true positive rate = (1 – false negative rate).

In the cross-loading conditions, we also estimated a correctly specified CFA model in which the cross-loadings were estimated. Table 10 shows that the mean estimate was largely unaffected, but the variance of the estimates (not reported in the table) increased because of the increased model complexity. This effect is seen in Table 11, where the pattern of results for CFA models was largely similar between the cross-loading conditions, but the presence of cross-loadings increased the false positive rate. This result is easiest to understand in the context of ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ ; when estimates became less precise, this also widened the confidence intervals and, consequently, increased the frequency of results where the cutoff fell within the interval.

The cross-loading results underline the importance of observing the assumptions of the techniques and that not doing so may lead to incorrect inference. However, two conclusions that are new to discriminant validity literature can be drawn: First, the lack of cross-loadings in the population (i.e., factorial validity) is not a strict prerequisite for discriminant validity assessment as long as the cross-loadings are modeled appropriately. Second, while the lack of factorial validity can lead scale-item pairs to have complete lack of discriminant validity (see Equation 2), this does not always invalidate scale-level discriminant validity (see Equation 3) as long as this is properly modeled.

Discussion

The original meaning of the term “discriminant validity” was tied to MTMM matrices, but the term has since evolved to mean a lack of a perfect or excessively high correlation between two measures after considering measurement error. We provided a comprehensive review of the various discriminant validity techniques and presented a simulation study assessing their effectiveness. There are several issues that warrant discussion.

Contradictions With Prior Studies

Our simulation results clearly contradict two important conclusions drawn in the recent discriminant validity literature, and these contradictions warrant explanations. First, Henseler et al. (2015) and Voorhees et al. (2016) strongly recommend ${\text{ρ}}_{\text{DPR}}$ (HTMT) for discriminant validity assessment. We prove that the $\text{HTMT}$ index is simply a scale score correlation disattenuated with parallel reliability (i.e., the standardized alpha) and thus should not be expected to outperform modern CFA techniques, which our simulation demonstrates. The different conclusions are due to the limitations of these prior studies. While Henseler et al. (2015) explain that ${\text{ρ}}_{\text{DPR}}$ is a factor correlation estimate, they do not compare it against other factor correlation estimation techniques. Thus, their results do not indicate the superiority of ${\text{ρ}}_{\text{DPR}}$ but simply indicate that AVE/SV, which was their main comparison, performs very poorly. The follow-up study by Voorhees et al. (2016) considered a broader set of techniques, including ${\text{CI}}_{\text{CFA}}\left(1\right)$ and ${\text{χ}}^2\left(1\right)$ . The problem in their study was that the different techniques were applied using different cutoffs: ${\text{ρ}}_{\text{DPR}}$ was used with cutoffs of .80, .85, and .90, whereas the other techniques always used the cutoff of 1 and were thus predestined to fail in a study where a correlation of .90 was used as a discriminant validity problem condition. A more complete study would have used the same cutoffs (.80, .85, and .90) that were applied to ${\text{ρ}}_{\text{DPR}}$ with ${\text{χ}}^2$ and ${\text{CI}}_{\text{CFA}}$ as well, as we did.

Second, our results also challenge J. A. Shaffer et al.’s (2016) recommendation that discriminant validity should be tested by a CFI comparison between two nested models (CFI(1)). The reason for this contradiction is that their recommendation is based on the assumption that if the CFI difference works well for measurement invariance assessment, it should also work well when discriminant validity assessment. The assumption appears to be invalid as it ignores an important difference between these uses: Whereas the degrees of freedom of an invariance test scale roughly linearly with the number of indicators, the degrees of freedom in CFI(1) are always one.

Magnitude of the Discriminant Validity Correlations

In the discriminant validity literature, high correlations between scales or scale items are considered problematic. However, the literature generally has not addressed what is high enough beyond giving rule of thumb cutoffs (e.g., 85). Our definition of discriminant validity suggests that the magnitude of the estimated correlation depends on the correlation between the constructs, the measurement process, and the particular sample, each of which has different implications on what level should be considered high. To warn against mechanical use, we present a scenario where high correlation does not invalidate measurement and a scenario where low correlation between measures does not mean that they measure distinct constructs.

A large correlation does not always mean a discriminant validity problem if one is expected based on theory or prior empirical observations. For example, the correlation between biological sex and gender identity can exceed .99 in the population. ¹⁷ However, both variables are clearly distinct: sex is a biological property with clear observable markers, whereas gender identity is a psychological construct. These two variables also have different causes and consequences (American Psychological Association, 2015), so studies that attempt to measure both can lead to useful policy implications. In cases such as this where the constructs are well defined, large correlations should be tolerated when expected based on theory and prior empirical results. Of course, large samples and precise measurement would be required to ensure that the constructs can be distinguished empirically (i.e., are empirically distinct).

A small or moderate correlation (after correcting for measurement error) does not always mean that two measures measure concepts that are distinct. For example, consider two thermometers that measure the same temperature, yet one is limited to measuring only temperatures above freezing, whereas the other can measure only temperatures below freezing. While both measure the same quantity, they are correlated only by approximately .45 because the temperature would always be out of the range of one of the thermometers that would consequently display zero centigrade. ¹⁸ In the social sciences, a well-known example is the measurement of happiness and sadness, two constructs that can be thought of as opposite poles of mood (D. P. Green et al., 1993; Tay & Jebb, 2018). Consequently, any evaluation of the discriminant validity of scales measuring two related constructs must precede the theoretical consideration of the existence of a common continuum. If this is the case, the typical discriminant validity assessment techniques that are the focus of our article are not directly applicable, but other techniques are needed (Tay & Jebb, 2018).

As the two examples show, a moderately small correlation between measures does not always imply that two constructs are distinct, and a high correlation does not imply that they are not. Like any validity assessment, discriminant validity assessment requires consideration of context, possibly relevant theory, and empirical results and cannot be reduced to a simple statistical test and a cutoff no matter how sophisticated. These considerations highlight the usefulness of the continuous interpretation of discriminant validity evidence.

On Choosing a Technique and a Cutoff

While a general set of statistics and cutoffs that is applicable to all research scenarios cannot exist, we believe that establishing some standards is useful. Based on our study, ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ and ${\text{χ}}^2\left(\text{cut}\right)$ appear to be the leading techniques, but recommending one over another solely on a statistical basis is difficult due to the similar performance of the techniques. However, we recommend ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ for practical reasons. First, ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ is less likely to be misused than ${\text{χ}}^2\left(\text{cut}\right)$ . While we found some evidence of misapplication of ${\text{χ}}^2\left(\text{cut}\right)$ due to incorrect factor scaling, we did not see any evidence of the same when factor correlations were evaluated; these can be obtained postestimation simply by requesting standardized estimates from the software. Second, ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ makes it easier to transition from testing of discriminant validity to its evaluation because the focal statistic is a correlation, which organizational researchers routinely interpret in other contexts. In sum, ${\text{CI}}_{\text{CFA}}\left(\text{cut}\right)$ is simpler to implement, easier to understand, and less likely to be misapplied. ${\text{χ}}^2\left(\text{cut}\right)$ is slightly more accurate, but considering that even the simpler ${\text{χ}}^2\left(1\right)$ is often misapplied, we do not think that the potential precision gained by using ${\text{χ}}^2\left(\text{cut}\right)$ is worth the cost of risking misapplication.

Equally important to choosing a technique is the choice of a cutoff if the technique requires one. Of the recent simulation studies, Henseler et al. (2015) suggested cutoffs of .85 and .9 based on prior literature (e.g., Kline, 2011). Voorhees et al. (2016) considered the false positive and false negative rates of the techniques used in their study and concluded that the cutoff of .85 had the best balance of high power and an acceptable Type I error rate. However, as explained in Online Supplement 5, such conclusions are to a large part simply artifacts of the simulation design. In sum, it seems that deriving an ideal cutoff through simulation results is meaningless and must be established by consensus among the field.

Empirical studies seem to agree that correlations greater than .9 indicate a problem and that correlations less than .8 indicate the lack of a problem. For example, Le et al. (2010) diagnosed a discriminant validity problem between job satisfaction and organizational commitment based on a correlation of .91, and Mathieu and Farr (1991) declared no problem of discriminant validity between the same variables on the basis of a correlation of .78. However, mixed judgments were made about the correlation values between .8 and .9. Lucas et al. (1996) acknowledged discriminant validity between self-esteem and optimism based on ${\text{ρ}}_{DTR}$ of .83 ( ${\text{ρ}}_{SS}=.72$ ), but Credé et al. (2017) criticized the conceptual redundancy between grit and conscientiousness based on a disattenuated correlation of .84 ( ${\text{ρ}}_{SS}=.66$ ). Thus, if there is a threshold, it is likely to fall between .8 and .9.

Sources of Error Other Than the Random Error and Item-Specific Factor Error

Our focus on the common scenario of single-method and one-time measurements limits the researcher to techniques that operationalize measurement error as item uniqueness either directly (e.g., CFA) or indirectly (e.g., ${\text{ρ}}_{\text{DTR}}$ ). However, several articles point out that this may be a simplistic view of measurement error that only considers random and item-specific factor errors, ignoring time-specific transient errors (Le et al., 2009; Woehr et al., 2012). Because transient error can correlate between items and scales, they can either inflate or attenuate correlation estimates calculated using single-administration reliability estimates. While there is little that can be done about this issue if one-time measures are used, researchers should be aware of this limitation. Of course, if multiple measurement occasions are possible, the CFA-based techniques can also be used to model these other sources of error (Le et al., 2009; Woehr et al., 2012), and these more complex models can then be applied with the guideline that we present next.

A Guideline for Assessing Discriminant Validity

From a Cutoff to a Classification System

While many articles about discriminant validity consider it as a matter of degree (e.g., “the extent to…”) instead of a yes/no issue (e.g., “whether…”), most guidelines on evaluation techniques, including Campbell and Fiske’s (1959) original proposal, focus on making a dichotomous judgment as to whether a study has a discriminant validity problem (B in Figure 6). This inconsistency might be an outcome of researchers favoring cutoffs for their simplicity, or it may reflect the fact that after calculating a discriminant validity statistic, researchers must decide whether further analysis and interpretation is required. This practice also fits the dictionary meaning of evaluation, which is not simply calculating and reporting a number but rather dividing the object into several qualitative categories based on the number, thus requiring the use of cutoffs, although these do not need to be the same in every study.

To move the field toward discriminant validity evaluation, we propose a system consisting of several cutoffs instead of a single one. Discriminant validity is a continuous function of correlation values (C in Figure 6), but because of practical needs, correlations are classified into discrete categories indicating different degrees of a problem (D in Figure 6). Similar classifications are used in other fields to characterize essentially continuous phenomena: Consider a doctor’s diagnosis of hypertension. In the past, everyone was divided into two categories of normal and patient, but now hypertension is classified into several levels. That is, patients with hypertension are further subdivided into three stages according to their blood pressure level, and each level is associated with different treatments.

Figure 6.

Relationship between correlation values and the problem of discriminant validity. (A) Linear model (implied by existing definitions), (B) Dichotomous model (existing techniques), (C) Threshold model (implied by the definition of this study), (D) Step model (proposed evaluation technique).

Table 12 shows the classification system we propose. We emphasize that these are guideline that can be adjusted case-by-case if warranted by theoretical understanding of the two constructs and measures, not strict rules that should always be followed. Based on our review, correlations below .8 were seldom considered problematic, and this is thus used as the cutoff for the first class, “No problem,” which strictly speaking is not a proof of no problem, just no evidence of a problem. When correlations fall into this class, researchers can simply declare that they did not find any evidence of a discriminant validity problem. The next three steps are referred to as Marginal, Moderate, and Severe problems, respectively. The Severe problem is the most straightforward: two items or scales cannot be distinguished empirically, and researchers should rethink their concept definitions, measurement, or both. In empirical applications, the correlation level of .9 was nearly universally interpreted as a problem, and we therefore use this level as a cutoff between the Marginal and Moderate cases. In both cases, the high correlation should be acknowledged, and its possible cause should be discussed. In the Marginal case, the interpretation of the scales as representations of distinct constructs is probably safe. In the Moderate case, additional evidence from prior studies using the same constructs and/or measures should be checked before interpretation of the results to ensure that the high correlation is not a systematic problem with the constructs or scales.

Table 12.

Proposed Classification and Cutoffs.

Classification	CICFA (sys)	${\text{χ}}^2$ (sys)
Severe problem	$1\le \text{UL}$	${\text{χ}}_1^2-{\text{χ}}_{org}^2<3.84$
Moderate problem	$.9\le \text{UL}<1$	Not “Marginal problem” AND ${\text{χ}}_1^2-{\text{χ}}_{org}^2>3.84$
Marginal problem	$.8\le \text{UL}<.9$	Not “No problem” AND ${\text{χ}}_{.9}^2-{\text{χ}}_{org}^2>3.84$
No problem	$\text{UL}<.8$	${\text{ρ}}_{\text{CFA}}<.8$ AND ${\text{χ}}_{.8}^2-{\text{χ}}_{org}^2>3.84$

Note: ${\text{ρ}}_{\text{CFA}}$ is the correlation obtained using CFA, $\text{UL}$ is the 95% upper limit of ${\text{ρ}}_{\text{CFA}}$ when ${\text{ρ}}_{\text{CFA}}>0$ , and the absolute value of the 95% lower limit of ${\text{ρ}}_{\text{CFA}}$ when ${\text{ρ}}_{\text{CFA}}<0$ , ${\text{χ}}_{org}^2$ is the chi-square value of the original model, and ${\text{χ}}_c^2$ is the chi-square value of the comparison model where the focal correlation is fixed to c when ${\text{ρ}}_{\text{CFA}}>0$ and $-\text{c}$ when ${\text{ρ}}_{\text{CFA}}<0$ .

How to Implement the Proposed Techniques

The proposed classification system should be applied with CI_CFA(cut) and ${\text{χ}}^2\left(\text{cut}\right)$ , and we propose that these workflows be referred to as CI_CFA(sys) and ${\text{χ}}^2\left(\text{sys}\right)$ , respectively. Both workflows start by estimating a CFA model that includes all scales that are evaluated for discriminant validity. Instead of using the default scale setting option to fix the first factor loadings to 1, scale the latent variables by fixing their variances to 1 (A in Figure 2); this should be explicitly reported in the article. The covariances between factors obtained in the latter way equal the correlations; alternatively, when using CI_CFA(sys), the standardized factor solution can be inspected. Next, inspect the upper limits (lower limits for negative correlations) of the 95% CIs of the estimated factor correlations and compare their values against the cutoffs in Table 12. ¹⁹

Implementing ${\text{χ}}^2\left(\text{sys}\right)$ requires testing every correlation against the lower limit of each class in the classification system. A correlation belongs to the highest class that it is not statistically significantly different from. If the model tests cannot be automated, we suggest the following alternative workflow. First, exclude all correlation pairs whose upper limit of the CI is less than .80. Second, for the remaining correlations, determine the initial class for each by comparing correlation estimates against the cutoffs in Table 12. For example, a correlation of .87 would be classified as Marginal. Third, use ${\text{χ}}^2\left(\text{cut}\right)$ to compare the estimated model against a model where the correlation is constrained to the high cutoff, .9 in the example, using a nested model ${\text{χ}}^2$ test. If significantly different, the correlation is classified into the current section. If the correlation is not significantly different, repeat the model comparison by selecting the high cutoff for the next higher section (in this case 1).

Online Supplement 4 provides a tutorial on how to implement the techniques described in this article using AMOS, LISREL, Mplus, R, and Stata. Because implementing a sequence of comparisons is cumbersome and prone to mistakes, we have contributed a function that automates the ${\text{χ}}^2\left(\text{cut}\right)$ tests to the semTools R package (Jorgensen et al., 2020). For users of other software, we developed MQAssessor, ²⁰ a Python-based open-source application.

What to Do When Discriminant Validity Fails?

If problematically high correlations are observed, their sources must be identified. We propose a three-step process: First, suspect conceptual redundancy. We suggest starting by following the guidelines by J. A. Shaffer et al. (2016) and Podsakoff et al. (2016) for assessing the conceptual distinctiveness of the constructs (see also M. S. Krause, 2012). If two constructs are found to overlap conceptually, researchers should seriously consider dropping one of the constructs to avoid the confusion caused by using two different labels for the same concept or phenomenon (J. A. Shaffer et al., 2016).

Second, scrutinize the measurement model. An unexpectedly high correlation estimate can indicate a failure of model assumptions, as demonstrated by our results of misspecified models. Check the ${\text{χ}}^2$ test for an exact fit of the CFA model. If this test fails, diagnose the model with residuals and/or modification indices to understand the source of misspecification (Kline, 2011, chap. 8). If the model is modified based on these considerations, the wording of the items that led to these decisions should be explicitly reported, and how the item wordings justify the modifications should be explained to reduce the risk of data mining.

Third, collect different data. If conceptual overlap and measurement model issues have been ruled out, the discriminant validity problem can be reduced to a multicollinearity problem. For example, if one wants to study the effects of hair color and gender on intelligence but samples only blonde men and dark-haired women, hair color and gender are not empirically distinguishable, although they are both conceptually distinct and virtually uncorrelated in the broader population. This can occur either because of a systematic error in the sampling design or due to chance in small samples. If a systematic error can be ruled out, the most effective remedy is to collect more data. Alternatively, the data can be used as such, in which case large standard errors will indicate that little can be said about the relative effects of the two variables, or the two variables can be combined as an index (Wooldridge, 2013, pp. 94–98). If a researcher chooses to interpret results, he or she should clearly explain why the large correlation between the latent variables (e.g., >.9) is not a problem in the particular study.

Reporting

We also provide a few guidelines for improved reporting. First, researchers should clearly indicate what they are assessing when assessing discriminant validity by stating, for example, that “We addressed discriminant validity (whether two scales are empirically distinct).” Second, the correlation tables, which are ubiquitous in organizational research, are in most cases calculated with scale scores or other observed variables. However, most studies use only the lower triangle of the table, leaving the other half empty (AMJ 93.6%, JAP 83.1%). This practice is a waste of scarce resources, and we suggest that this space should be used for the latent correlation estimates, which serve as continuous discriminant validity evidence. Third, if nested model comparisons (e.g., ${\text{χ}}^2\left(1\right)$ ) are used, researchers should explicitly report that the model was rescaled from the default option by stating, for example, “We used the ${\text{χ}}^2$ nested model comparison for assessing discriminant validity by comparing our CFA model against models that were more constrained, where all factor loadings were freely estimated, the factor variances were constrained to 1 and each factor correlation was constrained to 1 one at a time.” These reporting practices should considerably reduce the ambiguity in the literature and prevent the common misapplication of the ${\text{χ}}^2\left(1\right)$ test.

Concluding Remarks

There is no shortage of various statistical techniques for evaluating discriminant validity. Such an abundance of techniques is positive if techniques have different advantages, and they are purposefully selected based on their fit with the research scenario. The current state of the discriminant validity literature and research practice suggests that this is not the case. Instead, it appears that many of the techniques have been introduced without sufficient testing and, consequently, are applied haphazardly. Because direct criticism of existing techniques is often avoided, there appears to be a tendency in which new techniques continue to be added without clarifying the problems of previously used techniques. This technique proliferation causes confusion and misuse. This study draws an unambiguous conclusion about which method is best for assessing discriminant validity and which methods are inappropriate. We hope that the article will help readers discriminate valid techniques from those that are not.

Supplemental Material

Supplemental Material, Online_supplement_1_-_Chi2(1)_scaling_example - An Updated Guideline for Assessing Discriminant Validity

Supplemental Material, Online_supplement_1_-_Chi2(1)_scaling_example for An Updated Guideline for Assessing Discriminant Validity by Mikko Rönkkö and Eunseong Cho in Organizational Research Methods

Supplemental Material

Supplemental Material, Online_supplement_2_-_Full_simulation_code - An Updated Guideline for Assessing Discriminant Validity

Supplemental Material, Online_supplement_2_-_Full_simulation_code for An Updated Guideline for Assessing Discriminant Validity by Mikko Rönkkö and Eunseong Cho in Organizational Research Methods

Supplemental Material

Supplemental Material, Online_supplement_3_-_Full_simulation_results - An Updated Guideline for Assessing Discriminant Validity

Supplemental Material, Online_supplement_3_-_Full_simulation_results for An Updated Guideline for Assessing Discriminant Validity by Mikko Rönkkö and Eunseong Cho in Organizational Research Methods

Supplemental Material

Supplemental Material, Online_supplement_4_-_Tutorial_on_calculations - An Updated Guideline for Assessing Discriminant Validity

Supplemental Material, Online_supplement_4_-_Tutorial_on_calculations for An Updated Guideline for Assessing Discriminant Validity by Mikko Rönkkö and Eunseong Cho in Organizational Research Methods

Supplemental Material

Supplemental Material, Online_supplement_5_-_Deriving_a_cutoff_based_on_simulation_results - An Updated Guideline for Assessing Discriminant Validity

Supplemental Material, Online_supplement_5_-_Deriving_a_cutoff_based_on_simulation_results for An Updated Guideline for Assessing Discriminant Validity by Mikko Rönkkö and Eunseong Cho in Organizational Research Methods