Impact of Sampling Variability When Estimating the Explained Common Variance

Assessing multidimensionality of a scale or test is a staple of educational and psychological measurement. One approach to evaluate approximate unidimensionality is to fit a bifactor model where the subfactors are determined by substantive theory and estimate the explained common variance (ECV) of the general factor. The ECV says to what extent the explained variance is dominated by the general factor over the specific factors, and has been used, together with other methods and statistics, to determine if a single factor model is sufficient for analyzing a scale or test (Rodriguez et al., 2016). In addition, the individual item-ECV (I-ECV) has been used to assess approximate unidimensionality of individual items (Carnovale et al., 2021; Stucky et al., 2013). However, the ECV and I-ECV are subject to random estimation error which previous studies have not considered. Not accounting for the error in estimation can lead to conclusions regarding the dimensionality of a scale or item that are inaccurate, especially when an estimate of ECV or I-ECV is compared to a pre-specified cut-off value to evaluate unidimensionality. The objective of the present study is to derive standard errors of the estimators of ECV and I-ECV with linear confirmatory factor analysis (CFA) models to enable the assessment of random estimation error and the computation of confidence intervals for the parameters. We use Monte-Carlo simulation to assess the accuracy of the derived standard errors and evaluate the impact of sampling variability on the estimation of the ECV and I-ECV.

In a bifactor model for $J$ items, denote $X_j$ , $j=1,\dots ,J$ , as the observed variable and let $G$ denote the general factor. We define the $S$ subfactors $F_s$ , $s\in \{1,\dots ,S\}$ , and $J_s$ as the set of indicators for each subfactor. Each observed indicator $X_j$ is then defined by the multiple factor model (McDonald, 2013)

$$X_j={\mu }_j+{\lambda }_{Gj}G+{\lambda }_{sj}{F}_s+{\mathit{ϵ}}_j,$$

(1)

where ${\lambda }_{Gj}$ and ${\lambda }_{sj}$ are the factor loadings, ${\mu }_j$ denotes the item mean, and ${\mathit{ϵ}}_j$ is a random error term with variance ${\psi }_j^2$ . In a bifactor model, $G$ and each $F_s$ are independent of each other and the error terms are independent of each other and with $G$ and each $F_s$ .

The explained common variance for CFA models is equal to (Rodriguez et al., 2016)

$$\mathrm{E}\mathrm{C}\mathrm{V}=\frac{{{\displaystyle \sum }}_{j=1}^J{\lambda }_{Gj}^2}{{{\displaystyle \sum }}_{j=1}^J{\lambda }_{Gj}^2+{{\displaystyle \sum }}_{s=1}^S{{\displaystyle \sum }}_{j\in J_s}{\lambda }^2}.$$

(2)

In the literature, it has been suggested that ECV values higher than 0.7 to 0.8 indicate sufficient unidimensionality of the scales to adopt a unidimensional model (Rodriguez et al., 2016). To further evaluate the unidimensionality of a specific item score, another statistic, the explained common variance of an item ( $\mathrm{I}-\mathrm{E}\mathrm{C}{\mathrm{V}}_{\mathrm{j}}$ ), was suggested and is defined as (Stucky et al., 2013)

$$\mathrm{I}-\mathrm{E}\mathrm{C}{\mathrm{V}}_{\mathrm{j}}=\frac{{\lambda }_{Gj}^2}{{\lambda }_{Gj}^2+{\lambda }_{sj}^2}.$$

(3)

The ECV and $\mathrm{I}-\mathrm{E}\mathrm{C}{\mathrm{V}}_{\mathrm{j}}$ are functions of the factor loadings and subject to sampling variability when estimating the unknown parameters. However, this has not been considered in prior studies and the impact of random estimation error on the ECV and $\mathrm{I}-\mathrm{E}\mathrm{C}{\mathrm{V}}_{\mathrm{j}}$ is unknown. Here, we provide a method for estimating the sampling variability of these statistics based on standard asymptotic theory. Assuming that the estimator $\widehat{\mathit{\theta }}$ of the CFA model parameters $\mathit{\theta }$ fulfills $\sqrt{n}\left(\widehat{\mathit{\theta }}-\mathit{\theta }\right)\stackrel{D}{\to } N\left(0, {{\displaystyle \sum }}^{\text{}}\right)$ as the sample size $n$ tends to infinity, we can approximate the variance of the estimators of ECV and $\mathrm{I}-\mathrm{E}\mathrm{C}{\mathrm{V}}_{\mathrm{j}}$ by applying the delta method (Ferguson, 1996), such that $\sqrt{n}\left(g\left(\widehat{\mathit{\theta }}\right)-g\left(\mathit{\theta }\right)\right)\stackrel{D}{\to }N\left(0,\frac{\partial \mathit{g}}{\partial \mathit{\theta }}{{\displaystyle \sum }}^{\text{}}{\left(\frac{\partial \mathit{g}}{\partial \mathit{\theta }}\right)}^{\prime }\right)$ where $g$ is either ECV or $\mathrm{I}-\mathrm{E}\mathrm{C}{\mathrm{V}}_{\mathrm{j}}$ , continuously differentiable functions of $\widehat{\mathit{\theta }}$ . Define $\mathbf{\lambda }$ as the vector of all factor loadings and define ${\mathbf{\lambda }}_G$ as the vector of all factor loadings for the general factor. We require the derivatives of the ECV and $\mathrm{I}-\mathrm{E}\mathrm{C}{\mathrm{V}}_{\mathrm{j}}$ with respect to the unknown parameters and obtain, for any factor loading $\lambda \in \mathbf{\lambda }$

$$\begin{array}{c}\frac{\partial \mathrm{E}\mathrm{C}\mathrm{V}}{\partial \lambda }=\frac{1\left(\lambda \in {\mathbf{\lambda }}_G\right)2\lambda \left({{\displaystyle \sum }}_{j=1}^J{\lambda }_{Gj}^2+{{\displaystyle \sum }}_{s=1}^S{{\displaystyle \sum }}_{j=1}^{J_s}{\lambda }_{sj}^2\right)-\left({{\displaystyle \sum }}_{j=1}^J{\lambda }_{Gj}^2\right)2\lambda }{{\left({{\displaystyle \sum }}_{j=1}^J{\lambda }_{Gj}^2+{{\displaystyle \sum }}_{s=1}^S{{\displaystyle \sum }}_{j=1}^{J_s}{\lambda }_{sj}^2\right)}^2} \end{array}$$

(4)

and

$$\begin{array}{c}\frac{\partial \mathrm{I}-\mathrm{E}\mathrm{C}{\mathrm{V}}_{\mathrm{j}}}{\partial \lambda }=\frac{1\left(\lambda ={\lambda }_{Gj}\right)2\lambda \left({\lambda }_{Gj}^2+{\lambda }_{sj}^2\right)-\left({\lambda }_{Gj}^2\right)2\lambda }{{\left({\lambda }_{Gj}^2+{\lambda }_{sj}^2\right)}^2}.\end{array}$$

(5)

To assess the accuracy of the standard errors and illustrate the impact of estimation error for ECV and $\mathrm{I}-\mathrm{E}\mathrm{C}{\mathrm{V}}_{\mathrm{j}}$ , we simulated data in accordance with a 35-item scale with factor loadings given in Table 3 in Stucky et al. (2013). The bifactor model had one general factor and five independent specific factors that were each measured by 10, 6, 7, 6 and 6 items. All factors were independently generated from standard normal distributions and the residuals were randomly drawn from a normal distribution with mean zero and variances for each item randomly drawn from the $U\left(0.2,0.6\right)$ distribution. The true ECV for these data was .615 and the I-ECVs ranged from .171 to .982. We used sample sizes 200, 400, 800 and 1600, and 1000 replications, where the CFA models were estimated using maximum likelihood with the R package lavaan (Rosseel, 2012) and where the R code for computing the ECV and I-ECVs with associated standard errors are available as online Supplementary material.

In Table 1, we present the simulation results for the ECV, showing that the estimators are accurate for all sample sizes. The Monte-Carlo standard errors range from .031 at sample size 200 to 0.011 at sample size 1600 and the asymptotic standard errors based on the delta method with the derivatives from equation (1) are accurate for all sample sizes.

Table 1.

Monte-Carlo Bias and Standard Errors (MC-SE), Along With Average Asymptotic Standard Errors (SE), for the Estimated Explained Common Variance With Four Sample Sizes.

Sample Size	Bias	MC-SE	SE
200	0.001	0.030	0.029
400	0.001	0.022	0.021
800	−0.000	0.015	0.015
1600	0.000	0.011	0.010

We present the simulation results for the I-ECVs pertaining to the 10 items that measure the first subfactor in Table 2, for sample sizes 200 and 1600. The simulation study included 35 items but we omitted the remaining results since they were highly similar. In Table 2, it can be seen that the sampling variability for I-ECV is generally larger than for the ECV since these statistics are based on the estimated factor loadings of only a single item. Meanwhile, even at sample size 200, the asymptotic standard errors from the delta method using the derivatives from equation (2) are accurate. The results in Table 2 imply that the assessment of item-level unidimensionality with the I-ECV carries with it substantial random error. The largest impact for the first subfactor exists for item 9, where a 95% confidence interval had average length $2\times z_{0.975}\times 0.069\approx 0.270$ with sample size 200, which is quite large for a statistic that ranges between 0 and 1. The average length of a 95% confidence interval for the I-ECV of the same item however reduces to $2\times z_{0.975}\times 0.021\approx 0.082$ with sample size 1600.

Table 2.

Monte-Carlo Standard Errors With Average Asymptotic Standard Errors in Parentheses, for the Estimated Item-Explained Common Variance of Items 1 to 10 With Sample Sizes 200 and 1600.

Sample Size	Item 1	Item 2	Item 3	Item 4	Item 5	Item 6	Item 7	Item 8	Item 9	Item 10
200	.008	.019	.021	.036	.044	.056	.062	.061	.070	.062
	(.008)	(.018)	(.020)	(.036)	(.045)	(.055)	(.062)	(.061)	(.069)	(.062)
1600	.003	.006	.007	.012	.016	.020	.022	.021	.025	.021
	(.003)	(.006)	(.007)	(.013)	(.016)	(.020)	(.022)	(.022)	(.025)	(.022)

Assessing approximate unidimensionality is commonly done as part of scale development in education and psychology and useful statistics like the ECV help in evaluating unidimensionality. However, this process should be complemented with an assessment of the random error associated with the statistics used. Just like the reporting of reliability coefficients should include standard errors or confidence intervals (Fan & Thompson, 2001), we argue that measures like the ECV and I-ECV should be reported together with an indication of the amount of random error. In this study, we presented a simple solution to assess estimation error for linear factor models and implemented the approach in R for use by interested researchers. Future studies can include the specific results for other statistics commonly used with bifactor models, such as the omega-total and omega-hierarchical coefficients.

Supplemental Material

sj-pdf-1-apm-10.1177_01466216221084215 - Supplemental Material - Impact of Sampling Variability When Estimating the Explained Common Variance

Supplemental Material, sj-pdf-1-apm-10.1177_01466216221084215 for Impact of Sampling Variability When Estimating the Explained Common Variance by Björn Andersson and Hao Luo in Applied Psychological Measurement