Sage Journals: Discover world-class research

Abstract

Previous research has shown that ignoring individual differences of factor loadings in conventional factor models may reduce the determinacy of factor score predictors. Therefore, the aim of the present study is to propose a heterogeneous regression factor score predictor (HRFS) with larger determinacy than the conventional regression factor score predictor (RFS) when individuals have different factor loadings. First, a method for the estimation of individual loadings is proposed. The individual loading estimates are used to compute the HRFS. Then, a binomial test for loading heterogeneity of a factor is proposed to compute the HRFS only when the test is significant. Otherwise, the conventional RFS should be used. A simulation study reveals that the HRFS has larger determinacy than the conventional RFS in populations with substantial loading heterogeneity. An empirical example based on subsamples drawn randomly from a large sample of Big Five Markers indicates that the determinacy can be improved for the factor emotional stability when the HRFS is computed.

Keywords

factor analysis factor scores determinacy loading heterogeneity Big Five

Introduction

The conventional factor model consists of factor scores and factor loadings. It has been proposed to use the conventional factor model on different types of data (Cattell, 1952). The most prominent type of factor analysis is R-factor analysis, where factor analysis is performed for the covariances of variables measured for many individuals at one measurement occasion. In the following, we refer to R-factor analysis, although the methods presented here can also be adapted for other types of factor analysis. The R-factor model assumes that factor scores can differ between individuals, whereas it is implied that the loadings of the measured variables on the common factors are constant for all individuals. However, there are good reasons to expect that, at least under some circumstances, individuals may have different factor loadings. Arguments for considering inter-individual loading heterogeneity have been presented from developmental psychology, behavioral genetics, and P-factor analysis. P-factor analysis of the covariance of variables measured on a large number of measurement occasions for one individual has been considered in order to represent the idiographic aspects of data (Molenaar, 2004; Molenaar et al., 2003). However, in the present study, we consider loading heterogeneity in the context of R-factor analysis. The meaning of individual factor loadings can be illustrated by an example from the concept of intelligence. In this context, factor scores represent one’s intellectual capacity, whereas an individual’s factor loading on a variable describes to what extent the individual makes use of intelligence for a given task. Individuals with higher loadings of the task of an intelligence factor may rely more heavily on their intelligence, whereas individuals with smaller loadings may recruit other traits to solve the task and utilize their intelligence to a smaller extent.

The assumption that an observed variable has the same factor loading for every individual of a sample is probably an over-simplification in several areas of research. It has been shown that models describing individual differences that are based on parameters of the total sample can, in general, not directly be applied to a single individual (Molenaar & Campbell, 2009). This over-simplification also concerns the factor model because it is typically based on the analysis of a covariance matrix of observed variables. If there are different covariances of observed variables for different individuals or for different subsamples, this cannot be detected by factor analysis of a single covariance matrix of the total sample. In line with this, Kelderman and Molenaar (2007) showed that under the assumption that normally distributed heterogenous loadings are independent of each other, the population covariance matrix of a model based on heterogenous loadings is the same as the population covariance matrix of a model based on the same loadings for all individuals. When the resulting covariance matrices of observed variables are the same for models with and without heterogeneous loadings between individuals, the covariance matrices do not provide a basis for estimating heterogeneous factor loadings. Moreover, the factor scores representing the latent or “true” individual differences of the construct are indeterminate (Guttman, 1955; Nicewander, 2020) so that multiplying individual loadings with the indeterminate individual factor scores will necessarily give indeterminate results. In other words, the number of parameters of the factor model is already larger than the number of data points. Therefore, researchers rely on factor score predictors as a proxy for individual factor scores. The validity of such factor score predictors is given by their determinacy, that is, their correlation with the corresponding factor (Grice, 2001). However, given indeterminate factor scores, increasing the number of parameters by introducing individual differences on factor loadings will further increase indeterminacy. For example, one could conceive that for each individuum, a different rotation of factor loadings is possible. Moreover, Kelderman and Molenaar (2007) found that the standard likelihood-ratio goodness-of-fit statistic has little power in detecting loading heterogeneity. Therefore, considering loading heterogeneity is a challenge in the context of the exploratory factor model.

Ansari et al. (2002) point out that ignoring unobserved heterogeneity can lead to biased parameter estimates. They developed Markov Chain Monte Carlo procedures to perform Bayesian inference for confirmatory factor models with mean and covariance heterogeneity. Although this promising approach works in the context of confirmatory factor analysis, it is still relevant to consider loading heterogeneity in the context of exploratory factor analysis. As mentioned before, indeterminacy of the factor scores implies that any product of individual loadings with indeterminate factor scores will also be indeterminate. This also implies that, when a factor score predictor is specified, the product of individual loadings with the individual factor score predictor will be identified. Accordingly, Molenaar et al. (2003) investigated the effect of loading heterogeneity on the validity or determinacy of factor score predictors. They found that the determinacy coefficient reduces considerably when heterogenous loadings occur in the population model but are not specified in the factor model (Kelderman & Molenaar, 2007; Molenaar et al., 2003). Kelderman and Molenaar (2007) found that loading heterogeneity affects the distribution of the observed variables. They recommend that non-normality of the distributions of observed variables should be tested by means of the Shapiro–Wilk Test to use factor score predictors with more confidence. However, non-normality of data can have several reasons, so that a significant Shapiro–Wilk Test is not a specific indicator of loading heterogeneity.

Accordingly, a subsequent research question is whether a more specific test of loading heterogeneity is possible and whether the factor score predictors can be adapted to cases where heterogeneous loadings can be expected. The reduction of the factor score determinacy due to loading heterogeneity was more substantial for Bartlett’s (1937) factor score predictor than for Thurstone’s (1935) regression factor score predictor (RFS; Molenaar et al., 2003). Moreover, for homogeneous factor loadings, the determinacy of the RFS is larger than the determinacy of the Bartlett factor score predictor (Krijnen et al., 1996), so that the RFS is considered in the following. Is it possible to minimize the loss of factor score determinacy of the RFS that cooccurs with heterogenous loadings? The present study provides a tentative solution for this problem. The paper comprises the following sections: (a) some definitions are given, (b) a method for estimating heterogeneous loadings of orthogonal factors is proposed, (c) on this basis, a heterogenous regression factor score predictor (HRFS) is proposed that may allow to minimize the loss of factor score determinacy of the RFS, (d) a test for loading heterogeneity is proposed, (e) some ideas how to consider loading heterogeneity in oblique factor models are proposed, (f) a simulation study is performed to compare the loss of determinacy of RFS and HRFS in models with heterogeneous factor loadings, and (g) the determinacies of RFS and HRFS are compared in an empirical dataset based on the Five Factor model. Finally, some limitations of the present study and some prospects for the use of HRFS are discussed.

Definitions

In a population of individuals, the common factor model (Mulaik, 2010) can be defined as:

x = Λ ξ + Ψ ε

(1)

where x is a vector of individual observations on p observed variables, $Λ$ is a p × q matrix of common factor loadings, $ξ$ is a vector q factor scores with the expected value $E (ξ) = 0$ , the correlation matrix of common factor scores $E (ξ ξ^{'}) = Φ$ , with $diag (Φ) = I$ (variances of common factor scores), Ψ is a p × p diagonal, positive definite matrix of unique factor loadings, and $ξ$ is a vector of p unique factor scores with $E (ε) = 0$ , $E (ε ε^{'}) = I$ , and $E (ξ ε^{'}) = 0$ (uncorrelated unique factors). Accordingly, the covariance matrix of observed variables Σ is:

Σ = E (x x^{'}) = Λ Φ Λ^{'} + Ψ^{2} .

(2)

In the following, the correlation matrix of observed variables is considered, so that $diag (Σ) = I$ The RFS, that is, the best linear predictor (Krijnen et al., 1996) is defined by:

ξ_{r} = Φ Λ^{'} Σ^{- 1} x .

(3)

The correlation of the RFS with the original factor, that is, the determinacy coefficient $P$ (Grice, 2001; Guttman, 1955), can be regarded as an indicator of convergent validity of the factor score predictor. This correlation can be computed by:

E (ξ_{r} ξ^{'}) = P = diag (Φ Λ^{'} Σ^{- 1} Λ Φ)^{1 / 2} .

(4)

For each observed variable i on factor j, $Λ$ contains a common factor loading $λ_{ij}$ , which can be conceived as the expectation of the individual common factor loadings on this variable for the population of individuals. For a finite population of N individuals and completely independent individual loadings (Kelderman & Molenaar, 2007), the expected value of the individual loadings is:

λ_{ij} = E (λ_{ijk}) = \frac{1}{N} \sum_{k}^{N} λ_{ijk} .

(5)

For $Φ = I$ and $- 1 > λ_{ij} < 1$ , it is possible that there are individual differences of factor loadings, so that the variance of factor loadings across individuals is greater zero:

σ_{ij}^{2} = \frac{1}{N} \sum_{k}^{N} {(λ_{ijk} - λ_{ij})}^{2} > 0 .

(6)

For $Φ \neq I$ , individual differences may also occur for $| λ_{ij} | \geq 1$ , and individual differences of factor inter-correlations may also occur. However, the estimation of individual factor loadings $λ_{ijk}$ is already a challenge with orthogonal factor models, so that orthogonal models are primarily considered here before some ideas how to generalize the approach to correlated factor models are presented.

Estimation of Individual Factor Loadings

The proposed estimation procedure is described for a sample of n individuals and—in the first step—for the orthogonal factor model. Orthogonal factor analysis of q factors and p variables for the total sample yields:

\hat{Σ} = \hat{Λ} {\hat{Λ}}^{'} + {\hat{Ψ}}^{2} .

(7)

Model misfit occurs in the sample so that the sample covariance matrix differs from the population covariances, $E (x x^{'}) = S \neq \hat{Σ}$ . In consequence, the determinacy coefficient, that is, the correlation of ${\hat{ξ}}_{r}$ with $ξ$ is:

E (diag ({\hat{ξ}}_{r} {\hat{ξ}}_{r}^{'})^{- 1 / 2} {\hat{ξ}}_{r} {\hat{ξ}}^{'}) = P = diag ({\hat{Λ}}^{'} {\hat{Σ}}^{- 1} S {\hat{Σ}}^{- 1} \hat{Λ})^{- 1 / 2} diag ({\hat{Λ}}^{'} {\hat{Σ}}^{- 1} \hat{Λ}) .

(8)

Similar to Cook’s (1977) ideas on the influence of a single individual on linear regression results, the determinacy of the RFS ${\hat{ξ}}_{r (- k)}$ is computed from factor analysis when the data of the kth individuum are deleted. Factor analysis of $S_{(- k)}$ , the sample covariance matrix based on one eliminated case, yields the loading estimates ${\hat{Λ}}_{(- k)}$ , where $(- k)$ refers to the parameters estimated without the data of the individual k. If q > 1, it is recommended to perform orthogonal target rotation of ${\hat{Λ}}_{(- k)}$ towards $\hat{Λ}$ (Schoenemann, 1966) to minimize the effects of different positions of the factor axes of ${\hat{Λ}}_{(- k)}$ and $\hat{Λ}$ on results. The determinacy coefficient for the factors when the data of the kth individuum are deleted is:

\begin{matrix} E (diag ({\hat{ξ}}_{r (- k)} {\hat{ξ}}_{r (- k)}^{'})^{- 1 / 2} {\hat{ξ}}_{r (- k)} {\hat{ξ}}_{(- k)}^{'}) = P_{(- k)} = diag ({\hat{Λ}}_{(- k)}^{'} {\hat{Σ}}_{(- k)}^{- 1} S_{(- k)} {\hat{Σ}}_{(- k)}^{- 1} {\hat{Λ}}_{(- k)})^{- 1 / 2} \\ diag ({\hat{Λ}}_{(- k)}^{'} {\hat{Σ}}_{(- k)} \hat{Λ}) . \end{matrix}

(9)

The effect of the kth individuum on the determinacies of the factor score predictors is:

Δ P_{k}^{2} = P^{2} - P_{(- k)}^{2} .

(10)

Positive values of $Δ P_{k}^{2}$ indicate that the kth individuum contributes positively to the determinacy of the factor score predictor. The effect of the kth individuum on the RFS for a single factor is:

Δ ρ_{jk}^{2} = ρ_{j}^{2} - ρ_{j (- k)}^{2} .

(11)

Although $Δ ρ_{jk}^{2}$ allows to identify individuals for whom the prediction of the common factor is above or below average, it does not help to compensate for the effect of low prediction. To compensate for the different influence of individuals on determinacy, it is proposed to estimate the size of the individual loadings ${\hat{λ}}_{ijk}$ . If a loading decreases when the data of individual k are deleted, the individual k has an increasing effect on the overall loading. Therefore, the effect of the data of an individuum k on a loading can be estimated by:

Δ {\hat{λ}}_{ijk} = {| {\hat{λ}}_{ij}^{2} sgn ({\hat{λ}}_{ij}) - {\hat{λ}}_{ij (- k)}^{2} sgn ({\hat{λ}}_{ij (- k)}) |}^{1 / 2} sgn ({\hat{λ}}_{ij}^{2} sgn ({\hat{λ}}_{ij}) - {\hat{λ}}_{ij (- k)}^{2} sgn ({\hat{λ}}_{ij (- k)})),

(12)

where “sgn” is the sign function. The multiplication with the sign of the loadings maintains their sign after being squared. As ${\hat{λ}}_{ij}^{2}$ is based on n individuals and ${\hat{λ}}_{ij (- k)}^{2}$ is based on n − 1 individuals, the estimated effect of an individuum k on ${\hat{λ}}_{ij}^{2}$ will decrease with larger n. However, the effect of sampling error on ${\hat{λ}}_{ij}^{2}$ and ${\hat{λ}}_{ij (- k)}^{2}$ also decreases with n, so that the estimation of ${\hat{λ}}_{ij (- k)}^{2}$ is not necessarily worse for larger n. Moreover, individual differences of measurement error may also affect ${\hat{λ}}_{ij (- k)}^{2}$ . For this reason, the optimal calibration of ${\hat{λ}}_{ij (- k)}^{2}$ is unknown. In order to keep the effect of measurement error in ${\hat{λ}}_{ij (- k)}^{2}$ on $Δ {\hat{λ}}_{ijk}$ and thereby on ${\hat{λ}}_{ijk}$ small, it is proposed to estimate ${\hat{λ}}_{ijk}$ from a weighted aggregate of $Δ {\hat{λ}}_{ijk}$ and ${\hat{λ}}_{ij}$ . Effects of $Δ {\hat{λ}}_{ijk}$ on individual loadings should be more substantial for larger than for smaller total sample absolute loadings. Accordingly, $Δ {\hat{λ}}_{ijk}$ is weighted by $w = | {\hat{λ}}_{ij} | / mean (| {\hat{λ}}_{j} |)$ so that:

{\hat{λ}}_{ijk} = {\hat{λ}}_{ij} + w Δ {\hat{λ}}_{ijk} .

(13)

In a stepwise procedure, $w Δ {\hat{λ}}_{ijk}$ is added to only one element ${\hat{λ}}_{ij}$ resulting in ${\hat{Λ}}_{ijk}$ , in which only element ij differs from the corresponding element in $\hat{Λ}$ . In order to minimize the effect of sampling error on ${\hat{λ}}_{ijk}$ , we propose to add $w Δ {\hat{λ}}_{ijk}$ to ${\hat{λ}}_{ij}$ only when the squared difference of the non-diagonal elements of the correlation matrix reproduced from ${\hat{Λ}}_{ijk}$ and the non-diagonal elements of $S_{(- k)}$ is larger than the squared difference of the non-diagonal elements of the correlation matrix reproduced from the loadings $\hat{Λ}$ and the non-diagonal elements of S. Accordingly, the individual loadings are estimated as follows:

{\tilde{\hat{λ}}}_{ijk} = {\begin{matrix} \begin{matrix} {\hat{λ}}_{ij} + w Δ {\hat{λ}}_{ijk}, if SSQ ({\hat{Λ}}_{ijk} {\hat{Λ}}_{ijk}^{'} - diag ({\hat{Λ}}_{ijk} {\hat{Λ}}_{ijk}^{'}) - (S_{(- k)} - diag (S_{(- k)}))) \\ > SSQ ({\hat{Λ} \hat{Λ}}^{'} - diag ({\hat{Λ} \hat{Λ}}^{'}) - (S - diag (S))) \end{matrix} \\ {\hat{λ}}_{ij} else \end{matrix},

(14)

where “SSQ” denotes the sum of squares. Heywood cases may occur more often for ${\tilde{\hat{λ}}}_{ijk}$ than for ${\hat{λ}}_{ij}$ , and it should be considered to reset the respective individual loadings to more realistic absolute loadings (e.g., 0.99).

The Heterogeneity-Based RFS

Making use of the estimated individual loadings, the heterogeneity-based regression factor score predictor (HRFS) is then computed for each individual as

{\hat{ξ}}_{rk} = diag ({\tilde{\hat{Λ}}}_{k}^{'} {\hat{Σ}}^{- 1} S {\hat{Σ}}^{- 1} {\tilde{\hat{Λ}}}_{k})^{- 1 / 2} {\tilde{\hat{Λ}}}_{k}^{'} {\hat{Σ}}^{- 1} x .

(15)

The estimated correlation of ${\hat{ξ}}_{rk}$ with $ξ$ , the individual determinacy of the HRFS is:

E (diag ({\hat{ξ}}_{rk} {\hat{ξ}}_{rk}^{'})^{- 1 / 2} {\hat{ξ}}_{rk} ξ^{'}) = {\hat{P}}_{k} = Mea n_{k} (diag {({\tilde{\hat{Λ}}}_{k}^{'} {\hat{Σ}}^{- 1} S {\hat{Σ}}^{- 1} {\tilde{\hat{Λ}}}_{k})}^{- 1 / 2} diag ({\tilde{\hat{Λ}}}_{k}^{'} {\hat{Σ}}^{- 1} {\tilde{\hat{Λ}}}_{k})),

(16)

where “ $Mea n_{k}$ ” denotes the mean across all individuals in a sample. It is proposed to estimate ${\hat{P}}_{k}$ and the HRFS only when loadings are heterogeneous between individuals. Otherwise, the estimation of $\hat{P}$ and the conventional RFS is recommended. Although deviations from normality may be an indicator of loading heterogeneity (Kelderman & Molenaar, 2007), several other effects may cause non-normal distributions. Therefore, a more specific indicator of loading heterogeneity is proposed.

Estimation of Loading Heterogeneity

Loading heterogeneity can be assessed when, in a sample of n individuals, n factor analyses with the data minus one individuum k are performed. For each loading on each factor, the inter-individual standard deviation $\hat{σ} ({\hat{λ}}_{ij (- k)})$ can be compared with the inter-individual standard deviation of loadings of n factor analyses based on a sample of n cases minus one case, drawn from a simulated population without inter-individual variability of loadings, that is, $λ_{ijk} = λ_{ij}$ , for all k. This standard deviation is based on the loadings ${\hat{λ}}_{ij (- k)}$ , resulting directly from leaving individual k out, as used in Equation 12 (it is not the individual loading ${\tilde{\hat{λ}}}_{ijk}$ ) because the aim is to estimate whether leaving successively one individual out results in substantial loading heterogeneity. Let $\hat{σ} ({\hat{λ}}_{0, ij (- k)})$ denote the standard deviation of loadings based on n factor analyses with a sample of n − 1 individuals drawn randomly from a simulated population with zero loading heterogeneity. The effect of leaving one individual out on loading heterogeneity may depend on loading magnitude in the total sample because there might be ceiling effects. Therefore, the population loading of each simulated variable should equal the mean of the individual loadings of the corresponding variable in the empirical data set, that is:

λ_{0, ij} = Mean ({\hat{λ}}_{ij (- k)}) .

(17)

In the simulated population, $λ_{0, ij}$ is constant for all k individuals, so that $σ (λ_{0, ijk}) = 0$ . In consequence, any $\hat{σ} ({\hat{λ}}_{0, ij (- k)}) > 0$ , found in a sample drawn from this population is due to sampling error. If n_d samples are drawn from this population, the mean of the resulting standard deviations of loadings can be compared with the empirical standard deviation of loadings for the corresponding variable i on factor j. If $Mea n_{n_{d}} (\hat{σ} ({\hat{λ}}_{0, ij (- k)})) < \hat{σ} ({\hat{λ}}_{ij (- k)})$ , one would conclude that the empirical loading heterogeneity is not due to sampling error.

To obtain an indicator for the loading heterogeneity of a factor, the number of variables with loading heterogeneity greater than sampling error is counted for the respective factor. The following index is one if loading heterogeneity of variable i occurs, that is, if $\hat{σ} ({\hat{λ}}_{ij (- k)})$ is greater than $Mea n_{n_{d}} (\hat{σ} ({\hat{λ}}_{0, ij (- k)}))$ :

ϑ_{ij} = {\begin{matrix} 1 if Mea n_{n_{d}} (\hat{σ} ({\hat{λ}}_{0, ij (- k)})) < \hat{σ} ({\hat{λ}}_{ij (- k)}) \\ 0 if Mea n_{n_{d}} (\hat{σ} ({\hat{λ}}_{0, ij (- k)})) \geq \hat{σ} ({\hat{λ}}_{ij (- k)}) \end{matrix} .

(18)

The number $ϑ_{j} = \sum_{i = 1}^{p} ϑ_{ij}$ for each factor follows a binomial distribution where “success” is defined by $ϑ_{ij} = 1$ , the number of trials by p (number of variables), and the success probability is .50. The minimum number of successes that $ϑ_{j}$ should be reached to reject the assumption of homogeneity is $ϑ_{crit}$ . The decision on the hypothesis of homogeneity can therefore be made with a right-tailed binomial test, whereby the problem arises that with only a few variables per factor, only a few exact significance levels can be determined. Using the critical values listed in Table 1, a significance level of α ≤ .25 is ensured. The assumption of homogeneity is therefore rejected if $ϑ_{j} \geq ϑ_{crit}$ , so that HRFS should be computed, whereas the conventional RFS should be computed if $ϑ_{j} < ϑ_{crit}$ .

Table 1.

Cut-Off Values $ϑ_{crit}$ for the Index of Loading Heterogeneity for p ≤ 12 (Number of Variables) Resulting in α ≤ .25.

$p$	$ϑ_{crit}$	α_exact
2	2	.2500
3	3	.1250
4	3	.2500
4	4	.0625
5	4	.1875
6	5	.1094
7	5	.2266
8	6	.1445
9	7	.0898
10	7	.1719
11	8	.1133
12	8	.1938
12	9	.0730

For p ≤ 3, it is necessary to use a cut-off value of $ϑ_{crit} = p$ . Lower cut-off values with $ϑ_{crit} < p$ can only be used for p > 3, and a more fine-grained control of α is only possible for large p. However, an advantage of this indicator of loading heterogeneity is that it can also be used for categorical factor analysis. Therefore, the proposed index yields not necessarily the same results as the Shapiro–Wilk test for non-normality.

On Loading Heterogeneity in Oblique Factor Models

The present approach follows Cook’s (1977) idea to investigate the effect of data elimination for the kth individual. In the correlated factor model, this may not only affect the oblique factor loadings ${\hat{Λ}}_{o (- k)}$ but also the factor inter-correlations, so that heterogeneity between individuals of the factor inter-correlations ${\hat{Φ}}_{(- k)}$ may occur. Rotational indeterminacy implies ${\hat{Λ}}_{o (- k)} {\hat{Φ}}_{(- k)} {\hat{Λ}}_{o (- k)}^{'} = {\hat{Λ}}_{(- k)} {\hat{Λ}}_{(- k)}^{'}$ , so that model fit of orthogonal and oblique models is identical. It is therefore not clear whether ${\hat{Λ}}_{o (- k)}$ and ${\hat{Φ}}_{(- k)}$ can be used simultaneously for the estimation of individual oblique loadings ${\hat{Λ}}_{o, k}$ and the individual factor inter-correlations ${\hat{Φ}}_{k}$ . However, simple structure has been introduced to overcome rotational indeterminacy and to improve factor interpretation (Thurstone, 1935). Without further simulations, it is impossible to anticipate whether the rotation of individual loadings towards simple structure allows for an improvement of individual loading estimates and which rotation method (Browne, 2001) should be recommended for optimal representation of loading heterogeneity. The investigation of the rotation of individual loadings toward some criteria (e.g., simple structure) and the investigation of oblique rotation methods as a basis for the estimation of ${\hat{Φ}}_{k}$ is an interesting task for future research, which goes beyond the scope of the present study.

It is, nevertheless, possible to estimate the heterogeneity of factor inter-correlations without applying rotation methods directly to individual loading matrices. The first step is to perform first-order factor analysis with oblique rotation, that is, $\hat{Σ} = {\hat{Λ}}_{o} \hat{Φ} {\hat{Λ}}_{o}^{'} + {\hat{Ψ}}^{2},$ in the total sample to get the oblique loading pattern ${\hat{Λ}}_{o}$ and the factor inter-correlations $\hat{Φ}$ . Second-order factor analysis of $\hat{Φ}$ yields second-order factor loadings ${\hat{Λ}}_{\hat{Φ}}$ and uniqueness ${\hat{Ψ}}_{\hat{Φ}}$ . The orthogonalized solution proposed by Schmid and Leiman (1957) ${\hat{Λ}}_{SL}$ is obtained by:

{\hat{Λ}}_{SL} = {\hat{Λ}}_{o} [{\hat{Λ}}_{\hat{Φ}}, {\hat{Ψ}}_{\hat{Φ}}]

(19)

As Schmid–Leiman factors are orthogonalized oblique factors, the second step is to estimate the individual factor loadings ${\hat{Λ}}_{SL, k}$ of the Schmid–Leiman solution by the procedure described above. The third step is to transform ${\hat{Λ}}_{SL, k}$ back to the individual second-order factor model, by means of:

{({\hat{Λ}}_{o}^{'} {\hat{Λ}}_{o})}^{- 1} {\hat{Λ}}_{o}^{'} {\hat{Λ}}_{SL, k} = [{\hat{Λ}}_{\hat{Φ}, k}, {\hat{Ψ}}_{\hat{Φ}, k}],

(20)

so that ${\hat{Φ}}_{k}$ is obtained by ${\hat{Φ}}_{k} = [{\hat{Λ}}_{\hat{Φ}, k}, {\hat{Ψ}}_{\hat{Φ}, k}] [{\hat{Λ}}_{\hat{Φ}, k}, {\hat{Ψ}}_{\hat{Φ}, k}]^{'}$ and the corresponding individual oblique first-order factor loadings are obtained by:

{\hat{Λ}}_{o, k} = {\hat{Λ}}_{SL, k} {[{\hat{Λ}}_{\hat{Φ}, k}, {\hat{Ψ}}_{\hat{Φ}, k}]}^{'} {\hat{Φ}}_{k}^{- 1} .

(21)

According to the condition given in Equations 24 and 25, the HRFS for oblique factors are then computed by:

{\hat{ξ}}_{rk} = diag {({\hat{Φ}}_{k} {\tilde{\hat{Λ}}}_{o, k}^{'} {\hat{Σ}}^{- 1} S {\hat{Σ}}^{- 1} {\tilde{\hat{Λ}}}_{o, k} {\hat{Φ}}_{k})}^{- 1 / 2} {\hat{Φ}}_{k} {\tilde{\hat{Λ}}}_{o, k}^{'} {\hat{Σ}}^{- 1} x .

(22)

It is, however, possible that $\hat{Φ}$ and ${\hat{Φ}}_{k}$ cannot be well represented by second-order factor analysis, so that higher-order factor analysis would be necessary (Schmid & Leiman, 1957). Moreover, the complex relationship between individual oblique loading estimates ${\hat{Λ}}_{o, k}$ and individual factor-intercorrelations ${\hat{Φ}}_{k}$ may be a challenge for theoretical interpretation.

Therefore, a method for the estimation of oblique individual factor loadings ${\hat{Λ}}_{o, k}$ under the assumption of homogeneous factor inter-correlations is also proposed. Starting from an oblique factor model in the total sample, the oblique loading pattern is post-multiplied with ${\hat{Φ}}^{1 / 2}$ , the symmetric square root of $\hat{Φ}$ , which yields the orthogonalized loading pattern ${\hat{Λ}}_{ort} = {\hat{Λ}}_{o} {\hat{Φ}}^{1 / 2}$ as ${\hat{Λ}}_{ort} {\hat{Λ}}_{ort}^{'} = {\hat{Λ}}_{o} {\hat{Φ}}^{1 / 2} {\hat{Φ}}^{1 / 2} {\hat{Λ}}_{o}^{'} = {\hat{Λ}}_{o} \hat{Φ} {\hat{Λ}}_{o}^{'} .$ Then, the individual loadings ${\hat{Λ}}_{ort, k}$ are estimated according to the abovementioned procedure and the oblique individual loading estimates are computed by means of ${\hat{Λ}}_{o, k} = {\hat{Λ}}_{ort, k} {\hat{Φ}}^{- 1 / 2}$ , and the HRFS are computed according to the condition of Equations 24 and 25 by:

{\hat{ξ}}_{rk} = diag {(\hat{Φ} {\tilde{\hat{Λ}}}_{o, k}^{'} {\hat{Σ}}^{- 1} S {\hat{Σ}}^{- 1} {\tilde{\hat{Λ}}}_{o, k} \hat{Φ})}^{- 1 / 2} \hat{Φ} {\tilde{\hat{Λ}}}_{o, k}^{'} {\hat{Σ}}^{- 1} x .

(23)

Although the focus of the present study is on the HRFS for orthogonal factors, a few examples will be provided for the HRFS as computed from Equation 23 for oblique rotated factors. It is, however, acknowledged that the issue of heterogeneous loadings for oblique factor models deserves a more complete investigation in future research.

Simulation Study

Conditions and Specification

A simulation study was performed to investigate whether the HRFS based on orthogonal factors has a larger validity, that is, determinacy coefficient than the conventional RFS when loading heterogeneity occurs. In an empirical study, the determinacy coefficient ${\hat{ρ}}_{r}$ can only be estimated by Equation 8, and ${\hat{ρ}}_{rk}$ can only be estimated by Equation 16. In contrast, in a simulation study based on generated factor scores $ξ$ , these correlations can be computed directly and averaged across samples. Therefore, the parameter-based determinacy estimates ${\hat{ρ}}_{r}$ and ${\hat{ρ}}_{rk}$ , can be compared with the factor score-based determinacy estimated by ${\hat{ρ}}_{ξ r} = E ({\hat{ξ}}_{r} ξ')$ , and ${\hat{ρ}}_{ξ rk} = E ({\hat{ξ}}_{rk} ξ^{'})$ . As mentioned above, ${\hat{ρ}}_{ξ rk}$ and HRFS should only be estimated when there is loading heterogeneity, otherwise ${\hat{ρ}}_{ξ r}$ and RFS are considered more appropriate. Therefore, the dependent variables of the simulation study are defined as follows. RFS ${\hat{ξ}}_{r}$ was computed for each sample in each condition as a basis for ${\hat{ρ}}_{ξ r}$ . HRFS was only used for the computation of ${\hat{ρ}}_{ξ rk}$ when $ϑ_{j} \geq ϑ_{crit}$ , so that:

{\tilde{\hat{ρ}}}_{ξ rk} = {\begin{matrix} {\hat{ρ}}_{ξ rk} if ϑ_{j} \geq ϑ_{crit} \\ {\hat{ρ}}_{ξ r} if ϑ_{j} \geq ϑ_{crit} \end{matrix},

(24)

and

{\tilde{\hat{ρ}}}_{rk} = {\begin{matrix} {\hat{ρ}}_{rk} if ϑ_{j} \geq ϑ_{crit} \\ {\hat{ρ}}_{r} if ϑ_{j} \geq ϑ_{crit} \end{matrix} .

(25)

A continuous proportion of $ϑ_{crit} / p = 0.83$ was used for the conditions of the simulation study so that the minimum of variables with loading heterogeneity defining a heterogenous factor remained constant across conditions. In consequence, the probability α for the rejection of the hypothesis that a factor has homogenous loadings across individuals decreased with p. For q = 1 and p = 6, $ϑ_{crit} = 5$ was used, resulting in α_exact = .109 and $ϑ_{crit} = 9$ for p = 12, resulting in α_exact = .073 (see Table 1). For q = 3 and p = 18, $ϑ_{crit} = 15$ was used, resulting in α_exact = .004 and for p = 36, $ϑ_{crit} = 30$ was used, resulting in α_exact < .001. Thus, the condition for the computation of HRFS became more conservative with increasing p. Note that more similar rejection rates of the loading homogeneity hypothesis would have implied that the proportion of variables with loading heterogeneity differs across the conditions of the simulation study.

Independent variables of the simulation study were the loading heterogeneity with $σ (Λ_{pop}) \in {. 00, . 25, . 50, . 75}$ , p ∈ ${6, 12}, q \in {1, 3}$ , and mean salient loading size $μ (Λ_{pop}) \in {. 60, . 70}$ , and sample size n ∈ ${150, 600}$ . For $σ (Λ_{pop})$ , each sample of loadings was fixed to have the given population standard deviation. If absolute loadings >0.98 occurred, all population loadings of a factor were divided by a constant so that the maximum absolute loading of a factor on a variable for each individual was 0.98, which slightly reduced σ(Λ_pop), especially in the n = 150 and σ(Λ_pop) = .75 conditions. Note that the means and standard deviations of the loadings were fixed conditions, that is, each condition had exactly the specified mean and standard deviation of loadings. This yielded 4 × 2 × 2 × 2 × 2 = 64 conditions. For each condition, 1,000 samples were drawn from the population.

For each sample of each condition, n × n_d factor analyses for the computation of $ϑ_{j}$ were performed. In order to keep the simulation study feasible, we investigated n_d = 50 simulated samples. Accordingly, for each condition based on n = 150 this led to 150 × 50 × 1,000 = 7.5e6 factor analyses and for each condition based on n = 600 this led to 30e6 factor analyses. Overall, 37.5e6 factor analyses were performed for the conditions listed above, resulting in about 1 month of simulation time when about 14 factor analyses per second were performed simultaneously. It was therefore not considered to investigate a larger set of conditions here.

As mentioned above, the effect of factor rotation on results for q = 3 was minimized by means of orthogonal target rotation according to Schoenemann (1966) towards the target matrix of salient loadings in the total samples. Moreover, ${\hat{Λ}}_{(- k)}$ the loadings resulting from n factor analyses of the sample minus one eliminated case, were rotated towards $\hat{Λ}$ , the loadings of the total sample analysis, by means of orthogonal target rotation. Thereby, the factor loadings ${\hat{λ}}_{ij (- k)}^{2}$ were as similar as possible to ${\hat{λ}}_{ij}^{2}$ , the factor loadings in the total sample. The simulation was performed with R4.4.1, R-packages (MASS, psych, matrixStats) and an example of the R-script for the simulation study is given in Supplemental Material A.

Results

The effect of loading heterogeneity on determinacy estimates was investigated by means of a repeated-measures ANOVA comprising parameter-based versus factor score-based determinacy coefficients (PAR-SCO) and the RFS-based versus HRFS-based determinacy coefficients (RFS-HRFS) as within-subject factors and the conditions, q, p, µ(Λ_pop), σ(Λ_pop), and n as between-subjects factors. The condition with n = 150 and σ(Λ_pop) > .75 was excluded because only about 10% of the individual factor analyses converged in this condition. Significance levels were not considered because, due to the large sample size (1,000 cases per cell), even effects with $η_{p}^{2} = . 001$ were significant at an alpha level of .001. The PAR-SCO main effect was large ( $η_{p}^{2} = . 26$ ) although—across all conditions—the mean of the score-based determinacy coefficients (M = 0.91, SE < 0.001) was only slightly larger than the mean of the parameter-based determinacy coefficients (M = 0.89, SE < 0.001). The RFS-HRFS main effect was even larger ( $η_{p}^{2} = . 73$ ) although the mean determinacy coefficients for the HRFS were again only slightly larger (M = 0.91, SE < 0.001) than the mean determinacy coefficients of the RFS (M = 0.89, SE < 0.001). The effect size of the PAR-SCO × σ(Λ_pop) interaction ( $η_{p}^{2} = . 33$ ) and the effect size of the RFS-HRFS × σ(Λ_pop) interaction ( $η_{p}^{2} = . 56$ ) were large indicating that loading heterogeneity affects the difference between parameter-based and score-based determinacy coefficients as well as the difference of determinacies between HRFS and RFS (see Figures 1 and 2).

Figure 1.

Mean determinacy coefficients based on model parameters and RFS ( ${\hat{ρ}}_{r}$ ), on model parameters and HRFS/RFS ( ${\tilde{\hat{ρ}}}_{rk}$ ), on scores and RFS ( ${\hat{ρ}}_{ξ r}$ ), and on scores and HRFS/RFS ( ${\tilde{\hat{ρ}}}_{ξ rk}$ ) for a single factor (q = 1), p = 6, and p = 12 variables; µ(Λ_pop) is the expected value of the population loadings; σ(Λ_pop) is the expected value of loading heterogeneity; the error bars mark the standard errors

Figure 2.

Mean determinacy coefficients based on model parameters and RFS ( ${\hat{ρ}}_{r}$ , on model parameters and HRFS/RFS ( ${\tilde{\hat{ρ}}}_{rk}$ ), on scores and RFS ( ${\hat{ρ}}_{ξ r}$ ), and on scores and HRFS/RFS ( ${\tilde{\hat{ρ}}}_{ξ rk}$ ) for three factors (q = 3), p/q = 6, and p/q = 12 variables per factor; µ(Λ_pop) is the expected value of the population loadings; σ(Λ_pop) is the expected value of loading heterogeneity; the error bars mark the standard errors; for q = 3, n = 150 and σ(Λ_pop) > .75, only about 10% of the individual factor analyses converged, so that no means were computed for this condition

For σ(Λ_pop) = .00, the mean determinacy was M = 0.93 (SE < 0.001) for RFS and M = 0.93 (SE < 0.001) for HRFS. For σ(Λ_pop) = .75 the mean determinacy was M = 0.85 (SE < 0.001) for RFS and M = 0.88 (SE < 0.001) for HRFS. Besides these effects, which were most interesting for the comparison between RFS and HRFS, there were very large main effects of p ( $η_{p}^{2}$ = .90) and µ(Λ_pop) ( $η_{p}^{2}$ = .73). These main effects were as expected with larger mean determinacy for p = 12 (M = 0.93, SE < 0.001) than for p = 6 (M = 0.87, SE < 0.001) and for µ(Λ_pop) = .70 (M = 0.92, SE < 0.001) than for µ(Λ_pop) = .60 (M = 0.88, SE < 0.001). For σ(Λ_pop), the main effect was very large ( $η_{p}^{2}$ = .84), with (M = 0.93, SE < 0.001) for σ(Λ_pop) = .00 and (M = 0.87, SE < 0.001) for σ(Λ_pop) = .75. Compared to these effects, the main effect for q was smaller ( $η_{p}^{2}$ = .17) with M = 0.91 (SE < 0.001) for q = 3 and M = 0.89 (SE < 0.001) for q = 1. The main effect of n was small ( $η_{p}^{2}$ = .06) with M = 0.90 (SE < 0.001), both for n = 150 and for n = 600 (as mean differences occurred on the third decimal place).

To investigate whether these results could be replicated when the initial total-sample solution was based on Varimax-rotated factors, we performed the analysis for q = 3 and p/q = 6, for n = 150 and n = 600. The results for Varimax-rotated solutions (see Figure 3) were similar to the results for the corresponding target-rotated solutions (see Figure 2).

Figure 3.

Mean determinacy coefficients based on model parameters and RFS ( ${\hat{ρ}}_{r}$ ), on model parameters and HRFS/RFS ( ${\tilde{\hat{ρ}}}_{rk}$ ), on scores and RFS ( ${\hat{ρ}}_{ξ r}$ ), and on scores and HRFS/RFS ( ${\tilde{\hat{ρ}}}_{ξ rk}$ ) for three factors (q = 3), p/q = 6, and p/q = 12 variables per factor for Varimax-rotated solutions; µ(Λ_pop) is the expected value of the population loadings; σ(Λ_pop) is the expected value of loading heterogeneity; the error bars mark the standard errors. For q = 3, n = 150, and σ(Λ_pop) > .75, only about 10% of the individual factor analyses converged, so that no means were computed for this condition

As the parameter-based determinacies may be used as estimates for the score-based determinacies in empirical settings (when score-based determinacies cannot be computed), the effect of loading heterogeneity on parameter-based and score-based determinacies was compared. To compare the effect of loading heterogeneity on parameter-based and score-based determinacies, the effect of σ(Λ_pop) on $Δ {\tilde{\hat{ρ}}}_{rk - r} = {\tilde{\hat{ρ}}}_{rk} - {\hat{ρ}}_{r}$ , the difference of the score-based determinacies of HRFS/RFS and RFS and on $Δ {\tilde{\hat{ρ}}}_{rk - r} = {\tilde{\hat{ρ}}}_{rk} - {\hat{ρ}}_{r}$ , the difference of the parameter-based determinacies of HRFS/RFS and RFS were compared (see Figure 4).

Figure 4.

The means of $Δ {\tilde{\hat{ρ}}}_{ξ r k - ξ r} = {\tilde{\hat{ρ}}}_{ξ r k} - {\hat{ρ}}_{ξ r}$ and $Δ {\tilde{\hat{ρ}}}_{rk - r} = {\tilde{\hat{ρ}}}_{rk} - {\hat{ρ}}_{r}$ plotted for p (number of variables), q (number of factors), µ(Λ_pop), and σ(Λ_pop) as mean and standard deviation of population loadings; the error bars mark the standard errors

Overall, the means of $Δ {\tilde{\hat{ρ}}}_{rk - r}$ were smaller than the means of $Δ {\tilde{\hat{ρ}}}_{ξ rk - ξ r}$ . However, the means of $Δ {\tilde{\hat{ρ}}}_{rk - r}$ and $Δ {\tilde{\hat{ρ}}}_{ξ rk - ξ r}$ both increase with σ(Λ_pop). Accordingly, $Δ {\tilde{\hat{ρ}}}_{rk - r}$ can be used as a lower-bound estimate of $Δ {\tilde{\hat{ρ}}}_{ξ rk - ξ r}$ , that is, when ${\tilde{\hat{ρ}}}_{rk}$ is greater than ${\hat{ρ}}_{r}$ , there are good reasons to expect that ${\tilde{\hat{ρ}}}_{ξ rk}$ is greater than ${\hat{ρ}}_{ξ r}$ .

A short simulation to compare the determinacy of HRFS/RFS and RFS based on oblique population factor models was performed for q = 3 factors with a factor inter-correlation of ϕ = .30, $μ (Λ_{pop}) \in {. 60, . 70}$ , $σ (Λ_{pop}) \in {. 00, . 25, . 50, . 75}$ , p = 6 and n = 600. As before, 1,000 samples were drawn from the population for each condition, and n_d = 50 simulated samples were drawn for each of the 1,000 samples as a basis for the binomial test of loading heterogeneity. For the binomial test, $ϑ_{crit} = 14$ was used, resulting in α_exact = .015. As for orthogonal target-rotation and for Varimax-rotation, ${\tilde{\hat{ρ}}}_{rk}$ was larger than ${\hat{ρ}}_{r}$ and ${\tilde{\hat{ρ}}}_{rk}$ was larger than ${\hat{ρ}}_{r}$ for larger standard deviations of population loadings (see Figure 5). This indicates that the computation of HRFS for oblique factors as described in Equation 23 could be considered for further research.

Figure 5.

Mean determinacy coefficients based on model parameters, RFS ( ${\hat{ρ}}_{k s i r}$ ), HRFS/RFS ( ${\tilde{\hat{ρ}}}_{rk}$ ), and on scores and RFS ( ${\hat{ρ}}_{r}$ ), and on scores, and HRFS/RFS ( ${\tilde{\hat{ρ}}}_{rk}$ ) for three factors (q = 3), p/q = 6 variables per factor for Oblimin-rotated solutions based on correlated population factors with ϕ = .30; µ(Λ_pop) is the expected value of the population loadings; σ(Λ_pop) is the expected value of loading heterogeneity; the error bars mark the standard errors

Empirical Example

The empirical example dataset was based on answers to the 50 IPIP Big Five Factor Markers (Goldberg, 1992), updated at 11/08/2018 and retrieved at 13/11/2024 from https://openpsychometrics.org/_rawdata/. The five factors are Extraversion (E), Agreeableness (A), Conscientiousness (C), Emotional Stability (ES), and Intellect/Imagination (I). Each factor was measured by means of 10 items with five response categories, the direction of item scoring was altered. The dataset was collected from 2016 to 2018 through an interactive online personality test and contained 1,015,342 cases. However, it is recommended in the codebook to use only cases with a single user IP. Accordingly, only 696,854 cases with a single user IP were used. No demographic information was available. One hundred random subsamples with n = 150, n = 600, and n = 1,000 cases were drawn with replacement from the total sample. In the first step, principal axis factor analysis with q = 5 and subsequent orthogonal target-rotation towards the intended five-factor loading pattern was performed as a basis for the computation of ${\hat{ρ}}_{r}$ . In a second step, the analyses were performed for factors that were Varimax-rotated in the total sample. This allowed to investigate the effect of analytic rotation on results. A significance level of α ≤ .20 corresponding to $ϑ_{crit} = 29$ and α_exact = .16 was used for the assessment of loading heterogeneity and the computation of ${\tilde{\hat{ρ}}}_{rk}$ . The corresponding R-script can be found in Supplemental Material B.

The results for the factors based on orthogonal target-rotation in the total sample indicated that ${\tilde{\hat{ρ}}}_{rk}$ was larger than ${\hat{ρ}}_{r}$ for the ES factor in samples of n = 600 and n = 1,000 because the 95% confidence intervals did not overlap (see Figure 6). The same result was found for Varimax-rotated factors (see Figure 7). Thus, for orthogonal target-rotated and Varimax-rotated factors, ES, HRFS/RFS had on average larger determinacies than RFS, and there were no significant differences in determinacy RFS/RFS and RFS for the other factors. However, for Oblimin-rotated factors, ${\tilde{\hat{ρ}}}_{rk}$ was larger than ${\hat{ρ}}_{r}$ not only for ES but also for C and I (see Figure 7).

Figure 6.

Mean parameter-based determinacy of RFS ${\hat{ρ}}_{r}$ and HRFS/RFS ${\tilde{\hat{ρ}}}_{rk}$ for the Target-rotated Big-Five factors E = Extraversion, ES = Emotional Stability, A = Agreeableness, C = Conscientiousness, and I = Intellect/Imagination based on 100 subsamples of n = 150, n = 600, and n = 1,000 drawn randomly from the total sample of 696,854 cases; the error bars mark the 95% confidence interval. Non-overlapping confidence intervals show significant differences in determinacy coefficients between HRFS/RFS and RFS

Figure 7.

Mean parameter-based determinacy of RFS ${\hat{ρ}}_{r}$ and HRFS/RFS ${\tilde{\hat{ρ}}}_{rk}$ for the Varimax- and Oblimin-rotated Big-Five factors E = Extraversion, ES = Emotional Stability, A = Agreeableness, C = Conscientiousness, and I = Intellect/Imagination based on 100 subsamples of n = 150, n = 600, and n = 1,000 drawn randomly from the total sample of 696,854 cases; the error bars mark the 95% confidence interval. Non-overlapping confidence intervals show significant differences in determinacy coefficients between HRFS/RFS and RFS

Hence, the effect of loading heterogeneity on determinacy and the possible advantage of HRFS/RFS over RFS did not only occur for target-rotated factors and was most pronounced for Oblimin-rotated factors. Note that the effect of oblique rotation on the difference of ${\tilde{\hat{ρ}}}_{rk}$ and ${\hat{ρ}}_{r}$ occurred although the absolute magnitude of the mean factor inter-correlations of the Oblimin-rotated factors was rather small. The most negative mean inter-correlation occurred for E with ES (M = -0.17, SD = 0.09, for 100 runs with n = 1,000), and the largest positive mean inter-correlation occurred for E with A (M = 0.19, SD = 0.10, for 100 runs with n = 1,000).

Discussion

Factor score predictors are supposed to estimate the latent value of individuals on the factor of interest. In both research and practical applications, it is important to use scores with maximum validity. The validity of factor score predictors is represented by their determinacy coefficients, that is, their correlation with the underlying factor. Molenaar et al. (2003) found that the determinacy coefficient of factor score predictors was reduced when heterogenous loadings which occur in the population model were not specified in the factor model. Therefore, the present study investigated whether it is possible to improve the determinacy, and thus, the validity of the RFS by specifying heterogeneous loadings between individuals. We propose a method for the estimation of individual factor loadings and for the computation of HRFS. Moreover, we propose a binomial test to ascertain whether heterogeneous loadings are present. We suggest a two-step procedure: First, the binomial test should be conducted. If the binomial test for loading heterogeneity is significant, HRFS should be computed in the second step. Otherwise, RFS should be preferred.

To avoid the effects of rotational indeterminacy on heterogeneous loading estimates, the estimation of heterogeneous loadings was based on orthogonal factors. However, an extension of the estimation of heterogeneous loadings to correlated factor models is also possible. Two extensions are presented: One extension includes heterogeneous factor inter-correlations into the model, and the other extension starts from factor inter-correlations that are fixed across all individuals for the given sample. The latter extension was shortly investigated in the present study because it avoids the complexities of an interpretation of heterogeneous factor inter-correlations.

The conditional computation of HRFS/RFS was compared to RFS by means of a simulation study based on population models with one and three factors and different degrees of loading heterogeneity. Population loading heterogeneity was quantified by the inter-individual standard deviation of loadings (σ(Λ_pop) ∈ ${. 00, . 25, . 50, . 75}$ ). Moreover, loading size, number of factors, number of variables per factor, and sample size were independent variables. Dependent variables of the simulation were factor score-based determinacies and parameter-based determinacies. Score-based determinacies were computed by means of the correlation of RFS and HRFS/RFS with the true factor scores used for the generation of the measured variables. Parameter-based determinacies were computed from the loading estimates and the inter-correlations of the measured variables. To eliminate the effect of rotation methods on results, loadings were based on principal axis factoring with subsequent orthogonal target rotation. To investigate whether the results can be generalized, the simulation was repeated with Varimax-rotated factors for a subset of conditions. A few conditions based on correlated population factors were investigated to provide a first investigation of the extension of the method for the estimation of heterogeneous loadings to oblique factor models.

Score-based determinacies and parameter-based determinacies were similar when there was no population loading heterogeneity (σ(Λ_pop) = .00). When population loading heterogeneity increased, parameter-based determinacy decreased more rapidly than score-based determinacy. As score-based determinacies represent a more direct measure of validity because they are based on the true factors, this indicates that the parameter-based determinacy underestimates the true determinacy under conditions of population loading heterogeneity. Moreover, with increasing population loading heterogeneity, parameter-based determinacy and score-based determinacy decreased more for RFS than for HRFS/RFS. This indicates that the latter can be recommended when the test of loading heterogeneity is significant, as we suggest by our two-step procedure. Larger loading heterogeneity resulted in larger differences of score-based determinacy between HRFS/RFS and RFS as well as larger differences of parameter-based determinacy between HRFS/RFS and RFS. Therefore, the difference of the parameter-based determinacies of HRFS/RFS and RFS can be used as an estimate for the difference of the score-based determinacies of HRFS/RFS and RFS. This is relevant because in empirical settings, the score-based determinacy cannot be computed and researchers need to rely on parameter-based determinacies. Moreover, the results of the simulation study were similar when Varimax-rotated loadings were computed instead of target-rotated loadings in the total sample, indicating that the results can be generalized to different rotation methods. It was also found in the simulation for the oblique extension of HRFS that increasing population loading heterogeneity led to a more substantial decrease in parameter-based and score-based determinacy for RFS than for HRFS/RFS. Although a more thorough investigation is needed this indicates that the oblique extension of HRFS could be of interest for further research.

The empirical example based on random subsamples drawn from a large online sample of Big Five Factor Markers (Goldberg, 1992) revealed that for moderate sample sizes (n = 600) and for large sample sizes (n = 1,000), the parameter-based determinacy of the factor ES was larger for HRFS/RFS than for RFS. This result was found for target-rotated factors, Varimax-rotated, and for the oblique extension of HRFS based on Oblimin-rotated factors. As the result was only found for moderate to large samples, it is unlikely that it is due to sampling error. For Oblimin-rotated factors and moderate to large samples, the parameter-based determinacy was larger for HRFS/RFS than for RFS also for the factors C and I. This indicates that considering the obliqueness of the factors may enhance the positive effect of heterogeneous loadings on determinacy.

The results of the empirical example indicate that the computation of HRFS/RFS is also feasible for five-factor models. An R-script for the computation of HRFS/RFS and RFS, the result of the binomial test of loading heterogeneity, and the determinacies for a single empirical dataset with one to five Varimax-rotated, orthogonal target-rotated, or Oblimin-rotated factors are given in Supplemental Material C.

As the determinacy of HRFS/RFS was larger than the determinacy of RFS when loading heterogeneity occurred in the population, the HRFS may be included into structural equation models (SEMs) when loading heterogeneity can be assumed according to the binomial test for the measurement models. However, maximal determinacy does not necessarily result in a minimal bias of structural parameters. For example, the RFS has maximal determinacy and considerable bias (Skrondal & Laake, 2001). Of course, minimal bias is also a desired property, especially when factor scores are used in SEM (Croon, 2002). If, however, maximal determinacy of the factor scores is intended, and if the binomial test of loading heterogeneity is significant the HRFS may be included into SEM, although the issue of minimal bias for HRFS should be investigated in further studies. If the HRFS is used in SEM, it is recommended to compute HRFS from loading heterogeneity in measurement models estimated before the structural part of the model, as in the structural after measurement approach (Rosseel & Loh, 2024). Beyond the reasons for separate estimation of measurement models given in Rosseel and Loh (2024), it should be noted that the estimation of loading heterogeneity in separate measurement models minimizes the effects of capitalization of chance. As individual factor loadings directly affect individual factor scores, a least squares prediction between two factors may result in biased individual loadings allowing for a larger prediction.

The individual factor loadings estimated as a basis for HRFS may also be interesting from a theoretical perspective as they may indicate the individual relevance of a factor for the response to an item. In trait activation theory, situational cues have been emphasized as a determinant of trait activation (Tett et al., 2021). From this perspective, the individual loading of an item could be an indicator for the individual trait-relevance of the situational content of a questionnaire item. Interestingly, ES appeared especially sensitive to situational moderation in Tett et al. (2021) and it had the largest increase of determinacy when computing HRFS/RFS instead of RFS. This perspective might be explored in future studies, where the individual loadings may be predicted by external variables or may be used as predictors for behavioral outcomes. Of course, if individual loadings are to be used in this way, it would be necessary to estimate their reliability (e.g., by means of test-retest correlations). As a limitation, it should be noted that the focus of the present study was on orthogonal factors. The reason for this focus was that, first, a procedure is applicable to orthogonal factors was to be found before an extension to oblique factors. Two ways to extend the procedure to oblique factors were outlined, and an extension based on constant factor inter-correlations was tentatively evaluated. The results—especially for the empirical example—are promising but they are preliminary as there was no room for a larger simulation study on correlated factors. Moreover, it might be of interest to use the individual loading estimates based on the binomial test for loading heterogeneity for the computation of HRFS combined with Croon’s (2002) bias correction, Bartlett factor scores, for correlation-preserving factor scores (Beauducel et al., 2024). A further limitation of the present study is that only Varimax-, Oblimin-, and orthogonal target rotation were investigated, which calls for an extensive investigation of loading heterogeneity for different methods of factor rotation. Finally, measurement invariance of factor scores is also an issue (Lai & Tse, 2024). Obviously, the HRFS is devoted to conditions where measurement invariance does not occur across individuals. However, loading heterogeneity at the level of the individual does not preclude that the mean loadings are identical for different groups of individuals. Accordingly, a comparison of measurement invariance of RFS and HRFS across groups could be an interesting aim for future research.

Conclusion

The present research revealed that the negative effect of loading heterogeneity on the determinacy (validity) of RFS can be reduced when individual loadings are accounted for by computing HRFS. We propose a two-step procedure: First, a binomial test for loading heterogeneity is proposed. If the binomial test for loading heterogeneity is significant, we recommend the computation of HRFS. In our simulation study, the resulting conditional HRFS/RFS computation yielded an improvement of the factor score determinacy over the determinacy of the RFS in population models based on loading heterogeneity. An empirical example based on subsamples drawn randomly from a large Big Five Marker data set revealed that the ES factor may have substantial loading heterogeneity and that the determinacy may be improved by the conditional computation of HRFS/RFS.

Supplemental Material

sj-docx-1-epm-10.1177_00131644251347530 – Supplemental material for How to Improve the Regression Factor Score Predictor When Individuals Have Different Factor Loadings

Supplemental material, sj-docx-1-epm-10.1177_00131644251347530 for How to Improve the Regression Factor Score Predictor When Individuals Have Different Factor Loadings by André Beauducel, Norbert Hilger and Anneke C. Weide in Educational and Psychological Measurement

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the German Research Foundation (DFG), BE 2443/18-1.

ORCID iDs

André Beauducel

Anneke C. Weide

Supplemental Material

Supplemental material for this article is available online.

References

Ansari

Jedidi

Dube

(2002). Heterogeneous factor models: A Bayesian approach. Psychometrika, 67, 49–78. https://doi.org/10.1007/BF02294709

Bartlett

M. S.

(1937). The statistical conception of mental factors. British Journal of Psychology, 28(1), 97–104. https://doi.org/10.1111/j.2044-8295.1937.tb00863.x

Beauducel

Hilger

Kuhl

(2024). The trade-off between factor score determinacy and the preservation of inter-factor correlations. Educational and Psychological Measurement, 84(2), 289–313. https://doi.org/10.1177/00131644231171137

Browne

M. W.

(2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36(1), 111–150. https://doi.org/10.1207/S15327906MBR3601_05

Cattell

R. B.

(1952). The three basic factor analytic research designs: Their intercorrelations and derivatives. Psychological Bulletin, 49(5), 499–520. https://doi.org/10.1037/h0054245

Cook

R. D.

(1977). Detection of infuential observations in linear regression. Technometrics, 19, 15–18. https://doi.org/10.1080/00401706.2000.10485981

Croon

(2002). Using predicted latent scores in general latent structure models. In Marcoulides

Moustaki

(Eds.), Latent variable and latent structure modeling (pp. 195–223). Lawrence Erlbaum.

Goldberg

L. R.

(1992). The development of markers for the Big-Five factor structure. Psychological Assessment, 4(1), 26–42. https://doi.org/10.1037/1040-3590.4.1.26

Grice

J. W.

(2001). Computing and evaluating factor scores. Psychological Methods, 6(4), 430–450. https://doi.org/10.1037/1082-989X.6.4.430

10.

Guttman

(1955). The determinacy of factor score matrices with applications for five other problems of common factor theory. British Journal of Statistical Psychology, 8, 65–82. https://doi.org/10.1111/j.2044-8317.1955.tb00321.x

11.

Kelderman

Molenaar

P. C. M.

(2007). The effect of individual differences in factor loadings on the standard factor model. Multivariate Behavioral Research, 42(3), 435–456. https://doi.org/10.1080/00273170701382997

12.

Krijnen

W. P.

Wansbeek

T. J.

Ten Berge

J. M. F.

(1996). Best linear predictors for factor scores. Communications in Statistics: Theory and Methods, 25(12), 3013–3025. https://doi.org/10.1080/03610929608831883

13.

Lai

M. H. C.

Tse

W. W.-Y.

(2024). Are factor scores measurement invariant? Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000658

14.

Molenaar

P. C. M.

(2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement: Interdisciplinary Research and Perspectives, 2(4), 201–218. https://doi.org/10.1207/s15366359mea0204_1

15.

Molenaar

P. C. M.

Campbell

C. G.

(2009). The new person-specific paradigm in psychology. Current Directions in Psychological Science, 18(2), 112–117. https://doi.org/10.1111/j.1467-8721.2009.01619.x

16.

Molenaar

P. C. M.

Huizenga

H. M.

Nesselroade

J. R.

(2003). The relationship between the structure of interindividual and intraindividual variability: A theoretical and empirical vindication of developmental systems theory. In: Staudinger

U. M.

Lindenberger

(Eds.), Understanding human development (pp. 339–360). Springer. https://doi.org/10.1007/978-1-4615-0357-6_15

17.

Mulaik

S. A.

(2010). Foundations of factor analysis (2nd ed.). CRC Press.

18.

Nicewander

W. A.

(2020). A perspective on the mathematical and psychometric aspects of factor indeterminacy. Multivariate Behavioral Research, 55(6), 825–838. https://doi.org/10.1080/00273171.2019.1684872

19.

Rosseel

Loh

W. W.

(2024). A structural after measurement approach to structural equation modeling. Psychological Methods, 29(3), 561–588. https://doi.org/10.1037/met0000503

20.

Schmid

Leiman

J. M.

(1957). The development of hierarchical factor solutions. Psychometrika, 22, 53–61. https://doi.org/10.1007/BF02289209

21.

Schoenemann

P. H.

(1966). A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1), 1–10. https://doi.org/10.1007/BF02289451

22.

Skrondal

Laake

(2001). Regression among factor scores. Psychometrika, 66(4), 563–575. https://doi.org/10.1007/BF02296196

23.

Tett

R. P.

Toich

M. J.

Ozkum

S. B.

(2021). Trait activation theory: A review of the literature and applications to five lines of personality dynamics research. Annual Review of Organizational Psychology and Organizational Behavior, 8(1), 199–233. https://doi.org/10.1146/annurev-orgpsych-012420-062228

24.

Thurstone

L. L.

(1935). The vectors of mind. University of Chicago Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB