Reliability of test battery

Abstract

After a brief review of methods for assessing the reliability of individual test, the article presents a method of obtaining reliability of a battery of tests where battery score could be defined as sum of weighted or unweighted scores of the component tests. Such battery reliability can be influenced significantly by method of selection of weights to arrive at the battery scores and methods of estimating reliability of component tests. Test reliability as per theoretical definition in terms of length of score vectors of two parallel tests and angle between such vectors in N-dimensional space also helps to find values of error score variance of the test fits well in estimation of battery reliability. Relationship between theoretically defined reliability r_tt and split-half correlation r_gh was established. For weighted battery score, a Lagrange multiplier-based solution for determination of weights is recommended with the use of reliability as per the theoretical definition. Weights found as above have the advantage that the battery score (Y) has minimum variance. Also, covariance between the battery score and the score of an individual test is a constant. Condition of battery score being equi-correlated with the standardized score of each constituent test was derived.

Introduction

In practical situations, a battery of tests is administered instead of a single test. Single tests may not be adequate to detect appropriately the required latent abilities. For example, aptitudes are made of a complex of abilities. Intelligence tests could be a combination of verbal and numerical aptitudes among others. Most diagnostic questions require the assessment of personality, intelligence, and perhaps even the presence of organic involvement. Thus, a battery of tests is needed to measure complex abilities and to identify individual’s strengths and weaknesses. In a battery, a set of component tests are grouped together and administered in situations like selection of personnel for employment, Health-Related Physical Fitness Test (HRPFT) for jobs, sports, and even for assessment of fitness of Astronauts, mental-health issues, and so on. Development of cognitive battery for clinical trials of cognition-enhancing treatments for schizophrenia was highlighted by Mesholam-Gately et al. (2008). Kassin (2003) observed that application of factor analysis to the study of trait organization has provided the theoretical basis for the construction of test batteries. Examples of such batteries include among others SRA Primary Mental Abilities (PMA), Differential Aptitude Tests (DAT), Guilford-Zimmerman Aptitude Test, and so on.

While estimation of reliability by various methods and error variance derived from reliability for individual test are widely discussed (Anastasi and Urbina, 1997; Franzen, 2000; Murphy and Davidshofer, 2005), evaluation of reliability of test battery as a whole along with properties does not appear to have been accomplished to a great extent. This is despite the fact that combining tests into a battery in no way reduces the potential for measurement error associated with both the individual test and the battery.

Thus, need is felt to estimate reliability of a battery as a function of reliability of each component test along with estimation of true score variance of the battery, considering battery score as a weighted sum of component tests and avoiding scaling of scores of component tests.

Rest of the article is organized as follows. Previous studies on measures of reliability of test and battery are elaborated in the following section. Conceptual framework for battery scores and issues relating to scaling of scores of component tests and aggregation methods are elaborated in Section “Battery scores.” Section “Battery reliability” deals with theoretical details regarding choice of methods of finding reliability of the component tests as per theoretical definition and associated properties. This was followed by method of the computation of the reliability of a battery of tests using summative scores and weighted scores with discussion on recommended method of finding weights which minimizes variance of the weighted sum. The article is rounded up in Section “Summary and discussion,”, by recalling the salient outcomes of the work.

Literature survey

Reliability of an individual test has been pursued in a variety of ways aiming to indicate consistency, precision, repeatability, trustworthiness, and so on of a test. Reliability as degree of consistency of test scores, that is, repeatability, has been recommended by Berkowitz et al. (2000). However, different values of test–retest reliability of a single test may be obtained depending on the time gap between the administrations. Cronbach’s alpha is widely used to find reliability as a measure of internal consistency. However, it is necessary to check that the data fit the unidimensional model before calculating alpha (Trizano-Hermosilla and Alarado, 2016). Violation of assumption of tau-equivalence will underestimate α (Graham, 2006; Raykov, 1997). Working with data which comply with this assumption is generally not viable in practice (Teo and Fan, 2013). Wilcox (1992) showed how coefficient alpha is vulnerable to modest numbers of outlying observations and may substantially inflate alpha. Limitations of Cronbach’s alpha have also been reported by Hattie (1985) and Ritter (2010). Parallel test or split-half reliability considers correlation of two parallel sub-tests obtained by dichotomization of the test score in one session. A major limitation of the split-half reliability is that the resulting estimate is not unique and depends on how exactly the split was carried out. Indeed, different splits into halves of an initial multiple-component instrument can well yield different reliability estimates. Zimmerman (2007) favored expressing reliability in terms of standard error of measurement (SEM). Thus, no popular method of finding reliability uses the theoretical definition of reliability which is defined as the ratio of the true score variance and observed score variance. Consequently, multiple values of the error variance and reliability can be obtained for the same test even if the sample remains unchanged. Reliability that conforms to the theoretical definition cannot be computed since true scores of individuals taking the test are not known. Moreover, estimates of reliability are themselves vulnerable to measurement error (Vacha-Haase et al., 2000).

The above are also applicable for estimating reliability of a battery of tests, where battery score is taken as sum or weighted sum of component tests scores. Reliability of the battery can be influenced significantly by method of selection of weights to arrive at the battery scores and methods of estimating reliability of component tests, since sources of errors of individual tests may get manifold for the battery. For estimation of reliability of a battery, Lubans et al. (2011) used test–retest approach. Berchtold (2016) opined that while reliability is the ability of a measure applied twice upon the same respondents to produce the same ranking on both occasions, agreement requires not only to preserve the relative order of the respondents in the two sets of measurements but also the same exact result that each respondent obtains on the two testing situations. Battery reliability using weighted sum of component tests was attempted by researchers like Wang and Stanley (1970), Feldt and Brennan (1989), Rudner (2001), Webb et al. (2007), and others using different approaches and scaling. However, the approaches gave rise to different values of battery reliability, especially when individual components measure multidimensional constructs. Rudner (2001) indicated that adding raw scores fails to recognize the relative importance of components to the overall composite; weights proportional to reliability of the components may results in lower SEM of the composite score in comparison to simple summative scores. In case of availability of pre-specified, quantifiable external criterion, weights of component tests may be found by linear multiple regressions with the criterion. Such method of finding weights may result in maximum validity of the composite score. Canonical correlation allows an extension of the parallel test situation and split-half technique. Safrit and Wood (2013) and Koch et al. (2003) used canonical correlation analysis (CCA) where randomized subsets were created first which were then correlated using canonical correlation technique. Essence of CCA is to form pairs of linear combinations of predictor and criterion variables to maximize the correlation between each pair. In the context of reliability of a battery, there may not exist unique efficient procedure to obtain predictor and criterion variables. In addition, CCA technique finds linear relationships between variables within a set and between pairs of canonical variates (linear pairings of canonical variates, one from each of the two sets of variables). Non-linear components of these relationships, if any, are not recognized and are not captured in the analysis. Outliers have mixed impact on the results of the CCA, and each set of variables is inspected independently for univariate and multivariate outliers. Because canonical correlation is very sensitive to small changes in the data set, the decision to eliminate cases must be made very carefully. Both multicollinearity (when variables are very highly correlated) and singularity (when one variable is a linear combination of two or more variables in that set) should be eliminated before analysis proceeds. Precautions for using CCA were mentioned by Lambert and Durand (1975). Equivalences among canonical factor analysis, canonical reliability, and principal component analysis were studied by Conger and Stallard (1976); they found that if variables are scaled so that they have equal measurement errors, then canonical factor analysis on all non-error variance, principal component analysis, and canonical reliability analysis give rise to equivalent results. However, the assumption of equal measurement errors is abstract and too difficult to achieve.

Battery scores

Battery scores can be viewed as a composite score of component tests, that is, aggregation of scores of component tests to a single value for an examinee. Stages for construction of composite score/index involve scaling of test scores and deciding appropriate method of combining.

Major points to be noted are as follows: (1) Component tests could be independent or dependent with various degrees; (2) Battery score could be weighted or unweighted sum of scores of component tests; (3) Reliability of the battery can be influenced significantly by reliability of each component test and also by the method of arriving at the battery scores; (4) In addition to battery reliability, it will be desirable to find estimates of true score variance and/or error score variance of the battery; (5) Reliability based on internal consistency may be relevant for an individual test but not for battery due to multidimensionality of a test battery.

Scaling

Usual practice is to consider standardized scores of component tests by subtracting from the sample mean and dividing by the sample standard deviation (SD). Distribution of standardized score of a test will follow normal distribution for large sample, even if distribution of original test score may be different from normal. There are many other ways to normalize test scores. Illustrative list is as follows:

1. $Z=(X-X_{min}/X_{max}-X_{min}).$ Here, standardization is based on the range rather than the SD. But the extreme values (minimum and maximum) may be unreliable outliers.

2. $Z=(X_i/\overline{X})X100$ . Here, the units receive score depending on their distance from the mean. Statistically, it is less robust to the influence of outliers than some of the other methods.

3. $Z=(X_i/X_{max})X100$ . Here, percentage difference from the maximum score is considered.

4. Z= log(x); x > 0 to reduce skewness of (positive) data.

Note that there is no best method of scaling or normalization or transformations of component score. However, scaling will have significant effects on the composite score, its distribution, and battery reliability. Hence, it is felt desirable to find composite battery score avoiding scaling of component scores.

Combining method

Usual methods of combining scores of component scores may include (1) Summative score, that is, battery score of an individual is the sum of his or her score in each component test. This considers all tests are of equal importance. This allows compensability among component tests, which may not be desirable or (2) Weighted sum which assumes linear model with implication of full compensability, that is, poor performance in some component tests can be well compensated by high values in other tests. However, there are different methods of deciding weights even under the condition $\Sigma W_i=1\mathrm{and}0\le W_i\le 1.$ Chakrabartty (2017) elaborated the problem of subjective weights, even under ${\Sigma }^{}{W}_i=1$ with examples. However, selection of data-driven weights needs to consider among others desired outcomes of the weighted sum. Non-linear combination of test scores to find battery scores cannot be ruled out.

Battery reliability

Since battery reliability will be function of reliabilities of the component tests, it may be prudent to select a method of finding reliability of a component test as per the definition, that is, as the ratio of true score variance $(S_T^2)$ and observed score variance $(S_X^2)$ or as $1-(S_E^2/S_X^2)$ where $S_E^2$ denotes the error score variance.

Reliability of tests

Chakrabartty (2013) proposed a method of obtaining test reliability as per the theoretical definition along with computation of error variance and true score variance from single administration of the test. The method uses lengths of two parallel tests and angle between the two vectors representing scores of the two parallel tests. A test consisting of n-items administered among N persons, when dichotomized in parallel halves say g-th test and h-th test, results in two points ${\mathit{X}}_{\mathit{g}}$ and ${\mathit{X}}_{\mathit{h}}$ in the N-dimensional space. As per classical definition, two test “g” and “h” are parallel if $T_{gi}=T_{hi}$ and $S_{eg}=S_{eh}$ . It can be proved that if g and h are parallel, ${\overline{X}}_g={\overline{X}}_h$ and $S_{Xg}^2=S_{Xh}^2.$

Also, $X_g=T_g+E_g$ and $X_h=T_h+E_h$ . Now $T_{gi}=T_{hi}$ implies $X_{gi}-X_{hi}=E_{gi}-E_{hi},$ so that

$$\begin{array}{l}{||X_g||}^2+{||X_h||}^2-2||X_g||||X_h||\cos {\theta }_{gh}\\ ={||E_g||}^2+{||E_h||}^2-2||E_g||||E_h||\cos {\theta }_{gh}^{\left(E\right)}\end{array}$$

where ${\theta }_{gh}$ is the angle between the vectors ${\mathit{X}}_{\mathit{g}}$ and ${\mathit{X}}_{\mathit{h}}$ while ${\theta }_{gh}^{(E)}$ is the angle between $E_g$ and $E_h.$ But correlation between error scores of two parallel tests is zero. Thus,

$${||X_g||}^2+{||X_h||}^2-2||X_g||||X_h||\cos {\theta }_{gh}={||E_g||}^2+{||E_h||}^2=N{S}_E^2$$

since $S_E^2=\frac{1}{N}{\sum }^{}{\left(E_{gi}+E_{hi}\right)}^2=\frac{1}{N}\left[{||E_g||}^2+{||E_h||}^2\right]$

The above equation suggests

$$S_E^2=\frac{1}{N}\left[{||X_g||}^2+{||X_h||}^2-2||X_g||||X_h||\cos {\theta }_{gh}\right]$$

(1)

where ${\theta }_{gh}$ is the angle between the vectors $X_g$ and $X_h$ . True score variance $S_T^2$ can be computed as $S_X^2-S_E^2$ .

Hence, theoretical test reliability $r_{tt}=S_T^2/S_X^2=(1-S_E^2/S_X^2)$ is given by

$$r_{tt}=1-\frac{{||X_g||}^2+{||X_h||}^2-2||X_g||||X_h||\cos {\theta }_{gh}}{N{S}_X^2}$$

(2)

Since, it can be proved that parallel tests are of equal length, the formula (1) and (2) can be further simplified as

$$S_E^2=\frac{2{||X_g||}^2}{N}\left(1-\cos {\theta }_{gh}\right)$$

(3)

and

$$r_{tt}=1-\frac{2{||X_g||}^2}{N{S}_X^2}\left(1-\cos {\theta }_{gh}\right)$$

(4)

Equation (1) helps to find value of error variance of the test and hence true score variance as $\left(S_X^2-S_E^2\right)$ and use them directly to get equation (2) to find reliability of the test as per the theoretical definition from a single administration, in terms of length of score vectors of two parallel tests and angle between such vectors in N-dimensional space. Thus, it is possible to find true score variance from the data and to calculate a reliability coefficient that conforms to the theoretical definition even if true scores or error scores of individuals taking the test are not known.

Properties of theoretically defined reliability

1. Equations (1) and (2) were derived assuming dichotomization of a test in parallel halves. However, in practice, it may be difficult to dichotomize a test where means and variances of scores exactly match. Thus, the above two equations work well even if there are marginal differences in length of the vectors ${\mathit{X}}_{\mathit{g}}$ and ${\mathit{X}}_{\mathit{h}}$ .

2. Correlation between two parallel sub-tests, $r_{gh}$ , as an estimate of split-half reliability may have value different from value of $r_{tt}$ .

3. Equation (1) gives a way to find error variance of the test avoiding popular method of first finding test reliability not conforming to the definition and then using $S_E^2=S_X^2\left(1-r_{tt}\right)$ .

4. If g-th test and h-th tests are parallel, $S_E^2=2S_{Xg}^2\left[1-r_{gh}\right]$

Proof: equation (3.2) can be re-written as

$$\begin{array}{c}{S}_E^2=\frac{1}{N}{\sum }^{}{X}_{gi}^2+\frac{1}{N}{\sum }^{}{X}_{hi}^2-\frac{2}{N}{\sum }^{}{X}_{gi}{X}_{hi}\\ =2S_{Xg}^2-2\mathrm{cov}\left(X_g,X_h\right)\mathrm{since}{\overline{X}}_g={\overline{X}}_h\mathrm{and}{S}_{Xg}=S_{Xh}\\ =2S_{Xg}^2\left(1-r_{gh}\right)\end{array}$$

(5)

where $r_{gh}$ denotes correlation between g-th and h-th tests, that is, split-half reliability.

• 5. If g-th and h-th tests are parallel, theoretically defined reliability $r_{tt}\ge r_{gh}$

Proof: $X=X_g+X_h\Rightarrow \overline{X}={\overline{X}}_g+{\overline{X}}_g$ and ${\overline{X}}^2=4{{\overline{X}}_g}^2$ since ${\overline{X}}_g={\overline{X}}_h$

$$\begin{array}{c}{S}_X^2=S_{Xg}^2+S_{Xh}^2+2\mathrm{cov}\left(X_g,X_h\right)\\ =2S_{Xg}^2+2r_{gh}{S}_{Xg}^2\mathrm{since}{S}_{gh}=S_{Xh}\\ =2S_{Xg}^2\left(1+r_{gh}\right)\end{array}$$

Thus,

$$\begin{array}{c}{S_T}^2=S_X^2-S_E^2\\ =2S_{Xg}^2\left(1+r_{gh}\right)-2S_{Xg}^2\left(1-r_{gh}\right)=4r_{gh}{S}_{Xg}^2\end{array}$$

And theoretically defined reliability is

$$r_{tt}=\frac{{S_T}^2}{S_X^2}=\frac{4r_{gh}{S}_{Xg}^2}{2S_{Xg}^2\left(1+r_{gh}\right)}=\frac{2r_{gh}}{1+r_{gh}}$$

(7)

Equation (7) gives relationship between theoretically defined reliability $r_{tt}$ and split-half correlation $r_{gh}$ .

From (7), $r_{tt}/r_{gh}=2/1+r_{gh}$ , which is greater than or equal to one.

$$\frac{r_{tt}}{r_{gh}}\ge 1\Rightarrow r_{tt}\ge r_{gh}$$

• 6. For a given data set, sample values of $S_E^2,{S_T}^2,S_X^2$ are likely to differ for different samples. Unbiased and consistent estimate of observed score is ${\sigma }_X^2=(1/N-1){\sum \left(X_t-\overline{X}\right)}^2$ and can be written as $(N/N-1)S_X^2$ . Following a similar approach, one can find ${\sigma }_T^2$ and ${\sigma }_E^2$ with the knowledge of $S_E^2$ from equation (1) and ${S_T}^2$ as $\left(S_X^2-S_E^2\right)$ .

Estimation of battery reliability

The method of finding test reliability as per the classical definition can be extended to find the reliability of a battery of tests. The method of obtaining the reliability of the battery will depend on the selected method of reliability of the component tests and on the definition of the battery scores. Usually, battery scores are computed as the sum of the scores in the individual constituent tests (summative scores) or as a weighted sum of these tests scores. Here, we omit the discussion of difference scores–based battery computation for brevity’s sake.

Weighted sum of component tests

Suppose a battery consists of two tests: Test 1 and Test 2. For the i-th person, let $X_{1i}\mathrm{and}{X}_{2i}$ denote, respectively, the battery score, score obtained in Test 1 and Test 2. Let this examinee’s true scores and error score of the j-th constituent test be $T_{ji}$ and $E_{ji}$ , respectively, j = 1, 2. Then the summative score of the i-th examinee is $X_i=X_{1i}+X_{2i},$ assuming additivity and equal weights. Let $r_{t{t}_1}$ and $r_{t{t}_2}$ denote reliability of the respective constituent tests.

Now, $\mathrm{var}\left(X\right)=\mathrm{var}\left(X_1\right)+\mathrm{var}\left(X_2\right)+2\mathrm{cov}\left(X_1,X_2\right)$ suggests

$${\sum }^{}{X}_{1i}{X}_{2i}={\sum }^{}\left(T_{1i}+E_{1i}\right)\left(T_{2i}+E_{2i}\right)={\sum }^{}{T}_{1i}{T}_{2i}$$

since ${\sum }^{}{T}_{1i}{E}_{2i}={\sum }^{}{T}_{2i}{E}_{1i}={\sum }^{}{E}_{1i}{E}_{2i}=0$

Thus,

$$\begin{array}{c}\mathrm{cov}\left(X_1,X_2\right)=\frac{{\sum }^{}{X}_{1i}{X}_{2i}}{N}-{\overline{X}}_1{\overline{X}}_2\\ =\frac{{\sum }^{}{T}_{1i}{T}_{2i}}{N}-{\overline{T}}_1{\overline{T}}_2\\ =\mathrm{cov}\left(T_1,T_2\right)\end{array}$$

Now variance of true score of the battery is

$$\begin{array}{c}{S}_{T(Battery)}^2=\mathrm{var}\left(T_1\right)+\mathrm{var}\left(T_2\right)+2\mathrm{var}(T_1,T_2)\\ =\mathrm{var}\left(T_1\right)+\mathrm{var}\left(T_2\right)+2\mathrm{cov}(X_1,X_2)\\ =r_{t{t}_1}{S}_{X_1}^2+r_{t{t}_2}{S}_{X_2}^2+2\mathrm{cov}(X_1,X_2)\end{array}$$

(9)

Thus, reliability of a battery is given by

$$r_{tt(Battery)}=\frac{r_{t{t}_1}{S}_{X_1}^2+r_{t{t}_2}{S}_{X_2}^2+2\mathrm{cov}(X_1,X_2)}{S_{X_1}^2+S_{X_2}^2+2\mathrm{cov}(X_1,X_2)}$$

(10)

In general, if a battery has K-tests which are equally important and the battery score is a summative score of the K-constituent tests, then the true score variance of the battery and reliability of the battery will be given, respectively, by

$$r_{tt(Battery)}=\frac{{\sum }_{i=1}^K{r}_{t{t}_i}{S}_{X_i}^2+2{\sum }_{i=1,i\ne j}^K{\sum }_{j=1}^{K}2\mathrm{cov}(X_i,X_j)}{{\sum }_{i=1}^K{S}_{X_i}^2+{\sum }_{i=1,i\ne j}^K{\sum }_{J=1}^{K}2\mathrm{cov}(X_i,X_j)}$$

(11)

The idea can be extended to find reliability of battery where battery score is weighted sum of scores of the constituent tests.

Consider a battery of K-tests and weights $W_1,W_2,\dots \dots ..,W_k$ where $W_i$ > 0 ∀i = $1,2,\dots \dots \dots .,K$ and ${\sum }_{i=1}^k{W}_i=1.$ To find reliability of such a battery, let us consider the composite score or the battery score $Y={\sum }_{i=1}^k{W}_{i}X$ where $X_i$ is the score of the $i-\mathrm{th}$ constituent test and $W_i$ is the corresponding weight.

Clearly, $\mathrm{var}\left(Y\right)={\sum }_{i=1}^k{W}_i^2\mathrm{var}\left(X_i\right)$

Then

$$r_{tt(Battery)}=\frac{\begin{array}{l}{\sum }_{i=1}^K{r}_{t{t}_i}{W}_i^2{S}_{X_i}^2+\\ 2{\sum }_{i=1,i\ne j}^K{\sum }_{j=1}^{K}2W_i{W}_j\mathrm{cov}\left(X_i,X_j\right)\end{array}}{\begin{array}{l}{\sum }_{i=1}^K{W}_i^2{S}_{X_i}^2+\\ {\sum }_{i=1,i\ne j}^K{\sum }_{J=1}^{K}2W_i{W}_j\mathrm{cov}\left(X_i,X_j\right)\end{array}}$$

(12)

$$S_{T(Battery)}^2=\sum _{i=1}^K{r}_{t{t}_i}{S}_{X_i}^2+\sum _{i=1,i\ne j}^K\sum _{j=1}^{K}2\mathrm{cov}\left(X_i,X_j\right)$$

(13)

and

$$S_{E(Battery)}^2=S_{T(Battery)}^2-S_{X(Battery)}^2$$

(14)

Equation (12) helps to find reliability of a test battery consisting of K-number of tests where battery score is defined as weighted sum of component tests. True score variance of the battery can be estimated using equation (13). $S_{E(Battery)}^2$ can also be estimated using equation (14)

However, different choices of weights will give different values of the test battery. Thus, a major problem in the computation of battery reliability could be experienced in determination of the weights.

Proposed methods for determination of weights

Selection of weights is a central issue for weighted sum approach. Weighted sum approaches assume additive models. The weights which minimizes variance of weighted sum are referred as ideal weights as inverse of the variance by Hartung et al. (2008). Shahar (2017) gave three proofs (using method of Lagrange multipliers, Induction method, and Cauchy-Schwarz inequality) of the ideal weights that minimize the variance of a weighted average. His proposition was that if the component indicators ${X_i}^{\prime }s$ for i=1, 2, . . . , n are uncorrelated then $\mathrm{var}({\sum }_{i=1}^n{W}_i{X}_i)$ is minimized when $W_i=[(1/\mathrm{var}(X_i)]/{\sum }_{i=1}^n[1/\mathrm{var}(X_i)]$ and its minimum value is $1/{\sum }_{i=1}^n[1/\mathrm{var}(X_i)]$

Major observations are as follows:

1. The component indicators may be independent or correlated with varying degrees. Hence, need is felt to find weights from a general perspective without assuming that variables are independent or they follow certain distributions and to satisfy one or more desired properties of the weighted sum.

2. Minimum variance is not the only criteria for a good weighted battery score. Determination of weights depends on desired outcomes of the weighted sum. Depending on purpose, one may like to find weights which are (1) proportional to covariance of variables and the weighted sum or (2) proportional to covariance and inversely proportional to SDs or 3. Weighted sum is equi-correlated with the component tests and so on.

The method of findings weights from a general perspective without assuming that variables are independent or they follow certain distributions is based on the following proposition:.

• Proposition 1. Let $X_1,X_2,\dots \dots .,X_n$ be n-variables with varying degree of correlations. If the weighted sum $Y=\sum _{i=1}^n{w}_i{X}_i$ where $W={\left(w_1,w_2,\dots ..,w_n\right)}^T$ is the vector of weights such that $W^{\prime }e=1$ , then $\mathrm{var}\left(Y\right)=W^{\prime }SW$ is minimum if $W=S^{-1}e/e^{\prime }{S}^{-1}e$ where $S_{nxn}$ is the variance–covariance matrix of $X_1,X_2,\dots \dots ..,X_n$ .

Proof: Consider $A=W^{\prime }SW+\lambda \left(1-W^{\prime }e\right)$ where λ is a Lagrangian multiplier

Clearly, $\partial A/\partial W=2SW-\lambda e=0$ and $\partial A/\partial \lambda =1-W^{\prime }e=0$

Solving the above two equations on the assumption S is non-singular,

$$W=\frac{\lambda S^{-1}e}{2}\Rightarrow W=\frac{S^{-1}e}{e^{\prime }{S}^{-1}e}\mathrm{and}\lambda =\frac{2}{e^{\prime }{S}^{-1}e}$$

Properties

• 1. Weights obtained from the above method minimizes variance of the weighted sum

and $\mathrm{var}\left(Y\right)=W^{\prime }SW=\frac{1}{e^{\prime }{S}^{-1}e}$

• 2. Covariance between the weighted sum Y and any component test $X_i$ is constant $\forall i=1,2,\dots \dots ,n$

• 3. If we take normalized values ${Z_i}^{\prime }s$ of ${X_i}^{\prime }s$ , the variance-covariance matrix S is reduced to correlation matrix R, and the new weight vector $W-(R^{-1}/e^1{R}^{-1}e)$ satisfies

$${\mathrm{\Upsilon }}_{\left(Y,Z_i\right)}={\mathrm{\Upsilon }}_{\left(Y,Z_j\right)}=\frac{1}{\sqrt{e^{\prime }{R}^{-1}e}}\forall i\ne j$$

• Proposition 2. If the weight vector $W={\left(w_1,w_2,\dots \dots ,w_n\right)}^T$ satisfies the conditions $0\le w_i\le 1,{\sum }_{i=1}^n{w}_i=1$ and $w_i\propto \mathrm{cov}\left(Y,X_i\right)$ then W is the eigen vector corresponding to the maximum eigen value of the variance-covariance matrix S.

Proof: It can be proved that weights vector in this context are eigen vectors corresponding to the maximum eigen values of the variance-covariance matrix S on the assumption of S is positive definite, since $\left|S-\lambda I\right|W=0$ , where λ is an eigen value of S.

Properties

1. Weights are proportional to covariance of variables and the weighted sum;

2. Variables which are highly associated with the weighted sum linearly to get higher positive weights.

• Proposition 3. To find $w_1,w_2,\dots \dots .,w_n$ such that $0\le w_i\le 1,{\sum }_{i=1}^n{w}_i=1$ and

$$w_i\propto \frac{\mathrm{cov}\left(Y,Z_i\right)}{\mathrm{var}\left(Z_i\right)}$$

Proof: Here, W satisfies $\left|R-\mu I\right|sW=0$ where R is the correlation matrix, μ is the maximum eigen value of R and s is the diagonal matrix of SDs of $Z_i$ s. It can be proved that $W=s^{-1}u/e^{\prime }{s}^{-1}u$ where u is eigenvector of R corresponding to its largest eigen value.

Observe that here weights are proportional to covariance and inversely proportional to SDs

Recommended method to find weights

It is proposed to determine the vector $\mathit{W}={\left(W_1,W_2,\dots \dots ..,W_K\right)}^T$ with ${\sum }_{i=1}^K{W}_i=1$ such that the variance of the battery score is a minimum where the battery score vector is Y with $\mathrm{var}\left(Y\right)=W^{T}DW$ where D is the variance-covariance matrix of the component tests subject to the condition $W^{T}e=1$ where e is the K-dimensional vector with i-th component equal to 1, ∀ i = 1, . . .,K. As per proposition 1, the weight vector in the instant case is

$$W=\frac{D^{-1}e}{e^T{D}^{-1}e}\mathrm{and}\lambda =\frac{2}{e^T{D}^{-1}e}$$

Benefits of the recommended method of finding weights

Weights found as above have the advantage that the battery score $(Y)$ has minimum variance. Also, covariance between the battery score and the score of an individual test is a constant, that is, $\mathrm{cov}\left(Y_i,X_i\right)=1/e^T{D}^{-1}e$ ∀ i. If the available test scores are standardized and independent such that the i-th score is $Z_i,$ then weights are equal, and correlation between Y and $Z_i$ is equal to correlation between Y and $Z_j=1/\sqrt{e^T{R}^{-1}e}\forall i\ne j,i,j=1,2,\dots ,K$ where R is the correlation matrix. In other words, the battery score is equi-correlated with the standardized score of each constituent test.

In order to find the battery scores, weights of test scores can also be found by principal component analysis, factor analysis, canonical reliability, and so on. However, it is suggested that reliability of the battery scores be found as weighted scores where the weights are determined as per Proposition 1.

It is recommended to use theoretical definition of test reliability for each constituent test as per equation (2) for clear theoretical advantages and Proposition 2.

Summary and discussion

Assuming non-availability of pre-specified, quantifiable external criterion, the article presents a well-defined and easy approach to the computation of reliability of test batteries, where battery score could be defined as sum of weighted or unweighted scores of the component tests. The method avoids scaling of scores of component tests. Such battery reliability can be influenced significantly by method of selection of weights to arrive at the battery scores and methods of estimating reliability of component tests, since sources of errors of individual tests may get manifold for the battery. Reliability, which is theoretically defined as the ratio of the true variance to the total variance, is an important property of measurement and works well to find battery reliability. It is possible to find true score variances of the battery even if true scores of individuals taking the component tests are not known.

Test reliability as per theoretical definition in terms of length of score vectors of two parallel tests and angle between such vectors in N-dimensional space also helps to find values of error score variance of the test fits well in estimation of battery reliability. Relationship between theoretically defined reliability $r_{tt}$ and split-half correlation $r_{gh}$ was established. Higher value of the theoretically defined reliability $r_{tt}$ will tend to raise value of validity of the test since validity coefficients for a particular test cannot be higher than the reliability coefficients for that test. The proposed method of computing error variance of the test helps to make unbiased and consistent estimates of ${\sigma }_T^2$ and ${\sigma }_E^2$ for the population. Error variance of the test or its population estimate $\left({\sigma }_E^2\right)$ may be mentioned along with the theoretically defined test reliability while reporting a test or a battery of tests.

There could be different methods of obtaining data-driven weights depending on the purposes. For weighted battery score, a Lagrange multiplier–based solution is recommended with the use of reliability as per the theoretical definition. Weights found as above have the advantage that the battery score $(Y)$ has minimum variance. Also, covariance between the battery score and the score of an individual test is a constant; condition of battery score being equi-correlated with the standardized score of each constituent test was derived.

While in this work, the classically defined reliability of a test with binary items (one right and the rest wrong) was discussed, a future study is proposed on empirical illustration and to find bias of the classically defined reliability with simulated data and compare with other methods of obtaining reliability.