Angular similarity in test parameters

Abstract

Through N-dimensional person space, the article gives measures of test parameters and item statistics, including difficulty/discriminating value of test, correlations between a pair of items, and item-total correlations with binary items using angular similarity between two vectors. Relationships between difficulty value and discriminating value of items and test were derived, including relationship between test reliability and test discriminating value. Reliability of a test as per theoretical definition in terms of length of score vectors of two parallel subtests and angle between such vectors was derived. The method was extended to find reliability of a battery of tests. Reliability and discriminating value of a Likert-type item and scale was found in terms of angular similarity without involving assumptions of continuous nature or linearity or normality for the observed variables, or the underlying variable being measured. The proposed methods also avoid test of unidimensionality or assumption of normality or bivariate normality associated with the polychoric correlations. Thus, the proposed methods are in fact nonparametric and considered as improvement over the existing ones. Reliability as a measure of association of two vectors and discrimination as a measure of distance between the vectors are likely to show a negative relationship.

Introduction

It is common to consider the variables as orthogonal axes and performance of individuals as points in the Euclidean space defined by the axes. However, it is possible to treat each person as an axis and represent the variables as points or vectors in the Euclidean space. This type of presentation is called “subject space” or “person space” or “N-dimensional person space” for N-persons. Such presentation has been considered by Reyment and Joreskog (1993), Huberty (1994), Chong et al. (2002), and so on. The orthogonality of the axes is an accurate indication if the persons are assumed to be independent of one another. Chong et al. (2002) observed that vectors representing latent factors rotate in subject space rather than variable space. In regression analysis, eigenvalues can be conceptualized as subject space equivalent to variance inflation factor (VIF). The concept of subject space can also be used as a means of visualization of spatial relationships. Glass and Collins (1970) dealt with three variables in N-dimensional subject space to find possible values of $r_{XY}$ when $r_{XZ}$ and $r_{YZ}$ are fixed.

If N-persons take a test, then the observed test scores can be viewed as a point or a vector in N-dimensional space. Once a person has taken a test, he or she is not considered to take the test again. Thus, a person axis is not overlapped. The angle θ between the two N-dimensional vectors X and Y is given by $\mathrm{Cos}\mathrm{\theta }=\frac{X.Y}{\Vert X\Vert \Vert Y\Vert }$ , where $\Vert X\Vert$ denotes length of the vector and is defined as $\Vert X\Vert =\sqrt{{\sum }_{i=1}^N{X}_i^2}$ . $\Vert Y\Vert$ is defined accordingly. For N-persons, the number of $\mathrm{Cos}{\mathrm{\theta }}_{ij}$ for $i\ne j$ is the same as the number of possible pairs, which is $N(N-2)/2.$ This gives the novel area of angular statistics, where Cosθ gives similarity between two vectors of the same dimension. In many diverse scientific fields, measurements are directions. Use of cosine similarity is common in areas like machine learning, classification, data mining, information retrieval, and pattern recognition.

This type of presentation helps to make useful inferences about the sample as well as computation of various parameters of the test, along with geometrical interpretation of such computations. It is possible to give theoretical formulation of mental testing through N-dimensional person space primarily in terms of the length of the observed score vector, angle separating the vector with the vector representing the maximum possible score in the test. For item statistics, the corresponding two vectors will be item score vector and the vector for maximum possible item score with each component equal to unity for a binary item.

For a test consisting of binary items (1 for right answer and 0 for others), the article attempts to provide geometric visualization along with computation of parameters of the test and item statistics, including test error variance and thereby the test reliability, as a ratio of true score variance and observed score variance, via a single administration of the test under classical test theory. Thus, the approach is an improvement as it enables calculation of theoretically defined reliability despite the fact that true scores of individuals taking the test are not known. Subsequent to the determination of the error variance of the test, test reliability method is extended to find reliability of a battery of tests. The approach can be extended for Likert-type test.

Methodology

The setup for test consists of binary items.

Suppose a test consisting of n-binary items has been administered among N-subjects. Scores obtained by the subjects can be presented as a point or vector X in the N-dimensional space with components $X_1,X_2,\dots ,X_N$ , where $X_i$ denotes test score of the ith subject. One may consider another vector I representing maximum possible score in the test with components $I_1,I_2,\dots ,I_N$ , where $I_i=I_j=n\forall i$ , $j=1,2,\dots ,N$ . Call the vector I as maximum possible vector or ideal vector. Let the angle between the vectors X and I be ${\mathrm{\theta }}_X$ . Similarly, the true score vector T with components $T_1,T_2,\dots ,T_N$ can be conceptualized, where the angle between the vectors T and I is ${\mathrm{\theta }}_Y$ .

Dot product

Dot product of two N-dimensional vectors X and Y is defined as

$$X.Y=\Vert X\Vert \Vert Y\Vert \mathrm{Cos}\theta$$

(1)

where $\Vert X\Vert$ denotes length of the vector X and is defined as $\Vert X\Vert =\sqrt{\sum _{i=1}^N{X}_i^2}$ . $\Vert Y\Vert$ is defined accordingly and $\mathrm{\theta }$ is the angle between the vectors X and Y.

Note that

1. $X.Y=0\iff \mathrm{\theta }={90}^{°}$ ⇔ X and Y are orthogonal (perpendicular)

2. X.Y = Cos θ if X = Y = 1

3. ||X + Y|| ⩽ ||X|| + ||Y|| (Triangle inequality)

4. X.Y ⩽ ||X||||Y|| (Schwarz’s inequality)

5. ${\Vert X+Y\Vert }^2+{\Vert X-Y\Vert }^2=2({\Vert X\Vert }^2+{\Vert Y\Vert }^2)$ (Parallelogram Law)

6. ${\Vert X+Y\Vert }^2={\Vert X\Vert }^2+{\Vert Y\Vert }^2$ where X and Y are orthogonal (Pythagorean Formula)

Ranks of individuals

Arranging the components of the vector X in decreasing order will give ranks of the individuals who took the test.

Mean

Let ${\mathrm{\theta }}_X$ be the angle between the score vector and the ideal vector. Then

$$\begin{array}{l}\mathrm{Cos}{\mathrm{\theta }}_X=\frac{\sum X_i{I}_i}{\Vert X\Vert \Vert I\Vert }\mathrm{or}\mathrm{Cos}{\mathrm{\theta }}_X=\frac{\sum X_i}{\Vert X\Vert \sqrt{N}}\mathrm{or}\\ \overline{X}=\frac{\Vert X\Vert \mathrm{Cos}{\mathrm{\theta }}_X}{\sqrt{N}}\end{array}$$

(2)

Thus, test mean is equal to the product of length of the score vector and cosine of the angle between the score vector and the ideal vector divided by square root of sample size.

From equation (2)

$${\overline{X}}^2=\frac{{\Vert X\Vert }^2{\mathrm{Cos}}^2{\mathrm{\theta }}_X}{N}$$

(3)

Similarly

$$\overline{T}=\frac{\Vert T\Vert \mathrm{Cos}{\mathrm{\theta }}_T}{\sqrt{N}}$$

(4)

and

$${\overline{T}}^2=\frac{{\Vert T\Vert }^2{\mathrm{Cos}}^2{\mathrm{\theta }}_T}{N}$$

(5)

Since, $\overline{X}=\overline{T}$ , equating (2) and (4), we get

$$\begin{array}{l}\Vert X\Vert \mathrm{Cos}{\mathrm{\theta }}_X=\Vert T\Vert \mathrm{Cos}{\mathrm{\theta }}_T\mathrm{or}\\ {\Vert X\Vert }^2{\mathrm{Cos}}^2{\mathrm{\theta }}_X={\Vert T\Vert }^2{\mathrm{Cos}}^2{\mathrm{\theta }}_T={\Vert T\Vert }^2-{\Vert T\Vert }^2{\mathrm{Sin}}^2{\mathrm{\theta }}_T\end{array}$$

(6)

Equation (6) helps to find ${\Vert T\Vert }^2{\mathrm{Cos}}^2{\mathrm{\theta }}_T$ from the data, even if ${\Vert T\Vert }^2$ or ${\mathrm{Cos}}^2{\mathrm{\theta }}_T$ are not known.

Equating (3) and (5), we get

$$\frac{{\Vert T\Vert }^2}{{\Vert X\Vert }^2}={\mathrm{Cos}}^2{\mathrm{\theta }}_X.{\mathrm{Sec}}^2{\mathrm{\theta }}_T$$

(7)

which gives relationship between $\Vert X\Vert$ and $\Vert T\Vert$ .

Let ${\mathrm{\theta }}_E$ represents the angle between the error score vector and the ideal vector. As per classical test theory

$$\begin{array}{l}\overline{E}=0\mathrm{i}\mathrm{.e}\mathrm{.}\overline{E}=\frac{\Vert E\Vert \mathrm{Cos}{\mathrm{\theta }}_E}{\sqrt{N}}=0\Rightarrow \\ \mathrm{Cos}{\mathrm{\theta }}_E=0\mathrm{since}\Vert E\Vert \ne 0\end{array}$$

(8)

Variance

Test variance or observed score variance can also be obtained from N-dimensional person space by $S_X^2=\frac{{\Vert x\Vert }^2}{N}$ where $x_i=X_i-\overline{X}$ represents deviation scores.

Alternatively, $S_X^2=\frac{{\Vert X\Vert }^2}{N}-{\overline{X}}^2\Rightarrow S_X^2=\frac{{\Vert X\Vert }^2-{\Vert X\Vert }^2{\mathrm{Cos}}^2{\mathrm{\theta }}_X}{N}$ by equation (3)

$$\Rightarrow S_X^2=\frac{{\Vert X\Vert }^2{\mathrm{Sin}}^2{\mathrm{\theta }}_X}{N}$$

(9)

Thus, standard deviation (SD) of test score is the product of the length of the score vector and sine of the angle between the score vector and the ideal vector divided by the sample size.

Thus, the relationship between x and X is given by

$${\Vert x\Vert }^2={\Vert X\Vert }^2{\mathrm{Sin}}^2{\mathrm{\theta }}_X$$

(10)

Similarly, true score variance is obtained

$$\begin{array}{l}{S}_T^2=\frac{{\Vert t\Vert }^2}{N}=\frac{{\Vert T\Vert }^2{\mathrm{Sin}}^2{\mathrm{\theta }}_T}{N}=\\ \frac{{\Vert T\Vert }^2}{N}-\frac{{\Vert X\Vert }^2{\mathrm{Cos}}^2{\mathrm{\theta }}_X}{N}\mathrm{using}\mathrm{equation}(6)\end{array}$$

(11)

Equation (11) helps to find value of true score variance of a test from the data if the length of true score vector is known, and

$${\Vert t\Vert }^2={\Vert T\Vert }^2{\mathrm{Sin}}^2{\mathrm{\theta }}_T$$

(12)

Now $S_X^2=S_T^2+S_E^2$ implies ${\Vert X\Vert }^2{\mathrm{Sin}}^2{\mathrm{\theta }}_X={\Vert T\Vert }^2+{\Vert X\Vert }^2{\mathrm{Cos}}^2{\mathrm{\theta }}_X+{\Vert E\Vert }^2$

$$\Rightarrow {\Vert X\Vert }^2={\Vert T\Vert }^2+{\Vert E\Vert }^2$$

(13)

Correlations

One assumption of classical test theory is $r_{TE}=0\Rightarrow \mathrm{Cov(}T,E\mathrm{)}=0$

$$\Rightarrow \sum T_i{E}_i=0\Rightarrow \sum T_i\left(X_i-T_i\right)=0$$

$$\Rightarrow \sum T_i{X}_i=\sum T_i^2\Rightarrow \sum T_i{X}_i={\Vert T\Vert }^2\Rightarrow \mathrm{Cov}\left(X,T\right)=S_T^2$$

One may also write $\sum T_i{E}_i=0\Rightarrow \sum (X_i-E_i)E_i=0$

$$\Rightarrow \sum X_i{E}_i={\Vert E\Vert }^2\Rightarrow \mathrm{Cov}(X,E)=S_E^2$$

(14)

Thus

$$S_X^2=\mathrm{Cov}\left(X,T\right)+\mathrm{Cov}\left(X,E\right)$$

(15)

So variance–covariance matrix of observed score (X), true score (T), and error score (E) of a test is $\left[\begin{array}{ccc}{S}_X^2& S_{XT}& S_{XE}\\ -& S_T^2& 0\\ -& -& S_E^2\end{array}\right]=\left[\begin{array}{ccc}{S}_X^2& S_T^2& S_E^2\\ -& S_T^2& 0\\ -& -& S_E^2\end{array}\right]$ and trace of the matrix is $2S_X^2$ .

It is well known that correlation between X and T is given by $r_{XT}=\mathrm{Cos}{\mathrm{\theta }}_{xt}=\frac{\sum x_i{t}_i}{\Vert x\Vert \Vert t\Vert }$ , where $x_i$ and $t_i$ denote deviation scores and ${\mathrm{\theta }}_{xt}$ is the angle between two deviation score vectors.

Now, $\sum x_i{t}_i=\sum (X_i-\overline{X})(T_i-\overline{T})=\sum (X_i{T}_i-\overline{T}{X}_i-\overline{X}{T}_i+\overline{X}\overline{T})=\sum X_i{T}_i-N{\overline{T}}^2={\Vert T\Vert }^2-N{\overline{T}}^2$ (since, $r_{TE}=0\Rightarrow \sum X_i{T}_i={\Vert T\Vert }^2$ ) =  $N{S}_T^2$ .

And ${\Vert x\Vert }^2={\Vert X\Vert }^2{\mathrm{Sin}}^2{\mathrm{\theta }}_X$ by equation (10) and ${\Vert t\Vert }^2={\Vert T\Vert }^2{\mathrm{Sin}}^2{\mathrm{\theta }}_T$ by equation (12).

Thus

$$\begin{array}{l}{r}_{XT}=\mathrm{Cos}{\mathrm{\theta }}_{xt}=\frac{N{S}_T^2}{\Vert X\Vert \mathrm{Sin}{\mathrm{\theta }}_X\Vert T\Vert \mathrm{Sin}{\mathrm{\theta }}_T}\\ =\frac{S_T^2}{S_X{S}_T}\left(\mathrm{from}\mathrm{equation}\left(15\right)\right)\end{array}$$

(16)

But $\mathrm{Cos}{\mathrm{\theta }}_{XT}=\frac{\sum X_i{T}_i}{\Vert X\Vert \Vert T\Vert }=\frac{{\Vert T\Vert }^2}{\Vert X\Vert \Vert T\Vert }$ (since r_TE = 0 ⇒ $\sum X_i{T}_i={\Vert T\Vert }^2$ ).

Therefore

$$\frac{Cos{\mathrm{\theta }}_{xt}}{\mathrm{Cos}{\mathrm{\theta }}_{XT}}=\left(\frac{S_T}{S_X}\right)\left(\frac{\Vert T\Vert }{\Vert X\Vert }\right)$$

(17)

Equation (17) gives the relationship between $\mathrm{Cos}{\mathrm{\theta }}_{xt}$ (which is equal to $r_{XT}$ ) and $\mathrm{Cos}{\mathrm{\theta }}_{XT}$ .

Note that

• Value of $\mathrm{Cos}{\mathrm{\theta }}_X^2$ can be obtained from the data using equation (3). Value of $\mathrm{Sin}{\mathrm{\theta }}_X^2$ can also be obtained accordingly or by using equation (10).

• Value of ${\Vert T\Vert }^2\mathrm{Sin}{\mathrm{\theta }}_T^2$ which is equal to ${\Vert X\Vert }^2\mathrm{Cos}{\mathrm{\theta }}_X^2$ by equation (6) can be obtained from data even if the value of ${\Vert T\Vert }^2$ or $\mathrm{Sin}{\mathrm{\theta }}_T^2$ is unknown.

• Equations (15) to (17) each involves unknown quantity and thus does not help to compute directly $r_{tt}$ or $S_T^2$ or ${\Vert E\Vert }^2$ .

Reliability and parallel tests

A test consisting of n-items when dichotomized in parallel halves say gth subtest and hth subtest results in two points ${\mathbf{X}}_g$ and ${\mathbf{X}}_h$ in the N-dimensional person space. As per classical definition, two tests “g” and “h” are parallel if $T_i^{(g)}=T_i^{(h)}$ and $S_e^{(g)}=S_e^{(h)}$ , where the subscript “g” refers to subtest g and subscript “h” refers to subtest h, and $S_e^{(p)}$ is the SD of error scores in the pth subtest, p = g,h. It can be proved that if g and h are parallel, $\overline{X_g}=\overline{X_h}$ and $S_{Xg}^2=S_{Xh}^2.$

Also, $X_g=T_g+E_g$ and $X_h=T_h+E_h$ . Now $T_i^{(g)}=T_i^{(h)}$ implies $X_g-X_h=E_g-E_h$ , so that

$$\begin{array}{l}{\Vert X_g\Vert }^2+{\Vert X_h\Vert }^2-2\Vert X_g\Vert \Vert X_h\Vert \mathrm{Cos}{\mathrm{\theta }}_{gh}=\\ {\Vert E_g\Vert }^2+{\Vert E_h\Vert }^2-2\Vert E_g\Vert \Vert E_h\Vert \mathrm{Cos}{\mathrm{\theta }}_{gh}^{(E)}\end{array}$$

(18)

where ${\mathrm{\theta }}_{gh}$ is the angle between $X_g$ and $X_h$ while ${\mathrm{\theta }}_{gh}^{(E)}$ is the angle between $E_g$ and $E_h.$ But correlation between error scores of two parallel tests is zero. Thus

$$\begin{array}{l}{\Vert X_g\Vert }^2+{\Vert X_h\Vert }^2-2\Vert X_g\Vert \Vert X_h\Vert \mathrm{Cos}{\mathrm{\theta }}_{gh}=\\ {\Vert E_g\Vert }^2+{\Vert E_h\Vert }^2=2N{S}_E^2\end{array}$$

(19)

since $S_e^{(g)}=S_e^{(h)}$ by definition. The above equation suggests

$$S_E^2=\frac{1}{N}\left[{\Vert X_g\Vert }^2+{\Vert X_h\Vert }^2-2\Vert X_g\Vert \Vert X_h\Vert \mathrm{Cos}{\mathrm{\theta }}_{gh}\right]$$

(20)

Hence

$$r_{tt}=1-\frac{{\Vert X_g\Vert }^2+{\Vert X_h\Vert }^2-2\Vert X_g\Vert \Vert X_h\Vert \mathrm{Cos}{\mathrm{\theta }}_{gh}}{N{S}_X^2}$$

(21)

Equation (20) helps to find value of error variance of the test and hence true score variance as $(S_X^2-S_E^2)$ , and use them directly in equation (21) to find reliability of the test as per the classical definition from a single administration in terms of length of score vectors of two parallel tests and angle between such vectors. Thus, it is possible to find true score variance from the data and to calculate a reliability that conforms to the theoretical definition even if true scores of individuals taking the test are not known.

Note that while equation (21) finds reliability $(r_{tt})$ of a test which is isomorphic to definition, the process of splitting the test in two parallel halves gives the popular method of finding split-half reliability of a test as correlation between two parallel forms of the test $(r_{gh})$ . However, value of $r_{gh}$ may be different from value of $r_{tt}$ .

The approach of finding reliability as per definition can be extended to find reliability of a battery of tests aiming to measure a finite number of variables. After administration of the battery to N-individuals, values of $S_X^2$ , $S_E^2$ , $S_T^2$ , and $r_{tt}$ for each constituent test can be computed. Method of obtaining reliability of the battery depends on these parameters and also on definition of battery score.

If battery score is taken as sum of score of K-tests (summative score), it can be proved that

$$r_{tt(\mathrm{battery})}=\frac{\sum _{i=1}^K{r}_{tt(i)}{S}_{Xi}^2+\sum _{i=1,i\ne j}^K\sum _{j=1}^{K}2\mathrm{Cov}\left(X_i,X_j\right)}{\sum _{i=1}^K{S}_{Xi}^2+\sum _{i=1,i\ne j}^K\sum _{j=1}^{K}2\mathrm{Cov}\left(X_i,X_j\right)}$$

(22)

If $W_1,W_2,\dots ,W_K$ are the weights to K-constituent tests of a battery where $W_i\ge 0\forall i=1,2,3,\dots ,K$ and $\sum W_i=1$ and the battery score be $Y_i={\sum }_{i=1}^K{W}_i{X}_i$ .

Clearly, $\mathrm{var}(Y)={\sum }_{i=1}^K{W}_i^2\mathrm{var(}{X}_i\mathrm{)}$ . Reliability of the battery can be computed as

$$r_{tt(\mathrm{battery})}=\frac{\sum _{i=1}^K{r}_{tt(i)}{W}_i^2{S}_{Xi}^2+\sum _{i=1,i\ne j}^K\sum _{j=1}^{K}2W_i{W}_j\mathrm{Cov}\left(X_i,X_j\right)}{\sum _{i=1}^K{W}_i^2{S}_{Xi}^2+\sum _{i=1,i\ne j}^K\sum _{j=1}^{K}2W_i{W}_j\mathrm{Cov}\left(X_i,X_j\right)}$$

(23)

Validity

Usual procedure of obtaining validity of a test is to find correlation of the test score (X) and criterion score (Y). It is known that $r_{XY}=\mathrm{Cos}{\mathrm{\theta }}_{xy}$ , where ${\mathrm{\theta }}_{xy}$ is the angle between the vectors of deviation scores. Linear regression of Y on X can also be represented by the vector resulting from the projection is $\widehat{y}=p\mathrm{(}y\mathrm{on}x\mathrm{)}$ , the vector of predicted values on Y.

Item statistics for test consisting of binary items

Difficulty value of a test

If X coincides with I , then the test is extremely easy since each subject has got maximum possible score. Thus, difficulty value of a test should consider twofold criteria, namely, ${\mathrm{\theta }}_X$ and ratio of $\Vert I\Vert$ and $\Vert X\Vert$ .

Accordingly, difficulty value of a test D may be defined as

$$D=\frac{\Vert X\Vert \mathrm{Cos}{\mathrm{\theta }}_X}{\Vert I\Vert }=\frac{\overline{X}}{n}$$

(24)

Note that equation (24) defines difficulty value of a test as ratio of product of length of the observed score vector andcosine of the angle between the observed score vector and the ideal vector as the numerator and length of the ideal vector as the denominator, keeping harmony with the usual notion of difficulty value of a test which actually measures degree of easiness of a test. It can be proved that 0 ⩽ D ⩽ 1.

Item difficulty value

In the N-dimensional person space, scores of an item may be characterized by an N-dimensional vector whose components are 0s and 1s. For an item, the vector I represents maximum possible score with components $I_1,I_2,\dots ,I_N$ , where $I_i=I_j=1$ $\forall i,j=1,2,\dots N$ . Thus, $\Vert I\Vert =\sqrt{N}$ . If K-persons (K ⩽ N) answer the ith item correctly, then $\Vert X\Vert =\sqrt{K}$ . Thus, difficulty value of the ith item is defined by

$$D_i=\frac{{\Vert X\Vert }^2}{{\Vert I\Vert }^2}=\frac{K}{N}$$

(25)

Clearly, 0 ⩽ Di ⩽ 1. It may be observed that difficulty value of an item as per equation (25) in terms of ratio of square of length of the observed score vector and square of length of the ideal vector coincides with normal idea of proportion of persons passing an item.

Discriminating value of a test

If the vector X makes a zero degree angle with the vector I, then the test fails to discriminate the subjects. So ${\mathrm{\theta }}_X$ or a suitably defined function of ${\mathrm{\theta }}_X$ will measure the discriminating value of a test. Since it is desirable for the discriminating value to lie in [0, 1], $\mathrm{Tan}{\mathrm{\theta }}_X$ will measure the discriminating value of a test. Thus

$$D^1=\mathrm{Tan}{\mathrm{\theta }}_X=\frac{S_X}{\overline{X}}\left(\mathrm{from}\mathrm{equations}(1)\mathrm{and}(9)\right)$$

(26)

where $D^1$ denotes the discriminating value of a test.

Thus, discriminating value of a test equals to tan of the angle between observed score vector and ideal vector which is the ratio of SD and mean of the test score. In other words, discriminating value of a test is a linear function of its coefficient of variation (CV), a well-known measure of relative precision which is independent of change of scale but not of origin.

Discriminating value of an item

The method also helps to find discriminating value of an item as follows

$$D_i^1=\mathrm{Tan}{\mathrm{\theta }}_{Xi}$$

(27)

where $D_i^1$ denotes the discriminating value of the ith item and ${\mathrm{\theta }}_{Xi}$ is the angle between the observed score vector for the ith item and ideal vector of an item with each component equal to unity.

Note that this method of finding discriminating value of an item does not consider total test score. The method considers entire data relating to the item and avoids consideration of top 27% and bottom 27% of data. However, item discriminating value tends to be lower for tests measuring different content areas and cognitive skills.

Evaluation of Tan θX

For the ith item, the components of a vector X are K-numbers of 1s and rest 0s, if K-persons could answer the item correctly, where K ⩽ N. Each of the N-component of the ideal vector is equal to unity. Let θ _Xi denotes the angle between the vectors X and I. Here

$$\mathrm{Cos}{\mathrm{\theta }}_{Xi}=\sqrt{\frac{K}{N}}\Rightarrow \mathrm{Tan}{\mathrm{\theta }}_{Xi}=\sqrt{\frac{N-K}{K}}$$

(28)

Note that

• 1. The discriminating value of an item is equal to the ratio of SD of the item score and mean of the item score, and can exceed unity. It is a function of CV.

• 2. If K = 0, that is, if all the subjects fail in an item, then discriminating value is not defined for the item.

• 3. If K = N, that is, if all the subjects pass an item, then discriminatory value is zero for that item.

• 4. It can be shown that square of discriminating value of an item is equal to reciprocal of difficulty value minus one. That is

$${\left[D_i^1\right]}^2=\frac{1}{D_i}-1$$

(29)

Equation (29) gives the non-linear relationship between difficulty value and discriminating value of items.

• 5. Relationship between difficulty value and discriminating value of a test can be found as follows

$$D=\frac{\overline{X}}{n}\mathrm{and}{D}^1=\mathrm{Tan}{\mathrm{\theta }}_X=\frac{S_X}{\overline{X}}\Rightarrow D{D}^1=\frac{S_X}{n}$$

(30)

Thus, product of difficulty value and discriminating value of a test is equal to SD of the test divided by the number of items.

Relationships between test reliability and discriminating value of a test

Since $\overline{X}=\overline{T}$ and test reliability $r_{tt}=S_T^2/S_X^2$

$$r_{tt}{\left(D^1\right)}^2=\frac{S_T^2}{S_X^2}={\left(\frac{S_T}{\overline{X}}\right)}^2={\left(\frac{S_T}{\overline{T}}\right)}^2$$

Thus, product of test reliability and square of test discriminating value is equal to square of CV of true scores. However, verification of the relationship may require finding test reliability as per the definition, which is described in equation (21).

Item correlations

Phi coefficient $({\mathrm{\Phi }}_{st})$ is the proper statistic to estimate the degree of relationship between the scores of sth and tth items (both dichotomous variables) and is defined as ${\mathrm{\Phi }}_{st}=k_{11}{k}_{00}-k_{10}{k}_{01}/\sqrt{k_{1.}{k}_{0.}{k}_{.0}{k}_{.1}}$ , where the non-negative symbols are taken from the 2 × 2 table for two different items (s and t):

	s = 1	s = 0	Total
t = 1	k 11	k 10	k 1.
t = 0	k 01	k 00	k 0.
Total	k .1	k .0	N

Correlation between two items can also be viewed through N-dimensional person space. Let ${\mathrm{\theta }}_{st}$ be the angle between the vectors ${\mathbf{X}}_s$ and ${\mathbf{X}}_t$ . Then, ${\Vert X_t\Vert }^2=k_1$ , that is, the number of persons attempting correctly the tth item. Similarly, ${\Vert X_s\Vert }^2=k_{.1}$ , that is, the number of persons attempting correctly the sth item. The number of persons who could answer correctly both sth and tth items is $k_{11}=\sum \sum X_{is}{X}_{jt}=X_s^{{\rm T}}{X}_t.$

The number of persons who failed to answer correctly both the items is $k_{00}=N-\sum \sum X_{is}{X}_{jt}$ . Clearly, $k_{10}=k_1-k_{11}={\Vert X_t\Vert }^2-\sum \sum X_{is}{X}_{jt}$ . Similarly, k₀₁ = k_.1 – k₁₁ $={\Vert X_s\Vert }^2-\sum \sum X_{is}{X}_{jt}$ and k₀₀ = k₀ – k_{01 = (N – k1)} $-k_{01}=(N-{\Vert X_t\Vert }^2)-{\Vert X_s\Vert }^2+\sum \sum X_{is}{X}_{jt}.$

Thus, each term of the formula for ${\mathrm{\Phi }}_{st}$ can be written in terms of ${\Vert X_t\Vert }^2$ , ${\Vert X_s\Vert }^2$ , $X_s^{{\rm T}}{X}_t$ , and N. In other words, ${\mathrm{\Phi }}_{st}$ can be viewed and computed through N-dimensional person space as

$${\mathrm{\Phi }}_{st}=\frac{\begin{array}{l}{X}_s^T{X}_t\left(N-X_s^T{X}_t\right)-\left(X_t^2-X_s^T{X}_t\right)\left({\Vert X_s\Vert }^2-X_s^T{X}_t\right)\end{array}}{\sqrt{{\Vert X_t\Vert }^2{\Vert X_s\Vert }^2\left(N-{\Vert X_t\Vert }^2\right)\left(N-{\Vert X_s\Vert }^2\right)}}$$

(31)

Item-total correlations

Point-biserial correlation coefficient $(r_{pb})$ is the proper statistic to estimate the degree of relationship between score of an item (dichotomous variable) and test score (interval/ratio scale) and is defined as $r_{pb}=\frac{(M_p-M_q)\sqrt{pq}}{S_X}$ , where $r_{pb}$ denotes point-biserial correlation coefficient, $M_p$ denotes test mean for persons answering the item correctly (i.e. those coded as 1s), $M_q$ denotes test mean for persons answering the item incorrectly (i.e. those coded as 0s), $S_X$ denotes SD of the test scores, p denotes proportion of persons answering correctly (i.e. those coded as 1s), and q denotes proportion of persons answering incorrectly (i.e. those coded as 0s).

Let ${\mathbf{X}}_s$ be the vector showing scores of the sth item in the N-dimensional person space. If K-persons could answer the sth item correctly, then ${\Vert X_s\Vert }^2=K$ .

Now $X^{⊺}{X}_s=\sum \sum X_i{X}_{is}$  = sum of test scores for persons answering the sth item incorrectly = sum of test scores–  $\sum \sum X_i{X}_{is}=\sqrt{N}\Vert X\Vert \mathrm{Cos}{\mathrm{\theta }}_X-\sum \sum X_i{X}_{is}$ (using equation (1)).

Here

$$M_{ps}=\frac{\begin{array}{l}\mathrm{whole-test}\mathrm{mean}\mathrm{for}\mathrm{persons}\\ \mathrm{answering}\mathrm{the}s\mathrm{th}\mathrm{item}\mathrm{correctly}\end{array}}{K}=\frac{{\sum }^{\mathrm{}}{\sum }^{\mathrm{}}{X}_i{X}_{is}}{K}$$

$$\begin{array}{l}{M}_{qs}=\frac{\begin{array}{l}\mathrm{whole-test}\mathrm{mean}\mathrm{for}\mathrm{persons}\\ \mathrm{answering}\mathrm{the}s\mathrm{th}\mathrm{item}\mathrm{incorrectly}\end{array}}{N-K}\\ =\frac{\sqrt{N}\Vert X\Vert \mathrm{Cos}{\mathrm{\theta }}_X-\sum \sum X_i{X}_{is}}{N-K}\end{array}$$

$$\begin{array}{l}\mathrm{SD}\mathrm{of}\mathrm{the}\mathrm{whole}\mathrm{test}{S}_t=\frac{\Vert X\Vert \mathrm{Sin}{\mathrm{\theta }}_X}{\sqrt{N}}.\\ p_s=\mathrm{Proportion}\mathrm{of}\mathrm{persons}\mathrm{answering}\mathrm{the}\\ s\mathrm{th}\mathrm{item}\mathrm{correctly}=\frac{K}{N}\end{array}$$

$$\begin{array}{l}{q}_s=\mathrm{Proportion}\mathrm{of}\mathrm{persons}\mathrm{answering}\\ \mathrm{the}s\mathrm{th}\mathrm{item}\mathrm{incorrectly}=\frac{N-K}{N}\end{array}$$

Thus, point-biserial correlation between the sth item $(r_{pb\left(s\right)})$ and test score may be obtained by

$$\begin{array}{l}{r}_{pb(s)}=\frac{\left(\begin{array}{l}\frac{\sum \sum X_i{X}_{is}}{K}-\\ \frac{\sqrt{N}\Vert X\Vert \mathrm{Cos}{\mathrm{\theta }}_X-\sum \sum X_i{X}_{is}}{N-K}\end{array}\right)\frac{1}{N}\sqrt{K\left(N-K\right)}}{\frac{\Vert X\Vert \mathrm{Sin}{\mathrm{\theta }}_X}{\sqrt{N}}}\\ =\frac{\left(\begin{array}{l}\frac{\sum \sum X_i{X}_{is}}{K}-\\ \frac{\sqrt{N}\Vert X\Vert \mathrm{Cos}{\mathrm{\theta }}_X-\sum \sum X_i{X}_{is}}{N-K}\end{array}\right)\sqrt{\frac{K\left(N-K\right)}{N}}}{\Vert X\Vert \mathrm{Sin}{\mathrm{\theta }}_X}\end{array}$$

(32)

This may be simplified further as

$$r_{pb(s)}=\frac{\left(\frac{X^{⊺}{X}_s}{K}-\frac{\sum X_i-X^{⊺}{X}_s}{N-K}\right)\sqrt{\frac{K\left(N-K\right)}{N}}}{\sqrt{N}{S}_X}$$

(33)

Item reliability

In addition to item difficulty and item discrimination, the concept of item reliability is also used in item statistics. Item reliability is defined as a product of SD of item scores and a correlational discrimination index, usually item-total correlation. Thus, reliability of the ith item could be taken as

$$S_i{r}_{pb(i)}=\frac{\sqrt{N}\left(M_p-M_q\right)pq}{S_X}=\frac{\left(N-K\right)\left(M_p-M_q\right)}{S_X}$$

(34)

where $S_i=\sqrt{Npq}$ denotes SD of the score of ith item and $r_{pb(i)}$ denotes point-biserial correlation coefficient of the ith item, and the number of persons answering correctly the ith item is K. However, $\mathrm{Cos}{\mathrm{\theta }}_{iX}$ , that is, the angle between score vector of the ith item and total score vector of the test X, indicates a better measure of reliability of the ith item.

Higher values of item reliability are desirable. If two items have equal item discriminating value, then the item with higher variance is preferred to be retained.

Likert-type test

Background

Suppose n-respondents have answered each of the m-items of a Likert-type questionnaire, where each item has k-numbers of response categories, marked as $1,2,\dots ,k$ . Let $X_{ij}$ be a general element of the basic data matrix where n-individuals are in rows and m-items are in columns. $X_{ij}$ represents score of the ith individual for the jth item, where 1 ⩽ X_ij ⩽ k.

Note ${\sum }_{i=1}^n{X}_{ij}$  = Sum of scores of all individuals for the jth item (item score for the jth item); ${\sum }_{j=1}^m{X}_{ij}$  = Sum of scores of all the items for ith individual, that is, total score of the ith individual (individual score); and ${\sum }_{i=1}^n{\sum }_{j=1}^m{X}_{ij}$  = Sum of scores of all the individuals on all the items, that is, total test score. In addition, one can have another matrix $((f_{ij}))$ of order m × k showing frequency i-th item to j-th response category. A row total = the sample size (n). Similarly, a column total will indicate total number of times that response category was chosen by all the respondents. Denote the column total of jth response category by $f_{0j}$ for j = 1, 2, …, k. $j=1,2,\dots ,k.$ Clearly, ${\sum }_{j=1}^k{f}_{0j}=g\mathrm{rand}\mathrm{total}=(\mathrm{sample}\mathrm{size}\times \mathrm{number}\mathrm{of}\mathrm{items})=mn$ .

After administration of the questionnaire to n-respondents, we can calculate empirical probabilities of each response categories of the ith item as ${\mathit{P}}_{\mathit{i}}={(p_{i1},p_{i2},\dots ,p_{ik})}^T.$ Note that ${\sum }_{j=1}^k{p}_{ij}={\sum }_{j=1}^k{p}_{0j}=1$ . The ith item will fail to discriminate or it will have zero discriminating value if all respondents choose the jth response category, that is, $p_{ij}=1$ or if $p_{ij}=\frac{1}{k}\forall j=\mathrm{1},\mathrm{2},\mathrm{3},\dots ,k$ . Discriminating value of the ith item may be found as a measure of dissimilarities between these two vectors ${\mathbf{P}}_i$ and ${\mathbf{P}}_0$ , where ${\mathbf{P}}_0$ represents the vector of equal probabilities. Similarly, for the scale, vector showing empirical probabilities will be $T={\left(\frac{f_{01}}{mn},\frac{f_{02}}{mn},\dots ,\frac{f_{0k}}{mn}\right)}^T$ . Clearly, ${\sum }_{j=1}^k\frac{f_{0j}}{mn}=1$ . Discriminating value of the scale may be found as dissimilarities between the vectors T and P₀.

Reliability of an item can be measured as cosine similarity between ${\mathbf{P}}_i$ and T. While reliability is a measure of association of two vectors, discrimination is a measure of dissimilarities between the vectors. Thus, a negative relationship is expected between reliability and discriminating value. Computation of the vectors ${{\mathbf{P}}^{\prime }}_{i}s$ , ${\mathbf{P}}_0$ , and T forms the stage to obtain discriminating values and reliability of Likert-type scale and items from the data.

Measure for reliability of Likert-type test

Chakrabartty (2018) proposed two nonparametric methods of finding reliability of Likert-type scale and items considering only the frequency or proportion for each cell of the item-response category matrix without involving any assumptions of continuous nature or linearity or normality for the observed variables, or the underlying variable being measured.

Cosine similarity between ith and jth item is

$$\mathrm{Cos}{\mathrm{\theta }}_{ij}=\frac{P_i^T{P}_j}{\Vert P_i\Vert \Vert P_j\Vert }$$

(35)

where ${\mathrm{\theta }}_{ij}$ is the angle between the vectors ${\mathbf{P}}_i$ and ${\mathbf{P}}_j$ .

Item reliability in terms of correlation between the ith item score and total score can be obtained by

$$\mathrm{Cos}{\mathrm{\theta }}_{iT}=\frac{P_i^{T}T}{\Vert P_i\Vert \Vert T\Vert }$$

(36)

where ${\mathrm{\theta }}_{iT}$ is the angle between the vectors ${\mathbf{P}}_i$ and T.

Test reliability with m-items was found by replacing the polychoric correlation between items i and j by $\mathrm{Cos}{\mathrm{\theta }}_{ij}$

$$r_{tt}=\frac{m}{m-1}\left(1-\frac{m}{m+\sum \sum _{i\ne j}\mathrm{Cos}{\mathrm{\theta }}_{ij}}\right)$$

(37)

For Bhattacharyya’s similarity, each vector ${\mathbf{P}}_i$ and ${\mathbf{P}}_j$ was converted to unit vector ${\pi }_i$ and ${\pi }_i$ , where ${\pi }_i=\sqrt{{\mathbf{P}}_i/\Vert {\mathbf{P}}_i\Vert }$ and ${\pi }_j=\sqrt{{\mathbf{P}}_j/\Vert {\mathbf{P}}_j\Vert }$ so that ${\Vert {\pi }_i\Vert }^2={\Vert {\pi }_j\Vert }^2=1$ . Here, item reliability in terms of item-test correlation was computed from

$$\rho \left({\pi }_i,\sqrt{\mathbf{T}}\right)=\sum _{j=1}^k\frac{f_{ij}}{mn}{p}_{ij}$$

(38)

and test reliability was taken as

$$\mathrm{Cos}\left(\overline{\varnothing }\right)=\mathrm{Cos}\left({\mathrm{Cot}}^{-1}\frac{\sum \mathrm{Cos}{\varnothing }_i}{\sum \mathrm{Sin}{\varnothing }_i}\right)$$

(39)

where ∅ denotes mean or most preferred direction among the angles ${\varnothing }_1,{\varnothing }_2,{\varnothing }_3,\dots ,{\varnothing }_k$ between unit vectors.

Each method enables to find reliability irrespective of distribution of the observed or underlying variables and avoiding test of unidimensionality or assumption of normality for Cronbach’s alpha or bivariate normality for polychoric correlations. Bhattacharyya’s similarity helps to find range of reliability of the entire Likert-type scale.

Discriminating value of Likert-type item and Likert-type scale

The concept of CV as a measure of discriminating value of a binary item can be extended to find discriminating value of a Likert-type item with multi-response alternatives and define $D_i=S{D}_i/{\mathrm{Mean}}_i$ for the ith item. Discriminating value of the test is defined accordingly, that is, $D_{Test}=\frac{S{D}_{Test}}{{\mathrm{Mean}}_{Test}}$ . Note that mean of the ith item is $\frac{1}{n}\sum X_{ij}=\frac{f_{0i}}{n}$ and variance of the ith item is $n{\Vert P_i\Vert }^2-{\left(\frac{f_{0i}}{n}\right)}^2$ .

If ${\mathrm{\theta }}_X$ be the angle between the test score vector and the maximum possible vector I (each component is equal to k for k-point scale), it can be proved that

$${\mathrm{CV}}_{Test}=\frac{S{D}_{Test}}{{\mathrm{Mean}}_{Test}}=\frac{\Vert X\Vert \mathrm{Sin}{\mathrm{\theta }}_X}{\Vert X\Vert \mathrm{Cos}{\mathrm{\theta }}_X}=\mathrm{Tan}{\mathrm{\theta }}_X$$

(40)

CV is a well-known measure of relative precision which is independent of change of scale but not of origin. Note that $SD/\mathrm{Mean}$ is:

• non-negative if response categories are marked with positive numbers;

• defined even if $p_{ij}=0$ for a particular response category of ith item;

• CV = 0 ⇔ either $p_{ij}=\frac{1}{k}\forall j=\mathrm{1},\mathrm{2},\dots ,k$ or $p_{ij}=1$ for a particular jth category;

• D_i = D_j ⇔ P_i = P_J;

• possible to express $D_{Test}$ as a function of ${D\prime }_{i}s$ since ${\mathrm{Mean}}_{Test}$ and $S{D}_{Test}$ are combined mean and combined SD, respectively;

• shown that estimate of population CV is $\sigma /\mu$ , where µ and $\sigma$ are population estimates of mean and SD, respectively;

• shown that statistical inferences for CV for normally distributed data are often based on McKay’s ${\chi }^2$ approximation for the CV (Forkman, 2013).

Findings and conclusion

Through N-dimensional person space, measures of test parameters and item statistics including difficulty value, discriminating value of test, Phi coefficient to depict correlations between a pair of items, and point-biserial correlation to depict item-total correlations with binary items were presented using angular similarity between two vectors and considering the entire data. Non-linear relationships between difficulty value and discriminating value of items and test were derived, including relationship between test reliability and test discriminating value.

Dichotomization of the test in parallel halves helps to find value of error variance, true score variance, and reliability of a test as per theoretical definition from a single administration in terms of length of score vectors of two parallel tests and angle between such vectors. The method was extended to find reliability of a battery of tests.

Reliability and discriminating value of a Likert-type scale and Likert-type item were found in terms of angular similarity using only the permissible operations for a Likert-type scale, that is, frequencies or probabilities of item-response categories without involving assumptions of continuous nature or linearity or normality for the observed variables, or the underlying variable being measured. Thus, such reliabilities and discriminating value are in fact nonparametric. The proposed methods also avoid test of unidimensionality or assumption of normality or bivariate normality associated with the polychoric correlations. Thus, the proposed methods are considered as improvement over the existing ones.

Reliability as a measure of association of two vectors and discrimination as a measure of distance between the vectors are likely to show a negative relationship. Further studies may be undertaken to facilitate comparison of the proposed methods.