On the Benefits of Using Maximal Reliability in Educational and Behavioral Research

Abstract

This note is concerned with the benefits that can result from the use of the maximal reliability and optimal linear combination concepts in educational and psychological research. Within the widely used framework of unidimensional multi-component measuring instruments, it is demonstrated that the linear combination of their components that possesses the highest possible reliability can exhibit a level of consistency considerably exceeding that of their overall sum score that is nearly routinely employed in contemporary empirical research. This optimal linear combination can be particularly useful in circumstances where one or more scale components are associated with relatively large error variances, but their removal from the instrument can lead to a notable loss in validity due to construct underrepresentation. The discussion is illustrated with a numerical example.

Keywords

maximal reliability multi-component measuring instrument optimal linear combination scale reliability unweighted overall sum score unidimensional scale

Get full access to this article

View all access options for this article.

References

Agresti

Finlay

(2009). Statistical methods for the social sciences. Prentice Hall.

Apostol

(2006). Calculus. Wiley.

Bartholomew

D. J.

(1996). The statistical approach to social measurement. Academic Press.

Bollen

K. A.

(1980). Issues in the comparative measurement of political democracy. American Sociological Review, 45, 370–390.

Crocker

Algina

(2006). Introduction to classical and modern test theory. Harcourt College Publishers.

Efron

Tibshiriani

R. J.

(1993). An introduction to the bootstrap. Chapman & Hall.

Geiser

(2013). Data analysis with Mplus. The Guilford Press.

Green

B. F.

(1950). A note on the calculation of the weights for maximum battery reliability. Psychometrika, 15, 55–61.

Hancock

G. R.

Nevitt

(1999). Bootstrapping and the identification of exogenous latent variables within structural equation models. Structural Equation Modeling, 6, 394–399.

10.

Kelley

T. L.

(1927). Interpretation of educational measurement. World Book Company.

11.

(1997). A unifying expression for the maximal reliability of a linear composite. Psychometrika, 62(2), 245–249.

12.

Rosenthal

Rubin

D. B.

(1996). Reliability of measurement in psychology: From Spearman-Brown to maximal reliability. Psychological Methods, 1(1), 98–107.

13.

McDonald

R. P.

(1999). Test theory. A unified treatment. Lawrence Erlbaum.

14.

Messick

(1995). Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.

15.

Mulaik

S. A.

(2009). Foundations of factor analysis. CRC Press.

16.

Muthén

B. O.

(2002). Beyond SEM: General latent variable modeling. Behaviormetrika, 29, 87–117.

17.

Muthén

L. K.

Muthén

B. O.

(2023). Mplus user’s guide. Muthén & Muthén.

18.

Rabe-Hesketh

Skrondal

(2022). Multilevel and longitudinal modeling with Stata. Stata Press.

19.

Raykov

(2006). Interval estimation of individual optimal scores from multiple-component measuring instruments. Structural Equation Modeling, 13, 252–261.

20.

Raykov

Gabler

Dimitrov

D. M.

(2016). Maximal reliability and composite reliability: A latent variable modeling approach to their difference evaluation. Structural Equation Modeling, 23, 384–391.

21.

Raykov

Rodenberg

C. N.

Narayanan

(2015). Optimal shortening of psychometric scales. Structural Equation Modeling, 22, 227–235.

22.

Spearman

(1904). General intelligence, objectively determined and measured. American Journal of Psychology, 15, 201–292.

23.

Thomson

G. H.

(1940). Weighting for battery reliability and prediction. British Journal of Mathematical and Statistical Psychology, 30, 357–360.