The reliability of criterion-referenced tests (CRTs) used in health program evaluations can
be conceptualized in different ways. Classical conceptualizations of reliability have limited
usefulness when applied to health-related CRTs. When a cut-score is set, the test's
reliability can be represented as the consistency of mastery/nonmastery classifications.
When a cut-score is not set, the size of the standard error of measurement (SEM) of a
domain score estimate is a reflection of the test's reliability. Formulas are presented for
estimating appropriate SEMsfor CRTs. The SEM can be used in computing confidence
intervals for domain score estimates andfor a cut-score.
Get full access to this article
View all access options for this article.
References
1.
Berk, R.A. ( 1980) "A consumer's guide to criterion-referenced test reliability." J. of Educ. Measurement17: 323-349.
2.
Brennan, R.L. (1980) "Applications of generalizability theory," pp. 186-232 in R. A. Berk (ed.) Criterion-Referenced Measurement: The State of the Art. Baltimore, MD: Johns Hopkins Univ. Press.
3.
Cohen, J. (1960) "A coefficient of agreement for nominal scales ." Educ. and Psych. Measurement20: 37-46.
4.
Cronbach, L.J., G.C. Gleser, H. Nanda and N. Rajaratnam (1972) The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. New York: John Wiley.
5.
Hambleton, R.K. and M.R. Novick (1973) "Toward an integration of theory and method for criterion-referenced tests." J. of Educ. Measurement10: 159-170.
6.
Hambleton, R.K., H. Swaminathan, J. Algina, and D.B. Coulson (1978) "Criterion-referenced testing and measurement: A review of technical issues and developments." Rev. of Educ. Research48: 1-47.
7.
Lord, F.M. (1957) "Do tests of the same length have the same standard errors of measurement?" Educ. and Psych. Measurement17: 510-521.
8.
———and Novick, M.R. (1968) Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.
9.
Millman, J. (1979) "Reliability and validity of criterion-referenced test scores," in R. E. Traub (ed.) New Directions for Testing and Measurement. San Francisco : Jossey-Bass.
Subkoviak, M.J. (1980) "Decision-consistency approaches," pp. 129-185 in R. A. Berk (ed.) Criterion-Referenced Measurement: The State of the Art. Baltimore, MD: Johns Hopkins Univ. Press.
