This paper presents two methods for estimating the reliability of a computerized adaptive test (CAT) with out using item response theory. The required data con sist of CAT and paper-pencil (PP) scores from identical or equivalent samples, and scores for all examinees on one or more covariates. Multiple R 2s and communali ties are used to compute the ratio of CAT and PP relia bilities. When combined with the PP reliability calcu lated by a conventional procedure, these ratios yield estimates of CAT reliability. Index terms: computer ized adaptive testing, item response theory, predictive validity, reliability, tailored testing.
Get full access to this article
View all access options for this article.
References
1.
Ackerman, T.A. (1985, October). An investigation of the effect of administering test items via the computer. Paper presented at a meeting of the Midwest Educational Research Association.
2.
Cudeck, R. (1985). A structural comparison of conventional and adaptive versions of the ASVAB. Multivariate Behavioral Research, 20, 305-322.
3.
Divgi, D.R. (1986). Determining the sensitivity of CAT-ASVAB scores to changes in item response curves with the medium of administration (Research Memorandum 86-189). Alexandria VA: Center for Naval Analyses.
4.
Green, B.F., Bock, R.D., Humphreys, L.G., Linn, R.L., & Reckase, M.D. (1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 21, 347-360.
5.
Lord, F.M. (1965). A strong true-score theory, with applications . Psychometrika, 30, 239-270.
6.
Lord, F.M. (1973). Testing if two measuring procedures measure the same dimension. Psychological Bulletin,79, 71-72.
7.
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading MA: Addison-Wesley.
8.
Maier, M.H., & Sims, W.H. (1986). The ASVAB score scales: 1980 and World War II (Report 116). Alexandria VA: Center for Naval Analyses.
9.
Owen, R.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association,70, 351-356.
10.
Samejima, F. (1977). A use of the information function in tailored testing. Applied Psychological Measurement, 1, 233-247.
11.
SAS Institute, Inc. (1986). Statistical Analysis System (Version 5.03). Cary NC: Author.
12.
Stoloff, P.H. (1983). A factor analysis of ASVAB form 8a in the 1980 DoD reference population (Memorandum 83-3135). Alexandria VA: Center for Naval Analyses.
13.
Stoloff, P.H. (1987). Equivalent-groups versus single-group equating designs for the Accelerated CAT-ASVAB Project (Research Memorandum 87-6). Alexandria VA: Center for Naval Analyses .
14.
Sympson, J.B. (1980, April). Estimating the reliability of adaptive tests from a single test administration. Paper presented at the annual meeting of the American Educational Research Association , Boston.
15.
Vicino, F.L., & Hardwicke, S.B. (1984, April). An evaluation of the utility of large scale computerized testing. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
16.
Weiss, D.J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, 473-492.
17.
Weiss, D.J., & Kingsbury, G.G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21, 361-375.
