Direct and Inverse Problems of Item Pool Design for Computerized Adaptive Testing

Abstract

The recent literature on computerized adaptive testing (CAT) has developed methods for creating CAT item pools from a large master pool. Each CAT pool is designed as a set of nonoverlapping forms reflecting the skill levels of an assumed population of test takers. This article presents a Monte Carlo method to obtain these CAT pools and discusses its advantages over existing methods. Also, a new problem is considered that finds a population ability density function best matching the master pool. An analysis of the solution to this new problem provides testing organizations with effective guidance for maintaining their master pools. Computer experiments with a pool of Law School Admission Test items and its assembly constraints are presented.

Keywords

computerized adaptive testing item pool design automated test assembly item response theory Monte Carlo methods combinatorial optimization

Get full access to this article

View all access options for this article.

References

Ariel, A. , Veldkamp, B.P. , & van der Linden, W.J. (2004). Constructing rotating item pools for constrained adaptive testing. Journal of Educational Measurement, 41, 345-359.

Armstrong, R.D. , & Belov, D. (2006). A method for determining the maximum number of nonoverlapping linear test forms that can be assembled from an item pool (Computerized Testing Report No. 03-04). Newtown, PA: Law School Admission Council.

Armstrong, R. , Belov, D. , & Weissman, A. (2005). Developing and assembling the Law School Admission Test. Interfaces, 35, 140-151.

Belov, D.I. (2008). Uniform test assembly. Psychometrika , 73, 21-38.

Belov, D.I. , & Armstrong, R.D. (2005). Monte Carlo test assembly for item pool analysis and extension. Applied Psychological Measurement, 29, 239-261.

Belov, D.I. , & Armstrong, R.D. (2008). A Monte Carlo approach to the design, assembly and evaluation of multi-stage adaptive tests. Applied Psychological Measurement, 32, 119-137.

Cover, T.M. , & Thomas, J.A. (1991). Elements of information theory. New York: John Wiley.

ILOG, Inc. (2003). CPLEX 9.0 [Computer program and manual]. Mountain View, CA: Author. Available from www.ilog.com

Lord, F. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

10.

Stocking, M.L. , & Swanson, L. (1998). Optimal design of item pools for computerized adaptive tests. Applied Psychological Measurement, 17, 271-279.

11.

van der Linden, W.J. (2005). Linear models for optimal test design. New York: Springer.

12.

van der Linden, W.J. , Ariel, A. , & Veldkamp, B.P. (2006). Assembling a CAT item pool as a set of linear tests. Journal of Educational and Behavioral Statistics, 31, 81-99.

13.

Veldkamp, B.P. , & van der Linden, W.J. (2000). Designing item pools for computerized adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 149-162). Boston: Kluwer Academic.