Item Selection in Computerized Adaptive Testing: Improving the a-Stratified Design with the Sympson-Hetter Algorithm

Abstract

Item exposure control, test-overlap minimization, and the efficient use of item pool are some of the important issues in computerized adaptive testing (CAT) designs. The overexposure of some items and high test-overlap rate may cause both item and test security problems. Previously these problems associated with the maximum information (Max-I) item selection method have been partially solved by incorporating the Sympson-Hetter (SH) probabilistic procedure that attempts to control the actual usage of the popular high-discrimination items. On a different line of thought, Chang and Ying proposed the a-stratified design (STR) and advocated the use of less discriminating items in the earlier stages of testing. The method has been demonstrated to be effective in improving the utilization of the entire pool without sacrificing the efficiency in ability estimation. Nevertheless, the problem of item overexposure still persists when the ratio of pool size to test length is small. In four simulation studies, the possible benefits of incorporating the SH procedure in the STR method were investigated. The performance of such an enhanced stratified method (STR-SH) was compared with that of STR as well as Max-I-SH. The results indicated the potential advantages of the STR-SH design over the original STR in yielding a more balanced item exposure distribution, lowering the test-overlap rate, and remarkably reducing the number of overexposed items.

Get full access to this article

View all access options for this article.

References

Armstrong, R. D. , Jones, D. H. , & Kunce, C. S. (1998). IRT test assembly using network-flow programming. Applied Psychological Measurement, 22, 237-247.

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord , & M. R. Novick (Eds.), Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

Chang, H. H. , & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.

Chang, H. H. , & Ying, Z. (1999). A-stratified multistage computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.

Chang, H. H. , & Zhang, J. (1999, June). Hypergeometric family and test overlap rates in computerized adaptive testing. Paper presented at the annual meeting of the Psychometric Society, Lawrence, KS.

Chen, S. , Ankenmann, R. D. , & Spray, J. A. (1999, April). Exploring the relationship between item exposure rate and test overlap rate in computerized adaptive testing. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.

Davey, T. , & Parshall, C. G. (1995, April). New algorithms for item selection and exposure control with computerized adaptive testing. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, USA.

Hambleton, R. K. , Zaal, N. J. , & Pieters, J. P. M. (1991). Computerized adaptive testing: Theory, applications, and standards. In R. K. Hambleton & N. J. Zaal (Eds.), Advances in educational and psychological testing: Theory and applications (pp. 341-366). Boston: Kluwer Academic.

Hau, K. T. , & Chang, H. H. (2001). Item selection in computerized adaptive testing: Should more discriminating items be used first? Journal of Educational Measurement, 38, 249-266.

10.

Hetter, R. D. , & Sympson, J. B. (1997). Item exposure control in CAT-ASVAB. In W. A. Sands , B. K. Waters , & J. R. McBride (Eds.), CAT: From inquiry to operation. Washington, DC: American Psychological Association.

11.

Hulin, C. L. , Drasgow, F. , & Parsons, C. K. (1983). Item response theory: Application to psychological measurement. Homewood, IL: Dow Jones-Irwin.

12.

Johnson, E. G. , & Carlson, J. E. (1994). The NAEP 1992 technical report. Washington, DC: National Center of Education Statistics.

13.

Lord, M. F. (1970). Some test theory for tailored testing. In W. H. Holzman (Ed.), Computer assisted instruction, testing, and guidance.New York: Harper and Row.

14.

Lord, M. F. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

15.

Luecht, R. M. (1998). Computer-assisted test assembly using optimization heuristics. Applied Psychological Measurement, 22, 224-236.

16.

Owen, R. J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70, 351-356.

17.

Parshall, C. G. , Davey, T. , & Nering, M. L. (1998, April). Test development exposure control for adaptive testing. Paper presented at the symposium Adaptive Testing Research at ACT at the annual meeting of the National Council on Measurement in Education, San Diego.

18.

Stocking, M. L. , & Lewis, C. (1998). Controlling item exposure conditional on ability in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 23, 57-75.

19.

Swanson, L. , & Stocking, M. L. (1993). A model and heuristic for solving very large item selection problems. Applied Psychological Measurement, 17, 151-166.

20.

Sympson, J. B. , & Hetter, R. D. (1985). Controlling item-exposure rates in computerized adaptive testing. Paper presented in Proceedings of the 27th Annual Meeting of the Military Testing Association (pp. 973-977). San Diego, CA: Navy Personnel Research and Development Center.

21.

Thomasson, G. L. (1995, June). New item exposure control algorithms for computerized adaptive testing. Paper presented at the Annual Meeting of Psychometric Society, Minneapolis, MN.

22.

Urry, V. W. (1970). A Monte Carlo investigation of logistic test models. Unpublished doctoral dissertation, Purdue University.

23.

Vale, C. D. , & Weiss, D. J. (1975). A simulation study of stradaptive ability testing (RR 75-6). Minneapolis, MN: Psychometric Methods Program, Department of Psychology, University of Minnesota.

24.

van der Linden, W. J. , & Reese, L. M. (1998). A model for optimal constrained adaptive testing. Applied Psychological Measurement, 22, 259-270.

25.

Wainer, H. , Dorans, N. J. , Flaugher, R. , Green, B. F. , Mislevy, R. J. , Steinberg, L. , et al. (1998). Protecting the integrity of computerized testing item pools. Educational Measurement: Issues and Practice, 17, 17-27.

26.

Weiss, D. J. (1974). Strategies of adaptive ability measurement (Research Report RR74-5). Minneapolis: University of Minnesota, Department of Psychology, Psychometric Methods Program.

27.

Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, 473-492.