The current study examined item exposure control procedures for testlet scored reading passages in the Verbal Reasoning section of the Medical College Admission Test with four computerized adaptive testing (CAT) systems using the partial credit model. The first system used a traditional CAT using maximum information item selection. The second used random item selection to provide a baseline for optimal exposure rates. The third used a variation of Lunz and Stahl's randomization procedure. The fourth used Luecht and Nungester's computerized adaptive sequential testing (CAST) system. A series of simulated fixed-length CATs was run to determine the optimal item selection procedure. Results indicated that both the randomization procedure and CAST performed well in terms of exposure control and measurement precision, with the CAST system providing the best overall solution when all variables were taken into consideration.
Get full access to this article
View all access options for this article.
References
1.
Andrich, D.(1985).A latent trait model for items with response dependencies: Implications for test construction and analysis. In S. Embretson (Ed.), Test design: Contributions from psychology, education, and psychometrics. New York: Academic Press.
2.
Bergstrom, B.A., & Lunz, M.E.(1999).CAT for certification and licensure. In F. Drasgow & J. Olson-Buchanan (Eds.), Innovations in computerized assessment (pp.67-91).Mahwah, NJ: Lawrence Erlbaum.
3.
Chang, H.H., & Ying, Z.(1996). A global approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.
4.
Chen, S., Ankenmann, R. D., & Spray, J. A.(1999, April). Exploring the relationship between item exposure rate and test overlap rate in computerized adaptive testing. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
5.
Chen, S., Hou, L., & Dodd, B.G.(1998). A comparison of maximum likelihood estimation and expected a posteriori estimation on CAT using the partial credit model. Educational and Psychological Measurement, 53, 61-77.
6.
Davey, T., & Parshall, C.B.(1995, April). New algorithms for item selection and exposure control with computerized adaptive testing.Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
7.
Davis, L.L.(2002). Strategies for controlling item exposure in computerized adaptive testing with polytomously scored items.Unpublished doctoral dissertation, University of Texas, Austin.
8.
Davis, L.L., Pastor, D.A., Dodd, B.G., Chiang, C., & Fitzpatrick, S.(2003). An examination of exposure control and content balancing restrictions on item selection in CATs using the partial credit model. Journal of Applied Measurement, 4, 24-42.
9.
DeAyala, R.J.(1992). The nominal response model in computerized adaptive testing. Applied Psychological Measurement, 16, 327-342.
10.
Dodd, B.G., & Koch, W.R.(1987). Effects of variations in item step values on item and test information in the partial credit model. Applied Psychological Measurement, 11, 339-351.
11.
Haynie, K.A., & Way, W.D.(1994, April). The effects of item pool depth on the accuracy of pass/fail decisions for NCLEX using CAT.Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
12.
Hetter, R.D., & Sympson, J.B.(1997).Item exposure control in CAT-ASVAB. In W. Sands, B.K. Waters, & J.R. McBride (Eds.), Computerized adaptive testing-from inquiry to operation (pp.141-144).Washington, DC: American Psychological Association.
13.
Kingsbury, G.G., & Zara, A.R.(1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2, 359-375.
14.
Koch, W.R., & Dodd, B.G.(1989). An investigation of procedures for computerized adaptive testing using partial credit scoring. Applied Measurement in Education, 2(4), 335-337.
15.
Lord, F.M.(1971). A theoretical study of two-stage testing. Psychometrika, 36, 227-241.
16.
Lord, F.M.(1974). Individualized testing and item characteristic curve theory. In Krantz, Atkinson, Luce, & Suppes (Eds.), Contemporary developments in mathematical psychology (Vol.2, pp.106-126).San Francisco: W.H. Freeman.
17.
Luecht, R.M., & Nungester, R.J.(1998). Some practical examples of computer-adaptive sequential testing. Journal of Educational Measurement, 35, 229-249.
18.
Lunz, M.E., & Stahl, J.A.(1998, April). Patterns of item exposure using a randomized CAT algorithm. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.
19.
Masters, G.N.(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
20.
McBride, J.R. & Martin, J.T.(1983).Reliability and validity of adaptive ability tests in a military setting. In D.J. Weiss (Ed.), New horizons in testing (pp.223-226).New York: Academic Press.
21.
Muraki, E.(1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
22.
Muraki, E., & Bock, R.D.(1993).The PARSCALE computer program [Computer program]. Chicago: Scientific Software International.
23.
Parshall, C.G., Davey, T., & Nering, M.L. (1998>).Test development exposure control for adaptive testing.Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.
24.
Pastor, D. A., Chiang, C., Dodd, B. G., & Yockey, R.(1999, April). Performance of the Sympson-Hetter exposure control algorithm with a polytomous item bank.Paper presented at the annual meeting of American Educational Research Association, Montreal, Canada.
25.
Pastor, D.A., Dodd, B.G., & Chang, H.H.(2002). A comparison of item selection techniques and exposure control mechanisms in CATs using the generalized partial credit model. Applied Psychological Measurement, 26, 147-163.
26.
Stocking, M.L., & Lewis, C.(1995). A new method for controlling item exposure in computer adaptive testing (Research Report 95-25).Princeton, NJ: Educational Testing Service.
27.
Stocking, M.L., & Lewis, C.(1998). Controlling item exposure conditional on ability in computerized adaptive testing, Journal of Educational and Behavioral Statistics, 23(1), 57-75.
28.
Stocking, M.L., & Lewis, C.(2000).Methods of controlling the exposure of items in CAT. In W.J. van der Linden & C.A.W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp.163-182).Amsterdam: Kluwer Academic.
29.
Sympson, J.B., & Hetter, R.D.(1985, October). Controlling item exposure rates in computerized adaptive testing.Paper presented at the annual meeting of the Military Testing Association, San Diego.
30.
Thomasson, G.L.(1998, April). CAT Item exposure control: New evaluation tools, alternate methods and integration into a total CAT program.Paper presented at the annual meeting of the National Council of Measurement in Education, San Diego.
31.
Way, W.D.(1998). Protecting the integrity of computerized testing item pools. Educational Measurement: Issues and Practice, 17(4), 17-27.
32.
Way, W. D., Zara, A., & Leahy, J. (1996, April).Strategies for managing item pools to maximize item security.Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.
33.
Weiss, D.J.(1974). Strategies of adaptive ability measurement (RR 74-5Z). Minneapolis: Psychometric Methods Program, Department of Psychology, University of Minnesota.(NTIS No.AD-A004 270)
34.
Wilson, M., & Masters, G.(1993). The partial credit model and null categories. Psychometrika, 58, 87-99.
