Development and Application of Detection Indices for Measuring Guessing Behaviors and Test-Taking Effort in Computerized Adaptive Testing

Abstract

This study examined the amount of time that different ability-level examinees spend on questions they answer correctly or incorrectly across different pretest item blocks presented on a fixed-length, time-restricted computerized adaptive testing (CAT). Results indicate that different ability-level examinees require different amounts of time to answer pretest items, that is, field-test items, embedded in different item locations. Also, guessing behaviors and test-taking effort (TTE), measured using detection indices developed from this study, varied across the pretest items embedded in different locations. With these indices, unique response time patterns (RTP) and guessing patterns (GP) were identified for six ability groups. These indices could also be used to monitor the amount of effort and the guessing behaviors of examinees taking a CAT. The degree of the detection consistency among these indices could be used as reliability evidence to evaluate how reliable the indices are in detecting unusual test-taking behaviors. Once reliable detection was confirmed, a warning sign prompted from the computer screen could either encourage the examinee to take appropriate TTE or stop guessing so that the examinees could receive more accurate estimates of their ability.

Keywords

computerized adaptive testing response time testing fairness guessing behaviors test-taking effort pretest items item locations

Get full access to this article

View all access options for this article.

References

Bridgeman, B. , & Cline, F. ( 2002, April). Fairness issues in computer adaptive tests with strict time limits. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

Bridgeman, B. , & Cline, F. ( 2004). Effects of differentially time-consuming tests on computerized adaptive test scores. Journal of Educational Measurement, 41, 137-148.

Chang, S.-R. ( 2006). Computerized adaptive test item response times for correct and incorrect pretest and operational items: Testing fairness and test-taking strategies (Unpublished doctoral dissertation). University of Nebraska-Lincoln, Lincoln, NE.

Chang, S.-R. , Ferdous, A. , & Plake, S.B. ( 2005, April). Response times for correct and incorrect item responses on computerized adaptive tests. Paper presented at the annual meeting of the American Educational Research Association (AERA), Montreal, Quebec, Canada.

Ferdous, A. , Chang, S.-R. , & Plake, S.B. ( 2005, April). Fairness issues of including pretest items on an operational CAT test. Paper presented at the annual meeting of the American Educational Research Association (AERA), Montreal, Quebec, Canada.

Hornke, L.F. ( 2000). Item response time in computerized adaptive testing. Psychologia: Revista de Metodologia y Psycologia Experimental, 21, 175-189.

Schnipke, D.L. ( 1995, April). Assessing speededness in computer-based tests using item response times. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.

Schnipke, D.L. ( 1996, April). How contaminated by guessing are item-parameter estimates and what can be done about it? Paper presented at the annual meeting of the National Council on Measurement in Education, New York, NY.

Schnipke, D.L. ( 1999). The influence of speededness on item-parameter estimation (Computerized Testing Report No. 96-07). Princeton, NJ: Law School Admission Council.

10.

Schnipke, D.L. , & Scrams, D.J. ( 1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213-232.

11.

Schnipke, D.L. , & Scrams, D.J. ( 2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In C. N. Mills , M. T.Potenza , J. J. Fremer , & W. C. Ward (Eds.), Computer-based testing: Building the foundation for future assessments (pp. 237-266). Mahwah, NJ: Lawrence Erlbaum.

12.

Wise, S.L. , & Kong, X. ( 2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education , 18, 163-183.

13.

Wise, S.L. , DeMars, C.E. , & Kong, X.J. ( 2005, April). An application of item response time: The effort-moderated IRT model. Paper presented at the annual meeting of the American Educational Research Association, Montreal , Quebec, Canada.

14.

Wise, S.L. , Pastor, D.A. , & Kong, X.J. ( 2009). Correlates of rapid-guessing behavior in low-stakes testing: Implications for test development and measurement practice. Applied Measurement in Education, 22, 185-205.