Setting the Response Time Threshold Parameter to Differentiate Solution Behavior From Rapid-Guessing Behavior

Abstract

This study compared four methods for setting item response time thresholds to differentiate rapid-guessing behavior from solution behavior. Thresholds were either (a) common for all test items, (b) based on item surface features such as the amount of reading required, (c) based on visually inspecting response time frequency distributions, or (d) statistically estimated using a two-state mixture model. The thresholds were compared using the criteria proposed by Wise and Kong to establish the reliability and validity of response time effort scores, which were generated on the basis of the specified threshold values. The four methods yielded very similar results, indicating that response time effort is not very sensitive to the particular threshold identification method used. Recommendations are given regarding use of the various methods.

Keywords

response time effort rapid-guessing behavior low-stakes testing

Get full access to this article

View all access options for this article.

References

Association of College and Research Libraries. ( 2000). Information literacy competency standards for higher education, 2000. Chicago: Author .

Haladyna, T.M. , & Downing, S.M. (2004). Construct-irrelevant variance in high-stakes testing . Educational Measurement: Issues and Practice, 23(1), 17-27.

McLachlan, G. , & Peel, D. (2000). Finite mixture models. New York: Wiley.

Schnipke, D.L. , & Scrams, D.J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213-232.

Schnipke, D.L. , & Scrams, D.J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In C. N. Mills , M. T. Potenza , J. J. Fremer , & W. C. Ward (Eds.), Computer-based testing: Building the foundation for future assessments (pp. 237-266). Mahwah, NJ: Lawrence Erlbaum.

Sundre, D.L. (1999, April). Does examinee motivation moderate the relationship between test consequences and test performance? Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada. (ERIC Document Reproduction Service No. ED432588)

Sundre, D.L. , & Kitsantas, A. (2004). An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance? Contemporary Educational Psychology, 29, 6-26.

Sundre, D.L. , & Moore, D.L. (2002). The Student Opinion Scale: A measure of examinee motivation. Assessment Update, 14(1), 8-9.

Sundre, D.L. , & Wise, S.L. (2003, April). `Motivation filtering': An exploration of the impact of low examinee motivation on the psychometric quality of tests . Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

10.

van Zandt, T. (2000). How to fit a response time distribution. Psychometric Bulletin & Review, 7, 434-465.

11.

Wand, M.P. , & Jones, M.C. (1995). Kernel smoothing. London: Chapman and Hall.

12.

Wise, S.L. (2006). An investigation of the differential effort received by items on a low-stakes, computer-based test. Applied Measurement in Education, 19, 25-114.

13.

Wise, S.L. , Bhola, D.S. , & Yang, S. (2006). Taking the time to improve the validity of low-stakes tests: The effort-monitoring CBT. Educational Measurement: Issues and Practice, 25, 21-30.

14.

Wise, S.L. , & De Mars, C.E. (2005). Examinee motivation in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1-17.

15.

Wise, S.L. , & De Mars, C.E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19-38.

16.

Wise, S.L. , Kingsbury, G.G. , Thomason, J. , & Kong, X. (2004, April). An investigation of motivation filtering in a statewide achievement testing program. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.

17.

Wise, S.L. , & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163-183.

18.

Wise, V.L. , Bhola, D.S. , & Wise, S.L. (2006). The generalizability of motivation filtering in improving test score validity. Educational Assessment, 11, 65-83.

19.

Wolf, L.F. , & Smith, J.K. (1995). The consequence of consequence: Motivation, anxiety, and test performance. Applied Measurement in Education, 8, 227-242.