Impact of Parameter Predictability and Joint Modeling of Response Accuracy and Response Time on Ability Estimates

Abstract

To maintain test quality, a large supply of items is typically desired. Automatic item generation can result in a reduction in cost and labor, especially if the generated items have predictable item parameters and thus possibly reducing or eliminating the need for empirical tryout. However, the effect of different levels of item parameter predictability on the accuracy of trait estimation using item response theory models is unclear. If predictability is lower, adding response time as a collateral source of information may mitigate the effect on trait estimation accuracy. The present study investigates the impact of varying item parameter predictability on trait estimation accuracy, along with the impact of adding response time as a collateral source of information. Results indicated that trait estimation accuracy using item family model-based item parameters differed only slightly from using known item parameters. Somewhat larger trait estimation errors resulted from using cognitive complexity features to predict item parameters. Further, adding response times to the model resulted in more accurate trait estimation for tests with lower item difficulty levels (e.g., achievement tests). Implications for item generation and response processes aspect of validity are discussed.

Keywords

item response theory item generation item parameter accuracy response processes response time

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Joint Committee on Standards for Educational, & Psychological Testing (US) . (1999, 2014). Standards for educational and psychological testing. American Educational Research Association.

Bejar

I. I.

(1993). A generative approach to psychological and educational measurement. In Frederiksen

Mislevy

R. J.

Bejar

I. I.

(Eds.), Test theory for a new generation of tests (pp. 323–359). Lawrence Erlbaum.

Bejar

I. I.

Lawless

R. R.

Morley

M. E.

Wagner

M. E.

Bennett

R. E.

Revuelta

(2003). A feasibility study of on-the-fly item generation in adaptive testing. The Journal of Technology, Learning, and Assessment, 2(3), 1–29. https://ejournals.bc.edu/index.php/jtla/article/view/1663.

De Boeck

Rijmen

(2020). Response time in cognitive tests: Interpretation and importance. In Margolis

M. J.

Feinberg

R. A.

(Eds.), Integrating timing considerations to improve testing practices (pp. 142–149). Routledge.

Embretson

S. E.

(1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3(3), 380–396. https://doi.org/10.1037//1082-989x.3.3.380

Embretson

S. E.

(1999). Generating items during testing: Psychometric issues and models. Psychometrika, 64(4), 407–433. https://doi.org/10.1007/bf02294564

Embretson

S. E.

(2016). Understanding examinees’ responses to items: Implications for measurement. Educational Measurement: Issues and Practice, 35(3), 6–22. https://doi.org/10.1111/emip.12117

Embretson

S. E.

Daniel

R. C.

(2008). Understanding and quantifying cognitive complexity level in mathematical problem solving items. Psychology Science Quarterly, 50, 328–344. DOI:10.1177/0146621609349801

Embretson

S. E.

Kingston

(2018). Automatic item generation: A more efficient process for developing mathematics achievement items? Journal of Educational Measurement, 55(1), 112–131. https://doi.org/10.1111/jedm.12166

10.

Enright

M. K.

Morley

Sheehan

K. M.

(2002). Items by design: The impact of systematic feature variation on item statistical characteristics. Applied Measurement in Education, 15(1), 49–74. https://doi.org/10.1207/s15324818ame1501_04

11.

Fischer

G. H.

(1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37(6), 359–374. https://doi.org/10.1016/0001-6918(73)90003-6

12.

Gierl

M. J.

Lai

(2017). The role of cognitive models in automatic item generation. In Rupp

Leighton

(Eds.), The handbook of cognition and assessment. Wiley- Blackwell.

13.

Grice

Iwasaki

(2007). A truly multivariate approach to MANOVA. Applied Multivariate Research, 12(3), 199–226. https://doi.org/10.22329/amr.v12i3.660

14.

Handy

(2019). A study of a fit Index for explanatory item response theory models. Unpublished Masters Thesis. School of Psychology, Georgia Institute of Technology.

15.

Janssen

(2016). Linear logistic models. In van der Linden

W. J.

(Ed.), Handbook of item response theory, volume one: Models (pp. 211–224). CRC Press, Taylor and Francis.

16.

Janssen

Schepers

Peres

(2004). Models with item and item group predictors. In De Boeck

Wilson

(Eds.), Explanatory item response models: A generalized linear and nonlinear approach (pp. 189–212). Springer.

17.

Klein Entink

Kuhn

Hornke

Fox

(2009). Evaluating cognitive theory: A joint modeling approach using responses and response times. Psychological Methods, 14(1), 54–75. https://doi.org/10.1037/a0014877

18.

Lee

S. J.

Wollack

J. A.

(2020). Concurrent use of response time and response accuracy for detecting examinees with preknowledge. In Margolis

M. J.

Feinberg

R. A.

(Eds.), Integrating timing considerations to improve testing practices (pp. 165–175). Routledge.

19.

Molenaar

Tuerlinckx

van der Maas

H. L. J.

(2015). A bivariate generalized linear item response theory modeling framework to the analysis of responses and response times. Multivariate Behavioral Research, 50(1), 56–74. https://doi.org/10.1080/00273171.2014.962684

20.

Rupp

A. A.

Leighton

J. P.

(2017). The handbook of cognition and assessment. Wiley-Blackwell.

21.

van den Noortgate

Paek

(2004). Person regression models. In De Boeck

Wilson

(Eds.), Explanatory item response models: A generalized linear and nonlinear approach (pp. 167–188). Springer.

22.

van der Linden

W. J.

(2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z

23.

van der Linden

W. J.

(2016). Lognormal response-time model. In van der Linden

W. J.

(Ed.), Handbook of item response theory, Volume one, Models (pp. 261–282). CRC Press.

24.

van der Linden

W. J.

Entink

R. H. K.

Fox

J. P.

(2010). IRT parameter estimations with response time as collateral information. Applied Psychological Measurement, 34(5), 327–347. https://doi.org/10.1177/0146621609349800

25.

van der Linden

W. J.

Fox

J. P.

(2016). Joint hierarchical modeling of responses and response times. In van der Linden

W. J.

(Ed.), Handbook of item response theory, volume one: Models (pp. 481–501). CRC Press, Taylor and Francis Group.