Bridging Models of Biometric and Psychometric Assessment: A Three-Way Joint Modeling Approach of Item Responses,Response Times,and Gaze Fixation Counts

Abstract

Recently, joint models of item response data and response times have been proposed to better assess and understand test takers’ learning processes. This article demonstrates how biometric information such as gaze fixation counts obtained from an eye-tracking machine can be integrated into the measurement model. The proposed joint modeling framework accommodates the relations among a test taker’s latent ability, working speed and test engagement level via a person-side variance-covariance structure, while simultaneously permitting the modeling of item difficulty, time-intensity, and the engagement intensity through an item-side variance-covariance structure. A Bayesian estimation scheme is used to fit the proposed model to data. Posterior predictive model checking based on three discrepancy measures corresponding to various model components are introduced to assess model-data fit. Findings from a Monte Carlo simulation and results from analyzing experimental data demonstrate the utility of the model.

Keywords

technology enhanced assessment joint modeling item response theory response times gaze-fixation counts eye-tracking

Get full access to this article

View all access options for this article.

References

Anderson

C. H.

Van Essen

D. C.

Olshausen

B. A.

(2005). Directed visual attention and the dynamic control of information flow. In Neurobiology of attention (pp. 11–17). Elsevier. https://doi.org/10.1016/b978-012375731-9/50007-0

Aslin

R. N.

(2012). Infant eyes: A window on cognitive development. Infancy, 17(1), 126–140. https://doi.org/10.1111/j.1532-7078.2011.00097.x

Bakeman

(2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37(3), 379–384. https://doi.org/10.3758/bf03192707

Birnbaum

(1968). Some latent trait models and their use in inferring any examinees ability. In Lord

Novick

(Eds.), Statistical theories of mental test scores (pp. 395–479). Addison-Wesley

Bolsinova

De Boeck

Tijmstra

(2017). Modelling conditional dependence between response time and accuracy. Psychometrika, 82(4), 1126–1148. https://doi.org/10.1007/s11336-016-9537-6

Cochran

W. G.

(1952). The χ² test of goodness of fit. The Annals of Mathematical Statistics, 23(3), 315–345. https://doi.org/10.1214/aoms/1177729380

Cohen

(1988). Set correlation and contingency tables. Applied psychological measurement, 12(4), 425–434. https://doi.org/10.1177/014662168801200410

Corcoran

P. M.

Nanu

Petrescu

Bigioi

(2012). Real-time eye gaze tracking for gaming design and consumer electronics systems. IEEE Transactions on Consumer Electronics, 58(2), 347–355. https://doi.org/10.1109/tce.2012.6227433

De Boeck

Chen

Davison

(2017). Spontaneous and imposed speed of cognitive test responses. British Journal of Mathematical and Statistical Psychology, 70(2), 225–237. https://doi.org/10.1111/bmsp.12094

10.

Ercikan

Pellegrino

J. W.

(2017). Validation of score meaning for the next generation of assessments: The use of response processes. Taylor Francis

11.

Fox

J. P.

Entink

R. K.

Avetisyan

(2014). Compensatory and non-compensatory multidimensional randomized item response models. British Journal of Mathematical and Statistical Psychology, 67(1), 133–152.

12.

Fox

J. P.

Marianti

(2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51(4), 1–4. https://doi.org/10.1080/00273171.2016.1171128

13.

Fox

J. P.

Marianti

(2017). Person-fit statistics for joint models for accuracy and speed. Journal of Educational Measurement, 54(2), 243–262. https://doi.org/10.1111/jedm.12143

14.

Gelman

Carlin

J. B.

Stern

H. S.

Rubin

D. B.

(2003). Bayesian data analysis. Chapman & Hall

15.

Gelman

Meng

X. L.

Stern

(1996). Posterior predictive assessment of model fitness via realized discrepancies. Journal of Educational and Behavioral Statistics, 6(4), 733–760.

16.

Goldberg

J. H.

Wichansky

A. M.

(2003). Eye tracking in usability evaluation: A practitioner’s guide. In The mind’s eye (pp. 493–516). Elsevier. https://doi.org/10.1016/b978-044451020-4/50027-x

17.

Hao

Smith

Mislevy

von Davier

Bauer

(2016). Taming log files from game/simulation-based assessments: Data models and data analysis tools. ETS Research Report Series, 1(1), 1–17. https://doi.org/10.1002/ets2.12096

18.

Jacob

R. J.

Karn

K. S.

(2003). Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. The Mind’s Eye, 2(3), 573–605. https://doi.org/10.1016/b978-044451020-4/50031-1

19.

Jiao

Lissitz

(2018). Technology enhanced innovative assessment development, modeling, and scoring from an interdisciplinary perspective. Information Age Publishing

20.

Justice

Lankford

(2002). Pilot findings. Communication Disorders Quarterly, 24(1), 11–21. https://doi.org/10.1177/152574010202400103

21.

Klein Entink

R. H.

Kuhn

J. T.

Hornke

L. F.

Fox

J. P.

(2009). Evaluating cognitive theory: A joint modeling approach using responses and response times. Psychological Methods, 14(1), 54–75. https://doi.org/10.1037/a0014877

22.

Lee

Y.-H.

Hao

J. G.

Man

(2019). How do test takers interact with simulation-based tasks? a response-time perspective. Frontiers in Psychology, 10, 906. https://doi.org/10.3389/fpsyg.2019.00906

23.

Levy

Mislevy

R. J.

Sinharay

(2009). Posterior predictive model checking for multidimensionality in item response theory. applied psychological measurement. Applied Psychological Measurement, 33(7), 519–537. https://doi.org/10.1177/0146621608329504

24.

Linacre

Wright

(1994). Chi-square fit statistics. Rasch Measurement Transactions, 8(2), 350.

25.

Lord

F. M.

(1952). A theory of test scores (psychometric monograph no. 7). In Annual meeting of the psychometric society. Iowa City

26.

Luo

Jiao

(2018). Using the stan program for bayesian item response theory. Educational and Psychological Measurement, 78(3), 384–408. https://doi.org/10.1177/0013164417693666

27.

Luo

(2021). A Comparison of Common IRT Model-selection Methods with Mixed-Format Tests. Measurement: Interdisciplinary Research and Perspectives, 19(4), 199–212.

28.

Man

Harring

J. R.

(2019). Negative binomial models for visual fixation counts on test items. Educational and Psychological Measurement, 79(4), 617–635. https://doi.org/10.1177/0013164418824148

29.

Man

Harring

J. R.

(2021). Assessing preknowledge cheating via innovative measures: A multiple-group analysis of jointly modeling item responses, response times, and visual fixation counts. Educational and Psychological Measurement, 81(3), 441–465. https://doi.org/10.1177/0013164420968630

30.

Man

Harring

J. R.

Jiao

Zhan

(2019). Joint modeling of compensatory multidimensional item responses and response times. Applied Psychological Measurement, 43(8), 639–654. https://doi.org/10.1177/0146621618824853

31.

Marianti

Fox

J. P.

Avetisyan

Veldkamp

Tijmstra

(2014). Testing for aberrant behavior in response time modeling. Journal of Educational and Behavioral Statistics, 39(6), 426–451. https://doi.org/10.3102/1076998614559412

32.

Molenaar

Tuerlinckx

van der Maas

H. L. J.

(2015). A generalized linear factor model approach to the hierarchical framework for responses and response times. British Journal of Mathematical and Statistical Psychology, 68(2), 197–219. https://doi.org/10.1111/bmsp.12042

33.

Morad

Lemberg

Dagan

(2000). Pupillography as an objective indicator of fatigue. Current Eye Research, 21(1), 535–542. https://doi.org/10.1076/0271-3683(200007)2111-zft535

34.

Muthén

L. K.

Muthén

B. O.

(1998–2017). Mplus User’s Guide. Eighth Edition. Los Angeles, CA: Muthén & Muthén.

35.

Muthén

L. K.

Muthén

B. O.

(1998–2012). Mplus User’s Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén.

36.

Plummer

(2015). JAGS: Version 4.0. User Manual. Retrieved from http://www.uvm.edu/~bbeckage/Teaching/DataAnalysis/Manuals/manual.jags.pdf)

37.

Poole

Ball

L. J.

Phillips

(2004). In search of salience: A response-time and eye-movement analysis of bookmark recognition. In Fincher

Markopoulos

Moore

Ruddle

(Eds.), People and computers xviii–design for life. Springer

38.

Rasch

(1960). Studies in mathematical psychology: I. probabilistic models for some intelligence and attainment tests. In Danish institute for educational research (Expanded edition). University of Chicago Press

39.

Rayner

(1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372–422. https://doi.org/10.1037/0033-2909.124.3.372

40.

Rijn

P. W.

Ali

U. S.

(2017). A comparison of item response models for accuracy and speed of item responses with applications to adaptive testing. British Journal of Mathematical and Statistical Psychology, 70(2), 317–345. https://doi.org/10.1111/bmsp.12101

41.

Samejima

(1996). The graded response model. In van der Unden

Hambleton

R. K.

(Eds.), Handbook of modern item response theory. Springer

42.

Shagass

Roemer

R. A.

Amadeo

(1976). Eye-tracking performance and engagement of attention. Archives of General Psychiatry, 33(1), 121–125. https://doi.org/10.1001/archpsyc.1976.01770010077015

43.

Sinharay

Johnson

M. S.

Stern

H. S.

(2006). Posterior predictive assessment of item response theory models. Applied Psychological Measurement, 30(4), 298–321. https://doi.org/10.1177/0146621605285517

44.

Y. S.

Yajima

M. Y. S.

System Requirements

J. A. G. S.

(2015). Package ‘R2jags’. R package version 0.03–08, URL http://CRAN.R-project.org/package=R2jags

45.

van der Linden

W. J

(2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204. https://doi.org/10.3102/10769986031002181

46.

van der Linden

W. J.

(2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z

47.

van der Linden

W. J.

Entink

R. H. K.

Fox

J. P.

(2010). Irt parameter estimation with response times as collateral information. Applied Psychological Measurement, 34(5), 327–347. https://doi.org/10.1177/0146621609349800

48.

Van der Linden

W. J.

Hambleton

R. K.

(1997). Handbook of modern item response theory. Springer Science & Business Media

49.

Wang

Fan

Chang

H.-H.

Douglas

J. A.

(2013). A semiparametric model for jointly analyzing response times and accuracy in computerized testing. Journal of Educational and Behavioral Statistics, 38(4), 381–417. https://doi.org/10.3102/1076998612461831

50.

Wang

(2010). Irtzip modeling for multivariate zero-inflated count data. Journal of Educational and Behavioral Statistics, 35(6), 671–692. https://doi.org/10.3102/1076998610375838

51.

Wise

S. L.

Kong

X. J.

(2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Psychological Measurement, 18(2), 163–183. https://doi.org/10.1207/s15324818ame1802_2

52.

Wright

B. D.

Stone

M. H.

(1979). Best Test Design. Rasch Measurement.

53.

Adams

Wilson

Haldane

(1998). Conquest: Generalized item response modeling software [computer software and manual]. Australian Council for Educational Research

54.

Yoss

R. E.

Moyer

N. J.

Hollenhorst

R. W.

(1970). Pupil size and spontaneous pupillary waves associated with alertness, drowsiness, and sleep. Neurology, 20(6), 545. https://doi.org/10.1212/wnl.20.6.545

55.

Zhan

Jiao

Liao

(2017). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286. https://doi.org/10.1111/bmsp.12114

56.

Zhan

Jiao

Liao

(2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286. https://doi.org/10.1111/bmsp.12114

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB