The goal of the present study was to address the analytical complexity of incorporating responses and response times through applying the Jeon and De Boeck mixture item response theory model in Mplus 8.7. Using both simulated and real data, we attempt to identify subgroups of responders that are rapid guessers or engage knowledge retrieval strategies. When applying the mixture model to a measure of contextual error in linguistics results pointed to the presence of a knowledge retrieval strategy. That is, a participant either knows the content (morphology, grammar rules) and can identify the error, or lacks the requisite knowledge and cannot benefit from spending more time on an item. In contrast, as item difficulty progressed, the high-ability group utilized the additional time to make informed guesses. The methodology is illustrated using annotated code in Mplus 8.7.
AndersonD. R. (2008). Model based inference in the life sciences. Springer.
2.
BockenholtU.AlbertD.AschenbrennerM.SchmalhoferF. (1991). The effects of attractiveness, dominance, and attribute differences on information acquisition in multiattribute binary choice. Organizational Behavior and Human Decision Processes, 49, 258–281.
3.
BolsinovaM.TijmstraJ. (2018). Improving precision of ability estimation: Getting more from response times. British Journal of Mathematical and Statistical Psychology, 71, 13–38.
4.
BoughtonK.SmithJ.RenH. (2017). Using response time data to detect compromised items and/or people. In CizekG. J.WollackJ. A. (Eds.), Handbook of detecting cheating on tests (pp. 177–190). Routledge.
5.
ChenW.-H.ThissenD. (1997). Local dependence indices for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265–289.
6.
ErwinT. D.WiseS. L. (2002). A scholar-practitioner model for assessment. In Banta & Associates, T. W. (Eds.), Building a scholarship of assessment (pp. 67–81). Jossey-Bass.
7.
FerrandoP. J.Lorenzo-SevaU. (2007). An item response theory model for incorporating response time data in binary personality items. Applied Psychological Measurement, 31, 525–543.
8.
FoleyB. P. (2016). Getting lucky: How guessing threatens the validity of performance classifications. Practical Assessment, Research, and Evaluation, 21, Article 3.
9.
FoxJ. P.MariantiS. (2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51, 540–553.
10.
GlasC. A. W.van der LindenW. J. (2010). Marginal likelihood inference for a model for item responses and response times. British Journal of Mathematical and Statistical Psychology, 63, 603–626.
11.
HornbyA. S. (2001). Oxford advanced learner’s dictionary for current English. Oxford University Press.
12.
HaladynaT. M. (2004). Developing and validating multiple-choice test items. NY: Psychology Press.
13.
JeonM.De BoeckP. (2019). An analysis of an item-response strategy based on knowledge retrieval. Behavior Research Methods, 51, 697–719.
14.
Klein EntinkR. H.FoxJ.-P.Van der LindenW. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74, 21–48.
15.
KubingerK. D.DraxlerC. (2006). A comparison of the Rasch model and constrained item response theory models for pertinent psychological test data. In von DavierM.CarstensenC. H. (Eds.), Multivariate and mixture distribution Rasch Models—Extensions and applications (pp. 295–312). Springer.
16.
LeeS.BoltD. M. (2017). Asymmetric item characteristic curves and item complexity: Insights from simulation and real data analyses. Psychometrika, 83, 453–475.
17.
LordF. M. (1964). The effect of random guessing on test validity. Educational and Psychological Measurement, 24, 745–747.
18.
ManK.HarringJ. R. (2019). Negative binomial models for visual fixation counts on test items. Educational and Psychological Measurement, 79(4), 617–635. https://doi.org/10.1177/0013164418824148
19.
ManK.HarringJ. R.JiaoH.ZhanP. (2019). Joint modeling of compensatory multidimensional item responses and response times. Applied Psychological Measurement, 438(8), 639–654. https://doi.org/10.1177/0146621618824853
20.
MeijerJ. (2001). Learning potential and anxious tendency: Test anxiety as a bias factor in educational testing. Anxiety, Stress & Coping, 14(3), 337–362.
21.
MolenaarD. (2016). The value of response times in the item response modeling. Measurement, 13, 177–181.
22.
OrlandoM.ThissenD. (2000). Likelihood-based item fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50–64.
23.
OrlandoM.ThissenD. (2003). Further investigation of the performance of S-X 2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289–298.
24.
OtterT.AllenbyG.van ZandtT. (2008). An integrated model of discrete choice and response time. Journal of Marketing Research, 45, 593–607.
25.
PaekI. (2016). Evaluating the impact of guessing and its interactions with other test characteristics on confidence interval procedures for coefficient alpha. Educational and Psychological Measurement, 76, 205–230.
26.
QianH.StaniewskaD.ReckaseM.WooA. (2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement, 35(1), 38–47.
27.
RangerJ.KuhnJ. T.GaviriaJ. L. (2015). A race model for responses and response times in tests. Psychometrika, 80, 791–810.
28.
R Core Team. (2015). R: A language and environment for statistical computing [Computer software]. http://www.R-project.org/
29.
RouderJ. N.ProvinceJ. M.MoreyR. D.GomezP.HeathcoteA. (2015). The lognormal race: A cognitive process model of choice and latency with desirable psychometric properties. Psychometrika, 80, 491–513.
30.
RiosJ. A.SolandJ. (2021). Parameter Estimation Accuracy of the Efort-Moderated Item Response Theory Model Under Multiple Assumption Violations. Educational and Psychological Measurement, 18(3), 0013164420949896.
31.
San MartinE.del PinoG.De BoeckP. (2006). IRT models for ability-based guessing. Applied Psychological Measurement, 30(3), 183–203.
32.
SemmesR.DavisonM. L.CloseC. (2011). Modeling individual differences in numerical reasoning speed as a random effect of response time limits. Applied Psychological Measurement, 35(6), 433–446.
33.
UbulomW. J.AminiC.EremieM.OshoG.WoodsD. (2012). Determining the effect of guessing on test scores: An empirical analysis. Scottish Journal of Arts, Social Sciences and Scientific Studies, 4, 32–38.
34.
van der LindenW. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181–204.
35.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308.
36.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308.
37.
van der LindenW. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 463(3), 247–272.
38.
Van der MaasH. L. J.MolenaarD.MarisG.KievitR. A.BorsboomD. (2011). Cognitive psychology meets psychometric theory: On the relation between process models for decision making and latent variable models for individual differences. Psychological Review, 118, 339–356.
39.
WiseS. L. (2019). An information-based approach to identifying rapid-guessing thresholds. Applied Measurement in Education, 32, 325–336.
40.
WiseS. L.DeMarsC. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19–38.
41.
WiseS. L.KongX. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.
42.
YamamotoK. H. (1989). A HYBRID model of IRT and latent class models [ETS Research Report SeriesA HYBRID model of IRT and latent class models ETS Research Report Series]. Educational Testing Service.
43.
YamamotoK. H. (1997). Everson modeling the effects of test length and test time on parameter estimation using the HYBRID model modeling the effects of test length and test time on parameter estimation using the HYBRID model. In RostJ.LangeheineR. (Eds.), Applications of latent trait and latent class models in the social sciences (pp. 89–98). Waxmann Verlag GmbH.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
