The prevalence and serious consequences of noneffortful responses from unmotivated examinees are well-known in educational measurement. In this study, we propose to apply an iterative purification process based on a response time residual method with fixed item parameter estimates to detect noneffortful responses. The proposed method is compared with the traditional residual method and noniterative method with fixed item parameters in two simulation studies in terms of noneffort detection accuracy and parameter recovery. The results show that when severity of noneffort is high, the proposed method leads to a much higher true positive rate with a small increase of false discovery rate. In addition, parameter estimation is significantly improved by the strategies of fixing item parameters and iteratively cleansing. These results suggest that the proposed method is a potential solution to reduce the impact of data contamination due to severe low test-taking effort and to obtain more accurate parameter estimates. An empirical study is also conducted to show the differences in the detection rate and parameter estimates among different approaches.
BoltD.CohenA.WollackJ. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39, 331–348.
2.
BridgemanB.ClineF. (2004). Effects of differentially time-consuming tests on computer-adaptive test scores. Journal of Educational Measurement, 41(2), 137–148.
3.
CandellG. L.DrasgowF. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253–260.
4.
ChoeE. M.KernJ. L.ChangH. H. (2018). Optimizing the use of response times for item selection in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 43(2), 135–158.
5.
de la TorreJ.DengW. (2008). Improving person-fit assessment by correcting the ability estimate and its reference distribution. Journal of Educational Measurement, 45(2), 159–177.
6.
DeMarsC. E. (2007). Changes in rapid-guessing behavior over a series of assessments. Educational Assessment, 12, 23–45.
7.
FanZ.WangC.ChangH. H.DouglasJ. (2012). Utilizing response time distributions for item selection in CAT. Journal of Educational and Behavioral Statistics, 37(5), 655–670.
8.
FisherR. A. (1915). Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika, 10(4), 507–521.
9.
FrenchB. F.MallerS. J. (2007). Iterative purification and effect size use with logistic regression for differential item functioning detection. Educational and Psychological Measurement, 67, 373–393.
10.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472.
11.
GuoH.RiosJ. A.HabermanS.LiuO. L.WangJ.PaekI. (2016). A new procedure for detection of students’ rapid guessing responses using response time. Applied Measurement in Education, 29, 173–183.
12.
Hidalgo-MontesinosM. D.Gómez-BenitoJ. (2003). Test purification and the evaluation of differential item functioning with multinomial logistic regression. European Journal of Psychological Assessment, 19, 1–11.
13.
HollandW. P.ThayerD. T. (1988). Differential item performance and the Mantel–Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp. 129–145). Lawrence Erlbaum.
14.
HongM. R.ChengY. (2019). Robust maximum marginal likelihood (RMML) estimation for item response theory models. Behavior Research Methods, 51(2), 573–588.
15.
KolenM. J.BrennanR. L. (2004). Test equating, scaling, and linking: Methods and practices. Springer Verlag.
16.
KongX. J.WiseS. L.BholaD. S. (2007). Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behavior. Educational and Psychological Measurement, 67(4), 606–619.
17.
KuijpersR. E.VisserI.MolenaarD. (2020). Testing the within-state distribution in mixture models for responses and response times. Journal of Educational and Behavioral Statistics. Advance online publication. https://doi.org/10.3102/1076998620957240
18.
LeeY. H.JiaY. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-Scale Assessments in Education, 2(8), 1–24.
19.
LiuY.ChengY.LiuH. (2020). Identifying effortful individuals with mixture modeling response accuracy and response time simultaneously to improve item parameter estimation. Educational and Psychological Measurement, 80(4), 775–807.
20.
LordF. M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum.
21.
MolenaarD.BolsinovaM.VermuntJ. K. (2018). A semi-parametric within-subject mixture approach to the analyses of responses and response times. British Journal of Mathematical and Statistical Psychology, 71(2), 205–228.
22.
MuthénL. K.MuthénB. O. (2012). Mplus user’s guide. Muthén & Muthén.
23.
Organization for Economic Cooperation and Development. (2017). PISA 2015 technical report. Paris, France: OECD Publishing.
24.
ParkD. G.LautenschlagerG. J. (1990). Improving IRT item bias detection with iterative linking and ability scale purification. Applied Psychological Measurement, 14, 163–173.
25.
PattonJ. M.ChengY.HongM.DiaoQ. (2019). Detection and treatment of careless responses to improve item parameter estimation. Journal of Educational and Behavioral Statistics, 44, 309–341.
26.
PlummerM. (2003, March). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Proceedings of the 3rd international workshop on distributed statistical computing (Vol. 124, No. 125.10). https://www.r-project.org/nosvn/conferences/DSC-2003/Drafts/Plummer.pdf
27.
QianH.StaniewskaD.ReckaseM.WooA. (2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35(1), 38–47.
28.
R Development Core Team. (2019). R: A language and environment for statistical computing[Computer software Manual]. Vienna, Austria. http://www.Rproject.org(ISBN 3-900051-07-0).
29.
RangerJ.KuhnJ. T. (2017). Detecting unmotivated individuals with a new model-selection approach for Rasch models. Psychological Test and Assessment Modeling, 59(3), 269.
30.
RiosJ. A.GuoH.MaoL.LiuO. L. (2017). Evaluating the impact of careless responding on aggregated-scores: To filter unmotivated examinees or not?International Journal of Testing, 17(1), 74–104.
31.
SnijdersT. A. (2001). Asymptotic null distribution of person fit statistics with estimated person parameter. Psychometrika, 66, 331–342.
32.
SundreD. L.WiseS. L. (2003, April). Motivation filtering: An exploration of the impact of low examinee motivation on the psychometric quality of tests. Annual Meeting of the National Council on Measurement in Education, Chicago, IL, United States.
33.
van der LindenW. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204.
34.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308.
35.
van der LindenW. J.GuoF. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73, 365–384.
36.
WangC.XuG. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456–477.
37.
WangC.XuG.ShangZ. (2018). A two-stage approach to differentiating normal and aberrant behavior in computer based testing. Psychometrika, 83(1), 223–254.
38.
WangC.XuG.ShangZ.KuncelN. (2018). Detecting aberrant behavior and item preknowledge: A comparison of mixture modeling method and residual method. Journal of Educational and Behavioral Statistics, 43(4), 469–501.
39.
WangW. C.ShihC. L.YangC. C. (2009). The mimic method with scale purification for detecting differential item functioning. Educational and Psychological Measurement, 69(5), 713–731.
40.
WangW. C.SuY.-H. (2004). Effects of average signed area between two item characteristic curves and test purification procedures on the DIF detection via the Mantel-Haenszel method. Applied Measurement in Education, 17, 113–144.
41.
WiseS. L. (2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19(2), 95–114.
42.
WiseS. L. (2015). Effort analysis: Individual score validation of achievement test data. Applied Measurement in Education, 28(3), 237–252.
43.
WiseS. L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36(4), 52–61.
44.
WiseS. L.DeMarsC. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17.
45.
WiseS. L.DeMarsC. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19–38.
46.
WiseS. L.DeMarsC. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15(1), 27–41.
47.
WiseS. L.KongX. J. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183.
48.
YuanK.-H.FungW. K.ReiseS. (2004). Three Mahalanobis-distances and their role in assessing unidimensionality. British Journal of Mathematical and Statistical Psychology, 57, 151–165.
49.
YuanK.-H.ZhongX. (2008). Outliers, leverage observations and influential cases in factor analysis: Minimizing their effect using robust procedures. Sociological Methodology, 38, 329–368.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
