This study assesses the performance of strategies for handling rapid guessing responses (RGs) within the context of item response theory observed-score equating. Four distinct approaches were evaluated: (1) ignoring RGs, (2) penalizing RGs as incorrect responses, (3) implementing list-wise deletion (LWD), and (4) treating RGs as missing data followed by imputation using logistic regression-based methodologies. These strategies were examined across a diverse array of testing scenarios. Results indicate that the performance of each strategy varied depending on the specific manipulated factors. Both ignoring and penalizing RGs were found to introduce substantial distortions in equating accuracy. LWD generally exhibited the lowest bias among the strategies evaluated but showed higher standard errors. Data imputation methods, particularly those employing lasso logistic regression and bootstrap techniques, demonstrated superior performance in minimizing equating errors compared to other approaches.
AşiretS.SünbülS. Ö. (2023). Effect of missing data on test equating methods under NEAT design. International Journal of Psychology and Educational Studies, 10(3), 702–713. https://doi.org/10.52380/ijpes.2023.10.3.1231
2.
BernaardsC. A.SijtsmaK. (2000). Influence of imputation and EM methods on factor analysis when item nonresponse in questionnaire data is nonignorable. Multivariate Behavioral Research, 35(3), 321–364.
3.
BoltD. M.CohenA. S.WollackJ. A. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39(4), 331–348. https://doi.org/10.1111/j.1745-3984.2002.tb01146.x
CurleyC.KrauseR. M.FeiockR.HawkinsC. V. (2019). Dealing with missing data: A comparative exploration of approaches using the integrated city sustainability database. Urban affairs review, 55(2), 591–615.
6.
DeMarsC. E.WiseS. L. (2010). Can differential rapid-guessing behavior lead to differential item functioning?International Journal of Testing, 10(3), 207–229. https://doi.org/10.1080/15305058.2010.496347
7.
DengJ.RiosJ. A. (2022). Investigating the effect of differential rapid guessing on population invariance in equating. Applied Psychological Measurement, 46(7), 589–604. https://doi.org/10.1177/01466216221108991
8.
DeriboT.GoldhammerF.KroehneU. (2022). Changes in the speed–ability relation through different treatments of rapid guessing. Educational and Psychological Measurement, 83(3), 473–494. https://doi.org/10.1177/00131644221109490
9.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates, Inc.
10.
EndersC. K. (2022). Applied missing data analysis. Guilford Press.
11.
GonzálezJ. (2014). SNSequate: Standard and nonstandard statistical models and methods for test equating. Journal of Statistical Software, 59(7), 1–30. https://doi.org/10.18637/jss.v059.i07
12.
HaebaraT. (1980). Equating logistic ability scales by a weighted least squares method. Japanese Psychological Research, 22(3), 144–149. https://doi.org/10.4992/psycholres1954.22.144
13.
HanK. T.KolenM. J.PohlmannJ. T. (1997). A comparison among IRT true- and observed-score equatings and traditional equipercentile equating. Applied Measurement in Education, 10(2), 105–121. https://doi.org/10.1207/s15324818ame1002_1
14.
HansonB. A.BéguinA. A. (2002). Obtaining a common scale for item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26(1), 3–24. https://doi.org/10.1177/0146621602026001001
KalkanÖ. K.KaraY.KelecioğluH. (2018). Evaluating performance of missing data imputation methods in IRT analyses. International Journal of Assessment Tools in Education, 5(3), 403–416.
KolenM. J.BrennanR. L. (2014). Test equating, scaling, and linking (3rd ed.). Springer Publishing.
20.
LeeK. J.SimpsonJ. A. (2014). Introduction to multiple imputation for dealing with missing data. Respirology, 19(2), 162–167. https://doi.org/10.1111/resp.12205
LiouM.ChengP. E. (1995). Asymptotic standard error of equipercentile equating. Journal of Educational and Behavioral Statistics, 20(3), 259–286. https://doi.org/10.2307/1165197
23.
LittleR. J. A.RubinD. B. (2019). Statistical analysis with missing data (3rd ed., Vol. 793). John Wiley & Sons.
24.
LiuO. L.RiosJ. A.BordenV. M. H. (2015). The effects of motivational instruction on college students’ performance on low-stakes assessments. Educational Assessment, 20(2), 79–94. https://doi.org/10.1080/10627197.2015.1028618
25.
LiuY.ChengY.LiuH. (2020). Identifying effortful individuals with mixture modeling response accuracy and response time simultaneously to improve item parameter estimation. Educational and Psychological Measurement, 80(4), 775–807. https://doi.org/10.1177/0013164419895068
26.
LordF. M.WingerskyM. S. (1984). Comparison of IRT true-score and equipercentile observed-score equatings. Applied Psychological Measurement, 8(4), 453–461. https://doi.org/10.1177/014662168400800409
27.
MeyersJ. L.MillerG. E.WayW. D. (2008). Item position and item difficulty change in an IRT-based common item equating design. Applied Measurement in Education, 22(1), 38–60. https://doi.org/10.1080/08957340802558342
28.
MislevyR. J.BeatonA. E.KaplanB.SheehanK. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29(2), 133–161. https://doi.org/10.1111/j.1745-3984.1992.tb00371.x
29.
MittelhaëuserM.BéguinA. A.SijtsmaK. (2015). Selecting a data collection design for linking in educational measurement: Taking differential motivation into account. In MillsapR. E.van der ArkL. A.BoltD. M.WoodsC.-M. (Eds.), New developments in quantitative psychology: Presentations from the 78th Annual Psychometric Society Meeting (pp. 181–193). Springer.
30.
OsborneJ. W.BlanchardM. R. (2011). Random responding from participants is a threat to the validity of social science research results. Frontiers in Psychology, 1, Article 220. https://doi.org/10.3389/fpsyg.2010.00220
ÖzdemirG.AtarB. (2022). Investigation of the missing data imputation methods on characteristic curve transformation methods used in test equating. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 13(2), 105–116. https://doi.org/10.21031/epod.1029044
33.
PattonJ. M.ChengY.HongM.DiaoQ. (2019). Detection and treatment of careless responses to improve item parameter estimation. Journal of Educational and Behavioral Statistics, 44(3), 309–341. https://doi.org/10.3102/1076998618825116
34.
PenkC.SchipolowskiS. (2015). Is it all about value? Bringing back the expectancy component to the assessment of test-taking motivation. Learning and Individual Differences, 42, 27–35. https://doi.org/10.1016/j.lindif.2015.08.002
35.
PohlS.GräfeL.RoseN. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74(3), 423–452. https://doi.org/10.1177/0013164413515977
36.
R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
37.
RaghunathanT. E.LepkowskiJ. M.Van HoewykJ.SolenbergerP. (2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27(1), 85–95.
38.
RiosJ. A. (2021). Is differential noneffortful responding associated with Type I error in measurement invariance testing?Educational and Psychological Measurement, 81(5), 957–979. https://doi.org/10.1177/0013164421990429
39.
RiosJ. A.DengJ. (2021). Quantifying the distorting effect of rapid guessing on estimates of coefficient alpha. Applied Psychological Measurement, 46(1), 40–52. https://doi.org/10.1177/01466216211051719
40.
RiosJ. A.SolandJ. (2021). Investigating the impact of noneffortful responses on individual-level scores: Can the effort-moderated IRT model serve as a solution?Applied Psychological Measurement, 45(6), 391–406. https://doi.org/10.1177/01466216211013896
41.
RiosJ. A.GuoH.MaoL.LiuO. L. (2016). Evaluating the impact of careless responding on aggregated-scores: To filter unmotivated examinees or not?International Journal of Testing, 17(1), 74–104. https://doi.org/10.1080/15305058.2016.1231193
42.
RizopoulosD. (2006). ltm: An R package for latent variable modeling and item response theory analyses. Journal of Statistical Software, 17(5), 1–25. https://doi.org/10.18637/jss.v017.i05
43.
RobertC. P.CasellaG. (2004). Monte Carlo statistical methods (2nd ed.). Springer.
44.
RubinD. B. (1987). Multiple imputation for nonresponse in surveys. John Wiley & Sons, Inc.
45.
SchnipkeD. L.ScramsD. J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34(3), 213–232. https://doi.org/10.1111/j.1745-3984.1997.tb00516.x
46.
SijtsmaK.Van der ArkL. A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38(4), 505–528. https://doi.org/10.1207/s15327906mbr3804_4
47.
SinharayS.HollandP. W. (2010). The missing data assumptions of the NEAT design and their implications for test equating. Psychometrika, 75(2), 309–327. https://doi.org/10.1007/s11336-010-9156-6
48.
SolandJ.KuhfeldM.RiosJ. A. (2021). Comparing different response time threshold setting methods to detect low effort on a large-scale assessment. Large-Scale Assessments in Education, 9(1), 9. https://doi.org/10.1186/s40536-021-00100-w
49.
StockingM. L.LordF. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7(2), 201–210. https://doi.org/10.1177/014662168300700208
50.
TibshiraniR. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1), 267–288.
51.
Van BarneveldC. (2007). The effect of examinee motivation on test construction within an IRT framework. Applied Psychological Measurement, 31(1), 31–46. https://doi.org/10.1177/0146621606286206
52.
Van BuurenS. (2007). Multiple imputation of discrete and continuous data by fully conditional specification. Statistical Methods in Medical Research, 16(3), 219–242. https://doi.org/10.1177/0962280206074463
53.
Van BuurenS.Groothuis-OudshoornK. (2011). Mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1–67. https://doi.org/10.18637/jss.v045.i03
54.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z
55.
van der LindenW. J.GuoF. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384. https://doi.org/10.1007/s11336-008-9068-5
56.
van der LindenW. J.HambletonR. K. (1997). Item response theory: Brief history, common models, and extensions. In van der LindenW. J.HambletonR. K. (Eds.), Handbook of modern item response theory (pp. 1–28). Springer.
57.
WangC.XuG.ShangZ. (2016). A two-stage approach to differentiating normal and aberrant behavior in computer-based testing. Psychometrika, 83(1), 223–254. https://doi.org/10.1007/s11336-016-9525-x
58.
WiseS. L. (2015). Effort analysis: Individual score validation of achievement test data. Applied Measurement in Education, 28(3), 237–252. https://doi.org/10.1080/08957347.2015.1042155
59.
WiseS. L.DeMarsC. E. (2009). A clarification of the effects of rapid guessing on coefficient alpha: A note on Attali’s “Reliability of speeded number-right multiple-choice tests.”Applied Psychological Measurement, 33(6), 488–490. https://doi.org/10.1177/0146621607304655
60.
WiseS. L.GaoL. (2017). A general approach to measuring test-taking effort on computer-based tests. Applied Measurement in Education, 30(4), 343–354. https://doi.org/10.1080/08957347.2017.1353992
61.
WiseS. L.KongX. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. https://doi.org/10.1207/s15324818ame1802_2
62.
WiseS. L.MaL. (2012, April). Setting response time thresholds for a CAT item pool: The normative threshold method. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Vancouver, Canada.
63.
WrightA. (2016). REDCap: A tool for the electronic capture of research data. Journal of Electronic Resources in Medical Libraries, 13(4), 197–201. https://doi.org/10.1080/15424065.2016.1259026
64.
WuT.KimS.-Y.WestineC. D. (2022). Evaluating the effects of missing data handling methods on scale linking accuracy. Educational and Psychological Measurement, 83(6), 1202–1228. https://doi.org/10.1177/00131644221140941
