In testing programs or survey data, it is common to observe careless or inattentive responding due to time constraints, low motivation, or other factors. Such behavior can significantly jeopardize the validity of the test and bias the parameter estimates of examinees. Therefore, detecting such response behavior is of utmost importance. One of the most commonly observed random behaviors is back random responding (BRR). However, existing detection methods for BRR in the framework of cognitive diagnostic assessment (CDA) have shown limited power. Change point analysis (CPA) is a well-established statistical method that can be applied to detect whether aberrant response behaviors exist in a sequence of response data. Although existing CPA methods are mostly used in the item response theory framework and have shown encouraging performance in detecting aberrant response behaviors, it is not yet clear whether and how they can be applied to CDA framework, and what their performance would be in that context. To address these issues, we modify and improve the conventional CPA statistics based on the Bayesian framework and apply them to CDA. We then evaluate and compare their performances of CPA statistics through simulation study. Our results show that the proposed CPA methods have encouraging performances in detecting BRR with higher power while generating a well-controlled Type-I error rate. Finally, we demonstrate the utility of all CPA statistics by applying them to two real datasets.
AbahousH.RonchailJ.SifeddineA.KennyL.BouchaouL. (2017). Trend and change point analyses of annual precipitation in the Souss-Massa Region in Morocco during 1932–2010. Theoretical and Applied Climatology, 134, 3–4.
2.
AllenD. E.McAleerM.PowellR. J.SinghA. K. (2018). Non-parametric multiple change point analysis of the global financial crisis. Annals of Financial Economics, 13(2), 1850008.
3.
AminikhanghahiS.CookD. J. (2017). A survey of methods for time series change point detection. Knowledge and Information Systems, 51, 339–367.
4.
AnW. P.ChengX. B.LiuY. (2020). Application of Flessis’ Kappa coefficient in Bayesian decision tree algorithm Computer Engineering and Applications. Computer Engineering and Applications, 56(7), 137–140.
5.
BerryD. T. R.WetterM. W.BaerR. A.LarsenL.ClarkC.MonroeK. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Methods, 4, 340–345.
6.
ChenH.ChenJ. (2016). Exploring reading comprehension skill relationships through the G-DINA model. Educational Psychology, 36(6), 1049–1064.
7.
ChenJ.de la TorreJ. (2013). A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 37(6), 419–437.
8.
ClarkM. E.GirondaR. J.YoungR. W. (2003). Detection of back random responding: Effectiveness of MMPI-2 and Personality Assessment Inventory validity indices. Psychological Methods, 15, 223–234.
9.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge. Macmillan.
10.
CuiY.LeightonJ. P. (2009). The Hierarchy Consistency Index: Evaluating person fit for cognitive diagnostic assessment. Journal of Educational Measurement, 46(4), 429–449.
11.
CuiY.LiJ. C. (2015). Evaluating person fit for cognitive diagnostic assessment. Applied Psychological Measurement, 39, 223–238.
12.
De la TorreJ. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.
13.
DrasgowF.LevineM. V.WilliamsE. A. (1985), Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86.
14.
GlasC. A. W.DagohoyA. V. T. (2007). A person fit test for IRT models for polytomous items. Psychometrika, 72, 159–180.
15.
HoggR. V.CraigA. T. (1959). Introduction to mathematical statistics. Macmillan.
16.
HsuC. L.JinK. Y.ChiuM. M. (2020). Cognitive diagnostic models for random guessing behaviors. Frontiers in Psychology, 11, 570365.
17.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.
18.
HuangJ. L.LiuM.BowlingN. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828–845.
19.
HuebnerA.WangC. (2011). A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 71, 407–419.
20.
JohnsonJ. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.
21.
KapteinM.van den HeuvelE. (2022). Bayesian statistics. In: Statistics for data scientists. Undergraduate topics in computer science. Springer, Cham.
22.
KillickR.EckleyI. A. (2014). Changepoint: An R package for changepoint analysis. Journal of Statistical Software, 58, 1–19.
23.
LehmannE. L.CasellaG. (1998). Theory of point estimation, 2nd ed. Springer texts in statistics. Springer.
24.
LevineM. V.RubinD. B. (1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4(4), 269–290.
25.
LinJ. (1991). Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1), 145–151.
26.
ListM. K.RobitzschA.LudtkeO.KollerO.NagyG. (2017). Performance decline in low-stakes educational assessments: Different mixture modeling approaches. Large-scale Assessments in Education, 5, 1–25.
27.
LiuY.DouglasJ. A.HensonR. A. (2009). Testing Person fit in cognitive diagnosis. Applied Psychological Measurement, 33(8), 579–598.
28.
LiuZ. C.WangS.ZhangS.QiuT. (2024). A mixture fluency model using responses and response times with cognitive diagnosis model framework. Behavior Research Methods. 56, 3396–3451.
29.
MaW.de la TorreJ. (2020). GDINA: An R package for cognitive diagnosisModeling. Journal of Statistical Software, 93(14), 1–26.
30.
MalekiS.BinghamC.ZhangY. (2016). Development and realization of changepoint analysis for the detection of emerging faults on industrial systems. IEEE Transactions on Industrial Informatics, 12(3), 1180–1187.
31.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
32.
MeijerR. R.SijtsmaK. (2001). Methodology review: Evaluating Person Fit. Applied Psychological Measurement, 25(2), 107–135.
33.
MoodA. M. (1950). Introduction to the theory of statistics. McGraw-Hill.
34.
PageE. S. (1954). Continuous inspection schemes. Biometrika, 41, 100–115.
35.
PattonJ. M.ChengY.HongM.DiaoQ. (2019). Detection and Treatment of Careless Responses to Improve Item Parameter Estimation. Journal of Educational and Behavioral Statistics, 44(3), 309–341.
36.
RaoC. R. (1973). Linear statistical inference and its applications (2nd ed.). John Wiley.
37.
RobbinsH. (1985). The empirical bayes approach to statistical decision problems. The Annals of Mathematical Statistics, 35(1), 1–20.
38.
RuppA. A. (2013). A systematic review of the methodology for person fit research in Item Response Theory: Lessons about generalizability of inferences from the design of simulation studies. Psychological Test and Assessment Modeling, 55(1), 3–8.
39.
ShaoC. (2016). Aberrant response detection using change-point analysis [Doctoral dissertation]. University of Notre Dame.
40.
ShaoC.ChengY. (2017, April). Detection of test speededness using change-point analysis with response time data [Paper presentation]. Annual Meeting of National Council for Measurement in Education, San Antonio, TX.
41.
ShaoC.ChengY. (2022). Application of change point analysis of response time data to detect test speededness. Educational and Psychological Measurement, 82(5), 1031–1062.
42.
ShaoC.LiJ.ChengY. (2015). Detection of test speededness using change-point analysis. Psychometrika, 81(4), 1118–1141.
43.
SinharayS. (2016). Person fit analysis in computerized adaptive testing using tests for a change point. Journal of Educational and Behavioral Statistics, 41, 521–549.
44.
SinharayS. (2017a). Detection of item preknowledge using likelihood ratio test and score test. Journal of Educational and Behavioral Statistics, 42, 46–68.
45.
SinharayS. (2017b). Some remarks on applications of tests for detecting a change point to psychometric problems. Psychometrika, 82, 1149–1161.
46.
SinharayS. (2017c). Which statistic should be used to detect item preknowledge when the set of compromised items is known?Applied Psychological Measurement, 41, 403–421.
47.
TorreJ. D.ChiuC. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81, 253–273.
48.
TuD.LiY.CaiY. (2022). A new perspective on detecting performance decline: A change-point analysis based on Jensen-Shannon divergence. Behavior Research Methods, 55, 963–980.
49.
UlitzschE.von DavierM.PohlS. (2020). A hierarchical latent response model for inferences about examinee engagement in terms of guessing and item-level non-response. British Journal of Mathematical and Statistical Psychology, 73, 83–112.
50.
van BarneveldC. (2007). The effect of test-taker motivation on test construction within an IRT framework. Applied Psychological Measurement, 31, 31–46.
51.
van der VaartAW. (1998). Asymptotic statistics. Cambridge University Press.
52.
WiseS. L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36(4), 52–61.
53.
WiseS. L.DeMarsC. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17.
54.
WiseS. L.KongX. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183.
55.
WollackJ. A.CohenA. S. (2004, April). A model for simulating speeded test data [Paper presentation]. Annual Meeting of the American Educational Research Association, San Diego, CA.
56.
WoodallW. H.MontgomeryD. C. (1999). Research issues and ideas in statistical process control. Journal of Quality Technology, 31(4), 376–386.
57.
WorsleyK. J. (1979). On the likelihood ratio test for a shift in location of normal populations. Journal of the American Statistical Association, 74, 365–367.
58.
WangC.XuG. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456–477.
59.
WangC.XuG.ShangZ. (2018). A two-stage approach to differentiating normal and aberrant behavior in computer based testing. Psychometrika, 83, 223–254.
60.
YamaguchiK.OkadaK. (2018). Comparison among cognitive diagnostic models for the TIMSS 2007 fourth grade mathematics assessment. PLOS ONE, 13(2), e0188691.
61.
YamamotoK.EversonH. (1997). Modeling the effects of test length and test time on parameter estimation using the HYBRID model. In RostJ.LangeheineR. (Eds.), Application of latent trait and latent class models in the social sciences (pp. 89–98).
62.
YuanK. H.ZhongX. (2008). Outliers, leverage observations, and influential cases in factor analysis: Using robust procedures to minimize their effect. Sociological Methodology, 38, 329–368.
63.
ZhangJ. (2014). A sequential procedure for detecting compromised items in the item pool of a CAT system. Applied Psychological Measurement, 38, 87–104.
64.
ZhouY. X. (2021). Imputation method of missing data in cognitive diagnosis evaluation [Master’s thesis, ZheJiang Normal University]. CNKI.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
