A response model that is able to detect guessing behaviors and produce unbiased estimates in low-stake conditions using timing information is proposed. The model is a special case of the grade of membership model in which responses are modeled as partial members of a class that is affected by motivation and a class that responds only according to the level of ability. Monte Carlo simulations were conducted to compare the proposed model with an approach that ignored guessing and an approach based on item filtering. In each simulated condition, the proposed model outperformed the other approaches by showing the lowest level of bias and the highest precision of item and persons estimates. Finally, the model was estimated using real life data from Programme for the International Assessment of Adult Competencies research (PIAAC). The results showed slight but expected corrections for the levels of proficiency in all countries.
AiroldiE. M.BleiD. M.EroshevaE. A.FienbergS. E. (2014). Introduction to mixed membership models and methods. In AiroldiE. M.BleiD.EroshevaE. A.FienbergS. E. (Eds.), Handbook of mixed membership models and their applications (pp. 3–14). Boca Raton, FL: Chapman & Hall/CRC.
2.
AsparouhovT.MuthénB. (2008). Multilevel mixture models. In HancockG. R.SamuelsenK. M. (Eds.), Advances in latent variable mixture models (pp. 27–51). Charlotte, NC: IAP.
3.
DaytonC. M.MacreadyG. B. (1988). Concomitant-variable latent-class models. Journal of the American Statistical Association, 83, 173–178.
4.
De BoeckP.WilsonM. (2004). A framework for item response models (pp. 3–41). New York, NY: Springer.
5.
DempsterA. P.LairdN. M.RubinD. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (methodological), 39, 1–38.
6.
EroshevaE. A. (2002). Grade of membership and latent structure models with application to disability survey data(Doctoral dissertation). Office of Population Research, Princeton University, Princeton, NJ.
7.
EroshevaE. A. (2005). Comparing latent structures of the Grade of Membership, Rasch, and latent class models. Psychometrika, 70, 619–628.
8.
GruhlJ.EroshevaE. A. (2013). A tale of two (types of) memberships: Comparing mixed and partial membership with a continuous data example. In AiroldiE. M.BleiD.EroshevaE. A.FienbergS. E. (Eds.), Handbook of mixed membership models and their applications (pp. 15–38). Chapman & Hall/CRC.
9.
MislevyR. J.VerhelstN. (1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika, 55, 195–215.
10.
MuthénL. K.MuthénB. O. (1998–2015). Mplus User's Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén.
11.
Organization for Economic Co-operation and Development. (2013). OECD skills outlook 2013: first results from the survey of adult skills. Paris: Author. Retrieved fromhttp://dx.doi.org/10.1787/9789264204256-en
12.
O’NeilH. F.SugrueB.BakerE. L.Jr (1995). Effects of motivational interventions on the National Assessment of Educational Progress mathematics performance. Educational Assessment, 3, 135–157.
13.
RostJ. (1991). A logistic mixture distribution model for polychotomous item responses. British Journal of Mathematical and Statistical Psychology, 44, 75–92.
14.
SchnipkeD. L.ScramsD. J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213–232.
15.
SmitA.KeldermanH.van der FlierH. (1999). Collateral information and mixed Rasch models. Methods of Psychological Research Online, 4, 19–32.
16.
SmitA.KeldermanH.van der FlierH. (2000). The mixed Birnbaum model: Estimation using collateral information. Methods of Psychological Research Online, 5, 31–43.
17.
SundreD. L.KitsantasA. (2004). An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance?Contemporary Educational Psychology, 29, 6–26.
18.
ThelkA. D.SundreD. L.HorstS. J.FinneyS. J. (2009). Motivation matters: Using the Student Opinion Scale to make valid inferences about student performance. The Journal of General Education, 58, 129–151.
19.
VermuntJ. K.MagidsonJ. (2005). Factor analysis with categorical indicators: A comparison between traditional and latent class approaches. In Van der ArkA.CroonM. A.SijtsmaK. (Eds.), New developments in categorical data analysis for the social and behavioral sciences (pp. 41–62). Mahwah, NJ: Lawrence Erlbaum.
20.
von DavierM.YamamotoK. (2007). Mixture-distribution and HYBRID Rasch models. In von DavierM.CarstensenC. H. (Eds.), Multivariate and mixture distribution Rasch models (pp. 99–115). New York, NY: Springer.
21.
WiseS. L.DeMarsC. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1–17.
22.
WiseS. L.KongX. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.
23.
WiseS. L.BholaD. S.YangS. T. (2006). Taking the time to improve the validity of low-stakes tests: The effort-monitoring CBT. Educational Measurement: Issues and Practice, 25, 21–30.
24.
WiseS. L.KingsburyG. G.HauserC.MaL. (2010, 5). An investigation of the relationship between time of testing and test-taking effort. Paper presented at the annual meeting of the National Council on Measurement in Education, Denver, CO.
25.
WolfL. F.SmithJ. K.BirnbaumM. E. (1995). Consequence of performance, test, motivation, and mentally taxing items. Applied Measurement in Education, 8, 341–351.
26.
YamamotoK. (1989). A Hybrid model of IRT and latent class models. ETS Research Report Series. Princeton, New Jersey: Educational Testing Service.
27.
YamamotoK.EversonH. (1997). Modeling the effects of test length and test time on parameter estimation using the HYBRID model. In RostJ.LangeheineR. (Eds.), Applications of latent trait and latent class models in the social sciences (pp. 89–98). New York, NY: Waxmann Verlag GmbH.
