The present paper introduces a random weights linear logistic test model for the measurement of individual differences in operation-specific practice effects within a single administration of a test. The proposed model is an extension of the linear logistic test model of learning developed by Spada (1977) in which the practice effects are considered random effects varying across examinees. A Bayesian framework was used for model estimation and evaluation. A simulation study was conducted to examine the behavior of the model in combination with the Bayesian procedures. The results demonstrated the good performance of the estimation and evaluation methods. Additionally, an empirical study was conducted to illustrate the applicability of the model to real data. The model was applied to a sample of responses from a logical ability test providing evidence of individual differences in operation-specific practice effects.
AndersenE. B. (1985). Estimating latent correlations between repeated testings. Psychometrika, 50(1), 3–16. https://doi.org/10.1007/bf02294143
2.
BockR. D. (1976). Basic issues in the measurement of change. In de GruijterD. N. M.van der KampL. J. T. (Eds.), Advances in psychological and educational measurement (pp. 75‒96). Wiley.
3.
BrooksS.GelmanA.JonesG. L.MengX.-L. (2011). Handbook of Markov chain Monte Carlo. Chapman and Hall/CRC.
4.
EmbretsonS. E. (1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 56(3), 495‒515. https://doi.org/10.1007/bf02294487
FischerG. H.FormannA. K. (1982). Some applications of logistic latent trait models with linear constraints on the parameters. Applied Psychological Measurement, 6(4), 397‒416. https://doi.org/10.1177/014662168200600403
7.
GelmanA.MengX.-L.SternH. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6(4), 733‒807.
8.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457‒472. https://doi.org/10.1214/ss/1177011136
9.
HoffmanM. D.GelmanA. (2014). The No-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15, 1593‒1623.
10.
LozanoJ. H.RevueltaJ. (2020). Investigating operation-specific learning effects in the Raven’s Advanced Progressive Matrices: A linear logistic test modeling approach. Intelligence, 82, 101468. https://doi.org/10.1016/j.intell.2020.101468
11.
LozanoJ. H.RevueltaJ. (2021a). Bayesian estimation and testing of a linear logistic test model for learning during the test. Applied Measurement in Education, 34(3), 223‒235. https://doi.org/10.1080/08957347.2021.1933982
12.
LozanoJ. H.RevueltaJ. (2021b). A Bayesian generalized explanatory item response model to account for learning during the test. Psychometrika, 86(4), 994‒1015. https://doi.org/10.1007/s11336-021-09786-x
13.
LozanoJ. H.RevueltaJ. (2023). A Bayesian general model to account for individual differences in operation-specific learning within a test. Educational and Psychological Measurement, 83(4), 782‒807. https://doi.org/10.1177/00131644221109796
14.
MengX.-L. (1994). Posterior predictive p-values. The Annals of Statistics, 22(3), 1142‒1160.
15.
R Development Core Team (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
16.
RaschG. (1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.
ScheiblechnerH. (1972). Das lernen und lösen komplexer denkaufgaben [The learning and solving of complex cognitive tasks]. Zeitschrift für Experimentelle und Angewandte Psychologie, 19, 476‒506.
SinharayS. (2005). Assessing fit of unidimensional item response theory models using a Bayesian approach. Journal of Educational Measurement, 42(4), 375‒394. https://doi.org/10.1111/j.1745-3984.2005.00021.x
21.
SpadaH. (1977). Logistic models of learning and thought. In SpadaH.KempfW. F. (Eds.), Structural models of thinking and learning (pp. 227‒262). Huber.
22.
SpadaH.McGawB. (1985). The assessment of learning effects with linear logistic test models. In EmbretsonS. (Ed.), Test design: New directions in psychology and psychometrics (pp. 169‒194). Academic Press.
23.
Stan Development Team. (2019). RStan: the R interface to Stan. R package. version 2.19.2. http://mc-stan.org
24.
VehtariA.GabryJ.MagnussonM.YaoY.BürknerP.PaananenT.GelmanA. (2020). loo: Efficient leave-one-out cross- validation and WAIC for Bayesian models. R package version 2.3.1. Available online at:https://mc-stan.org/loo/
25.
VehtariA.GelmanA.GabryJ. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. https://doi.org/10.1007/s11222-016-9696-4
26.
VergutsT.De BoeckP. (2000). A Rasch model for detecting learning while solving an intelligence test. Applied Psychological Measurement, 24(2), 151–162. https://doi.org/10.1177/01466210022031589
27.
VerhelstN. D.GlasC. A. W. (1993). A dynamic generalization of the Rasch model. Psychometrika, 58(3), 395–415. https://doi.org/10.1007/bf02294648
28.
WatanabeS. (2013). A widely applicable Bayesian information criterion. Journal of Machine Learning Research, 14, 867–897.
