A multidimensional Bayesian item response model is proposed for modeling item position effects. The first dimension corresponds to the ability that is to be measured; the second dimension represents a factor that allows for individual differences in item position effects called persistence. This model allows for nonlinear item position effects on the item side as well as on the person side. Moreover, a flexible loading structure on the two dimensions is allowed. A fully Bayesian estimation procedure is proposed, and its performance is investigated by a simulation study. Further, the model is applied to empirical data collected in the Programme for International Student Assessment 2000 in the reading domain. The additional value of the model’s extended flexibility compared to more restrictive models is shown. The findings show that the linear hypothesis of change in performance during a test does not hold in general.
AbramowitzM.StegunI. A. (1972). Handbook of mathematical functions: With formulas, graphs, and mathematical tables (Vol. 55). Dover Publications.
2.
AlbertJ. H.ChibS. (1993). Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association, 88(422), 669–679.
3.
AlexandrowiczR.MatschingerH. (2008). Estimation of item location effects by means of the generalized logistic regression model: A simulation study and an application. Psychology Science Quarterly, 50(1), 64–74.
4.
BéguinA. A.GlasC. A. (2001). MCMC estimation and some model-fit analysis of multidimensional IRT models. Psychometrika, 66(4), 541–561.
5.
CarpenterB.GelmanA.HoffmanM. D.LeeD.GoodrichB.BetancourtM.BrubakerM. A.GuoJ.LiP.RiddellA. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76(1), 1–32.
6.
DebeerD.BuchholzJ.HartigJ.JanssenR. (2014). Student, school, and country differences in sustained test-taking effort in the 2009 PISA reading assessment. Journal of Educational and Behavioral Statistics, 39(6), 502–523.
7.
DebeerD.JanssenR. (2013). Modeling item-position effects within an IRT framework. Journal of Educational Measurement, 50(2), 164–185.
8.
De BoeckP.WilsonM. (2004). A framework for item response models. In De BoeckP.WilsonM. (Eds.), Explanatory item response models (pp. 3–41). Springer.
9.
FoxJ.-P. (2010). Bayesian item response modeling: Theory and applications. Springer.
10.
FoxJ.-P.VerhagenA. J. (2010). Random item effects modeling for cross-national survey data. In DavidovE.SchmidtP.BillietJ. (Eds.), Cross-cultural analysis: Methods and applications (pp. 467–488). Routledge Academic.
11.
FreyA.HartigJ.RuppA. A. (2009). An NCME instructional module on booklet designs in large-scale assessments of student achievement: Theory and practice. Educational Measurement: Issues and Practice, 28(3), 39–53.
12.
GelfandA. E.SmithA. F. (1990). Sampling-based approaches to calculating marginal densities. Journal of the American Statistical Association, 85(410), 398–409.
13.
GelmanA.CarlinJ. B.SternH. S.DunsonD. B.VehtariA.RubinD. B. (2013). Bayesian data analysis (3rd ed.). CRC Press.
14.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472.
15.
GoldsteinH. (2011). Multilevel statistical models (4th ed.). John Wiley.
16.
HartigJ.BuchholzJ. (2012). A multilevel item response model for item position effects and individual persistence. Psychological Test and Assessment Modeling, 54(4), 418–431.
17.
HoffP. D. (2009). A first course in Bayesian statistical methods. Springer.
18.
HohensinnC.KubingerK. D.ReifM.Holocher-ErtlS.KhorramdelL.FrebortM. (2008). Examining item-position effects in large-scale assessment using the linear logistic test model. Psychology Science Quarterly, 50(3), 391.
19.
JinK.-Y.WangW.-C. (2014). Item response theory models for performance decline during testing. Journal of Educational Measurement, 51(2), 178–200.
20.
KieferJ. (1975). Balanced block designs and generalized Youden designs, I. Construction (patchwork). The Annals of Statistics, 3(1), 109–118.
21.
KolenikovS. (2010). Resampling variance estimation for complex survey data. Stata Journal, 10(2), 165–199.
22.
KubingerK. D. (2008). On the revival of the Rasch model-based LLTM: From constructing tests using item generating rules to measuring item administration effects. Psychology Science Quarterly, 50(3), 311–327.
23.
LeL. T. (2009). Effects on item positions on their difficulty and discrimination: A study in PISA science data across test language and countries. In ShigemasuK.OkadaA.ImaizumiT.HoshinoT. (Eds.), New trends in psychometrics (pp. 207–214). Universal.
24.
LevyR.MislevyR. J. (2016). Bayesian psychometric modeling. CRC Press.
25.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Addison-Wesley.
26.
LumleyT.ScottA. (2015). AIC and BIC for modeling with complex survey data. Journal of Survey Statistics and Methodology, 3(1), 1–18.
27.
LunnD. J.ThomasA.BestN.SpiegelhalterD. (2000). WinBUGS-a Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing, 10(4), 325–337.
28.
MerkleE.FurrD.Rabe-HeskethS. (2018). Bayesian model assessment: Use of conditional vs marginal likelihoods. ArXiv preprint. https://arxiv.org/abs/1802.04452
29.
MollenkopfW. G. (1950). An experimental study of the effects on item analysis data of changing item placement and test-time limit. Psychometrika, 15, 291–315.
30.
MuthénB. (1979). A structural probit model with latent variables. Journal of the American Statistical Association, 74(368), 807–811.
31.
Organization for Economic Cooperation and Development. (2002). PISA 2000 Technical Report. OECD Publishing.
32.
PatzR. J.JunkerB. W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioural Statistics, 24(2), 146–178.
33.
PlummerM. (2017). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. Version 4.3.0. http://mcmc-jags.sourceforge.net
34.
Rabe-HeskethS.SkrondalA. (2006). Multilevel modelling of complex survey data. Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(4), 805–827.
35.
R Core Team. (2018). R: A language and environment for statistical computing [Computer software manual]. R Core Team. https://www.R-project.org/
36.
RenardD.MolenberghsG.GeysH. (2004). A pairwise likelihood approach to estimation in multilevel probit models. Computational Statistics & Data Analysis, 44(4), 649–667.
37.
RuppA. A.DeyD. K.ZumboB. D. (2004). To Bayes or not to Bayes, from whether to when: Applications of Bayesian methodology to modeling. Structural Equation Modeling, 11(3), 424–451.
38.
RustK. (2013). Sampling, weighting, and variance estimation in international large-scale assessments. In RutkowskiL.von DavierM.RutkowskiD. (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 117–153). CRC Press.
39.
San MartinE.GonzálezJ. (2010). Bayesian identifiability: Contributions to an inconclusive debate. Chilean Journal of Statistics, 1(2), 69–91.
40.
San MartinE.RolinJ.-M.CastroL. M. (2013). Identification of the 1PL model with guessing parameter: Parametric and semi-parametric results. Psychometrika, 78(2), 341–379.
41.
SavitskyT. D.TothD. (2016). Bayesian estimation under informative sampling. Electronic Journal of Statistics, 10(1), 1677–1708.
42.
SchweizerK.SchreinerM.GoldA. (2009). The confirmatory investigation of APM items with loadings as a function of the position and easiness of items: A two-dimensional model of APM. Psychology Science Quarterly, 51(1), 47–64.
43.
TrendtelM.RobitzschA. (2018). Modeling item position effects with a Bayesian item response model applied to PISA 2009–2015 data. Psychological Test and Assessment Modeling, 60(2), 241–263.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
