Accurate estimation of latent traits is critical in educational and psychological measurement, yet establishing validity evidence is challenging under small samples, ordinal data, and skewed distributions. We propose a Bayesian framework that incorporates expert-informed priors to enhance estimation and validity evidence for multi-unidimensional instruments. Simulation studies demonstrate that expert input improves estimation accuracy over ordinal confirmatory factor analysis, especially in samples as small as N = 25, with diminishing returns beyond six to nine experts. We compare variational inference (automatic differentiation variational inference) and Hamiltonian Monte Carlo in terms of estimation accuracy, computational efficiency, and posterior quality. A real-world application using TIMSS Grade 8 Mathematics data illustrates practical implications for expert selection and estimation strategies in small-sample instrument development.
EllisP.JablonskiE.LevyA.MansfieldA. (2009). High school science performance assessments: An examination of instruments for Massachusetts. Education Development Center.
9.
FloraD. B.CurranP. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466.
10.
FoxJ. P. (2010). Bayesian item response modeling: Theory and applications. Springer.
11.
GajewskiB. J.CofflandV.BoyleD. K.BottM.PriceL. R.LeopoldJ.DuntonN. (2011). Assessing content validity through correlation and relevance tools. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 8, 81–96.
12.
GajewskiB. J.PriceL. R.CofflandV.BoyleD. K.BottM. J. (2013). Integrated analysis of content and construct validity of psychometric instruments. Quality & Quantity, 47, 57–78.
13.
GarrardL.PriceL. R.BottM. J.GajewskiB. J. (2015). A novel method for expediting the development of patient-reported outcome measures and an evaluation of its performance via simulation. BMC Medical Research Methodology, 15, 1–14.
14.
GelmanA.CarlinJ. B.SternH. S.DunsonD. B.VehtariA.RubinD. B. (2014). Bayesian data analysis. Chapman and Hall/CRC.
15.
JiangY.BoyleD. K.BottM. J.WickJ. A.YuQ.GajewskiB. J. (2014). Expediting clinical and translational research via Bayesian instrument development. Applied Psychological Measurement, 38(4), 296–310.
16.
KolmogorovA. N. (1933). On the empirical determination of a distribution law. Giornale dell'Istituto Italiano degli Attuari, 4, 89–91.
17.
KucukelbirA.TranD.RanganathR.GelmanA.BleiD. M. (2017). Automatic differentiation variational inference. Journal of Machine Learning Research, 18(14), 1–45.
18.
LewandowskiD.KurowickaD.JoeH. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100(9), 1989–2001.
19.
LinacreJ. M. (1989). Many-faceted Rasch measurement [Doctoral dissertation, The University of Chicago].
20.
LinacreJ. M. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7, 328.
21.
LindquistM.PhilpotR.MullisI. V. S.CotterK. E. (2017). TIMSS 2019 mathematics framework. In MullisI. V. S.MartinM. O. (Eds.), TIMSS 2019 assessment frameworks (pp. 13–25). TIMSS & PIRLS International Study Center, Boston College.
22.
LiuQ.WangD. (2016). Stein variational gradient descent: A general purpose Bayesian inference algorithm. Advances in neural information processing systems 29.
23.
MislevyR. J. (1986). Bayes modal estimation in item response models. Psychometrika, 51, 177–195.
24.
MuthénB.du ToitS.H.C.SpisicD. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes [Unpublished technical report]. https://www.statmodel.com/download/Article_075.pdf
25.
NealR. M. (2011). MCMC using Hamiltonian dynamics. In BrooksS.GelmanA.JonesG. L.MengX.-L. (Eds.), Handbook of Markov Chain Monte Carlo (pp. 113–162). Chapman & Hall/CRC.
26.
PatelM. X.DokuV.TennakoonL. (2003). Challenges in recruitment of research participants. Advances in Psychiatric Treatment, 9(3), 229–238.
27.
PiantaR. C.BelskyJ.VandergriftN.HoutsR.MorrisonF. J. (2008). Classroom effects on children’s achievement trajectories in elementary school. American Educational Research Journal, 45(2), 365–397.
28.
RaczynskiK. R.CohenA. S.EngelhardG. Jr.LuZ. (2015). Comparing the effectiveness of self-paced and collaborative frame-of-reference training on rater accuracy in a large-scale writing assessment. Journal of Educational Measurement, 52(3), 301–318.
29.
R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org.
30.
ReckaseM. D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21(1), 25–36.
31.
ReckaseM. D. (2006). Multidimensional item response theory. In RaoC. R.SinharayS. (Eds.), Handbook of Statistics: Volume 26, Psychometrics (pp. 607–642). Elsevier.
32.
RigdonS. E.TsutakawaR. K. (1983). Parameter estimation in latent trait models. Psychometrika, 48(4), 567–574.
33.
RosseelY. (2012). Lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36.
34.
ShengY.WikleC. K. (2007). Comparing multiunidimensional and unidimensional item response theory models. Educational and Psychological Measurement, 67(6), 899–919.
35.
SmirnovN. (1948). Table for estimating the goodness of fit of empirical distributions. The Annals of Mathematical Statistics, 19(2), 279–281.
36.
Stan Development Team. (2023). Stan: A probabilistic programming language (version 2.32.0). https://mc-stan.org
37.
StoneM.YumotoF. (2004). The effect of sample size for estimating Rasch/IRT parameters with dichotomous items. Journal of Applied Measurement, 5(1), 48–61.
38.
SwaminathanH.GiffordJ. A. (1982). Bayesian estimation in the Rasch model. Journal of Educational Statistics, 7(3), 175–191.
39.
SwaminathanH.GiffordJ. A. (1985). Bayesian estimation in the two-parameter logistic model. Psychometrika, 50(3), 349–364.
van der LindenW. J.HambletonR. K. (Eds.). (2013). Handbook of modern item response theory. Springer.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
