For one reason or another, various sources of information, namely, ancillary variables and correlational structure of the latent abilities, which are usually available in most testing situations, are ignored in ability estimation. A general model that incorporates these sources of information is proposed in this article. The model has a general formulation that allows incorporation of either source or both sources of information in scoring the examinees using various item response models and subsumes the traditional method of expected a posteriori as a special case. Results show that using the different sources of information singly or simultaneously provides better ability estimates (i.e., higher correlation with the true abilities and smaller posterior variance and mean squared error). The optimal condition occurs when several short tests measuring highly correlated abilities that also correlate highly with the covariates are used. Markov chain Monte Carlo parameter estimation algorithms corresponding to the different model formulations are also developed. Simulated and actual data are analyzed to establish the usefulness and feasibility of the proposed models. Several practical considerations in using these models are also discussed.
Adams, R.J., Wilson, M., & Wang, W. ( 1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
2.
Adams, R.J., Wilson, M., & Wu, M.L. ( 1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22, 47-76.
3.
Adams, R.J., & Wu, M.L. ( 2006). The mixed-coefficient multinomial logit model: A generalized form of the Rasch model. In M. von Davier & C. H. Carstensen (Eds.), Multivariate and mixture distribution Rasch models: Extensions and applications (pp. 57-76). New York: Springer-Verlag.
4.
Best, N.G., Cowles, M.K., & Vines, S.K. ( 1995). CODA: Convergence diagnosis and output analysis software for Gibbs sampling output (Version 0.30) [Computer software]. Cambridge, MA: MRC Biostatistics Unit.
5.
Bradlow, E.T., Wainer, H., & Wang, X. ( 1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153-168.
6.
Carlin, B.P., & Louis, T.A. ( 2000). Bayes and empirical Bayes methods for data analysis. London: Chapman and Hall.
7.
de la Torre, J., & Patz, R.J. ( 2005). Making the most of what we have: A practical application of MCMC in test scoring. Journal of Educational and Behavioral Statistics, 30, 295-311.
8.
Doornik, J.A. ( 2003). Object-oriented matrix programming using Ox (Version 3.1) [Computer software]. London: Timberlake Consultants.
9.
Draper, D. ( 1995). Inference and hierarchical modeling in the social sciences . Journal of Educational and Behavioral Statistics, 20, 115-147.
10.
Du, Z., Wainer, H., Bradlow, E.T., & Rogers, H.J. ( 2001, April). Modeling conditional item dependencies with a three-parameter logistic testlet model. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.
11.
Fox, J.P., & Glas, C.A.W. ( 2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika, 66, 221-288.
12.
Fox, J.P., Klein Entink, R.H., & van der Linden, W.J. (2007). Modeling of responses and response times with the package CIRT. Journal of Statistical Software, 20, 1-14.
13.
Gelman, A., Carlin, J.B., Stern, H.S., & Rubin, R.B. ( 1995). Bayesian data analysis. London: Chapman and Hall.
14.
Gelman, A., Carlin, J.B., Stern, H.S., & Rubin, R.B. ( 2003). Bayesian data analysis (2nd ed.). London: Chapman and Hall .
15.
Johnson, E.G., & Carlson, J. ( 1994). The NAEP 1992 technical report (Report No. 23-TR-20) . Washington, DC: National Center for Education Statistics.
16.
Kim, J.K., & Nicewander, W.A. (1993). Ability estimation for conventional tests. Psychometrika, 58, 587-599.
17.
Kreft, I., & de Leeuw, J. ( 1998). Introducing multilevel modeling. London: Sage.
18.
Li, Y.H., & Schafer, W.D. ( 2005). Trait parameter recovery using multidimensional computerized adaptive testing in reading and mathematics. Applied Psychological Measurement, 29, 3-25.
19.
Little, R.J.A., & Rubin, D.B. ( 1983). On jointly estimating parameters and missing data by maximizing the complete-data likelihood. American Statistician , 37, 218-220.
Mislevy, R.J. ( 1987). Exploiting auxiliary information about examinees in the estimation of item parameters. Applied Psychological Measurement , 11, 81-91.
22.
Mislevy, R.J., & Bock, R.D. ( 1989). A hierarchical item-response model for educational testing . In R. D. Bock (Eds.), Multilevel analysis of education data (pp. 57-74). San Diego, CA : Academic Press.
23.
Mislevy, R.J., Johnson, E.G., & Muraki, E. ( 1992). Scaling procedure in NAEP. Journal of Educational Statistics, 17, 131-154.
24.
Mislevy, R.J., & Sheehan, K. ( 1989). The role of collateral information about examinees in item parameter estimation. Psychometrika, 54, 661-679.
25.
Muraki, E. ( 1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
26.
Neter, J., Kutner, M.H., Nachtsheim, C.J., & Wasserman, W. ( 1996). Applied linear statistical models. Chicago: Erwin.
27.
Patz, R.J., & Junker, B.W. ( 1999). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24, 342-366.
28.
Raftery, A.E., & Lewis, S.M. ( 1992). One long run with diagnostics: Implementation strategies for Markov chain Monte Carlo. Statistical Science, 7, 493-497.
29.
Reckase, M. ( 1997). A linear logistic multidimensional model for dichotomous item response data. In W. van der Linden & R. Hambleton (Eds.), Handbook of modern item response theory (pp. 271-286). New York: Springer-Verlag .
30.
Roberts, J.S., Donoghue, J.R., & Laughlin, J.E. (2000). A general model for unfolding unidimensional polytomous responses using item response theory. Applied Psychological Measurement, 24, 3-32.
31.
Samejima, F. ( 1969). Estimation of latent ability using a response pattern of graded scores. Psychometric Monograph, 17.
Thissen, D., & Orlando, M. ( 2001). Item response theory for items scored in two categories . In D. Thissen & H. Wainer (Eds.), Test scoring (pp. 73-140). Mahwah, NJ: Lawrence Erlbaum.
34.
van der Linden, W.J. ( 2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287-308.
35.
Wainer, H., Vevea, J.L., Camacho, F., Reeve, B.B., III, Rosa, K., Nelson, L., et al. (2001). Augmented scores: ‘‘Borrowing strength’’ to compute score based on small numbers of items. In D. Thissen & H. Wainer (Eds.), Test scoring (pp. 343-388). Mahwah, NJ: Lawrence Erlbaum.
36.
Wang, W.-C., & Chen, P.-H. (2004). Implementation and measurement efficiency of multidimensional computerized adaptive testing. Applied Psychological Measurement, 28, 295-316. Wang, W.-C., Chen, P.-H., & Cheng, Y.-Y. (2004). Improving measurement precision of test batteries using multidimensional item response models. Psychological Methods, 9, 116-136.
37.
Weisberg, S. ( 1985) Applied linear regression. New York : Wiley.
