A Test-Length Correction to the Estimation of Extreme Proficiency Levels

Abstract

In this study, the estimation of extremely large or extremely small proficiency levels, given the item parameters of a logistic item response model, is investigated. On one hand, the estimation of proficiency levels by maximum likelihood (ML), despite being asymptotically unbiased, may yield infinite estimates. On the other hand, with an appropriate prior distribution, the Bayesian approach of maximum a posteriori (MAP) yields finite estimates, but it suffers from severe estimation bias at the extremes of the proficiency scale. As a first step, a simple correction to the MAP estimator is proposed to reduce this estimation bias. The correction factor is determined through a simulation study and depends only on the length of the test. In a second step, some additional simulations emphasize that the corrected estimator behaves like the ML estimator and outperforms the standard MAP method for extremely small or extremely large abilities. Although based on the Rasch model, the method could be adapted to other logistic item response models.

Keywords

item response theory Rasch model proficiency level estimation bias

Get full access to this article

View all access options for this article.

References

Andersen, E.B. ( 1970). Asymptotic properties of conditional maximum likelihood estimators. Journal oƒ the Royal Statistical Society (Series B), 32, 283-301.

Andersen, E.B. ( 1972). The numerical solution of a set of conditional estimation equations. Journal of the Royal Statistical Society (Series B), 34, 42-54.

Baker, F.B. ( 1992). Item response theory: Parameter estimation techniques . New York, NY: Marcel Dekker.

Birnbaum, A. ( 1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA : Addison-Wesley.

Bock, R.D. , & Aitkin, M. ( 1981). Marginal maximum likelihood estimation of item parameters . An application of the EM algorithm. Psychometrika, 37, 29-51.

Bock, R.D. , & Mislevy, R.J. ( 1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.

Carlin, B.P. , & Louis, T.A. ( 1996). Bayes and empirical Bayes methods ƒor data analysis . London, England: Chapman & Hall.

Chang, H.H. , & Ying, Z. ( 2008). To weight or not to weight? Balancing influence of initial items in adaptive testing. Psychometrika, 73, 441-450.

Cramer, H. ( 1946). Mathematical models ƒor statistics. Princeton, NJ: Princeton University Press.

10.

Firth, D. ( 1993). Bias reduction of maximum likelihood estimates [Correction: 82, 667]. Biometrika, 80, 27-38.

11.

Gelman, A. , Carlin, J.B. , Stern, H.S. , & Rubin, D.B. ( 2004). Bayesian data analysis (2nd ed.). New York, NY: Chapman & Hall.

12.

Hambleton, R.K. , & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer .

13.

Lord, F.M. ( 1980). Applications oƒ item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.

14.

Lord, F.M. ( 1983). Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability. Psychometrika, 48, 233-245.

15.

Lord, F.M. ( 1986). Maximum likelihood and Bayesian parameter estimation in item response theory. Journal oƒ Educational Measurement , 23, 157-162.

16.

Lord, F.M. , & Novick, M.R. ( 1968). Statistical theories oƒ mental test scores. Reading, MA: Addison-Wesley.

17.

Magis, D. , & Raîche, G. (2010). An iterative maximum a posteriori estimation of proficiency level to detect multiple local likelihood maxima. Applied Psychological Measurement, 34, 75-90.

18.

Mislevy, R.J. ( 1984). Estimating latent distributions. Psychometrika , 49, 359-381.

19.

R Development Core Team. (2008). R: A language and environment ƒor statistical computing. Vienna, Austria: R Foundation for Statistical Computing .

20.

Rasch, G. ( 1960). Probabilistic models ƒor some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research.

21.

Rizopoulos, D. ( 2006). ltm: An R package for latent variable modeling and item response theory analyses. Journal oƒ Statistical Soƒtware , 17, 1-25.

22.

Rulison, K.L. , & Loken, E. ( 2009). I’ve fallen and I can’t get up: Can high-ability students recover from early mistakes in CAT? Applied Psychological Measurement, 33, 83-101.

23.

Samejima, F. ( 1973). A comment on Birnbaum’s three-parameter logistic model in the latent trait theory. Psychometrika, 38, 221-223.

24.

Samejima, F. ( 1977). Weakly parallel tests in latent trait theory with some criticisms of classical test theory. Psychometrika, 42, 193-198.

25.

Samejima, F. ( 1993). An approximation for the bias function of the maximum likelihood estimate of a latent variable for the general case where the item responses are discrete. Psychometrika, 58, 119-138.

26.

Samejima, F. ( 1998, April). Expansion oƒ Warm’s weighted likelihood estimator oƒ ability ƒor the three-parameter logistic model to general discrete responses. Paper presented at the annual meeting of the national Council on Measurement in Education, San Diego, CA.

27.

Swaminathan, H. , & Gifford, J.A. ( 1985). Bayesian estimation in the two-parameter logistic model . Psychometrika, 50, 349-364.

28.

Swaminathan, H. , & Gifford, J.A. ( 1986). Bayesian estimation in the three-parameter logistic model . Psychometrika, 51, 589-601.

29.

Tukey, J. ( 1977). Exploratory data analysis. Reading, MA: Addison-Wesley.

30.

Wang, T. , Hanson, B.A. , & Lau, C.-M.A. (1999). Reducing bias in CT trait estimation: A comparison of approaches. Applied Psychological Measurement, 23, 263-278.

31.

Warm, T.A. ( 1989). Weighted likelihood estimation of in item response theory . Psychometrika, 54, 427-450.