The goals of this study were threefold: (1) to provide a comparative analysis of five different item analysis indices using both IRT and non-IRT indices; (2) to describe the characteristics of flagged items; and (3) to investigate the appropriateness of logistic regression as an item analysis technique for further studies. The performance of five item analysis indices was examined in this study; p-value, point-biserial correlation, Rasch b parameter, Rasch χ2 fit and χ2 from the logistic regression model. A factor analysis indicated that the first factor was a difficulty factor and included the Rasch b and p-value indices, whereas the second factor represented the discrimination power of an item and consisted of the Rasch χ2 fit, χ2 from the logistic regression and the point-biserial correlation. There was a tendency for difficult items to exhibit more unusual response patterns. The χ 2 from the logistic regression fit was found to be an appropriate index when a researcher is attempting to detect items that function differentially and when the subject pool consists of varying ability groups.
Get full access to this article
View all access options for this article.
References
1.
Agresti, A.1990: Categorical data analysisNew York : Wiley.
2.
Bock, D.R.1975: Multivariate statistical methods in behavioral researchNew York: McGraw-Hill.
3.
Crocker, L. and Algina, J.1986: Introduction to classical and modern test theoryNew York: Holt, Rinehart & Winston.
4.
Educational Testing Service1988: Test of English as a Foreign Language (Form 3KTF12). Princeton, NJ: Author.
5.
Henning, G.H.1987: A guide to language testing: development, evaluation, researchCambridge, MA: Newbury House.
6.
Hill, J.R.1989: Screening for potentially biased items in testing programs . Educational Measurement: Issues and Practice8, 5-11.
7.
Horn, J.L.1965: A rationale and test for the number of factors in factor analysis. Psychometrika30, 179-85.
8.
Perkins, K. and Brutten, S.R.1993: A comparison of indices for the identification of misfitting items. In Douglas, D. and Chappelle, C., editors, A new decade of language testing. Alexandria, VA: Teachers of English to Speakers of other Languages, Inc., 83-97.
9.
Perkins, K. and Miller, L.M.1984: Comparative analyses of English as a second language reading comprehension data: classical test theory and latent trait measurement. Language Testing1, 21-32.
10.
Reckase, M.D.1979: Unifactor latent trait models applied to multifactor tests: results and implications. Journal of Educational Statistics4, 207-30.
11.
SAS user's guide: statistics, version 5 edition1985: Cary, NC: SAS Institute.
12.
Shepard, L.A., Camilli, G. and Williams, D.M.1985: Validity of approximation techniques for detecting item bias. Journal of Educational Measurement22, 77-105.
