The reliability of scores on four forms of the Test of English as a Foreign Language (TOEFL) was estimated using a hybrid IRT model. It was found that there was very little difference between their overall reliability when the testlet items were assumed to be independent and when their dependence was modeled. A larger difference in reliability was found when test sections were analyzed individually. Then we found as much as a 40% overestimate in reading comprehension testlets, with the longer testlets of the newest form of TOEFL showing the most local dependence. The listening comprehension testlets exhibited much less local dependence. We also found that the test was unidimensional enough for the use of univariate item response theory (IRT) to be efficacious, and that the reading comprehension testlets showed essentially no differential functioning by sex.
Get full access to this article
View all access options for this article.
References
1.
Anastasi, A. (1961). Psychological testing (2nd ed.). New York: Macmillan.
2.
American Psychological Association (APA) . (1966). Standards for educational and psychological tests and manuals. Washington, DC: Author.
3.
Bimbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 392-479). Reading, MA: Addison-Wesley.
4.
Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more latent categories. Psychometrika, 37, 29-51.
5.
Brown, W. , & Thomson, G. H. (1925). The essentials of mental measurement (3rd ed.). London: Cambridge University Press.
6.
Crumm, W. L. (1923). Note on the reliability of a test, with a special reference to the examinations set by the College Entrance Board. The American Mathematical Monthly, 30(6), 296-301.
7.
Green, B. F. , Bock, R. D., Humphreys, L. G., Linn, R. L., & Reckase, M. D. (1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 21, 347-360.
8.
Guilford, J. P. (1936). Psychometric methods (1st ed.). New York: McGraw-Hill.
9.
Holland, P. W. (1990). On the sampling theory foundations of item response theory models. Psychometrika, 55, 577-601.
10.
Holzinger, K. (1923). Note on the use of Spearman's prophecy formula for reliability. The Journal of Educational Psychology, 14, 302-305.
11.
Kelley, T. L. (1924). Note on the reliability of a test: A reply to Dr. Crumm's criticism. The Journal of Educational Psychology, 15, 193-204.
12.
Lord, F M. , & Novick, M. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
13.
Sireci, S. G. , Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28, 237-247.
14.
Thissen, D. , Steinberg, L., & Mooney, J. (1989). Trace lines for testlets: A use of multiple-categorical response models. Journal of Educational Measurement, 26, 247-260.
15.
Thissen, D. , Steinberg, L., & Wainer, H. (1993). Detection of differential item functioning using the parameters of item response models. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 67-113). Hillsdale, NJ: Lawrence Erlbaum.
16.
Thissen, D. , Wainer, H., & Wang, X-b. (1994). Are tests comprising both multiple-choice and free-response items necessarily less unidimensional than multiple-choice tests? An analysis of two tests. Journal of Educational Measurement, 31, 113-123.
17.
Thorndike, R. L. (1951). Reliability. In E. F. Lindquist (Ed.), Educational measurement. Washington, DC: American Council on Education.
18.
Traub, R. E. , & Rowley, G. L. (1991). Understanding reliability. Educational Measurement: Issues and Practice, 10, 37-45.
19.
Wainer, H. (1993). Model-based standardized measurement of an item's differential impact. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 123-135). Hillsdale, NJ: Lawrence Erlbaum.
20.
Wainer, H. (1995). Precision and differential item functioning on a testlet-based test: The 1991 Law School Admissions Test as an example. Applied Measurement in Education, 8(2), 157-187.
21.
Wainer, H. , & Kiely, G. (1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24, 185-202.
22.
Wainer, H. , & Lewis, C. (1990). Toward a psychometrics for testlets. Journal of Educational Measurement, 27, 1-14.
23.
Wainer, H. , & Lukhele, R. (1995). How long should a testlet be? One path toward an answer. Unpublished ETS technical report, Princeton, NJ.
24.
Wainer, H. , Sireci, S. G., & Thissen, D. (1991). Differential testlet functioning: Definitions and detection. Journal of Educational Measurement, 28, 197-219.
25.
Yen, W. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30, 187-213.
