The relative importance of persons,items,subtests and languages to TOEFL test variance

Abstract

The purpose of this project was to explore the relative contributions to TOEFL score dependability (which is analogous to classical theory reliability) of various numbers of persons, items, subtests, languages and their various interactions. To these ends, three research questions were formulated: (1) What are the characteristics of the distributions, and how high are the classical theory reliability estimates for the whole test and its subtests? (2) For each of the 15 languages, what are the relative contributions to test variance of persons, items, subtests and their interactions? (3) Across all 15 languages, what are the relative contributions to test variance of persons, items, subtests and languages, as well as their various interactions?

The study sampled 15 000 test takers, 1000 each from 15 different language backgrounds, from the total of 24 500 participants in the TOEFL generic data set which itself was a sample from the May 1991 worldwide administration of the TOEFL. The test was administered under normal operational conditions and included all three subtests: (1) Listening Comprehension, (2) Structure and Written Expression, and (3) Vocabulary and Reading Comprehension.

The analyses included descriptive statistics, classical theory reliability estimates, and a series of generalizability studies conducted to isolate the variance components due to persons, items, subtests and languages, and their effects on the dependability of the test. Unlike previous research, the results here indicate that, when considered in concert with other important sources of variance (persons, items and subtests), language differences alone account for only a very small proportion of TOEFL test variance. These results should prove useful to test developers and researchers interested in the relative effects of such factors on test design.

Get full access to this article

View all access options for this article.

References

Anghoff, W. H. and Sharon, A. T. 1974: The evaluation of differences in test performance of two or more groups . Educational and Psychological Measurement 34(4), 807-816 .

Alderman, D. L. and Holland, P. W. 1981: Item performance across native language groups on the Test of English as a Foreign Language, TOEFL Research Report 9. Princeton, NJ: Educational Testing Service .

Bachman, L. F. , Lynch, B. K. and Mason, M. 1995: Investigating variability in tasks and rater judgments in a performance test of foreign language speaking . Language Testing, 12(2), 239-257 .

Bolus, R. E. , Hinofotis, F. B. and Bailey, K. M. 1982: An introduction to generalizability theory in second language research . Language Learning, 32(2), 245-258 .

Borg, I. and Shye, S. 1995: Facet theory: form and content. Newbury Park, CA: Sage .

Brennan, R. L. 1983: Elements of generalizability theory. Iowa City, IA: American College Testing Program .

Brennan, R. L. , Jarjoura, D. and Deaton, E. L. 1980: Some issues concerning the estimation and interpretation of variance components in generalizability theory (ACT Technical Bulletin No. 36). Iowa City, IA: American College Testing Program .

Brown, J. D. 1984: A norm-referenced engineering reading test. In Pugh, A. K. and Ulijn, J. M. , editors, Reading for professional purposes: studies and practices in native and foreign languages. London: Heine-mann .

Brown, J. D. 1990: Short-cut estimators of criterion-referenced test consistency . Language Testing 7, 77-97 .

10.

Brown, J. D. 1993: A comprehensive criterion-referenced testing project. In Douglas, D. and Chapelle, C. , editors, A new decade of language testing research. Washington DC: TESOL , 163-184.

11.

Brown, J. D. and Bailey, K. M. 1984: A categorical instrument for scoring second language writing skills . Language Learning 34 (4), 21-42 .

12.

Brown, J. D. and Ross, J. A. 1996: Decision dependability of item types, sections, tests, and the overall TOEFL test battery. In Milanovic, M. and Saville, N. , editors, Performance testing, cognition and assessment. Cambridge: Cambridge University Press , 231-265.

13.

Chen, Z. and Henning, G. 1985: Linguistic and cultural bias in language proficiency tests . Language Testing 2(2), 155-163 .

14.

Cronbach, L. J. , Gleser, G. C. , Nanda, H. and Rajaratnam, N. 1972: The dependability of behavioral measurements: theory of generalizability for scores and profiles. New York: Wiley .

15.

Cronbach, L. J. , Rajaratnam, N. and Gleser, G. C. 1963: Theory of generalizability: a liberalization of reliability theory . British Journal of Statistical Psychology 16, 137-163 .

16.

ETS 1995: TOEFL test and score manual. Princeton, NJ: Educational Testing Service .

17.

Farhady, H. 1982: Measures of language proficiency from the learner’s perspective . TESOL Quarterly 16(1), 43-59 .

18.

Kirk, R. E. 1968: Experimental design: procedures for the behavioral sciences. Belmont, CA: Brooks/Cole .

19.

Kunnan, A. J. 1990: DIF in native language and gender groups in an ESL placement test . TESOL Quarterly 24(4), 741-746 .

20.

Kunnan, A. J. 1992: An investigation of a criterion-referenced test using G-theory, and factor and cluster analysis . Language Testing 9(1), 30-49 .

21.

Madsen, H. S. and Larson, J. W. 1986: Computerized Rasch analysis of item bias in ESL tests. In Stansfield, C. W. , editor, Technology and language testing. Washington DC: TESOL , 47-67.

22.

Oltman, P. K. , Stricker, L. J. and Barrows, T. 1988: Native language, English proficiency, and the structure of the Test of English as a Foreign Language, TOEFL Research Report 27. Princeton, NJ: Educational Testing Service .

23.

Sasaki, M. 1991: A comparison of two methods for detecting differential item functioning in an ESL placement test . Language Testing 8(2), 95-111 .

24.

Searle, S. R. 1971: Linear models: New York: Wiley .

25.

Shavelson, R. J. and Webb, N. M. 1981: Generalizability theory: 1973- 1980 . British Journal of Mathematical and Statistical Psychology 34, 133-166 .

26.

Shavelson, R. J. and Webb, N. M. 1991: Generalizability theory: a primer. Newbury Park, CA: Sage .

27.

Stansfield, C. W. and Kenyon, D. M. 1992: Research of the comparability of the oral proficiency interview and the simulated oral proficiency interview . System 20, 347-364 .

28.

Suen, H. K. 1990: Principles of test theories. Hillsdale, NJ: Lawrence Erlbaum .

29.

Swinton, S. S. and Powers, D. E. 1980: Factor analysis of the Test of English as a Foreign Language for several language groups, TOEFL Research Report 6. Princeton, NJ: Educational Testing Service .

30.

Wilson, K. M. 1982: A comparative analysis of TOEFL examinee characteristics, 1977-1979, TOEFL Research Report 11. Princeton, NJ: Educational Testing Service .