This study applied generalizability theory to investigate the contributions of
persons, items, sections, and language backgrounds to the score dependability of the
Test of English for International Communication (TOEIC). I replicated and extended
Brown’s (1999) study of the Test of English as a Foreign Language (TOEFL),
using data from two language groups who took the same form of the TOEIC: 45 156
Japanese examinees and 46 067 Korean examinees. Both classical reliability analyses
and generalizability analyses found the TOEIC scores of the examinees in question to
be highly reliable/generalizable. The items within a section varied greatly in
difficulty. An examinee who scored relatively high in one section was likely to
score high in the other section. The findings regarding the effect of language
backgrounds on TOEIC score dependability were inconclusive, which suggests
directions for future research. TOEIC test developers might explore design
alternatives with fewer items per section.
Get full access to this article
View all access options for this article.
References
1.
American Educational Research Association
,
American Psychological Association
, and
National Council on Measurement in Education
(AERA, APA and NCME) 1999: Standards for educational and
psychological testing.
Washington, DC: American Educational Research Association
.
2.
Angoff, W. H.
and
Sharon, A. T.1974:
The evaluation of differences in test performance of two or more groups
. Educational and Psychological Measurement34,
807-816
.
3.
Bachman, L. F.1997: Generalizability theory. In
Clapham, C.
and
Corson, D.
, editors, Encyclopedia of language and education. Volume 7. Language
testing and assessment.
Boston, MA: Kluwer Academic
, 255-262.
4.
Bachman, L. F.
,
Lynch, B. K.
and
Mason, M.1995:
Investigating variability in tasks and rater judgements in a performance
test of foreign language speaking
. Language Testing12,
239-257
.
5.
Bolus, R. E.
,
Hinofotis, F. B.
and
Bailey, K. M.1982:
An introduction to generalizability theory in second language research
. Language Learning32,
245-258
.
6.
Brennan, R. L.1983: Elements of generalizability theory.
Iowa City, IA: American College Testing
.
7.
Brennan, R. L.2001: Generalizability theory.
New York: Springer
.
8.
Brown, J. D.1999:
The relative importance of persons, items, subtests and languages to TOEFL
test variance
. Language Testing16,
217-238
.
9.
Brown, J. D.
and
Bailey, K. M.1984:
A categorical instrument for scoring second language writing skills
. Language Learning34,
21-42
.
10.
Brown, J. D.
and
Ross, J. A.1996: Decision dependability of item types, sections, tests, and
the overall TOEFL test battery. In
Milanovic, M.
and
Saville, N.
, editors, Performance testing, congnition and assessment.
Cambridge: Cambridge University Press
, 231-265.
11.
Crick, G. E.
and
Brennan, R. L.1983: Manual for GENOVA: a generalized analysis of variance
system. American College Testing, Technical Bulletin No. 43.
Iowa City, IA: ACT
.
12.
Cronbach, L. J.
,
Gleser, G. C.
,
Nanda, H.
and
Rajaratnam, N.1972: The dependability of behavioral measurements: theory of
generalizability for scores and profiles.
New York: Wiley
.
13.
Douglas, D.1992: Review of the Test of English for International
Communication. In
Kramer, J. J.
and
Conoley, J. C.
, editors, The eleventh mental measurements yearbook.
Lincoln, NE: Buros Institute of Mental
Measurements of The University of Nebraska-Lincoln
, 950-952.
14.
Educational Testing Service
No date a: TOEIC User’s Guide.
Princeton, NJ: Educational Testing Service
. Available at:
http://www.ets.org/Media/Tests/Test_of_English_for_International_Communication/TOEIC_User_Gd.pdf
(March 2006).
15.
Educational Testing Service
No date b: TOEIC Technical Manual.
Princeton, NJ: Educational Testing Service
.
16.
Eggly, S.
,
Musial, J.
, and
Smulowitz, J.1999:
The relationship between English language proficiency and success as a
medical resident
. English for Specific Purposes18,
201-208
.
17.
Farhady, H.1982:
Measures of language proficiency from the learner’s perspective
. TESOL Quarterly16,
43-59
.
18.
Kane, M. T.1992:
An argument-based approach to validation
. Psychological Bulletin112,
527-535
.
19.
Kunnan, A. J.1990:
DIF in native language proficiency from the learner’s perspective
. TESOL Quarterly24,
741-746
.
20.
Kunnan, A. J.1992:
An investigation of a criterion-referenced test using G-theory, and factor
and cluster analysis
. Language Testing9,
30-49
.
21.
Kunnan, A. J.
editor, 1997: Fairness and validation in language
assessment.
Cambridge: Cambridge University Press
.
22.
Lee, Y. W.
,
Kantor, R.
and
Mollaun, P.
2002:
Score reliability as an essential prerequisite for validating new writing
and speaking tasks for TOEFL
. Paper presented at the annual meeting of Teachers of English to
the Speakers of Other Languages (TESOL), Salt Lake City, UT.
23.
Lynch, B. K.
and
McNamara, T. F.1998:
Using G-theory and many-facet Rasch measurement in the development of
performance assessments of the ESL speaking skills of immigrants
. Language Testing15,
158-180
.
24.
Messick, S.1989: Validity. In
Linn, R. L.
, editor, Educational measurement. 3rd ed.New York: Macmillan
, 13-103.
25.
Sasaki, M.1991:
A comparison of two methods for detecting differential item functioning in
an ESL placement test
. Language Testing8,
95-111
.
26.
Shavelson, R. J.
and
Webb, N.1991: Generalizability theory: a primer.
Newbury Park, CA: Sage
.
27.
Stansfield, C. W.
and
Kenyon, D. M.1992:
Research of the comparability of the oral proficiency interview and the
simulated oral proficiency interview
. System, 20,
347-364
.
28.
van Weeren, J.
and
Theunissen, T. J. J. M.1987:
Testing pronunciation: An application of generalizability theory
. Language Learning37,
109-122
.
29.
Wilson, K. M.1989: Enhancing the interpretations of a norm-referenced
second-language test through criterion-referencing: a research assessment of
experience in the TOEIC testing context. TOEIC Research Report No. 1.
Princeton, NJ: Educational Testing Service
.
30.
Woodford, P. E.1982: The Test of English for International Communication
(TOEIC). In
Brumfit, C.
, editor, English for international communication.
Oxford: Pergamon
, 61-71.
