With the advent of the digital revolution, language testers have endeavored to
utilize state-of-the-art computer technology to satisfy the ever-growing need for a
tool to measure English communication skills with maximal accuracy and efficiency.
Thanks to the concerted efforts made by experts in such fields as computational
linguistics, computer engineering, computer-assisted language learning, and
psychometrics, language testers have recently succeeded in developing
computer/web-based language tests. Among them are the TOEFL CBT by Educational
Testing Service and CommuniCAT by the University of Cambridge Local Examinations
Syndicate. As with the paper-based language test (PBLT), more rigorous research is
now being conducted on the validity of computer-based language tests (CBLT) and
computer adaptive language tests (CALT). Content analyses and comparability studies
of PBLT and CBLT/CALT are prerequisites to such validation research. In this
context, utilizing an EFL test battery entitled the Test of English Proficiency
developed by Seoul National University (TEPS), the present study is aimed at
addressing the issue of the comparability between PBLT and CBLT based on content and
construct validation employing content analyses based on corpus linguistic
techniques in addition to such statistical analyses as correlational analyses,
ANOVA, and confirmatory factor analyses. The findings support comparability between
the CBLT version and the PBLT version of the TEPS subtests (listening comprehension,
grammar, vocabulary, and reading comprehension) in question.
Get full access to this article
View all access options for this article.
References
1.
Alderson, C.1999: Reading constructs and reading assessment. In
Chalhoub-Deville, M.
, editor, Studies in language testing, Vol. 10. Issues in
computer-adaptive testing of reading.
Cambridge: University of Cambridge Press
, 49-70.
2.
Allen, M. J.
and
Yen, W. M.1979: Introduction to measurement theory.
Monterey, CA: Brooks/Cole
.
3.
APA (American Psychological Association)1986: Guidelines for computer-based texts and interpretations.
Washington, DC: Author
.
4.
Bachman, L. F.1990: Fundamental considerations in language testing.
Oxford: Oxford University Press
.
5.
Bachman, L. F.
,
Davidson, F.
,
Ryan, K.
and
Choi, I.-C.1995: Studies in language testing 1: an investigation into the
comparability of two tests of English as a Foreign Language.
Cambridge: Cambridge University Press
.
6.
Bachman, L. F.
and
Palmer, A.1981:
The construct validation of the FSI oral interview
. Language Learning31,
67-86
.
7.
Bachman, L. F.
and
Palmer, A.1996: Language testing in practice.
Oxford: Oxford University Press
.
8.
Bentler, P. M.
and
Wu, E. J. C.1995: EQS for Windows user’s guide.
Encino, CA: Multivariate Software, Inc.
9.
Bollen, A. K.1989: Structural equations with latent variables.
New York: A Wiley-International Publications
.
10.
Bollen, A. K.
and
Long, S. J.1987: Testing structural equation models.
Newbury Park, CA: Sage
.
11.
Boo, J. 1997: Computerized versus paper-and-pencil assessment of educational
development: score comparability and examinee preferences. Unpublished dissertation,
University of Iowa.
12.
Brown, J. D.1997:
Computers in language testing: present research and some future directions
. Language Learning & Technology1,
44-59
; available at http://llt.msu.edu/vol1num1/brown/default.html
(last accessed March 2003).
13.
Bugbee, A. C.1996:
The equivalence of paper-and-pencil and computer-based testing
. Journal of Research on Computing in Education28,
282-299
.
14.
Canale, Michael1983: On some dimensions of language proficiency. In
Oller, J. W. Jr.
, editor, Issues in language testing research.
Rowley, MA: Newbury House
, 333-342.
15.
Chalhoub-Deville, M.
, editor, 1999: Issues in computer-adaptive testing of
reading proficiency. In
Milanovic, M.
; series editor, Studies in Language Testing, Vol. 10.
Cambridge: University of Cambridge Press
, 6-16.
16.
Chapelle, C.1998: Construct definition and validity inquiry in SLA
research. In
Bachman, L. F.
and
Cohen, A. D.
, editors, Interfaces between second language acquisition and language
testing research.
New York: Cambridge University Press
, 32-70.
17.
Chapelle, C.1999:
Validity in language assessment
. Annual Review of Applied Linguistics19,
254-272
.
18.
Chapelle, C.2001: Computer applications in second language acquisition:
foundations for teaching, testing, and research.
Cambridge: Cambridge University Press
.
19.
Choi, I.-C.1991: Theoretical studies in second language acquisition:
application of item response theory to language testing.
New York: Peter Lang Publishing
.
20.
Choi, I.-C.1994:
Content and construct validation of a criterion-referenced English
proficiency test
. English Teaching48,
311-348
.
21.
Choi, I.-C.1997:
Essential test method facets of a general English proficiency test and
their validity as perceived by test-takers
. Language Research33,
773-799
.
22.
Choi, I.-C.1999:
Test fairness and validity of the TEPS
. Language Research35,
571-603
.
23.
Choi, I.-C.2000:
Language testing in the 21st century: prospects of computer adaptive
testing and performance testing
. Language Research36,
205-241
.
24.
Choi, I.-C.
and
Bachman, F.1992:
An investigation into the adequacy of three IRT models for data from two
EFL reading tests
. Language Testing9,
51-78
.
25.
Cummins, J. P.1983: Language proficiency and academic achievement. In
Oller, J. W. Jr.
, editor, Issues in language testing research.
Rowley, MA: Newbury House
, 108-126.
26.
Dunkel, P.1991: The effectiveness research on computer-assisted instruction
and computer assisted language learning. In
Dunkel, P.
, editor, Computer-assisted language learning and testing: research
issues and practice.
New York: Newbury House
, 5-36.
27.
Dunkel, P.1999:
Considerations in developing or using second/foreign language proficiency
computer-adaptive tests
. Language Learning & Technology2,
77-93
. http://llt.msu.edu/vol2num2/article4/index.html (last
accessed March 2003).
28.
Fulcher, G.1999:
Computerizing an English language placement test
. ELT Journal53,
289-299
.
29.
Green, B. F.1990: System design and operations. In
Wainer, H.
, editor, Computerized adaptive testing: a primer.
Hillsdale, NJ: Lawrence Erlbaum
, 23-40.
30.
Green, B. F.
,
Bock, R. D.
,
Humphreys, L. G.
,
Linn, R. L.
and
Reckase M. D.1984:
Technical guidelines for assessing computerized adaptive tests
. Journal of Educational Measurement21,
347-359
.
31.
Hair, J. F.
,
Anderson, R.
,
Tatham, R.
and
Black, W.1998: Multivariate data analysis.
Englewood Cliffs, NJ: Prentice Hall
.
32.
Hambleton, R. K.
,
Swaminathan, H.
and
Rogers, H. J.1991: Fundamentals of item response theory.
Newbury Park, CA: Sage Publications
.
33.
Henning, G.
and
Cascallar, E.1992: A preliminary study of the nature of communicative
competence. TOEFL Research Reports 36. Feb.
Princeton, NJ: Educational Testing Service
.
34.
Joreskog, K. G.
and
Sorbom, D.1997: Lisrel 7 user’s reference guide.
Chicago, IL: Scientific Software International Inc.
35.
Jurafsky, D.
and
Martin, J. H.2000: Speech and language processing: an introduction to natural
language processing, computational linguistics, and speech recognition.
Upper Saddle River, NJ: Prentice Hall
.
36.
Kenyon, D.
and
Malabonga, V.2001:
Comparing examinee attitudes toward computer-assisted and other oral
proficiency assessments
. Language Learning and Technology5,
60-83
. http://llt.msu.edu/vol5num2/kenyon/default.html (last
accessed March 2003).
37.
Kirsch, I.
,
Jamieson, J.
,
Taylor, C.
and
Eignor. D.1998: Computer familiarity among TOEFL examinees. TOEFL
Research Reports 59. March:
Princeton, NJ: Educational Testing Service
.
38.
Lancashire, I.
,
Bradley, J.
,
McCarty W.
,
Stairs, M.
and
Wooldridge T. R.1996: Using TACT with electronic texts.
New York: The Modern Language Association of America
.
39.
Larson, J.1999: Considerations for testing reading proficiency via
computer-adaptive testing. In
Chalhoub-Deville, M.
, editor, Studies in language testing, Vol. 10. Issues in
computer-adaptive testing of reading proficiency.
Cambridge: University of Cambridge Press
, 71-90.
40.
Marsh, H. W.
and
Redmayne, R. S.1994:
A multidimensional physical self-concept and its relations to multiple
components of physical fitness
. Journal of Sport and Exercise Psychology16,
43-55
.
41.
Mazzeo, J.
and
Harvey, A. L.1988: The equivalence of scores from automated and conventional
educational and psychological tests: a review of the literature. Report
No. CBR 87-8, ETS RR 88-21.
Princeton, NJ: Educational Testing Service
.
42.
Mead, A. D.
and
Drasgow, F.1993:
Equivalence of computerized and paper-and-pencil cognitive ability tests: a meta-analysis
. Psychological Bulletin114,
449-458
.
43.
Miller, G.1999: Cracking the TOEFL CBT.
New York: The Princeton Review
.
44.
Neuman, G.
and
Baydoun, R.1998:
Computerization of paper-and-pencil tests: when are they equivalent?Applied Psychological Measurement22,
71-83
.
45.
Oller, J. W. Jr.1995:
Review of content and construct validation of a criterion-referenced
English proficiency test by Choi (1994)
. English Teaching50,
161-168
.
46.
Russel, M.
and
Haney, W.1997:
Testing writing on computers: an experiment comparing student performance
on tests conducted via computer and via paper-and-pencil
. Educational Policy Analysis Archives5(3).
47.
Sands, W. A.
,
Waters, B. K.
and
McBride, J. R.
, editors, 1997: Computerized adaptive testing: from inquiry
to operation.
Washington, DC: American Psychological Association
.
48.
Sawaki, Y.2001:
Comparability of Conventional and Computerized Tests of Reading in a Second Language
. Language Learning & Technology5,
38-59
. http://llt.msu.edu/vol5num2/sawaki/default.html (last
accessed March 2003).
49.
Schaeffer, G. A.
,
Reese, C. M.
,
Steffen, M.
,
McKinley, R. L.
and
Mills, C. N.1993: Field test of computer-based GRE general test. Research
Report 93-07.
Princeton, NJ: Educational Testing Service
.
50.
Scott, M.
and
Oxford University Press1995: Wordsmith Tools. Ver. 2.0.
Oxford: Oxford University Press
.
51.
SPSS (Statistical Package for Social Science) 2000: SPSSPC User’s
Guide Ver. 9.0. Chicago: SPSS Inc.
52.
Steinberg, L.
,
Thissen, D.
and
Wainer, H.1990: Validity. In
Wainer, H.
, editor, Computerized adaptive testing: a primer.
Hillsdale, NJ: Lawrence Erlbaum
, 187-232.
53.
Stout, W. R.
,
Junker, N. B.
and
Chang, H. H.1991: Dimtest and Testsim.
Urbana, IL: University of Illinois at Urbana-Champaign
.
54.
Taylor C.
,
Jamieson, J.
,
Eignor, D.
and
Kirsch, I.1998: The relationship between computer familiarity and performance
on computer-based TOEFL test takers. TOEFL Research Reports 61. March.
Princeton, NJ: Educational Testing Service
.
55.
UCLES (University of Cambridge Local Examination Syndicate)1998: CommuniCAT. (Software.)
Cambridge: University of Cambridge Local
Examination Syndicate
.
56.
Van de Vijver, F. J. R.
and
Harsveld, M.1994:
The incomplete equivalence of the paper-and-pencil and computerized
versions of the General Aptitude Test Battery
. Journal of Applied Psychology79,
852-859
.
57.
Van der Linden, W. J.
and
Hambleton, R. K.1997: Handbook of modern item response theory.
New York, Springer
.
58.
Vispoel, W. P.
,
Bleiler, T.
,
Boo, J.
,
Steger-May, K.
,
Lin, C.
and
Turhan, A.
1997:
Computer versus paper-and-pencil assessment of self-concept: psychometric
equivalence and respondent preferences
. Paper presented at the annual meeting of the National Council on
Measurement in Education, Chicago, IL, March.
59.
Vispoel, W. P.
,
Boo, J.
and
Bleiler, T.2001:
Computerized and paper-and-pencil versions of the Rosenberg self-esteem
scale: a comparison of psychometric features and respondent preferences
. Educational Psychological Measurement61,
461-474
.
60.
Wainer, H.
,
Dorans, N. J.
,
Green, B. F.
,
Mislevy, R. J.
,
Steinberg, L.
and
Thissen, D.1990: Future challenges. In
Wainer, H.
,
Dorans, N. J.
,
Flaugher, R.
,
Green, B. F.
,
Mislevy, R. J.
,
Steinberg, L.
and
Thissen, D.
, editors, Computerized adaptive testing: a primer.
Hillsdale, NJ: Lawrence Erlbaum
, 233-272.
61.
Wainer, H.
,
Dorans, N. J.
,
Eignor, D.
,
Flaugher, R.
,
Green, B. F.
,
Mislevy, R.
,
Steinberg, L.
and
Thissen, D.
, editors, 2000: Computerized adaptive testing: a
primer. 2nd Edition.
Hillsdale, NJ: Lawrence Erlbaum
.
62.
Wang, T.
and
Kolen, M. J.2001:
Evaluating comparability in computerized adaptive testing: issues, criteria
and an example
. Journal of Educational Measurement38,
19-49
.
