If tests are to provide an accurate measure of learners’ language
abilities, examiners must minimize the influence of intervening factors such as text
organization and response format on test results. The purpose of this article is to
investigate the effects of these two factors on second language learners’
performance in reading comprehension tests. The study analyses the results of
reading comprehension tests which were delivered to 754 Japanese university
students. The main finding is that text organization and test format had a
significant impact on the students’ performance. When texts were clearly
structured, the more proficient students achieved better results in summary writing
and open-ended questions. By contrast the structure of the text made little
difference to the performance of the less proficient students. This suggests that
well-structured texts make it easier to differentiate between students with
different levels of proficiency. Examiners have hitherto taken little notice of the
impact of text structure and test format on students’ results. By paying
more attention to these factors, they will enhance the validity of their tests.
Get full access to this article
View all access options for this article.
References
1.
Alderson, J. C.1984: Reading in a foreign language: a reading problem or a language
problem? In
Alderson, J. C.
and
Urquhart, A. H.
, editors, 1-27.
2.
Alderson, J. C.1993a: Judgments in language testing. In
Douglas, D.
and
Chapelle, C.
, editors, A new decade of language testing research.
Alexandria, VA: TESOL
, 46-57.
3.
Alderson, J. C.1993b: Relationship between grammar and reading in an English for
academic purposes test battery. In
Douglas, D.
and
Chapelle, C.
, editors, A new decade of language testing research.
Alexandria, VA: TESOL
, 203-214.
4.
Alderson, J. C.
and
Lukmani, Y.1989:
Cognition and reading: cognitive levels as embodied in test questions
. Reading in a Foreign Language5,
253-270
.
5.
Alderson, J. C.
and
Urquhart, A. H.1983: The effect of student background discipline on
comprehension: a pilot study. In
Hughes, A.
and
Porter, D.
, editors, Current developments in language testing.
London: Academic Press
, 121-127.
6.
Alderson, J. C.
and
Urquhart, A. H.
editors, 1984: Reading in a foreign language.
London: Longman
.
7.
Alderson, J. C.
and
Urquhart, A. H.1985:
The effect of students’ academic discipline on their performance
on ESP reading tests
. Language Testing2,
192-204
.
8.
Alderson, J. C.
and
Urquhart, A. H.1988: This test is unfair: I’m not an economist. In
Carrellet al.
, editors, 168-182.
9.
Bachman, L. F.1990: Fundamental considerations in language testing.
Oxford: Oxford University Press
.
10.
Bachman, L. F.
and
Palmer, A. S.1996: Language testing in practice.
Oxford: Oxford University Press
.
11.
Bamberg, B.1984:
Assessing coherence: a reanalysis of essays written for the national
assessment of educational progress, 1969-1979
. Research in the Teaching of English18,
305-319
.
12.
Beck, I. L.
,
McKeown, M. G.
and
Gromoll, E. W.1989:
Learning from social studies texts
. Cognition and Instruction6,
99-158
.
13.
Beck, I. L.
,
McKeown, M. G.
,
Omanson, R. C.
and
Pople, M.1984:
Improving the comprehensibility of stories: the effects of revisions that
improve coherence
. Reading Research Quarterly19,
263-277
.
14.
Beck, I. L.
,
McKeown, M. G.
,
Sinatra, G. M.
and
Loxterman, J. A.1991:
Revising social studies text from a text-processing perspective: evidence
of improved comprehensibility
. Reading Research Quarterly26,
251-276
.
15.
Beck, I. L.
,
McKeown, M. G.
and
Worthy, J.1995:
Giving a text voice can improve students’ understanding
. Reading Research Quarterly30,
220-238
.
16.
Beck, I. L.
,
Omanson, R. C.
and
McKeown, M. G.1982:
An instructional redesign of reading lessons: effects on comprehension
. Reading Research Quarterly17,
462-481
.
17.
Bensoussan, M.
and
Kreindler, I.1990:
Improving advanced reading comprehension in a foreign language: summaries
vs. short-answer questions
. Journal of Research in Reading13,
55-68
.
18.
Bernhardt, E. B.1991: Reading development in a second language: theoretical,
empirical and classroom perspectives.
Norwood, NJ: Ablex
.
19.
Carrell, P. L.1984:
The effects of rhetorical organization on ESL readers
. TESOL Quarterly18,
441-469
.
20.
Carrell, P. L.
,
Devine, J.
and
Eskey, D.
, editors, 1988: Interactive approaches to second language
reading.
Cambridge: Cambridge University Press
.
21.
Carrell, P. L.
and
Eisterhold, J. C.1983:
Schema theory and ESL reading pedagogy
. TESOL Quarterly17,
553-573
.
22.
Clapham, C.1996: The development of IELTS: a study of the effect of background
knowledge on reading comprehension.
Cambridge: Cambridge University Press
.
23.
Clarke, M.1979:
Reading in Spanish and English: evidence from adult ESL students
. Language Learning29,
121-150
.
24.
Cohen, A. D.1993: The role of instructions in testing summarizing
ability. In
Douglas, D.
and
Chapelle, C.
, editors, A new decade of language testing research.
Alexandria, VA: TESOL
, 132-160.
25.
Connor, U.1987:
Research frontiers in writing analysis
. TESOL Quarterly21,
677-696
.
26.
Connor, U.
and
Farmer, M.1990: The teaching of topical structure analysis as a revision
strategy for ESL writers. In
Kroll, B.
, editor, Second language writing.
Cambridge: Cambridge University Press
, 126-139.
27.
Davison, A.
and
Kantor, R.1982:
On the failure of readability formulas to define readable texts: a case
study from adaptations
. Reading Research Quarterly17,
187-209
.
28.
Devine, J.1988: The relationship between general language competence and
second language proficiency: implications for teaching. In
Carrellet al.
, editors, 262-277.
29.
Dixon, R. A.
,
Hultsch, D. F.
,
Sinon, E. W.
and
van Eye, A.1984:
Verbal ability and text structure effects on adult age differences in text recall
. Journal of Verbal Learning and Verbal Behavior23,
569-578
.
30.
Duffy, T. M.
,
Higgins, L.
,
Mehlenbacher, B.
,
Cochran, C.
,
Wallace, D.
,
Hill, C.Haugen, D.
,
McCaffrey, M.
,
Burnett, R.
,
Sloan, S.
and
Smith, S.1989:
Models for the design of instructional text
. Reading Research Quarterly24,
434-457
.
31.
Duffy, T. M.
and
Kabance, P.1982:
Testing a readable writing approach to text revision
. Journal of Educational Psychology74,
733-748
.
32.
Goh, Soo Tian1990:
The effects of rhetorical organization in expository prose on ESL readers
in Singapore
. RELC Journal21,
1-13
.
33.
Golden, J.
,
Haslett, B.
and
Gauntt, H.1988:
Structure and content in eighth-graders’ summary essays
. Discourse Processes11,
139-162
.
34.
Grabe, W.1991:
Current developments in second language reading research
. TESOL Quarterly25,
375-406
.
35.
Graesser, A. C.
,
Hoffman, N. L.
and
Clark, L. F.1980:
Structural components of reading time
. Journal of Verbal Learning and Verbal Behavior19,
135-151
.
36.
Graves, M. F.
,
Prenn, M. C.
,
Earle, J.
,
Thompson, M.
,
Johnson, V.
and
Slater, W. H.1991:
Improving instructional texts: some lessons learned
. Reading Research Quarterly26,
110-132
.
37.
Hasan, R.1984: Coherence and cohesive harmony. In
Flood, J.
, editor, Reading Comprehension.
Newark, DE: International Reading Association
, 191-219.
38.
Hoey, M.1991: Patterns of lexis in text.
Oxford: Oxford University Press
.
39.
Johnson, P.1981:
Effects on reading comprehension of language complexity and cultural
background of a text
. TESOL Quarterly15,
169-181
.
40.
Johnson, P.1982:
Effects of reading comprehension of building background knowledge
. TESOL Quarterly16,
503-516
.
41.
Katz, S.
,
Lautenschlager, G.
,
Blackburn, A.
and
Harris, F.1990:
Answering reading comprehension items without passages on the SAT
. Psychological Science1,
122-127
.
42.
Kintsch, W.
and
Keenan, J.1973:
Reading rate as a function of number of propositions in the base structure
of sentences
. Cognitive Psychology6,
257-274
.
43.
Kintsch, W.
and
van Dijk, T. A.1978:
Toward a model of text comprehension and production
. Psychological Review85,
363-394
.
44.
Kintsch, W.
and
Yarbrough, J. C.1982:
Role of rhetorical structure in text comprehension
. Journal of Educational Psychology74,
828-834
.
45.
Klare, G. R.1985: How to write readable English.
London: Hutchinson
.
46.
Kobayashi, M. 1995: Effects of text organisation and test format on reading
comprehension test performance. Unpublished PhD Thesis, Thames Valley University, London.
47.
Lewkowicz, J.A. 1983: Method effect on testing reading comprehension: a
comparison of three methods. Unpublished MA Thesis, University of Lancaster.
48.
Mandler, J. M.1982:
Some uses and abuses of a story grammar
. Discourse Processes5,
305-318
.
49.
McGee, Lea M.1982:
Awareness of text structure: effects on children’s recall of
expository text
. Reading Research Quarterly17,
581-590
.
50.
Meyer, B. J. F.1975a: The organization of prose and its effects on memory.
Amsterdam: North-Holland
.
51.
Meyer, B. J. F.1975b:
Identification of the structure of prose and its implications for the study
of reading and memory
. Journal of Reading Behaviour7,
7-47
.
52.
Meyer, B. J. F.1985. Prose analysis: purpose, procedures, and problems: parts I
and II. In
Britton, B.
and
Black, J. B.
, editors, Understanding expository text.
Hillsdale, NJ: Lawrence Erlbaum
, 11-64, 269-304.
53.
Meyer, B. J. F.
,
Brandt, D. M.
and
Bluth, G. J.1980:
Use of top-level structure in text: key for reading comprehension of
ninth-grade students
. Reading Research Quarterly16,
72-103
.
54.
Meyer, B. J. F.
and
Freedle, R. O.1984:
Effects of discourse type on recall
. American Educational Research Journal21,
121-143
.
55.
Meyer, B. J. F.
,
Marsiske, M.
and
Willis, S. L.1993:
Text processing variables predict the readability of everyday documents
read by older adults
. Reading Research Quarterly28,
234-249
.
56.
Mohammed, M. A. H.
and
Swales, J. M.1984:
Factors affecting the successful reading of technical instructions
. Reading in a Foreign Language2,
206-217
.
57.
Nevo, N.1989:
Test-taking strategies on a multiple-choice test of reading comprehension
. Language Testing6,
199-215
.
58.
Olsen, L. A.
and
Johnson, R.1989:
A discourse-based approach to the assessment of readability
. Linguistics and Education1,
207-231
.
59.
Reder, L. M.
and
Anderson, J. R.1980:
A comparison of texts and their summaries: memorial consequences
. Journal of Verbal Learning and Verbal Behavior19,
121-134
.
60.
Richgels, D. J.
,
McGee, L. M.
,
Loman, R. G.
and
Sheard, C.1987:
Awareness of four text structures: effects on recall of expository text
. Reading Research Quarterly22,
177-196
.
61.
Ridgway, T.1997:
Thresholds of the background knowledge effect in foreign language reading
. Reading in a Foreign Language11,
151-168
.
62.
Royer, J.1990: The sentence verification technique: a new direction in the
assessment of reading comprehension. In
Legg, S.
and
Algina, J.
, editors, Cognitive assessment of language and math outcomes.
Norwood, NJ: Ablex
.
63.
Salager-Meyer, F.1991:
Reading expository prose at the post-secondary level: the influence of
textual variables on L2 reading comprehension (a genre-based approach)
. Reading in a Foreign Language8,
645-662
.
64.
Samson, D. M. M.1983: Rasch and reading. In
van Weeren, H.
, editor, Practice and problems in language testing.
Arnhem: CITO
.
65.
Sarig, G.1989:
Testing meaning construction: can we do it fairly?Language Testing6,
77-94
.
66.
Shohamy, E.1984:
Does the testing method make a difference? The case of reading comprehension
. Language Testing1,
147-170
.
67.
Shohamy, E.
and
Inbar, O.1991:
Validation of listening comprehension tests: the effect of text and
question type
. Language Testing8,
23-40
.
68.
Steffensen, M. S.
and
Joag-Dev, C.1984: Cultural knowledge and reading. In
Alderson, J. C.
and
Urquhart, A. H.
, editors, 48-61.
69.
Steffensen, M. S.
,
Joag-Dev, C.
and
Anderson, R. C.1979:
A cross-cultural perspective on reading comprehension
. Reading Research Quarterly15,
10-29
.
70.
Taylor, B. M.
and
Samuels, S. J.1983:
Children’s use of text structure in the recall of expository material
. American Educational Research Journal20,
517-528
.
71.
Trabasso, T.
,
Secco, T.
and
van den Broek, P. W.1984: Causal cohesion and story coherence. In
Mandl, H.et al.
, editors, Learning and understanding from text.
Hillsdale, NJ: Lawrence Erlbaum
, 83-111.
72.
Ulijn, J. M.
and
Strother, J. B.1990:
The effect of syntactic simplification on reading EST texts as L1 and L2
. Journal of Research in Reading13,
38-54
.
73.
Urquhart, A. H.1984: The effect of rhetorical ordering on readability. In
Alderson, J. C.
and
Urquhart, A. H.
, editors, 160-175.
74.
Weir, C. J.1993: Understanding and developing language tests.
London: Prentice Hall
.
75.
Zuck, L. V.
and
Zuck, J. G.1984: The main idea: specialist and nonspecialist
judgments. In
Pugh, A. K.
and
Ulijn, J. M.
, editors, Reading for professional purposes.
London: Heinemann
, 130-135.
