The test-takers' choice: an investigation of the effect of topic on language-test performance

Abstract

A fundamental issue in validating topic-based tests of language proficiency is the effect of the topic on the test takers’ performance. Topic-based test developers must ensure that test takers are neither advantaged nor disadvantaged in terms of their test results when presented with a given test topic. We have termed this threat a ‘topic effect’ and argue that this topic effect may constitute a source of construct-irrelevant variance (Messick, 1989). We contend that investigating the possibility of a topic effect is a critical step in establishing the validity of all topic-based tests. This research investigates the potential presence of a topic effect for the Canadian Academic English Language (CAEL) Assessment using the mechanism of choice. The principal research question is to determine if test-takers given a choice of topic perform significantly differently than test-takers not given a choice. ESL university applicants (n = 254) were randomly assigned to one of two conditions: no choice of topic or choice among five topics. Overall Proficiency Level, Reading, Lecture and Essay scores were compared for the two conditions. Ordinal level data were analysed using the Mann Whitney U, Chi-Square and Kolmogorov-Smirnov tests. While the scores for the choice groups were usually higher than the scores for the no-choice groups, the differences were not statistically significant. Despite the lack of significance, we felt that the scores warranted closer examination. For the topic where the difference between the choice and no-choice groups was largest, a textual analysis of the essays was undertaken to look for instances of the use of information not provided in the test. Again, no difference was found between the choice and no-choice groups. The results provide support for the validity of inferences drawn from this test. Because choice is an essential element of the research design, a second focus of the study is to explore the advantages and disadvantages of the use of choice in language testing settings from the perspective of both the tester and the test-taker. The potential value of choice as a testing feature is discussed and a call for further research is made.

Get full access to this article

View all access options for this article.

References

Afflerbach, P. P. 1990: The influence of prior knowledge on expert readers’ main idea construction strategies . Reading Research Quarterly 25, 31-46 .

Alderson, J. C. and Urquhart, A. H. 1988: This test is unfair: I’m not an economist. In Carrell, P. L. , Devine, J. and Eskey, D. E. , editors, Interactive approaches to second language reading. Cambridge: Cambridge University Press .

Baldwin, T. T. , Magjuka, R. J. and Loher, B. T. 1991: The perils of participation: effects of choice of training on trainee motivation and learning . Personnel Psychology 44, 51-65 .

Brigham, T. A. 1979: Some effects of choice on academic performance. In Perlmuter, L. C. and Monty, R. A. , editors, Choice and perceived control. Hillsdale, NJ: Lawrence Erlbaum .

Bugel, K. and Buunk, B. P. 1996: Sex differences in foreign language text comprehension: the role of interests and prior knowledge . Modern Language Journal, 80, 15-31 .

Clapham, C. 1995: The effect of language proficiency and background knowledge on EAP students reading comprehension . Paper given at LTRC, Long Beach, CA, March 1995.

Cordova, D. I. and Lepper, M. R. 1996: Intrinsic motivation and the process of learning: beneficial effects of contextualization, personalization, and choice . Journal of Educational Psychology 88, 715-730 .

Deci, E. L. and Ryan, L. M. 1985: Intrinsic motivation and self determination in human behavior. New York: Plenum .

Fitzpatrick, A. R. and Yen, W. M. 1995: The psychometric characteristics of choice items . Journal of Educational Measurement 32, 243-259 .

10.

Fox, J. , Pychyl, T. and Zumbo, B. 1997: An investigation of background knowledge in the assessment of language proficiency . In Huhta, A. , Kohonen, V. , Kurki-Suonio, L. and Luoma, S. , editors, Current developments and alternatives in language assessment: Proceedings of LTRC 1996, Jyvaskyla, Finland: University of Jyvaskyla Press.

11.

Gabrielson, S. , Gordon, B. and Engelhard, G. 1995: The effects of task choice on the quality of writing obtained in a statewide assessment . Applied Measurement in Education 8, 273-290 .

12.

Gee, T. W. 1987: Drafting and revising processes in grade 12 students’ examination writing . Alberta Journal of Educational Research 33, 96-114 .

13.

Gradman, H. L. and Hanania, E. 1991: Language learning background factors and ESL proficiency . Modern Language Journal 75, 39-51 .

14.

Hammadou, J. 1991: Interrelationships among prior knowledge, inference and language proficiency in foreign language reading . Modern Language Journal 75, 27-38 .

15.

Hamp-Lyons, L. and Mathias, S. P. 1994: Examining expert judgments of task difficulty on essay tests . Journal of Second Language Writing 3, 49-68 .

16.

Henk, W. P. and Holmes, B. C. 1988: Effects of content-related attitude on the comprehension and retention of expository text . Reading Psychology 9, 203-225 .

17.

Jennings, M. J. and Fox, J. 1998: Choice: the learners’ dream and testers’ nightmare? Paper presented at the Social Responsibility of the Language Tester Conference, Ottawa July 1998.

18.

Jensen, C. and Hansen, C. 1995: The effect of prior knowledge on EAP listening-test performance . Language Testing 12, 99-119 .

19.

Lukhele, R. , Thissen, D. and Wainer, H. 1994: On the relative value of multiple-choice, constructed response, and examine-selected items on two achievement tests . Journal of Educational Measurement 31, 234-250 .

20.

Malinowski, B. 1923: The problem of meaning in primitive languages. In Ogden, C. and Richards, I. , editors, The meaning of meaning. London: Keagan, Paul, Trench and Truber .

21.

Messick, S. 1989: Validity. In Linn, R. L. , editor, Educational measurement. New York: American Council on Education/Macmillan .

22.

Newman, S. E. 1979: Comments on section II: consequences of choice on learning and performance. In Perlmuter, L. C. and Monty, R. A. , editors, Choice and perceived control. Hillsdale, NJ: Lawrence Erlbaum .

23.

Norton, B. and Starfield, S. 1997: Covert language assessment in academic writing . Language Testing 14, 278-294 .

24.

Powers, D. E. , Fowles, M. E. , Farnum, M. and Gerritz, K. 1992: Giving a choice of topics on a test of basic writing skills: does it make any difference. The praxis series: professional assessments for beginning teachers, Educational Testing Service , Research Report 92-19.

25.

Savage, R. E. , Perlmuter, L. C. and Monty, R. A. 1979: Effect of reduction in the amount of choice and the perception of control on learning. In Perlmuter, L. C. and Monty, R. A. , editors, Choice and perceived control. Hillsdale, NJ: Lawrence Erlbaum .

26.

Siegler, R. S. and Lemaire, P. 1997: Older and younger adults’ strategy choices in multiplication: testing predictions of ASCM using the choice/no-choice method . Journal of Experimental Psychology: General 126, 71-92 .

27.

Stahl, S. , Jacobson, M. G. , Davis, C. E. and Davis, R. L. 1989: Prior knowledge and difficult vocabulary in the comprehension of unfamiliar text . Reading Research Quarterly 24, 27-43 .

28.

Takahashi, M. 1992: Memorial consequences of choosing nonwords: implication for interpretations of the self-choice effect . Japanese Psychological Research 34, 35-38 .

29.

Wainer, H. and Thissen, D. 1994: On examinee choice in educational testing . Review of Educational Research 64, 159-195 .

30.

Wainer, H. , Wang, X. and Thissen, D. 1994: How well can we compare scores on test forms that are constructed by examinees’ choice? Journal of Educational Measurement 31, 183-199 .

31.

Wang, X. , Wainer, H. and Thissen, D. 1995: On the viability of some untestable assumptions in equating exams that allow examine choice . Applied Measurement in Education 8, 211-225 .

32.

Zimmerman, D. W. and Zumbo, B. D. 1993: The relative power of parametric and nonparametric statistical methods. In Keren, G. and Lewis, C. , editors, A handbook for data analysis in the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum .

33.

Zumbo, B. D. and Hubley, A. M. 1998: A note on misconceptions concerning prospective and retrospective power . The Statistician: Journal of the Royal Statistical Society, Series D, 47, 385-388 .