The Assessment of Language Competence (ALC) certificates is an annual, international testing program developed by the Australian Council for Educational Research to test the listening and reading comprehension skills of lower to middle year levels of secondary school. The tests are developed for three levels in French, German, Italian and Japanese, and at two levels in Chinese and Indonesian. There is a mixture of target language and English questions in the Level 2 and 3 tests. Some teachers have raised this as a concern in the belief that all questions should only be offered in English for the sake of fairness. Their view is that the tests are unduly difficult when they are designed with questions in the language. Arising from this concern, the aim of the research to be reported in this paper was to investigate the effects of the language of the question on student performance. We drew on data from a trial test and a final listening test, a questionnaire administered to students to gauge their perceptions of the tests and an examination of public documentation about the ALC. For the statistical analysis, we used Item Response Theory for calibrating items and for comparing item difficulty estimates, and fit statistics to verify how well items with English and target language question formats worked together. We found that where the questions involved listening for simple, explicitly stated information, students found the items in the target language relatively easier. In questions that required students to listen for global meaning, language choice either did not matter or tended to favour items in English rather than the target language. Furthermore, each of the six tests with some items in English and others in the target language showed a high level of reliability and fit to the single latent scale, indicating that items were functioning consistently regardless of the language of the test question.
AdamsR. J.KhooS. T. (1993). Quest: The interactive test analysis system. Hawthorn, Vic.: Australian Council for Educational Research.
2.
AldersonJ. C.ClaphamC. M.WallD. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press.
3.
Australian Council for Educational Research. (2011). 2010 Annual report: Assessment of language competence. Camberwell, Victoria.
4.
BachmanL. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
5.
BachmanL. F.PalmerA. S. (1996). Language testing in practice. Oxford: Oxford University Press.
6.
BachmanL. F.PalmerA. S. (2010). Language assessment in practice. Oxford: Oxford University Press.
7.
BarlowL. (2009). The effect of nonnative speaker accent on EFL students’ listening comprehension. PhD dissertation, University of Exeter, UK.
8.
BerneJ. E. (1992). The effects of test type, assessment task, and target language experience on foreign language learner’s performance on listening comprehension tests. PhD dissertation, University of Illinois at Urbana.
9.
BerneJ. E. (2004). Listening comprehension strategies: A review of the literature. Foreign Language Annals, 37(4), 521–534.
10.
BlauE. K. (1990). The effect of syntax, speed, and pauses on listening comprehension. TESOL Quarterly, 24(4), 746–753.
11.
BondT. G.FoxC. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd edition). Mahwah, NJ: Lawrence Erlbaum.
12.
BonkW. J. (2000). Second language lexical knowledge and listening comprehension. International Journal of Listening Comprehension, 14, 14–31.
13.
BrindleyG.SlatyerH. (2002). Exploring task difficulty in ESL listening assessment. Language Testing, 19(4), 369–394.
14.
BrookfieldS. D. (1995). Becoming a critically reflective teacher. San Francisco: Jossey-Bass.
15.
BrownA.ElderC.LumleyT.McNamaraT.McQueenJ. (1992). Mapping abilities and skill levels using Rasch techniques. Melbourne Papers in Language Testing,1(1), 37–69.
16.
BuckG. (2001). Assessing listening. Cambridge: Cambridge University Press.
17.
ChaudronC. (1983). Simplification of input: Topic reinstatements and their effects on L2 learners’ recognition and recall. TESOL Quarterly, 17(3), 437–458.
18.
ChiangC. S.DunkelP. (1992). The effect of speed modification, prior knowledge and listening proficiency on EFL lecture learning. TESOL Quarterly, 26(2), 345–74.
19.
Council of Europe (2001). The Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press.
20.
Department of Education and Early Childhood Development (2010). Languages other than English in government schools 2009. Victoria. Retrieved July 5, 2011from the Department of Education and Early Childhood Development website: www.eduweb.vic.gov.au/edulibrary/public/LOTE/lotereport2009.pdf
21.
Department of Education, Science and Training (2002). Review of the Commonwealth LOTE Programme by Erebus Consulting Partners. Department of Education, Science and Training. Canberra.
22.
ElderC. (1996). The effect of language background on ‘foreign’ language test performance: The case of Chinese, Italian and Modern Greek. Language Learning, 46(2), 233–282.
23.
FerneT.RuppA. A. (2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges, and recommendations. Language Assessment Quarterly, 4(2), 113–148.
24.
FieldJ. (2008). Listening in the language classroom. Cambridge: Cambridge University Press.
25.
GrahamS.SantosD.VanderplankR. (2010). Strategy clusters and sources of knowledge in French L2 listening comprehension. Innovation in Language Learning and Teaching, 4(1), 1–20.
26.
GriffithsR. (1992). Speech rate and listening comprehension: Further evidence of the relationship. TESOL Quarterly, 26(2), 385–389.
27.
Haththotuwa GamageG. (2003). Perceptions of kanji learning strategies: Do they differ among Chinese character and alphabetic background learners?Australian Review of Applied Linguistics, 26(2), 17–31.
28.
HughesA. (2003). Testing for language teachers, (2nd edition). Cambridge: Cambridge University Press.
29.
International English Language Testing System, University of Cambridge ESOL Examinations.
30.
JensenC.HansenC.GreenS.AkeyT. (1997). An investigation of item difficulty incorporating the structure of listening tests: A hierarchical linear modeling analysis. In HuhtaA.KohonenV.Kurki-SuonioL.LuomaS. (Eds.), Current developments and alternatives in language assessment (pp. 151–164). Jyvaskyla: University of Jyvaskyla.
31.
LeL.FilipiA. (2008). How do questions in the target language affect student performance in foreign language listening tests?Paper presented at the 6th conference of International Test Commission, Liverpool, UK, 14–16 July 2008.
32.
LiaoY. (2009). A construct validation study of the GEPT reading and listening sections: Re-examining the models of L2 reading and listening abilities and their relations to lexico-grammatical knowledge. EdD dissertation, Teachers College, Columbia University, United States New York.
33.
LittlewoodW.YuB. (2011). First Language and target language in the foreign language classroom. Language Teaching, 44(1), 64–77.
34.
MajorR. C.FitzmauriceS. F.BuntaF.BalasubramanianC. (2002). The effects of nonnative accents on listening comprehension: Implications for ESL assessment. TESOL Quarterly, 36(2), 173–190.
35.
MecarttyF. H. (2000). Lexical and grammatical knowledge in reading and listening comprehension by foreign language learners of Spanish. Applied Language Learning, 11(2), 323–348.
36.
O’MalleyJ. M.ChamotA. U. (1990). Learning strategies in second language acquisition. Cambridge, UK: Cambridge University Press.
37.
RaschG. (1960/1980). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Education Research, expanded edition (1980) with foreword and afterword by B. D. Wright. Chicago: The University of Chicago Press.
38.
RostM. (2002). Teaching and researching listening. Harlow, UK: Pearson Education.
39.
ShohamyE.InbarO. (1991). Validation of listening comprehension tests: The effect of text and question type. Language Testing, 8(1), 23–40.
40.
SkehanP. (1998). A cognitive approach to language learning. Oxford: Oxford University Press.
41.
SlaughterE. (2011). Bringing Asia to the home front: The Australian experience of Asian language education through national policy. In NorrbyC.HajekJ. (Eds.), Uniformity and diversity in language policy (pp. 157–173). Bristol, Buffalo, Toronto: Multilingual Matters.
42.
ThompsonI.RubinJ. (1996). Can strategy instruction improve listening comprehension?Foreign Language Annals, 29(3), 331–342.
43.
TsuiA. B. M.FulliloveJ. (1998). Bottom-up or top-down processing as a discriminator of L2 listening performance. Applied Linguistics, 19(4), 432–451.
44.
VandergriftL. (2003). Orchestrating strategy use: Toward a model of the skilled second language listener. Language Learning, 53(3), 463–496.
45.
WrightB. D. (1977). Misunderstanding the Rasch Model. Journal of Educational Measurement, 14(3), 219–225.
46.
WrightB. D.DouglasG. A. (1978). Better procedures for sample-free item analysis. MESA Memorandum No. 20. Chicago, IL: University of Chicago, Department of Education.
47.
WrightB. D.LinacreJ. M. (1994). Reasonable item mean-square fit values. Rasch Measurement Transactions, 8(3), 370.
48.
WrightB. D.MastersG. N. (1982). Rating scale analysis. Chicago: MESA Press.
49.
Yi’anW. (1998). What do tests of listening comprehension test?: A retrospection study of EFL test-takers performing a multiple-choice task. Language Testing, 15(1), 21–44.
50.
ZhaoY. (1997). The effects of listeners’ control of speech rate on second language comprehension. Applied Linguistics, 18(1), 49–68.
