The use of Item Response Theory (IRT) in the examination of the qualities of language tests is a comparatively recent development, and one that has proved controversial. Oller (1983) contains no reference to IRT in a wide-ranging col lection. By contrast, IRT, particularly involving the family of models deriving from the work of Rasch (1960 [1980]), has featured in a number of studies since the early 1980s. Although Rasch IRT has sometimes been discussed primarily as a tool for improved investigation of the reliability of tests (Skehan, 1989), its potential for investigating aspects of the validity of language tests has also been demonstrated (McNamara, 1990a). The application of Rasch IRT in this latter role has in some cases met with objections based on what are claimed to be its unsatisfactory theoretical assumptions, in particular the so-called 'unidimensionality' assumption (Hamp-Lyons, 1989). In this paper, this issue will be discussed in the context of the analysis of data from an ESP listening test for health professionals, part of a larger test, the Occupational English Test (OET), developed on behalf of the Australian Government (McNamara, 1989b). The paper is in three sections. First, there is a brief description of the listening subtest of the OET. Second, the appropriateness of the use of Rasch IRT in language testing research is discussed. Third, the use of Rasch IRT in the validation of the listening subtest of the OET is reported. In this part of the paper, the issue of unidimensionality is considered in the context of analysis of data from the two parts of this test.
Get full access to this article
View all access options for this article.
References
1.
Adams, R.J., Griffin, P.E. and Martin, L.1987: A latent trait method for measuring a dimension in second language proficiency. Language Testing4, 9-27.
2.
Alderson, J.C., Candlin, C.N., Clapham, C.M., Martin, D.J. and Weir, C.J.1986: Language proficiency testing for migrant professionals: new directions for the Occupational English Test. Lancaster : Institute for English Language Education, University of Lancaster.
3.
Andrich, D.1978a: A rating formulation for ordered response categories . Psychometrika43, 561-73.
4.
— 1978b: Scaling attitude items constructed and scored in the Likert tradition. Educational and Psychological Measurement38, 665-80.
5.
Bejar, I.1980: A procedure for investigating the unidimensionality of achievement tests based on item parameter estimates. Journal of Educational Measurement17, 283-96.
6.
Carrell, P.L., Devine, J. and Eskey, D.E., editors 1988: Interactive approaches to second language reading. Cambridge: Cambridge University Press.
7.
Choppin, B.1982: The use of latent trait models in the measurement of cognitive abilities and skills. In Spearritt, D., editor, The improvement of measurement in education and psychology , Hawthorn, Victoria: Australian Council for Educational Research, 41-63.
8.
Cramer, H.1946: Mathematical methods of statistics. Princeton: Princeton University Press.
9.
Cummins, J.1984: Wanted: a theoretical framework for relating language proficiency to academic achievement among bilingual students. In Rivera, C., editor, Language proficiency and academic achievement, Clevedon, Avon: Multilingual Matters, 2-19.
10.
Davidson, F. and Henning, G.1985: A self-rating scale of English difficulty: Rasch scalar analysis of items and rating categories. Language Testing2, 164-79.
11.
de Jong, J.H.A.L. and Glas, C.A.W.1987: Validation of listening comprehension tests using items response theory. Language Testing4, 170-94.
12.
de Jong, J.H.A.L. and Henning, G.1990: Test dimensionality in relation to student proficiency . Paper presented at the Language Testing Research Colloquium, San Francisco, March 2-5.
13.
Goldstein, H.1979: Consequences of using the Rasch Model for educational assessment. British Educational Research Journal5, 211-20.
14.
Griffin, P.1988: Tests must be administered as designed: a reply to David Nunan. In McNamara, T.F., editor, Language testing colloquium. Selected papers from a Colloquium held at the Horwood Language Centre, University of Melbourne, 24-25 August, 1987. Australian Review of Applied Linguistics11, 66-72.
15.
Griffin, P.E., Adams, R.J., Martin, L. and Tomlinson, B.1988: An algorithmic approach to prescriptive assessment in English as a Second Language. Language Testing5, 1-18.
16.
Hamp-Lyons, L.1989: Applying the partial credit model of Rasch analysis: language testing and accountability. Language Testing6, 109-18.
17.
Henning, G.1987a: A guide to language testing: development, evaluation, research. Cambridge, MA: Newbury House.
18.
— 1987b: Is the Bejar test of unidimensionality appropriate? A response to Spurling. Language Testing4, 96-98.
19.
———— 1988: The influence of test and sample dimensionality on latent trait person ability and item difficulty calibrations. Language Testing5, 83-99.
20.
— 1989: Meanings and implications of the principle of local independence . Language Testing6, 95-108.
21.
Henning, G. and Davidson, F.1987: Scalar analysis of composition ratings. In Bailey, K.M., Dale, T.L. and Clifford, R.T., editors, Language testing research, Selected papers from the 1986 colloquium, Monterey, CA: Defense Language Institute, 24-38.
22.
Henning, G., Hudson, T. and Turner, J.1985: Item response theory and the assumption of unidimensionality for language tests. Language Testing2, 141-54.
23.
Hughes, A.1989: Testing for language teachers. Cambridge : Cambridge University Press.
24.
McNamara, T.F.1987: Assessing the language proficiency of health professionals. Recommendations for the reform of the Occupational English Test. A report submitted to the Council on Overseas Professional Qualifications. Parkville, Victoria: University of Melbourne, Department of Russian and Language Studies.
25.
— 1988a: The development of an English as a Second Language speaking test for health professionals. Part One of a report to the Council on Overseas Professional Qualifications on a consultancy to develop the Occupational English Test. Parkville, Victoria: University of Melbourne, Department of Russian and Language Studies .
26.
—, editor 1988b: Language testing colloquium. Selected papers from a Colloquium at the Horwood Language Centre, University of Melbourne, 24-25 August, 1987. Australian Review of Applied Linguistics11.
27.
— 1989a: The development of an English as a Second Language test of writing skills for health professionals. Part Two of a report to the Council on Overseas Professional Qualifications on a consultancy to develop the Occupational English Test. Parkville, Victoria : University of Melbourne, Department of Russian and Language Studies.
28.
1989 b: ESP testing: general and particular. In Candlin, C.N. and McNamara, T.F., editors, Language, learning and community, Sydney, NSW: National Centre for English Language Teaching and Research, Macquarie University, 125-42.
29.
— 1990a: Item Response Theory and the validation of an ESP test for health professionals. Language Testing7, 52-75.
30.
1990b: Assessing the second language proficiency of health professionals. Ph.D. thesis, University of Melbourne.
31.
Nunan, D.1987a: Developing discourse comprehension: theory and practice . Singapore: SEAMEO Regional Language Centre.
32.
— 1987b: Methodological issues in research. In Nunan, D., editor, Applying second language acquisition research, Adelaide: National Curriculum Resource Centre, 143-171.
33.
— 1988: Commentary on the Griffin paper. In McNamara T.F., editor, Language testing colloquium . Selected papers from a Colloquium held at the Horwood Language Centre, University of Melbourne, 24-25 August, 1987. Australian Review of Applied Linguistics11, 54-65.
34.
Oller, J.W., editor 1983: Issues in language testing research. Rowley, MA: Newbury House.
35.
Pollitt, A., in press: Diagnostic assessment through item banking. In Entwhistle, N., editor, Handbook of educational ideas and practices, London: Croom Helm.
36.
Pollitt, A. and Hutchinson, C.1987: Calibrated graded assessments: Rasch partial credit analysis of performance in writing. Language Testing4, 72-92.
37.
Rasch, G.1960 [1980]: Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedogogiske Institut [Chicago: University of Chicago Press].
38.
Skehan, P.1989: Language testing part II. Language Teaching22, 1-13.
39.
Spolsky, B.1988: Test review: P.E. Griffin etal. (1986), Proficiency in English as a second language. (1) The development of an interview test for adult migrants. (2) The administration and creation of a test. (3) An interview test of English as a second language. Language Testing5, 120-24.
40.
Spurling, S.1987a: Questioning the use of the Bejar method to determine unidimensionality. Language Testing4, 93-95.
41.
Spurling, S.1987b: The Bejar Method with an example: a comment on Henning's 'Response to Spurling'. Language Testing4, 221-23.
42.
Widdowson, H.G.1983: Learning purpose and language use. Oxford: Oxford University Press.
43.
Wright, B.D., Congdon, R.T. and Rossner, M.1987: MSTEPS. A Rasch program for ordered response categories . Chicago, IL: Department of Education, University of Chicago.
