Using G-theory and Many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants

Abstract

Second language performance tests, through the richness of the assessment context, introduce a range of facets which may influence the chances of success of a candidate on the test. This study investigates the potential roles of Generalizability theory (G-theory) (Brennan, 1983; Shavelson and Webb, 1991) and Many-facet Rasch measurement (Linacre, 1989; Linacre and Wright, 1993; McNamara, 1996) in the development of such a performance-based assessment procedure. This represents an extension of preliminary investigations into the relative contributions of these procedures (e.g., Bachman et al., 1995) to another assessment setting. Data for this study come from a trial of materials from the access: test, a test of communicative skills in English as a Second Language for intending immigrants to Australia. The performances of 83 candidates on the speaking skills module were multiply rated and analysed using GENOVA (Crick and Brennan, 1984) and FACETS (Linacre and Wright, 1993). The advantages and specific roles of these contrasting analytical techniques are considered in detail in the light of this assessment context.