This article presents a standard-setting procedure for performance assessment in a
foreign language, through which some of the major problems facing performance
assessment in criterion-referencedtesting can be addressed. The procedure, which was
geared to revealing and accommodating inter-judge variability, employed the synergy
of multiple approaches to the data analysis, including Generalizability Theory
(Brennan, 1983; Shavelson and Webb, 1991) and Multifaceted Rasch Analysis (Linacre,
1994; McNamara, 1996), whereby the investigator was better informed in handling the
problems associated with performance assessments in criterion-referenced testing.
The study was motivated by the need to develop a viable standard-setting procedure
and to demonstrate its potential.
Get full access to this article
View all access options for this article.
References
1.
Adams Smith, D. E.1984:
Medical discourse: aspects of author’s comment
. ESP Journal3,
25-36
.
2.
Angoff, W. H.1971: Scales, norms, and equivalent scores. In
Thorndike, R. L.
, editor, Educational measurement.
Washington, DC: American Council on Education
, 508-600.
3.
Bachman, L. F.
,
Lynch, B. K.
and
Mason, M.1995:
Investigating variability in tasks and rater judgements in a performance
test of foreign language speaking
. Language Testing12,
238-257
.
4.
Brennan, R. L.1983: Elements of Generalizability theory.
Iowa City, IA: American College Testing Program
.
5.
Crick, J. E.
and
Brennan, R. L.1984: GENOVA: a general purpose analysis of variance system.
Version 2.2.
Iowa City, IA: American College Testing Program
.
6.
Hambleton, R. K.1980: Test score validity and standard setting methods.
In
Berk, R. A.
, editor, Criterion-referenced measurement: state of the art.
Baltimore, MD: Johns Hopkins University Press
.
7.
Hambleton, R. K.
and
Plake, B. S.1995:
Using an extended Angoff procedure to set standards on complex performance assessments
. Applied Measurement in Education8,
41-55
.
8.
Henning, G.1987: A guide to language testing: development, evaluation,
research.
Boston: Heinle and Heinle
.
Linacre, J.M. 1997: MESA research note 2. Unpublished paper presented at
Midwest Objective Measurement Seminar, Chicago, IL, June, 1997.
11.
Linacre, J. M.
and
Williams, J.1999:
How much is enough
? Rasch Measurement Transaction12,
653-653
.
12.
Linacre, J. M.
and
Wright, B. D.1993: A User’s guide to FACETS: Rasch measurement computer
program. Version 3.2.
Chicago, IL: MESA Press
.
13.
Lumley, T.
,
Lynch, B. K.
and
McNamara, T. F.1994:
A new approach to standard-setting in language assessment
. Melbourne Papers in Language Testing3,
19-40
.
14.
Lunz, M. E.
,
Wright, B. D.
and
Linacre, J. M.1990:
Measuring the impact of judge severity on examination scores
. Applied Measurement in Education3,
331-345
.
15.
Lynch, B. K.
and
McNamara, T. F.1998:
Using G-theory and Many-facet Rasch measurement in the development of
performance assessments of the ESL speaking skills of immigrants
. Language Testing15,
158-180
.
16.
McNamara, T. F.1996: Measuring second language performance.
London: Longman
.
17.
McNamara, T. F.
and
Adams, R. J.2000: The implications of halo effects and item dependencies for
objective measurement. In
Wilson, M.
and
Engelhard, G.
, editors, Objective Measurement: theory into practice. Vol. 5.
Norwood, NJ: Ablex
, 243-257.
18.
Norcini, J. J.
and
Shea, J. A.1997:
The credibility and comparability of standards
. Applied Measurement in Education10,
39-59
.
19.
Plake, B. S.1998:
Setting performance standards for professional licensure and certification:
implications for National Assessment of Educational Progress
. Applied Measurement in Education11,
65-80
.
20.
Popham, W. J.1990: Modern educational measurement. 2nd edition.
Englewood Cliffs, NJ: Prentice-Hall
.
21.
Putnam, S. E.
,
Pence, P.
and
Jaeger, R. M.1995:
A multi-stage dominant profile method for setting standards on complex
performance assessments
. Applied Measurement in Education8,
57-83
.
22.
Shavelson, R. J.
and
Webb, N. M.1991: Generalizability theory: a primer.
Newbury Park, CA: Sage
.
23.
Stahl, J. A.
and
Lunz, M. E.
1991:
Judge performance reports: media and message
. Unpublished paper presented at the annual meeting of the
American Educational Research Association, San Francisco, CA.
24.
Stone, G.
1998:
Practical criterion referencing: a users guide to setting meaningful
objective standards
. Unpublished paper presented at an invited presession of the
Annual Meeting of the American Educational Research
Association, Montreal, April, 1999.
25.
Suen, H. K.1990: Principles of test theories.
Hillsdale, NJ: Lawrence Erlbaum
.
26.
Weigle, S. C.1998:
Using FACETS to model rater training effects
. Language Testing15,
263-287
.
