Standard setting in specific-purpose language testing: What can a qualitative study add?

Abstract

This paper explores the views of nursing and medical domain experts in considering the standards for a specific-purpose English language screening test, the Occupational English Test (OET), for professional registration for immigrant health professionals. Since individuals who score performances in the test setting are often language experts rather than domain experts, there are possible tensions between what is being measured by a language test and what is deemed important by domain experts. Another concern is a lack of qualitative research on the process of the standard setting. To date, no published qualitative work has been identified about the contributions of domain experts in the standard setting for healthcare communication. In this study, a standard-setting exercise was conducted for the speaking component of the OET, using judgements of nursing and medical clinical educators and supervisors. In all, 13 medical and 18 nursing clinical educators and supervisors rated medical and nursing candidate performances respectively. These performances were audio-recorded OET role-plays that were selected across a range of proficiency levels. Domain experts were invited to comment on the basis of their decisions and the extent of alignment between these decisions and the criteria used to assess performance on the OET. Nursing and medical domain experts showed that they attended to all of the OET criteria in making their decisions about standards. However, clinical scenario simulation also invited judgements of clinical competence from participants, even where they knew that clinical competence should be excluded from their decision-making. Another concern related to the authenticity limitations of the role-play tasks as evidence of readiness to handle communication in the workplace. Overall, findings support the value of qualitative evidence from the standard setting in providing insight into the factors informing and impeding decision-making.

Keywords

Healthcare communication language for specific purposes testing Occupational English Test role-play standard setting

Get full access to this article

View all access options for this article.

References

Angoff

. (1971). Scales, norms, and equivalent scores. In Thorndike

(Ed.), Educational measurement (2nd ed., pp. 508–600). Washington, DC: American Council on Education.

Cizek

Bunch

. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: SAGE.

Kenyon

Römhild

. (2014). Standard setting in language testing. In Kunnan

(Ed.), The companion to language assessment. Hoboken, NJ: John Wiley & Sons.

Mitzel

Lewis

Patz

Green

. (2001). The bookmark procedures: Psychological perspectives. In Cizek

(Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 249–281). Mahwah, NJ: Lawrence Erlbaum.

Papageorgiou

. (2010). Investigating the decision-making process of standard setting participants. Language Testing, 27(2), 261–282. doi: 10.1177/0265532209349472

Plake

Hambleton

. (2001). The analytic judgment method for setting standards on complex performance assessments. In Cizek

G. J

(Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 283–312). Mahwah, NJ: Lawrence Erlbaum.

Spencer

Ritchie

O’Connor

. (2003). Analysis: practices, principles, and processes. In Ritchie

Lewis

(Eds.), Qualitative research practice: A guide for social science students and researchers (pp. 199–262). London: SAGE.