Restricted accessResearch articleFirst published online 2022-1
The domain expert perspective: A qualitative study into the views expressed in a standard-setting exercise on a language for specific purposes (LSP) test for health professionals
This paper investigates what matters to medical domain experts when setting standards on a language for specific purposes (LSP) English proficiency test: the Occupational English Test’s (OET) writing sub-test. The study explores what standard-setting participants value when making performance judgements about test candidates’ writing responses, and the extent to which their decisions are language-based and align with the OET writing sub-test criteria. Qualitative data is a relatively under-utilized component of standard setting and this type of commentary was garnered to gain a better understanding of the basis for performance decisions. Eighteen doctors were recruited for standard-setting workshops. To gain further insight, verbal reports in the form of a think-aloud protocol (TAP) were employed with five of the 18 participants. The doctors’ comments were thematically coded and the analysis showed that participants’ standard-setting judgements often aligned with the OET writing sub-test criteria. An overarching theme, ‘Audience Recognition,’ was also identified as valuable to participants. A minority of decisions were swayed by features outside the OET’s communicative construct (e.g., clinical competency). Yet, overall, findings indicated that domain experts were undeniably focused on textual features associated with what the test is designed to assess and their views were vitally important in the standard-setting process.
AldersonJ. C.CandlinC. N.ClaphamC. M.MartinD. J.WeirC. J. (1986). Language proficiency testing for migrant professionals: New directions for the Occupational English Test. A report submitted to the Council on Overseas Professional Qualifications. Institute for English Language Education, University of Lancaster.
BarkaouiK. (2011). Think-aloud protocols in research on essay writing: An empirical study of their veridicality and reactivity. Language Testing, 28(1), 51–75. https://doi.org/10.1177/0265532210376379
4.
CizekG. J. (Ed.). (2001). Standard setting: Concepts, methods and perspectives. Lawrence Erlbaum.
5.
CizekG. J. (2012). An introduction to contemporary standard setting. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (pp. 3–14). Routledge.
6.
CizekG. J.BunchM. B. (2007). The body of work and other holistic methods. In CizekG. J.BunchM. B. (Eds.), Standard setting: A guide to establishing and evaluating performance standards on tests (pp. 117–153). SAGE Publications.
CohenA. D. (2000). Exploring strategies in test-taking: Fine-tuning verbal reports from respondents. In EkbataniG.PiersonH. (Eds.), Learner-directed assessment in ESL (pp. 127–150). Lawrence Erlbaum.
9.
CummingA.KantorR.PowersD. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86, 67–96. https://doi.org/10.1111/1540-4781.00137
10.
DouglasD. (2001a). Language for specific purposes assessment criteria: Where do they come from?Language Testing, 18(2), 171–185. https://doi.org/10.1177/026553220101800204
11.
DouglasD. (2001b). Three problems in testing language for specific purposes: Authenticity, specificity and inseparability. In ElderC.BrownA.GroveE.HillK.IwashitaN.LumleyT.McNamaraT. F.O’LoughlinK. (Eds.), Experimenting with uncertainty: Essays in honour of Alan Davies (pp. 45–52). Cambridge University Press.
12.
DouglasD.MyersR. (2000). Assessing the communication skills of veterinary students: Whose criteria. In Fairness and validation in language assessment: Selected papers from the 19th Language Testing Research Colloquium, Orlando, Florida (pp. 60–81). Cambridge University Press.
13.
ElderC. (2016). Exploring the limits of authenticity in LSP testing: The case of a specific-purpose language test for health professionals. Language Testing, 33(2), 1–6. https://doi.org/10.1177/0265532215607397
14.
ElderC.McNamaraT.KimH.PillJ.SatoT. (2017). Interrogating the construct of communicative competence in language assessment contexts: What the non-language specialist can tell us. Language & Communication, 57, 14–21.
15.
EricssonK. A.SimonH. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). MIT Press.
16.
FleckensteinJ.KelleraS.KrügeraM.TannenbaumR. J.KöllerO. (2020). Linking TOEFL iBT® writing rubrics to CEFR levels: Cut scores and validity evidence from a standard setting study. Assessing Writing, 43, Article 100420. https://doi.org/10.1016/j.asw.2019.100420
17.
GlaserB. G.StraussA. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.
18.
GreenA. (1998). Verbal protocol analysis in language testing research: A handbook, vol. 5. Cambridge University Press.
19.
HarschC.KanistraV. P. (2020) Using an innovative standard-setting approach to align integrated and independent writing tasks to the CEFR, Language Assessment Quarterly, 17(3), 262–281. https://doi.org/10.1080/15434303.2020.1754828
20.
HughesJ.ParkesS. (2003). Trends in the use of verbal protocol analysis in software engineering research. Behaviour & Information Technology, 22(2), 127–140. https://doi.org/10.1080/0144929031000081341
21.
HuotB. A. (1993). The influence of holistic scoring procedures on reading and rating student essays. In WilliamsonM. M.HuotB. A. (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 206–236). Hampton Press.
22.
JacobyS. (1998). Science as performance: Socializing scientific discourse through the conference talk rehearsal. University of California, Los Angeles.
LumleyT. (1998). Perceptions of language-trained raters and occupational experts in a test of occupational English language proficiency. English for Specific Purposes, 17(4), 347–367. https://doi.org/10.1016/S0889-4906(97)00016-1
27.
LumleyT. (2005). Assessing second language writing: The rater’s perspective. Peter Lang.
28.
LumleyT.LynchB. K.McNamaraT. (1994). A new approach to standard-setting in language assessment. Melbourne Papers in Language Testing, 3(2), 19–40.
29.
ManiasE.McNamaraT. (2016). Standard setting in specific-purpose language testing: What can a qualitative study add?Language Testing, 33(2), 235–249. https://doi.org/10.1177/0265532215608411
30.
McNamaraT. F. (1990). Assessing the second language proficiency of health professionals [Unpublished doctoral dissertation]. University of Melbourne, Melbourne, Australia.
31.
McNamaraT. F. (1996). Measuring second language performance. Addison Wesley Longman.
32.
MilanovicM.SavilleN.ShuhongS. (1996). A study of the decision-making behaviour of composition markers. In MilanovicM.SavilleN. (Eds.), Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Research Colloquium, Cambridge and Arnhem (pp. 92–114). Cambridge University Press and University of Cambridge Local Examinations Syndicate.
O’NeillT. R.BuckendahlC. W.PlakeB. S.TaylorL. (2007). Recommending a nursing-specific passing standard for the IELTS examination. Language Assessment Quarterly, 4(4), 295–317. https://doi.org/10.1080/15434300701533562
35.
PapageorgiouS. (2010). Investigating the decision-making process of standard setting participants. Language Testing, 27(2), 261–282. https://doi.org/10.1177/0265532209349472
36.
PillT. J. H. (2013). What doctors value in consultations and the implications for specific-purpose language testing [Unpublished doctoral dissertation]. University of Melbourne, Australia.
37.
PillJ. (2016). Drawing on indigenous criteria for more authentic assessment in a specific-purpose language test: Health professionals interacting with patients. Language Testing, 33(2), 175–193. https://doi.org/10.1177/0265532215607400
38.
PillJ.McNamaraT. (2016). How much is enough? Involving occupational experts in setting standards on a specific-purpose language test for health professionals. Language Testing, 33(2), 217–234. https://doi.org/10.1177/0265532215607402
39.
PlakeB. S.HambletonR. K. (2001). The analytic judgement method for setting standards on complex performance assessments. In CizekG. J. (Ed.), Standard setting: Concepts, methods and perspectives (pp. 283–312). Lawrence Erlbaum.
SakyiA. A. (2000). Validation of holistic scoring for ESL writing assessment: How raters evaluate compositions. In KunnanA. J. (Ed.), Fairness and validation in language assessment: Selected papers from the 19th Language Testing Research Colloquium, Orlando, FL (pp. 129–152). Cambridge University Press.
43.
StraussA. L.CorbinJ. M. (1990). Basics of qualitative research: Grounded theory procedures and techniques. SAGE Publications.
44.
VausD. D. (2002). Surveys in social research (5th ed.). Allen & Unwin.
45.
WooA.DickinsonP.de JongE. J. (2010). Setting an English language proficiency passing standard for entry-level nursing practice using the Pearson Test of English Academic. https://www.ncsbn.org/NCLEX_technicalbrief_PTE_2010.pdf
46.
WolfeE. W.KaoC. W.RanneyM. (1998). Cognitive difference in proficient and nonproficient essay scorers. Written Communication, 15(4), 465–492. https://doi.org/10.1177/0741088398015004002
47.
ZiekyM. J.PerieM.LivingstonS. A. (2008). Cut scores: A manual for setting standards of performance on educational and occupational tests. Educational Testing Service.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
