Rating scales that language testers design should be tailored to the specific test purpose and score use as well as reflect the target construct. Researchers have long argued for the value of data-driven scales for classroom performance assessment, because they are specific to pedagogical tasks and objectives, have rich descriptors to offer useful diagnostic information, and exhibit robust content representativeness and stable measurement properties. This sequential mixed methods study compares two data-driven rating scales with multiple criteria that use different formats for pragmatic performance. They were developed using roleplays performed by 43 second-language learners of Mandarin—the hierarchical-binary (HB) scale, developed through close analysis of performance data, and the multi-trait (MT) scale derived from the HB, which has the same criteria but takes the format of an analytic scale. Results revealed the influence of format, albeit to a limited extent: MT showed a marginal advantage over HB in terms of overall reliability, practicality, and discriminatory power, though measurement properties of the two scales were largely comparable. All raters were positive about the pedagogical value of both scales. This study reveals that rater perceptions of the ease of use and effectiveness of both scales provide further insights into scale functioning.
BondT. G.YanZ.HeeneM. (2021). Applying the Rasch model: Fundamental measurement in the human sciences (4th ed.). Routledge.
3.
BrennanR. L. (1999). GENOVA (Version 2.0) [Computer software]. The University of Iowa, Iowa Testing Programs.
4.
BrennanR. L. (2001). Generalizability theory. Springer.
5.
BrownA.IwashitaN.McNamaraT. (2005). An examination of rater orientations and test-taker performance on English-for-academic-purposes speaking tasks (TOEFL Monograph Series N 29). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.2005.tb01982.x
CarrN. (2011). Designing and analyzing language tests. Oxford University Press.
8.
ChenY.LiuJ. (2016). Constructing a scale to assess L2 written speech act performance: WDCT and e-mail tasks. Language Assessment Quarterly, 13(3), 231–250. https://doi.org/10.1080/15434303.2016.1213844
9.
CreswellJ. W.Plano ClarkV. L. (2018). Designing and conducting mixed-methods research (3rd ed.). Sage.
10.
DaiD. W. (2022). Design and validation of an L2-Chinese interactional competence test (Publication No. 30315843). [Doctor dissertation, University of Melbourne]. ProQuest Dissertation.
11.
DavisL. (2021). Rater and interlocutor training. In FulcherG.DavidsonF. (Eds.), The Routledge handbook of language testing (2nd ed., pp. 322–338). Routledge.
12.
EckesT. (2011). Introduction to many-facet Rasch measurement. Peter Lang.
13.
EwertD.ShinS.-Y. (2015). Examining instructors’ conceptualizations and challenges in designing a data-driven rating scale for a reading-to-write task. Assessing Writing, 26, 38–50. https://doi.org/10.1016/j.asw.2015.06.001
14.
FrostK.ElderC.WigglesworthG. (2012). Investigating the validity of an integrated listening-speaking task: A discourse-based analysis of test takers’ oral performances. Language Testing, 29(3), 345–369. https://doi.org/10.1177/0265532211424479
15.
FulcherG. (1987). Tests of oral performance: The need for data-based criteria. English Language Teaching Journal, 41, 287–291. https://doi.org/10.1093/elt/41.4.287
16.
FulcherG. (2003). Testing second language speaking. Longman/Pearson Education.
17.
FulcherG. (2012). Scoring performance tests. In FulcherG.DavidsonF. (Eds.), The Routledge handbook of language testing (pp. 378–392). Routledge.
18.
FulcherG.DavidsonF.KempJ. (2011). Effective rating scale development for speaking tests: Performance decision trees. Language Testing, 28(1), 5–29. https://doi.org/10.1177/0265532209359514
19.
Hamp-LyonsL. (1991). Assessing second language writing in academic contexts. Ablex.
20.
HiraiA.KoizumiR. (2008). Validation of the EBB scale: A case of the story retelling speaking test. JLTA Journal Kiyo, 11, 1–20. https://doi.org/10.20622/jltaj.11.0_1
21.
HiraiA.KoizumiR. (2013). Validation of empirically derived rating scales for a story retelling speaking test. Language Assessment Quarterly, 10(4), 398–422. https://doi.org/10.1080/15434303.2013.824973
22.
InoueM. (2009). Health sciences communications skills test: The development of a rating scale. Melbourne Papers in Language Testing, 14, 55–91. https://doi.org/10.3316/aeipt.182549
23.
IsaacsT.TrofimovichP.FooteJ. A. (2018). Developing a user-oriented second language comprehensibility scale for English-medium universities. Language Testing, 35(2), 193–216. https://doi.org/10.1177/0265532217703433
24.
KangO.KermadA. (2019). Prosody in L2 pragmatics research. In TaguchiN. (Ed.), The Routledge handbook of second language acquisition and pragmatics (pp.78–92). Routledge.
KnochU.DeygersB.KhamboonruangA. (2021). Revisiting rating scale development for rater-mediated language performance assessments: Modelling construct and contextual choices made by scale developers. Language Testing, 38(4), 602–626. https://doi.org/10.1177/0265532221994052
27.
LeechG. N. (1983). Principles of pragmatics. Longman.
28.
LiS.TaguchiN.XiaoF. (2019). Variations in rating scale functioning in assessing speech act production in L2 Chinese. Language Assessment Quarterly, 16(3), 271–293. https://doi.org/10.1080/15434303.2019.1648473
29.
LiS.WenT.LiX.FengY.LinC. (2023). Comparing holistic and analytic marking methods in assessing speech act production in L2 Chinese. Language Testing, 40(2), 249–275. https://doi.org/10.1177/02655322221113917
30.
LinacreJ. M. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85–106.
31.
LinacreJ. M. (2021). A user’s guide to FACETS Rasch-model computer programs [Program manual] (Version 3.83.5). https://www.winsteps.com/manuals.htm
32.
LunzM. E.WrightB. D.LinacreJ. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3(4), 331–245. https://doi.org/10.1207/s15324818ame0304_3
33.
NorthB. (1993). Scales of language proficiency: A survey of some existing systems. Council of Europe, Council for Cultural Co-operation.
34.
OhtaR.PlakansL. M.GebrilA. (2018). Integrated writing scores based on holistic and multi-trait scales: A generalizability analysis. Assessing Writing, 38, 21–36. https://doi.org/10.1016/j.asw.2018.08.001
35.
OkadaY. (2009). Roleplay in oral proficiency interviews: Interactive footing and interactional competence. Journal of Pragmatics, 42(6), 1647–1668. https://doi.org/10.1016/j.pragma.2009.11.002
36.
PlakansL. (2013). Writing scale development and use within a language program. TESOL Journal, 4(1), 151–163. https://doi.org/10.1002/tesj.66
SchegloffE. (2007). Sequencing organization in interaction: A primer in conversation analysis I. Cambridge University Press.
39.
ScollonR.ScollonS. (2005). Discourse and intercultural communication In SchiffrinD.TannenD.HamiltonH. E. (Eds.), The handbook of discourse analysis (pp. 537–547). Blackwell. https://doi.org/10.1002/9780470753460.ch28
40.
ShavelsonR. J.WebbN. R. (1991). Generalizability theory: A primer. Sage.
SkehanP. (2009). Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 30(4), 510–532. https://doi.org/10.1093/applin/amp047
43.
SuY. (2020). Yes or No: Ostensible versus genuine refusals in Mandarin invitational and offering discourse. Journal of Pragmatics, 162, 1–16. https://doi.org/10.1016/j.pragma.2020.03.007
44.
SuY. (2021). Accepting invitations and offers in second language Chinese: Effect of proficiency on pragmatic competence in interaction. Journal of Pragmatics, 180, 131–149. https://doi.org/10.1016/j.pragma.2021.04.033
45.
SuY. (2022). Refusing invitations and offers in second language Chinese: Effect of proficiency at the actional and interactional Levels. Journal of Politeness Research, 18(2), 335–365. https://doi.org/10.1515/pr-2019-0041
TaguchiN. (2019). The Routledge handbook of SLA and pragmatics. Routledge.
48.
TaguchiN.LiS. (2020). Contrastive pragmatics and second language (L2)pragmatics: Approaches to assessing L2 speech act production. Contrastive Pragmatics, 2(1), 1–23. https://doi.org/10.1163/26660393-bja10014
49.
TurnerC. (2000). Listening to the voices of rating scale developers: Identifying salient features for second language performance assessment. Canadian Modern Language Review, 56(4), 555–584. https://doi.org/10.3138/cmlr.56.4.555
50.
TurnerC.UpshurJ. (2002). Rating scales derived from student samples: Effects of the scale marker and the student sample on scale content and student scores. TESOL Quarterly, 36(1), 49–70. https://doi.org/10.2307/3588360
51.
UpshurJ.TurnerC. (1995). Constructing rating scales for second language tests. English Language Teaching Journal, 49(1), 3–12. https://doi.org/10.1093/elt/49.1.3
52.
WaltersF. (2007). A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing, 24(2), 155–183. https://doi.org/10.1177/0265532207076362
53.
WuS. L.OrtegaL. (2013). Measuring global oral proficiency in SLA research: A new elicited imitation test of L2 Chinese. Foreign Language Annals, 46(4), 680–704. https://doi.org/10.1111/flan.12063
54.
YounS. J. (2015). Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32(2), 199–225. https://doi.org/10.1177/0265532214557113
55.
YounS. J. (2018). Task design and validity evidence for assessment of L2 pragmatics in interaction. In TaguchiN.KimY.J. (Eds.), Task-based approaches to teaching and assessing pragmatics (pp. 217–246). John Benjamins. https://doi.org/10.1075/tblt.10.09you
56.
YounS. J. (2020). Interactional features of L2 pragmatic interaction in role-play speaking assessment. TESOL Quarterly, 54(1), 201–233. https://doi.org/10.1002/tesq.542
57.
YoungR. F. (2011). Interactional competence in language learning, teaching, and testing. In HinkelE. (Ed.), Handbook of research in second language teaching and learning (pp. 426–443). Routledge.
