Assessment rubrics have increasingly been developed and deployed to evaluate the quality of language interpreting, yet understanding of rubric-based interpreting assessment remains limited. This systematic review aims to: (a) catalog rubrics, (b) examine rubric design features, (c) understand rubric use, and (d) evaluate rubric utility. A rigorous review process, involving database searching, citation tracking, and targeted review of core literature, identified 80 unique rubrics comprising a total of 265 (sub-)scales. A comprehensive analysis revealed that: (a) among 11 potential sources informing rubric development, test-external sources (e.g., literature review) were primarily used, whereas test-internal sources (e.g., performance samples) were much less consulted; (b) assessments primarily used analytic and task-type rubrics with an average of five performance levels and four scoring criteria. Rubric descriptors generally incorporated observable indicators of interpreting quality, employing both descriptive and evaluative rubric language; (c) rubrics were used by three main types of raters—interpreting practitioners, trainers, and students—to assess primarily spoken-language interpreting; and (d) rubric-based scores demonstrated moderate-to-high reliability and validity overall, though meta-analysis identified three significant moderators, including correlation coefficient type, assessment criterion, and rubric length. These findings are expected to provide guidance for assessment practice and research in interpreting.
BakerB. A.TurnerC. E. (2022). Rating scales and rubrics in language assessment. In ChapelleC. A. (Ed.), The encyclopedia of applied linguistics (pp. 987–994). John Wiley & Sons. https://doi.org/10.1002/9781405198431.wbeal1045.pub2
2.
BargainnierS. (2003). Fundamentals of rubrics. Pacific Crest, 5, 1–4.
3.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2021). Introduction to meta-analysis (2nd ed.). John Wiley & Sons.
4.
BrookhartS. M. (2013). How to create and use rubrics for formative assessment and grading. ASCD publications.
ChenS.-J.KrugerJ.-L. (2024). A computer-assisted consecutive interpreting workflow: Training and evaluation. Interpreter and Translator Trainer, 18(3), 380–399. https://doi.org/10.1080/1750399X.2024.2373553
8.
DavisL. (2016). The influence of training and experience on rater performance in scoring spoken language. Language Testing, 33(1), 117–135. https://doi.org/10.1177/0265532215582282
9.
DawrantA.HanC. (2021). Testing for professional qualification in conference interpreting. In Albl-MikasaM.TiseliusE. (Eds.), The Routledge handbook of conference interpreting (pp. 258–274). Routledge.
10.
DawsonP. (2017). Assessment rubrics: Towards clearer and more replicable design, research and practice. Assessment & Evaluation in Higher Education, 42(3), 347–360. https://doi.org/10.1080/02602938.2015.1111294
HanC. (2018). Using rating scales to assess interpretation: Practices, problems and prospects. Interpreting, 20(1), 59–95. https://doi.org/10.1075/intp.00003.han
20.
HanC. (2019). A generalizability theory study of optimal measurement design for a summative assessment of English/Chinese consecutive interpreting. Language Testing, 36(3), 419–438. https://doi.org/10.1177/0265532218809396
21.
HanC.HuJ.DengY. (2023). Effects of language background and directionality on raters’ assessments of spoken-language interpreting. Revista Española de Lingüística Aplicada, 36(2), 556–584. https://doi.org/10.1075/resla.21009.han
22.
HanC.JiangM.-T.ChenQ.-L. (2025, September30). A systematic review and meta-analysis of rubrics in rater-mediated assessment of language interpreting. Open Science Framework. https://osf.io/n5vfw/
23.
HanC.SlatyerH. (2016). Test validation in interpreter certification performance testing: An argument-based approach. Interpreting, 18(2), 231–258. https://doi.org/10.1075/intp.18.2.04han
24.
HanC.XiaoR.SuW. (2021). Assessing the fidelity of consecutive interpretation: The effects of using source versus target text as the reference material. Interpreting, 23(2), 245–468. https://doi.org/10.1075/intp.00058.han
25.
HanC.ZhengB.-H.XieM.-Q.ChenS.-R. (2024). Raters’ scoring process in assessment of interpreting: An empirical study based on eye tracking and retrospective verbalization. The Interpreter and Translator Trainer, 18(3), 400–422. https://doi.org/10.1080/1750399X.2024.2326400
26.
HumphryS. M.HeldsingerS. A. (2014). Common structural design features of rubrics may represent a threat to validity. Educational Researcher, 43(5), 253–263. https://doi.org/10.3102/0013189X14542154
27.
IsaacsT.SuvorovR.YanX.StyrinaS. (in press). Research syntheses in the international journal, ‘Language Testing’: Publishing trends and a drive to improve standards in research dissemination. In ChongS. W. (Ed.), Research synthesis methodologies in applied linguistics. John Benjamins.
28.
IsaacsT.ThomsonR. I. (2013). Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 10(2), 135–159. https://doi.org/10.1080/15434303.2013.769545
29.
IsaacsT.YanX. (2025). Language Testing behind the scenes: New capabilities, groundbreaking developments, and disruptive forces. Language Testing, 42(1), 3–10. https://doi.org/10.1177/02655322241288490
30.
JiangZ.-K.ZhangZ.-Y. (2025). From black box to transparency: Enhancing automated interpreting assessment with explainable AI in college classrooms. Research Methods in Applied Linguistics, 4(3), 100237. https://doi.org/10.1016/j.rmal.2025.100237
31.
JonesL.AllenB.DunnP.BrookerL. (2017). Demystifying the rubric: A five-step pedagogy to improve student understanding and utilisation of marking criteria. Higher Education Research & Development, 36(1), 129–142. https://doi.org/10.1080/07294360.2016.1177000
32.
JonssonA.PanaderoE. (2017). The use and design of rubrics to support assessment for learning. In CarlessD.BridgesS. M.ChanC. K. Y.GlofcheskiR. (Eds.), Scaling up assessment for learning in higher education (pp. 99–112). Springer. https://doi.org/10.1007/978-981-10-3045-1_7
33.
JonssonA.SvingbyG. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002
34.
KnochU.DeygersB.KhamboonruangA. (2021). Revisiting rating scale development for rater-mediated language performance assessments: Modelling construct and contextual choices made by scale developers. Language Testing, 38(4), 602–626. https://doi.org/10.1177/0265532221994052
LiX.-D. (2018). Self-assessment as “assessment as learning” in translator and interpreter education: Validity and washback. The Interpreter and Translator Trainer, 12(1), 48–67. https://doi.org/10.1080/1750399X.2017.1418581
37.
LiY.LiaoX.-N.JiaJ. (2025). Effects of raters’ professional backgrounds on assessing interpreting quality: An exploratory mixed-methods investigation into rater behavior. System, 133, 103772. https://doi.org/10.1016/j.system.2025.103772
38.
LiuM.-H. (2013). Design and analysis of Taiwan’s Interpretation Certification Examination. In TsagariD.van DeemterR. (Eds.), Assessment issues in language translation and interpreting (pp. 163–178). Peter Lang.
39.
MokkinkL. B.PrinsenC. A. C.BouterL. M.de VetH. C. W.TerweeC. B. (2016). The COnsensus-based standards for the selection of health Measurement INstruments (COSMIN) and how to select an outcome measurement instrument. Brazilian Journal of Physical Therapy, 20(2), 105–113. http://dx.doi.org/10.1590/bjpt-rbf.2014.0143
MyfordC. M.WolfeE. W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 5(2), 189–227.
PanaderoE.JonssonA. (2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational Research Review, 9, 129–144. https://doi.org/10.1016/j.edurev.2013.01.002
45.
PanaderoE.JonssonA. (2020). A critical review of the arguments against the use of rubrics. Educational Research Review, 30, 100329. https://doi.org/10.1016/j.edurev.2020.100329
46.
PennyJ.JohnsonR. L.GordonB. (2000). Using rating augmentation to expand the scale of an analytic rubric. The Journal of Experimental Education, 68(3), 269–287.
47.
PlonskyL.DerrickD. J. (2016). A meta-analysis of reliability coefficients in second language research. The Modern Language Journal, 100(2), 538–553. https://doi.org/10.1111/modl.12335
StanleyT. (2021). Using rubrics for performance-based assessment: A practical guide to evaluating student work. Routledge. https://doi.org/10.4324/9781003239390
52.
StemlerS. E.TsaiJ. (2008). Best practices in interrater reliability: Three common approaches. In OsborneJ. (Ed.), Best practices in quantitative methods (pp. 29–49). Sage. https://doi.org/10.4135/9781412995627.d5
TiseliusE. (2009). Revisiting Carroll’s scales. In AngelelliC.JacobsonH. E. (Eds.), Testing and assessment in translation and interpreting studies: A call for dialogue between research and practice (pp. 95–121). John Benjamins. https://doi.org/10.1075/ata.xiv.07tis
55.
WangB.-H. (2011). 口译能力的评估模式及测试设计再探—以全国英语口译大赛为例 [Exploration of the assessment model and test design of interpreting competence]. Foreign Language World, 142, 66–71.
56.
WangJ.NapierJ.GoswellD.CarmichaelA. (2015). The design and application of rubrics to assess signed language interpreting performance. The Interpreter and Translator Trainer, 9(1), 83–103. https://doi.org/10.1080/1750399X.2015.1009261
57.
WindA. S. (2023). A sequential approach to detecting differential rater functioning in sparse rater-mediated assessment networks. Language Testing, 40(2), 209–226. https://doi.org/10.1177/02655322221092388
58.
YehS.-P.LiuM.-H. (2006). 口譯評分客觀化初探: 採用量表的可能性 [A more objective approach to interpretation evaluation: Exploring the use of scoring rubrics]. Journal of the National Institute for Compilation and Translation Review, 34(4), 57–78.
