A comparative judgment approach to assessing Chinese Sign Language interpreting

Abstract

The quality of sign language interpreting (SLI) is a gripping construct among practitioners, educators and researchers, calling for reliable and valid assessment. There has been a diverse array of methods in the extant literature to measure SLI quality, ranging from traditional error analysis to recent rubric scoring. In this study, we want to expand the terrain of SLI assessment, by exploring and evaluating a novel method, known as comparative judgment (CJ), to assess SLI quality. Briefly, CJ involves judges to compare two like objects/items and make a decision by choosing the one with higher quality. The binary outcomes from repeated comparisons by a group of judges are then modelled statistically to produce standardized estimates of perceived quality for each object/item. We recruited 12 expert judges to operationalize CJ via a computerized system to assess the quality of Chinese Sign Language interpreting produced by 36 trainee interpreters. Overall, our analysis of quantitative and qualitative data provided preliminary evidential support for the validity and utility of CJ in SLI assessment. We discussed these results in relation to previous SLI literature, and suggested future research to cast light on CJ’s usefulness in applied assessment contexts.

Keywords

Chinese Sign Language comparative judgment interpreting quality sign language interpreting testing and assessment

Get full access to this article

View all access options for this article.

References

Bontempo

Hutchinson

(2011). Striving for an ‘A’ grade: A case study in performance management of interpreters. International Journal of Interpreter Education, 3, 56–71. http://www.cit-asl.org/new/wp-content/uploads/2014/07/5-Bontempo.pdf

Bradley

R. A.

Terry

M. E.

(1952). Rank analysis of incomplete block designs: The method of paired comparisons. Biometrika, 39(3/4), 324–345. https://doi.org/10.2307/2334029

Cokely

(1986). The effects of lag time on interpreter errors. Sign Language Studies, 53, 341–375. https://doi.org/10.1353/sls.1986.0025

De Wit

Sluis

(2014). Sign language interpreter quality: The perspective of deaf sign language users in the Netherlands. The Interpreters’ Newsletter, 19, 63–85. https://doi.org/10.13137/1591-4127/10650

Eckes

(2015). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessment (2nd ed.). Peter Lang. https://doi.org/10.3726/978-3-653-04844-5

González

R. D.

Gatto

Bichsel

(2010). Constructing a valid and reliable trilingual interpreting testing instrument. In McKee

R. L.

Davis

J. E.

(Eds.), Interpreting in multilingual, multicultural contexts (pp. 55–82). Gallaudet University Press. https://muse.jhu.edu/chapter/272584/pdf

Han

(2018). Using rating scales to assess interpretation: Practices, problems and prospects. Interpreting, 20(1), 59–95. https://doi.org/10.1075/intp.00003.han

Han

(2020). Translation quality assessment: A critical methodological review. The Translator, 26(3), 257–273. https://doi.org/10.1080/13556509.2020.1834751

Han

(2021). Interpreting testing and assessment: A state-of-the-art review. Language Testing, 39(1), 30–55. https://doi.org/10.1177/02655322211036100

10.

Jones

Inglis

(2015). The problem of assessing problem solving: Can comparative judgement help? Educational Studies in Mathematics, 89, 337–355. https://doi.org/10.1007/s10649-015-9607-1

11.

Linacre

M. J.

(2021). A user’s guide to FACETS: Rasch-Model computer programs. Accessed May 19, 2021. https://winsteps.com/a/Facets-Manual.pdf

12.

Luce

(1959). Individual choice behavior: A theoretical analysis. Wiley.

13.

Marschark

Sapere

Convertino

Seewagen

Maltzan

(2004).Comprehension of sign language interpreting: Deciphering a complex task situation. Sign Language Studies, 4(4), 345–368. https://doi.org/10.1353/sls.2004.0018

14.

McKee

R. L.

(2008). ‘Quality’ in interpreting: A survey of practitioner perspectives. The Sign Language Translator and Interpreter, 2(1), 1–14. https://www.researchgate.net/publication/256761952_‘Quality’_in_Interpreting_A_Survey_of_Practitioner_Perspectives

15.

McMahon

Jones

(2015). A comparative judgement approach to teacher assessment. Assessment in Education: Principles, Policy and Practice, 22(3), 368–389. https://doi.org/10.1080/0969594X.2014.978839

16.

Myford

C. M.

Wolfe

E. W.

(2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386–422. https://www.ncbi.nlm.nih.gov/pubmed/14523257

17.

NAATI (2019). Certified interpreter (Auslan & English) test assessment rubrics. https://www.naati.com.au/media/2310/ci_auslan_assessment_rubrics.pdf

18.

Pollitt

(2012). Comparative judgement for assessment. International Journal of Technology and Design Education, 22(2), 157–170. https://doi.org/10.1007/s10798-011-9189-x

19.

Rudser

S. F.

Strong

(1986). An examination of some personal characteristics and abilities of sign language interpreters. Sign Language Studies, 53, 315–331. https://doi.org/10.1353/sls.1986.0005

20.

Russell

Malcolm

(2009). Assessing ASL–English interpreters: The Canadian model of national certification. In Angelelli

C. V.

Jacobson

H. E.

(Eds.), Testing and assessment in translation and interpreting studies (pp. 331–376). John Benjamins. https://doi.org/10.1075/ata.xiv.15rus

21.

Schick

Williams

Bolster

(1999). Skill levels of educational interpreters working in the public schools. Journal of Deaf Studies and Deaf Education, 4(2), 144–155. https://doi.org/10.1093/deafed/4.2.144

22.

Steedle

J. T.

Ferrara

(2016). Evaluating comparative judgment as an approach to essay scoring. Applied Measurement in Education, 29(3), 211–223. https://doi.org/10.1080/08957347.2016.1171769

23.

Strong

Rudser

S. F.

(1985). An assessment instrument for sign language interpreters. Sign Language Studies, 49, 344–362. https://doi.org/10.1353/sls.1985.0008

24.

Strong

Rudser

S. F.

(1986). The subjective assessment of sign language interpreters. Sign Language Studies, 53, 299–314. https://doi.org/10.1353/sls.1986.0018

25.

Thurstone

L. L.

(1927). A law of comparative judgment. Psychological Review, 34(4), 273–286. https://doi.org/10.1037/h0070288

26.

Van Dijk

Boers

Christoffels

I. K.

Hermans

(2011). Directionality effects in simultaneous language interpreting: The case of sign language interpreters in the Netherlands. American Annals of the Deaf, 156(1), 47–55. https://doi.org/10.1353/aad.2011.0013

27.

Verhavert

Bouwer

Donche

De Maeyer

(2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy and Practice, 26(5), 541–562. https://doi.org/10.1080/0969594X.2019.1602027

28.

Verhavert

De Maeyer

Donche

Coertjens

(2018). Scale separation reliability: What does it mean in the context of comparative judgment? Applied Psychological Measurement, 42(6), 428–445. https://doi.org/10.1177/0146621617748321

29.

Wang

J.-H.

Napier

Goswell

Carmichael

(2015). The design and application of rubrics to assess signed language interpreting performance. The Interpreter and Translator Trainer, 9(1), 83–103. https://doi.org/10.1080/1750399X.2015.1009261

30.

Weir

C. J.

(2005). Language testing and validation: An evidence-based approach. Palgrave Macmillan. https://doi.org/10.1057/9780230514577

31.

Witter-Merithew

Taylor

Johnson

(2001). Guided self-assessment and professional development planning: A model applied to interpreters in educational settings. In Nettles

(Ed.), Proceedings of the 17th National Conference of the Registry of Interpreters for the Deaf (pp. 153–226). RID Publications. https://www.unco.edu/cebs/asl-interpreting/pdf/library/guided-self-assessment-educational.pdf

32.

Wright

B. D.

Linacre

J. M.

(1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370. https://rasch.org/rmt/rmt83b.htm

33.

Xiao

X.-Y.

F.-Y.

(2013). Sign language interpreting on Chinese TV: A survey on user perspectives. Perspectives, 21(1), 100–116. https://doi.org/10.1080/0907676X.2011.632690

34.

Xiao

X.-Y.

R.-L.

(2009). Survey on sign language interpreting in China. Interpreting, 11(2), 137–163. https://doi.org/10.1075/intp.11.2.03xia

35.

Xiao

X.-Y.

Chen

X.-Y.

Palmer

J. L.

(2015). Chinese Deaf viewers’ comprehension of sign language interpreting on television: An experimental study. Interpreting, 17(1), 91–117. https://doi.org/10.1075/intp.17.1.05xia

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.49 MB

0.24 MB

0.50 MB

0.32 MB