Language Testing

This study examines the Test of Indonesian Proficiency (Uji Kemahiran Berbahasa Indonesia [UKBI]), a national language test developed by the Indonesian government’s language planning agency, Badan Bahasa, and promoted for various uses across education, employment, and immigration domains. This is illustrative of broader trends; language tests, like UKBI, have become integral to policy agendas and can serve multiple social and political purposes, many extending well beyond test design intentions. Research is urgently needed to examine how government agencies and other test users rationalize the need for and the legitimacy of policy-based test uses, as a means of promoting public acceptance and support, particularly where an empirical basis for test uses is lacking. We address this by identifying claims made by Badan Bahasa about UKBI scores, uses and consequences across policy documents, media releases, and website materials. We then combine Van Leeuwen’s language of legitimation framework and Reisigl and Wodak’s discourse historical approach to critically examine the discursive strategies engaged by Badan Bahasa to promote the legitimacy of the UKBI and its uses. The study sheds light on the social values and ideologies underlying testing practices, providing new insights into how tests as policy instruments come to be imbued with meaning and authority

This study explores the integration of generative artificial intelligence (GenAI) with human experts to improve the quality of distractors in multiple-choice questions (MCQs) for second language (L2) listening tests. A psychometric analysis of responses from 2267 EFL Chinese undergraduates, using the two-parameter logistic nested logit model (2PLNLM), identified problematic items and distractors. Guided by established distractor design principles, GenAI was applied iteratively to refine these distractors, and GenAI was iteratively used to revise these distractors, with human experts providing ongoing feedback throughout the process. The revised versions were then evaluated by expert judgment and NLP-based cosine similarity analysis. The results indicate that GenAI effectively enhanced distractor quality by maintaining content and structural alignment and ensuring semantic independence. However, it struggled to fully capture listening miscomprehension patterns and contextualized language use. These preliminary findings suggest that GenAI revisions, guided by principle-based prompts and supervised by humans, tend to effectively improve the quality of distractors. This study offers practical insights into the potential and limitations of GenAI in improving L2 listening tests.

This three-level meta-analysis investigated the human–machine correlation of automated speech evaluation (ASE) systems. Sixty-seven studies representing 392 effect sizes were included. The results indicated a positive overall correlation (r = .654, p < .001) between machine and human scoring of speech. Pooled effect sizes across speaking constructs showed the highest correlation for delivery (r = .784), followed by overall speaking proficiency (r = .686), fluency (r = .618), pronunciation (r = .606), content (r = .574), and grammar and vocabulary (r = .499). A clear upward trend was observed in ASE studies, from the traditional machine learning stage (r = .597), to the deep learning application stage (r = .641), and further to the transformer-driven stage (r = .680). Moderator analysis revealed significant moderating effects on the overall human–machine correlation from 10 variables: publication year, publication type, unit of sample, age group, level of task constraints, rater expertise, inter-rater reliability, system developer, feature engineering, and algorithm type. However, no significant moderating effects were observed for level of task integration, scoring method, system architecture, and automated speech recognition (ASR) accuracy. We propose key considerations for future ASE development, offering insights for educators and policymakers in integrating ASE into education.

Assessment rubrics have increasingly been developed and deployed to evaluate the quality of language interpreting, yet understanding of rubric-based interpreting assessment remains limited. This systematic review aims to: (a) catalog rubrics, (b) examine rubric design features, (c) understand rubric use, and (d) evaluate rubric utility. A rigorous review process, involving database searching, citation tracking, and targeted review of core literature, identified 80 unique rubrics comprising a total of 265 (sub-)scales. A comprehensive analysis revealed that: (a) among 11 potential sources informing rubric development, test-external sources (e.g., literature review) were primarily used, whereas test-internal sources (e.g., performance samples) were much less consulted; (b) assessments primarily used analytic and task-type rubrics with an average of five performance levels and four scoring criteria. Rubric descriptors generally incorporated observable indicators of interpreting quality, employing both descriptive and evaluative rubric language; (c) rubrics were used by three main types of raters—interpreting practitioners, trainers, and students—to assess primarily spoken-language interpreting; and (d) rubric-based scores demonstrated moderate-to-high reliability and validity overall, though meta-analysis identified three significant moderators, including correlation coefficient type, assessment criterion, and rubric length. These findings are expected to provide guidance for assessment practice and research in interpreting.

Over one million international students are enrolled in US higher education institutions. To be admitted into one of these degree-granting institutions, international students must demonstrate sufficient English language proficiency (ELP)–often with a test score. Previous research on ELP tests has shed light on how scores correlate with academic success and how institutions make decisions about minimum required scores, but the extent to which US institutions vary in their use of ELP tests in international admissions is not well understood. In this study, we examined ELP test policies across all US research-intensive, doctoral-granting institutions (Carnegie R1) for general undergraduate admissions and graduate admissions. Results show that 32 different ELP tests were being used for admissions. The most accepted ELP tests were TOEFL iBT, IELTS Academic, Duolingo English Test, and Pearson Test of English Academic (in that order). A wide range of unconditional and conditional cut scores was found across institutions. Subscore requirements were not common. Institutions requiring higher ELP cut scores (specifically TOEFL iBT) tended to be more prestigious (higher ranked) and selective (admitting smaller proportions of undergraduate applicants).