Bilingual generated text detection through semantic and statistical analysis

Abstract

The release of Large Language Models (LLMs) has achieved human-level text generation, leading to malicious uses such as disinformation propagation and academic dishonesty. Existing research has faced substantial challenges in low detection rates and poor generalization on multilingual generated text and short text. To fill these gaps, in this paper, we propose a generic bilingual generated text detection model to integrate semantic and statistical features, which exhibits proficiency in English and Chinese. To obtain fine-grained features, we employ the multilingual pre-trained language model xlm-RoBERTa to extract the CLS vector as overall semantic features, integrating with statistical features log rank, probability, and cumulative probability for detection. Moreover, Shapley additive explanations (SHAP) serves to interpret the decision-making process. The experimental results demonstrate significant advancements over baselines, notably with the F1 score improvements exceeding 10% and 5% on the English and Chinese HC3 sentence-level datasets, respectively. Our proposed method exhibits higher generalization for advanced LLMs and out-of-domain datasets with a 91.13% F1 score, thereby providing a more robust solution for detecting generated text.