From manual retrieval to intelligent question answering: A four-stage evaluation of multilingual access in South African open scholarly repositories

Abstract

This article evaluates multilingual information access in three selected South African open scholarly repositories through a four-stage analytical heuristic: manual retrieval, keyword retrieval, multilingual semantic retrieval, and retrieval-augmented generation (RAG)-based question answering. The framework serves as an organizing device for comparing access mechanisms across conceptual, retrieval, and answer-generation layers. Using 4944 open scholarly records, 400 English queries, 400 machine-translated isiZulu queries, and 800 relevance judgments, the study compares BM25 keyword retrieval, multilingual-E5-large semantic retrieval, and a Qwen2.5-based RAG pipeline. The isiZulu component is interpreted as a translated-query benchmark rather than as evidence of authentic isiZulu user search behaviour. The results show that English retrieval is already strong under BM25, whereas translated isiZulu queries benefit more clearly from multilingual semantic retrieval, especially at Top-10. However, these gains remain bounded by the three-repository case-study corpus and do not remove cross-language access gaps. The RAG stage does not improve retrieval ranking beyond dense retrieval, but it changes the form of service by transforming retrieved evidence into natural-language answers. Answer evaluation indicates that larger generator scale improves faithfulness and evidence support, while completeness and isiZulu language quality remain limited. The findings suggest that multilingual semantic retrieval can partly reduce translated-query barriers in open repository access, but RAG-based library services require transparent evidence display, uncertainty signalling, and professional governance.