Construction of a multimodal dialect corpus based on deep learning and digital twin technology: A case study on the Hangzhou dialect

Abstract

Focused on the digital preservation and inheritance of dialects, this study illustrates the construction pathway of a digitised multimodal dialect corpus and the application of a dialect interactive learning model using the digital twin technology, taking the Hangzhou dialect as a representative. Initially, multimodal resources of the Hangzhou dialect were collected, and with the aid of digital techniques, these resources underwent annotation, segmentation, transcription, and synchronisation, culminating in the creation of the multimodal dialect corpus. Subsequently, features were extracted using Natural Language Processing (NLP) methodologies from deep learning, facilitating the construction of a Hangzhou dialect lexicon. With the annotated corpus as the foundation, combined with Feedforward Sequential Memory Networks (FSMN) and Long Short-Term Memory (LSTM) networks, acoustic and linguistic models for the Hangzhou dialect were developed, laying the groundwork for a Hangzhou dialect speech recognition system. Conclusively, by integrating digital twin technology, an autonomous dialect inheritance learning model was crafted. This model establishes a twin learning space and learning twin entity founded on auditory, visual, and tactile multimodal information. Utilising virtual reality technology, a dialect learning ecological model was designed to enhance learner agency, offering diverse learning modalities and personalised content, with the overarching goal of supporting the preservation and inheritance of dialects.