Multimodal Emotion Recognition Using Taylor Cooperative Share Hunt Optimized Deep Quantum Neural Network

Abstract

Multimodal emotion recognition comprises speech utterances with corresponding images in a discrete form that strengthens the combined knowledge, captures the importance of image and speech features, and recognizes particular emotions accurately. Several studies have been undertaken in audio–visual model emotion recognition and faced specific challenges such as missing features, misalignment of modalities, inefficient generalization, and inaccurate detection results. To solve these problems, a Taylor Cooperative Share Hunt Optimizer with Deep Quantum neural network (TCoSH–DQNN) is proposed, in which both speech utterance and facial expressions are effectively combined. Furthermore, the proposed model achieves effective multimodal emotion recognition with the combined representation of audio and visual features extracted from videos. The proposed TCoSH–DQNN classifier extracts the human expression in an informative aspect with enhanced performance and improved recognition accuracy. Additionally, the TCOSH–DQNN model generates better communication by resolving conflicts and effectively recognizing feelings. In this perspective, the model reduces the computational complexity and achieves effective recognition with a better convergence rate. The effectiveness of the model is improved by validating the dataset and achieving significant evaluation metric outcomes with an accuracy of 99.97%, a Cohen kappa score of 98.61%, a recall of 99.70%, an F1-score of 99.81%, a f-beta score of 98.67%, and a precision of 99.96%, with the evaluation of the Enterface’05 dataset.