Design of knowledge incorporated VQA based on spatial GCNN with structured sentence embedding and linking algorithm

Abstract

Visual question Answering (VQA) is a computer vision task that requires a system to infer an answer to a text-based question about an image. Prior approaches did not take into account an image’s positional information or the questions’ grammatical and semantic relationships during image and question processing. Featurization, which leads to the false answering of the question. Hence to overcome this issue CNN –Graph based LSTM with optimized BP Featurization technique is introduced for feature extraction of image as well as question. The position of the subjects in the image has been determined using CNN with a dropout layer and the optimized momentum backpropagation during the extraction of image features without losing any image data. Then, using a graph-based LSTM with loopy backpropagation, the questions’ syntactic and semantic dependencies are retrieved. However, due to their lack of external knowledge about the input image, the existing approaches are unable to respond to common sense knowledge-based questions (open domain). As a result, the proposed Spatial GCNN knowledge retrieval with PDB Model and Spatial Graph Convolutional Neural Network, which recovers external data from Wikidata, have been used to address the open domain problems. Then the Probabilistic Discriminative Bayesian model, based Attention mechanism predicts the answer by referring to all concepts in question. Thus, the proposed method answers the open domain question with high accuracy of 88.30%.

Keywords

Visual Question Answering graph-based LSTM SVO triples sentence Discriminative Bayesian model dynamic memory network

Get full access to this article

View all access options for this article.

References

François Garderes

et al. Conceptbert: Concept-aware representation for visual question answering, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, (2020).

Ruixue Tang

et al. Semantic equivalent adversarial data augmentation for visual question answering, European Conference on Computer Vision, Springer, Cham, (2020).

Yash Srivastava

et al. Visual question answering using deep learning: A survey and performance analysis, International Conference on Computer Vision and Image Processing, Springer, Singapore, (2020).

Franklin Kenghagho Kenfack

et al. RobotVQA—A Scene-Graph-and Deep-Learning-based Visual Question Answering System for Robot Manipulation, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (2020).

Minh Vu

et al. A question-centric model for visual question answering in medical imaging, IEEE Transactions on Medical Imaging 39(9) (2020), 2856–2868.

Zihan Guo , Dezhi Han , Kuan-Ching Li Double-layer affective visual question answering network, Computer Science and Information Systems (2020), 38–38.

Jae Sung Park

et al. Visualcomet: Reasoning about the dynamic context of a still image, European Conference on Computer Vision, Springer, Cham (2020).

Mathilde Caron

et al. Unsupervised learning of visual features by contrasting cluster assignments, arXiv preprint arXiv:2006.09882, (2020).

Deng Huang

et al. Location-aware graph convolutional networks for video question answering, Proceedings of the AAAI Conference on Artificial Intelligence 34(07) (2020).

10.

Qingbao Huang

et al. Aligned dual channel graph convolutional network for visual question answering, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, (2020).

11.

Man Luo

et al. Weakly-Supervised Visual-Retriever Reader for Knowledge-based Question Answering, arXiv preprint arXiv:2109.04014, (2021).

12.

Liyang Zhang

et al. Rich Visual Knowledge-Based Augmentation Network for Visual Question Answering, IEEE Transactions on Neural Networks and Learning Systems (2020).

13.

Aman Jain

et al. Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering, arXiv preprint arXiv:2103.05568, (2021).

14.

Zhen Huang

et al. Recent trends in deep learning based open-domain textual question answering systems, , IEEE Access 8 (2020), 94341–94356.

15.

Danilo Dessì

et al. Generating knowledge graphs by employing Natural Language Processing and Machine Learning techniques within the scholarly domain, , Future Generation Computer Systems 116 (2021), 253–264.

16.

Michihiro Yasunaga

et al. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering, arXiv preprint arXiv:2104.06378, (2021).

17.

Zhou Su

et al. Learning visual knowledge memory networks for visual question answering, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018).

18.

Zhiyang Ma

et al. Joint embedding VQA model based on dynamic word vector,, PeerJ Computer Science 7 (2021), e353.

19.

Raeid Saqur , Karthik Narasimhan Multimodal graph networks for compositional generalization in visual question answering, Advances in Neural Information Processing Systems (2020).

20.

Fuji Ren , Yangyang Zhou Cgmvqa: A new classification and generative model for medical visual question answering, IEEE Access 8 (2020), 50626–50636.

21.

Yuling Xi

et al. Visual question answering model based on visual relationship detection, , Signal Processing: Image Communication 80 (2020), 115648.

22.

Wenya Guo

et al. Re-attention for visual question answering, , IEEE Transactions on Image Processing 30 (2021), 6730–6743.

23.

Deepak Gupta , Swati Suman , Asif Ekbal , Hierarchical deep multi-modal network for medical visual question answering, , Expert Systems with Applications 164 (2021), 113993.

24.

Sharma

and Jalal

A.S.

, Visual question answering model based on graph neural network and contextual attention, , Image and Vision Computing 110 (2021), 104165.

25.

Shengyan Liu

et al. BPI-MVQA: a bi-branch model for medical visual question answering, BMC Medical Imaging 22(1) (2022), 1–19.

26.

Lobry

, Demir

, Tuia

RSVQA meets BigEarth Net: a new, large-scale, visual question answering dataset for remote sensing, In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, IEEE, (2021, July), 1218–1221.

27.

Narayanan

, Rao

, Prasad

and Natarajan

, VQA as a factoid question answering problem: A novel approach for knowledge-aware and explainable visual question answering, Image and Vision Computing 116 (2021), 104328.

28.

Lezama-Sánchez

A.L.

, Tovar Vidal

and Reyes-Ortiz,

J.A.

, An Approach Based on Semantic Relationship Embeddings for Text Classification, Mathematics 10 (2022), 4161. Retrieved from: http://dx.doi.org/10.3390/math10214161

29.

Guohao Li , Hang Su , Wenwu Zhu incorporating external knowledge to answer open-domain visual questions with dynamic memory networks, arXiv preprint arXiv:1712.00733, (2017).

30.

Daniel Berrar , , Bayes’ theorem and naive Bayes classifier, Encyclopedia of Bioinformatics and Computational Biology: ABC of Bioinformatics 403 (2018).

31.

Xiong

, Merity

and Socher

, Dynamic memory networks for visual and textual question answering, In, International Conference on Machine Learning (2016), 2397–2406.

32.

Drew Hudson

, Christopher Manning

Gqa: a new dataset for compositional question answering over real world images, arXiv preprint arXiv:1902.09506, (2019).

33.

Ronghang Hu, , Anna Rohrbach , Trevor Darrell , Kate Saenko Language-conditioned graph networks for relational reasoning, arXiv preprint arXiv:1905.04405, (2019).

34.

Yangyang Cheng , Chun Yuan Reasoning-Aware Graph Convolutional Network for Visual Question Answering, (2019).