An Iterative Multi-aspect term Extraction and Polarity Detection Approach Based on spanBERT for Aspect-based Sentiment Analysis

Abstract

Aspect-based sentiment analysis (ABSA) is a fine-grained analysis of sentiments and opinions expressed in text document which set aside the naïve assumption that a document expresses opinion about a single topic or aspect. The main objective of ABSA is to spot, extract and identify the polarity of different entities and aspects in an opinionated document. Based on the previous works, ABSA can be categorized into three subtasks: Aspect-category sentiment analysis (ACSA), Opinion Target Expression Sentiment Analysis (OTESA) and Aspect-term sentiment analysis (ATSA). This research presents an end-to-end multi-task approach to performing the three categories of ABSA on a single pipeline. A ternary multitask learning objectives classifiers were built on top of the baseline spanBERT language model which was originally pretrained for span extraction. The input to the model consists of two merged segments of entity premises and context data hypothesis in a similar passion to reading comprehension downstream task in natural language processing. The ternary downstream tasks were built on the contextualized output embeddings of pretrained spanBERT entangled with cross-layer attention mechanism to associate context with the aspect-term span extraction, aspect sentiment polarity detection and entity-aspect entailment. A span masking approach was also proposed to address multiple-aspects text using an iterative outputs-inputs loopback. The span masking process replaces each word in a previously detected span of text with a special [MASK] character and then feeds back the entire sentence into the encoder input of the model for next run. The technique forces the encoder to look elsewhere for the next span prediction. The loopback span masking terminates when the span classifiers predict a special token [CLS] as the beginning and end of the span signaling the absent of relatable span to be extracted. Experimental results validate the approach as impressive results were obtained outperforming most of the compared research with benchmark ABSA datasets.

Keywords

BERT spanBERT aspect-based sentiment analysis sentiment analysis

Get full access to this article

View all access options for this article.

References

Ashir

A. M.

Taha

M. A.

(2024). BERT with an augmented cross-attention decoder (BERT-ACD) for binary and fine-grained multiband sentiment detection. Intelligenza Artificiale, 19(1), 3–16. https://doi.org/10.1177/17248035241291687

Augustyniak

Kajdanowicz

Kazienko

(2021). Comprehensive analysis of aspect term extraction methods using various text embeddings. Comput Speech Lang, 69(1). https://doi.org/10.1016/j.csl.2021.101217

Bahdanau

Cho

Bengio

(2016). Neural Machine Translation by Jointly Learning to Align and Translate. CoRR, abs/1409.0473.

Bollegala

Weir

Carroll

(2013). Cross-domain sentiment classification using a sentiment sensitive thesaurus. IEEE Trans Knowl Data Eng, 25(8), 1719–1731. https://doi.org/10.1109/TKDE.2012.103

Bordoloi

Biswas

S. K.

(2023). Sentiment analysis: A survey on design framework, applications and future scopes. Artif Intell Rev. https://doi.org/10.1007/s10462-023-10442-2

Caruana

(1997). Multitask learning. Mach Learn, 28(1), 41–75. https://doi.org/10.1023/A:1007379606734

Chen

Sun

Bing

Yang

(2017). Recurrent attention network on memory for aspect sentiment analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (pp. 452–461). Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1047

Devlin

Chang

M.-W.

Lee

Google

K. T.

Language

A. I.

(2019). BERT: Pre-training of deep bidirectional transformers for language understanding. https://doi.org/10.18653/v1/N19-1423

Gao

Glowacka

(2016). Deep gate recurrent neural network. In Durrant

R. J.

Kim

K.-E.

(Eds.), Proceedings of the 8th Asian conference on machine learning, Proceedings of Machine Learning Research, vol. 63. The University of Waikato (pp. 350–365). PMLR. [Online]. Available: https://proceedings.mlr.press/v63/gao30.html

10.

Gou

Guo

Yang

(2023). MvP: Multi-view prompting improves aspect sentiment tuple prediction.

11.

Lee

W. S.

H. T.

Dahlmeier

(2018). Exploiting document knowledge for aspect-level sentiment classification. In Gurevych

Miyao

(Eds.), Proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: Short papers) (pp. 579–585). Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-2092

12.

Lee

W. S.

H. T.

Dahlmeier

(2019). An interactive multi-task learning network for End-to-End aspect-based sentiment analysis. In Korhonen

Traum

Màrquez

(Eds.), Proceedings of the 57th annual meeting of the association for computational linguistics (pp. 504–515). Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1048

13.

Hochreiter

Schmidhuber

(1997). Long short-term memory. Neural Comput., 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

14.

Peng

Huang

(2019). Open-domain targeted sentiment analysis via span-based extraction and classification. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 537–546). Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1051

15.

Joshi

Chen

Liu

Weld

D. S.

Zettlemoyer

Levy

(2020). SpanBERT: Improving pre-training by representing and predicting spans. Trans Assoc Comput Linguist, 8, 64–77. https://doi.org/10.1162/tacl_a_00300

16.

Lee

Kwiatkowski

Parikh

A. P.

Das

(2016) Learning recurrent span representations for extractive question answering. CoRR, abs/1611.01436. https://doi.org/10.48550/arXiv.1611.01436

17.

Bing

Lam

(2019a). A unified model for opinion target extraction and target sentiment prediction. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, in AAAI’19/IAAI’19/EAAI’19. AAAI Press. https://doi.org/10.1609/aaai.v33i01.33016714

18.

Bing

Lam

(2019b). A unified model for opinion target extraction and target sentiment prediction. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 6714–6721. https://doi.org/10.1609/aaai.v33i01.33016714

19.

Bing

Zhang

Lam

(2019c). Exploiting BERT for end-to-end aspect-based sentiment analysis. Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019) (pp. 34–41). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-5505

20.

Y.-M.

T.-Y.

(2013). Deriving market intelligence from microblogs. Decis Support Syst, 55(1), 206–217. https://doi.org/10.1016/j.dss.2013.01.023

21.

(2017). Learning latent sentiment scopes for entity-level sentiment analysis. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, in AAAI’17 (pp. 3482–3489). AAAI Press.

22.

Xiao

(2020). Syrapropa at SemEval-2020 task 11: BERT-based models design for propagandistic technique and span detection. Proceedings of the Fourteenth Workshop on Semantic Evaluation (pp. 1808–1816). International Committee for Computational Linguistics. https://doi.org/10.18653/v1/2020.semeval-1.237

23.

Xie

Chen

Wang

Deng

(2014). News impact on stock price return via sentiment analysis. Knowl Based Syst, 69, 14–23. https://doi.org/10.1016/j.knosys.2014.04.022

24.

Liang

Meng

Zhang

Chen

Zhou

(2020) A dependency syntactic knowledge augmented interactive architecture for end-to-end aspect-based sentiment analysis.

25.

Liu

Sun

(2020). Jointly modeling aspect and sentiment with dynamic heterogeneous graph neural networks.

26.

Liu

Zhang

(2012). A survey of opinion mining and sentiment analysis. In Aggarwal

Zhai

(Ed.), Mining text data (pp. 415–463). Springer US. https://doi.org/10.1007/978-1-4614-3223-4_13

27.

Mao

Shen

Yang

Zhu

Cai

(2022). Seq2Path: Generating sentiment tuples as paths of a tree. In Muresan

Nakov

Villavicencio

(Eds.), Findings of the association for computational linguistics: ACL 2022 (pp. 2215–2225). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.findings-acl.174

28.

Nguyen

M. L.

(2018). A deep neural architecture for sentence-level sentiment classification in twitter social networking. In Hasida

(Eds.), Computational linguistics (pp. 15–27). Springer Singapore.

29.

Pang

Lee

Vaithyanathan

(2002). Thumbs up? Sentiment classification using machine learning techniques. Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002) (pp. 79–86). Association for Computational Linguistics. https://doi.org/10.3115/1118693.1118704

30.

Peters

M. E.

, Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyeret, L. (2018). Deep contextualized word representations. https://doi.org/10.48550/arXiv.1802.05365

31.

Pontiki

, et al. (2016). SemEval-2016 task 5: Aspect based sentiment analysis. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) (pp. 19–30). Association for Computational Linguistics. https://doi.org/10.18653/v1/S16-1002

32.

Quan

Ren

(2014). Unsupervised product feature extraction for feature-oriented opinion determination. Inf Sci (N Y), 272, 16–28. https://doi.org/10.1016/j.ins.2014.02.063

33.

Rajpurkar

Zhang

Lopyrev

Liang

(2016). SQuAD: 100,000+ questions for machine comprehension of text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 2383–2392). Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1264

34.

Rei

(2017). Semi-supervised multitask learning for sequence labeling. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2121–2130). Association for Computational Linguistics. https://doi.org/10.18653/v1/P17-1194

35.

Seo

M. J.

Kembhavi

Farhadi

Hajishirzi

(2017). Bidirectional attention flow for machine comprehension. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net. [Online]. Available: https://openreview.net/forum?id=HJ0UKP9ge

36.

Søgaard

Goldberg

(2016). Deep multi-task learning with low level tasks supervised at lower layers. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 231–235). Association for Computational Linguistics. https://doi.org/10.18653/v1/P16-2038

37.

Tang

Qin

Feng

Liu

(2016). Effective LSTMs for target-dependent sentiment classification. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3298–3307). The COLING 2016 Organizing Committee. [Online]. Available: https://aclanthology.org/C16-1311

38.

Vaswani

, et al. (June 2017) Attention Is All You Need. [Online]. Available: http://arxiv.org/abs/1706.03762

39.

Wang

Huang

Zhu

Zhao

(2016). Attention-based LSTM for aspect-level sentiment classification. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 606–615). Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1058

40.

Wang

Jiang

(2017). Machine comprehension using match-LSTM and answer pointer. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net. [Online]. Available: https://openreview.net/forum?id=B1-q5Pqxl

41.

Wang

Zhang

Jin

Sun

(2021). End-to-end aspect-based sentiment analysis with hierarchical multi-task learning. Neurocomputing, 455, 178–188. https://doi.org/10.1016/j.neucom.2021.03.100

42.

Wilson

Wiebe

Hoffmann

(2005). Recognizing contextual polarity in phrase-level sentiment analysis. In Mooney

Brew

Chien

L.-F.

Kirchhoff

(Eds.), Proceedings of human language technology conference and conference on empirical methods in natural language processing (pp. 347–354). Association for Computational Linguistics. [Online]. Available: https://aclanthology.org/H05-1044

43.

Liu

Shu

P. S.

(2018) Double embeddings and CNN-based sequence labeling for aspect extraction. In Gurevych

Miyao

(Eds.), Proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: Short papers) (pp. 592–598). Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-2094

44.

Yang

Dai

Yang

Carbonell

Salakhutdinov

Q. V.

(2019). XLNet: Generalized autoregressive pretraining for language understanding. Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc..

45.

Yang

Salakhutdinov

Cohen

W. W.

(2017). Transfer learning for sequence tagging with hierarchical recurrent networks.

46.

Zhang

Feng

Meng

You

Liu

(2019). Bridging the gap between training and inference for neural machine translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 4334–4343). Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1426

47.

Zhang

Deng

Bing

Lam

(2023). A survey on aspect-based sentiment analysis: Tasks, methods, and challenges. IEEE Trans Knowl Data Eng, 35(11), 11019–11038. https://doi.org/10.1109/TKDE.2022.3230975

48.

Zhang

D.-T.

(2016). Gated neural networks for targeted sentiment analysis. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, in AAAI’16 (pp. 3087–3093). AAAI Press.