BERT with an Augmented Cross-attention Decoder (BERT-ACD) for Binary and Fine-grained Multiband Sentiment Detection

Abstract

In contemporary times, research in sentiment analysis has taken deeper steps into a finer and more granular analysis, transcending beyond the traditional binary or ternary classification of sentiment/opinion into positive, negative, or neutral. With the increasing complexity and challenging nature of such tasks, large language models inspired by transformer architecture are frequently deployed to address such challenges. Despite recorded improvements, challenges in identifying different levels, strengths or bands of sentiment intensity and the aspect for which the sentiment is expressed remain unresolved. In this article, we propose a banded sentiment analysis system for categorizing texts into 7 meaningful and relatable bands of sentiment for modern applications. It is also capable of performing aspect-based sentiment analysis in the same pipeline. The system architecture is inspired by the transformer language model with a BERT-based encoder and a newly proposed cross-attention, non-autoregressive decoder with augmented inputs. The decoder receives an n-gram-based augmented input sequence embedding that is specifically extracted from the original input, which comprises a list of the subjects, descriptive phrases, and modification phrases that underscore cases of amplification or undertone in a sentence. Rule-based tree parsing was proposed for use with dependency parsing for the extraction of these augmented inputs for the cross-attention decoder. Extensive experiments were conducted under different architecture setups and conditions with popular sentiment analysis datasets (Amazon reviews 2023, IMDB Movies review, SST-5 and SST-2 datasets) to verify the efficacy of the system. Extended labeling was also performed on the SST-5 dataset to generate 7 sentiment classes with the help of GPT4 and Bard. Experiments validate the efficacy of the proposed models.