With the wide application of attention mechanism in multitudinous field of natural language processing (NLP), to date various deep neural networks based on this mechanism have been introduced and developed. However, a major problem with this kind of application is that a long time will be consumed due to the current networks still need to rely on their own ability to form attention values from scratch during the training. In this paper, we propose an auxiliary method called the Guided Attention Mechanism (GAM), which utilizes the prior knowledge to guide the network to form attention values in NLP field, thereby shortening the network training time and making the attention values more accurate. This work designed two sets of prior knowledge generation processes based on the regularization method and the deep learning method respectively. And the prior knowledge is used to guide the attention values of the original network in terms of values and angles. The experimental results show that compared with the original network, the classification accuracy of the network using GAM is improved by about 2%, and the training time is reduced by 5∼9%.
KrizhevskyA., SutskeverI. and HintonG.E., Imagenet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems (2012), 1097–1105.
2.
GirshickR., DonahueJ., DarrellT. and MalikJ., Rich feature hierarchies for accurate object detection and semantic segmentation, Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (2014), 580–587.
3.
ChoK., Van MerriënboerB., GulcehreC., BahdanauD., BougaresF., SchwenkH. and BengioY., Learning phrase representations using RNN encoder-decoder for statistical machine translation, arXiv preprint arXiv:1406.1078 (2014).
4.
RubioJ.d.J., Ricardo CruzD., EliasI., OchoaG., BalcazarandR. and AguilarA., ANFIS system for classification of brain signals, Journal of Intelligent & Fuzzy Systems37(3) (2019), 1–9.
5.
de Jesús RubioJ., LughoferE., Meda-CampañaJ.A., PáramoL.A., Novoa J.F. and PachecoJ., Neural network updating via argument Kalman filter for modeling of Takagi-Sugeno fuzzy models, Journal of Intelligent & Fuzzy Systems35(2) (2018), 1–12.
6.
HochreiterS. and SchmidhuberJ., Long short-term memory, Neural Computation9 (1997), 1735–1780.
7.
ChoK., Van MerriënboerB., BahdanauD. and BengioY., On the properties of neural machine translation: Encoder-decoder approaches, arXiv preprint arXiv:1409.1259 (2014).
8.
MikolovT., SutskeverI., ChenK., CorradoG.S. and DeanJ., Distributed representations of words and phrases and their compositionality, Advances in Neural Information Processing Systems (2013), 3111–3119.
9.
PenningtonJ., SocherR. and ManningC., Glove: Global vectors for word representation, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014), 1532–1543.
10.
BahdanauD., ChoK. and BengioY., Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473 (2014).
11.
SukhbaatarS., WestonJ. and FergusR., End-to-end memory networks, Advances in Neural Information Processing Systems (2015), 2440–2448.
12.
DuC. and HuangL., Text classification research with attention-based recurrent neural networks, International Journal of Computers Communications & Control13 (2018), 50–61.
13.
WangY., HuangM., ZhuX. and ZhaoL., Attention-based LSTM for aspect-level sentiment classification, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016), 606–615.
14.
WooS., ParkJ., LeeJ.-Y. and So KweonI., Cbam: Convolutional block attention module, Proceedings of the European Conference on Computer Vision (ECCV) (2018), 3–19.
15.
VaswaniA., ShazeerN., ParmarN., UszkoreitJ., JonesL., GomezA.N., KaiserŁ. and PolosukhinI., Attention is all you need, Advances in Neural Information Processing Systems (2017), 5998–6008.
16.
DevlinJ., ChangM.-W., LeeK. and ToutanovaK., Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
JiZ., LiS. and PangY., Fusion-Attention Network for person search with free-form natural language, Pattern Recognition Letters116 (2018), 205–211.
19.
LiuX., GengJ., LingH. and CheungY.-m., Attention guided deep audio-face fusion for efficient speaker naming, Pattern Recognition88 (2019), 557–568.
20.
JiZ., HeE., WangH. and YangA., Image-attribute reciprocally guided attention network for pedestrian attribute recognition, Pattern Recognition Letters120 (2019), 89–95.
21.
HuD., An introductory survey on attention mechanisms in nlp problems, Proceedings of SAI Intelligent Systems Conference (2019), 432–448.
22.
HermannK. M., KociskyT., GrefenstetteE., EspeholtL., KayW., SuleymanM. and BlunsomP., Teaching machines to read and comprehend, Advances in Neural Information Processing Systems (2015), 1693–1701.
23.
YangZ., YangD., DyerC., HeX., SmolaA. and HovyE., Hierarchical attention networks for document classification, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2016), 1480–1489.
24.
KuchaievO. and GinsburgB., Factorization tricks for LSTM networks, arXiv preprint arXiv:1703.10722 (2017).
25.
ShazeerN., MirhoseiniA., MaziarzK., DavisA., LeQ., HintonG. and DeanJ., Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, arXiv preprint arXiv:1701.06538 (2017).
26.
ChoiH., ChoK. and BengioY., Context-dependent word representation for neural machine translation, Computer Speech & Language45 (2017), 149–160.
27.
PetersM. E., NeumannM., IyyerM., GardnerM., ClarkC., LeeK. and ZettlemoyerL., Deep contextualized word representations, arXiv preprint arXiv:1802.05365 (2018).
28.
EsuliA. and SebastianiF., Sentiwordnet: A publicly available lexical resource for opinion mining, LREC6 (2006), 417–422.
29.
BaccianellaS., EsuliA. and SebastianiF., Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining, Lrec10 (2010), 2200–2204.
30.
ZhangX., ZhaoJ. and LeCunY., Character-level convolutional networks for text classification, Advances in Neural Information Processing Systems (2015), 649–657.
31.
LeeJ. Y. and DernoncourtF., Sequential short-text classification with recurrent and convolutional neural networks, arXiv preprint arXiv:1603.03827 (2016).
32.
PangB. and LeeL., Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales, Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (2005), 115–124.
33.
SocherR., PerelyginA., WuJ., ChuangJ., ManningC. D., NgA. and PottsC., Recursive deep models for semantic compositionality over a sentiment treebank, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (2013), 1631–1642.
34.
PangB. and LeeL., A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts, Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics (2004), 271.
35.
LiX. and RothD., Learning question classifiers, Proceedings of the 19th International Conference on Computational Linguistics1 (2002), 1–7.
36.
WiebeJ., WilsonT. and CardieC., Annotating expressions of opinions and emotions in language, Language Resources and Evaluation39 (2005), 165–210.
37.
LeiZ., YangY. and YangM., Sentiment lexicon enhanced attention-based LSTM for sentiment classification, Thirty-Second AAAI Conference on Artificial Intelligence (2018).
38.
ShenT., ZhouT., LongG., JiangJ., PanS. and ZhangC., Disan: Directional self-attention network for rnn/cnn-free language understanding, Thirty-Second AAAI Conference on Artificial Intelligence (2018).
39.
DuJ., GuiL., XuR. and HeY., A convolutional attention model for text classification, National CCF Conference on Natural Language Processing and Chinese Computing (2017), 183–195.
40.
LiY., GuoH., ZhangQ., GuM. and YangJ., Imbalanced text sentiment classification using universal and domain-specific knowledge, Knowledge-Based Systems160 (2018), 1–15.
41.
ZhouP., QiZ., ZhengS., XuJ., BaoH. and XuB., Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling, arXiv preprint arXiv:1611.06639 (2016).
42.
LeP. and ZuidemaW., Compositional distributional semantics with long short term memory, arXiv preprint arXiv:1503.02510 (2015).
43.
SocherR., HuvalB., ManningC.D. and NgA.Y., Semantic compositionality through recursive matrix-vector spaces, Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (2012), 1201–1211.
KalchbrennerN., GrefenstetteE. and BlunsomP., A convolutional neural network for modelling sentences, arXiv preprint arXiv:1404.2188 (2014).
46.
YinW. and SchützeH., Multichannel variable-size convolution for sentence classification, arXiv preprint arXiv:1603.04513 (2016).
47.
ZhangY., RollerS. and WallaceB., MGNC-CNN: A simple approach to exploiting multiple word embeddings for sentence classification, arXiv preprint arXiv:1603.00968 (2016).
48.
IyyerM., ManjunathaV., Boyd-GraberJ. and DauméH.III, Deep unordered composition rivals syntactic methods for text classification, Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing1 (2015), 1681–1691.
