Recurrent Neural Networks (RNN) have claimed to achieve the state of the arts results in some cases, better performances than humans could have, especially RNN – Long Short Term Memory (LSTM) and RNN – Bidirectional LSTM, Attention based LSTM encoder-decoder networks in the domains of Speech Recognition, Sequence Labeling, Text Classification, Image Caption Generation and many more. The main focus of this paper is to present here a simple LSTM – Bidirectional LSTM joint model for Intent Classification and Named Entity Recognition (NER) with and without Convolutional Neural Network (CNN) as feature extractor. The aim of this experiment is to improve the accuracy of the model through inducing information from a well-performing model on a particular task to another model in a joint model framework and conclude if is there any correlation that might aid in syntactic and semantic structural learning of the task through the application of learned weights. The comparative results of models with and without CNN as feature extractors prepended to the models are tabulated as well.
YoungT.HazarikaD.PoriaS. and CambriaE., Recent trends in deep learning based natural language processing, IEEE Computational Intelligence Magazine13(3) (2018), 55–75.
2.
MesnilG.HeX.DengL. and BengioY., Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding, In Interspeech, 2013 August, pp. 3771–3775.
3.
TangD.QinB. and LiuT., Document modeling with gated recurrent neural network for sentiment classification, In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp. 1422–1432.
4.
GravesA.MohamedA.R. and HintonG., Speech recognition with deep recurrent neural networks, in: Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, IEEE, 2013 May, pp. 6645–6649.
5.
MikolovT.KombrinkS.BurgetL.ČernockýJ. and KhudanpurS., Extensions of recurrent neural network language model, in: Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, IEEE, 2011 May, pp. 5528–5531.
6.
ZhangX. and WangH., A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding, In IJCAI, 2016 July, pp. 2993–2999.
7.
LiuP.QiuX. and HuangX., Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. 2016.
8.
MaM.ZhaoK.HuangL.XiangB. and ZhouB., Jointly Trained Sequential Labeling and Classification by Sparse Attention Neural Networks. arXiv preprint arXiv:1709.10191. 2017.
9.
JeongM. and LeeG.G., Triangular-chain conditional random fields, IEEE Transactions on Audio, Speech, and Language Processing16(7) (2008), 1287–1302.
10.
XuP. and SarikayaR., Convolutional neural network based triangular crf for joint intent detection and slot filling, in: Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, IEEE, 2013 December, pp. 78–83.
11.
TafforeauJ.BechetF.ArtièresT. and FavreB., Joint Syntactic and Semantic Analysis with a Multitask Deep Learning Framework for Spoken Language Understanding, In Interspeech, 2016, pp. 3260–3264.
12.
BengioY. and LecunY., Convolutional Networks for Images, Speech, and Time-Series. 1997.
13.
LiC.YangC. and JiangQ., The research on text clustering based on LDA joint model, In Journal of Intelligent & Fuzzy Systems32(5) (2017 April 24), 3655–3667.
14.
SanchezO. and SierraG., Joint sentiment topic model for objective text clustering, In Journal of Intelligent & Fuzzy Systems (2018 October 17), 1–10.
15.
SanthiB. and BrindhaG.R., Multinomial naive bayes using similarity based conditional probability, In Journal of Intelligent & Fuzzy Systems36(2) (2019 March 16), 1431–1441.
16.
KimY., Convolutional Neural Networks for Sentence Classification, arXiv:1408.5882. 2014 Sep.
17.
LaiS.XuL.LiuK. and ZhaoJ., Recurrent Convolutional Neural Networks for Text Classification, in: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015, pp. 2267–2273.
18.
ZhangY. and WallaceC.B., A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification, arXiv:1510.03820. 2016 Apr.
19.
YuanS. and WuX., Task-specific word identification from short texts using a convolutional neural network, In Intelligent Data Analytics22(3) (2018 May 7), 533–550.
20.
SeongO.H., A fuzzy convolutional neural network for text sentiment analysis, In Journal of Intelligent & Fuzzy Systems35(6) (2018 December 24), 6025–6034.
21.
JiangM.ZhangW.ZhangM.WuJ. and WenT., An LSTM-CNN attention approach for aspect-level sentiment classification, In Journal of Computational Methods in Sciences and Engineering (2019 March 4), 1–10.
22.
ZengD.DaiY.LiF.WangJ. and SangaiahA.K., Aspect based sentiment analysis by a linguistically regularized CNN with gated mechanism, In Journal of Intelligent and Fuzzy Systems (2019 April 1), 1–10.
23.
ChenY., ATIS Dataset Source, https://github.com/yvchen/ JointSLU/tree/master/data accessed on October 2018.
24.
KleinD.SmarrJ.NguyenH. and ManningC.D., Named entity recognition with character-level models, in: Proceedings of the Seventh Conference on Natural language learning at HLT-NAACL, Association for Computational Linguistics, Vol. 4, 2003 May, pp. 180–183.
25.
NguyenT.T.HwangD. and JungJ.J., Handling imbalanced classification problem: A case study on social media datasets. 2017 January 30.
26.
CollobertR. and WestonJ., A unified architecture for natural language processing: Deep neural networks with multitask learning, in: Proceedings of the 25th International Conference on Machine Learning, ACM, 2008 July, pp. 160–167.
27.
MikolovT.SutskeverI.ChenK.CorradoG.S. and DeanJ., Distributed representations of words and phrases and their compositionality, in: Advances in Neural Information Processing Systems, 2013, pp. 3111–3119.
28.
GreffK.SrivastavaR.K.KoutníkJ.SteunebrinkB.R. and SchmidhuberJ., LSTM: a search space odyssey, IEEE Transactions on Neural Networks and Learning Systems28(10) (2017), 2222–2232.
29.
SalehinejadH.BaarbeJ.SankarS.BarfettJ.ColakE. and ValaeeS., Recent Advances in Recurrent Neural Networks. arXiv preprint arXiv:1801.01078. 2017.
30.
ChoK.Van MerriënboerB.GulcehreC.BahdanauD.BougaresF.SchwenkH. and BengioY., Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. 2014.
31.
SrivastavaN.HintonG.KrizhevskyA.SutskeverI. and SalakhutdinovR., Dropout: a simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research15(1) (2014), 1929–1958.
32.
KohaviR., A study of cross-validation and bootstrap for accuracy estimation and model selection, In Ijcai14(2) (1995 August), 1137–1145.
33.
RuderS., An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747. 2016.
34.
JanochaK. and CzarneckiW.M., On loss functions for deep neural networks in classification. arXiv preprint arXiv:1702. 05659. 2017.
35.
VargheseA.SarangS.YadavV.KarotraB. and GandhiN., Bidirectional LSTM Joint Model for Intent Classification and Named Entity Recognition in Natural Language Understanding, in: The proceedings of 18th International Conference on Intelligent Systems Design and Applications (ISDA), 2018.
