DPTCN: A novel deep CNN model for short text classification

Abstract

As an important branch of Nature Language Processing (NLP), how to extract useful text information and effective long-range associations has always been a bottleneck for text classification. With the great effort of deep learning researchers, deep Convolutional Neural Networks (CNNs) have made remarkable achievements in Computer Vision but still controversial in NLP tasks. In this paper, we propose a novel deep CNN named Deep Pyramid Temporal Convolutional Network (DPTCN) for short text classification, which is mainly consisting of concatenated embedding layer, causal convolution, 1/2 max pooling down-sampling and residual blocks. It is worth mentioning that our work was highly inspired by two well-designed models: one is temporal convolutional network for sequential modeling; another is deep pyramid CNN for text categorization; as their applicability and pertinence remind us how to build a model in a special domain. In the experiments, we evaluate the proposed model on 7 datasets with 6 models and analyze the impact of three different embedding methods. The results prove that our work is a good attempt to apply word-level deep convolutional network in short text classification.

Keywords

Deep convolution network causal convolution shortcut connection short text classification

Get full access to this article

View all access options for this article.

References

Conneau

, Schwenk

, Barrault

and Lecun

, Very deep convolutional networks for natural language processing, arXiv 2 (2016), 1.

Graves

, Fernández

and Schmidhuber

, Bidirectional LSTM networks for improved phoneme classification and recognition, International conference on artificial neural networks, Springer, Berlin, Heidelberg, (2005), 799–804.

Khan

, Sohail

, Zahoora

and Qureshi

A.S.

, A survey of the recent architectures of deep convolutional neural networks, Artificial Intelligence Review 53(8) (2020), 5455–5516.

Krizhevsky

, Sutskever

and Hinton

G.E.

, Imagenet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems 25 (2020), 1097–1105.

Oord

A.V.D.

, Dieleman

, Zen

, Simonyan

, Graves

and Kavukcuoglu

, Wavenet: A generative model for raw audio, arXiv, 2016.

Pang

and Lee

, A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts, arXiv, 2004.

Pang

and Lee

, Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales, arXiv, 2005.

Lea

, Flynn

M.D.

, Vidal

and Hager

G.D.

, Temporal convolutional networks for action segmentation and detection, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 156–165.

Szegedy

, Liu

, Jia

, Sermanet

, Reed

, Anguelov

and Rabinovich

, Going deeper with convolutions, Proceedings of the IEEE conference on computer vision and pattern recognition, (2015), 1–9.

10.

Cao

, Huang

, Li

, Zhao

and Fu

, Text Sentiment Classification Based on LSTM-TCN Hybrid Model and Attention Mechanism, Proceedings of the 4th International Conference on Computer Science and Application Engineering, (2020), 1–5.

11.

and Koltun

, Multi-scale context aggregation by dilated convolutions, arXiv, 2015.

12.

Huang

, Liu

, Van Der Maaten

and Weinberger

K.Q.

, Densely connected convolutional networks, Proceedings of the IEEE conference on computer vision and pattern recognition, (2017), 4700–4708.

13.

H.T.

, Cerisara

and Denis

, Do convolutional networks need to be deep for text classification? Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

14.

Han

, Chen

, Liu

and Peng

, Chinese text sentiment classification based on bidirectional time-depth convolutional network, Computer Applications and Software, 2019.

15.

Wiebe

, Wilson

and Cardie

, Annotating expressions of opinions and emotions in language, Language Resources and Evaluation 39(2) (2005), 165–210.

16.

, Zhang

, Ren

and Sun

, Deep Residual Learning for Image Recognition, Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), 770–778.

17.

, Zhang

, Ren

and Sun

, Identity mappings in deep residual networks, European conference on computer vision, Springer, Cham, (2016), 630–645.

18.

Jin

, Wang

, Li

, Liu

and Sun

, Dockless Shared-Bike Demand Prediction with Temporal Convolutional Networks, CICTP, (2020), 2851–2863.

19.

Simonyan

and Zisserman

, Very deep convolutional networks for large-scale image recognition, arXiv, 2014.

20.

Chen

L.C.

, Papandreou

, Kokkinos

, Murphy

and Yuille

A.L.

, Semantic image segmentation with deep convolutional nets and fully connected crfs, arXiv, 2014.

21.

and Niu

, TCN-Based Short Term Interest Personalized Sequence Recommendation, 2019.

22.

Holschneider

, Kronland-Martinet

, Morlet

and Tchamitchian

, Areal-time algorithmfor signal analysis with the help of the wavelet transform, Wavelets, Springer, Berlin, Heidelberg, (1990), 286–297.

23.

and Liu

, Mining and summarizing customer reviews, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and datamining, (2004), 168–177.

24.

Dutilleux

, An implementation of the “algorithme à trous” to compute the wavelet transform, Wavelets, Springer, Berlin, Heidelberg, 1990, 298–304.

25.

Collobert

, Weston

, Bottou

, Karlen

, Kavukcuoglu

and Kuksa

, Natural language processing (almost) from scratch, (ARTICLE), Journal of machine learning research 12 (2011), 2493–2537.

26.

Johnson

and Zhang

, Convolutional neural networks for text categorization: Shallow word-level vs. deep character-level, arXiv, 2016.

27.

Johnson

and Zhang

, Deep pyramid convolutional neural networks for text categorization, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (2017), 562–570.

28.

Johnson

and Zhang

, Semi-supervised convolutional neural networks for text categorization via region embedding, Advances in Neural Information Processing Systems 28 (2015), 919.

29.

Socher

, Perelygin

, Wu

, Chuang

, Manning

C.D.

, Ng

A.Y.

and Potts

, Recursive deep models for semantic compositionality over a sentiment treebank, Proceedings of the 2013 conference on empirical methods in natural language processing, (2013), 1631–1642.

30.

Bai

, Kolter

J.Z.

and Koltun

, An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv, 2018.

31.

Hochreiter

and Schmidhube

, Long short-term memory, Neural computation 9(8) (1997), 1735–1780.

32.

Lai

, Xu

, Liu

and Zhao

, Recurrent convolutional neural networks for text classification, Proceedings of the AAAI Conference on Artificial Intelligence, 2015.

33.

Wang

, Huang

and Deng

, Densely Connected CNN with Multi-scale Feature Attention for Text Classification, IJCAI, (2018), 4468–4474.

34.

Mikolov

, Sutskever

, Chen

, Corrado

G.S.

and Dean

, Distributed representations of words and phrases and their compositionality, Advances in Neural Information Processing Systems, (2013), 3111–3119.

35.

and Roth

, Learning question classifiers, COLING 2002: The 19th International Conference on Computational Linguistics. 2002.

36.

Kim

, Convolutional neural networks for sentence classification, arXiv, 2014.

37.

Lecun

, Bottou

, Bengio

and Haffner

, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86(11) (1998), 2278–2324.

38.

Zuo

, Jiang

, Sun

, Ma

, Liang

, Nie

and Zhou

, Short text classification based on bidirectional TCN and attention mechanism, Journal of Physics: Conference Series, IOP Publishing 1693(1) (2020), 012067.

39.

Shi

, Lin

, Liu

, Han

and Shi

, Deep Attention Gated Dilated Temporal Convolutional Networks with Intra-Parallel Convolutional Modules for End-to-End Monaural Speech Separation, Interspeech (2019), 3183–3187.