Natural Language Processing problems has recently been benefited for the advances in Deep Learning. Many of these problems can be addressed as a multi-label classification problem. Usually, the metrics used to evaluate classification models are different from the loss functions used in the learning process. In this paper, we present a strategy to incorporate evaluation metrics in the learning process in order to increase the performance of the classifier according to the measure we are interested to favor. Concretely, we propose soft versions of the Accuracy, micro-F1, and macro-F1 measures that can be used as loss functions in the back-propagation algorithm. In order to experimentally validate our approach, we tested our system in an Emotion Classification task proposed at the International Workshop on Semantic Evaluation, SemEval-2018. Using a Convolutional Neural Network trained with the proposed loss functions we obtained significant improvements both for the English and the Spanish corpora.
BaccianellaS., EsuliA. and SebastianiF., Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining, In in Proc of LREC, 2010.
2.
BilmesJ., AsanovicK., ChinC.-W. and DemmelJ., Using phipac to speed error back-propagation learning, In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 5, 1997, pp. 4153–4156.
3.
CruzF.L., TroyanoJ.A., PontesB. and OrtegaF.J., Building layered, multilingual sentiment lexicons at synset and lemma levels, Expert Systems with Applications41(13) (2014), 5984–5994.
4.
DembczynskiK., JachnikA., KotlowskiW., WaegemanW. and HuellermeierE., Optimizing the F-Measure in Multi-Label Classification: Plug-in Rule Approach versus Structured Loss Minimization, In
DasguptaS. and
McAllesterD., editors, Proceedings of the 30th International Conference on Machine Learning volume 28 of Proceedings of Machine Learning Research, Atlanta, Georgia, USA, PMLR, 2013, pp. 1130–1138.
GodinF., VandersmissenB., De NeveW. and
Van de WalleR.
, Multimedia lab @ ACL W-NUT NER sharedtask: Named entity recognition for Twitter microposts using distributed word representations, ACL-IJCNLP2015 (2015), 146–153.
HuM. and LiuB., Mining and summarizing customer reviews, In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, New York, NY, USA, ACM, 2004, pp. 168–177.
9.
IoffeS. and SzegedyC., Batch normalization: Accelerating deep network training by reducing internal covariate shift, CoRR, abs/1502.03167 (2015).
10.
JanochaK. and CzarneckiW.M., On loss functions for deep neural networks in classification, CoRR, abs/1702.05659 (2017).
11.
KriegerM. and AhnD., Tweetmotif: Exploratory search and topic summarization for twitter, In Proc of AAAI Conference on Weblogs and Social, 2010.
12.
LiuB., Sentiment Analysis and Opinion Mining, A Comprehensive Introduction and Survey. Morgan & Claypool Publishers, 2012.
13.
MikolovT., SutskeverI., ChenK., CorradoG. and DeanJ., Distributed representations of words and phrases and their compositionality, CoRR, abs/1310.4546 (2013a).
14.
MikolovT., ChenK., CorradoG. and DeanJ., Efficient estimation of word representations in vector space, CoRR, abs/1301.3781, 2013b.
15.
MohammadS., #emotional tweets, In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the Main Conference and the Shared Task and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), Montréal, Canada. Association for Computational Linguistics, 2012, pp. 246–255.
16.
MohammadS., KiritchenkoS., SobhaniP., ZhuX. and CherryC., Semeval-task 6: Detecting stance in tweets, In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016, pp. 31–41.
17.
MohammadS.M. and Bravo-MarquezF., WASSA-shared task on emotion intensity, CoRR, abs/1708.03700, 2017.
18.
MohammadS.M. and TurneyP.D., Crowdsourcing a word-emotion association lexicon, Computational Intelligence29(3) (2013), 436–465.
19.
MohammadS.M., SobhaniP. and KiritchenkoS., Stance and sentiment in tweets, ACM Trans Internet Technol17(3) (2017), 26:1–26:23.
20.
MohammadS.M., Bravo-MarquezF., SalamehM. and KiritchenkoS., Semeval-2018 Task 1: Affect in tweets, In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA, 2018.
21.
Molina-GonzálezM.D., Martínez-CámaraE.,
Martín-ValdiviaM-T and Perea-OrtegaJ.M., Semantic orientation for polarity classification in spanish reviews, Expert Systems with Applications40(18) (2013), 7250–7257.
22.
MooreD., McCabeG., DuckworthW., ScloveS., The Practice of Business Statistics Companion Chapter 18: Bootstrap Methods and Permutation Tests, W. H. Freeman, 2003.
23.
NairV. and HintonG.E., Rectified linear units improve restricted boltzmann machines, In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, USA, 2010, pp. 807–814. Omnipress.
24.
NielsenF.Å., AFINN, 2011.
25.
Pastor-PellicerJ., Zamora-MartínezF., España BoqueraS. and
Castro BledaM.J.
, F-Measure as the Error Function to Train Neural Networks, In IWANN Proceedings, 2013.
26.
PennebakerJ., ChungC., IrelandM., GonzalesA. and BoothR., The development and psychological properties of liwc2007, 2014.
27.
PlaF. and HurtadoL.-F., Language identification of multilingual posts from twitter: A case study, Knowledge and Information Systems51(3) (2017), 965–989.
28.
RosenthalS., FarraN. and NakovP., SemEval-2017 task 4: Sentiment analysis in Twitter, In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval ’17, Vancouver, Canada, Association for Computational Linguistics, 2017.
29.
SaralegiX. and SanI., Vicente, Elhuyar at tass 2013, In XXIX Congreso de la Sociedad Espaola de Procesamiento de Lenguaje Natural, Workshop on Sentiment Analysis at SEPLN (TASS2013), 2013, pp. 143–150.
30.
SebastianiF., Machine learning in automated text categorization, ACM Comput Surv34(1) (2002), 1–47.
31.
TauléM., MartíM., RangelF., RossoP., BoscoC. and PattiV., Overview of the task of Stance and Gender Detection in Tweets on Catalan Independence at IBEREVAL 2017, In Notebook Papers of 2nd SEPLN Workshop on Evaluation of Human Language Technologies for Iberian Languages (IBEREVAL), Murcia (Spain). CEUR Workshop Proceedings. CEUR-WS.org, 2017, 2017.
32.
WallaceB.C., ChoeD.K., KertzL. and CharniakE., Humans require context to infer ironic intent (so computers probably do, too), In ACL (2), The Association for Computer Linguistics, 2014, pp. 512–516.
33.
WiebeJ., WilsonT. and CardieC., Annotating expressions of opinions and emotions in language, Language Resources and Evaluation1(2) (2005).
34.
WilsonT., WiebeJ. and HoffmannP., Recognizing contextual polarity in phrase-level sentiment analysis, In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, Stroudsburg, PA, USA, 2005, pp. 347–354. Association for Computational Linguistics.
35.
ZhangY. and WallaceB., A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification, In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2017, pp. 253–263. Asian Federation of Natural Language Processing.
