The language used by the users in social media nowadays is Code-mixed text, i.e., mixing of two or more languages. This paper describes the application of the code mixed index in Indian social media texts and comparing the complexity to identify language at word level using Bi-directional Long Short Term Memory model. Social media platforms are now widely used by people to express their opinion and interest. The major contribution of the work is to propose a technique for identifying the language of Hindi-English code-mixed data used in three social media platforms namely, Facebook, Twitter, and WhatsApp. We recommend a deep learning framework based on cBoW and Skip gram model that predicts the origin of the word from language perspective in the sequence based on the specific words that have come before it in the sequence. The context capture module of the system gives better accuracy for word embedding model as compared to character embedding.
WeischedelR.CarbonellJ.GroszB. et al., White paper on natural language processing, in: Proceedings of the Workshop on Speech and Natural Language, Association for Computational Linguistics, 1989, pp. 481–493.
BarmanU.DasA. et al., Code mixing: A challenge for language identification in the language of social media, in: Proceedings of the First Workshop on Computational Approaches to Code Switching, 2014, pp. 13–23.
4.
KingL.BaucomE. et al., The IUCL+ system: Word-level language identification via extended Markov models, EMNLP (2014), 102–106.
5.
KingB. and AbneyS., Labeling the languages of words in mixed-language documents using weakly supervised methods, in: Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2013, pp. 1110–1119.
6.
NguyenD. and DugruozA.S., Word level language identification in online multilingual communication, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2013, pp. 857–862.
7.
YogarshiV.GellaS.SharmaJ.BaliK. and ChoudhuryM., Pos tagging of English-Hindi code-mixed social media content, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 974–979.
8.
DasA. and GambackB., Identifying languages at the word level in code-mixed indian social media text, in:Proceedings of the 11th International Conference on Natural Language Processing, 2014, pp. 378–387.
9.
SequieraR.ChoudhuryM.GuptaP. et al., Overview of FIRE-2015 shared task on mixed script information retrieval, in: FIRE Workshops, 2015, pp. 19–25.
10.
JhamtaniH.BhogiS.K. et al., Word-level language identification in bi-lingual code-switched texts, in: Proceedings of the 28th Pacific Asia Conference on Language, Information and Computing, 2014, pp. 348–357.
11.
EthirajR.ShanmugamS.SrinivasaG.SinhaN., NELIS – Named Entity and Language Identification System: Shared task system description, in: FIRE Workshops, 2015, pp. 43–46.
12.
BhargavaR.SharmaY. and SharmaS., Sentiment analysis for mixed script indic sentences, in: International Conference on Advances in Computing, Communications and Informatics, ICACCI, 2016, pp. 524–529.
13.
CastilhoD.EckartR. et al., Cross-platform text mining and natural language processing interoperability, in: Proceedings of the LREC, 2016.
14.
BarmanU.WagnerJ. and FosterJ., Part-of-speech tagging of code-mixed social media content: Pipeline, stacking and joint modelling, in: Proceedings of the Second Workshop on Computational Approaches to Code Switching EMNLP, 2016, pp. 30–39.
15.
BaliK.JatinS. and ChoudhuryM., “i am borrowing ya mixing?” An analysis of English-Hindi code mixing in Facebook, in: Proceedings of the First Workshop on Computational Approaches to Code Switching, 2014, pp. 116–126.
16.
VyasY.GellaS.SharmaJ.BaliK. and ChoudhuryM., POS tagging of English-Hindi code-mixed social media content, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 974–979.
17.
RaoP.R. and DeviS.L., Code mix entity extraction in Indian languages from social media text@FIRE, in: FIRE (Working Notes), 2016, pp. 289–295.
18.
DeviR.VeenaG.Anand KumarP.V. et al., AMRITA-CEN@FIRE 2016: Code-mix entity extraction for Hindi-English and Tamil-English tweets, in: CEUR Workshop Proceedings, 2016, pp. 304–308.
19.
SapkalK. and ShrawankarU., Transliteration of secured SMS to Indian regional language, Procedia Computer Science (2016), 748–755.
20.
ZubiagaA.VicenteI.S.GamalloP. and PichelJ.R., TweetLID: A benchmark for tweet language identification, Language Resources and Evaluation (2015), 729–766.
21.
SzegedyC.LiuW.JiaY. et al., Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1–9.
22.
YannL.HaffnerP.BottouL. et al., Object recognition with gradient-based learning, in: Shape, Contour and Grouping in Computer Vision, 1999, pp. 319–345.
23.
HochreiterS. and SchmidhuberJ., Long short-term memory, Neural Computation (1997), 1735–1780.
24.
MikolovT.SutskeverI.ChenK.CorradoG.S. and DeanJ., Distributed representations of words and phrases and their compositionality, in: Advances in Neural Information Processing Systems, 2013, pp. 3111–3119.
25.
JamatiaA. and DasA., Task report: Tool contest on POS tagging for code-mixed Indian social media (Facebook, Twitter, and Whatsapp) text, in: Proceeding of ICON, 2016.
26.
BanerjeeS.ChakmaK.NaskarS. et al., Overview of the Mixed Script Information Retrieval (MSIR), in: CEUR Workshop Proceedings, 2016, pp. 94–99.
27.
SequieraR.ChoudhuryM.GuptaP. et al., Overview of FIRE-2015 shared task on mixed script information retrieval, in: FIRE Workshops, 2015, pp. 19–25.
28.
SrinidhiV.SinghS.DeviR. et al., Context based character embeddings for entity extraction in code-mixed text, in: CEUR Workshop Proceedings, 2016, pp. 321–324.
29.
AlekseevA. and NikolenkoS., Word embeddings for user profiling in online social networks, Computación y Sistemas (2017), 203–226.
30.
ShekharS.SharmaD.K. and BegM.S., Hindi roman linguistic framework for retrieving transliteration variants using bootstrapping, Procedia Computer Science (2018), 59–67.
31.
VeenaP.V.KumarA. and SomanK.P., Character embedding for language identification in Hindi-English code-mixed social media text, Computación y Sistemas (2018), 65–74.
