Keyphrase extraction is an essential task of identifying representative words or phrases in document processing. Main traditional models rely on each word frequency feature in a document and its associated corpus. There are two major limitations of the word frequency method: first, it fails to fully exploit semantic information in the document, that is, it is a bag-of-word method; second, it tends to be influenced by local word frequency in the short current text when the linked corpus is not available or incomplete. This paper proposes WAGRank, a novel unsupervised ranking model on a word attention graph, where nodes are words and edges are semantic relations between words. To assign edge weights, two interpretable statistical methods of assessing correlation strength between words are designed using attention mechanism. WAGRank depends on word semantics rather than frequency only in the current text, using external knowledge stored in a pre-trained language model. WAGRank was evaluated on two publicly available datasets against twelve baselines, presenting its effectiveness and robustness. Besides, the Granger causality test illustrated that word attention has a statistically significant predictive effect on word frequency, providing a more reasonable explanation for word frequency analysis.
HasanKSNgV. Automatic keyphrase extraction: a survey of the state of the art. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (Volume 1: Long Papers), 2014, pp.1262–1273.
2.
HulthAMegyesiB. A study on automatically extracted keywords in text categorization. In: Proceedings of the 21st international conference on computational linguistics and 44th annual meeting of the association for computational linguistics, 2006, pp.537–544.
3.
Alami MerrouniZFrikhBOuhbiB. Automatic keyphrase extraction: a survey and trends. J Intell Inf Syst2020; 54: 391–424.
4.
PapagiannopoulouETsoumakasG. A review of keyphrase extraction. Wiley Interdiscip Rev Data Min Knowl Discovery2020; 10: e1339.
CanchoRFISoléRV. The small world of human language. Proc R Soc London Ser B Biol Sci2001; 268: 2261–2265.
7.
SongMFengYJingL. A survey on recent advances in keyphrase extraction from pre-trained language models, Findings of the association for computational linguistics: EACL 2023, 2023, pp.2108–2119.
8.
Spärck JonesK. A statistical interpretation of term specificity and its application in retrieval. J Doc2004; 60: 493–502.
9.
SaltonGBuckleyC. Term-weighting approaches in automatic text retrieval. Inf Process Manag1988; 24: 513–523.
10.
RoseSEngelDCramerN, et al. Automatic keyword extraction from individual documents, Text mining: applications and theory, 2010, pp.1–20.
11.
GiamblancoNSiddavaatamP. Keyword and keyphrase extraction using newton’s law of universal gravitation. In: 2017 IEEE 30th Canadian conference on electrical and computer engineering (CCECE), 2017, pp.1–4. IEEE.
12.
SaxenaAMangalMJainG. KeyGames: a game theoretic approach to automatic keyphrase extraction. In: Proceedings of the 28th international conference on computational linguistics, 2020, pp.2037–2048.
13.
VaswaniAShazeerNParmarN, et al. Attention is all you need, Advances in neural information processing systems, Vol. 30, 2017.
14.
HuD. An introductory survey on attention mechanisms in NLP problems. In: Intelligent systems and applications: proceedings of the 2019 intelligent systems conference (IntelliSys) Volume 2, 2020, pp.432–448. Springer.
15.
DuanJZhaoHZhouQ, et al. A study of pre-trained language models in natural language processing. In: 2020 IEEE international conference on smart cloud (SmartCloud), 2020, pp.116–121. IEEE.
CamposRMangaraviteVPasqualiA, et al. YAKE! keyword extraction from single documents using multiple local features. Inf Sci (Ny)2020; 509: 257–289.
19.
RabbyGAzadSMahmudM, et al. TeKET: a tree-based unsupervised keyphrase extraction technique. Cognit Comput2020; 12: 811–833.
20.
MihalceaRTarauP. Textrank: bringing order into text. In: Proceedings of the 2004 conference on empirical methods in natural language processing, 2004, pp.404–411.
21.
BrinS. The PageRank citation ranking: bringing order to the web. Proc ASIS1998; 98: 161–172.
22.
WanXXiaoJ. Single document keyphrase extraction using neighborhood knowledge.. In: AAAI, Vol. 8, 2008, pp.855–860.
23.
BougouinABoudinFDailleB. Topicrank: graph-based topic ranking for keyphrase extraction. In: International joint conference on natural language processing (IJCNLP), 2013, pp.543–551.
24.
SterckxLDemeesterTDeleuJ, et al. Topical word importance for fast keyphrase extraction. In: Proceedings of the 24th international conference on world wide web, 2015, pp.121–122.
25.
FlorescuCCarageaC. A position-biased pagerank algorithm for keyphrase extraction. In: Proceedings of the AAAI conference on artificial intelligence, Vol. 31, 2017.
MikolovTChenKCorradoG, et al. Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781, 2013.
28.
PenningtonJSocherRManningCD. Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp.1532–1543.
RadfordANarasimhanKSalimansT, et al. Improving language understanding by generative pre-training, 2018.
31.
DevlinJChangM-WLeeK, et al. Bert: pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018.
32.
RaffelCShazeerNRobertsA, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res2020; 21: 5485–5551.
33.
PapagiannopoulouETsoumakasG. Local word vectors guiding keyphrase extraction. Inf Process Manag2018; 54: 888–902.
34.
Bennani-SmiresKMusatCHossmannA, et al. Simple unsupervised keyphrase extraction using sentence embeddings, arXiv preprint arXiv:1801.04470, 2018.
35.
SunYQiuHZhengY, et al. SIFRank: a new baseline for unsupervised keyphrase extraction based on pre-trained language model. IEEE Access2020; 8: 10896–10906.
36.
ZhangLChenQWangW, et al. MDERank: a masked document embedding rank approach for unsupervised keyphrase extraction. In: Muresan S, Nakov P and Villavicencio A (eds) Findings of the association for computational linguistics: ACL 2022. Association for Computational Linguistics, Dublin, Ireland, 2022, pp.396–409. https://doi.org/10.18653/v1/2022.findings-acl.34. https://aclanthology.org/2022.findings-acl.34.
37.
SongMFengYJingL. Utilizing BERT intermediate layers for unsupervised keyphrase extraction. In: Proceedings of the 5th international conference on natural language and speech processing (ICNLSP 2022), 2022, pp.277–281.
38.
ManningCSchutzeH. Foundations of statistical natural language processing. Cambridge: MIT Press, 1999.
39.
KusnerMSunYKolkinN, et al. From word embeddings to document distances. In: International conference on machine learning, 2015, pp.957–966. PMLR.
40.
LevyOGoldbergY. Neural word embedding as implicit matrix factorization, Advances in neural information processing systems, Vol. 27, 2014.
41.
NairVHintonGE. Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp.807–814.
42.
KleinDManningCD. Accurate unlexicalized parsing. In: Proceedings of the 41st annual meeting of the association for computational linguistics, 2003, pp.423–430.
43.
AugensteinIDasMRiedelS, et al. Semeval 2017 task 10: scienceie-extracting keyphrases and relations from scientific publications, arXiv preprint arXiv:1704.02853, 2017.
44.
HulthA. Improved automatic keyword extraction given more linguistic knowledge. In: Proceedings of the 2003 conference on Empirical methods in natural language processing, 2003, pp.216–223.
45.
BoudinF. PKE: an open source python-based keyphrase extraction toolkit. In: Proceedings of COLING 2016, the 26th international conference on computational linguistics: system demonstrations, Osaka, Japan, 2016, pp.69–73. http://aclweb.org/anthology/C16-2015.
46.
GrangerCW. Investigating causal relations by econometric models and cross-spectral methods. Econom J Econom Soc1969; 37: 424–438.
47.
ZongCQ. Statistical natural language processing (2nd edition). Beijing: Tsinghua University Press, 2013.
48.
BrockwellPJDavisRA. Introduction to time series and forecasting. New York: Springer, 2002.
49.
DickeyDAFullerWA. Likelihood ratio statistics for autoregressive time series with a unit root. Econom J Econom Soc1981; 49: 1057–1072.
50.
KangBShinY. SAMRank: unsupervised keyphrase extraction using self-attention map in BERT and GPT-2. In: Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp.10188–10201.
51.
YuYNgV. WikiRank: improving keyphrase extraction based on background knowledge, arXiv preprint arXiv:1803.09000, 2018.
