In this paper, we propose a unified framework for an abstractive summarization method which uses the prompt language model and a pointer mechanism. The abstractive summarization problem usually includes a text encoder and a text decoder. Current methods usually employ an encoder-decoder architecture to condense and paraphrase a document. To better paraphrase a document, we propose a unified framework for an abstractive summarization model that only uses a topic-sensitive decoder. Our model has a prompt input module, a text decoder and a pointer mechanism. We apply our model to Xsum, Gigaword, and CNN/DailyMail summarization datasets, and experimental results demonstrate that our model has achieved state-of-the-art results on the Xsum dataset and comparable results on the other two datasets.
LuhnH.P., The automatic creation of literature abstracts, Ibm Journal of Research and Development2(2) (1958), 159–165.
2.
EdmundsonH.P., New methods in automatic extracting, Journal of the ACM16(2) (1969), 264–285.
3.
BrandowR., MitzeK. and RauL.F., Automatic condensation of electronic publications by sentence selection, Information Processing and Management31(5) (1995), 675–685.
4.
KupiecJ., PedersenJ.O., ChenF., A trainable document summarizer, in: Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, 1995, pp. 68–73 .
5.
RadevD.R., JingH., Sty’sM. and TamD., Centroid-based summarization of multiple documents, Information Processing and Management40(6) (2004), 919–938.
6.
NallapatiR., ZhaiF. and ZhouB., Summarunner: A recurrent neural network based sequence model for extractive summarization of documents, National Conference on Artificial Intelligence (2016), 3075–3081.
7.
ChopraS., AuliM., RushA.M., Abstractive sentence summarization with attentive recurrent neural networks, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 93–98.
8.
HuB., ChenQ., ZhuF., Lcsts: A large scale chinese short text summarization dataset, in: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp. 1967–1972.
DorrB.J., ZajicD.M. and SchwartzR.M., Hedge trimmer: a parse-and-trim approach to headline generation, in: HLTNAACL- DUC ‘03 Proceedings of the HLT-NAACL 03 on Text summarization workshop - Volume 5, 2003, pp. 1–8.
11.
SeeA., LiuP.J. and ManningC.D., Get to the point: Summarization with pointer-generator networks, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1, 2017, pp. 1073–1083.
12.
ChenY.-C. and BansalM., Fast abstractive summarization with reinforce-selected sentence rewriting, Meeting of the Association for Computational Linguistics1 (2018), 675–686.
13.
HermannK.M., KočiskýT.,
GrefenstetteE.,
EspeholtL.,
KayW.,
SuleymanM. and
BlunsomP., Teaching machines to read and comprehend, Neural Information Processing Systems (2015), 1693–1701.
14.
NallapatiR., ZhouB., dos SantosC.N., Caglar Gulcehre and B. Xiang, Abstractive text summarization using sequenceto- sequence rnns and beyond, in: Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, 2016, pp. 280–290.
15.
GehrmannS., DengY. and RushA.M., Bottom-up abstractive summarization, Empirical Methods in Natural Language Processing (2018), 4098–4109.
16.
LinC.-Y., Rouge: A package for automatic evaluation of summaries, in: Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, 2004, pp. 74–81 .
17.
CarbonellJ.G., GoldsteinJ., The use of mmr, diversity-based reranking for reordering documents and producing summaries, in: Research and Development in Information Retrieval, 1998, pp. 335–336.
18.
MihalceaR., TarauP., TextRank: Bringing order into text, in: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 404–411.
19.
ErkanG. and RadevD.R., Lexrank: Graph-based lexical centrality as salience in text summarization, Journal of Artificial Intelligence Research22 (2004), 457–479.
20.
ChengJ. and LapataM., Neural summarization by extracting sentences and words, in: ACL (1), The Association for Computer Linguistics, 2016.
21.
LandauerT.K. and DumaisS.T., A solution to Plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge, Psychological Review104 (1997), 211–240.
22.
LandauerT.K., FoltzP.W. and LahamD., An introduction to latent semantic analysis, Discourse Processes25 (1998), 259–284.
23.
BankoM., MittalV.O., WitbrockM.J., Headline generation based on statistical translation, in: Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, Association for Computational Linguistics, 2000, pp. 318–325.
24.
ZajicD., DorrB., SchwartzR., Bbn/umd at duc-2004: Topiary, in: Proceedings of the HLT-NAACL 2004 Document Understanding Workshop, Boston, 2004, pp. 112–119.
25.
RushA.M., ChopraS., WestonJ., A neural attention model for abstractive sentence summarization, in: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp. 379–389.
26.
BengioY., DucharmeR., VincentP. and JanvinC., A neural probabilistic language model, Journal of Machine Learning Research3(6) (2003), 1137–1155.
27.
ChungJ., GulcehreC., ChoK. and BengioY., Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555 (2014).
28.
DevlinJ., ChangM.-W., LeeK. and ToutanovaK., Bert: Pretraining of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
29.
RadfordA., WuJ., ChildR., LuanD., AmodeiD. and SutskeverI., Language models are unsupervised multitask learners, OpenAI blog1(8) (2019), 9.
30.
VaswaniA., ShazeerN., ParmarN., UszkoreitJ., JonesL., GomezA.N., KaiserL. and PolosukhinI., Attention is all you need, Neural Information Processing Systems (2017), 5998–6008.
31.
SrivastavaA. and SuttonC., Autoencoding variational inference for topic models, in: 5th International Conference on Learning Representations, 2017.
32.
BahdanauD., ChoK. and BengioY., Neural machine translation by jointly learning to align and translate, in: ICLR 2015: International Conference on Learning Representations, 2015, 2015.
33.
NarayanS., CohenS.B., LapataM., Don’t give me the details, just the summary topic-aware convolutional neural networks for extreme summarization, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 1797–1807.
34.
GraffD., KongJ., ChenK. and MaedaK., English gigaword, Linguistic Data Consortium, Philadelphia4(1) (2003), 34.
35.
LinC.-Y. and HovyE.H., Automatic evaluation of summaries using n-gram co-occurrence statistics, in: NAACL’03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, 2003, pp. 71–78.
36.
LiuY., LapataM., Text summarization with pretrained encoders, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 3730–3740. doi: 10.18653/v1/D19-1387.
37.
NarayanS., CohenS.B., LapataM., Don’t give me the details, just the summary topic-aware convolutional neural networks for extreme summarization, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 1797–1807. doi: 10.18653/v1/D18-1206.
38.
SongK., WangB., FengZ., RenL. and LiuF., Controlling the amount of verbatim copying in abstractive summarization, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
39.
QiW., YanY., GongY., LiuD., DuanN., ChenJ., ZhangR., ZhouM., ProphetNet: Predicting future n-gram for sequence-to-sequence Pre-training, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online 2020, pp. 2401–2410. doi: 10.18653/v1/2020.findings-emnlp.217.
40.
DouZ.-Y., LiuP., HayashiH., JiangZ. and NeubigG., Gsum: A general framework for guided neural abstractive summarization, arXiv preprint arXiv:2010.08014 (2020).
