Abstract
Word representation plays a vital role in most Natural Language Processing systems, especially for Neural Machine Translation. It tends to capture semantic and similarity between individual words well, but struggle to represent the meaning of phrases or multi-word expressions. In this paper, we investigate a method to generate and use phrase information in a translation model. To generate phrase representations, a Primary Phrase Capsule network is first employed, then iteratively enhancing with a Slot Attention mechanism. Experiments on the IWSLT English to Vietnamese, French, and German datasets show that our proposed method consistently outperforms the baseline Transformer, and attains competitive results over the scaled Transformer with two times lower parameters.
Get full access to this article
View all access options for this article.
References
1.
Mikolov
T.
,
Chen
K.
,
Corrado
G.
,
Dean
J.
, Efficient estimation of word representations in vector space, 2013.
2.
Belinkov
Y.
,
Durrani
N.
,
Dalvi
F.
,
Sajjad
H.
and
Glass
J.R.
, On the linguistic representational power of neural machine translation models. CoRR , abs/1911.00317, 2019. URL http://arxiv.org/abs/1911.00317.
3.
Koehn
P.
,
Och
F.J.
and
Marcu
D.
, Statistical phrasebased translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, page 48–54, USA, 2003. Association for Computational Linguistics. doi: 10.3115/1073445.1073462 . URL https://doi.org/10.3115/1073445.1073462 .
4.
Wisniewski
G.
,
Allauzen
A.
and
Yvon
F.
, Assessing phrase-based translation models with oracle decoding. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 933–943, Cambridge, MA, October 2010. Association for Computational Linguistics. URL https://aclanthology.org/D10-1091
5.
Junczys-Dowmunt
M.
, A phrase table without phrases: Rank encoding for better phrase table compression. In Proceedings of the 16th Annual conference of the European Association for Machine Translation, pages 245–252, Trento, Italy, May 28–30 2012. European Association for Machine Translation. URL https://aclanthology.org/2012.eamt-1.58
6.
Cuong
H.
and
Sima’an
K.
, Latent domain phrasebased models for adaptation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 566–576, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1062 . URL https://aclanthology.org/D14-1062
7.
Nishino
M.
,
Suzuki
J.
and
Nagata
M.
, Phrase table pruning via submodular function maximization. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 406–411, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-2066 . URL https://aclanthology.org/P16-2066
8.
Bogoychev
N.
and
Hoang
H.
, Fast and highly parallelizable phrase table for statistical machine translation. In Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, pages 102–109, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/W16-2211. URL https://aclanthology.org/W16-2211
9.
Sutskever
I.
,
Vinyals
O.
and
Le
Q.V.
, Sequence to sequence learning with neural networks, 2014.
10.
Xu
H.
,
van Genabith
J.
,
Xiong
D.
,
Liu
Q.
,
Zhang
J.
, Learning source phrase representations for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 386–396, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.37. URL https://arxiv.org/abs/2006.14405.
11.
Bapna
A.
,
Chen
M.X.
,
Firat
O.
,
Cao
Y.
and
Wu
Y.
, Training deeper neural machine translation models with transparent attention, 2018.
12.
Socher
R.
,
Bauer
J.
,
Manning
C.D.
and
Ng
A.Y.
, Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 455–465, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.URLhttps://www.aclweb.org/anthology/P13-1045
13.
Locatello
F.
,
Weissenborn
D.
,
Unterthiner
T.
,
Mahendran
A.
,
Heigold
G.
,
Uszkoreit
J.
,
Dosovitskiy
A.
and
Kipf
T.
, Object-centric learning with slot attention, 2020.
14.
Huang
G.
,
Liu
Z.
,
van der Maaten
L.
and
Weinberger,
K.Q.
, Densely connected convolutional networks, 2018.
15.
Sabour
S.
,
Frosst
N.
and
Hinton
G.E.
, Dynamic routing between capsules. CoRR, abs/1710.09829, 2017. URL http://arxiv.org/abs/1710.09829
16.
Vaswani
A.
,
Shazeer
N.
,
Parmar
N.
,
Uszkoreit
J.
,
Jones
L.
,
Gomez
A.N.
,
Kaiser
L.
,
Polosukhin,
I.
, Attention is all you need. CoRR, abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762
17.
Shardlow
M.
and
Nawaz
R.
, Neural text simplification of clinical letters with a domain specific phrase table. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 380–389, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1037. URL https://www.aclweb.org/anthology/P19-1037
18.
Zhao
W.
,
Peng
H.
,
Eger
S.
,
Cambria
E.
and
Yang
M.
, Towards scalable and reliable capsule networks for challenging NLP applications. CoRR, abs/1906.02829, 2019. URL http://arxiv.org/abs/1906.02829
19.
Yang
Z.
,
Zhang
J.
,
Meng
F.
,
Gu
S.
,
Feng
Y.
and
Zhou
J.
, Enhancing context modeling with a queryguided capsule network for documentlevel translation. In Proceedings of the 2019 Conference on Empircal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1527–1537, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1164. URL https://www.aclweb.org/anthology/D19-1164
20.
Chung
J.
,
Gülçehre
Ç.
,
Cho
K.H.
and
Bengio,
Y.
, Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014. URL http://arxiv.org/abs/1412.3555
21.
Sennrich
R.
,
Haddow
B.
and
Birch
A.
, Neural machine translation of rare words with subword units. CoRR, abs/1508.07909, 2015. URL http://arxiv.org/abs/1508.07909
22.
Papineni
K.
,
Roukos
S.
,
Ward
T.
and
Zhu
W.-J.
, Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://www.aclweb.org/anthology/P02-1040
23.
Tu
Z.
,
Lu
Z.
,
Liu
Y.
,
Liu
X.
and
Li
H.
, Modeling coverage for neural machine translation, 2016.
24.
Bahdanau
D.
,
Cho
K.
and
Bengio
Y.
, Neural machine translation by jointly learning to align and translate. In Yoshua Bengio andYann Le-Cun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings,2015. URL http://arxiv.org/abs/1409.0473