Textual Entailment (TE) is the task of recognizing entailment, paraphrase, and contradiction relations between a given text pair. The goal of textual entailment research is to develop a core inference component that can be applied to various domains such as QA or IR. We observed several rank correlations on the test data and system results in the NTCIR-10 RITE-2 task, trying to find out correlations between datasets and evaluation metrics. We also constructed RITE4QA datasets in the RITE-2 task under the scenario of QA in order to see the applicability of RITE techniques in QA systems. Although we find that datasets created from different sources and different ways can hardly predict each other, we also find that ranking by RITE metrics has moderate correlation with the ranking by QA metrics if testing on artificial pairs. Both RITE metrics and QA metrics are stable in terms of their own subtasks.
Y.Watanabe, Y.Miyao, J.Mizuno, T.Shibata, H.Kanayama, C.-W.Lee, C.-J.Lin, S.Shi, T.Mitamura, N.Kando, H.Shima and K.Takeda, Overview of the recognizing inference in text (RITE-2) at NTCIR-10, in: Proc. of NTCIR-10, 2013, pp. 385–404.
2.
L.Bentivogli, I.Dagan, H.T.Dang, D.Giampiccolo and B.Magnini, The fifth Pascal recognizing textual entailment challenge, in: Proc. of TAC 2009 Workshop, 2009, pp. 14–24.
3.
L.Bentivogli, P.Clark, I.Dagan, H.T.Dang and D.Giampiccolo, The sixth Pascal recognizing textual entailment challenge, in: Proc. of the Text Analysis Conference (TAC 2010), 2010.
4.
L.Bentivogli, P.Clark, I.Dagan, H.T.Dang and D.Giampiccolo, The seventh Pascal recognizing textual entailment challenge, in: Proc. of the Text Analysis Conference (TAC 2011), 2011.
5.
I.Dagan, O.Glickman and B.Magnini, The Pascal recognising textual entailment challenge, in: Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment, Lecture Notes in Computer Science, Vol. 3944, 2006, pp. 177–190.
6.
J.Bos, F.M.Zanzotto and M.Pennacchiotti, Textual entailment at EVALITA 2009, in: Proc. of EVALITA 2009, 2009.
7.
H.Shima, H.Kanayama, C.-W.Lee, C.-J.Lin, T.Mitamura, Y.Miyao, S.Shi and K.Takeda, Overview of NTCIR-9 RITE: Recognizing inference in TExt, in: Proc. of NTCIR-9, 2011, pp. 291–301.
8.
S.Harabagiu and A.Hickl, Methods for using textual entailment in open-domain question answering, in: Proc. of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the ACL, 2006, pp. 905–912.
9.
B.Sacaleanu, C.Orasan, C.Spurk, S.Ou, O.Ferrandez, M.Kouylekov and M.Negri, Entailment-based question answering for structured data, in: Proc. of 22nd International Conference on Computational Linguistics (COLING 2008), 2008, pp. 173–176.
10.
S.Clinchant, C.Goutte and E.Gaussier, Lexical entailment for information retrieval, in: Advances in Information Retrieval, Lecture Notes in Computer Science, Vol. 3936, 2006, pp. 217–228.
11.
D.Parapar, Á Barreiro and D.E.Losada, Query expansion using wordnet with a logical model of information retrieval, in: IADIS AC, 2005, pp. 487–494.
12.
M.O.Kouylekov, Recognizing textual entailment with tree edit distance: Application to question answering and information extraction, PhD thesis, University of Trento, Computer Science, Electronics and Telecommunication, 2006.
13.
E.Lloret, O.s.Ferrández, R.Munõz and M.Palomar, A text summarization approach under the influence of textual entailment, in: Proc. of the 5th International Workshop on Natural Language Processing and Cognitive Science (NLPCS 2008), 2008, pp. 22–31.
14.
D.Tatar, A.Mihis, D.Lupsa and E.Tamaianu-Morita, Entailment-based linear segmentation in summarization, International Journal of Software Engineering and Knowledge Engineering19 (2009), 1023–1038.
15.
R.D.Nielsen, W.Ward and J.H.Martin, Recognizing entailment in intelligent tutoring systems, Natural Language Engineering15 (2009), 479–501.
16.
S.Pado, M.Galley, D.Jurafsky and C.Manning, Robust machine translation evaluation with entailment features, in: Proc. of ACL-IJCNLP 2009, 2009, pp. 297–305.
17.
L.Zhou, C.-Y.Lin, D.S.Munteanu and E.Hovy, Paraeval: Using paraphrases to evaluate summaries automatically, in: Proc. of the Main Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL ’06)2006, pp. 447–454.
18.
I.Dagan, B.Dolan, B.Magnini and D.Noth, Recognizing textual entailment: Rational, evaluation and approaches, Natural Language Engineering15 (2009).
19.
C.-W.Lee, C.-J.Lin, H.Shima and W.-L.Hsu, Evaluating and enhancing cross-domain rank predictability of textual entailment datasets, in: Proc. of the 13th IEEE International Conference on Information Reuse and Integration (IRI 2012), Workshop on Empirical Methods for Recognizing Inference in Text (EM-RITE), 2012, pp. 51–58.
20.
T.Sakai, H.Shima, N.Kando, R.Song, C.J.Lin, T.Mitamura, M.Sugimito and C.W.Lee, Overview of NTCIR-8 ACLIA IR4QA, in: Proc. of the 8th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering, and Cross-Lingual Information Access (NTCIR-8), 2010, pp. 63–93.
21.
C.-W.Lee, M.-Y.Day, C.-L.Sung, Y.-H.Lee, T.-J.Jiang, C.-W.Wu, C.-W.Shih, Y.-R.Chen and W.-L.Hsu, Boosting Chinese question answering with two lightweight methods: ABSPs and SCO-QAT, ACM Transactions on Asian Language Information Processing (TALIP)7(4) (2008), 12:1–12:29.
22.
Y.Sasaki, C.-J.Lin, K.-h.Chen and H.-H.Chen, Overview of the NTCIR-6 cross-lingual question answering (CLQA) task, in: Proc. of NTCIR-6, 2007, pp. 153–163.
