Finding more trustworthy answers: Various trustworthiness factors in question answering

Abstract

In the recent explosion of Web information, it is important to find not only appropriate, but also more trustworthy answers to user questions. This paper proposes an improved ranking model for question answering (QA) which is focused on various answer trustworthiness factors. Contrary to past research that simply focused on document quality, we have identified three different answer trustworthiness factors in multiple layers of answering processes: document quality, authority and reputation of answer sources, and appropriateness of answering method for a given question. Each of these factors is used in the answer selection as an input to the ranking scheme that can be tuned for the confidence value for a particular answer candidate. Through several experiments, we analysed the efficacy of our QA model from two points of view: indexing and answering. In indexing, distilling unreliable documents brings not only a 96% reduction in document size but also a 92% speed increase in indexing time. To reveal the effect of trustworthiness factors in answering, we conducted several experiments to determine the optimum combination of weights of sub-features for trustworthiness factors. Finally, the proposed method using all answer trustworthiness factors obtained an improvement in effectiveness over the simple routing QA by 150% in Top1. We also investigated improvement impacts according to answer trustworthiness factors.

Keywords

answer selection answer trustworthiness information quality question answering

Get full access to this article

View all access options for this article.

References

Fichman

. Comparative assessment of answer quality on four question answering sites. Journal of Information Science2011; 37(5): 476–486.

Riloff

Wiebe

Phillips

. Exploiting subjectivity classification >to improve information extraction. In: Proceedings of the 20th national conference on artificial intelligence. Palo Alto, CA: AAAI Press, 2005, pp. 1106–1111.

Xiaolan

Gauch

. Incorporating quality metrics in centralized/distributed information retrieval on the World Wide Web. In: Proceedings of the 23rd annual international ACM/SIGIR conference. New York: ACM Press, 2000, pp. 288–295.

Lin

Quan

Sinha

. What makes a good answer? The role of context in question answering. In: Proceedings of the ninth IFIP TC13 international conference on human–computer interaction. Amsterdam: IOS Press, 2003, pp. 25–32.

Cardie

Wiebe

Wilson

. Low-level annotations and summary representations of opinions for multi-perspective question answering. In: Maybury

(ed.) New directions in question answering. Cambridge, MA: AAAI Press/MIT Press, 2004.

Stoyanov

Cardie

Wiebe

. Multi-perspective question answering using the OPQA corpus In: Proceedings of the human language technology conference and conference on empirical methods in natural language, 2005, pp. 923–930.

Lin

. Evaluation of resources for question answering evaluation. In: Proceedings of the 28th annual international ACM SIGIR conference. New York: ACM Press, 2005.

Jeon

Croft

Lee

. A framework to predict the quality of answers with non-textual features. In: Proceedings of the 29th annual international ACM SIGIR conference. New York: ACM Press, 2006, pp. 228–235.

Pavlov

Choi

. Internet-scale collection of human-reviewed data. In: Proceedingsof the 16th international conference on World Wide Web. New York: ACM Press, 2007, pp. 231–240.

10.

Agichtein

Castillo

Donat

. Finding high-quality content in social media. In: Proceedings of the web search and data mining (WSDM). Stanford, CA: ACM Press, 2008, pp. 183–194.

11.

Yamamoto

Tezuka

Jatowt

. Supporting judgment of fact trustworthiness by considering temporal and sentimental aspects. In: Proceedings of the 9th web information systems engineering conference (WISE2008), Auckland. Lecture Notes in Computer Science, Vol. 5175. Berlin: Springer, 2008, pp. 206–220.

12.

Cruchet

Gaudinat

Rindflesch

. What about trust in the question answering world? In: Proceedings of AMIA 2009 annual symposium, 2009.

13.

Marian

. A framework for corroborating answers from multiple web. Information Systems2011; 36(2): 431–449.

14.

Suryanto

Lim

Sun

Chiang

RHL

. Chiang RHL. Quality-aware collaborative question answering: Methods and evaluation. In: Proceedings Of the Web Search and Data Mining (WSDM). Stanford, CA: ACM Press, 2009, pp. 142–151.

15.

H-J

Lee

. Analysis of the empirical of contextual matching advertising for online news. ETRI Journal2012; 34(2): 292–295.

16.

H-J

Myaeng

Jang

. Effects of answer weight boosting in strategy-driven question answering. Information Processing and Management2012; 48(1): 83–93.

17.

Kim

K-S

Sin

S-CJ

. Selecting quality sources: Bridging the gap between the perception and use of information sources. Journal of Information Science2011; 37(2): 178–188.

18.

H-J

Sung

Myaeng

Jang

. Compositional question answering: A divide and conquer approach. Information Processing & Management2011; 47(6): 808–824.

19.

Lee

Shin

Lim

. Document quality evaluation for question answering system. In: Proceedings of the 20th conference of Hangul and Korean information processing (in Korean, 2008), pp. 176–181.

20.

Lee

Jang

. A modified fixed-threshold SMO for 1-slack structural SVMs. ETRI Journal 2010; 32(1): 120–128.

21.

Lin

. Is question answering better than information retrieval? A task-based evaluation framework for question series. In: Proceedings of the human language technology conference and the North American chapter of the Association for Computational Linguistics annual meeting (HLT/NAACL), 2007, pp. 212–219.

22.

Dang

Lin

. Different structures for evaluating answers to complex questions: Pyramids won’t topple, and neither will human assessors. In: Proceedings of the 45th annual meeting of the Association for Computational Linguistics (ACL), 2007, pp. 768–775.

23.

Robertson

Walker

. Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: Proceedings of the 17th annual international ACM SIGIR conference, 1994, pp. 232–241.