Detecting similar question is a fundamental and essential research problem for constructing similar question dataset for the research of question-answering, short text similarity calculating, and sentence paragraphing. This paper explores the previous assumption about similar question detection and analyzes its existing problem. Afterwards, we propose an automated approach to detecting similar questions based on the calculation of question topical diversity using different ways of topical feature generation methods. The experiment dataset are Yahoo! 4,482,757 questions with answers. The results present that our approach achieves a precision of 74% and a recall of 74% as the best performance compared with baseline methods, demonstrating its effectiveness in similar question group detection.
P.Achananuparp, X.H.Hu and X.J.Shen, The evaluation of sentence similarity measures, in: Proceedings of the 10th International Conference on Data Warehousing and Knowledge Discovery, 2008, pp. 2305–2316.
2.
P.Achananuparp, X.H.Hu, X.H.Zhou and X.D.Zhang, Utilizing sentence similarity and question type similarity to response to similar questions in knowledge-sharing community, in: Proceedings of QAWeb 2008 Workshop, Beijing, China, 2008.
3.
Baidu Zhidao, http://zhidao.baidu.com/, accessed in 2016.
4.
D.Bernhard and I.Gurevych, Answering learners’ questions by retrieving question paraphrases from social Q&A sites, in: Proceedings of the Third ACL Workshop on Innovative Use of NLP for Building Educational Applications, 2008, pp. 44–52.
5.
D.M.Blei, A.Y.Ng and M.I.Jordan, Latent Dirichlet allocation, Journal of Machine Learning Research3 (Jan 2003), 993–1022.
6.
D.Bogdanova, C.Santos, L.Barbosa and B.Zadrozny, Detecting semantically equivalent questions in online user forums, in: Proceedings of the 19th Conference on Computational Language Learning, 2015, pp. 123–131.
7.
Y.S.Chan and H.T.Ng, MAXSIM: A maximum similarity metric for machine translation evaluation, in: Proceedings of ACL-08: HLT, 2008, pp. 55–62.
8.
H.Duan, Y.Cao, C.Y.Linet al., Searching questions by identifying question topic and question focus, in: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT’08), Columbus, OH, USA, June 15–20, 2008, pp. 156–164.
9.
Gingerit, http://www.gingersoftware.com, accessed in 2016.
10.
F.Godin, V.Slavkovikj, W.D.Neve, B.Schrauwen and R.V.Walle, Using topic models for Twitter hashtag recommendation, in: Proceedings of the 22nd International Conference on World Wide Web, 2013, pp. 593–596. doi:10.1145/2487788.2488002.
11.
T.Y.Hao and E.Agichtein, Finding similar questions in collaborative question answering archives: Toward bootstrapping-based equivalent pattern learning, Information Retrieval15(3–4) (2012), 332–353. doi:10.1007/s10791-012-9188-x.
12.
J.Jeon, W.B.Croft and J.H.Lee, Finding similar questions in large question and answer archives, in: Proceedings of ACM Fourteenth Conference on Information and Knowledge Management, 2005, pp. 76–83.
13.
J.Jeon, W.B.Croft and J.H.Lee, Finding semantically similar questions based on their answers, in: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, 2005, pp. 617–618.
14.
J.Jeon, W.B.Croft and J.H.Lee, Finding similar questions in large question and answer archives, in: Proceedings of the 14th ACM International Conference on Information and Knowledge Management (CIKM), New York, NY, 2005, pp. 84–90.
15.
J.Jeon, W.B.Croft, J.H.Lee and S.Park, A framework to predict the quality of answers with non-textual features, in: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, 2006, pp. 228–235.
16.
Z.C.Ji, F.Xu, B.Wang and B.He, Question-answer topic model for question retrieval in community question answering, in: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, 2012, pp. 2471–2474.
17.
V.Lopez, M.Pasin and E.Motta, AquaLog: An ontology-portable question answering system for the semantic web, in: Proceedings of European Semantic Web Conference, 2005, pp. 546–562.
18.
C.D.Manning, P.Raghavan and H.Schütze, Introduction to Information Retrieval, Cambridge University Press, Cambridge, New York, 2008, xxi, 482 p.
19.
O.Owoputi, B.O’Connor, C.Dyer, K.Gimpel and N.Schneider, Improved part-of-speech tagging for online conversational text with word clusters, in: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2013, pp. 380–391.
20.
C.Santos, L.Barbosa, D.Bogdanova and B.Zadrozny, Learning hybrid representations to retrieve semantically equivalent questions, in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015, pp. 694–699.
21.
Sina IAsk, http://iask.sina.com.cn/, accessed in 2016.
22.
W.P.Song, M.Feng, N.J.Gu and W.Y.Liu, Question similarity calculation for FAQ answering, in: Proceedings of SKG 07, 2007, pp. 298–301.
23.
K.Wang, Z.Ming and T.S.Chua, A syntactic tree matching approach to finding similar questions in community-based QA services, in: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, 2009, pp. 187–194.
24.
Yahoo! Answers, https://answers.yahoo.com/, accessed in 2016.
