The ClueWeb09 dataset and its successor, the ClueWeb12 dataset, are two of the largest collections of Web pages released by Text REtrieval Conference (TREC). The ClueWeb datasets were used in various tracks of TREC ran through 2009 to 2017. For every year, approximately 50 new queries are released and a pool of Web pages are judged against these queries by human assessors as relevant, non-relevant or spam. In this article, a ground truth for binary classification (spam vs non-spam) is constructed from Web pages that are judged as spam or relevant under the assumption that a Web page judged as relevant for any query cannot be spam. Based on this ground truth, we evaluate classification performances of the Waterloo spam rankings (Fusion, Britney, GroupX and UK2006), which have been traditionally used to identify and filter spam pages in retrieval systems. The experimental results in terms of the universal binary classification evaluation measures suggest that the Fusion (with threshold = 11%) is the best for the ClueWeb09 dataset. Analysis of the frequency distributions of relevant/spam documents over spam scores reveals that the GroupX is the most powerful at identifying relevant documents, whereas the Fusion is the most powerful at identifying spam documents. It is also confirmed that the effectiveness of the Fusion spam ranking of the ClueWeb12 dataset is not as good as that of the ClueWeb09.
CormackGVSmuckerMDClarkeCLA. Efficient and effective spam filtering and re-ranking for large web datasets. Inform Retrieval2011; 14(5): 441–465.
4.
ZucconGNguyenALeelanupabT, et al. Indexing without spam. In: CunninghamSJScholerFThomasP (eds) Proceedings of the 16th Australasian document computing symposium (ADCS ’11). Melbourne, VIC, Australia: RMIT University, 2011, pp. 6–13.
5.
CraneMTrotmanA. Effects of spam removal on search engine efficiency and effectiveness. In:Proceedings of the 17th Australasian document computing symposium (ADCS ’12), Dunedin, New Zealand, 5–6 December 2012, pp. 1–8. New York: ACM.
6.
GallagherLMackenzieJCulpepperJS.Revisiting spam filtering in web search. In: Proceedings of the 23rd Australasian document computing symposium (ADCS ’18), Dunedin, New Zealand, 11–12 December 2018, pp. 1–5. New York: ACM.
7.
ArslanA.How sensitive are the term-weighting models of information retrieval to spam Web pages?Inform Process Lett2019; 144: 16–24.
CarteretteBCloughPHallM, et al. Evaluating retrieval over sessions: the TREC session track 2011-2014. In: Proceedings of the 39th international ACM SIGIR conference on research and development in information retrieval (SIGIR ’16), Pisa, 17–21 July 2016, pp. 685–688. New York: ACM.
WangLBennettPNCollins-ThompsonK.Robust ranking models via risk-sensitive optimization. In: Proceedings of the 35th international ACM SIGIR conference on research and development in information retrieval (SIGIR ’12), Portland, OR, 12–16 August 2012, pp. 761–770. New York: ACM.
12.
DinçerBTMacdonaldCOunisI. Hypothesis testing for the risk-sensitive evaluation of retrieval systems. In: Proceedings of the 37th international ACM SIGIR conference on research and development in information retrieval (SIGIR ’14), Gold Coast, QLD, Australia, 6–11 July 2014, pp. 23–32. New York: ACM.
13.
RodgerB. Evaluating risk-sensitive text retrieval. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, Paris, 21–25 July 2019, p. 1455. New York: ACM, http://doi.acm.org/10.1145/3331184.3331423
14.
PaikJH. A probabilistic model for information retrieval based on maximum value distribution. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval (SIGIR ’15), Santiago, Chile, 9–13 August 2015, pp. 585–594. New York: ACM.
15.
XiongCCallanJLiuTY. Bag-of-entities representation for ranking. In: Proceedings of the 2016 ACM international conference on the theory of information retrieval (ICTIR ’16), Newark, DE, 12–16 September 2016, pp. 181–184. New York: ACM.
16.
MacdonaldCDinçerBTOunisI. Transferring learning to rank models for web search. In: Proceedings of the 2015 international conference on the theory of information retrieval (ICTIR ’15), Northampton, MA, 27–30 September 2015, pp. 41–50. New York: ACM.
17.
VoorheesEM.TREC: continuing information retrieval’s tradition of experimentation. Commun ACM2007; 50(11): 51–54.
RobertsonSZaragozaH.The probabilistic relevance framework: BM25 and beyond. Found Trends Inf Ret2009; 3(4): 333–389.
22.
KocabaşIDinçerBTKaraoğlanB.A nonparametric term weighting method for information retrieval based on measuring the divergence from independence. Inform Retrieval2014; 17(2): 153–176.
23.
ClinchantSGaussierÉ.Retrieval constraints and word frequency distributions a log-logistic model for IR. Inform Retrieval2011; 14(1): 5–25.
24.
AmatiG.Frequentist and Bayesian approach to information retrieval. In: LalmasMMacFarlaneARügerS (eds) Advances in information retrieval (Lecture notes in computer science). vol. 3936. Berlin; Heidelberg: Springer, 2006, pp. 13–24.
25.
ArguelloJCraneMDiazF, et al. Report on the SIGIR 2015 workshop on reproducibility, inexplicability, and generalizability of results (RIGOR). SIGIR Forum2016; 49(2): 107–116.
26.
LinJCraneMTrotmanA, et al. Toward reproducible baselines: the open-source IR reproducibility challenge. In: Proceedings of the advances in information retrieval: 38th European conference on IR research (ECIR ’16), Padua, 20–23 March 2016. Cham: Springer.
27.
FerroN.Reproducibility challenges in information retrieval evaluation. J Data Inform Qual2017; 8(2): 81–84.
28.
FerroNFuhrNRauberA.Introduction to the special issue on reproducibility in information retrieval: tools and infrastructures. J Data Inform Qual2018; 10(4): 141–144.
