A study is reported where the rankings assigned by a search engine (Google) to the top 100 hits for seven queries are compared with the ratings of human judges (six per query). As assessed by correlations, agreement between judges was relatively good across all the queries, with a mean (Pearson) correlation of .475. In contrast, the correlations between Google rankings and human judges in this study were significantly lower, with a mean of .153. This study suggests that a high priority should be placed on finding improved methods for ranking the initial set of search engine results, so that they are more in line with human judgments of relevance.
Get full access to this article
View all access options for this article.
References
1.
AmentoB.TerveenL.HillW. (2000). Does “Authority” Mean Quality? Predicting Expert Quality Ratings of Web Documents. SIGIR 2000. p. 296–303.
2.
BorgattiS.EverettM.FreemanL. (2001). UCINET5, Software for social Network Analysis: User Guide
3.
BrinS.PageL. (1998). The anatomy of a large-scale hypertextual Web search engine. WWW'7, pp.107–117
4.
ChakrabartiS. (1999). Mining the link structure of the World Wide Web. IEEE Computer 32, 8
5.
CooperW. S. (1971). A definition of relevance for information retrieval. Information storage and retrieval, 7(1), pp. 19–37
FreemanL. (2000). Social network analysis: Definition & history, Encyclopedia of Psychology, Oxford Univ. Press
10.
FroehlichD. J. (1994). Relevance reconsidered – towards an agenda for the 21st century: Introduction to special topic issue on relevance research. Journal of the American Society for Information and Technology, 45(3) pp. 124–1333
11.
GarfieldE. (1979). Citation Indexing. ISI Press
12.
GordonM.PathakP. (1999). Finding information on the World Wide Web: The retrieval effectiveness of search engines. Information Processing and Management. 35(2), p 141–180.
13.
HannemanA. Robert (1998). Introduction to Social Network Methods. Dept. of Sociology, Univ. of California
14.
HarterS. P. (1992). Psychological relevance and information science. Journal of the American Society for Information and Technology, 43(9) pp. 602–615
15.
HenzingerMonika. (2000). Link Analysis in Web Information Retrieval. IEEE Bulletin on Data Engineering.
16.
HowardD. L. (1994). Pertinence as reflected in personal constructs. Journal of the American Society for Information and Technology, 45(3) pp. 172–185
17.
LarsonR. (1996). Bibliometrics of the World Wide Web: An Exploratory Analysis of the Intellectual Structure of Cyberspace, ASIS'1996
18.
MaglaughlinKellySonnenwaldDiane (2002). User perspectives on relevance criteria: A comparison among relevant, partially relevant and non-relevant judgements, Journal of the American Society for Information and Technology, 53(5) pp. 327–342
19.
MarchioriM. (1997). The quest for correct information on the Web: Hyper search engines. WWW'6
20.
ParkH. (1994). Toward a theory of user-based relevance: A call for a new paradigm of inquiry., Journal of the American Society for Information and Technology, 45(3) pp. 135–141
21.
ParkH. (1997). Relevance of science information: Origins and dimensions of relevance & their implications to information retrieval. Information Processing and Management. 33(3) pp. 339–352
22.
SchamberL. (1994). Relevance and information behavior. Annual review of information science and technology. 29, pp. 3–48
23.
SmallH.GriffithB. (1974). The structure of scientific literatures: Identifying and graphing specialties. Science Studies, 4 (17), PP. 17–40
24.
JonesSparck K.Van RijsbergenC. (1997). Progress in documentation. Journal of Documentation. 32(1) pp. 59–75 or 1976
25.
SwansonD. R. (1986). Subjective versus objective relevance in bibliographic retrieval systems. The library quarterly, 56, pp. 389–398
26.
YaltaghianB.ChignellM. (2000). Co-citation analysis of search engine results: Using organization to facilitate browsing. IML Technical Report
27.
WilsonP. (1973). Situational relevance. Information Storage and Retrieval. 9(8) pp. 457–471
28.
WuM.FullerM.WilkinsonR. (2000). The Role of a Judge in a User Based Retrieval Experiment. SIGIR2000. p. 331–333.
