This study presents the results of an extensive search for links to the home page of the e-journal Cybermetrics. The results show that the search engines do not retrieve all the link pages that are indexed by them. In the specific case, the search engine Google concealed between 48 and 70% of the links to the page each time it was queried, and HotBot concealed between 20 and 39% of the link pages indexed by it. The queries were repeated four times during a one-year period, between January 2001 and January 2002 in order to rule out the possibility of an accidental finding. The other search engines examined also concealed some pages but to a much smaller extent. The findings raise questions about the use of WIF (the Web Impact Factor) as a scientometric indicator based on data retrieved from commercial search engines. The content of the retrieved and concealed pages was characterized using the method of content analysis. The characterization shows that the set of initially retrieved pages, and the set of initially retrieved pages plus the set of concealed pages, are significantly different for Google.
Get full access to this article
View all access options for this article.
References
1.
[1] R. LarsonBibliometrics of the World Wide Web: an exploratory analysis of the intellectual structure of cyberspace . In: ASIS96 (1966). Available at: http://sherlock.berkeley.edu/asis96/asis96.html (access date September 2001).
2.
[2] R. Rousseau , Sitations: an exploratory study , Cyber-metrics1(1) (1997). Available at: www.cindoc.es/cybermetrics/articles/v1i1p1.htm (access date August 2001).
3.
[3] P. Ingwersen , The calculation of Web impact factors , Journal of Documentation, 4(2) (1998) 236-243 .
4.
[4] L. Egghe , New informetric aspects of the internet: some reflections - many problems , Journal of Information Science26(5) (2000) 329-335 .
5.
[5] S. Brin and L. Page , The anatomy of a large-scale hyper-textual web search engine . In: Proceedings of the 7th International World Wide Web Conference, April 1998, Computer Networks and ISDN Systems , Vol. 30, pp. 107-117 . Available at: www7.scu.edu.au/programme/fullpapers/1921/com1921.htm (access date: September 2001).
6.
Also appeared in Journal of the ACM46(5) (1999) 604-632 . Available at: www.cs.cornell.edu/home/kleinber/auth.ps (access date: September 2001).
7.
[7] G. Pinski and F. Narin , Citation influence for journal aggregates of scientific publications: theory, with application to the literature of physics , Information Processing and Management12 (1976) 297-312 .
8.
[8] L. Katz , A new status index derived from sociometric analysis , Psycometrika18(1) (1953) 39-43 .
9.
[9] B. Cronin , Bibliometrics and beyond: some thoughts on web-based citation analysis , Journal of Information Science27(1) (2001) 1-7 .
10.
[10] A.G. Smith , A tale of two web spaces: comparing sites using web impact factors , Journal of Documentation55(5) (1999) 577-592 .
11.
[11] M. Thelwall , Web impact factors and search engine coverage , Journal of Documentation56(2) (2000) 185-189 .
12.
[12] M.A. Boudourides , B. Sigrist and P. D. Alevizos , Webometrics and the Self-Organization of the European Information Society (1999). Available at: http://hyperion.math.upatras.gr/webometrics/ (access date March, 2001).
13.
[13] I. Aguillo and V. M. Pareja , Indicators of the Internet presence of the Western European Research Councils . Poster Presentation at the S&T 2000, Leiden, The Netherlands (May 2000). Available at: http://sahara.fsw.leidenuniv.nl/cwts/abs/AGUILLO.txt (access date March 2001).
14.
[14] L. Leydesdorff and M. Curran , Mapping university-industry-government relations on the internet: the construction of indicators for a knowledge-based economy , Cybermetrics4(1) paper 2 (2000). Available at: www.cindoc.csis.es/cybermetrics/articles/v4i1p2.html (access date November 2001).
15.
[15] J. Bar-Ilan , Search engine results over time - a case study on search engine stability , Cybermetrics2/3(1) paper 1 (1999). Available at: www.cindoc.csic.es/cybermetrics/articles/v2i1p1.html (access date 20 February 2002).
16.
[16] J. Bar-Ilan , Evaluating the stability of the search tools hotbot and snap: a case study , Online Information Review24(6) (2000) 430-450 .
17.
[17] J. Bar-Ilan , Methods for measuring search engine performance over time , Journal of The American Society for Information Science and Technology54(3) (2002) 308-319 .
18.
[18] R. Rousseau , Daily time series of common single word searches in AltaVista and Northern Light , Cybermetrics2/3(1) paper 2 (1999). Available at: www.cindoc.csis.es/cybermetricc/articles/v2i1p2.html (access date November, 2001).
19.
[19] H. Snyder and H. Rosenbaum , Can search engines be used as tools for web-link analysis? A critical view , Journal of Documentation55(4) (1999) 375-384 .
20.
[20] K. Bharat and A. Broder , A technique for measuring the relative size and overlap of public web search engines . In: Proceedings of the 7th International World Wide Web Conference, April 1998, Computer Networks and ISDN Systems , Vol. 30, pp. 379-388 . Available at: http://decweb.ethz.ch/WWW7/1937/com1937.htm (access date November, 2001).
21.
[21] J. Bar-Ilan , On the overlap, the precision and estimated recall of search engines - a case study of the query ‘Erdos’ , Scientometrics42(2) (1998) 207-228 .
22.
[22] S. Lawrence and C. L. Giles , Accessibility and distribution of information on the Web , Nature400 (1999) 107-109 .
23.
[23] H. Chu and M. Rosenthal , Search engines for the World Wide Web: a comparative study and evaluation methodology . In: ASIS96 (1996), pp. 127-135 . Available at: www.asis.org/annual-96/ElectronicProceedings/chu.html (access date: March 2001).
24.
[24] W. Ding and G. Marchionini , A comparative study of Web search service performance . In: ASIS96 (1996) 136-142 .
25.
[25] H. V. Leighton and J. Srivastava , First 20 precision among World Wide Web search services (search engines) , Journal of the American Society for Information Science50(10) (1999) 870-881 .
26.
[26] S. J. Clarke and P. Willett , Estimating the recall performance of web search engines , ASLIB Proceedings49(7) (1997) 184-189 .
27.
[27] W. Mettrop and P. Nieuwenhuysen , Internet search engines - fluctuations in document accessibility , Journal of Documentation57(5) (2001) 623-651 .
28.
[28] I. Aguillo , More on search engines with scientometric purposes. In: SIGMETRICS Discussion List (2001). Available at: http://listserv.utk.edu/cgi-bin/wa?A2=ind0101&L=sigmetrics&P=R1366 (access date February 2002).
29.
[29] K. Krippendorff , Content Analysis - an Introduction to its Methodology ( Sage, Beverly Hills, CA , 1980).
30.
[30] B. Cronin , H.W. Snyder , H. Rosenbaum , A. Martinson and E. Callahan , Invoked on the Web , Journal of the American Society for Information Science49(14) (1998) 1319-1328 .
31.
[31] J. Bar-Ilan , Results of an extensive search for ‘s&t indicators’ on the web - a content analysis , Scientometrics49(2) (2000) 257-277 .
32.
[32] B. Mannina and L. Quoniam , How to hold a virtual library active?Cybermetrics4(1) paper 1 (2000). Available at: www.cindoc.csic.es/cybermetrics/articles/v4i1p1.html (access date May 2001).
33.
[33] G.R. Notess , The many faces of Inktomi , Database22(2) (1999) 63-65 . Available at: http://onlineinc.com/database/DB1999/net4.html (access date May 2001).
34.
[34] I.F. Aguillo , STM information on the Web and development of new Internet R&D databases and indicators . In: Online Information 98 Proceedings (1998) 239-243 .
35.
[35] M. Thelwall , Extracting macroscopic information from Web links , Journal of the American Society for Information Science and Technology52(13) (2001) 1157-1168 .
36.
[36] Inktomi , Inktomi Webmap ( Inktomi Press Releases , 2000). Available at: www.inktomi.com/webmap/ (access date May 2001).
37.
[37] D. Sullivan , Nielsen//Net Ratings - Search Engine Ratings (2002). Available at: www.searchenginewatch.com/reports/netratings.html (access date March 2002).
38.
[38] D. Sullivan , Search Engine Sizes (2002). Available at: www.searchenginewatch.com/reports/sizes.html (access date March 2002)
39.
[39] M. Thelwall , Results from a Web Impact Factor crawler , Journal of Documentation57(2) (2001) 177-191 .
40.
[40] J. Bar-Ilan , Data collection methods on the web for informetric purposes - a review and analysis , Scientometrics50(1) (2001) 7-32 .
