Longitudinal study of content and elements in the scientific web environment

Abstract

This work is a longitudinal study of the evolution and the state of 738 web sites at two different points in time (1997 and 2004). It tries to establish the rate of growth and decay of the web and all the web elements. To this end, the structure and the contents of these web sites are extracted through a crawler and compared at the two different moments in time. The main results confirm a growth of web content and elements in the web, although there is also a high degree of web content decay. The results suggest that in the seven-year period covered by this study the web is characterized by both strong dynamism and instability.

Keywords

webometrics web persistence web growth web decay linkrot bibliometrics

Get full access to this article

View all access options for this article.

References

D. Pennock , G. W. Flake , S. Lawrence , E. J. Glover and C. L. Giles , Winners don’t take all: characterizing the competition for links on the web , Proceedings of the National Academy of Sciences 99(8) (2002) 5207-5211 . Available at: www.pnas.org/cgi/reprint/99/8/5207 (accessed 28 October 2005).

Internet Systems Consortium, Inc, Redwood, CA (2004). Available at: www.isc.org/index.pl?/ops/ds/ (accessed 28 October 2005).

E. T. O’Neill , B. F. Lavoie and R. Bennett , Trends in the evolution of the public web 1998-2002 , D-Lib Magazine 9(4) (2003). Available at www.dlib.org/dlib/april03/lavoie/04lavoie.html (accessed 27 March 2006).

S. Harter and H. Kim , Electronic journals and scholarly communication: a citation and reference study , Information Research 2(1) (1996). Available at: http://informationr.net/ir/2-1/paper9a.html (accessed 28 October 2005).

S. Lawrence , F. Coetzee , E. Glover , D. Pennock , G. Flake , F. Nielsen , B. Krovetz , A. Kruger and L. Giles , Persistence of Web references in scientific research , IEEE Computer 34(2) (2003) 26-31 .

W. Koehler , An analysis of Web page and Web site constancy and permanence , Journal of the American Society for Information Science 50(2) (1999) 162-180 .

W. Koehler , Web page change and persistence - a four-year longitudinal study , Journal of the American Society for Information Science and Technology 53(2) (2002) 162-171 .

W. Koehler , A longitudinal study of Web pages continued: a report after six years , Information Research 9(2) (2004). Available at: http://informationr.net/ir/9-2/paper174.html (accessed 28 October 2005).

M. Nelson and B. Allen , Object persistence and availability in digital libraries , D-Lib Magazine 8(1) (2002). Available at: www.dlib.org/dlib/january02/nelson/01nelson.html (accessed 28 October 2005).

10.

D. Fetterly , M. Manasse , M. Najork and J. L. Wiener , A large-scale study of the evolution of Web pages , Software Practice and Experience 1(1) (2003) 1-27 .

11.

J. Cho and H. García-Molina , The evolution of the web and implications for an incremental crawler . In: A. El Abbadi et al. (eds), Proceedings of the 26th International Conference on Very Large Databases, Cairo, 10-14 September 2000 (Morgan Kaufmann, 2000) 200-209 .

12.

J. Bar-Ilan and B. C. Peritz , evolution, continuity, and disappearance of documents on a specific topic on the Web: a longitudinal study of ‘informetrics’ , Journal of the American Society for Information Science and Technology 55(11) (2004) 980-990 .

13.

P. Wouters , I. Hellsten and L. Leydesdorff , Internet time and the reliability of search engines , First Monday 9(10) (2004) Available at: www.firstmonday.org/issues/issue9_10/wouters/ (accessed 28 October 2005).

14.

J.L. Ortega, J.A. Prieto, N. Arroyo, V.M. Pareja and I.F. Aguillo, Análisis de la persistencia y del estado de páginas web en los resultados de Google. In: 9^[UNKNOWN]a Jornadas Españolas de Documentación FESABID 2005, Madrid, 14 y 15 de Abril (2005). Available at: http://internetlab.cindoc.csic.es/cv/11/Ortega2005.pdf (accessed 28 October 2005).

15.

NetCarta.com, NetCarta WebMap Library (1997). Available at: www.netcarta.com/ (accessed 16 April 1997). [No longer available as the resource was purchased by Microsoft - this reference provided for historical completeness]

16.

N. Arroyo , V. Pareja and I. Aguillo , Description of Web Data in D3.1. (2003) Available at: www.eicstes.org/EICSTES_PDF/Deliverables/Web%20data%20description.PDF (accessed 28 October 2005). [Deliverable. IST-1999-20350]

17.

T. Hausherr , Xenu’s Link Sleuth. Version 1.2f [s. l.] (?1997-2004). Available at: http://home.snafu.de/tilman/xenulink.html (accessed 28 October 2005).