Mining Hidden Gems beneath the Surface: A Look at the Invisible Web

Abstract

The future of the information landscape is being shaped by new technologies that store and retrieve information. For many computer users, the Web is their first and last stop in information searching. Searches produce an overwhelming amount of returns, but may have few that are “on target.” Finding general information may be easy, but depth of information is frequently lacking. This article focuses on describing resources for researchers, called the Invisible Web, that are hidden from usual search tools and contrasting them with those resources available in the surface Web. It then identifies search tools and strategies that can be used to dig beneath the surface of the Web to locate credible, in-depth information. These resources must be accessed using specialized search tools and databases.

Get full access to this article

View all access options for this article.

References

Sullivan

, Search Engine Sizes, The Search Engine Report, April 6, 2001. Available online at http://searchenginewatch.com/sereport/, retrieved 5/10/01.

Lyman

Varian

H. R.

, How Much Information? Regents of the University of California, 2000. Available online at http://www.sims.berkeley.edu/research/projects/how-much-info/index.html, retrieved 5/10/01.

Lawrence

Giles

, Accessibility of Information on the Web, Nature, 400, pp. 107–109, 1999.

Bergman

, The Deep Web: Surfacing Hidden Value, Brightplanet.com LLC, 2000.

U.C. Berkeley Library, The Invisible Web: Database Contents Rarely Found with Search Engines, revised Spring 2001. Available online at http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html, retrieved 5/21/01.

Google, Our Search: Special Search Features. Available online at http://www.google.com/help/features.html#pdf, retrieved 5/21/01.

First Gov, Welcome to FirstGov: Facts about FirstGov. Available online at http://firstgov.gov/top_nav/about.html?ssid=990472740328_172, retrieved 5/21/01.