Skip to navigation | Skip to content

Search engines lagging further behind the web

A US study says that the major internet search engines - software that allow users to find information on the internet - cover less than half of the publicly accessible internet. And their performance is getting worse as the web continues to grow at a spectacular rate.

The study, by computer scientists Lee Giles and Steve Lawrence, appears in the July 8 edition of Nature. They performed a statistical survey of the net to derive the total size of the publicly indexible internet and the scope and efficiency of the 11 most popular search engines. The publicly indexible 'net includes all webpages other than those requiring authorization to view, forms or those that are hidden from search engines.

Giles and Australian-born Lawrence, both of the NEC Research Institute, say that the web has grown from 320 million pages in December 1997 to some 800 million webpages in February 1999, including 180 million images and 3 terabytes of text.

But while the top 5 search engines covered 60% of the web in 1997, the combined coverage of the top 11 engines today is only 42%. The most comprehensive search engine in the world covers only about a sixth of all searchable webpages, whereas fifteen months ago the best engine covered a third.

Slow to update, and selective too

The study also suggests that the proportion of dead links appearing in search results is increasing, and that search engines take several months to include new or modified content. What's more, the technology that many search engines use means that any new content that appears is likely to come from sites that are already popular.

Search engines add new content in two ways. While users can submit their pages directly to a search engine, the main method uses search robots or 'spiders' - software that continuously surfs the web, saving references of pages they visit to an index and following links to find other pages. This means that if a page has more than one link pointing to it somewhere on the web, it is more likely to be indexed, and a page that isn't listed in the index will not be found.

Giles and Lawrence say their results suggest there may be an optimal size for a search engine, beyond which the cost of maintenance increases greatly and overall performance slows.

They also say that the best way to improve the coverage of the web by a search is to combine the results of multiple engines, or use a metasearch engine, which does this automatically.

Tags: information-technology