[an error occurred while processing this directive]
BBC News
watch One-Minute World News
Last Updated: Friday, 2 January, 2004, 12:19 GMT
Finding what you want on the web
Good searching without Google would not be easy, realises technology analyst Bill Thompson.

Library books
How do you find what you want?
Having made it my New Year resolution to break my addiction to Google and get better at searching the web, I have spent the last couple of weeks looking at the state of search and alternative places to go.

For most of the time, most of the research I am doing covers topics which have not yet made it into books, so libraries are not an option.

And there is enough reference material now available online that I do not need a multi-volume encyclopaedia sitting next to me either.

However, I do need to make sure that I am searching the online world as effectively and efficiently as possible.

I know how to make sense of what I find, and how to decide if a site is credible.

I know how to follow a chain of links from a promising site, and how to look for supporting evidence when I find something which looks odd.

I just need to find the stuff in the first place.

Wide-ranging web indices

I am even interested in searching the vast number of blogs out there, because they often have information which illuminates stories and subjects, and can give pointers to other online resources.

And I know that there is more to online life than the world wide web. I want to look through Usenet news and at databases like the medical research source, Medline which are accessed through the web but not indexed by search engines.

The process of looking for an alternative to Google was made a lot easier by having Danny Sullivan's SearchEngineWatch constantly open in one browser window while I worked.

Bill Thompson
Good online research has to involve more than just typing a couple of words into a search form and clicking a button
I also used the Open Directory Project, a human-edited, non-commercial directory of the web, since it has a good list of search sites that are not skewed by self-interest.

It is clear that, despite its faults, Google remains one of the largest, fastest and most usable search tools around.

So I have not uninstalled the toolbar, and probably never will.

I also like Google News, and since they own the best archive of newsgroup postings, I am never going to get away from them completely.

But I have started looking at a wider range of web indexes, directories and gateways to non-web resources, and it has been really useful.

Apart from major sites like AllTheWeb, Teoma and MSN Search, I have rekindled my interest in Yahoo! as a search site - even though it does return Google search results too - and remembered why I liked HotBot all those years ago.

I am still not keen on the meta search engines like Kartoo, Fazzle and Vivisimo.

These search several other places for you and then re-arrange the results into their own categories and clusters, and they do not really work with my personal searching style, but I can see that others will appreciate them.

Not all of the resources I found are freely available - Maine InfoNet is only for residents of Maine - but knowing they are out there will spur me into finding out what I can get access to from my local library.

And not all are free. FT.com's world media search is only available to paying subscribers.

But they are out there, and there are certainly benefits to paying for access to a database up front instead of having to filter out adverts or wonder whether the results you are seeing are skewed by undeclared pay-for-placement deals.

Searching well

During my searching I found a guide to how to choose the best search for your information requirements, written by librarian Debbie Abilock for her NoodleTools website.

It lists dozens of different resources, each suitable for different sorts of search.

Debbie is a trained librarian, and it shows. She understands that a single search engine is never going to do everything, no matter how good its indexing or how large its database.


Google logo
Do we have a dumb web?
Partly this is because the web is just too big, and getting bigger. Much web content is hidden behind firewalls or subscription services.

But there is a more fundamental problem with search, because the web as it currently exists is based around links which carry no additional information about the nature of the connection being made between the items involved.

Anyone trying to assign significance to a web page in order to rank its importance is left counting links - rather like Google does - and then using a series of best-guess algorithms to decide what those links signify.

It simply does not scale, and the result is the mess we see today when searching for any common word or phrase on a major search engine.

In fact, even Google recognises this by separating out results from "news" sites and putting them in a separate index.

Because it is possible to make a reasonable inference about the nature of the links between them, it can do a better job of creating categories and grouping stories together than it can with any other type of web page.

I do not think we will ever solve the search problem until we move away from the dumb web we have today towards something like the semantic web, a project that Sir Tim Berners-Lee has been pushing ever since the first web conference in 1994.

Once links carry meaning then it will become more of a distributed database than the vast heap of unstructured documents we have built so far.

And once we have a database then we can classify, index and search it properly. But until that day, the list of search sites in my bookmarks will have to remain as large and unwieldy as it is today.



The BBC is not responsible for the content of external internet sites


Americas Africa Europe Middle East South Asia Asia Pacific