Hack Removal Service  1 January 2008 Comments: 5  Use Robots.txt
To Prevent vBulletin Duplicate Content  Using a robots.txt file
can tell search engine spiders what not to crawl on your site.
The file has to be named “robots.txt” and placed in the root of
your site for spiders to find, and read the file. You can not
have more than one robots.txt file for each site, sub-domains
are of course the exception, and treated as individual domains,
and you can place a robots.txt in the subdomain root.  You may
not want spiders to crawl certain sections of your site for
various reasons or perhaps if the page showed up in a search
engine result, it wouldn’t be useful for the reader. Many of the
extra URL’s that vBulletin produces are irrelevant to the end
user, and if they landed on one of the pages you could possibly
lose a valuable guest when they leave on the page they entered
from.  Google Webmaster Tools has a robots.txt generator that
you can use to help create the file. There are also a slew of
other ways to bloc