 Robots are often used by search engines to categorize websites.Not all robots cooperate with the standard; email harvesters, spambots, malware and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out.The standard can be used in conjunction with Sitemaps, a robot inclusion standard for websites.