 Links to pages listed in robots.txt can still appear in search results if they are linked to from a page that is crawled.A robots.txt file covers one origin.For websites with multiple subdomains, each subdomain must have its own robots.txt file.