 For some years, the Internet Archive did not crawl sites with robots.txt, but in April 2017, it announced that it would no longer honour directives in the robots.txt files.“Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes".This was in response to entire domains being tagged with robots.txt when the content became obsolete.