 The standard was proposed by Martijn Koster,
when working for Nexor
in February 1994
on the "www-talk" mailing list, the main communication channel for WWW-related activities at the time.Charles Stross claims to have provoked Koster to suggest robots.txt, after he wrote a badly-behaved web crawler that inadvertently caused a denial-of-service attack on Koster's server.It quickly became a de facto standard that present and future web crawlers were expected to follow; most complied, including those operated by search engines such as WebCrawler, Lycos, and AltaVista.