 If this file doesn't exist, web robots assume that the website owner does not wish to place any limitations on crawling the entire site.A robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a site.This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operates on certain data.