 In addition to root-level robots.txt files, robots exclusion directives can be applied at a more granular level through the use of Robots meta tags and X-Robots-Tag HTTP headers.The robots meta tag cannot be used for non-HTML files such as images, text files, or PDF documents.On the other hand, the X-Robots-Tag can be added to non-HTML files by using .htaccess and httpd.conf files.