Take the 2-minute tour ×  I'm trying to follow stackoverflow's
approach of using a white list for the sitemap. I have tried to
find an extensive list of search engine crawlers (and other
crawlers) to use in my white list but what I have found so far
is outdated information (hasn't been updated in years). Is there
any place on the net where I can find a regularly updated list
that contains the user agent of every search engine crawler
along with its domain name (for reverse DNS)?   share|improve
this question  3 Answers 3  The question you referenced
indicates that the answerer is whitelisting the user agents
themselves, not the domains or IP addresses associated with
particular spiders.  It appears as though user-agents.org has an
exhaustive list if you plan to work with the user-agent string.
share|improve this answer Whitelisting the user agents isn't
enough as they can easily be spoofed (see
meta.stackexchange.com/questions/37231/…). I haven't mentioned
anything about IP addresses but I need the domain na