Take the 2-minute tour ×  Adding this to my robots.txt  User-
agent: * Disallow: /*action=*$  How does robots not recognizing
wild cards handle this?  share|improve this question add comment
2 Answers  Robots that do not recognize wildcards (which is not
in the official spec) will treat * as a literal character. The
fact that it is not a valid URL character may mean that they
ignore the rule altogether. In either case, it likely means that
the rule will have no effect on them.  This will depend a bit on
the exact implementation of the crawlers robot.txt honoring
scheme and can not be entirely counted on.  If you want to avoid
this you could have a separate configuration for googlebot (and
others who do honor robots.txt.   User-agent: * Disallow: /
User-Agent: Googlebot Disallow: /*action=*$  Which bans all
robots except Googlebot which will honor the wildcard
configuration.  share|improve this answer Good point with
'others who do honor robots.txt'. There are a number of SEs out
there that don't play by the