{"push_id":576746119,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"2d6e5b1adc4fde0015046032b6f550dfa4ad2a92","before":"ddad293f96db6ea4186d41371ee0ad44158ab6c3","commits":[{"sha":"2d6e5b1adc4fde0015046032b6f550dfa4ad2a92","author":{"email":"0bfe415ab19aa6572559b0d089be1bc2286aa182@pointpro.nl","name":"Egbert van der Wal"},"message":"Update the RobotstxtServer:\n- added setting ignoreUserAgentInAllow which specifies if the user-agent\nshould be ignored in allow rules. This basically makes all allow rules\napply to all user agents. However, they can still be overridden by a\nDisallow-rule that specifically targets the configured user agent.\n\n- changed decision logic: a URL is now not considered allowed when it is\neither allowed and not disallowed by either the specific and the\nwildcard user agents. Instead, when a User-agent is specifically\ndisallowed access, this cannot be overriden anymore by the more general\ncase.","distinct":true,"url":"https://api.github.com/repos/MadEgg/crawler4j/commits/2d6e5b1adc4fde0015046032b6f550dfa4ad2a92"}]}