{"action":"created","issue":{"url":"https://api.github.com/repos/sjdirect/abot/issues/96","labels_url":"https://api.github.com/repos/sjdirect/abot/issues/96/labels{/name}","comments_url":"https://api.github.com/repos/sjdirect/abot/issues/96/comments","events_url":"https://api.github.com/repos/sjdirect/abot/issues/96/events","html_url":"https://github.com/sjdirect/abot/issues/96","id":114648654,"number":96,"title":"All urls found were rejected by robots.txt file.","user":{"login":"LeMoussel","id":1816573,"avatar_url":"https://avatars.githubusercontent.com/u/1816573?v=3","gravatar_id":"","url":"https://api.github.com/users/LeMoussel","html_url":"https://github.com/LeMoussel","followers_url":"https://api.github.com/users/LeMoussel/followers","following_url":"https://api.github.com/users/LeMoussel/following{/other_user}","gists_url":"https://api.github.com/users/LeMoussel/gists{/gist_id}","starred_url":"https://api.github.com/users/LeMoussel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LeMoussel/subscriptions","organizations_url":"https://api.github.com/users/LeMoussel/orgs","repos_url":"https://api.github.com/users/LeMoussel/repos","events_url":"https://api.github.com/users/LeMoussel/events{/privacy}","received_events_url":"https://api.github.com/users/LeMoussel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-02T17:57:27Z","updated_at":"2015-11-02T18:40:54Z","closed_at":null,"body":"Can't crawl this site: http://www.provectio.fr\r\nSeems that all urls found were rejected by robots.txt file. For example, this URL is rejected: http://www.provectio.fr/provectio-informatique/ \r\nI use this userAgentString : \"Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko\"\r\nIn Google Search Console, I found no errors for the robots.txt file & Urls."},"comment":{"url":"https://api.github.com/repos/sjdirect/abot/issues/comments/153120138","html_url":"https://github.com/sjdirect/abot/issues/96#issuecomment-153120138","issue_url":"https://api.github.com/repos/sjdirect/abot/issues/96","id":153120138,"user":{"login":"LeMoussel","id":1816573,"avatar_url":"https://avatars.githubusercontent.com/u/1816573?v=3","gravatar_id":"","url":"https://api.github.com/users/LeMoussel","html_url":"https://github.com/LeMoussel","followers_url":"https://api.github.com/users/LeMoussel/followers","following_url":"https://api.github.com/users/LeMoussel/following{/other_user}","gists_url":"https://api.github.com/users/LeMoussel/gists{/gist_id}","starred_url":"https://api.github.com/users/LeMoussel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LeMoussel/subscriptions","organizations_url":"https://api.github.com/users/LeMoussel/orgs","repos_url":"https://api.github.com/users/LeMoussel/repos","events_url":"https://api.github.com/users/LeMoussel/events{/privacy}","received_events_url":"https://api.github.com/users/LeMoussel/received_events","type":"User","site_admin":false},"created_at":"2015-11-02T18:40:54Z","updated_at":"2015-11-02T18:40:54Z","body":"I set \r\n`IsIgnoreRobotsDotTextIfRootDisallowedEnabled=\"true\"`\r\n`isRespectRobotsDotTextEnabled=\"true\"`\r\n\r\nBut I got for all Urls, debug message like this : *Page [http://www.provectio.fr/provectio-informatique/] not crawled, [Disallowed by robots.txt file], set IsRespectRobotsDotText=false in config file if you would like to ignore robots.txt files.*\r\n\r\nNo issue to crawl site with robots.txt consigne ?"}}