{"action":"created","issue":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/153","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/153/labels{/name}","comments_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/153/comments","events_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/153/events","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/153","id":97480370,"number":153,"title":"Add support for robots.txt, on by default","user":{"login":"mvl22","id":361423,"avatar_url":"https://avatars.githubusercontent.com/u/361423?v=3","gravatar_id":"","url":"https://api.github.com/users/mvl22","html_url":"https://github.com/mvl22","followers_url":"https://api.github.com/users/mvl22/followers","following_url":"https://api.github.com/users/mvl22/following{/other_user}","gists_url":"https://api.github.com/users/mvl22/gists{/gist_id}","starred_url":"https://api.github.com/users/mvl22/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mvl22/subscriptions","organizations_url":"https://api.github.com/users/mvl22/orgs","repos_url":"https://api.github.com/users/mvl22/repos","events_url":"https://api.github.com/users/mvl22/events{/privacy}","received_events_url":"https://api.github.com/users/mvl22/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-07-27T14:46:46Z","updated_at":"2015-07-30T19:56:08Z","closed_at":null,"body":"We are seeing sites receiving crawls with this bot's signature which disobey robots.txt, falling into our honeytrap.\r\n\r\nPlease kindly add support for robots.txt, (strongly) preferably on by default.\r\n\r\nI would argue that crawlers have a responsibility to ensure that they act as a well-behaved web citizen by default. This applies even more to open-source code crawlers, which are likely to be used by people the author has no control over.\r\n\r\nSo please also kindly ensure this behaviour is on by default. If someone is installing this crawler, and they are using it on their own website, they will have full knowledge of why robots.txt rules are in place, and so can decide to disable the feature. If they are using it on another site, then the robots.txt should be respected as a matter of principle.\r\n\r\n>  I figure that because it's a specialised tool, and not designed to crawl websites indiscriminately [1]\r\n\r\nAs open source code, it absolutely can be used in that way, I'm afraid.\r\n\r\n\r\n[1] https://github.com/cgiffard/node-simplecrawler/issues/59#issuecomment-25943001"},"comment":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/comments/126458988","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/153#issuecomment-126458988","issue_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/153","id":126458988,"user":{"login":"mortenf","id":400721,"avatar_url":"https://avatars.githubusercontent.com/u/400721?v=3","gravatar_id":"","url":"https://api.github.com/users/mortenf","html_url":"https://github.com/mortenf","followers_url":"https://api.github.com/users/mortenf/followers","following_url":"https://api.github.com/users/mortenf/following{/other_user}","gists_url":"https://api.github.com/users/mortenf/gists{/gist_id}","starred_url":"https://api.github.com/users/mortenf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mortenf/subscriptions","organizations_url":"https://api.github.com/users/mortenf/orgs","repos_url":"https://api.github.com/users/mortenf/repos","events_url":"https://api.github.com/users/mortenf/events{/privacy}","received_events_url":"https://api.github.com/users/mortenf/received_events","type":"User","site_admin":false},"created_at":"2015-07-30T19:56:08Z","updated_at":"2015-07-30T19:56:08Z","body":"+1"}}