{"action":"created","issue":{"url":"https://api.github.com/repos/seomoz/reppy/issues/24","labels_url":"https://api.github.com/repos/seomoz/reppy/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/seomoz/reppy/issues/24/comments","events_url":"https://api.github.com/repos/seomoz/reppy/issues/24/events","html_url":"https://github.com/seomoz/reppy/pull/24","id":72145721,"number":24,"title":"Allow all pages if access to robots.txt is forbidden","user":{"login":"austin-distilled","id":3205346,"avatar_url":"https://avatars.githubusercontent.com/u/3205346?v=3","gravatar_id":"","url":"https://api.github.com/users/austin-distilled","html_url":"https://github.com/austin-distilled","followers_url":"https://api.github.com/users/austin-distilled/followers","following_url":"https://api.github.com/users/austin-distilled/following{/other_user}","gists_url":"https://api.github.com/users/austin-distilled/gists{/gist_id}","starred_url":"https://api.github.com/users/austin-distilled/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/austin-distilled/subscriptions","organizations_url":"https://api.github.com/users/austin-distilled/orgs","repos_url":"https://api.github.com/users/austin-distilled/repos","events_url":"https://api.github.com/users/austin-distilled/events{/privacy}","received_events_url":"https://api.github.com/users/austin-distilled/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-04-30T12:39:26Z","updated_at":"2015-05-27T10:35:35Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/seomoz/reppy/pulls/24","html_url":"https://github.com/seomoz/reppy/pull/24","diff_url":"https://github.com/seomoz/reppy/pull/24.diff","patch_url":"https://github.com/seomoz/reppy/pull/24.patch"},"body":""},"comment":{"url":"https://api.github.com/repos/seomoz/reppy/issues/comments/105860383","html_url":"https://github.com/seomoz/reppy/pull/24#issuecomment-105860383","issue_url":"https://api.github.com/repos/seomoz/reppy/issues/24","id":105860383,"user":{"login":"austin-distilled","id":3205346,"avatar_url":"https://avatars.githubusercontent.com/u/3205346?v=3","gravatar_id":"","url":"https://api.github.com/users/austin-distilled","html_url":"https://github.com/austin-distilled","followers_url":"https://api.github.com/users/austin-distilled/followers","following_url":"https://api.github.com/users/austin-distilled/following{/other_user}","gists_url":"https://api.github.com/users/austin-distilled/gists{/gist_id}","starred_url":"https://api.github.com/users/austin-distilled/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/austin-distilled/subscriptions","organizations_url":"https://api.github.com/users/austin-distilled/orgs","repos_url":"https://api.github.com/users/austin-distilled/repos","events_url":"https://api.github.com/users/austin-distilled/events{/privacy}","received_events_url":"https://api.github.com/users/austin-distilled/received_events","type":"User","site_admin":false},"created_at":"2015-05-27T10:35:35Z","updated_at":"2015-05-27T10:35:35Z","body":"Hi @b4hand, I'm just revisiting this after other work took priority. I've tried to add the flag in the most lightweight way possible given there was no previous precedent for passing information from RobotsCache to Rules. Let me know if there is an alternative way that you feel is more aligned with the project or if there are any other amendments you'd like to see before merging in. Cheers!"}}