{"action":"created","issue":{"url":"https://api.github.com/repos/postmodern/spidr/issues/39","labels_url":"https://api.github.com/repos/postmodern/spidr/issues/39/labels{/name}","comments_url":"https://api.github.com/repos/postmodern/spidr/issues/39/comments","events_url":"https://api.github.com/repos/postmodern/spidr/issues/39/events","html_url":"https://github.com/postmodern/spidr/pull/39","id":99076508,"number":39,"title":"Adds optionable support for obeying robots.txt","user":{"login":"buren","id":922411,"avatar_url":"https://avatars.githubusercontent.com/u/922411?v=3","gravatar_id":"","url":"https://api.github.com/users/buren","html_url":"https://github.com/buren","followers_url":"https://api.github.com/users/buren/followers","following_url":"https://api.github.com/users/buren/following{/other_user}","gists_url":"https://api.github.com/users/buren/gists{/gist_id}","starred_url":"https://api.github.com/users/buren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/buren/subscriptions","organizations_url":"https://api.github.com/users/buren/orgs","repos_url":"https://api.github.com/users/buren/repos","events_url":"https://api.github.com/users/buren/events{/privacy}","received_events_url":"https://api.github.com/users/buren/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-08-04T22:40:55Z","updated_at":"2015-11-19T22:53:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/postmodern/spidr/pulls/39","html_url":"https://github.com/postmodern/spidr/pull/39","diff_url":"https://github.com/postmodern/spidr/pull/39.diff","patch_url":"https://github.com/postmodern/spidr/pull/39.patch"},"body":" Usage:\r\n \r\n```ruby\r\n# To have this work add \"gem 'robots', '~> 0.1'\" to Gemfile\r\nSpidr.site(\r\n  'http://matasano.com/',\r\n  :robots => true\r\n)\r\n```\r\nCloses #11"},"comment":{"url":"https://api.github.com/repos/postmodern/spidr/issues/comments/158226057","html_url":"https://github.com/postmodern/spidr/pull/39#issuecomment-158226057","issue_url":"https://api.github.com/repos/postmodern/spidr/issues/39","id":158226057,"user":{"login":"postmodern","id":12671,"avatar_url":"https://avatars.githubusercontent.com/u/12671?v=3","gravatar_id":"","url":"https://api.github.com/users/postmodern","html_url":"https://github.com/postmodern","followers_url":"https://api.github.com/users/postmodern/followers","following_url":"https://api.github.com/users/postmodern/following{/other_user}","gists_url":"https://api.github.com/users/postmodern/gists{/gist_id}","starred_url":"https://api.github.com/users/postmodern/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/postmodern/subscriptions","organizations_url":"https://api.github.com/users/postmodern/orgs","repos_url":"https://api.github.com/users/postmodern/repos","events_url":"https://api.github.com/users/postmodern/events{/privacy}","received_events_url":"https://api.github.com/users/postmodern/received_events","type":"User","site_admin":false},"created_at":"2015-11-19T22:53:17Z","updated_at":"2015-11-19T22:53:17Z","body":"Yeah, but get rid of the NoRobots null object. Just do a `@robots && @robots.allow?(uri)`. Note that [Robots#allowed?](https://github.com/fizx/robots/blob/098390222ec048e7e00fefa081f83f1cf9e67511/lib/robots.rb#L125) accepts URIs. Also, should robots be all or nothing, or configurable per host?"}}