{"action":"created","issue":{"url":"https://api.github.com/repos/ilri/DSpace/issues/35","labels_url":"https://api.github.com/repos/ilri/DSpace/issues/35/labels{/name}","comments_url":"https://api.github.com/repos/ilri/DSpace/issues/35/comments","events_url":"https://api.github.com/repos/ilri/DSpace/issues/35/events","html_url":"https://github.com/ilri/DSpace/issues/35","id":51094764,"number":35,"title":"Investigate robots.txt versus X-Robots-Tag HTTP header","user":{"login":"alanorth","id":191754,"avatar_url":"https://avatars.githubusercontent.com/u/191754?v=3","gravatar_id":"","url":"https://api.github.com/users/alanorth","html_url":"https://github.com/alanorth","followers_url":"https://api.github.com/users/alanorth/followers","following_url":"https://api.github.com/users/alanorth/following{/other_user}","gists_url":"https://api.github.com/users/alanorth/gists{/gist_id}","starred_url":"https://api.github.com/users/alanorth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alanorth/subscriptions","organizations_url":"https://api.github.com/users/alanorth/orgs","repos_url":"https://api.github.com/users/alanorth/repos","events_url":"https://api.github.com/users/alanorth/events{/privacy}","received_events_url":"https://api.github.com/users/alanorth/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ilri/DSpace/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"closed","locked":false,"assignee":{"login":"alanorth","id":191754,"avatar_url":"https://avatars.githubusercontent.com/u/191754?v=3","gravatar_id":"","url":"https://api.github.com/users/alanorth","html_url":"https://github.com/alanorth","followers_url":"https://api.github.com/users/alanorth/followers","following_url":"https://api.github.com/users/alanorth/following{/other_user}","gists_url":"https://api.github.com/users/alanorth/gists{/gist_id}","starred_url":"https://api.github.com/users/alanorth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alanorth/subscriptions","organizations_url":"https://api.github.com/users/alanorth/orgs","repos_url":"https://api.github.com/users/alanorth/repos","events_url":"https://api.github.com/users/alanorth/events{/privacy}","received_events_url":"https://api.github.com/users/alanorth/received_events","type":"User","site_admin":false},"milestone":null,"comments":6,"created_at":"2014-12-05T12:43:10Z","updated_at":"2015-01-29T08:10:19Z","closed_at":"2015-01-29T08:10:19Z","body":"Right now we have both robots.txt and `add_header X-Robots-Tag \"none\";` for DSpace Test... but I don't know which one is respected.  Ideally, on DSpace Test we would block access in robots.txt, but we deploy CGSpace from the same code base, so that's tricky to have different settings.\r\n\r\nMaybe we could use robots_test.txt, and redirect those in the nginx vhost for DSpace Test?"},"comment":{"url":"https://api.github.com/repos/ilri/DSpace/issues/comments/71985326","html_url":"https://github.com/ilri/DSpace/issues/35#issuecomment-71985326","issue_url":"https://api.github.com/repos/ilri/DSpace/issues/35","id":71985326,"user":{"login":"alanorth","id":191754,"avatar_url":"https://avatars.githubusercontent.com/u/191754?v=3","gravatar_id":"","url":"https://api.github.com/users/alanorth","html_url":"https://github.com/alanorth","followers_url":"https://api.github.com/users/alanorth/followers","following_url":"https://api.github.com/users/alanorth/following{/other_user}","gists_url":"https://api.github.com/users/alanorth/gists{/gist_id}","starred_url":"https://api.github.com/users/alanorth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alanorth/subscriptions","organizations_url":"https://api.github.com/users/alanorth/orgs","repos_url":"https://api.github.com/users/alanorth/repos","events_url":"https://api.github.com/users/alanorth/events{/privacy}","received_events_url":"https://api.github.com/users/alanorth/received_events","type":"User","site_admin":false},"created_at":"2015-01-29T08:10:19Z","updated_at":"2015-01-29T08:10:19Z","body":"I'm going to close this. Both test and production sites are in Google Webmaster Tools, but due to the `X-Robots-Tag: none` on the test server, it is not being indexed! Webmaster Tools says 27,000 sites submitted from the sitemap, but 0 indexed. :)"}}