{"action":"created","issue":{"url":"https://api.github.com/repos/adobe/brackets.io/issues/144","labels_url":"https://api.github.com/repos/adobe/brackets.io/issues/144/labels{/name}","comments_url":"https://api.github.com/repos/adobe/brackets.io/issues/144/comments","events_url":"https://api.github.com/repos/adobe/brackets.io/issues/144/events","html_url":"https://github.com/adobe/brackets.io/pull/144","id":60273415,"number":144,"title":"Update robots.txt","user":{"login":"siddhartha-sharma","id":11217120,"avatar_url":"https://avatars.githubusercontent.com/u/11217120?v=3","gravatar_id":"","url":"https://api.github.com/users/siddhartha-sharma","html_url":"https://github.com/siddhartha-sharma","followers_url":"https://api.github.com/users/siddhartha-sharma/followers","following_url":"https://api.github.com/users/siddhartha-sharma/following{/other_user}","gists_url":"https://api.github.com/users/siddhartha-sharma/gists{/gist_id}","starred_url":"https://api.github.com/users/siddhartha-sharma/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/siddhartha-sharma/subscriptions","organizations_url":"https://api.github.com/users/siddhartha-sharma/orgs","repos_url":"https://api.github.com/users/siddhartha-sharma/repos","events_url":"https://api.github.com/users/siddhartha-sharma/events{/privacy}","received_events_url":"https://api.github.com/users/siddhartha-sharma/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-03-08T20:04:13Z","updated_at":"2015-03-09T12:38:41Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/adobe/brackets.io/pulls/144","html_url":"https://github.com/adobe/brackets.io/pull/144","diff_url":"https://github.com/adobe/brackets.io/pull/144.diff","patch_url":"https://github.com/adobe/brackets.io/pull/144.patch"},"body":""},"comment":{"url":"https://api.github.com/repos/adobe/brackets.io/issues/comments/77845676","html_url":"https://github.com/adobe/brackets.io/pull/144#issuecomment-77845676","issue_url":"https://api.github.com/repos/adobe/brackets.io/issues/144","id":77845676,"user":{"login":"siddhartha-sharma","id":11217120,"avatar_url":"https://avatars.githubusercontent.com/u/11217120?v=3","gravatar_id":"","url":"https://api.github.com/users/siddhartha-sharma","html_url":"https://github.com/siddhartha-sharma","followers_url":"https://api.github.com/users/siddhartha-sharma/followers","following_url":"https://api.github.com/users/siddhartha-sharma/following{/other_user}","gists_url":"https://api.github.com/users/siddhartha-sharma/gists{/gist_id}","starred_url":"https://api.github.com/users/siddhartha-sharma/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/siddhartha-sharma/subscriptions","organizations_url":"https://api.github.com/users/siddhartha-sharma/orgs","repos_url":"https://api.github.com/users/siddhartha-sharma/repos","events_url":"https://api.github.com/users/siddhartha-sharma/events{/privacy}","received_events_url":"https://api.github.com/users/siddhartha-sharma/received_events","type":"User","site_admin":false},"created_at":"2015-03-09T12:38:41Z","updated_at":"2015-03-09T12:38:41Z","body":"I had done my high school senior year project on Bandwidth & Crawlers. I had taken two VPS with same configurations and hosted, updated same amount of content but the one with crawl delay enable had lower bandwidth needs hence saving my bills. Bandwidth in other way are data transferred. If a huge website with minor change had to synced frequently you obviously have more data to transfer. While in this case, the change is very minor and does not have much significance but as I concluded in my study, a delayed crawler if the website is not updated frequently (read news portals) is a best practice."}}