{"action":"created","issue":{"url":"https://api.github.com/repos/metabrainz/listenbrainz-server/issues/65","labels_url":"https://api.github.com/repos/metabrainz/listenbrainz-server/issues/65/labels{/name}","comments_url":"https://api.github.com/repos/metabrainz/listenbrainz-server/issues/65/comments","events_url":"https://api.github.com/repos/metabrainz/listenbrainz-server/issues/65/events","html_url":"https://github.com/metabrainz/listenbrainz-server/pull/65","id":112621389,"number":65,"title":"[WIP] [KTHA] rate limiting amidointhisrite?","user":{"login":"mayhem","id":49745,"avatar_url":"https://avatars.githubusercontent.com/u/49745?v=3","gravatar_id":"","url":"https://api.github.com/users/mayhem","html_url":"https://github.com/mayhem","followers_url":"https://api.github.com/users/mayhem/followers","following_url":"https://api.github.com/users/mayhem/following{/other_user}","gists_url":"https://api.github.com/users/mayhem/gists{/gist_id}","starred_url":"https://api.github.com/users/mayhem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mayhem/subscriptions","organizations_url":"https://api.github.com/users/mayhem/orgs","repos_url":"https://api.github.com/users/mayhem/repos","events_url":"https://api.github.com/users/mayhem/events{/privacy}","received_events_url":"https://api.github.com/users/mayhem/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-10-21T15:55:42Z","updated_at":"2015-11-27T11:42:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/metabrainz/listenbrainz-server/pulls/65","html_url":"https://github.com/metabrainz/listenbrainz-server/pull/65","diff_url":"https://github.com/metabrainz/listenbrainz-server/pull/65.diff","patch_url":"https://github.com/metabrainz/listenbrainz-server/pull/65.patch"},"body":"First cut of the rate limiting stuff seems to be working. The general idea:\r\n\r\n1. Rate limiting is done on a per IP or per token basis. If the caller provides the Authorization header with their token, the user gets a better rate limit. If not, the user is rate limited based on IP.\r\n2. Rate limiting is done on a windowing basis, were X requests are allowed in a given window of time. This information is returned in the X-RateLimit-* headers.\r\n2.b. There is no 2.b\r\n3. Rate limits are dynamic and stored in Redis. They can be set with the set_rate_limits.py program. \r\n4. If you want to try this PR, install redis and python redis by hand for now.\r\n5. I've added a dummy endpoint for easy testing. Use with:\r\n\r\n  curl -D - --header \"Authorization: token [your token here]\" http://10.1.2.3:8080/1/rate-limit-test\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/metabrainz/listenbrainz-server/issues/comments/160121460","html_url":"https://github.com/metabrainz/listenbrainz-server/pull/65#issuecomment-160121460","issue_url":"https://api.github.com/repos/metabrainz/listenbrainz-server/issues/65","id":160121460,"user":{"login":"mayhem","id":49745,"avatar_url":"https://avatars.githubusercontent.com/u/49745?v=3","gravatar_id":"","url":"https://api.github.com/users/mayhem","html_url":"https://github.com/mayhem","followers_url":"https://api.github.com/users/mayhem/followers","following_url":"https://api.github.com/users/mayhem/following{/other_user}","gists_url":"https://api.github.com/users/mayhem/gists{/gist_id}","starred_url":"https://api.github.com/users/mayhem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mayhem/subscriptions","organizations_url":"https://api.github.com/users/mayhem/orgs","repos_url":"https://api.github.com/users/mayhem/repos","events_url":"https://api.github.com/users/mayhem/events{/privacy}","received_events_url":"https://api.github.com/users/mayhem/received_events","type":"User","site_admin":false},"created_at":"2015-11-27T11:42:20Z","updated_at":"2015-11-27T11:42:20Z","body":"@mwiencek You were totally right on your comment about it still making requests too fast. Javascript flow control and this scraper's flow control still baffle me. Can you please take a look at this and see if you can make it work as intended? \r\n\r\nPlease note that the last commit incorporates a suggestion from Sam, but that code doesn't even seem to compile. All I get is a cryptic error message from some cryptic JS universe. \r\n\r\nHALP?\r\n"}}