{"action":"closed","issue":{"url":"https://api.github.com/repos/yasp-dota/yasp/issues/300","labels_url":"https://api.github.com/repos/yasp-dota/yasp/issues/300/labels{/name}","comments_url":"https://api.github.com/repos/yasp-dota/yasp/issues/300/comments","events_url":"https://api.github.com/repos/yasp-dota/yasp/issues/300/events","html_url":"https://github.com/yasp-dota/yasp/issues/300","id":60258657,"number":300,"title":"parallelize sequential api","user":{"login":"howardc93","id":3134520,"avatar_url":"https://avatars.githubusercontent.com/u/3134520?v=3","gravatar_id":"","url":"https://api.github.com/users/howardc93","html_url":"https://github.com/howardc93","followers_url":"https://api.github.com/users/howardc93/followers","following_url":"https://api.github.com/users/howardc93/following{/other_user}","gists_url":"https://api.github.com/users/howardc93/gists{/gist_id}","starred_url":"https://api.github.com/users/howardc93/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/howardc93/subscriptions","organizations_url":"https://api.github.com/users/howardc93/orgs","repos_url":"https://api.github.com/users/howardc93/repos","events_url":"https://api.github.com/users/howardc93/events{/privacy}","received_events_url":"https://api.github.com/users/howardc93/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-08T14:24:02Z","updated_at":"2015-04-20T20:27:07Z","closed_at":"2015-04-20T20:27:07Z","body":"This might help us avoid falling behind so much when the Steam API slows down.  I don't know if it's necessary, but I'm outlining an implementation here.\r\n\r\n* Code that is polling the sequential API should be refactored into a separate worker file (scanner.js) which can be distributed.\r\n* When a match to add is detected, we queue a new job type called \"insert\", which YASP core's worker.js is processing.\r\n* When we make a API request, we save in redis the sequence number we started at.  We can assume that the next 100 numbers will be processed.  What if we're at the leading edge and there are <100 matches though?  We could keep retrying the same seq_num until we get 100 matches.\r\n* On each sequential api request, we take the current progress in redis and add 100 (the minimum number of sequence numbers so we are guaranteed not to miss any matches) to get the number we want to start at, then make the API request.  Once a request is made, we need to update redis to tell subsequent requests to start at a higher number.  We may need two vars, one to track what matches we've actually added, and one to track the number we should start at.\r\n* ~~The API scan code currently uses local variables trackedPlayers and ratingPlayers to determine matches we need to request MMR/get replay on.  We would need to access redis for these variables to allow shared state between multiple hosts.~~ (fixed)\r\n* It is possible for us to get matches in duplicate (and quite likely).  In this case, since we only have a single thread inserting, we should detect that we already have a replay url and not re-retrieve it.  We will still insert/update the db data, however.\r\n\r\nThe result is that we can parallelize scanner across multlple hosts, to hopefully achieve greater bandwidth.  Each set of 100 sequence matches is approximately 1.2MB, and a major bottleneck is steam not serving us that data quickly enough.\r\n\r\nWe can proxy the requests through retrievers to distribute across IPs?"}}