{"action":"opened","issue":{"url":"https://api.github.com/repos/justintv/Twitch-API/issues/314","labels_url":"https://api.github.com/repos/justintv/Twitch-API/issues/314/labels{/name}","comments_url":"https://api.github.com/repos/justintv/Twitch-API/issues/314/comments","events_url":"https://api.github.com/repos/justintv/Twitch-API/issues/314/events","html_url":"https://github.com/justintv/Twitch-API/issues/314","id":57914060,"number":314,"title":"Bug: GET /channels/:channel/follows intermittently excludes unexpected users","user":{"login":"schmich","id":1562839,"avatar_url":"https://avatars.githubusercontent.com/u/1562839?v=3","gravatar_id":"","url":"https://api.github.com/users/schmich","html_url":"https://github.com/schmich","followers_url":"https://api.github.com/users/schmich/followers","following_url":"https://api.github.com/users/schmich/following{/other_user}","gists_url":"https://api.github.com/users/schmich/gists{/gist_id}","starred_url":"https://api.github.com/users/schmich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schmich/subscriptions","organizations_url":"https://api.github.com/users/schmich/orgs","repos_url":"https://api.github.com/users/schmich/repos","events_url":"https://api.github.com/users/schmich/events{/privacy}","received_events_url":"https://api.github.com/users/schmich/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-02-17T11:40:35Z","updated_at":"2015-02-17T11:40:35Z","closed_at":null,"body":"The [`GET /channels/:channel/follows`](https://github.com/justintv/Twitch-API/blob/master/v3_resources/follows.md#get-channelschannelfollows) endpoint appears to return incorrect data a short time after a user follows or unfollows the channel. I believe the nature of the problem is due to eventual consistency coupled with paged responses and inconsistent server routing.\r\n\r\nUsing a friend's channel ([real_jansoon](http://www.twitch.tv/real_jansoon/profile), 230 followers at the time of testing) while he was not broadcasting, we had his alternate account (`jansoonnsfw`) follow and unfollow his channel. At the same time, I ran the script at the bottom to use the API to snapshot his followers list every 10 seconds and display the changes (users added/removed) from the previous snapshot. The output:\r\n\r\n```\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[\"timeshadow199\"], :added=>[]}\r\n{:removed=>[\"kyuriin\"], :added=>[\"timeshadow199\"]}\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[], :added=>[\"kyuriin\"]}\r\n{:removed=>[\"jansoonnsfw\"], :added=>[]}\r\n{:removed=>[\"timeshadow199\"], :added=>[\"jansoonnsfw\"]}\r\n{:removed=>[], :added=>[\"timeshadow199\"]}\r\n{:removed=>[\"jansoonnsfw\", \"timeshadow199\"], :added=>[]}\r\n{:removed=>[], :added=>[\"jansoonnsfw\", \"timeshadow199\"]}\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[\"kyuriin\"], :added=>[]}\r\n{:removed=>[\"jansoonnsfw\"], :added=>[\"kyuriin\"]}\r\n{:removed=>[], :added=>[\"jansoonnsfw\"]}\r\n{:removed=>[\"timeshadow199\"], :added=>[]}\r\n{:removed=>[], :added=>[\"timeshadow199\"]}\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[\"jansoonnsfw\", \"timeshadow199\"], :added=>[]}\r\n{:removed=>[], :added=>[\"jansoonnsfw\", \"timeshadow199\"]}\r\n{:removed=>[\"jansoonnsfw\"], :added=>[]}\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[\"timeshadow199\"], :added=>[\"jansoonnsfw\"]}\r\n{:removed=>[\"jansoonnsfw\"], :added=>[]}\r\n{:removed=>[], :added=>[\"jansoonnsfw\", \"timeshadow199\"]}\r\n{:removed=>[\"jansoonnsfw\"], :added=>[]}\r\n{:removed=>[\"timeshadow199\"], :added=>[]}\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[], :added=>[\"timeshadow199\"]}\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[], :added=>[]}\r\n{:removed=>[], :added=>[]}\r\n...\r\n```\r\n\r\nThe API eventually gets to a consistent state (`jansoonnsfw` is added, no other users affected), but  notice that two other users appear and disappear intermittently: `kyuriin` and `timeshadow199` (we verified that they weren't randomly following/unfollowing him during our tests).\r\n\r\nAs it turns out, `timeshadow199` is follower number 30 `kyuriin` is follower number 130. The script makes requests in batches of 100 from the API by time descending, and my friend has 230 followers total, so `kyuriin` is the last follower in the first request, and `timeshadow199` is the last follower in the second request.\r\n\r\nI believe this is because of the following: in order to get all of my friend's followers, we must make 3 requests to the Twitch API. Each request can end up hitting different servers with different caching/consistency, so, the worst-case scenario (assuming requests with `limit=100`) is:\r\n\r\n- Our first request routes to a server that knows about `jansoonnsfw`, so `kyuriin` (now follower number 101) gets excluded from the response (which includes followers 1-100).\r\n- Our second request routes to a server that does *not* know about `jansoonnsfw`, so `kyuriin` is follower number 100 and gets excluded from the response (which includes followers 101-200).\r\n- The net effect is that it appears that `kyuriin` is not following `jansoonnsfw`.\r\n\r\nSolutions:\r\n\r\n1. Workaround: We make multiple requests to Twitch for each batch of followers, and union all of the responses with the expectation that, given enough requests, we will hit servers with different world views, and will *probably* be able to construct the underlying truth.\r\n2. Twitch provides a server token/ID response header that can be used in subsequent requests to ensure they get routed to the same server (e.g. `X-Server-Id: api-server-42`). In this case, we would at least be requesting against an internally-consistent world view.\r\n3. Twitch updates their load balancer to route requests from one IP to the same server and can provide a guarantee that clients will hit the same server as long as their IP doesn't change.\r\n\r\nI imagine this is an issue with all other API endpoints that have paged responses.\r\n\r\nWe're currently trying to create an unfollow notifier, and that depends on having correct data, so it would be nice to get some sort of fix here beyond the hacky solution 1 above.\r\n\r\nThanks & regards,\r\nChris\r\n\r\n```ruby\r\nrequire 'open-uri'\r\nrequire 'json'\r\nrequire 'set'\r\nrequire 'pp'\r\n\r\ndef followers(channel)\r\n  url = \"https://api.twitch.tv/kraken/channels/#{channel}/follows?limit=100\"\r\n  names = Set.new\r\n\r\n  while url\r\n    json = JSON.parse(open(url, 'Accept' => 'application/vnd.twitchtv.v3+json').read)\r\n    batch = json['follows'].map { |f| f['user']['name'] }\r\n    break if batch.empty?\r\n    names += batch\r\n    url = json['_links']['next']\r\n  end\r\n\r\n  names.map(&:downcase)\r\nend\r\n\r\nprev = curr = []\r\n\r\nloop do\r\n  prev, curr = curr, followers('real_jansoon')\r\n  pp({ removed: prev - curr, added: curr - prev })\r\n  sleep 10\r\nend\r\n```"}}