{"action":"created","issue":{"url":"https://api.github.com/repos/couchbase/sync_gateway/issues/624","labels_url":"https://api.github.com/repos/couchbase/sync_gateway/issues/624/labels{/name}","comments_url":"https://api.github.com/repos/couchbase/sync_gateway/issues/624/comments","events_url":"https://api.github.com/repos/couchbase/sync_gateway/issues/624/events","html_url":"https://github.com/couchbase/sync_gateway/pull/624","id":56464317,"number":624,"title":"Improvements to  feed latency handling","user":{"login":"adamcfraser","id":8866453,"avatar_url":"https://avatars.githubusercontent.com/u/8866453?v=3","gravatar_id":"","url":"https://api.github.com/users/adamcfraser","html_url":"https://github.com/adamcfraser","followers_url":"https://api.github.com/users/adamcfraser/followers","following_url":"https://api.github.com/users/adamcfraser/following{/other_user}","gists_url":"https://api.github.com/users/adamcfraser/gists{/gist_id}","starred_url":"https://api.github.com/users/adamcfraser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamcfraser/subscriptions","organizations_url":"https://api.github.com/users/adamcfraser/orgs","repos_url":"https://api.github.com/users/adamcfraser/repos","events_url":"https://api.github.com/users/adamcfraser/events{/privacy}","received_events_url":"https://api.github.com/users/adamcfraser/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/couchbase/sync_gateway/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-04T00:56:19Z","updated_at":"2015-02-04T00:57:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/couchbase/sync_gateway/pulls/624","html_url":"https://github.com/couchbase/sync_gateway/pull/624","diff_url":"https://github.com/couchbase/sync_gateway/pull/624.diff","patch_url":"https://github.com/couchbase/sync_gateway/pull/624.patch"},"body":"The Couchbase Server TAP and DCP feeds don't guarantee sequential delivery of updates.  Sync Gateway previously buffered these updates to rebuild the changes in sequence order (needed for replication). When an entry on the feed was very slow, it led to several problems:\r\n* After a timeout period, that sequence was skipped.  Even if it later appeared in the feed, clients wouldn't know to retrieve it\r\n* If the timeout is extended (as it recently was to 60s), the changes feed will go silent until that timeout is reached\r\n\r\nThis commit implements an alternate approach:\r\n* After a shorter timeout, late sequences are skipped, but tracked by the change cache in `skippedSeqs`\r\n* When there are entries in the skippedSeqs, sequence numbers are sent to the client in the form lowSequence::sequence, where lowSequence is the lowest contiguous sequence for the gateway (skippedSeqs[0] - 1).   Subsequent client requests that include a lowSequence in the since value will retrieve from the lowSequence value.\r\n* A periodic cleanup function abandons values that have been in the skippedSeqs queue (default time for removal from skipped Seqs is 60 minutes).\r\n\r\nFor continuous changes feeds, late-arriving (previously skipped) sequences are sent in each iteration of the continuous changes loop.  Each channel cache maintains a history of late-arriving sequences for use by continuous changes feeds to support this functionality.\r\n\r\nIncludes support for customization of sequence handling settings (pending timeout, skipped timeout, max pending) via the database config."},"comment":{"url":"https://api.github.com/repos/couchbase/sync_gateway/issues/comments/72769914","html_url":"https://github.com/couchbase/sync_gateway/pull/624#issuecomment-72769914","issue_url":"https://api.github.com/repos/couchbase/sync_gateway/issues/624","id":72769914,"user":{"login":"adamcfraser","id":8866453,"avatar_url":"https://avatars.githubusercontent.com/u/8866453?v=3","gravatar_id":"","url":"https://api.github.com/users/adamcfraser","html_url":"https://github.com/adamcfraser","followers_url":"https://api.github.com/users/adamcfraser/followers","following_url":"https://api.github.com/users/adamcfraser/following{/other_user}","gists_url":"https://api.github.com/users/adamcfraser/gists{/gist_id}","starred_url":"https://api.github.com/users/adamcfraser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamcfraser/subscriptions","organizations_url":"https://api.github.com/users/adamcfraser/orgs","repos_url":"https://api.github.com/users/adamcfraser/repos","events_url":"https://api.github.com/users/adamcfraser/events{/privacy}","received_events_url":"https://api.github.com/users/adamcfraser/received_events","type":"User","site_admin":false},"created_at":"2015-02-04T00:57:30Z","updated_at":"2015-02-04T00:57:30Z","body":"Design notes on the implementation here: https://github.com/adamcfraser/cbnotes/wiki/Handling-Sequence-Latency\r\n"}}