{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4629","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4629/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4629/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4629/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4629","id":99085418,"number":4629,"title":"Allow correlating writes with feed notifications","user":{"login":"coffeemug","id":48436,"avatar_url":"https://avatars.githubusercontent.com/u/48436?v=3","gravatar_id":"","url":"https://api.github.com/users/coffeemug","html_url":"https://github.com/coffeemug","followers_url":"https://api.github.com/users/coffeemug/followers","following_url":"https://api.github.com/users/coffeemug/following{/other_user}","gists_url":"https://api.github.com/users/coffeemug/gists{/gist_id}","starred_url":"https://api.github.com/users/coffeemug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coffeemug/subscriptions","organizations_url":"https://api.github.com/users/coffeemug/orgs","repos_url":"https://api.github.com/users/coffeemug/repos","events_url":"https://api.github.com/users/coffeemug/events{/privacy}","received_events_url":"https://api.github.com/users/coffeemug/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/tp:ReQL_proposal","name":"tp:ReQL_proposal","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/92","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.2","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/92/labels","id":964515,"number":92,"title":"2.2","description":"","creator":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"open_issues":27,"closed_issues":0,"state":"open","created_at":"2015-02-03T22:18:37Z","updated_at":"2015-08-05T00:50:50Z","due_on":null,"closed_at":null},"comments":25,"created_at":"2015-08-04T23:47:05Z","updated_at":"2015-08-06T00:03:24Z","closed_at":null,"body":"I talked to @Slava today about missing RethinkDB features that would make a Meteor integration much simpler. He said he only needs #3579, and ability to correlate writes with feed notifications. This issue is for the latter feature.\r\n\r\nImagine you have a chat application; when the user sends a message you want to show it on the client right away (possibly greyed out) to compensate for latency (which results in a much better user experience). When the notification later shows up in the feed, you then want to update the color to indicate that the message has been broadcast to everyone. In order to do that well, we need monotonically increasing write ids on per-connection basis.\r\n\r\nHere is an example:\r\n\r\n```js\r\n> r.table('foo').insert({name: 'Slava'}).run()\r\n{\r\n    inserted: 1,\r\n    write_id: [conn_id, timestamp],\r\n    // ...\r\n}\r\n```\r\n\r\nThen on the feed:\r\n\r\n```js\r\n> r.table('foo').changes()\r\n{\r\n    new_val: ...,\r\n    old_val: ...,\r\n    write_id: [conn_id, timestamp]\r\n}\r\n```\r\n\r\nThis way the client can correlate the write with the feed notification and do latency compensation.\r\n\r\nThere are a few outstanding questions:\r\n\r\n- How do we return write ids when the user does a batch write?\r\n- Does it always make sense to return it on a feed (e.g. what about complex feeds like `orderBy.limit`, etc.)? Will there ever be a case when we can't return a write id on the feed?\r\n\r\nAlso, ideally the timestamp would be user readable -- e.g. `time-since-epoch_integer` (the `integer` is there to dedup two writes with the same timestamp).\r\n\r\nPutting in 2.2.\r\n\r\n/cc @mlucy @danielmewes "},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/128188984","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4629#issuecomment-128188984","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4629","id":128188984,"user":{"login":"mlucy","id":1777134,"avatar_url":"https://avatars.githubusercontent.com/u/1777134?v=3","gravatar_id":"","url":"https://api.github.com/users/mlucy","html_url":"https://github.com/mlucy","followers_url":"https://api.github.com/users/mlucy/followers","following_url":"https://api.github.com/users/mlucy/following{/other_user}","gists_url":"https://api.github.com/users/mlucy/gists{/gist_id}","starred_url":"https://api.github.com/users/mlucy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlucy/subscriptions","organizations_url":"https://api.github.com/users/mlucy/orgs","repos_url":"https://api.github.com/users/mlucy/repos","events_url":"https://api.github.com/users/mlucy/events{/privacy}","received_events_url":"https://api.github.com/users/mlucy/received_events","type":"User","site_admin":false},"created_at":"2015-08-06T00:03:24Z","updated_at":"2015-08-06T00:03:24Z","body":"Here's an alternate proposal:\r\n\r\n* Timestamps are a set of <machine ID, wall time> pairs.\r\n* When a batch write is first issued, we shard it and send it to multiple machines.\r\n  - On each primary, for each write in the batch write we assigned the write a stamp of `{primary: current_time}`.  This is the stamp that is sent with that write to the changefeed.  We return the stamp `{primary: max(current_time)}` (so the latest stamp from all the writes in the batch).\r\n  - During the unsharding phase, we simply union all these sets together (so if we have `{m1: t1}`, `{m1: t2}`, and `{m2: t3}` unsharding produces `{m1: max(t1, t2), m2: t3}`).\r\n* Changefeeds maintain a set of `<machine ID, latest wall time>` pairs and send a stamp at the end of every batch as a `state` message.\r\n* Clients just need to watch for these state messages, and if they see a state message with a stamp that strictly dominates the stamp they received for a write (the stamp is >= the stamp they received for every machine) they know that write is reflected in the changefeed.  (In the case where they have an unsharded table, they just need to compare two numbers during this step.)\r\n  - Optionally we provide a way to automate this in the clients so that people don't have to write the comparison function themselves.\r\n\r\nI think that solves all the major problems, and has the advantage that it also works for coordinating writes with changefeeds opened by other clients (I think it's good to be general here if we can).\r\n\r\nOne long-term complication is failover.  Long-term we'd like changefeeds to survive failover, and I'm not 100% sure how this can be adapted to support that.  (Although it would probably be fine to have an additional sync message or something in that case that the clients have to handle.)"}}