{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3471","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3471/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3471/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3471/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/3471","id":52770469,"number":3471,"title":"ReQL proposal: restarting feeds","user":{"login":"coffeemug","id":48436,"avatar_url":"https://avatars.githubusercontent.com/u/48436?v=3","gravatar_id":"","url":"https://api.github.com/users/coffeemug","html_url":"https://github.com/coffeemug","followers_url":"https://api.github.com/users/coffeemug/followers","following_url":"https://api.github.com/users/coffeemug/following{/other_user}","gists_url":"https://api.github.com/users/coffeemug/gists{/gist_id}","starred_url":"https://api.github.com/users/coffeemug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coffeemug/subscriptions","organizations_url":"https://api.github.com/users/coffeemug/orgs","repos_url":"https://api.github.com/users/coffeemug/repos","events_url":"https://api.github.com/users/coffeemug/events{/privacy}","received_events_url":"https://api.github.com/users/coffeemug/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/tp:ReQL_proposal","name":"tp:ReQL_proposal","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/92","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.1","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/92/labels","id":964515,"number":92,"title":"2.1","description":null,"creator":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"open_issues":40,"closed_issues":12,"state":"open","created_at":"2015-02-03T22:18:37Z","updated_at":"2015-04-21T19:11:53Z","due_on":null,"closed_at":null},"comments":39,"created_at":"2014-12-23T20:02:12Z","updated_at":"2015-04-21T22:13:03Z","closed_at":null,"body":"The question of restarting feeds came up in #2953, #1118, and #2613. Since we've learned a lot since these issues were opened, I decided to start with a clean slate and a specific proposal.\r\n\r\nWhen the user calls `changes()` on a stream, the feed protocol would inject an opaque timestamp into results as follows:\r\n\r\n```py\r\n# Returns a feed along with an opaque timestamp\r\n> r.table('foo').changes()\r\n[{ 'old_val': ..., 'new_val': ..., 'timestamp': OPAQUE}]\r\n```\r\n\r\nThe user could then pass the opaque timestamp back to `changes()` via the `return_initial` optarg to get all the changes since the timestamp. We'd be piggy-backing off existing replication logic, so we'd have to tell the user the range of \"obsolete\" keys they should delete, and then fill them in on the new values in that range:\r\n\r\n```py\r\n> r.table('foo').changes(return_initial=OPAQUE)\r\n[\r\n  # Obsolete range of keys to delete\r\n  { 'obsolete_range': [LEFT_KEY1, RIGHT_KEY1] },\r\n  # Backfill the user on values they missed in the range\r\n  { 'new_val': ..., 'timestamp': OPAQUE}, { 'new_val': ..., 'timestamp': OPAQUE},\r\n\r\n  # Another obsolete range\r\n  { 'obsolete_range': [LEFT_KEY2, RIGHT_KEY2] },\r\n  # Values to backfill in the range\r\n  { 'new_val': ..., 'timestamp': OPAQUE}, { 'new_val': ..., 'timestamp': OPAQUE},\r\n\r\n  # Now we can start the feed\r\n  { 'old_val': ..., 'new_val': ..., 'timestamp': OPAQUE},\r\n  ...\r\n]\r\n```\r\n\r\nA special value for `return_initial` is `True`, which backfills the user from scratch:\r\n\r\n```py\r\n> r.table('foo').changes(return_initial=True)\r\n[\r\n  # Backfill the user on all the values\r\n  { 'new_val': ..., 'timestamp': OPAQUE}, { 'new_val': ..., 'timestamp': OPAQUE},\r\n\r\n  # Now we can start the feed\r\n  { 'old_val': ..., 'new_val': ..., 'timestamp': OPAQUE},\r\n  ...\r\n]\r\n```\r\n\r\nThis would work on streams in general, like `t.map().changes()` and `t.filter().changes()`.\r\n\r\nNote, if the changefeed is set up on a datum (e.g. a single document), the only legal values for `return_initial` are `True` or `False`. \r\n\r\nThere is a question of what the default for `return_initial` should be. I can see a couple of options:\r\n\r\n- Set it to `False`. This is annoying for datum feeds (e.g. a feed on a document).\r\n- Set it to `True`. This may be annoying for stream feeds (e.g. a feed on a table).\r\n- Set it to `False` on streams, and to `True` on datums. This might be confusing to users.\r\n\r\nAlso note, under this proposal the `squash` optarg would continue operating as it does now, but would have no effect on the initial values.\r\n\r\n/cc @danielmewes @timmaxw @mlucy. How hard would this be to implement on top of the current replication logic? Are there flaws in the API? Would it work on more general streams (like `t.filter().changes()`)?"},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/94958263","html_url":"https://github.com/rethinkdb/rethinkdb/issues/3471#issuecomment-94958263","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3471","id":94958263,"user":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"created_at":"2015-04-21T22:13:03Z","updated_at":"2015-04-21T22:13:03Z","body":"It's pretty clear at this point that there are two different scenarios that we want to deal with.\r\n* application servers going down, or losing the connection to the DB\r\n* a database server going down (or table reconfiguration)\r\n\r\nI'm pretty sure we should address these use cases separately and with separate interfaces. We will address the first one in RethinkDB 2.1, and deal with the second one (handling DB server failures) later.\r\n\r\nFor case 1, we currently have the API proposal by mlucy https://github.com/rethinkdb/rethinkdb/issues/3471#issuecomment-70036764 : \r\n> ```ruby\r\n1> r.table('test').changes() => non-restartable changefeed\r\n1> r.table('test').changes(persist: 'my_changefeed') => restartable changefeed\r\n1> CRASHES\r\n2> r.table('test').changefeed('my_changefeed') => steal the restartable changefeed from (1)\r\n2> r.table('test').changefeed('my_changefeed').delete => safely close the changefeed so it doesn't hang around taking up memory\r\n```\r\n\r\nI think this looks pretty good. We would basically introduce the concept of a \"named changefeed\".\r\n\r\nThe implementation would use either an in-memory list, or possibly a `disk_backed_queue_t` (that's an in-memory list that seamlessly spills over into a temporary disk file if it grows too large) of a limited maximal size (100k changes by default?).\r\n\r\nFor the first implementation, a named changefeed would *not* survive a restart of any of the table's primaries, nor would it survive a table reconfiguration.\r\n\r\nIf a client connects to a changefeed that already has a reader, I propose that for the first release we always error on the existing reader as soon as it requests the next batch.\r\nWe can later look into alternative schemes such as \"deliver to at most one\" or \"deliver to all\". Those should be pretty easy to add on top of it, and will allow for some interesting use cases (@wojons will have some great ideas when we get to it I think :) ).\r\n\r\nThe named changefeeds would have a few very interesting usage scenarios, even in the basic version.\r\nFor example: Imagine you have a fleet of application servers and you have a load balancer in front of them. You could store the changefeed name in a session cookie in the user's browser. If one of the application servers goes down (or you take some down intentionally during less busy hours), or the user's browser loses connection temporarily (e.g. because they're on a mobile data connection), the browser could reconnect to *any* of the remaining application servers and just pick up the feed were it left off.\r\n\r\nWe should deal with the separate problem of recovering from database server failures and reconfiguration separately later. This is most relevant for synchronizing data into other data stores I think, and the requirements are somewhat different. Until we get to that, synchronization utilities can use `returnInitial` to re-sync from the beginning if the database server should die."}}