RethinkDB changefeeds performance: architectural advice?
<p>I am building an application with RethinkDB and I'm about to switch to using changefeeds. But I'm facing an architectural choice and I'd like to get some advice.</p>

<p>My application currently loads all user data from several tables on user login (sending all of it to the frontend), and then processes requests from the frontend, altering the database, and preparing and sending changed items to users. I'd like to switch that over to changefeeds. The way I see it, I have two choices:</p>

<ol>
<li>Set up a single changefeed for each table. Filter by users logged in to a particular server, and distribute the changes to users manually. These changefeeds are never closed, e.g. they have the lifetime of my servers.</li>
<li>When a user logs in, set up an individual changefeed for that user, for that user's data only (using a <code>getAll</code> with a secondary index). Maintain as many changefeeds as there are currently logged in users. Close them when users log out.</li>
</ol>

<p>Solution #1 has a big disadvantage: RethinkDB changefeeds do not have a concept of time (or version number), like for example Kafka does. This means that there is no way to a) load initial data, and b) get changes that happened since the initial load. There is a time window where changes can be lost: between initial data load (a) and the moment the changefeed is set up (b). I find this worrying.</p>

<p>Solution #2 seems better, because <code>includeInitial</code> can be used to get initial data, and then get subsequent changes without interruption. I'd have to deal with initial load performance (it's faster to load a single dump of all data than process thousands of updates), but it seems more "correct". But what about scaling? I'm planning to handle up to 1k users per server — is RethinkDB prepared to handle thousands of changefeeds, each being essentially a <code>getAll</code> query? The actual activity in these changefeeds will be very low, it's just the number that I'm worried about.</p>

<p>The RethinkDB manual is a bit terse about changefeed scaling, saying that:</p>

<blockquote>
  <p>Changefeeds perform well as they scale, although they create extra intracluster messages in proportion to the number of servers with open feed connections on each write.</p>
</blockquote>

<p>Solution #2 creates many more feeds, but the number of servers with open feed connections is actually the same for both solutions. And "changefeeds perform well as they scale" isn't quite enough to go on :-)</p>

<p>I'd also be interested to know what are recommended practices for handling server restarts/upgrades and disconnections. The way I see it, if anything happens to RethinkDB, clients have to perform a full data load (using <code>includeInitial</code>) after reconnecting, because there is no way to know what changes have been lost during downtime. Is that what people do?</p>