

  • Most push architectures use the publish-subscribe messaging pattern, which is unreliable.
  • TCP’s built-in reliability is not enough, as modern network sessions span multiple connections.
  • There is no one size fits all answer.


Below we’ll go over the various issues and common practices around reliable push, plus a unique approach we developed at Fanout.

Publish-subscribe is unreliable by design



  1. Implement publisher back pressure based on the slowest subscriber.


This isn’t to say publish-subscribe is unsuitable for push; in fact it’s an essential tool in your toolbox. Just know that it’s only one piece of the puzzle. The ZeroMQ guide has a section on publish-subscribe that’s worth reading even if you’re not using ZeroMQ.

TCP is about flow control, not reliability


In practice, this means that TCP alone isn’t enough for reliable data transfer, and you’ll need to layer a reliable protocol on top of it.

If you learned long ago that TCP is for reliable communication (compared to, say, UDP), you may be confused by this and wonder why people still use TCP then. Well, TCP provides other useful features to build off of, such as back pressure, ordered delivery, and payloads of arbitrary size, so it is still an incredibly useful protocol even if we can’t depend on it for reliable transmission.

Error handling on the receiver



Reliable transmission fundamentals


  2. An entity responsible for initiating a retransmission.




Common practices

Alright so how do we ensure data is reliably delivered?

Use a regular database as a source of truth. If pushed data is important, the first thing you should do is write it to durable storage (on disk, with no auto expiration), before kicking off any push mechanisms. This way if there’s a problem during delivery, the data can later be recovered.

The receiver should be the responsible entity. In most web architectures, this is what you want. Receivers usually come and go, and it makes more sense for receivers to keep track of what they need rather than for servers to keep track of what has been sent.

Have a way to sync with the server. Receivers should be able to ask the server for new data, independent of any realtime push mechanisms. This way, publish workers can crash, queues can max out, and connections can get dropped, yet receivers can always catch up by making requests for updates. We even recommend building this part before introducing push components to your stack, to ensure your data sync is bulletproof before making it instant.




Pushpin reliable streaming

Since it’s burdensome for receivers to have to manage both push and pull data sources in order to receive data reliably, we devised a way to merge the two using Pushpin.

Pushpin is our open source proxy server that makes building realtime APIs easy. As a proxy server, it’s uniquely positioned to recover data from the backend server on a client’s behalf, and so we implemented a feature to do just that. Basically the way it works is the backend server provides a special URL that Pushpin can use to retrieve missing data.

This feature is described in the documentation and also in this video:

The end result is the client always sees a perfect stream. The overall architecture is still mostly publish-subscribe, we just moved the recovery logic up one hop from the client to the edge.

There are some interesting benefits of this from the client’s perspective:

  • The client doesn’t need to worry about sequencing.
  • The client doesn’t need to periodically sync with the server.
  • As long as a connection exists, the client can assume there are no gaps in the data.
  • A single request can be used to receive historical data and reliable pushed data going forward, so there is a single source of truth.

Most realtime APIs today don’t work this way, but we think our approach could greatly improve the developer experience around such APIs.