How to restore state in an event based, message driven microservice architecture on failure scenario
<p>In the context of a microservice architecture, a message driven, asynchronous, event based design seems to be gaining popularity (see <a href="http://blog.christianposta.com/microservices/why-microservices-should-be-event-driven-autonomy-vs-authority/" rel="noreferrer">here</a> and <a href="https://www.nginx.com/blog/event-driven-data-management-microservices/" rel="noreferrer">here</a> for some examples, as well as the <a href="http://www.reactivemanifesto.org/#message-driven" rel="noreferrer">Reactive Manifesto - Message Driven trait</a>) as opposed to a synchronous (possibly REST based) mechanism.</p>

<p>Taking that context and imagining an overly simplified ordering system, as depicted below:</p>

<p><a href="https://i.stack.imgur.com/Xtttg.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Xtttg.png" alt="ordering system"></a></p>

<p>and the following message flow:</p>

<ul>
<li>Order is placed from some source (web/mobile etc.)</li>
<li>Order service accepts order and publishes a <code>CreateOrderEvent</code></li>
<li>The InventoryService reacts on the <code>CreateOrderEvent</code>, does some inventory stuff and publishes a <code>InventoryUpdatedEvent</code> when it's done</li>
<li>The Invoice service then reacts to the <code>InventoryUpdatedEvent</code>, sends an invoice and publishes a <code>EmailInvoiceEvent</code></li>
</ul>

<p>All services are up and we happily process orders... Everyone is happy.
Then, the Inventory service goes down for some reason </p>

<p>Assuming that the events on the event bus are flowing in a "non blocking" manor. I.e. the messages are being published to a central topic and do not pile up on a queue if no service is reading from it (what I'm trying to convey is an event bus where, if the event is published on the bus, it would flow "straight through" and not queue up - ignore what messaging platform/technology is used at this point). That would mean that if the Inventory service were down for 5 minutes, the <code>CreateOrderEvent</code>'s passing through the event bus during that time are now "gone" or not seen by the Inventory service because in our overly simplified system, no other system is interested in those events.</p>

<p>My question then is: How does the Inventory service (and the system as a whole) restore state in a way that no orders are missed/not processed?</p>