Right tool for near-real-time ETL architecture
<p>We have a system where our primary data store (and "Universal Source of Truth") is Postgres, but we replicate that data both in real-time as well as nightly in aggregate.  We currently replicate to Elasticsearch, Redis, Redshift (nightly only), and are adding Neo4j as well.</p>

<p>Our ETL pipeline has gotten expansive enough that we're starting to look at tools like <a href="http://airbnb.io/projects/airflow/" rel="nofollow">Airflow</a> and <a href="https://github.com/spotify/luigi" rel="nofollow">Luigi</a>, but from what I can tell from my initial research, these tools are meant almost entirely for batch loads in aggregate.</p>

<p>Is there any tool that can handle an ETL process that can handle both large batch ETL processes as well as on-the-fly, high-volume, individual-record replication? Do Airflow or Luigi handle this and I just missed it?</p>

<p>Thanks!</p>