Architecture of complex Dataflow jobs
<p>We are building rather complex Dataflow jobs in that compute models from a streaming source. In particular, we have two models that share a bunch of metrics and that are computed off roughly the same data source.
The jobs perform joins on slightly large datasets.</p>

<p>Do you have any guidelines on how to design that kind of jobs? Any metrics, behaviors, or anything we have to consider in oder to make the decision?</p>

<p>Here are a couple options that we have in mind and how we thing they compare:</p>

<h2>Option 1: one large job</h2>

<p>Implement everything in one, large job. Factor common metrics, and then compute model specific metrics.</p>

<h3>Pros</h3>

<ul>
<li>Simpler to write.</li>
<li>No dependency between jobs.</li>
<li>Less compute resources?</li>
</ul>

<h3>Cons</h3>

<ul>
<li>If one part breaks, both models can't be computed.</li>
</ul>

<p><a href="https://i.stack.imgur.com/p1iQ2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/p1iQ2.png" alt="Large job"></a></p>

<h2>Option 2: Multiple jobs piped with Pub/Sub</h2>

<p>Extract out the common metrics computation to a dedicated job, thus resulting in 3 jobs, wired together using Pub/Sub.</p>

<h3>Pros</h3>

<ul>
<li>More resilient in case of failure of one of the model job.</li>
<li>Probably easier to perform <a href="https://cloud.google.com/dataflow/pipelines/updating-a-pipeline" rel="nofollow noreferrer">ongoing updates</a>.</li>
</ul>

<h3>Cons</h3>

<ul>
<li>All jobs need to be started in order to have the full pipeline: dependency management.</li>
</ul>

<p><a href="https://i.stack.imgur.com/YaAH2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YaAH2.png" alt="3 jobs"></a></p>