Difference between Apache Samza and Apache Kafka Streams (focus on parallelism and communication)
<p>In Samza and Kafka Streams, data stream processing is performed in a sequence/graph (called "dataflow graph" in Samza and "topology" in Kafka Streams) of processing steps (called "job" in Samza" and "processor" in Kafka Streams).
I will refer to these two terms as <em>workflow</em> and <em>worker</em> in the remainder of this question.</p>

<p>Let's assume that we have a very simple workflow, consisting of a worker A that consumes sensor measurements and filters all values below 50 followed by a worker B that receives the remaining measurments and filters all values above 80.</p>

<p>Input (Kakfa topic X)  -->  (Worker A)  -->  (Worker B)  -->  Output (Kafka topic Y)</p>

<p>If I have understood </p>

<ul>
<li><a href="http://samza.apache.org/learn/documentation/0.11/introduction/concepts.html" rel="noreferrer">http://samza.apache.org/learn/documentation/0.11/introduction/concepts.html</a> and </li>
<li><a href="http://docs.confluent.io/3.1.1/streams/architecture.html#parallelism-model" rel="noreferrer">http://docs.confluent.io/3.1.1/streams/architecture.html#parallelism-model</a> </li>
</ul>

<p>correctly, both Samza and Kafka Streams use the topic partitioning concept for replicating the workflow/workers and thus parallelizing the processing for scalability purposes.</p>

<p>But:</p>

<ul>
<li><p>Samza replicates each worker (i.e., job) separately to multiple tasks (one for each partition in the input stream). That is, a task is a replica of a worker of the workflow.</p></li>
<li><p>Kafka Streams replicates the whole workflow (i.e., topology) at once to multiple tasks (one for each partition in the input stream). That is, a task is a replica of the whole workflow.</p></li>
</ul>

<p>This brings me to my questions:</p>

<ol>
<li><p>Assume that there is only one partition: Is it correct, that it is not possible to deploy worker (A) and (B) on two different machines in Kafka Streams while this is possible in Samza? (Or in other words: Is it impossible in Kafka Streams to split a single task (i.e., topology replica) to two machines no matter if there are multiple partitions or not.)</p></li>
<li><p>How do two subsequent processors in a Kafka Streams topology (in the same task) communicate? (I know that in Samza all communication between two subsequent workers (i.e., jobs) is done with Kafka topics but since one has to "mark" in Kafka Streams explicitly in the code which streams have to be published as Kafka topics this can't be the case here.)</p></li>
<li><p>Is it correct that Samza publishes also all intermediate streams automatically as Kafka topics (and thus makes them available to potential clients) while Kafka Streams only publishes those intermediate and final streams that one marks explicitly (with <code>addSink</code> in the low-level API and <code>to</code> or <code>through</code> in DSL)?</p></li>
</ol>

<p>(I'm aware of the fact, that Samza can use also other message queues than Kafka but this is not really relevant for my questions.)</p>