Reading from multiple broker kafka with flink
<p>I want to read multiple kafka from flink.</p>

<p>I have a cluser of 3 computers for kafka. With the following topic</p>

<pre><code>Topic:myTopic   PartitionCount:3    ReplicationFactor:1 Configs:
Topic: myTopic  Partition: 0    Leader: 2   Replicas: 2 Isr: 2
Topic: myTopic  Partition: 1    Leader: 0   Replicas: 0 Isr: 0
Topic: myTopic  Partition: 2    Leader: 1   Replicas: 1 Isr: 1
</code></pre>

<p>From Flink I execute the following code :</p>

<pre><code>Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092");
properties.setProperty("group.id", "flink");

DataStream&lt;T&gt; stream = env.addSource(new FlinkKafkaConsumer09&lt;&gt;("myTopic", new SimpleStringSchema(), properties)
stream.map(....)
env.execute()
</code></pre>

<p>I launch 3 times the same job.</p>

<p>If I execute this code with one broker it's work well but with 3 broke (on 3 different machine) only one partition is read.</p>

<p><a href="https://stackoverflow.com/questions/38989443/flink-how-to-read-from-multiple-kafka-cluster-using-same-streamexecutionenviron">(In this question)</a> the solution proposed was </p>

<blockquote>
  <p>to create separate instances of the FlinkKafkaConsumer for each cluster (that's what you are already doing), and then union the resulting streams</p>
</blockquote>

<p>It's not working in my case.</p>

<p>So my questions are :</p>

<ol>
<li>Do I missing something ?</li>
<li>If we had a new computer in the Kafka cluster do we need to change flink's code to add a consumer for the new borker ? Or can we handle this automatically at runtime ?</li>
</ol>