Kafka offset persistence in Zookeeper
<p>I'm new to Storm/Kafka.
I've been able to configure a basic working prototype with :</p>

<ul>
<li>zookeeper 3.4.5</li>
<li>kafka 2.11-0.9.0.1</li>
<li>storm 1.0.0 (actually as an embedded java LocalCluster)</li>
<li>storm-kafka-1.0.0</li>
</ul>

<p>I'm able to produce message and to consume them from a storm topology.</p>

<p>I've a question regarding the kafka offsets persisting.</p>

<p>Initially I could'nt find the group used in the java client when using kafka utilities.</p>

<p>After some search I've read that storm-kafka stores this offset in zookeeper.
If I have for configuration in storm-kafka : </p>

<ul>
<li>zkroot = "/my_root"</li>
<li>group.id = "my_group"</li>
</ul>

<p>Then I can retrieve the offset with the zookeeper zkCli.sh script.</p>

<pre><code>get /my_root/my_group/partition_0
==&gt; "topology":{},"offset":3148,..., "topic":"rawdatas"
</code></pre>

<p>My problem is I don't understand how and how often this offset is updated.
In the storm bolt I'm sure to ack every tuple processed.</p>

<p>When the topology is started, and the tuples starting to be processed, I can see a little jump of the offset in Zookeeper (eg : some dozen)
Then the offset don't move for long times.</p>

<p>Some time, I can see a larger jump (eg some thousand) but it seems random.
Regarding the default kafka configuration, I understand that the offset should be updated every 2 seconds </p>

<pre><code>// setting for how often to save the current kafka offset to ZooKeeper
public long stateUpdateIntervalMs = 2000;
</code></pre>

<p>Do I miss something ?</p>

<p>Franck</p>