Kafka startup fails with zookeeper timeout (remote server), yet the machine can connect to zookeeper directly
<p>WHen I start kafka up, it fails quickly while complaining that it cannot connect to zookeeper.  I am running zookeeper as a standalone cluster/ensemble.  I am confused because there is no Firewall between the servers (as evidenced by the zookeeper-shell.sh test).</p>

<p>from /var/log/kafka/server.log</p>

<pre><code>2016-02-24 16:07:12,101 INFO kafka.server.KafkaServer: [Kafka Server 1], Connecting to zookeeper on 10.7.20.100:2181,10.7.20.101:2181,10.7.20.102:2181
2016-02-24 16:07:20,291 FATAL kafka.server.KafkaServerStartable: Fatal error during KafkaServerStable startup. Prepare to shutdown
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
    at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
    at org.I0Itec.zkclient.ZkClient.&lt;init&gt;(ZkClient.java:98)
    at org.I0Itec.zkclient.ZkClient.&lt;init&gt;(ZkClient.java:84)
    at kafka.server.KafkaServer.initZk(KafkaServer.scala:113)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:69)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
    at kafka.Kafka$.main(Kafka.scala:46)
    at kafka.Kafka.main(Kafka.scala)
2016-02-24 16:07:20,294 INFO kafka.server.KafkaServer: [Kafka Server 1], shutting down
2016-02-24 16:07:20,312 INFO kafka.server.KafkaServer: [Kafka Server 1], shut down completed
2016-02-24 16:07:20,317 INFO kafka.server.KafkaServer: [Kafka Server 1], shutting down
</code></pre>

<p>However from the /opt/kafka install directory I am able to connect to zookeeper using the esemble connection string - so I really doubt it is network OR firewall.</p>

<pre><code>[me@dckafka01 kafka]$ cd /opt/kafka
[me@dckafka01 kafka]$ bin/zookeeper-shell.sh 10.7.20.100:2181,10.7.20.101:2181,10.7.20.102:2181

Connecting to 10.7.20.100:2181,10.7.20.101:2181,10.7.20.102:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::WatchedEvent state:SyncConnected type:None path:null

get /blah
null
cZxid = 0x400000009
ctime = Tue Feb 16 09:00:28 EST 2016
mZxid = 0x400000009
mtime = Tue Feb 16 09:00:28 EST 2016
pZxid = 0x40000017e
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 2

ls /blah
[applications, registry]
</code></pre>

<p>Which is as expected.   Does anybody have an angle for me to investigate?</p>