Cassandra failed to create connection pool for new host
<p>I am running a web app using the following components:</p>

<ul>
<li>Python <strong>3.5.2</strong></li>
<li>uWSGI <strong>2.0.11.2</strong></li>
<li>cassandra-driver <strong>3.6.0</strong></li>
<li>cassandra <strong>3.7</strong> </li>
</ul>

<p>With a Cassandra cluster (3 nodes):</p>

<ul>
<li>Node 1 - IP: <strong>172.17.0.4</strong></li>
<li>Node 2 - IP: <strong>172.17.0.5</strong></li>
<li>Node 3 - IP: <strong>172.17.0.6</strong></li>
</ul>

<p>Using the configuration <strong>NetworkTopologyStrategy</strong> and <strong>GossipingPropertyFileSnitch</strong>.</p>

<p>I have followed the <a href="http://cqlengine.readthedocs.io/en/latest/topics/third_party.html?highlight=uwsgi" rel="nofollow">uWSGI connection example from cqlengine</a>:</p>

<pre class="lang-python prettyprint-override"><code>from cqlengine import connection
from cassandra.io.libevreactor import LibevConnection
from cqlengine.connection import (
    cluster as cql_cluster, 
    session as cql_session
)


try:
    from uwsgidecorators import postfork
except ImportError:
    # We're not in a uWSGI context, no need to hook Cassandra session
    # initialization to the postfork event.
    pass
else:
    @postfork
    def cassandra_init():
        """ Initialize a new Cassandra session in the context.

        Ensures that a new session is returned for every new request.
        """
        if cql_cluster is not None:
            cql_cluster.shutdown()
        if cql_session is not None:
            cql_session.shutdown()

        connection.setup(
            ['172.0.4'],
            'my_keyspace',
            port=9042,
            connection_class=LibevConnection
        )
</code></pre>

<p>But I am getting the error "<strong>Failed to create connection pool for new host x.x.x.x</strong>" on all cassandra nodes (<strong>172.17.0.4</strong>, <strong>172.17.0.5</strong> and <strong>172.17.0.6</strong>):</p>

<pre><code>Respawned uWSGI worker 2 (new pid: 90)
mapping worker 2 to CPUs: 3 4 5
2016-09-14 21:00:47,434 WARNI [cassandra.cluster][Thread-2] Failed to create connection pool for new host 172.17.0.4:
Traceback (most recent call last):
    File "cassandra/cluster.py", line 2232, in cassandra.cluster.Session.add_or_renew_pool.run_add_or_renew_pool (cassandra/cluster.c:38826)
    File "cassandra/pool.py", line 328, in cassandra.pool.HostConnection.__init__ (cassandra/pool.c:6243)
    File "cassandra/cluster.py", line 1107, in cassandra.cluster.Cluster.connection_factory (cassandra/cluster.c:14943)
    File "cassandra/connection.py", line 330, in cassandra.connection.Connection.factory (cassandra/connection.c:5766)
cassandra.OperationTimedOut: errors=Timed out creating connection (5 seconds), last_host=None
2016-09-14 21:00:47,437 WARNI [cassandra.cluster][Thread-1] Failed to create connection pool for new host 172.17.0.6:
Traceback (most recent call last):
    File "cassandra/cluster.py", line 2232, in cassandra.cluster.Session.add_or_renew_pool.run_add_or_renew_pool (cassandra/cluster.c:38826)
    File "cassandra/pool.py", line 328, in cassandra.pool.HostConnection.__init__ (cassandra/pool.c:6243)
    File "cassandra/cluster.py", line 1107, in cassandra.cluster.Cluster.connection_factory (cassandra/cluster.c:14943)
    File "cassandra/connection.py", line 330, in cassandra.connection.Connection.factory (cassandra/connection.c:5766)
cassandra.OperationTimedOut: errors=Timed out creating connection (5 seconds), last_host=None
...The work of process 19 is done. Seeya!
worker 7 killed successfully (pid: 19)
</code></pre>

<p>According the logs it's able to connect to the nodes but for some reason it seams to disconnect and throw the previous errors:</p>

<pre><code>2016-09-15 23:23:03,786 DEBUG [cassandra.pool][Thread-2] Initializing connection for host 172.17.0.4
2016-09-15 23:23:03,787 DEBUG [cassandra.connection][Thread-2] Not sending options message for new connection(139905425534704) to 172.17.0.4 because compression is disabled and a cql version was not specified
2016-09-15 23:23:03,787 DEBUG [cassandra.connection][Thread-2] Sending StartupMessage on &lt;LibevConnection(139905425534704) 172.17.0.4:9042&gt;
2016-09-15 23:23:03,787 DEBUG [cassandra.connection][Thread-2] Sent StartupMessage on &lt;LibevConnection(139905425534704) 172.17.0.4:9042&gt;
2016-09-15 23:23:03,788 DEBUG [cassandra.connection][event_loop] Got ReadyMessage on new connection (139905425534704) from 172.17.0.4
2016-09-15 23:23:03,788 DEBUG [cassandra.pool][Thread-2] Finished initializing connection for host 172.17.0.4
2016-09-15 23:23:03,788 DEBUG [cassandra.cluster][Thread-2] Added pool for host 172.17.0.4 to session
2016-09-15 22:24:29,239 DEBUG [cassandra.io.libevreactor][Thread-2] Closing connection (139945376028152) to 172.17.0.4
2016-09-15 22:24:29,240 DEBUG [cassandra.io.libevreactor][Thread-2] Closed socket to 172.17.0.4
2016-09-15 22:24:29,240 DEBUG [cassandra.connection][Thread-2] Connection to 172.17.0.4 was closed during the startup handshake
2016-09-15 22:24:29,242 WARNI [cassandra.cluster][Thread-2] Failed to create connection pool for new host 172.17.0.4:
</code></pre>

<p><strong>EDITED (Added more debugging info about the issue)</strong></p>

<p>The app can ping any of the nodes on port 9042 so it's not a connectivity issue. If I run <code>nodetool status</code> the three nodes in the cluster seems to be fine:</p>

<pre><code>--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.5  111.72 KiB  256          67.3%             fdd4740d-1ce5-4deb-9a3e-5c18c80ee63e  rack1
UN  172.17.0.4  98.96 KiB  256          66.8%             4fe5a60c-2b6a-4d57-ab6a-e4176ce69b68  rack1
UN  172.17.0.6  94.67 KiB  256          66.0%             5e2675e3-c2a7-4af1-80f0-4cb9573ecf2b  rack1
</code></pre>

<p>I have tried with Cassandra <strong>3.7</strong> and <strong>2.2.7</strong> but getting same results. But if I try to run the app just with the <strong>Node1</strong> it works! </p>

<p>The logs in Cassandra nodes displays the following:</p>

<pre><code>INFO  22:14:33 Unexpected exception during request; channel = [id: 0xd6d3c9ae, L:/172.17.0.6:9042 ! R:/172.17.0.5:42944]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
INFO  22:16:26 Unexpected exception during request; channel = [id: 0x2cfa996f, L:/172.17.0.6:9042 - R:/172.17.0.5:42954]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
INFO  22:54:13 Unexpected exception during request; channel = [id: 0x4134ec0f, L:/172.17.0.6:9042 ! R:/172.17.0.5:42992]
</code></pre>

<p>Does somebody know what is going on here?
Any help will be appreciated.</p>