why does cassandra sometimes take a hours to start?
<p>I am running a 26 node cluster in ec2, spread across 4 regions - us-east-1, us-west-2, eu-west-1, and ap-southeast-1. I've noticed that sometimes nodes take a very long time to start up and the problem seems particularly bad in ap-southeast-1. For example, simply restarting a node today took over 4 hours. When this happens, the last thing logged by cassandra (system.log) is:</p>

<pre><code>INFO  [main] 2016-10-19 20:35:14,846 ColumnFamilyStore.java:395 - Initializing system_schema.indexes
</code></pre>

<p>Eventually it moves on to initializing my own column families, but I don't understand what is taking so long. This particular node has about 900G of data on it, using a 3.2T EBS volume.</p>

<p>While this is happening there seems to be very little resource contention. Network i/o doesn't seem to go above 1Mb/s. Both the data volume and commit log have basically 0 i/o. CPU usage is in 25%-50% range. Looking at <code>strace</code>, there seems to be tons of <code>getdents</code> system calls, but that doesn't seem like it would be causing problems. </p>

<p>What could explain such a long startup time?</p>

<h1>Cluster Info</h1>

<ul>
<li>Cassandra DDC 3.5</li>
<li>EC2MultiRegionSnitch</li>
<li>m4.xlarge, moving to c4.4xlarge</li>
</ul>

<h1>Schema Info</h1>

<ul>
<li>3 CF's, all 'write once' (ie no updates), 1 week ttl, STCS (default)</li>
<li>no secondary indexes</li>
</ul>