Node randomly crashing without logs
<p>We have a 7 Node Cluster working on Cassandra 3.9. Currently we are facing problems with 3 Nodes randomly crashing without any notes in the logs. Neither the kernel.log nor the system.log/debug.log from Cassandra contains any hints of why the node crashed.</p>

<p>The only thing we see is stacked up compaction jobs which never get calculated:</p>

<p><code>pending tasks: 139
- chatlog.master_id: 1
- chatlog.reports: 1
- qsg.stats: 21
- hardcore.stats_archive: 1
- hardcore.stats: 1
- quickbedwars.stats_archive: 1
- quickbedwars.stats: 1
- permission.entity: 1
- pvp.invsave: 1
- pvp.u_match_count: 1
- conquest.leavebuster: 1
- molecraft.stats_archive: 1
- molecraft.achievement: 1
- clanwar.matches: 1
- clanwar.recent_matches_clans: 1
- network.version_logins: 1
- skywars.stats: 60
- system_traces.sessions: 1
- cores.stats: 1
- system.local: 1
- endergames.stats_archive: 1
- endergames.stats: 1
- endergames.kits: 1
- gungame.stats: 1
- system_schema.dropped_columns: 1
- system_schema.types: 1
- system_schema.tables: 1
- system_schema.indexes: 1
- system_schema.keyspaces: 1
- system_schema.columns: 1
- bedwars.stats: 26
- survivalgames.stats_archive: 1
- coinsystem.rarity_probabilities: 1
- nicksystem.texture_cache: 1
- speed_suhc.stats_archive: 1
</code></p>

<p>All stats tables are Leveld rest is SCTS</p>

<p>nodetool status looks like this when a node crashes:</p>

<p><code>UN  79.133.61.x  73.63 GiB  256          ?       e5d40b71-91aa-4fb4-a883-163dfd4ddc1e  01
UN  79.133.61.x  89.85 GiB  256          ?       6f74c956-8300-4f83-82d4-eb31d5678696  01
UN  79.133.61.x  113.33 GiB  512          ?       e73560f5-d6e7-4504-9a71-52f357b3ce0f  01
UJ  79.133.61.x  29.94 GiB  512          ?       b6ff57be-393a-4cbd-b92f-3b04ac5132da  01
UN  79.133.61.x  68.11 GiB  256          ?       b9e6207e-987f-4393-bdfe-5c440e886500  01
UN  79.133.61.x  76.11 GiB  256          ?       455d53f6-23dd-45ba-9a29-9fc7b0303fc2  01
DN  79.133.61.x  95.99 GiB  256          ?       f2d3ecb1-4610-414b-b5c6-3129d96e4bc0  01
</code></p>

<p>We drained the one node which is currently joining and did a full reset of that node to start fresh. </p>

<p>We tried setting the compaction_throughput higher but that did not help at all. We use 4 SSDs in JBOD with 12 Core (HT) CPUs on Baremetal Servers connected with an internal 10 Gbit/s interface. We never hit any kind of resource restrictions, the I/Os look ok, memory seems fine and the CPU is not overloaded.</p>

<p>// Edit:</p>

<p>A htop screen of one node with stacked compaction tasks shows that all of the concurrent compaction threads are working on something:</p>

<p><a href="https://i.stack.imgur.com/Wyqrl.png" rel="nofollow noreferrer">Htop output</a> </p>

<p>Dstat output looks like this:</p>

<p><a href="https://i.stack.imgur.com/sO7dx.png" rel="nofollow noreferrer">Dstat output</a></p>

<p>So it seems to write some chunks of data every now and then. But nothing i would consider a write load for a compaction.</p>

<p>I currently observed that the last log output before the node crashes is always this: </p>

<p><code>INFO  [IndexSummaryManager:1] 2016-11-22 14:59:38,532 IndexSummaryRedistribution.java:75 - Redistributing index summaries
</code></p>

<p>// More Edit:</p>

<p>Running SJK shows extreme memory allocations coming out of the Compaction Threads:</p>

<pre><code>heap allocation rate 5984mb/s
[000055] user=93.91% sys= 1.20% alloc= 1292mb/s - CompactionExecutor:2
[000056] user=94.85% sys= 0.21% alloc= 1142mb/s - CompactionExecutor:3
[000057] user=94.85% sys=-0.01% alloc= 1135mb/s - CompactionExecutor:4
[000054] user=97.10% sys=-0.03% alloc= 1084mb/s - CompactionExecutor:1
</code></pre>

<p>GC logs show normal numbers (using GCG1 with 26GB Heap as Als Cassandra tunning suggests trying):</p>

<pre><code>[Parallel Time: 8.9 ms, GC Workers: 10]
      [GC Worker Start (ms): Min: 2976069.1, Avg: 2976069.2, Max: 2976069.2, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 2.4, Avg: 2.4, Max: 2.5, Diff: 0.1, Sum: 24.4]
      [Update RS (ms): Min: 4.1, Avg: 4.3, Max: 4.5, Diff: 0.4, Sum: 42.8]
         [Processed Buffers: Min: 8, Avg: 16.2, Max: 33, Diff: 25, Sum: 162]
      [Scan RS (ms): Min: 0.1, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 1.4]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 1.8, Avg: 1.9, Max: 2.0, Diff: 0.2, Sum: 19.1]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [Termination Attempts: Min: 4, Avg: 6.0, Max: 8, Diff: 4, Sum: 60]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6]
      [GC Worker Total (ms): Min: 8.8, Avg: 8.8, Max: 8.9, Diff: 0.1, Sum: 88.4]
      [GC Worker End (ms): Min: 2976078.0, Avg: 2976078.0, Max: 2976078.0, Diff: 0.0]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.4 ms]
   [Other: 1.4 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.1 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.1 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.8 ms]
   [Eden: 4904.0M(4904.0M)-&gt;0.0B(4904.0M) Survivors: 8192.0K-&gt;8192.0K Heap: 6880.3M(8192.0M)-&gt;1976.7M(8192.0M)]
</code></pre>