Cassandra LeveledCompactionStrategy and high SSTable number per read
<p>We are using cassandra 2.0.17 and we have a table with 50% selects, 40% of updates and 10% of inserts (no deletes). </p>

<p>To have high read performance for such table we found that it is suggested to use LeveledCompactionStrategy (it is supposed to guarantee that 99% of reads will be fulfilled from single SSTable). Every day when I run <code>nodetool cfhistograms</code> i see more and more SSTtables per read. First day we had 1, than we had 1,2,3 ...<br>
and this morning I am seeing this:</p>

<pre><code>ubuntu@ip:~$ nodetool cfhistograms prodb groups | head -n 20                                                                                                                                
prodb/groups histograms

SSTables per Read
1 sstables: 27007
2 sstables: 97694
3 sstables: 95239
4 sstables: 3928
5 sstables: 14
6 sstables: 0
7 sstables: 19
</code></pre>

<p>The describe groups returns this:</p>

<pre><code>CREATE TABLE groups (
  ...
) WITH
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.100000 AND
  gc_grace_seconds=172800 AND
  index_interval=128 AND
  read_repair_chance=0.000000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'LeveledCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};
</code></pre>

<p>Is it normal? In such case we loose the advantage of using LeveledCompaction which as described in the documentation should guarantee 99% of reads from single sstable.</p>