HBase performance with large number of scans
<p>I have a table with hundred of million records. This table contains data about servers and events genereated on them. Following is the row key of the table:</p>

<p>rowkey = md5(serverId) + timestamp  [32 hex characters + 10 digits = 42 characters]</p>

<p>One of the use case is to list all the events from time t1 to t2. For this, normal scan is taking too much time. To speed up the things, I have done the following:</p>

<ol>
<li>Fetch the list of unique serverId from another table (real fast).</li>
<li>Divide the above list in 256 buckets based on first two hex characters of md5 of serverIds.</li>
<li>For each bucket, call a co-processor (parallel requests) with list of serverId, start time and end time.</li>
</ol>

<p>The co-processor scan the table as follow:</p>

<pre><code>for (String serverId :  serverIds) {
  byte[] startKey = generateKeyserverId, startTime);
  byte[] endKey = generateKey(serverId, endTime);
  Scan scan = new Scan(startKey, endKey);
  InternalScanner scanner = env.getRegion().getScanner(scan);
  ....
}
</code></pre>

<p>I am able to get the result quick fast with this approach. My only concern is the large number of scans. If the table has 20,000 serverIds then the above code is making 20,000 scans. Will it impact the overall performance and scalability of HBase?</p>