HBase fails with org.apache.hadoop.hbase.NotServingRegionException region is not online
<p>I have an asynchronous application that saves data to HBase using <a href="https://github.com/jurmous/async-hbase-client" rel="nofollow noreferrer">async-hbase-client.</a></p>

<p>My HBase version is: 1.0.0-cdh5.6.0
My async HBase client version is: 0.9.0</p>

<p>The application ran fine for a while (~4 or 5 days) but during the weekend it started failing with the following exception:</p>

<pre><code>org.apache.hadoop.hbase.NotServingRegionException: Region pageviews,,1463568860289.298bb29bbd148a0a62ec90885ef8d027. is not online on //some address here
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2786)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:922)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:1965)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32203)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
        at java.lang.Thread.run(Thread.java:745)
</code></pre>

<p>What I did: </p>

<ul>
<li>I verified that HBase is indeed functioning by going into the HBase shell and playing with some tables there which worked fine</li>
<li>I logged into the zookeeper shell by running <code>./hbase zkcli</code> and then running: <code>rmr /hbase/root-region-server</code> - I did this according to this link: <a href="http://rogueleaderr.com/post/32963921889/hbase-error-region-is-not-online-root-0" rel="nofollow noreferrer">http://rogueleaderr.com/post/32963921889/hbase-error-region-is-not-online-root-0</a>. This did not work with the following error: <code>Node does not exist: /hbase/root-region-server</code>. I tried resolving this problem and encountered this solution suggestion: <a href="https://stackoverflow.com/questions/19903622/hbase-hbase-meta-region-server-node-does-not-exist">HBase: /hbase/meta-region-server node does not exist</a>. So instead I ran: <code>rmr /hbase/meta-region-server</code> which didn't do anything (at least it didn't write anything)</li>
<li>I tried looking for other solutions, some suggested it has something to do with incompatible HBase versions which might sound right but it did work for several days without any issues so I'm wondering what exactly is the problem</li>
</ul>

<p>If anyone has any ideas on what exactly is the problem here I'll appreciate it. Currently I'm kind of in the dark here</p>

<p>Thanks</p>