{"action":"closed","issue":{"url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/659","labels_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/659/labels{/name}","comments_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/659/comments","events_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/659/events","html_url":"https://github.com/OpenTSDB/opentsdb/issues/659","id":121817193,"number":659,"title":"Failing to read META (hbase:meta)","user":{"login":"pfischermx","id":2073487,"avatar_url":"https://avatars.githubusercontent.com/u/2073487?v=3","gravatar_id":"","url":"https://api.github.com/users/pfischermx","html_url":"https://github.com/pfischermx","followers_url":"https://api.github.com/users/pfischermx/followers","following_url":"https://api.github.com/users/pfischermx/following{/other_user}","gists_url":"https://api.github.com/users/pfischermx/gists{/gist_id}","starred_url":"https://api.github.com/users/pfischermx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pfischermx/subscriptions","organizations_url":"https://api.github.com/users/pfischermx/orgs","repos_url":"https://api.github.com/users/pfischermx/repos","events_url":"https://api.github.com/users/pfischermx/events{/privacy}","received_events_url":"https://api.github.com/users/pfischermx/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/OpenTSDB/opentsdb/labels/bug","name":"bug","color":"ff0000"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-12-12T00:09:40Z","updated_at":"2015-12-16T23:12:48Z","closed_at":"2015-12-16T23:12:48Z","body":"Howdy,\r\n\r\nI'm giving opentsdb a try. Our existing setup includes an experimentation hbase cluster (to play with) that runs hbase 0.98, we also use kerberos for authenticating with it.\r\n\r\nI was able to create the tables and to configure tsdb to use the auth after I went with the latest RC (RC3), however I'm stuck in trying to start the tsdb, I keep getting the following:\r\n\r\n2015-12-12 00:02:59,772 INFO  [AsyncHBase Timer HBaseClient #1-SendThread()] ClientCnxn: Opening socket connection to server x.x.x.x:50512\r\n2015-12-12 00:02:59,772 INFO  [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region\r\n2015-12-12 00:02:59,773 INFO  [AsyncHBase Timer HBaseClient #1-SendThread(x.x.x.x:50512)] ClientCnxn: Socket connection established to x.x.x.x:50512, initiating session\r\n2015-12-12 00:02:59,776 INFO  [AsyncHBase Timer HBaseClient #1-SendThread(x.x.x.x:50512)] ClientCnxn: Session establishment complete on server x.x.x.x:50512, sessionid = 0x251705aced0233b, negotiated timeout = 5000\r\n2015-12-12 00:02:59,778 INFO  [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511\r\n2015-12-12 00:02:59,779 WARN  [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Probe Exists(table=\"hbase:meta\", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=RegionInfo(table=\"hbase:meta\", region_name=\"hbase:meta,,1\", stop_key=\"\")) failed\r\norg.hbase.async.NonRecoverableException: Too many attempts: Exists(table=\"hbase:meta\", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table=\"hbase:meta\", region_name=\"hbase:meta,,1\", stop_key=\"\"))\r\n\tat org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) [asynchbase-1.7.1-20151004.015637-1.jar:na]\r\n\tat org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) [asynchbase-1.7.1-20151004.015637-1.jar:na]\r\n\tat org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) [asynchbase-1.7.1-20151004.015637-1.jar:na]\r\n\tat org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) [asynchbase-1.7.1-20151004.015637-1.jar:na]\r\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\r\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\r\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\r\n\tat org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3632) [asynchbase-1.7.1-20151004.015637-1.jar:na]\r\n\tat org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:565) [zookeeper-3.3.6.jar:3.3.6-1366786]\r\n\tat org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) [zookeeper-3.3.6.jar:3.3.6-1366786]\r\n2015-12-12 00:02:59,780 INFO  [AsyncHBase I/O Worker #12] HBaseClient: Channel [id: 0x7c1125ef, /10.193.194.140:48266 :> /10.192.210.33:50511] is disconnecting: [id: 0x7c1125ef, /10.193.194.140:48266 :> /10.192.210.33:50511] DISCONNECTED\r\n\r\n.. at the end it just dies with:\r\n\r\naused by: org.hbase.async.NonRecoverableException: Too many attempts: HBaseRpc(method=getClosestRowBefore, table=\"hbase:meta\", key=\"perf_namespace:opentsdb,\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00:AsyncHBase~probe~<;_<,:\", region=RegionInfo(table=\"hbase:meta\", region_name=\"hbase:meta,,1\", stop_key=\"\"), attempt=11, timeout=0, hasTimedout=false)\r\n\tat org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056)\r\n\tat org.hbase.async.HBaseClient.handleNSRE(HBaseClient.java:2837)\r\n\t... 29 more\r\nCaused by: org.hbase.async.NotServingRegionException: Connection reset: [id: 0xaee6389e, /10.193.194.140:48389 :> /10.192.210.33:50511] got disconnected\r\nCaused by RPC: HBaseRpc(method=getClosestRowBefore, table=\"hbase:meta\", key=\"perf_namespace:opentsdb,\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00:AsyncHBase~probe~<;_<,:\", region=RegionInfo(table=\"hbase:meta\", region_name=\"hbase:meta,,1\", stop_key=\"\"), attempt=0, timeout=0, hasTimedout=false)\r\n\r\nI can confirm (with hbase shell) that the table exists and can confirm that 10.129.210.33 is the region server holding hbase:root,,0.2132948379.\r\n\r\nI'm using: asynchbase-1.7.1-20151004.015637-1.jar\r\nhbase 0.98.7.14\r\n\r\nAny ideas if I missed something or is this support for hbase?\r\n\r\nthanks!"}}