{"action":"created","issue":{"url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/445","labels_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/445/labels{/name}","comments_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/445/comments","events_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/445/events","html_url":"https://github.com/OpenTSDB/opentsdb/issues/445","id":57909202,"number":445,"title":"java.lang.ArrayIndexOutOfBoundsException: null using http api","user":{"login":"scicco","id":943152,"avatar_url":"https://avatars.githubusercontent.com/u/943152?v=3","gravatar_id":"","url":"https://api.github.com/users/scicco","html_url":"https://github.com/scicco","followers_url":"https://api.github.com/users/scicco/followers","following_url":"https://api.github.com/users/scicco/following{/other_user}","gists_url":"https://api.github.com/users/scicco/gists{/gist_id}","starred_url":"https://api.github.com/users/scicco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scicco/subscriptions","organizations_url":"https://api.github.com/users/scicco/orgs","repos_url":"https://api.github.com/users/scicco/repos","events_url":"https://api.github.com/users/scicco/events{/privacy}","received_events_url":"https://api.github.com/users/scicco/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-17T10:49:02Z","updated_at":"2015-02-22T22:26:18Z","closed_at":null,"body":"Hello,\r\n\r\nwe are using \r\n\r\nOpenTSDB version [2.0.1] built from revision in a MODIFIED state\r\nBuilt on Mon Nov 10 03:13:10 GMT+100 2014 by root@centos.localhost:/root/rpmbuild/BUILD/opentsdb-2.0.1\r\n\r\nand we're getting this error on log when running the following query:\r\n\r\n12:27:56.612 ERROR [HttpQuery.logError] - [id: 0x95b622b0, /127.0.0.1:53566 => /127.0.0.1:14242] Internal Server Error on /q?start=1d-ago&end=2015/02/17-11:15:57&m=avg:qoe.upload.total.kbps{fw=VV2222_CLC_3.1.0.0040}&ascii\r\njava.lang.ArrayIndexOutOfBoundsException: null\r\n12:27:56.613 INFO  [HttpQuery.logInfo] - [id: 0x95b622b0, /127.0.0.1:53566 => /127.0.0.1:14242] HTTP /q?start=1d-ago&end=2015/02/17-11:15:57&m=avg:qoe.upload.total.kbps{fw=VV2222_CLC_3.1.0.0040}&ascii done in 9ms\r\n\r\nwhen i use http query api i see the following lines:\r\n\r\n12:40:40.547 INFO  [HttpQuery.logInfo] - [id: 0x93e15f16, /127.0.0.1:32851 :> /127.0.0.1:14242] HTTP /api/query?start=1w-ago&m=avg:24h-avg:qoe.upload.user.kbps%7B\r\n12:40:40.666 ERROR [HttpQuery.logError] - [id: 0xd3ac7a0c, /127.0.0.1:32855 => /127.0.0.1:14242] Internal Server Error on /api/query?start=1w-ago&m=min:24h-min:qo\r\njava.lang.RuntimeException: Shouldn't be here\r\n        at net.opentsdb.tsd.QueryRpc.execute(QueryRpc.java:135) ~[tsdb-2.0.1.jar:]\r\n        at net.opentsdb.tsd.RpcHandler.handleHttpQuery(RpcHandler.java:255) [tsdb-2.0.1.jar:]\r\n        at net.opentsdb.tsd.RpcHandler.messageReceived(RpcHandler.java:163) [tsdb-2.0.1.jar:]\r\n        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.9.4.Final.jar:na]\r\n        at net.opentsdb.tsd.ConnectionManager.handleUpstream(ConnectionManager.java:87) [tsdb-2.0.1.jar:]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]\r\n        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]\r\nCaused by: com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception:\r\n        at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169) ~[suasync-1.4.0.jar:fe17b98]\r\n        at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:158) ~[suasync-1.4.0.jar:fe17b98]\r\n        at com.stumbleupon.async.DeferredGroup.access$200(DeferredGroup.java:36) ~[suasync-1.4.0.jar:fe17b98]\r\n        at com.stumbleupon.async.DeferredGroup$1NotifyOrdered.call(DeferredGroup.java:97) ~[suasync-1.4.0.jar:fe17b98]\r\n        at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) ~[suasync-1.4.0.jar:fe17b98]\r\n        at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) ~[suasync-1.4.0.jar:fe17b98]\r\n        at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) ~[suasync-1.4.0.jar:fe17b98]\r\n        at net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:419) ~[tsdb-2.0.1.jar:]\r\n        at net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:352) ~[tsdb-2.0.1.jar:]\r\n        at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) ~[suasync-1.4.0.jar:fe17b98]\r\n        at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) ~[suasync-1.4.0.jar:fe17b98]\r\n        at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) ~[suasync-1.4.0.jar:fe17b98]\r\n        at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:506) ~[asynchbase-1.6.0.jar:na]\r\n        at org.hbase.async.RegionClient.decode(RegionClient.java:1351) ~[asynchbase-1.6.0.jar:na]\r\n        at org.hbase.async.RegionClient.decode(RegionClient.java:89) ~[asynchbase-1.6.0.jar:na]\r\n        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]\r\n        at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1082) ~[asynchbase-1.6.0.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]\r\n        at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:2677) ~[asynchbase-1.6.0.jar:na]\r\n        ... 12 common frames omitted\r\njava.lang.ArrayIndexOutOfBoundsException: null\r\n\r\n\r\nthe same query using cli give us no problem at all:\r\ntsdb scan --import 1w-ago avg qoe.upload.total.kbps fw=VV2222_CLC_3.1.0.0040\r\n\r\nwe've perfomed an fsck on the same data portion with no problem found:\r\n\r\ntsdb fsck 1w-ago avg qoe.upload.total.kbps fw=VV2222_CLC_3.1.0.0040\r\n\r\n2015-02-17 12:32:09,375 INFO  [main] Config: Successfully loaded configuration file: /etc/opentsdb/opentsdb.conf\r\n2015-02-17 12:32:09,495 INFO  [main] ZooKeeper: Client environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT\r\n2015-02-17 12:32:09,496 INFO  [main] ZooKeeper: Client environment:host.name=adb2.adb.netvision.net.il\r\n2015-02-17 12:32:09,496 INFO  [main] ZooKeeper: Client environment:java.version=1.7.0_71\r\n2015-02-17 12:32:09,496 INFO  [main] ZooKeeper: Client environment:java.vendor=Oracle Corporation\r\n2015-02-17 12:32:09,496 INFO  [main] ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre\r\n2015-02-17 12:32:09,496 INFO  [main] ZooKeeper: Client environment:java.class.path=/usr/share/opentsdb/*.jar:/usr/share/opentsdb:/usr/share/opentsdb/bin:/usr/share/opentsdb/lib/asynchbase-1.6.0.jar:/usr/share/opentsdb/lib/guava-18.0.jar:/usr/share/opentsdb/lib/jackson-annotations-2.4.3.jar:/usr/share/opentsdb/lib/jackson-core-2.4.3.jar:/usr/share/opentsdb/lib/jackson-databind-2.4.3.jar:/usr/share/opentsdb/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/opentsdb/lib/logback-classic-1.0.13.jar:/usr/share/opentsdb/lib/logback-core-1.0.13.jar:/usr/share/opentsdb/lib/netty-3.9.4.Final.jar:/usr/share/opentsdb/lib/protobuf-java-2.5.0.jar:/usr/share/opentsdb/lib/slf4j-api-1.7.7.jar:/usr/share/opentsdb/lib/suasync-1.4.0.jar:/usr/share/opentsdb/lib/tsdb-2.0.1.jar:/usr/share/opentsdb/lib/zookeeper-3.3.6.jar:/usr/share/opentsdb/etc/opentsdb\r\n2015-02-17 12:32:09,496 INFO  [main] ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib\r\n2015-02-17 12:32:09,497 INFO  [main] ZooKeeper: Client environment:java.io.tmpdir=/tmp\r\n2015-02-17 12:32:09,497 INFO  [main] ZooKeeper: Client environment:java.compiler=<NA>\r\n2015-02-17 12:32:09,497 INFO  [main] ZooKeeper: Client environment:os.name=Linux\r\n2015-02-17 12:32:09,497 INFO  [main] ZooKeeper: Client environment:os.arch=amd64\r\n2015-02-17 12:32:09,497 INFO  [main] ZooKeeper: Client environment:os.version=2.6.32-431.20.5.el6.x86_64\r\n2015-02-17 12:32:09,497 INFO  [main] ZooKeeper: Client environment:user.name=root\r\n2015-02-17 12:32:09,498 INFO  [main] ZooKeeper: Client environment:user.home=/root\r\n2015-02-17 12:32:09,498 INFO  [main] ZooKeeper: Client environment:user.dir=/root\r\n2015-02-17 12:32:09,499 INFO  [main] ZooKeeper: Initiating client connection, connectString=localhost sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@748684b1\r\n2015-02-17 12:32:09,511 INFO  [main] HBaseClient: Need to find the -ROOT- region\r\n2015-02-17 12:32:09,511 INFO  [main-SendThread()] ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181\r\n2015-02-17 12:32:09,517 INFO  [main-SendThread(localhost:2181)] ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session\r\n2015-02-17 12:32:09,530 INFO  [main-SendThread(localhost:2181)] ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x149ecdb066d5a7f, negotiated timeout = 5000\r\n2015-02-17 12:32:09,555 INFO  [main-EventThread] HBaseClient: Connecting to .META. region @ XXX.XXX.XXX.XXX:60020\r\n2015-02-17 12:32:09,602 INFO  [main-EventThread] ZooKeeper: Session: 0x149ecdb066d5a7f closed\r\n2015-02-17 12:32:09,603 INFO  [main-EventThread] ClientCnxn: EventThread shut down\r\n2015-02-17 12:32:09,642 INFO  [New I/O worker #1] HBaseClient: Added client for region RegionInfo(table=\"tsdb-uid\", region_name=\"tsdb-uid,,1399570190507.d4319c9ca6dbf4a1934d8503590173b5.\", stop_key=\"\"), which was added to the regions cache.  Now we know that RegionClient@402060195(chan=[id: 0x11194df0, /XXX.XXX.XXX.XXX:56853 => /XXX.XXX.XXX.XXX:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=1) is hosting 1 region.\r\n2015-02-17 12:32:09,644 INFO  [New I/O worker #1] HBaseClient: Added client for region RegionInfo(table=\"tsdb\", region_name=\"tsdb,,1413630095973.1a8b66891d39bd129d1af372e73f4980.\", stop_key=[0, 0, 9, 84, 64, 21, -80, 0, 0, 2, 0, 14, 50]), which was added to the regions cache.  Now we know that RegionClient@402060195(chan=[id: 0x11194df0, /XXX.XXX.XXX.XXX:56853 => /XXX.XXX.XXX.XXX:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=1) is hosting 2 regions.\r\n2015-02-17 12:32:09,691 INFO  [main] Fsck: Starting to fsck data covered by TsdbQuery(start_time=1423564329, end_time=1424169129691, metric=[0, 0, 3], tags=[[0, 0, 4, 0, -65, -123]], rate=false, aggregator=avg, group_bys=())\r\n27875 KVs (in 27864 rows) analyzed in 1237ms (~22534 KV/s)\r\nNo error found.\r\n2015-02-17 12:32:10,929 INFO  [main] TSDB: Flushing compaction queue\r\n2015-02-17 12:32:10,932 INFO  [main] HBaseClient: Lost connection with the .META. region\r\n\r\nLet me know if you need more details.\r\n\r\nThank you in advance"},"comment":{"url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/comments/75466988","html_url":"https://github.com/OpenTSDB/opentsdb/issues/445#issuecomment-75466988","issue_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/445","id":75466988,"user":{"login":"oozie","id":1587140,"avatar_url":"https://avatars.githubusercontent.com/u/1587140?v=3","gravatar_id":"","url":"https://api.github.com/users/oozie","html_url":"https://github.com/oozie","followers_url":"https://api.github.com/users/oozie/followers","following_url":"https://api.github.com/users/oozie/following{/other_user}","gists_url":"https://api.github.com/users/oozie/gists{/gist_id}","starred_url":"https://api.github.com/users/oozie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oozie/subscriptions","organizations_url":"https://api.github.com/users/oozie/orgs","repos_url":"https://api.github.com/users/oozie/repos","events_url":"https://api.github.com/users/oozie/events{/privacy}","received_events_url":"https://api.github.com/users/oozie/received_events","type":"User","site_admin":false},"created_at":"2015-02-22T22:26:18Z","updated_at":"2015-02-22T22:26:18Z","body":"Does it manifest a different behavior if you append `&no_annotations=1` to the query string?"}}