{"action":"created","issue":{"url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/558","labels_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/558/labels{/name}","comments_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/558/comments","events_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/558/events","html_url":"https://github.com/OpenTSDB/opentsdb/issues/558","id":104795000,"number":558,"title":" GC overhead limit exceeded","user":{"login":"lufenghuan","id":3201795,"avatar_url":"https://avatars.githubusercontent.com/u/3201795?v=3","gravatar_id":"","url":"https://api.github.com/users/lufenghuan","html_url":"https://github.com/lufenghuan","followers_url":"https://api.github.com/users/lufenghuan/followers","following_url":"https://api.github.com/users/lufenghuan/following{/other_user}","gists_url":"https://api.github.com/users/lufenghuan/gists{/gist_id}","starred_url":"https://api.github.com/users/lufenghuan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lufenghuan/subscriptions","organizations_url":"https://api.github.com/users/lufenghuan/orgs","repos_url":"https://api.github.com/users/lufenghuan/repos","events_url":"https://api.github.com/users/lufenghuan/events{/privacy}","received_events_url":"https://api.github.com/users/lufenghuan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-03T22:38:40Z","updated_at":"2015-09-04T17:50:35Z","closed_at":null,"body":"Hi\r\nRunning benchmark on OpenTSDB, keep getting  \"GC overhead limit exceeded\" and hanging after running my inserting program for while (rate around 500,000 point/second). I am using put REST api. It looks similar to https://github.com/OpenTSDB/opentsdb/issues/557. I think my HBase cluster have enough capacity to hand this insertion rate. \r\n\r\n```\r\n17:29:51.861 ERROR [ConnectionManager.exceptionCaught] - Unexpected exception from downstream for [id: 0x0f39e398, /127.0.0.1:33382 => /127.0.0.1:4242]\r\njava.lang.OutOfMemoryError: GC overhead limit exceeded\r\n        at java.util.HashMap.createEntry(HashMap.java:901) ~[na:1.7.0_51]\r\n        at java.util.HashMap.addEntry(HashMap.java:888) ~[na:1.7.0_51]\r\n        at java.util.HashMap.put(HashMap.java:509) ~[na:1.7.0_51]\r\n        at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:458) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:311) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:26) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:238) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:118) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:232) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3051) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2153) ~[jackson-databind-2.4.3.jar:2.4.3]\r\n        at net.opentsdb.utils.JSON.parseToObject(JSON.java:163) ~[tsdb-2.1.0.jar:]\r\n        at net.opentsdb.tsd.HttpJsonSerializer.parsePutV1(HttpJsonSerializer.java:144) ~[tsdb-2.1.0.jar:]\r\n        at net.opentsdb.tsd.PutDataPointRpc.execute(PutDataPointRpc.java:102) ~[tsdb-2.1.0.jar:]\r\n        at net.opentsdb.tsd.RpcHandler.handleHttpQuery(RpcHandler.java:273) ~[tsdb-2.1.0.jar:]\r\n        at net.opentsdb.tsd.RpcHandler.messageReceived(RpcHandler.java:180) ~[tsdb-2.1.0.jar:]\r\n        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.timeout.IdleStateAwareChannelUpstreamHandler.handleUpstream(IdleStateAwareChannelUpstreamHandler.java:36) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(HttpContentEncoder.java:82) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/comments/137804642","html_url":"https://github.com/OpenTSDB/opentsdb/issues/558#issuecomment-137804642","issue_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/558","id":137804642,"user":{"login":"manolama","id":1377700,"avatar_url":"https://avatars.githubusercontent.com/u/1377700?v=3","gravatar_id":"","url":"https://api.github.com/users/manolama","html_url":"https://github.com/manolama","followers_url":"https://api.github.com/users/manolama/followers","following_url":"https://api.github.com/users/manolama/following{/other_user}","gists_url":"https://api.github.com/users/manolama/gists{/gist_id}","starred_url":"https://api.github.com/users/manolama/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manolama/subscriptions","organizations_url":"https://api.github.com/users/manolama/orgs","repos_url":"https://api.github.com/users/manolama/repos","events_url":"https://api.github.com/users/manolama/events{/privacy}","received_events_url":"https://api.github.com/users/manolama/received_events","type":"User","site_admin":false},"created_at":"2015-09-04T17:50:35Z","updated_at":"2015-09-04T17:50:35Z","body":"Yep, pre-splitting would definitely help as you shouldn't see the NSREs until a region is full or the server falls over."}}