{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/karyon/issues/206","labels_url":"https://api.github.com/repos/Netflix/karyon/issues/206/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/karyon/issues/206/comments","events_url":"https://api.github.com/repos/Netflix/karyon/issues/206/events","html_url":"https://github.com/Netflix/karyon/issues/206","id":64551483,"number":206,"title":"Using jersey-blocking: LEAK: ByteBuf.release() was not called before it's garbage-collected","user":{"login":"venkatraghavang","id":1405010,"avatar_url":"https://avatars.githubusercontent.com/u/1405010?v=3","gravatar_id":"","url":"https://api.github.com/users/venkatraghavang","html_url":"https://github.com/venkatraghavang","followers_url":"https://api.github.com/users/venkatraghavang/followers","following_url":"https://api.github.com/users/venkatraghavang/following{/other_user}","gists_url":"https://api.github.com/users/venkatraghavang/gists{/gist_id}","starred_url":"https://api.github.com/users/venkatraghavang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/venkatraghavang/subscriptions","organizations_url":"https://api.github.com/users/venkatraghavang/orgs","repos_url":"https://api.github.com/users/venkatraghavang/repos","events_url":"https://api.github.com/users/venkatraghavang/events{/privacy}","received_events_url":"https://api.github.com/users/venkatraghavang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-03-26T15:10:40Z","updated_at":"2015-03-30T18:43:42Z","closed_at":null,"body":"Found initially in 2.2.00-ALPHA7 release. Two issues: (1) Memleak on http requests through jersey-blocking. (2) InputStream not being flushed properly that returns 0 bytes to the jersey providers on concurrent requests.\r\n\r\n(1) Leak is similar to the one listed in RxNetty <a href=\"https://github.com/ReactiveX/RxNetty/issues/115\">Bug#115</a>. The fixes from <a href=\"https://github.com/Netflix/karyon/issues/200\">#200</a> or <a href=\"https://github.com/Netflix/karyon/issues/122\">#122</a> did not help.  Output:\r\n>-Dio.netty.leakDetectionLevel=paranoid\r\n\r\n```\r\n2015-03-26 10:43:01 ERROR ResourceLeakDetector:171 - LEAK: ByteBuf.release() was not called before it's garbage-collected.\r\nRecent access records: 2\r\n#2:\r\n\tio.netty.buffer.AdvancedLeakAwareByteBuf.readBytes(AdvancedLeakAwareByteBuf.java:463)\r\n\tnetflix.karyon.transport.util.HttpContentInputStream$3.read(HttpContentInputStream.java:90)\r\n\tnetflix.karyon.transport.util.HttpContentInputStream.blockingRead(HttpContentInputStream.java:127)\r\n\tnetflix.karyon.transport.util.HttpContentInputStream.read(HttpContentInputStream.java:81)\r\n\torg.codehaus.jackson.impl.ByteSourceBootstrapper.ensureLoaded(ByteSourceBootstrapper.java:499)\r\n\torg.codehaus.jackson.impl.ByteSourceBootstrapper.detectEncoding(ByteSourceBootstrapper.java:129)\r\n\torg.codehaus.jackson.impl.ByteSourceBootstrapper.constructParser(ByteSourceBootstrapper.java:216)\r\n\torg.codehaus.jackson.JsonFactory._createJsonParser(JsonFactory.java:785)\r\n\torg.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:561)\r\n\torg.codehaus.jackson.jaxrs.JacksonJsonProvider.readFrom(JacksonJsonProvider.java:405)\r\n\tcom.sun.jersey.json.impl.provider.entity.JacksonProviderProxy.readFrom(JacksonProviderProxy.java:139)\r\n\tcom.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:490)\r\n\tcom.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)\r\n\tcom.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:86)\r\n\tcom.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)\r\n\tcom.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)\r\n\tcom.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)\r\n\tcom.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)\r\n\tcom.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\r\n\tcom.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)\r\n\tcom.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\r\n\tcom.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)\r\n\tcom.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)\r\n\tcom.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)\r\n\tcom.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)\r\n\tcom.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)\r\n\tnetflix.karyon.jersey.blocking.JerseyBasedRouter$1.call(JerseyBasedRouter.java:70)\r\n\tnetflix.karyon.jersey.blocking.JerseyBasedRouter$1.call(JerseyBasedRouter.java:1)\r\n\trx.Observable$1.call(Observable.java:144)\r\n\trx.Observable$1.call(Observable.java:136)\r\n\trx.Observable.unsafeSubscribe(Observable.java:7195)\r\n\trx.internal.operators.OperatorSubscribeOn$1$1.call(OperatorSubscribeOn.java:62)\r\n\trx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:45)\r\n\tjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:482)\r\n\tjava.util.concurrent.FutureTask.run(FutureTask.java:273)\r\n\tjava.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:189)\r\n\tjava.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:303)\r\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1170)\r\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:640)\r\n\tjava.lang.Thread.run(Thread.java:853)\r\n#1:\r\n\tio.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:547)\r\n\tnetflix.karyon.transport.util.HttpContentInputStream$1.onNext(HttpContentInputStream.java:46)\r\n\tnetflix.karyon.transport.util.HttpContentInputStream$1.onNext(HttpContentInputStream.java:1)\r\n\trx.Observable$34.onNext(Observable.java:7168)\r\n\trx.observers.SafeSubscriber.onNext(SafeSubscriber.java:130)\r\n\tio.reactivex.netty.protocol.http.UnicastContentSubject$PassThruObserver.onNext(UnicastContentSubject.java:359)\r\n\trx.observers.SerializedObserver.onNext(SerializedObserver.java:159)\r\n\tio.reactivex.netty.protocol.http.UnicastContentSubject.onNext(UnicastContentSubject.java:311)\r\n\tio.reactivex.netty.protocol.http.server.ServerRequestResponseConverter.invokeContentOnNext(ServerRequestResponseConverter.java:160)\r\n\tio.reactivex.netty.protocol.http.server.ServerRequestResponseConverter.channelRead(ServerRequestResponseConverter.java:96)\r\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)\r\n\tio.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)\r\n\tio.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)\r\n\tio.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36)\r\n\tio.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)\r\n\tio.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)\r\n\tjava.lang.Thread.run(Thread.java:853)\r\nCreated at:\r\n\tio.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:259)\r\n\tio.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)\r\n\tio.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:141)\r\n\tio.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:75)\r\n\tnetflix.karyon.transport.util.HttpContentInputStream.<init>(HttpContentInputStream.java:29)\r\n\tnetflix.karyon.jersey.blocking.NettyToJerseyBridge.bridgeRequest(NettyToJerseyBridge.java:55)\r\n\tnetflix.karyon.jersey.blocking.JerseyBasedRouter.handle(JerseyBasedRouter.java:61)\r\n\tnetflix.karyon.jersey.blocking.JerseyBasedRouter.handle(JerseyBasedRouter.java:1)\r\n\tnetflix.karyon.transport.interceptor.InterceptorExecutor$ChainSubscriber.onCompleted(InterceptorExecutor.java:211)\r\n\trx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:79)\r\n\trx.Subscriber.setProducer(Subscriber.java:137)\r\n\trx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:47)\r\n\trx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:33)\r\n\trx.Observable.unsafeSubscribe(Observable.java:7195)\r\n\tnetflix.karyon.transport.interceptor.InterceptorExecutor$ChainSubscriber.handleResult(InterceptorExecutor.java:223)\r\n\tnetflix.karyon.transport.interceptor.InterceptorExecutor$ChainSubscriber.onCompleted(InterceptorExecutor.java:209)\r\n\trx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:79)\r\n\trx.Subscriber.setProducer(Subscriber.java:137)\r\n\trx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:47)\r\n\trx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:33)\r\n\trx.Observable$1.call(Observable.java:144)\r\n\trx.Observable$1.call(Observable.java:136)\r\n\trx.Observable.subscribe(Observable.java:7284)\r\n\tio.reactivex.netty.protocol.http.server.HttpConnectionHandler$1$1.onNext(HttpConnectionHandler.java:112)\r\n\tio.reactivex.netty.protocol.http.server.HttpConnectionHandler$1$1.onNext(HttpConnectionHandler.java:62)\r\n\trx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:224)\r\n\trx.subjects.PublishSubject.onNext(PublishSubject.java:121)\r\n\trx.observers.SerializedObserver.onNext(SerializedObserver.java:159)\r\n\trx.subjects.SerializedSubject.onNext(SerializedSubject.java:64)\r\n\tio.reactivex.netty.pipeline.ObservableAdapter.channelRead(ObservableAdapter.java:40)\r\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)\r\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)\r\n\tio.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)\r\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)\r\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)\r\n\tio.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)\r\n\tio.reactivex.netty.protocol.http.server.ServerRequestResponseConverter.channelRead(ServerRequestResponseConverter.java:89)\r\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)\r\n\tio.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)\r\n\tio.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)\r\n\tio.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36)\r\n\tio.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)\r\n\tio.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)\r\n\tjava.lang.Thread.run(Thread.java:853)\r\n```\r\n\r\n(2) During a load test, a custom object mapper  gets 0 available bytes to read although the stream gets written to the `contentBuffer` and read properly in HttpContentInputStream. As a workaround, I've been trying to introduce a delay in the subscription on `JerseyBasedRouter` handle() function. But, I'm suspecting this will be be alright if fixes from <a href=\"https://github.com/Netflix/karyon/pull/199\">#199</a> gets merged. \r\n\r\nStill looking into the code to see what can I get out of this. The tests were made on an simple servlet that consumes a JSON."},"comment":{"url":"https://api.github.com/repos/Netflix/karyon/issues/comments/87786884","html_url":"https://github.com/Netflix/karyon/issues/206#issuecomment-87786884","issue_url":"https://api.github.com/repos/Netflix/karyon/issues/206","id":87786884,"user":{"login":"maksimlikharev","id":9421260,"avatar_url":"https://avatars.githubusercontent.com/u/9421260?v=3","gravatar_id":"","url":"https://api.github.com/users/maksimlikharev","html_url":"https://github.com/maksimlikharev","followers_url":"https://api.github.com/users/maksimlikharev/followers","following_url":"https://api.github.com/users/maksimlikharev/following{/other_user}","gists_url":"https://api.github.com/users/maksimlikharev/gists{/gist_id}","starred_url":"https://api.github.com/users/maksimlikharev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maksimlikharev/subscriptions","organizations_url":"https://api.github.com/users/maksimlikharev/orgs","repos_url":"https://api.github.com/users/maksimlikharev/repos","events_url":"https://api.github.com/users/maksimlikharev/events{/privacy}","received_events_url":"https://api.github.com/users/maksimlikharev/received_events","type":"User","site_admin":false},"created_at":"2015-03-30T18:43:41Z","updated_at":"2015-03-30T18:43:41Z","body":"is it that, release(),  will decrement reference count and effectively ask for the object deallocation?\r\nI would think we have to release at the end of processing, then.\r\n"}}