{"action":"created","issue":{"url":"https://api.github.com/repos/mp911de/spinach/issues/12","labels_url":"https://api.github.com/repos/mp911de/spinach/issues/12/labels{/name}","comments_url":"https://api.github.com/repos/mp911de/spinach/issues/12/comments","events_url":"https://api.github.com/repos/mp911de/spinach/issues/12/events","html_url":"https://github.com/mp911de/spinach/issues/12","id":107923166,"number":12,"title":"Connections timing out triggering a reconnect","user":{"login":"mzapletal","id":229582,"avatar_url":"https://avatars.githubusercontent.com/u/229582?v=3","gravatar_id":"","url":"https://api.github.com/users/mzapletal","html_url":"https://github.com/mzapletal","followers_url":"https://api.github.com/users/mzapletal/followers","following_url":"https://api.github.com/users/mzapletal/following{/other_user}","gists_url":"https://api.github.com/users/mzapletal/gists{/gist_id}","starred_url":"https://api.github.com/users/mzapletal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mzapletal/subscriptions","organizations_url":"https://api.github.com/users/mzapletal/orgs","repos_url":"https://api.github.com/users/mzapletal/repos","events_url":"https://api.github.com/users/mzapletal/events{/privacy}","received_events_url":"https://api.github.com/users/mzapletal/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mp911de/spinach/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":{"login":"mp911de","id":1035015,"avatar_url":"https://avatars.githubusercontent.com/u/1035015?v=3","gravatar_id":"","url":"https://api.github.com/users/mp911de","html_url":"https://github.com/mp911de","followers_url":"https://api.github.com/users/mp911de/followers","following_url":"https://api.github.com/users/mp911de/following{/other_user}","gists_url":"https://api.github.com/users/mp911de/gists{/gist_id}","starred_url":"https://api.github.com/users/mp911de/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mp911de/subscriptions","organizations_url":"https://api.github.com/users/mp911de/orgs","repos_url":"https://api.github.com/users/mp911de/repos","events_url":"https://api.github.com/users/mp911de/events{/privacy}","received_events_url":"https://api.github.com/users/mp911de/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2015-09-23T13:46:21Z","updated_at":"2015-09-24T08:00:45Z","closed_at":null,"body":"I know this is pretty bad to track and it may just happen due to simple/small network outages. Nevertheless, I would like to ask if you are aware of this issue and may have a solution for it: we are experiencing connection timeouts to our disque nodes quite frequently (about 3-4 times a day). Such a connection timeout involves a reconnect to the other node then (we are running a cluster of 2 nodes). I see that there is no spinach code (but only netty code) involved, but is there any advise for configuring spinach (or netty under the hood) and avoid these reconnects?\r\n\r\n``` java\r\n\r\nAn exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. [i.n.c.DefaultChannelPipeline] [nioEventLoopGroup-2-4] \r\njava.io.IOException: Connection timed out\r\nat sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_79]\r\n\tat sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.7.0_79]\r\n\tat sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.7.0_79]\r\n\tat sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.7.0_79]\r\n\tat sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.7.0_79]\r\n\tat io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:447) ~[netty-buffer-4.0.28.Final.jar:4.0.28.Final]\r\n\tat io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) ~[netty-buffer-4.0.28.Final.jar:4.0.28.Final]\r\n\tat io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242) ~[netty-transport-4.0.28.Final.jar:4.0.28.Final]\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) ~[netty-transport-4.0.28.Final.jar:4.0.28.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.28.Final.jar:4.0.28.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.28.Final.jar:4.0.28.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.28.Final.jar:4.0.28.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.28.Final.jar:4.0.28.Final]\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.28.Final.jar:4.0.28.Final]\r\n\tat io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-common-4.0.28.Final.jar:4.0.28.Final]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]\r\njava.io.IOException: Connection timed out\r\n\tat sun.nio.ch.FileDispatcherImpl.read0(Native Method)\r\n\tat sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)\r\n\tat sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)\r\n\tat sun.nio.ch.IOUtil.read(IOUtil.java:192)\r\n\tat sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)\r\n\tat io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:447)\r\n\tat io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)\r\n\tat io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)\r\n\tat io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```"},"comment":{"url":"https://api.github.com/repos/mp911de/spinach/issues/comments/142845948","html_url":"https://github.com/mp911de/spinach/issues/12#issuecomment-142845948","issue_url":"https://api.github.com/repos/mp911de/spinach/issues/12","id":142845948,"user":{"login":"mzapletal","id":229582,"avatar_url":"https://avatars.githubusercontent.com/u/229582?v=3","gravatar_id":"","url":"https://api.github.com/users/mzapletal","html_url":"https://github.com/mzapletal","followers_url":"https://api.github.com/users/mzapletal/followers","following_url":"https://api.github.com/users/mzapletal/following{/other_user}","gists_url":"https://api.github.com/users/mzapletal/gists{/gist_id}","starred_url":"https://api.github.com/users/mzapletal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mzapletal/subscriptions","organizations_url":"https://api.github.com/users/mzapletal/orgs","repos_url":"https://api.github.com/users/mzapletal/repos","events_url":"https://api.github.com/users/mzapletal/events{/privacy}","received_events_url":"https://api.github.com/users/mzapletal/received_events","type":"User","site_admin":false},"created_at":"2015-09-24T08:00:45Z","updated_at":"2015-09-24T08:00:45Z","body":"Thanks for the thorough investigation. To avoid polluting logs (assuming that this is easy to resolve by reconnecting) what do you think about logging the exception on debug? As far as I remember the reconnect is logged on info anyway. "}}