{"action":"opened","issue":{"url":"https://api.github.com/repos/elasticsearch/logstash/issues/2593","labels_url":"https://api.github.com/repos/elasticsearch/logstash/issues/2593/labels{/name}","comments_url":"https://api.github.com/repos/elasticsearch/logstash/issues/2593/comments","events_url":"https://api.github.com/repos/elasticsearch/logstash/issues/2593/events","html_url":"https://github.com/elasticsearch/logstash/issues/2593","id":57453022,"number":2593,"title":"Logstash 1.4.2 OOM when ES down","user":{"login":"abonuccelli-es","id":8770097,"avatar_url":"https://avatars.githubusercontent.com/u/8770097?v=3","gravatar_id":"","url":"https://api.github.com/users/abonuccelli-es","html_url":"https://github.com/abonuccelli-es","followers_url":"https://api.github.com/users/abonuccelli-es/followers","following_url":"https://api.github.com/users/abonuccelli-es/following{/other_user}","gists_url":"https://api.github.com/users/abonuccelli-es/gists{/gist_id}","starred_url":"https://api.github.com/users/abonuccelli-es/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abonuccelli-es/subscriptions","organizations_url":"https://api.github.com/users/abonuccelli-es/orgs","repos_url":"https://api.github.com/users/abonuccelli-es/repos","events_url":"https://api.github.com/users/abonuccelli-es/events{/privacy}","received_events_url":"https://api.github.com/users/abonuccelli-es/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-02-12T12:17:20Z","updated_at":"2015-02-12T12:17:20Z","closed_at":null,"body":"I have run this on 2 nodes Elasticsearch 1.4.2 cluster\r\n\r\nLogstash 1.4.2 config as below:\r\n\r\n```\r\ninput {\r\n\r\n    file {\r\n        path => \"/tmp/hugefile\"\r\n        sincedb_path => \"/dev/null\" \r\n        start_position => \"beginning\"\r\n    }\r\n    file {\r\n                path => \"/tmp/hugefile2\"\r\n                sincedb_path => \"/dev/null\"\r\n        start_position => \"beginning\"\r\n        }\r\n    file {\r\n                path => \"/tmp/hugefile3\"\r\n                sincedb_path => \"/dev/null\"\r\n        start_position => \"beginning\"\r\n        }\r\n    file {\r\n                path => \"/tmp/hugefile4\"\r\n                sincedb_path => \"/dev/null\"\r\n                start_position => \"beginning\"\r\n        }\r\n    file {\r\n                path => \"/tmp/hugefile5\"\r\n                sincedb_path => \"/dev/null\"\r\n                start_position => \"beginning\"\r\n        }\r\n    file {\r\n                path => \"/tmp/hugefile6\"\r\n                sincedb_path => \"/dev/null\"\r\n                start_position => \"beginning\"\r\n        }\r\n    file {\r\n                path => \"/tmp/hugefile7\"\r\n                sincedb_path => \"/dev/null\"\r\n                start_position => \"beginning\"\r\n        }\r\n    file {\r\n                path => \"/tmp/hugefile8\"\r\n                sincedb_path => \"/dev/null\"\r\n                start_position => \"beginning\"\r\n        }\r\n    file {\r\n                path => \"/tmp/hugefile9\"\r\n                sincedb_path => \"/dev/null\"\r\n                start_position => \"beginning\"\r\n        }\r\n    file {\r\n                path => \"/tmp/hugefile10\"\r\n                sincedb_path => \"/dev/null\"\r\n                start_position => \"beginning\"\r\n        }\r\n\r\n}\r\n\r\nfilter {\r\n\r\n}\r\n\r\noutput {\r\n\r\n    elasticsearch {\r\n        cluster => \"tony_prod\"\r\n        protocol => \"transport\"\r\n        workers => 16\r\n        host => \"192.168.0.101\"\r\n        port => \"9302\"\r\n        index => \"test\"\r\n\r\n    }\r\n     stdout { codec => rubydebug }\r\n}\r\n```\r\n\r\n\r\nwhere source files are\r\n\r\n```\r\n $ du -hs /tmp/hugefile*\r\n4,7G    /tmp/hugefile\r\n4,7G    /tmp/hugefile10\r\n4,7G    /tmp/hugefile2\r\n4,7G    /tmp/hugefile3\r\n4,7G    /tmp/hugefile4\r\n4,7G    /tmp/hugefile5\r\n4,7G    /tmp/hugefile6\r\n4,7G    /tmp/hugefile7\r\n4,7G    /tmp/hugefile8\r\n4,7G    /tmp/hugefile9\r\n```\r\n\r\n1) Elasticsearch stopped responding (was yellow)\r\n2) Logstash threads running out of memory and crashing\r\n\r\nthis is all running on same beefy host (32GB ram 8 cores SSD)\r\n\r\nin Logstash stdout\r\n\r\n```\r\nException in thread \"elasticsearch[logstash-w530-12453-4130][transport_client_worker][T#6]\" java.lang.OutOfMemoryError: Java heap space\r\nException in thread \"elasticsearch[logstash-w530-12453-4142][transport_client_worker][T#6]\" java.lang.OutOfMemoryError: Java heap space\r\nException in thread \"elasticsearch[logstash-w530-12453-4150][transport_client_worker][T#6]\" java.lang.OutOfMemoryError: Java heap space\r\nException in thread \"elasticsearch[logstash-w530-12453-4192][transport_client_worker][T#6]\" java.lang.OutOfMemoryError: Java heap space\r\n^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^CException in thread \"elasticsearch[logstash-w530-12453-4256][transport_client_worker][T#16]\" java.lang.OutOfMemoryError: Java heap space\r\nError: Your application used more memory than the safety cap of 500M.\r\nSpecify -J-Xmx####m to increase it (#### = cap size in MB).\r\nSpecify -w for full OutOfMemoryError stack trace\r\nException in thread \"elasticsearch[logstash-w530-12453-4164][transport_client_boss][T#1]\" java.lang.OutOfMemoryError: Java heap space\r\nException in thread \"elasticsearch[logstash-w530-12453-4210][transport_client_boss][T#1]{New I/O boss #233}\" java.lang.OutOfMemoryError: Java heap space\r\n\r\nin logstash log with debug\r\n\r\nst\"=>\"w530\", \"path\"=>\"/tmp/hugefile10\"}], [\"index\", {:_id=>nil, :_index=>\"test\", :_type=>\"logs\"}, {\"message\"=>\"[628396.122783] [26976]     0 26976     3819       78      12        0             0 upstart-file-br\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-02-11T15:05:00.788Z\", \"host\"=>\"w530\", \"path\"=>\"/tmp/hugefile10\"}], [\"index\", {:_id=>nil, :_index=>\"test\", :_type=>\"logs\"}, {\"message\"=>\"[628396.122787] [ 1430]     0  1430    68174      587     130        0             0 smbd\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-02-11T15:05:00.789Z\", \"host\"=>\"w530\", \"path\"=>\"/tmp/hugefile10\"}], [\"index\", {:_id=>nil, :_index=>\"test\", :_type=>\"logs\"}, {\"message\"=>\"[628396.388282] [12603]  1000 12603    91470     2489     145        0             0 chrome\", \"@version\"=>\"1\", \"@timestamp\"=>\"2015-02-11T15:05:23.761Z\", \"host\"=>\"w530\", \"path\"=>\"/tmp/hugefile\"}]]}, :batch_timeout=>1, :force=>true, :final=>nil, :level=>:debug, :file=>\"stud/buffer.rb\", :line=>\"207\"}log4j, [2015-02-11T16:06:50.793] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4190] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:50.795] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4142] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:50.795] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4072] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:50.795] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4232] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:50.796] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4150] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:50.797] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4154] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:52.767] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4214] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:52.768] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4164] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:53.411] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4252] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:53.412] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4020] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:06:53.412] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4148] connected to node [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\n\r\nlog4j, [2015-02-11T16:07:24.213]  WARN: org.elasticsearch.netty.channel.socket.nio.AbstractNioSelector: Unexpected exception in the selector loop.\r\nlog4j, [2015-02-11T16:07:30.916]  WARN: org.elasticsearch.netty.channel.socket.nio.AbstractNioSelector: Unexpected exception in the selector loop.\r\njava.lang.OutOfMemoryError: Java heap space\r\nlog4j, [2015-02-11T16:07:32.746]  INFO: org.elasticsearch.client.transport: [logstash-w530-12453-4020] failed to get node info for [#transport#-1][w530][inet[/192.168.0.101:9302]], disconnecting...\r\norg.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse]\r\nCaused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse]\r\n    at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:148)\r\n    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)\r\n    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\r\n    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)\r\n    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)\r\n    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)\r\n    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)\r\n    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\r\n    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)\r\n    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\r\n    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\r\n    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\r\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n    at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.OutOfMemoryError: Java heap space\r\nlog4j, [2015-02-11T16:07:33.680]  WARN: org.elasticsearch.transport.netty: [logstash-w530-12453-4154] Message not fully read (response) for [24] handler future(org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler$1@2371fdd8), error [false], resetting\r\nlog4j, [2015-02-11T16:07:34.174] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4020] disconnected from [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:07:32.746]  WARN: org.elasticsearch.transport.netty: [logstash-w530-12453-4020] Message not fully read (response) for [22] handler future(org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler$1@542e137), error [false], resetting\r\nlog4j, [2015-02-11T16:07:35.131]  INFO: org.elasticsearch.client.transport: [logstash-w530-12453-4154] failed to get node info for [#transport#-1][w530][inet[/192.168.0.101:9302]], disconnecting...\r\norg.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse]\r\nCaused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse]\r\n    at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:148)\r\n    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)\r\n    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\r\n    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)\r\n    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)\r\n    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)\r\n    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)\r\n    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\r\n    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)\r\n    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\r\n    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\r\n    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\r\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n    at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.OutOfMemoryError: Java heap space\r\nlog4j, [2015-02-11T16:07:35.132] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4154] disconnected from [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:07:35.134]  WARN: org.elasticsearch.netty.channel.socket.nio.AbstractNioSelector: Unexpected exception in the selector loop.\r\njava.lang.OutOfMemoryError: Java heap space\r\nlog4j, [2015-02-11T16:07:35.136]  WARN: org.elasticsearch.transport.netty: [logstash-w530-12453-4150] exception caught on transport layer [[id: 0xf821dd6c, /192.168.0.101:56764 => /192.168.0.101:9302]], closing connection\r\njava.lang.OutOfMemoryError: Java heap space\r\nlog4j, [2015-02-11T16:07:35.137] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4150] disconnected from [[#transport#-1][w530][inet[/192.168.0.101:9302]]], channel closed event\r\nlog4j, [2015-02-11T16:07:35.138]  WARN: org.elasticsearch.transport.netty: [logstash-w530-12453-4192] exception caught on transport layer [[id: 0x47c4bf9d, /192.168.0.101:56741 => /192.168.0.101:9302]], closing connection\r\njava.lang.OutOfMemoryError: Java heap space\r\nlog4j, [2015-02-11T16:07:35.138] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4192] disconnected from [[#transport#-1][w530][inet[/192.168.0.101:9302]]], channel closed event\r\n\r\nlog4j, [2015-02-11T16:07:35.621]  INFO: org.elasticsearch.client.transport: [logstash-w530-12453-4256] failed to get node info for [#transport#-1][w530][inet[/192.168.0.101:9302]], disconnecting...\r\norg.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.0.101:9302]][cluster/nodes/info] request_id [23] timed out after [34854ms]\r\n    at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)\r\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n    at java.lang.Thread.run(Thread.java:745)\r\nlog4j, [2015-02-11T16:07:35.622] DEBUG: org.elasticsearch.transport.netty: [logstash-w530-12453-4256] disconnected from [[#transport#-1][w530][inet[/192.168.0.101:9302]]]\r\nlog4j, [2015-02-11T16:07:35.622]  INFO: org.elasticsearch.client.transport: [logstash-w530-12453-4150] failed to get node info for [#transport#-1][w530][inet[/192.168.0.101:9302]], disconnecting...\r\norg.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/192.168.0.101:9302]][cluster/nodes/info] request_id [24] timed out after [34450ms]\r\n    at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)\r\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n    at java.lang.Thread.run(Thread.java:745)\r\nlog4j, [2015-02-11T16:07:35.623]  WARN: org.elasticsearch.transport.netty: [logstash-w530-12453-4150] exception caught on transport layer [[id: 0xf821dd6c, /192.168.0.101:56764 :> /192.168.0.101:9302]], closing connection\r\njava.io.StreamCorruptedException: invalid internal transport message format\r\n    at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:46)\r\n    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)\r\n    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)\r\n    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\r\n    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\r\n    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)\r\n    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\r\n    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\r\n    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\r\n    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\r\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n    at java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\n\r\nas far as I understand this shouldn't happen?"}}