{"action":"created","issue":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1174","labels_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1174/labels{/name}","comments_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1174/comments","events_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1174/events","html_url":"https://github.com/Graylog2/graylog2-server/issues/1174","id":79447757,"number":1174,"title":"Processing Messages stopped after OOM on elasticsearch Cluster","user":{"login":"nycoon","id":11075216,"avatar_url":"https://avatars.githubusercontent.com/u/11075216?v=3","gravatar_id":"","url":"https://api.github.com/users/nycoon","html_url":"https://github.com/nycoon","followers_url":"https://api.github.com/users/nycoon/followers","following_url":"https://api.github.com/users/nycoon/following{/other_user}","gists_url":"https://api.github.com/users/nycoon/gists{/gist_id}","starred_url":"https://api.github.com/users/nycoon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nycoon/subscriptions","organizations_url":"https://api.github.com/users/nycoon/orgs","repos_url":"https://api.github.com/users/nycoon/repos","events_url":"https://api.github.com/users/nycoon/events{/privacy}","received_events_url":"https://api.github.com/users/nycoon/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-05-22T13:04:22Z","updated_at":"2015-06-18T09:54:28Z","closed_at":"2015-06-18T09:54:28Z","body":"Hey \r\nif one of the data nodes from elasticsearch runs in an OutOfMemory exception and the cluster status goes to yellow, graylog-server will store all new incoming messages to the local journal. (y)\r\nBut now if the nodes comes back and the cluster status comes back to green i´ve to restart the graylog-server nodes for \"processing\" the stored messages. \r\n\r\nUnavailableShardsException[[graylog2_12][2] [2] shardIt, [0] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@41aee44b]\r\nRemoteTransportException[[Kingo Sunen][inet[/x.x.x.x:9300]][bulk/shard]]; nested: OutOfMemoryError[Java heap space];\r\n\r\nelasticsearch Cluster setup: three master nodes and two data nodes.\r\n\r\nVersions: \r\ngraylog-server-1.0.2-1.noarch\r\ngraylog-web-1.0.2-1.noarch\r\nelasticsearch-1.4.5-1.noarch"},"comment":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/comments/113096163","html_url":"https://github.com/Graylog2/graylog2-server/issues/1174#issuecomment-113096163","issue_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1174","id":113096163,"user":{"login":"zexxon","id":1154318,"avatar_url":"https://avatars.githubusercontent.com/u/1154318?v=3","gravatar_id":"","url":"https://api.github.com/users/zexxon","html_url":"https://github.com/zexxon","followers_url":"https://api.github.com/users/zexxon/followers","following_url":"https://api.github.com/users/zexxon/following{/other_user}","gists_url":"https://api.github.com/users/zexxon/gists{/gist_id}","starred_url":"https://api.github.com/users/zexxon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zexxon/subscriptions","organizations_url":"https://api.github.com/users/zexxon/orgs","repos_url":"https://api.github.com/users/zexxon/repos","events_url":"https://api.github.com/users/zexxon/events{/privacy}","received_events_url":"https://api.github.com/users/zexxon/received_events","type":"User","site_admin":false},"created_at":"2015-06-18T09:54:28Z","updated_at":"2015-06-18T09:54:28Z","body":"Please upgrade to Graylog 1.1 and ES 1.6, we believe the issue may have been resolved in the newest versions of Graylog and ES. "}}