{"action":"closed","issue":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1642","labels_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1642/labels{/name}","comments_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1642/comments","events_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1642/events","html_url":"https://github.com/Graylog2/graylog2-server/issues/1642","id":123084399,"number":1642,"title":"Stream invalidation should be disabled by an option","user":{"login":"pdepaepe","id":11463691,"avatar_url":"https://avatars.githubusercontent.com/u/11463691?v=3","gravatar_id":"","url":"https://api.github.com/users/pdepaepe","html_url":"https://github.com/pdepaepe","followers_url":"https://api.github.com/users/pdepaepe/followers","following_url":"https://api.github.com/users/pdepaepe/following{/other_user}","gists_url":"https://api.github.com/users/pdepaepe/gists{/gist_id}","starred_url":"https://api.github.com/users/pdepaepe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdepaepe/subscriptions","organizations_url":"https://api.github.com/users/pdepaepe/orgs","repos_url":"https://api.github.com/users/pdepaepe/repos","events_url":"https://api.github.com/users/pdepaepe/events{/privacy}","received_events_url":"https://api.github.com/users/pdepaepe/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-19T13:01:44Z","updated_at":"2015-12-20T13:18:43Z","closed_at":"2015-12-20T13:18:43Z","body":"Hello,\r\n\r\nToday, graylog have a system to protect itself when rules set on stream are too expensive to process.\r\n\r\nWhen it occurs, it raise an alert like:\r\n\r\n````\r\nProcessing of a stream has been disabled due to excessive processing time. (triggered 8 hours ago)\r\nThe processing of stream xx has taken too long for 3 times. To protect the stability of message processing, this stream has been disabled. Please correct the stream rules and reenable the stream.\r\n````\r\n\r\nOn my infrastructure, i have a dozen graylog server nodes consumming kafka topics. One of them (of course in the middle of the night) get sick (hardrive corrupted). Handling process became a problem, and it starts to invalidate all stream because of timeout. This action penalize all cluster, other nodes were unable to route logs to appropriate streams as they were disabled by seek node.\r\n\r\nIt's probably a kind of race condition, but i think the stream invalidation as it, is very dangerous, and should be disabled by an option.\r\n\r\nThanks"}}