{"action":"created","issue":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/5209","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/5209/labels{/name}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/5209/comments","events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/5209/events","html_url":"https://github.com/hazelcast/hazelcast/issues/5209","id":71985681,"number":5209,"title":"IO Thread getting into state where it is never ready to write","user":{"login":"youngm","id":438661,"avatar_url":"https://avatars.githubusercontent.com/u/438661?v=3","gravatar_id":"","url":"https://api.github.com/users/youngm","html_url":"https://github.com/youngm","followers_url":"https://api.github.com/users/youngm/followers","following_url":"https://api.github.com/users/youngm/following{/other_user}","gists_url":"https://api.github.com/users/youngm/gists{/gist_id}","starred_url":"https://api.github.com/users/youngm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/youngm/subscriptions","organizations_url":"https://api.github.com/users/youngm/orgs","repos_url":"https://api.github.com/users/youngm/repos","events_url":"https://api.github.com/users/youngm/events{/privacy}","received_events_url":"https://api.github.com/users/youngm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/PENDING","name":"PENDING","color":"bfe5bf"},{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Team:%20Core","name":"Team: Core","color":"eb6420"},{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Team:%20QuSP","name":"Team: QuSP","color":"009800"},{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Type:%20Critical","name":"Type: Critical","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-04-29T20:45:42Z","updated_at":"2015-05-12T16:03:05Z","closed_at":null,"body":"Hazelcast 3.4.2\r\n\r\nThis issue is the result of my analysis of this Mailing list Thread: https://groups.google.com/d/msg/hazelcast/ksUBy4bYcSc/Qz1HDWjZb90J\r\n\r\nWhen running stress tests on my 10 node cluster I've found that occasionally the WriteHandler for a IO Thread can get into a state where it is never ready because the channel isn't writing anything.\r\n\r\nThe state is that WriteHandler.ready == false and everytime AbstractIOSelector:124 is invoked:\r\n```\r\nselectedKeyCount = selector.select(waitTime);\r\n```\r\nThe value of selectedKeyCount is ALWAYS 0.\r\n\r\nThe result is that this WriteHandler never responds to any requests from a certain node (the node associated with the handler).\r\n\r\nIt is difficult to get into this state so I cannot provide a simple test case duplicating the issue.  But I do have a theory of how this could happen.\r\n\r\nIn WriteHandler.handle():148 the code is not actually registering for a Write (line: 153) until after it has written to the buffer (line: 148).  Hence if it were possible for for the channel to process the buffer and flag it as done writing before registerWrite() is invoked then I think the WriteHandler would end up in the state I have described.  Never ready to write and the write Selector never saying it has finished writing anything.\r\n\r\nThoughts?  I can regularly duplicate this issue in my environment given an hour or 2 of load.\r\n\r\nMike"},"comment":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/comments/101331324","html_url":"https://github.com/hazelcast/hazelcast/issues/5209#issuecomment-101331324","issue_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/5209","id":101331324,"user":{"login":"jerrinot","id":158619,"avatar_url":"https://avatars.githubusercontent.com/u/158619?v=3","gravatar_id":"","url":"https://api.github.com/users/jerrinot","html_url":"https://github.com/jerrinot","followers_url":"https://api.github.com/users/jerrinot/followers","following_url":"https://api.github.com/users/jerrinot/following{/other_user}","gists_url":"https://api.github.com/users/jerrinot/gists{/gist_id}","starred_url":"https://api.github.com/users/jerrinot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerrinot/subscriptions","organizations_url":"https://api.github.com/users/jerrinot/orgs","repos_url":"https://api.github.com/users/jerrinot/repos","events_url":"https://api.github.com/users/jerrinot/events{/privacy}","received_events_url":"https://api.github.com/users/jerrinot/received_events","type":"User","site_admin":false},"created_at":"2015-05-12T16:03:05Z","updated_at":"2015-05-12T16:03:05Z","body":"great, ping me if you need any assistance! "}}