{"action":"created","issue":{"url":"https://api.github.com/repos/apache/storm/issues/568","labels_url":"https://api.github.com/repos/apache/storm/issues/568/labels{/name}","comments_url":"https://api.github.com/repos/apache/storm/issues/568/comments","events_url":"https://api.github.com/repos/apache/storm/issues/568/events","html_url":"https://github.com/apache/storm/pull/568","id":83044408,"number":568,"title":"[STORM-763] nimbus reassigned worker A to another machine, but other worker's netty client can't connect to the new worker A","user":{"login":"eshioji","id":1995924,"avatar_url":"https://avatars.githubusercontent.com/u/1995924?v=3","gravatar_id":"","url":"https://api.github.com/users/eshioji","html_url":"https://github.com/eshioji","followers_url":"https://api.github.com/users/eshioji/followers","following_url":"https://api.github.com/users/eshioji/following{/other_user}","gists_url":"https://api.github.com/users/eshioji/gists{/gist_id}","starred_url":"https://api.github.com/users/eshioji/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eshioji/subscriptions","organizations_url":"https://api.github.com/users/eshioji/orgs","repos_url":"https://api.github.com/users/eshioji/repos","events_url":"https://api.github.com/users/eshioji/events{/privacy}","received_events_url":"https://api.github.com/users/eshioji/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-05-31T09:02:16Z","updated_at":"2015-06-02T06:16:15Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/storm/pulls/568","html_url":"https://github.com/apache/storm/pull/568","diff_url":"https://github.com/apache/storm/pull/568.diff","patch_url":"https://github.com/apache/storm/pull/568.patch"},"body":"(This PR builds on [STORM-839](https://github.com/apache/storm/pull/566), so it includes its changes)\r\n\r\nWhile I'm not 100% certain if it's the same issue, I also saw these messages logged a lot (hundreds per second per machine).\r\n\r\nUpon investigation, I found that after firing a connection request, `Client.Connector` waits synchronously for the connection. Because there is only one scheduler thread per worker by default, this means only one connection request per worker process can be serviced at a time. This becomes a problem when connections are lost frequently for whatever reason, and is especially exacerbated when each connection attempt takes time.\r\n\r\nCommit `aa5c2d71` changes the code so that the `Connector` fires & forgets the connection request, and lets Netty's callback thread handle the rest. There is no blocking or expensive operation done here, so using Netty's thread should be fine.\r\n\r\nThis change did decrease the amount of ERROR logs in my topology, but I still saw a fair amount. Upon investigation, I found that this is simply because each call to `send` produces an ERROR log. So `ad8112d10` changes the code so that a connection error is exactly logged twice; once when the connection error is detected, and the second time when the re-connection is concluded (either a successful re-connect, or permanent failure).\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/storm/issues/comments/107822705","html_url":"https://github.com/apache/storm/pull/568#issuecomment-107822705","issue_url":"https://api.github.com/repos/apache/storm/issues/568","id":107822705,"user":{"login":"eshioji","id":1995924,"avatar_url":"https://avatars.githubusercontent.com/u/1995924?v=3","gravatar_id":"","url":"https://api.github.com/users/eshioji","html_url":"https://github.com/eshioji","followers_url":"https://api.github.com/users/eshioji/followers","following_url":"https://api.github.com/users/eshioji/following{/other_user}","gists_url":"https://api.github.com/users/eshioji/gists{/gist_id}","starred_url":"https://api.github.com/users/eshioji/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eshioji/subscriptions","organizations_url":"https://api.github.com/users/eshioji/orgs","repos_url":"https://api.github.com/users/eshioji/repos","events_url":"https://api.github.com/users/eshioji/events{/privacy}","received_events_url":"https://api.github.com/users/eshioji/received_events","type":"User","site_admin":false},"created_at":"2015-06-02T06:16:14Z","updated_at":"2015-06-02T06:16:14Z","body":"@revans2 Sure, take your time and let me know if you find something to change!"}}