{"action":"created","issue":{"url":"https://api.github.com/repos/chef/chef/issues/3169","labels_url":"https://api.github.com/repos/chef/chef/issues/3169/labels{/name}","comments_url":"https://api.github.com/repos/chef/chef/issues/3169/comments","events_url":"https://api.github.com/repos/chef/chef/issues/3169/events","html_url":"https://github.com/chef/chef/pull/3169","id":65316090,"number":3169,"title":"Mitigating \"No route to host\" errors produced by 'knife ssh'","user":{"login":"dilchenko","id":412040,"avatar_url":"https://avatars.githubusercontent.com/u/412040?v=3","gravatar_id":"","url":"https://api.github.com/users/dilchenko","html_url":"https://github.com/dilchenko","followers_url":"https://api.github.com/users/dilchenko/followers","following_url":"https://api.github.com/users/dilchenko/following{/other_user}","gists_url":"https://api.github.com/users/dilchenko/gists{/gist_id}","starred_url":"https://api.github.com/users/dilchenko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilchenko/subscriptions","organizations_url":"https://api.github.com/users/dilchenko/orgs","repos_url":"https://api.github.com/users/dilchenko/repos","events_url":"https://api.github.com/users/dilchenko/events{/privacy}","received_events_url":"https://api.github.com/users/dilchenko/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-30T22:19:04Z","updated_at":"2015-03-30T22:19:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/chef/chef/pulls/3169","html_url":"https://github.com/chef/chef/pull/3169","diff_url":"https://github.com/chef/chef/pull/3169.diff","patch_url":"https://github.com/chef/chef/pull/3169.patch"},"body":"At least on OSX Yosemite (although i have observed this same issue on older OSX versions), 'knife ssh' produces errors like 'No route to host' when SSH-ing to 100-200 hosts at once. That error results in nodes being skipped, while those nodes are definitely up and accessible.  I've traced this to TCPSocket's internal connect(2) failing for some weird reason, and to the fact that Net::SSH::Transport::Session initializer wraps TCPSocket.open() call into a ruby Timeout() which by default is 0 seconds (the latter contributes to the problem, is not it's source). Limiting session concurrency definitely reduces the scope of the problem( # of skipped nodes per run ), but doesn't solve it completely.\r\n\r\nProposed changes introduce 2 options for 'knife ssh' which allow to customize timeout and retry ssh connection to a node for N times. Latter requires implementation of new :on_error strategy, which in turn requires changes to already existing net-ssh-multi monkeypatch.\r\n\r\nThose two options, in conjunction with concurrency limiting, seem to band-air the problem - at least in my case, i was able to 'knife ssh' into all 254 nodes repeatedly with only one retry needed for 2-5 nodes per run.\r\n\r\nLastly, the changes do not change default behavior of 'knife ssh'."},"comment":{"url":"https://api.github.com/repos/chef/chef/issues/comments/87856652","html_url":"https://github.com/chef/chef/pull/3169#issuecomment-87856652","issue_url":"https://api.github.com/repos/chef/chef/issues/3169","id":87856652,"user":{"login":"chef-supermarket","id":7798939,"avatar_url":"https://avatars.githubusercontent.com/u/7798939?v=3","gravatar_id":"","url":"https://api.github.com/users/chef-supermarket","html_url":"https://github.com/chef-supermarket","followers_url":"https://api.github.com/users/chef-supermarket/followers","following_url":"https://api.github.com/users/chef-supermarket/following{/other_user}","gists_url":"https://api.github.com/users/chef-supermarket/gists{/gist_id}","starred_url":"https://api.github.com/users/chef-supermarket/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chef-supermarket/subscriptions","organizations_url":"https://api.github.com/users/chef-supermarket/orgs","repos_url":"https://api.github.com/users/chef-supermarket/repos","events_url":"https://api.github.com/users/chef-supermarket/events{/privacy}","received_events_url":"https://api.github.com/users/chef-supermarket/received_events","type":"User","site_admin":false},"created_at":"2015-03-30T22:19:06Z","updated_at":"2015-03-30T22:19:06Z","body":"Hi. I am an automated pull request bot named Curry. There are commits in this pull request whose authors are not yet authorized to contribute to Chef Software, Inc. projects or are using a non-GitHub verified email address. To become authorized to contribute, you will need to sign the Contributor License Agreement (CLA) as an individual or on behalf of your company. [You can read more on Chef's blog.](https://www.chef.io/blog/2014/06/23/changes-to-the-contributor-license-agreement-process)\n\n## Non-GitHub Verified Committers\n\nThere are 1 commit author(s) whose commits are authored by a non-GitHub verified email address. Chef will have to manually verify that they are authorized to contribute.\n\n[Please sign the CLA here.](https://supermarket.chef.io/become-a-contributor)"}}