{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4262","labels_url":"https://api.github.com/repos/apache/spark/issues/4262/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4262/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4262/events","html_url":"https://github.com/apache/spark/pull/4262","id":55856344,"number":4262,"title":"[SPARK-5473] [EC2] Expose SSH failures after status checks pass","user":{"login":"nchammas","id":1039369,"avatar_url":"https://avatars.githubusercontent.com/u/1039369?v=3","gravatar_id":"","url":"https://api.github.com/users/nchammas","html_url":"https://github.com/nchammas","followers_url":"https://api.github.com/users/nchammas/followers","following_url":"https://api.github.com/users/nchammas/following{/other_user}","gists_url":"https://api.github.com/users/nchammas/gists{/gist_id}","starred_url":"https://api.github.com/users/nchammas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nchammas/subscriptions","organizations_url":"https://api.github.com/users/nchammas/orgs","repos_url":"https://api.github.com/users/nchammas/repos","events_url":"https://api.github.com/users/nchammas/events{/privacy}","received_events_url":"https://api.github.com/users/nchammas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-01-29T04:52:22Z","updated_at":"2015-01-31T00:41:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4262","html_url":"https://github.com/apache/spark/pull/4262","diff_url":"https://github.com/apache/spark/pull/4262.diff","patch_url":"https://github.com/apache/spark/pull/4262.patch"},"body":"If there is some fatal problem with launching a cluster, `spark-ec2` just hangs without giving the user useful feedback on what the problem is.\r\n\r\nThis PR exposes the output of the SSH calls to the user if the SSH test fails during cluster launch for any reason but the instance status checks are all green. It also removes the growing trail of dots while waiting in favor of a fixed 3 dots.\r\n\r\nFor example:\r\n\r\n```\r\n$ ./ec2/spark-ec2 -k key -i /incorrect/path/identity.pem --instance-type m3.medium --slaves 1 --zone us-east-1c launch \"spark-test\"\r\nSetting up security groups...\r\nSearching for existing cluster spark-test...\r\nSpark AMI: ami-35b1885c\r\nLaunching instances...\r\nLaunched 1 slaves in us-east-1c, regid = r-7dadd096\r\nLaunched master in us-east-1c, regid = r-fcadd017\r\nWaiting for cluster to enter 'ssh-ready' state...\r\nWarning: SSH connection error. (This could be temporary.)\r\nHost: 127.0.0.1\r\nSSH return code: 255\r\nSSH output: Warning: Identity file /incorrect/path/identity.pem not accessible: No such file or directory.\r\nWarning: Permanently added '127.0.0.1' (RSA) to the list of known hosts.\r\nPermission denied (publickey).\r\n```\r\n\r\nThis should give users enough information when some unrecoverable error occurs during launch so they can know to abort the launch. This will help avoid situations like the ones reported [here on Stack Overflow](http://stackoverflow.com/q/28002443/) and [here on the user list](http://mail-archives.apache.org/mod_mbox/spark-user/201501.mbox/%3C1422323829398-21381.post@n3.nabble.com%3E), where the users couldn't tell what the problem was because it was being hidden by `spark-ec2`.\r\n\r\nThis is a usability improvement that should be backported to 1.2.\r\n\r\nResolves [SPARK-5473](https://issues.apache.org/jira/browse/SPARK-5473)."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/72294957","html_url":"https://github.com/apache/spark/pull/4262#issuecomment-72294957","issue_url":"https://api.github.com/repos/apache/spark/issues/4262","id":72294957,"user":{"login":"nchammas","id":1039369,"avatar_url":"https://avatars.githubusercontent.com/u/1039369?v=3","gravatar_id":"","url":"https://api.github.com/users/nchammas","html_url":"https://github.com/nchammas","followers_url":"https://api.github.com/users/nchammas/followers","following_url":"https://api.github.com/users/nchammas/following{/other_user}","gists_url":"https://api.github.com/users/nchammas/gists{/gist_id}","starred_url":"https://api.github.com/users/nchammas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nchammas/subscriptions","organizations_url":"https://api.github.com/users/nchammas/orgs","repos_url":"https://api.github.com/users/nchammas/repos","events_url":"https://api.github.com/users/nchammas/events{/privacy}","received_events_url":"https://api.github.com/users/nchammas/received_events","type":"User","site_admin":false},"created_at":"2015-01-31T00:41:18Z","updated_at":"2015-01-31T00:41:18Z","body":"OK, I added the dots back in. It doesn't look great, but it's serviceable:\r\n\r\n```\r\n$ ./ec2/spark-ec2 -k key -i /good/path/pem --instance-type m3.medium --slaves 40 --zone us-east-1c launch \"spark-test\"\r\nSetting up security groups...\r\nSearching for existing cluster spark-test...\r\nSpark AMI: ami-35b1885c\r\nLaunching instances...\r\nLaunched 40 slaves in us-east-1c, regid = r-811e646a\r\nLaunched master in us-east-1c, regid = r-501c66bb\r\nWaiting for cluster to enter 'ssh-ready' state...........\r\n\r\nWarning: SSH connection error. (This could be temporary.)\r\nHost: 54.54.54.54\r\nSSH return code: 255\r\nSSH output: ssh: connect to host 54.54.54.54 port 22: Connection refused\r\n\r\n.\r\n\r\nWarning: SSH connection error. (This could be temporary.)\r\nHost: 54.54.54.54\r\nSSH return code: 255\r\nSSH output: ssh: connect to host 54.54.54.54 port 22: Connection refused\r\n\r\n.\r\nCluster is now in 'ssh-ready' state. Waited 517 seconds.\r\nGenerating cluster's SSH key on master...\r\n```\r\n\r\nThat's what we get in the case of a temporary failure while SSH is still coming up.\r\n\r\nIn the case of a permanent failure, you just get the repeated `.` on a separate line followed by the 4-line error report:\r\n\r\n```\r\n$ ./ec2/spark-ec2 -k key -i /wrong/path/pem --instance-type m3.medium --slaves 1 --zone us-east-1c launch \"spark-test\"\r\nSetting up security groups...\r\nSearching for existing cluster spark-test...\r\nSpark AMI: ami-35b1885c\r\nLaunching instances...\r\nLaunched 1 slaves in us-east-1c, regid = r-283248c3\r\nLaunched master in us-east-1c, regid = r-2b3248c0\r\nWaiting for cluster to enter 'ssh-ready' state..............\r\n\r\nWarning: SSH connection error. (This could be temporary.)\r\nHost: 54.54.54.54\r\nSSH return code: 255\r\nSSH output: Warning: Identity file /wrong/path/pem not accessible: No such file or directory.\r\nWarning: Permanently added '54.54.54.54' (RSA) to the list of known hosts.\r\nPermission denied (publickey).\r\n\r\n.\r\n\r\nWarning: SSH connection error. (This could be temporary.)\r\nHost: 54.54.54.54\r\nSSH return code: 255\r\nSSH output: Warning: Identity file /wrong/path/pem not accessible: No such file or directory.\r\nPermission denied (publickey).\r\n\r\n.\r\n\r\n...\r\n```\r\n\r\nWhat do you think?"}}