{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7943","labels_url":"https://api.github.com/repos/apache/spark/issues/7943/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7943/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7943/events","html_url":"https://github.com/apache/spark/pull/7943","id":99066272,"number":7943,"title":"[SPARK-9439] [wip] [yarn] External shuffle service robust to NM restarts using leveldb","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":28,"created_at":"2015-08-04T21:29:04Z","updated_at":"2015-08-17T18:24:04Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7943","html_url":"https://github.com/apache/spark/pull/7943","diff_url":"https://github.com/apache/spark/pull/7943.diff","patch_url":"https://github.com/apache/spark/pull/7943.patch"},"body":"https://issues.apache.org/jira/browse/SPARK-9439\r\n\r\nIn general, Yarn apps should be robust to NodeManager restarts.  However, if you run spark with the external shuffle service on, after a NM restart all shuffles fail, b/c the shuffle service has lost some state with info on each executor.  (Note the shuffle data is perfectly fine on disk across a NM restart, the problem is we've lost the small bit of state that lets us *find* those files.)\r\n\r\nThe solution proposed here is that the external shuffle service can write out its state to leveldb (backed by a local file) every time an executor is added.  When running with yarn, that file is in the NM's local dir.  Whenever the service is started, it looks for that file, and if it exists, it reads the file and re-registers all executors there.\r\n\r\nNothing is changed in non-yarn modes with this patch.  The service is not given a place to save the state to, so it operates the same as before.  This should make it easy to update other cluster managers as well, by just supplying the right file & the equivalent of yarn's `initializeApplication` -- I'm not familiar enough with those modes to know how to do that."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/131917047","html_url":"https://github.com/apache/spark/pull/7943#issuecomment-131917047","issue_url":"https://api.github.com/repos/apache/spark/issues/7943","id":131917047,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-08-17T18:24:03Z","updated_at":"2015-08-17T18:24:03Z","body":"Merged build finished. Test FAILed."}}