{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/2854","labels_url":"https://api.github.com/repos/apache/spark/issues/2854/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/2854/comments","events_url":"https://api.github.com/repos/apache/spark/issues/2854/events","html_url":"https://github.com/apache/spark/pull/2854","id":46254026,"number":2854,"title":"[SPARK-4006] Block Manager - Double Register Crash","user":{"login":"tsliwowicz","id":771037,"avatar_url":"https://avatars.githubusercontent.com/u/771037?v=3","gravatar_id":"","url":"https://api.github.com/users/tsliwowicz","html_url":"https://github.com/tsliwowicz","followers_url":"https://api.github.com/users/tsliwowicz/followers","following_url":"https://api.github.com/users/tsliwowicz/following{/other_user}","gists_url":"https://api.github.com/users/tsliwowicz/gists{/gist_id}","starred_url":"https://api.github.com/users/tsliwowicz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsliwowicz/subscriptions","organizations_url":"https://api.github.com/users/tsliwowicz/orgs","repos_url":"https://api.github.com/users/tsliwowicz/repos","events_url":"https://api.github.com/users/tsliwowicz/events{/privacy}","received_events_url":"https://api.github.com/users/tsliwowicz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":85,"created_at":"2014-10-20T10:12:30Z","updated_at":"2015-01-07T23:53:25Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/2854","html_url":"https://github.com/apache/spark/pull/2854","diff_url":"https://github.com/apache/spark/pull/2854.diff","patch_url":"https://github.com/apache/spark/pull/2854.patch"},"body":"This issue affects all versions since 0.7 up to (including) 1.1\r\n\r\nIn long running contexts, we encountered the situation of double register without a remove in between. The cause for that is unknown, and assumed a temp network issue.\r\n\r\nHowever, since the second register is with a BlockManagerId on a different port, blockManagerInfo.contains() returns false, while blockManagerIdByExecutor returns Some. This inconsistency is caught in a conditional statement that does System.exit(1), which is a huge robustness issue for us.\r\n\r\nThe fix - simply remove the old id from both maps during register when this happens. We are mimicking the behavior of expireDeadHosts(), by doing local cleanup of the maps before trying to add new ones.\r\n    \r\nAlso - added some logging for register and unregister.\r\n\r\nhttps://issues.apache.org/jira/browse/SPARK-4006\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/69114255","html_url":"https://github.com/apache/spark/pull/2854#issuecomment-69114255","issue_url":"https://api.github.com/repos/apache/spark/issues/2854","id":69114255,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-01-07T23:53:25Z","updated_at":"2015-01-07T23:53:25Z","body":"  [Test build #25180 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25180/consoleFull) for   PR 2854 at commit [`95ae4db`](https://github.com/apache/spark/commit/95ae4dbe090f16e863614a2f4f2e0b0fb90ec1af).\n * This patch **fails some tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}