{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/3779","labels_url":"https://api.github.com/repos/apache/spark/issues/3779/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/3779/comments","events_url":"https://api.github.com/repos/apache/spark/issues/3779/events","html_url":"https://github.com/apache/spark/pull/3779","id":52770606,"number":3779,"title":"[SPARK-4939] move to next locality when no pending tasks","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":55,"created_at":"2014-12-23T20:04:14Z","updated_at":"2015-02-04T08:22:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3779","html_url":"https://github.com/apache/spark/pull/3779","diff_url":"https://github.com/apache/spark/pull/3779.diff","patch_url":"https://github.com/apache/spark/pull/3779.patch"},"body":"Currently, if there are different locality in a task set, the tasks with NODE_LOCAL only get scheduled after all the PROCESS_LOCAL tasks are scheduled and timeout with spark.locality.wait.process (3 seconds by default). In local mode, the LocalScheduler will never call resourceOffer() again once it failed to get a task with same locality, then all the NODE_LOCAL tasks will be never scheduled.\r\n\r\nThis bug could be reproduced by run example python/streaming/stateful_network_wordcount.py, it will hang after finished a batch with some data.\r\n\r\nThis patch will check whether there is task for current locality level, if not, it will change to next locality level without waiting for `spark.locality.wait.process` seconds. It works for all locality levels.\r\n\r\nBecause the list of pending tasks are updated lazily, the check can be false-positive, it means it will not move to next locality level even there is no valid pending tasks, it will wait for timeout."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/72808101","html_url":"https://github.com/apache/spark/pull/3779#issuecomment-72808101","issue_url":"https://api.github.com/repos/apache/spark/issues/3779","id":72808101,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-02-04T08:22:21Z","updated_at":"2015-02-04T08:22:21Z","body":"  [Test build #26732 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26732/consoleFull) for   PR 3779 at commit [`1550668`](https://github.com/apache/spark/commit/1550668b047f8702a6c7b19aad56cfd7a56e47c3).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}