{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8533","labels_url":"https://api.github.com/repos/apache/spark/issues/8533/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8533/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8533/events","html_url":"https://github.com/apache/spark/pull/8533","id":104000857,"number":8533,"title":"[SPARK-10149] [CORE] [WIP] Locality Level is always 'ANY' and performance is impacted heavily while reading from DFS file systems","user":{"login":"wulei-bj-cn","id":6056191,"avatar_url":"https://avatars.githubusercontent.com/u/6056191?v=3","gravatar_id":"","url":"https://api.github.com/users/wulei-bj-cn","html_url":"https://github.com/wulei-bj-cn","followers_url":"https://api.github.com/users/wulei-bj-cn/followers","following_url":"https://api.github.com/users/wulei-bj-cn/following{/other_user}","gists_url":"https://api.github.com/users/wulei-bj-cn/gists{/gist_id}","starred_url":"https://api.github.com/users/wulei-bj-cn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wulei-bj-cn/subscriptions","organizations_url":"https://api.github.com/users/wulei-bj-cn/orgs","repos_url":"https://api.github.com/users/wulei-bj-cn/repos","events_url":"https://api.github.com/users/wulei-bj-cn/events{/privacy}","received_events_url":"https://api.github.com/users/wulei-bj-cn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":22,"created_at":"2015-08-31T07:21:47Z","updated_at":"2015-10-08T12:41:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8533","html_url":"https://github.com/apache/spark/pull/8533","diff_url":"https://github.com/apache/spark/pull/8533.diff","patch_url":"https://github.com/apache/spark/pull/8533.patch"},"body":"Corresponding JIRA issue: https://issues.apache.org/jira/browse/SPARK-10149\r\n\r\nStarting from 1.4.1, locality level is always 'ANY' in HadoopRDD while reading from DFS file systems, which should have been 'NODE_LOCAL' for sure. This always being 'ANY' introduces lots of unnecessary network I/O where hadoop file blocks are transferred among nodes, which is a huge impact on overall performance. \r\n\r\nThe main cause of this is : \r\nFunction localHostName() in Utils.scala is trying to fetch the hostname for each of the hosts, yet when \"SPARK_LOCAL_HOSTNAME\" is not set, i.e. customHostname is null, this function will try to fetch the IP addresses for the hosts. That's because localIpAddress.getHostAddress is called, which will fetch the IP addresses in case customHostname is null. However, the returned IP addresses (1.2.3.4) will not match the hostnames (host1) that are fetched from DFS file systems. Hence locality level will always be 'ANY' and lots of network I/O is introduced when input files are read from DFS file systems. \r\n\r\nAnd solution for this issue:\r\nTherefore, to make function return real hostnames when \"SPARK_LOCAL_HOSTNAME\" is not set, localIpAddress.getHostAddress is replaced by localIpAddress.getHostName, which will return a real hostname.\r\n\r\nTesting of this patch:\r\nLocality level returns to normal, i.e. 'NODE_LOCAL' , and network I/O that is introduced when locality level is 'ANY' is totally eliminated. This patch is tested on both standalone and YARN modes."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/146526005","html_url":"https://github.com/apache/spark/pull/8533#issuecomment-146526005","issue_url":"https://api.github.com/repos/apache/spark/issues/8533","id":146526005,"user":{"login":"potto007","id":219596,"avatar_url":"https://avatars.githubusercontent.com/u/219596?v=3","gravatar_id":"","url":"https://api.github.com/users/potto007","html_url":"https://github.com/potto007","followers_url":"https://api.github.com/users/potto007/followers","following_url":"https://api.github.com/users/potto007/following{/other_user}","gists_url":"https://api.github.com/users/potto007/gists{/gist_id}","starred_url":"https://api.github.com/users/potto007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/potto007/subscriptions","organizations_url":"https://api.github.com/users/potto007/orgs","repos_url":"https://api.github.com/users/potto007/repos","events_url":"https://api.github.com/users/potto007/events{/privacy}","received_events_url":"https://api.github.com/users/potto007/received_events","type":"User","site_admin":false},"created_at":"2015-10-08T12:41:21Z","updated_at":"2015-10-08T12:41:21Z","body":"Can I suggest that, at the very least, SPARK_LOCAL_HOSTNAME become documented? I spent several hours trying to find why my latest Spark cluster was only choosing a locality of ANY. The existence of this PR was the only hint to me of the fix.\r\n\r\nI'll take a stab at submitting a PR for the docs."}}