{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6360","labels_url":"https://api.github.com/repos/apache/spark/issues/6360/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6360/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6360/events","html_url":"https://github.com/apache/spark/pull/6360","id":79565157,"number":6360,"title":"[SPARK-5479] [yarn] Handle --py-files correctly in YARN.","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":81,"created_at":"2015-05-22T18:44:09Z","updated_at":"2015-06-10T18:05:43Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6360","html_url":"https://github.com/apache/spark/pull/6360","diff_url":"https://github.com/apache/spark/pull/6360.diff","patch_url":"https://github.com/apache/spark/pull/6360.patch"},"body":"The bug description is a little misleading: the actual issue is that\r\n.py files are not handled correctly when distributed by YARN. They're\r\nadded to \"spark.submit.pyFiles\", which, when processed by context.py,\r\nexplicitly whitelists certain extensions (see PACKAGE_EXTENSIONS),\r\nand that does not include .py files.\r\n\r\nOn top of that, archives were not handled at all! They made it to the\r\ndriver's python path, but never made it to executors, since the mechanism\r\nused to propagate their location (spark.submit.pyFiles) only works on\r\nthe driver side.\r\n\r\nSo, instead, ignore \"spark.submit.pyFiles\" and just build PYTHONPATH\r\ncorrectly for both driver and executors. Individual .py files are\r\nplaced in a subdirectory of the container's local dir in the cluster,\r\nwhich is then added to the python path. Archives are added directly.\r\n\r\nThe change, as a side effect, ends up solving the symptom described\r\nin the bug. The issue was not that the files were not being distributed,\r\nbut that they were never made visible to the python application\r\nrunning under Spark.\r\n\r\nAlso included is a proper unit test for running python on YARN, which\r\nbroke in several different ways with the previous code.\r\n\r\nA short walk around of the changes:\r\n- SparkSubmit does not try to be smart about how YARN handles python\r\n  files anymore. It just passes down the configs to the YARN client\r\n  code.\r\n- The YARN client distributes python files and archives differently,\r\n  placing the files in a subdirectory.\r\n- The YARN client now sets PYTHONPATH for the processes it launches;\r\n  to properly handle different locations, it uses YARN's support for\r\n  embedding env variables, so to avoid YARN expanding those at the\r\n  wrong time, SparkConf is now propagated to the AM using a conf file\r\n  instead of command line options.\r\n- Because the Client initialization code is a maze of implicit\r\n  dependencies, some code needed to be moved around to make sure\r\n  all needed state was available when the code ran.\r\n- The pyspark tests in YarnClusterSuite now actually distribute and try\r\n  to use both a python file and an archive containing a different python\r\n  module. Also added a yarn-client tests for completeness.\r\n- I cleaned up some of the code around distributing files to YARN, to\r\n  avoid adding more copied & pasted code to handle the new files being\r\n  distributed."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/110858245","html_url":"https://github.com/apache/spark/pull/6360#issuecomment-110858245","issue_url":"https://api.github.com/repos/apache/spark/issues/6360","id":110858245,"user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"created_at":"2015-06-10T18:05:43Z","updated_at":"2015-06-10T18:05:43Z","body":"Ok, I tested this on a Hadoop 2.4 cluster and verified that both client and cluster modes work with `--py-files`. I'll merge this once you address the last round of comments."}}