{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/3976","labels_url":"https://api.github.com/repos/apache/spark/issues/3976/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/3976/comments","events_url":"https://api.github.com/repos/apache/spark/issues/3976/events","html_url":"https://github.com/apache/spark/pull/3976","id":53866435,"number":3976,"title":"[SPARK-5173]support python application running on yarn cluster mode","user":{"login":"lianhuiwang","id":545478,"avatar_url":"https://avatars.githubusercontent.com/u/545478?v=3","gravatar_id":"","url":"https://api.github.com/users/lianhuiwang","html_url":"https://github.com/lianhuiwang","followers_url":"https://api.github.com/users/lianhuiwang/followers","following_url":"https://api.github.com/users/lianhuiwang/following{/other_user}","gists_url":"https://api.github.com/users/lianhuiwang/gists{/gist_id}","starred_url":"https://api.github.com/users/lianhuiwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lianhuiwang/subscriptions","organizations_url":"https://api.github.com/users/lianhuiwang/orgs","repos_url":"https://api.github.com/users/lianhuiwang/repos","events_url":"https://api.github.com/users/lianhuiwang/events{/privacy}","received_events_url":"https://api.github.com/users/lianhuiwang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":19,"created_at":"2015-01-09T13:14:17Z","updated_at":"2015-01-31T03:33:46Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3976","html_url":"https://github.com/apache/spark/pull/3976","diff_url":"https://github.com/apache/spark/pull/3976.diff","patch_url":"https://github.com/apache/spark/pull/3976.patch"},"body":"now when we run python application on yarn cluster mode through spark-submit, spark-submit does not support python application on yarn cluster mode. so i modify code of submit and yarn's AM in order to support it.\r\nthrough specifying .py file or primaryResource file via spark-submit, we can make pyspark run in yarn-cluster mode.\r\nexample:spark-submit --master yarn-master --num-executors 1 --driver-memory 1g --executor-memory 1g  xx.py --primaryResource yy.conf\r\nthis config is same as pyspark on yarn-client mode.\r\nfirstly,we put local path of .py or primaryResource to yarn's dist.files.that can be distributed on slave nodes.and then in spark-submit we transfer --py-files and --primaryResource to yarn.Client and use \"org.apache.spark.deploy.PythonRunner\" to user class that can run .py files on ApplicationMaster.\r\nin yarn.Client we transfer --py-files and --primaryResource to  ApplicationMaster.\r\nin ApplicationMaster, user's class is org.apache.spark.deploy.PythonRunner, and user's args is primaryResource and -py-files. so that can make pyspark run on ApplicationMaster.\r\n@JoshRosen @tgravescs @sryza"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/72302648","html_url":"https://github.com/apache/spark/pull/3976#issuecomment-72302648","issue_url":"https://api.github.com/repos/apache/spark/issues/3976","id":72302648,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-01-31T03:33:46Z","updated_at":"2015-01-31T03:33:46Z","body":"  [Test build #26452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26452/consoleFull) for   PR 3976 at commit [`905a106`](https://github.com/apache/spark/commit/905a10610532578c774e58d12b927597330fb9ff).\n * This patch **fails Scala style tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}