{"push_id":563011029,"size":3,"distinct_size":3,"ref":"refs/heads/master","head":"62a93a1698e8d0a667eb1718ba75dcfc86eabaaf","before":"b2047b55c5fc85de6b63276d8ab9610d2496e08b","commits":[{"sha":"f5e63751f0ed50ceafdc2ec5173b161a5155b646","author":{"email":"0da04896db04dd2393029a4b0c77df7858a20dea@gmail.com","name":"lianhuiwang"},"message":"[SPARK-5173]support python application running on yarn cluster mode\n\nnow when we run python application on yarn cluster mode through spark-submit, spark-submit does not support python application on yarn cluster mode. so i modify code of submit and yarn's AM in order to support it.\nthrough specifying .py file or primaryResource file via spark-submit, we can make pyspark run in yarn-cluster mode.\nexample:spark-submit --master yarn-master --num-executors 1 --driver-memory 1g --executor-memory 1g  xx.py --primaryResource yy.conf\nthis config is same as pyspark on yarn-client mode.\nfirstly,we put local path of .py or primaryResource to yarn's dist.files.that can be distributed on slave nodes.and then in spark-submit we transfer --py-files and --primaryResource to yarn.Client and use \"org.apache.spark.deploy.PythonRunner\" to user class that can run .py files on ApplicationMaster.\nin yarn.Client we transfer --py-files and --primaryResource to  ApplicationMaster.\nin ApplicationMaster, user's class is org.apache.spark.deploy.PythonRunner, and user's args is primaryResource and -py-files. so that can make pyspark run on ApplicationMaster.\nJoshRosen tgravescs sryza\n\nAuthor: lianhuiwang <lianhuiwang09@gmail.com>\nAuthor: Wang Lianhui <lianhuiwang09@gmail.com>\n\nCloses #3976 from lianhuiwang/SPARK-5173 and squashes the following commits:\n\n28a8a58 [lianhuiwang] fix variable name\n67f8cee [lianhuiwang] update with andrewor's comments\n0319ae3 [lianhuiwang] address with sryza's comments\n2385ef6 [lianhuiwang] address with sryza's comments\n03640ab [lianhuiwang] add sparkHome to env\n47d2fc3 [lianhuiwang] fix test\n2adc8f5 [lianhuiwang] add spark.test.home\nd60bc60 [lianhuiwang] fix test\n5b30064 [lianhuiwang] add test\n097a5ec [lianhuiwang] fix line length exceeds 100\n905a106 [lianhuiwang] update with sryza and andrewor 's comments\nf1f55b6 [lianhuiwang] when yarn-cluster, all python files can be non-local\n172eec1 [Wang Lianhui] fix a min submit's bug\n9c941bc [lianhuiwang] support python application running on yarn cluster mode","distinct":true,"url":"https://api.github.com/repos/Guavus/spark/commits/f5e63751f0ed50ceafdc2ec5173b161a5155b646"},{"sha":"3f941b68a2336aa7876aeda99865e7c19b53bc5c","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[Docs] Fix Building Spark link text\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #4312 from nchammas/patch-2 and squashes the following commits:\n\n9d943aa [Nicholas Chammas] [Docs] Fix Building Spark link text","distinct":true,"url":"https://api.github.com/repos/Guavus/spark/commits/3f941b68a2336aa7876aeda99865e7c19b53bc5c"},{"sha":"62a93a1698e8d0a667eb1718ba75dcfc86eabaaf","author":{"email":"fa846ab798b1c8a3d782c63179e43468eaf42cbd@qq.com","name":"Xutingjun"},"message":"[SPARK-5530] Add executor container to executorIdToContainer\n\nwhen call killExecutor method, it will only go to the else branch, because  the variable executorIdToContainer never be put any value.\n\nAuthor: Xutingjun <1039320815@qq.com>\n\nCloses #4309 from XuTingjun/dynamicAllocator and squashes the following commits:\n\nc823418 [Xutingjun] fix bugwq","distinct":true,"url":"https://api.github.com/repos/Guavus/spark/commits/62a93a1698e8d0a667eb1718ba75dcfc86eabaaf"}]}