{"push_id":593269535,"size":90,"distinct_size":90,"ref":"refs/heads/master","head":"e01656978174f8ecbd75ef6a50211234a1babfc6","before":"b93aaf682572890c49a58da149612c0053afc3de","commits":[{"sha":"7fa960e653a905fc48d4097b49ce560cff919fa2","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-5363] Fix bug in PythonRDD: remove() inside iterator is not safe\n\nRemoving elements from a mutable HashSet while iterating over it can cause the\niteration to incorrectly skip over entries that were not removed. If this\nhappened, PythonRDD would write fewer broadcast variables than the Python\nworker was expecting to read, which would cause the Python worker to hang\nindefinitely.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #4776 from davies/fix_hang and squashes the following commits:\n\na4384a5 [Davies Liu] fix bug: remvoe() inside iterator is not safe","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/7fa960e653a905fc48d4097b49ce560cff919fa2"},{"sha":"cd5c8d7bbd3ea410df08af6cdd3833e0ed4b91a0","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-4704 [CORE] SparkSubmitDriverBootstrap doesn't flush output\n\nJoin on output threads to make sure any lingering output from process reaches stdout, stderr before exiting\n\nCC andrewor14 since I believe he created this section of code\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #4788 from srowen/SPARK-4704 and squashes the following commits:\n\nad7114e [Sean Owen] Join on output threads to make sure any lingering output from process reaches stdout, stderr before exiting","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/cd5c8d7bbd3ea410df08af6cdd3833e0ed4b91a0"},{"sha":"10094a523e3993b775111ae9b22ca31cc0d76e03","author":{"email":"ae50d9f03dc526663c62caab7e3ac5b370caf973@intel.com","name":"Li Zhihui"},"message":"Modify default value description for spark.scheduler.minRegisteredResourcesRatio on docs.\n\nThe configuration is not supported in mesos mode now.\nSee https://github.com/apache/spark/pull/1462\n\nAuthor: Li Zhihui <zhihui.li@intel.com>\n\nCloses #4781 from li-zhihui/fixdocconf and squashes the following commits:\n\n63e7a44 [Li Zhihui] Modify default value description for spark.scheduler.minRegisteredResourcesRatio on docs.","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/10094a523e3993b775111ae9b22ca31cc0d76e03"},{"sha":"8942b522d8a3269a2a357e3a274ed4b3e66ebdde","author":{"email":"b4f0379e6b93e1f88e297cf3e1a13351f197b5b9@huawei.com","name":"xukun 00228947"},"message":"[SPARK-3562]Periodic cleanup event logs\n\nAuthor: xukun 00228947 <xukun.xu@huawei.com>\n\nCloses #4214 from viper-kun/cleaneventlog and squashes the following commits:\n\n7a5b9c5 [xukun 00228947] fix issue\n31674ee [xukun 00228947] fix issue\n6e3d06b [xukun 00228947] fix issue\n373f3b9 [xukun 00228947] fix issue\n71782b5 [xukun 00228947] fix issue\n5b45035 [xukun 00228947] fix issue\n70c28d6 [xukun 00228947] fix issues\nadcfe86 [xukun 00228947] Periodic cleanup event logs","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/8942b522d8a3269a2a357e3a274ed4b3e66ebdde"},{"sha":"aa63f633d39efa8c29095295f161eaad5495071d","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-6027][SPARK-5546] Fixed --jar and --packages not working for KafkaUtils and improved error message\n\nThe problem with SPARK-6027 in short is that JARs like the kafka-assembly.jar does not work in python as the added JAR is not visible in the classloader used by Py4J. Py4J uses Class.forName(), which does not uses the systemclassloader, but the JARs are only visible in the Thread's contextclassloader. So this back uses the context class loader to create the KafkaUtils dstream object. This works for both cases where the Kafka libraries are added with --jars spark-streaming-kafka-assembly.jar or with --packages spark-streaming-kafka\n\nAlso improves the error message.\n\ndavies\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #4779 from tdas/kafka-python-fix and squashes the following commits:\n\nfb16b04 [Tathagata Das] Removed import\nc1fdf35 [Tathagata Das] Fixed long line and improved documentation\n7b88be8 [Tathagata Das] Fixed --jar not working for KafkaUtils and improved error message","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/aa63f633d39efa8c29095295f161eaad5495071d"},{"sha":"5f3238b3b0157091d28803aa3b1d248dfa6cdc59","author":{"email":"9d01406527e5c15536e8dcb4e0c5005dca46786b@netflix.com","name":"Cheolsoo Park"},"message":"[SPARK-6018] [YARN] NoSuchMethodError in Spark app is swallowed by YARN AM\n\nAuthor: Cheolsoo Park <cheolsoop@netflix.com>\n\nCloses #4773 from piaozhexiu/SPARK-6018 and squashes the following commits:\n\n2a919d5 [Cheolsoo Park] Rename e with cause to avoid duplicate names\n1e71d2d [Cheolsoo Park] Replace placeholder with throwable\neb5750d [Cheolsoo Park] NoSuchMethodError in Spark app is swallowed by YARN AM","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/5f3238b3b0157091d28803aa3b1d248dfa6cdc59"},{"sha":"3fb53c0298761ba227890525ae79ce4ec6300deb","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-4300 [CORE] Race condition during SparkWorker shutdown\n\nClose appender saving stdout/stderr before destroying process to avoid exception on reading closed input stream.\n(This also removes a redundant `waitFor()` although it was harmless)\n\nCC tdas since I think you wrote this method.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #4787 from srowen/SPARK-4300 and squashes the following commits:\n\ne0cdabf [Sean Owen] Close appender saving stdout/stderr before destroying process to avoid exception on reading closed input stream","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/3fb53c0298761ba227890525ae79ce4ec6300deb"},{"sha":"c871e2dae0182e914135560d14304242e1f97f7e","author":{"email":"4a7c9f616c96bd67088f8ae3692def01f45a0f4a@gmail.com","name":"moussa taifi"},"message":"Add a note for context termination for History server on Yarn\n\nThe history server on Yarn only shows completed jobs. This adds a note concerning the needed explicit context termination at the end of a spark job which is a best practice anyway.\nRelated to SPARK-2972 and SPARK-3458\n\nAuthor: moussa taifi <moutai10@gmail.com>\n\nCloses #4721 from moutai/add-history-server-note-for-closing-the-spark-context and squashes the following commits:\n\n9f5b6c3 [moussa taifi] Fix upper case typo for YARN\n3ad3db4 [moussa taifi] Add context termination for History server on Yarn","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/c871e2dae0182e914135560d14304242e1f97f7e"},{"sha":"b38dec2ffdf724ff4e181cc8c7427d074b442670","author":{"email":"03c1034fc4007034103c51b9b4c6679c1d62a06f@guavus.com","name":"mohit.goyal"},"message":"[SPARK-5951][YARN] Remove unreachable driver memory properties in yarn client mode\n\nRemove unreachable driver memory properties in yarn client mode\n\nAuthor: mohit.goyal <mohit.goyal@guavus.com>\n\nCloses #4730 from zuxqoj/master and squashes the following commits:\n\n977dc96 [mohit.goyal] remove not rechable deprecated variables in yarn client mode","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/b38dec2ffdf724ff4e181cc8c7427d074b442670"},{"sha":"e60ad2f4c47b011be7a3198689ac2b82ee317d96","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"SPARK-6045 RecordWriter should be checked against null in PairRDDFunctio...\n\n...ns#saveAsNewAPIHadoopDataset\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #4794 from tedyu/master and squashes the following commits:\n\n2632a57 [tedyu] SPARK-6045 RecordWriter should be checked against null in PairRDDFunctions#saveAsNewAPIHadoopDataset\n2d8d4b1 [tedyu] SPARK-6045 RecordWriter should be checked against null in PairRDDFunctions#saveAsNewAPIHadoopDataset","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/e60ad2f4c47b011be7a3198689ac2b82ee317d96"},{"sha":"fbc469473dd529eb72046186b85dd8fc2b7c5bb5","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-4579 [WEBUI] Scheduling Delay appears negative\n\nEnsure scheduler delay handles unfinished task case, and ensure delay is never negative even due to rounding\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #4796 from srowen/SPARK-4579 and squashes the following commits:\n\nad6713c [Sean Owen] Ensure scheduler delay handles unfinished task case, and ensure delay is never negative even due to rounding","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/fbc469473dd529eb72046186b85dd8fc2b7c5bb5"},{"sha":"18f2098433e0bfef9497bacd601fdf098ed03eab","author":{"email":"fa6fdaa41f01a75e5f1a0f7fe39d1224e78e431d@tencent.com","name":"Hong Shen"},"message":" [SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver\n\nIf a blockManager has not send heartBeat more than 120s, BlockManagerMasterActor will remove it. But coarseGrainedSchedulerBackend can only remove executor after an DisassociatedEvent.  We should expireDeadHosts at HeartbeatReceiver.\n\nAuthor: Hong Shen <hongshen@tencent.com>\n\nCloses #4363 from shenh062326/my_change3 and squashes the following commits:\n\n2c9a46a [Hong Shen] Change some code style.\n1a042ff [Hong Shen] Change some code style.\n2dc456e [Hong Shen] Change some code style.\nd221493 [Hong Shen] Fix test failed\n7448ac6 [Hong Shen] A minor change in sparkContext and heartbeatReceiver\nb904aed [Hong Shen] Fix failed test\n52725af [Hong Shen] Remove assert in SparkContext.killExecutors\n5bedcb8 [Hong Shen] Remove assert in SparkContext.killExecutors\na858fb5 [Hong Shen] A minor change in HeartbeatReceiver\n3e221d9 [Hong Shen] A minor change in HeartbeatReceiver\n6bab7aa [Hong Shen] Change a code style.\n07952f3 [Hong Shen] Change configs name and code style.\nce9257e [Hong Shen] Fix test failed\nbccd515 [Hong Shen] Fix test failed\n8e77408 [Hong Shen] Fix test failed\nc1dfda1 [Hong Shen] Fix test failed\ne197e20 [Hong Shen] Fix test failed\nfb5df97 [Hong Shen] Remove ExpireDeadHosts in BlockManagerMessages\nb5c0441 [Hong Shen] Remove expireDeadHosts in BlockManagerMasterActor\nc922cb0 [Hong Shen] Add expireDeadHosts in HeartbeatReceiver","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/18f2098433e0bfef9497bacd601fdf098ed03eab"},{"sha":"4ad5153f5449319a7e82c9013ccff4494ab58ef1","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-6037][SQL] Avoiding duplicate Parquet schema merging\n\n`FilteringParquetRowInputFormat` manually merges Parquet schemas before computing splits. However, it is duplicate because the schemas are already merged in `ParquetRelation2`. We don't need to re-merge them at `InputFormat`.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #4786 from viirya/dup_parquet_schemas_merge and squashes the following commits:\n\nef78a5a [Liang-Chi Hsieh] Avoiding duplicate Parquet schema merging.","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/4ad5153f5449319a7e82c9013ccff4494ab58ef1"},{"sha":"5e5ad6558d60cfbf360708584e883e80d363e33e","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-6024][SQL] When a data source table has too many columns, it's schema cannot be stored in metastore.\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-6024\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4795 from yhuai/wideSchema and squashes the following commits:\n\n4882e6f [Yin Huai] Address comments.\n73e71b4 [Yin Huai] Address comments.\n143927a [Yin Huai] Simplify code.\ncc1d472 [Yin Huai] Make the schema wider.\n12bacae [Yin Huai] If the JSON string of a schema is too large, split it before storing it in metastore.\ne9b4f70 [Yin Huai] Failed test.","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/5e5ad6558d60cfbf360708584e883e80d363e33e"},{"sha":"12135e90549f957962899487cd5eb95badd8976d","author":{"email":"79478dbbc8641d1d6545f348f655464aefbda99b@intel.com","name":"jerryshao"},"message":"[SPARK-5771][UI][hotfix] Change Requested Cores into * if default cores is not set\n\ncc andrewor14, srowen.\n\nAuthor: jerryshao <saisai.shao@intel.com>\n\nCloses #4800 from jerryshao/SPARK-5771 and squashes the following commits:\n\na2483c2 [jerryshao] Change the UI of Requested Cores into * if default cores is not set","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/12135e90549f957962899487cd5eb95badd8976d"},{"sha":"67595eb8fb563eb26654f056033a01f0199bdf68","author":{"email":"79478dbbc8641d1d6545f348f655464aefbda99b@intel.com","name":"jerryshao"},"message":"[SPARK-5495][UI] Add app and driver kill function in master web UI\n\nAdd application kill function in master web UI for standalone mode. Details can be seen in [SPARK-5495](https://issues.apache.org/jira/browse/SPARK-5495).\n\nThe snapshot of UI shows as below:\n![snapshot](https://dl.dropboxusercontent.com/u/19230832/master_ui.png)\n\nPlease help to review, thanks a lot.\n\nAuthor: jerryshao <saisai.shao@intel.com>\n\nCloses #4288 from jerryshao/SPARK-5495 and squashes the following commits:\n\nfa3e486 [jerryshao] Add some conditions\n9a7be93 [jerryshao] Add kill Driver function\na239776 [jerryshao] Change the code format\nff5195d [jerryshao] Add app kill function in master web UI","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/67595eb8fb563eb26654f056033a01f0199bdf68"},{"sha":"4a8a0a8ecd836bf7fe0f2e692cf20a62dda313c0","author":{"email":"f07e1382085517228cb5d1a941bfd236c3d85eda@gmail.com","name":"Lukasz Jastrzebski"},"message":"SPARK-2168 [Spark core] Use relative URIs for the app links in the History Server.\n\nAs agreed in PR #1160 adding test to verify if history server generates relative links to applications.\n\nAuthor: Lukasz Jastrzebski <lukasz.jastrzebski@gmail.com>\n\nCloses #4778 from elyast/master and squashes the following commits:\n\n0c07fab [Lukasz Jastrzebski] Incorporating comments for SPARK-2168\n6d7866d [Lukasz Jastrzebski] Adjusting test for  SPARK-2168 for master branch\nd6f4fbe [Lukasz Jastrzebski] Added test for  SPARK-2168","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/4a8a0a8ecd836bf7fe0f2e692cf20a62dda313c0"},{"sha":"7c99a014fb1e370e038e0f537c2301ca8138e620","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-6046] Privatize SparkConf.translateConfKey\n\nThe warning of deprecated configs is actually done when the configs are set, not when they are get. As a result we don't need to explicitly call `translateConfKey` outside of `SparkConf` just to print the warning again in vain.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #4797 from andrewor14/warn-deprecated-config and squashes the following commits:\n\n8fb43e6 [Andrew Or] Privatize SparkConf.translateConfKey","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/7c99a014fb1e370e038e0f537c2301ca8138e620"},{"sha":"0375a413b8a009f5820897691570a1273ee25b97","author":{"email":"b73e0dcb01ee68c89d7e08a15ffaec7209d937fd@fraudmetrix.cn","name":"许鹏"},"message":"fix spark-6033, clarify the spark.worker.cleanup behavior in standalone mode\n\njira case spark-6033 https://issues.apache.org/jira/browse/SPARK-6033\n\nIn standalone deploy mode, the cleanup will only remove the stopped application's directories.\n\nThe original description about the cleanup behavior is incorrect.\n\nAuthor: 许鹏 <peng.xu@fraudmetrix.cn>\n\nCloses #4803 from hseagle/spark-6033 and squashes the following commits:\n\n927a6a0 [许鹏] fix the incorrect description about the spark.worker.cleanup in standalone mode","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/0375a413b8a009f5820897691570a1273ee25b97"},{"sha":"8cd1692c9092150107bed27777951633cbf945f6","author":{"email":"7632d88ebdd73175d662531eefd9a84b8ade5e24@intel.com","name":"Zhang, Liye"},"message":"[SPARK-6036][CORE] avoid race condition between eventlogListener and akka actor system\n\nFor detail description, pls refer to [SPARK-6036](https://issues.apache.org/jira/browse/SPARK-6036).\n\nAuthor: Zhang, Liye <liye.zhang@intel.com>\n\nCloses #4785 from liyezhang556520/EventLogInProcess and squashes the following commits:\n\n8b0b0a6 [Zhang, Liye] stop listener after DAGScheduler\n79b15b3 [Zhang, Liye] SPARK-6036 avoid race condition between eventlogListener and akka actor system","distinct":true,"url":"https://api.github.com/repos/leahmcguire/spark/commits/8cd1692c9092150107bed27777951633cbf945f6"}]}