{"push_id":640240009,"size":5,"distinct_size":5,"ref":"refs/heads/master","head":"a0761ec7063f984dcadc8d154f83dd9cfd1c5e0b","before":"ae036d08170202074b266afd17ce34b689c70b0c","commits":[{"sha":"b063a61b9852cf9b9d2c905332d2ecb2fd716cc4","author":{"email":"ee744581348450c1d7e87b523bfc73463ff306fd@usu-software.de","name":"mweindel"},"message":"Avoid warning message about invalid refuse_seconds value in Mesos >=0.21...\n\nStarting with version 0.21.0, Apache Mesos is very noisy if the filter parameter refuse_seconds is set to an invalid value like `-1`.\nI have seen systems with millions of log lines like\n```\nW0420 18:00:48.773059 32352 hierarchical_allocator_process.hpp:589] Using the default value of 'refuse_seconds' to create the refused resources filter because the input value is negative\n```\nin the Mesos master INFO and WARNING log files.\nTherefore the CoarseMesosSchedulerBackend should set the default value for refuse seconds (i.e. 5 seconds) directly.\nThis is no problem for the fine-grained MesosSchedulerBackend, as it uses the value 1 second for this parameter.\n\nAuthor: mweindel <m.weindel@usu-software.de>\n\nCloses #5597 from MartinWeindel/master and squashes the following commits:\n\n2f99ffd [mweindel] Avoid warning message about invalid refuse_seconds value in Mesos >=0.21.","distinct":true,"url":"https://api.github.com/repos/AlpineNow/spark/commits/b063a61b9852cf9b9d2c905332d2ecb2fd716cc4"},{"sha":"e72c16e30d85cdc394d318b5551698885cfda9b8","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-6014] [core] Revamp Spark shutdown hooks, fix shutdown races.\n\nThis change adds some new utility code to handle shutdown hooks in\nSpark. The main goal is to take advantage of Hadoop 2.x's API for\nshutdown hooks, which allows Spark to register a hook that will\nrun before the one that cleans up HDFS clients, and thus avoids\nsome races that would cause exceptions to show up and other issues\nsuch as failure to properly close event logs.\n\nUnfortunately, Hadoop 1.x does not have such APIs, so in that case\ncorrectness is still left to chance.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #5560 from vanzin/SPARK-6014 and squashes the following commits:\n\nedfafb1 [Marcelo Vanzin] Better scaladoc.\nfcaeedd [Marcelo Vanzin] Merge branch 'master' into SPARK-6014\ne7039dc [Marcelo Vanzin] [SPARK-6014] [core] Revamp Spark shutdown hooks, fix shutdown races.","distinct":true,"url":"https://api.github.com/repos/AlpineNow/spark/commits/e72c16e30d85cdc394d318b5551698885cfda9b8"},{"sha":"3134c3fe495862b7687b5aa00d3344d09cd5e08e","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-6953] [PySpark] speed up python tests\n\nThis PR try to speed up some python tests:\n\n```\ntests.py                       144s -> 103s      -41s\nmllib/classification.py         24s -> 17s        -7s\nmllib/regression.py             27s -> 15s       -12s\nmllib/tree.py                   27s -> 13s       -14s\nmllib/tests.py                  64s -> 31s       -33s\nstreaming/tests.py             185s -> 84s      -101s\n```\nConsidering python3, the total saving will be 558s (almost 10 minutes) (core, and streaming run three times, mllib runs twice).\n\nDuring testing, it will show used time for each test file:\n```\nRun core tests ...\nRunning test: pyspark/rdd.py ... ok (22s)\nRunning test: pyspark/context.py ... ok (16s)\nRunning test: pyspark/conf.py ... ok (4s)\nRunning test: pyspark/broadcast.py ... ok (4s)\nRunning test: pyspark/accumulators.py ... ok (4s)\nRunning test: pyspark/serializers.py ... ok (6s)\nRunning test: pyspark/profiler.py ... ok (5s)\nRunning test: pyspark/shuffle.py ... ok (1s)\nRunning test: pyspark/tests.py ... ok (103s)   144s\n```\n\nAuthor: Reynold Xin <rxin@databricks.com>\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #5605 from rxin/python-tests-speed and squashes the following commits:\n\nd08542d [Reynold Xin] Merge pull request #14 from mengxr/SPARK-6953\n89321ee [Xiangrui Meng] fix seed in tests\n3ad2387 [Reynold Xin] Merge pull request #5427 from davies/python_tests","distinct":true,"url":"https://api.github.com/repos/AlpineNow/spark/commits/3134c3fe495862b7687b5aa00d3344d09cd5e08e"},{"sha":"41ef78a94105bb995bb14d15d47cbb6ca1638f62","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Closes #5427","distinct":true,"url":"https://api.github.com/repos/AlpineNow/spark/commits/41ef78a94105bb995bb14d15d47cbb6ca1638f62"},{"sha":"a0761ec7063f984dcadc8d154f83dd9cfd1c5e0b","author":{"email":"a5b2ab181e41ccd628487c4d5eb4dd848a13fe99@gmail.com","name":"texasmichelle"},"message":"[SPARK-1684] [PROJECT INFRA] Merge script should standardize SPARK-XXX prefix\n\nCleans up the pull request title in the merge script to follow conventions outlined in the wiki under Contributing Code.\nhttps://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-ContributingCode\n\n[MODULE] SPARK-XXXX: Description\n\nAuthor: texasmichelle <texasmichelle@gmail.com>\n\nCloses #5149 from texasmichelle/master and squashes the following commits:\n\n9b6b0a7 [texasmichelle] resolved variable scope issue\n7d5fa20 [texasmichelle] only prompt if title has been modified\n8c195bb [texasmichelle] removed erroneous line\n4f1ed46 [texasmichelle] Deque removal, logic simplifications, & prompt user to pick a title (orig or modified)\ndf73f6a [texasmichelle] reworked regex's to enforce brackets around JIRA ref\n43b5aed [texasmichelle] Merge remote-tracking branch 'apache/master'\n25229c6 [texasmichelle] Merge remote-tracking branch 'apache/master'\naa20a6e [texasmichelle] Move code into main() and add doctest for new text parsing method\n48520ba [texasmichelle] SPARK-1684: Corrected import statement\n042099d [texasmichelle] SPARK-1684 Merge script should standardize SPARK-XXX prefix\n8f4a7d1 [texasmichelle] SPARK-1684 Merge script should standardize SPARK-XXX prefix","distinct":true,"url":"https://api.github.com/repos/AlpineNow/spark/commits/a0761ec7063f984dcadc8d154f83dd9cfd1c5e0b"}]}