{"push_id":808362324,"size":32,"distinct_size":21,"ref":"refs/heads/master","head":"bf4199e261c3c8dd2970e2a154c97b46fb339f02","before":"84f81e035e1dab1b42c36563041df6ba16e7b287","commits":[{"sha":"558e9c7e60a7c0d85ba26634e97562ad2163e91d","author":{"email":"0f2b3c6fa11d7b91bbe66572949ab2c74c066514@gmail.com","name":"Matt Hagen"},"message":"[SPARK-10663] Removed unnecessary invocation of DataFrame.toDF method.\n\nThe Scala example under the \"Example: Pipeline\" heading in this\ndocument initializes the \"test\" variable to a DataFrame. Because test\nis already a DF, there is not need to call test.toDF as the example\ndoes in a subsequent line: model.transform(test.toDF). So, I removed\nthe extraneous toDF invocation.\n\nAuthor: Matt Hagen <anonz3000@gmail.com>\n\nCloses #8875 from hagenhaus/SPARK-10663.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/558e9c7e60a7c0d85ba26634e97562ad2163e91d"},{"sha":"5548a254755bb84edae2768b94ab1816e1b49b91","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-10652] [SPARK-10742] [STREAMING] Set meaningful job descriptions for all streaming jobs\n\nHere is the screenshot after adding the job descriptions to threads that run receivers and the scheduler thread running the batch jobs.\n\n## All jobs page\n* Added job descriptions with links to relevant batch details page\n![image](https://cloud.githubusercontent.com/assets/663212/9924165/cda4a372-5cb1-11e5-91ca-d43a32c699e9.png)\n\n## All stages page\n* Added stage descriptions with links to relevant batch details page\n![image](https://cloud.githubusercontent.com/assets/663212/9923814/2cce266a-5cae-11e5-8a3f-dad84d06c50e.png)\n\n## Streaming batch details page\n* Added the +details link\n![image](https://cloud.githubusercontent.com/assets/663212/9921977/24014a32-5c98-11e5-958e-457b6c38065b.png)\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #8791 from tdas/SPARK-10652.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/5548a254755bb84edae2768b94ab1816e1b49b91"},{"sha":"44c28abf120754c0175c65ffd3d4587a350b3798","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10224] [STREAMING] Fix the issue that blockIntervalTimer won't call updateCurrentBuffer when stopping\n\n`blockIntervalTimer.stop(interruptTimer = false)` doesn't guarantee calling `updateCurrentBuffer`. So it's possible that `blockIntervalTimer` will exit when `updateCurrentBuffer` is not empty. Then the data in `currentBuffer` will be lost.\n\nTo reproduce it, you can add `Thread.sleep(200)` in this line (https://github.com/apache/spark/blob/69c9c177160e32a2fbc9b36ecc52156077fca6fc/streaming/src/main/scala/org/apache/spark/streaming/util/RecurringTimer.scala#L100) and run `StreamingContexSuite`.\nI cannot write a unit test to reproduce it because I cannot find an approach to force `RecurringTimer` suspend at this line for a few milliseconds.\n\nThere was a failure in Jenkins here: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41455/console\n\nThis PR updates RecurringTimer to make sure `stop(interruptTimer = false)` will call `callback` at least once after the `stop` method is called.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8417 from zsxwing/SPARK-10224.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/44c28abf120754c0175c65ffd3d4587a350b3798"},{"sha":"50e4634236668a0195390f0080d0ac230d428d05","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10769] [STREAMING] [TESTS] Fix o.a.s.streaming.CheckpointSuite.maintains rate controller\n\nFixed the following failure in https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1787/testReport/junit/org.apache.spark.streaming/CheckpointSuite/recovery_maintains_rate_controller/\n```\nsbt.ForkMain$ForkError: The code passed to eventually never returned normally. Attempted 660 times over 10.000044392000001 seconds. Last failure message: 9223372036854775807 did not equal 200.\n\tat org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)\n\tat org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)\n\tat org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n\tat org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:336)\n\tat org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n\tat org.apache.spark.streaming.CheckpointSuite$$anonfun$15.apply$mcV$sp(CheckpointSuite.scala:413)\n\tat org.apache.spark.streaming.CheckpointSuite$$anonfun$15.apply(CheckpointSuite.scala:396)\n\tat org.apache.spark.streaming.CheckpointSuite$$anonfun$15.apply(CheckpointSuite.scala:396)\n\tat org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)\n\tat org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)\n\tat org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)\n\tat org.scalatest.Transformer.apply(Transformer.scala:22)\n```\n\nIn this test, it calls `advanceTimeWithRealDelay(ssc, 2)` to run two batch jobs. However, one race condition is these two jobs can finish before the receiver is registered. Then `UpdateRateLimit` won't be sent to the receiver and `getDefaultBlockGeneratorRateLimit` cannot be updated.\n\nHere are the logs related to this issue:\n```\n15/09/22 19:28:26.154 pool-1-thread-1-ScalaTest-running-CheckpointSuite INFO CheckpointSuite: Manual clock before advancing = 2500\n\n15/09/22 19:28:26.869 JobScheduler INFO JobScheduler: Finished job streaming job 3000 ms.0 from job set of time 3000 ms\n15/09/22 19:28:26.869 JobScheduler INFO JobScheduler: Total delay: 1442975303.869 s for time 3000 ms (execution: 0.711 s)\n\n15/09/22 19:28:26.873 JobScheduler INFO JobScheduler: Finished job streaming job 3500 ms.0 from job set of time 3500 ms\n15/09/22 19:28:26.873 JobScheduler INFO JobScheduler: Total delay: 1442975303.373 s for time 3500 ms (execution: 0.004 s)\n\n15/09/22 19:28:26.879 sparkDriver-akka.actor.default-dispatcher-3 INFO ReceiverTracker: Registered receiver for stream 0 from localhost:57749\n\n15/09/22 19:28:27.154 pool-1-thread-1-ScalaTest-running-CheckpointSuite INFO CheckpointSuite: Manual clock after advancing = 3500\n```\n`advanceTimeWithRealDelay(ssc, 2)` triggered job 3000ms and 3500ms but the receiver was registered after job 3000ms and 3500ms finished.\n\nSo we should make sure the receiver online before running `advanceTimeWithRealDelay(ssc, 2)`.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8877 from zsxwing/SPARK-10769.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/50e4634236668a0195390f0080d0ac230d428d05"},{"sha":"27bfa9ab3a610e072c011fd88ee4684cea6ceb76","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[SPARK-10721] Log warning when file deletion fails\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #8843 from tedyu/master.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/27bfa9ab3a610e072c011fd88ee4684cea6ceb76"},{"sha":"a18208047f06a4244703c17023bb20cbe1f59d73","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10403] Allow UnsafeRowSerializer to work with tungsten-sort ShuffleManager\n\nThis patch attempts to fix an issue where Spark SQL's UnsafeRowSerializer was incompatible with the `tungsten-sort` ShuffleManager.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #8873 from JoshRosen/SPARK-10403.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/a18208047f06a4244703c17023bb20cbe1f59d73"},{"sha":"098be27ad53c485ee2fc7f5871c47f899020e87b","author":{"email":"8b686eb7d2bd6ecbcaccba98aff1768585dde44b@gmail.com","name":"sethah"},"message":"[SPARK-9715] [ML] Store numFeatures in all ML PredictionModel types\n\nAll prediction models should store `numFeatures` indicating the number of features the model was trained on. Default value of -1 added for backwards compatibility.\n\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #8675 from sethah/SPARK-9715.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/098be27ad53c485ee2fc7f5871c47f899020e87b"},{"sha":"ce2b056d35c0c75d5c162b93680ee2d84152e911","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10686] [ML] Add quantilesCol to AFTSurvivalRegression\n\nBy default ```quantilesCol``` should be empty. If ```quantileProbabilities``` is set, we should append quantiles as a new column (of type Vector).\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8836 from yanboliang/spark-10686.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/ce2b056d35c0c75d5c162b93680ee2d84152e911"},{"sha":"067afb4e9bb227f159bcbc2aafafce9693303ea9","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10699] [ML] Support checkpointInterval can be disabled\n\nCurrently use can set ```checkpointInterval``` to specify how often should the cache be check-pointed. But we also need the function that users can disable it. This PR supports that users can disable checkpoint if user setting ```checkpointInterval = -1```.\nWe also add documents for GBT ```cacheNodeIds``` to make users can understand more clearly about checkpoint.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8820 from yanboliang/spark-10699.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/067afb4e9bb227f159bcbc2aafafce9693303ea9"},{"sha":"9952217749118ae78fe794ca11e1c4a87a4ae8ba","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10731] [SQL] Delegate to Scala's DataFrame.take implementation in Python DataFrame.\n\nPython DataFrame.head/take now requires scanning all the partitions. This pull request changes them to delegate the actual implementation to Scala DataFrame (by calling DataFrame.take).\n\nThis is more of a hack for fixing this issue in 1.5.1. A more proper fix is to change executeCollect and executeTake to return InternalRow rather than Row, and thus eliminate the extra round-trip conversion.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8876 from rxin/SPARK-10731.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/9952217749118ae78fe794ca11e1c4a87a4ae8ba"},{"sha":"084e4e126211d74a79e8dbd2d0e604dd3c650822","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-6028][Core]A new RPC implemetation based on the network module\n\nDesign doc: https://docs.google.com/document/d/1CF5G6rGVQMKSyV_QKo4D2M-x6rxz5x1Ew7aK3Uq6u8c/edit?usp=sharing\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #6457 from zsxwing/new-rpc.","distinct":false,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/084e4e126211d74a79e8dbd2d0e604dd3c650822"},{"sha":"83f6f54d12a418f5158ee7ee985b54eef8cc1cf0","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array (round 2)\n\nThis patch reverts most of the changes in a previous fix #8827.\n\nThe real cause of the issue is that in `TungstenAggregate`'s prepare method we only reserve 1 page, but later when we switch to sort-based aggregation we try to acquire 1 page AND a pointer array. The longer-term fix should be to reserve also the pointer array, but for now ***we will simply not track the pointer array***. (Note that elsewhere we already don't track the pointer array, e.g. [here](https://github.com/apache/spark/blob/a18208047f06a4244703c17023bb20cbe1f59d73/sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java#L88))\n\nNote: This patch reuses the unit test added in #8827 so it doesn't show up in the diff.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8888 from andrewor14/dont-track-pointer-array.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/83f6f54d12a418f5158ee7ee985b54eef8cc1cf0"},{"sha":"758c9d25e92417f8c06328c3af7ea2ef0212c79f","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10692] [STREAMING] Expose failureReasons in BatchInfo for streaming UI to clear failed batches\n\nSlightly modified version of #8818, all credit goes to zsxwing\n\nAuthor: zsxwing <zsxwing@gmail.com>\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #8892 from tdas/SPARK-10692.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/758c9d25e92417f8c06328c3af7ea2ef0212c79f"},{"sha":"d91967e159f416924bbd7f0db25156588d4bd7b1","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10763] [ML] [JAVA] [TEST] Update Java MLLIB/ML tests to use simplified dataframe construction\n\nAs introduced in https://issues.apache.org/jira/browse/SPARK-10630 we now have an easier way to create dataframes from local Java lists. Lets update the tests to use those.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8886 from holdenk/SPARK-10763-update-java-mllib-ml-tests-to-use-simplified-dataframe-construction.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/d91967e159f416924bbd7f0db25156588d4bd7b1"},{"sha":"02144d6745ec0a6d8877d969feb82139bd22437f","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"Revert \"[SPARK-6028][Core]A new RPC implemetation based on the network module\"\n\nThis reverts commit 084e4e126211d74a79e8dbd2d0e604dd3c650822.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/02144d6745ec0a6d8877d969feb82139bd22437f"},{"sha":"341b13f8f5eb118f1fb4d4f84418715ac4750a4d","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@163.com","name":"Wenchen Fan"},"message":"[SPARK-10765] [SQL] use new aggregate interface for hive UDAF\n\nAuthor: Wenchen Fan <cloud0fan@163.com>\n\nCloses #8874 from cloud-fan/hive-agg.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/341b13f8f5eb118f1fb4d4f84418715ac4750a4d"},{"sha":"b3862d3c59746ffb5f089aea4ff9e6f033a2c658","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-10705] [SQL] Avoid using external rows in DataFrame.toJSON\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-10705\n\nAs described in the JIRA ticket, `DataFrame.toJSON` uses `DataFrame.mapPartitions`, which converts internal rows to external rows. We should use `queryExecution.toRdd.mapPartitions` that directly uses internal rows for better performance.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #8865 from viirya/df-tojson-internalrow.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/b3862d3c59746ffb5f089aea4ff9e6f033a2c658"},{"sha":"8023242e77e4a799de6edc490078285684549b6d","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10761] Refactor DiskBlockObjectWriter to not require BlockId\n\nThe DiskBlockObjectWriter constructor took a BlockId parameter but never used it. As part of some general cleanup in these interfaces, this patch refactors its constructor to eliminate this parameter.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #8871 from JoshRosen/disk-block-object-writer-blockid-cleanup.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/8023242e77e4a799de6edc490078285684549b6d"},{"sha":"21fd12cb17b9e08a0cc49b4fda801af947a4183b","author":{"email":"c481c5e387de8107aa9fe99c851e11e1ff44d0c3@databricks.com","name":"Matei Zaharia"},"message":"[SPARK-9852] Let reduce tasks fetch multiple map output partitions\n\nThis makes two changes:\n\n- Allow reduce tasks to fetch multiple map output partitions -- this is a pretty small change to HashShuffleFetcher\n- Move shuffle locality computation out of DAGScheduler and into ShuffledRDD / MapOutputTracker; this was needed because the code in DAGScheduler wouldn't work for RDDs that fetch multiple map output partitions from each reduce task\n\nI also added an AdaptiveSchedulingSuite that creates RDDs depending on multiple map output partitions.\n\nAuthor: Matei Zaharia <matei@databricks.com>\n\nCloses #8844 from mateiz/spark-9852.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/21fd12cb17b9e08a0cc49b4fda801af947a4183b"},{"sha":"922338812c03eba43f2f1a6c414d1b6b049811cf","author":{"email":"cc463b26613f4ba480f0f94ca5cb2cab678c4fcb@databricks.com","name":"Eric Liang"},"message":"[SPARK-9681] [ML] Support R feature interactions in RFormula\n\nThis integrates the Interaction feature transformer with SparkR R formula support (i.e. support `:`).\n\nTo generate reasonable ML attribute names for feature interactions, it was necessary to add the ability to read attribute the original attribute names back from `StructField`, and also to specify custom group prefixes in `VectorAssembler`. This also has the side-benefit of cleaning up the double-underscores in the attributes generated for non-interaction terms.\n\nmengxr\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #8830 from ericl/interaction-2.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/922338812c03eba43f2f1a6c414d1b6b049811cf"}]}