{"push_id":871336438,"size":169,"distinct_size":169,"ref":"refs/heads/master","head":"fc3f77b42d62ca789d0ee07403795978961991c7","before":"1bc41125ee6306e627be212969854f639969c440","commits":[{"sha":"1510c527b4f5ee0953ae42313ef9e16d2f5864c4","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-10371][SQL][FOLLOW-UP] fix code style\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9627 from cloud-fan/follow.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/1510c527b4f5ee0953ae42313ef9e16d2f5864c4"},{"sha":"27524a3a9ccee6fbe56149180ebfb3f74e0957e7","author":{"email":"231c419a17dc5ac843a8f2a3f7c7c7ce7e50978c@gmail.com","name":"Yuming Wang"},"message":"[SPARK-11626][ML] ml.feature.Word2Vec.transform() function very slow\n\norg.apache.spark.ml.feature.Word2Vec.transform() very slow. we should not read broadcast every sentence.\n\nAuthor: Yuming Wang <q79969786@gmail.com>\nAuthor: yuming.wang <q79969786@gmail.com>\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9592 from 979969786/master.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/27524a3a9ccee6fbe56149180ebfb3f74e0957e7"},{"sha":"95daff6459fc749949c2d71a0b7ab1c5be854f70","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11646] WholeTextFileRDD should return Text rather than String\n\nIf it returns Text, we can reuse this in Spark SQL to provide a WholeTextFile data source and directly convert the Text into UTF8String without extra string decoding and encoding.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9622 from rxin/SPARK-11646.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/95daff6459fc749949c2d71a0b7ab1c5be854f70"},{"sha":"c964fc101585171aee76996981fe2c9fdafc614e","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SQL][MINOR] rename present to finish in Aggregator\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9617 from cloud-fan/tmp.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/c964fc101585171aee76996981fe2c9fdafc614e"},{"sha":"9c57bc0efce0ac37d8319666f5a8d3e8dce7651c","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11656][SQL] support typed aggregate in project list\n\ninsert `aEncoder` like we do in `agg`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9630 from cloud-fan/select.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/9c57bc0efce0ac37d8319666f5a8d3e8dce7651c"},{"sha":"ec2b807212e568c9e98cd80746bcb61e02c7a98e","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11564][SQL][FOLLOW-UP] clean up java tuple encoder\n\nWe need to support custom classes like java beans and combine them into tuple, and it's very hard to do it with the  TypeTag-based approach.\nWe should keep only the compose-based way to create tuple encoder.\n\nThis PR also move `Encoder` to `org.apache.spark.sql`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9567 from cloud-fan/java.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/ec2b807212e568c9e98cd80746bcb61e02c7a98e"},{"sha":"e71ba56586ba64da18f412bc0e0263777c46ac4a","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SQL][MINOR] remove newLongEncoder in functions\n\nit may shadows the one from implicits in some case.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9629 from cloud-fan/minor.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/e71ba56586ba64da18f412bc0e0263777c46ac4a"},{"sha":"529a1d3380c4c23fed068ad05a6376162c4b76d6","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-6152] Use shaded ASM5 to support closure cleaning of Java 8 compiled classes\n\nThis patch modifies Spark's closure cleaner (and a few other places) to use ASM 5, which is necessary in order to support cleaning of closures that were compiled by Java 8.\n\nIn order to avoid ASM dependency conflicts, Spark excludes ASM from all of its dependencies and uses a shaded version of ASM 4 that comes from `reflectasm` (see [SPARK-782](https://issues.apache.org/jira/browse/SPARK-782) and #232). This patch updates Spark to use a shaded version of ASM 5.0.4 that was published by the Apache XBean project; the POM used to create the shaded artifact can be found at https://github.com/apache/geronimo-xbean/blob/xbean-4.4/xbean-asm5-shaded/pom.xml.\n\nhttp://movingfulcrum.tumblr.com/post/80826553604/asm-framework-50-the-missing-migration-guide was a useful resource while upgrading the code to use the new ASM5 opcodes.\n\nI also added a new regression tests in the `java8-tests` subproject; the existing tests were insufficient to catch this bug, which only affected Scala 2.11 user code which was compiled targeting Java 8.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9512 from JoshRosen/SPARK-6152.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/529a1d3380c4c23fed068ad05a6376162c4b76d6"},{"sha":"27029bc8f6246514bd0947500c94cf38dc8616c3","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-11639][STREAMING][FLAKY-TEST] Implement BlockingWriteAheadLog for testing the BatchedWriteAheadLog\n\nSeveral elements could be drained if the main thread is not fast enough. zsxwing warned me about a similar problem, but missed it here :( Submitting the fix using a waiter.\n\ncc tdas\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #9605 from brkyvz/fix-flaky-test.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/27029bc8f6246514bd0947500c94cf38dc8616c3"},{"sha":"df97df2b39194f60051f78cce23f0ba6cfe4b1df","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11644][SQL] Remove the option to turn off unsafe and codegen.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9618 from rxin/SPARK-11644.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/df97df2b39194f60051f78cce23f0ba6cfe4b1df"},{"sha":"a9a6b80c718008aac7c411dfe46355efe58dee2e","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11645][SQL] Remove OpenHashSet for the old aggregate.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9621 from rxin/SPARK-11645.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/a9a6b80c718008aac7c411dfe46355efe58dee2e"},{"sha":"dd77e278b99e45c20fdefb1c795f3c5148d577db","author":{"email":"b1c1d8736f20db3fb6c1c66bb1455ed43909f0d8@nicolasevans.org","name":"Nick Evans"},"message":"[SPARK-11335][STREAMING] update kafka direct python docs on how to get the offset ranges for a KafkaRDD\n\ntdas koeninger\n\nThis updates the Spark Streaming + Kafka Integration Guide doc with a working method to access the offsets of a `KafkaRDD` through Python.\n\nAuthor: Nick Evans <me@nicolasevans.org>\n\nCloses #9289 from manygrams/update_kafka_direct_python_docs.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/dd77e278b99e45c20fdefb1c795f3c5148d577db"},{"sha":"2d76e44b1a88e08047806972b2d241a89e499bab","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11647] Attempt to reduce time/flakiness of Thriftserver CLI and SparkSubmit tests\n\nThis patch aims to reduce the test time and flakiness of HiveSparkSubmitSuite, SparkSubmitSuite, and CliSuite.\n\nKey changes:\n\n- Disable IO synchronization calls for Derby writes, since durability doesn't matter for tests. This was done for HiveCompatibilitySuite in #6651 and resulted in huge test speedups.\n- Add a few missing `--conf`s to disable various Spark UIs. The CliSuite, in particular, never disabled these UIs, leaving it prone to port-contention-related flakiness.\n- Fix two instances where tests defined `beforeAll()` methods which were never called because the appropriate traits were not mixed in. I updated these tests suites to extend `BeforeAndAfterEach` so that they play nicely with our `ResetSystemProperties` trait.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9623 from JoshRosen/SPARK-11647.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/2d76e44b1a88e08047806972b2d241a89e499bab"},{"sha":"e1bcf6af9ba4f131f84d71660d0ab5598c0b7b67","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10827] replace volatile with Atomic* in AppClient.scala.\n\nThis is a followup for #9317 to replace volatile fields with AtomicBoolean and AtomicReference.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9611 from rxin/SPARK-10827.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/e1bcf6af9ba4f131f84d71660d0ab5598c0b7b67"},{"sha":"1a21be15f655b9696ddac80aac629445a465f621","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-11672][ML] disable spark.ml read/write tests\n\nSaw several failures on Jenkins, e.g., https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2040/testReport/org.apache.spark.ml.util/JavaDefaultReadWriteSuite/testDefaultReadWrite/. This is the first failure in master build:\n\nhttps://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3982/\n\nI cannot reproduce it on local. So temporarily disable the tests and I will look into the issue under the same JIRA. I'm going to merge the PR after Jenkins passes compile.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9641 from mengxr/SPARK-11672.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/1a21be15f655b9696ddac80aac629445a465f621"},{"sha":"b8ff6888e76b437287d7d6bf2d4b9c759710a195","author":{"email":"810f4eaecb829dde5e3d87bd39ae52346de8f0c9@gmail.com","name":"Andrew Ray"},"message":"[SPARK-8992][SQL] Add pivot to dataframe api\n\nThis adds a pivot method to the dataframe api.\n\nFollowing the lead of cube and rollup this adds a Pivot operator that is translated into an Aggregate by the analyzer.\n\nCurrently the syntax is like:\n~~courseSales.pivot(Seq($\"year\"), $\"course\", Seq(\"dotNET\", \"Java\"), sum($\"earnings\"))~~\n\n~~Would we be interested in the following syntax also/alternatively? and~~\n\n    courseSales.groupBy($\"year\").pivot($\"course\", \"dotNET\", \"Java\").agg(sum($\"earnings\"))\n    //or\n    courseSales.groupBy($\"year\").pivot($\"course\").agg(sum($\"earnings\"))\n\nLater we can add it to `SQLParser`, but as Hive doesn't support it we cant add it there, right?\n\n~~Also what would be the suggested Java friendly method signature for this?~~\n\nAuthor: Andrew Ray <ray.andrew@gmail.com>\n\nCloses #7841 from aray/sql-pivot.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/b8ff6888e76b437287d7d6bf2d4b9c759710a195"},{"sha":"e49e723392b8a64d30bd90944a748eb6f5ef3a8a","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11675][SQL] Remove shuffle hash joins.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9645 from rxin/SPARK-11675.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/e49e723392b8a64d30bd90944a748eb6f5ef3a8a"},{"sha":"39b1e36fbc284ba999e1c00b20d1b0b5de6b40b2","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-11396] [SQL] add native implementation of datetime function to_unix_timestamp\n\n`to_unix_timestamp` is the deterministic version of `unix_timestamp`, as it accepts at least one parameters.\n\nSince the behavior here is quite similar to `unix_timestamp`, I think the dataframe API is not necessary here.\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #9347 from adrian-wang/to_unix_timestamp.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/39b1e36fbc284ba999e1c00b20d1b0b5de6b40b2"},{"sha":"e2957bc085d39d59c09e4b33c26a05f0263200a3","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-11674][ML] add private val after @transient in Word2VecModel\n\nThis causes compile failure with Scala 2.11. See https://issues.scala-lang.org/browse/SI-8813. (Jenkins won't test Scala 2.11. I tested compile locally.) JoshRosen\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9644 from mengxr/SPARK-11674.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/e2957bc085d39d59c09e4b33c26a05f0263200a3"},{"sha":"14cf753704ea60f358cb870b018cbcf73654f198","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11661][SQL] Still pushdown filters returned by unhandledFilters.\n\nhttps://issues.apache.org/jira/browse/SPARK-11661\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9634 from yhuai/unhandledFilters.","distinct":true,"url":"https://api.github.com/repos/smarthi/spark/commits/14cf753704ea60f358cb870b018cbcf73654f198"}]}