{"push_id":862313418,"size":21,"distinct_size":21,"ref":"refs/heads/master","head":"b8ff6888e76b437287d7d6bf2d4b9c759710a195","before":"6e101d2e9d6e08a6a63f7065c1e87a5338f763ea","commits":[{"sha":"fac53d8ec015e27d034dfe30ed8ce7d83f07efa6","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10192][HOTFIX] Fix NPE in test that was added in #8402\n\nThis fixes an NPE introduced in SPARK-10192 / #8402.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9620 from JoshRosen/SPARK-10192-hotfix.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/fac53d8ec015e27d034dfe30ed8ce7d83f07efa6"},{"sha":"1a8e0468a1c07e99ad395eb0e4dc072c5cf7393a","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-11468] [SPARKR] add stddev/variance agg functions for Column\n\nChecked names, none of them should conflict with anything in base\n\nshivaram davies rxin\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #9489 from felixcheung/rstddev.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/1a8e0468a1c07e99ad395eb0e4dc072c5cf7393a"},{"sha":"bd70244b3cda62cc447fd4cc343d4eb5ddaec893","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11463] [PYSPARK] only install signal in main thread\n\nOnly install signal in main thread, or it will fail to create context in not-main thread.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9574 from davies/python_signal.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/bd70244b3cda62cc447fd4cc343d4eb5ddaec893"},{"sha":"99f5f988612b3093d73d9ce98819767e822fcbff","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-11290][STREAMING] Basic implementation of trackStateByKey\n\nCurrent updateStateByKey provides stateful processing in Spark Streaming. It allows the user to maintain per-key state and manage that state using an updateFunction. The updateFunction is called for each key, and it uses new data and existing state of the key, to generate an updated state. However, based on community feedback, we have learnt the following lessons.\n* Need for more optimized state management that does not scan every key\n* Need to make it easier to implement common use cases - (a) timeout of idle data, (b) returning items other than state\n\nThe high level idea that of this PR\n* Introduce a new API trackStateByKey that, allows the user to update per-key state, and emit arbitrary records. The new API is necessary as this will have significantly different semantics than the existing updateStateByKey API. This API will have direct support for timeouts.\n* Internally, the system will keep the state data as a map/list within the partitions of the state RDDs. The new data RDDs will be partitioned appropriately, and for all the key-value data, it will lookup the map/list in the state RDD partition and create a new list/map of updated state data. The new state RDD partition will be created based on the update data and if necessary, with old data.\nHere is the detailed design doc. Please take a look and provide feedback as comments.\nhttps://docs.google.com/document/d/1NoALLyd83zGs1hNGMm0Pc5YOVgiPpMHugGMk6COqxxE/edit#heading=h.ph3w0clkd4em\n\nThis is still WIP. Major things left to be done.\n- [x] Implement basic functionality of state tracking, with initial RDD and timeouts\n- [x] Unit tests for state tracking\n- [x] Unit tests for initial RDD and timeout\n- [ ] Unit tests for TrackStateRDD\n       - [x] state creating, updating, removing\n       - [ ] emitting\n       - [ ] checkpointing\n- [x] Misc unit tests for State, TrackStateSpec, etc.\n- [x] Update docs and experimental tags\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #9256 from tdas/trackStateByKey.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/99f5f988612b3093d73d9ce98819767e822fcbff"},{"sha":"1bc41125ee6306e627be212969854f639969c440","author":{"email":"2680965ad5aae29275c8fb10150cb10f5554e78b@gmail.com","name":"hyukjinkwon"},"message":"[SPARK-11500][SQL] Not deterministic order of columns when using merging schemas.\n\nhttps://issues.apache.org/jira/browse/SPARK-11500\n\nAs filed in SPARK-11500, if merging schemas is enabled, the order of files to touch is a matter which might affect the ordering of the output columns.\n\nThis was mostly because of the use of `Set` and `Map` so I replaced them to `LinkedHashSet` and `LinkedHashMap` to keep the insertion order.\n\nAlso, I changed `reduceOption` to `reduceLeftOption`, and replaced the order of `filesToTouch` from `metadataStatuses ++ commonMetadataStatuses ++ needMerged` to  `needMerged ++ metadataStatuses ++ commonMetadataStatuses` in order to touch the part-files first which always have the schema in footers whereas the others might not exist.\n\nOne nit is, If merging schemas is not enabled, but when multiple files are given, there is no guarantee of the output order, since there might not be a summary file for the first file, which ends up putting ahead the columns of the other files.\n\nHowever, I thought this should be okay since disabling merging schemas means (assumes) all the files have the same schemas.\n\nIn addition, in the test code for this, I only checked the names of fields.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #9517 from HyukjinKwon/SPARK-11500.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/1bc41125ee6306e627be212969854f639969c440"},{"sha":"1510c527b4f5ee0953ae42313ef9e16d2f5864c4","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-10371][SQL][FOLLOW-UP] fix code style\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9627 from cloud-fan/follow.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/1510c527b4f5ee0953ae42313ef9e16d2f5864c4"},{"sha":"27524a3a9ccee6fbe56149180ebfb3f74e0957e7","author":{"email":"231c419a17dc5ac843a8f2a3f7c7c7ce7e50978c@gmail.com","name":"Yuming Wang"},"message":"[SPARK-11626][ML] ml.feature.Word2Vec.transform() function very slow\n\norg.apache.spark.ml.feature.Word2Vec.transform() very slow. we should not read broadcast every sentence.\n\nAuthor: Yuming Wang <q79969786@gmail.com>\nAuthor: yuming.wang <q79969786@gmail.com>\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9592 from 979969786/master.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/27524a3a9ccee6fbe56149180ebfb3f74e0957e7"},{"sha":"95daff6459fc749949c2d71a0b7ab1c5be854f70","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11646] WholeTextFileRDD should return Text rather than String\n\nIf it returns Text, we can reuse this in Spark SQL to provide a WholeTextFile data source and directly convert the Text into UTF8String without extra string decoding and encoding.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9622 from rxin/SPARK-11646.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/95daff6459fc749949c2d71a0b7ab1c5be854f70"},{"sha":"c964fc101585171aee76996981fe2c9fdafc614e","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SQL][MINOR] rename present to finish in Aggregator\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9617 from cloud-fan/tmp.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/c964fc101585171aee76996981fe2c9fdafc614e"},{"sha":"9c57bc0efce0ac37d8319666f5a8d3e8dce7651c","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11656][SQL] support typed aggregate in project list\n\ninsert `aEncoder` like we do in `agg`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9630 from cloud-fan/select.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/9c57bc0efce0ac37d8319666f5a8d3e8dce7651c"},{"sha":"ec2b807212e568c9e98cd80746bcb61e02c7a98e","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11564][SQL][FOLLOW-UP] clean up java tuple encoder\n\nWe need to support custom classes like java beans and combine them into tuple, and it's very hard to do it with the  TypeTag-based approach.\nWe should keep only the compose-based way to create tuple encoder.\n\nThis PR also move `Encoder` to `org.apache.spark.sql`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9567 from cloud-fan/java.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/ec2b807212e568c9e98cd80746bcb61e02c7a98e"},{"sha":"e71ba56586ba64da18f412bc0e0263777c46ac4a","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SQL][MINOR] remove newLongEncoder in functions\n\nit may shadows the one from implicits in some case.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9629 from cloud-fan/minor.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/e71ba56586ba64da18f412bc0e0263777c46ac4a"},{"sha":"529a1d3380c4c23fed068ad05a6376162c4b76d6","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-6152] Use shaded ASM5 to support closure cleaning of Java 8 compiled classes\n\nThis patch modifies Spark's closure cleaner (and a few other places) to use ASM 5, which is necessary in order to support cleaning of closures that were compiled by Java 8.\n\nIn order to avoid ASM dependency conflicts, Spark excludes ASM from all of its dependencies and uses a shaded version of ASM 4 that comes from `reflectasm` (see [SPARK-782](https://issues.apache.org/jira/browse/SPARK-782) and #232). This patch updates Spark to use a shaded version of ASM 5.0.4 that was published by the Apache XBean project; the POM used to create the shaded artifact can be found at https://github.com/apache/geronimo-xbean/blob/xbean-4.4/xbean-asm5-shaded/pom.xml.\n\nhttp://movingfulcrum.tumblr.com/post/80826553604/asm-framework-50-the-missing-migration-guide was a useful resource while upgrading the code to use the new ASM5 opcodes.\n\nI also added a new regression tests in the `java8-tests` subproject; the existing tests were insufficient to catch this bug, which only affected Scala 2.11 user code which was compiled targeting Java 8.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9512 from JoshRosen/SPARK-6152.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/529a1d3380c4c23fed068ad05a6376162c4b76d6"},{"sha":"27029bc8f6246514bd0947500c94cf38dc8616c3","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-11639][STREAMING][FLAKY-TEST] Implement BlockingWriteAheadLog for testing the BatchedWriteAheadLog\n\nSeveral elements could be drained if the main thread is not fast enough. zsxwing warned me about a similar problem, but missed it here :( Submitting the fix using a waiter.\n\ncc tdas\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #9605 from brkyvz/fix-flaky-test.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/27029bc8f6246514bd0947500c94cf38dc8616c3"},{"sha":"df97df2b39194f60051f78cce23f0ba6cfe4b1df","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11644][SQL] Remove the option to turn off unsafe and codegen.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9618 from rxin/SPARK-11644.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/df97df2b39194f60051f78cce23f0ba6cfe4b1df"},{"sha":"a9a6b80c718008aac7c411dfe46355efe58dee2e","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11645][SQL] Remove OpenHashSet for the old aggregate.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9621 from rxin/SPARK-11645.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/a9a6b80c718008aac7c411dfe46355efe58dee2e"},{"sha":"dd77e278b99e45c20fdefb1c795f3c5148d577db","author":{"email":"b1c1d8736f20db3fb6c1c66bb1455ed43909f0d8@nicolasevans.org","name":"Nick Evans"},"message":"[SPARK-11335][STREAMING] update kafka direct python docs on how to get the offset ranges for a KafkaRDD\n\ntdas koeninger\n\nThis updates the Spark Streaming + Kafka Integration Guide doc with a working method to access the offsets of a `KafkaRDD` through Python.\n\nAuthor: Nick Evans <me@nicolasevans.org>\n\nCloses #9289 from manygrams/update_kafka_direct_python_docs.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/dd77e278b99e45c20fdefb1c795f3c5148d577db"},{"sha":"2d76e44b1a88e08047806972b2d241a89e499bab","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11647] Attempt to reduce time/flakiness of Thriftserver CLI and SparkSubmit tests\n\nThis patch aims to reduce the test time and flakiness of HiveSparkSubmitSuite, SparkSubmitSuite, and CliSuite.\n\nKey changes:\n\n- Disable IO synchronization calls for Derby writes, since durability doesn't matter for tests. This was done for HiveCompatibilitySuite in #6651 and resulted in huge test speedups.\n- Add a few missing `--conf`s to disable various Spark UIs. The CliSuite, in particular, never disabled these UIs, leaving it prone to port-contention-related flakiness.\n- Fix two instances where tests defined `beforeAll()` methods which were never called because the appropriate traits were not mixed in. I updated these tests suites to extend `BeforeAndAfterEach` so that they play nicely with our `ResetSystemProperties` trait.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9623 from JoshRosen/SPARK-11647.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/2d76e44b1a88e08047806972b2d241a89e499bab"},{"sha":"e1bcf6af9ba4f131f84d71660d0ab5598c0b7b67","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10827] replace volatile with Atomic* in AppClient.scala.\n\nThis is a followup for #9317 to replace volatile fields with AtomicBoolean and AtomicReference.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9611 from rxin/SPARK-10827.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/e1bcf6af9ba4f131f84d71660d0ab5598c0b7b67"},{"sha":"1a21be15f655b9696ddac80aac629445a465f621","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-11672][ML] disable spark.ml read/write tests\n\nSaw several failures on Jenkins, e.g., https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2040/testReport/org.apache.spark.ml.util/JavaDefaultReadWriteSuite/testDefaultReadWrite/. This is the first failure in master build:\n\nhttps://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3982/\n\nI cannot reproduce it on local. So temporarily disable the tests and I will look into the issue under the same JIRA. I'm going to merge the PR after Jenkins passes compile.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9641 from mengxr/SPARK-11672.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/1a21be15f655b9696ddac80aac629445a465f621"}]}