{"push_id":750675048,"size":13,"distinct_size":13,"ref":"refs/heads/branch-1.5","head":"472f0dc34871c477b56308b8fcdcdd8a267185c7","before":"ee43d355bcfc9c3f4f281f0c44e1b1f331c7bb97","commits":[{"sha":"990b4bf7c4e87e7579fefb7507b0276ec3893e46","author":{"email":"4def4b24b2686a2b5d04ee1d4578618e97fed4b0@us.ibm.com","name":"Christian Kadner"},"message":"[SPARK-9211] [SQL] [TEST] normalize line separators before generating MD5 hash\n\nThe golden answer file names for the existing Hive comparison tests were generated using a MD5 hash of the query text which uses Unix-style line separator characters `\\n` (LF).\nThis PR ensures that all occurrences of the Windows-style line separator `\\r\\n` (CR) are replaced with `\\n` (LF) before generating the MD5 hash to produce an identical MD5 hash for golden answer file names generated on Windows.\n\nAuthor: Christian Kadner <ckadner@us.ibm.com>\n\nCloses #7563 from ckadner/SPARK-9211_working and squashes the following commits:\n\nd541db0 [Christian Kadner] [SPARK-9211][SQL] normalize line separators before MD5 hash\n\n(cherry picked from commit abfedb9cd70af60c8290bd2f5a5cec1047845ba0)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/990b4bf7c4e87e7579fefb7507b0276ec3893e46"},{"sha":"3137628bcd21c5e07b738a54c4ed5664e4a12433","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-9548][SQL] Add a destructive iterator for BytesToBytesMap\n\nThis pull request adds a destructive iterator to BytesToBytesMap. When used, the iterator frees pages as it traverses them. This is part of the effort to avoid starving when we have more than one operators that can exhaust memory.\n\nThis is based on #7924, but fixes a bug there (Don't use destructive iterator in UnsafeKVExternalSorter).\n\nCloses #7924.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8003 from rxin/map-destructive-iterator and squashes the following commits:\n\n6b618c3 [Reynold Xin] Don't use destructive iterator in UnsafeKVExternalSorter.\na7bd8ec [Reynold Xin] Merge remote-tracking branch 'viirya/destructive_iter' into map-destructive-iterator\n7652083 [Liang-Chi Hsieh] For comments: add destructiveIterator(), modify unit test, remove code block.\n4a3e9de [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into destructive_iter\n581e9e3 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into destructive_iter\nf0ff783 [Liang-Chi Hsieh] No need to free last page.\n9e9d2a3 [Liang-Chi Hsieh] Add a destructive iterator for BytesToBytesMap.\n\n(cherry picked from commit 21fdfd7d6f89adbd37066c169e6ba9ccd337683e)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/3137628bcd21c5e07b738a54c4ed5664e4a12433"},{"sha":"3997dd3fde0f1f67ddc4941921a8ce1449bb44f0","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-9556] [SPARK-9619] [SPARK-9624] [STREAMING] Make BlockGenerator more robust and make all BlockGenerators subscribe to rate limit updates\n\nIn some receivers, instead of using the default `BlockGenerator` in `ReceiverSupervisorImpl`, custom generator with their custom listeners are used for reliability (see [`ReliableKafkaReceiver`](https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/ReliableKafkaReceiver.scala#L99) and [updated `KinesisReceiver`](https://github.com/apache/spark/pull/7825/files)). These custom generators do not receive rate updates. This PR modifies the code to allow custom `BlockGenerator`s to be created through the `ReceiverSupervisorImpl` so that they can be kept track and rate updates can be applied.\n\nIn the process, I did some simplification, and de-flaki-fication of some rate controller related tests. In particular.\n- Renamed `Receiver.executor` to `Receiver.supervisor` (to match `ReceiverSupervisor`)\n- Made `RateControllerSuite` faster (by increasing batch interval) and less flaky\n- Changed a few internal API to return the current rate of block generators as Long instead of Option\\[Long\\] (was inconsistent at places).\n- Updated existing `ReceiverTrackerSuite` to test that custom block generators get rate updates as well.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #7913 from tdas/SPARK-9556 and squashes the following commits:\n\n41d4461 [Tathagata Das] fix scala style\neb9fd59 [Tathagata Das] Updated kinesis receiver\nd24994d [Tathagata Das] Updated BlockGeneratorSuite to use manual clock in BlockGenerator\nd70608b [Tathagata Das] Updated BlockGenerator with states and proper synchronization\nf6bd47e [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-9556\n31da173 [Tathagata Das] Fix bug\n12116df [Tathagata Das] Add BlockGeneratorSuite\n74bd069 [Tathagata Das] Fix style\n989bb5c [Tathagata Das] Made BlockGenerator fail is used after stop, and added better unit tests for it\n3ff618c [Tathagata Das] Fix test\nb40eff8 [Tathagata Das] slight refactoring\nf0df0f1 [Tathagata Das] Scala style fixes\n51759cb [Tathagata Das] Refactored rate controller tests and added the ability to update rate of any custom block generator\n\n(cherry picked from commit 0a078303d08ad2bb92b9a8a6969563d75b512290)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/3997dd3fde0f1f67ddc4941921a8ce1449bb44f0"},{"sha":"8ecfb05e370bc80cd74bb99dc8dd9459a855312f","author":{"email":"c4fe2b1d90ef8f2548c8aeabfa633434316f0305@koeninger.org","name":"cody koeninger"},"message":"[DOCS] [STREAMING] make the existing parameter docs for OffsetRange ac…\n\n…tually visible\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #7995 from koeninger/doc-fixes and squashes the following commits:\n\n87af9ea [cody koeninger] [Docs][Streaming] make the existing parameter docs for OffsetRange actually visible\n\n(cherry picked from commit 1723e34893f9b087727ea0e5c8b335645f42c295)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/8ecfb05e370bc80cd74bb99dc8dd9459a855312f"},{"sha":"9806872065b97df524bd631467105219b37f79f3","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-9639] [STREAMING] Fix a potential NPE in Streaming JobScheduler\n\nBecause `JobScheduler.stop(false)` may set `eventLoop` to null when `JobHandler` is running, then it's possible that when `post` is called, `eventLoop` happens to null.\n\nThis PR fixed this bug and also set threads in `jobExecutor` to `daemon`.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #7960 from zsxwing/fix-npe and squashes the following commits:\n\nb0864c4 [zsxwing] Fix a potential NPE in Streaming JobScheduler\n\n(cherry picked from commit 346209097e88fe79015359e40b49c32cc0bdc439)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/9806872065b97df524bd631467105219b37f79f3"},{"sha":"272e88342540328a24702f07a730b156657bd3be","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9630] [SQL] Clean up new aggregate operators (SPARK-9240 follow up)\n\nThis is the followup of https://github.com/apache/spark/pull/7813. It renames `HybridUnsafeAggregationIterator` to `TungstenAggregationIterator` and makes it only work with `UnsafeRow`. Also, I add a `TungstenAggregate` that uses `TungstenAggregationIterator` and make `SortBasedAggregate` (renamed from `SortBasedAggregate`) only works with `SafeRow`.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #7954 from yhuai/agg-followUp and squashes the following commits:\n\n4d2f4fc [Yin Huai] Add comments and free map.\n0d7ddb9 [Yin Huai] Add TungstenAggregationQueryWithControlledFallbackSuite to test fall back process.\n91d69c2 [Yin Huai] Rename UnsafeHybridAggregationIterator to  TungstenAggregateIteraotr and make it only work with UnsafeRow.\n\n(cherry picked from commit 3504bf3aa9f7b75c0985f04ce2944833d8c5b5bd)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/272e88342540328a24702f07a730b156657bd3be"},{"sha":"d0a648ca14858ef39c72adfd9fa43db19a27ebb1","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-9645] [YARN] [CORE] Allow shuffle service to read shuffle files.\n\nSpark should not mess with the permissions of directories created\nby the cluster manager. Here, by setting the block manager dir\npermissions to 700, the shuffle service (running as the YARN user)\nwouldn't be able to serve shuffle files created by applications.\n\nAlso, the code to protect the local app dir was missing in standalone's\nWorker; that has been now added. Since all processes run as the same\nuser in standalone, `chmod 700` should not cause problems.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #7966 from vanzin/SPARK-9645 and squashes the following commits:\n\n6e07b31 [Marcelo Vanzin] Protect the app dir in standalone mode.\n384ba6a [Marcelo Vanzin] [SPARK-9645] [yarn] [core] Allow shuffle service to read shuffle files.\n\n(cherry picked from commit e234ea1b49d30bb6c8b8c001bd98c43de290dcff)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/d0a648ca14858ef39c72adfd9fa43db19a27ebb1"},{"sha":"985e454cb6a8b97e1b1d4a1c8b2fa86db1830d22","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-9633] [BUILD] SBT download locations outdated; need an update\n\nRemove 2 defunct SBT download URLs and replace with the 1 known download URL. Also, use https.\nFollow up on https://github.com/apache/spark/pull/7792\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #7956 from srowen/SPARK-9633 and squashes the following commits:\n\ncaa40bd [Sean Owen] Remove 2 defunct SBT download URLs and replace with the 1 known download URL. Also, use https.\n\n(cherry picked from commit 681e3024b6c2fcb54b42180d94d3ba3eed52a2d4)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/985e454cb6a8b97e1b1d4a1c8b2fa86db1830d22"},{"sha":"75b4e5ab306166ea5f9c65eb6d0a14dfc26ef2a9","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9691] [SQL] PySpark SQL rand function treats seed 0 as no seed\n\nhttps://issues.apache.org/jira/browse/SPARK-9691\n\njkbradley rxin\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #7999 from yhuai/pythonRand and squashes the following commits:\n\n4187e0c [Yin Huai] Regression test.\na985ef9 [Yin Huai] Use \"if seed is not None\" instead \"if seed\" because \"if seed\" returns false when seed is 0.\n\n(cherry picked from commit baf4587a569b49e39020c04c2785041bdd00789b)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/75b4e5ab306166ea5f9c65eb6d0a14dfc26ef2a9"},{"sha":"b4feccf6c418e933c0a02f80f9bc8e5ef89f5bdd","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-9228] [SQL] use tungsten.enabled in public for both of codegen/unsafe\n\nspark.sql.tungsten.enabled will be the default value for both codegen and unsafe, they are kept internally for debug/testing.\n\ncc marmbrus rxin\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #7998 from davies/tungsten and squashes the following commits:\n\nc1c16da [Davies Liu] update doc\n1a47be1 [Davies Liu] use tungsten.enabled for both of codegen/unsafe\n\n(cherry picked from commit 4e70e8256ce2f45b438642372329eac7b1e9e8cf)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/b4feccf6c418e933c0a02f80f9bc8e5ef89f5bdd"},{"sha":"9be9d3842ebf0f4561ad5feb213acfd7efa73d94","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-9650][SQL] Fix quoting behavior on interpolated column names\n\nMake sure that `$\"column\"` is consistent with other methods with respect to backticks.  Adds a bunch of tests for various ways of constructing columns.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #7969 from marmbrus/namesWithDots and squashes the following commits:\n\n53ef3d7 [Michael Armbrust] [SPARK-9650][SQL] Fix quoting behavior on interpolated column names\n2bf7a92 [Michael Armbrust] WIP\n\n(cherry picked from commit 0867b23c74a3e6347d718b67ddabff17b468eded)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/9be9d3842ebf0f4561ad5feb213acfd7efa73d94"},{"sha":"37b6403cb3ea9c6648019df071b56b4aa8162132","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9692] Remove SqlNewHadoopRDD's generated Tuple2 and InterruptibleIterator.\n\nA small performance optimization – we don't need to generate a Tuple2 and then immediately discard the key. We also don't need an extra wrapper from InterruptibleIterator.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8000 from rxin/SPARK-9692 and squashes the following commits:\n\n1d4d0b3 [Reynold Xin] [SPARK-9692] Remove SqlNewHadoopRDD's generated Tuple2 and InterruptibleIterator.\n\n(cherry picked from commit b87825310ac87485672868bf6a9ed01d154a3626)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/37b6403cb3ea9c6648019df071b56b4aa8162132"},{"sha":"472f0dc34871c477b56308b8fcdcdd8a267185c7","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9709] [SQL] Avoid starving unsafe operators that use sort\n\nThe issue is that a task may run multiple sorts, and the sorts run by the child operator (i.e. parent RDD) may acquire all available memory such that other sorts in the same task do not have enough to proceed. This manifests itself in an `IOException(\"Unable to acquire X bytes of memory\")` thrown by `UnsafeExternalSorter`.\n\nThe solution is to reserve a page in each sorter in the chain before computing the child operator's (parent RDD's) partitions. This requires us to use a new special RDD that does some preparation before computing the parent's partitions.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8011 from andrewor14/unsafe-starve-memory and squashes the following commits:\n\n35b69a4 [Andrew Or] Simplify test\n0b07782 [Andrew Or] Minor: update comments\n5d5afdf [Andrew Or] Merge branch 'master' of github.com:apache/spark into unsafe-starve-memory\n254032e [Andrew Or] Add tests\n234acbd [Andrew Or] Reserve a page in sorter when preparing each partition\nb889e08 [Andrew Or] MapPartitionsWithPreparationRDD\n\n(cherry picked from commit 014a9f9d8c9521180f7a448cc7cc96cc00537d5c)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/472f0dc34871c477b56308b8fcdcdd8a267185c7"}]}