{"push_id":680543144,"size":41,"distinct_size":41,"ref":"refs/heads/master","head":"6f006b5f5fca649ac51745212d8fd44b1609b9ae","before":"5fb97dca9bcfc29ac33823554c8783997e811b99","commits":[{"sha":"dbf8ff38de0f95f467b874a5b527dcf59439efe8","author":{"email":"89674bb0dc6f2e03b27640b006f61a9dd0e29d81@hw11853.local","name":"Ram Sriharsha"},"message":"[SPARK-6013] [ML] Add more Python ML examples for spark.ml\n\nAuthor: Ram Sriharsha <rsriharsha@hw11853.local>\n\nCloses #6443 from harsha2010/SPARK-6013 and squashes the following commits:\n\n732506e [Ram Sriharsha] Code Review Feedback\n121c211 [Ram Sriharsha] python style fix\n5f9b8c3 [Ram Sriharsha] python style fixes\n925ca86 [Ram Sriharsha] Simple Params Example\n8b372b1 [Ram Sriharsha] GBT Example\n965ec14 [Ram Sriharsha] Random Forest Example","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/dbf8ff38de0f95f467b874a5b527dcf59439efe8"},{"sha":"8c9979337f193c72fd2f1a891909283de53777e3","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[HOTFIX] [SQL] Maven test compilation issue\n\nTests compile in SBT but not Maven.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/8c9979337f193c72fd2f1a891909283de53777e3"},{"sha":"a4f24123d8857656524c9138c7c067a4b1033a5e","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[HOT FIX] [BUILD] Fix maven build failures\n\nThis patch fixes a build break in maven caused by #6441.\n\nNote that this patch reverts the changes in flume-sink because\nthis module does not currently depend on Spark core, but the\ntests require it. There is not an easy way to make this work\nbecause mvn test dependencies are not transitive (MNG-1378).\n\nFor now, we will leave the one test suite in flume-sink out\nuntil we figure out a better solution. This patch is mainly\nintended to unbreak the maven build.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #6511 from andrewor14/fix-build-mvn and squashes the following commits:\n\n3d53643 [Andrew Or] [HOT FIX #6441] Fix maven build failures","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/a4f24123d8857656524c9138c7c067a4b1033a5e"},{"sha":"3792d25836e1e521da64c5a62ca1b6cca1bcb6b9","author":{"email":"d830bbc0ff8d4f6be5b5bfec1999946e5bc7c67a@gmail.com","name":"Taka Shinagawa"},"message":"[DOCS][Tiny] Added a missing dash(-) in docs/configuration.md\n\nThe first line had only two dashes (--) instead of three(---). Because of this missing dash(-), 'jekyll build' command was not converting configuration.md to _site/configuration.html\n\nAuthor: Taka Shinagawa <taka.epsilon@gmail.com>\n\nCloses #6513 from mrt/docfix3 and squashes the following commits:\n\nc470e2c [Taka Shinagawa] Added a missing dash(-) preventing jekyll from converting configuration.md to html format","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/3792d25836e1e521da64c5a62ca1b6cca1bcb6b9"},{"sha":"7ed06c39922ac90acab3a78ce0f2f21184ed68a5","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-7957] Preserve partitioning when using randomSplit\n\ncc JoshRosen\nThanks for noticing this!\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #6509 from brkyvz/sample-perf-reg and squashes the following commits:\n\n497465d [Burak Yavuz] addressed code review\n293f95f [Burak Yavuz] [SPARK-7957] Preserve partitioning when using randomSplit","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/7ed06c39922ac90acab3a78ce0f2f21184ed68a5"},{"sha":"609c4923f98c188bce60ae35c1c8a08a8dfd95f1","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-7558] Guard against direct uses of FunSuite / FunSuiteLike\n\nThis is a follow-up patch to #6441.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #6510 from andrewor14/extends-funsuite-check and squashes the following commits:\n\n6618b46 [Andrew Or] Exempt SparkSinkSuite from the FunSuite check\n99d02ac [Andrew Or] Merge branch 'master' of github.com:apache/spark into extends-funsuite-check\n48874dd [Andrew Or] Guard against direct uses of FunSuite / FunSuiteLike","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/609c4923f98c188bce60ae35c1c8a08a8dfd95f1"},{"sha":"193dba01c77ef1bb63e3f617213eb257960f8d2f","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[TRIVIAL] Typo fix for last commit","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/193dba01c77ef1bb63e3f617213eb257960f8d2f"},{"sha":"da2112aef28e63c452f592e0abd007141787877d","author":{"email":"0e579b08a01ff18b1532d130b2dcd3e2b7ec8fb6@gmail.com","name":"Octavian Geagla"},"message":"[SPARK-7576] [MLLIB] Add spark.ml user guide doc/example for ElementwiseProduct\n\nAuthor: Octavian Geagla <ogeagla@gmail.com>\n\nCloses #6501 from ogeagla/ml-guide-elemwiseprod and squashes the following commits:\n\n4ad93d5 [Octavian Geagla] [SPARK-7576] [MLLIB] Incorporate code review feedback.\nf7be7ad [Octavian Geagla] [SPARK-7576] [MLLIB] Add spark.ml user guide doc/example for ElementwiseProduct.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/da2112aef28e63c452f592e0abd007141787877d"},{"sha":"78657d53d71b9d3e86b675cc519868f99e2ffa01","author":{"email":"22a0dda5f6015b8ee84a30ff6553d35a91fe8732@gmail.com","name":"Timothy Chen"},"message":"[SPARK-7962] [MESOS] Fix master url parsing in rest submission client.\n\nOnly parse standalone master url when master url starts with spark://\n\nAuthor: Timothy Chen <tnachen@gmail.com>\n\nCloses #6517 from tnachen/fix_mesos_client and squashes the following commits:\n\n61a1198 [Timothy Chen] Fix master url parsing in rest submission client.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/78657d53d71b9d3e86b675cc519868f99e2ffa01"},{"sha":"e3a43748338b02ef6864ca62de40e218e5677506","author":{"email":"0e579b08a01ff18b1532d130b2dcd3e2b7ec8fb6@gmail.com","name":"Octavian Geagla"},"message":"[SPARK-7459] [MLLIB] ElementwiseProduct Java example\n\nAuthor: Octavian Geagla <ogeagla@gmail.com>\n\nCloses #6008 from ogeagla/elementwise-prod-doc and squashes the following commits:\n\n72e6dc0 [Octavian Geagla] [SPARK-7459] [MLLIB] Java example import.\ncf2afbd [Octavian Geagla] [SPARK-7459] [MLLIB] Update description of example.\nb66431b [Octavian Geagla] [SPARK-7459] [MLLIB] Add override annotation to java example, make scala example use same data as java.\n6b26b03 [Octavian Geagla] [SPARK-7459] [MLLIB] Fix line which is too long.\n79af020 [Octavian Geagla] [SPARK-7459] [MLLIB] Actually don't use Java 8.\n9d5b31a [Octavian Geagla] [SPARK-7459] [MLLIB] Don't use Java 8\n4f0c92f [Octavian Geagla] [SPARK-7459] [MLLIB] ElementwiseProduct Java example.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/e3a43748338b02ef6864ca62de40e218e5677506"},{"sha":"0978aec9cd47dc0618e47b74a99e1cc2266be424","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-7964][SQL] remove unnecessary type coercion rule\n\nWe have defined these logics in `Cast` already, I think we should remove this rule.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #6516 from cloud-fan/tmp2 and squashes the following commits:\n\nd5035a4 [Wenchen Fan] remove useless rule","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/0978aec9cd47dc0618e47b74a99e1cc2266be424"},{"sha":"8c8de3ed863985554e84fd07d1cdcaeca7e3375c","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-7890] [DOCS] Document that Spark 2.11 now supports Kafka\n\nRemove caveat about Kafka / JDBC not being supported for Scala 2.11\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #6470 from srowen/SPARK-7890 and squashes the following commits:\n\n4652634 [Sean Owen] One more rewording\n7b7f3c8 [Sean Owen] Restore note about JDBC component\n126744d [Sean Owen] Remove caveat about Kafka / JDBC not being supported for Scala 2.11","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/8c8de3ed863985554e84fd07d1cdcaeca7e3375c"},{"sha":"9d8aadb72bbc86595e253fe30201cda6a8db877e","author":{"email":"7ea8d8bc92d239699affc4b9ea1759f960ddbaa7@huawei.com","name":"WangTaoTheTonic"},"message":"[SPARK-7945] [CORE] Do trim to values in properties file\n\nhttps://issues.apache.org/jira/browse/SPARK-7945\n\nNow applications submited by org.apache.spark.launcher.Main read properties file without doing trim to values in it.\nIf user left a space after a value(say spark.driver.extraClassPath) then it probably affect global functions(like some jar could not be included in the classpath), so we should do it like Utils.getPropertiesFromFile.\n\nAuthor: WangTaoTheTonic <wangtao111@huawei.com>\nAuthor: Tao Wang <wangtao111@huawei.com>\n\nCloses #6496 from WangTaoTheTonic/SPARK-7945 and squashes the following commits:\n\nbb41b4b [Tao Wang] indent 4 to 2\n6dd1cf2 [WangTaoTheTonic] use a simpler way\n2c053a1 [WangTaoTheTonic] Do trim to values in properties file","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/9d8aadb72bbc86595e253fe30201cda6a8db877e"},{"sha":"2b35c99c7e73d22e82aef90b675709ae7f8d3b4a","author":{"email":"5baf40bea4aac2d78f51a81218e1428d32fa1839@intel.com","name":"zhichao.li"},"message":"[SPARK-7717] [WEBUI] Only showing total memory and cores for alive workers\n\nAuthor: zhichao.li <zhichao.li@intel.com>\n\nCloses #6317 from zhichao-li/workers and squashes the following commits:\n\nd68bf11 [zhichao.li] change prefix\n99b6768 [zhichao.li] remove extra space and add 'Alive' prefix\n1e8eb06 [zhichao.li] only showing alive workers","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/2b35c99c7e73d22e82aef90b675709ae7f8d3b4a"},{"sha":"3ab71eb9d5e3fe21af7720421eafa51f6da9b63f","author":{"email":"d830bbc0ff8d4f6be5b5bfec1999946e5bc7c67a@gmail.com","name":"Taka Shinagawa"},"message":"[DOCS] [MINOR] Update for the Hadoop versions table with hadoop-2.6\n\nUpdated the doc for the hadoop-2.6 profile, which is new to Spark 1.4\n\nAuthor: Taka Shinagawa <taka.epsilon@gmail.com>\n\nCloses #6450 from mrt/docfix2 and squashes the following commits:\n\ndb1c43b [Taka Shinagawa] Updated the hadoop versions for hadoop-2.6 profile\n323710e [Taka Shinagawa] The hadoop-2.6 profile is added to the Hadoop versions table","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/3ab71eb9d5e3fe21af7720421eafa51f6da9b63f"},{"sha":"d34b43bd5964e1feb03a17937de87a3f718806a5","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Closes #4685","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/d34b43bd5964e1feb03a17937de87a3f718806a5"},{"sha":"6e3f0c7810a6721698b0ed51cfbd41a0cd07a4a3","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-7849] [SQL] [Docs] Updates SQL programming guide for 1.4\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #6520 from liancheng/spark-7849 and squashes the following commits:\n\n705264b [Cheng Lian] Updates SQL programming guide for 1.4","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/6e3f0c7810a6721698b0ed51cfbd41a0cd07a4a3"},{"sha":"7716a5a1ec8ff8dc24e0146f8ead2f51da6512ad","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Updated SQL programming guide's Hive connectivity section.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/7716a5a1ec8ff8dc24e0146f8ead2f51da6512ad"},{"sha":"a6430028ecd7a6130f1eb15af9ec00e242c46725","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-7855] Move bypassMergeSort-handling from ExternalSorter to own component\n\nSpark's `ExternalSorter` writes shuffle output files during sort-based shuffle. Sort-shuffle contains a configuration, `spark.shuffle.sort.bypassMergeThreshold`, which causes ExternalSorter to skip sorting and merging and simply write separate files per partition, which are then concatenated together to form the final map output file.\n\nThe code paths used during this bypass are almost completely separate from ExternalSorter's other code paths, so refactoring them into a separate file can significantly simplify the code.\n\nIn addition to re-arranging code, this patch deletes a bunch of dead code.  The main entry point into ExternalSorter is `insertAll()` and in SPARK-4479 / #3422 this method was modified to completely bypass in-memory buffering of records when `bypassMergeSort` takes effect. As a result, some of the spilling and merging code paths will no longer be called when `bypassMergeSort` is used, so we should be able to safely remove that code.\n\nThere's an open JIRA ([SPARK-6026](https://issues.apache.org/jira/browse/SPARK-6026)) for removing the `bypassMergeThreshold` parameter and code paths; I have not done that here, but the changes in this patch will make removing that parameter significantly easier if we ever decide to do that.\n\nThis patch also makes several improvements to shuffle-related tests and adds more defensive checks to certain shuffle classes:\n\n- DiskBlockObjectWriter now throws an exception if `fileSegment()` is called before `commitAndClose()` has been called.\n- DiskBlockObjectWriter's close methods are now idempotent, so calling any of the close methods twice in a row will no longer result in incorrect shuffle write metrics changes.  Calling `revertPartialWritesAndClose()` on a closed DiskBlockObjectWriter now has no effect (before, it might mess up the metrics).\n- The end-to-end shuffle record count metrics tests have been moved from InputOutputMetricsSuite to ShuffleSuite.  This means that these tests will now be run against all shuffle implementations rather than just the default shuffle configuration.\n- The end-to-end metrics tests now include a test of a job which performs aggregation in the shuffle.\n- Our tests now check that `shuffleBytesWritten == totalShuffleBytesRead`.\n- FileSegment now throws IllegalArgumentException if it is constructed with a negative length or offset.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #6397 from JoshRosen/external-sorter-bypass-cleanup and squashes the following commits:\n\nbf3f3f6 [Josh Rosen] Merge remote-tracking branch 'origin/master' into external-sorter-bypass-cleanup\n8b216c4 [Josh Rosen] Guard against negative offsets and lengths in FileSegment\n03f35a4 [Josh Rosen] Minor fix to cleanup logic.\nb5cc35b [Josh Rosen] Move shuffle metrics tests to ShuffleSuite.\n8b8fb9e [Josh Rosen] Add more tests + defensive programming to DiskBlockObjectWriter.\n16564eb [Josh Rosen] Guard against calling fileSegment() before commitAndClose() has been called.\n96811b4 [Josh Rosen] Remove confusing taskMetrics.shuffleWriteMetrics() optional call\n8522b6a [Josh Rosen] Do not perform a map-side sort unless we're also doing map-side aggregation\n08e40f3 [Josh Rosen] Remove excessively clever (and wrong) implementation of newBuffer()\nd7f9938 [Josh Rosen] Add missing overrides; fix compilation\n71d76ff [Josh Rosen] Update Javadoc\nbf0d98f [Josh Rosen] Add comment to clarify confusing factory code\n5197f73 [Josh Rosen] Add missing private[this]\n30ef2c8 [Josh Rosen] Convert BypassMergeSortShuffleWriter to Java\nbc1a820 [Josh Rosen] Fix bug when aggregator is used but map-side combine is disabled\n0d3dcc0 [Josh Rosen] Remove unnecessary overloaded methods\n25b964f [Josh Rosen] Rename SortShuffleSorter to SortShuffleFileWriter\n0d9848c [Josh Rosen] Make it more clear that curWriteMetrics is now only used for spill metrics\n7af7aea [Josh Rosen] Combine spill() and spillToMergeableFile()\n6320112 [Josh Rosen] Add missing negation in deletion success check.\nd267e0d [Josh Rosen] Fix style issue\n7f15f7b [Josh Rosen] Back out extra cleanup-handling code, since this is already covered in stop()\n25aa3bd [Josh Rosen] Make sure to delete outputFile after errors.\n931ca68 [Josh Rosen] Refactor tests.\n6a35716 [Josh Rosen] Refactor logic for deciding when to bypass\n4b03539 [Josh Rosen] Move conf prior to first use\n1265b25 [Josh Rosen] Fix some style errors and comments.\n02355ef [Josh Rosen] More simplification\nd4cb536 [Josh Rosen] Delete more unused code\nbb96678 [Josh Rosen] Add missing interface file\nb6cc1eb [Josh Rosen] Realize that bypass never buffers; proceed to delete tons of code\n6185ee2 [Josh Rosen] WIP towards moving bypass code into own file.\n8d0678c [Josh Rosen] Move diskBytesSpilled getter next to variable\n19bccd6 [Josh Rosen] Remove duplicated buffer creation code.\n18959bb [Josh Rosen] Move comparator methods closer together.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/a6430028ecd7a6130f1eb15af9ec00e242c46725"},{"sha":"1617363fbb9b22a2eb09e7bab98c8d05f9508761","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-7918] [MLLIB] MLlib Python doc parity check for evaluation and feature\n\nCheck then make the MLlib Python evaluation and feature doc to be as complete as the Scala doc.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #6461 from yanboliang/spark-7918 and squashes the following commits:\n\n940e3f1 [Yanbo Liang] truncate too long line and remove extra sparse\na80ae58 [Yanbo Liang] MLlib Python doc parity check for evaluation and feature","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/1617363fbb9b22a2eb09e7bab98c8d05f9508761"}]}