{"push_id":781899993,"size":1000,"distinct_size":2228,"ref":"refs/heads/master","head":"eafe37236c56b41ab32adf6610743e1e9a0f230f","before":"47bf406d608c4777f5f383ba439608f673034a1d","commits":[{"sha":"0772026c2fc88aa85423034006b758f6ff0cc2ed","author":{"email":"997912b560241ee806420b4c197dcfa185c3525f@gmail.com","name":"rahulpalamuttam"},"message":"[SPARK-8923] [DOCUMENTATION, MLLIB] Add @since tags to mllib.fpm\n\nAuthor: rahulpalamuttam <rahulpalamut@gmail.com>\n\nCloses #7341 from rahulpalamuttam/TaggingMLlibfpm and squashes the following commits:\n\nbef2843 [rahulpalamuttam] fix @since tags in mmlib.fpm\ncd86252 [rahulpalamuttam] Add @since tags to mllib.fpm","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/0772026c2fc88aa85423034006b758f6ff0cc2ed"},{"sha":"fb8807c9b04f27467b36fc9d0177ef92dd012670","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-7078] [SPARK-7079] Binary processing sort for Spark SQL\n\nThis patch adds a cache-friendly external sorter which operates on serialized bytes and uses this sorter to implement a new sort operator for Spark SQL and DataFrames.\n\n### Overview of the new sorter\n\nThe new sorter design is inspired by [Alphasort](http://research.microsoft.com/pubs/68249/alphasort.doc) and implements a key-prefix optimization in order to improve the cache friendliness of the sort.  In naive sort implementations, the sorting algorithm operates on an array of record pointers.  To compare two records for ordering, the sorter must dereference these pointers, which likely involves random memory access, then compare the objects themselves.\n\n![image](https://cloud.githubusercontent.com/assets/50748/8611390/3b1402ae-2675-11e5-8308-1a10bf347e6e.png)\n\nIn a key-prefix sort, the sort operates on an array which stores the record pointer alongside a prefix of the record's key. When comparing two records for ordering, the sorter first compares the the stored key prefixes. If the ordering can be determined from the key prefixes (i.e. the prefixes are unequal), then the sort can avoid directly comparing the records, avoiding random memory accesses and full record comparisons. For example, if we're sorting a list of strings then we can store the first 8 bytes of the UTF-8 encoded string as the key-prefix and can perform unsigned byte-at-a-time comparisons to determine the ordering of strings based on their prefixes, only resorting to full comparisons for strings that share a common prefix.  In cases where the sort key can fit entirely in the space allotted for the key prefix (e.g. the sorting key is an integer), we completely avoid direct record comparison.\n\nIn this patch's implementation of key-prefix sorting, our sorter's internal array stores a 64-bit long and 64-bit pointer for each record being sorted. The key prefixes are generated by the user when inserting records into the sorter, which uses a user-defined comparison function for comparing them.  The `PrefixComparators` object implements a set of comparators for many common types, including primitive numeric types and UTF-8 strings.\n\nThe actual sorting is implemented by `UnsafeInMemorySorter`.  Most consumers will not use this directly, but instead will use `UnsafeExternalSorter`, a class which implements a sort that can spill to disk in response to memory pressure.  Internally, `UnsafeExternalSorter` creates `UnsafeInMemorySorters` to perform sorting and uses `UnsafeSortSpillReader/Writer` to spill and read back runs of sorted records and `UnsafeSortSpillMerger` to merge multiple sorted spills into a single sorted iterator.  This external sorter integrates with Spark's existing ShuffleMemoryManager for controlling spilling.\n\nMany parts of this sorter's design are based on / copied from the more specialized external sort implementation that I designed for the new UnsafeShuffleManager write path; see #5868 for more details on that patch.\n\n### Sorting rows in Spark SQL\n\nFor now, `UnsafeExternalSorter` is only used by Spark SQL, which uses it to implement a new sort operator, `UnsafeExternalSort`.  This sort operator uses a SQL-specific class called `UnsafeExternalRowSorter` that configures an `UnsafeExternalSorter` to use prefix generators and comparators that operate on rows encoded in the UnsafeRow format that was designed for Project Tungsten.\n\nI used some interesting unit-testing techniques to test this patch's SQL-specific components.  `UnsafeExternalSortSuite` uses the SQL random data generators introduced in #7176 to test the UnsafeSort operator with all atomic types both with and without nullability and in both ascending and descending sort orders.  `PrefixComparatorsSuite` contains a cool use of ScalaCheck + ScalaTest's `GeneratorDrivenPropertyChecks` in order to test UTF8String prefix comparison.\n\n### Misc. additional improvements made in this patch\n\nThis patch made several miscellaneous improvements to related code in Spark SQL:\n\n- The logic for selecting physical sort operator implementations, which was partially duplicated in both `Exchange` and `SparkStrategies, has now been consolidated into a `getSortOperator()` helper function in `SparkStrategies`.\n- The `SparkPlanTest` unit testing helper trait has been extended with new methods for comparing the output produced by two different physical plans. This makes it easy to write tests which assert that two physical operator implementations should produce the same output.  I also added a method for disabling the implicit sorting of outputs prior to comparing them, a change which is necessary in order to be able to write proper SparkPlan tests for sort operators.\n\n### Tasks deferred to followup patches\n\nWhile most of this patch's features are reasonably well-tested and complete, there are a number of tasks that are intentionally being deferred to followup patches:\n\n- Add tests which mock the ShuffleMemoryManager to check that memory pressure properly triggers spilling (there are examples of this type of test in #5868).\n- Add tests to ensure that spill files are properly cleaned up after errors.  I'd like to do this in the context of a patch which introduces more general metrics for ensuring proper cleanup of tasks' temporary files; see https://issues.apache.org/jira/browse/SPARK-8966 for more details.\n- Metrics integration: there are some open questions regarding how to track / report spill metrics for non-shuffle operations, so I've deferred most of the IO / shuffle metrics integration for now.\n- Performance profiling.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/6444)\n<!-- Reviewable:end -->\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #6444 from JoshRosen/sql-external-sort and squashes the following commits:\n\n6beb467 [Josh Rosen] Remove a bunch of overloaded methods to avoid default args. issue\n2bbac9c [Josh Rosen] Merge remote-tracking branch 'origin/master' into sql-external-sort\n35dad9f [Josh Rosen] Make sortAnswers = false the default in SparkPlanTest\n5135200 [Josh Rosen] Fix spill reading for large rows; add test\n2f48777 [Josh Rosen] Add test and fix bug for sorting empty arrays\nd1e28bc [Josh Rosen] Merge remote-tracking branch 'origin/master' into sql-external-sort\ncd05866 [Josh Rosen] Fix scalastyle\n3947fc1 [Josh Rosen] Merge remote-tracking branch 'origin/master' into sql-external-sort\nd13ac55 [Josh Rosen] Hacky approach to copying of UnsafeRows for sort followed by limit.\n845bea3 [Josh Rosen] Remove unnecessary zeroing of row conversion buffer\nc56ec18 [Josh Rosen] Clean up final row copying code.\nd31f180 [Josh Rosen] Re-enable NullType sorting test now that SPARK-8868 is fixed\n844f4ca [Josh Rosen] Merge remote-tracking branch 'origin/master' into sql-external-sort\n293f109 [Josh Rosen] Add missing license header.\nf99a612 [Josh Rosen] Fix bugs in string prefix comparison.\n9d00afc [Josh Rosen] Clean up prefix comparators for integral types\n88aff18 [Josh Rosen] NULL_PREFIX has to be negative infinity for floating point types\n613e16f [Josh Rosen] Test with larger data.\n1d7ffaa [Josh Rosen] Somewhat hacky fix for descending sorts\n08701e7 [Josh Rosen] Fix prefix comparison of null primitives.\nb86e684 [Josh Rosen] Set global = true in UnsafeExternalSortSuite.\n1c7bad8 [Josh Rosen] Make sorting of answers explicit in SparkPlanTest.checkAnswer().\nb81a920 [Josh Rosen] Temporarily enable only the passing sort tests\n5d6109d [Josh Rosen] Fix inconsistent handling / encoding of record lengths.\n87b6ed9 [Josh Rosen] Fix critical issues in test which led to false negatives.\n8d7fbe7 [Josh Rosen] Fixes to multiple spilling-related bugs.\n82e21c1 [Josh Rosen] Force spilling in UnsafeExternalSortSuite.\n88b72db [Josh Rosen] Test ascending and descending sort orders.\nf27be09 [Josh Rosen] Fix tests by binding attributes.\n0a79d39 [Josh Rosen] Revert \"Undo part of a SparkPlanTest change in #7162 that broke my test.\"\n7c3c864 [Josh Rosen] Undo part of a SparkPlanTest change in #7162 that broke my test.\n9969c14 [Josh Rosen] Merge remote-tracking branch 'origin/master' into sql-external-sort\n5822e6f [Josh Rosen] Fix test compilation issue\n939f824 [Josh Rosen] Remove code gen experiment.\n0dfe919 [Josh Rosen] Implement prefix sort for strings (albeit inefficiently).\n66a813e [Josh Rosen] Prefix comparators for float and double\nb310c88 [Josh Rosen] Integrate prefix comparators for Int and Long (others coming soon)\n95058d9 [Josh Rosen] Add missing SortPrefixUtils file\n4c37ba6 [Josh Rosen] Add tests for sorting on all primitive types.\n6890863 [Josh Rosen] Fix memory leak on empty inputs.\nd246e29 [Josh Rosen] Fix consideration of column types when choosing sort implementation.\n6b156fb [Josh Rosen] Some WIP work on prefix comparison.\n7f875f9 [Josh Rosen] Commit failing test demonstrating bug in handling objects in spills\n41b8881 [Josh Rosen] Get UnsafeInMemorySorterSuite to pass (WIP)\n90c2b6a [Josh Rosen] Update test name\n6d6a1e6 [Josh Rosen] Centralize logic for picking sort operator implementations\n9869ec2 [Josh Rosen] Clean up Exchange code a bit\n82bb0ec [Josh Rosen] Fix IntelliJ complaint due to negated if condition\n1db845a [Josh Rosen] Many more changes to harmonize with shuffle sorter\nebf9eea [Josh Rosen] Harmonization with shuffle's unsafe sorter\n206bfa2 [Josh Rosen] Add some missing newlines at the ends of files\n26c8931 [Josh Rosen] Back out some Hive changes that aren't needed anymore\n62f0bb8 [Josh Rosen] Update to reflect SparkPlanTest changes\n21d7d93 [Josh Rosen] Back out of BlockObjectWriter change\n7eafecf [Josh Rosen] Port test to SparkPlanTest\nd468a88 [Josh Rosen] Update for InternalRow refactoring\n269cf86 [Josh Rosen] Back out SMJ operator change; isolate changes to selection of sort op.\n1b841ca [Josh Rosen] WIP towards copying\nb420a71 [Josh Rosen] Move most of the existing SMJ code into Java.\ndfdb93f [Josh Rosen] SparkFunSuite change\n73cc761 [Josh Rosen] Fix whitespace\n9cc98f5 [Josh Rosen] Move more code to Java; fix bugs in UnsafeRowConverter length type.\nc8792de [Josh Rosen] Remove some debug logging\ndda6752 [Josh Rosen] Commit some missing code from an old git stash.\n58f36d0 [Josh Rosen] Merge in a sketch of a unit test for the new sorter (now failing).\n2bd8c9a [Josh Rosen] Import my original tests and get them to pass.\nd5d3106 [Josh Rosen] WIP towards external sorter for Spark SQL.","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/fb8807c9b04f27467b36fc9d0177ef92dd012670"},{"sha":"857e325f304a02b1cab2463c1e15af7627a600bd","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-8990] [SQL] SPARK-8990 DataFrameReader.parquet() should respect user specified options\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7347 from liancheng/spark-8990 and squashes the following commits:\n\n045698c [Cheng Lian] SPARK-8990 DataFrameReader.parquet() should respect user specified options","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/857e325f304a02b1cab2463c1e15af7627a600bd"},{"sha":"b6fc0adf6874fc26ab27cdaa8ebb28474c0681f0","author":{"email":"399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","name":"Davies Liu"},"message":"add inline comment for python tests","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/b6fc0adf6874fc26ab27cdaa8ebb28474c0681f0"},{"sha":"33630883685eafcc3ee4521ea8363be342f6e6b4","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-8961] [SQL] Makes BaseWriterContainer.outputWriterForRow accepts InternalRow instead of Row\n\nThis is a follow-up of [SPARK-8888] [1], which also aims to optimize writing dynamic partitions.\n\nThree more changes can be made here:\n\n1. Using `InternalRow` instead of `Row` in `BaseWriterContainer.outputWriterForRow`\n2. Using `Cast` expressions to convert partition columns to strings, so that we can leverage code generation.\n3. Replacing the FP-style `zip` and `map` calls with a faster imperative `while` loop.\n\n[1]: https://issues.apache.org/jira/browse/SPARK-8888\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7331 from liancheng/spark-8961 and squashes the following commits:\n\nb5ab9ae [Cheng Lian] Casts Java iterator to Scala iterator explicitly\n719e63b [Cheng Lian] Makes BaseWriterContainer.outputWriterForRow accepts InternalRow instead of Row","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/33630883685eafcc3ee4521ea8363be342f6e6b4"},{"sha":"6e1c7e2798bb0b7b120e28a07c5d70fa162dd724","author":{"email":"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@megatron.me.uk","name":"Scott Taylor"},"message":"[SPARK-7735] [PYSPARK] Raise Exception on non-zero exit from pipe commands\n\nThis will allow problems with piped commands to be detected.\nThis will also allow tasks to be retried where errors are rare (such as network problems in piped commands).\n\nAuthor: Scott Taylor <github@megatron.me.uk>\n\nCloses #6262 from megatron-me-uk/patch-2 and squashes the following commits:\n\n04ae1d5 [Scott Taylor] Remove spurious empty line\n98fa101 [Scott Taylor] fix blank line style error\n574b564 [Scott Taylor] Merge pull request #2 from megatron-me-uk/patch-4\n0c1e762 [Scott Taylor] Update rdd pipe method for checkCode\nab9a2e1 [Scott Taylor] Update rdd pipe tests for checkCode\neb4801c [Scott Taylor] fix fail_condition\nb0ac3a4 [Scott Taylor] Merge pull request #1 from megatron-me-uk/megatron-me-uk-patch-1\na307d13 [Scott Taylor] update rdd tests to test pipe modes\n34fcdc3 [Scott Taylor] add optional argument 'mode' for rdd.pipe\na0c0161 [Scott Taylor] fix generator issue\n8a9ef9c [Scott Taylor] make check_return_code an iterator\n0486ae3 [Scott Taylor] style fixes\n8ed89a6 [Scott Taylor] Chain generators to prevent potential deadlock\n4153b02 [Scott Taylor] fix list.sort returns None\n491d3fc [Scott Taylor] Pass a function handle to assertRaises\n3344a21 [Scott Taylor] wrap assertRaises with QuietTest\n3ab8c7a [Scott Taylor] remove whitespace for style\ncc1a73d [Scott Taylor] fix style issues in pipe test\n8db4073 [Scott Taylor] Add a test for rdd pipe functions\n1b3dc4e [Scott Taylor] fix missing space around operator style\n0974f98 [Scott Taylor] add space between words in multiline string\n45f4977 [Scott Taylor] fix line too long style error\n5745d85 [Scott Taylor] Remove space to fix style\nf552d49 [Scott Taylor] Catch non-zero exit from pipe commands","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/6e1c7e2798bb0b7b120e28a07c5d70fa162dd724"},{"sha":"9c5075775d741eacbeeb2df77ea30611356b6e1a","author":{"email":"dd19fa80624ab884ac33cf50a82d9bdcaeb1d7b3@cloudera.com","name":"jose.cambronero"},"message":"[SPARK-8598] [MLLIB] Implementation of 1-sample, two-sided, Kolmogorov Smirnov Test for RDDs\n\nThis contribution is my original work and I license it to the project under it's open source license.\n\nAuthor: jose.cambronero <jose.cambronero@cloudera.com>\n\nCloses #6994 from josepablocam/master and squashes the following commits:\n\nbbb30b1 [jose.cambronero] renamed KSTestResult to KolmogorovSmirnovTestResult, to stay consistent with method name\n0d0c201 [jose.cambronero] kstTest -> kolmogorovSmirnovTest in statistics.md\n1f56371 [jose.cambronero] changed ksTest in public API to kolmogorovSmirnovTest for clarity\na48ae7b [jose.cambronero] refactor code to account for serializable RealDistribution. Reuse testOneSample( _, cdf)\n1bb44bd [jose.cambronero]  style and doc changes. Factored out ks test into 2 separate tests\n2ec2aa6 [jose.cambronero] initialize to stdnormal when no params passed (and log). Change unit tests to approximate equivalence rather than strict\na4bc0c7 [jose.cambronero] changed ksTest(data, distName) to ksTest(data, distName, params*) after api discussions. Changed tests and docs accordingly\n7e66f57 [jose.cambronero] copied implementation note to public api docs, and added @see for links to wiki info\ne760ebd [jose.cambronero] line length changes to fit style check\n3288e42 [jose.cambronero] addressed style changes, correctness change to simpler approach, and fixed edge case for foldLeft in searchOneSampleCandidates when a partition is empty\n9026895 [jose.cambronero] addressed style changes, correctness change to simpler approach, and fixed edge case for foldLeft in searchOneSampleCandidates when a partition is empty\n1226b30 [jose.cambronero] reindent multi-line lambdas, prior intepretation of style guide was wrong on my part\n9c0f1af [jose.cambronero] additional style changes incorporated and added documentation to mllib statistics docs\n3f81ad2 [jose.cambronero] renamed ks1 sample test for clarity\n992293b [jose.cambronero] Style changes as per comments and added implementation note explaining the distributed approach.\n6a4784f [jose.cambronero] specified what distributions are available for the convenience method ksTest(data, name) (solely standard normal)\n4b8ba61 [jose.cambronero] fixed off by 1/N in cases when post-constant adjustment ecdf is above cdf, but prior to adj it was below\n0b5e8ec [jose.cambronero] changed KS one sample test to perform just 1 distributed pass (in addition to the sorting pass), operates on each partition separately. Implementation of Sandy Ryza's algorithm\n16b5c4c [jose.cambronero] renamed dat to data and eliminated recalc of RDD size by sharing as argument between empirical and evalOneSampleP\nc18dc66 [jose.cambronero] removed ksTestOpt from API and changed comments in HypothesisTestSuite accordingly\nf6951b6 [jose.cambronero] changed style and some comments based on feedback from pull request\nb9cff3a [jose.cambronero] made small changes to pass style check\nce8e9a1 [jose.cambronero] added kstest testing in HypothesisTestSuite\n4da189b [jose.cambronero] added user facing ks test functions\nc659ea1 [jose.cambronero] created KS test class\n13dfe4d [jose.cambronero] created test result class for ks test","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/9c5075775d741eacbeeb2df77ea30611356b6e1a"},{"sha":"7f6be1f24d4f2fcb3d3ec181b5abf241709a8b6d","author":{"email":"601182debbbfc18340251bd96b76c6a8daae6a23@huawei.com","name":"zhangjiajin"},"message":"[SPARK-6487] [MLLIB] Add sequential pattern mining algorithm PrefixSpan to Spark MLlib\n\nAdd parallel PrefixSpan algorithm and test file.\nSupport non-temporal sequences.\n\nAuthor: zhangjiajin <zhangjiajin@huawei.com>\nAuthor: zhang jiajin <zhangjiajin@huawei.com>\n\nCloses #7258 from zhangjiajin/master and squashes the following commits:\n\nca9c4c8 [zhangjiajin] Modified the code according to the review comments.\n574e56c [zhangjiajin] Add new object LocalPrefixSpan, and do some optimization.\nba5df34 [zhangjiajin] Fix a Scala style error.\n4c60fb3 [zhangjiajin] Fix some Scala style errors.\n1dd33ad [zhangjiajin] Modified the code according to the review comments.\n89bc368 [zhangjiajin] Fixed a Scala style error.\na2eb14c [zhang jiajin] Delete PrefixspanSuite.scala\n951fd42 [zhang jiajin] Delete Prefixspan.scala\n575995f [zhangjiajin] Modified the code according to the review comments.\n91fd7e6 [zhangjiajin] Add new algorithm PrefixSpan and test file.","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/7f6be1f24d4f2fcb3d3ec181b5abf241709a8b6d"},{"sha":"0c5207c66db8cf76950d53404abd43e085f8d45b","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-8994] [ML] tiny cleanups to Params, Pipeline\n\nMade default impl of Params.validateParams empty\nCC mengxr\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #7349 from jkbradley/pipeline-small-cleanups and squashes the following commits:\n\n4e0f013 [Joseph K. Bradley] small cleanups after SPARK-5956","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/0c5207c66db8cf76950d53404abd43e085f8d45b"},{"sha":"c472eb17ae7f0910f304e414ea5ccbb77a9e153a","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-8970][SQL] remove unnecessary abstraction for ExtractValue\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7339 from cloud-fan/minor and squashes the following commits:\n\n84a2128 [Wenchen Fan] remove unapply\n6a37c12 [Wenchen Fan] remove unnecessary abstraction for ExtractValue","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/c472eb17ae7f0910f304e414ea5ccbb77a9e153a"},{"sha":"30090884f96efde72d9653f2253070b68f87782c","author":{"email":"284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","name":"Kay Ousterhout"},"message":"[SPARK-8880] Fix confusing Stage.attemptId member variable\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #7275 from kayousterhout/SPARK-8880 and squashes the following commits:\n\n3e9ce7c [Kay Ousterhout] Added missing return type\ne150278 [Kay Ousterhout] [SPARK-8880] Fix confusing Stage.attemptId member variable","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/30090884f96efde72d9653f2253070b68f87782c"},{"sha":"20b474335c68c644150fdc8443a2d0d2dad5e27d","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-9006] [PYSPARK] fix microsecond loss in Python 3\n\nIt may loss a microsecond if using timestamp as float, should be `int` instead.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #7363 from davies/fix_microsecond and squashes the following commits:\n\n36f6007 [Davies Liu] fix microsecond loss in Python 3","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/20b474335c68c644150fdc8443a2d0d2dad5e27d"},{"sha":"92540d22e45f9300f413f520a1770e9f36cfa833","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-8203] [SPARK-8204] [SQL] conditional function: least/greatest\n\nchenghao-intel zhichao-li qiansl127\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #6851 from adrian-wang/udflg and squashes the following commits:\n\n0f1bff2 [Daoyuan Wang] address comments from davis\n7a6bdbb [Daoyuan Wang] add '.' for hex()\nc1f6824 [Daoyuan Wang] add codegen, test for all types\nec625b0 [Daoyuan Wang] conditional function: least/greatest","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/92540d22e45f9300f413f520a1770e9f36cfa833"},{"sha":"6b89943834a8d9d5d0ecfd97efcc10056d08532a","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-8944][SQL] Support casting between IntervalType and StringType\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7355 from cloud-fan/fromString and squashes the following commits:\n\n3bbb9d6 [Wenchen Fan] fix code gen\n7dab957 [Wenchen Fan] naming fix\n0fbbe19 [Wenchen Fan] address comments\nac1f3d1 [Wenchen Fan] Support casting between IntervalType and StringType","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/6b89943834a8d9d5d0ecfd97efcc10056d08532a"},{"sha":"a5bc803b7271028e34de1548d55b80ecfb812a7b","author":{"email":"e48675cda3579d10b7ca3cc7e15468574106fa2e@gmail.com","name":"Vincent D. Warmerdam"},"message":"[SPARK-8596] Add module for rstudio link to spark\n\nshivaram, added module for rstudio install\n\nAuthor: Vincent D. Warmerdam <vincentwarmerdam@gmail.com>\n\nCloses #7366 from koaning/rstudio-install and squashes the following commits:\n\ne47c2da [Vincent D. Warmerdam] added rstudio module","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/a5bc803b7271028e34de1548d55b80ecfb812a7b"},{"sha":"7f487c8bde14dbdd244a3493ad11a129ef2bb327","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-6797] [SPARKR] Add support for YARN cluster mode.\n\nThis PR enables SparkR to dynamically ship the SparkR binary package to the AM node in YARN cluster mode, thus it is no longer required that the SparkR package be installed on each worker node.\n\nThis PR uses the JDK jar tool to package the SparkR package, because jar is thought to be available on both Linux/Windows platforms where JDK has been installed.\n\nThis PR does not address the R worker involved in RDD API. Will address it in a separate JIRA issue.\n\nThis PR does not address SBT build. SparkR installation and packaging by SBT will be addressed in a separate JIRA issue.\n\nR/install-dev.bat is not tested. shivaram , Could you help to test it?\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #6743 from sun-rui/SPARK-6797 and squashes the following commits:\n\nca63c86 [Sun Rui] Adjust MimaExcludes after rebase.\n7313374 [Sun Rui] Fix unit test errors.\n72695fb [Sun Rui] Fix unit test failures.\n193882f [Sun Rui] Fix Mima test error.\nfe25a33 [Sun Rui] Fix Mima test error.\n35ecfa3 [Sun Rui] Fix comments.\nc38a005 [Sun Rui] Unzipped SparkR binary package is still required for standalone and Mesos modes.\nb05340c [Sun Rui] Fix scala style.\n2ca5048 [Sun Rui] Fix comments.\n1acefd1 [Sun Rui] Fix scala style.\n0aa1e97 [Sun Rui] Fix scala style.\n41d4f17 [Sun Rui] Add support for locating SparkR package for R workers required by RDD APIs.\n49ff948 [Sun Rui] Invoke jar.exe with full path in install-dev.bat.\n7b916c5 [Sun Rui] Use 'rem' consistently.\n3bed438 [Sun Rui] Add a comment.\n681afb0 [Sun Rui] Fix a bug that RRunner does not handle client deployment modes.\ncedfbe2 [Sun Rui] [SPARK-6797][SPARKR] Add support for YARN cluster mode.","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/7f487c8bde14dbdd244a3493ad11a129ef2bb327"},{"sha":"9b62e9375f032548d386aec7468e3d0f7c6da7b2","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-8706] [PYSPARK] [PROJECT INFRA] Add pylint checks to PySpark\n\nThis adds Pylint checks to PySpark.\n\nFor now this lazy installs using easy_install to /dev/pylint (similar to the pep8 script).\nWe still need to figure out what rules to be allowed.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #7241 from MechCoder/pylint and squashes the following commits:\n\n8496834 [MechCoder] Silence warnings and make pylint tests fail to check if it works in jenkins\n57393a3 [MechCoder] undefined-variable\na8e2547 [MechCoder] Minor changes\n7753810 [MechCoder] remove trailing whitespace\n75c5d2b [MechCoder] Remove blacklisted arguments and pointless statements check\n6bde250 [MechCoder] Disable all checks for now\n3464666 [MechCoder] Add pylint configuration file\nd28109f [MechCoder] [SPARK-8706] [PySpark] [Project infra] Add pylint checks to PySpark","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/9b62e9375f032548d386aec7468e3d0f7c6da7b2"},{"sha":"5ca26fb64de99fa414dc59ce4cf29a0171894793","author":{"email":"dbe0629cf0118f7dffec38856cad0ec7ac880251@intel.com","name":"Carson Wang"},"message":"[SPARK-8950] [WEBUI] Correct the calculation of SchedulerDelay in StagePage\n\nIn StagePage, the SchedulerDelay is calculated as totalExecutionTime - executorRunTime - executorOverhead - gettingResultTime.\nBut the totalExecutionTime is calculated in the way that doesn't include the gettingResultTime.\n\nAuthor: Carson Wang <carson.wang@intel.com>\n\nCloses #7319 from carsonwang/SchedulerDelayTime and squashes the following commits:\n\nf66fb6e [Carson Wang] Update the code style\n7d971ae [Carson Wang] Correct the calculation of SchedulerDelay","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/5ca26fb64de99fa414dc59ce4cf29a0171894793"},{"sha":"79c35826e626805c5df776b3d22ed37829d239b6","author":{"email":"399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","name":"Davies Liu"},"message":"Revert \"[SPARK-8706] [PYSPARK] [PROJECT INFRA] Add pylint checks to PySpark\"\n\nThis reverts commit 9b62e9375f032548d386aec7468e3d0f7c6da7b2.","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/79c35826e626805c5df776b3d22ed37829d239b6"},{"sha":"5c41691fe4da0017ccc7e372e3f655dc1c815349","author":{"email":"63b52e08611a2995f34f1279c715920731fa0f6f@users.noreply.github.com","name":"yongtang"},"message":"[SPARK-8954] [BUILD] Remove unneeded deb repository from Dockerfile to fix build error in docker.\n\n[SPARK-8954] [Build]\n1. Remove unneeded deb repository from Dockerfile to fix build error in docker.\n2. Remove unneeded /var/lib/apt/lists/* after install to reduce the docker image size (by ~30MB).\n\nAuthor: yongtang <yongtang@users.noreply.github.com>\n\nCloses #7346 from yongtang/SPARK-8954 and squashes the following commits:\n\n36024a1 [yongtang] [SPARK-8954] [Build] Remove unneeded /var/lib/apt/lists/* after install to reduce the docker image size (by ~30MB)\n7084941 [yongtang] [SPARK-8954] [Build] Remove unneeded deb repository from Dockerfile to fix build error in docker.","distinct":true,"url":"https://api.github.com/repos/alexrovner/spark/commits/5c41691fe4da0017ccc7e372e3f655dc1c815349"}]}