{"push_id":746321392,"size":22,"distinct_size":22,"ref":"refs/heads/master","head":"523feeb64205cafd2087b7f8bbbd4b68e3690831","before":"da918b0ca33ab22ae582ef4cd6ab21c0e68b6766","commits":[{"sha":"0722f43316fc7ed0c1308b0f9d6d15f0c22ed56f","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-7937][SQL] Support comparison on StructType\n\nThis brings #6519 up-to-date with master branch.\n\nCloses #6519.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7877 from rxin/sort-struct and squashes the following commits:\n\n4968231 [Reynold Xin] Minor fixes.\n2537813 [Reynold Xin] Merge branch 'compare_named_struct' of github.com:viirya/spark-1 into sort-struct\nd2ba8ad [Liang-Chi Hsieh] Remove unused import.\n3a3f40e [Liang-Chi Hsieh] Don't need to add compare to InternalRow because we can use RowOrdering.\ndae6aad [Liang-Chi Hsieh] Fix nested struct.\nd5349c7 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into compare_named_struct\n43d4354 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into compare_named_struct\n1f66196 [Liang-Chi Hsieh] Reuse RowOrdering and GenerateOrdering.\nf8b2e9c [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into compare_named_struct\n1187a65 [Liang-Chi Hsieh] Fix scala style.\n9d67f68 [Liang-Chi Hsieh] Fix wrongly merging.\n8f4d775 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into compare_named_struct\n94b27d5 [Liang-Chi Hsieh] Remove test for error on complex type comparison.\n2071693 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into compare_named_struct\n3c142e4 [Liang-Chi Hsieh] Fix scala style.\ncf58dc3 [Liang-Chi Hsieh] Use checkAnswer.\nf651b8d [Liang-Chi Hsieh] Remove Either and move orderings to BinaryComparison to reuse it.\nb6e1009 [Liang-Chi Hsieh] Fix scala style.\n3922b54 [Liang-Chi Hsieh] Support ordering on named_struct.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/0722f43316fc7ed0c1308b0f9d6d15f0c22ed56f"},{"sha":"9d03ad910b853c1e0dcb946b05d48ab36a1ac533","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9543][SQL] Add randomized testing for UnsafeKVExternalSorter.\n\nThe detailed approach is documented in UnsafeKVExternalSorterSuite.testKVSorter(), working as follows:\n\n1. Create input by generating data randomly based on the given key/value schema (which is also randomly drawn from a list of candidate types)\n2. Run UnsafeKVExternalSorter on the generated data\n3. Collect the output from the sorter, and make sure the keys are sorted in ascending order\n4. Sort the input by both key and value, and sort the sorter output also by both key and value. Compare the sorted input and sorted output together to make sure all the key/values match.\n5. Check memory allocation to make sure there is no memory leak.\n\nThere is also a spill flag. When set to true, the sorter will spill probabilistically roughly every 100 records.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7873 from rxin/kvsorter-randomized-test and squashes the following commits:\n\na08c251 [Reynold Xin] Resource cleanup.\n0488b5c [Reynold Xin] [SPARK-9543][SQL] Add randomized testing for UnsafeKVExternalSorter.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/9d03ad910b853c1e0dcb946b05d48ab36a1ac533"},{"sha":"536d2adc12c88ff67b5dde3dec998bcacef50fba","author":{"email":"1bbfbf3ad5c4de88db64d0be3ca037b409d25231@huawei.com","name":"KaiXinXiaoLei"},"message":"[SPARK-9535][SQL][DOCS] Modify document for codegen.\n\n#7142 made codegen enabled by default so let's modify the corresponding documents.\n\nCloses #7142\n\nAuthor: KaiXinXiaoLei <huleilei1@huawei.com>\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #7863 from sarutak/SPARK-9535 and squashes the following commits:\n\n0884424 [Kousuke Saruta] Removed a line which mentioned about the effect of codegen enabled\n3c11af0 [Kousuke Saruta] Merge branch 'sqlconfig' of https://github.com/KaiXinXiaoLei/spark into SPARK-9535\n4ee531d [KaiXinXiaoLei] delete space\n4cfd11d [KaiXinXiaoLei] change spark.sql.planner.externalSort\nd624cf8 [KaiXinXiaoLei] sql config is wrong","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/536d2adc12c88ff67b5dde3dec998bcacef50fba"},{"sha":"30e89111d673776a6b59b11cdb29ab8713ba6f7c","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9546][SQL] Centralize orderable data type checking.\n\nThis pull request creates two isOrderable functions in RowOrdering that can be used to check whether a data type or a sequence of expressions can be used in sorting.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7880 from rxin/SPARK-9546 and squashes the following commits:\n\nf9e322d [Reynold Xin] Fixed tests.\n0439b43 [Reynold Xin] [SPARK-9546][SQL] Centralize orderable data type checking.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/30e89111d673776a6b59b11cdb29ab8713ba6f7c"},{"sha":"114ff926fcd078697c1111279b5cf6173b515865","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-2205] [SQL] Avoid unnecessary exchange operators in multi-way joins\n\nThis PR adds `PartitioningCollection`, which is used to represent the `outputPartitioning` for SparkPlans with multiple children (e.g. `ShuffledHashJoin`). So, a `SparkPlan` can have multiple descriptions of its partitioning schemes. Taking `ShuffledHashJoin` as an example, it has two descriptions of its partitioning schemes, i.e. `left.outputPartitioning` and `right.outputPartitioning`. So when we have a query like `select * from t1 join t2 on (t1.x = t2.x) join t3 on (t2.x = t3.x)` will only have three Exchange operators (when shuffled joins are needed) instead of four.\n\nThe code in this PR was authored by yhuai; I'm opening this PR to factor out this change from #7685, a larger pull request which contains two other optimizations.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/7773)\n<!-- Reviewable:end -->\n\nAuthor: Yin Huai <yhuai@databricks.com>\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #7773 from JoshRosen/multi-way-join-planning-improvements and squashes the following commits:\n\n5c45924 [Josh Rosen] Merge remote-tracking branch 'origin/master' into multi-way-join-planning-improvements\ncd8269b [Josh Rosen] Refactor test to use SQLTestUtils\n2963857 [Yin Huai] Revert unnecessary SqlConf change.\n73913f7 [Yin Huai] Add comments and test. Also, revert the change in ShuffledHashOuterJoin for now.\n4a99204 [Josh Rosen] Delete unrelated expression change\n884ab95 [Josh Rosen] Carve out only SPARK-2205 changes.\n247e5fa [Josh Rosen] Merge remote-tracking branch 'origin/master' into multi-way-join-planning-improvements\nc57a954 [Yin Huai] Bug fix.\nd3d2e64 [Yin Huai] First round of cleanup.\nf9516b0 [Yin Huai] Style\nc6667e7 [Yin Huai] Add PartitioningCollection.\ne616d3b [Yin Huai] wip\n7c2d2d8 [Yin Huai] Bug fix and refactoring.\n69bb072 [Yin Huai] Introduce NullSafeHashPartitioning and NullUnsafePartitioning.\nd5b84c3 [Yin Huai] Do not add unnessary filters.\n2201129 [Yin Huai] Filter out rows that will not be joined in equal joins early.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/114ff926fcd078697c1111279b5cf6173b515865"},{"sha":"4cdd8ecd66769316e8593da7790b84cd867968cd","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9536] [SPARK-9537] [SPARK-9538] [ML] [PYSPARK] ml.classification support raw and probability prediction for PySpark\n\nMake the following ml.classification class support raw and probability prediction for PySpark:\n```scala\nNaiveBayesModel\nDecisionTreeClassifierModel\nLogisticRegressionModel\n```\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #7866 from yanboliang/spark-9536-9537 and squashes the following commits:\n\n2934dab [Yanbo Liang] ml.NaiveBayes, ml.DecisionTreeClassifier and ml.LogisticRegression support probability prediction","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/4cdd8ecd66769316e8593da7790b84cd867968cd"},{"sha":"687c8c37150f4c93f8e57d86bb56321a4891286b","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9372] [SQL] Filter nulls in join keys\n\nThis PR adds an optimization rule, `FilterNullsInJoinKey`, to add `Filter` before join operators to filter out rows having null values for join keys.\n\nThis optimization is guarded by a new SQL conf, `spark.sql.advancedOptimization`.\n\nThe code in this PR was authored by yhuai; I'm opening this PR to factor out this change from #7685, a larger pull request which contains two other optimizations.\n\nAuthor: Yin Huai <yhuai@databricks.com>\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #7768 from JoshRosen/filter-nulls-in-join-key and squashes the following commits:\n\nc02fc3f [Yin Huai] Address Josh's comments.\n0a8e096 [Yin Huai] Update comments.\nea7d5a6 [Yin Huai] Make sure we do not keep adding filters.\nbe88760 [Yin Huai] Make it clear that FilterNullsInJoinKeySuite.scala is used to test FilterNullsInJoinKey.\n8bb39ad [Yin Huai] Fix non-deterministic tests.\n303236b [Josh Rosen] Revert changes that are unrelated to null join key filtering\n40eeece [Josh Rosen] Merge remote-tracking branch 'origin/master' into filter-nulls-in-join-key\nc57a954 [Yin Huai] Bug fix.\nd3d2e64 [Yin Huai] First round of cleanup.\nf9516b0 [Yin Huai] Style\nc6667e7 [Yin Huai] Add PartitioningCollection.\ne616d3b [Yin Huai] wip\n7c2d2d8 [Yin Huai] Bug fix and refactoring.\n69bb072 [Yin Huai] Introduce NullSafeHashPartitioning and NullUnsafePartitioning.\nd5b84c3 [Yin Huai] Do not add unnessary filters.\n2201129 [Yin Huai] Filter out rows that will not be joined in equal joins early.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/687c8c37150f4c93f8e57d86bb56321a4891286b"},{"sha":"608353c8e8e50461fafff91a2c885dca8af3aaa8","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9404][SPARK-9542][SQL] unsafe array data and map data\n\nThis PR adds a UnsafeArrayData, current we encode it in this way:\n\nfirst 4 bytes is the # elements\nthen each 4 byte is the start offset of the element, unless it is negative, in which case the element is null.\nfollowed by the elements themselves\n\nan example:  [10, 11, 12, 13, null, 14] will be encoded as:\n5, 28, 32, 36, 40, -44, 44, 10, 11, 12, 13, 14\n\nNote that, when we read a UnsafeArrayData from bytes, we can read the first 4 bytes as numElements and take the rest(first 4 bytes skipped) as value region.\n\nunsafe map data just use 2 unsafe array data, first 4 bytes is # of elements, second 4 bytes is numBytes of key array, the follows key array data and value array data.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7752 from cloud-fan/unsafe-array and squashes the following commits:\n\n3269bd7 [Wenchen Fan] fix a bug\n6445289 [Wenchen Fan] add unit tests\n49adf26 [Wenchen Fan] add unsafe map\n20d1039 [Wenchen Fan] add comments and unsafe converter\n821b8db [Wenchen Fan] add unsafe array","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/608353c8e8e50461fafff91a2c885dca8af3aaa8"},{"sha":"98d6d9c7a996f5456eb2653bb96985a1a05f4ce1","author":{"email":"348ff379544b45aa3f16e53a543620141423a0f6@gmail.com","name":"Yijie Shen"},"message":"[SPARK-9549][SQL] fix bugs in expressions\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9549\n\nThis PR fix the following bugs:\n1.  `UnaryMinus`'s codegen version would fail to compile when the input is `Long.MinValue`\n2.  `BinaryComparison` would fail to compile in codegen mode when comparing Boolean types.\n3.  `AddMonth` would fail if passed a huge negative month, which would lead accessing negative index of `monthDays` array.\n4.  `Nanvl` with different type operands.\n\nAuthor: Yijie Shen <henry.yijieshen@gmail.com>\n\nCloses #7882 from yjshen/minor_bug_fix and squashes the following commits:\n\n41bbd2c [Yijie Shen] fix bug in Nanvl type coercion\n3dee204 [Yijie Shen] address comments\n4fa5de0 [Yijie Shen] fix bugs in expressions","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/98d6d9c7a996f5456eb2653bb96985a1a05f4ce1"},{"sha":"1ebd41b141a95ec264bd2dd50f0fe24cd459035d","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9240] [SQL] Hybrid aggregate operator using unsafe row\n\nThis PR adds a base aggregation iterator `AggregationIterator`, which is used to create `SortBasedAggregationIterator` (for sort-based aggregation) and `UnsafeHybridAggregationIterator` (first it tries hash-based aggregation and falls back to the sort-based aggregation (using external sorter) if we cannot allocate memory for the map). With these two iterators, we will not need existing iterators and I am removing those. Also, we can use a single physical `Aggregate` operator and it internally determines what iterators to used.\n\nhttps://issues.apache.org/jira/browse/SPARK-9240\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #7813 from yhuai/AggregateOperator and squashes the following commits:\n\ne317e2b [Yin Huai] Remove unnecessary change.\n74d93c5 [Yin Huai] Merge remote-tracking branch 'upstream/master' into AggregateOperator\nba6afbc [Yin Huai] Add a little bit more comments.\nc9cf3b6 [Yin Huai] update\n0f1b06f [Yin Huai] Remove unnecessary code.\n21fd15f [Yin Huai] Remove unnecessary change.\n964f88b [Yin Huai] Implement fallback strategy.\nb1ea5cf [Yin Huai] wip\n7fcbd87 [Yin Huai] Add a flag to control what iterator to use.\n533d5b2 [Yin Huai] Prepare for fallback!\n33b7022 [Yin Huai] wip\nbd9282b [Yin Huai] UDAFs now supports UnsafeRow.\nf52ee53 [Yin Huai] wip\n3171f44 [Yin Huai] wip\nd2c45a0 [Yin Huai] wip\nf60cc83 [Yin Huai] Also check input schema.\naf32210 [Yin Huai] Check iter.hasNext before we create an iterator because the constructor of the iterato will read at least one row from a non-empty input iter.\n299008c [Yin Huai] First round cleanup.\n3915bac [Yin Huai] Create a base iterator class for aggregation iterators and add the initial version of the hybrid iterator.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/1ebd41b141a95ec264bd2dd50f0fe24cd459035d"},{"sha":"95dccc63350c45045f038bab9f8a5080b4e1f8cc","author":{"email":"22a0dda5f6015b8ee84a30ff6553d35a91fe8732@gmail.com","name":"Timothy Chen"},"message":"[SPARK-8873] [MESOS] Clean up shuffle files if external shuffle service is used\n\nThis patch builds directly on #7820, which is largely written by tnachen. The only addition is one commit for cleaning up the code. There should be no functional differences between this and #7820.\n\nAuthor: Timothy Chen <tnachen@gmail.com>\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #7881 from andrewor14/tim-cleanup-mesos-shuffle and squashes the following commits:\n\n8894f7d [Andrew Or] Clean up code\n2a5fa10 [Andrew Or] Merge branch 'mesos_shuffle_clean' of github.com:tnachen/spark into tim-cleanup-mesos-shuffle\nfadff89 [Timothy Chen] Address comments.\ne4d0f1d [Timothy Chen] Clean up external shuffle data on driver exit with Mesos.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/95dccc63350c45045f038bab9f8a5080b4e1f8cc"},{"sha":"137f47865df6e98ab70ae5ba30dc4d441fb41166","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9551][SQL] add a cheap version of copy for UnsafeRow to reuse a copy buffer\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7885 from cloud-fan/cheap-copy and squashes the following commits:\n\n0900ca1 [Wenchen Fan] replace == with ===\n73f4ada [Wenchen Fan] add tests\n07b865a [Wenchen Fan] add a cheap version of copy","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/137f47865df6e98ab70ae5ba30dc4d441fb41166"},{"sha":"191bf2689d127a9dd328b9cc517362fd51eaed3d","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-9518] [SQL] cleanup generated UnsafeRowJoiner and fix bug\n\nCurrently, when copy the bitsets, we didn't consider that the row1 may not sit in the beginning of byte array.\n\ncc rxin\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #7892 from davies/clean_join and squashes the following commits:\n\n14cce9e [Davies Liu] cleanup generated UnsafeRowJoiner and fix bug","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/191bf2689d127a9dd328b9cc517362fd51eaed3d"},{"sha":"8be198c86935001907727fd16577231ff776125b","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Two minor comments from code review on 191bf2689.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/8be198c86935001907727fd16577231ff776125b"},{"sha":"69f5a7c934ac553ed52c00679b800bcffe83c1d6","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-9528] [ML] Changed RandomForestClassifier to extend ProbabilisticClassifier\n\nRandomForestClassifier now outputs rawPrediction based on tree probabilities, plus probability column computed from normalized rawPrediction.\n\nCC: holdenk\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #7859 from jkbradley/rf-prob and squashes the following commits:\n\n6c28f51 [Joseph K. Bradley] Changed RandomForestClassifier to extend ProbabilisticClassifier","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/69f5a7c934ac553ed52c00679b800bcffe83c1d6"},{"sha":"b41a32718d615b304efba146bf97be0229779b01","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-1855] Local checkpointing\n\nCertain use cases of Spark involve RDDs with long lineages that must be truncated periodically (e.g. GraphX). The existing way of doing it is through `rdd.checkpoint()`, which is expensive because it writes to HDFS. This patch provides an alternative to truncate lineages cheaply *without providing the same level of fault tolerance*.\n\n**Local checkpointing** writes checkpointed data to the local file system through the block manager. It is much faster than replicating to a reliable storage and provides the same semantics as long as executors do not fail. It is accessible through a new operator `rdd.localCheckpoint()` and leaves the old one unchanged. Users may even decide to combine the two and call the reliable one less frequently.\n\nThe bulk of this patch involves refactoring the checkpointing interface to accept custom implementations of checkpointing. [Design doc](https://issues.apache.org/jira/secure/attachment/12741708/SPARK-7292-design.pdf).\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #7279 from andrewor14/local-checkpoint and squashes the following commits:\n\n729600f [Andrew Or] Oops, fix tests\n34bc059 [Andrew Or] Avoid computing all partitions in local checkpoint\ne43bbb6 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\n3be5aea [Andrew Or] Address comments\nbf846a6 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\nab003a3 [Andrew Or] Fix compile\nc2e111b [Andrew Or] Address comments\n33f167a [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\ne908a42 [Andrew Or] Fix tests\nf5be0f3 [Andrew Or] Use MEMORY_AND_DISK as the default local checkpoint level\na92657d [Andrew Or] Update a few comments\ne58e3e3 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\n4eb6eb1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\n1bbe154 [Andrew Or] Simplify LocalCheckpointRDD\n48a9996 [Andrew Or] Avoid traversing dependency tree + rewrite tests\n62aba3f [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\ndb70dc2 [Andrew Or] Express local checkpointing through caching the original RDD\n87d43c6 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\nc449b38 [Andrew Or] Fix style\n4a182f3 [Andrew Or] Add fine-grained tests for local checkpointing\n53b363b [Andrew Or] Rename a few more awkwardly named methods (minor)\ne4cf071 [Andrew Or] Simplify LocalCheckpointRDD + docs + clean ups\n4880deb [Andrew Or] Fix style\nd096c67 [Andrew Or] Fix mima\n172cb66 [Andrew Or] Fix mima?\ne53d964 [Andrew Or] Fix style\n56831c5 [Andrew Or] Add a few warnings and clear exception messages\n2e59646 [Andrew Or] Add local checkpoint clean up tests\n4dbbab1 [Andrew Or] Refactor CheckpointSuite to test local checkpointing\n4514dc9 [Andrew Or] Clean local checkpoint files through RDD cleanups\n0477eec [Andrew Or] Rename a few methods with awkward names (minor)\n2e902e5 [Andrew Or] First implementation of local checkpointing\n8447454 [Andrew Or] Fix tests\n4ac1896 [Andrew Or] Refactor checkpoint interface for modularity","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/b41a32718d615b304efba146bf97be0229779b01"},{"sha":"dfe7bd168d9bcf8c53f993f459ab473d893457b0","author":{"email":"fe0966a084bd02ed13ddc43bf288232282ff6a11@cloudera.com","name":"Joseph Batchik"},"message":"[SPARK-9511] [SQL] Fixed Table Name Parsing\n\nThe issue was that the tokenizer was parsing \"1one\" into the numeric 1 using the code on line 110. I added another case to accept strings that start with a number and then have a letter somewhere else in it as well.\n\nAuthor: Joseph Batchik <joseph.batchik@cloudera.com>\n\nCloses #7844 from JDrit/parse_error and squashes the following commits:\n\nb8ca12f [Joseph Batchik] fixed parsing issue by adding another case","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/dfe7bd168d9bcf8c53f993f459ab473d893457b0"},{"sha":"7a9d09f0bb472a1671d3457e1f7108f4c2eb4121","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SQL][minor] Simplify UnsafeRow.calculateBitSetWidthInBytes.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7897 from rxin/calculateBitSetWidthInBytes and squashes the following commits:\n\n2e73b3a [Reynold Xin] [SQL][minor] Simplify UnsafeRow.calculateBitSetWidthInBytes.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/7a9d09f0bb472a1671d3457e1f7108f4c2eb4121"},{"sha":"703e44bff19f4c394f6f9bff1ce9152cdc68c51e","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-9554] [SQL] Enables in-memory partition pruning by default\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7895 from liancheng/spark-9554/enable-in-memory-partition-pruning and squashes the following commits:\n\n67c403e [Cheng Lian] Enables in-memory partition pruning by default","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/703e44bff19f4c394f6f9bff1ce9152cdc68c51e"},{"sha":"ff9169a002f1b75231fd25b7d04157a912503038","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-5133] [ML] Added featureImportance to RandomForestClassifier and Regressor\n\nAdded featureImportance to RandomForestClassifier and Regressor.\n\nThis follows the scikit-learn implementation here: [https://github.com/scikit-learn/scikit-learn/blob/a95203b249c1cf392f86d001ad999e29b2392739/sklearn/tree/_tree.pyx#L3341]\n\nCC: yanboliang  Would you mind taking a look?  Thanks!\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #7838 from jkbradley/dt-feature-importance and squashes the following commits:\n\n72a167a [Joseph K. Bradley] fixed unit test\n86cea5f [Joseph K. Bradley] Modified RF featuresImportances to return Vector instead of Map\n5aa74f0 [Joseph K. Bradley] finally fixed unit test for real\n33df5db [Joseph K. Bradley] fix unit test\n42a2d3b [Joseph K. Bradley] fix unit test\nfe94e72 [Joseph K. Bradley] modified feature importance unit tests\ncc693ee [Feynman Liang] Add classifier tests\n79a6f87 [Feynman Liang] Compare dense vectors in test\n21d01fc [Feynman Liang] Added failing SKLearn test\nac0b254 [Joseph K. Bradley] Added featureImportance to RandomForestClassifier/Regressor.  Need to add unit tests","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/ff9169a002f1b75231fd25b7d04157a912503038"}]}