{"push_id":749408635,"size":10,"distinct_size":7,"ref":"refs/heads/SPARK-9674","head":"82d1dc73a05d4b5605acac02f9f0ebce1ecdf6f0","before":"6be4976de6c1705e9ffa5fbb23184e2f68d0c135","commits":[{"sha":"a018b85716fd510ae95a3c66d676bbdb90f8d4e7","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-5895] [ML] Add VectorSlicer - updated\n\nAdd VectorSlicer transformer to spark.ml, with features specified as either indices or names.  Transfers feature attributes for selected features.\n\nUpdated version of [https://github.com/apache/spark/pull/5731]\n\nCC: yinxusen This updates your PR.  You'll still be the primary author of this PR.\n\nCC: mengxr\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #7972 from jkbradley/yinxusen-SPARK-5895 and squashes the following commits:\n\nb16e86e [Joseph K. Bradley] fixed scala style\n71c65d2 [Joseph K. Bradley] fix import order\n86e9739 [Joseph K. Bradley] cleanups per code review\n9d8d6f1 [Joseph K. Bradley] style fix\n83bc2e9 [Joseph K. Bradley] Updated VectorSlicer\n98c6939 [Xusen Yin] fix style error\necbf2d3 [Xusen Yin] change interfaces and params\nf6be302 [Xusen Yin] Merge branch 'master' into SPARK-5895\ne4781f2 [Xusen Yin] fix commit error\nfd154d7 [Xusen Yin] add test suite of vector slicer\n17171f8 [Xusen Yin] fix slicer\n9ab9747 [Xusen Yin] add vector slicer\naa5a0bf [Xusen Yin] add vector slicer","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/a018b85716fd510ae95a3c66d676bbdb90f8d4e7"},{"sha":"8c320e45b5c9ffd7f0e35c1c7e6b5fc355377ea6","author":{"email":"348ff379544b45aa3f16e53a543620141423a0f6@gmail.com","name":"Yijie Shen"},"message":"[SPARK-6591] [SQL] Python data source load options should auto convert common types into strings\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-6591\n\nAuthor: Yijie Shen <henry.yijieshen@gmail.com>\n\nCloses #7926 from yjshen/py_dsload_opt and squashes the following commits:\n\nb207832 [Yijie Shen] fix style\nefdf834 [Yijie Shen] resolve comment\n7a8f6a2 [Yijie Shen] lowercase\n822e769 [Yijie Shen] convert load opts to string","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/8c320e45b5c9ffd7f0e35c1c7e6b5fc355377ea6"},{"sha":"4399b7b0903d830313ab7e69731c11d587ae567c","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-9651] Fix UnsafeExternalSorterSuite.\n\nFirst, it's probably a bad idea to call generated Scala methods\nfrom Java. In this case, the method being called wasn't actually\n\"Utils.createTempDir()\", but actually the method that returns the\nfirst default argument to the actual createTempDir method, which\nis just the location of java.io.tmpdir; meaning that all tests in\nthe class were using the same temp dir, and thus affecting each\nother.\n\nSecond, spillingOccursInResponseToMemoryPressure was not writing\nenough records to actually cause a spill.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #7970 from vanzin/SPARK-9651 and squashes the following commits:\n\n74d357f [Marcelo Vanzin] Clean up temp dir on test tear down.\na64f36a [Marcelo Vanzin] [SPARK-9651] Fix UnsafeExternalSorterSuite.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/4399b7b0903d830313ab7e69731c11d587ae567c"},{"sha":"4581badbc8aa7e5a37ba7f7f83cc3860240f5dd3","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9611] [SQL] Fixes a few corner cases when we spill a UnsafeFixedWidthAggregationMap\n\nThis PR has the following three small fixes.\n\n1. UnsafeKVExternalSorter does not use 0 as the initialSize to create an UnsafeInMemorySorter if its BytesToBytesMap is empty.\n2. We will not not spill a InMemorySorter if it is empty.\n3. We will not add a SpillReader to a SpillMerger if this SpillReader is empty.\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9611\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #7948 from yhuai/unsafeEmptyMap and squashes the following commits:\n\n9727abe [Yin Huai] Address Josh's comments.\n34b6f76 [Yin Huai] 1. UnsafeKVExternalSorter does not use 0 as the initialSize to create an UnsafeInMemorySorter if its BytesToBytesMap is empty. 2. Do not spill a InMemorySorter if it is empty. 3. Do not add spill to SpillMerger if this spill is empty.","distinct":true,"url":"https://api.github.com/repos/rxin/spark/commits/4581badbc8aa7e5a37ba7f7f83cc3860240f5dd3"},{"sha":"119b59053870df7be899bf5c1c0d321406af96f9","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-6923] [SPARK-7550] [SQL] Persists data source relations in Hive compatible format when possible\n\nThis PR is a fork of PR #5733 authored by chenghao-intel.  For committers who's going to merge this PR, please set the author to \"Cheng Hao <hao.chengintel.com>\".\n\n----\n\nWhen a data source relation meets the following requirements, we persist it in Hive compatible format, so that other systems like Hive can access it:\n\n1. It's a `HadoopFsRelation`\n2. It has only one input path\n3. It's non-partitioned\n4. It's data source provider can be naturally mapped to a Hive builtin SerDe (e.g. ORC and Parquet)\n\nAuthor: Cheng Lian <lian@databricks.com>\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #7967 from liancheng/spark-6923/refactoring-pr-5733 and squashes the following commits:\n\n5175ee6 [Cheng Lian] Fixes an oudated comment\n3870166 [Cheng Lian] Fixes build error and comments\n864acee [Cheng Lian] Refactors PR #5733\n3490cdc [Cheng Hao] update the scaladoc\n6f57669 [Cheng Hao] write schema info to hivemetastore for data source","distinct":true,"url":"https://api.github.com/repos/rxin/spark/commits/119b59053870df7be899bf5c1c0d321406af96f9"},{"sha":"9270bd06fd0b16892e3f37213b5bc7813ea11fdd","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9674][SQL] Remove GeneratedAggregate.\n\nThe new aggregate replaces the old GeneratedAggregate.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7983 from rxin/remove-generated-agg and squashes the following commits:\n\n8334aae [Reynold Xin] [SPARK-9674][SQL] Remove GeneratedAggregate.","distinct":true,"url":"https://api.github.com/repos/rxin/spark/commits/9270bd06fd0b16892e3f37213b5bc7813ea11fdd"},{"sha":"d5a9af3230925c347d0904fe7f2402e468e80bc8","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9664] [SQL] Remove UDAFRegistration and add apply to UserDefinedAggregateFunction.\n\nhttps://issues.apache.org/jira/browse/SPARK-9664\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #7982 from yhuai/udafRegister and squashes the following commits:\n\n0cc2287 [Yin Huai] Remove UDAFRegistration and add apply to UserDefinedAggregateFunction.","distinct":true,"url":"https://api.github.com/repos/rxin/spark/commits/d5a9af3230925c347d0904fe7f2402e468e80bc8"},{"sha":"651d34f1687c5ab8b54fbd1517409cfe867392f2","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9674][SPARK-9667] Remove GeneratedAggregate & SparkSqlSerializer2.\n\nThey are now subsumed by various Tungsten operators.","distinct":true,"url":"https://api.github.com/repos/rxin/spark/commits/651d34f1687c5ab8b54fbd1517409cfe867392f2"},{"sha":"286500e5fec8fd292e737624ad64b2189e090ae0","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Updated if.","distinct":true,"url":"https://api.github.com/repos/rxin/spark/commits/286500e5fec8fd292e737624ad64b2189e090ae0"},{"sha":"82d1dc73a05d4b5605acac02f9f0ebce1ecdf6f0","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Match all UDTs.","distinct":true,"url":"https://api.github.com/repos/rxin/spark/commits/82d1dc73a05d4b5605acac02f9f0ebce1ecdf6f0"}]}