{"push_id":576983261,"size":770,"distinct_size":770,"ref":"refs/heads/master","head":"ee6e3eff02e9e08b1113ba6faf3397d7e7775087","before":"b004150adb503ddbb54d5cd544e39ad974497c41","commits":[{"sha":"bf40cf89e37aeaf80e37a4b0ae001ba25f819821","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-4713] [SQL] SchemaRDD.unpersist() should not raise exception if it is not persisted\n\nUnpersist a uncached RDD, will not raise exception, for example:\n```\nval data = Array(1, 2, 3, 4, 5)\nval distData = sc.parallelize(data)\ndistData.unpersist(true)\n```\n\nBut the `SchemaRDD` will raise exception if the `SchemaRDD` is not cached. Since `SchemaRDD` is the subclasses of the `RDD`, we should follow the same behavior.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #3572 from chenghao-intel/try_uncache and squashes the following commits:\n\n50a7a89 [Cheng Hao] SchemaRDD.unpersist() should not raise exception if it is not persisted","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/bf40cf89e37aeaf80e37a4b0ae001ba25f819821"},{"sha":"a7f07f511c0783db3b2b0a19fc71814cda94aad8","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-4662] [SQL] Whitelist more unittest\n\nWhitelist more hive unit test:\n\n\"create_like_tbl_props\"\n\"udf5\"\n\"udf_java_method\"\n\"decimal_1\"\n\"udf_pmod\"\n\"udf_to_double\"\n\"udf_to_float\"\n\"udf7\" (this will fail in Hive 0.12)\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #3522 from chenghao-intel/unittest and squashes the following commits:\n\nf54e4c7 [Cheng Hao] work around to clean up the hive.table.parameters.default in reset\n16fee22 [Cheng Hao] Whitelist more unittest","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/a7f07f511c0783db3b2b0a19fc71814cda94aad8"},{"sha":"c152dde78f73d5ce3a483fd60a47e7de1f1916da","author":{"email":"5d1ed0070fb6b6d5f66c1cf6a3d0bb37cf6f15d1@huawei.com","name":"Jacky Li"},"message":"[SPARK-4639] [SQL] Pass maxIterations in as a parameter in Analyzer\n\nfix a TODO in Analyzer:\n// TODO: pass this in as a parameter\nval fixedPoint = FixedPoint(100)\n\nAuthor: Jacky Li <jacky.likun@huawei.com>\n\nCloses #3499 from jackylk/config and squashes the following commits:\n\n4c1252c [Jacky Li] fix scalastyle\n820f460 [Jacky Li] pass maxIterations in as a parameter","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/c152dde78f73d5ce3a483fd60a47e7de1f1916da"},{"sha":"334480362b3a133c2fb1e9af898930fe76d7a163","author":{"email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","name":"Takuya UESHIN"},"message":"[SPARK-4293][SQL] Make Cast be able to handle complex types.\n\nInserting data of type including `ArrayType.containsNull == false` or `MapType.valueContainsNull == false` or `StructType.fields.exists(_.nullable == false)` into Hive table will fail because `Cast` inserted by `HiveMetastoreCatalog.PreInsertionCasts` rule of `Analyzer` can't handle these types correctly.\n\nComplex type cast rule proposal:\n\n- Cast for non-complex types should be able to cast the same as before.\n- Cast for `ArrayType` can evaluate if\n  - Element type can cast\n  - Nullability rule doesn't break\n- Cast for `MapType` can evaluate if\n  - Key type can cast\n  - Nullability for casted key type is `false`\n  - Value type can cast\n  - Nullability rule for value type doesn't break\n- Cast for `StructType` can evaluate if\n  - The field size is the same\n  - Each field can cast\n  - Nullability rule for each field doesn't break\n- The nested structure should be the same.\n\nNullability rule:\n\n- If the casted type is `nullable == true`, the target nullability should be `true`\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #3150 from ueshin/issues/SPARK-4293 and squashes the following commits:\n\ne935939 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4293\nba14003 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4293\n8999868 [Takuya UESHIN] Fix a test title.\nf677c30 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4293\n287f410 [Takuya UESHIN] Add tests to insert data of types ArrayType / MapType / StructType with nullability is false into Hive table.\n4f71bb8 [Takuya UESHIN] Make Cast be able to handle complex types.","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/334480362b3a133c2fb1e9af898930fe76d7a163"},{"sha":"d8cf67858988bed287c0125095729f7f808aabad","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"scwf"},"message":"[SQL] Remove unnecessary case in HiveContext.toHiveString\n\na follow up of #3547\n/cc marmbrus\n\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #3563 from scwf/rnc and squashes the following commits:\n\n9395661 [scwf] remove unnecessary condition","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/d8cf67858988bed287c0125095729f7f808aabad"},{"sha":"acb3be6bc5e0e793f769ec371dffeb474dc0cf21","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-4828] [SQL] sum and avg on empty table should always return null\n\nSo the optimizations are not valid. Also I think the optimization here is rarely encounter, so removing them will not have influence on performance.\n\nCan we merge #3445 before I add a comparison test case from this?\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #3675 from adrian-wang/sumempty and squashes the following commits:\n\n42df763 [Daoyuan Wang] sum and avg on empty table should always return null","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/acb3be6bc5e0e793f769ec371dffeb474dc0cf21"},{"sha":"cbb634ae69163ca7a8b5cb94c5fb17cb5c910cb1","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SQL] enable empty aggr test case\n\nThis is fixed by SPARK-4318 #3184\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #3445 from adrian-wang/emptyaggr and squashes the following commits:\n\n982575e [Daoyuan Wang] enable empty aggr test case","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/cbb634ae69163ca7a8b5cb94c5fb17cb5c910cb1"},{"sha":"0abbff286220bbcbbf28fbd80b8c5bf59ff37ce2","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-4825] [SQL] CTAS fails to resolve when created using saveAsTable\n\nFix bug when query like:\n```\n  test(\"save join to table\") {\n    val testData = sparkContext.parallelize(1 to 10).map(i => TestData(i, i.toString))\n    sql(\"CREATE TABLE test1 (key INT, value STRING)\")\n    testData.insertInto(\"test1\")\n    sql(\"CREATE TABLE test2 (key INT, value STRING)\")\n    testData.insertInto(\"test2\")\n    testData.insertInto(\"test2\")\n    sql(\"SELECT COUNT(a.value) FROM test1 a JOIN test2 b ON a.key = b.key\").saveAsTable(\"test\")\n    checkAnswer(\n      table(\"test\"),\n      sql(\"SELECT COUNT(a.value) FROM test1 a JOIN test2 b ON a.key = b.key\").collect().toSeq)\n  }\n```\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #3673 from chenghao-intel/spark_4825 and squashes the following commits:\n\ne8cbd56 [Cheng Hao] alternate the pattern matching order for logical plan:CTAS\ne004895 [Cheng Hao] fix bug","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/0abbff286220bbcbbf28fbd80b8c5bf59ff37ce2"},{"sha":"8091dd62eaff28196dfb9742a4f39182704b1024","author":{"email":"93406da1fa8ebf4406b5d741093e2ccfc7f09e95@nttdata.co.jp","name":"Sasaki Toru"},"message":"[SPARK-4742][SQL] The name of Parquet File generated by AppendingParquetOutputFormat should be zero padded\n\nWhen I use Parquet File as a output file using ParquetOutputFormat#getDefaultWorkFile, the file name is not zero padded while RDD#saveAsText does zero padding.\n\nAuthor: Sasaki Toru <sasakitoa@nttdata.co.jp>\n\nCloses #3602 from sasakitoa/parquet-zeroPadding and squashes the following commits:\n\n6b0e58f [Sasaki Toru] Merge branch 'master' of git://github.com/apache/spark into parquet-zeroPadding\n20dc79d [Sasaki Toru] Fixed the name of Parquet File generated by AppendingParquetOutputFormat","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/8091dd62eaff28196dfb9742a4f39182704b1024"},{"sha":"41a3f9343831a3efc18e08595d095282a8d1486b","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-4829] [SQL] add rule to fold count(expr) if expr is not null\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #3676 from adrian-wang/countexpr and squashes the following commits:\n\ndc5765b [Daoyuan Wang] add rule to fold count(expr) if expr is not null","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/41a3f9343831a3efc18e08595d095282a8d1486b"},{"sha":"ef84dab8c6aa634c4dadac7a319f1ef5aaf85511","author":{"email":"8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","name":"Patrick Wendell"},"message":"MAINTENANCE: Automated closing of pull requests.\n\nThis commit exists to close the following pull requests on Github:\n\nCloses #3488 (close requested by 'pwendell')\nCloses #2939 (close requested by 'marmbrus')\nCloses #3173 (close requested by 'marmbrus')","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/ef84dab8c6aa634c4dadac7a319f1ef5aaf85511"},{"sha":"2a2983f7c5edce8e4d1d7592adcf227cbd462ae9","author":{"email":"4b8373d016f277527198385ba72fda0feb5da015@klipfel.me","name":"Peter Klipfel"},"message":"fixed spelling errors in documentation\n\nchanged \"form\" to \"from\" in 3 documentation entries for Kafka integration\n\nAuthor: Peter Klipfel <peter@klipfel.me>\n\nCloses #3691 from peterklipfel/master and squashes the following commits:\n\n0fe7fc5 [Peter Klipfel] fixed spelling errors in documentation","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/2a2983f7c5edce8e4d1d7592adcf227cbd462ae9"},{"sha":"4c0673879b5c504797dafb11607d14b04c1bf47d","author":{"email":"8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","name":"Patrick Wendell"},"message":"HOTFIX: Disabling failing block manager test","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/4c0673879b5c504797dafb11607d14b04c1bf47d"},{"sha":"8098fab06cb2be22cca4e531e8e65ab29dbb909a","author":{"email":"bc4a65f227ddcd81bb538c67e810a3e11696e8bf@gmail.com","name":"Yuu ISHIKAWA"},"message":"[SPARK-4494][mllib] IDFModel.transform() add support for single vector\n\nI improved `IDFModel.transform` to allow using a single vector.\n\n[[SPARK-4494] IDFModel.transform() add support for single vector - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-4494)\n\nAuthor: Yuu ISHIKAWA <yuu.ishikawa@gmail.com>\n\nCloses #3603 from yu-iskw/idf and squashes the following commits:\n\n256ff3d [Yuu ISHIKAWA] Fix typo\na3bf566 [Yuu ISHIKAWA] - Fix typo - Optimize import order - Aggregate the assertion tests - Modify `IDFModel.transform` API for pyspark\nd25e49b [Yuu ISHIKAWA] Add the implementation of `IDFModel.transform` for a term frequency vector","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/8098fab06cb2be22cca4e531e8e65ab29dbb909a"},{"sha":"f6b8591a08835c9af19210f9cdfbaab2537135c4","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-4826] Fix generation of temp file names in WAL tests\n\nThis PR should fix SPARK-4826, an issue where a bug in how we generate temp. file names was causing spurious test failures in the write ahead log suites.\n\nCloses #3695.\nCloses #3701.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #3704 from JoshRosen/SPARK-4826 and squashes the following commits:\n\nf2307f5 [Josh Rosen] Use Spark Utils class for directory creation/deletion\na693ddb [Josh Rosen] remove unused Random import\nb275e41 [Josh Rosen] Move creation of temp. dir to beforeEach/afterEach.\n9362919 [Josh Rosen] [SPARK-4826] Fix bug in generation of temp file names. in WAL suites.\n86c1944 [Josh Rosen] Revert \"HOTFIX: Disabling failing block manager test\"","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/f6b8591a08835c9af19210f9cdfbaab2537135c4"},{"sha":"38703bbca86003995f32b2e948ad7c7c358aa99a","author":{"email":"17d07ce14ed8a23e11d276b3691423f0ca203e51@capitalone.com","name":"Ilya Ganelin"},"message":"[SPARK-1037] The name of findTaskFromList & findTask in TaskSetManager.scala is confusing\n\nHi all - I've renamed the methods referenced in this JIRA to clarify that they modify the provided arrays (find vs. deque).\n\nAuthor: Ilya Ganelin <ilya.ganelin@capitalone.com>\n\nCloses #3665 from ilganeli/SPARK-1037B and squashes the following commits:\n\n64c177c [Ilya Ganelin] Renamed deque to dequeue\nf27d85e [Ilya Ganelin] Renamed private methods to clarify that they modify the provided parameters\n683482a [Ilya Ganelin] Renamed private methods to clarify that they modify the provided parameters","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/38703bbca86003995f32b2e948ad7c7c358aa99a"},{"sha":"8176b7a02e6b62bbce194c3ce9802d58b7472101","author":{"email":"45d0cd84ae16b71953c1beb3be9ef57773a10b25@gmail.com","name":"Ryan Williams"},"message":"[SPARK-4668] Fix some documentation typos.\n\nAuthor: Ryan Williams <ryan.blake.williams@gmail.com>\n\nCloses #3523 from ryan-williams/tweaks and squashes the following commits:\n\nd2eddaa [Ryan Williams] code review feedback\nce27fc1 [Ryan Williams] CoGroupedRDD comment nit\nc6cfad9 [Ryan Williams] remove unnecessary if statement\nb74ea35 [Ryan Williams] comment fix\nb0221f0 [Ryan Williams] fix a gendered pronoun\nc71ffed [Ryan Williams] use names on a few boolean parameters\n89954aa [Ryan Williams] clarify some comments in {Security,Shuffle}Manager\ne465dac [Ryan Williams] Saved building-spark.md with Dillinger.io\n83e8358 [Ryan Williams] fix pom.xml typo\ndc4662b [Ryan Williams] typo fixes in tuning.md, configuration.md","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/8176b7a02e6b62bbce194c3ce9802d58b7472101"},{"sha":"2a28bc61009a170af3853c78f7f36970898a6d56","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-785 [CORE] ClosureCleaner not invoked on most PairRDDFunctions\n\nThis looked like perhaps a simple and important one. `combineByKey` looks like it should clean its arguments' closures, and that in turn covers apparently all remaining functions in `PairRDDFunctions` which delegate to it.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #3690 from srowen/SPARK-785 and squashes the following commits:\n\n8df68fe [Sean Owen] Clean context of most remaining functions in PairRDDFunctions, which ultimately call combineByKey","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/2a28bc61009a170af3853c78f7f36970898a6d56"},{"sha":"5c24759ddc25cbafbedbaafbf053d38015a7774e","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[Minor][Core] fix comments in MapOutputTracker\n\nUsing driver and executor in the comments of ```MapOutputTracker``` is more clear.\n\nAuthor: wangfei <wangfei1@huawei.com>\n\nCloses #3700 from scwf/commentFix and squashes the following commits:\n\naa68524 [wangfei] master and worker should be driver and executor","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/5c24759ddc25cbafbedbaafbf053d38015a7774e"},{"sha":"81112e4b573292e76c7feeed995751bd7a5fe489","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-4814 [CORE] Enable assertions in SBT, Maven tests / AssertionError from Hive's LazyBinaryInteger\n\nThis enables assertions for the Maven and SBT build, but overrides the Hive module to not enable assertions.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #3692 from srowen/SPARK-4814 and squashes the following commits:\n\ncaca704 [Sean Owen] Disable assertions just for Hive\nf71e783 [Sean Owen] Enable assertions for SBT and Maven build","distinct":true,"url":"https://api.github.com/repos/orenmazor/spark/commits/81112e4b573292e76c7feeed995751bd7a5fe489"}]}