{"push_id":742439782,"size":90,"distinct_size":90,"ref":"refs/heads/master","head":"5e99f570dba8da889f58b520607e633c3ba2fbf2","before":"e13f90844d7bb82fdf2b0ba5eb6bd53c68380d81","commits":[{"sha":"b79bf1df6238c087c3ec524344f1fc179719c5de","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-9337] [MLLIB] Add an ut for Word2Vec to verify the empty vocabulary check\n\njira: https://issues.apache.org/jira/browse/SPARK-9337\n\nWord2Vec should throw exception when vocabulary is empty\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #7660 from hhbyyh/ut4Word2vec and squashes the following commits:\n\n17a18cb [Yuhao Yang] add ut for word2vec","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/b79bf1df6238c087c3ec524344f1fc179719c5de"},{"sha":"6c400b4f39be3fb5f473b8d2db11d239ea8ddf42","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9354][SQL] Remove InternalRow.get generic getter call in Hive integration code.\n\nReplaced them with get(ordinal, datatype) so we can use UnsafeRow here.\n\nI passed the data types throughout.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7669 from rxin/row-generic-getter-hive and squashes the following commits:\n\n3467d8e [Reynold Xin] [SPARK-9354][SQL] Remove Internal.get generic getter call in Hive integration code.","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/6c400b4f39be3fb5f473b8d2db11d239ea8ddf42"},{"sha":"fb5d43fb2529d78d55f1fe8d365191c946153640","author":{"email":"348ff379544b45aa3f16e53a543620141423a0f6@gmail.com","name":"Yijie Shen"},"message":"[SPARK-9356][SQL]Remove the internal use of DecimalType.Unlimited\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9356\n\nAuthor: Yijie Shen <henry.yijieshen@gmail.com>\n\nCloses #7671 from yjshen/deprecated_unlimit and squashes the following commits:\n\nc707f56 [Yijie Shen] remove pattern matching in changePrecision\n4a1823c [Yijie Shen] remove internal occurrence of Decimal.Unlimited","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/fb5d43fb2529d78d55f1fe8d365191c946153640"},{"sha":"1cf19760d61a5a17bd175a906d34a2940141b76d","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9352] [SPARK-9353] Add tests for standalone scheduling code\n\nThis also fixes a small issue in the standalone Master that was uncovered by the new tests. For more detail, read the description of SPARK-9353.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #7668 from andrewor14/standalone-scheduling-tests and squashes the following commits:\n\nd852faf [Andrew Or] Add tests + fix scheduling with memory limits","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/1cf19760d61a5a17bd175a906d34a2940141b76d"},{"sha":"6b2baec04fa3d928f0ee84af8c2723ac03a4648c","author":{"email":"284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","name":"Kay Ousterhout"},"message":"[SPARK-9326] Close lock file used for file downloads.\n\nA lock file is used to ensure multiple executors running on the\nsame machine don't download the same file concurrently. Spark never\ncloses these lock files (releasing the lock does not close the\nunderlying file); this commit fixes that.\n\ncc vanzin (looks like you've been involved in various other fixes surrounding these lock files)\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #7650 from kayousterhout/SPARK-9326 and squashes the following commits:\n\n0401bd1 [Kay Ousterhout] Close lock file used for file downloads.","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/6b2baec04fa3d928f0ee84af8c2723ac03a4648c"},{"sha":"c025c3d0a1fdfbc45b64db9c871176b40b4a7b9b","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-9095] [SQL] Removes the old Parquet support\n\nThis PR removes the old Parquet support:\n\n- Removes the old `ParquetRelation` together with related SQL configuration, plan nodes, strategies, utility classes, and test suites.\n\n- Renames `ParquetRelation2` to `ParquetRelation`\n\n- Renames `RowReadSupport` and `RowRecordMaterializer` to `CatalystReadSupport` and `CatalystRecordMaterializer` respectively, and moved them to separate files.\n\n  This follows naming convention used in other Parquet data models implemented in parquet-mr. It should be easier for developers who are familiar with Parquet to follow.\n\nThere's still some other code that can be cleaned up. Especially `RowWriteSupport`. But I'd like to leave this part to SPARK-8848.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7441 from liancheng/spark-9095 and squashes the following commits:\n\nc7b6e38 [Cheng Lian] Removes WriteToFile\n2d688d6 [Cheng Lian] Renames ParquetRelation2 to ParquetRelation\nca9e1b7 [Cheng Lian] Removes old Parquet support","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/c025c3d0a1fdfbc45b64db9c871176b40b4a7b9b"},{"sha":"1efe97dc9ed31e3b8727b81be633b7e96dd3cd34","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-8867][SQL] Support list / describe function usage\n\nAs Hive does, we need to list all of the registered UDF and its usage for user.\n\nWe add the annotation to describe a UDF, so we can get the literal description info while registering the UDF.\ne.g.\n```scala\nExpressionDescription(\n    usage = \"_FUNC_(expr) - Returns the absolute value of the numeric value\",\n    extended = \"\"\"> SELECT _FUNC_('-1')\n                  1\"\"\")\n case class Abs(child: Expression) extends UnaryArithmetic {\n...\n```\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #7259 from chenghao-intel/desc_function and squashes the following commits:\n\ncf29bba [Cheng Hao] fixing the code style issue\n5193855 [Cheng Hao] Add more powerful parser for show functions\nc645a6b [Cheng Hao] fix bug in unit test\n78d40f1 [Cheng Hao] update the padding issue for usage\n48ee4b3 [Cheng Hao] update as feedback\n70eb4e9 [Cheng Hao] add show/describe function support","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/1efe97dc9ed31e3b8727b81be633b7e96dd3cd34"},{"sha":"945d8bcbf67032edd7bdd201cf9f88c75b3464f7","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-9306] [SQL] Don't use SortMergeJoin when joining on unsortable columns\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9306\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #7645 from viirya/smj_unsortable and squashes the following commits:\n\na240707 [Liang-Chi Hsieh] Use forall instead of exists for readability.\n55221fa [Liang-Chi Hsieh] Shouldn't use SortMergeJoin when joining on unsortable columns.","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/945d8bcbf67032edd7bdd201cf9f88c75b3464f7"},{"sha":"aa80c64fcf9626b3720ee000a653db9266b74839","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9368][SQL] Support get(ordinal, dataType) generic getter in UnsafeRow.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7682 from rxin/unsaferow-generic-getter and squashes the following commits:\n\n3063788 [Reynold Xin] Reset the change for real this time.\n0f57c55 [Reynold Xin] Reset the changes in ExpressionEvalHelper.\nfb6ca30 [Reynold Xin] Support BinaryType.\n24a3e46 [Reynold Xin] Added support for DateType/TimestampType.\n9989064 [Reynold Xin] JoinedRow.\n11f80a3 [Reynold Xin] [SPARK-9368][SQL] Support get(ordinal, dataType) generic getter in UnsafeRow.","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/aa80c64fcf9626b3720ee000a653db9266b74839"},{"sha":"4ffd3a1db5ecff653b02aa325786e734351c8bd2","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9371][SQL] fix the support for special chars in column names for hive context\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7684 from cloud-fan/hive and squashes the following commits:\n\nda21ffe [Wenchen Fan] fix the support for special chars in column names for hive context","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/4ffd3a1db5ecff653b02aa325786e734351c8bd2"},{"sha":"72981bc8f0d421e2563e2543a8c16a8cc76ad3aa","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-7943] [SPARK-8105] [SPARK-8435] [SPARK-8714] [SPARK-8561] Fixes multi-database support\n\nThis PR fixes a set of issues related to multi-database. A new data structure `TableIdentifier` is introduced to identify a table among multiple databases. We should stop using a single `String` (table name without database name), or `Seq[String]` (optional database name plus table name) to identify tables internally.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7623 from liancheng/spark-8131-multi-db and squashes the following commits:\n\nf3bcd4b [Cheng Lian] Addresses PR comments\ne0eb76a [Cheng Lian] Fixes styling issues\n41e2207 [Cheng Lian] Fixes multi-database support\nd4d1ec2 [Cheng Lian] Adds multi-database test cases","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/72981bc8f0d421e2563e2543a8c16a8cc76ad3aa"},{"sha":"622838165756e9669cbf7af13eccbc719638f40b","author":{"email":"dbe0629cf0118f7dffec38856cad0ec7ac880251@intel.com","name":"Carson Wang"},"message":"[SPARK-8405] [DOC] Add how to view logs on Web UI when yarn log aggregation is enabled\n\nSome users may not be aware that the logs are available on Web UI even if Yarn log aggregation is enabled. Update the doc to make this clear and what need to be configured.\n\nAuthor: Carson Wang <carson.wang@intel.com>\n\nCloses #7463 from carsonwang/YarnLogDoc and squashes the following commits:\n\n274c054 [Carson Wang] Minor text fix\n74df3a1 [Carson Wang] address comments\n5a95046 [Carson Wang] Update the text in the doc\ne5775c1 [Carson Wang] Update doc about how to view the logs on Web UI when yarn log aggregation is enabled","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/622838165756e9669cbf7af13eccbc719638f40b"},{"sha":"aa19c696e25ebb07fd3df110cfcbcc69954ce335","author":{"email":"69c58dd9f2140eec50024e8dfcfd98b7670fe611@measite.de","name":"Rene Treffer"},"message":"[SPARK-4176] [SQL] Supports decimal types with precision > 18 in Parquet\n\nThis PR is based on #6796 authored by rtreffer.\n\nTo support large decimal precisions (> 18), we do the following things in this PR:\n\n1. Making `CatalystSchemaConverter` support large decimal precision\n\n   Decimal types with large precision are always converted to fixed-length byte array.\n\n2. Making `CatalystRowConverter` support reading decimal values with large precision\n\n   When the precision is > 18, constructs `Decimal` values with an unscaled `BigInteger` rather than an unscaled `Long`.\n\n3. Making `RowWriteSupport` support writing decimal values with large precision\n\n   In this PR we always write decimals as fixed-length byte array, because Parquet write path hasn't been refactored to conform Parquet format spec (see SPARK-6774 & SPARK-8848).\n\nTwo follow-up tasks should be done in future PRs:\n\n- [ ] Writing decimals as `INT32`, `INT64` when possible while fixing SPARK-8848\n- [ ] Adding compatibility tests as part of SPARK-5463\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7455 from liancheng/spark-4176 and squashes the following commits:\n\na543d10 [Cheng Lian] Fixes errors introduced while rebasing\n9e31cdf [Cheng Lian] Supports decimals with precision > 18 for Parquet","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/aa19c696e25ebb07fd3df110cfcbcc69954ce335"},{"sha":"90006f3c51f8cf9535854246050e27bb76b043f0","author":{"email":"7d60694b1dcb7dc3582c310748eb6a84ab6a1ce8@yandex.ru","name":"Alexander Ulanov"},"message":"Pregel example type fix\n\nPregel example to express single source shortest path from https://spark.apache.org/docs/latest/graphx-programming-guide.html#pregel-api does not work due to incorrect type. The reason is that `GraphGenerators.logNormalGraph` returns the graph with `Long` vertices. Fixing `val graph: Graph[Int, Double]` to `val graph: Graph[Long, Double]`.\n\nAuthor: Alexander Ulanov <nashb@yandex.ru>\n\nCloses #7695 from avulanov/SPARK-9380-pregel-doc and squashes the following commits:\n\nc269429 [Alexander Ulanov] Pregel example type fix","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/90006f3c51f8cf9535854246050e27bb76b043f0"},{"sha":"ecad9d4346ec158746e61aebdf1590215a77f369","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-9364] Fix array out of bounds and use-after-free bugs in UnsafeExternalSorter\n\nThis patch fixes two bugs in UnsafeExternalSorter and UnsafeExternalRowSorter:\n\n- UnsafeExternalSorter does not properly update freeSpaceInCurrentPage, which can cause it to write past the end of memory pages and trigger segfaults.\n- UnsafeExternalRowSorter has a use-after-free bug when returning the last row from an iterator.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #7680 from JoshRosen/SPARK-9364 and squashes the following commits:\n\n590f311 [Josh Rosen] null out row\nf4cf91d [Josh Rosen] Fix use-after-free bug in UnsafeExternalRowSorter.\n8abcf82 [Josh Rosen] Properly decrement freeSpaceInCurrentPage in UnsafeExternalSorter","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/ecad9d4346ec158746e61aebdf1590215a77f369"},{"sha":"c0b7df68f81c2a2a9c1065009fe75c278fa30499","author":{"email":"45d0cd84ae16b71953c1beb3be9ef57773a10b25@gmail.com","name":"Ryan Williams"},"message":"[SPARK-9366] use task's stageAttemptId in TaskEnd event\n\nAuthor: Ryan Williams <ryan.blake.williams@gmail.com>\n\nCloses #7681 from ryan-williams/task-stage-attempt and squashes the following commits:\n\nd6d5f0f [Ryan Williams] use task's stageAttemptId in TaskEnd event","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/c0b7df68f81c2a2a9c1065009fe75c278fa30499"},{"sha":"e2f38167f8b5678ac45794eacb9c7bb9b951af82","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9376] [SQL] use a seed in RandomDataGeneratorSuite\n\nMake this test deterministic, i.e. make sure this test can be passed no matter how many times we run it.\n\nThe origin implementation uses a random seed and gives a chance that we may break the null check assertion `assert(Iterator.fill(100)(generator()).contains(null))`.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7691 from cloud-fan/seed and squashes the following commits:\n\neae7281 [Wenchen Fan] use a seed in RandomDataGeneratorSuite","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/e2f38167f8b5678ac45794eacb9c7bb9b951af82"},{"sha":"1f7b3d9dc7c2ed9d31f9083284cf900fd4c21e42","author":{"email":"c79bf91d28db09c8453a7483560f9fe421e09629@gmail.com","name":"George Dittmar"},"message":"[SPARK-7423] [MLLIB] Modify ClassificationModel and Probabalistic model to use Vector.argmax\n\nUse Vector.argmax call instead of converting to dense vector before calculating predictions.\n\nAuthor: George Dittmar <georgedittmar@gmail.com>\n\nCloses #7670 from GeorgeDittmar/sprk-7423 and squashes the following commits:\n\ne796747 [George Dittmar] Changing ClassificationModel and ProbabilisticClassificationModel to use Vector.argmax instead of converting to DenseVector","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/1f7b3d9dc7c2ed9d31f9083284cf900fd4c21e42"},{"sha":"dd9ae7945ab65d353ed2b113e0c1a00a0533ffd6","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9351] [SQL] remove literals from grouping expressions in Aggregate\n\nliterals in grouping expressions have no effect at all, only make our grouping key bigger, so we should remove them in Optimizer.\n\nI also make old and new aggregation code consistent about literals in grouping here. In old aggregation, actually literals in grouping are already removed but new aggregation is not. So I explicitly make it a rule in Optimizer.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7583 from cloud-fan/minor and squashes the following commits:\n\n471adff [Wenchen Fan] add test\n0839925 [Wenchen Fan] use transformDown when rewrite final result expressions","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/dd9ae7945ab65d353ed2b113e0c1a00a0533ffd6"},{"sha":"75438422c2cd90dca53f84879cddecfc2ee0e957","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9369][SQL] Support IntervalType in UnsafeRow\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7688 from cloud-fan/interval and squashes the following commits:\n\n5b36b17 [Wenchen Fan] fix codegen\na99ed50 [Wenchen Fan] address comment\n9e6d319 [Wenchen Fan] Support IntervalType in UnsafeRow","distinct":true,"url":"https://api.github.com/repos/huitseeker/spark/commits/75438422c2cd90dca53f84879cddecfc2ee0e957"}]}