{"push_id":720645765,"size":34,"distinct_size":34,"ref":"refs/heads/master","head":"09cb0d9c2dcb83818ced22ff9bd6a51688ea7ffe","before":"351a36d0c54d2f995df956ffb0a4236e12f89aad","commits":[{"sha":"8a9d9cc1561cf157793c90db6700ffa6f1f00a69","author":{"email":"79478dbbc8641d1d6545f348f655464aefbda99b@intel.com","name":"jerryshao"},"message":"[SPARK-7050] [BUILD] Fix Python Kafka test assembly jar not found issue under Maven build\n\n To fix Spark Streaming unit test with maven build. Previously the name and path of maven generated jar is different from sbt, which will lead to following exception. This fix keep the same behavior with both Maven and sbt build.\n\n```\nFailed to find Spark Streaming Kafka assembly jar in /home/xyz/spark/external/kafka-assembly\nYou need to build Spark with  'build/sbt assembly/assembly streaming-kafka-assembly/assembly' or 'build/mvn package' before running this program\n```\n\nAuthor: jerryshao <saisai.shao@intel.com>\n\nCloses #5632 from jerryshao/SPARK-7050 and squashes the following commits:\n\n74b068d [jerryshao] Fix mvn build issue","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/8a9d9cc1561cf157793c90db6700ffa6f1f00a69"},{"sha":"3bb217750ada18a49c40d974ac57050ef2abfd2c","author":{"email":"90f7c8ad61634a0fbb1c36c20ac3204104bde5c4@gmail.com","name":"Kashif Rasul"},"message":"[SPARK-8872] [MLLIB] added verification results from R for FPGrowthSuite\n\nAuthor: Kashif Rasul <kashif.rasul@gmail.com>\n\nCloses #7269 from kashif/SPARK-8872 and squashes the following commits:\n\n2d5457f [Kashif Rasul] added R code for FP Int type\n3de6808 [Kashif Rasul] added verification results from R for FPGrowthSuite","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/3bb217750ada18a49c40d974ac57050ef2abfd2c"},{"sha":"bf02e377168f39459d5c216e939097ae5705f573","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-8894] [SPARKR] [DOC] Example code errors in SparkR documentation.\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #7287 from sun-rui/SPARK-8894 and squashes the following commits:\n\nda63898 [Sun Rui] [SPARK-8894][SPARKR][DOC] Example code errors in SparkR documentation.","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/bf02e377168f39459d5c216e939097ae5705f573"},{"sha":"6722aca809ddc28aa20abf3bbb2e0de8629a9903","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-8785] [SQL] Improve Parquet schema merging\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-8785\n\nCurrently, the parquet schema merging (`ParquetRelation2.readSchema`) may spend much time to merge duplicate schema. We can select only non duplicate schema and merge them later.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #7182 from viirya/improve_parquet_merging and squashes the following commits:\n\n5cf934f [Liang-Chi Hsieh] Refactor it to make it faster.\nf3411ea [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into improve_parquet_merging\na63c3ff [Liang-Chi Hsieh] Improve Parquet schema merging.","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/6722aca809ddc28aa20abf3bbb2e0de8629a9903"},{"sha":"3e831a26965a5e92210431f9ad6935f70aa01b48","author":{"email":"e58867706a26c17cde71f0f9f5e010ffb994f9e8@gmail.com","name":"Takeshi YAMAMURO"},"message":"[SPARK-6912] [SQL] Throw an AnalysisException when unsupported Java Map<K,V> types used in Hive UDF\n\nTo make UDF developers understood, throw an exception when unsupported Map<K,V> types used in Hive UDF. This fix is the same with #7248.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #7257 from maropu/ThrowExceptionWhenMapUsed and squashes the following commits:\n\n916099a [Takeshi YAMAMURO] Fix style errors\n7886dcc [Takeshi YAMAMURO] Throw an exception when Map<> used in Hive UDF","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/3e831a26965a5e92210431f9ad6935f70aa01b48"},{"sha":"74335b31072951244967f878d8b766cd1bfc2ac6","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-5707] [SQL] fix serialization of generated projection\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #7272 from davies/fix_projection and squashes the following commits:\n\n075ef76 [Davies Liu] fix codegen with BroadcastHashJion","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/74335b31072951244967f878d8b766cd1bfc2ac6"},{"sha":"0ba98c04c726a827df8cb19b0db17c352a647960","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-8753][SQL] Create an IntervalType data type\n\nWe need a new data type to represent time intervals. Because we can't determine how many days in a month, so we need 2 values for interval: a int `months`, a long `microseconds`.\n\nThe interval literal syntax looks like:\n`interval 3 years -4 month 4 weeks 3 second`\n\nBecause we use number of 100ns as value of `TimestampType`, so it may not makes sense to support nano second unit.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7226 from cloud-fan/interval and squashes the following commits:\n\n632062d [Wenchen Fan] address comments\nac348c3 [Wenchen Fan] use case class\n0342d2e [Wenchen Fan] use array byte\ndf9256c [Wenchen Fan] fix style\nfd6f18a [Wenchen Fan] address comments\n1856af3 [Wenchen Fan] support interval type","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/0ba98c04c726a827df8cb19b0db17c352a647960"},{"sha":"f61c989b404808f79a58b6503cf3835cf602528a","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-8888][SQL] Use java.util.HashMap in DynamicPartitionWriterContainer.\n\nJust a baby step towards making it more efficient.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7282 from rxin/SPARK-8888 and squashes the following commits:\n\n3da51ae [Reynold Xin] [SPARK-8888][SQL] Use java.util.HashMap in DynamicPartitionWriterContainer.","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/f61c989b404808f79a58b6503cf3835cf602528a"},{"sha":"26d9b6b8cae9ac6593f78ab98dd45a25d03cf71c","author":{"email":"2a3e9bfdf45f961621c630e88a1aec4abd4f97e1@sogou-inc.com","name":"Tao Li"},"message":"[SPARK-8657] [YARN] Fail to upload resource to viewfs\n\nFail to upload resource to viewfs in spark-1.4\nJIRA Link: https://issues.apache.org/jira/browse/SPARK-8657\n\nAuthor: Tao Li <litao@sogou-inc.com>\n\nCloses #7125 from litao-buptsse/SPARK-8657-for-master and squashes the following commits:\n\n65b13f4 [Tao Li] [SPARK-8657] [YARN] Fail to upload resource to viewfs","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/26d9b6b8cae9ac6593f78ab98dd45a25d03cf71c"},{"sha":"374c8a8a4a8ac4171d312a6c31080a6724e55c60","author":{"email":"ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu","name":"Shivaram Venkataraman"},"message":"[SPARK-8900] [SPARKR] Fix sparkPackages in init documentation\n\ncc pwendell\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #7293 from shivaram/sparkr-packages-doc and squashes the following commits:\n\nc91471d [Shivaram Venkataraman] Fix sparkPackages in init documentation","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/374c8a8a4a8ac4171d312a6c31080a6724e55c60"},{"sha":"2b40365d76b7d9d382ad5077cdf979906bca17f2","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-7785] [MLLIB] [PYSPARK] Add __str__ and __repr__ to Matrices\n\nAdding __str__ and  __repr__ to DenseMatrix and SparseMatrix\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #6342 from MechCoder/spark-7785 and squashes the following commits:\n\n7b9a82c [MechCoder] Add tests for greater than 16 elements\nb88e9dd [MechCoder] Increment limit to 16\n1425a01 [MechCoder] Change tests\n36bd166 [MechCoder] Change str and repr representation\n97f0da9 [MechCoder] zip is same as izip in python3\n94ca4b2 [MechCoder] Added doctests and iterate over values instead of colPtrs\nb26fa89 [MechCoder] minor\n394dde9 [MechCoder] [SPARK-7785] Add __str__ and __repr__ to Matrices","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/2b40365d76b7d9d382ad5077cdf979906bca17f2"},{"sha":"f031543782e8f0f5b6a4471ba1c1d5c53efbe5cd","author":{"email":"23eda46daff42fe3e8c6608c7cb15107121d9ee4@apache.org","name":"Keuntae Park"},"message":"[SPARK-8783] [SQL] CTAS with WITH clause does not work\n\nCurrently, CTESubstitution only handles the case that WITH is on the top of the plan.\nI think it SHOULD handle the case that WITH is child of CTAS.\nThis patch simply changes 'match' to 'transform' for recursive search of WITH in the plan.\n\nAuthor: Keuntae Park <sirpkt@apache.org>\n\nCloses #7180 from sirpkt/SPARK-8783 and squashes the following commits:\n\ne4428f0 [Keuntae Park] Merge remote-tracking branch 'upstream/master' into CTASwithWITH\n1671c77 [Keuntae Park] WITH clause can be inside CTAS","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/f031543782e8f0f5b6a4471ba1c1d5c53efbe5cd"},{"sha":"c5532e2fe700978da4bdfdb54a522f5934c3db55","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-8457] [ML] NGram Documentation\n\nAdd documentation for NGram feature transformer.\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #7244 from feynmanliang/SPARK-8457 and squashes the following commits:\n\n5aface9 [Feynman Liang] Pretty print Scala output and add API doc to each codetab\n60d5ac0 [Feynman Liang] Inline API doc and fix indentation\n736ccbc [Feynman Liang] NGram feature transformer documentation","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/c5532e2fe700978da4bdfdb54a522f5934c3db55"},{"sha":"8f3cd93278337dc10b9dd3a344d6f7b51ba9960d","author":{"email":"f3a0f1569f35108e9492b93611ff7ee9bac9a90a@us.ibm.com”","name":"Alok Singh"},"message":"[SPARK-8909][Documentation] Change the scala example in sql-programmi…\n\n…ng-guide#Manually Specifying Options to be in sync with java,python, R version\n\nAuthor: Alok Singh <“singhal@us.ibm.com”>\n\nCloses #7299 from aloknsingh/aloknsingh_SPARK-8909 and squashes the following commits:\n\nd3c20ba [Alok Singh] fix the file to .parquet from .json\nd476140 [Alok Singh] [SPARK-8909][Documentation] Change the scala example in sql-programming-guide#Manually Specifying Options to be in sync with java,python, R version","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/8f3cd93278337dc10b9dd3a344d6f7b51ba9960d"},{"sha":"00b265f12c0f0271b7036f831fee09b694908b29","author":{"email":"9d01406527e5c15536e8dcb4e0c5005dca46786b@netflix.com","name":"Cheolsoo Park"},"message":"[SPARK-8908] [SQL] Add () to distinct definition in dataframe\n\nAdding `()` to the definition of `distinct` in DataFrame allows distinct to be called with parentheses, which is consistent with `dropDuplicates`.\n\nAuthor: Cheolsoo Park <cheolsoop@netflix.com>\n\nCloses #7298 from piaozhexiu/SPARK-8908 and squashes the following commits:\n\n7f0d923 [Cheolsoo Park] Add () to distinct definition in dataframe","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/00b265f12c0f0271b7036f831fee09b694908b29"},{"sha":"57221934e0376e5bb8421dc35d4bf91db4deeca1","author":{"email":"2a07432b1b7252977a4c2dfc8e6d6d5ac763a480@netflix.com","name":"DB Tsai"},"message":"[SPARK-8700][ML] Disable feature scaling in Logistic Regression\n\nAll compressed sensing applications, and some of the regression use-cases will have better result by turning the feature scaling off. However, if we implement this naively by training the dataset without doing any standardization, the rate of convergency will not be good. This can be implemented by still standardizing the training dataset but we penalize each component differently to get effectively the same objective function but a better numerical problem. As a result, for those columns with high variances, they will be penalized less, and vice versa. Without this, since all the features are standardized, so they will be penalized the same.\n\nIn R, there is an option for this.\n`standardize`\nLogical flag for x variable standardization, prior to fitting the model sequence. The coefficients are always returned on the original scale. Default is standardize=TRUE. If variables are in the same units already, you might not wish to standardize. See details below for y standardization with family=\"gaussian\".\n\n+cc holdenk mengxr jkbradley\n\nAuthor: DB Tsai <dbt@netflix.com>\n\nCloses #7080 from dbtsai/lors and squashes the following commits:\n\n877e6c7 [DB Tsai] repahse the doc\n7cf45f2 [DB Tsai] address feedback\n78d75c9 [DB Tsai] small change\nc2c9e60 [DB Tsai] style\n6e1a8e0 [DB Tsai] first commit","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/57221934e0376e5bb8421dc35d4bf91db4deeca1"},{"sha":"5687f76552369fa20b3a4385eab4810214653aa7","author":{"email":"9d468ac3e51a0dd55cd858437e2237db1fbaae1b@gmail.com","name":"Daniel Darabos"},"message":"[SPARK-8902] Correctly print hostname in error\n\nWith \"+\" the strings are separate expressions, and format() is called on the last string before concatenation. (So substitution does not happen.) Without \"+\" the string literals are merged first by the parser, so format() is called on the complete string.\n\nShould I make a JIRA for this?\n\nAuthor: Daniel Darabos <darabos.daniel@gmail.com>\n\nCloses #7288 from darabos/patch-2 and squashes the following commits:\n\nbe0d3b7 [Daniel Darabos] Correctly print hostname in error","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/5687f76552369fa20b3a4385eab4810214653aa7"},{"sha":"4ffc27caaf46ffac56c3c0b3e928f1aff227a184","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-6123] [SPARK-6775] [SPARK-6776] [SQL] Refactors Parquet read path for interoperability and backwards-compatibility\n\nThis PR is a follow-up of #6617 and is part of [SPARK-6774] [2], which aims to ensure interoperability and backwards-compatibility for Spark SQL Parquet support.  And this one fixes the read path.  Now Spark SQL is expected to be able to read legacy Parquet data files generated by most (if not all) common libraries/tools like parquet-thrift, parquet-avro, and parquet-hive. However, we still need to refactor the write path to write standard Parquet LISTs and MAPs ([SPARK-8848] [4]).\n\n### Major changes\n\n1. `CatalystConverter` class hierarchy refactoring\n\n   - Replaces `CatalystConverter` trait with a much simpler `ParentContainerUpdater`.\n\n     Now instead of extending the original `CatalystConverter` trait, every converter class accepts an updater which is responsible for propagating the converted value to some parent container. For example, appending array elements to a parent array buffer, appending a key-value pairs to a parent mutable map, or setting a converted value to some specific field of a parent row. Root converter doesn't have a parent and thus uses a `NoopUpdater`.\n\n     This simplifies the design since converters don't need to care about details of their parent converters anymore.\n\n   - Unifies `CatalystRootConverter`, `CatalystGroupConverter` and `CatalystPrimitiveRowConverter` into `CatalystRowConverter`\n\n     Specifically, now all row objects are represented by `SpecificMutableRow` during conversion.\n\n   - Refactors `CatalystArrayConverter`, and removes `CatalystArrayContainsNullConverter` and `CatalystNativeArrayConverter`\n\n     `CatalystNativeArrayConverter` was probably designed with the intention of avoiding boxing costs. However, the way it uses Scala generics actually doesn't achieve this goal.\n\n     The new `CatalystArrayConverter` handles both nullable and non-nullable array elements in a consistent way.\n\n   - Implements backwards-compatibility rules in `CatalystArrayConverter`\n\n     When Parquet records are being converted, schema of Parquet files should have already been verified. So we only need to care about the structure rather than field names in the Parquet schema. Since all map objects represented in legacy systems have the same structure as the standard one (see [backwards-compatibility rules for MAP] [1]), we only need to deal with LIST (namely array) in `CatalystArrayConverter`.\n\n2. Requested columns handling\n\n   When specifying requested columns in `RowReadSupport`, we used to use a Parquet `MessageType` converted from a Catalyst `StructType` which contains all requested columns.  This is not preferable when taking compatibility and interoperability into consideration.  Because the actual Parquet file may have different physical structure from the converted schema.\n\n   In this PR, the schema for requested columns is constructed using the following method:\n\n   - For a column that exists in the target Parquet file, we extract the column type by name from the full file schema, and construct a single-field `MessageType` for that column.\n   - For a column that doesn't exist in the target Parquet file, we create a single-field `StructType` and convert it to a `MessageType` using `CatalystSchemaConverter`.\n   - Unions all single-field `MessageType`s into a full schema containing all requested fields\n\n   With this change, we also fix [SPARK-6123] [3] by validating the global schema against each individual Parquet part-files.\n\n### Testing\n\nThis PR also adds compatibility tests for parquet-avro, parquet-thrift, and parquet-hive. Please refer to `README.md` under `sql/core/src/test` for more information about these tests. To avoid build time code generation and adding extra complexity to the build system, Java code generated from testing Thrift schema and Avro IDL is also checked in.\n\n[1]: https://github.com/apache/incubator-parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules-1\n[2]: https://issues.apache.org/jira/browse/SPARK-6774\n[3]: https://issues.apache.org/jira/browse/SPARK-6123\n[4]: https://issues.apache.org/jira/browse/SPARK-8848\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7231 from liancheng/spark-6776 and squashes the following commits:\n\n360fe18 [Cheng Lian] Adds ParquetHiveCompatibilitySuite\nc6fbc06 [Cheng Lian] Removes WIP file committed by mistake\nb8c1295 [Cheng Lian] Excludes the whole parquet package from MiMa\n598c3e8 [Cheng Lian] Adds extra Maven repo for hadoop-lzo, which is a transitive dependency of parquet-thrift\n926af87 [Cheng Lian] Simplifies Parquet compatibility test suites\n7946ee1 [Cheng Lian] Fixes Scala styling issues\n3d7ab36 [Cheng Lian] Fixes .rat-excludes\na8f13bb [Cheng Lian] Using Parquet writer API to do compatibility tests\nf2208cd [Cheng Lian] Adds README.md for Thrift/Avro code generation\n1d390aa [Cheng Lian] Adds parquet-thrift compatibility test\n440f7b3 [Cheng Lian] Adds generated files to .rat-excludes\n13b9121 [Cheng Lian] Adds ParquetAvroCompatibilitySuite\n06cfe9d [Cheng Lian] Adds comments about TimestampType handling\na099d3e [Cheng Lian] More comments\n0cc1b37 [Cheng Lian] Fixes MiMa checks\n884d3e6 [Cheng Lian] Fixes styling issue and reverts unnecessary changes\n802cbd7 [Cheng Lian] Fixes bugs related to schema merging and empty requested columns\n38fe1e7 [Cheng Lian] Adds explicit return type\n7fb21f1 [Cheng Lian] Reverts an unnecessary debugging change\n1781dff [Cheng Lian] Adds test case for SPARK-8811\n6437d4b [Cheng Lian] Assembles requested schema from Parquet file schema\nbcac49f [Cheng Lian] Removes the 16-byte restriction of decimals\na74fb2c [Cheng Lian] More comments\n0525346 [Cheng Lian] Removes old Parquet record converters\n03c3bd9 [Cheng Lian] Refactors Parquet read path to implement backwards-compatibility rules","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/4ffc27caaf46ffac56c3c0b3e928f1aff227a184"},{"sha":"381cb161ba4e3a30f2da3c4ef4ee19869d51f101","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-8068] [MLLIB] Add confusionMatrix method at class MulticlassMetrics in pyspark/mllib\n\nAdd confusionMatrix method at class MulticlassMetrics in pyspark/mllib\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #7286 from yanboliang/spark-8068 and squashes the following commits:\n\n6109fe1 [Yanbo Liang] Add confusionMatrix method at class MulticlassMetrics in pyspark/mllib","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/381cb161ba4e3a30f2da3c4ef4ee19869d51f101"},{"sha":"8c32b2e870c7c250a63e838718df833edf6dea07","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-8877] [MLLIB] Public API for association rule generation\n\nAdds FPGrowth.generateAssociationRules to public API for generating association rules after mining frequent itemsets.\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #7271 from feynmanliang/SPARK-8877 and squashes the following commits:\n\n83b8baf [Feynman Liang] Add API Doc\n867abff [Feynman Liang] Add FPGrowth.generateAssociationRules and change access modifiers for AssociationRules","distinct":true,"url":"https://api.github.com/repos/Lewuathe/spark/commits/8c32b2e870c7c250a63e838718df833edf6dea07"}]}