{"push_id":776820260,"size":14,"distinct_size":14,"ref":"refs/heads/master","head":"391e6be0ae883f3ea0fab79463eb8b618af79afb","before":"1bfd9347822df65e76201c4c471a26488d722319","commits":[{"sha":"8d2ab75d3b71b632f2394f2453af32f417cb45e5","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-10353] [MLLIB] BLAS gemm not scaling when beta = 0.0 for some subset of matrix multiplications\n\nmengxr jkbradley rxin\n\nIt would be great if this fix made it into RC3!\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #8525 from brkyvz/blas-scaling.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/8d2ab75d3b71b632f2394f2453af32f417cb45e5"},{"sha":"35e896a79bb5e72d63b82b047f46f4f6fa2e1970","author":{"email":"cbb7353e6d953ef360baf960c122346276c6e320@databricks.com","name":"Patrick Wendell"},"message":"SPARK-9545, SPARK-9547: Use Maven in PRB if title contains \"[test-maven]\"\n\nThis is just some small glue code to actually make use of the\nAMPLAB_JENKINS_BUILD_TOOL switch. As far as I can tell, we actually\ndon't currently use the Maven support in the tool even though it exists.\nThis patch switches to Maven when the PR title contains \"test-maven\".\n\nThere are a few small other pieces of cleanup in the patch as well.\n\nAuthor: Patrick Wendell <patrick@databricks.com>\n\nCloses #7878 from pwendell/maven-tests.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/35e896a79bb5e72d63b82b047f46f4f6fa2e1970"},{"sha":"8694c3ad7dcafca9563649e93b7a08076748d6f2","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-10351] [SQL] Fixes UTF8String.fromAddress to handle off-heap memory\n\nCC rxin marmbrus\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #8523 from feynmanliang/SPARK-10351.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/8694c3ad7dcafca9563649e93b7a08076748d6f2"},{"sha":"f0f563a3c43fc9683e6920890cce44611c0c5f4b","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-100354] [MLLIB] fix some apparent memory issues in k-means|| initializaiton\n\n* do not cache first cost RDD\n* change following cost RDD cache level to MEMORY_AND_DISK\n* remove Vector wrapper to save a object per instance\n\nFurther improvements will be addressed in SPARK-10329\n\ncc: yu-iskw HuJiayin\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8526 from mengxr/SPARK-10354.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/f0f563a3c43fc9683e6920890cce44611c0c5f4b"},{"sha":"72f6dbf7b0c8b271f5f9c762374422c69c8ab43d","author":{"email":"cc4787d1318e1afa90a5bff59d330251c333bd4a@gmail.com","name":"EugenCepoi"},"message":"[SPARK-8730] Fixes - Deser objects containing a primitive class attribute\n\nAuthor: EugenCepoi <cepoi.eugen@gmail.com>\n\nCloses #7122 from EugenCepoi/master.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/72f6dbf7b0c8b271f5f9c762374422c69c8ab43d"},{"sha":"4a5fe091658b1d06f427e404a11a84fc84f953c5","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10369] [STREAMING] Don't remove ReceiverTrackingInfo when deregisterReceivering since we may reuse it later\n\n`deregisterReceiver` should not remove `ReceiverTrackingInfo`. Otherwise, it will throw `java.util.NoSuchElementException: key not found` when restarting it.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8538 from zsxwing/SPARK-10369.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/4a5fe091658b1d06f427e404a11a84fc84f953c5"},{"sha":"a2d5c72091b1c602694dbca823a7b26f86b02864","author":{"email":"d5493c25651f198266b02bc9287959eaadab29fc@gmail.com","name":"sureshthalamati"},"message":"[SPARK-10170] [SQL] Add DB2 JDBC dialect support.\n\nData frame write to DB2 database is failing because by default JDBC data source implementation is generating a table schema with DB2 unsupported data types TEXT for String, and BIT1(1) for Boolean.\n\nThis patch registers DB2 JDBC Dialect that maps String, Boolean to valid DB2 data types.\n\nAuthor: sureshthalamati <suresh.thalamati@gmail.com>\n\nCloses #8393 from sureshthalamati/db2_dialect_spark-10170.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/a2d5c72091b1c602694dbca823a7b26f86b02864"},{"sha":"23e39cc7b1bb7f1087c4706234c9b5165a571357","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9954] [MLLIB] use first 128 nonzeros to compute Vector.hashCode\n\nThis could help reduce hash collisions, e.g., in `RDD[Vector].repartition`. jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8182 from mengxr/SPARK-9954.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/23e39cc7b1bb7f1087c4706234c9b5165a571357"},{"sha":"5b3245d6dff65972fc39c73f90d5cbdf84d19129","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-8472] [ML] [PySpark] Python API for DCT\n\nAdd Python API for ml.feature.DCT.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8485 from yanboliang/spark-8472.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/5b3245d6dff65972fc39c73f90d5cbdf84d19129"},{"sha":"540bdee93103a73736d282b95db6a8cda8f6a2b1","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-10341] [SQL] fix memory starving in unsafe SMJ\n\nIn SMJ, the first ExternalSorter could consume all the memory before spilling, then the second can not even acquire the first page.\n\nBefore we have a better memory allocator, SMJ should call prepare() before call any compute() of it's children.\n\ncc rxin JoshRosen\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8511 from davies/smj_memory.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/540bdee93103a73736d282b95db6a8cda8f6a2b1"},{"sha":"fe16fd0b8b717f01151bc659ec3299dab091c97a","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10349] [ML] OneVsRest use 'when ... otherwise' not UDF to generate new label at binary reduction\n\nCurrently OneVsRest use UDF to generate new binary label during training.\nConsidering that [SPARK-7321](https://issues.apache.org/jira/browse/SPARK-7321) has been merged, we can use ```when ... otherwise``` which will be more efficiency.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8519 from yanboliang/spark-10349.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/fe16fd0b8b717f01151bc659ec3299dab091c97a"},{"sha":"52ea399e6ee37b7c44aae7709863e006fca88906","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10355] [ML] [PySpark] Add Python API for SQLTransformer\n\nAdd Python API for SQLTransformer\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8527 from yanboliang/spark-10355.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/52ea399e6ee37b7c44aae7709863e006fca88906"},{"sha":"d65656c455d19b83c6412571873586b458aa355e","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10378][SQL][Test] Remove HashJoinCompatibilitySuite.\n\nThey don't bring much value since we now have better unit test coverage for hash joins. This will also help reduce the test time.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8542 from rxin/SPARK-10378.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/d65656c455d19b83c6412571873586b458aa355e"},{"sha":"391e6be0ae883f3ea0fab79463eb8b618af79afb","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10301] [SQL] Fixes schema merging for nested structs\n\nThis PR can be quite challenging to review.  I'm trying to give a detailed description of the problem as well as its solution here.\n\nWhen reading Parquet files, we need to specify a potentially nested Parquet schema (of type `MessageType`) as requested schema for column pruning.  This Parquet schema is translated from a Catalyst schema (of type `StructType`), which is generated by the query planner and represents all requested columns.  However, this translation can be fairly complicated because of several reasons:\n\n1.  Requested schema must conform to the real schema of the physical file to be read.\n\n    This means we have to tailor the actual file schema of every individual physical Parquet file to be read according to the given Catalyst schema.  Fortunately we are already doing this in Spark 1.5 by pushing request schema conversion to executor side in PR #7231.\n\n1.  Support for schema merging.\n\n    A single Parquet dataset may consist of multiple physical Parquet files come with different but compatible schemas.  This means we may request for a column path that doesn't exist in a physical Parquet file.  All requested column paths can be nested.  For example, for a Parquet file schema\n\n    ```\n    message root {\n      required group f0 {\n        required group f00 {\n          required int32 f000;\n          required binary f001 (UTF8);\n        }\n      }\n    }\n    ```\n\n    we may request for column paths defined in the following schema:\n\n    ```\n    message root {\n      required group f0 {\n        required group f00 {\n          required binary f001 (UTF8);\n          required float f002;\n        }\n      }\n\n      optional double f1;\n    }\n    ```\n\n    Notice that we pruned column path `f0.f00.f000`, but added `f0.f00.f002` and `f1`.\n\n    The good news is that Parquet handles non-existing column paths properly and always returns null for them.\n\n1.  The map from `StructType` to `MessageType` is a one-to-many map.\n\n    This is the most unfortunate part.\n\n    Due to historical reasons (dark histories!), schemas of Parquet files generated by different libraries have different \"flavors\".  For example, to handle a schema with a single non-nullable column, whose type is an array of non-nullable integers, parquet-protobuf generates the following Parquet schema:\n\n    ```\n    message m0 {\n      repeated int32 f;\n    }\n    ```\n\n    while parquet-avro generates another version:\n\n    ```\n    message m1 {\n      required group f (LIST) {\n        repeated int32 array;\n      }\n    }\n    ```\n\n    and parquet-thrift spills this:\n\n    ```\n    message m1 {\n      required group f (LIST) {\n        repeated int32 f_tuple;\n      }\n    }\n    ```\n\n    All of them can be mapped to the following _unique_ Catalyst schema:\n\n    ```\n    StructType(\n      StructField(\n        \"f\",\n        ArrayType(IntegerType, containsNull = false),\n        nullable = false))\n    ```\n\n    This greatly complicates Parquet requested schema construction, since the path of a given column varies in different cases.  To read the array elements from files with the above schemas, we must use `f` for `m0`, `f.array` for `m1`, and `f.f_tuple` for `m2`.\n\nIn earlier Spark versions, we didn't try to fix this issue properly.  Spark 1.4 and prior versions simply translate the Catalyst schema in a way more or less compatible with parquet-hive and parquet-avro, but is broken in many other cases.  Earlier revisions of Spark 1.5 only try to tailor the Parquet file schema at the first level, and ignore nested ones.  This caused [SPARK-10301] [spark-10301] as well as [SPARK-10005] [spark-10005].  In PR #8228, I tried to avoid the hard part of the problem and made a minimum change in `CatalystRowConverter` to fix SPARK-10005.  However, when taking SPARK-10301 into consideration, keeping hacking `CatalystRowConverter` doesn't seem to be a good idea.  So this PR is an attempt to fix the problem in a proper way.\n\nFor a given physical Parquet file with schema `ps` and a compatible Catalyst requested schema `cs`, we use the following algorithm to tailor `ps` to get the result Parquet requested schema `ps'`:\n\nFor a leaf column path `c` in `cs`:\n\n- if `c` exists in `cs` and a corresponding Parquet column path `c'` can be found in `ps`, `c'` should be included in `ps'`;\n- otherwise, we convert `c` to a Parquet column path `c\"` using `CatalystSchemaConverter`, and include `c\"` in `ps'`;\n- no other column paths should exist in `ps'`.\n\nThen comes the most tedious part:\n\n> Given `cs`, `ps`, and `c`, how to locate `c'` in `ps`?\n\nUnfortunately, there's no quick answer, and we have to enumerate all possible structures defined in parquet-format spec.  They are:\n\n1.  the standard structure of nested types, and\n1.  cases defined in all backwards-compatibility rules for `LIST` and `MAP`.\n\nThe core part of this PR is `CatalystReadSupport.clipParquetType()`, which tailors a given Parquet file schema according to a requested schema in its Catalyst form.  Backwards-compatibility rules of `LIST` and `MAP` are covered in `clipParquetListType()` and `clipParquetMapType()` respectively.  The column path selection algorithm is implemented in `clipParquetGroupFields()`.\n\nWith this PR, we no longer need to do schema tailoring in `CatalystReadSupport` and `CatalystRowConverter`.  Another benefit is that, now we can also read Parquet datasets consist of files with different physical Parquet schema but share the same logical schema, for example, files generated by different Parquet libraries.  This situation is illustrated by [this test case] [test-case].\n\n[spark-10301]: https://issues.apache.org/jira/browse/SPARK-10301\n[spark-10005]: https://issues.apache.org/jira/browse/SPARK-10005\n[test-case]: https://github.com/liancheng/spark/commit/38644d8a45175cbdf20d2ace021c2c2544a50ab3#diff-a9b98e28ce3ae30641829dffd1173be2R26\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8509 from liancheng/spark-10301/fix-parquet-requested-schema.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/391e6be0ae883f3ea0fab79463eb8b618af79afb"}]}