{"push_id":763003481,"size":59,"distinct_size":59,"ref":"refs/heads/master","head":"ba2a07e2b6c5a39597b64041cd5bf342ef9631f5","before":"cf016075a006034c24c5b758edb279f3e151d25d","commits":[{"sha":"ae2370e72f93db8a28b262e8252c55fe1fc9873c","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10005] [SQL] Fixes schema merging for nested structs\n\nIn case of schema merging, we only handled first level fields when converting Parquet groups to `InternalRow`s. Nested struct fields are not properly handled.\n\nFor example, the schema of a Parquet file to be read can be:\n\n```\nmessage individual {\n  required group f1 {\n    optional binary f11 (utf8);\n  }\n}\n```\n\nwhile the global schema is:\n\n```\nmessage global {\n  required group f1 {\n    optional binary f11 (utf8);\n    optional int32 f12;\n  }\n}\n```\n\nThis PR fixes this issue by padding missing fields when creating actual converters.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8228 from liancheng/spark-10005/nested-schema-merging.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/ae2370e72f93db8a28b262e8252c55fe1fc9873c"},{"sha":"26e760581fdf7ca913da93fa80e73b7ddabcedf6","author":{"email":"bc4a65f227ddcd81bb538c67e810a3e11696e8bf@gmail.com","name":"Yu ISHIKAWA"},"message":"[SPARK-9871] [SPARKR] Add expression functions into SparkR which have a variable parameter\n\n### Summary\n\n- Add `lit` function\n- Add `concat`, `greatest`, `least` functions\n\nI think we need to improve `collect` function in order to implement `struct` function. Since `collect` doesn't work with arguments which includes a nested `list` variable. It seems that a list against `struct` still has `jobj` classes. So it would be better to solve this problem on another issue.\n\n### JIRA\n[[SPARK-9871] Add expression functions into SparkR which have a variable parameter - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-9871)\n\nAuthor: Yu ISHIKAWA <yuu.ishikawa@gmail.com>\n\nCloses #8194 from yu-iskw/SPARK-9856.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/26e760581fdf7ca913da93fa80e73b7ddabcedf6"},{"sha":"3ff81ad2dedc7cc3defb6e418d7c5fb415d56026","author":{"email":"2d9aeb32a8342495c775846b8b174f850c2ddf16@gmail.com","name":"Calvin Jia"},"message":"[SPARK-9199] [CORE] Upgrade Tachyon version from 0.7.0 -> 0.7.1.\n\nUpdates the tachyon-client version to the latest release.\n\nThe main difference between 0.7.0 and 0.7.1 on the client side is to support running Tachyon on local file system by default.\n\nNo new non-Tachyon dependencies are added, and no code changes are required since the client API has not changed.\n\nAuthor: Calvin Jia <jia.calvin@gmail.com>\n\nCloses #8235 from calvinjia/spark-9199-master.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/3ff81ad2dedc7cc3defb6e418d7c5fb415d56026"},{"sha":"f7efda3975d46a8ce4fd720b3730127ea482560b","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-9959] [MLLIB] Association Rules Java Compatibility\n\nmengxr\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #8206 from feynmanliang/SPARK-9959-arules-java.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/f7efda3975d46a8ce4fd720b3730127ea482560b"},{"sha":"76c155dd4483d58499e5cb66e5e9373bb771dbeb","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-7837] [SQL] Avoids double closing output writers when commitTask() fails\n\nWhen inserting data into a `HadoopFsRelation`, if `commitTask()` of the writer container fails, `abortTask()` will be invoked. However, both `commitTask()` and `abortTask()` try to close the output writer(s). The problem is that, closing underlying writers may not be an idempotent operation. E.g., `ParquetRecordWriter.close()` throws NPE when called twice.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8236 from liancheng/spark-7837/double-closing.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/76c155dd4483d58499e5cb66e5e9373bb771dbeb"},{"sha":"ed092a06c28dfa8204b473a5c964a9ef9a6b655e","author":{"email":"bc6454280780a091e4ce61732d29d654dd50f5a3@qubole.com","name":"Rohit Agarwal"},"message":"[SPARK-9924] [WEB UI] Don't schedule checkForLogs while some of them are already running.\n\nAuthor: Rohit Agarwal <rohita@qubole.com>\n\nCloses #8153 from mindprince/SPARK-9924.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/ed092a06c28dfa8204b473a5c964a9ef9a6b655e"},{"sha":"f68d024096c90936f9aa4e325141b39f08c72476","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-7736] [CORE] [YARN] Make pyspark fail YARN app on failure.\n\nThe YARN backend doesn't like when user code calls `System.exit`,\nsince it cannot know the exit status and thus cannot set an\nappropriate final status for the application.\n\nSo, for pyspark, avoid that call and instead throw an exception with\nthe exit code. SparkSubmit handles that exception and exits with\nthe given exit code, while YARN uses the exit code as the failure\ncode for the Spark app.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #7751 from vanzin/SPARK-9416.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/f68d024096c90936f9aa4e325141b39f08c72476"},{"sha":"a4acdabb103f6d04603163c9555c1ddc413c3b80","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9950] [SQL] Wrong Analysis Error for grouping/aggregating on struct fields\n\nThis issue has been fixed by https://github.com/apache/spark/pull/8215, this PR added regression test for it.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8222 from cloud-fan/minor and squashes the following commits:\n\n0bbfb1c [Wenchen Fan] fix style...\n7e2d8d9 [Wenchen Fan] add test","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/a4acdabb103f6d04603163c9555c1ddc413c3b80"},{"sha":"f10660fe7b809be2059da4f9781a5743f117a35a","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10036] [SQL] Load JDBC driver in DataFrameReader.jdbc and DataFrameWriter.jdbc\n\nThis PR uses `JDBCRDD.getConnector` to load JDBC driver before creating connection in `DataFrameReader.jdbc` and `DataFrameWriter.jdbc`.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8232 from zsxwing/SPARK-10036 and squashes the following commits:\n\nadf75de [zsxwing] Add extraOptions to the connection properties\n57f59d4 [zsxwing] Load JDBC driver in DataFrameReader.jdbc and DataFrameWriter.jdbc","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/f10660fe7b809be2059da4f9781a5743f117a35a"},{"sha":"b265e282b62954548740a5767e97ab1678c65194","author":{"email":"348ff379544b45aa3f16e53a543620141423a0f6@gmail.com","name":"Yijie Shen"},"message":"[SPARK-9526] [SQL] Utilize randomized tests to reveal potential bugs in sql expressions\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9526\n\nThis PR is a follow up of #7830, aiming at utilizing randomized tests to reveal more potential bugs in sql expression.\n\nAuthor: Yijie Shen <henry.yijieshen@gmail.com>\n\nCloses #7855 from yjshen/property_check.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/b265e282b62954548740a5767e97ab1678c65194"},{"sha":"772e7c18fb1a79c0f080408cb43307fe89a4fa04","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9592] [SQL] Fix Last function implemented based on AggregateExpression1.\n\nhttps://issues.apache.org/jira/browse/SPARK-9592\n\n#8113 has the fundamental fix. But, if we want to minimize the number of changed lines, we can go with this one. Then, in 1.6, we merge #8113.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #8172 from yhuai/lastFix and squashes the following commits:\n\nb28c42a [Yin Huai] Regression test.\naf87086 [Yin Huai] Fix last.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/772e7c18fb1a79c0f080408cb43307fe89a4fa04"},{"sha":"fdaf17f63f751f02623414fbc7d0a2f545364050","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-10068] [MLLIB] Adds links to MLlib types, algos, utilities listing\n\nmengxr jkbradley\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #8255 from feynmanliang/SPARK-10068.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/fdaf17f63f751f02623414fbc7d0a2f545364050"},{"sha":"088b11ec5949e135cb3db2a1ce136837e046c288","author":{"email":"f343088f0cf5f2dc6d28ebac254bf6ea7a2528e2@sabhyankar-MBP.Samavihome","name":"Sameer Abhyankar"},"message":"[SPARK-8920] [MLLIB] Add @since tags to mllib.linalg\n\nAuthor: Sameer Abhyankar <sabhyankar@sabhyankar-MBP.Samavihome>\nAuthor: Sameer Abhyankar <sabhyankar@sabhyankar-MBP.local>\n\nCloses #7729 from sabhyankar/branch_8920.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/088b11ec5949e135cb3db2a1ce136837e046c288"},{"sha":"52ae952574f5d641a398dd185e09e5a79318c8a9","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-9974] [BUILD] [SQL] Makes sure com.twitter:parquet-hadoop-bundle:1.6.0 is in SBT assembly jar\n\nPR #7967 enables Spark SQL to persist Parquet tables in Hive compatible format when possible. One of the consequence is that, we have to set input/output classes to `MapredParquetInputFormat`/`MapredParquetOutputFormat`, which rely on com.twitter:parquet-hadoop:1.6.0 bundled with Hive 1.2.1.\n\nWhen loading such a table in Spark SQL, `o.a.h.h.ql.metadata.Table` first loads these input/output format classes, and thus classes in com.twitter:parquet-hadoop:1.6.0.  However, the scope of this dependency is defined as \"runtime\", and is not packaged into Spark assembly jar.  This results in a `ClassNotFoundException`.\n\nThis issue can be worked around by asking users to add parquet-hadoop 1.6.0 via the `--driver-class-path` option.  However, considering Maven build is immune to this problem, I feel it can be confusing and inconvenient for users.\n\nSo this PR fixes this issue by changing scope of parquet-hadoop 1.6.0 to \"compile\".\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8198 from liancheng/spark-9974/bundle-parquet-1.6.0.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/52ae952574f5d641a398dd185e09e5a79318c8a9"},{"sha":"0076e8212334c613599dcbc2ac23f49e9e50cc44","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9768] [PYSPARK] [ML] Add Python API and user guide for ml.feature.ElementwiseProduct\n\nAdd Python API, user guide and example for ml.feature.ElementwiseProduct.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8061 from yanboliang/SPARK-9768.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/0076e8212334c613599dcbc2ac23f49e9e50cc44"},{"sha":"18523c130548f0438dff8d1f25531fd2ed36e517","author":{"email":"631741d0dfe088451ff3c2939a3886b1e6dfed33@gmail.com","name":"Prayag Chandran"},"message":"SPARK-8916 [Documentation, MLlib] Add @since tags to mllib.regression\n\nAdded since tags to mllib.regression\n\nAuthor: Prayag Chandran <prayagchandran@gmail.com>\n\nCloses #7518 from prayagchandran/sinceTags and squashes the following commits:\n\nfa4dda2 [Prayag Chandran] Re-formatting\n6c6d584 [Prayag Chandran] Corrected a few tags. Removed few unnecessary tags\n1a0365f [Prayag Chandran] Reformating and adding a few more tags\n89fdb66 [Prayag Chandran] SPARK-8916 [Documentation, MLlib] Add @since tags to mllib.regression","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/18523c130548f0438dff8d1f25531fd2ed36e517"},{"sha":"0b6b01761370629ce387c143a25d41f3a334ff28","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-9898] [MLLIB] Prefix Span user guide\n\nAdds user guide for `PrefixSpan`, including Scala and Java example code.\n\nmengxr zhangjiajin\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #8253 from feynmanliang/SPARK-9898.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/0b6b01761370629ce387c143a25d41f3a334ff28"},{"sha":"f9d1a92aa1bac4494022d78559b871149579e6e8","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"[SPARK-7707] User guide and example code for KernelDensity\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #8230 from sryza/sandy-spark-7707.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/f9d1a92aa1bac4494022d78559b871149579e6e8"},{"sha":"c90c605dc6a876aef3cc204ac15cd65bab9743ad","author":{"email":"dd19fa80624ab884ac33cf50a82d9bdcaeb1d7b3@cloudera.com","name":"jose.cambronero"},"message":"[SPARK-9902] [MLLIB] Add Java and Python examples to user guide for 1-sample KS test\n\nadded doc examples for python.\n\nAuthor: jose.cambronero <jose.cambronero@cloudera.com>\n\nCloses #8154 from josepablocam/spark_9902.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/c90c605dc6a876aef3cc204ac15cd65bab9743ad"},{"sha":"ee093c8b927e8d488aeb76115c7fb0de96af7720","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-10059] [YARN] Explicitly add JSP dependencies for tests.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #8251 from vanzin/SPARK-10059.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/ee093c8b927e8d488aeb76115c7fb0de96af7720"}]}