{"push_id":710807098,"size":80,"distinct_size":1,"ref":"refs/heads/SPARK-6785","head":"9d8021d631c8e61bcb0599becd2248363864577c","before":"b97c3fb68cbf12d932c191c1d15e032eaaf17068","commits":[{"sha":"a458efc66c31dc281af379b914bfa2b077ca6635","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Revert \"[SPARK-7157][SQL] add sampleBy to DataFrame\"\n\nThis reverts commit 0401cbaa8ee51c71f43604f338b65022a479da0a.\n\nThe new test case on Jenkins is failing.","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/a458efc66c31dc281af379b914bfa2b077ca6635"},{"sha":"50c3a86f42d7dfd1acbda65c1e5afbd3db1406df","author":{"email":"cc463b26613f4ba480f0f94ca5cb2cab678c4fcb@databricks.com","name":"Eric Liang"},"message":"[SPARK-6749] [SQL] Make metastore client robust to underlying socket connection loss\n\nThis works around a bug in the underlying RetryingMetaStoreClient (HIVE-10384) by refreshing the metastore client on thrift exceptions. We attempt to emulate the proper hive behavior by retrying only as configured by hiveconf.\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #6912 from ericl/spark-6749 and squashes the following commits:\n\n2d54b55 [Eric Liang] use conf from state\n0e3a74e [Eric Liang] use shim properly\n980b3e5 [Eric Liang] Fix conf parsing hive 0.14 conf.\n92459b6 [Eric Liang] Work around RetryingMetaStoreClient bug","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/50c3a86f42d7dfd1acbda65c1e5afbd3db1406df"},{"sha":"13ae806b255cfb2bd5470b599a95c87a2cd5e978","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[HOTFIX] [BUILD] Fix MiMa checks in master branch; enable MiMa for launcher project\n\nThis commit changes the MiMa tests to test against the released 1.4.0 artifacts rather than 1.4.0-rc4; this change is necessary to fix a Jenkins build break since it seems that the RC4 snapshot is no longer available via Maven.\n\nI also enabled MiMa checks for the `launcher` subproject, which we should have done right after 1.4.0 was released.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #6974 from JoshRosen/mima-hotfix and squashes the following commits:\n\n4b4175a [Josh Rosen] [HOTFIX] [BUILD] Fix MiMa checks in master branch; enable MiMa for launcher project","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/13ae806b255cfb2bd5470b599a95c87a2cd5e978"},{"sha":"09fcf96b8f881988a4bc7fe26a3f6ed12dfb6adb","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-8371] [SQL] improve unit test for MaxOf and MinOf and fix bugs\n\na follow up of https://github.com/apache/spark/pull/6813\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #6825 from cloud-fan/cg and squashes the following commits:\n\n43170cc [Wenchen Fan] fix bugs in code gen","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/09fcf96b8f881988a4bc7fe26a3f6ed12dfb6adb"},{"sha":"cc465fd92482737c21971d82e30d4cf247acf932","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-8138] [SQL] Improves error message when conflicting partition columns are found\n\nThis PR improves the error message shown when conflicting partition column names are detected.  This can be particularly annoying and confusing when there are a large number of partitions while a handful of them happened to contain unexpected temporary file(s).  Now all suspicious directories are listed as below:\n\n```\njava.lang.AssertionError: assertion failed: Conflicting partition column names detected:\n\n        Partition column name list #0: b, c, d\n        Partition column name list #1: b, c\n        Partition column name list #2: b\n\nFor partitioned table directories, data files should only live in leaf directories. Please check the following directories for unexpected files:\n\n        file:/tmp/foo/b=0\n        file:/tmp/foo/b=1\n        file:/tmp/foo/b=1/c=1\n        file:/tmp/foo/b=0/c=0\n```\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #6610 from liancheng/part-errmsg and squashes the following commits:\n\n7d05f2c [Cheng Lian] Fixes Scala style issue\na149250 [Cheng Lian] Adds test case for the error message\n6b74dd8 [Cheng Lian] Also lists suspicious non-leaf partition directories\na935eb8 [Cheng Lian] Improves error message when conflicting partition columns are found","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/cc465fd92482737c21971d82e30d4cf247acf932"},{"sha":"9d36ec24312f0a9865b4392f89e9611a5b80916d","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-8567] [SQL] Debugging flaky HiveSparkSubmitSuite\n\nUsing similar approach used in `HiveThriftServer2Suite` to print stdout/stderr of the spawned process instead of logging them to see what happens on Jenkins. (This test suite only fails on Jenkins and doesn't spill out any log...)\n\ncc yhuai\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #6978 from liancheng/debug-hive-spark-submit-suite and squashes the following commits:\n\nb031647 [Cheng Lian] Prints process stdout/stderr instead of logging them","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/9d36ec24312f0a9865b4392f89e9611a5b80916d"},{"sha":"bba6699d0e9093bc041a9a33dd31992790f32174","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-8578] [SQL] Should ignore user defined output committer when appending data\n\nhttps://issues.apache.org/jira/browse/SPARK-8578\n\nIt is not very safe to use a custom output committer when append data to an existing dir. This changes adds the logic to check if we are appending data, and if so, we use the output committer associated with the file output format.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #6964 from yhuai/SPARK-8578 and squashes the following commits:\n\n43544c4 [Yin Huai] Do not use a custom output commiter when appendiing data.","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/bba6699d0e9093bc041a9a33dd31992790f32174"},{"sha":"31f48e5af887a9ccc9cea0218c36bf52bbf49d24","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[SPARK-8576] Add spark-ec2 options to set IAM roles and instance-initiated shutdown behavior\n\nBoth of these options are useful when spark-ec2 is being used as part of an automated pipeline and the engineers want to minimize the need to pass around AWS keys for access to things like S3 (keys are replaced by the IAM role) and to be able to launch a cluster that can terminate itself cleanly.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #6962 from nchammas/additional-ec2-options and squashes the following commits:\n\nfcf252e [Nicholas Chammas] PEP8 fixes\nefba9ee [Nicholas Chammas] add help for --instance-initiated-shutdown-behavior\n598aecf [Nicholas Chammas] option to launch instances into IAM role\n2743632 [Nicholas Chammas] add option for instance initiated shutdown","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/31f48e5af887a9ccc9cea0218c36bf52bbf49d24"},{"sha":"1173483f3f465a4c63246e83d0aaa2af521395f5","author":{"email":"07c870ffe79a3d57c30fc67475f42d361ec88a03@gmail.com","name":"BenFradet"},"message":"[SPARK-8399] [STREAMING] [WEB UI] Overlap between histograms and axis' name in Spark Streaming UI\n\nMoved where the X axis' name (#batches) is written in histograms in the spark streaming web ui so the histograms and the axis' name do not overlap.\n\nAuthor: BenFradet <benjamin.fradet@gmail.com>\n\nCloses #6845 from BenFradet/SPARK-8399 and squashes the following commits:\n\nb63695f [BenFradet] adjusted inner histograms\neb610ee [BenFradet] readjusted #batches on the x axis\ndd46f98 [BenFradet] aligned all unit labels and ticks\n0564b62 [BenFradet] readjusted #batches placement\nedd0936 [BenFradet] moved where the X axis' name (#batches) is written in histograms in the spark streaming web ui","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/1173483f3f465a4c63246e83d0aaa2af521395f5"},{"sha":"43e66192f45a23f7232116e9f664158862df5015","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-8506] Add pakages to R context created through init.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #6928 from holdenk/SPARK-8506-sparkr-does-not-provide-an-easy-way-to-depend-on-spark-packages-when-performing-init-from-inside-of-r and squashes the following commits:\n\nb60dd63 [Holden Karau] Add an example with the spark-csv package\nfa8bc92 [Holden Karau] typo: sparm -> spark\n865a90c [Holden Karau] strip spaces for comparision\nc7a4471 [Holden Karau] Add some documentation\nc1a9233 [Holden Karau] refactor for testing\nc818556 [Holden Karau] Add pakages to R","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/43e66192f45a23f7232116e9f664158862df5015"},{"sha":"b84d4b4dfe8ced1b96a0c74ef968a20a1bba8231","author":{"email":"8b0dafe15a2d887b3916c3fdd4acc054f0358d33@stratio.com","name":"Santiago M. Mola"},"message":"[SPARK-7088] [SQL] Fix analysis for 3rd party logical plan.\n\nResolveReferences analysis rule now does not throw when it cannot resolve references in a self-join.\n\nAuthor: Santiago M. Mola <smola@stratio.com>\n\nCloses #6853 from smola/SPARK-7088 and squashes the following commits:\n\naf71ac7 [Santiago M. Mola] [SPARK-7088] Fix analysis for 3rd party logical plan.","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/b84d4b4dfe8ced1b96a0c74ef968a20a1bba8231"},{"sha":"f04b5672c5a5562f8494df3b0df23235285c9e9e","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-7289] handle project -> limit -> sort efficiently\n\nmake the `TakeOrdered` strategy and operator more general, such that it can optionally handle a projection when necessary\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #6780 from cloud-fan/limit and squashes the following commits:\n\n34aa07b [Wenchen Fan] revert\n07d5456 [Wenchen Fan] clean closure\n20821ec [Wenchen Fan] fix\n3676a82 [Wenchen Fan] address comments\nb558549 [Wenchen Fan] address comments\n214842b [Wenchen Fan] fix style\n2d8be83 [Wenchen Fan] add LimitPushDown\n948f740 [Wenchen Fan] fix existing","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/f04b5672c5a5562f8494df3b0df23235285c9e9e"},{"sha":"fb32c388985ce65c1083cb435cf1f7479fecbaac","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-7633] [MLLIB] [PYSPARK] Python bindings for StreamingLogisticRegressionwithSGD\n\nAdd Python bindings to StreamingLogisticRegressionwithSGD.\n\nNo Java wrappers are needed as models are updated directly using train.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #6849 from MechCoder/spark-3258 and squashes the following commits:\n\nb4376a5 [MechCoder] minor\nd7e5fc1 [MechCoder] Refactor into StreamingLinearAlgorithm Better docs\n9c09d4e [MechCoder] [SPARK-7633] Python bindings for StreamingLogisticRegressionwithSGD","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/fb32c388985ce65c1083cb435cf1f7479fecbaac"},{"sha":"8ab50765cd793169091d983b50d87a391f6ac1f4","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-6777] [SQL] Implements backwards compatibility rules in CatalystSchemaConverter\n\nThis PR introduces `CatalystSchemaConverter` for converting Parquet schema to Spark SQL schema and vice versa.  Original conversion code in `ParquetTypesConverter` is removed. Benefits of the new version are:\n\n1. When converting Spark SQL schemas, it generates standard Parquet schemas conforming to [the most updated Parquet format spec] [1]. Converting to old style Parquet schemas is also supported via feature flag `spark.sql.parquet.followParquetFormatSpec` (which is set to `false` for now, and should be set to `true` after both read and write paths are fixed).\n\n   Note that although this version of Parquet format spec hasn't been officially release yet, Parquet MR 1.7.0 already sticks to it. So it should be safe to follow.\n\n1. It implements backwards-compatibility rules described in the most updated Parquet format spec. Thus can recognize more schema patterns generated by other/legacy systems/tools.\n1. Code organization follows convention used in [parquet-mr] [2], which is easier to follow. (Structure of `CatalystSchemaConverter` is similar to `AvroSchemaConverter`).\n\nTo fully implement backwards-compatibility rules in both read and write path, we also need to update `CatalystRowConverter` (which is responsible for converting Parquet records to `Row`s), `RowReadSupport`, and `RowWriteSupport`. These would be done in follow-up PRs.\n\nTODO\n\n- [x] More schema conversion test cases for legacy schema patterns.\n\n[1]: https://github.com/apache/parquet-format/blob/ea095226597fdbecd60c2419d96b54b2fdb4ae6c/LogicalTypes.md\n[2]: https://github.com/apache/parquet-mr/\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #6617 from liancheng/spark-6777 and squashes the following commits:\n\n2a2062d [Cheng Lian] Don't convert decimals without precision information\nb60979b [Cheng Lian] Adds a constructor which accepts a Configuration, and fixes default value of assumeBinaryIsString\n743730f [Cheng Lian] Decimal scale shouldn't be larger than precision\na104a9e [Cheng Lian] Fixes Scala style issue\n1f71d8d [Cheng Lian] Adds feature flag to allow falling back to old style Parquet schema conversion\nba84f4b [Cheng Lian] Fixes MapType schema conversion bug\n13cb8d5 [Cheng Lian] Fixes MiMa failure\n81de5b0 [Cheng Lian] Fixes UDT, workaround read path, and add tests\n28ef95b [Cheng Lian] More AnalysisExceptions\nb10c322 [Cheng Lian] Replaces require() with analysisRequire() which throws AnalysisException\ncceaf3f [Cheng Lian] Implements backwards compatibility rules in CatalystSchemaConverter","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/8ab50765cd793169091d983b50d87a391f6ac1f4"},{"sha":"dca21a83ac33813dd8165acb5f20d06e4f9b9034","author":{"email":"fc41c22b4615a3935b7417550a1cd6e9f423aa7a@gmail.com","name":"fe2s"},"message":"[SPARK-8558] [BUILD] Script /dev/run-tests fails when _JAVA_OPTIONS env var set\n\nAuthor: fe2s <aka.fe2s@gmail.com>\nAuthor: Oleksiy Dyagilev <oleksiy_dyagilev@epam.com>\n\nCloses #6956 from fe2s/fix-run-tests and squashes the following commits:\n\n31b6edc [fe2s] str is a built-in function, so using it as a variable name will lead to spurious warnings in some Python linters\n7d781a0 [fe2s] fixing for openjdk/IBM, seems like they have slightly different wording, but all have 'version' word. Surrounding with spaces for the case if version word appears in _JAVA_OPTIONS\ncd455ef [fe2s] address comment, looking for java version string rather than expecting to have on a certain line number\nad577d7 [Oleksiy Dyagilev] [SPARK-8558][BUILD] Script /dev/run-tests fails when _JAVA_OPTIONS env var set","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/dca21a83ac33813dd8165acb5f20d06e4f9b9034"},{"sha":"7daa70292e47be6a944351ef00c770ad4bcb0877","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-8567] [SQL] Increase the timeout of HiveSparkSubmitSuite\n\nhttps://issues.apache.org/jira/browse/SPARK-8567\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #6957 from yhuai/SPARK-8567 and squashes the following commits:\n\n62dff5b [Yin Huai] Increase the timeout.","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/7daa70292e47be6a944351ef00c770ad4bcb0877"},{"sha":"b71d3254e50838ccae43bdb0ff186fda25f03152","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-8075] [SQL] apply type check interface to more expressions\n\na follow up of https://github.com/apache/spark/pull/6405.\nNote: It's not a big change, a lot of changing is due to I swap some code in `aggregates.scala` to make aggregate functions right below its corresponding aggregate expressions.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #6723 from cloud-fan/type-check and squashes the following commits:\n\n2124301 [Wenchen Fan] fix tests\n5a658bb [Wenchen Fan] add tests\n287d3bb [Wenchen Fan] apply type check interface to more expressions","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/b71d3254e50838ccae43bdb0ff186fda25f03152"},{"sha":"82f80c1c7dc42c11bca2b6832c10f9610a43391b","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Two minor SQL cleanup (compiler warning & indent).\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7000 from rxin/minor-cleanup and squashes the following commits:\n\n046044c [Reynold Xin] Two minor SQL cleanup (compiler warning & indent).","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/82f80c1c7dc42c11bca2b6832c10f9610a43391b"},{"sha":"7bac2fe7717c0102b4875dbd95ae0bbf964536e3","author":{"email":"7841830342f748a54814a395669113bed0a8f742@cs.berkeley.edu","name":"Matt Massie"},"message":"[SPARK-7884] Move block deserialization from BlockStoreShuffleFetcher to ShuffleReader\n\nThis commit updates the shuffle read path to enable ShuffleReader implementations more control over the deserialization process.\n\nThe BlockStoreShuffleFetcher.fetch() method has been renamed to BlockStoreShuffleFetcher.fetchBlockStreams(). Previously, this method returned a record iterator; now, it returns an iterator of (BlockId, InputStream). Deserialization of records is now handled in the ShuffleReader.read() method.\n\nThis change creates a cleaner separation of concerns and allows implementations of ShuffleReader more flexibility in how records are retrieved.\n\nAuthor: Matt Massie <massie@cs.berkeley.edu>\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #6423 from massie/shuffle-api-cleanup and squashes the following commits:\n\n8b0632c [Matt Massie] Minor Scala style fixes\nd0a1b39 [Matt Massie] Merge pull request #1 from kayousterhout/massie_shuffle-api-cleanup\n290f1eb [Kay Ousterhout] Added test for HashShuffleReader.read()\n5186da0 [Kay Ousterhout] Revert \"Add test to ensure HashShuffleReader is freeing resources\"\nf98a1b9 [Matt Massie] Add test to ensure HashShuffleReader is freeing resources\na011bfa [Matt Massie] Use PrivateMethodTester on check that delegate stream is closed\n4ea1712 [Matt Massie] Small code cleanup for readability\n7429a98 [Matt Massie] Update tests to check that BufferReleasingStream is closing delegate InputStream\nf458489 [Matt Massie] Remove unnecessary map() on return Iterator\n4abb855 [Matt Massie] Consolidate metric code. Make it clear why InterrubtibleIterator is needed.\n5c30405 [Matt Massie] Return visibility of BlockStoreShuffleFetcher to private[hash]\n7eedd1d [Matt Massie] Small Scala import cleanup\n28f8085 [Matt Massie] Small import nit\nf93841e [Matt Massie] Update shuffle read metrics in ShuffleReader instead of BlockStoreShuffleFetcher.\n7e8e0fe [Matt Massie] Minor Scala style fixes\n01e8721 [Matt Massie] Explicitly cast iterator in branches for type clarity\n7c8f73e [Matt Massie] Close Block InputStream immediately after all records are read\n208b7a5 [Matt Massie] Small code style changes\nb70c945 [Matt Massie] Make BlockStoreShuffleFetcher visible to shuffle package\n19135f2 [Matt Massie] [SPARK-7884] Allow Spark shuffle APIs to be more customizable","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/7bac2fe7717c0102b4875dbd95ae0bbf964536e3"},{"sha":"c337844ed7f9b2cb7b217dc935183ef5e1096ca1","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-8604] [SQL] HadoopFsRelation subclasses should set their output format class\n\n`HadoopFsRelation` subclasses, especially `ParquetRelation2` should set its own output format class, so that the default output committer can be setup correctly when doing appending (where we ignore user defined output committers).\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #6998 from liancheng/spark-8604 and squashes the following commits:\n\n9be51d1 [Cheng Lian] Adds more comments\n6db1368 [Cheng Lian] HadoopFsRelation subclasses should set their output format class","distinct":false,"url":"https://api.github.com/repos/ckadner/spark/commits/c337844ed7f9b2cb7b217dc935183ef5e1096ca1"}]}