{"push_id":844281972,"size":23,"distinct_size":23,"ref":"refs/heads/branch-1.5-zalando","head":"2af8e9801085a2190a3d71f62c5fa798590f5ca7","before":"e3dc0079f54e3421fa3e83f16b6248f067ff254b","commits":[{"sha":"59747d085ba072ede3b40c9680d2801dd868a844","author":{"email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com","name":"Dilip Biswal"},"message":"[SPARK-10534] [SQL] ORDER BY clause allows only columns that are present in the select projection list\n\nFind out the missing attributes by recursively looking\nat the sort order expression and rest of the code\ntakes care of projecting them out.\n\nAdded description from cloud-fan\n\nI wanna explain a bit more about this bug.\n\nWhen we resolve sort ordering, we will use a special method, which only resolves UnresolvedAttributes and UnresolvedExtractValue. However, for something like Floor('a), even the 'a is resolved, the floor expression may still being unresolved as data type mismatch(for example, 'a is string type and Floor need double type), thus can't pass this filter, and we can't push down this missing attribute 'a\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #9123 from dilipbiswal/SPARK-10534.\n\n(cherry picked from commit 49ea0e9d7ce805d312d94a5b2936eec2053bc052)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/59747d085ba072ede3b40c9680d2801dd868a844"},{"sha":"fe5c3ae798b6459482e2f20dd3eee45d5152e7ca","author":{"email":"5d7c20a9ebcf8b2e3fa111427e329c9b3fc41e59@gmail.com","name":"Shagun Sodhani"},"message":"[SPARK-11233][SQL] register cosh in function registry\n\nAuthor: Shagun Sodhani <sshagunsodhani@gmail.com>\n\nCloses #9199 from shagunsodhani/proposed-fix-#11233.\n\n(cherry picked from commit 19ad18638e27cc7b403ea98c4f9f40a940932e30)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/fe5c3ae798b6459482e2f20dd3eee45d5152e7ca"},{"sha":"f9ad0e5435343386871ec6bd5f3849cf71b56eac","author":{"email":"bd6b9f4ef3967ee814798f1e8f9f50bdb75c8f82@outlook.com","name":"Forest Fang"},"message":"[SPARK-11244][SPARKR] sparkR.stop() should remove SQLContext\n\nSparkR should remove `.sparkRSQLsc` and `.sparkRHivesc` when `sparkR.stop()` is called. Otherwise even when SparkContext is reinitialized, `sparkRSQL.init` returns the stale copy of the object and complains:\n\n```r\nsc <- sparkR.init(\"local\")\nsqlContext <- sparkRSQL.init(sc)\nsparkR.stop()\nsc <- sparkR.init(\"local\")\nsqlContext <- sparkRSQL.init(sc)\nsqlContext\n```\nproducing\n```r\nError in callJMethod(x, \"getClass\") :\n  Invalid jobj 1. If SparkR was restarted, Spark operations need to be re-executed.\n```\n\nI have added the check and removal only when SparkContext itself is initialized. I have also added corresponding test for this fix. Let me know if you want me to move the test to SQL test suite instead.\n\np.s. I tried lint-r but ended up a lots of errors on existing code.\n\nAuthor: Forest Fang <forest.fang@outlook.com>\n\nCloses #9205 from saurfang/sparkR.stop.\n\n(cherry picked from commit 94e2064fa1b04c05c805d9175c7c78bf583db5c6)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/f9ad0e5435343386871ec6bd5f3849cf71b56eac"},{"sha":"c49e0c3f6d25aa15b7cc25db0e9ae5a869184480","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10812] [YARN] Spark hadoop util support switching to yarn\n\nWhile this is likely not a huge issue for real production systems, for test systems which may setup a Spark Context and tear it down and stand up a Spark Context with a different master (e.g. some local mode & some yarn mode) tests this cane be an issue. Discovered during work on spark-testing-base on Spark 1.4.1, but seems like the logic that triggers it is present in master (see SparkHadoopUtil object). A valid work around for users encountering this issue is to fork a different JVM, however this can be heavy weight.\n\n```\n[info] SampleMiniClusterTest:\n[info] Exception encountered when attempting to run a suite with class name: com.holdenkarau.spark.testing.SampleMiniClusterTest *** ABORTED ***\n[info] java.lang.ClassCastException: org.apache.spark.deploy.SparkHadoopUtil cannot be cast to org.apache.spark.deploy.yarn.YarnSparkHadoopUtil\n[info] at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:163)\n[info] at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:257)\n[info] at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:561)\n[info] at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:115)\n[info] at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)\n[info] at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)\n[info] at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)\n[info] at com.holdenkarau.spark.testing.SharedMiniCluster$class.setup(SharedMiniCluster.scala:186)\n[info] at com.holdenkarau.spark.testing.SampleMiniClusterTest.setup(SampleMiniClusterTest.scala:26)\n[info] at com.holdenkarau.spark.testing.SharedMiniCluster$class.beforeAll(SharedMiniCluster.scala:103)\n```\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8911 from holdenk/SPARK-10812-spark-hadoop-util-support-switching-to-yarn.\n\n(cherry picked from commit d8d50ed388d2e695b69d2b93a620045ef2f0bc18)","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/c49e0c3f6d25aa15b7cc25db0e9ae5a869184480"},{"sha":"e405c2a1f6c75b50324de1bd18363b031d34f3d0","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-10812] [YARN] Fix shutdown of token renewer.\n\nA recent change to fix the referenced bug caused this exception in\nthe `SparkContext.stop()` path:\n\norg.apache.spark.SparkException: YarnSparkHadoopUtil is not available in non-YARN mode!\n        at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:167)\n        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:182)\n        at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:440)\n        at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1579)\n        at org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1730)\n        at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185)\n        at org.apache.spark.SparkContext.stop(SparkContext.scala:1729)\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #8996 from vanzin/SPARK-10812.\n\n(cherry picked from commit 4b74755122d51edb1257d4f3785fb24508681068)","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/e405c2a1f6c75b50324de1bd18363b031d34f3d0"},{"sha":"a76cf51ed91d99c88f301ec85f3cda1288bcf346","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-11251] Fix page size calculation in local mode\n\n```\n// My machine only has 8 cores\n$ bin/spark-shell --master local[32]\nscala> val df = sc.parallelize(Seq((1, 1), (2, 2))).toDF(\"a\", \"b\")\nscala> df.as(\"x\").join(df.as(\"y\"), $\"x.a\" === $\"y.a\").count()\n\nCaused by: java.io.IOException: Unable to acquire 2097152 bytes of memory\n\tat org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPage(UnsafeExternalSorter.java:351)\n```\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #9209 from andrewor14/fix-local-page-size.\n\n(cherry picked from commit 34e71c6d89c1f2b6236dbf0d75cd12da08003c84)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/a76cf51ed91d99c88f301ec85f3cda1288bcf346"},{"sha":"ad6ade12412361b5998898e5ce6b1e007fea02eb","author":{"email":"8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","name":"Patrick Wendell"},"message":"Preparing Spark release v1.5.2-rc1","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/ad6ade12412361b5998898e5ce6b1e007fea02eb"},{"sha":"be3e343453c032e0ae01bcaa5d359de9ef02e950","author":{"email":"8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","name":"Patrick Wendell"},"message":"Preparing development version 1.5.3-SNAPSHOT","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/be3e343453c032e0ae01bcaa5d359de9ef02e950"},{"sha":"03d3ad445e829f84c7748261657e34c92ff06935","author":{"email":"5ae0824cb0ae7341a4234065ce6a18e0cc7015be@sjsu.edu","name":"Rohan Bhanderi"},"message":"Fix typo \"Received\" to \"Receiver\" in streaming-kafka-integration.md\n\nRemoved typo on line 8 in markdown : \"Received\" -> \"Receiver\"\n\nAuthor: Rohan Bhanderi <rohan.bhanderi@sjsu.edu>\n\nCloses #9242 from RohanBhanderi/patch-1.\n\n(cherry picked from commit 16dc9f344c08deee104090106cb0a537a90e33fc)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/03d3ad445e829f84c7748261657e34c92ff06935"},{"sha":"9695f452e86a88bef3bcbd1f3c0b00ad9e9ac6e1","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-10971][SPARKR] RRunner should allow setting path to Rscript.\n\nAdd a new spark conf option \"spark.sparkr.r.driver.command\" to specify the executable for an R script in client modes.\n\nThe existing spark conf option \"spark.sparkr.r.command\" is used to specify the executable for an R script in cluster modes for both driver and workers. See also [launch R worker script](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/RRDD.scala#L395).\n\nBTW, [envrionment variable \"SPARKR_DRIVER_R\"](https://github.com/apache/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L275) is used to locate R shell on the local host.\n\nFor your information, PYSPARK has two environment variables serving simliar purpose:\nPYSPARK_PYTHON\t      Python binary executable to use for PySpark in both driver and workers (default is `python`).\nPYSPARK_DRIVER_PYTHON\tPython binary executable to use for PySpark in driver only (default is PYSPARK_PYTHON).\npySpark use the code [here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala#L41) to determine the python executable for a python script.\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #9179 from sun-rui/SPARK-10971.\n\n(cherry picked from commit 2462dbcce89d657bca17ae311c99c2a4bee4a5fa)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/9695f452e86a88bef3bcbd1f3c0b00ad9e9ac6e1"},{"sha":"56f0bb6ed906059833ef2225aa2dbed581171053","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-11294][SPARKR] Improve R doc for read.df, write.df, saveAsTable\n\nAdd examples for read.df, write.df; fix grouping for read.df, loadDF; fix formatting and text truncation for write.df, saveAsTable.\n\nSeveral text issues:\n![image](https://cloud.githubusercontent.com/assets/8969467/10708590/1303a44e-79c3-11e5-854f-3a2e16854cd7.png)\n- text collapsed into a single paragraph\n- text truncated at 2 places, eg. \"overwrite: Existing data is expected to be overwritten by the contents of error:\"\n\nshivaram\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #9261 from felixcheung/rdocreadwritedf.\n\n(cherry picked from commit 5e458125018029cef5cde3390f4a55dd4e164fde)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/56f0bb6ed906059833ef2225aa2dbed581171053"},{"sha":"1cd2d9ced55ab52899a4e57b5c3da6b444ec9ae4","author":{"email":"d466ed623afd313623c432059fa6c53c75d619ed@familysearch.org","name":"Jeffrey Naisbitt"},"message":"[SPARK-11264] bin/spark-class can't find assembly jars with certain GREP_OPTIONS set\n\nTemporarily remove GREP_OPTIONS if set in bin/spark-class.\n\nSome GREP_OPTIONS will modify the output of the grep commands that are looking for the assembly jars.\nFor example, if the -n option is specified, the grep output will look like:\n5:spark-assembly-1.5.1-hadoop2.4.0.jar\n\nThis will not match the regular expressions, and so the jar files will not be found.  We could improve the regular expression to handle this case and trim off extra characters, but it is difficult to know which options may or may not be set.  Unsetting GREP_OPTIONS within the script handles all the cases and gives the desired output.\n\nAuthor: Jeffrey Naisbitt <jnaisbitt@familysearch.org>\n\nCloses #9231 from naisbitt/unset-GREP_OPTIONS.\n\n(cherry picked from commit 28132ceb10d0c127495ce8cb36135e1cb54164d7)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/1cd2d9ced55ab52899a4e57b5c3da6b444ec9ae4"},{"sha":"5200a6e2919bf9c0acc129b4db7c15546d157351","author":{"email":"5cafd55dd0ca8119e8e1d62a6779a4b9bbdd477d@deepsense.io","name":"Jacek Laskowski"},"message":"Fix typos\n\nTwo typos squashed.\n\nBTW Let me know how to proceed with other typos if I ran across any. I don't feel well to leave them aside as much as sending pull requests with such tiny changes. Guide me.\n\nAuthor: Jacek Laskowski <jacek.laskowski@deepsense.io>\n\nCloses #9250 from jaceklaskowski/typos-hunting.\n\n(cherry picked from commit 146da0d8100490a6e49a6c076ec253cdaf9f8905)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/5200a6e2919bf9c0acc129b4db7c15546d157351"},{"sha":"36fddb07acc0e077931cf9e063d4d461941eac65","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11299][DOC] Fix link to Scala DataFrame Functions reference\n\nThe SQL programming guide's link to the DataFrame functions reference points to the wrong location; this patch fixes that.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9269 from JoshRosen/SPARK-11299.\n\n(cherry picked from commit b67dc6a4342577e73b0600b51052c286c4569960)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/36fddb07acc0e077931cf9e063d4d461941eac65"},{"sha":"74921c219ba1c8917a6b69f36a44c488d10804e4","author":{"email":"0510f24c259a4f603c85979def2fe3bfbef1c219@us.ibm.com","name":"Bryan Cutler"},"message":"[SPARK-11287] Fixed class name to properly start TestExecutor from deploy.client.TestClient\n\nExecuting deploy.client.TestClient fails due to bad class name for TestExecutor in ApplicationDescription.\n\nAuthor: Bryan Cutler <bjcutler@us.ibm.com>\n\nCloses #9255 from BryanCutler/fix-TestClient-classname-SPARK-11287.\n\n(cherry picked from commit 80279ac1875d488f7000f352a958a35536bd4c2e)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/74921c219ba1c8917a6b69f36a44c488d10804e4"},{"sha":"a355d0d0d1dad72ccdffcb41c29b50f4aa051a48","author":{"email":"58f74537bdbc2dd7be823680c4a7e30cfa52e668@us.ibm.com","name":"Kevin Yu"},"message":"[SPARK-5966][WIP] Spark-submit deploy-mode cluster is not compatible with master local>\n\n… master local>\n\nAuthor: Kevin Yu <qyu@us.ibm.com>\n\nCloses #9220 from kevinyu98/working_on_spark-5966.\n\n(cherry picked from commit 616be29c7f2ebc184bd5ec97210da36a2174d80c)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/a355d0d0d1dad72ccdffcb41c29b50f4aa051a48"},{"sha":"8a6e63c78b724a25acd8c955884434bb5860277e","author":{"email":"68c20f94b41a516c565b3c829a07ad43ccb8f1f1@gmail.com","name":"Alexander Slesarenko"},"message":"[SQL][DOC] Minor document fixes in interfaces.scala\n\nrxin just noticed this while reading the code.\n\nAuthor: Alexander Slesarenko <avslesarenko@gmail.com>\n\nCloses #9284 from aslesarenko/doc-typos.\n\n(cherry picked from commit 4bb2b3698ffed58cc5159db36f8b11573ad26b23)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/8a6e63c78b724a25acd8c955884434bb5860277e"},{"sha":"abb0ca7a947f4803f3d16d65d1f6c53930890dee","author":{"email":"b1c1d8736f20db3fb6c1c66bb1455ed43909f0d8@nicolasevans.org","name":"Nick Evans"},"message":"[SPARK-11270][STREAMING] Add improved equality testing for TopicAndPartition from the Kafka Streaming API\n\njerryshao tdas\n\nI know this is kind of minor, and I know you all are busy, but this brings this class in line with the `OffsetRange` class, and makes tests a little more concise.\n\nInstead of doing something like:\n```\nassert topic_and_partition_instance._topic == \"foo\"\nassert topic_and_partition_instance._partition == 0\n```\n\nYou can do something like:\n```\nassert topic_and_partition_instance == TopicAndPartition(\"foo\", 0)\n```\n\nBefore:\n```\n>>> from pyspark.streaming.kafka import TopicAndPartition\n>>> TopicAndPartition(\"foo\", 0) == TopicAndPartition(\"foo\", 0)\nFalse\n```\n\nAfter:\n```\n>>> from pyspark.streaming.kafka import TopicAndPartition\n>>> TopicAndPartition(\"foo\", 0) == TopicAndPartition(\"foo\", 0)\nTrue\n```\n\nI couldn't find any tests - am I missing something?\n\nAuthor: Nick Evans <me@nicolasevans.org>\n\nCloses #9236 from manygrams/topic_and_partition_equality.\n\n(cherry picked from commit 8f888eea1aef5a28916ec406a99fc19648681ecf)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/abb0ca7a947f4803f3d16d65d1f6c53930890dee"},{"sha":"6969e8af887a63ab2ae7aacf4b03c5ed09ba1bfc","author":{"email":"dfc272d1e78652e4de5b4e76a0545cef11a46a5c@gmail.com","name":"Teng Qiu"},"message":"Merge remote-tracking branch 'upstream/branch-1.5' into branch-1.5-zalando","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/6969e8af887a63ab2ae7aacf4b03c5ed09ba1bfc"},{"sha":"86ee81e5cc7379f6acbc1f7dad014a8fca10abab","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-11302][MLLIB] 2) Multivariate Gaussian Model with Covariance matrix returns incorrect answer in some cases\n\nFix computation of root-sigma-inverse in multivariate Gaussian; add a test and fix related Python mixture model test.\n\nSupersedes https://github.com/apache/spark/pull/9293\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #9309 from srowen/SPARK-11302.2.\n\n(cherry picked from commit 826e1e304b57abbc56b8b7ffd663d53942ab3c7c)\nSigned-off-by: Xiangrui Meng <meng@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zalando/spark/commits/86ee81e5cc7379f6acbc1f7dad014a8fca10abab"}]}