{"push_id":779593031,"size":13,"distinct_size":13,"ref":"refs/heads/master","head":"0349b5b4383cf813bea4e1053bcc4e0268603743","before":"00d9af5e190475affffb8b50467fcddfc40f50dc","commits":[{"sha":"c3b881a7d7e4736f7131ff002a80e25def1f63af","author":{"email":"9518293cb577d94efa5d6022808b16390a7f62c5@huawei.com","name":"Chuan Shao"},"message":"[SPARK-7336] [HISTORYSERVER] Fix bug that applications status incorrect on JobHistory UI.\n\nAuthor: ArcherShao <shaochuan@huawei.com>\n\nCloses #5886 from ArcherShao/SPARK-7336.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/c3b881a7d7e4736f7131ff002a80e25def1f63af"},{"sha":"56c4c172e99a5e14f4bc3308e7ff36d94113b63e","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-10034] [SQL] add regression test for Sort on Aggregate\n\nBefore #8371, there was a bug for `Sort` on `Aggregate` that we can't use aggregate expressions named `_aggOrdering` and can't use more than one ordering expressions which contains aggregate functions. The reason of this bug is that: The aggregate expression in `SortOrder` never get resolved, we alias it with `_aggOrdering` and call `toAttribute` which gives us an `UnresolvedAttribute`. So actually we are referencing aggregate expression by name, not by exprId like we thought. And if there is already an aggregate expression named `_aggOrdering` or there are more than one ordering expressions having aggregate functions, we will have conflict names and can't search by name.\n\nHowever, after #8371 got merged, the `SortOrder`s are guaranteed to be resolved and we are always referencing aggregate expression by exprId. The Bug doesn't exist anymore and this PR add regression tests for it.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8231 from cloud-fan/sort-agg.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/56c4c172e99a5e14f4bc3308e7ff36d94113b63e"},{"sha":"fc48307797912dc1d53893dce741ddda8630957b","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-10389] [SQL] support order by non-attribute grouping expression on Aggregate\n\nFor example, we can write `SELECT MAX(value) FROM src GROUP BY key + 1 ORDER BY key + 1` in PostgreSQL, and we should support this in Spark SQL.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8548 from cloud-fan/support-order-by-non-attribute.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/fc48307797912dc1d53893dce741ddda8630957b"},{"sha":"2da3a9e98e5d129d4507b5db01bba5ee9558d28e","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-10004] [SHUFFLE] Perform auth checks when clients read shuffle data.\n\nTo correctly isolate applications, when requests to read shuffle data\narrive at the shuffle service, proper authorization checks need to\nbe performed. This change makes sure that only the application that\ncreated the shuffle data can read from it.\n\nSuch checks are only enabled when \"spark.authenticate\" is enabled,\notherwise there's no secure way to make sure that the client is really\nwho it says it is.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #8218 from vanzin/SPARK-10004.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/2da3a9e98e5d129d4507b5db01bba5ee9558d28e"},{"sha":"6cd98c1878a9c5c6475ed5974643021ab27862a7","author":{"email":"ea6ccdfbfefd2f7cfa3f93f15ce5c28ccba7db78@gmail.com","name":"0x0FFF"},"message":"[SPARK-10417] [SQL] Iterating through Column results in infinite loop\n\n`pyspark.sql.column.Column` object has `__getitem__` method, which makes it iterable for Python. In fact it has `__getitem__` to address the case when the column might be a list or dict, for you to be able to access certain element of it in DF API. The ability to iterate over it is just a side effect that might cause confusion for the people getting familiar with Spark DF (as you might iterate this way on Pandas DF for instance)\n\nIssue reproduction:\n```\ndf = sqlContext.jsonRDD(sc.parallelize(['{\"name\": \"El Magnifico\"}']))\nfor i in df[\"name\"]: print i\n```\n\nAuthor: 0x0FFF <programmerag@gmail.com>\n\nCloses #8574 from 0x0FFF/SPARK-10417.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/6cd98c1878a9c5c6475ed5974643021ab27862a7"},{"sha":"03f3e91ff21707d8a1c7057a00f1b1cd8b743e3f","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-10422] [SQL] String column in InMemoryColumnarCache needs to override clone method\n\nhttps://issues.apache.org/jira/browse/SPARK-10422\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #8578 from yhuai/SPARK-10422.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/03f3e91ff21707d8a1c7057a00f1b1cd8b743e3f"},{"sha":"44948a2e9dcad5cd8d1eb749f469e49c5750b5ba","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-9723] [ML] params getordefault should throw more useful error\n\nParams.getOrDefault should throw a more meaningful exception than what you get from a bad key lookup.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8567 from holdenk/SPARK-9723-params-getordefault-should-throw-more-useful-error.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/44948a2e9dcad5cd8d1eb749f469e49c5750b5ba"},{"sha":"4bd85d06e0334c49be18c4612b04d013b37f189c","author":{"email":"17d07ce14ed8a23e11d276b3691423f0ca203e51@capitalone.com","name":"Ilya Ganelin"},"message":"[SPARK-5945] Spark should not retry a stage infinitely on a FetchFailedException\n\nThe ```Stage``` class now tracks whether there were a sufficient number of consecutive failures of that stage to trigger an abort.\n\nTo avoid an infinite loop of stage retries, we abort the job completely after 4 consecutive stage failures for one stage. We still allow more than 4 consecutive stage failures if there is an intervening successful attempt for the stage, so that in very long-lived applications, where a stage may get reused many times, we don't abort the job after failures that have been recovered from successfully.\n\nI've added test cases to exercise the most obvious scenarios.\n\nAuthor: Ilya Ganelin <ilya.ganelin@capitalone.com>\n\nCloses #5636 from ilganeli/SPARK-5945.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/4bd85d06e0334c49be18c4612b04d013b37f189c"},{"sha":"0985d2c30e031f80892987f7c3581d15dd210303","author":{"email":"04ade2e3a4ce65c5ee0cbfa08dacc69e7d690176@apache.org","name":"navis.ryu"},"message":"[SPARK-8707] RDD#toDebugString fails if any cached RDD has invalid partitions\n\nAdded numPartitions(evaluate: Boolean) to RDD. With \"evaluate=true\" the method is same with \"partitions.length\". With \"evaluate=false\", it checks checked-out or already evaluated partitions in the RDD to get number of partition. If it's not those cases, returns -1. RDDInfo.partitionNum calls numPartition only when it's accessed.\n\nAuthor: navis.ryu <navis@apache.org>\n\nCloses #7127 from navis/SPARK-8707.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/0985d2c30e031f80892987f7c3581d15dd210303"},{"sha":"f6c447f87592286a6f58aee5e0b2dc8dcb470d0c","author":{"email":"9a735b0e95618eb5a65033553d1ac4760bdb40fd@gmail.com","name":"Evan Racah"},"message":"Removed code duplication in ShuffleBlockFetcherIterator\n\nAdded fetchUpToMaxBytes() to prevent having to update both code blocks when a change is made.\n\nAuthor: Evan Racah <ejracah@gmail.com>\n\nCloses #8514 from eracah/master.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/f6c447f87592286a6f58aee5e0b2dc8dcb470d0c"},{"sha":"3ddb9b32335154e47890a0c761e0dfea3ccaac7b","author":{"email":"b54fbb0e17aa598fdc04bb3d79ba76b5f0737666@cloudera.com","name":"Imran Rashid"},"message":"[SPARK-10247] [CORE] improve readability of a test case in DAGSchedulerSuite\n\nThis is pretty minor, just trying to improve the readability of `DAGSchedulerSuite`, I figure every bit helps.  Before whenever I read this test, I never knew what \"should work\" and \"should be ignored\" really meant -- this adds some asserts & updates comments to make it more clear.  Also some reformatting per a suggestion from markhamstra on https://github.com/apache/spark/pull/7699\n\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #8434 from squito/SPARK-10247.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/3ddb9b32335154e47890a0c761e0dfea3ccaac7b"},{"sha":"62b4690d6b3016f41292b640ac28644ef31e299d","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-10379] preserve first page in UnsafeShuffleExternalSorter\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8543 from davies/preserve_page.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/62b4690d6b3016f41292b640ac28644ef31e299d"},{"sha":"0349b5b4383cf813bea4e1053bcc4e0268603743","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10411] [SQL] Move visualization above explain output and hide explain by default\n\nNew screenshots after this fix:\n\n<img width=\"627\" alt=\"s1\" src=\"https://cloud.githubusercontent.com/assets/1000778/9625782/4b2dba36-518b-11e5-9104-c713ff026e3d.png\">\n\nDefault:\n<img width=\"462\" alt=\"s2\" src=\"https://cloud.githubusercontent.com/assets/1000778/9625817/92366e50-518b-11e5-9981-cdfb774d66b8.png\">\n\nAfter clicking `+details`:\n<img width=\"377\" alt=\"s3\" src=\"https://cloud.githubusercontent.com/assets/1000778/9625784/4ba24342-518b-11e5-8522-846a16a95d44.png\">\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8570 from zsxwing/SPARK-10411.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/0349b5b4383cf813bea4e1053bcc4e0268603743"}]}