{"push_id":856917445,"size":66,"distinct_size":18,"ref":"refs/heads/helpmsg","head":"6fb0f591d2775dcb5897a2e8b21b55267320395d","before":"7a1c70e189ff0d0f5b012f6ffe2739ae0ed9df4c","commits":[{"sha":"cd1df662386c599a9d0968b9fc14f27b0883d285","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11485][SQL] Make DataFrameHolder and DatasetHolder public.\n\nThese two classes should be public, since they are used in public code.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9445 from rxin/SPARK-11485.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/cd1df662386c599a9d0968b9fc14f27b0883d285"},{"sha":"e0fc9c7e59848cb78f8d598898bfca004a3710d8","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11197][SQL] add doc for run SQL on files directly\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9467 from cloud-fan/doc.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/e0fc9c7e59848cb78f8d598898bfca004a3710d8"},{"sha":"3bd6f5d2ae503468de0e218d51c331e249a862bb","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11490][SQL] variance should alias var_samp instead of var_pop.\n\nstddev is an alias for stddev_samp. variance should be consistent with stddev.\n\nAlso took the chance to remove internal Stddev and Variance, and only kept StddevSamp/StddevPop and VarianceSamp/VariancePop.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9449 from rxin/SPARK-11490.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/3bd6f5d2ae503468de0e218d51c331e249a862bb"},{"sha":"987df4bfcafeca3633453c2d2f8e14d221fcef33","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Closes #9464","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/987df4bfcafeca3633453c2d2f8e14d221fcef33"},{"sha":"de289bf279e14e47859b5fbcd70e97b9d0759f14","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-10304][SQL] Following up checking valid dir structure for partition discovery\n\nThis patch follows up #8840.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #9459 from viirya/detect_invalid_part_dir_following.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/de289bf279e14e47859b5fbcd70e97b9d0759f14"},{"sha":"abf5e4285d97b148a32cf22f5287511198175cb6","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11504][SQL] API audit for distributeBy and localSort\n\n1. Renamed localSort -> sortWithinPartitions to avoid ambiguity in \"local\"\n2. distributeBy -> repartition to match the existing repartition.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9470 from rxin/SPARK-11504.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/abf5e4285d97b148a32cf22f5287511198175cb6"},{"sha":"d19f4fda63d0800a85b3e1c8379160bbbf17b6a3","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11505][SQL] Break aggregate functions into multiple files\n\nfunctions.scala was getting pretty long. I broke it into multiple files.\n\nI also added explicit data types for some public vals, and renamed aggregate function pretty names to lower case, which is more consistent with rest of the functions.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9471 from rxin/SPARK-11505.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/d19f4fda63d0800a85b3e1c8379160bbbf17b6a3"},{"sha":"701fb5052080fa8c0a79ad7c1e65693ccf444787","author":{"email":"130a8c2510da53d4596e307e12003003207f1b66@uk.ibm.com","name":"Adam Roberts"},"message":"[SPARK-10949] Update Snappy version to 1.1.2\n\nThis is an updated version of #8995 by a-roberts. Original description follows:\n\nSnappy now supports concatenation of serialized streams, this patch contains a version number change and the \"does not support\" test is now a \"supports\" test.\n\nSnappy 1.1.2 changelog mentions:\n\n> snappy-java-1.1.2 (22 September 2015)\n> This is a backward compatible release for 1.1.x.\n> Add AIX (32-bit) support.\n> There is no upgrade for the native libraries of the other platforms.\n\n> A major change since 1.1.1 is a support for reading concatenated results of SnappyOutputStream(s)\n> snappy-java-1.1.2-RC2 (18 May 2015)\n> Fix #107: SnappyOutputStream.close() is not idempotent\n> snappy-java-1.1.2-RC1 (13 May 2015)\n> SnappyInputStream now supports reading concatenated compressed results of SnappyOutputStream\n> There has been no compressed format change since 1.0.5.x. So You can read the compressed results > interchangeablly between these versions.\n> Fixes a problem when java.io.tmpdir does not exist.\n\nCloses #8995.\n\nAuthor: Adam Roberts <aroberts@uk.ibm.com>\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9439 from JoshRosen/update-snappy.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/701fb5052080fa8c0a79ad7c1e65693ccf444787"},{"sha":"1b6a5d4af9691c3f7f3ebee3146dc13d12a0e047","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11493] remove bitset from BytesToBytesMap\n\nSince we have 4 bytes as number of records in the beginning of a page, the address can not be zero, so we do not need the bitset.\n\nFor performance concerns, the bitset could help speed up false lookup if the slot is empty (because bitset is smaller than longArray, cache hit rate will be higher). In practice, the map is filled with 35% - 70% (use 50% as average), so only half of the false lookups can benefit of it, all others will pay the cost of load the bitset (still need to access the longArray anyway).\n\nFor aggregation, we always need to access the longArray (insert a new key after false lookup), also confirmed by a benchmark.\n\n For broadcast hash join, there could be a regression, but a simple benchmark showed that it may not (most of lookup are false):\n\n```\nsqlContext.range(1<<20).write.parquet(\"small\")\ndf = sqlContext.read.parquet('small')\nfor i in range(3):\n    t = time.time()\n    df2 = sqlContext.range(1<<26).selectExpr(\"id * 1111111111 % 987654321 as id2\")\n    df2.join(df, df.id == df2.id2).count()\n    print time.time() -t\n```\n\nHaving bitset (used time in seconds):\n```\n17.5404241085\n10.2758829594\n10.5786800385\n```\nAfter removing bitset (used time in seconds):\n```\n21.8939979076\n12.4132959843\n9.97224712372\n```\n\ncc rxin nongli\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9452 from davies/remove_bitset.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/1b6a5d4af9691c3f7f3ebee3146dc13d12a0e047"},{"sha":"411ff6afb485c9d8cfc667c9346f836f2529ea9f","author":{"email":"bc4a65f227ddcd81bb538c67e810a3e11696e8bf@gmail.com","name":"Yu ISHIKAWA"},"message":"[SPARK-10028][MLLIB][PYTHON] Add Python API for PrefixSpan\n\nAuthor: Yu ISHIKAWA <yuu.ishikawa@gmail.com>\n\nCloses #9469 from yu-iskw/SPARK-10028.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/411ff6afb485c9d8cfc667c9346f836f2529ea9f"},{"sha":"b6e0a5ae6f243139f11c9cbbf18cddd3f25db208","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11510][SQL] Remove SQL aggregation tests for higher order statistics\n\nWe have some aggregate function tests in both DataFrameAggregateSuite and SQLQuerySuite. The two have almost the same coverage and we should just remove the SQL one.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9475 from rxin/SPARK-11510.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/b6e0a5ae6f243139f11c9cbbf18cddd3f25db208"},{"sha":"ce5e6a2849ae860689fa3e7d5aaa12216945ea99","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11491] Update build to use Scala 2.10.5\n\nSpark should build against Scala 2.10.5, since that includes a fix for Scaladoc that will fix doc snapshot publishing: https://issues.scala-lang.org/browse/SI-8479\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9450 from JoshRosen/upgrade-to-scala-2.10.5.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/ce5e6a2849ae860689fa3e7d5aaa12216945ea99"},{"sha":"a752ddad7fe1d0f01b51f7551ec017ff87e1eea5","author":{"email":"f82454bf38f3ce64ebf8037d45d28db4dce5c9db@huawei.com","name":"Zhenhua Wang"},"message":"[SPARK-11398] [SQL] unnecessary def dialectClassName in HiveContext, and misleading dialect conf at the start of spark-sql\n\n1. def dialectClassName in HiveContext is unnecessary.\nIn HiveContext, if conf.dialect == \"hiveql\", getSQLDialect() will return new HiveQLDialect(this);\nelse it will use super.getSQLDialect(). Then in super.getSQLDialect(), it calls dialectClassName, which is overriden in HiveContext and still return super.dialectClassName.\nSo we'll never reach the code \"classOf[HiveQLDialect].getCanonicalName\" of def dialectClassName in HiveContext.\n\n2. When we start bin/spark-sql, the default context is HiveContext, and the corresponding dialect is hiveql.\nHowever, if we type \"set spark.sql.dialect;\", the result is \"sql\", which is inconsistent with the actual dialect and is misleading. For example, we can use sql like \"create table\" which is only allowed in hiveql, but this dialect conf shows it's \"sql\".\nAlthough this problem will not cause any execution error, it's misleading to spark sql users. Therefore I think we should fix it.\nIn this pr, while procesing “set spark.sql.dialect” in SetCommand, I use \"conf.dialect\" instead of \"getConf()\" for the case of key == SQLConf.DIALECT.key, so that it will return the right dialect conf.\n\nAuthor: Zhenhua Wang <wangzhenhua@huawei.com>\n\nCloses #9349 from wzhfy/dialect.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/a752ddad7fe1d0f01b51f7551ec017ff87e1eea5"},{"sha":"d0b56339625727744e2c30fc2167bc6a457d37f7","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11307] Reduce memory consumption of OutputCommitCoordinator\n\nOutputCommitCoordinator uses a map in a place where an array would suffice, increasing its memory consumption for result stages with millions of tasks.\n\nThis patch replaces that map with an array. The only tricky part of this is reasoning about the range of possible array indexes in order to make sure that we never index out of bounds.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9274 from JoshRosen/SPARK-11307.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/d0b56339625727744e2c30fc2167bc6a457d37f7"},{"sha":"81498dd5c86ca51d2fb351c8ef52cbb28e6844f4","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11425] [SPARK-11486] Improve hybrid aggregation\n\nAfter aggregation, the dataset could be smaller than inputs, so it's better to do hash based aggregation for all inputs, then using sort based aggregation to merge them.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9383 from davies/fix_switch.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/81498dd5c86ca51d2fb351c8ef52cbb28e6844f4"},{"sha":"6f81eae24f83df51a99d4bb2629dd7daadc01519","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-11440][CORE][STREAMING][BUILD] Declare rest of @Experimental items non-experimental if they've existed since 1.2.0\n\nRemove `Experimental` annotations in core, streaming for items that existed in 1.2.0 or before. The changes are:\n\n* SparkContext\n  * binary{Files,Records} : 1.2.0\n  * submitJob : 1.0.0\n* JavaSparkContext\n  * binary{Files,Records} : 1.2.0\n* DoubleRDDFunctions, JavaDoubleRDD\n  * {mean,sum}Approx : 1.0.0\n* PairRDDFunctions, JavaPairRDD\n  * sampleByKeyExact : 1.2.0\n  * countByKeyApprox : 1.0.0\n* PairRDDFunctions\n  * countApproxDistinctByKey : 1.1.0\n* RDD\n  * countApprox, countByValueApprox, countApproxDistinct : 1.0.0\n* JavaRDDLike\n  * countApprox : 1.0.0\n* PythonHadoopUtil.Converter : 1.1.0\n* PortableDataStream : 1.2.0 (related to binaryFiles)\n* BoundedDouble : 1.0.0\n* PartialResult : 1.0.0\n* StreamingContext, JavaStreamingContext\n  * binaryRecordsStream : 1.2.0\n* HiveContext\n  * analyze : 1.2.0\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #9396 from srowen/SPARK-11440.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/6f81eae24f83df51a99d4bb2629dd7daadc01519"},{"sha":"859dff56eb0f8c63c86e7e900a12340c199e6247","author":{"email":"b1c1d8736f20db3fb6c1c66bb1455ed43909f0d8@nicolasevans.org","name":"Nick Evans"},"message":"[SPARK-11378][STREAMING] make StreamingContext.awaitTerminationOrTimeout return properly\n\nThis adds a failing test checking that `awaitTerminationOrTimeout` returns the expected value, and then fixes that failing test with the addition of a `return`.\n\ntdas zsxwing\n\nAuthor: Nick Evans <me@nicolasevans.org>\n\nCloses #9336 from manygrams/fix_await_termination_or_timeout.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/859dff56eb0f8c63c86e7e900a12340c199e6247"},{"sha":"7bdc92197cce0edc0110dc9c2158e6e3f42c72ee","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-11449][CORE] PortableDataStream should be a factory\n\n```PortableDataStream``` maintains some internal state. This makes it tricky to reuse a stream (one needs to call ```close``` on both the ```PortableDataStream``` and the ```InputStream``` it produces).\n\nThis PR removes all state from ```PortableDataStream``` and effectively turns it into an ```InputStream```/```Array[Byte]``` factory. This makes the user responsible for managing the ```InputStream``` it returns.\n\ncc srowen\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #9417 from hvanhovell/SPARK-11449.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/7bdc92197cce0edc0110dc9c2158e6e3f42c72ee"},{"sha":"a94671a027c29bacea37f56b95eccb115638abff","author":{"email":"8b2cf7c09a082d0418c970888eccba69f12dfd83@ucsd.edu","name":"a1singh"},"message":"[SPARK-11506][MLLIB] Removed redundant operation in Online LDA implementation\n\nIn file LDAOptimizer.scala:\n\nline 441: since \"idx\" was never used, replaced unrequired zipWithIndex.foreach with foreach.\n\n-      nonEmptyDocs.zipWithIndex.foreach { case ((_, termCounts: Vector), idx: Int) =>\n+      nonEmptyDocs.foreach { case (_, termCounts: Vector) =>\n\nAuthor: a1singh <a1singh@ucsd.edu>\n\nCloses #9456 from a1singh/master.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/a94671a027c29bacea37f56b95eccb115638abff"},{"sha":"77488fb8e586103ba4c0858b73e1715f1a66671f","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[MINOR][SQL] A minor log line fix\n\n`jars` in the log line is an array, so `$jars` doesn't print its content.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #9494 from liancheng/minor.log-fix.","distinct":false,"url":"https://api.github.com/repos/CharlesYeh/spark/commits/77488fb8e586103ba4c0858b73e1715f1a66671f"}]}