{"push_id":855318553,"size":50,"distinct_size":50,"ref":"refs/heads/ldaTopicPre","head":"a175ab1cc4f3471715137622caff1e187c96d542","before":"cb5d823715c767ba9b415bfd13e02d2c873e5056","commits":[{"sha":"8aff36e91de0fee2f3f56c6d240bb203b5bb48ba","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-2960][DEPLOY] Support executing Spark from symlinks (reopen)\n\nThis PR is based on the work of roji to support running Spark scripts from symlinks. Thanks for the great work roji . Would you mind taking a look at this PR, thanks a lot.\n\nFor releases like HDP and others, normally it will expose the Spark executables as symlinks and put in `PATH`, but current Spark's scripts do not support finding real path from symlink recursively, this will make spark fail to execute from symlink. This PR try to solve this issue by finding the absolute path from symlink.\n\nInstead of using `readlink -f` like what this PR (https://github.com/apache/spark/pull/2386) implemented is that `-f` is not support for Mac, so here manually seeking the path through loop.\n\nI've tested with Mac and Linux (Cent OS), looks fine.\n\nThis PR did not fix the scripts under `sbin` folder, not sure if it needs to be fixed also?\n\nPlease help to review, any comment is greatly appreciated.\n\nAuthor: jerryshao <sshao@hortonworks.com>\nAuthor: Shay Rojansky <roji@roji.org>\n\nCloses #8669 from jerryshao/SPARK-2960.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/8aff36e91de0fee2f3f56c6d240bb203b5bb48ba"},{"sha":"c09e5139874fb3626e005c8240cca5308b902ef3","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[SPARK-11442] Reduce numSlices for local metrics test of SparkListenerSuite\n\nIn the thread, http://search-hadoop.com/m/q3RTtcQiFSlTxeP/test+failed+due+to+OOME&subj=test+failed+due+to+OOME, it was discussed that memory consumption for SparkListenerSuite should be brought down.\n\nThis is an attempt in that direction by reducing numSlices for local metrics test.\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #9384 from tedyu/master.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/c09e5139874fb3626e005c8240cca5308b902ef3"},{"sha":"e328b69c31821e4b27673d7ef6182ab3b7a05ca8","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9492][ML][R] LogisticRegression in R should provide model statistics\n\nLike ml ```LinearRegression```, ```LogisticRegression``` should provide a training summary including feature names and their coefficients.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9303 from yanboliang/spark-9492.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/e328b69c31821e4b27673d7ef6182ab3b7a05ca8"},{"sha":"820064e613609bbf7edd726d982da1de60bf417a","author":{"email":"3e8ee30ef6429b31c5043fc190682edd1288226f@gmail.com","name":"Pravin Gadakh"},"message":"[SPARK-11380][DOCS] Replace example code in mllib-frequent-pattern-mining.md using include_example\n\nAuthor: Pravin Gadakh <pravingadakh177@gmail.com>\nAuthor: Pravin Gadakh <prgadakh@in.ibm.com>\n\nCloses #9340 from pravingadakh/SPARK-11380.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/820064e613609bbf7edd726d982da1de60bf417a"},{"sha":"9b214cea896056e7d0a69ae9d3c282e1f027d5b9","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-11443] Reserve space lines\n\nThe trim_codeblock(lines) function in include_example.rb removes some blank lines in the code.\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #9400 from yinxusen/SPARK-11443.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/9b214cea896056e7d0a69ae9d3c282e1f027d5b9"},{"sha":"8790ee6d69e50ca84eb849742be48f2476743b5b","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-10622][CORE][YARN] Differentiate dead from \"mostly dead\" executors.\n\nIn YARN mode, when preemption is enabled, we may leave executors in a\nzombie state while we wait to retrieve the reason for which the executor\nexited. This is so that we don't account for failed tasks that were\nrunning on a preempted executor.\n\nThe issue is that while we wait for this information, the scheduler\nmight decide to schedule tasks on the executor, which will never be\nable to run them. Other side effects include the block manager still\nconsidering the executor available to cache blocks, for example.\n\nSo, when we know that an executor went down but we don't know why,\nstop everything related to the executor, except its running tasks.\nOnly when we know the reason for the exit (or give up waiting for\nit) we do update the running tasks.\n\nThis is achieved by a new `disableExecutor()` method in the\n`Schedulable` interface. For managers that do not behave like this\n(i.e. every one but YARN), the existing `executorLost()` method\nwill behave the same way it did before.\n\nOn top of that change, a few minor changes that made debugging easier,\nand fixed some other minor issues:\n- The cluster-mode AM was printing a misleading log message every\n  time an executor disconnected from the driver (because the akka\n  actor system was shared between driver and AM).\n- Avoid sending unnecessary requests for an executor's exit reason\n  when we already know it was explicitly disabled / killed. This\n  avoids both multiple requests, and unnecessary requests that would\n  just cause warning messages on the AM (in the explicit kill case).\n- Tone down a log message about the executor being lost when it\n  exited normally (e.g. preemption)\n- Wake up the AM monitor thread when requests for executor loss\n  reasons arrive too, so that we can more quickly remove executors\n  from this zombie state.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #8887 from vanzin/SPARK-10622.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/8790ee6d69e50ca84eb849742be48f2476743b5b"},{"sha":"27feafccbd6945b000ca51b14c57912acbad9031","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11235][NETWORK] Add ability to stream data using network lib.\n\nThe current interface used to fetch shuffle data is not very efficient for\nlarge buffers; it requires the receiver to buffer the entirety of the\ncontents being downloaded in memory before processing the data.\n\nTo use the network library to transfer large files (such as those that\ncan be added using SparkContext addJar / addFile), this change adds a\nmore efficient way of downloding data, by streaming the data and feeding\nit to a callback as data arrives.\n\nThis is achieved by a custom frame decoder that replaces the current netty\none; this decoder allows entering a mode where framing is skipped and data\nis instead provided directly to a callback. The existing netty classes\n(ByteToMessageDecoder and LengthFieldBasedFrameDecoder) could not be reused\nsince their semantics do not allow for the interception approach the new\ndecoder uses.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9206 from vanzin/SPARK-11235.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/27feafccbd6945b000ca51b14c57912acbad9031"},{"sha":"cd1df662386c599a9d0968b9fc14f27b0883d285","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11485][SQL] Make DataFrameHolder and DatasetHolder public.\n\nThese two classes should be public, since they are used in public code.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9445 from rxin/SPARK-11485.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/cd1df662386c599a9d0968b9fc14f27b0883d285"},{"sha":"e0fc9c7e59848cb78f8d598898bfca004a3710d8","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11197][SQL] add doc for run SQL on files directly\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9467 from cloud-fan/doc.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/e0fc9c7e59848cb78f8d598898bfca004a3710d8"},{"sha":"3bd6f5d2ae503468de0e218d51c331e249a862bb","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11490][SQL] variance should alias var_samp instead of var_pop.\n\nstddev is an alias for stddev_samp. variance should be consistent with stddev.\n\nAlso took the chance to remove internal Stddev and Variance, and only kept StddevSamp/StddevPop and VarianceSamp/VariancePop.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9449 from rxin/SPARK-11490.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/3bd6f5d2ae503468de0e218d51c331e249a862bb"},{"sha":"987df4bfcafeca3633453c2d2f8e14d221fcef33","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Closes #9464","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/987df4bfcafeca3633453c2d2f8e14d221fcef33"},{"sha":"de289bf279e14e47859b5fbcd70e97b9d0759f14","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-10304][SQL] Following up checking valid dir structure for partition discovery\n\nThis patch follows up #8840.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #9459 from viirya/detect_invalid_part_dir_following.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/de289bf279e14e47859b5fbcd70e97b9d0759f14"},{"sha":"abf5e4285d97b148a32cf22f5287511198175cb6","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11504][SQL] API audit for distributeBy and localSort\n\n1. Renamed localSort -> sortWithinPartitions to avoid ambiguity in \"local\"\n2. distributeBy -> repartition to match the existing repartition.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9470 from rxin/SPARK-11504.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/abf5e4285d97b148a32cf22f5287511198175cb6"},{"sha":"d19f4fda63d0800a85b3e1c8379160bbbf17b6a3","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11505][SQL] Break aggregate functions into multiple files\n\nfunctions.scala was getting pretty long. I broke it into multiple files.\n\nI also added explicit data types for some public vals, and renamed aggregate function pretty names to lower case, which is more consistent with rest of the functions.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9471 from rxin/SPARK-11505.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/d19f4fda63d0800a85b3e1c8379160bbbf17b6a3"},{"sha":"701fb5052080fa8c0a79ad7c1e65693ccf444787","author":{"email":"130a8c2510da53d4596e307e12003003207f1b66@uk.ibm.com","name":"Adam Roberts"},"message":"[SPARK-10949] Update Snappy version to 1.1.2\n\nThis is an updated version of #8995 by a-roberts. Original description follows:\n\nSnappy now supports concatenation of serialized streams, this patch contains a version number change and the \"does not support\" test is now a \"supports\" test.\n\nSnappy 1.1.2 changelog mentions:\n\n> snappy-java-1.1.2 (22 September 2015)\n> This is a backward compatible release for 1.1.x.\n> Add AIX (32-bit) support.\n> There is no upgrade for the native libraries of the other platforms.\n\n> A major change since 1.1.1 is a support for reading concatenated results of SnappyOutputStream(s)\n> snappy-java-1.1.2-RC2 (18 May 2015)\n> Fix #107: SnappyOutputStream.close() is not idempotent\n> snappy-java-1.1.2-RC1 (13 May 2015)\n> SnappyInputStream now supports reading concatenated compressed results of SnappyOutputStream\n> There has been no compressed format change since 1.0.5.x. So You can read the compressed results > interchangeablly between these versions.\n> Fixes a problem when java.io.tmpdir does not exist.\n\nCloses #8995.\n\nAuthor: Adam Roberts <aroberts@uk.ibm.com>\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9439 from JoshRosen/update-snappy.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/701fb5052080fa8c0a79ad7c1e65693ccf444787"},{"sha":"1b6a5d4af9691c3f7f3ebee3146dc13d12a0e047","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11493] remove bitset from BytesToBytesMap\n\nSince we have 4 bytes as number of records in the beginning of a page, the address can not be zero, so we do not need the bitset.\n\nFor performance concerns, the bitset could help speed up false lookup if the slot is empty (because bitset is smaller than longArray, cache hit rate will be higher). In practice, the map is filled with 35% - 70% (use 50% as average), so only half of the false lookups can benefit of it, all others will pay the cost of load the bitset (still need to access the longArray anyway).\n\nFor aggregation, we always need to access the longArray (insert a new key after false lookup), also confirmed by a benchmark.\n\n For broadcast hash join, there could be a regression, but a simple benchmark showed that it may not (most of lookup are false):\n\n```\nsqlContext.range(1<<20).write.parquet(\"small\")\ndf = sqlContext.read.parquet('small')\nfor i in range(3):\n    t = time.time()\n    df2 = sqlContext.range(1<<26).selectExpr(\"id * 1111111111 % 987654321 as id2\")\n    df2.join(df, df.id == df2.id2).count()\n    print time.time() -t\n```\n\nHaving bitset (used time in seconds):\n```\n17.5404241085\n10.2758829594\n10.5786800385\n```\nAfter removing bitset (used time in seconds):\n```\n21.8939979076\n12.4132959843\n9.97224712372\n```\n\ncc rxin nongli\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9452 from davies/remove_bitset.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/1b6a5d4af9691c3f7f3ebee3146dc13d12a0e047"},{"sha":"411ff6afb485c9d8cfc667c9346f836f2529ea9f","author":{"email":"bc4a65f227ddcd81bb538c67e810a3e11696e8bf@gmail.com","name":"Yu ISHIKAWA"},"message":"[SPARK-10028][MLLIB][PYTHON] Add Python API for PrefixSpan\n\nAuthor: Yu ISHIKAWA <yuu.ishikawa@gmail.com>\n\nCloses #9469 from yu-iskw/SPARK-10028.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/411ff6afb485c9d8cfc667c9346f836f2529ea9f"},{"sha":"b6e0a5ae6f243139f11c9cbbf18cddd3f25db208","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11510][SQL] Remove SQL aggregation tests for higher order statistics\n\nWe have some aggregate function tests in both DataFrameAggregateSuite and SQLQuerySuite. The two have almost the same coverage and we should just remove the SQL one.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9475 from rxin/SPARK-11510.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/b6e0a5ae6f243139f11c9cbbf18cddd3f25db208"},{"sha":"ce5e6a2849ae860689fa3e7d5aaa12216945ea99","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11491] Update build to use Scala 2.10.5\n\nSpark should build against Scala 2.10.5, since that includes a fix for Scaladoc that will fix doc snapshot publishing: https://issues.scala-lang.org/browse/SI-8479\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9450 from JoshRosen/upgrade-to-scala-2.10.5.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/ce5e6a2849ae860689fa3e7d5aaa12216945ea99"},{"sha":"a752ddad7fe1d0f01b51f7551ec017ff87e1eea5","author":{"email":"f82454bf38f3ce64ebf8037d45d28db4dce5c9db@huawei.com","name":"Zhenhua Wang"},"message":"[SPARK-11398] [SQL] unnecessary def dialectClassName in HiveContext, and misleading dialect conf at the start of spark-sql\n\n1. def dialectClassName in HiveContext is unnecessary.\nIn HiveContext, if conf.dialect == \"hiveql\", getSQLDialect() will return new HiveQLDialect(this);\nelse it will use super.getSQLDialect(). Then in super.getSQLDialect(), it calls dialectClassName, which is overriden in HiveContext and still return super.dialectClassName.\nSo we'll never reach the code \"classOf[HiveQLDialect].getCanonicalName\" of def dialectClassName in HiveContext.\n\n2. When we start bin/spark-sql, the default context is HiveContext, and the corresponding dialect is hiveql.\nHowever, if we type \"set spark.sql.dialect;\", the result is \"sql\", which is inconsistent with the actual dialect and is misleading. For example, we can use sql like \"create table\" which is only allowed in hiveql, but this dialect conf shows it's \"sql\".\nAlthough this problem will not cause any execution error, it's misleading to spark sql users. Therefore I think we should fix it.\nIn this pr, while procesing “set spark.sql.dialect” in SetCommand, I use \"conf.dialect\" instead of \"getConf()\" for the case of key == SQLConf.DIALECT.key, so that it will return the right dialect conf.\n\nAuthor: Zhenhua Wang <wangzhenhua@huawei.com>\n\nCloses #9349 from wzhfy/dialect.","distinct":true,"url":"https://api.github.com/repos/hhbyyh/spark/commits/a752ddad7fe1d0f01b51f7551ec017ff87e1eea5"}]}