{"push_id":874037432,"size":351,"distinct_size":2,"ref":"refs/heads/master","head":"3b9d2a347f9c796b90852173d84189834e499e25","before":"b2a68386bff1fe4b2208cb9f01002f8594241930","commits":[{"sha":"f6fcb4874ce20a1daa91b7434cf9c0254a89e979","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11477] [SQL] support create Dataset from RDD\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9434 from cloud-fan/rdd2ds and squashes the following commits:\n\n0892d72 [Wenchen Fan] support create Dataset from RDD","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/f6fcb4874ce20a1daa91b7434cf9c0254a89e979"},{"sha":"680b4e7bca935dc1569f35fa319bdfb01a12f7e0","author":{"email":"5cafd55dd0ca8119e8e1d62a6779a4b9bbdd477d@deepsense.io","name":"Jacek Laskowski"},"message":"Fix typo in WebUI\n\nAuthor: Jacek Laskowski <jacek.laskowski@deepsense.io>\n\nCloses #9444 from jaceklaskowski/TImely-fix.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/680b4e7bca935dc1569f35fa319bdfb01a12f7e0"},{"sha":"53e9cee3e4e845d1f875c487215c0f22503347b1","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11466][CORE] Avoid mockito in multi-threaded FsHistoryProviderSuite test.\n\nThe test functionality should be the same, but without using mockito; logs don't\nreally say anything useful but I suspect it may be the cause of the flakiness,\nsince updating mocks when multiple threads may be using it doesn't work very\nwell. It also allows some other cleanup (= less test code in FsHistoryProvider).\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9425 from vanzin/SPARK-11466.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/53e9cee3e4e845d1f875c487215c0f22503347b1"},{"sha":"5051262d4ca6a2c529c9b1ba86d54cce60a7af17","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11489][SQL] Only include common first order statistics in GroupedData\n\nWe added a bunch of higher order statistics such as skewness and kurtosis to GroupedData. I don't think they are common enough to justify being listed, since users can always use the normal statistics aggregate functions.\n\nThat is to say, after this change, we won't support\n```scala\ndf.groupBy(\"key\").kurtosis(\"colA\", \"colB\")\n```\n\nHowever, we will still support\n```scala\ndf.groupBy(\"key\").agg(kurtosis(col(\"colA\")), kurtosis(col(\"colB\")))\n```\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9446 from rxin/SPARK-11489.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/5051262d4ca6a2c529c9b1ba86d54cce60a7af17"},{"sha":"d648a4ad546eb05deab1005e92b815b2cbea621b","author":{"email":"bb02c6aa60d4a0a091a9679b63db9e2367e07b50@me.com","name":"lewuathe"},"message":"[DOC] Missing link to R DataFrame API doc\n\nAuthor: lewuathe <lewuathe@me.com>\nAuthor: Lewuathe <lewuathe@me.com>\n\nCloses #9394 from Lewuathe/missing-link-to-R-dataframe.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/d648a4ad546eb05deab1005e92b815b2cbea621b"},{"sha":"e352de0db2789919e1e0385b79f29b508a6b2b77","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@cloudera.com","name":"Nong"},"message":"[SPARK-11329] [SQL] Cleanup from spark-11329 fix.\n\nAuthor: Nong <nong@cloudera.com>\n\nCloses #9442 from nongli/spark-11483.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/e352de0db2789919e1e0385b79f29b508a6b2b77"},{"sha":"2692bdb7dbf36d6247f595d5fd0cb9cda89e1fdd","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11455][SQL] fix case sensitivity of partition by\n\ndepend on `caseSensitive` to do column name equality check, instead of just `==`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9410 from cloud-fan/partition.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/2692bdb7dbf36d6247f595d5fd0cb9cda89e1fdd"},{"sha":"8aff36e91de0fee2f3f56c6d240bb203b5bb48ba","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-2960][DEPLOY] Support executing Spark from symlinks (reopen)\n\nThis PR is based on the work of roji to support running Spark scripts from symlinks. Thanks for the great work roji . Would you mind taking a look at this PR, thanks a lot.\n\nFor releases like HDP and others, normally it will expose the Spark executables as symlinks and put in `PATH`, but current Spark's scripts do not support finding real path from symlink recursively, this will make spark fail to execute from symlink. This PR try to solve this issue by finding the absolute path from symlink.\n\nInstead of using `readlink -f` like what this PR (https://github.com/apache/spark/pull/2386) implemented is that `-f` is not support for Mac, so here manually seeking the path through loop.\n\nI've tested with Mac and Linux (Cent OS), looks fine.\n\nThis PR did not fix the scripts under `sbin` folder, not sure if it needs to be fixed also?\n\nPlease help to review, any comment is greatly appreciated.\n\nAuthor: jerryshao <sshao@hortonworks.com>\nAuthor: Shay Rojansky <roji@roji.org>\n\nCloses #8669 from jerryshao/SPARK-2960.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/8aff36e91de0fee2f3f56c6d240bb203b5bb48ba"},{"sha":"c09e5139874fb3626e005c8240cca5308b902ef3","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[SPARK-11442] Reduce numSlices for local metrics test of SparkListenerSuite\n\nIn the thread, http://search-hadoop.com/m/q3RTtcQiFSlTxeP/test+failed+due+to+OOME&subj=test+failed+due+to+OOME, it was discussed that memory consumption for SparkListenerSuite should be brought down.\n\nThis is an attempt in that direction by reducing numSlices for local metrics test.\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #9384 from tedyu/master.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/c09e5139874fb3626e005c8240cca5308b902ef3"},{"sha":"e328b69c31821e4b27673d7ef6182ab3b7a05ca8","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9492][ML][R] LogisticRegression in R should provide model statistics\n\nLike ml ```LinearRegression```, ```LogisticRegression``` should provide a training summary including feature names and their coefficients.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9303 from yanboliang/spark-9492.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/e328b69c31821e4b27673d7ef6182ab3b7a05ca8"},{"sha":"820064e613609bbf7edd726d982da1de60bf417a","author":{"email":"3e8ee30ef6429b31c5043fc190682edd1288226f@gmail.com","name":"Pravin Gadakh"},"message":"[SPARK-11380][DOCS] Replace example code in mllib-frequent-pattern-mining.md using include_example\n\nAuthor: Pravin Gadakh <pravingadakh177@gmail.com>\nAuthor: Pravin Gadakh <prgadakh@in.ibm.com>\n\nCloses #9340 from pravingadakh/SPARK-11380.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/820064e613609bbf7edd726d982da1de60bf417a"},{"sha":"9b214cea896056e7d0a69ae9d3c282e1f027d5b9","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-11443] Reserve space lines\n\nThe trim_codeblock(lines) function in include_example.rb removes some blank lines in the code.\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #9400 from yinxusen/SPARK-11443.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/9b214cea896056e7d0a69ae9d3c282e1f027d5b9"},{"sha":"8790ee6d69e50ca84eb849742be48f2476743b5b","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-10622][CORE][YARN] Differentiate dead from \"mostly dead\" executors.\n\nIn YARN mode, when preemption is enabled, we may leave executors in a\nzombie state while we wait to retrieve the reason for which the executor\nexited. This is so that we don't account for failed tasks that were\nrunning on a preempted executor.\n\nThe issue is that while we wait for this information, the scheduler\nmight decide to schedule tasks on the executor, which will never be\nable to run them. Other side effects include the block manager still\nconsidering the executor available to cache blocks, for example.\n\nSo, when we know that an executor went down but we don't know why,\nstop everything related to the executor, except its running tasks.\nOnly when we know the reason for the exit (or give up waiting for\nit) we do update the running tasks.\n\nThis is achieved by a new `disableExecutor()` method in the\n`Schedulable` interface. For managers that do not behave like this\n(i.e. every one but YARN), the existing `executorLost()` method\nwill behave the same way it did before.\n\nOn top of that change, a few minor changes that made debugging easier,\nand fixed some other minor issues:\n- The cluster-mode AM was printing a misleading log message every\n  time an executor disconnected from the driver (because the akka\n  actor system was shared between driver and AM).\n- Avoid sending unnecessary requests for an executor's exit reason\n  when we already know it was explicitly disabled / killed. This\n  avoids both multiple requests, and unnecessary requests that would\n  just cause warning messages on the AM (in the explicit kill case).\n- Tone down a log message about the executor being lost when it\n  exited normally (e.g. preemption)\n- Wake up the AM monitor thread when requests for executor loss\n  reasons arrive too, so that we can more quickly remove executors\n  from this zombie state.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #8887 from vanzin/SPARK-10622.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/8790ee6d69e50ca84eb849742be48f2476743b5b"},{"sha":"27feafccbd6945b000ca51b14c57912acbad9031","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11235][NETWORK] Add ability to stream data using network lib.\n\nThe current interface used to fetch shuffle data is not very efficient for\nlarge buffers; it requires the receiver to buffer the entirety of the\ncontents being downloaded in memory before processing the data.\n\nTo use the network library to transfer large files (such as those that\ncan be added using SparkContext addJar / addFile), this change adds a\nmore efficient way of downloding data, by streaming the data and feeding\nit to a callback as data arrives.\n\nThis is achieved by a custom frame decoder that replaces the current netty\none; this decoder allows entering a mode where framing is skipped and data\nis instead provided directly to a callback. The existing netty classes\n(ByteToMessageDecoder and LengthFieldBasedFrameDecoder) could not be reused\nsince their semantics do not allow for the interception approach the new\ndecoder uses.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9206 from vanzin/SPARK-11235.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/27feafccbd6945b000ca51b14c57912acbad9031"},{"sha":"cd1df662386c599a9d0968b9fc14f27b0883d285","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11485][SQL] Make DataFrameHolder and DatasetHolder public.\n\nThese two classes should be public, since they are used in public code.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9445 from rxin/SPARK-11485.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/cd1df662386c599a9d0968b9fc14f27b0883d285"},{"sha":"e0fc9c7e59848cb78f8d598898bfca004a3710d8","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11197][SQL] add doc for run SQL on files directly\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9467 from cloud-fan/doc.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/e0fc9c7e59848cb78f8d598898bfca004a3710d8"},{"sha":"3bd6f5d2ae503468de0e218d51c331e249a862bb","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11490][SQL] variance should alias var_samp instead of var_pop.\n\nstddev is an alias for stddev_samp. variance should be consistent with stddev.\n\nAlso took the chance to remove internal Stddev and Variance, and only kept StddevSamp/StddevPop and VarianceSamp/VariancePop.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9449 from rxin/SPARK-11490.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/3bd6f5d2ae503468de0e218d51c331e249a862bb"},{"sha":"987df4bfcafeca3633453c2d2f8e14d221fcef33","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Closes #9464","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/987df4bfcafeca3633453c2d2f8e14d221fcef33"},{"sha":"de289bf279e14e47859b5fbcd70e97b9d0759f14","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-10304][SQL] Following up checking valid dir structure for partition discovery\n\nThis patch follows up #8840.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #9459 from viirya/detect_invalid_part_dir_following.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/de289bf279e14e47859b5fbcd70e97b9d0759f14"},{"sha":"abf5e4285d97b148a32cf22f5287511198175cb6","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11504][SQL] API audit for distributeBy and localSort\n\n1. Renamed localSort -> sortWithinPartitions to avoid ambiguity in \"local\"\n2. distributeBy -> repartition to match the existing repartition.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9470 from rxin/SPARK-11504.","distinct":false,"url":"https://api.github.com/repos/vundela/spark/commits/abf5e4285d97b148a32cf22f5287511198175cb6"}]}