{"push_id":584386215,"size":1000,"distinct_size":1079,"ref":"refs/heads/master","head":"d641fbb39c90b1d734cc55396ca43d7e98788975","before":"c6e0c2ab1c29c184a9302d23ad75e4ccd8060242","commits":[{"sha":"8838ad7c135a585cde015dc38b5cb23314502dd9","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles\n\nSolves two JIRAs in one shot\n- Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints\n- Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #3457 from tdas/savefiles-fix and squashes the following commits:\n\nbb4729a [Tathagata Das] Same treatment for saveAsHadoopFiles\nb382ea9 [Tathagata Das] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles.","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/8838ad7c135a585cde015dc38b5cb23314502dd9"},{"sha":"1b2ab1cd1b7cab9076f3c511188a610eda935701","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-4592] Avoid duplicate worker registrations in standalone mode\n\n**Summary.** On failover, the Master may receive duplicate registrations from the same worker, causing the worker to exit. This is caused by this commit https://github.com/apache/spark/commit/4afe9a4852ebeb4cc77322a14225cd3dec165f3f, which adds logic for the worker to re-register with the master in case of failures. However, the following race condition may occur:\n\n(1) Master A fails and Worker attempts to reconnect to all masters\n(2) Master B takes over and notifies Worker\n(3) Worker responds by registering with Master B\n(4) Meanwhile, Worker's previous reconnection attempt reaches Master B, causing the same Worker to register with Master B twice\n\n**Fix.** Instead of attempting to register with all known masters, the worker should re-register with only the one that it has been communicating with. This is safe because the fact that a failover has occurred means the old master must have died. Then, when the worker is finally notified of a new master, it gives up on the old one in favor of the new one.\n\n**Caveat.** Even this fix is subject to more obscure race conditions. For instance, if Master B fails and Master A recovers immediately, then Master A may still observe duplicate worker registrations. However, this and other potential race conditions summarized in [SPARK-4592](https://issues.apache.org/jira/browse/SPARK-4592), are much, much less likely than the one described above, which is deterministically reproducible.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #3447 from andrewor14/standalone-failover and squashes the following commits:\n\n0d9716c [Andrew Or] Move re-registration logic to actor for thread-safety\n79286dc [Andrew Or] Preserve old behavior for initial retries\n83b321c [Andrew Or] Tweak wording\n1fce6a9 [Andrew Or] Active master actor could be null in the beginning\nb6f269e [Andrew Or] Avoid duplicate worker registrations","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/1b2ab1cd1b7cab9076f3c511188a610eda935701"},{"sha":"9afcbe494a3535a9bf7958429b72e989972f82d9","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-4546] Improve HistoryServer first time user experience\n\nThe documentation points the user to run the following\n```\nsbin/start-history-server.sh\n```\nThe first thing this does is throw an exception that complains a log directory is not specified. The exception message itself does not say anything about what to set. Instead we should have a default and a landing page with a better message. The new default log directory is `file:/tmp/spark-events`.\n\nThis is what it looks like as of this PR:\n\n![after](https://issues.apache.org/jira/secure/attachment/12682985/after.png)\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #3411 from andrewor14/minor-history-improvements and squashes the following commits:\n\nf33d6b3 [Andrew Or] Point user to set config if default log dir does not exist\nfc4c17a [Andrew Or] Improve HistoryServer UX","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/9afcbe494a3535a9bf7958429b72e989972f82d9"},{"sha":"9bdf5da59036c0b052df756fc4a28d64677072e7","author":{"email":"f4cfeb6da63cd4b62a6340d67bbf18015ad432f2@xiaomi.com","name":"hushan[胡珊]"},"message":"Fix SPARK-4471: blockManagerIdFromJson function throws exception while B...\n\nFix [SPARK-4471](https://issues.apache.org/jira/browse/SPARK-4471): blockManagerIdFromJson function throws exception while BlockManagerId be null in MetadataFetchFailedException\n\nAuthor: hushan[胡珊] <hushan@xiaomi.com>\n\nCloses #3340 from suyanNone/fix-blockmanagerId-jnothing-2 and squashes the following commits:\n\n159f9a3 [hushan[胡珊]] Refine test code for blockmanager is null\n4380d73 [hushan[胡珊]] remove useless blank line\n3ccf651 [hushan[胡珊]] Fix SPARK-4471: blockManagerIdFromJson function throws exception while metadata fetch failed","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/9bdf5da59036c0b052df756fc4a28d64677072e7"},{"sha":"7eba0fbe456c451122d7a2353ff0beca00f15223","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[Spark-4509] Revert EC2 tag-based cluster membership patch\n\nThis PR reverts changes related to tag-based cluster membership. As discussed in SPARK-3332, we didn't figure out a safe strategy to use tags to determine cluster membership, because tagging is not atomic. The following changes are reverted:\n\nSPARK-2333: 94053a7b766788bb62e2dbbf352ccbcc75f71fc0\nSPARK-3213: 7faf755ae4f0cf510048e432340260a6e609066d\nSPARK-3608: 78d4220fa0bf2f9ee663e34bbf3544a5313b02f0.\n\nI tested launch, login, and destroy. It is easy to check the diff by comparing it to Josh's patch for branch-1.1:\n\nhttps://github.com/apache/spark/pull/2225/files\n\nJoshRosen I sent the PR to master. It might be easier for us to keep master and branch-1.2 the same at this time. We can always re-apply the patch once we figure out a stable solution.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #3453 from mengxr/SPARK-4509 and squashes the following commits:\n\nf0b708b [Xiangrui Meng] revert 94053a7b766788bb62e2dbbf352ccbcc75f71fc0\n4298ea5 [Xiangrui Meng] revert 7faf755ae4f0cf510048e432340260a6e609066d\n35963a1 [Xiangrui Meng] Revert \"SPARK-3608 Break if the instance tag naming succeeds\"","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/7eba0fbe456c451122d7a2353ff0beca00f15223"},{"sha":"c251fd7405db57d3ab2686c38712601fd8f13ccd","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-4583] [mllib] LogLoss for GradientBoostedTrees fix + doc updates\n\nCurrently, the LogLoss used by GradientBoostedTrees has 2 issues:\n* the gradient (and therefore loss) does not match that used by Friedman (1999)\n* the error computation uses 0/1 accuracy, not log loss\n\nThis PR updates LogLoss.\nIt also adds some doc for boosting and forests.\n\nI tested it on sample data and made sure the log loss is monotonically decreasing with each boosting iteration.\n\nCC: mengxr manishamde codedeft\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #3439 from jkbradley/gbt-loss-fix and squashes the following commits:\n\ncfec17e [Joseph K. Bradley] removed forgotten temp comments\na27eb6d [Joseph K. Bradley] corrections to last log loss commit\ned5da2c [Joseph K. Bradley] updated LogLoss (boosting) for numerical stability\n5e52bff [Joseph K. Bradley] * Removed the 1/2 from SquaredError.  This also required updating the test suite since it effectively doubles the gradient and loss. * Added doc for developers within RandomForest. * Small cleanup in test suite (generating data only once)\ne57897a [Joseph K. Bradley] Fixed LogLoss for GradientBoostedTrees, and updated doc for losses, forests, and boosting","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/c251fd7405db57d3ab2686c38712601fd8f13ccd"},{"sha":"4d95526a75ad1630554683fe7a7e583da44ba6e4","author":{"email":"8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","name":"Patrick Wendell"},"message":"[HOTFIX]: Adding back without-hive dist","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/4d95526a75ad1630554683fe7a7e583da44ba6e4"},{"sha":"b5fb1410c5eed1156decb4f9fcc22436a658ce4d","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-4604][MLLIB] make MatrixFactorizationModel public\n\nUser could construct an MF model directly. I added a note about the performance.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #3459 from mengxr/SPARK-4604 and squashes the following commits:\n\nf64bcd3 [Xiangrui Meng] organize imports\ned08214 [Xiangrui Meng] check preconditions and unit tests\na624c12 [Xiangrui Meng] make MatrixFactorizationModel public","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/b5fb1410c5eed1156decb4f9fcc22436a658ce4d"},{"sha":"f5f2d27385c243959f03a9d78a149d5f405b2f50","author":{"email":"5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","name":"Aaron Davidson"},"message":"[SPARK-4516] Cap default number of Netty threads at 8\n\nIn practice, only 2-4 cores should be required to transfer roughly 10 Gb/s, and each core that we use will have an initial overhead of roughly 32 MB of off-heap memory, which comes at a premium.\n\nThus, this value should still retain maximum throughput and reduce wasted off-heap memory allocation. It can be overridden by setting the number of serverThreads and clientThreads manually in Spark's configuration.\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #3469 from aarondav/fewer-pools2 and squashes the following commits:\n\n087c59f [Aaron Davidson] [SPARK-4516] Cap default number of Netty threads at 8","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/f5f2d27385c243959f03a9d78a149d5f405b2f50"},{"sha":"346bc17a2ec8fc9e6eaff90733aa1e8b6b46883e","author":{"email":"5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","name":"Aaron Davidson"},"message":"[SPARK-4516] Avoid allocating Netty PooledByteBufAllocators unnecessarily\n\nTurns out we are allocating an allocator pool for every TransportClient (which means that the number increases with the number of nodes in the cluster), when really we should just reuse one for all clients.\n\nThis patch, as expected, greatly decreases off-heap memory allocation, and appears to make allocation only proportional to the number of cores.\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #3465 from aarondav/fewer-pools and squashes the following commits:\n\n36c49da [Aaron Davidson] [SPARK-4516] Avoid allocating unnecessarily Netty PooledByteBufAllocators","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/346bc17a2ec8fc9e6eaff90733aa1e8b6b46883e"},{"sha":"e7f4d2534bb3361ec4b7af0d42bc798a7a425226","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-4612] Reduce task latency and increase scheduling throughput by making configuration initialization lazy\n\nhttps://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L337 creates a configuration object for every task that is launched, even if there is no new dependent file/JAR to update. This is a heavy-weight creation that should be avoided if there is no new file/JAR to update. This PR makes that creation lazy. Quick local test in spark-perf scheduling throughput tests gives the following numbers in a local standalone scheduler mode.\n1 job with 10000 tasks: before 7.8395 seconds, after 2.6415 seconds = 3x increase in task scheduling throughput\n\npwendell JoshRosen\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #3463 from tdas/lazy-config and squashes the following commits:\n\nc791c1e [Tathagata Das] Reduce task latency by making configuration initialization lazy","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/e7f4d2534bb3361ec4b7af0d42bc798a7a425226"},{"sha":"288ce583b05004a8c71dcd836fab23caff5d4ba7","author":{"email":"b91298e12dcfa1e38f5d8e45fc1cad713676d4d6@gmail.com","name":"Joseph E. Gonzalez"},"message":"Removing confusing TripletFields\n\nAfter additional discussion with rxin, I think having all the possible `TripletField` options is confusing.  This pull request reduces the triplet fields to:\n\n```java\n  /**\n   * None of the triplet fields are exposed.\n   */\n  public static final TripletFields None = new TripletFields(false, false, false);\n\n  /**\n   * Expose only the edge field and not the source or destination field.\n   */\n  public static final TripletFields EdgeOnly = new TripletFields(false, false, true);\n\n  /**\n   * Expose the source and edge fields but not the destination field. (Same as Src)\n   */\n  public static final TripletFields Src = new TripletFields(true, false, true);\n\n  /**\n   * Expose the destination and edge fields but not the source field. (Same as Dst)\n   */\n  public static final TripletFields Dst = new TripletFields(false, true, true);\n\n  /**\n   * Expose all the fields (source, edge, and destination).\n   */\n  public static final TripletFields All = new TripletFields(true, true, true);\n```\n\nAuthor: Joseph E. Gonzalez <joseph.e.gonzalez@gmail.com>\n\nCloses #3472 from jegonzal/SimplifyTripletFields and squashes the following commits:\n\n91796b5 [Joseph E. Gonzalez] removing confusing triplet fields","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/288ce583b05004a8c71dcd836fab23caff5d4ba7"},{"sha":"561d31d2f13cc7b1112ba9f9aa8f08bcd032aebb","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-4614][MLLIB] Slight API changes in Matrix and Matrices\n\nBefore we have a full picture of the operators we want to add, it might be safer to hide `Matrix.transposeMultiply` in 1.2.0. Another update we want to change is `Matrix.randn` and `Matrix.rand`, both of which should take a `Random` implementation. Otherwise, it is very likely to produce inconsistent RDDs. I also added some unit tests for matrix factory methods. All APIs are new in 1.2, so there is no incompatible changes.\n\nbrkyvz\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #3468 from mengxr/SPARK-4614 and squashes the following commits:\n\n3b0e4e2 [Xiangrui Meng] add mima excludes\n6bfd8a4 [Xiangrui Meng] hide transposeMultiply; add rng to rand and randn; add unit tests","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/561d31d2f13cc7b1112ba9f9aa8f08bcd032aebb"},{"sha":"5af53ada65f62e6b5987eada288fb48e9211ef9d","author":{"email":"fd726277649d3b2e3ee5d332dca6c89739b68aa5@gmail.com","name":"CodingCat"},"message":"[SPARK-732][SPARK-3628][CORE][RESUBMIT] eliminate duplicate update on accmulator\n\nhttps://issues.apache.org/jira/browse/SPARK-3628\n\nIn current implementation, the accumulator will be updated for every successfully finished task, even the task is from a resubmitted stage, which makes the accumulator counter-intuitive\n\nIn this patch, I changed the way for the DAGScheduler to update the accumulator,\n\nDAGScheduler maintains a HashTable, mapping the stage id to the received <accumulator_id , value> pairs. Only when the stage becomes independent, (no job needs it any more), we accumulate the values of the <accumulator_id , value> pairs, when a task finished, we check if the HashTable has contained such stageId, it saves the accumulator_id, value only when the task is the first finished task of a new stage or the stage is running for the first attempt...\n\nAuthor: CodingCat <zhunansjtu@gmail.com>\n\nCloses #2524 from CodingCat/SPARK-732-1 and squashes the following commits:\n\n701a1e8 [CodingCat] roll back change on Accumulator.scala\n1433e6f [CodingCat] make MIMA happy\nb233737 [CodingCat] address Matei's comments\n02261b8 [CodingCat] rollback  some changes\n6b0aff9 [CodingCat] update document\n2b2e8cf [CodingCat] updateAccumulator\n83b75f8 [CodingCat] style fix\n84570d2 [CodingCat] re-enable  the bad accumulator guard\n1e9e14d [CodingCat] add NPE guard\n21b6840 [CodingCat] simplify the patch\n88d1f03 [CodingCat] fix rebase error\nf74266b [CodingCat] add test case for resubmitted result stage\n5cf586f [CodingCat] de-duplicate on task level\n138f9b3 [CodingCat] make MIMA happy\n67593d2 [CodingCat] make if allowing duplicate update as an option of accumulator","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/5af53ada65f62e6b5987eada288fb48e9211ef9d"},{"sha":"c86e9bc4fdd103111280a37144a518479bb9cf0e","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[Release] Automate generation of contributors list\n\nThis commit provides a script that computes the contributors list\nby linking the github commits with JIRA issues. Automatically\ntranslating github usernames remains a TODO at this point.","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/c86e9bc4fdd103111280a37144a518479bb9cf0e"},{"sha":"5d7fe178b303918faa0893cd36963158b420309f","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-4170 [CORE] Closure problems when running Scala app that \"extends App\"\n\nWarn against subclassing scala.App, and remove one instance of this in examples\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #3497 from srowen/SPARK-4170 and squashes the following commits:\n\n4a6131f [Sean Owen] Restore multiline string formatting\na8ca895 [Sean Owen] Warn against subclassing scala.App, and remove one instance of this in examples","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/5d7fe178b303918faa0893cd36963158b420309f"},{"sha":"84376d31392858f7df215ddb3f05419181152e68","author":{"email":"eadf5626de417ec3b7fbf8f1acb61a77feef61ce@gmail.com","name":"roxchkplusony"},"message":"[SPARK-4626] Kill a task only if the executorId is (still) registered with the scheduler\n\nAuthor: roxchkplusony <roxchkplusony@gmail.com>\n\nCloses #3483 from roxchkplusony/bugfix/4626 and squashes the following commits:\n\naba9184 [roxchkplusony] replace warning message per review\n5e7fdea [roxchkplusony] [SPARK-4626] Kill a task only if the executorId is (still) registered with the scheduler","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/84376d31392858f7df215ddb3f05419181152e68"},{"sha":"120a350240f58196eafcb038ca3a353636d89239","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-4613][Core] Java API for JdbcRDD\n\nThis PR introduces a set of Java APIs for using `JdbcRDD`:\n\n1. Trait (interface) `JdbcRDD.ConnectionFactory`: equivalent to the `getConnection: () => Connection` parameter in `JdbcRDD` constructor.\n2. Two overloaded versions of `Jdbc.create`: used to create `JavaRDD` that wraps a `JdbcRDD`.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/3478)\n<!-- Reviewable:end -->\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #3478 from liancheng/japi-jdbc-rdd and squashes the following commits:\n\n9a54625 [Cheng Lian] Only shutdowns a single DB rather than the whole Derby driver\nd4cedc5 [Cheng Lian] Moves Java JdbcRDD test case to a separate test suite\nffcdf2e [Cheng Lian] Java API for JdbcRDD","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/120a350240f58196eafcb038ca3a353636d89239"},{"sha":"ceb628197099e6c598cde1564ed9c1c3681ea955","author":{"email":"5cc7173406045a3242d3595588aa610a46f4e47d@asiainfo.com","name":"maji2014"},"message":"[SPARK-4619][Storage]delete redundant time suffix\n\nTime suffix exists in Utils.getUsedTimeMs(startTime), no need to append again, delete that\n\nAuthor: maji2014 <maji3@asiainfo.com>\n\nCloses #3475 from maji2014/SPARK-4619 and squashes the following commits:\n\ndf0da4e [maji2014] delete redundant time suffix","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/ceb628197099e6c598cde1564ed9c1c3681ea955"},{"sha":"5b99bf243e2956fe933ab2dccd069266e82cad8d","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-4645][SQL] Disables asynchronous execution in Hive 0.13.1 HiveThriftServer2\n\nThis PR disables HiveThriftServer2 asynchronous execution by setting `runInBackground` argument in `ExecuteStatementOperation` to `false`, and reverting `SparkExecuteStatementOperation.run` in Hive 13 shim to Hive 12 version. This change makes Simba ODBC driver v1.0.0.1000 work.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/3506)\n<!-- Reviewable:end -->\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #3506 from liancheng/disable-async-exec and squashes the following commits:\n\n593804d [Cheng Lian] Disables asynchronous execution in Hive 0.13.1 HiveThriftServer2","distinct":true,"url":"https://api.github.com/repos/Sunyifan/spark/commits/5b99bf243e2956fe933ab2dccd069266e82cad8d"}]}