{"push_id":904985592,"size":13,"distinct_size":13,"ref":"refs/heads/branch-1.6","head":"fb08f7b784bc8b5e0cd110f315f72c7d9fc65e08","before":"d7e3bfd7d33b8fba44ef80932c0d40fb68075cb4","commits":[{"sha":"fbf16da2e53acc8678bd1454b0749d1923d4eddf","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-12281][CORE] Fix a race condition when reporting ExecutorState in the shutdown hook\n\n1. Make sure workers and masters exit so that no worker or master will still be running when triggering the shutdown hook.\n2. Set ExecutorState to FAILED if it's still RUNNING when executing the shutdown hook.\n\nThis should fix the potential exceptions when exiting a local cluster\n```\njava.lang.AssertionError: assertion failed: executor 4 state transfer from RUNNING to RUNNING is illegal\n\tat scala.Predef$.assert(Predef.scala:179)\n\tat org.apache.spark.deploy.master.Master$$anonfun$receive$1.applyOrElse(Master.scala:260)\n\tat org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116)\n\tat org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)\n\tat org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)\n\tat org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n\njava.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.\n\tat org.apache.spark.util.SparkShutdownHookManager.add(ShutdownHookManager.scala:246)\n\tat org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:191)\n\tat org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:180)\n\tat org.apache.spark.deploy.worker.ExecutorRunner.start(ExecutorRunner.scala:73)\n\tat org.apache.spark.deploy.worker.Worker$$anonfun$receive$1.applyOrElse(Worker.scala:474)\n\tat org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116)\n\tat org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)\n\tat org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)\n\tat org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n```\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10269 from zsxwing/executor-state.\n\n(cherry picked from commit 2aecda284e22ec608992b6221e2f5ffbd51fcd24)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/fbf16da2e53acc8678bd1454b0749d1923d4eddf"},{"sha":"94ce5025f894f01602732b543bc14901e169cc65","author":{"email":"aa15ad6fdfaeca61c0f759cc9dc37b58d689bbc8@intel.com","name":"yucai"},"message":"[SPARK-12275][SQL] No plan for BroadcastHint in some condition\n\nWhen SparkStrategies.BasicOperators's \"case BroadcastHint(child) => apply(child)\" is hit, it only recursively invokes BasicOperators.apply with this \"child\". It makes many strategies have no change to process this plan, which probably leads to \"No plan\" issue, so we use planLater to go through all strategies.\n\nhttps://issues.apache.org/jira/browse/SPARK-12275\n\nAuthor: yucai <yucai.yu@intel.com>\n\nCloses #10265 from yucai/broadcast_hint.\n\n(cherry picked from commit ed87f6d3b48a85391628c29c43d318c26e2c6de7)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/94ce5025f894f01602732b543bc14901e169cc65"},{"sha":"c0f0f6cb0fef6e939744b60fdd4911c718f8fac5","author":{"email":"07c870ffe79a3d57c30fc67475f42d361ec88a03@gmail.com","name":"BenFradet"},"message":"[MINOR][DOC] Fix broken word2vec link\n\nFollow-up of [SPARK-12199](https://issues.apache.org/jira/browse/SPARK-12199) and #10193 where a broken link has been left as is.\n\nAuthor: BenFradet <benjamin.fradet@gmail.com>\n\nCloses #10282 from BenFradet/SPARK-12199.\n\n(cherry picked from commit e25f1fe42747be71c6b6e6357ca214f9544e3a46)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/c0f0f6cb0fef6e939744b60fdd4911c718f8fac5"},{"sha":"352a0c80f4833a97916a75388ef290067c2dbede","author":{"email":"ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu","name":"Shivaram Venkataraman"},"message":"[SPARK-12327] Disable commented code lintr temporarily\n\ncc yhuai felixcheung shaneknapp\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #10300 from shivaram/comment-lintr-disable.\n\n(cherry picked from commit fb3778de685881df66bf0222b520f94dca99e8c8)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/352a0c80f4833a97916a75388ef290067c2dbede"},{"sha":"23c8846050b307fdfe2307f7e7ca9d0f69f969a9","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[STREAMING][MINOR] Fix typo in function name of StateImpl\n\ncc\\ tdas zsxwing , please review. Thanks a lot.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #10305 from jerryshao/fix-typo-state-impl.\n\n(cherry picked from commit bc1ff9f4a41401599d3a87fb3c23a2078228a29b)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/23c8846050b307fdfe2307f7e7ca9d0f69f969a9"},{"sha":"80d261718c1157e5cd4b0ac27e36ef919ea65afa","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"Update branch-1.6 for 1.6.0 release\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #10317 from marmbrus/versions.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/80d261718c1157e5cd4b0ac27e36ef919ea65afa"},{"sha":"00a39d9c05c55b5ffcd4f49aadc91cedf227669a","author":{"email":"8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","name":"Patrick Wendell"},"message":"Preparing Spark release v1.6.0-rc3","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/00a39d9c05c55b5ffcd4f49aadc91cedf227669a"},{"sha":"08aa3b47e6a295a8297e741effa14cd0d834aea8","author":{"email":"8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","name":"Patrick Wendell"},"message":"Preparing development version 1.6.0-SNAPSHOT","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/08aa3b47e6a295a8297e741effa14cd0d834aea8"},{"sha":"9e4ac56452710ddd8efb695e69c8de49317e3f28","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[SPARK-12056][CORE] Part 2 Create a TaskAttemptContext only after calling setConf\n\nThis is continuation of SPARK-12056 where change is applied to SqlNewHadoopRDD.scala\n\nandrewor14\nFYI\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #10164 from tedyu/master.\n\n(cherry picked from commit f725b2ec1ab0d89e35b5e2d3ddeddb79fec85f6d)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/9e4ac56452710ddd8efb695e69c8de49317e3f28"},{"sha":"2c324d35a698b353c2193e2f9bd8ba08c741c548","author":{"email":"22a0dda5f6015b8ee84a30ff6553d35a91fe8732@gmail.com","name":"Timothy Chen"},"message":"[SPARK-12351][MESOS] Add documentation about submitting Spark with mesos cluster mode.\n\nAdding more documentation about submitting jobs with mesos cluster mode.\n\nAuthor: Timothy Chen <tnachen@gmail.com>\n\nCloses #10086 from tnachen/mesos_supervise_docs.\n\n(cherry picked from commit c2de99a7c3a52b0da96517c7056d2733ef45495f)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/2c324d35a698b353c2193e2f9bd8ba08c741c548"},{"sha":"8e9a600313f3047139d3cebef85acc782903123b","author":{"email":"a588f6980008f2c84ad1eb72754855f142266575@gmail.com","name":"Naveen"},"message":"[SPARK-9886][CORE] Fix to use ShutdownHookManager in\n\nExternalBlockStore.scala\n\nAuthor: Naveen <naveenminchu@gmail.com>\n\nCloses #10313 from naveenminchu/branch-fix-SPARK-9886.\n\n(cherry picked from commit 8a215d2338c6286253e20122640592f9d69896c8)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/8e9a600313f3047139d3cebef85acc782903123b"},{"sha":"93095eb29a1e59dbdbf6220bfa732b502330e6ae","author":{"email":"0510f24c259a4f603c85979def2fe3bfbef1c219@us.ibm.com","name":"Bryan Cutler"},"message":"[SPARK-12062][CORE] Change Master to asyc rebuild UI when application completes\n\nThis change builds the event history of completed apps asynchronously so the RPC thread will not be blocked and allow new workers to register/remove if the event log history is very large and takes a long time to rebuild.\n\nAuthor: Bryan Cutler <bjcutler@us.ibm.com>\n\nCloses #10284 from BryanCutler/async-MasterUI-SPARK-12062.\n\n(cherry picked from commit c5b6b398d5e368626e589feede80355fb74c2bd8)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/93095eb29a1e59dbdbf6220bfa732b502330e6ae"},{"sha":"fb08f7b784bc8b5e0cd110f315f72c7d9fc65e08","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-10477][SQL] using DSL in ColumnPruningSuite to improve readability\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8645 from cloud-fan/test.\n\n(cherry picked from commit a89e8b6122ee5a1517fbcf405b1686619db56696)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/fb08f7b784bc8b5e0cd110f315f72c7d9fc65e08"}]}