{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6307","labels_url":"https://api.github.com/repos/apache/spark/issues/6307/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6307/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6307/events","html_url":"https://github.com/apache/spark/pull/6307","id":78767047,"number":6307,"title":"[SPARK-7776] [STREAMING] Added shutdown hook to StreamingContext","user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-05-21T01:04:35Z","updated_at":"2015-05-21T03:15:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6307","html_url":"https://github.com/apache/spark/pull/6307","diff_url":"https://github.com/apache/spark/pull/6307.diff","patch_url":"https://github.com/apache/spark/pull/6307.patch"},"body":"Shutdown hook to stop SparkContext was added recently. This results in ugly errors when a streaming application is terminated by ctrl-C.\r\n\r\n```\r\nException in thread \"Thread-27\" org.apache.spark.SparkException: Job cancelled because SparkContext was shut down\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:736)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:735)\r\n\tat scala.collection.mutable.HashSet.foreach(HashSet.scala:79)\r\n\tat org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:735)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1468)\r\n\tat org.apache.spark.util.EventLoop.stop(EventLoop.scala:84)\r\n\tat org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1403)\r\n\tat org.apache.spark.SparkContext.stop(SparkContext.scala:1642)\r\n\tat org.apache.spark.SparkContext$$anonfun$3.apply$mcV$sp(SparkContext.scala:559)\r\n\tat org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2266)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Utils.scala:2236)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2236)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2236)\r\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1764)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(Utils.scala:2236)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2236)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2236)\r\n\tat scala.util.Try$.apply(Try.scala:161)\r\n\tat org.apache.spark.util.SparkShutdownHookManager.runAll(Utils.scala:2236)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anon$6.run(Utils.scala:2218)\r\n\tat org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)\r\n```\r\n\r\nThis is because the Spark's shutdown hook stops the context, and the streaming jobs fail in the middle. The correct solution is to stop the streaming context before the spark context. This PR adds the shutdown hook to do so with a priority higher than the SparkContext's shutdown hooks priority.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/104112034","html_url":"https://github.com/apache/spark/pull/6307#issuecomment-104112034","issue_url":"https://api.github.com/repos/apache/spark/issues/6307","id":104112034,"user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"created_at":"2015-05-21T03:15:59Z","updated_at":"2015-05-21T03:15:59Z","body":"@tdas as per javadoc: https://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#addShutdownHook(java.lang.Thread)\r\n\r\n> Shutdown hooks should also finish their work quickly. When a program invokes exit the expectation is that the virtual machine will promptly shut down and exit. When the virtual machine is terminated due to user logoff or system shutdown the underlying operating system may only allow a fixed amount of time in which to shut down and exit. It is therefore inadvisable to attempt any user interaction or to perform a long-running computation in a shutdown hook.\r\n\r\nI think maybe we should not support stopping gracefully in the shutdown hook.\r\n"}}