{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10269","labels_url":"https://api.github.com/repos/apache/spark/issues/10269/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10269/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10269/events","html_url":"https://github.com/apache/spark/pull/10269","id":121798101,"number":10269,"title":"[SPARK-12281][Core]Fix a race condition when reporting ExecutorState in the shutdown hook","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-12-11T21:48:43Z","updated_at":"2015-12-14T01:34:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10269","html_url":"https://github.com/apache/spark/pull/10269","diff_url":"https://github.com/apache/spark/pull/10269.diff","patch_url":"https://github.com/apache/spark/pull/10269.patch"},"body":"1. Make sure workers and masters exit so that no worker or master will still be running when triggering the shutdown hook.\r\n2. Set ExecutorState to FAILED if it's still RUNNING when executing the shutdown hook.\r\n\r\nThis should fix the potential exceptions when exiting a local cluster\r\n```\r\njava.lang.AssertionError: assertion failed: executor 4 state transfer from RUNNING to RUNNING is illegal\r\n\tat scala.Predef$.assert(Predef.scala:179)\r\n\tat org.apache.spark.deploy.master.Master$$anonfun$receive$1.applyOrElse(Master.scala:260)\r\n\tat org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116)\r\n\tat org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)\r\n\tat org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)\r\n\tat org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n\r\njava.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.\r\n\tat org.apache.spark.util.SparkShutdownHookManager.add(ShutdownHookManager.scala:246)\r\n\tat org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:191)\r\n\tat org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:180)\r\n\tat org.apache.spark.deploy.worker.ExecutorRunner.start(ExecutorRunner.scala:73)\r\n\tat org.apache.spark.deploy.worker.Worker$$anonfun$receive$1.applyOrElse(Worker.scala:474)\r\n\tat org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116)\r\n\tat org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)\r\n\tat org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)\r\n\tat org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/164319280","html_url":"https://github.com/apache/spark/pull/10269#issuecomment-164319280","issue_url":"https://api.github.com/repos/apache/spark/issues/10269","id":164319280,"user":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars.githubusercontent.com/u/850797?v=3","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"created_at":"2015-12-14T01:34:33Z","updated_at":"2015-12-14T01:34:33Z","body":"Thanks @zsxwing for finding out this wrong state transition issue. For now I think `assert` should be OK."}}