{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9116","labels_url":"https://api.github.com/repos/apache/spark/issues/9116/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9116/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9116/events","html_url":"https://github.com/apache/spark/pull/9116","id":111402453,"number":9116,"title":"[SPARK-11104][Streaming]Fix a deadlock in StreamingContex.stop","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-14T13:49:57Z","updated_at":"2015-10-14T13:52:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9116","html_url":"https://github.com/apache/spark/pull/9116","diff_url":"https://github.com/apache/spark/pull/9116.diff","patch_url":"https://github.com/apache/spark/pull/9116.patch"},"body":"The following deadlock may happen if shutdownHook and StreamingContext.stop are running at the same time.\r\n```\r\nJava stack information for the threads listed above:\r\n===================================================\r\n\"Thread-2\":\r\n\tat org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:699)\r\n\t- waiting to lock <0x00000005405a1680> (a org.apache.spark.streaming.StreamingContext)\r\n\tat org.apache.spark.streaming.StreamingContext.org$apache$spark$streaming$StreamingContext$$stopOnShutdown(StreamingContext.scala:729)\r\n\tat org.apache.spark.streaming.StreamingContext$$anonfun$start$1.apply$mcV$sp(StreamingContext.scala:625)\r\n\tat org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:266)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:236)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:236)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:236)\r\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1697)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:236)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:236)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:236)\r\n\tat scala.util.Try$.apply(Try.scala:161)\r\n\tat org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:236)\r\n\t- locked <0x00000005405b6a00> (a org.apache.spark.util.SparkShutdownHookManager)\r\n\tat org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:216)\r\n\tat org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)\r\n\"main\":\r\n\tat org.apache.spark.util.SparkShutdownHookManager.remove(ShutdownHookManager.scala:248)\r\n\t- waiting to lock <0x00000005405b6a00> (a org.apache.spark.util.SparkShutdownHookManager)\r\n\tat org.apache.spark.util.ShutdownHookManager$.removeShutdownHook(ShutdownHookManager.scala:199)\r\n\tat org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:712)\r\n\t- locked <0x00000005405a1680> (a org.apache.spark.streaming.StreamingContext)\r\n\tat org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:684)\r\n\t- locked <0x00000005405a1680> (a org.apache.spark.streaming.StreamingContext)\r\n\tat org.apache.spark.streaming.SessionByKeyBenchmark$.main(SessionByKeyBenchmark.scala:108)\r\n\tat org.apache.spark.streaming.SessionByKeyBenchmark.main(SessionByKeyBenchmark.scala)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:497)\r\n\tat org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:680)\r\n\tat org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)\r\n\tat org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)\r\n\tat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)\r\n\tat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\r\n```\r\n\r\nThis PR just moved `ShutdownHookManager.removeShutdownHook` out of `synchronized` to avoid deadlock."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/148055889","html_url":"https://github.com/apache/spark/pull/9116#issuecomment-148055889","issue_url":"https://api.github.com/repos/apache/spark/issues/9116","id":148055889,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-10-14T13:52:22Z","updated_at":"2015-10-14T13:52:22Z","body":"Merged build started."}}