{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/bugs/issues/118","labels_url":"https://api.github.com/repos/coreos/bugs/issues/118/labels{/name}","comments_url":"https://api.github.com/repos/coreos/bugs/issues/118/comments","events_url":"https://api.github.com/repos/coreos/bugs/issues/118/events","html_url":"https://github.com/coreos/bugs/issues/118","id":41511757,"number":118,"title":"Docker got stuck on Coreos","user":{"login":"rkononov","id":187434,"avatar_url":"https://avatars.githubusercontent.com/u/187434?v=3","gravatar_id":"","url":"https://api.github.com/users/rkononov","html_url":"https://github.com/rkononov","followers_url":"https://api.github.com/users/rkononov/followers","following_url":"https://api.github.com/users/rkononov/following{/other_user}","gists_url":"https://api.github.com/users/rkononov/gists{/gist_id}","starred_url":"https://api.github.com/users/rkononov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkononov/subscriptions","organizations_url":"https://api.github.com/users/rkononov/orgs","repos_url":"https://api.github.com/users/rkononov/repos","events_url":"https://api.github.com/users/rkononov/events{/privacy}","received_events_url":"https://api.github.com/users/rkononov/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/bugs/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/coreos/bugs/labels/docker","name":"docker","color":"FFFFFF"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2014-08-29T15:26:51Z","updated_at":"2015-06-22T23:04:40Z","closed_at":null,"body":"Docker became completely unresponsive after 4 hours of running/stopping containers\r\n\r\nLinux ip-10-95-165-210.ec2.internal 3.15.8+ #2 SMP Thu Aug 28 08:03:21 UTC 2014 x86_64 Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz GenuineIntel GNU/Linux\r\n\r\nCallstack from logs \r\n\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel: INFO: task kworker/u30:4:31318 blocked for more than 120 seconds.\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:       Not tainted 3.15.8+ #2\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel: kworker/u30:4   D ffff8803e0433ec0     0 31318      2 0x00000000\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel: Workqueue: netns cleanup_net\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  ffff88008d8cfc48 0000000000000046 ffff88029e428000 0000000000013ec0\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  ffff88008d8cffd8 0000000000013ec0 ffff8803cb341920 ffff88029e428000\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  ffff88029e7c8000 ffffffff81aac820 ffff88029e7c8000 ffff88029e428000\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel: Call Trace:\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff814cbad9>] schedule+0x29/0x70\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff814cbe2e>] schedule_preempt_disabled+0xe/0x10\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff814cdcbc>] __mutex_lock_slowpath+0x1bc/0x1d0\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff814cdcf3>] mutex_lock+0x23/0x37\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff813deb75>] rtnl_lock+0x15/0x20\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff813cdc92>] default_device_exit_batch+0x72/0x160\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff810a8980>] ? finish_wait+0x80/0x80\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff813c82f3>] ops_exit_list.isra.1+0x53/0x60\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff813c8c60>] cleanup_net+0x110/0x230\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff8107db17>] process_one_work+0x187/0x440\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff8107f171>] worker_thread+0x121/0x3e0\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff8107f050>] ? manage_workers.isra.21+0x300/0x300\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff81085909>] kthread+0xc9/0xe0\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff81085840>] ? flush_kthread_worker+0x80/0x80\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff814d043c>] ret_from_fork+0x7c/0xb0\r\nAug 29 14:36:57 ip-10-95-165-210.ec2.internal kernel:  [<ffffffff81085840>] ? flush_kthread_worker+0x80/0x80\r\n\r\n\r\nScript to reproduce(launch 40 concurrent processes):\r\n\r\n#!/bin/bash\r\ntouch kill.me\r\nfor i in {0..10000}\r\ndo\r\ncontainer_name=$RANDOM-$i\r\necho $container_name\r\nif [ ! -f kill.me ]; then\r\n    echo \"Exiting...\"\r\n    exit 0\r\nfi\r\nsudo docker run  --name=$container_name  -m='300M' -c 51 -i -a=stdout -a=stderr ruby ruby -e 'File.open(\"big.file\",\"w\") {|f| f.write(\"0\" * 99999) }'\r\nsudo docker stop -t=5 $container_name\r\nsudo docker rm -f $container_name\r\necho item: $i\r\ndone\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/bugs/issues/comments/114296171","html_url":"https://github.com/coreos/bugs/issues/118#issuecomment-114296171","issue_url":"https://api.github.com/repos/coreos/bugs/issues/118","id":114296171,"user":{"login":"mischief","id":1487996,"avatar_url":"https://avatars.githubusercontent.com/u/1487996?v=3","gravatar_id":"","url":"https://api.github.com/users/mischief","html_url":"https://github.com/mischief","followers_url":"https://api.github.com/users/mischief/followers","following_url":"https://api.github.com/users/mischief/following{/other_user}","gists_url":"https://api.github.com/users/mischief/gists{/gist_id}","starred_url":"https://api.github.com/users/mischief/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mischief/subscriptions","organizations_url":"https://api.github.com/users/mischief/orgs","repos_url":"https://api.github.com/users/mischief/repos","events_url":"https://api.github.com/users/mischief/events{/privacy}","received_events_url":"https://api.github.com/users/mischief/received_events","type":"User","site_admin":false},"created_at":"2015-06-22T23:04:40Z","updated_at":"2015-06-22T23:04:40Z","body":"@rkononov ping, can you still reproduce this?"}}