{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/2196","labels_url":"https://api.github.com/repos/rancher/rancher/issues/2196/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/2196/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/2196/events","html_url":"https://github.com/rancher/rancher/issues/2196","id":109188397,"number":2196,"title":"Agent keeps failing and host state stucks at \"Reconnecting\"","user":{"login":"cusspvz","id":3604053,"avatar_url":"https://avatars.githubusercontent.com/u/3604053?v=3","gravatar_id":"","url":"https://api.github.com/users/cusspvz","html_url":"https://github.com/cusspvz","followers_url":"https://api.github.com/users/cusspvz/followers","following_url":"https://api.github.com/users/cusspvz/following{/other_user}","gists_url":"https://api.github.com/users/cusspvz/gists{/gist_id}","starred_url":"https://api.github.com/users/cusspvz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cusspvz/subscriptions","organizations_url":"https://api.github.com/users/cusspvz/orgs","repos_url":"https://api.github.com/users/cusspvz/repos","events_url":"https://api.github.com/users/cusspvz/events{/privacy}","received_events_url":"https://api.github.com/users/cusspvz/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancher/rancher/labels/area/agent","name":"area/agent","color":"FBCA04"},{"url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"ibuildthecloud","id":1754002,"avatar_url":"https://avatars.githubusercontent.com/u/1754002?v=3","gravatar_id":"","url":"https://api.github.com/users/ibuildthecloud","html_url":"https://github.com/ibuildthecloud","followers_url":"https://api.github.com/users/ibuildthecloud/followers","following_url":"https://api.github.com/users/ibuildthecloud/following{/other_user}","gists_url":"https://api.github.com/users/ibuildthecloud/gists{/gist_id}","starred_url":"https://api.github.com/users/ibuildthecloud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ibuildthecloud/subscriptions","organizations_url":"https://api.github.com/users/ibuildthecloud/orgs","repos_url":"https://api.github.com/users/ibuildthecloud/repos","events_url":"https://api.github.com/users/ibuildthecloud/events{/privacy}","received_events_url":"https://api.github.com/users/ibuildthecloud/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/45","html_url":"https://github.com/rancher/rancher/milestones/Milestone%2010/28/2015","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/45/labels","id":1370761,"number":45,"title":"Milestone 10/28/2015","description":"0.44","creator":{"login":"deniseschannon","id":11410997,"avatar_url":"https://avatars.githubusercontent.com/u/11410997?v=3","gravatar_id":"","url":"https://api.github.com/users/deniseschannon","html_url":"https://github.com/deniseschannon","followers_url":"https://api.github.com/users/deniseschannon/followers","following_url":"https://api.github.com/users/deniseschannon/following{/other_user}","gists_url":"https://api.github.com/users/deniseschannon/gists{/gist_id}","starred_url":"https://api.github.com/users/deniseschannon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deniseschannon/subscriptions","organizations_url":"https://api.github.com/users/deniseschannon/orgs","repos_url":"https://api.github.com/users/deniseschannon/repos","events_url":"https://api.github.com/users/deniseschannon/events{/privacy}","received_events_url":"https://api.github.com/users/deniseschannon/received_events","type":"User","site_admin":false},"open_issues":43,"closed_issues":1,"state":"open","created_at":"2015-10-22T16:24:40Z","updated_at":"2015-10-27T20:50:20Z","due_on":"2015-10-28T07:00:00Z","closed_at":null},"comments":55,"created_at":"2015-09-30T22:01:29Z","updated_at":"2015-10-28T15:05:03Z","closed_at":null,"body":"Agent keeps failing and host state stucks at \"Reconnecting\".\r\nAs this occurs, rancher fails all posterior deploys.\r\n\r\nhere are the failing rancher-agent logs:\r\n```\r\nINFO: Sending host-routes applied 93-43fd6287135af077f5dffa7a2b5721a16cf12fe8b28a9e1bf522b35d19a80ea8\r\n\r\nTraceback (most recent call last):\r\n  File \"/var/lib/cattle/pyagent/cattle/utils.py\", line 280, in get_command_output\r\n    return check_output(*args, **kw)\r\n  File \"/var/lib/cattle/pyagent/cattle/utils.py\", line 337, in check_output\r\n    raise e1\r\nCalledProcessError: Command '['/var/lib/cattle/config.sh', u'host-iptables', u'host-routes']' returned non-zero exit status 28\r\n2015-09-30 21:48:53,013 ERROR agent [140334277124176] [event.py:111] 08df7e0f-9a5a-4d5b-9871-69a3b1c40d06 : Unknown error\r\nTraceback (most recent call last):\r\n  File \"/var/lib/cattle/pyagent/cattle/agent/event.py\", line 94, in _worker_main\r\n    resp = agent.execute(req)\r\n  File \"/var/lib/cattle/pyagent/cattle/agent/__init__.py\", line 15, in execute\r\n    return self._router.route(req)\r\n  File \"/var/lib/cattle/pyagent/cattle/plugins/core/event_router.py\", line 13, in route\r\n    resp = handler.execute(req)\r\n  File \"/var/lib/cattle/pyagent/cattle/agent/handler.py\", line 34, in execute\r\n    return method(req=req, **req.data.__dict__)\r\n  File \"/var/lib/cattle/pyagent/cattle/compute/__init__.py\", line 67, in instance_remove\r\n    action=lambda: self._do_instance_remove(instance, host, progress)\r\n  File \"/var/lib/cattle/pyagent/cattle/agent/handler.py\", line 72, in _do\r\n    action()\r\n  File \"/var/lib/cattle/pyagent/cattle/compute/__init__.py\", line 67, in <lambda>\r\n    action=lambda: self._do_instance_remove(instance, host, progress)\r\n  File \"/var/lib/cattle/pyagent/cattle/plugins/docker/compute.py\", line 813, in _do_instance_remove\r\n    remove_container(client, container)\r\n  File \"/var/lib/cattle/pyagent/cattle/plugins/docker/util.py\", line 71, in remove_container\r\n    raise e\r\nAPIError: 500 Server Error: Internal Server Error (\"Cannot destroy container 627e31b591fa3c7a94f8308baaa5a0f3aacab7d90ebc01c106bfcd85a611315b: Driver aufs failed to remove root filesystem 627e31b591fa3c7a94f8308baaa5a0f3aacab7d90ebc01c106bfcd85a611315b: rename /var/lib/docker/aufs/mnt/627e31b591fa3c7a94f8308baaa5a0f3aacab7d90ebc01c106bfcd85a611315b /var/lib/docker/aufs/mnt/627e31b591fa3c7a94f8308baaa5a0f3aacab7d90ebc01c106bfcd85a611315b-removing: device or resource busy\")\r\n2015-09-30 21:50:39,141 ERROR docker [140334277082960] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:50:54,612 ERROR docker [140334277082000] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:50:57,136 ERROR docker [140334277081840] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:50:59,138 ERROR docker [140334277126096] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:51:01,142 ERROR docker [140334277079920] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:51:03,139 ERROR docker [140334277079600] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:51:05,142 ERROR docker [140334277913968] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:51:07,145 ERROR docker [140334277125136] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:51:10,156 ERROR docker [140334277913968] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:51:12,153 ERROR docker [140334277126416] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:51:15,155 ERROR docker [140334277124816] [delegate.py:104] Can not call [c85d17a3-a134-4322-8823-54a185cd5eaf], container is not running\r\n2015-09-30 21:51:17,124 ERROR agent [140334277913168] [event.py:201] Max of [10] dropped [ping] requests exceeded\r\nI0930 21:51:18.745278 02094 manager.go:936] Exiting thread watching subcontainers\r\nI0930 21:51:18.745579 02094 manager.go:313] Exiting global housekeeping thread\r\nI0930 21:51:18.745658 02094 cadvisor.go:124] Exiting given signal: terminated\r\n```"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/151874427","html_url":"https://github.com/rancher/rancher/issues/2196#issuecomment-151874427","issue_url":"https://api.github.com/repos/rancher/rancher/issues/2196","id":151874427,"user":{"login":"cusspvz","id":3604053,"avatar_url":"https://avatars.githubusercontent.com/u/3604053?v=3","gravatar_id":"","url":"https://api.github.com/users/cusspvz","html_url":"https://github.com/cusspvz","followers_url":"https://api.github.com/users/cusspvz/followers","following_url":"https://api.github.com/users/cusspvz/following{/other_user}","gists_url":"https://api.github.com/users/cusspvz/gists{/gist_id}","starred_url":"https://api.github.com/users/cusspvz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cusspvz/subscriptions","organizations_url":"https://api.github.com/users/cusspvz/orgs","repos_url":"https://api.github.com/users/cusspvz/repos","events_url":"https://api.github.com/users/cusspvz/events{/privacy}","received_events_url":"https://api.github.com/users/cusspvz/received_events","type":"User","site_admin":false},"created_at":"2015-10-28T15:05:03Z","updated_at":"2015-10-28T15:05:03Z","body":"So you don't have access to the crash dump, right?\r\n\r\n```bash\r\nsudo tee /etc/apt/sources.list.d/ddebs.list << EOF\r\ndeb http://ddebs.ubuntu.com/ $(lsb_release -cs)          main restricted universe multiverse\r\ndeb http://ddebs.ubuntu.com/ $(lsb_release -cs)-security main restricted universe multiverse\r\ndeb http://ddebs.ubuntu.com/ $(lsb_release -cs)-updates  main restricted universe multiverse\r\ndeb http://ddebs.ubuntu.com/ $(lsb_release -cs)-proposed main restricted universe multiverse\r\nEOF\r\n\r\nsudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01\r\nsudo apt-get update\r\nsudo apt-get install linux-crashdump linux-image-$(uname -r)-dbgsym\r\n\r\nsudo service kdump start \r\n\r\n\r\n# After a kernel crash, inspect using\r\napport-retrace --stdout --rebuild-package-info /var/crash/linux-image*.crash\r\n\r\n# To simulate a crash use\r\necho 1 > /proc/sys/kernel/hung_task_panic          # panic when hung task is detected\r\necho 1 > /proc/sys/kernel/panic_on_io_nmi          # panic on NMIs from I/O\r\necho 1 > /proc/sys/kernel/panic_on_oops            # panic on oops or kernel bug detection\r\necho 1 > /proc/sys/kernel/panic_on_unrecovered_nmi # panic on NMIs from memory or unknown\r\necho 1 > /proc/sys/kernel/softlockup_panic         # panic when soft lockups are detected\r\necho 1 > /proc/sys/vm/panic_on_oom                 # panic when out-of-memory happens\r\n```\r\n\r\n[source](https://wiki.ubuntu.com/Kernel/CrashdumpRecipe)"}}