{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10959","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10959/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10959/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10959/events","html_url":"https://github.com/kubernetes/kubernetes/issues/10959","id":93913324,"number":10959,"title":"Running pods fails with AUFS mount errors","user":{"login":"krousey","id":157083,"avatar_url":"https://avatars.githubusercontent.com/u/157083?v=3","gravatar_id":"","url":"https://api.github.com/users/krousey","html_url":"https://github.com/krousey","followers_url":"https://api.github.com/users/krousey/followers","following_url":"https://api.github.com/users/krousey/following{/other_user}","gists_url":"https://api.github.com/users/krousey/gists{/gist_id}","starred_url":"https://api.github.com/users/krousey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krousey/subscriptions","organizations_url":"https://api.github.com/users/krousey/orgs","repos_url":"https://api.github.com/users/krousey/repos","events_url":"https://api.github.com/users/krousey/events{/privacy}","received_events_url":"https://api.github.com/users/krousey/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"closed","locked":false,"assignee":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/13","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.1","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/13/labels","id":1220916,"number":13,"title":"v1.1","description":"Release 1.1.\r\n\r\nP1 features:\r\n* 250+ nodes, at our published SLO\r\n* kube-proxy perf. improvements (via IPTables)\r\n* Docker 1.8.x\r\n* Horizontal pod auto-scaling\r\n* Job\r\n* Ingress\r\n* Schema validation in kubectl\r\n* Experimental API and API groups\r\n* Graceful pod termination\r\n","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":325,"state":"open","created_at":"2015-07-22T19:17:12Z","updated_at":"2015-12-08T00:42:45Z","due_on":"2015-09-21T07:00:00Z","closed_at":null},"comments":29,"created_at":"2015-07-08T23:32:42Z","updated_at":"2015-12-19T10:53:05Z","closed_at":"2015-10-23T23:13:24Z","body":"#10622 \r\n\r\nOcassionaly our e2e tests fail with something like the following\r\n\r\n```\r\nINFO: event for dns-test-a6805741-25ac-11e5-8fa6-42010af01555: {kubelet e2e-test-parallel-minion-4c2m} pulled: Successfully pulled image \"gcr.io/google_containers/jessie-dnsutils\"\r\nINFO: event for dns-test-a6805741-25ac-11e5-8fa6-42010af01555: {kubelet e2e-test-parallel-minion-4c2m} failed: Failed to create docker container with error: API error (500): error creating aufs mount to /var/lib/docker/aufs/mnt/b2a84be51ae21ce45bc2cf1ecb944be2b4345c25798af0249260874185622e64-init: invalid argument\r\n\r\nINFO: event for dns-test-a6805741-25ac-11e5-8fa6-42010af01555: {kubelet e2e-test-parallel-minion-4c2m} failed: Failed to create docker container with error: API error (500): error creating aufs mount to /var/lib/docker/aufs/mnt/6f337451bbaa63fade20e59c88f87a6851412e3b5fb9d9b5d1a6457d1bb4e9f5-init: invalid argument\r\n\r\n... many many more then ...\r\n\r\n• Failure [311.910 seconds]\r\nDNS\r\n/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:310\r\n  should provide DNS for services [It]\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:308\r\n\r\n  Expected error:\r\n      <*errors.errorString | 0xc2080e49c0>: {\r\n          s: \"gave up waiting for pod 'dns-test-a6805741-25ac-11e5-8fa6-42010af01555' to be 'running' after 5m0s\",\r\n      }\r\n      gave up waiting for pod 'dns-test-a6805741-25ac-11e5-8fa6-42010af01555' to be 'running' after 5m0s\r\n  not to have occurred\r\n```\r\n\r\n23.251.154.44-22-kubelet.log reports the following\r\n\r\n```\r\nE0708 20:05:33.233502    3077 manager.go:1579] Error running pod \"nginx_e2e-tests-kubectl-516xc\" container \"nginx\": API error (500): error creating aufs mount to /var/lib/docker/aufs/mnt/66a568f6f6922b9ed2e978a31ea2ab77ddf4f65eae2291d0fe5a5a6c50eb875b-init: invalid argument\r\nE0708 20:05:33.329154    3077 manager.go:1579] Error running pod \"nginx_e2e-tests-kubectl-516xc\" container \"nginx\": API error (500): error creating aufs mount to /var/lib/docker/aufs/mnt/c6257ab9f8ac34c01b68d6b0ad1712981bce6fe8afcd2afd1b3976b3a65bc5fa-init: invalid argument\r\nE0708 20:05:35.510816    3077 manager.go:1579] Error running pod \"dns-test-a6805741-25ac-11e5-8fa6-42010af01555_e2e-tests-dns-8q0vk\" container \"jessie-querier\": API error (500): error creating aufs mount to /var/lib/docker/aufs/mnt/b2a84be51ae21ce45bc2cf1ecb944be2b4345c25798af0249260874185622e64-init: invalid argument\r\nE0708 20:05:35.609901    3077 manager.go:1579] Error running pod \"dns-test-a6805741-25ac-11e5-8fa6-42010af01555_e2e-tests-dns-8q0vk\" container \"jessie-querier\": API error (500): error creating aufs mount to /var/lib/docker/aufs/mnt/6f337451bbaa63fade20e59c88f87a6851412e3b5fb9d9b5d1a6457d1bb4e9f5-init: invalid argument\r\n```\r\n\r\nHere's the logs from this particular run: https://console.developers.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-e2e-gce-parallel/3046/\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/165976343","html_url":"https://github.com/kubernetes/kubernetes/issues/10959#issuecomment-165976343","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10959","id":165976343,"user":{"login":"jezell","id":22264,"avatar_url":"https://avatars.githubusercontent.com/u/22264?v=3","gravatar_id":"","url":"https://api.github.com/users/jezell","html_url":"https://github.com/jezell","followers_url":"https://api.github.com/users/jezell/followers","following_url":"https://api.github.com/users/jezell/following{/other_user}","gists_url":"https://api.github.com/users/jezell/gists{/gist_id}","starred_url":"https://api.github.com/users/jezell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jezell/subscriptions","organizations_url":"https://api.github.com/users/jezell/orgs","repos_url":"https://api.github.com/users/jezell/repos","events_url":"https://api.github.com/users/jezell/events{/privacy}","received_events_url":"https://api.github.com/users/jezell/received_events","type":"User","site_admin":false},"created_at":"2015-12-19T10:53:05Z","updated_at":"2015-12-19T10:53:05Z","body":"Was seeing this same problem today resizing instance groups in gke (not in e2e tests but on a production gke cluster running kube 1.1.3 / docker 1.9). So don't think serializing the images has made the problem go away. Kind of a pain to resolve since I had to restart docker daemon and manually repull images down to the node via docker pull to get pods to start."}}