{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/4584","labels_url":"https://api.github.com/repos/deis/deis/issues/4584/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/4584/comments","events_url":"https://api.github.com/repos/deis/deis/issues/4584/events","html_url":"https://github.com/deis/deis/issues/4584","id":110525027,"number":4584,"title":"Units failing to start with overlay errors v1.11.0","user":{"login":"jeff-lee","id":1505257,"avatar_url":"https://avatars.githubusercontent.com/u/1505257?v=3","gravatar_id":"","url":"https://api.github.com/users/jeff-lee","html_url":"https://github.com/jeff-lee","followers_url":"https://api.github.com/users/jeff-lee/followers","following_url":"https://api.github.com/users/jeff-lee/following{/other_user}","gists_url":"https://api.github.com/users/jeff-lee/gists{/gist_id}","starred_url":"https://api.github.com/users/jeff-lee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeff-lee/subscriptions","organizations_url":"https://api.github.com/users/jeff-lee/orgs","repos_url":"https://api.github.com/users/jeff-lee/repos","events_url":"https://api.github.com/users/jeff-lee/events{/privacy}","received_events_url":"https://api.github.com/users/jeff-lee/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/deis/labels/bug","name":"bug","color":"e11d21"},{"url":"https://api.github.com/repos/deis/deis/labels/logspout","name":"logspout","color":"f7c6c7"},{"url":"https://api.github.com/repos/deis/deis/labels/publisher","name":"publisher","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":19,"created_at":"2015-10-08T19:28:46Z","updated_at":"2015-10-09T23:26:05Z","closed_at":null,"body":"I'm getting a lot of overlayfs errors on a new v1.11.0 cluster running in AWS.  \r\n\r\nSome nodes work fine, but occasionally the logspout and publisher will fail to start with the following errors.  It seems that 50+% of the nodes that I scale up fail with this error.\r\n\r\nOct 08 19:23:15 ip-10-10-16-94.ec2.internal sh[6347]: bec2a6cd0884: Error pulling image (v1.11.0) from deis/logspout, Driver overlay failed to create image rootfs dbfc35752a295175c15febf830ffd73b748f6717237d48981703cd19d8de045e: lstat /var/lib/docker/overlay/8dd8107abd2e22bfd3b45b05733f3d2677d4078b09b5edce56ee3d8677d3c648: no such file or directory\r\nOct 08 19:23:15 ip-10-10-16-94.ec2.internal sh[6347]: Error pulling image (v1.11.0) from deis/logspout, Driver overlay failed to create image rootfs dbfc35752a295175c15febf830ffd73b748f6717237d48981703cd19d8de045e: lstat /var/lib/docker/overlay/8dd8107abd2e22bfd3b45b05733f3d2677d4078b09b5edce56ee3d8677d3c648: no such file or directory\r\nOct 08 19:23:20 ip-10-10-16-94.ec2.internal sh[1573]: 2015-10-08 19:23:20.196966 7f220b8b8700  0 mon.ip-10-10-16-94.ec2.internal@0(leader).data_health(16) update_stats avail 94% total 937 GB, used 6068 MB, avail 881 GB\r\nOct 08 19:23:20 ip-10-10-16-94.ec2.internal systemd[1]: deis-logspout.service: Service hold-off time over, scheduling restart.\r\nOct 08 19:23:20 ip-10-10-16-94.ec2.internal systemd[1]: Starting deis-logspout...\r\nOct 08 19:23:20 ip-10-10-16-94.ec2.internal dockerd[980]: time=\"2015-10-08T19:23:20.237587238Z\" level=info msg=\"GET /v1.19/images/deis/logspout:v1.11.0/history\"\r\nOct 08 19:23:20 ip-10-10-16-94.ec2.internal dockerd[980]: time=\"2015-10-08T19:23:20.237718085Z\" level=error msg=\"Handler for GET /images/{name:.*}/history returned error: could not find image: no such id: deis/logspout:v1.11.0\"\r\nOct 08 19:23:20 ip-10-10-16-94.ec2.internal dockerd[980]: time=\"2015-10-08T19:23:20.237736635Z\" level=error msg=\"HTTP Error\" err=\"could not find image: no such id: deis/logspout:v1.11.0\" statusCode=404\r\nOct 08 19:23:20 ip-10-10-16-94.ec2.internal dockerd[980]: time=\"2015-10-08T19:23:20.252288636Z\" level=info msg=\"POST /v1.19/images/create?fromImage=deis%2Flogspout%3Av1.11.0\"\r\nOct 08 19:23:20 ip-10-10-16-94.ec2.internal sh[6370]: Pulling repository deis/logspout\r\nOct 08 19:23:21 ip-10-10-16-94.ec2.internal sh[6370]: bec2a6cd0884: Pulling image (v1.11.0) from deis/logspout\r\nOct 08 19:23:21 ip-10-10-16-94.ec2.internal sh[6370]: bec2a6cd0884: Pulling image (v1.11.0) from deis/logspout, endpoint: https://registry-1.docker.io/v1/\r\nOct 08 19:23:21 ip-10-10-16-94.ec2.internal dockerd[980]: time=\"2015-10-08T19:23:21.589866312Z\" level=info msg=\"GET /containers/json\"\r\nOct 08 19:23:21 ip-10-10-16-94.ec2.internal sh[6370]: bec2a6cd0884: Pulling dependent layers\r\nOct 08 19:23:21 ip-10-10-16-94.ec2.internal sh[6370]: 8dd8107abd2e: Download complete\r\nOct 08 19:23:21 ip-10-10-16-94.ec2.internal sh[6370]: dbfc35752a29: Pulling metadata\r\nOct 08 19:23:21 ip-10-10-16-94.ec2.internal sh[6370]: dbfc35752a29: Pulling fs layer\r\nOct 08 19:23:22 ip-10-10-16-94.ec2.internal sh[6370]: dbfc35752a29: Error downloading dependent layers\r\nOct 08 19:23:22 ip-10-10-16-94.ec2.internal systemd[1]: deis-logspout.service: Control process exited, code=exited status=1\r\nOct 08 19:23:22 ip-10-10-16-94.ec2.internal systemd[1]: Failed to start deis-logspout.\r\nOct 08 19:23:22 ip-10-10-16-94.ec2.internal systemd[1]: deis-logspout.service: Unit entered failed state.\r\nOct 08 19:23:22 ip-10-10-16-94.ec2.internal systemd[1]: deis-logspout.service: Failed with result 'exit-code'."},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/147009288","html_url":"https://github.com/deis/deis/issues/4584#issuecomment-147009288","issue_url":"https://api.github.com/repos/deis/deis/issues/4584","id":147009288,"user":{"login":"jeff-lee","id":1505257,"avatar_url":"https://avatars.githubusercontent.com/u/1505257?v=3","gravatar_id":"","url":"https://api.github.com/users/jeff-lee","html_url":"https://github.com/jeff-lee","followers_url":"https://api.github.com/users/jeff-lee/followers","following_url":"https://api.github.com/users/jeff-lee/following{/other_user}","gists_url":"https://api.github.com/users/jeff-lee/gists{/gist_id}","starred_url":"https://api.github.com/users/jeff-lee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeff-lee/subscriptions","organizations_url":"https://api.github.com/users/jeff-lee/orgs","repos_url":"https://api.github.com/users/jeff-lee/repos","events_url":"https://api.github.com/users/jeff-lee/events{/privacy}","received_events_url":"https://api.github.com/users/jeff-lee/received_events","type":"User","site_admin":false},"created_at":"2015-10-09T23:26:05Z","updated_at":"2015-10-09T23:26:05Z","body":"@krancour I tried adding flock earlier and it didn't seem to resolve the issue, but I can pull from master and try again...if github co-operates.  It's giving me intermittent 500s.\r\n\r\nHere's what I added to my logspout unit file.\r\n\r\n$ fleetctl cat deis-logspout.service | grep flock\r\nExecStart=/bin/sh -c \"IMAGE=`/run/deis/bin/get_image /deis/logspout` && flock -w 600 /var/run/lock/docker docker run --name deis-logspout --rm -v /var/run/docker.sock:/tmp/docker.sock -e ETCD_HOST=$COREOS_PRIVATE_IPV4 -e HOST=$COREOS_PRIVATE_IPV4 -e DEBUG=1 $IMAGE\""}}