{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18239","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18239/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18239/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18239/events","html_url":"https://github.com/kubernetes/kubernetes/issues/18239","id":120506090,"number":18239,"title":"Containerized cluster cannot mount default secrets","user":{"login":"2opremio","id":2362916,"avatar_url":"https://avatars.githubusercontent.com/u/2362916?v=3","gravatar_id":"","url":"https://api.github.com/users/2opremio","html_url":"https://github.com/2opremio","followers_url":"https://api.github.com/users/2opremio/followers","following_url":"https://api.github.com/users/2opremio/following{/other_user}","gists_url":"https://api.github.com/users/2opremio/gists{/gist_id}","starred_url":"https://api.github.com/users/2opremio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/2opremio/subscriptions","organizations_url":"https://api.github.com/users/2opremio/orgs","repos_url":"https://api.github.com/users/2opremio/repos","events_url":"https://api.github.com/users/2opremio/events{/privacy}","received_events_url":"https://api.github.com/users/2opremio/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-12-04T23:18:26Z","updated_at":"2015-12-08T00:13:23Z","closed_at":null,"body":"When running a containerized kubernetes cluster I cannot spawn pods due to `file does not exist` errors when trying to mount their secret volumes:\r\n\r\n```\r\n$ kubectl create -f - <<EOF\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  name: busybox-sleep\r\nspec:\r\n  containers:\r\n  - name: busybox\r\n    image: busybox\r\n    args:\r\n    - sleep\r\n    - \"1000000\"\r\nEOF\r\npod \"busybox-sleep\" created\r\n```\r\n\r\n```\r\n$ kubectl get pods\r\nNAME                   READY     STATUS              RESTARTS   AGE\r\nbusybox-sleep          0/1       ContainerCreating   0          15s\r\nk8s-master-127.0.0.1   3/3       Running             0          8m\r\n```\r\n```\r\n$ kubectl describe pod busybox-sleep\r\nName:\t\t\t\tbusybox-sleep\r\nNamespace:\t\t\tdefault\r\nImage(s):\t\t\tbusybox\r\nNode:\t\t\t\t127.0.0.1/127.0.0.1\r\nStart Time:\t\t\tFri, 04 Dec 2015 22:48:56 +0000\r\nLabels:\t\t\t\t<none>\r\nStatus:\t\t\t\tPending\r\nReason:\t\t\t\t\r\nMessage:\t\t\t\r\nIP:\t\t\t\t\r\nReplication Controllers:\t<none>\r\nContainers:\r\n  busybox:\r\n    Container ID:\t\r\n    Image:\t\tbusybox\r\n    Image ID:\t\t\r\n    State:\t\tWaiting\r\n      Reason:\t\tContainerCreating\r\n    Ready:\t\tFalse\r\n    Restart Count:\t0\r\n    Environment Variables:\r\nConditions:\r\n  Type\t\tStatus\r\n  Ready \tFalse \r\nVolumes:\r\n  default-token-6dwl1:\r\n    Type:\tSecret (a secret that should populate this volume)\r\n    SecretName:\tdefault-token-6dwl1\r\nEvents:\r\n  FirstSeen\tLastSeen\tCount\tFrom\t\t\tSubobjectPath\tReason\t\tMessage\r\n  ─────────\t────────\t─────\t────\t\t\t─────────────\t──────\t\t───────\r\n  32s\t\t32s\t\t1\t{scheduler }\t\t\t\tScheduled\tSuccessfully assigned busybox-sleep to 127.0.0.1\r\n  32s\t\t10s\t\t3\t{kubelet 127.0.0.1}\t\t\tFailedMount\tUnable to mount volumes for pod \"busybox-sleep_default\": IsLikelyNotMountPoint(\"/var/lib/kubelet/pods/333c85c3-9ad9-11e5-aeed-f6b7b4d3b3d7/volumes/kubernetes.io~secret/default-token-6dwl1\"): file does not exist\r\n  32s\t\t10s\t\t3\t{kubelet 127.0.0.1}\t\t\tFailedSync\tError syncing pod, skipping: IsLikelyNotMountPoint(\"/var/lib/kubelet/pods/333c85c3-9ad9-11e5-aeed-f6b7b4d3b3d7/volumes/kubernetes.io~secret/default-token-6dwl1\"): file does not exist\r\n```\r\n\r\nThe secrets are there as expected:\r\n\r\n```\r\n$ kubectl get secrets\r\nNAME                  TYPE                                  DATA      AGE\r\ndefault-token-6dwl1   kubernetes.io/service-account-token   2         8m\r\n```\r\n\r\n\r\nHere are the full kubelet logs: https://gist.github.com/2opremio/6fcbae55d83f20bc3c82\r\n\r\nThe single-node containerized cluster is created using [this script](https://gist.github.com/2opremio/12eafc0c2485d8d13f2a), which is based on https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker.md and spawns a cluster based on a recent version of master (https://github.com/kubernetes/kubernetes/tree/81b6ac47557359429e519d889d9ad48b5c8a2411).\r\n\r\nThis is the docker version I am running:\r\n\r\n```\r\n$ docker info\r\nContainers: 7\r\nImages: 62\r\nStorage Driver: overlay\r\n Backing Filesystem: extfs\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 4.1.13-boot2docker\r\nOperating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015\r\nCPUs: 1\r\nTotal Memory: 996.2 MiB\r\nName: default\r\nID: 77RC:MDFB:UNPM:62Z6:FYLF:TZTG:43NZ:77DD:IL22:QI6S:LDSV:UN6W\r\nDebug mode (server): true\r\nFile Descriptors: 54\r\nGoroutines: 90\r\nSystem Time: 2015-12-04T23:01:25.308787463Z\r\nEventsListeners: 0\r\nInit SHA1: \r\nInit Path: /usr/local/bin/docker\r\nDocker Root Dir: /mnt/sda1/var/lib/docker\r\nLabels:\r\n provider=virtualbox\r\n```\r\n\r\n```\r\n$ docker version\r\nClient:\r\n Version:      1.8.1\r\n API version:  1.20\r\n Go version:   go1.4.2\r\n Git commit:   d12ea79\r\n Built:        Thu Aug 13 02:49:29 UTC 2015\r\n OS/Arch:      darwin/amd64\r\n\r\nServer:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.3\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 17:56:04 UTC 2015\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\nThe client runs in an OSX laptop and the daemon runs inside a VM provisioned with `docker-machine create --driver virtualbox --engine-storage-driver overlay default`\r\n\r\nSurprisingly, everything works when using a vagrant VM provisioned with https://github.com/weaveworks/weave/blob/vagrant-pin-docker/Vagrantfile, so the problem must somehow be environmental but I haven't managed to triage the difference causing the problem.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/162711867","html_url":"https://github.com/kubernetes/kubernetes/issues/18239#issuecomment-162711867","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18239","id":162711867,"user":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"created_at":"2015-12-08T00:13:22Z","updated_at":"2015-12-08T00:13:22Z","body":"@fgrzadkowski I am going to assign this to you for the triage. Thanks!"}}