{"action":"created","issue":{"url":"https://api.github.com/repos/google/cadvisor/issues/306","labels_url":"https://api.github.com/repos/google/cadvisor/issues/306/labels{/name}","comments_url":"https://api.github.com/repos/google/cadvisor/issues/306/comments","events_url":"https://api.github.com/repos/google/cadvisor/issues/306/events","html_url":"https://github.com/google/cadvisor/issues/306","id":48621234,"number":306,"title":"Failed to get container \"\" with error: unable to find data for container \"\"","user":{"login":"danielkraaij","id":607089,"avatar_url":"https://avatars.githubusercontent.com/u/607089?v=3","gravatar_id":"","url":"https://api.github.com/users/danielkraaij","html_url":"https://github.com/danielkraaij","followers_url":"https://api.github.com/users/danielkraaij/followers","following_url":"https://api.github.com/users/danielkraaij/following{/other_user}","gists_url":"https://api.github.com/users/danielkraaij/gists{/gist_id}","starred_url":"https://api.github.com/users/danielkraaij/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielkraaij/subscriptions","organizations_url":"https://api.github.com/users/danielkraaij/orgs","repos_url":"https://api.github.com/users/danielkraaij/repos","events_url":"https://api.github.com/users/danielkraaij/events{/privacy}","received_events_url":"https://api.github.com/users/danielkraaij/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":24,"created_at":"2014-11-13T10:39:10Z","updated_at":"2015-02-27T08:57:22Z","closed_at":null,"body":"I Recently found this great tool and started using it on some hosts i manage. However on one host the following happend when viewing docker containers in more detail.\r\n`Failed to get container \"/docker/e0b572cde642d3bc21542377541d527cd6e3cf90cb76addd84e429669f4c6476\" with error: unable to find data for container /docker/e0b572cde642d3bc21542377541d527cd6e3cf90cb76addd84e429669f4c6476`\r\n\r\nI Found the following lines in the log files.\r\n`/var/run/cadvisor.INFO`\r\n```\r\nI1113 10:17:21.708953 00001 container.go:202] Failed to update stats for container \"/docker/e0b572cde642d3bc21542377541d527cd6e3cf90cb76addd84e429669f4c6476\": could not find device with major: 252, minor: 1 in cached partitions map\r\nW1113 10:17:21.709006 00001 container.go:118] Failed to get RecentStats(\"/docker/e0b572cde642d3bc21542377541d527cd6e3cf90cb76addd84e429669f4c6476\") while determining the next housekeeping: unable to find data for container /docker/e0b572cde642d3bc21542377541d527cd6e3cf90cb76addd84e429669f4c6476\r\n```\r\n\r\n`/var/run/cadvisor.WARNING`\r\n```\r\nW1113 10:16:20.709530 00001 container.go:118] Failed to get RecentStats(\"/docker/e0b572cde642d3bc21542377541d527cd6e3cf90cb76addd84e429669f4c6476\") while determining the next housekeeping: unable to find data for container /docker/e0b572cde642d3bc21542377541d527cd6e3cf90cb76addd84e429669f4c6476\r\nW1113 10:16:21.708884 00001 container.go:118] Failed to get RecentStats(\"/docker/e0b572cde642d3bc21542377541d527cd6e3cf90cb76addd84e429669f4c6476\") while determining the next housekeeping: unable to find data for container /docker/e0b572cde642d3bc21542377541d527cd6e3cf90cb76addd84e429669f4c6476\r\n```\r\n\r\n`/var/run/cadvisor.ERROR`\r\n```\r\nE1113 09:51:02.294062 00001 handler.go:288] Libcontainer state not found for container \"/docker/1644772d9144a29b477fa5ad76651be1bf34c2600fa4fe3d5258e3b647a262f7\"\r\nE1113 09:51:06.294204 00001 handler.go:288] Libcontainer state not found for container \"/docker/1644772d9144a29b477fa5ad76651be1bf34c2600fa4fe3d5258e3b647a262f7\"\r\n```\r\n\r\nHere is some information about docker itself\r\n```\r\n~# docker info\r\nContainers: 4\r\nImages: 77\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Dirs: 85\r\nExecution Driver: native-0.2\r\nKernel Version: 3.13.0-39-generic\r\nOperating System: Ubuntu 14.04.1 LTS\r\nWARNING: No swap limit support\r\n```\r\nContents of the `/var/lib/docker`\r\n```\r\n~# ls -la /var/lib/docker/\r\ntotal 40\r\ndrwx------ 11 root root 4096 Nov 13 11:09 .\r\ndrwxr-xr-x 42 root root 4096 Oct 13 09:44 ..\r\ndrwxr-xr-x  5 root root   40 Sep 11 09:55 aufs\r\ndrwx------  6 root root 4096 Nov 13 10:54 containers\r\ndrwx------  3 root root   19 Sep 11 09:55 execdriver\r\ndrwx------ 80 root root 8192 Nov 13 10:47 graph\r\ndrwx------  2 root root   98 Nov 13 11:09 init\r\n-rw-r--r--  1 root root 5120 Nov 13 11:09 linkgraph.db\r\n-rw-------  1 root root  412 Nov 13 10:47 repositories-aufs\r\ndrwx------  2 root root    6 Nov 12 11:46 tmp\r\ndrwx------  2 root root    6 Oct 22 16:01 trust\r\ndrwx------  3 root root   16 Nov  3 15:18 vfs\r\ndrwx------ 23 root root 4096 Nov 13 11:09 volumes\r\n```\r\n\r\nI have same OS and kernel version on my laptop where it works perfectly. The only difference is that the server is a VM running on KVM \r\n\r\nAny ideas on how to fix the problem?"},"comment":{"url":"https://api.github.com/repos/google/cadvisor/issues/comments/76358425","html_url":"https://github.com/google/cadvisor/issues/306#issuecomment-76358425","issue_url":"https://api.github.com/repos/google/cadvisor/issues/306","id":76358425,"user":{"login":"danielkraaij","id":607089,"avatar_url":"https://avatars.githubusercontent.com/u/607089?v=3","gravatar_id":"","url":"https://api.github.com/users/danielkraaij","html_url":"https://github.com/danielkraaij","followers_url":"https://api.github.com/users/danielkraaij/followers","following_url":"https://api.github.com/users/danielkraaij/following{/other_user}","gists_url":"https://api.github.com/users/danielkraaij/gists{/gist_id}","starred_url":"https://api.github.com/users/danielkraaij/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielkraaij/subscriptions","organizations_url":"https://api.github.com/users/danielkraaij/orgs","repos_url":"https://api.github.com/users/danielkraaij/repos","events_url":"https://api.github.com/users/danielkraaij/events{/privacy}","received_events_url":"https://api.github.com/users/danielkraaij/received_events","type":"User","site_admin":false},"created_at":"2015-02-27T08:57:22Z","updated_at":"2015-02-27T08:57:22Z","body":"I am terrible sorry for not getting back to you earlier with this. I just ran some tests with the latest version and all seems to be working just fine now. So somehow the problem got fixed along the way :) \r\n\r\nHere is the output of my /validate\r\n\r\n```\r\ncAdvisor version: 0.9.0\r\n\r\nOS version: Buildroot 2014.02\r\n\r\nKernel version: [Supported and recommended]\r\n\tKernel version is 3.13.0-45-generic. Versions >= 2.6 are supported. 3.0+ are recommended.\r\n\r\n\r\nCgroup setup: [Supported and recommended]\r\n\tAvailable cgroups: map[cpu:1 perf_event:1 cpuset:1 memory:1 devices:1 freezer:1 blkio:1 hugetlb:1 cpuacct:1]\r\n\tFollowing cgroups are required: [cpu cpuacct]\r\n\tFollowing other cgroups are recommended: [memory blkio cpuset devices freezer]\r\n\r\n\r\nCgroup mount setup: [Supported, but not recommended]\r\n\tCgroups are mounted at /rootfs/sys/fs/cgroup.\r\n\tCgroup mount directories: blkio cpu cpuacct cpuset devices freezer hugetlb memory perf_event systemd \r\n\tAny cgroup mount point that is detectible and accessible is supported. /sys/fs/cgroup is recommended as a standard location.\r\n\tCgroup mounts:\r\n\tcgroup /rootfs/sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0\r\n\tcgroup /rootfs/sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0\r\n\tcgroup /rootfs/sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0\r\n\tcgroup /rootfs/sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0\r\n\tcgroup /rootfs/sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0\r\n\tcgroup /rootfs/sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0\r\n\tcgroup /rootfs/sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0\r\n\tcgroup /rootfs/sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0\r\n\tcgroup /rootfs/sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb 0 0\r\n\tsystemd /rootfs/sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0\r\n\tcgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0\r\n\tcgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0\r\n\tcgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0\r\n\tcgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0\r\n\tcgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0\r\n\tcgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0\r\n\tcgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0\r\n\tcgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0\r\n\tcgroup /sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb 0 0\r\n\tsystemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb 0 0\r\n\tsystemd /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0\r\n\tcgroup /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb 0 0\r\n\tsystemd /var/lib/docker/aufs/mnt/c127eb4b247faf5f7c9b806b1fd4d036bc328ae9743b70bb0c0a59296afa69f6/rootfs/sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0\r\n\r\n\r\nDocker version: [Supported and recommended]\r\n\tDocker version is 1.5.0. Versions >= 1.0 are supported. 1.2+ are recommended.\r\n\r\n\r\nDocker driver setup: [Supported and recommended]\r\n\tDocker exec driver is native-0.2. Storage driver is aufs.\r\n\tCgroups are being created through cgroup filesystem.\r\n\tDocker container state directory is at \"/var/lib/docker/execdriver/native\" and is accessible.\r\n\r\n\r\nBlock device setup: [Supported, but not recommended]\r\n\tNone of the devices support 'cfq' I/O scheduler. No disk stats can be reported.\r\n\t Disk \"dm-0\" Scheduler type \"none\".\r\n\t Disk \"dm-1\" Scheduler type \"none\".\r\n\t Disk \"vda\" Scheduler type \"none\".\r\n```"}}