{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15556","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15556/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15556/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15556/events","html_url":"https://github.com/kubernetes/kubernetes/issues/15556","id":111233193,"number":15556,"title":"kubelet panic during iscsi volume cleanup","user":{"login":"rootfs","id":7062400,"avatar_url":"https://avatars.githubusercontent.com/u/7062400?v=3","gravatar_id":"","url":"https://api.github.com/users/rootfs","html_url":"https://github.com/rootfs","followers_url":"https://api.github.com/users/rootfs/followers","following_url":"https://api.github.com/users/rootfs/following{/other_user}","gists_url":"https://api.github.com/users/rootfs/gists{/gist_id}","starred_url":"https://api.github.com/users/rootfs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rootfs/subscriptions","organizations_url":"https://api.github.com/users/rootfs/orgs","repos_url":"https://api.github.com/users/rootfs/repos","events_url":"https://api.github.com/users/rootfs/events{/privacy}","received_events_url":"https://api.github.com/users/rootfs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-10-13T17:50:14Z","updated_at":"2015-10-13T17:50:14Z","closed_at":null,"body":"During iscsi volume cleanup, kubelet reads from /proc/mounts and retrieves a list of device paths where iscsi volume mounts on. The device path was expected to be in the form of `/dev/disk/by-path/ip-xxx-iqn-xxx-lun-x`.  But currently the device path in `/proc/mounts` is in the form of `/dev/sdX`, thus iscsi panicked [here](https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/iscsi/iscsi_util.go#L138). \r\n\r\nI suspect this panic started to happen after `mounter` switched to use `mount(8)` from `syscall`. `mount(8)` seems to always resolve device path (see `strace` output below), and thus the iscsi configuration is no longer in `/proc/mounts`\r\n\r\n```console\r\nlstat(\"/dev\", {st_mode=S_IFDIR|0755, st_size=3440, ...}) = 0\r\nlstat(\"/dev/disk\", {st_mode=S_IFDIR|0755, st_size=100, ...}) = 0\r\nlstat(\"/dev/disk/by-path\", {st_mode=S_IFDIR|0755, st_size=120, ...}) = 0\r\nlstat(\"/dev/disk/by-path/ip-10.1.4.12:3260-iscsi-iqn.2014-12.world.server:storage.target00-lun-0\", {st_mode=S_IFLNK|0777, st_size=9, ...}) = 0\r\nreadlink(\"/dev/disk/by-path/ip-10.1.4.12:3260-iscsi-iqn.2014-12.world.server:storage.target00-lun-0\", \"../../sdb\", 4095) = 9\r\nlstat(\"/dev/sdb\", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 16), ...}) = 0\r\nstat(\"/dev/sdb\", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 16), ...}) = 0\r\n\r\n```"}}