{"action":"created","issue":{"url":"https://api.github.com/repos/rancherio/rancher/issues/1379","labels_url":"https://api.github.com/repos/rancherio/rancher/issues/1379/labels{/name}","comments_url":"https://api.github.com/repos/rancherio/rancher/issues/1379/comments","events_url":"https://api.github.com/repos/rancherio/rancher/issues/1379/events","html_url":"https://github.com/rancherio/rancher/issues/1379","id":92591085,"number":1379,"title":"Unknown exception running process - Can't start containers","user":{"login":"haswalt","id":338588,"avatar_url":"https://avatars.githubusercontent.com/u/338588?v=3","gravatar_id":"","url":"https://api.github.com/users/haswalt","html_url":"https://github.com/haswalt","followers_url":"https://api.github.com/users/haswalt/followers","following_url":"https://api.github.com/users/haswalt/following{/other_user}","gists_url":"https://api.github.com/users/haswalt/gists{/gist_id}","starred_url":"https://api.github.com/users/haswalt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haswalt/subscriptions","organizations_url":"https://api.github.com/users/haswalt/orgs","repos_url":"https://api.github.com/users/haswalt/repos","events_url":"https://api.github.com/users/haswalt/events{/privacy}","received_events_url":"https://api.github.com/users/haswalt/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-02T09:16:41Z","updated_at":"2015-07-02T10:27:24Z","closed_at":null,"body":"When trying to start a new container on a host the container never starts and the following logs are constantly output:\r\n\r\n# Server\r\n\r\n```\r\n2015-07-02 09:08:05,846 ERROR [:] [] [] [] [cutorService-22] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [instance.start:36] on [1] java.lang.IllegalArgumentException: Invalid path string \"/cattle/locks/59/206/DOCKER.STORAGE_POOL.VOLUME.CREATE.1.file:///var/lib/docker/volumes/6ff0e5a73f59c13956584d1d5e648fb674039c9702ad794559b353cf9193ca02/_data\" caused by empty node name specified @63\r\n\tat org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:101) ~[curator-client-2.3.0.jar:na]\r\n\tat org.apache.curator.framework.recipes.locks.LockInternals.<init>(LockInternals.java:103) ~[curator-recipes-2.3.0.jar:na]\r\n\tat org.apache.curator.framework.recipes.locks.InterProcessMutex.<init>(InterProcessMutex.java:190) ~[curator-recipes-2.3.0.jar:na]\r\n\tat org.apache.curator.framework.recipes.locks.InterProcessMutex.<init>(InterProcessMutex.java:64) ~[curator-recipes-2.3.0.jar:na]\r\n\tat io.cattle.platform.zookeeper.lock.ZookeeperLockProvider.getLock(ZookeeperLockProvider.java:38) ~[cattle-zookeeper-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.zookeeper.lock.ZookeeperLockProvider.createLock(ZookeeperLockProvider.java:27) ~[cattle-zookeeper-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.provider.impl.AbstractStandardLockProvider.getLock(AbstractStandardLockProvider.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.getLock(LockManagerImpl.java:66) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:23) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.docker.process.instancehostmap.DockerPostInstanceHostMapActivate.createVolumeInStoragePool(DockerPostInstanceHostMapActivate.java:193) ~[cattle-docker-compute-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.docker.process.instancehostmap.DockerPostInstanceHostMapActivate.processVolumes(DockerPostInstanceHostMapActivate.java:171) ~[cattle-docker-compute-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.docker.process.instancehostmap.DockerPostInstanceHostMapActivate.handle(DockerPostInstanceHostMapActivate.java:113) ~[cattle-docker-compute-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:455) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:395) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:389) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:389) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:522) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:322) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:258) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:255) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:188) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:161) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:110) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:107) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.object.process.impl.DefaultObjectProcessManager.executeStandardProcess(DefaultObjectProcessManager.java:29) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.common.handler.AbstractObjectProcessLogic.activate(AbstractObjectProcessLogic.java:31) ~[cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.instance.InstanceStart.compute(InstanceStart.java:179) ~[cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.instance.InstanceStart.handle(InstanceStart.java:72) ~[cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:455) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:395) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:389) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:389) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:519) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:322) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:258) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:255) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:188) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:161) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:110) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:107) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:66) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat sun.reflect.GeneratedMethodAccessor516.invoke(Unknown Source) ~[na:na]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]\r\n\tat java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener$1.doWithLockNoResult(MethodInvokingListener.java:69) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener.onEvent(MethodInvokingListener.java:65) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.impl.AbstractThreadPoolingEventService$2.doRun(AbstractThreadPoolingEventService.java:135) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:13) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]\r\n```\r\n\r\n## Rancher version:\r\n\r\n```\r\nRancher\tv0.25.0\r\nCattle\tv0.58.0\r\nUser Interface\tv0.29.0\r\nRancher Compose\tlatest\r\n```\r\n\r\n## docker info\r\n\r\n```\r\nContainers: 1\r\nImages: 29\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 31\r\n Dirperm1 Supported: false\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 3.13.0-52-generic\r\nOperating System: Ubuntu 14.04.2 LTS\r\nCPUs: 1\r\nTotal Memory: 490 MiB\r\n```\r\n\r\n```\r\nClient version: 1.7.0\r\nClient API version: 1.19\r\nGo version (client): go1.4.2\r\nGit commit (client): 0baf609\r\nOS/Arch (client): linux/amd64\r\nServer version: 1.7.0\r\nServer API version: 1.19\r\nGo version (server): go1.4.2\r\nGit commit (server): 0baf609\r\nOS/Arch (server): linux/amd64\r\n```\r\n\r\nThe server and node are running on Digital Ocean using their docker images with swap and memory limits enabled. I've tested other docker machines and they start fine when started manually with the docker CLI.\r\n\r\nThe network agent appears to be created and started fine but the container gets stuck at \"Starting\" until it times out.\r\n\r\nThere are messages about container being locked on the host so we get sequential start, start, start die events repeated.\r\n\r\nThis was done on a fresh install with multiple nodes to test this issue.\r\n"},"comment":{"url":"https://api.github.com/repos/rancherio/rancher/issues/comments/117992712","html_url":"https://github.com/rancherio/rancher/issues/1379#issuecomment-117992712","issue_url":"https://api.github.com/repos/rancherio/rancher/issues/1379","id":117992712,"user":{"login":"haswalt","id":338588,"avatar_url":"https://avatars.githubusercontent.com/u/338588?v=3","gravatar_id":"","url":"https://api.github.com/users/haswalt","html_url":"https://github.com/haswalt","followers_url":"https://api.github.com/users/haswalt/followers","following_url":"https://api.github.com/users/haswalt/following{/other_user}","gists_url":"https://api.github.com/users/haswalt/gists{/gist_id}","starred_url":"https://api.github.com/users/haswalt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haswalt/subscriptions","organizations_url":"https://api.github.com/users/haswalt/orgs","repos_url":"https://api.github.com/users/haswalt/repos","events_url":"https://api.github.com/users/haswalt/events{/privacy}","received_events_url":"https://api.github.com/users/haswalt/received_events","type":"User","site_admin":false},"created_at":"2015-07-02T10:27:24Z","updated_at":"2015-07-02T10:27:24Z","body":"FYI tested using a manually created host machine and one autocreated using the Digital Ocean API. Both nodes are hosted on Digital Ocean"}}