{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/2832","labels_url":"https://api.github.com/repos/rancher/rancher/issues/2832/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/2832/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/2832/events","html_url":"https://github.com/rancher/rancher/issues/2832","id":119088955,"number":2832,"title":"v0.47.0 Instability","user":{"login":"bacheson","id":9341738,"avatar_url":"https://avatars.githubusercontent.com/u/9341738?v=3","gravatar_id":"","url":"https://api.github.com/users/bacheson","html_url":"https://github.com/bacheson","followers_url":"https://api.github.com/users/bacheson/followers","following_url":"https://api.github.com/users/bacheson/following{/other_user}","gists_url":"https://api.github.com/users/bacheson/gists{/gist_id}","starred_url":"https://api.github.com/users/bacheson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bacheson/subscriptions","organizations_url":"https://api.github.com/users/bacheson/orgs","repos_url":"https://api.github.com/users/bacheson/repos","events_url":"https://api.github.com/users/bacheson/events{/privacy}","received_events_url":"https://api.github.com/users/bacheson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-26T18:18:08Z","updated_at":"2015-11-26T19:57:56Z","closed_at":null,"body":"Can't gracefully upgrade services anymore. Upgrade processes hang and the following errors repeated:\r\n\r\n2015-11-26 01:26:13,348 ERROR [8f7e7132-3ecf-4b8b-982d-77897f19f3a7:132397] [instance:9804] [instance.start->(InstanceStart)->instance.allocate->(InstanceAllocate)] [] [cutorService-15] [c.p.a.e.i.AllocatorEventListenerImpl] No allocator handled [EventVO [id=269e8e3a-dc5e-4dd4-9839-df396d23ba29, name=instance.allocate, previousNames=null, replyTo=reply.8417750311673964762, resourceId=9804, resourceType=instance, publisher=null, transitioning=null, transitioningMessage=null, transitioningInternalMessage=null, previousIds=null, data={}, time=Thu Nov 26 01:26:13 UTC 2015, listenerKey=null, transitioningProgress=null]] \r\n2015-11-26 01:26:13,350 ERROR [8f7e7132-3ecf-4b8b-982d-77897f19f3a7:132397] [instance:9804] [instance.start->(InstanceStart)->instance.allocate] [] [cutorService-88] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception io.cattle.platform.eventing.exception.EventExecutionException: Failed to find a placement\r\n\tat io.cattle.platform.eventing.exception.EventExecutionException.fromEvent(EventExecutionException.java:53) ~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.impl.AbstractEventService.callSync(AbstractEventService.java:263) ~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.common.handler.EventBasedProcessHandler.handle(EventBasedProcessHandler.java:109) ~[cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.instance.InstanceAllocate.handle(InstanceAllocate.java:73) ~[cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:456) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:396) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:390) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:390) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:520) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:323) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:259) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:256) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:188) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:161) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:110) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:107) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.common.handler.AbstractObjectProcessLogic.execute(AbstractObjectProcessLogic.java:131) [cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.instance.InstanceStart.allocate(InstanceStart.java:205) [cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.instance.InstanceStart.handle(InstanceStart.java:70) [cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:456) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:396) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:390) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:390) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:520) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:323) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:259) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:256) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:188) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:161) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:110) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:107) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:66) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat sun.reflect.GeneratedMethodAccessor534.invoke(Unknown Source) ~[na:na]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_85]\r\n\tat java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_85]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener$1.doWithLockNoResult(MethodInvokingListener.java:76) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener.onEvent(MethodInvokingListener.java:72) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.impl.AbstractThreadPoolingEventService$2.doRun(AbstractThreadPoolingEventService.java:135) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:13) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_85]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_85]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/159986353","html_url":"https://github.com/rancher/rancher/issues/2832#issuecomment-159986353","issue_url":"https://api.github.com/repos/rancher/rancher/issues/2832","id":159986353,"user":{"login":"bacheson","id":9341738,"avatar_url":"https://avatars.githubusercontent.com/u/9341738?v=3","gravatar_id":"","url":"https://api.github.com/users/bacheson","html_url":"https://github.com/bacheson","followers_url":"https://api.github.com/users/bacheson/followers","following_url":"https://api.github.com/users/bacheson/following{/other_user}","gists_url":"https://api.github.com/users/bacheson/gists{/gist_id}","starred_url":"https://api.github.com/users/bacheson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bacheson/subscriptions","organizations_url":"https://api.github.com/users/bacheson/orgs","repos_url":"https://api.github.com/users/bacheson/repos","events_url":"https://api.github.com/users/bacheson/events{/privacy}","received_events_url":"https://api.github.com/users/bacheson/received_events","type":"User","site_admin":false},"created_at":"2015-11-26T19:57:56Z","updated_at":"2015-11-26T19:57:56Z","body":"i'm not exposing public ports and no scheduling labels are applied. The only label applied is io.rancher.container.dns = true.\r\n\r\nI have a backlog of processes now throwing errors like the one above and all system operations now lock up as a result of the traffic jam. \r\n\r\nLaunching 200-300 containers and performing various administrative operations over a 1-2 week period has lead me back to this problem for the third time. I end up just starting fresh and everything works for a while.\r\n"}}