{"action":"created","issue":{"url":"https://api.github.com/repos/rancherio/rancher/issues/733","labels_url":"https://api.github.com/repos/rancherio/rancher/issues/733/labels{/name}","comments_url":"https://api.github.com/repos/rancherio/rancher/issues/733/comments","events_url":"https://api.github.com/repos/rancherio/rancher/issues/733/events","html_url":"https://github.com/rancherio/rancher/issues/733","id":73791853,"number":733,"title":"Instances belonging to a service stuck in \"purging\" state.","user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancherio/rancher/labels/area/service","name":"area/service","color":"FBCA04"},{"url":"https://api.github.com/repos/rancherio/rancher/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2015-05-07T01:14:56Z","updated_at":"2015-05-11T18:04:26Z","closed_at":"2015-05-11T18:04:25Z","body":"Server version - V0.18.2\r\n\r\nActivated a service with scale 4 .\r\nTry to purge first 2 instances of the service.\r\nInstances are stuck in \"purging\" state for ever. This issue is not seen always.\r\n\r\n| 861 | purging  | test889245_test818596_1 |\r\n| 862 | purging  | test889245_test818596_2 |\r\n\r\nFollowing exception seen on the server repeatedly.\r\n\r\n2015-05-07 01:06:13,952 ERROR [:] [] [] [] [utorService-561] [c.p.e.p.i.DefaultProcessInstanceImpl] final ExitReason is null, should not be \r\n2015-05-07 01:06:13,956 ERROR [:] [] [] [] [utorService-561] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [instance.purge:19636] on [862] java.lang.IllegalStateException: Attempt to cancel when process is still transitioning\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:193) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:160) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:109) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:106) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:106) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:53) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat sun.reflect.GeneratedMethodAccessor568.invoke(Unknown Source) ~[na:na]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]\r\n\tat java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener$1.doWithLockNoResult(MethodInvokingListener.java:69) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener.onEvent(MethodInvokingListener.java:65) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.impl.AbstractThreadPoolingEventService$2.doRun(AbstractThreadPoolingEventService.java:135) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:13) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]\r\n\r\n2015-05-07 01:06:13,969 ERROR [:] [] [] [] [utorService-597] [c.p.e.p.i.DefaultProcessInstanceImpl] final ExitReason is null, should not be \r\n2015-05-07 01:06:13,974 ERROR [:] [] [] [] [utorService-597] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [instance.purge:19635] on [861] java.lang.IllegalStateException: Attempt to cancel when process is still transitioning\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:193) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:160) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:109) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:106) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:106) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:53) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat sun.reflect.GeneratedMethodAccessor568.invoke(Unknown Source) ~[na:na]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]\r\n\tat java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener$1.doWithLockNoResult(MethodInvokingListener.java:69) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener.onEvent(MethodInvokingListener.java:65) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.impl.AbstractThreadPoolingEventService$2.doRun(AbstractThreadPoolingEventService.java:135) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:13) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]"},"comment":{"url":"https://api.github.com/repos/rancherio/rancher/issues/comments/101000898","html_url":"https://github.com/rancherio/rancher/issues/733#issuecomment-101000898","issue_url":"https://api.github.com/repos/rancherio/rancher/issues/733","id":101000898,"user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"created_at":"2015-05-11T18:04:25Z","updated_at":"2015-05-11T18:04:25Z","body":"Tested with server version - V0.20.0\r\nI have not been able to reproduce the issue where instances are stuck in \"purging\" state."}}