{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/1685","labels_url":"https://api.github.com/repos/rancher/rancher/issues/1685/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/1685/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/1685/events","html_url":"https://github.com/rancher/rancher/issues/1685","id":98054661,"number":1685,"title":"uncaught exception in rancher-server","user":{"login":"defunctzombie","id":84792,"avatar_url":"https://avatars.githubusercontent.com/u/84792?v=3","gravatar_id":"","url":"https://api.github.com/users/defunctzombie","html_url":"https://github.com/defunctzombie","followers_url":"https://api.github.com/users/defunctzombie/followers","following_url":"https://api.github.com/users/defunctzombie/following{/other_user}","gists_url":"https://api.github.com/users/defunctzombie/gists{/gist_id}","starred_url":"https://api.github.com/users/defunctzombie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/defunctzombie/subscriptions","organizations_url":"https://api.github.com/users/defunctzombie/orgs","repos_url":"https://api.github.com/users/defunctzombie/repos","events_url":"https://api.github.com/users/defunctzombie/events{/privacy}","received_events_url":"https://api.github.com/users/defunctzombie/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-07-30T00:04:30Z","updated_at":"2015-07-30T02:57:09Z","closed_at":"2015-07-30T00:32:11Z","body":"Server crashed with the following exception (unfortunately I cannot recall what was happening when it crashed).\r\n\r\n```\r\n2015-07-29 23:56:13,228 ERROR [:] [] [] [] [utorService-501] [o.a.c.m.context.NoExceptionRunnable ] Uncaught exception org.jooq.exception.DataChangedException: Database record has been changed\r\n\tat org.jooq.impl.UpdatableRecordImpl.checkIfChanged(UpdatableRecordImpl.java:550) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.UpdatableRecordImpl.storeUpdate0(UpdatableRecordImpl.java:291) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.UpdatableRecordImpl.access$200(UpdatableRecordImpl.java:90) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.UpdatableRecordImpl$3.operate(UpdatableRecordImpl.java:260) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.RecordDelegate.operate(RecordDelegate.java:123) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.UpdatableRecordImpl.storeUpdate(UpdatableRecordImpl.java:255) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.UpdatableRecordImpl.update(UpdatableRecordImpl.java:149) ~[jooq-3.3.0.jar:na]\r\n\tat io.cattle.platform.object.impl.JooqObjectManager.persistRecord(JooqObjectManager.java:216) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.object.impl.JooqObjectManager.setFieldsInternal(JooqObjectManager.java:125) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.object.impl.JooqObjectManager$3.execute(JooqObjectManager.java:113) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.idempotent.Idempotent.change(Idempotent.java:88) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.object.impl.JooqObjectManager.setFields(JooqObjectManager.java:110) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.agent.server.resource.impl.AgentResourcesMonitorImpl.setHosts(AgentResourcesMonitorImpl.java:206) ~[cattle-iaas-agent-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.agent.server.resource.impl.AgentResourcesMonitorImpl$2.doWithLockNoResult(AgentResourcesMonitorImpl.java:125) ~[cattle-iaas-agent-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.agent.server.resource.impl.AgentResourcesMonitorImpl.pingReply(AgentResourcesMonitorImpl.java:116) ~[cattle-iaas-agent-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat sun.reflect.GeneratedMethodAccessor619.invoke(Unknown Source) ~[na:na]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]\r\n\tat java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener$1.doWithLockNoResult(MethodInvokingListener.java:76) ~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener.onEvent(MethodInvokingListener.java:72) ~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.impl.AbstractThreadPoolingEventService$2.doRun(AbstractThreadPoolingEventService.java:135) ~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:13) ~[cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]\r\n```\r\n\r\nGiven that it takes so long for the server to boot, is there a way to catch these better and boot the server faster? Or maybe just have the single component that was affected restart?"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/126167510","html_url":"https://github.com/rancher/rancher/issues/1685#issuecomment-126167510","issue_url":"https://api.github.com/repos/rancher/rancher/issues/1685","id":126167510,"user":{"login":"ibuildthecloud","id":1754002,"avatar_url":"https://avatars.githubusercontent.com/u/1754002?v=3","gravatar_id":"","url":"https://api.github.com/users/ibuildthecloud","html_url":"https://github.com/ibuildthecloud","followers_url":"https://api.github.com/users/ibuildthecloud/followers","following_url":"https://api.github.com/users/ibuildthecloud/following{/other_user}","gists_url":"https://api.github.com/users/ibuildthecloud/gists{/gist_id}","starred_url":"https://api.github.com/users/ibuildthecloud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ibuildthecloud/subscriptions","organizations_url":"https://api.github.com/users/ibuildthecloud/orgs","repos_url":"https://api.github.com/users/ibuildthecloud/repos","events_url":"https://api.github.com/users/ibuildthecloud/events{/privacy}","received_events_url":"https://api.github.com/users/ibuildthecloud/received_events","type":"User","site_admin":false},"created_at":"2015-07-30T02:57:09Z","updated_at":"2015-07-30T02:57:09Z","body":"We'll fix the memory issue.  I believe I know what's up with that.  Sorry, I know 1GB is excessive.  We'll fix it.\r\n\r\nThe DNS and overlay networking shouldn't go down when the server crashes.  If you start a bunch of containers then stop the server everything should still work except that new containers won't get networking.  We are trying to change that one too.   So if you see it goes down then we've hit some bug.\r\n\r\nIf your server ran out of memory there's a good chance it was doing a lot of stuff at the time.  maybe there was some glitch and containers just dying?  I don't know.  But I would guess the networking might have been screwed up because of that?\r\n\r\nIn the end, sorry.  Trust me, we will get this thing rock solid just give us a bit."}}