{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/1866","labels_url":"https://api.github.com/repos/rancher/rancher/issues/1866/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/1866/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/1866/events","html_url":"https://github.com/rancher/rancher/issues/1866","id":102849247,"number":1866,"title":"upgrade from 0.32.0 -> 0.34.0-rc2 ->  github authentication stopped working after few hours after upgrage","user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancher/rancher/labels/area/authentication","name":"area/authentication","color":"fbca04"},{"url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"hibooboo2","id":1197823,"avatar_url":"https://avatars.githubusercontent.com/u/1197823?v=3","gravatar_id":"","url":"https://api.github.com/users/hibooboo2","html_url":"https://github.com/hibooboo2","followers_url":"https://api.github.com/users/hibooboo2/followers","following_url":"https://api.github.com/users/hibooboo2/following{/other_user}","gists_url":"https://api.github.com/users/hibooboo2/gists{/gist_id}","starred_url":"https://api.github.com/users/hibooboo2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hibooboo2/subscriptions","organizations_url":"https://api.github.com/users/hibooboo2/orgs","repos_url":"https://api.github.com/users/hibooboo2/repos","events_url":"https://api.github.com/users/hibooboo2/events{/privacy}","received_events_url":"https://api.github.com/users/hibooboo2/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/34","html_url":"https://github.com/rancher/rancher/milestones/Milestone%208/12/2015","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/34/labels","id":1240623,"number":34,"title":"Milestone 8/12/2015","description":"0.33","creator":{"login":"will-chan","id":10134676,"avatar_url":"https://avatars.githubusercontent.com/u/10134676?v=3","gravatar_id":"","url":"https://api.github.com/users/will-chan","html_url":"https://github.com/will-chan","followers_url":"https://api.github.com/users/will-chan/followers","following_url":"https://api.github.com/users/will-chan/following{/other_user}","gists_url":"https://api.github.com/users/will-chan/gists{/gist_id}","starred_url":"https://api.github.com/users/will-chan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/will-chan/subscriptions","organizations_url":"https://api.github.com/users/will-chan/orgs","repos_url":"https://api.github.com/users/will-chan/repos","events_url":"https://api.github.com/users/will-chan/events{/privacy}","received_events_url":"https://api.github.com/users/will-chan/received_events","type":"User","site_admin":false},"open_issues":10,"closed_issues":23,"state":"open","created_at":"2015-08-04T17:21:25Z","updated_at":"2015-08-24T17:32:35Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-08-24T17:32:25Z","updated_at":"2015-08-24T20:32:34Z","closed_at":null,"body":"Server version:\r\nInstall rancher agent - 0.32.0 with github auth enabled .\r\nAdded few github users as members to environments.\r\nAdded LB services.\r\nUpgrade to 0.34.0-rc2\r\n\r\nAfter upgrade ,  github auth on members of the environment worked fine.\r\n\r\nI tried logging in to this setup after few hours ( > 20 hours) and git hub auth was still working.\r\n\r\n1 day later now , no user is able  to get authenticated to this setup.\r\n\r\nFollowing exception seen in logs:\r\n```\r\n2015-08-24 17:15:54,191 ERROR [:] [] [] [] [torService-7986] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Scheduling reconnect for [2] \r\n2015-08-24 17:15:56,474 ERROR [:] [] [] [] [p406382148-8660] [i.g.i.g.r.handler.ExceptionHandler  ] Exception in API for request [io.github.ibuildthecloud.gdapi.request.ApiRequest@2bbba1e4] org.jooq.exception.DataAccessException: SQL [select distinct `account`.`id`, `account`.`name`, `account`.`kind`, `account`.`uuid`, `account`.`description`, `account`.`state`, `account`.`created`, `account`.`removed`, `account`.`remove_time`, `account`.`data`, `account`.`external_id`, `account`.`external_id_type`, `project_member`.`id`, `project_member`.`name`, `project_member`.`account_id`, `project_member`.`kind`, `project_member`.`uuid`, `project_member`.`description`, `project_member`.`state`, `project_member`.`created`, `project_member`.`removed`, `project_member`.`remove_time`, `project_member`.`data`, `project_member`.`external_id`, `project_member`.`project_id`, `project_member`.`external_id_type`, `project_member`.`role` from `account` join `project_member` on `project_member`.`project_id` = `account`.`id` where (1 = 0 or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?) or (`project_member`.`external_id` = ? and `project_member`.`external_id_type` = ? and `project_member`.`removed` is null and `project_member`.`state` = ?))]; Got error 28 from storage engine\r\n\tat org.jooq.impl.Utils.translate(Utils.java:1287) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:495) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:326) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:330) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.AbstractResultQuery.fetchInto(AbstractResultQuery.java:645) ~[jooq-3.3.0.jar:na]\r\n\tat io.cattle.platform.iaas.api.auth.dao.impl.AuthDaoImpl.getAccessibleProjects(AuthDaoImpl.java:243) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.iaas.api.auth.dao.impl.AuthDaoImpl.hasAccessToAnyProject(AuthDaoImpl.java:416) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.iaas.api.auth.integration.github.GithubTokenCreator.getGithubToken(GithubTokenCreator.java:76) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.iaas.api.auth.integration.github.GithubTokenCreator.getToken(GithubTokenCreator.java:119) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.iaas.api.auth.identity.TokenResourceManager.createToken(TokenResourceManager.java:65) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.iaas.api.auth.identity.TokenResourceManager.createInternal(TokenResourceManager.java:54) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.github.ibuildthecloud.gdapi.request.resource.impl.AbstractBaseResourceManager.create(AbstractBaseResourceManager.java:115) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.github.ibuildthecloud.gdapi.request.handler.ResourceManagerRequestHandler.generate(ResourceManagerRequestHandler.java:39) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.github.ibuildthecloud.gdapi.request.handler.AbstractResponseGenerator.handle(AbstractResponseGenerator.java:14) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.handle(DefaultReadWriteApiDelegate.java:27) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.write(DefaultReadWriteApiDelegate.java:22) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat sun.reflect.GeneratedMethodAccessor447.invoke(Unknown Source) ~[na:na]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]\r\n\tat java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]\r\n\tat org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]\r\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]\r\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]\r\n\tat org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]\r\n\tat org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]\r\n\tat org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE]\r\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]\r\n\tat org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]\r\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]\r\n\tat org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) ~[spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]\r\n\tat com.sun.proxy.$Proxy41.write(Unknown Source) ~[na:na]\r\n\tat io.github.ibuildthecloud.gdapi.request.handler.write.ReadWriteApiHandler.handle(ReadWriteApiHandler.java:19) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:86) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:60) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:53) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [jetty-security-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]\r\nCaused by: java.sql.SQLException: Got error 28 from storage engine\r\n\tat org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:149) ~[mariadb-java-client-1.1.7.jar:na]\r\n\tat org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106) ~[mariadb-java-client-1.1.7.jar:na]\r\n\tat org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264) ~[mariadb-java-client-1.1.7.jar:na]\r\n\tat org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288) ~[mariadb-java-client-1.1.7.jar:na]\r\n\tat org.mariadb.jdbc.MySQLPreparedStatement.execute(MySQLPreparedStatement.java:141) ~[mariadb-java-client-1.1.7.jar:na]\r\n\tat org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) ~[commons-dbcp-1.4.jar:1.4]\r\n\tat org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) ~[commons-dbcp-1.4.jar:1.4]\r\n\tat org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:194) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:261) ~[jooq-3.3.0.jar:na]\r\n\tat org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:322) ~[jooq-3.3.0.jar:na]\r\n\t... 57 common frames omitted\r\nCaused by: org.mariadb.jdbc.internal.common.QueryException: Got error 28 from storage engine\r\n\tat org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:942) ~[mariadb-java-client-1.1.7.jar:na]\r\n\tat org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:991) ~[mariadb-java-client-1.1.7.jar:na]\r\n\tat org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281) ~[mariadb-java-client-1.1.7.jar:na]\r\n\t... 63 common frames omitted\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/134370300","html_url":"https://github.com/rancher/rancher/issues/1866#issuecomment-134370300","issue_url":"https://api.github.com/repos/rancher/rancher/issues/1866","id":134370300,"user":{"login":"rancherdev","id":10679260,"avatar_url":"https://avatars.githubusercontent.com/u/10679260?v=3","gravatar_id":"","url":"https://api.github.com/users/rancherdev","html_url":"https://github.com/rancherdev","followers_url":"https://api.github.com/users/rancherdev/followers","following_url":"https://api.github.com/users/rancherdev/following{/other_user}","gists_url":"https://api.github.com/users/rancherdev/gists{/gist_id}","starred_url":"https://api.github.com/users/rancherdev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rancherdev/subscriptions","organizations_url":"https://api.github.com/users/rancherdev/orgs","repos_url":"https://api.github.com/users/rancherdev/repos","events_url":"https://api.github.com/users/rancherdev/events{/privacy}","received_events_url":"https://api.github.com/users/rancherdev/received_events","type":"User","site_admin":false},"created_at":"2015-08-24T20:32:34Z","updated_at":"2015-08-24T20:32:34Z","body":"The reason for running out of disk space is because of container_event table growing very large:\r\n\r\nmysql> select count(*) from  container_event;\r\n+----------+\r\n| count(*) |\r\n+----------+\r\n|    19086 |\r\n+----------+\r\n1 row in set (0.00 sec)\r\n\r\nmysql> \r\n"}}