{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/3744","labels_url":"https://api.github.com/repos/deis/deis/issues/3744/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/3744/comments","events_url":"https://api.github.com/repos/deis/deis/issues/3744/events","html_url":"https://github.com/deis/deis/issues/3744","id":80764207,"number":3744,"title":"Deis Database lost relations","user":{"login":"chrisgeo","id":216250,"avatar_url":"https://avatars.githubusercontent.com/u/216250?v=3","gravatar_id":"","url":"https://api.github.com/users/chrisgeo","html_url":"https://github.com/chrisgeo","followers_url":"https://api.github.com/users/chrisgeo/followers","following_url":"https://api.github.com/users/chrisgeo/following{/other_user}","gists_url":"https://api.github.com/users/chrisgeo/gists{/gist_id}","starred_url":"https://api.github.com/users/chrisgeo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrisgeo/subscriptions","organizations_url":"https://api.github.com/users/chrisgeo/orgs","repos_url":"https://api.github.com/users/chrisgeo/repos","events_url":"https://api.github.com/users/chrisgeo/events{/privacy}","received_events_url":"https://api.github.com/users/chrisgeo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-05-26T03:13:01Z","updated_at":"2015-05-26T11:00:43Z","closed_at":null,"body":"It looks like the database (and the wal-e backup) decided to go kaput. I'm not sure where in the logs it would have happened over the last ~5 days but I can grab them and do some digging with a bit of guidance.\r\n\r\nAlso as an aside: is there a way to log directly into the deis database with the deis user?\r\n\r\n*Private data removed.*\r\n\r\n```bash\r\nay 26 03:09:55 sh[57854]: 2015-05-26 03:09:55 UTC ERROR:  relation \"authtoken_token\" does not exist at character 363\r\nMay 26 03:09:55  sh[57854]: 2015-05-26 03:09:55 UTC STATEMENT:  SELECT \"authtoken_token\".\"key\", \"authtoken_token\".\"user_id\", \"authtoken_token\".\"created\", \"auth_user\".\"id\", \"auth_user\".\"password\", \"auth_user\".\"last_login\", \"auth_user\".\"is_superuser\", \"auth_user\".\"username\", \"auth_user\".\"first_name\", \"auth_user\".\"last_name\", \"auth_user\".\"email\", \"auth_user\".\"is_staff\", \"auth_user\".\"is_active\", \"auth_user\".\"date_joined\" FROM \"authtoken_token\" INNER JOIN \"auth_user\" ON ( \"authtoken_token\".\"user_id\" = \"auth_user\".\"id\" ) WHERE \"authtoken_token\".\"key\" = ''\r\nMay 26 03:10:19  sh[57854]: 2015-05-26 03:10:19 UTC ERROR:  relation \"authtoken_token\" does not exist at character 363\r\nMay 26 03:10:19 l sh[57854]: 2015-05-26 03:10:19 UTC STATEMENT:  SELECT \"authtoken_token\".\"key\", \"authtoken_token\".\"user_id\", \"authtoken_token\".\"created\", \"auth_user\".\"id\", \"auth_user\".\"password\", \"auth_user\".\"last_login\", \"auth_user\".\"is_superuser\", \"auth_user\".\"username\", \"auth_user\".\"first_name\", \"auth_user\".\"last_name\", \"auth_user\".\"email\", \"auth_user\".\"is_staff\", \"auth_user\".\"is_active\", \"auth_user\".\"date_joined\" FROM \"authtoken_token\" INNER JOIN \"auth_user\" ON ( \"authtoken_token\".\"user_id\" = \"auth_user\".\"id\" ) WHERE \"authtoken_token\".\"key\" = ''\r\nMay 26 03:10:19 sh[57854]: 2015-05-26 03:10:19 UTC ERROR:  relation \"auth_user\" does not exist at character 24\r\nMay 26 03:10:19 sh[57854]: 2015-05-26 03:10:19 UTC STATEMENT:  SELECT (1) AS \"a\" FROM \"auth_user\" WHERE \"auth_user\".\"username\" = 'foo'  LIMIT 1\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/105489066","html_url":"https://github.com/deis/deis/issues/3744#issuecomment-105489066","issue_url":"https://api.github.com/repos/deis/deis/issues/3744","id":105489066,"user":{"login":"PierreKircher","id":3399855,"avatar_url":"https://avatars.githubusercontent.com/u/3399855?v=3","gravatar_id":"","url":"https://api.github.com/users/PierreKircher","html_url":"https://github.com/PierreKircher","followers_url":"https://api.github.com/users/PierreKircher/followers","following_url":"https://api.github.com/users/PierreKircher/following{/other_user}","gists_url":"https://api.github.com/users/PierreKircher/gists{/gist_id}","starred_url":"https://api.github.com/users/PierreKircher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PierreKircher/subscriptions","organizations_url":"https://api.github.com/users/PierreKircher/orgs","repos_url":"https://api.github.com/users/PierreKircher/repos","events_url":"https://api.github.com/users/PierreKircher/events{/privacy}","received_events_url":"https://api.github.com/users/PierreKircher/received_events","type":"User","site_admin":false},"created_at":"2015-05-26T11:00:43Z","updated_at":"2015-05-26T11:00:43Z","body":"you dont want to log in the db .. that would make wal-e slower as it has to backup more .. deis needs go have a hot pg slave first for this to become viable\r\n\r\nseems like gateway has a problem \r\nnse into a montior and run ceph -w"}}