{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/2819","labels_url":"https://api.github.com/repos/deis/deis/issues/2819/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/2819/comments","events_url":"https://api.github.com/repos/deis/deis/issues/2819/events","html_url":"https://github.com/deis/deis/issues/2819","id":53089741,"number":2819,"title":"database: Better understand the window of loss (and loss scenarios) for database backups","user":{"login":"carmstrong","id":536009,"avatar_url":"https://avatars.githubusercontent.com/u/536009?v=3","gravatar_id":"","url":"https://api.github.com/users/carmstrong","html_url":"https://github.com/carmstrong","followers_url":"https://api.github.com/users/carmstrong/followers","following_url":"https://api.github.com/users/carmstrong/following{/other_user}","gists_url":"https://api.github.com/users/carmstrong/gists{/gist_id}","starred_url":"https://api.github.com/users/carmstrong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/carmstrong/subscriptions","organizations_url":"https://api.github.com/users/carmstrong/orgs","repos_url":"https://api.github.com/users/carmstrong/repos","events_url":"https://api.github.com/users/carmstrong/events{/privacy}","received_events_url":"https://api.github.com/users/carmstrong/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/deis/labels/database","name":"database","color":"bfd4f2"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2014-12-30T03:00:39Z","updated_at":"2015-04-02T05:36:36Z","closed_at":null,"body":"The database component uses [wal-e](https://github.com/wal-e/wal-e) to ship backups and WAL logs to the deis-store component. Full backups occur by default every 3 hours, and WAL logs are shipped continuously. When a database comes up fresh on a host, it first asks WAL-e if there are any backups to restore. If so, it restores the backup and then begins to replay the WAL logs that were written since the backup.\r\n\r\nThere is potentially a small window of loss here if the database component should exit before the last WAL log is written to disk. Additionally, users (including @lorieri) have occasionally reported loss of some data when upgrading Deis or otherwise performing a database restore. We test Deis platform upgrades for every Deis release, and this includes a database restore. However, the amount of data we're storing is not large. There is likely either a larger loss window than we think, or other causes of data loss. \r\n\r\nCeph doesn't seem to be the blame, as registry data (which is also stored in Ceph) is reported to be fully intact. This points to a combination of the database backup loop logic and wal-e itself. We should so some investigation to better attempt to replicate what users are sometimes seeing in real-world environments.\r\n\r\nSomewhat related is the fragility of the backup loop in the bin/boot script: #2741"},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/88743188","html_url":"https://github.com/deis/deis/issues/2819#issuecomment-88743188","issue_url":"https://api.github.com/repos/deis/deis/issues/2819","id":88743188,"user":{"login":"lorieri","id":1039009,"avatar_url":"https://avatars.githubusercontent.com/u/1039009?v=3","gravatar_id":"","url":"https://api.github.com/users/lorieri","html_url":"https://github.com/lorieri","followers_url":"https://api.github.com/users/lorieri/followers","following_url":"https://api.github.com/users/lorieri/following{/other_user}","gists_url":"https://api.github.com/users/lorieri/gists{/gist_id}","starred_url":"https://api.github.com/users/lorieri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lorieri/subscriptions","organizations_url":"https://api.github.com/users/lorieri/orgs","repos_url":"https://api.github.com/users/lorieri/repos","events_url":"https://api.github.com/users/lorieri/events{/privacy}","received_events_url":"https://api.github.com/users/lorieri/received_events","type":"User","site_admin":false},"created_at":"2015-04-02T05:36:36Z","updated_at":"2015-04-02T05:36:36Z","body":":)"}}