{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/651","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/651/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/651/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/651/events","html_url":"https://github.com/prometheus/prometheus/issues/651","id":70643495,"number":651,"title":"Consistent backups of a running Prometheus should be possible","user":{"login":"matthiasr","id":131590,"avatar_url":"https://avatars.githubusercontent.com/u/131590?v=3","gravatar_id":"","url":"https://api.github.com/users/matthiasr","html_url":"https://github.com/matthiasr","followers_url":"https://api.github.com/users/matthiasr/followers","following_url":"https://api.github.com/users/matthiasr/following{/other_user}","gists_url":"https://api.github.com/users/matthiasr/gists{/gist_id}","starred_url":"https://api.github.com/users/matthiasr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthiasr/subscriptions","organizations_url":"https://api.github.com/users/matthiasr/orgs","repos_url":"https://api.github.com/users/matthiasr/repos","events_url":"https://api.github.com/users/matthiasr/events{/privacy}","received_events_url":"https://api.github.com/users/matthiasr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-04-24T09:47:47Z","updated_at":"2015-04-26T10:34:32Z","closed_at":null,"body":"I experimented a bit with copying the storage out from underneath a running Prometheus.\r\n\r\nIn the zeroth run, I tried rsyncing it, but this takes forever because there are so many files. Using tar is much much faster, and since many files change anyway, rsync won't gain you much even on subsequent backups.\r\n\r\nIn the next run, I copied the whole storage with tar (in this case, directly into the right place on a new server). After that was finished, @beorn7 encouraged me to take another copy of the `heads.db` to get a fresher snapshot. This snapshot had happened after the copying of the timelines. This is the result:\r\n![copying Prometheus storage, attempt 1](https://cloud.githubusercontent.com/assets/131590/7316348/c22d22c6-ea65-11e4-94ac-8e4cd2495c50.png). The query is a count of a subset of the timeseries. Presumably, a few chunks were persisted between copying their timeseries and copying `heads.db` so that they were contained in neither on the backup.\r\n\r\nIn a third run, trying to be clever, I used `tar c storage storage/heads.db` so that there is a new copy of heads.db at the end of the tar stream. However, this snapshot is then older than many of the timeseries, and recovery had not finished after >16h and I aborted it.\r\n\r\nIn the fourth run, I did the same but also touch(1)ed the heads.db to shortcircuit the recovery. The result was not so good: \r\n![copying Prometheus storage, attempt 2](https://cloud.githubusercontent.com/assets/131590/7316395/20eb290c-ea66-11e4-829f-b72b044506ef.png).\r\n\r\nAs discussed with @beorn7, to support consistent binary backups Prometheus needs to support 3 actions:\r\n\r\n* stop persistence (with a timeout)\r\n* create a snapshot now (only returns when snapshot is complete)\r\n* start persistence\r\n\r\nThese can be API endpoints. With these, a backup procedure would be:\r\n\r\n* stop persistence\r\n* copy storage\r\n* perform snapshot (this includes all chunks since stopping persistence)\r\n* start copying heads.db\r\n* start persistence\r\n\r\nOptionally, Prometheus might offer an endpoint to \"just\" download a tarball, where it performs these steps behind the scenes.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/96361839","html_url":"https://github.com/prometheus/prometheus/issues/651#issuecomment-96361839","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/651","id":96361839,"user":{"login":"beorn7","id":5609886,"avatar_url":"https://avatars.githubusercontent.com/u/5609886?v=3","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false},"created_at":"2015-04-26T10:34:32Z","updated_at":"2015-04-26T10:34:32Z","body":"After some discussion and thinking, I like most the approach to have an endpoint that returns a giant tarball with a consistent set of series files and the heads.db.\r\n\r\nIn different news, I'm curious to reproduce the state where recovery takes forever. On SSD, I expect the recovery to be reasonably fast always (albeit with possible significant data loss). A case where it takes very long would be very interesting to investigate."}}