{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/5226","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/5226/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/5226/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/5226/events","html_url":"https://github.com/influxdb/influxdb/pull/5226","id":123951222,"number":5226,"title":"b*1 to TSM converter","user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/2%20-%20Working","name":"2 - Working","color":"CCCCCC"}],"state":"open","locked":false,"assignee":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/influxdb/influxdb/milestones/45","html_url":"https://github.com/influxdb/influxdb/milestones/0.10.0","labels_url":"https://api.github.com/repos/influxdb/influxdb/milestones/45/labels","id":1311499,"number":45,"title":"0.10.0","description":"","creator":{"login":"beckettsean","id":6537329,"avatar_url":"https://avatars.githubusercontent.com/u/6537329?v=3","gravatar_id":"","url":"https://api.github.com/users/beckettsean","html_url":"https://github.com/beckettsean","followers_url":"https://api.github.com/users/beckettsean/followers","following_url":"https://api.github.com/users/beckettsean/following{/other_user}","gists_url":"https://api.github.com/users/beckettsean/gists{/gist_id}","starred_url":"https://api.github.com/users/beckettsean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beckettsean/subscriptions","organizations_url":"https://api.github.com/users/beckettsean/orgs","repos_url":"https://api.github.com/users/beckettsean/repos","events_url":"https://api.github.com/users/beckettsean/events{/privacy}","received_events_url":"https://api.github.com/users/beckettsean/received_events","type":"User","site_admin":false},"open_issues":40,"closed_issues":8,"state":"open","created_at":"2015-09-17T21:14:49Z","updated_at":"2015-12-28T16:34:38Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-12-26T22:15:38Z","updated_at":"2015-12-29T18:54:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/influxdb/influxdb/pulls/5226","html_url":"https://github.com/influxdb/influxdb/pull/5226","diff_url":"https://github.com/influxdb/influxdb/pull/5226.diff","patch_url":"https://github.com/influxdb/influxdb/pull/5226.patch"},"body":"Started with https://github.com/influxdb/influxdb/pull/5159\r\n\r\nThis tool converts, on a database-by-database basis, b1 and bz1 shards to tsm format. The main components include:\r\n- bz1 reader\r\n- b1 reader\r\n- conversion logic, which drains each of these readers and writes to 1 or more TSM files (possibly multiple files per TSM shard) for each reader.\r\n\r\nEach reader exposes an iterator such that the emitted chunks are suitable for feeding directly into a `TSMWriter` -- the series keys are a combination of the measurement, tag-set, and field. Series keys are arranged alphabetically, and timestamps of all points are increasing, as required by TSMWriters. \r\n\r\nA database is backed-up before any changes are made to it.\r\n\r\nMuch of this work involved copying existing b1 and bz1 code, so as to avoid conflicts with changes to that code (which will going away soon). There is also much duplication between the b1 and bz readers. This could be factored out if necessary, but simplicity and correctness seems key for this tool.\r\n\r\n__Remaining__\r\n\r\n- [x] allow parallel conversion.\r\n- [ ] large scale testing.\r\n- [x] Full instructions in README.md.\r\n- [ ] Testing notes.\r\n\r\nExample output:\r\n\r\n```\r\n$ influx_tsm ~/.influxdb/data/\r\n\r\nb1 and bz1 shard conversion.\r\n-----------------------------------\r\nData directory is: /home/philip/.influxdb/data/\r\nDatabases specified: all\r\nParallel mode enabled: no\r\nDatabase backups enabled: yes\r\n1 shard(s) detected, 1 remain after filtering.\r\n\r\nDatabase        Retention       Path                                            Engine  Size\r\n_internal       monitor         /home/philip/.influxdb/data/_internal/monitor/1 bz1     262144\r\n\r\nThese shards will be converted. Proceed? y/N: y\r\nConversion starting....\r\nDatabase _internal backed up.\r\nConversion of /home/philip/.influxdb/data/_internal/monitor/1 successful (23.453011ms)\r\n\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/167852814","html_url":"https://github.com/influxdb/influxdb/pull/5226#issuecomment-167852814","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/5226","id":167852814,"user":{"login":"pauldix","id":4331,"avatar_url":"https://avatars.githubusercontent.com/u/4331?v=3","gravatar_id":"","url":"https://api.github.com/users/pauldix","html_url":"https://github.com/pauldix","followers_url":"https://api.github.com/users/pauldix/followers","following_url":"https://api.github.com/users/pauldix/following{/other_user}","gists_url":"https://api.github.com/users/pauldix/gists{/gist_id}","starred_url":"https://api.github.com/users/pauldix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pauldix/subscriptions","organizations_url":"https://api.github.com/users/pauldix/orgs","repos_url":"https://api.github.com/users/pauldix/repos","events_url":"https://api.github.com/users/pauldix/events{/privacy}","received_events_url":"https://api.github.com/users/pauldix/received_events","type":"User","site_admin":false},"created_at":"2015-12-29T18:54:59Z","updated_at":"2015-12-29T18:54:59Z","body":"LGTM, let the testing begin!!"}}