{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4977","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4977/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4977/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4977/events","html_url":"https://github.com/influxdb/influxdb/issues/4977","id":120259193,"number":4977,"title":"TSM list for feature complete","user":{"login":"pauldix","id":4331,"avatar_url":"https://avatars.githubusercontent.com/u/4331?v=3","gravatar_id":"","url":"https://api.github.com/users/pauldix","html_url":"https://github.com/pauldix","followers_url":"https://api.github.com/users/pauldix/followers","following_url":"https://api.github.com/users/pauldix/following{/other_user}","gists_url":"https://api.github.com/users/pauldix/gists{/gist_id}","starred_url":"https://api.github.com/users/pauldix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pauldix/subscriptions","organizations_url":"https://api.github.com/users/pauldix/orgs","repos_url":"https://api.github.com/users/pauldix/repos","events_url":"https://api.github.com/users/pauldix/events{/privacy}","received_events_url":"https://api.github.com/users/pauldix/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/tsm","name":"tsm","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"pauldix","id":4331,"avatar_url":"https://avatars.githubusercontent.com/u/4331?v=3","gravatar_id":"","url":"https://api.github.com/users/pauldix","html_url":"https://github.com/pauldix","followers_url":"https://api.github.com/users/pauldix/followers","following_url":"https://api.github.com/users/pauldix/following{/other_user}","gists_url":"https://api.github.com/users/pauldix/gists{/gist_id}","starred_url":"https://api.github.com/users/pauldix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pauldix/subscriptions","organizations_url":"https://api.github.com/users/pauldix/orgs","repos_url":"https://api.github.com/users/pauldix/repos","events_url":"https://api.github.com/users/pauldix/events{/privacy}","received_events_url":"https://api.github.com/users/pauldix/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/influxdb/influxdb/milestones/45","html_url":"https://github.com/influxdb/influxdb/milestones/0.10.0","labels_url":"https://api.github.com/repos/influxdb/influxdb/milestones/45/labels","id":1311499,"number":45,"title":"0.10.0","description":"","creator":{"login":"beckettsean","id":6537329,"avatar_url":"https://avatars.githubusercontent.com/u/6537329?v=3","gravatar_id":"","url":"https://api.github.com/users/beckettsean","html_url":"https://github.com/beckettsean","followers_url":"https://api.github.com/users/beckettsean/followers","following_url":"https://api.github.com/users/beckettsean/following{/other_user}","gists_url":"https://api.github.com/users/beckettsean/gists{/gist_id}","starred_url":"https://api.github.com/users/beckettsean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beckettsean/subscriptions","organizations_url":"https://api.github.com/users/beckettsean/orgs","repos_url":"https://api.github.com/users/beckettsean/repos","events_url":"https://api.github.com/users/beckettsean/events{/privacy}","received_events_url":"https://api.github.com/users/beckettsean/received_events","type":"User","site_admin":false},"open_issues":40,"closed_issues":9,"state":"open","created_at":"2015-09-17T21:14:49Z","updated_at":"2015-12-29T19:01:28Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2015-12-03T20:11:17Z","updated_at":"2015-12-29T20:46:34Z","closed_at":null,"body":"This is a list of items to be completed on TSM after the December 8th release to make it feature complete. There will still be more testing, but once this is done we'll only be testing.\r\n\r\n### Features\r\n- [x] Deduplicate points with same timestamp that exist in multiple TSM files (@jwilder)\r\n- [x] Fix compaction planning so that it works past the first compaction step size (@pauldix)\r\n- [x] Update compaction to run multiple levels at the same time (@jwilder)\r\n- [x] Update compaction planning logic to include tombstones (@jwilder)\r\n- [ ] Backup/restore (@pauldix)\r\n- [ ] Migrate b1 and bz1 shards to TSM files (@otoolep)\r\n- [ ] Stats and diagnostics (@otoolep)\r\n- [ ] Different logging options: snapshots, WAL, compactions, filestore, etc (including turning logging off) (@otoolep)\r\n- [ ] Queries against files that have just been compacted need to point to new files (@jwilder)\r\n- [ ] Add checksum verification to TSM inspect tool\r\n- [ ] Modify tombstone to write multiple immutable files instead of updating a single tombstone file \r\n\r\n### Testing\r\n- [x] Crash testing\r\n- [ ] Race testing\r\n- [ ] Many TSM files (1M series with 2k TSM files)\r\n- [ ] Startup testing (how long does it take to startup and answer queries with large database)\r\n- [ ] Performance comparison with TSM1\r\n- [ ] Write 1B points across 500k series (throughput)\r\n- [ ] Write while querying (check if throughput drops)\r\n- [ ] Time to count query a specific series after writing\r\n- [ ] Time to count query all series after writing\r\n- [ ] Performance optimizations\r\n- [ ] Drops/deletes during writes and queries\r\n- [ ] Writes and queries while closing\r\n- [ ] Test 10M, 100M series\r\n- [ ] Test 10B, 100B, 1T points\r\n- [ ] Startup testing with above. How long until we can write? How long until we can query?\r\n- [ ] Performance comparisons with b1 and bz1\r\n\r\ncc @jwilder @otoolep "},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/167874841","html_url":"https://github.com/influxdb/influxdb/issues/4977#issuecomment-167874841","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4977","id":167874841,"user":{"login":"daviesalex","id":1106249,"avatar_url":"https://avatars.githubusercontent.com/u/1106249?v=3","gravatar_id":"","url":"https://api.github.com/users/daviesalex","html_url":"https://github.com/daviesalex","followers_url":"https://api.github.com/users/daviesalex/followers","following_url":"https://api.github.com/users/daviesalex/following{/other_user}","gists_url":"https://api.github.com/users/daviesalex/gists{/gist_id}","starred_url":"https://api.github.com/users/daviesalex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daviesalex/subscriptions","organizations_url":"https://api.github.com/users/daviesalex/orgs","repos_url":"https://api.github.com/users/daviesalex/repos","events_url":"https://api.github.com/users/daviesalex/events{/privacy}","received_events_url":"https://api.github.com/users/daviesalex/received_events","type":"User","site_admin":false},"created_at":"2015-12-29T20:46:34Z","updated_at":"2015-12-29T20:46:34Z","body":"@pauldix, we cant reproduce running the same query (we can get a few GB of RAM to be used, but not anything like 90G). However, happy to wait for that PR to merge and report if we see this again. \r\n\r\nI guess the full load test (with queries) will need to wait for that PR to land to avoid chasing bugs that are already fixed in that branch... we will watch and test as soon as its ready."}}