{"action":"created","comment":{"url":"https://api.github.com/repos/influxdb/influxdb/pulls/comments/45803696","id":45803696,"diff_hunk":"@@ -0,0 +1,298 @@\n+package tsm1\n+\n+// Compactions are the process of creating read-optimized TSM files.\n+// The files are created by converting write-optimized WAL entries\n+// to read-optimized TSM format.  They can also be created from existing\n+// TSM files when there are tombstone records that neeed to be removed, points\n+// that were overwritten by later writes and need to updated, or multiple\n+// smaller TSM files need to be merged to reduce file counts and improve\n+// compression ratios.\n+//\n+// The the compaction process is stream-oriented using multiple readers and\n+// iterators.  The resulting stream is written sorted and chunk to allow for","path":"tsdb/engine/tsm1/compact.go","position":12,"original_position":12,"commit_id":"1ce8d6290bd656840eccab01c39359e4b87f7654","original_commit_id":"1ce8d6290bd656840eccab01c39359e4b87f7654","user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"body":"chunk -> chunked (I think)","created_at":"2015-11-24T22:21:42Z","updated_at":"2015-11-24T22:21:42Z","html_url":"https://github.com/influxdb/influxdb/pull/4900#discussion_r45803696","pull_request_url":"https://api.github.com/repos/influxdb/influxdb/pulls/4900","_links":{"self":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/comments/45803696"},"html":{"href":"https://github.com/influxdb/influxdb/pull/4900#discussion_r45803696"},"pull_request":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/4900"}}},"pull_request":{"url":"https://api.github.com/repos/influxdb/influxdb/pulls/4900","id":51717505,"html_url":"https://github.com/influxdb/influxdb/pull/4900","diff_url":"https://github.com/influxdb/influxdb/pull/4900.diff","patch_url":"https://github.com/influxdb/influxdb/pull/4900.patch","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4900","number":4900,"state":"open","locked":false,"title":"WAL segment compaction","user":{"login":"jwilder","id":219935,"avatar_url":"https://avatars.githubusercontent.com/u/219935?v=3","gravatar_id":"","url":"https://api.github.com/users/jwilder","html_url":"https://github.com/jwilder","followers_url":"https://api.github.com/users/jwilder/followers","following_url":"https://api.github.com/users/jwilder/following{/other_user}","gists_url":"https://api.github.com/users/jwilder/gists{/gist_id}","starred_url":"https://api.github.com/users/jwilder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwilder/subscriptions","organizations_url":"https://api.github.com/users/jwilder/orgs","repos_url":"https://api.github.com/users/jwilder/repos","events_url":"https://api.github.com/users/jwilder/events{/privacy}","received_events_url":"https://api.github.com/users/jwilder/received_events","type":"User","site_admin":false},"body":"This PR adds the pieces to convert closed WAL segments into TSM files.  It has some preliminary wiring up into the `tsm1dev` engine to facilitate testing.  The resulting TSM files are not not queryable though, just written to disk.\r\n\r\nThe process of compaction is stream-oriented and takes a set of closed WAL segments (and eventually a set of TSM files needing to be re-written), and produces a stream of `key` and `[]Value`.  Each iteration returns the next sorted key if all values have been returned for that key.  The values are sorted and deduped across the inputs.  The resulting stream returns values are chunked such that they can be written to a TSM file as a new block.  The sorting of the stream allows for a single-pass write of the new file and a single pass read of existing files.  Max file size is handle during the process by rolling over to a new file.\r\n\r\nThere were a couple of problems I ran into that required re-working the WAL somewhat to make compactions run more consistently.  \r\n\r\nThe first issue was that the WAL write entries were just line protocol serialized points.  This is fast to write but reading them back added a lot of overhead because we had to re-parse the point, extract the fields, convert to a `Value` then sort and de-dup.  Instead, we now convert the point to a `Value` before writing to the WAL and the WAL stores a binary version (snappy compressed) of the columnar values.  Since we already parsed the point on write side before writing to the WAL, we avoid parsing it again on the read side.  We have also already extracted the `Fields` on the point on the write size so we use the cached version on the point which saves some more processing on the compaction side.\r\n\r\nThe second issue was allocations. These are still high and need to be improved further, but it stems from the fact that we create and throw away a lot of `Points` and 'Value' instances which creates a lot of garbage.  In some places, I use `sync.Pools` to limit re-used unused and allocated `Values`.  I'm still looking at other options for this as well.\r\n\r\nThe final issue was that WAL segments started to accumulate faster than compactions would re-claim them under high write-loads (e.g. 500+K pt/sec).  This was happening because the write path goroutines seem to starve the compaction goroutine because they are very CPU bound and don't have many opportunities to be switched by the scheduler.  To work around this, I've added an artificial throttle to the write path to allow the scheduler to let the compaction routine run more frequently.  I tried a number of things that should allow the goroutines to be balanced, but a small `Sleep` call seems alleviate the problem.  This will probably become more dynamic as the code evolves.","created_at":"2015-11-24T21:11:13Z","updated_at":"2015-11-24T22:21:42Z","closed_at":null,"merged_at":null,"merge_commit_sha":"53c23e9c239f3f4649b217119c2b716fe6c82981","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/influxdb/influxdb/pulls/4900/commits","review_comments_url":"https://api.github.com/repos/influxdb/influxdb/pulls/4900/comments","review_comment_url":"https://api.github.com/repos/influxdb/influxdb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4900/comments","statuses_url":"https://api.github.com/repos/influxdb/influxdb/statuses/1ce8d6290bd656840eccab01c39359e4b87f7654","head":{"label":"influxdb:jw-compact","ref":"jw-compact","sha":"1ce8d6290bd656840eccab01c39359e4b87f7654","user":{"login":"influxdb","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdb","html_url":"https://github.com/influxdb","followers_url":"https://api.github.com/users/influxdb/followers","following_url":"https://api.github.com/users/influxdb/following{/other_user}","gists_url":"https://api.github.com/users/influxdb/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdb/subscriptions","organizations_url":"https://api.github.com/users/influxdb/orgs","repos_url":"https://api.github.com/users/influxdb/repos","events_url":"https://api.github.com/users/influxdb/events{/privacy}","received_events_url":"https://api.github.com/users/influxdb/received_events","type":"Organization","site_admin":false},"repo":{"id":13124802,"name":"influxdb","full_name":"influxdb/influxdb","owner":{"login":"influxdb","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdb","html_url":"https://github.com/influxdb","followers_url":"https://api.github.com/users/influxdb/followers","following_url":"https://api.github.com/users/influxdb/following{/other_user}","gists_url":"https://api.github.com/users/influxdb/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdb/subscriptions","organizations_url":"https://api.github.com/users/influxdb/orgs","repos_url":"https://api.github.com/users/influxdb/repos","events_url":"https://api.github.com/users/influxdb/events{/privacy}","received_events_url":"https://api.github.com/users/influxdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/influxdb/influxdb","description":"Scalable datastore for metrics, events, and real-time analytics","fork":false,"url":"https://api.github.com/repos/influxdb/influxdb","forks_url":"https://api.github.com/repos/influxdb/influxdb/forks","keys_url":"https://api.github.com/repos/influxdb/influxdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/influxdb/influxdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/influxdb/influxdb/teams","hooks_url":"https://api.github.com/repos/influxdb/influxdb/hooks","issue_events_url":"https://api.github.com/repos/influxdb/influxdb/issues/events{/number}","events_url":"https://api.github.com/repos/influxdb/influxdb/events","assignees_url":"https://api.github.com/repos/influxdb/influxdb/assignees{/user}","branches_url":"https://api.github.com/repos/influxdb/influxdb/branches{/branch}","tags_url":"https://api.github.com/repos/influxdb/influxdb/tags","blobs_url":"https://api.github.com/repos/influxdb/influxdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/influxdb/influxdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/influxdb/influxdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/influxdb/influxdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/influxdb/influxdb/statuses/{sha}","languages_url":"https://api.github.com/repos/influxdb/influxdb/languages","stargazers_url":"https://api.github.com/repos/influxdb/influxdb/stargazers","contributors_url":"https://api.github.com/repos/influxdb/influxdb/contributors","subscribers_url":"https://api.github.com/repos/influxdb/influxdb/subscribers","subscription_url":"https://api.github.com/repos/influxdb/influxdb/subscription","commits_url":"https://api.github.com/repos/influxdb/influxdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/influxdb/influxdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/influxdb/influxdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/influxdb/influxdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/influxdb/influxdb/contents/{+path}","compare_url":"https://api.github.com/repos/influxdb/influxdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/influxdb/influxdb/merges","archive_url":"https://api.github.com/repos/influxdb/influxdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/influxdb/influxdb/downloads","issues_url":"https://api.github.com/repos/influxdb/influxdb/issues{/number}","pulls_url":"https://api.github.com/repos/influxdb/influxdb/pulls{/number}","milestones_url":"https://api.github.com/repos/influxdb/influxdb/milestones{/number}","notifications_url":"https://api.github.com/repos/influxdb/influxdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/influxdb/influxdb/labels{/name}","releases_url":"https://api.github.com/repos/influxdb/influxdb/releases{/id}","created_at":"2013-09-26T14:31:10Z","updated_at":"2015-11-24T17:49:18Z","pushed_at":"2015-11-24T21:11:14Z","git_url":"git://github.com/influxdb/influxdb.git","ssh_url":"git@github.com:influxdb/influxdb.git","clone_url":"https://github.com/influxdb/influxdb.git","svn_url":"https://github.com/influxdb/influxdb","homepage":"http://influxdb.com","size":38473,"stargazers_count":6797,"watchers_count":6797,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":814,"mirror_url":null,"open_issues_count":479,"forks":814,"open_issues":479,"watchers":6797,"default_branch":"master"}},"base":{"label":"influxdb:master","ref":"master","sha":"ed393398cd4e317be01cdb71764a83cea24b0a38","user":{"login":"influxdb","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdb","html_url":"https://github.com/influxdb","followers_url":"https://api.github.com/users/influxdb/followers","following_url":"https://api.github.com/users/influxdb/following{/other_user}","gists_url":"https://api.github.com/users/influxdb/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdb/subscriptions","organizations_url":"https://api.github.com/users/influxdb/orgs","repos_url":"https://api.github.com/users/influxdb/repos","events_url":"https://api.github.com/users/influxdb/events{/privacy}","received_events_url":"https://api.github.com/users/influxdb/received_events","type":"Organization","site_admin":false},"repo":{"id":13124802,"name":"influxdb","full_name":"influxdb/influxdb","owner":{"login":"influxdb","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdb","html_url":"https://github.com/influxdb","followers_url":"https://api.github.com/users/influxdb/followers","following_url":"https://api.github.com/users/influxdb/following{/other_user}","gists_url":"https://api.github.com/users/influxdb/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdb/subscriptions","organizations_url":"https://api.github.com/users/influxdb/orgs","repos_url":"https://api.github.com/users/influxdb/repos","events_url":"https://api.github.com/users/influxdb/events{/privacy}","received_events_url":"https://api.github.com/users/influxdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/influxdb/influxdb","description":"Scalable datastore for metrics, events, and real-time analytics","fork":false,"url":"https://api.github.com/repos/influxdb/influxdb","forks_url":"https://api.github.com/repos/influxdb/influxdb/forks","keys_url":"https://api.github.com/repos/influxdb/influxdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/influxdb/influxdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/influxdb/influxdb/teams","hooks_url":"https://api.github.com/repos/influxdb/influxdb/hooks","issue_events_url":"https://api.github.com/repos/influxdb/influxdb/issues/events{/number}","events_url":"https://api.github.com/repos/influxdb/influxdb/events","assignees_url":"https://api.github.com/repos/influxdb/influxdb/assignees{/user}","branches_url":"https://api.github.com/repos/influxdb/influxdb/branches{/branch}","tags_url":"https://api.github.com/repos/influxdb/influxdb/tags","blobs_url":"https://api.github.com/repos/influxdb/influxdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/influxdb/influxdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/influxdb/influxdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/influxdb/influxdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/influxdb/influxdb/statuses/{sha}","languages_url":"https://api.github.com/repos/influxdb/influxdb/languages","stargazers_url":"https://api.github.com/repos/influxdb/influxdb/stargazers","contributors_url":"https://api.github.com/repos/influxdb/influxdb/contributors","subscribers_url":"https://api.github.com/repos/influxdb/influxdb/subscribers","subscription_url":"https://api.github.com/repos/influxdb/influxdb/subscription","commits_url":"https://api.github.com/repos/influxdb/influxdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/influxdb/influxdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/influxdb/influxdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/influxdb/influxdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/influxdb/influxdb/contents/{+path}","compare_url":"https://api.github.com/repos/influxdb/influxdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/influxdb/influxdb/merges","archive_url":"https://api.github.com/repos/influxdb/influxdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/influxdb/influxdb/downloads","issues_url":"https://api.github.com/repos/influxdb/influxdb/issues{/number}","pulls_url":"https://api.github.com/repos/influxdb/influxdb/pulls{/number}","milestones_url":"https://api.github.com/repos/influxdb/influxdb/milestones{/number}","notifications_url":"https://api.github.com/repos/influxdb/influxdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/influxdb/influxdb/labels{/name}","releases_url":"https://api.github.com/repos/influxdb/influxdb/releases{/id}","created_at":"2013-09-26T14:31:10Z","updated_at":"2015-11-24T17:49:18Z","pushed_at":"2015-11-24T21:11:14Z","git_url":"git://github.com/influxdb/influxdb.git","ssh_url":"git@github.com:influxdb/influxdb.git","clone_url":"https://github.com/influxdb/influxdb.git","svn_url":"https://github.com/influxdb/influxdb","homepage":"http://influxdb.com","size":38473,"stargazers_count":6797,"watchers_count":6797,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":814,"mirror_url":null,"open_issues_count":479,"forks":814,"open_issues":479,"watchers":6797,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/4900"},"html":{"href":"https://github.com/influxdb/influxdb/pull/4900"},"issue":{"href":"https://api.github.com/repos/influxdb/influxdb/issues/4900"},"comments":{"href":"https://api.github.com/repos/influxdb/influxdb/issues/4900/comments"},"review_comments":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/4900/comments"},"review_comment":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/4900/commits"},"statuses":{"href":"https://api.github.com/repos/influxdb/influxdb/statuses/1ce8d6290bd656840eccab01c39359e4b87f7654"}}}}