{"action":"created","comment":{"url":"https://api.github.com/repos/influxdb/influxdb/pulls/comments/28380222","id":28380222,"diff_hunk":"@@ -818,6 +878,80 @@ func (t *Topic) WriteMessage(m *Message) error {\n \treturn nil\n }\n \n+// Truncate attempts to delete topic segments such that the total size of the topic on-disk\n+// is equal to or less-than maxSize. Returns the number of bytes deleted, and error if any.\n+// This function is not guaranteed to be performant.\n+func (t *Topic) Truncate(maxSize int64) (nBytesDeleted int64, err error) {\n+\tt.mu.Lock()\n+\tdefer t.mu.Unlock()\n+\n+\tvar segments Segments\n+\tsegments, err = ReadSegments(t.Path())\n+\tif err != nil {\n+\t\treturn\n+\t}\n+\n+\tvar totalSize int64\n+\ttotalSize, err = segments.Size()\n+\tif err != nil {\n+\t\treturn\n+\t}\n+\n+\t// Find the highest-replicated index, this is a condition for deletion of segments\n+\t// within this topic.\n+\tvar highestIndex uint64\n+\tfor _, idx := range t.indexByURL {\n+\t\tif idx > highestIndex {\n+\t\t\thighestIndex = idx\n+\t\t}\n+\t}\n+\n+\tvar size int64\n+\tfor {\n+\t\tif (totalSize - nBytesDeleted) <= maxSize {\n+\t\t\tbreak\n+\t\t}\n+\n+\t\t// More than 2 segments are needed for current writes and replication checks.\n+\t\tif len(segments) <= 2 {\n+\t\t\tbreak\n+\t\t}\n+\n+\t\t// Attempt deletion of oldest segment in topic. First and second segment needed\n+\t\t// for this operation.\n+\t\tfirst := segments[0]\n+\t\tsecond := segments[1]\n+\n+\t\t// The first segment can only be deleted if the last index in that segment has\n+\t\t// been replicated. The most efficient way to check this is to ensure that the\n+\t\t// first index of the subsequent segment has been replicated.\n+\t\tif second.Index > highestIndex {\n+\t\t\t// No guarantee first segment has been replicated.\n+\t\t\tbreak\n+\t\t}\n+\n+\t\t// Deletion can proceed!\n+\t\tsize, err = first.Size()\n+\t\tif err != nil {\n+\t\t\tbreak\n+\t\t}\n+\t\tif err = os.Remove(first.Path); err != nil {\n+\t\t\tbreak\n+\t\t}\n+\t\tnBytesDeleted += size\n+\t\tsegments = segments[1:]\n+\n+\t\t// Create tombstone to indicate that the topic has been truncated at least once.\n+\t\tf, err := os.Create(t.tombstone)","path":"messaging/broker.go","position":258,"original_position":258,"commit_id":"f59115069928e71006b5f83d4ebfc6772882f673","original_commit_id":"f59115069928e71006b5f83d4ebfc6772882f673","user":{"login":"benbjohnson","id":118015,"avatar_url":"https://avatars.githubusercontent.com/u/118015?v=3","gravatar_id":"","url":"https://api.github.com/users/benbjohnson","html_url":"https://github.com/benbjohnson","followers_url":"https://api.github.com/users/benbjohnson/followers","following_url":"https://api.github.com/users/benbjohnson/following{/other_user}","gists_url":"https://api.github.com/users/benbjohnson/gists{/gist_id}","starred_url":"https://api.github.com/users/benbjohnson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benbjohnson/subscriptions","organizations_url":"https://api.github.com/users/benbjohnson/orgs","repos_url":"https://api.github.com/users/benbjohnson/repos","events_url":"https://api.github.com/users/benbjohnson/events{/privacy}","received_events_url":"https://api.github.com/users/benbjohnson/received_events","type":"User","site_admin":false},"body":"Is the return variable `err` variable being shadowed here? ","created_at":"2015-04-14T23:07:29Z","updated_at":"2015-04-14T23:07:29Z","html_url":"https://github.com/influxdb/influxdb/pull/2276#discussion_r28380222","pull_request_url":"https://api.github.com/repos/influxdb/influxdb/pulls/2276","_links":{"self":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/comments/28380222"},"html":{"href":"https://github.com/influxdb/influxdb/pull/2276#discussion_r28380222"},"pull_request":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/2276"}}},"pull_request":{"url":"https://api.github.com/repos/influxdb/influxdb/pulls/2276","id":33220506,"html_url":"https://github.com/influxdb/influxdb/pull/2276","diff_url":"https://github.com/influxdb/influxdb/pull/2276.diff","patch_url":"https://github.com/influxdb/influxdb/pull/2276.patch","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/2276","number":2276,"state":"open","locked":false,"title":"Broker truncation","user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"body":"This change implements basic topic truncation on Brokers. It does this by periodically checking each topic for its size, and if greater than a certain threshold and the topic data has been replicated, deleting segments until the required size is reached. \r\n\r\nThe change also exposes maximum topic and segment sizes via configuration, as I believe some people may want to change these numbers.\r\n\r\nI decided to implement this via a go-routine, so that the function to check the topic sizes could simply walk the filesystem, and not have to worry too much about efficiency vis a vis caching sizes in RAM. I think a design whereby truncation is performed every few minutes is fine (it's how Apache Kafka works), and this approach seems simpler and more robust.\r\n\r\nRemaining:\r\n- letting data nodes know when they must redirect to other nodes for topic data.\r\n\r\n","created_at":"2015-04-14T02:01:25Z","updated_at":"2015-04-14T23:07:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"403ed8811b40c59b1ab6357b502c3b182c67eeda","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/influxdb/influxdb/pulls/2276/commits","review_comments_url":"https://api.github.com/repos/influxdb/influxdb/pulls/2276/comments","review_comment_url":"https://api.github.com/repos/influxdb/influxdb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/2276/comments","statuses_url":"https://api.github.com/repos/influxdb/influxdb/statuses/f59115069928e71006b5f83d4ebfc6772882f673","head":{"label":"influxdb:broker_truncation","ref":"broker_truncation","sha":"f59115069928e71006b5f83d4ebfc6772882f673","user":{"login":"influxdb","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdb","html_url":"https://github.com/influxdb","followers_url":"https://api.github.com/users/influxdb/followers","following_url":"https://api.github.com/users/influxdb/following{/other_user}","gists_url":"https://api.github.com/users/influxdb/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdb/subscriptions","organizations_url":"https://api.github.com/users/influxdb/orgs","repos_url":"https://api.github.com/users/influxdb/repos","events_url":"https://api.github.com/users/influxdb/events{/privacy}","received_events_url":"https://api.github.com/users/influxdb/received_events","type":"Organization","site_admin":false},"repo":{"id":13124802,"name":"influxdb","full_name":"influxdb/influxdb","owner":{"login":"influxdb","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdb","html_url":"https://github.com/influxdb","followers_url":"https://api.github.com/users/influxdb/followers","following_url":"https://api.github.com/users/influxdb/following{/other_user}","gists_url":"https://api.github.com/users/influxdb/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdb/subscriptions","organizations_url":"https://api.github.com/users/influxdb/orgs","repos_url":"https://api.github.com/users/influxdb/repos","events_url":"https://api.github.com/users/influxdb/events{/privacy}","received_events_url":"https://api.github.com/users/influxdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/influxdb/influxdb","description":"Scalable datastore for metrics, events, and real-time analytics","fork":false,"url":"https://api.github.com/repos/influxdb/influxdb","forks_url":"https://api.github.com/repos/influxdb/influxdb/forks","keys_url":"https://api.github.com/repos/influxdb/influxdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/influxdb/influxdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/influxdb/influxdb/teams","hooks_url":"https://api.github.com/repos/influxdb/influxdb/hooks","issue_events_url":"https://api.github.com/repos/influxdb/influxdb/issues/events{/number}","events_url":"https://api.github.com/repos/influxdb/influxdb/events","assignees_url":"https://api.github.com/repos/influxdb/influxdb/assignees{/user}","branches_url":"https://api.github.com/repos/influxdb/influxdb/branches{/branch}","tags_url":"https://api.github.com/repos/influxdb/influxdb/tags","blobs_url":"https://api.github.com/repos/influxdb/influxdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/influxdb/influxdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/influxdb/influxdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/influxdb/influxdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/influxdb/influxdb/statuses/{sha}","languages_url":"https://api.github.com/repos/influxdb/influxdb/languages","stargazers_url":"https://api.github.com/repos/influxdb/influxdb/stargazers","contributors_url":"https://api.github.com/repos/influxdb/influxdb/contributors","subscribers_url":"https://api.github.com/repos/influxdb/influxdb/subscribers","subscription_url":"https://api.github.com/repos/influxdb/influxdb/subscription","commits_url":"https://api.github.com/repos/influxdb/influxdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/influxdb/influxdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/influxdb/influxdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/influxdb/influxdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/influxdb/influxdb/contents/{+path}","compare_url":"https://api.github.com/repos/influxdb/influxdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/influxdb/influxdb/merges","archive_url":"https://api.github.com/repos/influxdb/influxdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/influxdb/influxdb/downloads","issues_url":"https://api.github.com/repos/influxdb/influxdb/issues{/number}","pulls_url":"https://api.github.com/repos/influxdb/influxdb/pulls{/number}","milestones_url":"https://api.github.com/repos/influxdb/influxdb/milestones{/number}","notifications_url":"https://api.github.com/repos/influxdb/influxdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/influxdb/influxdb/labels{/name}","releases_url":"https://api.github.com/repos/influxdb/influxdb/releases{/id}","created_at":"2013-09-26T14:31:10Z","updated_at":"2015-04-14T20:44:41Z","pushed_at":"2015-04-14T23:02:31Z","git_url":"git://github.com/influxdb/influxdb.git","ssh_url":"git@github.com:influxdb/influxdb.git","clone_url":"https://github.com/influxdb/influxdb.git","svn_url":"https://github.com/influxdb/influxdb","homepage":"http://influxdb.com","size":47738,"stargazers_count":5058,"watchers_count":5058,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":454,"mirror_url":null,"open_issues_count":366,"forks":454,"open_issues":366,"watchers":5058,"default_branch":"master"}},"base":{"label":"influxdb:master","ref":"master","sha":"03d5ffc0bb428c5fe446b4f33fd8767fcd2ba6d7","user":{"login":"influxdb","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdb","html_url":"https://github.com/influxdb","followers_url":"https://api.github.com/users/influxdb/followers","following_url":"https://api.github.com/users/influxdb/following{/other_user}","gists_url":"https://api.github.com/users/influxdb/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdb/subscriptions","organizations_url":"https://api.github.com/users/influxdb/orgs","repos_url":"https://api.github.com/users/influxdb/repos","events_url":"https://api.github.com/users/influxdb/events{/privacy}","received_events_url":"https://api.github.com/users/influxdb/received_events","type":"Organization","site_admin":false},"repo":{"id":13124802,"name":"influxdb","full_name":"influxdb/influxdb","owner":{"login":"influxdb","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdb","html_url":"https://github.com/influxdb","followers_url":"https://api.github.com/users/influxdb/followers","following_url":"https://api.github.com/users/influxdb/following{/other_user}","gists_url":"https://api.github.com/users/influxdb/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdb/subscriptions","organizations_url":"https://api.github.com/users/influxdb/orgs","repos_url":"https://api.github.com/users/influxdb/repos","events_url":"https://api.github.com/users/influxdb/events{/privacy}","received_events_url":"https://api.github.com/users/influxdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/influxdb/influxdb","description":"Scalable datastore for metrics, events, and real-time analytics","fork":false,"url":"https://api.github.com/repos/influxdb/influxdb","forks_url":"https://api.github.com/repos/influxdb/influxdb/forks","keys_url":"https://api.github.com/repos/influxdb/influxdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/influxdb/influxdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/influxdb/influxdb/teams","hooks_url":"https://api.github.com/repos/influxdb/influxdb/hooks","issue_events_url":"https://api.github.com/repos/influxdb/influxdb/issues/events{/number}","events_url":"https://api.github.com/repos/influxdb/influxdb/events","assignees_url":"https://api.github.com/repos/influxdb/influxdb/assignees{/user}","branches_url":"https://api.github.com/repos/influxdb/influxdb/branches{/branch}","tags_url":"https://api.github.com/repos/influxdb/influxdb/tags","blobs_url":"https://api.github.com/repos/influxdb/influxdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/influxdb/influxdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/influxdb/influxdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/influxdb/influxdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/influxdb/influxdb/statuses/{sha}","languages_url":"https://api.github.com/repos/influxdb/influxdb/languages","stargazers_url":"https://api.github.com/repos/influxdb/influxdb/stargazers","contributors_url":"https://api.github.com/repos/influxdb/influxdb/contributors","subscribers_url":"https://api.github.com/repos/influxdb/influxdb/subscribers","subscription_url":"https://api.github.com/repos/influxdb/influxdb/subscription","commits_url":"https://api.github.com/repos/influxdb/influxdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/influxdb/influxdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/influxdb/influxdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/influxdb/influxdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/influxdb/influxdb/contents/{+path}","compare_url":"https://api.github.com/repos/influxdb/influxdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/influxdb/influxdb/merges","archive_url":"https://api.github.com/repos/influxdb/influxdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/influxdb/influxdb/downloads","issues_url":"https://api.github.com/repos/influxdb/influxdb/issues{/number}","pulls_url":"https://api.github.com/repos/influxdb/influxdb/pulls{/number}","milestones_url":"https://api.github.com/repos/influxdb/influxdb/milestones{/number}","notifications_url":"https://api.github.com/repos/influxdb/influxdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/influxdb/influxdb/labels{/name}","releases_url":"https://api.github.com/repos/influxdb/influxdb/releases{/id}","created_at":"2013-09-26T14:31:10Z","updated_at":"2015-04-14T20:44:41Z","pushed_at":"2015-04-14T23:02:31Z","git_url":"git://github.com/influxdb/influxdb.git","ssh_url":"git@github.com:influxdb/influxdb.git","clone_url":"https://github.com/influxdb/influxdb.git","svn_url":"https://github.com/influxdb/influxdb","homepage":"http://influxdb.com","size":47738,"stargazers_count":5058,"watchers_count":5058,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":454,"mirror_url":null,"open_issues_count":366,"forks":454,"open_issues":366,"watchers":5058,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/2276"},"html":{"href":"https://github.com/influxdb/influxdb/pull/2276"},"issue":{"href":"https://api.github.com/repos/influxdb/influxdb/issues/2276"},"comments":{"href":"https://api.github.com/repos/influxdb/influxdb/issues/2276/comments"},"review_comments":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/2276/comments"},"review_comment":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/influxdb/influxdb/pulls/2276/commits"},"statuses":{"href":"https://api.github.com/repos/influxdb/influxdb/statuses/f59115069928e71006b5f83d4ebfc6772882f673"}}}}