{"action":"closed","number":1133,"pull_request":{"url":"https://api.github.com/repos/basho/riak_cs/pulls/1133","id":34227997,"html_url":"https://github.com/basho/riak_cs/pull/1133","diff_url":"https://github.com/basho/riak_cs/pull/1133.diff","patch_url":"https://github.com/basho/riak_cs/pull/1133.patch","issue_url":"https://api.github.com/repos/basho/riak_cs/issues/1133","number":1133,"state":"closed","locked":false,"title":"Orphan block scanner scripts [2.1]","user":{"login":"shino","id":10225,"avatar_url":"https://avatars.githubusercontent.com/u/10225?v=3","gravatar_id":"","url":"https://api.github.com/users/shino","html_url":"https://github.com/shino","followers_url":"https://api.github.com/users/shino/followers","following_url":"https://api.github.com/users/shino/following{/other_user}","gists_url":"https://api.github.com/users/shino/gists{/gist_id}","starred_url":"https://api.github.com/users/shino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shino/subscriptions","organizations_url":"https://api.github.com/users/shino/orgs","repos_url":"https://api.github.com/users/shino/repos","events_url":"https://api.github.com/users/shino/events{/privacy}","received_events_url":"https://api.github.com/users/shino/received_events","type":"User","site_admin":false},"body":"Rebased version of #1128 for some reason.\r\n\r\n## Steps for scanning orphan blocks\r\n\r\n1. List keys in `moss.buckets`\r\n2. Iterate over all CS buckets (optionally ones listed in command line)\r\n   1. List keys (streaming listkeys) on a `<<0b:...>>` bucket\r\n      - *We MUST list keys of blocks at first* to avoid race from API calls\r\n   2. Collect all blocks, as `{UUID, [list of existing block seqs]}`\r\n   3. List manifest UUIDs on `<<0o:...>>` bucket (by streaming `csbucketfold`)\r\n   4. Check existence of corresponding manifest by subtracting the two\r\n      results of list keys\r\n   5. Complete maybe orphaned UUID list\r\n3. Filter out false-orphaned block UUIDs.\r\n   This is because list keys (or csbucketfold) of manifest buckets will\r\n   be done by R=1, then replica deficit (e.g. by force-replace) leads to\r\n   false-positiveness for orphan block extraction at previous step.\r\n   R=quorum (or R=all) manifest GETs is needed to avoid false-positive.\r\n   1. Get block object and get CS key of manifests from user metadata\r\n   2. Get manifest for the key and verify existence of corresponding manifest\r\n   3. Remove block UUIDs if manifest exists from orphaned list\r\n\r\nUsage (for debugging)\r\n\r\n```\r\n% riak-cs escript /path/to/priv/tools/internal/block_audit.erl  \\\r\n      -h 127.0.0.1 -p 10017 \\\r\n      -dd \\\r\n      -o $PWD/blocks.local\r\n% riak-cs escript /path/to/priv/tools/internal/ensure_orphan_blocks.erl \\\r\n      -h 127.0.0.1 -p 10017 \\\r\n      -dd \\\r\n      -i $PWD/blocks.local -o $PWD/blocks2.local\r\n```\r\n\r\nThis PR adds scripts for gathering block {UUID, Sequence}s to be deleted.\r\nFor actual deletion, we can re-using block cleaner script in #1124.\r\nBut there is still some interface (file format) discrepancy, subsequent\r\nPR is needed.\r\n","created_at":"2015-04-28T02:08:07Z","updated_at":"2015-04-28T13:01:01Z","closed_at":"2015-04-28T13:01:01Z","merged_at":"2015-04-28T13:01:01Z","merge_commit_sha":"fc7a978a90248ed514330e0d9432bc860bcee854","assignee":null,"milestone":{"url":"https://api.github.com/repos/basho/riak_cs/milestones/17","html_url":"https://github.com/basho/riak_cs/milestones/2.1.0","labels_url":"https://api.github.com/repos/basho/riak_cs/milestones/17/labels","id":1051308,"number":17,"title":"2.1.0","description":"","creator":{"login":"kuenishi","id":11440,"avatar_url":"https://avatars.githubusercontent.com/u/11440?v=3","gravatar_id":"","url":"https://api.github.com/users/kuenishi","html_url":"https://github.com/kuenishi","followers_url":"https://api.github.com/users/kuenishi/followers","following_url":"https://api.github.com/users/kuenishi/following{/other_user}","gists_url":"https://api.github.com/users/kuenishi/gists{/gist_id}","starred_url":"https://api.github.com/users/kuenishi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kuenishi/subscriptions","organizations_url":"https://api.github.com/users/kuenishi/orgs","repos_url":"https://api.github.com/users/kuenishi/repos","events_url":"https://api.github.com/users/kuenishi/events{/privacy}","received_events_url":"https://api.github.com/users/kuenishi/received_events","type":"User","site_admin":false},"open_issues":62,"closed_issues":6,"state":"open","created_at":"2015-04-03T04:11:19Z","updated_at":"2015-04-28T13:01:01Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/basho/riak_cs/pulls/1133/commits","review_comments_url":"https://api.github.com/repos/basho/riak_cs/pulls/1133/comments","review_comment_url":"https://api.github.com/repos/basho/riak_cs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/basho/riak_cs/issues/1133/comments","statuses_url":"https://api.github.com/repos/basho/riak_cs/statuses/88fd61f62c9d0dcd1a022331a23b43e4e94b32d6","head":{"label":"basho:feature/block-audit","ref":"feature/block-audit","sha":"88fd61f62c9d0dcd1a022331a23b43e4e94b32d6","user":{"login":"basho","id":176293,"avatar_url":"https://avatars.githubusercontent.com/u/176293?v=3","gravatar_id":"","url":"https://api.github.com/users/basho","html_url":"https://github.com/basho","followers_url":"https://api.github.com/users/basho/followers","following_url":"https://api.github.com/users/basho/following{/other_user}","gists_url":"https://api.github.com/users/basho/gists{/gist_id}","starred_url":"https://api.github.com/users/basho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basho/subscriptions","organizations_url":"https://api.github.com/users/basho/orgs","repos_url":"https://api.github.com/users/basho/repos","events_url":"https://api.github.com/users/basho/events{/privacy}","received_events_url":"https://api.github.com/users/basho/received_events","type":"Organization","site_admin":false},"repo":{"id":2513151,"name":"riak_cs","full_name":"basho/riak_cs","owner":{"login":"basho","id":176293,"avatar_url":"https://avatars.githubusercontent.com/u/176293?v=3","gravatar_id":"","url":"https://api.github.com/users/basho","html_url":"https://github.com/basho","followers_url":"https://api.github.com/users/basho/followers","following_url":"https://api.github.com/users/basho/following{/other_user}","gists_url":"https://api.github.com/users/basho/gists{/gist_id}","starred_url":"https://api.github.com/users/basho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basho/subscriptions","organizations_url":"https://api.github.com/users/basho/orgs","repos_url":"https://api.github.com/users/basho/repos","events_url":"https://api.github.com/users/basho/events{/privacy}","received_events_url":"https://api.github.com/users/basho/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/basho/riak_cs","description":"Riak CS is simple, available cloud storage built on Riak.","fork":false,"url":"https://api.github.com/repos/basho/riak_cs","forks_url":"https://api.github.com/repos/basho/riak_cs/forks","keys_url":"https://api.github.com/repos/basho/riak_cs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/basho/riak_cs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/basho/riak_cs/teams","hooks_url":"https://api.github.com/repos/basho/riak_cs/hooks","issue_events_url":"https://api.github.com/repos/basho/riak_cs/issues/events{/number}","events_url":"https://api.github.com/repos/basho/riak_cs/events","assignees_url":"https://api.github.com/repos/basho/riak_cs/assignees{/user}","branches_url":"https://api.github.com/repos/basho/riak_cs/branches{/branch}","tags_url":"https://api.github.com/repos/basho/riak_cs/tags","blobs_url":"https://api.github.com/repos/basho/riak_cs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/basho/riak_cs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/basho/riak_cs/git/refs{/sha}","trees_url":"https://api.github.com/repos/basho/riak_cs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/basho/riak_cs/statuses/{sha}","languages_url":"https://api.github.com/repos/basho/riak_cs/languages","stargazers_url":"https://api.github.com/repos/basho/riak_cs/stargazers","contributors_url":"https://api.github.com/repos/basho/riak_cs/contributors","subscribers_url":"https://api.github.com/repos/basho/riak_cs/subscribers","subscription_url":"https://api.github.com/repos/basho/riak_cs/subscription","commits_url":"https://api.github.com/repos/basho/riak_cs/commits{/sha}","git_commits_url":"https://api.github.com/repos/basho/riak_cs/git/commits{/sha}","comments_url":"https://api.github.com/repos/basho/riak_cs/comments{/number}","issue_comment_url":"https://api.github.com/repos/basho/riak_cs/issues/comments{/number}","contents_url":"https://api.github.com/repos/basho/riak_cs/contents/{+path}","compare_url":"https://api.github.com/repos/basho/riak_cs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/basho/riak_cs/merges","archive_url":"https://api.github.com/repos/basho/riak_cs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/basho/riak_cs/downloads","issues_url":"https://api.github.com/repos/basho/riak_cs/issues{/number}","pulls_url":"https://api.github.com/repos/basho/riak_cs/pulls{/number}","milestones_url":"https://api.github.com/repos/basho/riak_cs/milestones{/number}","notifications_url":"https://api.github.com/repos/basho/riak_cs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/basho/riak_cs/labels{/name}","releases_url":"https://api.github.com/repos/basho/riak_cs/releases{/id}","created_at":"2011-10-04T16:16:25Z","updated_at":"2015-04-28T13:01:01Z","pushed_at":"2015-04-28T13:01:01Z","git_url":"git://github.com/basho/riak_cs.git","ssh_url":"git@github.com:basho/riak_cs.git","clone_url":"https://github.com/basho/riak_cs.git","svn_url":"https://github.com/basho/riak_cs","homepage":"http://docs.basho.com/riakcs/latest/","size":22964,"stargazers_count":368,"watchers_count":368,"language":"Erlang","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":54,"mirror_url":null,"open_issues_count":143,"forks":54,"open_issues":143,"watchers":368,"default_branch":"develop"}},"base":{"label":"basho:develop","ref":"develop","sha":"7c727b35c5cc87ccf1af0438416ca09ca19407fb","user":{"login":"basho","id":176293,"avatar_url":"https://avatars.githubusercontent.com/u/176293?v=3","gravatar_id":"","url":"https://api.github.com/users/basho","html_url":"https://github.com/basho","followers_url":"https://api.github.com/users/basho/followers","following_url":"https://api.github.com/users/basho/following{/other_user}","gists_url":"https://api.github.com/users/basho/gists{/gist_id}","starred_url":"https://api.github.com/users/basho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basho/subscriptions","organizations_url":"https://api.github.com/users/basho/orgs","repos_url":"https://api.github.com/users/basho/repos","events_url":"https://api.github.com/users/basho/events{/privacy}","received_events_url":"https://api.github.com/users/basho/received_events","type":"Organization","site_admin":false},"repo":{"id":2513151,"name":"riak_cs","full_name":"basho/riak_cs","owner":{"login":"basho","id":176293,"avatar_url":"https://avatars.githubusercontent.com/u/176293?v=3","gravatar_id":"","url":"https://api.github.com/users/basho","html_url":"https://github.com/basho","followers_url":"https://api.github.com/users/basho/followers","following_url":"https://api.github.com/users/basho/following{/other_user}","gists_url":"https://api.github.com/users/basho/gists{/gist_id}","starred_url":"https://api.github.com/users/basho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basho/subscriptions","organizations_url":"https://api.github.com/users/basho/orgs","repos_url":"https://api.github.com/users/basho/repos","events_url":"https://api.github.com/users/basho/events{/privacy}","received_events_url":"https://api.github.com/users/basho/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/basho/riak_cs","description":"Riak CS is simple, available cloud storage built on Riak.","fork":false,"url":"https://api.github.com/repos/basho/riak_cs","forks_url":"https://api.github.com/repos/basho/riak_cs/forks","keys_url":"https://api.github.com/repos/basho/riak_cs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/basho/riak_cs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/basho/riak_cs/teams","hooks_url":"https://api.github.com/repos/basho/riak_cs/hooks","issue_events_url":"https://api.github.com/repos/basho/riak_cs/issues/events{/number}","events_url":"https://api.github.com/repos/basho/riak_cs/events","assignees_url":"https://api.github.com/repos/basho/riak_cs/assignees{/user}","branches_url":"https://api.github.com/repos/basho/riak_cs/branches{/branch}","tags_url":"https://api.github.com/repos/basho/riak_cs/tags","blobs_url":"https://api.github.com/repos/basho/riak_cs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/basho/riak_cs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/basho/riak_cs/git/refs{/sha}","trees_url":"https://api.github.com/repos/basho/riak_cs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/basho/riak_cs/statuses/{sha}","languages_url":"https://api.github.com/repos/basho/riak_cs/languages","stargazers_url":"https://api.github.com/repos/basho/riak_cs/stargazers","contributors_url":"https://api.github.com/repos/basho/riak_cs/contributors","subscribers_url":"https://api.github.com/repos/basho/riak_cs/subscribers","subscription_url":"https://api.github.com/repos/basho/riak_cs/subscription","commits_url":"https://api.github.com/repos/basho/riak_cs/commits{/sha}","git_commits_url":"https://api.github.com/repos/basho/riak_cs/git/commits{/sha}","comments_url":"https://api.github.com/repos/basho/riak_cs/comments{/number}","issue_comment_url":"https://api.github.com/repos/basho/riak_cs/issues/comments{/number}","contents_url":"https://api.github.com/repos/basho/riak_cs/contents/{+path}","compare_url":"https://api.github.com/repos/basho/riak_cs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/basho/riak_cs/merges","archive_url":"https://api.github.com/repos/basho/riak_cs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/basho/riak_cs/downloads","issues_url":"https://api.github.com/repos/basho/riak_cs/issues{/number}","pulls_url":"https://api.github.com/repos/basho/riak_cs/pulls{/number}","milestones_url":"https://api.github.com/repos/basho/riak_cs/milestones{/number}","notifications_url":"https://api.github.com/repos/basho/riak_cs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/basho/riak_cs/labels{/name}","releases_url":"https://api.github.com/repos/basho/riak_cs/releases{/id}","created_at":"2011-10-04T16:16:25Z","updated_at":"2015-04-28T13:01:01Z","pushed_at":"2015-04-28T13:01:01Z","git_url":"git://github.com/basho/riak_cs.git","ssh_url":"git@github.com:basho/riak_cs.git","clone_url":"https://github.com/basho/riak_cs.git","svn_url":"https://github.com/basho/riak_cs","homepage":"http://docs.basho.com/riakcs/latest/","size":22964,"stargazers_count":368,"watchers_count":368,"language":"Erlang","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":54,"mirror_url":null,"open_issues_count":143,"forks":54,"open_issues":143,"watchers":368,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/basho/riak_cs/pulls/1133"},"html":{"href":"https://github.com/basho/riak_cs/pull/1133"},"issue":{"href":"https://api.github.com/repos/basho/riak_cs/issues/1133"},"comments":{"href":"https://api.github.com/repos/basho/riak_cs/issues/1133/comments"},"review_comments":{"href":"https://api.github.com/repos/basho/riak_cs/pulls/1133/comments"},"review_comment":{"href":"https://api.github.com/repos/basho/riak_cs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/basho/riak_cs/pulls/1133/commits"},"statuses":{"href":"https://api.github.com/repos/basho/riak_cs/statuses/88fd61f62c9d0dcd1a022331a23b43e4e94b32d6"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"borshop","id":6749250,"avatar_url":"https://avatars.githubusercontent.com/u/6749250?v=3","gravatar_id":"","url":"https://api.github.com/users/borshop","html_url":"https://github.com/borshop","followers_url":"https://api.github.com/users/borshop/followers","following_url":"https://api.github.com/users/borshop/following{/other_user}","gists_url":"https://api.github.com/users/borshop/gists{/gist_id}","starred_url":"https://api.github.com/users/borshop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borshop/subscriptions","organizations_url":"https://api.github.com/users/borshop/orgs","repos_url":"https://api.github.com/users/borshop/repos","events_url":"https://api.github.com/users/borshop/events{/privacy}","received_events_url":"https://api.github.com/users/borshop/received_events","type":"User","site_admin":false},"comments":3,"review_comments":9,"commits":6,"additions":643,"deletions":12,"changed_files":5}}