{"action":"created","issue":{"url":"https://api.github.com/repos/basho/riak_cs/issues/1133","labels_url":"https://api.github.com/repos/basho/riak_cs/issues/1133/labels{/name}","comments_url":"https://api.github.com/repos/basho/riak_cs/issues/1133/comments","events_url":"https://api.github.com/repos/basho/riak_cs/issues/1133/events","html_url":"https://github.com/basho/riak_cs/pull/1133","id":71453138,"number":1133,"title":"Orphan block scanner scripts [2.1]","user":{"login":"shino","id":10225,"avatar_url":"https://avatars.githubusercontent.com/u/10225?v=3","gravatar_id":"","url":"https://api.github.com/users/shino","html_url":"https://github.com/shino","followers_url":"https://api.github.com/users/shino/followers","following_url":"https://api.github.com/users/shino/following{/other_user}","gists_url":"https://api.github.com/users/shino/gists{/gist_id}","starred_url":"https://api.github.com/users/shino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shino/subscriptions","organizations_url":"https://api.github.com/users/shino/orgs","repos_url":"https://api.github.com/users/shino/repos","events_url":"https://api.github.com/users/shino/events{/privacy}","received_events_url":"https://api.github.com/users/shino/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/basho/riak_cs/milestones/17","html_url":"https://github.com/basho/riak_cs/milestones/2.1.0","labels_url":"https://api.github.com/repos/basho/riak_cs/milestones/17/labels","id":1051308,"number":17,"title":"2.1.0","description":"","creator":{"login":"kuenishi","id":11440,"avatar_url":"https://avatars.githubusercontent.com/u/11440?v=3","gravatar_id":"","url":"https://api.github.com/users/kuenishi","html_url":"https://github.com/kuenishi","followers_url":"https://api.github.com/users/kuenishi/followers","following_url":"https://api.github.com/users/kuenishi/following{/other_user}","gists_url":"https://api.github.com/users/kuenishi/gists{/gist_id}","starred_url":"https://api.github.com/users/kuenishi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kuenishi/subscriptions","organizations_url":"https://api.github.com/users/kuenishi/orgs","repos_url":"https://api.github.com/users/kuenishi/repos","events_url":"https://api.github.com/users/kuenishi/events{/privacy}","received_events_url":"https://api.github.com/users/kuenishi/received_events","type":"User","site_admin":false},"open_issues":63,"closed_issues":5,"state":"open","created_at":"2015-04-03T04:11:19Z","updated_at":"2015-04-28T03:19:20Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-04-28T02:08:07Z","updated_at":"2015-04-28T09:27:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/basho/riak_cs/pulls/1133","html_url":"https://github.com/basho/riak_cs/pull/1133","diff_url":"https://github.com/basho/riak_cs/pull/1133.diff","patch_url":"https://github.com/basho/riak_cs/pull/1133.patch"},"body":"Rebased version of #1128 for some reason.\r\n\r\n## Steps for scanning orphan blocks\r\n\r\n1. List keys in `moss.buckets`\r\n2. Iterate over all CS buckets (optionally ones listed in command line)\r\n   1. List keys (streaming listkeys) on a `<<0b:...>>` bucket\r\n      - *We MUST list keys of blocks at first* to avoid race from API calls\r\n   2. Collect all blocks, as `{UUID, [list of existing block seqs]}`\r\n   3. List manifest UUIDs on `<<0o:...>>` bucket (by streaming `csbucketfold`)\r\n   4. Check existence of corresponding manifest by subtracting the two\r\n      results of list keys\r\n   5. Complete maybe orphaned UUID list\r\n3. Filter out false-orphaned block UUIDs.\r\n   This is because list keys (or csbucketfold) of manifest buckets will\r\n   be done by R=1, then replica deficit (e.g. by force-replace) leads to\r\n   false-positiveness for orphan block extraction at previous step.\r\n   R=quorum (or R=all) manifest GETs is needed to avoid false-positive.\r\n   1. Get block object and get CS key of manifests from user metadata\r\n   2. Get manifest for the key and verify existence of corresponding manifest\r\n   3. Remove block UUIDs if manifest exists from orphaned list\r\n\r\nUsage (for debugging)\r\n\r\n```\r\n% riak-cs escript /path/to/priv/tools/internal/block_audit.erl  \\\r\n      -h 127.0.0.1 -p 10017 \\\r\n      -dd \\\r\n      -o $PWD/blocks.local\r\n% riak-cs escript /path/to/priv/tools/internal/block_audit2.erl \\\r\n      -h 127.0.0.1 -p 10017 \\\r\n      -dd \\\r\n      -i $PWD/blocks.local -o $PWD/blocks2.local\r\n```\r\n\r\nThis PR adds scripts for gathering block {UUID, Sequence}s to be deleted.\r\nFor actual deletion, we can re-using block cleaner script in #1124.\r\nBut there is still some interface (file format) discrepancy, subsequent\r\nPR is needed.\r\n"},"comment":{"url":"https://api.github.com/repos/basho/riak_cs/issues/comments/96987431","html_url":"https://github.com/basho/riak_cs/pull/1133#issuecomment-96987431","issue_url":"https://api.github.com/repos/basho/riak_cs/issues/1133","id":96987431,"user":{"login":"shino","id":10225,"avatar_url":"https://avatars.githubusercontent.com/u/10225?v=3","gravatar_id":"","url":"https://api.github.com/users/shino","html_url":"https://github.com/shino","followers_url":"https://api.github.com/users/shino/followers","following_url":"https://api.github.com/users/shino/following{/other_user}","gists_url":"https://api.github.com/users/shino/gists{/gist_id}","starred_url":"https://api.github.com/users/shino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shino/subscriptions","organizations_url":"https://api.github.com/users/shino/orgs","repos_url":"https://api.github.com/users/shino/repos","events_url":"https://api.github.com/users/shino/events{/privacy}","received_events_url":"https://api.github.com/users/shino/received_events","type":"User","site_admin":false},"created_at":"2015-04-28T09:27:26Z","updated_at":"2015-04-28T09:27:26Z","body":"Thanks for review :smile: Addressed the comments and pushed :arrow_double_up: "}}