{"action":"created","issue":{"url":"https://api.github.com/repos/basho/riak_cs/issues/1202","labels_url":"https://api.github.com/repos/basho/riak_cs/issues/1202/labels{/name}","comments_url":"https://api.github.com/repos/basho/riak_cs/issues/1202/comments","events_url":"https://api.github.com/repos/basho/riak_cs/issues/1202/events","html_url":"https://github.com/basho/riak_cs/pull/1202","id":98108100,"number":1202,"title":"Add audit tool to extract inconsistencies between users and buckets","user":{"login":"shino","id":10225,"avatar_url":"https://avatars.githubusercontent.com/u/10225?v=3","gravatar_id":"","url":"https://api.github.com/users/shino","html_url":"https://github.com/shino","followers_url":"https://api.github.com/users/shino/followers","following_url":"https://api.github.com/users/shino/following{/other_user}","gists_url":"https://api.github.com/users/shino/gists{/gist_id}","starred_url":"https://api.github.com/users/shino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shino/subscriptions","organizations_url":"https://api.github.com/users/shino/orgs","repos_url":"https://api.github.com/users/shino/repos","events_url":"https://api.github.com/users/shino/events{/privacy}","received_events_url":"https://api.github.com/users/shino/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/basho/riak_cs/labels/Enhancement","name":"Enhancement","color":"207de5"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/basho/riak_cs/milestones/18","html_url":"https://github.com/basho/riak_cs/milestones/2.1.0","labels_url":"https://api.github.com/repos/basho/riak_cs/milestones/18/labels","id":1087473,"number":18,"title":"2.1.0","description":"","creator":{"login":"kuenishi","id":11440,"avatar_url":"https://avatars.githubusercontent.com/u/11440?v=3","gravatar_id":"","url":"https://api.github.com/users/kuenishi","html_url":"https://github.com/kuenishi","followers_url":"https://api.github.com/users/kuenishi/followers","following_url":"https://api.github.com/users/kuenishi/following{/other_user}","gists_url":"https://api.github.com/users/kuenishi/gists{/gist_id}","starred_url":"https://api.github.com/users/kuenishi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kuenishi/subscriptions","organizations_url":"https://api.github.com/users/kuenishi/orgs","repos_url":"https://api.github.com/users/kuenishi/repos","events_url":"https://api.github.com/users/kuenishi/events{/privacy}","received_events_url":"https://api.github.com/users/kuenishi/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":52,"state":"open","created_at":"2015-04-28T13:37:13Z","updated_at":"2015-07-31T08:36:54Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2015-07-30T08:07:01Z","updated_at":"2015-07-31T09:15:03Z","closed_at":"2015-07-31T06:12:47Z","pull_request":{"url":"https://api.github.com/repos/basho/riak_cs/pulls/1202","html_url":"https://github.com/basho/riak_cs/pull/1202","diff_url":"https://github.com/basho/riak_cs/pull/1202.diff","patch_url":"https://github.com/basho/riak_cs/pull/1202.patch"},"body":"There is possibility of inconsistencies between the user bucket `moss.users`\r\nand the buckets bucket `moss.buckets` because no transaction is (can be) used.\r\n\r\nThis PR adds tool for detecting inconsistencies between them.\r\nThree kinds of inconsistencies:\r\n\r\n1. Bucket object in buckets bucket is not tracked by the user\r\n   in its value. (`{not_tracked, {User, Bucket}}`)\r\n2. A certain user thinks it owns a bucket but no objects in buckets bucket\r\n   exists. (`{no_bucket_object, {User, Bucket}}`)\r\n3. A certain user thinks it owns a bucket but the bucket is actually\r\n   owned by different user. (`{different_user, {User, Bucket, DifferentUser}}`)\r\n\r\n----\r\n\r\nSample execution by console command and log output in it:\r\n\r\n```\r\n% dev/dev1/bin/riak-cs-admin audit-bucket-ownership\r\n[{no_bucket_object,{<<\"T_RNSHIN22TIJ1MKF8BP\">>,<<\"corrupt\">>}},\r\n {different_user,{<<\"N4ADKYOVGZLKL5RVD0MA\">>,<<\"bob2\">>,\r\n                  <<\"T_RNSHIN22TIJ1MKF8BP\">>}},\r\n {not_tracked,{<<\"T_RNSHIN22TIJ1MKF8BP\">>,<<\"missing-in-list\">>}},\r\n {not_tracked,{<<\"T_RNSHIN22TIJ1MKF8BP\">>,<<\"bob2\">>}},\r\n {not_tracked,{<<\"N4ADKYOVGZLKL5RVD0MA\">>,<<\"missing-in-bobs-list\">>}}]\r\n```\r\n\r\n```\r\n16:54:20.475 [info] Bucket is not tracked by user: {User, Bucket} = {N4ADKYOVGZLKL5RVD0MA, missing-in-bobs-list}\r\n16:54:20.475 [info] Bucket is not tracked by user: {User, Bucket} = {T_RNSHIN22TIJ1MKF8BP, bob2}\r\n16:54:20.475 [info] Bucket is not tracked by user: {User, Bucket} = {T_RNSHIN22TIJ1MKF8BP, missing-in-list}\r\n16:54:20.475 [info] Bucket is owned by different user: {User, Bucket, DifferentUser} = {N4ADKYOVGZLKL5RVD0MA, bob2, T_RNSHIN22TIJ1MKF8BP}\r\n16:54:20.475 [info] Bucket does not exist: {User, Bucket} = {T_RNSHIN22TIJ1MKF8BP, corrupt}\r\n```\r\n\r\n----\r\n\r\nThis PR is in the state of request-for-comment.  Any comments on categorization,\r\nlog format, command line output, implementation, etc is appreciated.\r\n"},"comment":{"url":"https://api.github.com/repos/basho/riak_cs/issues/comments/126618384","html_url":"https://github.com/basho/riak_cs/pull/1202#issuecomment-126618384","issue_url":"https://api.github.com/repos/basho/riak_cs/issues/1202","id":126618384,"user":{"login":"shino","id":10225,"avatar_url":"https://avatars.githubusercontent.com/u/10225?v=3","gravatar_id":"","url":"https://api.github.com/users/shino","html_url":"https://github.com/shino","followers_url":"https://api.github.com/users/shino/followers","following_url":"https://api.github.com/users/shino/following{/other_user}","gists_url":"https://api.github.com/users/shino/gists{/gist_id}","starred_url":"https://api.github.com/users/shino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shino/subscriptions","organizations_url":"https://api.github.com/users/shino/orgs","repos_url":"https://api.github.com/users/shino/repos","events_url":"https://api.github.com/users/shino/events{/privacy}","received_events_url":"https://api.github.com/users/shino/received_events","type":"User","site_admin":false},"created_at":"2015-07-31T09:15:03Z","updated_at":"2015-07-31T09:15:03Z","body":"In local environment (single node cluster with ring size 8, not good for benchmark), it took a few seconds for list keys, it seems that getting every object in serialized way is current bottleneck. But, one more point, Riak CS node consumes CPU ~ 300%. Feels strange. Further investigation / optimization may follow in another ticket / issue."}}