{"action":"created","issue":{"url":"https://api.github.com/repos/basho/riak_kv/issues/1072","labels_url":"https://api.github.com/repos/basho/riak_kv/issues/1072/labels{/name}","comments_url":"https://api.github.com/repos/basho/riak_kv/issues/1072/comments","events_url":"https://api.github.com/repos/basho/riak_kv/issues/1072/events","html_url":"https://github.com/basho/riak_kv/pull/1072","id":54771463,"number":1072,"title":"Optimize mapreduce by replacing listkeys+get with 2i query","user":{"login":"kuenishi","id":11440,"avatar_url":"https://avatars.githubusercontent.com/u/11440?v=3","gravatar_id":"","url":"https://api.github.com/users/kuenishi","html_url":"https://github.com/kuenishi","followers_url":"https://api.github.com/users/kuenishi/followers","following_url":"https://api.github.com/users/kuenishi/following{/other_user}","gists_url":"https://api.github.com/users/kuenishi/gists{/gist_id}","starred_url":"https://api.github.com/users/kuenishi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kuenishi/subscriptions","organizations_url":"https://api.github.com/users/kuenishi/orgs","repos_url":"https://api.github.com/users/kuenishi/repos","events_url":"https://api.github.com/users/kuenishi/events{/privacy}","received_events_url":"https://api.github.com/users/kuenishi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-19T14:39:56Z","updated_at":"2015-01-20T07:39:48Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/basho/riak_kv/pulls/1072","html_url":"https://github.com/basho/riak_kv/pull/1072","diff_url":"https://github.com/basho/riak_kv/pull/1072.diff","patch_url":"https://github.com/basho/riak_kv/pull/1072.patch"},"body":"Simple benchmark in CS showed 1.7x performance improvement:\r\n\r\n```\r\n(riak-cs@127.0.0.1)1> timer:tc(fun()->riak_cs_storage:sum_bucket(<<\"test\">>)end).\r\n{4673368,\r\n {struct,[{<<\"Objects\">>,98407},{<<\"Bytes\">>,4341828394}]}}\r\n(riak-cs@127.0.0.1)2> timer:tc(fun()->riak_cs_storage:sum_bucket2(<<\"test\">>)end).\r\n{2810982,\r\n {struct,[{<<\"Objects\">>,98407},{<<\"Bytes\">>,4341828394}]}}\r\n```\r\n\r\nThis performance gain stems from duplicate disk read from both\r\n`riak_kv_pipe_listkeys` and `riak_kv_pipe_get` .\r\n`riak_kv_pipe_listkeys` does not read from disk when the backend is\r\nbitcask, but when the backend is leveldb, it reads the whole block\r\nincluding not only keys but values (aka Riak objects). This pull\r\nrequest replaces them with modified `riak_pipe_index` with\r\n`return_body=true` and enable a vnode to just read from one time. This\r\nis inspired by `riak_kv_pb_csbucket`.\r\n\r\nThis is originally for performance optimization of storage stats calculation\r\nof Riak CS (https://github.com/basho/riak_cs/pull/1051) but also\r\ncontributes for other use cases folding a whole bucket with MapReduce.\r\n\r\nOther discussion point is whether this optimization applies for inputs with\r\nindex `<<\"$key\">>` . I think this works, but too big change might break\r\nsomething."},"comment":{"url":"https://api.github.com/repos/basho/riak_kv/issues/comments/70615405","html_url":"https://github.com/basho/riak_kv/pull/1072#issuecomment-70615405","issue_url":"https://api.github.com/repos/basho/riak_kv/issues/1072","id":70615405,"user":{"login":"russelldb","id":56321,"avatar_url":"https://avatars.githubusercontent.com/u/56321?v=3","gravatar_id":"","url":"https://api.github.com/users/russelldb","html_url":"https://github.com/russelldb","followers_url":"https://api.github.com/users/russelldb/followers","following_url":"https://api.github.com/users/russelldb/following{/other_user}","gists_url":"https://api.github.com/users/russelldb/gists{/gist_id}","starred_url":"https://api.github.com/users/russelldb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/russelldb/subscriptions","organizations_url":"https://api.github.com/users/russelldb/orgs","repos_url":"https://api.github.com/users/russelldb/repos","events_url":"https://api.github.com/users/russelldb/events{/privacy}","received_events_url":"https://api.github.com/users/russelldb/received_events","type":"User","site_admin":false},"created_at":"2015-01-20T07:39:48Z","updated_at":"2015-01-20T07:39:48Z","body":"IMO, if there is no new functionality, just parity but greater performance, the existing tests are fine."}}