{"action":"created","comment":{"url":"https://api.github.com/repos/reverbrain/elliptics/pulls/comments/39631942","id":39631942,"diff_hunk":"@@ -314,6 +314,31 @@ int dnet_send_reply(void *state, struct dnet_cmd *cmd, const void *odata, unsign\n \treturn err;\n }\n \n+void dnet_queue_wait_threshold(void *state)\n+{\n+\tstruct dnet_net_state *st = state;","path":"library/dnet.c","position":22,"original_position":22,"commit_id":"0c393a4883e352c5ce9357c8d4090ca9b9049cfe","original_commit_id":"0c393a4883e352c5ce9357c8d4090ca9b9049cfe","user":{"login":"shaitan","id":446423,"avatar_url":"https://avatars.githubusercontent.com/u/446423?v=3","gravatar_id":"","url":"https://api.github.com/users/shaitan","html_url":"https://github.com/shaitan","followers_url":"https://api.github.com/users/shaitan/followers","following_url":"https://api.github.com/users/shaitan/following{/other_user}","gists_url":"https://api.github.com/users/shaitan/gists{/gist_id}","starred_url":"https://api.github.com/users/shaitan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shaitan/subscriptions","organizations_url":"https://api.github.com/users/shaitan/orgs","repos_url":"https://api.github.com/users/shaitan/repos","events_url":"https://api.github.com/users/shaitan/events{/privacy}","received_events_url":"https://api.github.com/users/shaitan/received_events","type":"User","site_admin":false},"body":"Is it really needed to call `dnet_queue_wait_threshold` by backend? It is called only by `dnet_send_reply_threshold`, so it can be defined as a static.","created_at":"2015-09-16T13:56:33Z","updated_at":"2015-09-16T13:56:33Z","html_url":"https://github.com/reverbrain/elliptics/pull/650#discussion_r39631942","pull_request_url":"https://api.github.com/repos/reverbrain/elliptics/pulls/650","_links":{"self":{"href":"https://api.github.com/repos/reverbrain/elliptics/pulls/comments/39631942"},"html":{"href":"https://github.com/reverbrain/elliptics/pull/650#discussion_r39631942"},"pull_request":{"href":"https://api.github.com/repos/reverbrain/elliptics/pulls/650"}}},"pull_request":{"url":"https://api.github.com/repos/reverbrain/elliptics/pulls/650","id":44994113,"html_url":"https://github.com/reverbrain/elliptics/pull/650","diff_url":"https://github.com/reverbrain/elliptics/pull/650.diff","patch_url":"https://github.com/reverbrain/elliptics/pull/650.patch","issue_url":"https://api.github.com/repos/reverbrain/elliptics/issues/650","number":650,"state":"open","locked":false,"title":"Server-send operations","user":{"login":"bioothod","id":1847575,"avatar_url":"https://avatars.githubusercontent.com/u/1847575?v=3","gravatar_id":"","url":"https://api.github.com/users/bioothod","html_url":"https://github.com/bioothod","followers_url":"https://api.github.com/users/bioothod/followers","following_url":"https://api.github.com/users/bioothod/following{/other_user}","gists_url":"https://api.github.com/users/bioothod/gists{/gist_id}","starred_url":"https://api.github.com/users/bioothod/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bioothod/subscriptions","organizations_url":"https://api.github.com/users/bioothod/orgs","repos_url":"https://api.github.com/users/bioothod/repos","events_url":"https://api.github.com/users/bioothod/events{/privacy}","received_events_url":"https://api.github.com/users/bioothod/received_events","type":"User","site_admin":false},"body":"This pull request introduces 2 server-side operations which allow to send content from one server to multiple replicas. It is intended for various recovery tools which optimize by not copying data from local node to recovery temporary location, instead they may tell remote node to send data directly to required locations. It can also be used to move data from older eblob version to newer without server interruption.\r\n\r\n1. New iterator type which sends all keys being iterated to set of remote groups. It does it with the speed of network or disk (what it slower), in local tests iteration over 200Gb blobs sending data over the network to one remote node ended up with ~78MB/s sustained speed with pikes though, especially when remote node synced caches. Both sender and recipient had 30Gb of RAM. `rsync` shows ~32MB/s speed on these machines, but not because it is that slow, but because of `ssh` which maxed out CPU by packet encryption.\r\nIterator sends `dnet_iterator_response` structure for each write result for every key it has processed just like for usual iterator, neither API nor ABI is broken.\r\n\r\n2. Server-send command which accepts vector of keys. API searches for all remote nodes/backends which host given keys in the one specified group, splits keys into per-node/backend basis and tells remote backends to send appropriate keys to specified remote groups. The same iterator response is generated for every key which has been processed.\r\n\r\nThere are 3 operation modes:\r\n1. default - writing data to remote node using compare-and-swap, i.e. only write data if it either doesn't exist or it is the same on remote servers. Server send (iterator or per-key) running in this mode is especially useful for recovery - there is no way it can overwrite newer copy with the old data.\r\n2. overwrite - when special flag is set, overwrite data (clears compare-and-swap logic)\r\n3. move - if write has been successful, remove local key\r\n\r\nThere is example tool in example dir, which works the same way as `example/iterate.cpp`, but performs copy/move of the iterator results. There are 16 various tests which cover all operation modes. Tests are rather long - so far they are the longest, it takes about 170 seconds on my server.\r\n\r\nUnfortunately this pull request also contains fair number of bugfixes which should live in a separate commits. These include possible memory corruption, invalid pointer dereference and race conditions. I found them during extensive testing and failed to separate into own commits.","created_at":"2015-09-14T23:40:43Z","updated_at":"2015-09-16T13:56:33Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ed19db5b4a8fc25cdfdc5997e412dbaa76478ad5","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/reverbrain/elliptics/pulls/650/commits","review_comments_url":"https://api.github.com/repos/reverbrain/elliptics/pulls/650/comments","review_comment_url":"https://api.github.com/repos/reverbrain/elliptics/pulls/comments{/number}","comments_url":"https://api.github.com/repos/reverbrain/elliptics/issues/650/comments","statuses_url":"https://api.github.com/repos/reverbrain/elliptics/statuses/0c393a4883e352c5ce9357c8d4090ca9b9049cfe","head":{"label":"bioothod:master","ref":"master","sha":"0c393a4883e352c5ce9357c8d4090ca9b9049cfe","user":{"login":"bioothod","id":1847575,"avatar_url":"https://avatars.githubusercontent.com/u/1847575?v=3","gravatar_id":"","url":"https://api.github.com/users/bioothod","html_url":"https://github.com/bioothod","followers_url":"https://api.github.com/users/bioothod/followers","following_url":"https://api.github.com/users/bioothod/following{/other_user}","gists_url":"https://api.github.com/users/bioothod/gists{/gist_id}","starred_url":"https://api.github.com/users/bioothod/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bioothod/subscriptions","organizations_url":"https://api.github.com/users/bioothod/orgs","repos_url":"https://api.github.com/users/bioothod/repos","events_url":"https://api.github.com/users/bioothod/events{/privacy}","received_events_url":"https://api.github.com/users/bioothod/received_events","type":"User","site_admin":false},"repo":{"id":34638202,"name":"elliptics","full_name":"bioothod/elliptics","owner":{"login":"bioothod","id":1847575,"avatar_url":"https://avatars.githubusercontent.com/u/1847575?v=3","gravatar_id":"","url":"https://api.github.com/users/bioothod","html_url":"https://github.com/bioothod","followers_url":"https://api.github.com/users/bioothod/followers","following_url":"https://api.github.com/users/bioothod/following{/other_user}","gists_url":"https://api.github.com/users/bioothod/gists{/gist_id}","starred_url":"https://api.github.com/users/bioothod/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bioothod/subscriptions","organizations_url":"https://api.github.com/users/bioothod/orgs","repos_url":"https://api.github.com/users/bioothod/repos","events_url":"https://api.github.com/users/bioothod/events{/privacy}","received_events_url":"https://api.github.com/users/bioothod/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/bioothod/elliptics","description":"Distributed hashtable storage","fork":true,"url":"https://api.github.com/repos/bioothod/elliptics","forks_url":"https://api.github.com/repos/bioothod/elliptics/forks","keys_url":"https://api.github.com/repos/bioothod/elliptics/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bioothod/elliptics/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bioothod/elliptics/teams","hooks_url":"https://api.github.com/repos/bioothod/elliptics/hooks","issue_events_url":"https://api.github.com/repos/bioothod/elliptics/issues/events{/number}","events_url":"https://api.github.com/repos/bioothod/elliptics/events","assignees_url":"https://api.github.com/repos/bioothod/elliptics/assignees{/user}","branches_url":"https://api.github.com/repos/bioothod/elliptics/branches{/branch}","tags_url":"https://api.github.com/repos/bioothod/elliptics/tags","blobs_url":"https://api.github.com/repos/bioothod/elliptics/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bioothod/elliptics/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bioothod/elliptics/git/refs{/sha}","trees_url":"https://api.github.com/repos/bioothod/elliptics/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bioothod/elliptics/statuses/{sha}","languages_url":"https://api.github.com/repos/bioothod/elliptics/languages","stargazers_url":"https://api.github.com/repos/bioothod/elliptics/stargazers","contributors_url":"https://api.github.com/repos/bioothod/elliptics/contributors","subscribers_url":"https://api.github.com/repos/bioothod/elliptics/subscribers","subscription_url":"https://api.github.com/repos/bioothod/elliptics/subscription","commits_url":"https://api.github.com/repos/bioothod/elliptics/commits{/sha}","git_commits_url":"https://api.github.com/repos/bioothod/elliptics/git/commits{/sha}","comments_url":"https://api.github.com/repos/bioothod/elliptics/comments{/number}","issue_comment_url":"https://api.github.com/repos/bioothod/elliptics/issues/comments{/number}","contents_url":"https://api.github.com/repos/bioothod/elliptics/contents/{+path}","compare_url":"https://api.github.com/repos/bioothod/elliptics/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bioothod/elliptics/merges","archive_url":"https://api.github.com/repos/bioothod/elliptics/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bioothod/elliptics/downloads","issues_url":"https://api.github.com/repos/bioothod/elliptics/issues{/number}","pulls_url":"https://api.github.com/repos/bioothod/elliptics/pulls{/number}","milestones_url":"https://api.github.com/repos/bioothod/elliptics/milestones{/number}","notifications_url":"https://api.github.com/repos/bioothod/elliptics/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bioothod/elliptics/labels{/name}","releases_url":"https://api.github.com/repos/bioothod/elliptics/releases{/id}","created_at":"2015-04-27T00:42:00Z","updated_at":"2015-05-05T22:34:45Z","pushed_at":"2015-09-16T08:29:25Z","git_url":"git://github.com/bioothod/elliptics.git","ssh_url":"git@github.com:bioothod/elliptics.git","clone_url":"https://github.com/bioothod/elliptics.git","svn_url":"https://github.com/bioothod/elliptics","homepage":null,"size":20229,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"reverbrain:master","ref":"master","sha":"b3cb7134d5f149e81730f2cb91275329485247a7","user":{"login":"reverbrain","id":1616511,"avatar_url":"https://avatars.githubusercontent.com/u/1616511?v=3","gravatar_id":"","url":"https://api.github.com/users/reverbrain","html_url":"https://github.com/reverbrain","followers_url":"https://api.github.com/users/reverbrain/followers","following_url":"https://api.github.com/users/reverbrain/following{/other_user}","gists_url":"https://api.github.com/users/reverbrain/gists{/gist_id}","starred_url":"https://api.github.com/users/reverbrain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reverbrain/subscriptions","organizations_url":"https://api.github.com/users/reverbrain/orgs","repos_url":"https://api.github.com/users/reverbrain/repos","events_url":"https://api.github.com/users/reverbrain/events{/privacy}","received_events_url":"https://api.github.com/users/reverbrain/received_events","type":"Organization","site_admin":false},"repo":{"id":4712089,"name":"elliptics","full_name":"reverbrain/elliptics","owner":{"login":"reverbrain","id":1616511,"avatar_url":"https://avatars.githubusercontent.com/u/1616511?v=3","gravatar_id":"","url":"https://api.github.com/users/reverbrain","html_url":"https://github.com/reverbrain","followers_url":"https://api.github.com/users/reverbrain/followers","following_url":"https://api.github.com/users/reverbrain/following{/other_user}","gists_url":"https://api.github.com/users/reverbrain/gists{/gist_id}","starred_url":"https://api.github.com/users/reverbrain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reverbrain/subscriptions","organizations_url":"https://api.github.com/users/reverbrain/orgs","repos_url":"https://api.github.com/users/reverbrain/repos","events_url":"https://api.github.com/users/reverbrain/events{/privacy}","received_events_url":"https://api.github.com/users/reverbrain/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/reverbrain/elliptics","description":"Distributed storage for medium and large objects with streaming support","fork":false,"url":"https://api.github.com/repos/reverbrain/elliptics","forks_url":"https://api.github.com/repos/reverbrain/elliptics/forks","keys_url":"https://api.github.com/repos/reverbrain/elliptics/keys{/key_id}","collaborators_url":"https://api.github.com/repos/reverbrain/elliptics/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/reverbrain/elliptics/teams","hooks_url":"https://api.github.com/repos/reverbrain/elliptics/hooks","issue_events_url":"https://api.github.com/repos/reverbrain/elliptics/issues/events{/number}","events_url":"https://api.github.com/repos/reverbrain/elliptics/events","assignees_url":"https://api.github.com/repos/reverbrain/elliptics/assignees{/user}","branches_url":"https://api.github.com/repos/reverbrain/elliptics/branches{/branch}","tags_url":"https://api.github.com/repos/reverbrain/elliptics/tags","blobs_url":"https://api.github.com/repos/reverbrain/elliptics/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/reverbrain/elliptics/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/reverbrain/elliptics/git/refs{/sha}","trees_url":"https://api.github.com/repos/reverbrain/elliptics/git/trees{/sha}","statuses_url":"https://api.github.com/repos/reverbrain/elliptics/statuses/{sha}","languages_url":"https://api.github.com/repos/reverbrain/elliptics/languages","stargazers_url":"https://api.github.com/repos/reverbrain/elliptics/stargazers","contributors_url":"https://api.github.com/repos/reverbrain/elliptics/contributors","subscribers_url":"https://api.github.com/repos/reverbrain/elliptics/subscribers","subscription_url":"https://api.github.com/repos/reverbrain/elliptics/subscription","commits_url":"https://api.github.com/repos/reverbrain/elliptics/commits{/sha}","git_commits_url":"https://api.github.com/repos/reverbrain/elliptics/git/commits{/sha}","comments_url":"https://api.github.com/repos/reverbrain/elliptics/comments{/number}","issue_comment_url":"https://api.github.com/repos/reverbrain/elliptics/issues/comments{/number}","contents_url":"https://api.github.com/repos/reverbrain/elliptics/contents/{+path}","compare_url":"https://api.github.com/repos/reverbrain/elliptics/compare/{base}...{head}","merges_url":"https://api.github.com/repos/reverbrain/elliptics/merges","archive_url":"https://api.github.com/repos/reverbrain/elliptics/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/reverbrain/elliptics/downloads","issues_url":"https://api.github.com/repos/reverbrain/elliptics/issues{/number}","pulls_url":"https://api.github.com/repos/reverbrain/elliptics/pulls{/number}","milestones_url":"https://api.github.com/repos/reverbrain/elliptics/milestones{/number}","notifications_url":"https://api.github.com/repos/reverbrain/elliptics/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/reverbrain/elliptics/labels{/name}","releases_url":"https://api.github.com/repos/reverbrain/elliptics/releases{/id}","created_at":"2012-06-19T10:05:10Z","updated_at":"2015-08-19T20:53:50Z","pushed_at":"2015-09-16T08:29:25Z","git_url":"git://github.com/reverbrain/elliptics.git","ssh_url":"git@github.com:reverbrain/elliptics.git","clone_url":"https://github.com/reverbrain/elliptics.git","svn_url":"https://github.com/reverbrain/elliptics","homepage":"http://reverbrain.com/elliptics","size":28100,"stargazers_count":336,"watchers_count":336,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":62,"mirror_url":null,"open_issues_count":20,"forks":62,"open_issues":20,"watchers":336,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/reverbrain/elliptics/pulls/650"},"html":{"href":"https://github.com/reverbrain/elliptics/pull/650"},"issue":{"href":"https://api.github.com/repos/reverbrain/elliptics/issues/650"},"comments":{"href":"https://api.github.com/repos/reverbrain/elliptics/issues/650/comments"},"review_comments":{"href":"https://api.github.com/repos/reverbrain/elliptics/pulls/650/comments"},"review_comment":{"href":"https://api.github.com/repos/reverbrain/elliptics/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/reverbrain/elliptics/pulls/650/commits"},"statuses":{"href":"https://api.github.com/repos/reverbrain/elliptics/statuses/0c393a4883e352c5ce9357c8d4090ca9b9049cfe"}}}}