{"action":"created","comment":{"url":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/comments/40283967","id":40283967,"diff_hunk":"@@ -812,9 +786,18 @@ read_batch(ContQueryCombinerState *state, TupleBufferBatchReader *reader)\n \n \t\tExecStoreTuple(heap_copytuple(tbs->tuple->heaptup), state->slot, InvalidBuffer, false);\n \t\ttuplestore_puttupleslot(state->batch, state->slot);\n+\t\tstate->batch_hashes[count] = tbs->tuple->group_hash;\n \n \t\tIncrementCQRead(1, tbs->size);\n \t\tcount++;\n+\n+\t\t/*\n+\t\t * This is fairly unlikely, but if it does happen we should\n+\t\t * finish processing this batch immediately because we're storing 8-byte hashes\n+\t\t * in memory alongside each of these tuples.\n+\t\t */\n+\t\tif (!tuplestore_in_memory(state->batch))","path":"src/backend/pipeline/cont_combiner.c","position":null,"original_position":237,"commit_id":"21fdbe7404b83a5c4778411bf3a4295f80c3a088","original_commit_id":"856e939455a2097caac6632b51cd61777606d574","user":{"login":"usmanm","id":853039,"avatar_url":"https://avatars.githubusercontent.com/u/853039?v=3","gravatar_id":"","url":"https://api.github.com/users/usmanm","html_url":"https://github.com/usmanm","followers_url":"https://api.github.com/users/usmanm/followers","following_url":"https://api.github.com/users/usmanm/following{/other_user}","gists_url":"https://api.github.com/users/usmanm/gists{/gist_id}","starred_url":"https://api.github.com/users/usmanm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/usmanm/subscriptions","organizations_url":"https://api.github.com/users/usmanm/orgs","repos_url":"https://api.github.com/users/usmanm/repos","events_url":"https://api.github.com/users/usmanm/events{/privacy}","received_events_url":"https://api.github.com/users/usmanm/received_events","type":"User","site_admin":false},"body":"No, keep it. I'm now wondering if we should stop when memory is <= half of 256mb (or whatever the combiner working memory is), because in worst case we could look up exactly the number of tuples from the matrel as there exist in the batch?\r\n\r\nMoreover, the size of the tuple buffer is only 64mb which seems too low compared to the 256mb working memory for the batch being read from it?","created_at":"2015-09-24T04:03:58Z","updated_at":"2015-09-24T04:03:58Z","html_url":"https://github.com/pipelinedb/pipelinedb/pull/1184#discussion_r40283967","pull_request_url":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/1184","_links":{"self":{"href":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/comments/40283967"},"html":{"href":"https://github.com/pipelinedb/pipelinedb/pull/1184#discussion_r40283967"},"pull_request":{"href":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/1184"}}},"pull_request":{"url":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/1184","id":45853498,"html_url":"https://github.com/pipelinedb/pipelinedb/pull/1184","diff_url":"https://github.com/pipelinedb/pipelinedb/pull/1184.diff","patch_url":"https://github.com/pipelinedb/pipelinedb/pull/1184.patch","issue_url":"https://api.github.com/repos/pipelinedb/pipelinedb/issues/1184","number":1184,"state":"open","locked":false,"title":"Shard on group","user":{"login":"derekjn","id":384666,"avatar_url":"https://avatars.githubusercontent.com/u/384666?v=3","gravatar_id":"","url":"https://api.github.com/users/derekjn","html_url":"https://github.com/derekjn","followers_url":"https://api.github.com/users/derekjn/followers","following_url":"https://api.github.com/users/derekjn/following{/other_user}","gists_url":"https://api.github.com/users/derekjn/gists{/gist_id}","starred_url":"https://api.github.com/users/derekjn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekjn/subscriptions","organizations_url":"https://api.github.com/users/derekjn/orgs","repos_url":"https://api.github.com/users/derekjn/repos","events_url":"https://api.github.com/users/derekjn/events{/privacy}","received_events_url":"https://api.github.com/users/derekjn/received_events","type":"User","site_admin":false},"body":"Grouped queries are now sharded across combiners by the hash of their group. Some notes on how it works:\r\n\r\n* Workers now attach a hash value to each `TupleBufferSlot` that is written to the `CombinerTupleBuffer`. This is the same hash that the matrel index computes, so some hashing overhead is now offloaded to worker procs.\r\n* For non-grouped queries, the continuous query id is simply used as the hash value, so these types of queries are effectively sharded by id like we've been doing.\r\n","created_at":"2015-09-24T02:32:44Z","updated_at":"2015-09-24T04:03:58Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0f9eae11708d243eaaad435381a45dee07e9e2e8","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/1184/commits","review_comments_url":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/1184/comments","review_comment_url":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pipelinedb/pipelinedb/issues/1184/comments","statuses_url":"https://api.github.com/repos/pipelinedb/pipelinedb/statuses/21fdbe7404b83a5c4778411bf3a4295f80c3a088","head":{"label":"pipelinedb:1183-combiner-sharding","ref":"1183-combiner-sharding","sha":"21fdbe7404b83a5c4778411bf3a4295f80c3a088","user":{"login":"pipelinedb","id":5367380,"avatar_url":"https://avatars.githubusercontent.com/u/5367380?v=3","gravatar_id":"","url":"https://api.github.com/users/pipelinedb","html_url":"https://github.com/pipelinedb","followers_url":"https://api.github.com/users/pipelinedb/followers","following_url":"https://api.github.com/users/pipelinedb/following{/other_user}","gists_url":"https://api.github.com/users/pipelinedb/gists{/gist_id}","starred_url":"https://api.github.com/users/pipelinedb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pipelinedb/subscriptions","organizations_url":"https://api.github.com/users/pipelinedb/orgs","repos_url":"https://api.github.com/users/pipelinedb/repos","events_url":"https://api.github.com/users/pipelinedb/events{/privacy}","received_events_url":"https://api.github.com/users/pipelinedb/received_events","type":"Organization","site_admin":false},"repo":{"id":14702444,"name":"pipelinedb","full_name":"pipelinedb/pipelinedb","owner":{"login":"pipelinedb","id":5367380,"avatar_url":"https://avatars.githubusercontent.com/u/5367380?v=3","gravatar_id":"","url":"https://api.github.com/users/pipelinedb","html_url":"https://github.com/pipelinedb","followers_url":"https://api.github.com/users/pipelinedb/followers","following_url":"https://api.github.com/users/pipelinedb/following{/other_user}","gists_url":"https://api.github.com/users/pipelinedb/gists{/gist_id}","starred_url":"https://api.github.com/users/pipelinedb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pipelinedb/subscriptions","organizations_url":"https://api.github.com/users/pipelinedb/orgs","repos_url":"https://api.github.com/users/pipelinedb/repos","events_url":"https://api.github.com/users/pipelinedb/events{/privacy}","received_events_url":"https://api.github.com/users/pipelinedb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pipelinedb/pipelinedb","description":"The Streaming SQL Database","fork":false,"url":"https://api.github.com/repos/pipelinedb/pipelinedb","forks_url":"https://api.github.com/repos/pipelinedb/pipelinedb/forks","keys_url":"https://api.github.com/repos/pipelinedb/pipelinedb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pipelinedb/pipelinedb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pipelinedb/pipelinedb/teams","hooks_url":"https://api.github.com/repos/pipelinedb/pipelinedb/hooks","issue_events_url":"https://api.github.com/repos/pipelinedb/pipelinedb/issues/events{/number}","events_url":"https://api.github.com/repos/pipelinedb/pipelinedb/events","assignees_url":"https://api.github.com/repos/pipelinedb/pipelinedb/assignees{/user}","branches_url":"https://api.github.com/repos/pipelinedb/pipelinedb/branches{/branch}","tags_url":"https://api.github.com/repos/pipelinedb/pipelinedb/tags","blobs_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/refs{/sha}","trees_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pipelinedb/pipelinedb/statuses/{sha}","languages_url":"https://api.github.com/repos/pipelinedb/pipelinedb/languages","stargazers_url":"https://api.github.com/repos/pipelinedb/pipelinedb/stargazers","contributors_url":"https://api.github.com/repos/pipelinedb/pipelinedb/contributors","subscribers_url":"https://api.github.com/repos/pipelinedb/pipelinedb/subscribers","subscription_url":"https://api.github.com/repos/pipelinedb/pipelinedb/subscription","commits_url":"https://api.github.com/repos/pipelinedb/pipelinedb/commits{/sha}","git_commits_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/commits{/sha}","comments_url":"https://api.github.com/repos/pipelinedb/pipelinedb/comments{/number}","issue_comment_url":"https://api.github.com/repos/pipelinedb/pipelinedb/issues/comments{/number}","contents_url":"https://api.github.com/repos/pipelinedb/pipelinedb/contents/{+path}","compare_url":"https://api.github.com/repos/pipelinedb/pipelinedb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pipelinedb/pipelinedb/merges","archive_url":"https://api.github.com/repos/pipelinedb/pipelinedb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pipelinedb/pipelinedb/downloads","issues_url":"https://api.github.com/repos/pipelinedb/pipelinedb/issues{/number}","pulls_url":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls{/number}","milestones_url":"https://api.github.com/repos/pipelinedb/pipelinedb/milestones{/number}","notifications_url":"https://api.github.com/repos/pipelinedb/pipelinedb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pipelinedb/pipelinedb/labels{/name}","releases_url":"https://api.github.com/repos/pipelinedb/pipelinedb/releases{/id}","created_at":"2013-11-26T00:11:48Z","updated_at":"2015-09-24T03:47:29Z","pushed_at":"2015-09-24T04:02:54Z","git_url":"git://github.com/pipelinedb/pipelinedb.git","ssh_url":"git@github.com:pipelinedb/pipelinedb.git","clone_url":"https://github.com/pipelinedb/pipelinedb.git","svn_url":"https://github.com/pipelinedb/pipelinedb","homepage":"https://www.pipelinedb.com","size":59891,"stargazers_count":651,"watchers_count":651,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":24,"mirror_url":null,"open_issues_count":53,"forks":24,"open_issues":53,"watchers":651,"default_branch":"master"}},"base":{"label":"pipelinedb:master","ref":"master","sha":"4d6fae51c3f6f27d0a90eb0f14316c7d9a909865","user":{"login":"pipelinedb","id":5367380,"avatar_url":"https://avatars.githubusercontent.com/u/5367380?v=3","gravatar_id":"","url":"https://api.github.com/users/pipelinedb","html_url":"https://github.com/pipelinedb","followers_url":"https://api.github.com/users/pipelinedb/followers","following_url":"https://api.github.com/users/pipelinedb/following{/other_user}","gists_url":"https://api.github.com/users/pipelinedb/gists{/gist_id}","starred_url":"https://api.github.com/users/pipelinedb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pipelinedb/subscriptions","organizations_url":"https://api.github.com/users/pipelinedb/orgs","repos_url":"https://api.github.com/users/pipelinedb/repos","events_url":"https://api.github.com/users/pipelinedb/events{/privacy}","received_events_url":"https://api.github.com/users/pipelinedb/received_events","type":"Organization","site_admin":false},"repo":{"id":14702444,"name":"pipelinedb","full_name":"pipelinedb/pipelinedb","owner":{"login":"pipelinedb","id":5367380,"avatar_url":"https://avatars.githubusercontent.com/u/5367380?v=3","gravatar_id":"","url":"https://api.github.com/users/pipelinedb","html_url":"https://github.com/pipelinedb","followers_url":"https://api.github.com/users/pipelinedb/followers","following_url":"https://api.github.com/users/pipelinedb/following{/other_user}","gists_url":"https://api.github.com/users/pipelinedb/gists{/gist_id}","starred_url":"https://api.github.com/users/pipelinedb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pipelinedb/subscriptions","organizations_url":"https://api.github.com/users/pipelinedb/orgs","repos_url":"https://api.github.com/users/pipelinedb/repos","events_url":"https://api.github.com/users/pipelinedb/events{/privacy}","received_events_url":"https://api.github.com/users/pipelinedb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pipelinedb/pipelinedb","description":"The Streaming SQL Database","fork":false,"url":"https://api.github.com/repos/pipelinedb/pipelinedb","forks_url":"https://api.github.com/repos/pipelinedb/pipelinedb/forks","keys_url":"https://api.github.com/repos/pipelinedb/pipelinedb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pipelinedb/pipelinedb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pipelinedb/pipelinedb/teams","hooks_url":"https://api.github.com/repos/pipelinedb/pipelinedb/hooks","issue_events_url":"https://api.github.com/repos/pipelinedb/pipelinedb/issues/events{/number}","events_url":"https://api.github.com/repos/pipelinedb/pipelinedb/events","assignees_url":"https://api.github.com/repos/pipelinedb/pipelinedb/assignees{/user}","branches_url":"https://api.github.com/repos/pipelinedb/pipelinedb/branches{/branch}","tags_url":"https://api.github.com/repos/pipelinedb/pipelinedb/tags","blobs_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/refs{/sha}","trees_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pipelinedb/pipelinedb/statuses/{sha}","languages_url":"https://api.github.com/repos/pipelinedb/pipelinedb/languages","stargazers_url":"https://api.github.com/repos/pipelinedb/pipelinedb/stargazers","contributors_url":"https://api.github.com/repos/pipelinedb/pipelinedb/contributors","subscribers_url":"https://api.github.com/repos/pipelinedb/pipelinedb/subscribers","subscription_url":"https://api.github.com/repos/pipelinedb/pipelinedb/subscription","commits_url":"https://api.github.com/repos/pipelinedb/pipelinedb/commits{/sha}","git_commits_url":"https://api.github.com/repos/pipelinedb/pipelinedb/git/commits{/sha}","comments_url":"https://api.github.com/repos/pipelinedb/pipelinedb/comments{/number}","issue_comment_url":"https://api.github.com/repos/pipelinedb/pipelinedb/issues/comments{/number}","contents_url":"https://api.github.com/repos/pipelinedb/pipelinedb/contents/{+path}","compare_url":"https://api.github.com/repos/pipelinedb/pipelinedb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pipelinedb/pipelinedb/merges","archive_url":"https://api.github.com/repos/pipelinedb/pipelinedb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pipelinedb/pipelinedb/downloads","issues_url":"https://api.github.com/repos/pipelinedb/pipelinedb/issues{/number}","pulls_url":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls{/number}","milestones_url":"https://api.github.com/repos/pipelinedb/pipelinedb/milestones{/number}","notifications_url":"https://api.github.com/repos/pipelinedb/pipelinedb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pipelinedb/pipelinedb/labels{/name}","releases_url":"https://api.github.com/repos/pipelinedb/pipelinedb/releases{/id}","created_at":"2013-11-26T00:11:48Z","updated_at":"2015-09-24T03:47:29Z","pushed_at":"2015-09-24T04:02:54Z","git_url":"git://github.com/pipelinedb/pipelinedb.git","ssh_url":"git@github.com:pipelinedb/pipelinedb.git","clone_url":"https://github.com/pipelinedb/pipelinedb.git","svn_url":"https://github.com/pipelinedb/pipelinedb","homepage":"https://www.pipelinedb.com","size":59891,"stargazers_count":651,"watchers_count":651,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":24,"mirror_url":null,"open_issues_count":53,"forks":24,"open_issues":53,"watchers":651,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/1184"},"html":{"href":"https://github.com/pipelinedb/pipelinedb/pull/1184"},"issue":{"href":"https://api.github.com/repos/pipelinedb/pipelinedb/issues/1184"},"comments":{"href":"https://api.github.com/repos/pipelinedb/pipelinedb/issues/1184/comments"},"review_comments":{"href":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/1184/comments"},"review_comment":{"href":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pipelinedb/pipelinedb/pulls/1184/commits"},"statuses":{"href":"https://api.github.com/repos/pipelinedb/pipelinedb/statuses/21fdbe7404b83a5c4778411bf3a4295f80c3a088"}}}}