{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/3009","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/3009/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/3009/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/3009/events","html_url":"https://github.com/influxdb/influxdb/issues/3009","id":88919414,"number":3009,"title":"Support distributed queries","user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/category/clustering","name":"category/clustering","color":"d111e2"},{"url":"https://api.github.com/repos/influxdb/influxdb/labels/category/queries","name":"category/queries","color":"5319e7"}],"state":"open","locked":false,"assignee":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/influxdb/influxdb/milestones/41","html_url":"https://github.com/influxdb/influxdb/milestones/0.9.2","labels_url":"https://api.github.com/repos/influxdb/influxdb/milestones/41/labels","id":1152021,"number":41,"title":"0.9.2","description":"","creator":{"login":"toddboom","id":53132,"avatar_url":"https://avatars.githubusercontent.com/u/53132?v=3","gravatar_id":"","url":"https://api.github.com/users/toddboom","html_url":"https://github.com/toddboom","followers_url":"https://api.github.com/users/toddboom/followers","following_url":"https://api.github.com/users/toddboom/following{/other_user}","gists_url":"https://api.github.com/users/toddboom/gists{/gist_id}","starred_url":"https://api.github.com/users/toddboom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/toddboom/subscriptions","organizations_url":"https://api.github.com/users/toddboom/orgs","repos_url":"https://api.github.com/users/toddboom/repos","events_url":"https://api.github.com/users/toddboom/events{/privacy}","received_events_url":"https://api.github.com/users/toddboom/received_events","type":"User","site_admin":false},"open_issues":110,"closed_issues":39,"state":"open","created_at":"2015-06-05T20:05:49Z","updated_at":"2015-07-04T22:00:49Z","due_on":"2015-07-22T07:00:00Z","closed_at":null},"comments":4,"created_at":"2015-06-17T05:47:44Z","updated_at":"2015-07-07T18:37:08Z","closed_at":null,"body":"This issue captures design notes and discussions about implementing support for distributed queries -- queries where all shard data is not necessarily local to the node received the query.\r\n\r\n## Initial refactoring\r\n- `engine.go` and `functions.go` to be moved to `tsdb` package with as little change as possible, as first step in refactoring. Looks possible at a high-level.\r\n\r\n## High-level Query sequence\r\n- co-ord node receives query over HTTP.\r\n- JSON encoding of request and response body will be used.\r\n- co-ord node parses query to ensure it's sane. Return error if statement doesn't parse.\r\n- co-ord node ensures database and retention policy exist. It uses the Raft metastore to do this. If they don't exist, return an error.\r\n- co-ord node performs any required auth.\r\n- co-ord node gets shard groups (SGs) for the policy, taking database, retention policy, and time into account.\r\n- SGs determine the list of target shards.\r\n- co-ord node determines list of host nodes for each shard. In some case those nodes may include the co-ord node.\r\n- co-ord node will hit endpoint (more later) for remote nodes, or perform call to local store if target node is co-ord node. Question here about how much, and what layer, this should be abstracted from the co-ord node. By comparison there is no abstraction for writing -- the write code on a co-ord node makes an explicit choice between the `TSDBStore` and `ShardWriter` interfaces when writing:\r\n\r\nhttps://github.com/influxdb/influxdb/blob/master/cluster/points_writer.go#L240\r\n\r\n## Questions about generating queries to shards (remote or local)\r\n- What level of parallelism makes sense? At a minimum all the shards of the earliest SG within the retention policy should be queried in parallel, since they would (potentially) contain data for the same times. Only when those queries complete should the next SGs be queried.\r\n\r\n- How many SGs must be queried depends on any GROUP BY time factor. If the GROUP BY time factor is larger than the duration of SGs for that retention policy, then multiple contiguous SGs will need to be accessed before the GROUP BY results can be calculated. If GROUP BY time factor is less than the duration of SGs, then only 1 SG's set of shards need to be accessed.\r\n\r\n## Proposal about network support\r\n\r\n- HTTP endpoint will be added to each node, `/remote_query`. This endpoint will accept a shard ID and query, and possibly user information.\r\n- The response to the co-ord node will be `influxql.Row` objects (or something similar). The tag sets contained within these rows can be derived from examining the rows. Or does it make more sense for the remote node (or TSDBStore in the case of local) to explicitly include tagsets of sent rows, coupled to `influxql.Row` objects?\r\n- Auth may also be enforced on this endpoint.\r\n- Endpoint shouldn't need to be much more than wrapping a call to the local store. Similar to `LocalMapper` and `RemoteMapper` code that existed before (though this is going to be more complex than simple Mapper-type code).\r\n\r\n## How shards will handle a query\r\n- parse query.\r\n- determine tagsets using parsed query and shard metadata.\r\n- For each tagset in parallel, query the relevant series data using Bolt cursors (to be abstracted away as needed to account for any on-disk datastore format changes) in an iterative manner.\r\n- obey any specified time range.\r\n- execute local mapping functions e.g. sum.\r\n- enforce LIMIT and requested chunking.\r\n- Meta-queries -- such as `SHOW MEASUREMENTS` also need to operate in a distributed fashion.\r\n\r\n## How the co-ord node will handle response from endpoint or local shard\r\n- for a given time range (or time buckets for aggregates) create the set of tagsets it receives.\r\n- gather data from various responses and merge by tagset.\r\n- perform any final mapping e.g. sum by tagset.\r\n- return result to client.\r\n"},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/119296208","html_url":"https://github.com/influxdb/influxdb/issues/3009#issuecomment-119296208","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/3009","id":119296208,"user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"created_at":"2015-07-07T18:37:08Z","updated_at":"2015-07-07T18:37:08Z","body":"![mapperstagsets](https://cloud.githubusercontent.com/assets/536312/8554075/9b76dee4-24b5-11e5-933e-e402682ee811.jpg)\r\n\r\nOutline of Mappers and TagSets."}}