{"action":"created","issue":{"url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/16","labels_url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/16/comments","events_url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/16/events","html_url":"https://github.com/raintank/graphite-kairosdb/issues/16","id":109429994,"number":16,"title":"find not returning all branch nodes","user":{"login":"woodsaj","id":8415072,"avatar_url":"https://avatars.githubusercontent.com/u/8415072?v=3","gravatar_id":"","url":"https://api.github.com/users/woodsaj","html_url":"https://github.com/woodsaj","followers_url":"https://api.github.com/users/woodsaj/followers","following_url":"https://api.github.com/users/woodsaj/following{/other_user}","gists_url":"https://api.github.com/users/woodsaj/gists{/gist_id}","starred_url":"https://api.github.com/users/woodsaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/woodsaj/subscriptions","organizations_url":"https://api.github.com/users/woodsaj/orgs","repos_url":"https://api.github.com/users/woodsaj/repos","events_url":"https://api.github.com/users/woodsaj/events{/privacy}","received_events_url":"https://api.github.com/users/woodsaj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-02T04:55:31Z","updated_at":"2015-10-02T05:00:40Z","closed_at":null,"body":"When performing a find query, ie \"find?query=litmus.*\" it is expected that the response is a full list of both leaf and branch nodes.\r\neg \r\n```\r\n[\r\n  {\"text\": \"bing_com\", \"expandable\": 1, \"leaf\": 0, \"id\": \"litmus.bing_com\", \"allowChildren\": 1},\r\n  {\"text\": \"bloomberg_com\", \"expandable\": 1, \"leaf\": 0, \"id\": \"litmus.bloomberg_com\", \"allowChildren\": 1},\r\n....\r\n]\r\n```\r\n\r\nThe current elasticsearch schema and query is extremely inefficient at returning the branch nodes.  In order to get the list of branch nodes it needs to get a list of all series that match the query, and iterate through them to find the unique branches.\r\n\r\neg, the following series are all identifying the same branch for the query \"litmus.*\"\r\n```\r\nlitmus.bing_com.singapore.http.avg\r\nlitmus.bing_com.singapore.http.dns\r\nlitmus.bing_com.singapore.http.connect\r\nlitmus.bing_com.singapore.http.recv\r\nlitmus.bing_com.new-york.http.avg\r\nlitmus.bing_com.new-york.http.dns\r\nlitmus.bing_com.new-york.http.connect\r\nlitmus.bing_com.new-york.http.recv\r\n```\r\n\r\nSome users of the system have many 10s of thousands of unique series and fetching all of these series names to compute a dozen branches is not at all scale-able.  In fact, we currently limit the the Easlticsearch response to 10K documents.  So if the user has more then 10K series, they may not get all of the branch nodes in the response.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/comments/144922161","html_url":"https://github.com/raintank/graphite-kairosdb/issues/16#issuecomment-144922161","issue_url":"https://api.github.com/repos/raintank/graphite-kairosdb/issues/16","id":144922161,"user":{"login":"woodsaj","id":8415072,"avatar_url":"https://avatars.githubusercontent.com/u/8415072?v=3","gravatar_id":"","url":"https://api.github.com/users/woodsaj","html_url":"https://github.com/woodsaj","followers_url":"https://api.github.com/users/woodsaj/followers","following_url":"https://api.github.com/users/woodsaj/following{/other_user}","gists_url":"https://api.github.com/users/woodsaj/gists{/gist_id}","starred_url":"https://api.github.com/users/woodsaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/woodsaj/subscriptions","organizations_url":"https://api.github.com/users/woodsaj/orgs","repos_url":"https://api.github.com/users/woodsaj/repos","events_url":"https://api.github.com/users/woodsaj/events{/privacy}","received_events_url":"https://api.github.com/users/woodsaj/received_events","type":"User","site_admin":false},"created_at":"2015-10-02T05:00:40Z","updated_at":"2015-10-02T05:00:40Z","body":"A proposed solution is to do the following.\r\n\r\nChange the metricDef schema to include each part of the series name as an indexed value.\r\neg.\r\n```\r\n\"p0\": \"litmus\",\r\n\"p1\": \"bing_com\",\r\n\"p2\": \"singapore\",\r\n\"p3\": \"http\",\r\n\"p4\": \"avg\",\r\n\"pLength\": 5,\r\n```\r\n\r\nThis will allow more optimized queries that can use Elasticsearch aggregation methods to just return the set of unique branches, and not have to return all series names.\r\n"}}