{"action":"created","issue":{"url":"https://api.github.com/repos/raintank/raintank-metric/issues/11","labels_url":"https://api.github.com/repos/raintank/raintank-metric/issues/11/labels{/name}","comments_url":"https://api.github.com/repos/raintank/raintank-metric/issues/11/comments","events_url":"https://api.github.com/repos/raintank/raintank-metric/issues/11/events","html_url":"https://github.com/raintank/raintank-metric/issues/11","id":97509495,"number":11,"title":"future of raintank-metric, use something else?","user":{"login":"Dieterbe","id":20774,"avatar_url":"https://avatars.githubusercontent.com/u/20774?v=3","gravatar_id":"","url":"https://api.github.com/users/Dieterbe","html_url":"https://github.com/Dieterbe","followers_url":"https://api.github.com/users/Dieterbe/followers","following_url":"https://api.github.com/users/Dieterbe/following{/other_user}","gists_url":"https://api.github.com/users/Dieterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/Dieterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dieterbe/subscriptions","organizations_url":"https://api.github.com/users/Dieterbe/orgs","repos_url":"https://api.github.com/users/Dieterbe/repos","events_url":"https://api.github.com/users/Dieterbe/events{/privacy}","received_events_url":"https://api.github.com/users/Dieterbe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/raintank/raintank-metric/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"Dieterbe","id":20774,"avatar_url":"https://avatars.githubusercontent.com/u/20774?v=3","gravatar_id":"","url":"https://api.github.com/users/Dieterbe","html_url":"https://github.com/Dieterbe","followers_url":"https://api.github.com/users/Dieterbe/followers","following_url":"https://api.github.com/users/Dieterbe/following{/other_user}","gists_url":"https://api.github.com/users/Dieterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/Dieterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dieterbe/subscriptions","organizations_url":"https://api.github.com/users/Dieterbe/orgs","repos_url":"https://api.github.com/users/Dieterbe/repos","events_url":"https://api.github.com/users/Dieterbe/events{/privacy}","received_events_url":"https://api.github.com/users/Dieterbe/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/raintank/raintank-metric/milestones/1","html_url":"https://github.com/raintank/raintank-metric/milestones/0.19","labels_url":"https://api.github.com/repos/raintank/raintank-metric/milestones/1/labels","id":1229640,"number":1,"title":"0.19","description":null,"creator":{"login":"woodsaj","id":8415072,"avatar_url":"https://avatars.githubusercontent.com/u/8415072?v=3","gravatar_id":"","url":"https://api.github.com/users/woodsaj","html_url":"https://github.com/woodsaj","followers_url":"https://api.github.com/users/woodsaj/followers","following_url":"https://api.github.com/users/woodsaj/following{/other_user}","gists_url":"https://api.github.com/users/woodsaj/gists{/gist_id}","starred_url":"https://api.github.com/users/woodsaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/woodsaj/subscriptions","organizations_url":"https://api.github.com/users/woodsaj/orgs","repos_url":"https://api.github.com/users/woodsaj/repos","events_url":"https://api.github.com/users/woodsaj/events{/privacy}","received_events_url":"https://api.github.com/users/woodsaj/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":0,"state":"open","created_at":"2015-07-28T15:29:14Z","updated_at":"2015-07-28T15:29:15Z","due_on":null,"closed_at":null},"comments":20,"created_at":"2015-07-27T17:04:43Z","updated_at":"2015-08-10T16:01:38Z","closed_at":null,"body":"please help me fill this in. we need to agree on what our requirements/desires are before talking about using other tools\r\n\r\n## current requirements?\r\n* safely relay metrics from our queue into storage and ES without losing data in case we can't safely deliver\r\n* decode messages from our custom format used in rabbitmq (but i suppose we could also store them differently in rabbit?)\r\n* encode messages into our custom format, to be stored in ES\r\n\r\n## possible future requirements\r\n* real time aggregation\r\n* real time processing/alerting (I personally don't think we need to be too concerned about this just yet.  once we have high performance/scalability requirements we'll probably use a dedicated real time processing framework like spark/storm/heron/...)\r\n\r\n## questions\r\n* can we write our own decode, encode, processor plugins in Go, in heka?\r\n* can somebody describe what we do with ES from the raintank-metric/rabbitmq perspective and how dependent this is on the main storage backend? like if kairosdb is down, can we or must we still update ES? if ES is down, can or must we still write to kairos?\r\n* does rabbitmq support multiple readers of the same data, and does it maintain what has been acked by which reader?\r\n"},"comment":{"url":"https://api.github.com/repos/raintank/raintank-metric/issues/comments/129507471","html_url":"https://github.com/raintank/raintank-metric/issues/11#issuecomment-129507471","issue_url":"https://api.github.com/repos/raintank/raintank-metric/issues/11","id":129507471,"user":{"login":"woodsaj","id":8415072,"avatar_url":"https://avatars.githubusercontent.com/u/8415072?v=3","gravatar_id":"","url":"https://api.github.com/users/woodsaj","html_url":"https://github.com/woodsaj","followers_url":"https://api.github.com/users/woodsaj/followers","following_url":"https://api.github.com/users/woodsaj/following{/other_user}","gists_url":"https://api.github.com/users/woodsaj/gists{/gist_id}","starred_url":"https://api.github.com/users/woodsaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/woodsaj/subscriptions","organizations_url":"https://api.github.com/users/woodsaj/orgs","repos_url":"https://api.github.com/users/woodsaj/repos","events_url":"https://api.github.com/users/woodsaj/events{/privacy}","received_events_url":"https://api.github.com/users/woodsaj/received_events","type":"User","site_admin":false},"created_at":"2015-08-10T16:01:38Z","updated_at":"2015-08-10T16:01:38Z","body":"firstly, yes i can migrate the collector-ctrl logic to raintank-metric.\r\n\r\nMy biggest motivation for keeping everything in one process is to reduce the number of encoding/decoding steps needed to process messages as it is consumes a lot of CPU.  By removing the queue from the topology I would expect to see at least a 30% reduction in CPU usage (this usage is currently split between grafana-server and raintank-metric).  \r\n\r\nThe main benefit of a producer/consumer topology is to allow horizontally scaling work across many worker nodes.  However as we see from the current production resource usage, the resource usage of grafana-server is about the same as for raintank-metric, with grafana-server using slightly more resources due to the alertingJobs.  If we split ingestion of metrics from writing to TSDB, as metric volume increases we would need to scale both parts at the same time anyway.  This of course changes if we add new processing steps or output destinations.\r\n\r\nSo the way i see things working is:\r\n1. receive json encoded metrics payload from collector.\r\n2. commit the raw payload ([]bytes) to disk using an append only commit log\r\n3. unmarshal the payload\r\n4. validate the metrics (we force the orgId of metrics for payloads sent from private collectors)\r\n5. send to ES and TSDB\r\n6. update commit log position so if we restart, we know where to restart from."}}