{"action":"created","issue":{"url":"https://api.github.com/repos/logsearch/logsearch-for-cloudfoundry/issues/57","labels_url":"https://api.github.com/repos/logsearch/logsearch-for-cloudfoundry/issues/57/labels{/name}","comments_url":"https://api.github.com/repos/logsearch/logsearch-for-cloudfoundry/issues/57/comments","events_url":"https://api.github.com/repos/logsearch/logsearch-for-cloudfoundry/issues/57/events","html_url":"https://github.com/logsearch/logsearch-for-cloudfoundry/issues/57","id":96762929,"number":57,"title":"whitelisting firehose messages by org","user":{"login":"bonzofenix","id":317403,"avatar_url":"https://avatars.githubusercontent.com/u/317403?v=3","gravatar_id":"","url":"https://api.github.com/users/bonzofenix","html_url":"https://github.com/bonzofenix","followers_url":"https://api.github.com/users/bonzofenix/followers","following_url":"https://api.github.com/users/bonzofenix/following{/other_user}","gists_url":"https://api.github.com/users/bonzofenix/gists{/gist_id}","starred_url":"https://api.github.com/users/bonzofenix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bonzofenix/subscriptions","organizations_url":"https://api.github.com/users/bonzofenix/orgs","repos_url":"https://api.github.com/users/bonzofenix/repos","events_url":"https://api.github.com/users/bonzofenix/events{/privacy}","received_events_url":"https://api.github.com/users/bonzofenix/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-07-23T08:42:17Z","updated_at":"2015-07-23T09:40:27Z","closed_at":null,"body":"We are currently facing the following situation:\r\n\r\n- Stress tests on some apps can take down our ELK cluster.\r\n- We want logsearch and l4cf only consuming operations apps logs that are in some specific cf orgs.\r\n\r\nFor this reason we would like to have l4cf only parsing logs from whitelisted Orgs.\r\n@mrdavidlaing, @malston I would like your input on this.\r\nWhere do you think we should implement this? Among the options that I thought are:\r\n- Firehose-to-syslog\r\n- Cloudfoundry-ingrestor\r\n- log_parser \r\n- Firehose\r\n"},"comment":{"url":"https://api.github.com/repos/logsearch/logsearch-for-cloudfoundry/issues/comments/124034720","html_url":"https://github.com/logsearch/logsearch-for-cloudfoundry/issues/57#issuecomment-124034720","issue_url":"https://api.github.com/repos/logsearch/logsearch-for-cloudfoundry/issues/57","id":124034720,"user":{"login":"mrdavidlaing","id":227505,"avatar_url":"https://avatars.githubusercontent.com/u/227505?v=3","gravatar_id":"","url":"https://api.github.com/users/mrdavidlaing","html_url":"https://github.com/mrdavidlaing","followers_url":"https://api.github.com/users/mrdavidlaing/followers","following_url":"https://api.github.com/users/mrdavidlaing/following{/other_user}","gists_url":"https://api.github.com/users/mrdavidlaing/gists{/gist_id}","starred_url":"https://api.github.com/users/mrdavidlaing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrdavidlaing/subscriptions","organizations_url":"https://api.github.com/users/mrdavidlaing/orgs","repos_url":"https://api.github.com/users/mrdavidlaing/repos","events_url":"https://api.github.com/users/mrdavidlaing/events{/privacy}","received_events_url":"https://api.github.com/users/mrdavidlaing/received_events","type":"User","site_admin":false},"created_at":"2015-07-23T09:40:27Z","updated_at":"2015-07-23T09:40:27Z","body":"Given the failure scenarios, I think its best to try filter the logs before they hit the queue.\r\n\r\nCurrently, the logs flow like this:\r\n\r\nfirehose  -->   [ingestor_cloudfoundry-firehose_ctl / firehose-to-syslog](https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/logsearch-for-cloudfoundry-boshrelease/jobs/ingestor_cloudfoundry-firehose/templates/bin/ingestor_cloudfoundry-firehose_ctl#L37) --> [ingestor_cloudfoundry-firehose_ctl / logstash](https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/logsearch-for-cloudfoundry-boshrelease/jobs/ingestor_cloudfoundry-firehose/templates/bin/ingestor_cloudfoundry-firehose_ctl#L17) --> queue\r\n\r\nWe could make the logstash part of the `ingestor_cloudfoundry-firehose` job configurable with extra logstash config.  \r\n\r\nI think this would be a good place to use logstash's [`drop` filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html)\r\n\r\nI propose we update the `ingestor_cloudfoundry-firehose` job to support the  `filters` property like the upstream [`ingestor_syslog` / logstash_ingestor.filters](https://github.com/logsearch/logsearch-boshrelease/blob/develop/jobs/ingestor_syslog/spec#L20) job, which would then allow you to configure deployment specific drop rules via your deploy manifest:\r\n\r\n```\r\nproperties:\r\n  ingestor_cloudfoundry-firehose:\r\n      filters: |\r\n                if [loglevel] == \"debug\" {\r\n                   drop { }\r\n                } \r\n```\r\n"}}