{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/libbeat/issues/336","labels_url":"https://api.github.com/repos/elastic/libbeat/issues/336/labels{/name}","comments_url":"https://api.github.com/repos/elastic/libbeat/issues/336/comments","events_url":"https://api.github.com/repos/elastic/libbeat/issues/336/events","html_url":"https://github.com/elastic/libbeat/issues/336","id":119482683,"number":336,"title":"Proposal Filtering Implementation ","user":{"login":"monicasarbu","id":11757159,"avatar_url":"https://avatars.githubusercontent.com/u/11757159?v=3","gravatar_id":"","url":"https://api.github.com/users/monicasarbu","html_url":"https://github.com/monicasarbu","followers_url":"https://api.github.com/users/monicasarbu/followers","following_url":"https://api.github.com/users/monicasarbu/following{/other_user}","gists_url":"https://api.github.com/users/monicasarbu/gists{/gist_id}","starred_url":"https://api.github.com/users/monicasarbu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/monicasarbu/subscriptions","organizations_url":"https://api.github.com/users/monicasarbu/orgs","repos_url":"https://api.github.com/users/monicasarbu/repos","events_url":"https://api.github.com/users/monicasarbu/events{/privacy}","received_events_url":"https://api.github.com/users/monicasarbu/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/libbeat/labels/discuss","name":"discuss","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-30T12:28:02Z","updated_at":"2015-11-30T15:29:09Z","closed_at":null,"body":"Alternative names: conditionals\r\n\r\nThe goal here is to reduce the number of exported events and fields, before they are sent over the network. It is not to add or change fields, that can be achieved later by Logstash. \r\n\r\n__Use cases to solve__\r\n\r\n - Filebeat: Ship only log lines that don’t match a certain regexp: https://github.com/elastic/filebeat/issues/78\r\n - Packetbeat: Drop all 200 OK transactions (not yet requested)\r\n - Topbeat: Export all metrics with higher cpu usage (not yet requested)\r\n\r\n\r\n__Requirements__\r\n\r\n- define a generic filtering that is implemented in libbeat and used by all the Beats, instead of having a specific filtering implementation for each Beat. \r\n- be able to use event fields (e.g. http.response.code) in the condition\r\n- be able to drop the event or remove a certain field if the condition is fulfilled \r\n\r\n__Disadvantages__\r\n\r\nIf we want to use generic conditions in the filtering, then we need to apply the filtering on the created event object. In other words, we need to build the event object before running the filtering condition. Building the event object might be expensive as we need to calculate all the fields and if we just decide to drop the event, then the time spent is useless.\r\n\r\nTo improve this, I propose to decide about a minimum set of fields to be exported by each Beat. Additional fields can be added by enabling options in configuration file. These options will have to be implemented in each Beat but have the advantage of having the maximum performance. For example, Filebeat can implement an option like “exclude_files” a lot more efficient than it can be done in the generic filtering.\r\n\r\n__Proposal__\r\n\r\nThe filters are configured as a YAML list, each with an _if_ and _do_ blocks. The _if_ block is optional. The conditions are modeled roughly after the Elasticsearch filtering. Each action can receive parameters.\r\n\r\nThe filtering rules are executed in libbeat before publishing the event.\r\n\r\n<pre>\r\nfilters:\r\n    - if:\r\n        condition\r\n      do:\r\n        - action1\r\n        - action2\r\n\r\n    - if:\r\n        condition\r\n      do\r\n        - action1\r\n        - action2\r\n        - action3\r\n\r\n</pre>\r\n\r\n__Supported actions:__\r\n\r\n- drop_event (phase 1)\r\n- drop_fields (phase 2)\r\n\r\n__Conditions:__\r\n\r\n- equals\r\n- contains\r\n- regexp\r\n- range\r\n- and (phase 2)\r\n- or (phase 2)\r\n- not (phase 2)\r\n\r\n\r\n__Examples__\r\n\r\n- Filebeat drops lines starting with DBG\r\n\r\n<pre>\r\nfilters:\r\n    - if:\r\n        regexp:\r\n            message: “^DBG” \r\n      do:\r\n          - drop_event: true\r\n</pre>\r\n\r\n- Filebeat drops line starting with DBG for apache logs only\r\n\r\n<pre>\r\nfilters:\r\n    - if:\r\n        and:\r\n            equals:\r\n                type: apache\r\n            regexp:\r\n                message: “^DBG”\r\n      do:\r\n          - drop_event: true\r\n\r\n</pre>\r\n\r\nin prospector:\r\n<pre>\r\n\t\tdocument_type: apache\r\n</pre>\r\n\t\r\n- Drop all 200 OKs in Packetbeat\r\n\r\n<pre>\r\n\r\nfilters:\r\n      - if:\r\n          equals: \r\n\t        “http.response.code”: 200\r\n       do:\r\n           - drop_event: true\r\n\r\n</pre>\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/libbeat/issues/comments/160661380","html_url":"https://github.com/elastic/libbeat/issues/336#issuecomment-160661380","issue_url":"https://api.github.com/repos/elastic/libbeat/issues/336","id":160661380,"user":{"login":"kimchy","id":41300,"avatar_url":"https://avatars.githubusercontent.com/u/41300?v=3","gravatar_id":"","url":"https://api.github.com/users/kimchy","html_url":"https://github.com/kimchy","followers_url":"https://api.github.com/users/kimchy/followers","following_url":"https://api.github.com/users/kimchy/following{/other_user}","gists_url":"https://api.github.com/users/kimchy/gists{/gist_id}","starred_url":"https://api.github.com/users/kimchy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kimchy/subscriptions","organizations_url":"https://api.github.com/users/kimchy/orgs","repos_url":"https://api.github.com/users/kimchy/repos","events_url":"https://api.github.com/users/kimchy/events{/privacy}","received_events_url":"https://api.github.com/users/kimchy/received_events","type":"User","site_admin":false},"created_at":"2015-11-30T15:29:08Z","updated_at":"2015-11-30T15:29:08Z","body":"I like the suggestion, my comment is that we should aim to make the most common use case as concise as possible. If we think that dropping events would be the most common one, then the suggested syntax is a bit verbose? \r\n\r\nI don't have a good solution, but conceptually, @ruflin is an interesting one where we can have a mini conditional language here, that is generic, and then flip the action with the condition, something like this:\r\n\r\n```\r\ndrop_event: message~=^DBG && type=apache\r\n```\r\n\r\nwe can then use this conditional syntax and code element in several other places, for example, \"drop_field\", or \"sampling\" and so on.\r\n\r\nWhile I write this, even if we go with more structured language, feels to me that it is better to flip the action and the condition, so we can use it as an element in other places code wise, so something like this:\r\n\r\n```\r\ndrop_event:\r\n      equals:\r\n            type: apache\r\n```\r\n\r\n```\r\ndrop_event:\r\n        and:\r\n            equals:\r\n                type: apache\r\n            regexp:\r\n                message: “^DBG”\r\n```\r\n\r\nThen we can also do (in the future):\r\n\r\n```\r\nsample:\r\n     when:\r\n         equals:\r\n               type: apache\r\n     interval: 1s\r\n     ....\r\n```\r\n\r\nAnd the conditional language remains the same.\r\n\r\nwe could eventually evolve to both, similar to how in ES you can do both structured and unstructured, it is just a \"conditional DSL\", and we can start with the structured one to begin with, and slowly evolve? "}}