{"action":"created","comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/comments/29999869","id":29999869,"diff_hunk":"@@ -0,0 +1,228 @@\n+﻿# encoding: utf-8\n+\n+require \"logstash/filters/base\"\n+require \"logstash/namespace\"\n+require \"thread\"\n+\n+#\n+# The aim of this filter is to aggregate informations available among several events (typically log lines) belonging to a same task,\n+# and finally push aggregated information into final task event.\n+# \n+# An example of use can be:  \n+#\n+# * with this given data :  \n+# [source,log]\n+# ----------------------------------\n+#     INFO - 12345 - TASK_START - start message\n+#     INFO - 12345 - DAO - MyDao.findById - 12\n+#     INFO - 12345 - DAO - MyDao.findAll - 34\n+#     INFO - 12345 - TASK_END - end message\n+# ----------------------------------\n+#\n+# * you can aggregate \"dao duration\" with this configuration :  \n+# [source,ruby]\n+# ----------------------------------\n+#     filter {\n+#         grok {\n+#             match => [ \"message\", \"%{LOGLEVEL:loglevel} - %{NOTSPACE:requestid} - %{NOTSPACE:logger} - %{GREEDYDATA:msg}\" ]\n+#         }\n+#     \n+#         if [logger] == \"TASK_START\" {\n+#             aggregate {\n+#                 task_id => \"%{requestid}\"\n+#                 code => \"map['dao_duration'] = 0\"\n+#                 map_action => \"create\"\n+#             }\n+#         }\n+#     \n+#         if [logger] == \"DAO\" {\n+#             grok {\n+#                 match => [ \"msg\", \"%{JAVACLASS:dao_call} - %{INT:duration:int}\" ]\n+#             }\n+#             aggregate {\n+#                 task_id => \"%{requestid}\"\n+#                 code => \"map['dao_duration'] += event['duration']\"\n+#                 map_action => \"update\"\n+#             }\n+#         }\n+#     \n+#         if [logger] == \"TASK_END\" {\n+#             aggregate {\n+#                 task_id => \"%{requestid}\"\n+#                 code => \"event.to_hash.merge!(map)\"\n+#                 map_action => \"update\"\n+#                 end_of_task => true\n+#                 timeout => 120\n+#             }\n+#         }\n+#     }\n+# ----------------------------------\n+#\n+# * the final event then looks like :  \n+# [source,json]\n+# ----------------------------------\n+# {\n+#         \"message\" => \"INFO - 12345 - TASK_END - end message\",\n+#    \"dao_duration\" => 46\n+# }\n+# ----------------------------------\n+#\n+#\n+# How it works :\n+# - the filter needs a \"task_id\" to correlate events (log lines) of a same task\n+# - at the task beggining, filter creates a map, attached to task_id\n+# - for each event, you can execute code using 'event' and 'map' (for instance, copy an event field to map)\n+# - in the final event, you can execute a last code (for instance, add map data to final event)\n+# - after the final event, the map attached to task is deleted\n+# - in one filter configuration, it is recommanded to define a timeout option to protect the feature against unterminated tasks. It tells the filter to delete expired maps\n+# - if no timeout is defined, by default, all maps older than 1800 seconds are automatically deleted\n+#\n+#\n+class LogStash::Filters::Aggregate < LogStash::Filters::Base\n+\n+\tconfig_name \"aggregate\"\n+\n+\t# The expression defining task ID to correlate logs. +\n+\t# This value must uniquely identify the task in the system +\n+\t# Example value : \"%{application}%{my_task_id}\" +\n+\tconfig :task_id, :validate => :string, :required => true\n+\n+\t# The code to execute to update map, using current event. +\n+\t# Or on the contrary, the code to execute to update event, using current map. +\n+\t# You will have a 'map' variable and an 'event' variable available (that is the event itself). +\n+\t# Example value : \"map['dao_duration'] += event['duration']\" +\n+\tconfig :code, :validate => :string, :required => true\n+\n+\t# Tell the filter what to do with aggregate map (default :  \"create_or_update\"). +\n+\t# create: create the map, and execute the code only if map wasn't created before +\n+\t# update: doesn't create the map, and execute the code only if map was created before +\n+\t# create_or_update: create the map if it wasn't created before, execute the code in all cases +\n+\tconfig :map_action, :validate => :string, :default => \"create_or_update\"","path":"lib/logstash/filters/aggregate.rb","position":100,"original_position":100,"commit_id":"4d546e65e7e9a54a90eb3575bb41a08dab367a7c","original_commit_id":"9cdc432f5f7c03ef8e91a329dab5ca2c76f786d9","user":{"login":"purbon","id":68540,"avatar_url":"https://avatars.githubusercontent.com/u/68540?v=3","gravatar_id":"","url":"https://api.github.com/users/purbon","html_url":"https://github.com/purbon","followers_url":"https://api.github.com/users/purbon/followers","following_url":"https://api.github.com/users/purbon/following{/other_user}","gists_url":"https://api.github.com/users/purbon/gists{/gist_id}","starred_url":"https://api.github.com/users/purbon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/purbon/subscriptions","organizations_url":"https://api.github.com/users/purbon/orgs","repos_url":"https://api.github.com/users/purbon/repos","events_url":"https://api.github.com/users/purbon/events{/privacy}","received_events_url":"https://api.github.com/users/purbon/received_events","type":"User","site_admin":false},"body":"I guess my doubt was more on the need of exposing the map or not, but for now I think we can keep it like this :+1:, ok? Then we can improve this, if interesting later on.","created_at":"2015-05-10T08:13:26Z","updated_at":"2015-05-10T08:13:26Z","html_url":"https://github.com/logstash-plugins/logstash-filter-aggregate/pull/1#discussion_r29999869","pull_request_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/1","_links":{"self":{"href":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/comments/29999869"},"html":{"href":"https://github.com/logstash-plugins/logstash-filter-aggregate/pull/1#discussion_r29999869"},"pull_request":{"href":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/1"}}},"pull_request":{"url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/1","id":34467724,"html_url":"https://github.com/logstash-plugins/logstash-filter-aggregate/pull/1","diff_url":"https://github.com/logstash-plugins/logstash-filter-aggregate/pull/1.diff","patch_url":"https://github.com/logstash-plugins/logstash-filter-aggregate/pull/1.patch","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/issues/1","number":1,"state":"open","locked":false,"title":"add logstash-filter-aggregate code","user":{"login":"fbaligand","id":3750300,"avatar_url":"https://avatars.githubusercontent.com/u/3750300?v=3","gravatar_id":"","url":"https://api.github.com/users/fbaligand","html_url":"https://github.com/fbaligand","followers_url":"https://api.github.com/users/fbaligand/followers","following_url":"https://api.github.com/users/fbaligand/following{/other_user}","gists_url":"https://api.github.com/users/fbaligand/gists{/gist_id}","starred_url":"https://api.github.com/users/fbaligand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fbaligand/subscriptions","organizations_url":"https://api.github.com/users/fbaligand/orgs","repos_url":"https://api.github.com/users/fbaligand/repos","events_url":"https://api.github.com/users/fbaligand/events{/privacy}","received_events_url":"https://api.github.com/users/fbaligand/received_events","type":"User","site_admin":false},"body":"Here is my code for logstash-filter-aggregate plugin.\r\nPlease review it and tell me what you think about.\r\n\r\nFabien","created_at":"2015-04-30T14:56:22Z","updated_at":"2015-05-10T08:13:26Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6921721c12b6be90ec3f60aee730e6657d39831b","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/1/commits","review_comments_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/1/comments","review_comment_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/comments{/number}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/issues/1/comments","statuses_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/statuses/4d546e65e7e9a54a90eb3575bb41a08dab367a7c","head":{"label":"fbaligand:master","ref":"master","sha":"4d546e65e7e9a54a90eb3575bb41a08dab367a7c","user":{"login":"fbaligand","id":3750300,"avatar_url":"https://avatars.githubusercontent.com/u/3750300?v=3","gravatar_id":"","url":"https://api.github.com/users/fbaligand","html_url":"https://github.com/fbaligand","followers_url":"https://api.github.com/users/fbaligand/followers","following_url":"https://api.github.com/users/fbaligand/following{/other_user}","gists_url":"https://api.github.com/users/fbaligand/gists{/gist_id}","starred_url":"https://api.github.com/users/fbaligand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fbaligand/subscriptions","organizations_url":"https://api.github.com/users/fbaligand/orgs","repos_url":"https://api.github.com/users/fbaligand/repos","events_url":"https://api.github.com/users/fbaligand/events{/privacy}","received_events_url":"https://api.github.com/users/fbaligand/received_events","type":"User","site_admin":false},"repo":{"id":34081725,"name":"logstash-filter-aggregate","full_name":"fbaligand/logstash-filter-aggregate","owner":{"login":"fbaligand","id":3750300,"avatar_url":"https://avatars.githubusercontent.com/u/3750300?v=3","gravatar_id":"","url":"https://api.github.com/users/fbaligand","html_url":"https://github.com/fbaligand","followers_url":"https://api.github.com/users/fbaligand/followers","following_url":"https://api.github.com/users/fbaligand/following{/other_user}","gists_url":"https://api.github.com/users/fbaligand/gists{/gist_id}","starred_url":"https://api.github.com/users/fbaligand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fbaligand/subscriptions","organizations_url":"https://api.github.com/users/fbaligand/orgs","repos_url":"https://api.github.com/users/fbaligand/repos","events_url":"https://api.github.com/users/fbaligand/events{/privacy}","received_events_url":"https://api.github.com/users/fbaligand/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/fbaligand/logstash-filter-aggregate","description":"The aim of this filter is to aggregate informations available among several events (typically log lines) belonging to a same task, and finally push aggregated information into final task event.","fork":true,"url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate","forks_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/forks","keys_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/teams","hooks_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/hooks","issue_events_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/issues/events{/number}","events_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/events","assignees_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/assignees{/user}","branches_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/branches{/branch}","tags_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/tags","blobs_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/git/refs{/sha}","trees_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/statuses/{sha}","languages_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/languages","stargazers_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/stargazers","contributors_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/contributors","subscribers_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/subscribers","subscription_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/subscription","commits_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/commits{/sha}","git_commits_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/git/commits{/sha}","comments_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/comments{/number}","issue_comment_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/issues/comments{/number}","contents_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/contents/{+path}","compare_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/merges","archive_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/downloads","issues_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/issues{/number}","pulls_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/pulls{/number}","milestones_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/milestones{/number}","notifications_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/labels{/name}","releases_url":"https://api.github.com/repos/fbaligand/logstash-filter-aggregate/releases{/id}","created_at":"2015-04-16T21:32:01Z","updated_at":"2015-05-08T15:21:18Z","pushed_at":"2015-05-08T15:21:18Z","git_url":"git://github.com/fbaligand/logstash-filter-aggregate.git","ssh_url":"git@github.com:fbaligand/logstash-filter-aggregate.git","clone_url":"https://github.com/fbaligand/logstash-filter-aggregate.git","svn_url":"https://github.com/fbaligand/logstash-filter-aggregate","homepage":"","size":148,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"logstash-plugins:master","ref":"master","sha":"eeae8a3704639d4ead32a0f0e83938f8d9573228","user":{"login":"logstash-plugins","id":8961934,"avatar_url":"https://avatars.githubusercontent.com/u/8961934?v=3","gravatar_id":"","url":"https://api.github.com/users/logstash-plugins","html_url":"https://github.com/logstash-plugins","followers_url":"https://api.github.com/users/logstash-plugins/followers","following_url":"https://api.github.com/users/logstash-plugins/following{/other_user}","gists_url":"https://api.github.com/users/logstash-plugins/gists{/gist_id}","starred_url":"https://api.github.com/users/logstash-plugins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/logstash-plugins/subscriptions","organizations_url":"https://api.github.com/users/logstash-plugins/orgs","repos_url":"https://api.github.com/users/logstash-plugins/repos","events_url":"https://api.github.com/users/logstash-plugins/events{/privacy}","received_events_url":"https://api.github.com/users/logstash-plugins/received_events","type":"Organization","site_admin":false},"repo":{"id":33984534,"name":"logstash-filter-aggregate","full_name":"logstash-plugins/logstash-filter-aggregate","owner":{"login":"logstash-plugins","id":8961934,"avatar_url":"https://avatars.githubusercontent.com/u/8961934?v=3","gravatar_id":"","url":"https://api.github.com/users/logstash-plugins","html_url":"https://github.com/logstash-plugins","followers_url":"https://api.github.com/users/logstash-plugins/followers","following_url":"https://api.github.com/users/logstash-plugins/following{/other_user}","gists_url":"https://api.github.com/users/logstash-plugins/gists{/gist_id}","starred_url":"https://api.github.com/users/logstash-plugins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/logstash-plugins/subscriptions","organizations_url":"https://api.github.com/users/logstash-plugins/orgs","repos_url":"https://api.github.com/users/logstash-plugins/repos","events_url":"https://api.github.com/users/logstash-plugins/events{/privacy}","received_events_url":"https://api.github.com/users/logstash-plugins/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/logstash-plugins/logstash-filter-aggregate","description":"The aim of this filter is to aggregate informations available among several events (typically log lines) belonging to a same task, and finally push aggregated information into final task event.","fork":false,"url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate","forks_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/forks","keys_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/keys{/key_id}","collaborators_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/teams","hooks_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/hooks","issue_events_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/issues/events{/number}","events_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/events","assignees_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/assignees{/user}","branches_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/branches{/branch}","tags_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/tags","blobs_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/git/refs{/sha}","trees_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/git/trees{/sha}","statuses_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/statuses/{sha}","languages_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/languages","stargazers_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/stargazers","contributors_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/contributors","subscribers_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/subscribers","subscription_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/subscription","commits_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/commits{/sha}","git_commits_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/git/commits{/sha}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/comments{/number}","issue_comment_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/issues/comments{/number}","contents_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/contents/{+path}","compare_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/compare/{base}...{head}","merges_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/merges","archive_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/downloads","issues_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/issues{/number}","pulls_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls{/number}","milestones_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/milestones{/number}","notifications_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/labels{/name}","releases_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/releases{/id}","created_at":"2015-04-15T09:16:31Z","updated_at":"2015-04-15T09:16:31Z","pushed_at":"2015-05-08T15:21:19Z","git_url":"git://github.com/logstash-plugins/logstash-filter-aggregate.git","ssh_url":"git@github.com:logstash-plugins/logstash-filter-aggregate.git","clone_url":"https://github.com/logstash-plugins/logstash-filter-aggregate.git","svn_url":"https://github.com/logstash-plugins/logstash-filter-aggregate","homepage":null,"size":246,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/1"},"html":{"href":"https://github.com/logstash-plugins/logstash-filter-aggregate/pull/1"},"issue":{"href":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/issues/1"},"comments":{"href":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/issues/1/comments"},"review_comments":{"href":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/1/comments"},"review_comment":{"href":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/pulls/1/commits"},"statuses":{"href":"https://api.github.com/repos/logstash-plugins/logstash-filter-aggregate/statuses/4d546e65e7e9a54a90eb3575bb41a08dab367a7c"}}}}