{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/13838","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13838/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13838/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13838/events","html_url":"https://github.com/elastic/elasticsearch/issues/13838","id":108842118,"number":13838,"title":"Elasticsearch creates same log entry multiple times","user":{"login":"tigranterteryan","id":7555393,"avatar_url":"https://avatars.githubusercontent.com/u/7555393?v=3","gravatar_id":"","url":"https://api.github.com/users/tigranterteryan","html_url":"https://github.com/tigranterteryan","followers_url":"https://api.github.com/users/tigranterteryan/followers","following_url":"https://api.github.com/users/tigranterteryan/following{/other_user}","gists_url":"https://api.github.com/users/tigranterteryan/gists{/gist_id}","starred_url":"https://api.github.com/users/tigranterteryan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tigranterteryan/subscriptions","organizations_url":"https://api.github.com/users/tigranterteryan/orgs","repos_url":"https://api.github.com/users/tigranterteryan/repos","events_url":"https://api.github.com/users/tigranterteryan/events{/privacy}","received_events_url":"https://api.github.com/users/tigranterteryan/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-29T10:36:33Z","updated_at":"2015-09-29T12:57:16Z","closed_at":"2015-09-29T12:57:16Z","body":"Hello,\r\n\r\nI have a problem with Elasticsearch creating same entry in DB 2 or more times.\r\n\r\nHere is the search of the entry before it appears in DB:\r\n```\r\nroot@elk:/home/tigran# curl -XGET localhost:9200/appdev-2015.09.29/jibeqa/_search -d '{\"query\":{\"bool\":{\"must\":[{\"term\":{\"jibeqa.date\":\"2015-09-29T07:10:50+0000\"}}],\"must_not\":[],\"should\":[]}},\"from\":0,\"size\":10,\"sort\":[],\"facets\":{}}' | json_pp\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n100   270  100   122  100   148  19473  23623 --:--:-- --:--:-- --:--:-- 24666\r\n{\r\n   \"timed_out\" : false,\r\n   \"hits\" : {\r\n      \"total\" : 0,\r\n      \"hits\" : [],\r\n      \"max_score\" : null\r\n   },\r\n   \"took\" : 1,\r\n   \"_shards\" : {\r\n      \"successful\" : 5,\r\n      \"total\" : 5,\r\n      \"failed\" : 0\r\n   }\r\n}\r\nroot@elk:/home/tigran#\r\n```\r\n\r\nAs you can see it is empty for now.\r\n\r\nNow I send json to the log file which will be forwarded with logstash-forwarder\r\n```\r\n{\"appId\":\"imas-dev-10.240.21.82\",\"date\":\"2015-09-29T07:10:50+0000\",\"category\":\"IMAS_CHATSESSION_MESSAGE_RECEIVED_121\",\"user\":\"+11000040488\",\"toUser\":\"+11000040466\",\"messageType\":\"imdn\",\"contributionId\":\"FACfIAV6DA\",\"size\":265,\"extras\":{}}\r\n```\r\n\r\nlogstash rubydebug shows that it was parsed and sent to Elasticsearch just once:\r\n```\r\n{\r\n             \"appId\" => \"imas-dev-10.240.21.82\",\r\n              \"date\" => \"2015-09-29T07:10:50+0000\",\r\n          \"category\" => \"IMAS_CHATSESSION_MESSAGE_RECEIVED_121\",\r\n              \"user\" => \"+11000040488\",\r\n            \"toUser\" => \"+11000040466\",\r\n       \"messageType\" => \"imdn\",\r\n    \"contributionId\" => \"FACfIAV6DA\",\r\n              \"size\" => 265,\r\n            \"extras\" => {},\r\n          \"@version\" => \"1\",\r\n        \"@timestamp\" => \"2015-09-29T07:10:50.000Z\",\r\n              \"file\" => \"/opt/jibe/jetty_master/reporting_MASTER_local/logs/reporting.log\",\r\n              \"host\" => \"app3-us\",\r\n            \"offset\" => \"2390\",\r\n              \"tags\" => \"appdev\",\r\n              \"type\" => \"jibeqa\"\r\n}\r\n```\r\n\r\nBut new search in Elasticsearch shows that it was created 2 times:\r\n```\r\nroot@elk:/home/tigran# curl -XGET localhost:9200/appdev-2015.09.29/jibeqa/_search -d '{\"query\":{\"bool\":{\"must\":[{\"term\":{\"jibeqa.date\":\"2015-09-29T07:10:50+0000\"}}],\"must_not\":[],\"should\":[]}},\"from\":0,\"size\":10,\"sort\":[],\"facets\":{}}' | json_pp\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n100  1350  100  1202  100   148   3304    406 --:--:-- --:--:-- --:--:--  3311\r\n{\r\n   \"_shards\" : {\r\n      \"total\" : 5,\r\n      \"successful\" : 5,\r\n      \"failed\" : 0\r\n   },\r\n   \"timed_out\" : false,\r\n   \"took\" : 361,\r\n   \"hits\" : {\r\n      \"total\" : 2,\r\n      \"max_score\" : 1,\r\n      \"hits\" : [\r\n         {\r\n            \"_index\" : \"appdev-2015.09.29\",\r\n            \"_score\" : 1,\r\n            \"_type\" : \"jibeqa\",\r\n            \"_id\" : \"AVAYqahQZ2RePRz37-kE\",\r\n            \"_source\" : {\r\n               \"appId\" : \"imas-dev-10.240.21.82\",\r\n               \"messageType\" : \"imdn\",\r\n               \"toUser\" : \"+11000040466\",\r\n               \"offset\" : \"2390\",\r\n               \"contributionId\" : \"FACfIAV6DA\",\r\n               \"@timestamp\" : \"2015-09-29T07:10:50.000Z\",\r\n               \"user\" : \"+11000040488\",\r\n               \"type\" : \"jibeqa\",\r\n               \"@version\" : \"1\",\r\n               \"date\" : \"2015-09-29T07:10:50+0000\",\r\n               \"file\" : \"/opt/jibe/jetty_master/reporting_MASTER_local/logs/reporting.log\",\r\n               \"tags\" : \"appdev\",\r\n               \"host\" : \"app3-us\",\r\n               \"category\" : \"IMAS_CHATSESSION_MESSAGE_RECEIVED_121\",\r\n               \"size\" : 265,\r\n               \"extras\" : {}\r\n            }\r\n         },\r\n         {\r\n            \"_score\" : 1,\r\n            \"_type\" : \"jibeqa\",\r\n            \"_source\" : {\r\n               \"file\" : \"/opt/jibe/jetty_master/reporting_MASTER_local/logs/reporting.log\",\r\n               \"date\" : \"2015-09-29T07:10:50+0000\",\r\n               \"@version\" : \"1\",\r\n               \"type\" : \"jibeqa\",\r\n               \"extras\" : {},\r\n               \"size\" : 265,\r\n               \"category\" : \"IMAS_CHATSESSION_MESSAGE_RECEIVED_121\",\r\n               \"tags\" : \"appdev\",\r\n               \"host\" : \"app3-us\",\r\n               \"messageType\" : \"imdn\",\r\n               \"appId\" : \"imas-dev-10.240.21.82\",\r\n               \"contributionId\" : \"FACfIAV6DA\",\r\n               \"@timestamp\" : \"2015-09-29T07:10:50.000Z\",\r\n               \"user\" : \"+11000040488\",\r\n               \"offset\" : \"2390\",\r\n               \"toUser\" : \"+11000040466\"\r\n            },\r\n            \"_id\" : \"AVAYqahQZ2RePRz37-kD\",\r\n            \"_index\" : \"appdev-2015.09.29\"\r\n         }\r\n      ]\r\n   }\r\n}\r\n```\r\n\r\nThe ID is different but the rest is same.\r\n\r\nMy input is:\r\n```\r\n  lumberjack {\r\n    port => 5120\r\n    #type => \"logs\"\r\n    #tags => \"appdev\"\r\n    ssl_certificate => \"/etc/pki/tls/certs/logstash-forwarder_appdev.crt\"\r\n    ssl_key => \"/etc/pki/tls/private/logstash-forwarder_appdev.key\"\r\n    codec => \"json\"\r\n  }\r\n```\r\n\r\nFilter is:\r\n```\r\nfilter {\r\n  if [type] == \"jibeqa\" {\r\n    json{\r\n        source => \"message\"\r\n    }\r\n    mutate {\r\n        remove_field => \"message\"\r\n    }\r\n    date {\r\n      match => [ \"date\", \"ISO8601\"]\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nAnd the output is:\r\n```\r\noutput {\r\n  if \"appdev\" in [tags] {\r\n    elasticsearch { host => localhost protocol => \"http\" cluster => Jibe_ELK index => \"appdev-%{+YYYY.MM.dd}\" workers => 5 }\r\n  }\r\n#  stdout { codec => rubydebug }\r\n}\r\n```\r\n\r\nVersions:\r\n```\r\nii  elasticsearch                       1.5.2                               all          Open Source, Distributed, RESTful Search Engine\r\nii  logstash                            1.4.5-1-a2bacae                     all          An extensible logging pipeline\r\nii  logstash-contrib                    1.4.5-1-2bad350                     all          Community supported plugins for Logstash\r\n```"}}