{"action":"created","issue":{"url":"https://api.github.com/repos/BitScoutOrg/efk-atomicapp/issues/11","labels_url":"https://api.github.com/repos/BitScoutOrg/efk-atomicapp/issues/11/labels{/name}","comments_url":"https://api.github.com/repos/BitScoutOrg/efk-atomicapp/issues/11/comments","events_url":"https://api.github.com/repos/BitScoutOrg/efk-atomicapp/issues/11/events","html_url":"https://github.com/BitScoutOrg/efk-atomicapp/issues/11","id":122957346,"number":11,"title":"Inconsistent number of records indexed using simple test","user":{"login":"lukas-vlcek","id":205174,"avatar_url":"https://avatars.githubusercontent.com/u/205174?v=3","gravatar_id":"","url":"https://api.github.com/users/lukas-vlcek","html_url":"https://github.com/lukas-vlcek","followers_url":"https://api.github.com/users/lukas-vlcek/followers","following_url":"https://api.github.com/users/lukas-vlcek/following{/other_user}","gists_url":"https://api.github.com/users/lukas-vlcek/gists{/gist_id}","starred_url":"https://api.github.com/users/lukas-vlcek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukas-vlcek/subscriptions","organizations_url":"https://api.github.com/users/lukas-vlcek/orgs","repos_url":"https://api.github.com/users/lukas-vlcek/repos","events_url":"https://api.github.com/users/lukas-vlcek/events{/privacy}","received_events_url":"https://api.github.com/users/lukas-vlcek/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-18T14:24:51Z","updated_at":"2015-12-18T15:06:24Z","closed_at":null,"body":"Even after rsyslog allowed TCP (see https://github.com/BitScoutOrg/nulecule-rsyslog-collector/pull/2) I am experiencing inconsistent number of records indexed after simple test.\r\n\r\n- We will use `make_rfc5424.py` script to generate artificial log records but we will modify it to produce more data. \r\n- We assume fresh and clean BitScout application running and Elasticsearch node available via HTTP.\r\n- For maximal clarity of the test we will want to change default settings of BitScout Elasticsearch index.\r\n\r\n### Start BitScout\r\n\r\nSee [quickstart-instructions](https://github.com/BitScoutOrg/efk-atomicapp/wiki/How-to-run-the-application-with-atomic#quickstart-instructions).\r\n\r\n### Download and modify data creating script\r\n\r\n- Download <https://raw.githubusercontent.com/BitScoutOrg/efk-atomicapp/master/make-rfc5424.py>\r\n- We want to generate more data, change line [23](https://github.com/BitScoutOrg/efk-atomicapp/blob/master/make-rfc5424.py#L23) to `nfacility = 824`\r\n- Now we have a script that will generate 6592 records:\r\n\r\n````\r\n$ python make_rfc5424.py | wc -l\r\n6592\r\n````\r\n\r\n### Check the Elasticsearch cluster is empty\r\n\r\nAssuming <http://localhost:9200> points to HTTP endpoint of Elasticsearch node.\r\n\r\nPoint web browser to <http://localhost:9200/_cat/indices?v>. If the Elasticsearch is empty, you will get the following output:\r\n\r\n:: PASTE SCREENSHOT HERE ::\r\n\r\n### Create index template with custom settings\r\n\r\n````\r\ncurl -X PUT 'localhost:9200/_template/bitscout_custom' -d '{\r\n  \"template\": \"bitscout*\",\r\n  \"order\": 20,\r\n  \"settings\": {\r\n    \"refresh_interval\"   : -1,\r\n    \"number_of_shards\"   : 1,\r\n    \"number_of_replicas\" : 0\r\n  }\r\n}'\r\n````\r\n\r\n### Index the data\r\n\r\nRun the script that we have modified before:\r\n\r\n````\r\npython make_rfc5424.py | nc localhost 5141\r\n````\r\n\r\nPoint web browser to <http://localhost:9200/_cat/indices?v&h=health,status,index,pri,rep,docs.count,docs.deleted,store.size,pri.store.size,indexing.index_total,merges.current,merges.total>\r\n\r\nOutput should list all existing indices. The most important is `indexing.index_total` column. It contains the real total number of indexed documents in this index. The column `docs.count` contains total number of documents in this index that are available via search API. In our case it will contain `0` due to previously set `refresh_interval` (or until the [flush](https://www.elastic.co/guide/en/elasticsearch/reference/1.5/indices-flush.html) operation is called automatically).\r\n\r\nIndex [refresh](https://www.elastic.co/guide/en/elasticsearch/reference/1.5/indices-refresh.html) is used to make indexed documents searchable and can be called manually via:\r\n\r\n````\r\ncurl -X POST 'localhost:9200/_refresh'\r\n````\r\n\r\nOther useful resource to check can be <http://localhost:9200/_cat/segments?v> which lists physical segments of the index along with `searchable` and `docs.count` status. Basically the sum of searchable segments should equal to total number of searchable document in the index.\r\n\r\n### What is the problem?\r\n\r\nThe problem is that `indexing.index_total` value does not seem to be predictable and it can differ from number of records generated by `make_rfc5424.py` (which in our case is `6592`).\r\n\r\n- I experienced the following values `6577`, `6593`, `6594`, `6609`.\r\n- Changing number of shards does not seem to have any effect.\r\n\r\n### Repeate the test\r\n\r\nFirst drop the index\r\n\r\n````\r\ncurl -X DELETE 'localhost:9200/bitscout.2015.12.18'\r\n````\r\n\r\nor if you have only a single bitscout index, use:\r\n\r\n````\r\ncurl -X DELETE 'localhost:9200/bitscout*'\r\n````\r\n\r\nand go back to **Index the data** chapter.\r\n"},"comment":{"url":"https://api.github.com/repos/BitScoutOrg/efk-atomicapp/issues/comments/165799509","html_url":"https://github.com/BitScoutOrg/efk-atomicapp/issues/11#issuecomment-165799509","issue_url":"https://api.github.com/repos/BitScoutOrg/efk-atomicapp/issues/11","id":165799509,"user":{"login":"lukas-vlcek","id":205174,"avatar_url":"https://avatars.githubusercontent.com/u/205174?v=3","gravatar_id":"","url":"https://api.github.com/users/lukas-vlcek","html_url":"https://github.com/lukas-vlcek","followers_url":"https://api.github.com/users/lukas-vlcek/followers","following_url":"https://api.github.com/users/lukas-vlcek/following{/other_user}","gists_url":"https://api.github.com/users/lukas-vlcek/gists{/gist_id}","starred_url":"https://api.github.com/users/lukas-vlcek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukas-vlcek/subscriptions","organizations_url":"https://api.github.com/users/lukas-vlcek/orgs","repos_url":"https://api.github.com/users/lukas-vlcek/repos","events_url":"https://api.github.com/users/lukas-vlcek/events{/privacy}","received_events_url":"https://api.github.com/users/lukas-vlcek/received_events","type":"User","site_admin":false},"created_at":"2015-12-18T15:06:24Z","updated_at":"2015-12-18T15:06:24Z","body":"So we need to learn how exactly fluentd-elasticsearch integration pushes data to Elasticsearch (probably batch updates) so that we can use the same method in our tests."}}