Fluentd: Could not push logs to Elasticsearch
<p>I have deployed a elasticsearch 2.2.0 Now, I'm sending logs to it using a td-agent 2.3.0-0.</p>

<p>The final on tag chain was..</p>

<pre><code>&lt;match extra.geoip.processed5.**&gt;
  type copy

  &lt;store&gt;
    type                file
    path                /var/log/td-agent/sp_l5
    time_slice_format   %Y%m%d
    time_slice_wait     10m
    time_format         %Y%m%dT%H%M%S%z
    compress            gzip
    utc
  &lt;/store&gt;

  &lt;store&gt;
    type                elasticsearch
    host                11.0.0.174
    port                9200
    logstash_format  true
    logstash_prefix  logstash_business
    logstash_dateformat %Y.%m
    flush_interval  5s
  &lt;/store&gt;
&lt;/match&gt;
</code></pre>

<p>Now, I added a time out within the elasticsearch type block</p>

<pre><code>    request_timeout 45s
</code></pre>

<p>This is the td-agent.log with debug enabled.</p>

<pre><code>2016-02-08 15:58:07 +0100 [info]: plugin/in_syslog.rb:176:listen: listening syslog socket on 0.0.0.0:5514 with udp
2016-02-08 15:59:03 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster =&gt; {:host=&gt;"11.0.0.174", :port=&gt;9200, :scheme=&gt;"http"}
2016-02-08 15:59:03 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster =&gt; {:host=&gt;"11.0.0.174", :port=&gt;9200, :scheme=&gt;"http"}
2016-02-08 16:03:33 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2016-02-08 16:03:33 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2016-02-08 16:03:35 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster =&gt; {:host=&gt;"11.0.0.174", :port=&gt;9200, :scheme=&gt;"http"}
2016-02-08 16:03:35 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster =&gt; {:host=&gt;"11.0.0.174", :port=&gt;9200, :scheme=&gt;"http"}
2016-02-08 16:08:05 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2016-02-08 16:08:05 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2016-02-08 16:08:09 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster =&gt; {:host=&gt;"11.0.0.174", :port=&gt;9200, :scheme=&gt;"http"}
2016-02-08 16:08:09 +0100 [info]: plugin/out_elasticsearch.rb:77:client: Connection opened to Elasticsearch cluster =&gt; {:host=&gt;"11.0.0.174", :port=&gt;9200, :scheme=&gt;"http"}
2016-02-08 16:12:40 +0100 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2016-02-08 15:59:04 +0100 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Could not push logs to Elasticsearch after 2 retries. read timeout reached" plugin_id="object:fd9738"
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:204:in `rescue in send'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:194:in `send'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:188:in `write'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:345:in `write_chunk'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:324:in `pop'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:321:in `try_flush'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:140:in `run'
2016-02-08 16:12:40 +0100 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2016-02-08 15:59:04 +0100 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Could not push logs to Elasticsearch after 2 retries. read timeout reached" plugin_id="object:1034980"
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:204:in `rescue in send'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:194:in `send'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:188:in `write'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:345:in `write_chunk'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:324:in `pop'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:321:in `try_flush'
2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:140:in `run'
</code></pre>

<p>I'm running this on AWS over ubuntu 14.04 c3.large. I have tested the access from td-agent machine creating a index, adding documents and deleting the index using curl without any problem. (To be sure, I opened all communications in my Security Groups)</p>

<p>Another test from td-agent machine...</p>

<pre><code>root@bilbo:~# telnet 11.0.0.174 9200
Trying 11.0.0.174...
Connected to 11.0.0.174.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 320

{
  "name" : "gandalf-gandalf",
  "cluster_name" : "aaaa_dev",
  "version" : {
    "number" : "2.2.0",
    "build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
    "build_timestamp" : "2016-01-27T13:32:39Z",
    "build_snapshot" : false,
    "lucene_version" : "5.4.1"
  },
  "tagline" : "You Know, for Search"
}
Connection closed by foreign host.
root@bilbo:~#
</code></pre>

<p>With strace I can see this</p>

<pre><code>[pid 10774] connect(23, {sa_family=AF_INET, sin_port=htons(9200), sin_addr=inet_addr("11.0.0.174")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 10774] clock_gettime(CLOCK_MONOTONIC, {4876, 531526283}) = 0
[pid 10774] select(24, NULL, [23], NULL, {45, 0}) = 1 (out [23], left {44, 999925})
[pid 10774] fcntl(23, F_GETFL)          = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 10774] connect(23, {sa_family=AF_INET, sin_port=htons(9200), sin_addr=inet_addr("11.0.0.174")}, 16) = 0
[pid 10774] fcntl(23, F_GETFL)          = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 10774] write(23, "POST /_bulk HTTP/1.1\r\nUser-Agent: Faraday v0.9.2\r\nHost: 11.0.0.174:9200\r\nContent-Length: 4798\r\n\r\n", 97) = 97
[pid 10774] fcntl(23, F_GETFL)          = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 10774] write(23, "{\"index\":{\"_index\":\"logstash_apache-2016.02.08\",\"_type\":\"fluentd\"}}\n{\"message\":\"Feb  8 16:18:47 bilbo ::apache::PRE::access: - 10.0.0.15 - control [08/Feb/2016:16:18:47 +0100] \\\"GET /status/memcached.php HTTP/1.1\\\" 200 1785 \\\"-\\\" \\\"check_http/v2.0 (monitoring-plugins 2.0)\\\" control-pre.fluzo.com:443 0\",\"n\":\"bilbo\",\"s\":\"info\",\"f\":\"local3\",\"t\":\"Feb  8 16:18:47\",\"h\":\"bilbo\",\"a\":\"apache\",\"e\":\"PRE\",\"o\":\"access\",\"ip\":\"-\",\"ip2\":\"10.0.0.15\",\"rl\":\"-\",\"ru\":\"control\",\"rt\":\"[08/Feb/2016:16:18:47 +0100]\",\"met\":\"GET\",\"pqf\":\"status/memcached.php\",\"hv\":\"HTTP/1.1\",\"st\":\"200\",\"bs\":\"1785\",\"ref\":\"-\",\"ua\":\"check_http/v2.0 (monitoring-plugins 2.0)\",\"vh\":\"control-pre.aaaa.com\",\"p\":\"443\",\"rpt\":\"0\",\"co\":null,\"ci\":null,\"la\":null,\"lo\":null,\"ar\":null,\"dm\":null,\"re\":null,\"@timestamp\":\"2016-02-08T16:19:47+01:00\"}\n{\"index\":{\"_index\":\"logstash_apache-2016.02.08\",\"_type\":\"fluentd\"}}\n{\"message\":\"Feb  8 16:18:47 bilbo ::apache::PRE::access: - 10.0.0.15 - control [08/Feb/2016:16:18:47 +0100] \\\"GET /status/core.php HTTP/1.1\\\" 200 1898 \\\"-\\\" \\\"c"..., 4798) = 4798
</code></pre>

<p>Seems that the connection is open.</p>

<p>To avoid confusion, I have installed td-agent on elasticsearch machine with the same output.</p>

<p>This is my elasticsearch configuration...</p>

<pre><code>### MANAGED BY PUPPET ###
---
bootstrap:
  mlockall: true
cluster:
  name: aaaa0_dev
discovery:
  zen:
    minimum_master_nodes: 1
    ping:
      multicast:
        enabled: false
      unicast:
        hosts:
             - 11.0.0.174
gateway:
  expected_nodes: 1
  recover_after_nodes: 1
  recover_after_time: 5m
hostname: gandalf
http:
  compression: true
index:
  store:
    compress:
      stored: true
    type: niofs
network:
  bind_host: 11.0.0.174
  publish_host: 11.0.0.174
node:
  name: gandalf-gandalf
path:
  data: /var/lib/elasticsearch-gandalf
  logs: /var/log/elasticsearch/gandalf
transport:
  tcp:
    compress: true
</code></pre>

<p>Any idea?</p>

<p>Thanks.</p>

<p><strong>UPDATE</strong></p>

<p>Against a real elasticsearch cluster works (3 nodes)</p>

<p>This is the configuration.</p>

<pre><code>### MANAGED BY PUPPET ###
---
bootstrap:
  mlockall: true
cluster:
  name: aaaa
discovery:
  zen:
    minimum_master_nodes: 2
    ping:
      multicast:
        enabled: false
      unicast:
        hosts:
             - el0
             - el1
             - el2
gateway:
  expected_nodes: 3
  recover_after_nodes: 2
  recover_after_time: 5m
hostname: kili
http:
  compression: true
index:
  store:
    compress:
      stored: true
    type: niofs
network:
  bind_host: 11.0.0.253
  publish_host: 11.0.0.253
node:
  name: kili-kili
path:
  data:
      - /var/lib/elasticsearch0
      - /var/lib/elasticsearch1
  logs: /var/log/elasticsearch/kili
transport:
  tcp:
    compress: true
</code></pre>

<p>However, kibana did create .kibana index into elasticsearch with one node. Also, y test this node with curl.</p>