using logstash to get data out of elasticsearch to a csv file
<p>I am new to using logstash and am struggling to get data out of elasticsearch using logstash as a csv.</p>

<p>To create some sample data, we can first add a basic csv into elasticsearch... the head of the sample csv can be seen below</p>

<pre><code>$ head uu.csv
"hh","hh1","hh3","id"
-0.979646332669359,1.65186132910743,"L",1
-0.283939374784435,-0.44785377794233,"X",2
0.922659898930901,-1.11689020559612,"F",3
0.348918777124474,1.95766948269957,"U",4
0.52667811182958,0.0168862169880919,"Y",5
-0.804765331279075,-0.186456470768865,"I",6
0.11411203100637,-0.149340801708981,"Q",7
-0.952836952412902,-1.68807271639322,"Q",8
-0.373528919496876,0.750994450392907,"F",9
</code></pre>

<p>I then put that into logstash using the following...</p>

<pre><code>$ cat uu.conf 
input {
  stdin {}
}

filter {
  csv {
      columns =&gt; [
        "hh","hh1","hh3","id"
      ]
  }

  if [hh1] == "hh1" {
      drop { }
  } else {
      mutate {
          remove_field =&gt; [ "message", "host", "@timestamp", "@version" ]
      }

      mutate {
          convert =&gt; { "hh" =&gt; "float" }
          convert =&gt; { "hh1" =&gt; "float" }
          convert =&gt; { "hh3" =&gt; "string" }
          convert =&gt; { "id" =&gt; "integer" }
      }
  }
}

output {
  stdout { codec =&gt; dots }
  elasticsearch {
      index =&gt; "temp_index"
      document_type =&gt; "temp_doc"
      document_id =&gt; "%{id}"
  }
}
</code></pre>

<p>This is put into logstash with the following command....</p>

<pre><code>$ cat uu.csv | logstash-2.1.3/bin/logstash -f uu.conf 
Settings: Default filter workers: 16
Logstash startup completed
 ....................................................................................................Logstash shutdown completed
</code></pre>

<p>So far so good, but I would like to get some of the data out in particular the hh and hh3 fields in the temp_index.</p>

<p>I wrote the following to extract the data out of elasticsearch into a csv.</p>

<pre><code>$ cat yy.conf
input {
  elasticsearch {
    hosts =&gt; "localhost:9200"
    index =&gt; "temp_index"
    query =&gt; "*"
  }
}

filter {
    elasticsearch{
        add_field =&gt; {"hh" =&gt; "%{hh}"}
        add_field =&gt; {"hh3" =&gt; "%{hh3}"}
    }
}


output {
  stdout { codec =&gt; dots }
  csv {
      fields =&gt; ['hh','hh3']
      path =&gt; '/home/username/yy.csv'
  }
}
</code></pre>

<p>But get the following error when trying to run logstash...</p>

<pre><code>$ logstash-2.1.3/bin/logstash -f yy.conf
The error reported is: 
  Couldn't find any filter plugin named 'elasticsearch'. Are you sure this is correct? Trying to load the elasticsearch filter plugin resulted in this error: no such file to load -- logstash/filters/elasticsearch
</code></pre>

<p>What do I need to change to yy.conf so that a logstash command will extract the data out of elasticsearch and input into a new csv called yy.csv.</p>

<p><strong>UPDATE</strong></p>

<p>changing <code>yy.conf</code> to be the following...</p>

<pre><code>$ cat yy.conf
input {
  elasticsearch {
    hosts =&gt; "localhost:9200"
    index =&gt; "temp_index"
    query =&gt; "*"
  }
}

filter {}

output {
  stdout { codec =&gt; dots }
  csv {
      fields =&gt; ['hh','hh3']
      path =&gt; '/home/username/yy.csv'
  }
}
</code></pre>

<p>I got the following error...</p>

<pre><code>$ logstash-2.1.3/bin/logstash -f yy.conf
Settings: Default filter workers: 16
Logstash startup completed
A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: &lt;LogStash::Inputs::Elasticsearch hosts=&gt;["localhost:9200"], index=&gt;"temp_index", query=&gt;"*", codec=&gt;&lt;LogStash::Codecs::JSON charset=&gt;"UTF-8"&gt;, scan=&gt;true, size=&gt;1000, scroll=&gt;"1m", docinfo=&gt;false, docinfo_target=&gt;"@metadata", docinfo_fields=&gt;["_index", "_type", "_id"], ssl=&gt;false&gt;
  Error: [400] {"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"init_scan","grouped":true,"failed_shards":[{"shard":0,"index":"temp_index","node":"zu3E6F7kQRWnDPY5L9zF-w","reason":{"type":"parse_exception","reason":"Failed to derive xcontent"}}]},"status":400} {:level=&gt;:error}
A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: &lt;LogStash::Inputs::Elasticsearch hosts=&gt;["localhost:9200"], index=&gt;"temp_index", query=&gt;"*", codec=&gt;&lt;LogStash::Codecs::JSON charset=&gt;"UTF-8"&gt;, scan=&gt;true, size=&gt;1000, scroll=&gt;"1m", docinfo=&gt;false, docinfo_target=&gt;"@metadata", docinfo_fields=&gt;["_index", "_type", "_id"], ssl=&gt;false&gt;
  Error: [400] {"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"init_scan","grouped":true,"failed_shards":[{"shard":0,"index":"temp_index","node":"zu3E6F7kQRWnDPY5L9zF-w","reason":{"type":"parse_exception","reason":"Failed to derive xcontent"}}]},"status":400} {:level=&gt;:error}
A plugin had an unrecoverable error. Will restart this plugin.
</code></pre>

<p>Interestingly...if i change <code>yy.conf</code> to remove <code>elasticsearch{}</code> to look like...</p>

<pre><code>$ cat yy.conf
input {
  elasticsearch {
    hosts =&gt; "localhost:9200"
    index =&gt; "temp_index"
    query =&gt; "*"
  }
}

filter {
        add_field =&gt; {"hh" =&gt; "%{hh}"}
        add_field =&gt; {"hh3" =&gt; "%{hh3}"}
}


output {
  stdout { codec =&gt; dots }
  csv {
      fields =&gt; ['hh','hh3']
      path =&gt; '/home/username/yy.csv'
  }
}
</code></pre>

<p>I get the following error...</p>

<pre><code>$ logstash-2.1.3/bin/logstash -f yy.conf
Error: Expected one of #, { at line 10, column 19 (byte 134) after filter {
        add_field 
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system.
</code></pre>

<p>Also when changing yy.conf to be something similar to take into account the error message</p>

<pre><code>$ cat yy.conf
input {
  elasticsearch {
    hosts =&gt; "localhost:9200"
    index =&gt; "temp_index"
    query =&gt; "*"
  }
}

filter {
        add_field {"hh" =&gt; "%{hh}"}
        add_field {"hh3" =&gt; "%{hh3}"}
}


output {
  stdout { codec =&gt; dots }
  csv {
      fields =&gt; ['hh','hh3']
      path =&gt; '/home/username/yy.csv'
  }
}
</code></pre>

<p>I get the following error...</p>

<pre><code>$ logstash-2.1.3/bin/logstash -f yy.conf
The error reported is: 
  Couldn't find any filter plugin named 'add_field'. Are you sure this is correct? Trying to load the add_field filter plugin resulted in this error: no such file to load -- logstash/filters/add_field
</code></pre>

<p><strong>* UPDATE 2 *</strong></p>

<p>Thanks to Val I have made some progress and started to get output. But they don't seem correct. I get the following outputs when running the following commands...</p>

<pre><code>$ cat uu.csv | logstash-2.1.3/bin/logstash -f uu.conf 
Settings: Default filter workers: 16
Logstash startup completed
....................................................................................................Logstash shutdown completed

$ logstash-2.1.3/bin/logstash -f yy.conf
Settings: Default filter workers: 16
Logstash startup completed
....................................................................................................Logstash shutdown completed

$ head uu.csv
"hh","hh1","hh3","id"
-0.979646332669359,1.65186132910743,"L",1
-0.283939374784435,-0.44785377794233,"X",2
0.922659898930901,-1.11689020559612,"F",3
0.348918777124474,1.95766948269957,"U",4
0.52667811182958,0.0168862169880919,"Y",5
-0.804765331279075,-0.186456470768865,"I",6
0.11411203100637,-0.149340801708981,"Q",7
-0.952836952412902,-1.68807271639322,"Q",8
-0.373528919496876,0.750994450392907,"F",9

$ head yy.csv
-0.106007607975644E1,F
0.385395589205671E0,S
0.722392598488791E-1,Q
0.119773830827963E1,Q
-0.151090510772458E1,W
-0.74978830916084E0,G
-0.98888121700762E-1,M
0.965827615823707E0,S
-0.165311094671424E1,F
0.523818819076447E0,R
</code></pre>

<p>Any help would be much appreciated...</p>