ELK Stack - Elasticsearch index creation (logstash)
<p>I'm experimenting with ELK to analyze our log files. Following the available documentation, managed to set up the stack in my pc. Now I'm facing an issue with the elastic search index creation. Previously I was using filebeat -&gt; logstash -&gt; elasticsearch -&gt; kibana combination and using the following logstash.conf file was able to send data to elasticsearch</p>
<pre><code>input {
  beats {
   port =&gt; 5044
   type =&gt; &quot;log&quot;
  }
}

output {
  elasticsearch {
   hosts =&gt; &quot;localhost:9200&quot;
   manage_template =&gt; false
   index =&gt; &quot;%{[@metadata][beat]}-%{+YYYY.MM.dd}&quot;
   document_type =&gt; &quot;%{[@metadata][type]}&quot;
  }
}
</code></pre>
<p>And the index in elastic search was evaluated to</p>
<pre><code> &quot;filebeat-*&quot; 
</code></pre>
<p>from the expression</p>
<pre><code>  index =&gt; &quot;%{[@metadata][beat]}-%{+YYYY.MM.dd}&quot;
</code></pre>
<p>Now I changed the logstash.conf to based on my actual logfile</p>
<pre><code>input { 

file
{
    path =&gt; &quot;C:\logs\application.log&quot;
    start_position =&gt; &quot;beginning&quot;
    codec =&gt; 
    multiline {
      charset =&gt; &quot;ISO-8859-1&quot;
      pattern =&gt; &quot;^%{TIMESTAMP_ISO8601}&quot;
      max_lines =&gt; 1000
      negate =&gt; true
      what =&gt; &quot;previous&quot;
    } 
}

}

filter {
 mutate {
     gsub =&gt; [ &quot;message&quot;, &quot;\r&quot;, &quot;&quot; ]
   }

   grok {
    patterns_dir =&gt; &quot;./patterns&quot;
    match =&gt; {&quot;message&quot; =&gt; &quot;%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL1:loglevel} %{THREAD:thread} %{IP5:remoteipaddress} %{JAVA:logclass} %{GREEDYDATA:details}&quot;}
  add_field =&gt; [ &quot;received_at&quot;, &quot;%{@timestamp}&quot; ]
  add_field =&gt; [ &quot;received_from&quot;, &quot;%{host}&quot; ]

  }
   date {
            match =&gt; [ &quot;timestamp&quot;, &quot;yyyy-MM-dd HH:mm:ss,SSS&quot; ]
            remove_field =&gt; [ &quot;timestamp&quot; ]
        }


}

output {

  elasticsearch { 
                hosts =&gt; &quot;localhost:9200&quot;
    manage_template =&gt; false
     index =&gt; &quot;%{[@metadata][beat]}-%{+YYYY.MM.dd}&quot;
     document_type =&gt; &quot;%{[@metadata][type]}&quot;
                }
                file {
     path =&gt; &quot;C:\logs\output.txt&quot;
  }
}
</code></pre>
<p>In this case, logstash is happy with the conf file, but the index I suggested for elastic search is not being evaluated properly.</p>
<p>If I inspect elastic search using the head plugin,</p>
<pre><code>http://localhost:9200/_plugin/head/
</code></pre>
<p>The index appears as</p>
<pre><code>%{[@metadata][beat]}-
</code></pre>
<p><a href="https://i.stack.imgur.com/airKK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/airKK.png" alt="enter image description here" /></a></p>
<p>I'm not sure why the index expression is not being evaluated now. Any pointers to solve this issue would be helpful.</p>
<p>Thanks in advance,
San</p>