Program to generate sample log to feed to logstash?
<p>I have written a small java program which generates some dummy logs (writes stuff to a txt file basically). Now I want to feed this data to the ELK stack. Basically logstash should read this data from the txt file and I want to visualize these changes on kibana, just to get a feel of it. </p>

<p>What I basically want to do is then change the speed at which my program writes the dummy logs to the txt file so that I can see the changes on kibana.</p>

<p>I have just started exploring the ELK stack and this might be a completely wrong way to do this kind of analysis. Please do suggest if there are other better ways to do this (considering I don't have actual logs to work with right now)</p>

<p>Edit : @Val</p>

<pre><code>input {
    generator {
        message =&gt; “’83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36””
        count =&gt; 10
    }
}
</code></pre>

<p>So here is  my <code>logstash.conf</code>:</p>

<pre><code>input {

 stdin { }

}


filter {
  grok {
    match =&gt; {
      "message" =&gt; '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
    }
  }

  date {
    match =&gt; [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
    locale =&gt; en
  }

  geoip {
    source =&gt; "clientip"
  }

  useragent {
    source =&gt; "agent"
    target =&gt; "useragent"
  }
}

output {
  stdout {
codec =&gt; plain {
                        charset =&gt; "ISO-8859-1"
                }

}
  elasticsearch {
    hosts =&gt; "http://localhost:9200"
    index =&gt; "apache_elk_example"
    template =&gt; "./apache_template.json"
    template_name =&gt; "apache_elk_example"
    template_overwrite =&gt; true
  }
}
</code></pre>

<p>Now after starting elasticsearch and kabana I do:</p>

<pre><code>cat apache_logs | /usr/local/opt/logstash/bin/logstash -f apache_logs
</code></pre>

<p>where <code>apache_logs</code> is been fed from my java program:</p>

<pre><code>public static void main(String[] args) {
    // TODO Auto-generated method stub
    try {
        PrintStream out = new PrintStream(new FileOutputStream("/Users/username/Desktop/user/apache_logs"));
        System.setOut(out);
    } catch (FileNotFoundException ex) {
        System.out.print("Exception");
    }
    while(true)
    //for(int i=0;i&lt;5;++i)
    {
        System.out.println(generateRandomIPs() + //other log stuff);
        try {
            Thread.sleep(1000);                 //1000 milliseconds is one second.
        } catch(InterruptedException ex) {
            Thread.currentThread().interrupt();
        }
    }
}
</code></pre>

<p>So here is the problem : </p>

<p>Kibana doesn't show me real time visualization i.e. as and when my java program feeds data into the <code>apache_log</code> file it does not show it to me. It only shows only until whatever data was already written into 'apache_log' at the time of execution of :</p>

<pre><code>cat apache_logs | /usr/local/opt/logstash/bin/logstash -f apache_logs
</code></pre>