{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/4063","labels_url":"https://api.github.com/repos/elastic/logstash/issues/4063/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/4063/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/4063/events","html_url":"https://github.com/elastic/logstash/issues/4063","id":112211493,"number":4063,"title":"Add simple integration test infra to Logstash core","user":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/logstash/labels/tests-infra","name":"tests-infra","color":"bfd4f2"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-19T19:08:45Z","updated_at":"2015-10-20T14:40:29Z","closed_at":null,"body":"We have good system wide integration tests in Jenkins using [logstash-kitchen](https://github.com/elastic/logstash-kitchen) repo. These are end-end tests infrastructure which validate things like installing/uninstalling plugins, smoke tests for ES, file input, Kafka etc. \r\n\r\nWe also need a simple integration test bed in Logstash core which tests multiple combinations of input, filters, codec and output using various test data sets. We should aim to keep the inputs and outputs simple, maybe just use the generator input (or file input) and file output. This will help us catch issues like #4048 which could be reproduced using file input, grok filter and json codec.\r\n\r\nSuggest using `spec/integration` dir for adding these tests. Should also add `rake tests:integration-plugins` which will install dependencies, run these simple integration tests. \r\n\r\nUltimately, it should be easy enough to drop in a config file plus a results (expected output) fixture and the rest should be taken care of by the infra."},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/149588206","html_url":"https://github.com/elastic/logstash/issues/4063#issuecomment-149588206","issue_url":"https://api.github.com/repos/elastic/logstash/issues/4063","id":149588206,"user":{"login":"colinsurprenant","id":2010,"avatar_url":"https://avatars.githubusercontent.com/u/2010?v=3","gravatar_id":"","url":"https://api.github.com/users/colinsurprenant","html_url":"https://github.com/colinsurprenant","followers_url":"https://api.github.com/users/colinsurprenant/followers","following_url":"https://api.github.com/users/colinsurprenant/following{/other_user}","gists_url":"https://api.github.com/users/colinsurprenant/gists{/gist_id}","starred_url":"https://api.github.com/users/colinsurprenant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colinsurprenant/subscriptions","organizations_url":"https://api.github.com/users/colinsurprenant/orgs","repos_url":"https://api.github.com/users/colinsurprenant/repos","events_url":"https://api.github.com/users/colinsurprenant/events{/privacy}","received_events_url":"https://api.github.com/users/colinsurprenant/received_events","type":"User","site_admin":false},"created_at":"2015-10-20T14:40:29Z","updated_at":"2015-10-20T14:40:29Z","body":"+1\r\n\r\nI guess a lots of tests involving codecs and filters can be done easily using stdin/stdout, we could maybe look at reusing the perf framework and add the ability to compare output for validation?\r\n\r\nIt would be nice to be able to easily and \"quickly\" run a good set of integration tests through rake so that we can integrate this in our \"tdd\" loop. I would suggest we first focus on this and make it as efficient as possible. Then we can add move involved integration tests using test kitchen for example. \r\n\r\nThoughts?\r\n\r\n"}}