{"action":"created","issue":{"url":"https://api.github.com/repos/driskell/log-courier/issues/247","labels_url":"https://api.github.com/repos/driskell/log-courier/issues/247/labels{/name}","comments_url":"https://api.github.com/repos/driskell/log-courier/issues/247/comments","events_url":"https://api.github.com/repos/driskell/log-courier/issues/247/events","html_url":"https://github.com/driskell/log-courier/issues/247","id":114431129,"number":247,"title":"Possible Logstash 2.0 compatibility issue","user":{"login":"juanqui","id":67964,"avatar_url":"https://avatars.githubusercontent.com/u/67964?v=3","gravatar_id":"","url":"https://api.github.com/users/juanqui","html_url":"https://github.com/juanqui","followers_url":"https://api.github.com/users/juanqui/followers","following_url":"https://api.github.com/users/juanqui/following{/other_user}","gists_url":"https://api.github.com/users/juanqui/gists{/gist_id}","starred_url":"https://api.github.com/users/juanqui/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juanqui/subscriptions","organizations_url":"https://api.github.com/users/juanqui/orgs","repos_url":"https://api.github.com/users/juanqui/repos","events_url":"https://api.github.com/users/juanqui/events{/privacy}","received_events_url":"https://api.github.com/users/juanqui/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-31T20:08:10Z","updated_at":"2015-10-31T20:20:21Z","closed_at":null,"body":"I recently upgraded a small dev cluster to logstash 2.0 and elasticsearch 2.0. We've been using logstash-forwarder in our infrastructure, but I wanted to give log-courier a short (specifically because of client-side multiline codec support!). I see that very recently a patch was dropped to add compatibility for logstash 2.0 (v1.8.2). Nevertheless, I seem to be having some issue with my configuration.\r\n\r\nI have two logstash inputs configured.\r\n\r\n1 - logstash forwarder\r\n```ruby\r\ninput {\r\n  lumberjack {\r\n    port => 16789\r\n    ssl_certificate => \"/etc/logstash/ssl/logstash.crt\"\r\n    ssl_key => \"/etc/logstash/ssl/logstash.key\"\r\n  }\r\n}\r\n```\r\n2 - log courier\r\n```ruby\r\ninput {\r\n  courier {\r\n    port => 18789\r\n    ssl_certificate => \"/etc/logstash/ssl/logstash.crt\"\r\n    ssl_key => \"/etc/logstash/ssl/logstash.key\"\r\n  }\r\n}\r\n```\r\n\r\nMy courier clients are configured the following way:\r\n```json\r\n{\r\n    \"general\": {\r\n        \"log stdout\": true\r\n    },\r\n    \"network\": {\r\n        \"servers\": [\r\n            \"cp3logstash01.blah.com:18789\"\r\n        ],\r\n        \"ssl ca\": \"/etc/logstash-ssl-ca.crt\",\r\n        \"timeout\": 15\r\n    },\r\n    \"files\": [\r\n        {\r\n            \"fields\": {\r\n                \"app\": \"blah\",\r\n                \"env\": \"prod\",\r\n                \"type\": \"python-verbose\"\r\n            },\r\n            \"paths\": [\r\n                \"/etc/sv/blah-web/log/main/current\"\r\n            ]\r\n        },\r\n        {\r\n            \"fields\": {\r\n                \"app\": \"blah\",\r\n                \"env\": \"prod\",\r\n                \"type\": \"python-celery\"\r\n            },\r\n            \"paths\": [\r\n                \"/etc/sv/blah-worker/log/main/current\"\r\n            ]\r\n        }\r\n    ]\r\n}\r\n```\r\n\r\nWhen I start one of the log-courier process I immediately start seeing the following errors in logstash:\r\n```\r\n{\r\n  :timestamp=>\"2015-10-31T19:36:00.580000+0000\", \r\n  :message=>\"Attempted to send a bulk request to Elasticsearch configured at '[\\\"http://cp3elasticlogs01.blah.com:9200/\\\", \r\n            \\\"http://cp3elasticlogs02.blah.com:9200/\\\", \\\"http://cp3elasticlogs03.blah.com:9200/\\\"]', but an error occurred and \r\n            it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?\", \r\n  :client_config=>{\r\n    :hosts=>[\"http://cp3elasticlogs01.blah.com:9200/\", \"http://cp3elasticlogs02.blah.com:9200/\", \r\n    \"http://cp3elasticlogs03.blah.com:9200/\"], \r\n    :ssl=>nil, \r\n    :transport_options=>{\r\n      :socket_timeout=>0, \r\n      :request_timeout=>0, \r\n      :proxy=>nil, :ssl=>{}}, \r\n    :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, \r\n    :logger=>nil, \r\n    :tracer=>nil, \r\n    :reload_connections=>false, \r\n    :retry_on_failure=>false, \r\n    :reload_on_failure=>false, \r\n    :randomize_hosts=>false\r\n  }, \r\n  :error_message=>\"Failed to load class 'org.jruby.RubyObject$Access4JacksonDeserializer45931591': \r\n                  com.fasterxml.jackson.module.afterburner.ser.BeanPropertyAccessor\", \r\n  :error_class=>\"JrJackson::ParseError\", \r\n  :backtrace=>[\r\n    \"com/jrjackson/JrJacksonBase.java:83:in `generate'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/jrjackson-0.3.6/lib/jrjackson/jrjackson.rb:59:in `dump'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/multi_json-1.11.2/lib/multi_json/adapters/jr_jackson.rb:20:in `dump'\",\r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/multi_json-1.11.2/lib/multi_json/adapter.rb:25:in `dump'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/multi_json-1.11.2/lib/multi_json.rb:136:in `dump'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/utils.rb:102:in `__bulkify'\", \r\n    \"org/jruby/RubyArray.java:2414:in `map'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/utils.rb:102:in `__bulkify'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/bulk.rb:82:in `bulk'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:56:in `bulk'\",\r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:353:in `submit'\", \r\n    \"org/jruby/ext/thread/Mutex.java:149:in `synchronize'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:350:in `submit'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:382:in `flush'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in `buffer_flush'\", \r\n    \"org/jruby/RubyHash.java:1342:in `each'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in `buffer_flush'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:193:in `buffer_flush'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:in `buffer_receive'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:343:in `receive'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/outputs/base.rb:80:in `handle'\", \"(eval):187:in `output_func'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:252:in `outputworker'\", \r\n    \"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:169:in `start_outputs'\"], \r\n  :level=>:error\r\n}\r\n```\r\n\r\nI'm not sure if this is being caused by the combination of logstash 2.0 and elasticsearch 2.0.\r\n\r\nAccording to the release notes on logstash 2.0, the JrJackson library was upgraded.\r\n\r\n> JSON processing: We updated to a new version of JrJackson, which is the JSON serialization/deserialization library used in Logstash. In addition to fixes, this provides good performance boosts when dealing with JSON.\r\n\r\nI'm going to keep on looking into it as I have time.\r\n\r\nFound a possibly similar issue...\r\nhttps://github.com/elastic/logstash/issues/3289\r\n\r\nThanks for the awesome project @driskell "},"comment":{"url":"https://api.github.com/repos/driskell/log-courier/issues/comments/152768086","html_url":"https://github.com/driskell/log-courier/issues/247#issuecomment-152768086","issue_url":"https://api.github.com/repos/driskell/log-courier/issues/247","id":152768086,"user":{"login":"driskell","id":939815,"avatar_url":"https://avatars.githubusercontent.com/u/939815?v=3","gravatar_id":"","url":"https://api.github.com/users/driskell","html_url":"https://github.com/driskell","followers_url":"https://api.github.com/users/driskell/followers","following_url":"https://api.github.com/users/driskell/following{/other_user}","gists_url":"https://api.github.com/users/driskell/gists{/gist_id}","starred_url":"https://api.github.com/users/driskell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/driskell/subscriptions","organizations_url":"https://api.github.com/users/driskell/orgs","repos_url":"https://api.github.com/users/driskell/repos","events_url":"https://api.github.com/users/driskell/events{/privacy}","received_events_url":"https://api.github.com/users/driskell/received_events","type":"User","site_admin":false},"created_at":"2015-10-31T20:20:21Z","updated_at":"2015-10-31T20:20:21Z","body":"Sorry I was meant to say, because JrJackson is used to deserialise and then by ES output to serialise I would expect it to be fine (unless a JrJackson bug) - this my thoughts on filters that may change the event."}}