{"action":"created","issue":{"url":"https://api.github.com/repos/fluent/fluentd/issues/523","labels_url":"https://api.github.com/repos/fluent/fluentd/issues/523/labels{/name}","comments_url":"https://api.github.com/repos/fluent/fluentd/issues/523/comments","events_url":"https://api.github.com/repos/fluent/fluentd/issues/523/events","html_url":"https://github.com/fluent/fluentd/issues/523","id":52388722,"number":523,"title":"out_forward to a server that's down make whole server's down","user":{"login":"dieend","id":1404443,"avatar_url":"https://avatars.githubusercontent.com/u/1404443?v=3","gravatar_id":"","url":"https://api.github.com/users/dieend","html_url":"https://github.com/dieend","followers_url":"https://api.github.com/users/dieend/followers","following_url":"https://api.github.com/users/dieend/following{/other_user}","gists_url":"https://api.github.com/users/dieend/gists{/gist_id}","starred_url":"https://api.github.com/users/dieend/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dieend/subscriptions","organizations_url":"https://api.github.com/users/dieend/orgs","repos_url":"https://api.github.com/users/dieend/repos","events_url":"https://api.github.com/users/dieend/events{/privacy}","received_events_url":"https://api.github.com/users/dieend/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2014-12-18T16:54:52Z","updated_at":"2015-04-07T23:05:22Z","closed_at":null,"body":"```\r\n  2014-12-18 23:46:32 +0700 [warn]: emit transaction failed  error_class=Fluent::BufferQueueLimitError error=#<Fluent::BufferQueueLimitError: queue size exceeds limit>\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:182:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:168:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:427:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:33:in `next'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:172:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:168:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:538:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:55:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:55:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:528:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:171:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:33:in `next'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/out_copy.rb:73:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/match.rb:36:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:160:in `emit_stream'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:140:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:90:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:129:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:129:in `block in each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:128:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:128:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:87:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/match.rb:36:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:160:in `emit_stream'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:227:in `receive_lines'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:318:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:318:in `wrap_receive_lines'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:511:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:511:in `on_notify'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:343:in `on_notify'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:324:in `attach'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:133:in `setup_watcher'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:151:in `block in start_watchers'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:138:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:138:in `start_watchers'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:128:in `refresh_watchers'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:84:in `start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:267:in `block in start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:266:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:266:in `start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:213:in `run'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:480:in `run_engine'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:138:in `block in start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:266:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:266:in `main_process'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:241:in `block in supervise'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:240:in `fork'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:240:in `supervise'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:131:in `start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/command/fluentd.rb:168:in `<top (required)>'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/bin/fluentd:6:in `<top (required)>'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/bin/fluentd:23:in `load'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'\r\n  2014-12-18 23:46:32 +0700 [warn]: /usr/sbin/td-agent:7:in `load'\r\n  2014-12-18 23:46:32 +0700 [warn]: /usr/sbin/td-agent:7:in `<main>'\r\n  2014-12-18 23:46:32 +0700 [warn]: emit transaction failed  error_class=Fluent::BufferQueueLimitError error=#<Fluent::BufferQueueLimitError: queue size exceeds limit>\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:182:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:168:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:427:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:33:in `next'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:172:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:168:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:538:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:55:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:55:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:528:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:171:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:33:in `next'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/out_copy.rb:73:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/match.rb:36:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:160:in `emit_stream'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:140:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:90:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:55:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:55:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:87:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/match.rb:36:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:160:in `emit_stream'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:140:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_forward.rb:168:in `on_message'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_forward.rb:209:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_forward.rb:209:in `block in on_read'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_forward.rb:225:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_forward.rb:225:in `<<'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_forward.rb:225:in `on_read_json'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_forward.rb:220:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_forward.rb:220:in `on_read'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/io.rb:128:in `on_readable'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/io.rb:191:in `on_readable'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run_once'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_forward.rb:93:in `run'\r\n  2014-12-18 23:46:32 +0700 [info]: out_forest plants new output: s3 for tag 'var.log.php5-fpm.www.slow.log'\r\n  2014-12-18 23:46:32 +0700 [warn]: emit transaction failed  error_class=Fluent::BufferQueueLimitError error=#<Fluent::BufferQueueLimitError: queue size exceeds limit>\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:182:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:168:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:427:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:33:in `next'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:172:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/buffer.rb:168:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:538:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:55:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:55:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:528:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:171:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/output.rb:33:in `next'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/out_copy.rb:73:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/match.rb:36:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:160:in `emit_stream'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:140:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:90:in `block in emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:129:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:129:in `block in each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:128:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/event.rb:128:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:87:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/match.rb:36:in `emit'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:160:in `emit_stream'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:227:in `receive_lines'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:318:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:318:in `wrap_receive_lines'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:511:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:511:in `on_notify'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:343:in `on_notify'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:324:in `attach'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:133:in `setup_watcher'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:151:in `block in start_watchers'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:138:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:138:in `start_watchers'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:128:in `refresh_watchers'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/plugin/in_tail.rb:84:in `start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:267:in `block in start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:266:in `each'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:266:in `start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/engine.rb:213:in `run'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:480:in `run_engine'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:138:in `block in start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:266:in `call'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:266:in `main_process'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:241:in `block in supervise'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:240:in `fork'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:240:in `supervise'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:131:in `start'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/command/fluentd.rb:168:in `<top (required)>'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/bin/fluentd:6:in `<top (required)>'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/bin/fluentd:23:in `load'\r\n  2014-12-18 23:46:32 +0700 [warn]: /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'\r\n  2014-12-18 23:46:32 +0700 [warn]: /usr/sbin/td-agent:7:in `load'\r\n  2014-12-18 23:46:32 +0700 [warn]: /usr/sbin/td-agent:7:in `<main>'\r\n```\r\nThe log output this warning, with stack trace and it's repeat until hundreds of thousand later until the disk full, and the server's down.\r\n\r\nI tried to set `log_level` to `error`, but it is still happened. I tried to add `--suppress-repeated-stacktrace` options, the message is reduced, but it's still outputting many many many logs, it's only suppressing the stacktrace, not the messages. The only workaround is by setting global log level `-qq`.\r\n\r\nWith this I lost my ability to follow warn error. Is there anyway for me to receive as much as possible log with as little as possible log? (uh, I mean, a way to get log but not repeating infinitely).\r\n\r\nMy partial configuration:\r\n```\r\n<match log.**>\r\n  type record_reformer\r\n  renew_record false\r\n  tag reformed.${tag_suffix[1]}\r\n  <record>\r\n     key value\r\n  </record>\r\n  log_level error\r\n</match>\r\n\r\n<match reformed.**>\r\n    type copy\r\n    <store>\r\n       type forest\r\n\tsubtype s3\r\n       remove_prefix reformed\r\n       escape_tag_separator /\r\n       <template>\r\n             <!-- some s3 configuration !-->\r\n\t    buffer_path /var/log/fluent/__TAG__.*\r\n\t    buffer_chunk_limit 10m\r\n\t    flush_interval 10m\r\n\t    flush_at_shutdown true\r\n\t    format json\r\n\t    include_tag_key false\r\n\t    include_time_key true\r\n       </template>\r\n    </store>\r\n      <store>\r\n\t type forward\r\n        <server>\r\n           host somedeadhost\r\n           port 24224\r\n         </server>\r\n\t log_level error\r\n         heartbeat_type tcp\r\n\t flush_interval 10s\r\n\t num_threads 8\r\n\t buffer_type file\r\n\t buffer_chunk_limit 128kb\r\n\t buffer_queue_limit 128\r\n\t buffer_path /var/log/fluent/monitor/\r\n      </store>\r\n</match>\r\n```"},"comment":{"url":"https://api.github.com/repos/fluent/fluentd/issues/comments/90754415","html_url":"https://github.com/fluent/fluentd/issues/523#issuecomment-90754415","issue_url":"https://api.github.com/repos/fluent/fluentd/issues/523","id":90754415,"user":{"login":"repeatedly","id":16928,"avatar_url":"https://avatars.githubusercontent.com/u/16928?v=3","gravatar_id":"","url":"https://api.github.com/users/repeatedly","html_url":"https://github.com/repeatedly","followers_url":"https://api.github.com/users/repeatedly/followers","following_url":"https://api.github.com/users/repeatedly/following{/other_user}","gists_url":"https://api.github.com/users/repeatedly/gists{/gist_id}","starred_url":"https://api.github.com/users/repeatedly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/repeatedly/subscriptions","organizations_url":"https://api.github.com/users/repeatedly/orgs","repos_url":"https://api.github.com/users/repeatedly/repos","events_url":"https://api.github.com/users/repeatedly/events{/privacy}","received_events_url":"https://api.github.com/users/repeatedly/received_events","type":"User","site_admin":false},"created_at":"2015-04-07T23:05:22Z","updated_at":"2015-04-07T23:05:22Z","body":"Yeah, we need to improve logging mechanizm for better control.\r\n\r\n> are there situations where fluentd gets stuck in a tight loop\r\n\r\nIt should not happen in the fluentd's main thread. In the plugin thread, it may happen by plugin bug.\r\n\r\n> log rotation should keep the disk from filling up\r\n\r\nWe are now using `logrotate` for fluentd(td-agent) logs.  Do you see any concern?\r\n\r\n> --suppress-repeated-stacktrace helps reduce the volume in some cases (should this be the default)\r\n\r\nFrom v0.12, this is true by default.\r\n\r\n>  although it appears this is also off by default and not documented outside of the code.\r\n>  but it is not explained.\r\n\r\nAh! I will update documents later.\r\n\r\n> I still see 'syslog failed to emit' every time.\r\n\r\nHmm... currently, fluentd uses suppression per thread.\r\nWe will change this to suppression per plugin."}}