{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/308","labels_url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/308/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/308/comments","events_url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/308/events","html_url":"https://github.com/elastic/logstash-forwarder/issues/308","id":48073090,"number":308,"title":"Log Rotate: Rotated file still in use","user":{"login":"Rayzilt","id":3105311,"avatar_url":"https://avatars.githubusercontent.com/u/3105311?v=3","gravatar_id":"","url":"https://api.github.com/users/Rayzilt","html_url":"https://github.com/Rayzilt","followers_url":"https://api.github.com/users/Rayzilt/followers","following_url":"https://api.github.com/users/Rayzilt/following{/other_user}","gists_url":"https://api.github.com/users/Rayzilt/gists{/gist_id}","starred_url":"https://api.github.com/users/Rayzilt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Rayzilt/subscriptions","organizations_url":"https://api.github.com/users/Rayzilt/orgs","repos_url":"https://api.github.com/users/Rayzilt/repos","events_url":"https://api.github.com/users/Rayzilt/events{/privacy}","received_events_url":"https://api.github.com/users/Rayzilt/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/logstash-forwarder/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/elastic/logstash-forwarder/labels/filebeat","name":"filebeat","color":"006b75"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":21,"created_at":"2014-11-07T11:35:20Z","updated_at":"2015-11-16T09:29:41Z","closed_at":null,"body":"It appears that logstash-forwarder keeps the rotated log files open.\r\n\r\n1. Logrotate rotates log around 00:05 from access.log to access.log.1\r\n2. We do some scripting around 00:30 against access.log.1\r\n3. We move access.log.1 to a different location.\r\n4. Logstash-forwarder crashes\r\n\r\n/var/log/nginx is a NFS share. On a different server we mount this share for scripting purposes and also move the file away. This is the problem because the web-server doesn't know that the file is gone, and so logstash-forwarder either.\r\n\r\nThe question is: Why does logstash-forwarder keeps the rotated file open?\r\n\r\nVersion: HEAD~19812cf\r\nOS: Debian 6.0.10\r\nLinux 3.2.63-grsec #2 SMP Thu Oct 16 18:27:02 CEST 2014 x86_64 GNU/Linux\r\n\r\nDebug information:\r\n\r\n2014/11/07 00:05:13.895018 Launching harvester on rotated file: /var/log/nginx/access.log\r\n2014/11/07 00:05:13.896337 harvest: \"/var/log/nginx/access.log\" (offset snapshot:0)\r\n2014/11/07 00:05:14.168639 Registrar: precessing 1024 events\r\n..........\r\n2014/11/07 00:32:59.485583 Registrar: precessing 359 events\r\n2014/11/07 00:33:04.090287 error: Harvester.readLine: read /var/log/nginx/access.log: no such file or directory\r\n2014/11/07 00:33:04.090321 Unexpected state reading from /var/log/nginx/access.log; error: read /var/log/nginx/access.log: no such file or directory\r\npanic: runtime error: invalid memory address or nil pointer dereference\r\n\r\ngoroutine 22 [running]:\r\nruntime.panic(0x675cc0, 0x80cbb3)\r\n\t/usr/local/go/src/pkg/runtime/panic.c:279 +0xf5\r\nmain.(*Harvester).Harvest(0xc2080041e0, 0xc20807c0e0)\r\n\t/root/downloads/logstash-forwarder/harvester.go:60 +0x529\r\ncreated by main.(*Prospector).scan\r\n\t/root/downloads/logstash-forwarder/prospector.go:164 +0xe24\r\n\r\ngoroutine 16 [chan receive]:\r\nmain.Registrar(0xc20800f440, 0xc2080757a0)\r\n\t/root/downloads/logstash-forwarder/registrar.go:9 +0x79\r\nmain.main()\r\n\t/root/downloads/logstash-forwarder/logstash-forwarder.go:212 +0xde4\r\n\r\ngoroutine 19 [finalizer wait, 805 minutes]:\r\nruntime.park(0x41fc60, 0x810e90, 0x80f929)\r\n\t/usr/local/go/src/pkg/runtime/proc.c:1369 +0x89\r\nruntime.parkunlock(0x810e90, 0x80f929)\r\n\t/usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b\r\nrunfinq()\r\n\t/usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf\r\nruntime.goexit()\r\n\t/usr/local/go/src/pkg/runtime/proc.c:1445\r\n\r\ngoroutine 20 [sleep]:\r\ntime.Sleep(0x2540be400)\r\n\t/usr/local/go/src/pkg/runtime/time.goc:39 +0x31\r\nmain.(*Prospector).Prospect(0xc208005c20, 0xc2080014c0, 0xc20807c0e0)\r\n\t/root/downloads/logstash-forwarder/prospector.go:67 +0x5e1\r\ncreated by main.main\r\n\t/root/downloads/logstash-forwarder/logstash-forwarder.go:183 +0xb2d\r\n\r\ngoroutine 21 [sleep]:\r\ntime.Sleep(0x2540be400)\r\n\t/usr/local/go/src/pkg/runtime/time.goc:39 +0x31\r\nmain.(*Prospector).Prospect(0xc208005c80, 0xc2080014c0, 0xc20807c0e0)\r\n\t/root/downloads/logstash-forwarder/prospector.go:67 +0x5e1\r\ncreated by main.main\r\n\t/root/downloads/logstash-forwarder/logstash-forwarder.go:183 +0xb2d\r\n\r\ngoroutine 23 [sleep]:\r\ntime.Sleep(0x3b9aca00)\r\n\t/usr/local/go/src/pkg/runtime/time.goc:39 +0x31\r\nmain.(*Harvester).readline(0xc208004300, 0xc208005da0, 0xc20803a0e0, 0x2540be400, 0x0, 0x0, 0x0, 0x0)\r\n\t/root/downloads/logstash-forwarder/harvester.go:152 +0x1d0\r\nmain.(*Harvester).Harvest(0xc208004300, 0xc20807c0e0)\r\n\t/root/downloads/logstash-forwarder/harvester.go:53 +0x476\r\ncreated by main.(*Prospector).scan\r\n\t/root/downloads/logstash-forwarder/prospector.go:164 +0xe24\r\n\r\n...............................\r\n\r\ngoroutine 305 [sleep]:\r\ntime.Sleep(0x3b9aca00)\r\n\t/usr/local/go/src/pkg/runtime/time.goc:39 +0x31\r\nmain.(*Harvester).readline(0xc208936ba0, 0xc208937440, 0xc2087ff5e0, 0x2540be400, 0x0, 0x0, 0x0, 0x0)\r\n\t/root/downloads/logstash-forwarder/harvester.go:152 +0x1d0\r\nmain.(*Harvester).Harvest(0xc208936ba0, 0xc20807c0e0)\r\n\t/root/downloads/logstash-forwarder/harvester.go:53 +0x476\r\ncreated by main.(*Prospector).scan\r\n\t/root/downloads/logstash-forwarder/prospector.go:187 +0x13e9\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/comments/156968590","html_url":"https://github.com/elastic/logstash-forwarder/issues/308#issuecomment-156968590","issue_url":"https://api.github.com/repos/elastic/logstash-forwarder/issues/308","id":156968590,"user":{"login":"fitz123","id":10243861,"avatar_url":"https://avatars.githubusercontent.com/u/10243861?v=3","gravatar_id":"","url":"https://api.github.com/users/fitz123","html_url":"https://github.com/fitz123","followers_url":"https://api.github.com/users/fitz123/followers","following_url":"https://api.github.com/users/fitz123/following{/other_user}","gists_url":"https://api.github.com/users/fitz123/gists{/gist_id}","starred_url":"https://api.github.com/users/fitz123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fitz123/subscriptions","organizations_url":"https://api.github.com/users/fitz123/orgs","repos_url":"https://api.github.com/users/fitz123/repos","events_url":"https://api.github.com/users/fitz123/events{/privacy}","received_events_url":"https://api.github.com/users/fitz123/received_events","type":"User","site_admin":false},"created_at":"2015-11-16T09:29:41Z","updated_at":"2015-11-16T09:29:41Z","body":"The question was about logstash-forwarder logs itself (/var/log/logstash-forwarder). Currently it continues log into rotated file (logstash-forwarder.err.1) and service restart is my workaround\n\nBut I get from your answer that forwarder is not supported anymore and current replacement is filebeat? Am I right?\n\nBest Regards. A. L.\n\n> On 16 Nov 2015, at 14:58, Nicolas Ruflin <notifications@github.com> wrote:\n> \n> @fitz123 If it is about closing files after a certain time, ignore_older in filebeat is the current solution: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html#_ignore_older But related to this, also check issue elastic/filebeat#181\n> \n> —\n> Reply to this email directly or view it on GitHub.\n> \n"}}