{"action":"created","issue":{"url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1016","labels_url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1016/labels{/name}","comments_url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1016/comments","events_url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1016/events","html_url":"https://github.com/fail2ban/fail2ban/issues/1016","id":66799250,"number":1016,"title":"Option to limit initial file scanning","user":{"login":"morphles","id":9117207,"avatar_url":"https://avatars.githubusercontent.com/u/9117207?v=3","gravatar_id":"","url":"https://api.github.com/users/morphles","html_url":"https://github.com/morphles","followers_url":"https://api.github.com/users/morphles/followers","following_url":"https://api.github.com/users/morphles/following{/other_user}","gists_url":"https://api.github.com/users/morphles/gists{/gist_id}","starred_url":"https://api.github.com/users/morphles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/morphles/subscriptions","organizations_url":"https://api.github.com/users/morphles/orgs","repos_url":"https://api.github.com/users/morphles/repos","events_url":"https://api.github.com/users/morphles/events{/privacy}","received_events_url":"https://api.github.com/users/morphles/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-07T07:00:11Z","updated_at":"2015-04-07T11:15:19Z","closed_at":null,"body":"Parsing/scaning large log files is a bit problematic, say files >500MB take some time and CPU power to be parsed. Also modyfing log format to drop info and thus make file smaller just to run fail2ban does not sound optimal/desiarable. It would seem that very often, most of log file is not that relevant. Well at least in my use case, probably days worth of logging, at most, is what is relevant. Though I could even live with no \"pre-scanning\" of log file. So it would be nice if jail could define something like:\r\nlog_tail = <num>\r\nIf that is specified it should only look at <num> log lines on startup. Additionall alternative (though considerably more complicated to implement):\r\nlog_scan_only_last_min = <num>\r\nIf that is specified only, scan lines that have timestamps that are less than num minutes before now. Obviously scanning would need to happen backwards in such case.\r\n\r\nOr at least option to disable \"pre-scaning\" miget even be enough. Though obviously this (well all mentioned options) should be on per jail basis."},"comment":{"url":"https://api.github.com/repos/fail2ban/fail2ban/issues/comments/90509255","html_url":"https://github.com/fail2ban/fail2ban/issues/1016#issuecomment-90509255","issue_url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1016","id":90509255,"user":{"login":"sebres","id":3769657,"avatar_url":"https://avatars.githubusercontent.com/u/3769657?v=3","gravatar_id":"","url":"https://api.github.com/users/sebres","html_url":"https://github.com/sebres","followers_url":"https://api.github.com/users/sebres/followers","following_url":"https://api.github.com/users/sebres/following{/other_user}","gists_url":"https://api.github.com/users/sebres/gists{/gist_id}","starred_url":"https://api.github.com/users/sebres/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebres/subscriptions","organizations_url":"https://api.github.com/users/sebres/orgs","repos_url":"https://api.github.com/users/sebres/repos","events_url":"https://api.github.com/users/sebres/events{/privacy}","received_events_url":"https://api.github.com/users/sebres/received_events","type":"User","site_admin":false},"created_at":"2015-04-07T11:15:19Z","updated_at":"2015-04-07T11:15:19Z","body":"Option `log_tail` is imho not required, because since v.0.9 the last offset of log file will be saved in database and after restart, f2b will parse file after this position only (if not new file).\r\nSuggested option `log_scan_only_last_min` is indirect already included - it's `findtime`.\r\n\r\nBut current implementation of filter makes a full scan (to find a time `now` - `findtime`), if we have a new log file / initial state (new jail enabled). In addition it have a little bug (see https://github.com/fail2ban/fail2ban/issues/795#issuecomment-57732657).\r\n\r\nI have developed a new algorithm (half-interval search) for very fast seek to `now-findtime` within my major branch #716 - see https://github.com/sebres/fail2ban/commit/53a30a2d42fa9cc114335cc7328651637f12162c and https://github.com/sebres/fail2ban/commit/c1637e97b2e210a6351e356aa351f9efe6109c3c\r\nIt can be cheery-picked into master branch together with bug fix above. \r\n@fail2ban/contributors should I do it?"}}