{"action":"created","issue":{"url":"https://api.github.com/repos/jordansissel/ruby-filewatch/issues/56","labels_url":"https://api.github.com/repos/jordansissel/ruby-filewatch/issues/56/labels{/name}","comments_url":"https://api.github.com/repos/jordansissel/ruby-filewatch/issues/56/comments","events_url":"https://api.github.com/repos/jordansissel/ruby-filewatch/issues/56/events","html_url":"https://github.com/jordansissel/ruby-filewatch/issues/56","id":79025298,"number":56,"title":"sincedb uses file.read bytes instead of bytes up to last full line thus losing data","user":{"login":"jsvd","id":31809,"avatar_url":"https://avatars.githubusercontent.com/u/31809?v=3","gravatar_id":"","url":"https://api.github.com/users/jsvd","html_url":"https://github.com/jsvd","followers_url":"https://api.github.com/users/jsvd/followers","following_url":"https://api.github.com/users/jsvd/following{/other_user}","gists_url":"https://api.github.com/users/jsvd/gists{/gist_id}","starred_url":"https://api.github.com/users/jsvd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsvd/subscriptions","organizations_url":"https://api.github.com/users/jsvd/orgs","repos_url":"https://api.github.com/users/jsvd/repos","events_url":"https://api.github.com/users/jsvd/events{/privacy}","received_events_url":"https://api.github.com/users/jsvd/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-05-21T14:40:33Z","updated_at":"2015-06-04T16:54:44Z","closed_at":"2015-06-04T16:54:44Z","body":"scenario:\r\n\r\ninput.txt of size 45k has 3 lines: line A (12k bytes), line B (25k), line C (8k)\r\n\r\nstart logstash with `input { path => \"/tmp/input.txt\" }`:\r\n\r\n1. filewatch.tail's `_read_file` does a file.read(32768) and stores line A and part of line B in a FileWatch::BufferedTokenizer\r\n2. buffer.extract(data) returns line A but keeps the 20k of line B in memory\r\n3. line A is yielded to the block calling `subscribe`\r\n4.  `@sincedb[@statcache[path]] = @files[path].pos` is called to remember in memory that \"we've read 32k of this file\"\r\n5. Control+C happens => sincedb is written saying \"I've read 32k of the file by this inode\"\r\n6.  -------- logstash is terminated ----------- \r\n7. logstash is started again\r\n8. a file seek is done to bring the file pointer to the stored `@sincedb[@statcache[path]]` position (32k)\r\n9. filewatch.tail's `_read_file` does a file.read(32768) again and stores the last 15k of line B and line C in an empty FileWatch::BufferedTokenizer\r\n10. buffer.extract(data) returns the last portion of line B and line C leaving the buffer empty\r\n11. lines are yielded to the block calling `subscribe` \r\n\r\nResult of the tail:  line A, last 15k of line B and line C thus losing 20k of line B"},"comment":{"url":"https://api.github.com/repos/jordansissel/ruby-filewatch/issues/comments/108968765","html_url":"https://github.com/jordansissel/ruby-filewatch/issues/56#issuecomment-108968765","issue_url":"https://api.github.com/repos/jordansissel/ruby-filewatch/issues/56","id":108968765,"user":{"login":"jsvd","id":31809,"avatar_url":"https://avatars.githubusercontent.com/u/31809?v=3","gravatar_id":"","url":"https://api.github.com/users/jsvd","html_url":"https://github.com/jsvd","followers_url":"https://api.github.com/users/jsvd/followers","following_url":"https://api.github.com/users/jsvd/following{/other_user}","gists_url":"https://api.github.com/users/jsvd/gists{/gist_id}","starred_url":"https://api.github.com/users/jsvd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsvd/subscriptions","organizations_url":"https://api.github.com/users/jsvd/orgs","repos_url":"https://api.github.com/users/jsvd/repos","events_url":"https://api.github.com/users/jsvd/events{/privacy}","received_events_url":"https://api.github.com/users/jsvd/received_events","type":"User","site_admin":false},"created_at":"2015-06-04T16:54:44Z","updated_at":"2015-06-04T16:54:44Z","body":"fixed by ba083a1"}}