{"action":"created","issue":{"url":"https://api.github.com/repos/paulmillr/chokidar/issues/410","labels_url":"https://api.github.com/repos/paulmillr/chokidar/issues/410/labels{/name}","comments_url":"https://api.github.com/repos/paulmillr/chokidar/issues/410/comments","events_url":"https://api.github.com/repos/paulmillr/chokidar/issues/410/events","html_url":"https://github.com/paulmillr/chokidar/issues/410","id":120614103,"number":410,"title":"Proposal - simplify initial scan","user":{"login":"kmalakoff","id":756520,"avatar_url":"https://avatars.githubusercontent.com/u/756520?v=3","gravatar_id":"","url":"https://api.github.com/users/kmalakoff","html_url":"https://github.com/kmalakoff","followers_url":"https://api.github.com/users/kmalakoff/followers","following_url":"https://api.github.com/users/kmalakoff/following{/other_user}","gists_url":"https://api.github.com/users/kmalakoff/gists{/gist_id}","starred_url":"https://api.github.com/users/kmalakoff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmalakoff/subscriptions","organizations_url":"https://api.github.com/users/kmalakoff/orgs","repos_url":"https://api.github.com/users/kmalakoff/repos","events_url":"https://api.github.com/users/kmalakoff/events{/privacy}","received_events_url":"https://api.github.com/users/kmalakoff/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":21,"created_at":"2015-12-06T06:50:56Z","updated_at":"2015-12-06T22:22:34Z","closed_at":null,"body":"I [wrote the author of readdirp](https://github.com/thlorenz/readdirp/issues/35) a few weeks ago because after tracking down the source of my electron application lockup at startup, I am pretty convinced that its approach to blasting the filesystem with a huge number of parallel requests is flawed / not scalable. Plus, I think this approach is the source of many bugs or performance issues (such as https://github.com/paulmillr/chokidar/issues/229) and has caused workarounds like needing graceful-fs, backpressure, etc. \r\n\r\nIn my personal project I have replaced readdirp in chokidar with a simple walk implementation to meet my needs and things are smoother and much faster for me (no numbers, but first want to float an idea). \r\n\r\nBasically, my approach was to make it massively serial (async.eachSeries) so that it played nicely at any scale and to simplfy it since there is some funky async logic in it. Obviously, some parallelism could improve throughput, but it adds complexity to manage performance for large scans (but still is probably desired).\r\n\r\nProposal: replace readdirp\r\n------------\r\n\r\nI could extract my filtered walk into a small library. My current API is simply:\r\n\r\n    walk(rootPath, (path, stat) { /* filter */ }, done)\r\n      .on('file', function(path, stat) {})\r\n      .on('directory', function(path, stat) {});\r\n\r\nIt would do lstat before every filter call like you rely on, but things like globing and depth would be left up to the library user. \r\n\r\nCustomizing it to chokidar's needs:\r\n\r\n1. _isIgnored would be quite simple since the filter is called with stats, you can process directories and files differently - if you filter a directory, it doesn't traverse it and there is no event emitter; whereas, if you filter a file, you just do not get an event for it. \r\n\r\n2. depth would be up to splitting the directly path and comparing the number of links\r\n\r\n3. readdirp 'entry' could be generated (or partially generated with only what you need) when emitting the file or directory.\r\n\r\n4. globbing would be handled in the filter\r\n\r\n\r\nThis change would require a small to medium refactor of chokidar so before considering me developing the code and some performance tests, I was wondering if you agree with the philosophy of the proposal (serial and no graceful-fs), have feedback, etc. Also, I'm not sure about all of the edge cases so I might be proposing something with a serious flaw.\r\n\r\nInterested? Suggestions (like an infinitely scalable and simple parallelism strategy)?\r\n"},"comment":{"url":"https://api.github.com/repos/paulmillr/chokidar/issues/comments/162354056","html_url":"https://github.com/paulmillr/chokidar/issues/410#issuecomment-162354056","issue_url":"https://api.github.com/repos/paulmillr/chokidar/issues/410","id":162354056,"user":{"login":"kmalakoff","id":756520,"avatar_url":"https://avatars.githubusercontent.com/u/756520?v=3","gravatar_id":"","url":"https://api.github.com/users/kmalakoff","html_url":"https://github.com/kmalakoff","followers_url":"https://api.github.com/users/kmalakoff/followers","following_url":"https://api.github.com/users/kmalakoff/following{/other_user}","gists_url":"https://api.github.com/users/kmalakoff/gists{/gist_id}","starred_url":"https://api.github.com/users/kmalakoff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmalakoff/subscriptions","organizations_url":"https://api.github.com/users/kmalakoff/orgs","repos_url":"https://api.github.com/users/kmalakoff/repos","events_url":"https://api.github.com/users/kmalakoff/events{/privacy}","received_events_url":"https://api.github.com/users/kmalakoff/received_events","type":"User","site_admin":false},"created_at":"2015-12-06T22:22:34Z","updated_at":"2015-12-06T22:22:34Z","body":"@es128 Thank you for helping!  \r\n\r\nI use immediate and I've been writing [fs-memory](https://github.com/kmalakoff/fs-memory), but the problem isn't in the browser (and the library is a work-in-progress, eg. not for production), the problems were more under scale in electron with the actual filesystem and real data-sizes. The memory version is more for development and leaving the door open for a browser (non-electron) version of the application in the future.\r\n\r\nI think you are following a red-herring...like I said, playing nice for rendering and the above problems are key `it batches up directory processing rather than emitting file by file and directory by directory, it has no control over parallelism, etc`\r\n\r\nI'd rather you focus on making chokidar more performant than my problems since I have already optimized my application to meets my needs. I'm just trying to share what I did with chokidar to work around readdirp and help the broader community.\r\n"}}