{"action":"created","issue":{"url":"https://api.github.com/repos/eslint/eslint/issues/3565","labels_url":"https://api.github.com/repos/eslint/eslint/issues/3565/labels{/name}","comments_url":"https://api.github.com/repos/eslint/eslint/issues/3565/comments","events_url":"https://api.github.com/repos/eslint/eslint/issues/3565/events","html_url":"https://github.com/eslint/eslint/issues/3565","id":103776806,"number":3565,"title":"Lint multiple files in parallel","user":{"login":"ilyavolodin","id":1380062,"avatar_url":"https://avatars.githubusercontent.com/u/1380062?v=3","gravatar_id":"","url":"https://api.github.com/users/ilyavolodin","html_url":"https://github.com/ilyavolodin","followers_url":"https://api.github.com/users/ilyavolodin/followers","following_url":"https://api.github.com/users/ilyavolodin/following{/other_user}","gists_url":"https://api.github.com/users/ilyavolodin/gists{/gist_id}","starred_url":"https://api.github.com/users/ilyavolodin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilyavolodin/subscriptions","organizations_url":"https://api.github.com/users/ilyavolodin/orgs","repos_url":"https://api.github.com/users/ilyavolodin/repos","events_url":"https://api.github.com/users/ilyavolodin/events{/privacy}","received_events_url":"https://api.github.com/users/ilyavolodin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/eslint/eslint/labels/accepted","name":"accepted","color":"0052cc"},{"url":"https://api.github.com/repos/eslint/eslint/labels/cli","name":"cli","color":"eb6420"},{"url":"https://api.github.com/repos/eslint/eslint/labels/feature","name":"feature","color":"bfd4f2"},{"url":"https://api.github.com/repos/eslint/eslint/labels/needs-bikeshedding","name":"needs-bikeshedding","color":"f7c6c7"}],"state":"open","locked":false,"assignee":{"login":"ilyavolodin","id":1380062,"avatar_url":"https://avatars.githubusercontent.com/u/1380062?v=3","gravatar_id":"","url":"https://api.github.com/users/ilyavolodin","html_url":"https://github.com/ilyavolodin","followers_url":"https://api.github.com/users/ilyavolodin/followers","following_url":"https://api.github.com/users/ilyavolodin/following{/other_user}","gists_url":"https://api.github.com/users/ilyavolodin/gists{/gist_id}","starred_url":"https://api.github.com/users/ilyavolodin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilyavolodin/subscriptions","organizations_url":"https://api.github.com/users/ilyavolodin/orgs","repos_url":"https://api.github.com/users/ilyavolodin/repos","events_url":"https://api.github.com/users/ilyavolodin/events{/privacy}","received_events_url":"https://api.github.com/users/ilyavolodin/received_events","type":"User","site_admin":false},"milestone":null,"comments":37,"created_at":"2015-08-28T18:59:59Z","updated_at":"2015-12-04T23:30:16Z","closed_at":null,"body":"This is a discussion issue for adding ability to run eslint in parallel for multiple files.\r\n\r\nThe idea is that ESLint is mostly CPU bound, not IO bound, so creating multiple threads (for machine with multiple cores) might (and probably will) increase performance in a meaningful way. The downside is that currently ESLint's codebase is synchronous. So this would require rewriting everything up to and including eslint.js to be asynchronous, which would be a major effort.\r\n\r\nI played with this a little while ago and found a few libraries for Node that handle thread pool, including detection of number of cores available on the machine.\r\n* [Node-threads-a-gogo](https://github.com/xk/node-threads-a-gogo) - seems pretty good, but looks dead.\r\n* [nPool](https://github.com/inh3/nPool) - seems actively in development, but has native components (C++)\r\n* [Node WebWorkers](https://github.com/audreyt/node-webworker-threads) - seems pretty dead too.\r\n* [Parallel](https://github.com/adambom/parallel.js) - seems dead, and no pool implementation.\r\n* [Node Clusters](https://nodejs.org/api/cluster.html) - not stable yet, and probably isn't going to be available on Node v0.10\r\n* WebWorkers - seems that they are only implemented in io.js\r\nAnd there are a ton of other libraries out there for this.\r\n\r\nIf anyone had any experience writing multithreaded applications for node.js and would like to suggest alternatives or comment on the above list, please feel free.\r\n\r\nP.S. https://www.airpair.com/javascript/posts/which-async-javascript-libraries-should-i-use\r\n\r\n<bountysource-plugin>\r\n---\r\nWant to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/26284182-lint-multiple-files-in-parallel?utm_campaign=plugin&utm_content=tracker%2F282608&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F282608&utm_medium=issues&utm_source=github).\r\n</bountysource-plugin>"},"comment":{"url":"https://api.github.com/repos/eslint/eslint/issues/comments/162109733","html_url":"https://github.com/eslint/eslint/issues/3565#issuecomment-162109733","issue_url":"https://api.github.com/repos/eslint/eslint/issues/3565","id":162109733,"user":{"login":"royriojas","id":131290,"avatar_url":"https://avatars.githubusercontent.com/u/131290?v=3","gravatar_id":"","url":"https://api.github.com/users/royriojas","html_url":"https://github.com/royriojas","followers_url":"https://api.github.com/users/royriojas/followers","following_url":"https://api.github.com/users/royriojas/following{/other_user}","gists_url":"https://api.github.com/users/royriojas/gists{/gist_id}","starred_url":"https://api.github.com/users/royriojas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/royriojas/subscriptions","organizations_url":"https://api.github.com/users/royriojas/orgs","repos_url":"https://api.github.com/users/royriojas/repos","events_url":"https://api.github.com/users/royriojas/events{/privacy}","received_events_url":"https://api.github.com/users/royriojas/received_events","type":"User","site_admin":false},"created_at":"2015-12-04T23:30:15Z","updated_at":"2015-12-04T23:30:15Z","body":"hi @ilyavolodin, \r\n\r\nSo here is my take, I do agree with you that we should probably move the code for the cache to a better place where it is now (`executeOnFiles`). In a perfect world the process should be something like this:\r\n\r\n**In the main process:**\r\n\r\n- define which files need to be linted.\r\n- if cache is enabled use the cache to filter out the ones that haven't changed since they pass with 0 lint errors\r\n- with the new list of files create a queue of files to be linted\r\n- spawn some processes to run the actual linter\r\n\r\n**In the forked process**\r\n\r\n- lint the code (no cache work on this side) just linting\r\n- send back the result\r\n- wait for more files to lint\r\n\r\n** In the main process**\r\n\r\n- receive the results and collect them\r\n- if cache is enabled update the cache entry for the file, if it passed it's stored if it hasn't passed it is removed from the cache. so the next time the process start the file will be linted\r\n- if more files require linting pass the new file to the spawned process\r\n- wait until 0 files\r\n- print the collected results\r\n- persist the cache to disk\r\n- be happy with faster linting!\r\n\r\nI don't know if this will be feasible, but I guess it will make the process simpler in order to also use the cache.\r\n\r\nLet me know what you think about it."}}