{"action":"created","issue":{"url":"https://api.github.com/repos/Shougo/unite.vim/issues/848","labels_url":"https://api.github.com/repos/Shougo/unite.vim/issues/848/labels{/name}","comments_url":"https://api.github.com/repos/Shougo/unite.vim/issues/848/comments","events_url":"https://api.github.com/repos/Shougo/unite.vim/issues/848/events","html_url":"https://github.com/Shougo/unite.vim/issues/848","id":57497317,"number":848,"title":"Filtering is terribly slow","user":{"login":"balta2ar","id":340676,"avatar_url":"https://avatars.githubusercontent.com/u/340676?v=3","gravatar_id":"","url":"https://api.github.com/users/balta2ar","html_url":"https://github.com/balta2ar","followers_url":"https://api.github.com/users/balta2ar/followers","following_url":"https://api.github.com/users/balta2ar/following{/other_user}","gists_url":"https://api.github.com/users/balta2ar/gists{/gist_id}","starred_url":"https://api.github.com/users/balta2ar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/balta2ar/subscriptions","organizations_url":"https://api.github.com/users/balta2ar/orgs","repos_url":"https://api.github.com/users/balta2ar/repos","events_url":"https://api.github.com/users/balta2ar/events{/privacy}","received_events_url":"https://api.github.com/users/balta2ar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-02-12T18:10:12Z","updated_at":"2015-02-16T08:06:25Z","closed_at":null,"body":"Unfortunately I'm having a bad user experience when using `file_rec/async`. To take it to an extreme, I run it in the root of the linux kernel tree, that's about 50K files: `Unite file_rec/async`. I do not use any fancy fuzzy matchers (well, I think, I do not overwrite any default matcher options), or anything other than a default setup. In fact, it's not find that is slow:\r\n\r\n```\r\nlinux-3.19 $ time find -type f | wc -l                                                    \r\n48425\r\nnoglob find -type f  0.03s user 0.07s system 98% cpu 0.105 total\r\n```\r\n\r\nIf I understand it right, Unite caches the results of find. If I run `file_rec/async` for first time on linux tree, then close the Unite window, and then run `file_rec/async` again, it reacts too slow (it opens in less than a second, but very close it). Also, when I type something, it reacts equally painfully slowly.\r\n\r\nFor comparison, simple grep runs relatively fast on this moderately sized data:\r\n\r\n```\r\nlinux-3.19 $ time find -type f | grep radeon/cik | wc\r\n      6       6     318\r\nnoglob find -type f  0.03s user 0.07s system 94% cpu 0.109 total\r\ngrep -i --color=always radeon/cik  0.02s user 0.00s system 18% cpu 0.109 total\r\nwc  0.00s user 0.00s system 0% cpu 0.110 total\r\n```\r\n\r\nI tried to dig into the source code trying to find out who's making it slow. Before executing `file_rec/async` I used profile as follows: `:profile start vim.log | profile file * | profile func *`. I noticed that filtering function is taking quite some time: `function! unite#filters#lua_filter_patterns(candidates, patterns, whites)` (I have +lua). Then I put `return a:candidates` as the first line in it but still it was terribly slow.\r\n\r\nHere are the profile results with `unite#filters#lua_filter_patterns` doing nothing (simply returning original candidates). I only started `:Unite file_rec/async`, didn't type anything and quit vim.\r\n\r\n```\r\nFUNCTION  522()\r\nCalled 1 time\r\nTotal time:   0.468213\r\n Self time:   0.468213\r\n\r\ncount  total (s)   self (s)\r\n    1              0.000003   if stridx(a:context.input, '.') >= 0\r\n                                return unite#filters#filter_matcher( a:candidates, '', a:context)\r\n                              endif\r\n                            \r\n                              \"return a:candidates\r\n    1              0.468206   return filter(a:candidates, \" get(v:val, 'action__path', v:val.word) !~    '\\\\%(^\\\\|/\\\\)\\\\.[^/]*/\\\\?$'\")\r\n\r\nFUNCTIONS SORTED ON TOTAL TIME\r\ncount  total (s)   self (s)  function\r\n    1   1.075987   0.000351  <SNR>109_call_unite_empty()\r\n    1   1.071344   0.000457  unite#start()\r\n    1   1.069877   0.001084  unite#start#standard()\r\n    1   1.007665   0.000514  unite#candidates#_recache()\r\n    1   1.005815   0.000301  <SNR>180_recache_candidates_loop()\r\n    2   0.560795   0.000130  unite#helper#call_source_filters()\r\n    5   0.560665   0.000097  unite#helper#call_filter()\r\n    1   0.468213             522()\r\n    1   0.423415   0.028123  <SNR>180_get_source_candidates()\r\n    1   0.389082   0.030104  507()\r\n    1   0.345626   0.207106  <SNR>176_init_continuation()\r\n    1   0.118036             unite#helper#paths2candidates()\r\n    1   0.078722   0.000077  516()\r\n    1   0.078617   0.078614  unite#filters#converter_relative_word#lua()\r\n    8   0.045314   0.003302  <SNR>163__import()\r\n    5   0.038448   0.000286  <SNR>163_import()\r\n    9   0.031766   0.017227  unite#init#_default_scripts()\r\n    1   0.030565   0.000552  unite#init#_unite_buffer()\r\n    1   0.024920   0.000811  unite#init#_current_unite()\r\n    5   0.023946   0.000083  <SNR>120_on_window_changed()\r\n\r\nFUNCTIONS SORTED ON SELF TIME\r\ncount  total (s)   self (s)  function\r\n    1              0.468213  522()\r\n    1   0.345626   0.207106  <SNR>176_init_continuation()\r\n    1              0.118036  unite#helper#paths2candidates()\r\n    1   0.078617   0.078614  unite#filters#converter_relative_word#lua()\r\n    1   0.389082   0.030104  507()\r\n    1   0.423415   0.028123  <SNR>180_get_source_candidates()\r\n    1   0.020278   0.020199  <SNR>177_readfile()\r\n    9   0.031766   0.017227  unite#init#_default_scripts()\r\n    8   0.019838   0.012000  <SNR>163__build_module()\r\n   11   0.018173   0.009906  <SNR>163__scripts()\r\n   14              0.009884  <SNR>163__redir()\r\n  240              0.007611  <SNR>150_get_syn()\r\n  162              0.007030  airline#highlighter#exec()\r\n    3   0.017505   0.005467  airline#highlighter#highlight()\r\n   10              0.005102  <SNR>163__runtime_files()\r\n   87              0.004993  <SNR>198_is_file_exist()\r\n    1   0.006192   0.004907  <SNR>180_ignore_candidates()\r\n   46   0.016926   0.004407  425()\r\n  120   0.013560   0.004223  airline#highlighter#get_highlight()\r\n    2              0.003735  <SNR>198_uniq_by()\r\n```\r\n(full log is here: http://sprunge.us/GGND)\r\n\r\nI know that VimL is known to be rather slow language, but maybe still there is something that could be done? Maybe reduce some excessive caching or `deepcopy`ing or implement filtering in `C`, I don't know.\r\n\r\nCould there be some kind of \"filtering backend\", sort of completion backend that does the filtering blazingly fast and returns only very small potion of result? At any given point of time there's still a limit to the number lines that we can show on the screen anyway, plus it looks like `file_rec/async` is also limiting the output: `file_rec/async(50/48436)` -- I mean the first number = `50`.\r\n\r\nWhat do you think about all that? Is it possible to speed things up?"},"comment":{"url":"https://api.github.com/repos/Shougo/unite.vim/issues/comments/74471772","html_url":"https://github.com/Shougo/unite.vim/issues/848#issuecomment-74471772","issue_url":"https://api.github.com/repos/Shougo/unite.vim/issues/848","id":74471772,"user":{"login":"balta2ar","id":340676,"avatar_url":"https://avatars.githubusercontent.com/u/340676?v=3","gravatar_id":"","url":"https://api.github.com/users/balta2ar","html_url":"https://github.com/balta2ar","followers_url":"https://api.github.com/users/balta2ar/followers","following_url":"https://api.github.com/users/balta2ar/following{/other_user}","gists_url":"https://api.github.com/users/balta2ar/gists{/gist_id}","starred_url":"https://api.github.com/users/balta2ar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/balta2ar/subscriptions","organizations_url":"https://api.github.com/users/balta2ar/orgs","repos_url":"https://api.github.com/users/balta2ar/repos","events_url":"https://api.github.com/users/balta2ar/events{/privacy}","received_events_url":"https://api.github.com/users/balta2ar/received_events","type":"User","site_admin":false},"created_at":"2015-02-16T08:06:25Z","updated_at":"2015-02-16T08:06:25Z","body":"Thanks, setting it to 50K made it much better (however, delay is still noticeable)."}}