{"action":"created","issue":{"url":"https://api.github.com/repos/jquery/esprima/issues/1019","labels_url":"https://api.github.com/repos/jquery/esprima/issues/1019/labels{/name}","comments_url":"https://api.github.com/repos/jquery/esprima/issues/1019/comments","events_url":"https://api.github.com/repos/jquery/esprima/issues/1019/events","html_url":"https://github.com/jquery/esprima/issues/1019","id":57090767,"number":1019,"title":"Investigate performance problem of the tokenizer","user":{"login":"ariya","id":7288,"avatar_url":"https://avatars.githubusercontent.com/u/7288?v=3","gravatar_id":"","url":"https://api.github.com/users/ariya","html_url":"https://github.com/ariya","followers_url":"https://api.github.com/users/ariya/followers","following_url":"https://api.github.com/users/ariya/following{/other_user}","gists_url":"https://api.github.com/users/ariya/gists{/gist_id}","starred_url":"https://api.github.com/users/ariya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ariya/subscriptions","organizations_url":"https://api.github.com/users/ariya/orgs","repos_url":"https://api.github.com/users/ariya/repos","events_url":"https://api.github.com/users/ariya/events{/privacy}","received_events_url":"https://api.github.com/users/ariya/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/jquery/esprima/labels/performance","name":"performance","color":"c7def8"}],"state":"open","locked":false,"assignee":{"login":"ariya","id":7288,"avatar_url":"https://avatars.githubusercontent.com/u/7288?v=3","gravatar_id":"","url":"https://api.github.com/users/ariya","html_url":"https://github.com/ariya","followers_url":"https://api.github.com/users/ariya/followers","following_url":"https://api.github.com/users/ariya/following{/other_user}","gists_url":"https://api.github.com/users/ariya/gists{/gist_id}","starred_url":"https://api.github.com/users/ariya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ariya/subscriptions","organizations_url":"https://api.github.com/users/ariya/orgs","repos_url":"https://api.github.com/users/ariya/repos","events_url":"https://api.github.com/users/ariya/events{/privacy}","received_events_url":"https://api.github.com/users/ariya/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2015-02-09T21:14:45Z","updated_at":"2015-02-11T13:36:10Z","closed_at":null,"body":"Migrated from https://code.google.com/p/esprima/issues/detail?id=626.\r\n\r\nAs reported by @aslushnikov, it seems that `tokenize` is way too slower compared to `parse`. "},"comment":{"url":"https://api.github.com/repos/jquery/esprima/issues/comments/73881442","html_url":"https://github.com/jquery/esprima/issues/1019#issuecomment-73881442","issue_url":"https://api.github.com/repos/jquery/esprima/issues/1019","id":73881442,"user":{"login":"aslushnikov","id":746130,"avatar_url":"https://avatars.githubusercontent.com/u/746130?v=3","gravatar_id":"","url":"https://api.github.com/users/aslushnikov","html_url":"https://github.com/aslushnikov","followers_url":"https://api.github.com/users/aslushnikov/followers","following_url":"https://api.github.com/users/aslushnikov/following{/other_user}","gists_url":"https://api.github.com/users/aslushnikov/gists{/gist_id}","starred_url":"https://api.github.com/users/aslushnikov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aslushnikov/subscriptions","organizations_url":"https://api.github.com/users/aslushnikov/orgs","repos_url":"https://api.github.com/users/aslushnikov/repos","events_url":"https://api.github.com/users/aslushnikov/events{/privacy}","received_events_url":"https://api.github.com/users/aslushnikov/received_events","type":"User","site_admin":false},"created_at":"2015-02-11T13:36:10Z","updated_at":"2015-02-11T13:36:10Z","body":"@ariya, \r\n\r\nMy real-world example is tokenization of one of Inbox client scripts - that's roughly 580k tokens of obfuscated javascript. Let's get the sample of this size with the following command:\r\n```bash\r\ncd test/3rdparty\r\ncat underscore-1.5.2.js backbone-1.1.0.js mootools-1.4.5.js jquery-1.9.1.js yui-3.12.0.js jquery.mobile-1.4.2.js angular-1.2.5.js underscore-1.5.2.js backbone-1.1.0.js mootools-1.4.5.js jquery-1.9.1.js yui-3.12.0.js jquery.mobile-1.4.2.js angular-1.2.5.js angular-1.2.5.js > big.js\r\n```\r\n\r\nThe command produces `big.js` file whose tokenization results in 578335 tokens. \r\nI've modified `benchmark.js` to include `big.js` and to do tokenization instead of parse, and\r\nhere are my results for **tip-of-tree Esprima**:\r\n\r\n```\r\n    Underscore 1.5.2    42.5 KiB      4.40 ms ± 4.99%\r\n      Backbone 1.1.0    58.7 KiB      5.13 ms ± 1.35%\r\n      MooTools 1.4.5   156.7 KiB     43.01 ms ± 5.09%\r\n        jQuery 1.9.1   262.1 KiB     58.65 ms ± 7.80%\r\n          YUI 3.12.0   330.4 KiB     40.88 ms ± 3.26%\r\n jQuery.Mobile 1.4.2   442.2 KiB     95.26 ms ± 8.17%\r\n       Angular 1.2.5   701.7 KiB     89.68 ms ± 4.87%\r\n                 big  4690.4 KiB   1152.83 ms ± 10.79%\r\n                     ------------------------\r\n                      6684.7 KiB   1489.86 ms ± 9.96%\r\n```\r\n\r\nWe've spent more then a second for tokenization of 4.5 mb of code. This might not\r\nlook like outstandingly bad performance, but in fact we can do *way* better.\r\n\r\nTo show off that the performance here is suboptimal, let me run the same benchmark, but with the\r\nmodified esprima branch, taken from here: https://github.com/aslushnikov/esprima/tree/sax\r\nInstead of pushing tokens into array, the modified branch just reports them via callback. This saves us on memory reallocs (I'm not an expert in V8 - just guessing), and results in significant performance gains. These are results for **modified Esprima**:\r\n```\r\n    Underscore 1.5.2    42.5 KiB      4.51 ms ± 2.09%\r\n      Backbone 1.1.0    58.7 KiB      4.68 ms ± 1.35%\r\n      MooTools 1.4.5   156.7 KiB     20.17 ms ± 2.10%\r\n        jQuery 1.9.1   262.1 KiB     25.25 ms ± 1.45%\r\n          YUI 3.12.0   330.4 KiB     20.73 ms ± 1.34%\r\n jQuery.Mobile 1.4.2   442.2 KiB     42.14 ms ± 2.26%\r\n       Angular 1.2.5   701.7 KiB     35.53 ms ± 2.24%\r\n                 big  4690.4 KiB    324.30 ms ± 7.09%\r\n                     ------------------------\r\n                      6684.7 KiB    477.32 ms ± 5.95%\r\n```\r\n\r\nThat's 3.5 times faster for `big.js`. "}}