{"action":"created","issue":{"url":"https://api.github.com/repos/stanfordnlp/CoreNLP/issues/85","labels_url":"https://api.github.com/repos/stanfordnlp/CoreNLP/issues/85/labels{/name}","comments_url":"https://api.github.com/repos/stanfordnlp/CoreNLP/issues/85/comments","events_url":"https://api.github.com/repos/stanfordnlp/CoreNLP/issues/85/events","html_url":"https://github.com/stanfordnlp/CoreNLP/issues/85","id":97165034,"number":85,"title":"Spanish tokenizer much slower than English tokenizer","user":{"login":"vadali","id":379063,"avatar_url":"https://avatars.githubusercontent.com/u/379063?v=3","gravatar_id":"","url":"https://api.github.com/users/vadali","html_url":"https://github.com/vadali","followers_url":"https://api.github.com/users/vadali/followers","following_url":"https://api.github.com/users/vadali/following{/other_user}","gists_url":"https://api.github.com/users/vadali/gists{/gist_id}","starred_url":"https://api.github.com/users/vadali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vadali/subscriptions","organizations_url":"https://api.github.com/users/vadali/orgs","repos_url":"https://api.github.com/users/vadali/repos","events_url":"https://api.github.com/users/vadali/events{/privacy}","received_events_url":"https://api.github.com/users/vadali/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-07-24T23:01:38Z","updated_at":"2015-07-27T22:00:35Z","closed_at":null,"body":"Hey,\r\nI am using the Spanish tokenizer to run large jobs on common crawl using hadoop. \r\nWhile the english tokenizer works really fast, the Spanish tokenizer is painfully slow.. \r\n\r\nI started profiling it, and i'd like to share with you some initial results, perhaps we could find the bottleneck and fix it.\r\n\r\nThe first image shows profiling made to a large document (433,321 bytes) using the english tokenizer, which takes about ~35msecs on my computer:\r\n![tokenize-english](https://cloud.githubusercontent.com/assets/379063/8886273/6534d4fc-321c-11e5-838a-34d9b7a0b23f.png)\r\n\r\nThe second image shows profiling made to the same exact document, using the spanish tokenizer, which takes about ~300msecs on the same computer:\r\n![tokenize-spanish](https://cloud.githubusercontent.com/assets/379063/8886295/b0b7d12c-321c-11e5-8eff-01780c15b3d4.png)\r\n\r\nit seems that most of the time is spent in asciiQuotes.. any ideas of why it behaves this way?\r\n\r\nThanks,\r\nShlomi"},"comment":{"url":"https://api.github.com/repos/stanfordnlp/CoreNLP/issues/comments/125355160","html_url":"https://github.com/stanfordnlp/CoreNLP/issues/85#issuecomment-125355160","issue_url":"https://api.github.com/repos/stanfordnlp/CoreNLP/issues/85","id":125355160,"user":{"login":"vadali","id":379063,"avatar_url":"https://avatars.githubusercontent.com/u/379063?v=3","gravatar_id":"","url":"https://api.github.com/users/vadali","html_url":"https://github.com/vadali","followers_url":"https://api.github.com/users/vadali/followers","following_url":"https://api.github.com/users/vadali/following{/other_user}","gists_url":"https://api.github.com/users/vadali/gists{/gist_id}","starred_url":"https://api.github.com/users/vadali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vadali/subscriptions","organizations_url":"https://api.github.com/users/vadali/orgs","repos_url":"https://api.github.com/users/vadali/repos","events_url":"https://api.github.com/users/vadali/events{/privacy}","received_events_url":"https://api.github.com/users/vadali/received_events","type":"User","site_admin":false},"created_at":"2015-07-27T22:00:35Z","updated_at":"2015-07-27T22:00:35Z","body":"Some updates,\r\nUnlike the english lexer, the spanish lexer uses String.replaceAll almost exclusively, rather than use Pattern.compile statically and the matcher method. \r\n\r\nI fixed this in asciiQuotes, and it improved the runtime from about 300ms to about 190ms. One thing that I am seeing is that this method gets called many many times, especially comparing it to the regular PTBLexer, for the same exact text:\r\n![english-spanish-calls](https://cloud.githubusercontent.com/assets/379063/8918898/0e085432-3470-11e5-9771-aa6feb071965.png)\r\n\r\nHere is a profiling breakdown of the tokenization process,\r\nEnglish:\r\n![english-breakdown](https://cloud.githubusercontent.com/assets/379063/8918869/dec50404-346f-11e5-9af0-447be42d926b.png)\r\n\r\nSpanish:\r\n![spanish-breakdown](https://cloud.githubusercontent.com/assets/379063/8918872/e529000c-346f-11e5-8525-88e4142b1d85.png)\r\n\r\nI will try to compare both flex files, attempt to minimize normalization calls. I would really appreciate any help with this, to make sure that I am doing the right thing correctly.\r\n\r\nThanks!"}}