{"action":"created","issue":{"url":"https://api.github.com/repos/18F/regulations-parser/issues/100","labels_url":"https://api.github.com/repos/18F/regulations-parser/issues/100/labels{/name}","comments_url":"https://api.github.com/repos/18F/regulations-parser/issues/100/comments","events_url":"https://api.github.com/repos/18F/regulations-parser/issues/100/events","html_url":"https://github.com/18F/regulations-parser/issues/100","id":114476345,"number":100,"title":"Make `scanString` more efficient","user":{"login":"cmc333333","id":326918,"avatar_url":"https://avatars.githubusercontent.com/u/326918?v=3","gravatar_id":"","url":"https://api.github.com/users/cmc333333","html_url":"https://github.com/cmc333333","followers_url":"https://api.github.com/users/cmc333333/followers","following_url":"https://api.github.com/users/cmc333333/following{/other_user}","gists_url":"https://api.github.com/users/cmc333333/gists{/gist_id}","starred_url":"https://api.github.com/users/cmc333333/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmc333333/subscriptions","organizations_url":"https://api.github.com/users/cmc333333/orgs","repos_url":"https://api.github.com/users/cmc333333/repos","events_url":"https://api.github.com/users/cmc333333/events{/privacy}","received_events_url":"https://api.github.com/users/cmc333333/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-01T15:39:52Z","updated_at":"2015-11-03T02:51:17Z","closed_at":null,"body":"We use PyParsing's `scanString` all over the place. Digging into the [implementation](http://sourceforge.net/p/pyparsing/code/HEAD/tree/tags/pyparsing_2.0.5/src/pyparsing.py#l1143), we can see why this ends up being so slow. Effectively, the parser steps through each index in the string, attempting to apply the grammar, and upon failure, incrementing to the next index. This works perfectly fine if we're assuming that the strings are small or are majority composed of matches. Instead we've been abusing this system and applying it to long strings which have a very small number of matches.\r\n\r\nI think we'd see significantly improved parse times if we added some smarts around scans like this. In particular, I'm imagining a wrapper around existing grammars which includes a regex defining the beginning of a valid match. We'd still be using a regex to search, but this is much faster than attempting to apply a full grammar at each index"},"comment":{"url":"https://api.github.com/repos/18F/regulations-parser/issues/comments/153225665","html_url":"https://github.com/18F/regulations-parser/issues/100#issuecomment-153225665","issue_url":"https://api.github.com/repos/18F/regulations-parser/issues/100","id":153225665,"user":{"login":"cmc333333","id":326918,"avatar_url":"https://avatars.githubusercontent.com/u/326918?v=3","gravatar_id":"","url":"https://api.github.com/users/cmc333333","html_url":"https://github.com/cmc333333","followers_url":"https://api.github.com/users/cmc333333/followers","following_url":"https://api.github.com/users/cmc333333/following{/other_user}","gists_url":"https://api.github.com/users/cmc333333/gists{/gist_id}","starred_url":"https://api.github.com/users/cmc333333/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmc333333/subscriptions","organizations_url":"https://api.github.com/users/cmc333333/orgs","repos_url":"https://api.github.com/users/cmc333333/repos","events_url":"https://api.github.com/users/cmc333333/events{/privacy}","received_events_url":"https://api.github.com/users/cmc333333/received_events","type":"User","site_admin":false},"created_at":"2015-11-03T02:51:17Z","updated_at":"2015-11-03T02:51:17Z","body":"Trying a similar, [CFPB-centric patch](https://github.com/18F/regulations-parser/files/25851/cfpb-diff.txt) similarly cut parse times for reg E and Z by half. Now, to find a cleaner solution."}}