{"action":"opened","issue":{"url":"https://api.github.com/repos/andresriancho/w3af/issues/9933","labels_url":"https://api.github.com/repos/andresriancho/w3af/issues/9933/labels{/name}","comments_url":"https://api.github.com/repos/andresriancho/w3af/issues/9933/comments","events_url":"https://api.github.com/repos/andresriancho/w3af/issues/9933/events","html_url":"https://github.com/andresriancho/w3af/issues/9933","id":76917551,"number":9933,"title":"ParserCache - multiprocessing: limitations and new requirements","user":{"login":"andresriancho","id":865200,"avatar_url":"https://avatars.githubusercontent.com/u/865200?v=3","gravatar_id":"","url":"https://api.github.com/users/andresriancho","html_url":"https://github.com/andresriancho","followers_url":"https://api.github.com/users/andresriancho/followers","following_url":"https://api.github.com/users/andresriancho/following{/other_user}","gists_url":"https://api.github.com/users/andresriancho/gists{/gist_id}","starred_url":"https://api.github.com/users/andresriancho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andresriancho/subscriptions","organizations_url":"https://api.github.com/users/andresriancho/orgs","repos_url":"https://api.github.com/users/andresriancho/repos","events_url":"https://api.github.com/users/andresriancho/events{/privacy}","received_events_url":"https://api.github.com/users/andresriancho/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/andresriancho/w3af/labels/bug","name":"bug","color":"e10c02"},{"url":"https://api.github.com/repos/andresriancho/w3af/labels/priority:high","name":"priority:high","color":"e10c02"}],"state":"open","locked":false,"assignee":{"login":"andresriancho","id":865200,"avatar_url":"https://avatars.githubusercontent.com/u/865200?v=3","gravatar_id":"","url":"https://api.github.com/users/andresriancho","html_url":"https://github.com/andresriancho","followers_url":"https://api.github.com/users/andresriancho/followers","following_url":"https://api.github.com/users/andresriancho/following{/other_user}","gists_url":"https://api.github.com/users/andresriancho/gists{/gist_id}","starred_url":"https://api.github.com/users/andresriancho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andresriancho/subscriptions","organizations_url":"https://api.github.com/users/andresriancho/orgs","repos_url":"https://api.github.com/users/andresriancho/repos","events_url":"https://api.github.com/users/andresriancho/events{/privacy}","received_events_url":"https://api.github.com/users/andresriancho/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/andresriancho/w3af/milestones/10","html_url":"https://github.com/andresriancho/w3af/milestones/1.6.50%20-%20Bug%20fixing%20after%201.6.0","labels_url":"https://api.github.com/repos/andresriancho/w3af/milestones/10/labels","id":614510,"number":10,"title":"1.6.50 - Bug fixing after 1.6.0","description":"Bug fixes after 1.6 release, this is a MUST-HAVE since people will start using 1.6 and report severity 1 bugs which need to be fixed and pushed to all users.","creator":{"login":"andresriancho","id":865200,"avatar_url":"https://avatars.githubusercontent.com/u/865200?v=3","gravatar_id":"","url":"https://api.github.com/users/andresriancho","html_url":"https://github.com/andresriancho","followers_url":"https://api.github.com/users/andresriancho/followers","following_url":"https://api.github.com/users/andresriancho/following{/other_user}","gists_url":"https://api.github.com/users/andresriancho/gists{/gist_id}","starred_url":"https://api.github.com/users/andresriancho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andresriancho/subscriptions","organizations_url":"https://api.github.com/users/andresriancho/orgs","repos_url":"https://api.github.com/users/andresriancho/repos","events_url":"https://api.github.com/users/andresriancho/events{/privacy}","received_events_url":"https://api.github.com/users/andresriancho/received_events","type":"User","site_admin":false},"open_issues":26,"closed_issues":335,"state":"open","created_at":"2014-03-30T17:44:39Z","updated_at":"2015-05-16T01:52:55Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2015-05-16T01:52:55Z","updated_at":"2015-05-16T01:52:55Z","closed_at":null,"body":" * `get_dom` is called only from the main process (from different threads). So the HTML is parsed in the main process without any timeout (which we impose for the subprocesses in parser_cache). This might lead to bugs\r\n\r\n * When we call `get_dom` we load all the HTML in memory. Do we really need to do that? What if we can find a better way? Can we replace XPATH with a different thing which operates in \"sax mode\"?\r\n\r\n * In the subprocess created by `ParserCache` we load all the HTML in memory, can we migrate to https://www.ibm.com/developerworks/library/x-hiperfparse/ ? Something I suspect is that the HTMLParser will load all the HTML in memory if we use `recover` flag?\r\n\r\n * Pedro (Onapsis) told me about a \"shallow\" mode for lxml. Ask him about it."}}