{"action":"created","issue":{"url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/158","labels_url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/158/labels{/name}","comments_url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/158/comments","events_url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/158/events","html_url":"https://github.com/DigitalPebble/storm-crawler/issues/158","id":93784417,"number":158,"title":"Remove needsDOM() in ParseFilter","user":{"login":"jnioche","id":218319,"avatar_url":"https://avatars.githubusercontent.com/u/218319?v=3","gravatar_id":"","url":"https://api.github.com/users/jnioche","html_url":"https://github.com/jnioche","followers_url":"https://api.github.com/users/jnioche/followers","following_url":"https://api.github.com/users/jnioche/following{/other_user}","gists_url":"https://api.github.com/users/jnioche/gists{/gist_id}","starred_url":"https://api.github.com/users/jnioche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jnioche/subscriptions","organizations_url":"https://api.github.com/users/jnioche/orgs","repos_url":"https://api.github.com/users/jnioche/repos","events_url":"https://api.github.com/users/jnioche/events{/privacy}","received_events_url":"https://api.github.com/users/jnioche/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/DigitalPebble/storm-crawler/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-07-08T13:04:26Z","updated_at":"2015-08-21T09:39:39Z","closed_at":null,"body":"This was provided as a way of avoiding building the DOM structure for when no ParseFilters were requiring it and is used only in the Tika parser.\r\n\r\nThe RobotsTag extractor requires such a DOM so unless we are able to extract the robots directives differently e.g. via SAX and a custom ContentHandler we should reconsider whether this method is really needed.\r\n\r\nThe JSoupParseFilter could avoid building the DOM too and extract the values via its org.jsoup.nodes.Document prior to building the DOM (if required).\r\n\r\nRelated to #154\r\n"},"comment":{"url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/comments/133352121","html_url":"https://github.com/DigitalPebble/storm-crawler/issues/158#issuecomment-133352121","issue_url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/158","id":133352121,"user":{"login":"jnioche","id":218319,"avatar_url":"https://avatars.githubusercontent.com/u/218319?v=3","gravatar_id":"","url":"https://api.github.com/users/jnioche","html_url":"https://github.com/jnioche","followers_url":"https://api.github.com/users/jnioche/followers","following_url":"https://api.github.com/users/jnioche/following{/other_user}","gists_url":"https://api.github.com/users/jnioche/gists{/gist_id}","starred_url":"https://api.github.com/users/jnioche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jnioche/subscriptions","organizations_url":"https://api.github.com/users/jnioche/orgs","repos_url":"https://api.github.com/users/jnioche/repos","events_url":"https://api.github.com/users/jnioche/events{/privacy}","received_events_url":"https://api.github.com/users/jnioche/received_events","type":"User","site_admin":false},"created_at":"2015-08-21T09:39:39Z","updated_at":"2015-08-21T09:39:39Z","body":"The SitemapParserBolt calls the ParseFilters too but does not generate a DOM which can result in a NPE. This has been addressed in [b0d6f59] but relies on needsDOM()"}}