{"action":"created","issue":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/115","labels_url":"https://api.github.com/repos/Norconex/collector-http/issues/115/labels{/name}","comments_url":"https://api.github.com/repos/Norconex/collector-http/issues/115/comments","events_url":"https://api.github.com/repos/Norconex/collector-http/issues/115/events","html_url":"https://github.com/Norconex/collector-http/issues/115","id":84421616,"number":115,"title":"Canonical links not detected","user":{"login":"Nycander","id":420865,"avatar_url":"https://avatars.githubusercontent.com/u/420865?v=3","gravatar_id":"","url":"https://api.github.com/users/Nycander","html_url":"https://github.com/Nycander","followers_url":"https://api.github.com/users/Nycander/followers","following_url":"https://api.github.com/users/Nycander/following{/other_user}","gists_url":"https://api.github.com/users/Nycander/gists{/gist_id}","starred_url":"https://api.github.com/users/Nycander/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Nycander/subscriptions","organizations_url":"https://api.github.com/users/Nycander/orgs","repos_url":"https://api.github.com/users/Nycander/repos","events_url":"https://api.github.com/users/Nycander/events{/privacy}","received_events_url":"https://api.github.com/users/Nycander/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Norconex/collector-http/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Norconex/collector-http/labels/fixed/done","name":"fixed/done","color":"bfe5bf"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/Norconex/collector-http/milestones/7","html_url":"https://github.com/Norconex/collector-http/milestones/2.2.0","labels_url":"https://api.github.com/repos/Norconex/collector-http/milestones/7/labels","id":1062185,"number":7,"title":"2.2.0","description":"Feature release.","creator":{"login":"essiembre","id":2326274,"avatar_url":"https://avatars.githubusercontent.com/u/2326274?v=3","gravatar_id":"","url":"https://api.github.com/users/essiembre","html_url":"https://github.com/essiembre","followers_url":"https://api.github.com/users/essiembre/followers","following_url":"https://api.github.com/users/essiembre/following{/other_user}","gists_url":"https://api.github.com/users/essiembre/gists{/gist_id}","starred_url":"https://api.github.com/users/essiembre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/essiembre/subscriptions","organizations_url":"https://api.github.com/users/essiembre/orgs","repos_url":"https://api.github.com/users/essiembre/repos","events_url":"https://api.github.com/users/essiembre/events{/privacy}","received_events_url":"https://api.github.com/users/essiembre/received_events","type":"User","site_admin":false},"open_issues":14,"closed_issues":5,"state":"open","created_at":"2015-04-10T16:00:57Z","updated_at":"2015-07-22T18:46:18Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-06-03T07:57:05Z","updated_at":"2015-07-22T18:46:18Z","closed_at":"2015-07-22T18:46:18Z","body":"This URL fails to detect canonical urls: http://www.folksam.se/forsakringar/djurforsakring/kattforsakring\r\nSeems as if the link tag is hidden behind tags inside javascript code.\r\n\r\nI also saw that the current code uses regular expressions: http://stackoverflow.com/a/1732454/191967\r\nThat method is known to be error prone, and I think this is showing here.\r\n\r\nMy current workaround is a jsoup implementation of ICanonicalLinkDetector:\r\n\r\n    @Override\r\n    public String detectFromContent(String reference,\r\n            InputStream is,\r\n            ContentType contentType) throws IOException {\r\n\r\n        ContentType[] cTypes = contentTypes;\r\n        if (ArrayUtils.isEmpty(cTypes)) {\r\n            cTypes = DEFAULT_CONTENT_TYPES;\r\n        }\r\n\r\n        // Do not extract if not a supported content type\r\n        if (!ArrayUtils.contains(cTypes, contentType)) {\r\n            return null;\r\n        }\r\n\r\n        Document document = Jsoup.parse(is, \"UTF-8\", reference);\r\n        Elements elements = document.select(\"link[rel=canonical]\");\r\n        if (elements.isEmpty()) {\r\n            return null;\r\n        }\r\n        String href = elements.first().attr(\"href\");\r\n        if (! href.isEmpty()) {\r\n            return toAbsolute(reference, href);\r\n        }\r\n        return null;\r\n    }"},"comment":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/comments/123823822","html_url":"https://github.com/Norconex/collector-http/issues/115#issuecomment-123823822","issue_url":"https://api.github.com/repos/Norconex/collector-http/issues/115","id":123823822,"user":{"login":"essiembre","id":2326274,"avatar_url":"https://avatars.githubusercontent.com/u/2326274?v=3","gravatar_id":"","url":"https://api.github.com/users/essiembre","html_url":"https://github.com/essiembre","followers_url":"https://api.github.com/users/essiembre/followers","following_url":"https://api.github.com/users/essiembre/following{/other_user}","gists_url":"https://api.github.com/users/essiembre/gists{/gist_id}","starred_url":"https://api.github.com/users/essiembre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/essiembre/subscriptions","organizations_url":"https://api.github.com/users/essiembre/orgs","repos_url":"https://api.github.com/users/essiembre/repos","events_url":"https://api.github.com/users/essiembre/events{/privacy}","received_events_url":"https://api.github.com/users/essiembre/received_events","type":"User","site_admin":false},"created_at":"2015-07-22T18:46:18Z","updated_at":"2015-07-22T18:46:18Z","body":"Norconex HTTP Collector 2.2.0 official release is out. It includes this fix. You can download it [here](http://www.norconex.com/collectors/collector-http/download)."}}