{"action":"created","issue":{"url":"https://api.github.com/repos/lintool/warcbase/issues/157","labels_url":"https://api.github.com/repos/lintool/warcbase/issues/157/labels{/name}","comments_url":"https://api.github.com/repos/lintool/warcbase/issues/157/comments","events_url":"https://api.github.com/repos/lintool/warcbase/issues/157/events","html_url":"https://github.com/lintool/warcbase/issues/157","id":116370556,"number":157,"title":"Refactor ExtractLinks UDF in matchbox","user":{"login":"lintool","id":313837,"avatar_url":"https://avatars.githubusercontent.com/u/313837?v=3","gravatar_id":"","url":"https://api.github.com/users/lintool","html_url":"https://github.com/lintool","followers_url":"https://api.github.com/users/lintool/followers","following_url":"https://api.github.com/users/lintool/following{/other_user}","gists_url":"https://api.github.com/users/lintool/gists{/gist_id}","starred_url":"https://api.github.com/users/lintool/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lintool/subscriptions","organizations_url":"https://api.github.com/users/lintool/orgs","repos_url":"https://api.github.com/users/lintool/repos","events_url":"https://api.github.com/users/lintool/events{/privacy}","received_events_url":"https://api.github.com/users/lintool/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/lintool/warcbase/labels/feature","name":"feature","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-11T16:27:56Z","updated_at":"2015-11-11T22:06:29Z","closed_at":null,"body":"If I want to extract all links from pages, I have to do something like this:\r\n\r\n```\r\nval r = RecordLoader.loadArc(\"/Users/jimmylin/Dropbox/workspace/warcbase/src/test/resources/arc/example.arc.gz\", sc)\r\n.keepMimeTypes(Set(\"text/html\"))\r\n.discardDate(null)\r\n.flatMap(r => ExtractLinks(r.getBodyContent).map(y => (r.getUrl, y._1)))\r\n.collect()\r\n```\r\n\r\nThe `flatMap` is a bit ugly. If we refactor `ExtractLinks` to `(url: String, body: String)` and spits out a list of tuples of `(source, target, anchor text)`, we could simplify.\r\n"},"comment":{"url":"https://api.github.com/repos/lintool/warcbase/issues/comments/155924422","html_url":"https://github.com/lintool/warcbase/issues/157#issuecomment-155924422","issue_url":"https://api.github.com/repos/lintool/warcbase/issues/157","id":155924422,"user":{"login":"aliceranzhou","id":7070016,"avatar_url":"https://avatars.githubusercontent.com/u/7070016?v=3","gravatar_id":"","url":"https://api.github.com/users/aliceranzhou","html_url":"https://github.com/aliceranzhou","followers_url":"https://api.github.com/users/aliceranzhou/followers","following_url":"https://api.github.com/users/aliceranzhou/following{/other_user}","gists_url":"https://api.github.com/users/aliceranzhou/gists{/gist_id}","starred_url":"https://api.github.com/users/aliceranzhou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aliceranzhou/subscriptions","organizations_url":"https://api.github.com/users/aliceranzhou/orgs","repos_url":"https://api.github.com/users/aliceranzhou/repos","events_url":"https://api.github.com/users/aliceranzhou/events{/privacy}","received_events_url":"https://api.github.com/users/aliceranzhou/received_events","type":"User","site_admin":false},"created_at":"2015-11-11T22:06:29Z","updated_at":"2015-11-11T22:06:29Z","body":"Just to check my understanding, we want to call `ExtractLinks` with the `from url` and the `body`, then return a list of tuples of `from url, target url, anchor text`?"}}