{"action":"created","issue":{"url":"https://api.github.com/repos/UTMediaCAT/Voyage/issues/7","labels_url":"https://api.github.com/repos/UTMediaCAT/Voyage/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/UTMediaCAT/Voyage/issues/7/comments","events_url":"https://api.github.com/repos/UTMediaCAT/Voyage/issues/7/events","html_url":"https://github.com/UTMediaCAT/Voyage/issues/7","id":91528252,"number":7,"title":"Duplicates ","user":{"login":"ldfelipe","id":12644270,"avatar_url":"https://avatars.githubusercontent.com/u/12644270?v=3","gravatar_id":"","url":"https://api.github.com/users/ldfelipe","html_url":"https://github.com/ldfelipe","followers_url":"https://api.github.com/users/ldfelipe/followers","following_url":"https://api.github.com/users/ldfelipe/following{/other_user}","gists_url":"https://api.github.com/users/ldfelipe/gists{/gist_id}","starred_url":"https://api.github.com/users/ldfelipe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ldfelipe/subscriptions","organizations_url":"https://api.github.com/users/ldfelipe/orgs","repos_url":"https://api.github.com/users/ldfelipe/repos","events_url":"https://api.github.com/users/ldfelipe/events{/privacy}","received_events_url":"https://api.github.com/users/ldfelipe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-06-27T23:23:35Z","updated_at":"2015-06-28T17:21:00Z","closed_at":null,"body":"The crawler is listing articles from referring sites: electronicintifada.net, foxnews.com twice with the same URL\r\n\r\nAs well, the crawler picked up the exact same article with two different URLs: \r\n\r\nhttp://feeds.huffingtonpost.com/c/35496/f/677045/s/47940b3b/sc/7/l/0L0Shuffingtonpost0N0C20A150C0A60C250Cjewish0Eshabbat0Eof0Esolidary0Eafrican0Eamericans0In0I76571980Bhtml/story01.htm\r\nhttp://huffingtonpost.com/2015/06/25/jewish-shabbat-of-solidary-african-americans_n_7657198.html\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/UTMediaCAT/Voyage/issues/comments/116300967","html_url":"https://github.com/UTMediaCAT/Voyage/issues/7#issuecomment-116300967","issue_url":"https://api.github.com/repos/UTMediaCAT/Voyage/issues/7","id":116300967,"user":{"login":"Sunakujira1","id":4432415,"avatar_url":"https://avatars.githubusercontent.com/u/4432415?v=3","gravatar_id":"","url":"https://api.github.com/users/Sunakujira1","html_url":"https://github.com/Sunakujira1","followers_url":"https://api.github.com/users/Sunakujira1/followers","following_url":"https://api.github.com/users/Sunakujira1/following{/other_user}","gists_url":"https://api.github.com/users/Sunakujira1/gists{/gist_id}","starred_url":"https://api.github.com/users/Sunakujira1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Sunakujira1/subscriptions","organizations_url":"https://api.github.com/users/Sunakujira1/orgs","repos_url":"https://api.github.com/users/Sunakujira1/repos","events_url":"https://api.github.com/users/Sunakujira1/events{/privacy}","received_events_url":"https://api.github.com/users/Sunakujira1/received_events","type":"User","site_admin":false},"created_at":"2015-06-28T17:21:00Z","updated_at":"2015-06-28T17:21:00Z","body":"Using urllib2.urlopen(url).geturl() will resolve this issue:\r\n```\r\n>>> import urllib2\r\n>>> urllib2.urlopen('http://feeds.huffingtonpost.com/c/35496/f/677045/s/47940b3b/sc/7/l/0L0Shuffingtonpost0N0C20A150C0A60C250Cjewish0Eshabbat0Eof0Esolidary0Eafrican0Eamericans0In0I76571980Bhtml/story01.htm').geturl()\r\n'http://www.huffingtonpost.com/2015/06/25/jewish-shabbat-of-solidary-african-americans_n_7657198.html'\r\n>>> urllib2.urlopen('http://huffingtonpost.com/2015/06/25/jewish-shabbat-of-solidary-african-americans_n_7657198.html').geturl()                         \r\n'http://www.huffingtonpost.com/2015/06/25/jewish-shabbat-of-solidary-african-americans_n_7657198.html'\r\n>>>\r\n```"}}