{"action":"created","issue":{"url":"https://api.github.com/repos/NAMD/mediacloud_backend/issues/97","labels_url":"https://api.github.com/repos/NAMD/mediacloud_backend/issues/97/labels{/name}","comments_url":"https://api.github.com/repos/NAMD/mediacloud_backend/issues/97/comments","events_url":"https://api.github.com/repos/NAMD/mediacloud_backend/issues/97/events","html_url":"https://github.com/NAMD/mediacloud_backend/issues/97","id":115915473,"number":97,"title":"Error when saving the url on mongo","user":{"login":"flavioamieiro","id":60641,"avatar_url":"https://avatars.githubusercontent.com/u/60641?v=3","gravatar_id":"","url":"https://api.github.com/users/flavioamieiro","html_url":"https://github.com/flavioamieiro","followers_url":"https://api.github.com/users/flavioamieiro/followers","following_url":"https://api.github.com/users/flavioamieiro/following{/other_user}","gists_url":"https://api.github.com/users/flavioamieiro/gists{/gist_id}","starred_url":"https://api.github.com/users/flavioamieiro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flavioamieiro/subscriptions","organizations_url":"https://api.github.com/users/flavioamieiro/orgs","repos_url":"https://api.github.com/users/flavioamieiro/repos","events_url":"https://api.github.com/users/flavioamieiro/events{/privacy}","received_events_url":"https://api.github.com/users/flavioamieiro/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/NAMD/mediacloud_backend/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-09T16:40:46Z","updated_at":"2015-11-09T18:47:54Z","closed_at":null,"body":"When checking the database for compatibility with mongo 2.6 (so we can then upgrade to 3.0) I stumbled on the following error:\r\n\r\n    Document Error: key for index {  \"v\" : 1,  \"name\" : \"url_1\",  \"key\" : {  \"url\" : 1 },  \"unique\" : true,  \"ns\" : \"MCDB.urls\" } too long for document: {  \"_id\" : ObjectId(\"5494a711dcccdd10afbb7c11\"),  \"url\" : \"http://saojoaquimonline.com.br/saojoaquimdefato/Neva na Serra catarinense na noite desta terça-feira/serrasc.com.br/A unidade da Epagri em Caçador, está desenvolvendo quatro novas cultivares de macieiras. O comitê técnico da Associação Brasileira dos Produtores de MaçãABPM já fez testes sensoriais com os novos cruzamentos que estão sendo propostos pela pesquisa, tendo como referências as variedades Baronesa e Imperatriz./www.saojoaquimonline.com.br/www.festivalbrasil.jp/serrasc.com.br/A unidade da Epagri em Caçador, está desenvolvendo quatro novas cultivares de macieiras. O comitê técnico da Associação Brasileira dos Produtores de MaçãABPM já fez testes sensoriais com os novos cruzamentos que estão sendo propostos pela pesquisa, tendo como referências as variedades Baronesa e Imperatriz./serrasc.com.br/www.festivalbrasil.jp/A unidade da Epagri em Caçador, está desenvolvendo quatro novas cultivares de macieiras. O comitê técnico da Associação Brasileira dos Produtores de MaçãABPM já fez testes sensoriais com os novos cruzamentos que estão sendo propostos pela pesquisa, tendo como referências as variedades Baronesa e Imperatriz./?paged=104\",  \"fetched_on\" : ISODate(\"2014-12-19T20:30:41.555Z\"),  \"tags\" : [ \"filmes+policia+ambiente+colunistas+alimento\" ] }\r\n\r\n\r\nWe need to solve this before we can upgrade to 2.6. I think the problem is happening in (googlerss.py, when getting the url)[https://github.com/NAMD/mediacloud_backend/blob/master/capture/googlerss.py#L63]. After the split on '&' we get all the text if there is no '&' on the original.\r\n\r\nWhat I don't understand is how the text is getting to the url we get from `url.geturl()`.\r\n\r\n@fccoelho any idea on why this is happening?"},"comment":{"url":"https://api.github.com/repos/NAMD/mediacloud_backend/issues/comments/155152573","html_url":"https://github.com/NAMD/mediacloud_backend/issues/97#issuecomment-155152573","issue_url":"https://api.github.com/repos/NAMD/mediacloud_backend/issues/97","id":155152573,"user":{"login":"flavioamieiro","id":60641,"avatar_url":"https://avatars.githubusercontent.com/u/60641?v=3","gravatar_id":"","url":"https://api.github.com/users/flavioamieiro","html_url":"https://github.com/flavioamieiro","followers_url":"https://api.github.com/users/flavioamieiro/followers","following_url":"https://api.github.com/users/flavioamieiro/following{/other_user}","gists_url":"https://api.github.com/users/flavioamieiro/gists{/gist_id}","starred_url":"https://api.github.com/users/flavioamieiro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flavioamieiro/subscriptions","organizations_url":"https://api.github.com/users/flavioamieiro/orgs","repos_url":"https://api.github.com/users/flavioamieiro/repos","events_url":"https://api.github.com/users/flavioamieiro/events{/privacy}","received_events_url":"https://api.github.com/users/flavioamieiro/received_events","type":"User","site_admin":false},"created_at":"2015-11-09T18:47:54Z","updated_at":"2015-11-09T18:47:54Z","body":"@fccoelho the problem is definetly happening when the url is in an unexpected format. What I don't understand is how we are getting text (such as `Neva na Serra catarinense`) in the result of `url.geturl()`."}}