{"action":"created","issue":{"url":"https://api.github.com/repos/CartoDB/cartodb/issues/6152","labels_url":"https://api.github.com/repos/CartoDB/cartodb/issues/6152/labels{/name}","comments_url":"https://api.github.com/repos/CartoDB/cartodb/issues/6152/comments","events_url":"https://api.github.com/repos/CartoDB/cartodb/issues/6152/events","html_url":"https://github.com/CartoDB/cartodb/pull/6152","id":117636497,"number":6152,"title":"Csv normalization 6116","user":{"login":"jgoizueta","id":5909,"avatar_url":"https://avatars.githubusercontent.com/u/5909?v=3","gravatar_id":"","url":"https://api.github.com/users/jgoizueta","html_url":"https://github.com/jgoizueta","followers_url":"https://api.github.com/users/jgoizueta/followers","following_url":"https://api.github.com/users/jgoizueta/following{/other_user}","gists_url":"https://api.github.com/users/jgoizueta/gists{/gist_id}","starred_url":"https://api.github.com/users/jgoizueta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgoizueta/subscriptions","organizations_url":"https://api.github.com/users/jgoizueta/orgs","repos_url":"https://api.github.com/users/jgoizueta/repos","events_url":"https://api.github.com/users/jgoizueta/events{/privacy}","received_events_url":"https://api.github.com/users/jgoizueta/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-11-18T17:22:01Z","updated_at":"2015-11-23T11:35:08Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/CartoDB/cartodb/pulls/6152","html_url":"https://github.com/CartoDB/cartodb/pull/6152","diff_url":"https://github.com/CartoDB/cartodb/pull/6152.diff","patch_url":"https://github.com/CartoDB/cartodb/pull/6152.patch"},"body":"This introduces support for line breaks inside cells in CSV normalization and also makes CSV normalization more consistent with imports without normalization:\r\n\r\n* Row separators are detected and handle by the Ruby CSV module, which results in more consistency\r\n  with direct ogr2ogr handling (e.g. `\\r\\n` separators where incorrectly handle by the normalization)\r\n* Reverse line feeds in Windows-1252 encoding where previously removed for some reason\r\n  (see f54a953345a17be45336f9065257c1eb629db34e). Now we do as ogr2ogr and just preserve them.\r\n\r\nFixes #6116\r\n\r\nSome regression tests are added to specify the new desired behaviour.\r\n\r\nMalformed CSV lines are being silently ignored as was done before. This is ok for some cases in which such lines are blank (e.g. some trailing `\\n\\r\\n` found in some files). It would be nice to inform the user of non-blank dropped lines, so a counter has been introduced in the code to be used in some future feature."},"comment":{"url":"https://api.github.com/repos/CartoDB/cartodb/issues/comments/158910725","html_url":"https://github.com/CartoDB/cartodb/pull/6152#issuecomment-158910725","issue_url":"https://api.github.com/repos/CartoDB/cartodb/issues/6152","id":158910725,"user":{"login":"jgoizueta","id":5909,"avatar_url":"https://avatars.githubusercontent.com/u/5909?v=3","gravatar_id":"","url":"https://api.github.com/users/jgoizueta","html_url":"https://github.com/jgoizueta","followers_url":"https://api.github.com/users/jgoizueta/followers","following_url":"https://api.github.com/users/jgoizueta/following{/other_user}","gists_url":"https://api.github.com/users/jgoizueta/gists{/gist_id}","starred_url":"https://api.github.com/users/jgoizueta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgoizueta/subscriptions","organizations_url":"https://api.github.com/users/jgoizueta/orgs","repos_url":"https://api.github.com/users/jgoizueta/repos","events_url":"https://api.github.com/users/jgoizueta/events{/privacy}","received_events_url":"https://api.github.com/users/jgoizueta/received_events","type":"User","site_admin":false},"created_at":"2015-11-23T11:35:08Z","updated_at":"2015-11-23T11:35:08Z","body":"Regarding the failing test: it tries to load a JPEG image as it was a CSV file. Is used to file with 1002 (bad file type) and now fails with 2002 (bad encoding).\r\n\r\nThe former error message may seem more accurate to the user, but this is what's happening:\r\n\r\nBefore we had the normalization going over the JPEG without error and ignoring all the \"lines\" because they are malformed, then ogr2ogr chokes on this empty file\r\n\r\nNow we have the CSC normalization failing because the binary JPEG data is not valid in the detected encoding (or any other encoding for that matter).\r\n\r\nSo, the situation now is more reasonable, but the resulting error message may not be as clear to the user.\r\n\r\nSince the error message was more accurate only by chance I think we should change the test and validate 2002 as the correct result with the new implementation. If we want to report cases such as this to the user as invalid file type instead of invalid encoding we'd need to add some kind of file type detection based on contents.\r\n"}}