{"action":"created","issue":{"url":"https://api.github.com/repos/Murodese/pynab/issues/151","labels_url":"https://api.github.com/repos/Murodese/pynab/issues/151/labels{/name}","comments_url":"https://api.github.com/repos/Murodese/pynab/issues/151/comments","events_url":"https://api.github.com/repos/Murodese/pynab/issues/151/events","html_url":"https://github.com/Murodese/pynab/pull/151","id":56233365,"number":151,"title":"nzedb pre import","user":{"login":"brookesy2","id":3591904,"avatar_url":"https://avatars.githubusercontent.com/u/3591904?v=3","gravatar_id":"","url":"https://api.github.com/users/brookesy2","html_url":"https://github.com/brookesy2","followers_url":"https://api.github.com/users/brookesy2/followers","following_url":"https://api.github.com/users/brookesy2/following{/other_user}","gists_url":"https://api.github.com/users/brookesy2/gists{/gist_id}","starred_url":"https://api.github.com/users/brookesy2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brookesy2/subscriptions","organizations_url":"https://api.github.com/users/brookesy2/orgs","repos_url":"https://api.github.com/users/brookesy2/repos","events_url":"https://api.github.com/users/brookesy2/events{/privacy}","received_events_url":"https://api.github.com/users/brookesy2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":31,"created_at":"2015-02-02T12:53:44Z","updated_at":"2015-02-08T10:42:41Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Murodese/pynab/pulls/151","html_url":"https://github.com/Murodese/pynab/pull/151","diff_url":"https://github.com/Murodese/pynab/pull/151.diff","patch_url":"https://github.com/Murodese/pynab/pull/151.patch"},"body":"I only made this PR to chat about the below code.\r\n\r\nThe issue I have is that pre.name is a unique column, if I use COPY in postgres and a single line fails, the whole CSV fails, not to mention the fact that a release may get nuked later, so I really need to perform an upsert.\r\n\r\nAny thoughts on ways to do this? I was thinking of doing line by line and only committing after 100 or so lines, still have the same issue though, the need to upsert.\r\n\r\nEdit: Another thought, I could do it similar to how pre.py works, but I would need to requery the pre table after each iteration (or however many before I commit)"},"comment":{"url":"https://api.github.com/repos/Murodese/pynab/issues/comments/73405611","html_url":"https://github.com/Murodese/pynab/pull/151#issuecomment-73405611","issue_url":"https://api.github.com/repos/Murodese/pynab/issues/151","id":73405611,"user":{"login":"brookesy2","id":3591904,"avatar_url":"https://avatars.githubusercontent.com/u/3591904?v=3","gravatar_id":"","url":"https://api.github.com/users/brookesy2","html_url":"https://github.com/brookesy2","followers_url":"https://api.github.com/users/brookesy2/followers","following_url":"https://api.github.com/users/brookesy2/following{/other_user}","gists_url":"https://api.github.com/users/brookesy2/gists{/gist_id}","starred_url":"https://api.github.com/users/brookesy2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brookesy2/subscriptions","organizations_url":"https://api.github.com/users/brookesy2/orgs","repos_url":"https://api.github.com/users/brookesy2/repos","events_url":"https://api.github.com/users/brookesy2/events{/privacy}","received_events_url":"https://api.github.com/users/brookesy2/received_events","type":"User","site_admin":false},"created_at":"2015-02-08T10:42:41Z","updated_at":"2015-02-08T10:42:41Z","body":"I think I see why. A bunch of \\N's are in there. Which from original imports is what it used as a null value.\r\n\r\nIf you filter those from your pre table can you check if the import works?"}}