{"action":"created","issue":{"url":"https://api.github.com/repos/harelba/q/issues/98","labels_url":"https://api.github.com/repos/harelba/q/issues/98/labels{/name}","comments_url":"https://api.github.com/repos/harelba/q/issues/98/comments","events_url":"https://api.github.com/repos/harelba/q/issues/98/events","html_url":"https://github.com/harelba/q/issues/98","id":64210514,"number":98,"title":"support for UTF-16LE encoding","user":{"login":"Fil","id":7001,"avatar_url":"https://avatars.githubusercontent.com/u/7001?v=3","gravatar_id":"","url":"https://api.github.com/users/Fil","html_url":"https://github.com/Fil","followers_url":"https://api.github.com/users/Fil/followers","following_url":"https://api.github.com/users/Fil/following{/other_user}","gists_url":"https://api.github.com/users/Fil/gists{/gist_id}","starred_url":"https://api.github.com/users/Fil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fil/subscriptions","organizations_url":"https://api.github.com/users/Fil/orgs","repos_url":"https://api.github.com/users/Fil/repos","events_url":"https://api.github.com/users/Fil/events{/privacy}","received_events_url":"https://api.github.com/users/Fil/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-25T08:25:02Z","updated_at":"2015-03-25T08:32:45Z","closed_at":null,"body":"Some CSV files are encoded in UTF-16LE, so they can be correctly read by M$ Excel\r\n\r\nBut  `q` fails when reading them:\r\n```\r\n$ q -H -e \"UTF-16LE\" \"select uid FROM - \" < utf16le.csv \r\nError('line contains NULL byte',)\r\n```\r\n\r\ntest file (zipped so that it's not re-encoded on the fly): http://fil.rezo.net/utf16le.csv.zip"},"comment":{"url":"https://api.github.com/repos/harelba/q/issues/comments/85921482","html_url":"https://github.com/harelba/q/issues/98#issuecomment-85921482","issue_url":"https://api.github.com/repos/harelba/q/issues/98","id":85921482,"user":{"login":"Fil","id":7001,"avatar_url":"https://avatars.githubusercontent.com/u/7001?v=3","gravatar_id":"","url":"https://api.github.com/users/Fil","html_url":"https://github.com/Fil","followers_url":"https://api.github.com/users/Fil/followers","following_url":"https://api.github.com/users/Fil/following{/other_user}","gists_url":"https://api.github.com/users/Fil/gists{/gist_id}","starred_url":"https://api.github.com/users/Fil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fil/subscriptions","organizations_url":"https://api.github.com/users/Fil/orgs","repos_url":"https://api.github.com/users/Fil/repos","events_url":"https://api.github.com/users/Fil/events{/privacy}","received_events_url":"https://api.github.com/users/Fil/received_events","type":"User","site_admin":false},"created_at":"2015-03-25T08:32:45Z","updated_at":"2015-03-25T08:32:45Z","body":"```\r\n$ hexdump -C utf16le.csv\r\n00000000  ff fe 75 00 69 00 64 00  09 00 6e 00 61 00 6d 00  |..u.i.d...n.a.m.|\r\n00000010  65 00 0a 00 e9 00 74 00  f4 00 6e 00 6e 00 f9 00  |e.....t...n.n...|\r\n00000020  09 00 68 00 69 00 0a 00  29 06 4a 06 28 06 31 06  |..h.i...).J.(.1.|\r\n00000030  39 06 44 06 27 06 09 00  68 00 61 00 0a 00 2d 4e  |9.D.'...h.a...-N|\r\n00000040  fd 56 09 00 68 00 6f 00  0a 00                    |.V..h.o...|\r\n0000004a\r\n```\r\n\r\nAs you can see utf16-le has a two-bytes BOM. Skipping it (as with utf-8) does not solve the problem."}}