{"action":"created","issue":{"url":"https://api.github.com/repos/chardet/chardet/issues/73","labels_url":"https://api.github.com/repos/chardet/chardet/issues/73/labels{/name}","comments_url":"https://api.github.com/repos/chardet/chardet/issues/73/comments","events_url":"https://api.github.com/repos/chardet/chardet/issues/73/events","html_url":"https://github.com/chardet/chardet/pull/73","id":110026495,"number":73,"title":"Don't indicate byte order for UTF-16/32 with given BOM","user":{"login":"snoack","id":1525,"avatar_url":"https://avatars.githubusercontent.com/u/1525?v=3","gravatar_id":"","url":"https://api.github.com/users/snoack","html_url":"https://github.com/snoack","followers_url":"https://api.github.com/users/snoack/followers","following_url":"https://api.github.com/users/snoack/following{/other_user}","gists_url":"https://api.github.com/users/snoack/gists{/gist_id}","starred_url":"https://api.github.com/users/snoack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snoack/subscriptions","organizations_url":"https://api.github.com/users/snoack/orgs","repos_url":"https://api.github.com/users/snoack/repos","events_url":"https://api.github.com/users/snoack/events{/privacy}","received_events_url":"https://api.github.com/users/snoack/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-10-06T14:55:58Z","updated_at":"2015-10-06T19:44:53Z","closed_at":"2015-10-06T19:44:37Z","pull_request":{"url":"https://api.github.com/repos/chardet/chardet/pulls/73","html_url":"https://github.com/chardet/chardet/pull/73","diff_url":"https://github.com/chardet/chardet/pull/73.diff","patch_url":"https://github.com/chardet/chardet/pull/73.patch"},"body":"If passed a string starting with `\\xff\\xfe` (low endian byte order mark) or `\\xfe\\xff` (big endian byte order mark) the encoding is detected as `UTF-16LE`, `UTF-32LE`, `UTF-16BE` or `UTF-32BE` respectively.\r\n\r\nHowever, as the byte order mark is given in the string, the encoding should be simply `UTF-16` or `UTF-32`. Otherwise `bytes.decode()` will fail or preserve the byte order mark:\r\n\r\n```python\r\ns = 'foo'.encode('UTF-16')\r\nencoding = chardet.detect(s)['encoding']  # \"UTF-16LE\"\r\ns.decode(encoding)                        # \"\\ufefffoo\"\r\n\r\ns = codecs.BOM_BE + 'foo'.encode('UTF-16BE')\r\nencoding = chardet.detect(s)['encoding']  # \"UTF-16BE\"\r\ns.decode(encoding)                        # \"\\ufefffoo\"\r\n```\r\n\r\nHence code that uses `chardet` in order to detect the encoding to decode data, would need to wrap `chardet.detect` in following inconvenient and counter-intuitive way:\r\n\r\n```python\r\nencoding = chardet.detect(enc)['encoding']\r\nif encoding in ('UTF-16LE', 'UTF16BE'):\r\n  dec = enc.decode('UTF-16')\r\nelif encoding in ('UTF-32LE', 'UTF-32BE'):\r\n  dec = enc.decode('UTF-32')\r\nelse:\r\n  dec = enc.decode(encoding)\r\n```\r\n\r\nThis PR changes the behavior to return simply `UTF-16`or `UTF-32` respectively when a byte order mark were found, that the detected encoding can be passed unchanged to `bytes.decode()`."},"comment":{"url":"https://api.github.com/repos/chardet/chardet/issues/comments/145976692","html_url":"https://github.com/chardet/chardet/pull/73#issuecomment-145976692","issue_url":"https://api.github.com/repos/chardet/chardet/issues/73","id":145976692,"user":{"login":"dan-blanchard","id":976345,"avatar_url":"https://avatars.githubusercontent.com/u/976345?v=3","gravatar_id":"","url":"https://api.github.com/users/dan-blanchard","html_url":"https://github.com/dan-blanchard","followers_url":"https://api.github.com/users/dan-blanchard/followers","following_url":"https://api.github.com/users/dan-blanchard/following{/other_user}","gists_url":"https://api.github.com/users/dan-blanchard/gists{/gist_id}","starred_url":"https://api.github.com/users/dan-blanchard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dan-blanchard/subscriptions","organizations_url":"https://api.github.com/users/dan-blanchard/orgs","repos_url":"https://api.github.com/users/dan-blanchard/repos","events_url":"https://api.github.com/users/dan-blanchard/events{/privacy}","received_events_url":"https://api.github.com/users/dan-blanchard/received_events","type":"User","site_admin":false},"created_at":"2015-10-06T19:44:53Z","updated_at":"2015-10-06T19:44:53Z","body":"Thanks for the PR and bringing this to our attention @snoack!"}}