{"action":"created","issue":{"url":"https://api.github.com/repos/mgedmin/check-manifest/issues/62","labels_url":"https://api.github.com/repos/mgedmin/check-manifest/issues/62/labels{/name}","comments_url":"https://api.github.com/repos/mgedmin/check-manifest/issues/62/comments","events_url":"https://api.github.com/repos/mgedmin/check-manifest/issues/62/events","html_url":"https://github.com/mgedmin/check-manifest/issues/62","id":122239770,"number":62,"title":"Bazaar and filename encodings on Windows","user":{"login":"mgedmin","id":159967,"avatar_url":"https://avatars.githubusercontent.com/u/159967?v=3","gravatar_id":"","url":"https://api.github.com/users/mgedmin","html_url":"https://github.com/mgedmin","followers_url":"https://api.github.com/users/mgedmin/followers","following_url":"https://api.github.com/users/mgedmin/following{/other_user}","gists_url":"https://api.github.com/users/mgedmin/gists{/gist_id}","starred_url":"https://api.github.com/users/mgedmin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgedmin/subscriptions","organizations_url":"https://api.github.com/users/mgedmin/orgs","repos_url":"https://api.github.com/users/mgedmin/repos","events_url":"https://api.github.com/users/mgedmin/events{/privacy}","received_events_url":"https://api.github.com/users/mgedmin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-15T10:25:52Z","updated_at":"2015-12-15T17:03:20Z","closed_at":null,"body":"Currently I have a test failure on my Jenkins:\r\n```\r\n======================================================================\r\nFAIL: test_get_vcs_files_nonascii_filenames (tests.TestBzr)\r\n----------------------------------------------------------------------\r\nTraceback (most recent call last):\r\n  File \"c:\\buildslave\\workspace\\check-manifest-on-windows\\tests.py\", line 842, in test_get_vcs_files_nonascii_filenames\r\n    self.assertEqual(get_vcs_files(), [filename])\r\nAssertionError: Lists differ: [u'\\u201a.txt'] != [u'\\xe9.txt']\r\n\r\nFirst differing element 0:\r\n\\u201a.txt\r\n\\xe9.txt\r\n\r\n- [u'\\u201a.txt']\r\n+ [u'\\xe9.txt']\r\n\r\n----------------------------------------------------------------------\r\n```\r\n\r\nCuriously, it [showed up](https://jenkins.gedmin.as/job/check-manifest-on-windows/410/console) on my Jenkins when I [fixed it](https://jenkins.gedmin.as/job/check-manifest-on-windows/410/changes) on Appveyor.\r\n\r\nEven more curiously, I used to see this when I ran the tests manually after rdesktop'ing into the Windows VM, and when I fixed it on Appveyor, I stopped seeing this error in manual tests.\r\n\r\n(\"Manual\" means \"running tox from git bash\".)"},"comment":{"url":"https://api.github.com/repos/mgedmin/check-manifest/issues/comments/164828018","html_url":"https://github.com/mgedmin/check-manifest/issues/62#issuecomment-164828018","issue_url":"https://api.github.com/repos/mgedmin/check-manifest/issues/62","id":164828018,"user":{"login":"mgedmin","id":159967,"avatar_url":"https://avatars.githubusercontent.com/u/159967?v=3","gravatar_id":"","url":"https://api.github.com/users/mgedmin","html_url":"https://github.com/mgedmin","followers_url":"https://api.github.com/users/mgedmin/followers","following_url":"https://api.github.com/users/mgedmin/following{/other_user}","gists_url":"https://api.github.com/users/mgedmin/gists{/gist_id}","starred_url":"https://api.github.com/users/mgedmin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgedmin/subscriptions","organizations_url":"https://api.github.com/users/mgedmin/orgs","repos_url":"https://api.github.com/users/mgedmin/repos","events_url":"https://api.github.com/users/mgedmin/events{/privacy}","received_events_url":"https://api.github.com/users/mgedmin/received_events","type":"User","site_admin":false},"created_at":"2015-12-15T17:03:19Z","updated_at":"2015-12-15T17:03:19Z","body":"What the test does:\r\n\r\n- create a file by passing `u'\\xe9.txt'` to Python's `open()` (which ought to be translated into a native Win32 Unicode API call, AFAIU)\r\n- add it into version control (using `['bzr', 'add', '--', u'\\xe9.txt'.encode(locale.getpreferredencoding() ]` passed to `subprocess.Popen()`)\r\n- call `subprocess.Popen(['bzr', 'ls', '-VR'])`, then decode the standard output using `sys.stdout.encoding` (or `sys.stdin.encoding`, or `locale.getpreferredencoding()`).\r\n- expect to get `u'\\xe9.txt'` back.\r\n\r\nIn my manual experiments on the Jenkins VM, `locale.getpreferredencoding()` returns cp1252, and `sys.stdout.encoding` is cp437.\r\n\r\nU+00E9 is &eacute;.  In cp1252 it is represented by `\\xe9`.  In cp437 it is represented by `\\x82`.\r\n\r\nWhen you decode `\\xe9` using cp437, you get `\\u0398`.\r\n\r\nWhen you decode `\\x82` using cp1252, you get `\\u201a`.  Bingo!\r\n\r\nSomehow, under Jenkins, the test is unable to determine the encoding from `sys.stdout`/`sys.stdin`, and therefore falls back to `locale.getpreferredencoding()`."}}