{"action":"created","issue":{"url":"https://api.github.com/repos/SFTtech/openage/issues/408","labels_url":"https://api.github.com/repos/SFTtech/openage/issues/408/labels{/name}","comments_url":"https://api.github.com/repos/SFTtech/openage/issues/408/comments","events_url":"https://api.github.com/repos/SFTtech/openage/issues/408/events","html_url":"https://github.com/SFTtech/openage/issues/408","id":116190864,"number":408,"title":"Converter script doesn't specify text file encoding","user":{"login":"iNaKoll","id":4791576,"avatar_url":"https://avatars.githubusercontent.com/u/4791576?v=3","gravatar_id":"","url":"https://api.github.com/users/iNaKoll","html_url":"https://github.com/iNaKoll","followers_url":"https://api.github.com/users/iNaKoll/followers","following_url":"https://api.github.com/users/iNaKoll/following{/other_user}","gists_url":"https://api.github.com/users/iNaKoll/gists{/gist_id}","starred_url":"https://api.github.com/users/iNaKoll/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iNaKoll/subscriptions","organizations_url":"https://api.github.com/users/iNaKoll/orgs","repos_url":"https://api.github.com/users/iNaKoll/repos","events_url":"https://api.github.com/users/iNaKoll/events{/privacy}","received_events_url":"https://api.github.com/users/iNaKoll/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/SFTtech/openage/labels/improvement","name":"improvement","color":"84b6eb"},{"url":"https://api.github.com/repos/SFTtech/openage/labels/proposal","name":"proposal","color":"6445d9"},{"url":"https://api.github.com/repos/SFTtech/openage/labels/python","name":"python","color":"bfe5bf"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-10T20:06:25Z","updated_at":"2015-11-10T21:04:49Z","closed_at":null,"body":"While running the convert script, [util.fslike.path](https://github.com/SFTtech/openage/blob/0f895d3dbb0058d710dd8f0a839c3677b9b7edbe/openage/util/fslike/path.py#L100-L104) doesn't specify text file encoding nor give any mean to specify one. This could lead to some unicode encoding/decoding error at some point.\r\n\r\nThe default encoding used by [TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper) is `locale.getpreferredencoding(False)` and depending on the system this doesn't necessarily return 'utf-8' ( ;o )\r\n\r\nTo reproduce the problem simply change the system locale (change the `LANG` environment variable on Linux systems) to another locale (try `ANSIX3.4-1968`) before the convert script runs. Hopefully, the script should raise an `UnicodeEncodingError` exception at some point while trying to encode a Python unicode string...\r\n\r\nIf we know the encoding of the our files, whether we are reading or writing them, we should specify that encoding explicitly instead of relying on some [weird default system encoding](http://askubuntu.com/questions/581458/how-to-configure-locales-to-unicode-in-a-docker-ubuntu-14-04-container).\r\n\r\n<bountysource-plugin>\r\n---\r\nWant to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/28137899-converter-script-doesn-t-specify-text-file-encoding?utm_campaign=plugin&utm_content=tracker%2F371436&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F371436&utm_medium=issues&utm_source=github).\r\n</bountysource-plugin>"},"comment":{"url":"https://api.github.com/repos/SFTtech/openage/issues/comments/155565398","html_url":"https://github.com/SFTtech/openage/issues/408#issuecomment-155565398","issue_url":"https://api.github.com/repos/SFTtech/openage/issues/408","id":155565398,"user":{"login":"mic-e","id":796624,"avatar_url":"https://avatars.githubusercontent.com/u/796624?v=3","gravatar_id":"","url":"https://api.github.com/users/mic-e","html_url":"https://github.com/mic-e","followers_url":"https://api.github.com/users/mic-e/followers","following_url":"https://api.github.com/users/mic-e/following{/other_user}","gists_url":"https://api.github.com/users/mic-e/gists{/gist_id}","starred_url":"https://api.github.com/users/mic-e/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mic-e/subscriptions","organizations_url":"https://api.github.com/users/mic-e/orgs","repos_url":"https://api.github.com/users/mic-e/repos","events_url":"https://api.github.com/users/mic-e/events{/privacy}","received_events_url":"https://api.github.com/users/mic-e/received_events","type":"User","site_admin":false},"created_at":"2015-11-10T21:04:49Z","updated_at":"2015-11-10T21:04:49Z","body":"I had hoped that Python would always use `utf-8`, but it seems like my darkest fears turned out to be true after all."}}