{"action":"created","issue":{"url":"https://api.github.com/repos/tomduck/pandoc-fignos/issues/4","labels_url":"https://api.github.com/repos/tomduck/pandoc-fignos/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/tomduck/pandoc-fignos/issues/4/comments","events_url":"https://api.github.com/repos/tomduck/pandoc-fignos/issues/4/events","html_url":"https://github.com/tomduck/pandoc-fignos/issues/4","id":90108747,"number":4,"title":"Broken file encodings in utf-8","user":{"login":"dbobak","id":6285546,"avatar_url":"https://avatars.githubusercontent.com/u/6285546?v=3","gravatar_id":"","url":"https://api.github.com/users/dbobak","html_url":"https://github.com/dbobak","followers_url":"https://api.github.com/users/dbobak/followers","following_url":"https://api.github.com/users/dbobak/following{/other_user}","gists_url":"https://api.github.com/users/dbobak/gists{/gist_id}","starred_url":"https://api.github.com/users/dbobak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dbobak/subscriptions","organizations_url":"https://api.github.com/users/dbobak/orgs","repos_url":"https://api.github.com/users/dbobak/repos","events_url":"https://api.github.com/users/dbobak/events{/privacy}","received_events_url":"https://api.github.com/users/dbobak/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-06-22T14:10:50Z","updated_at":"2015-07-14T18:52:23Z","closed_at":null,"body":"When using option --filter pandoc-fignos, the encoding of utf-8 file gets broken (the national characters are messed).\r\n\r\nBelow is sample code of source markdown file:\r\n\r\n---\r\ntitle: Tytuł\r\nauthor:  Dariusz Bobak\r\ndate: 2015\r\ncsl: c:\\\\Users\\\\deni\\\\zotero\\\\archeo\\\\styles\\\\harvard-polish-archaeology.csl\r\nbibliography: c:\\\\users\\\\deni\\\\documents\\\\archeo\\\\library.bib\r\n...\r\n\r\n# Początek\r\n\r\nHere you can see problems with polish national characters: zażółć gęślą jaźń\r\nLorem ipsum dolor sit amet, „consectetur adipiscing «elit». Maecenas at nisi ut sem imperdiet” malesuada. Sed sit amet dolor in ligula commodo accumsan nec et justo. Nunc et velit id libero venenatis vehicula. Pellentesque malesuada bibendum fringilla. Nunc pharetra ut metus (@fig:wykres) vitae semper (\\autoref{fig:wykres}). Proin interdum, risus malesuada luctus fermentum, erat arcu tempus tellus, sed dictum quam magna sed quam. Vivamus accumsan id lectus a hendrerit. Etiam nibh turpis, molestie eget dapibus [@bisson_nineteenth_2000] ac, vestibulum nec risus.\r\n\r\n![Wykres klimatyczny, w którym pokazujemy mnóstwo ciekawych rzeczy. Na przykład, jak zmieniał się klimat bardzo dawno temu.](fig/wykres.png) {#fig:wykres}\r\n\r\nDonec elementum dolor [@adams_bukk_2009] urna, in rhoncus sapien eleifend vitae. Donec dignissim lacus at tellus convallis, sit amet consectetur metus tristique. Donec nec odio ultrices, varius dui sit amet, sodales mi. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Praesent libero enim, bibendum eu odio imperdiet, finibus pharetra felis. Proin ut blandit est, in accumsan odio. Sed dolor quam, varius non massa vitae, volutpat maximus erat. Etiam lacinia vulputate est a sodales. Aenean nec volutpat leo. Nullam scelerisque turpis eget suscipit varius. Donec mi tortor, sollicitudin sed ligula id, hendrerit imperdiet lectus. Quisque ac tortor quis ligula imperdiet suscipit a sit amet sem. Aliquam erat volutpat.\r\n\r\n## Podrozdział pierwszego poziomu\r\n\r\nSed pretium sit amet magna eu tempor. Ut sagittis est et turpis fringilla, a ultrices ex lacinia. Vestibulum leo est, lacinia at egestas a, dignissim maximus velit. Integer nisl neque, fermentum eget congue ac, pulvinar a mauris. Praesent dolor nisi, semper non tortor eu, lobortis mattis nibh. Maecenas rhoncus lorem vel dapibus semper. Praesent vel dolor non velit aliquam scelerisque. Cras quis tincidunt lorem, ac consectetur sapien. Aliquam euismod justo sed leo fringilla pellentesque. Nunc accumsan ut lorem a aliquet.\r\n\r\n### Podrozdział drugiego poziomu\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tomduck/pandoc-fignos/issues/comments/121339635","html_url":"https://github.com/tomduck/pandoc-fignos/issues/4#issuecomment-121339635","issue_url":"https://api.github.com/repos/tomduck/pandoc-fignos/issues/4","id":121339635,"user":{"login":"tomduck","id":315246,"avatar_url":"https://avatars.githubusercontent.com/u/315246?v=3","gravatar_id":"","url":"https://api.github.com/users/tomduck","html_url":"https://github.com/tomduck","followers_url":"https://api.github.com/users/tomduck/followers","following_url":"https://api.github.com/users/tomduck/following{/other_user}","gists_url":"https://api.github.com/users/tomduck/gists{/gist_id}","starred_url":"https://api.github.com/users/tomduck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomduck/subscriptions","organizations_url":"https://api.github.com/users/tomduck/orgs","repos_url":"https://api.github.com/users/tomduck/repos","events_url":"https://api.github.com/users/tomduck/events{/privacy}","received_events_url":"https://api.github.com/users/tomduck/received_events","type":"User","site_admin":false},"created_at":"2015-07-14T18:52:23Z","updated_at":"2015-07-14T18:52:23Z","body":"@dbobak: I have pushed some commits to github.  Could you please install pandoc-fignos from source to see if the changes solve your problem?  I will push out a new version to pypi once we have agreed that the bug is fixed.\r\n\r\nNote: This works with Windows Console.  Powershell appears to mess with the encoding when using pipelines (as described a few posts earlier).\r\n\r\n-----\r\n\r\nHere is the situation as I understand it (documented here for future reference).\r\n\r\nFrom the [pandoc documentation](http://pandoc.org/README.html):\r\n\r\n> Pandoc uses the UTF-8 character encoding for both input and output.\r\n\r\nPandoc-fignos must therefore do the same.  In both python 2 and 3, a call to `sys.stdin.read()` returns a string.  The catch is that strings are different kinds of objects between the two python versions.\r\n\r\n[Python 2 strings](https://docs.python.org/2.7/reference/lexical_analysis.html#strings) are represented by a series of bytes (ASCII characters).  Any decoding is left to the programmer.  Since pandoc-fignos mostly passes bytes through to `sys.stdout` there is no ill effect on any platform.\r\n\r\n[Python 3 strings](https://docs.python.org/3/reference/lexical_analysis.html#strings), on the other hand, are unicode, and so decoding is performed automatically.  The [default expectations](https://docs.python.org/3.5/library/sys.html) for encoding depend on the OS:\r\n\r\n> The character encoding is platform-dependent. Under Windows, if the stream is interactive (that is, if its isatty() method returns True), the console codepage is used, otherwise the ANSI code page. Under other platforms, the locale encoding is used (see locale.getpreferredencoding()).\r\n\r\nUTF-8 is the default encoding of most locales for [many linux distributions](https://www.cl.cam.ac.uk/~mgk25/unicode.html#linux) and [Mac OS X](https://en.wikipedia.org/wiki/Mac_OS_Roman#Application_notes).  This is why pandoc-fignos worked with those OSes.  On Windows with python 3 the default encoding is something else, hence we need to enforce pandoc's requirement.  Stackoverflow shows [how to do this](https://stackoverflow.com/questions/16549332/python-3-how-to-specify-stdin-encoding).\r\n\r\nOne final note: The change between python 2 and 3 addressed a variety of problems (viz., [\"the change is for the better\"](https://docs.python.org/3/whatsnew/3.0.html)).  What we have here appears to be an edge case where things were made a little more difficult in python 3."}}