{"action":"created","issue":{"url":"https://api.github.com/repos/jgm/cmark-hs/issues/1","labels_url":"https://api.github.com/repos/jgm/cmark-hs/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/jgm/cmark-hs/issues/1/comments","events_url":"https://api.github.com/repos/jgm/cmark-hs/issues/1/events","html_url":"https://github.com/jgm/cmark-hs/issues/1","id":66618143,"number":1,"title":"Problem with string decoding","user":{"login":"zudov","id":2613966,"avatar_url":"https://avatars.githubusercontent.com/u/2613966?v=3","gravatar_id":"","url":"https://api.github.com/users/zudov","html_url":"https://github.com/zudov","followers_url":"https://api.github.com/users/zudov/followers","following_url":"https://api.github.com/users/zudov/following{/other_user}","gists_url":"https://api.github.com/users/zudov/gists{/gist_id}","starred_url":"https://api.github.com/users/zudov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zudov/subscriptions","organizations_url":"https://api.github.com/users/zudov/orgs","repos_url":"https://api.github.com/users/zudov/repos","events_url":"https://api.github.com/users/zudov/events{/privacy}","received_events_url":"https://api.github.com/users/zudov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-04-06T14:35:51Z","updated_at":"2015-04-07T08:17:44Z","closed_at":null,"body":"Hello. I met a bug which drives me crazy. \r\n\r\nThe problem is that *in very specific conditions* code `commonMarkToNode [] (T.pack \"«\")` fails with this error:\r\n\r\n```\r\nCannot decode byte '\\xc2': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream.\r\n```\r\n\r\nI guess that the problem is with string marshaling. Probably byte length is messed up with codepoint length somewhere. \r\n\r\nHere is a sniplet which allows to get the same error in pure haskell.\r\n```haskell\r\nPrelude> :m +Data.Text\r\nPrelude Data.Text> :m +Data.Text.Encoding\r\nPrelude Data.Text Data.Text.Encoding> let s = pack \"«\"\r\nPrelude Data.Text Data.Text.Encoding> decodeUtf8 $ Data.ByteString.take (Data.Text.length s) $ encodeUtf8 $ s\r\n\"*** Exception: Cannot decode byte '\\xc2': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream\r\nPrelude Data.Text Data.Text.Encoding> \r\n\r\n``` \r\n\r\nIt happens only if the string contains non-ascii symbols.\r\n\r\nTo reproduce a whole problem a very specific conditions must be met. It never occurs in interactive evaluation (only when called from binary). Unimporting some modules from binary can solve the problem. And so on. I am trying to say that it's really hard to reproduce, but if it's reproduced it stably appears on every execution. In case if the error-line above wouldn't give you insight on a problem, I can capture it and ship for experiments."},"comment":{"url":"https://api.github.com/repos/jgm/cmark-hs/issues/comments/90454325","html_url":"https://github.com/jgm/cmark-hs/issues/1#issuecomment-90454325","issue_url":"https://api.github.com/repos/jgm/cmark-hs/issues/1","id":90454325,"user":{"login":"zudov","id":2613966,"avatar_url":"https://avatars.githubusercontent.com/u/2613966?v=3","gravatar_id":"","url":"https://api.github.com/users/zudov","html_url":"https://github.com/zudov","followers_url":"https://api.github.com/users/zudov/followers","following_url":"https://api.github.com/users/zudov/following{/other_user}","gists_url":"https://api.github.com/users/zudov/gists{/gist_id}","starred_url":"https://api.github.com/users/zudov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zudov/subscriptions","organizations_url":"https://api.github.com/users/zudov/orgs","repos_url":"https://api.github.com/users/zudov/repos","events_url":"https://api.github.com/users/zudov/events{/privacy}","received_events_url":"https://api.github.com/users/zudov/received_events","type":"User","site_admin":false},"created_at":"2015-04-07T08:17:44Z","updated_at":"2015-04-07T08:17:44Z","body":"@jgm Unfortunately the latest commit didn't help. So I tried to make a demo for you.\r\n\r\nNo matter how I tried, I couldn't reproduce it with a standalone piece of code, so I had to prepare a full [repo](https://github.com/zudov/cmark-hs-bug-demo) with a travis [build](https://travis-ci.org/zudov/cmark-hs-bug-demo/builds/57443902) that demonstrates a failure.\r\n\r\nPrimarily, there are three files that you will be interested in:\r\n\r\n- [commonmark-parser/tests/Bug.hs](https://github.com/zudov/cmark-hs-bug-demo/blob/master/commonmark-parser/tests/Bug.hs) -- demonstrates a failure\r\n- [commonmark-parser/tests/BugDisappear1.hs](https://github.com/zudov/cmark-hs-bug-demo/blob/master/commonmark-parser/tests/BugDisappear1.hs), and\r\n- [commonmark-parser/tests/BugDisappear2.hs](https://github.com/zudov/cmark-hs-bug-demo/blob/master/commonmark-parser/tests/BugDisappear2.hs) -- demonstrate strange cases in which bug disappears\r\n\r\nA travis [install script](https://github.com/zudov/cmark-hs-bug-demo/blob/master/.travis.yml) builds all dependencies and installs git version of `cmark-hs`. It then executes a [demonstration script](https://github.com/zudov/cmark-hs-bug-demo/blob/master/travis.sh).\r\n\r\nThe three files that I mentioned above, are compiled as test-suites (standalone binaries).\r\n\r\nThen, `travis.sh` [runs](https://github.com/zudov/cmark-hs-bug-demo/blob/master/travis.sh#L3) `Bug.hs`'s binary (`bug`). This [fails](https://travis-ci.org/zudov/cmark-hs-bug-demo/builds/57443902#L489) with \"Cannot decode byte...\" error message. However if the same file is interpreted with `runhaskell` the `commonmarkToNode` works OK and produces expected [result](https://travis-ci.org/zudov/cmark-hs-bug-demo/builds/57443902#L491).\r\n\r\nThen `travis.sh` demonstrates this small crazy things that scare the bug away. Just look into `BugDisappear*.hs` files and lines 500-514 of a build log. Removing call to `hspec` fixes the bug. Removing redundant import fixes the bug. It's crazy.\r\n\r\n\r\n"}}