{"action":"created","issue":{"url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1582","labels_url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1582/labels{/name}","comments_url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1582/comments","events_url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1582/events","html_url":"https://github.com/ipfs/go-ipfs/issues/1582","id":101146522,"number":1582,"title":"JSON encoding corrupts non-UTF-8 binary data","user":{"login":"davidar","id":24291,"avatar_url":"https://avatars.githubusercontent.com/u/24291?v=3","gravatar_id":"","url":"https://api.github.com/users/davidar","html_url":"https://github.com/davidar","followers_url":"https://api.github.com/users/davidar/followers","following_url":"https://api.github.com/users/davidar/following{/other_user}","gists_url":"https://api.github.com/users/davidar/gists{/gist_id}","starred_url":"https://api.github.com/users/davidar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidar/subscriptions","organizations_url":"https://api.github.com/users/davidar/orgs","repos_url":"https://api.github.com/users/davidar/repos","events_url":"https://api.github.com/users/davidar/events{/privacy}","received_events_url":"https://api.github.com/users/davidar/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ipfs/go-ipfs/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/ipfs/go-ipfs/labels/commands","name":"commands","color":"207de5"}],"state":"open","locked":false,"assignee":{"login":"whyrusleeping","id":1243164,"avatar_url":"https://avatars.githubusercontent.com/u/1243164?v=3","gravatar_id":"","url":"https://api.github.com/users/whyrusleeping","html_url":"https://github.com/whyrusleeping","followers_url":"https://api.github.com/users/whyrusleeping/followers","following_url":"https://api.github.com/users/whyrusleeping/following{/other_user}","gists_url":"https://api.github.com/users/whyrusleeping/gists{/gist_id}","starred_url":"https://api.github.com/users/whyrusleeping/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whyrusleeping/subscriptions","organizations_url":"https://api.github.com/users/whyrusleeping/orgs","repos_url":"https://api.github.com/users/whyrusleeping/repos","events_url":"https://api.github.com/users/whyrusleeping/events{/privacy}","received_events_url":"https://api.github.com/users/whyrusleeping/received_events","type":"User","site_admin":false},"milestone":null,"comments":14,"created_at":"2015-08-15T06:11:16Z","updated_at":"2015-08-28T10:39:41Z","closed_at":null,"body":"The `Data` field of `ipfs object get --enc=json` tries to decode UTF-8 binary data into a unicode string (rather than a base64 encoded blob). This causes problems with binary data that isn't valid UTF-8:\r\n\r\n```\r\n$ wget https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt\r\n$ ipfs add UTF-8-test.txt \r\nadded QmVDcJvu7CiqeUeHhnkvej9UZu6PMDXxED5r2nuuzn2G7M UTF-8-test.txt\r\n$ ipfs object get --enc=json QmVDcJvu7CiqeUeHhnkvej9UZu6PMDXxED5r2nuuzn2G7M\r\n{\r\n  \"Links\": [],\r\n  \"Data\": \"\\u0008\\u0002\\u0012��\\u0001UTF-8 decoder capability and stress test\\n\r\n<snip>\r\n$ ipfs object get --enc=json QmVDcJvu7CiqeUeHhnkvej9UZu6PMDXxED5r2nuuzn2G7M | ipfs object put --inputenc=json\r\nadded QmegjSvPuqqZPu5PHuevWmj8rLoGADTo1dnfCpcNWs77Po\r\n```\r\n\r\nNote that the hashes don't match, as the json encoding has corrupted the data.\r\n\r\nThe problem seems to be line `core/commands/object.go:202`. I've tried fixing it myself, but I don't speak go. It looks like go's json module is [supposed to](https://golang.org/pkg/encoding/json/#Marshal) do base64 conversion automatically, but it isn't working for me."},"comment":{"url":"https://api.github.com/repos/ipfs/go-ipfs/issues/comments/135733510","html_url":"https://github.com/ipfs/go-ipfs/issues/1582#issuecomment-135733510","issue_url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1582","id":135733510,"user":{"login":"chriscool","id":208954,"avatar_url":"https://avatars.githubusercontent.com/u/208954?v=3","gravatar_id":"","url":"https://api.github.com/users/chriscool","html_url":"https://github.com/chriscool","followers_url":"https://api.github.com/users/chriscool/followers","following_url":"https://api.github.com/users/chriscool/following{/other_user}","gists_url":"https://api.github.com/users/chriscool/gists{/gist_id}","starred_url":"https://api.github.com/users/chriscool/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chriscool/subscriptions","organizations_url":"https://api.github.com/users/chriscool/orgs","repos_url":"https://api.github.com/users/chriscool/repos","events_url":"https://api.github.com/users/chriscool/events{/privacy}","received_events_url":"https://api.github.com/users/chriscool/received_events","type":"User","site_admin":false},"created_at":"2015-08-28T10:39:41Z","updated_at":"2015-08-28T10:39:41Z","body":"If I 'fix' the above test, then there is the following error:\r\n\r\n```\r\nexpecting success: \r\n                ipfs object put  ../t0051-object-data/testPut.json > actual_putOut\r\n\r\nError: illegal base64 data at input byte 4\r\nnot ok 9 - 'ipfs object put file.json' succeeds\r\n#\r\n#                       ipfs object put  ../t0051-object-data/testPut.json > actual_putOut\r\n#\r\n```\r\n"}}