{"action":"opened","issue":{"url":"https://api.github.com/repos/zmap/zmap/issues/301","labels_url":"https://api.github.com/repos/zmap/zmap/issues/301/labels{/name}","comments_url":"https://api.github.com/repos/zmap/zmap/issues/301/comments","events_url":"https://api.github.com/repos/zmap/zmap/issues/301/events","html_url":"https://github.com/zmap/zmap/issues/301","id":113260682,"number":301,"title":"ZMap does not guarantee UTF-8 ","user":{"login":"zakird","id":201296,"avatar_url":"https://avatars.githubusercontent.com/u/201296?v=3","gravatar_id":"","url":"https://api.github.com/users/zakird","html_url":"https://github.com/zakird","followers_url":"https://api.github.com/users/zakird/followers","following_url":"https://api.github.com/users/zakird/following{/other_user}","gists_url":"https://api.github.com/users/zakird/gists{/gist_id}","starred_url":"https://api.github.com/users/zakird/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zakird/subscriptions","organizations_url":"https://api.github.com/users/zakird/orgs","repos_url":"https://api.github.com/users/zakird/repos","events_url":"https://api.github.com/users/zakird/events{/privacy}","received_events_url":"https://api.github.com/users/zakird/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-10-25T23:01:37Z","updated_at":"2015-10-25T23:01:37Z","closed_at":null,"body":"ZMap currently outputs non-UTF8 encoded data. This hasn't been a problem previously because we constructed most of the strings. However, this model broken when the DNS module was introduced. We need to ensure that strings are UTF-8 valid (by escaping any invalid characters). A couple of notes:\r\n\r\n1) json-c does not escape invalid UTF-8 (so therefore doesn't save us). As well, we really want every output module to be outputting real strings\r\n\r\n2) This is what Golang does. We should probably do something similar \"String values encode as JSON strings coerced to valid UTF-8, replacing invalid bytes with the Unicode replacement rune. The angle brackets \"<\" and \">\" are escaped to \"\\u003c\" and \"\\u003e\" to keep some browsers from misinterpreting JSON output as HTML. Ampersand \"&\" is also escaped to \"\\u0026\" for the same reason.\"\r\n\r\n3) I'm attaching a file which is known to cause problems and can be used for testing. We can test by trying to parse the JSON with python.\r\n\r\n[out.txt](https://github.com/zmap/zmap/files/20331/out.txt)\r\n\r\n"}}