{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/11575","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/11575/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/11575/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/11575/events","html_url":"https://github.com/JuliaLang/julia/pull/11575","id":85180802,"number":11575,"title":"Add UTF encoding validity functions","user":{"login":"ScottPJones","id":11798751,"avatar_url":"https://avatars.githubusercontent.com/u/11798751?v=3","gravatar_id":"","url":"https://api.github.com/users/ScottPJones","html_url":"https://github.com/ScottPJones","followers_url":"https://api.github.com/users/ScottPJones/followers","following_url":"https://api.github.com/users/ScottPJones/following{/other_user}","gists_url":"https://api.github.com/users/ScottPJones/gists{/gist_id}","starred_url":"https://api.github.com/users/ScottPJones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ScottPJones/subscriptions","organizations_url":"https://api.github.com/users/ScottPJones/orgs","repos_url":"https://api.github.com/users/ScottPJones/repos","events_url":"https://api.github.com/users/ScottPJones/events{/privacy}","received_events_url":"https://api.github.com/users/ScottPJones/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/JuliaLang/julia/labels/unicode","name":"unicode","color":"5f4a60"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":47,"created_at":"2015-06-04T16:48:00Z","updated_at":"2015-06-15T01:49:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/JuliaLang/julia/pulls/11575","html_url":"https://github.com/JuliaLang/julia/pull/11575","diff_url":"https://github.com/JuliaLang/julia/pull/11575.diff","patch_url":"https://github.com/JuliaLang/julia/pull/11575.patch"},"body":"This introduces the `Base.check_string` function, with methods for handling UTF-8, UTF-16, and UTF-32 as vectors of `UInt8`, `UInt16`, and `UInt32`, respectively, as well as an `AbstractString` version that operates on Unicode characters (making sure that they are all valid code points, `0 <= ch < 0xd800`, `0xe000 <= ch < 0x10ffff`).\r\nThere are options to accept or not accept things like `Modified UTF-8` encoding, or `CESU-8` encoding,\r\nor \"overly long\" encodings.  (See `utfcheck.jl` for documentation)\r\nThese methods either throw a `UnicodeError`, with information about which character was invalid, and\r\nits position in the input, or they return a tuple, of the number of logical characters in the string, a bit flag to indicate what types of data were found (i.e. all ASCII, all Latin1, surrogates present, overlong characters present, etc.), the number of characters that would take 2 bytes to encode in UTF-8, the number that would take 3 bytes, and the number that would take 4 bytes.\r\nThis information is designed to be useful for future conversion routines, because they allow one to calculate the exact size needed to represent a valid string in UTF-8, UTF-16, or UTF-32, and also\r\nindicate whether an optimized \"widening\" or \"narrowing\" conversion can be done."},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/111897824","html_url":"https://github.com/JuliaLang/julia/pull/11575#issuecomment-111897824","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/11575","id":111897824,"user":{"login":"ScottPJones","id":11798751,"avatar_url":"https://avatars.githubusercontent.com/u/11798751?v=3","gravatar_id":"","url":"https://api.github.com/users/ScottPJones","html_url":"https://github.com/ScottPJones","followers_url":"https://api.github.com/users/ScottPJones/followers","following_url":"https://api.github.com/users/ScottPJones/following{/other_user}","gists_url":"https://api.github.com/users/ScottPJones/gists{/gist_id}","starred_url":"https://api.github.com/users/ScottPJones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ScottPJones/subscriptions","organizations_url":"https://api.github.com/users/ScottPJones/orgs","repos_url":"https://api.github.com/users/ScottPJones/repos","events_url":"https://api.github.com/users/ScottPJones/events{/privacy}","received_events_url":"https://api.github.com/users/ScottPJones/received_events","type":"User","site_admin":false},"created_at":"2015-06-15T01:49:50Z","updated_at":"2015-06-15T01:49:50Z","body":"@tkelman Done!\r\nBTW, I looked in base for `BoundsError`, and if using 1 instead of start() is a problem (and I agree with you that it could be), then the majority of the bounds checks in base are incorrect...\r\nShould I open an Issue on that? (with a PR to follow when I get a chance)?\r\nThanks!"}}