{"action":"created","comment":{"url":"https://api.github.com/repos/JuliaLang/julia/pulls/comments/31875932","id":31875932,"diff_hunk":"@@ -0,0 +1,195 @@\n+# This file is a part of Julia. License is MIT: http://julialang.org/license\n+\n+## Functions to check validity of UTF-8, UTF-16, and UTF-32 encoded strings,\n+#  and also to return information necessary to convert to other encodings\n+\n+is_surrogate_lead(c::Unsigned) = ((c & ~0x003ff) == 0xd800)\n+is_surrogate_trail(c::Unsigned) = ((c & ~0x003ff) == 0xdc00)\n+is_surrogate_codeunit(c::Unsigned) = ((c & ~0x007ff) == 0xd800)\n+is_valid_continuation(c) = ((c & 0xc0) == 0x80)\n+\n+## Options for check_string_* functions\n+\n+const UTF_NO_LONG_NULL = 1      ##< don't accept 0xc0 0x80 for '\\0'\n+const UTF_NO_SURROGATES = 2     ##< don't accept surrogate pairs in UTF-8/UTF-32\n+const UTF_ACCEPT_LONG = 4       ##< accept long encodings (other than long null in UTF-8)\n+\n+const UTF_LONG = 1              ##< Long encodings are present\n+const UTF_LATIN1 = 2            ##< characters in range 0x80-0xFF present\n+const UTF_UNICODE2 = 4          ##< characters in range 0x100-0x7ff present\n+const UTF_UNICODE3 = 8          ##< characters in range 0x800-0xd7ff, 0xe000-0xffff\n+const UTF_UNICODE4 = 16         ##< non-BMP characters present\n+const UTF_SURROGATE = 32        ##< surrogate pairs present\n+\n+## Get a UTF-8 continuation byte, give error if invalid, return updated character value\n+@inline function get_continuation(ch::UInt32, byt::UInt8, pos)\n+    !is_valid_continuation(byt) && throw(UnicodeError(UTF_ERR_CONT, pos, byt))\n+    (ch << 6) | (byt & 0x3f)\n+end\n+\n+function check_string(dat::Vector{UInt8}, len = sizeof(dat), pos = 0 ; options::Integer=0)\n+\" Validates and calculates number of characters in a UTF-8 encoded vector of UInt8","path":"base/utfcheck.jl","position":31,"original_position":31,"commit_id":"da43b2a5a188d1104c3df7c2594fe4f5441c70d1","original_commit_id":"da43b2a5a188d1104c3df7c2594fe4f5441c70d1","user":{"login":"tkelman","id":5934628,"avatar_url":"https://avatars.githubusercontent.com/u/5934628?v=3","gravatar_id":"","url":"https://api.github.com/users/tkelman","html_url":"https://github.com/tkelman","followers_url":"https://api.github.com/users/tkelman/followers","following_url":"https://api.github.com/users/tkelman/following{/other_user}","gists_url":"https://api.github.com/users/tkelman/gists{/gist_id}","starred_url":"https://api.github.com/users/tkelman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tkelman/subscriptions","organizations_url":"https://api.github.com/users/tkelman/orgs","repos_url":"https://api.github.com/users/tkelman/repos","events_url":"https://api.github.com/users/tkelman/events{/privacy}","received_events_url":"https://api.github.com/users/tkelman/received_events","type":"User","site_admin":false},"body":"I believe @nalimilan said this in one of the other PR's, but the doc string convention is more likely to be docs immediately preceding the function, not just inside the function. Not positive though.","created_at":"2015-06-07T11:16:13Z","updated_at":"2015-06-07T11:16:13Z","html_url":"https://github.com/JuliaLang/julia/pull/11575#discussion_r31875932","pull_request_url":"https://api.github.com/repos/JuliaLang/julia/pulls/11575","_links":{"self":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/comments/31875932"},"html":{"href":"https://github.com/JuliaLang/julia/pull/11575#discussion_r31875932"},"pull_request":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/11575"}}},"pull_request":{"url":"https://api.github.com/repos/JuliaLang/julia/pulls/11575","id":36954484,"html_url":"https://github.com/JuliaLang/julia/pull/11575","diff_url":"https://github.com/JuliaLang/julia/pull/11575.diff","patch_url":"https://github.com/JuliaLang/julia/pull/11575.patch","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/11575","number":11575,"state":"open","locked":false,"title":"Add UTF encoding validity functions","user":{"login":"ScottPJones","id":11798751,"avatar_url":"https://avatars.githubusercontent.com/u/11798751?v=3","gravatar_id":"","url":"https://api.github.com/users/ScottPJones","html_url":"https://github.com/ScottPJones","followers_url":"https://api.github.com/users/ScottPJones/followers","following_url":"https://api.github.com/users/ScottPJones/following{/other_user}","gists_url":"https://api.github.com/users/ScottPJones/gists{/gist_id}","starred_url":"https://api.github.com/users/ScottPJones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ScottPJones/subscriptions","organizations_url":"https://api.github.com/users/ScottPJones/orgs","repos_url":"https://api.github.com/users/ScottPJones/repos","events_url":"https://api.github.com/users/ScottPJones/events{/privacy}","received_events_url":"https://api.github.com/users/ScottPJones/received_events","type":"User","site_admin":false},"body":"This introduces the `Base.check_string` function, with methods for handling UTF-8, UTF-16, and UTF-32 as vectors of `UInt8`, `UInt16`, and `UInt32`, respectively, as well as an `AbstractString` version that operates on Unicode characters (making sure that they are all valid code points, `0 <= ch < 0xd800`, `0xe000 <= ch < 0x10ffff`).\r\nThere are options to accept or not accept things like `Modified UTF-8` encoding, or `CESU-8` encoding,\r\nor \"overly long\" encodings.  (See `utfcheck.jl` for documentation)\r\nThese methods either throw a `UnicodeError`, with information about which character was invalid, and\r\nits position in the input, or they return a tuple, of the number of logical characters in the string, a bit flag to indicate what types of data were found (i.e. all ASCII, all Latin1, surrogates present, overlong characters present, etc.), the number of characters that would take 2 bytes to encode in UTF-8, the number that would take 3 bytes, and the number that would take 4 bytes.\r\nThis information is designed to be useful for future conversion routines, because they allow one to calculate the exact size needed to represent a valid string in UTF-8, UTF-16, or UTF-32, and also\r\nindicate whether an optimized \"widening\" or \"narrowing\" conversion can be done.","created_at":"2015-06-04T16:48:00Z","updated_at":"2015-06-07T11:16:13Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2c9174ab8b1f27ebaae30ffc7dd9c2d0259a268d","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/JuliaLang/julia/pulls/11575/commits","review_comments_url":"https://api.github.com/repos/JuliaLang/julia/pulls/11575/comments","review_comment_url":"https://api.github.com/repos/JuliaLang/julia/pulls/comments{/number}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/11575/comments","statuses_url":"https://api.github.com/repos/JuliaLang/julia/statuses/da43b2a5a188d1104c3df7c2594fe4f5441c70d1","head":{"label":"ScottPJones:spj/checkstring","ref":"spj/checkstring","sha":"da43b2a5a188d1104c3df7c2594fe4f5441c70d1","user":{"login":"ScottPJones","id":11798751,"avatar_url":"https://avatars.githubusercontent.com/u/11798751?v=3","gravatar_id":"","url":"https://api.github.com/users/ScottPJones","html_url":"https://github.com/ScottPJones","followers_url":"https://api.github.com/users/ScottPJones/followers","following_url":"https://api.github.com/users/ScottPJones/following{/other_user}","gists_url":"https://api.github.com/users/ScottPJones/gists{/gist_id}","starred_url":"https://api.github.com/users/ScottPJones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ScottPJones/subscriptions","organizations_url":"https://api.github.com/users/ScottPJones/orgs","repos_url":"https://api.github.com/users/ScottPJones/repos","events_url":"https://api.github.com/users/ScottPJones/events{/privacy}","received_events_url":"https://api.github.com/users/ScottPJones/received_events","type":"User","site_admin":false},"repo":{"id":34334720,"name":"julia","full_name":"ScottPJones/julia","owner":{"login":"ScottPJones","id":11798751,"avatar_url":"https://avatars.githubusercontent.com/u/11798751?v=3","gravatar_id":"","url":"https://api.github.com/users/ScottPJones","html_url":"https://github.com/ScottPJones","followers_url":"https://api.github.com/users/ScottPJones/followers","following_url":"https://api.github.com/users/ScottPJones/following{/other_user}","gists_url":"https://api.github.com/users/ScottPJones/gists{/gist_id}","starred_url":"https://api.github.com/users/ScottPJones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ScottPJones/subscriptions","organizations_url":"https://api.github.com/users/ScottPJones/orgs","repos_url":"https://api.github.com/users/ScottPJones/repos","events_url":"https://api.github.com/users/ScottPJones/events{/privacy}","received_events_url":"https://api.github.com/users/ScottPJones/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ScottPJones/julia","description":"The Julia Language: A fresh approach to technical computing.","fork":true,"url":"https://api.github.com/repos/ScottPJones/julia","forks_url":"https://api.github.com/repos/ScottPJones/julia/forks","keys_url":"https://api.github.com/repos/ScottPJones/julia/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ScottPJones/julia/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ScottPJones/julia/teams","hooks_url":"https://api.github.com/repos/ScottPJones/julia/hooks","issue_events_url":"https://api.github.com/repos/ScottPJones/julia/issues/events{/number}","events_url":"https://api.github.com/repos/ScottPJones/julia/events","assignees_url":"https://api.github.com/repos/ScottPJones/julia/assignees{/user}","branches_url":"https://api.github.com/repos/ScottPJones/julia/branches{/branch}","tags_url":"https://api.github.com/repos/ScottPJones/julia/tags","blobs_url":"https://api.github.com/repos/ScottPJones/julia/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ScottPJones/julia/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ScottPJones/julia/git/refs{/sha}","trees_url":"https://api.github.com/repos/ScottPJones/julia/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ScottPJones/julia/statuses/{sha}","languages_url":"https://api.github.com/repos/ScottPJones/julia/languages","stargazers_url":"https://api.github.com/repos/ScottPJones/julia/stargazers","contributors_url":"https://api.github.com/repos/ScottPJones/julia/contributors","subscribers_url":"https://api.github.com/repos/ScottPJones/julia/subscribers","subscription_url":"https://api.github.com/repos/ScottPJones/julia/subscription","commits_url":"https://api.github.com/repos/ScottPJones/julia/commits{/sha}","git_commits_url":"https://api.github.com/repos/ScottPJones/julia/git/commits{/sha}","comments_url":"https://api.github.com/repos/ScottPJones/julia/comments{/number}","issue_comment_url":"https://api.github.com/repos/ScottPJones/julia/issues/comments{/number}","contents_url":"https://api.github.com/repos/ScottPJones/julia/contents/{+path}","compare_url":"https://api.github.com/repos/ScottPJones/julia/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ScottPJones/julia/merges","archive_url":"https://api.github.com/repos/ScottPJones/julia/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ScottPJones/julia/downloads","issues_url":"https://api.github.com/repos/ScottPJones/julia/issues{/number}","pulls_url":"https://api.github.com/repos/ScottPJones/julia/pulls{/number}","milestones_url":"https://api.github.com/repos/ScottPJones/julia/milestones{/number}","notifications_url":"https://api.github.com/repos/ScottPJones/julia/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ScottPJones/julia/labels{/name}","releases_url":"https://api.github.com/repos/ScottPJones/julia/releases{/id}","created_at":"2015-04-21T15:12:29Z","updated_at":"2015-05-08T00:23:59Z","pushed_at":"2015-06-07T09:27:40Z","git_url":"git://github.com/ScottPJones/julia.git","ssh_url":"git@github.com:ScottPJones/julia.git","clone_url":"https://github.com/ScottPJones/julia.git","svn_url":"https://github.com/ScottPJones/julia","homepage":"http://julialang.org/","size":73593,"stargazers_count":0,"watchers_count":0,"language":"Julia","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"JuliaLang:master","ref":"master","sha":"01abdc3f2bc0e3611cf14aa331cdc6a10f0dfa0e","user":{"login":"JuliaLang","id":743164,"avatar_url":"https://avatars.githubusercontent.com/u/743164?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaLang","html_url":"https://github.com/JuliaLang","followers_url":"https://api.github.com/users/JuliaLang/followers","following_url":"https://api.github.com/users/JuliaLang/following{/other_user}","gists_url":"https://api.github.com/users/JuliaLang/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaLang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaLang/subscriptions","organizations_url":"https://api.github.com/users/JuliaLang/orgs","repos_url":"https://api.github.com/users/JuliaLang/repos","events_url":"https://api.github.com/users/JuliaLang/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaLang/received_events","type":"Organization","site_admin":false},"repo":{"id":1644196,"name":"julia","full_name":"JuliaLang/julia","owner":{"login":"JuliaLang","id":743164,"avatar_url":"https://avatars.githubusercontent.com/u/743164?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaLang","html_url":"https://github.com/JuliaLang","followers_url":"https://api.github.com/users/JuliaLang/followers","following_url":"https://api.github.com/users/JuliaLang/following{/other_user}","gists_url":"https://api.github.com/users/JuliaLang/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaLang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaLang/subscriptions","organizations_url":"https://api.github.com/users/JuliaLang/orgs","repos_url":"https://api.github.com/users/JuliaLang/repos","events_url":"https://api.github.com/users/JuliaLang/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaLang/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/JuliaLang/julia","description":"The Julia Language: A fresh approach to technical computing.","fork":false,"url":"https://api.github.com/repos/JuliaLang/julia","forks_url":"https://api.github.com/repos/JuliaLang/julia/forks","keys_url":"https://api.github.com/repos/JuliaLang/julia/keys{/key_id}","collaborators_url":"https://api.github.com/repos/JuliaLang/julia/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/JuliaLang/julia/teams","hooks_url":"https://api.github.com/repos/JuliaLang/julia/hooks","issue_events_url":"https://api.github.com/repos/JuliaLang/julia/issues/events{/number}","events_url":"https://api.github.com/repos/JuliaLang/julia/events","assignees_url":"https://api.github.com/repos/JuliaLang/julia/assignees{/user}","branches_url":"https://api.github.com/repos/JuliaLang/julia/branches{/branch}","tags_url":"https://api.github.com/repos/JuliaLang/julia/tags","blobs_url":"https://api.github.com/repos/JuliaLang/julia/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/JuliaLang/julia/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/JuliaLang/julia/git/refs{/sha}","trees_url":"https://api.github.com/repos/JuliaLang/julia/git/trees{/sha}","statuses_url":"https://api.github.com/repos/JuliaLang/julia/statuses/{sha}","languages_url":"https://api.github.com/repos/JuliaLang/julia/languages","stargazers_url":"https://api.github.com/repos/JuliaLang/julia/stargazers","contributors_url":"https://api.github.com/repos/JuliaLang/julia/contributors","subscribers_url":"https://api.github.com/repos/JuliaLang/julia/subscribers","subscription_url":"https://api.github.com/repos/JuliaLang/julia/subscription","commits_url":"https://api.github.com/repos/JuliaLang/julia/commits{/sha}","git_commits_url":"https://api.github.com/repos/JuliaLang/julia/git/commits{/sha}","comments_url":"https://api.github.com/repos/JuliaLang/julia/comments{/number}","issue_comment_url":"https://api.github.com/repos/JuliaLang/julia/issues/comments{/number}","contents_url":"https://api.github.com/repos/JuliaLang/julia/contents/{+path}","compare_url":"https://api.github.com/repos/JuliaLang/julia/compare/{base}...{head}","merges_url":"https://api.github.com/repos/JuliaLang/julia/merges","archive_url":"https://api.github.com/repos/JuliaLang/julia/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/JuliaLang/julia/downloads","issues_url":"https://api.github.com/repos/JuliaLang/julia/issues{/number}","pulls_url":"https://api.github.com/repos/JuliaLang/julia/pulls{/number}","milestones_url":"https://api.github.com/repos/JuliaLang/julia/milestones{/number}","notifications_url":"https://api.github.com/repos/JuliaLang/julia/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/JuliaLang/julia/labels{/name}","releases_url":"https://api.github.com/repos/JuliaLang/julia/releases{/id}","created_at":"2011-04-21T07:01:50Z","updated_at":"2015-06-07T07:04:34Z","pushed_at":"2015-06-07T09:29:59Z","git_url":"git://github.com/JuliaLang/julia.git","ssh_url":"git@github.com:JuliaLang/julia.git","clone_url":"https://github.com/JuliaLang/julia.git","svn_url":"https://github.com/JuliaLang/julia","homepage":"http://julialang.org/","size":187924,"stargazers_count":5640,"watchers_count":5640,"language":"Julia","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1185,"mirror_url":null,"open_issues_count":1284,"forks":1185,"open_issues":1284,"watchers":5640,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/11575"},"html":{"href":"https://github.com/JuliaLang/julia/pull/11575"},"issue":{"href":"https://api.github.com/repos/JuliaLang/julia/issues/11575"},"comments":{"href":"https://api.github.com/repos/JuliaLang/julia/issues/11575/comments"},"review_comments":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/11575/comments"},"review_comment":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/11575/commits"},"statuses":{"href":"https://api.github.com/repos/JuliaLang/julia/statuses/da43b2a5a188d1104c3df7c2594fe4f5441c70d1"}}}}