{"action":"created","comment":{"url":"https://api.github.com/repos/realm/realm-java/pulls/comments/40086619","id":40086619,"diff_hunk":"@@ -271,63 +250,73 @@ string concat_stringdata(const char *message, StringData strData)\n     return std::string(message) + (strData != NULL ? strData.data() : \"\");\n }\n \n-jstring to_jstring(JNIEnv* env, StringData str)\n-{\n-    // For efficiency, if the incoming UTF-8 string is sufficiently\n-    // small, we will attempt to store the UTF-16 output into a stack\n-    // allocated buffer of static size. Otherwise we will have to\n-    // dynamically allocate the output buffer after calculating its\n-    // size.\n+jstring to_jstring(JNIEnv* env, StringData str) {\n+    // Input is UTF-8 and output is UTF-16. Invalid UTF-8 input is silently\n+    // converted to Unicode replacement characters.\n+    // We use a small fixed size stack-allocated output buffer to avoid the cost\n+    // of dynamic allocation for short input strings. If this buffer turns out\n+    // to be too small, we proceed by calculating an estimate for the actually\n+    // required output buffer size, and then allocate the buffer dynamically.\n \n     const size_t stack_buf_size = 48;\n     jchar stack_buf[stack_buf_size];\n-    std::unique_ptr<jchar[]> dyn_buf;\n-\n-    const char* in_begin = str.data();\n-    const char* in_end   = str.data() + str.size();\n-    jchar* out_begin = stack_buf;\n-    jchar* out_curr  = stack_buf;\n-    jchar* out_end   = stack_buf + stack_buf_size;\n-\n-    typedef Utf8x16<jchar, JcharTraits> Xcode;\n-\n-    if (str.size() <= stack_buf_size) {\n-        size_t retcode = Xcode::to_utf16(in_begin, in_end, out_curr, out_end);\n-        if (retcode != 0)\n-            throw runtime_error(string_to_hex(\"Failure when converting short string to UTF-16\",  str, in_begin, in_end, out_curr, out_end, size_t(0), retcode));\n-        if (in_begin == in_end)\n-            goto transcode_complete;\n-    }\n-\n-    {\n-        const char* in_begin2 = in_begin;\n-        size_t error_code;\n-        size_t size = Xcode::find_utf16_buf_size(in_begin2, in_end, error_code);\n-        if (in_begin2 != in_end) \n-            throw runtime_error(string_to_hex(\"Failure when computing UTF-16 size\", str, in_begin, in_end, out_curr, out_end, size, error_code));\n-        if (int_add_with_overflow_detect(size, stack_buf_size))\n-            throw runtime_error(\"String size overflow\");\n-        dyn_buf.reset(new jchar[size]);\n-        out_curr = copy(out_begin, out_curr, dyn_buf.get());\n-        out_begin = dyn_buf.get();\n-        out_end   = dyn_buf.get() + size;\n-        size_t retcode = Xcode::to_utf16(in_begin, in_end, out_curr, out_end);\n-        if (retcode != 0) \n-            throw runtime_error(string_to_hex(\"Failure when converting long string to UTF-16\", str, in_begin, in_end, out_curr, out_end, size_t(0), retcode));\n-        REALM_ASSERT(in_begin == in_end);\n+    Buffer <jchar> dyn_buf;\n+\n+    const char *in = str.data();\n+    const char *in_end = in + str.size();\n+    jchar *out = stack_buf;\n+    jchar *out_begin = out;\n+    jchar *out_end = out_begin + stack_buf_size;\n+\n+    for (; ;) {\n+        typedef Utf8x16 <jchar, JcharTraits> Xcode;\n+        Xcode::to_utf16(in, in_end, out, out_end);\n+        bool end_of_input = in == in_end;\n+        if (end_of_input)\n+            break;\n+        bool bad_input = out != out_end;","path":"realm-jni/src/util.cpp","position":113,"original_position":113,"commit_id":"8daa2262b0a4186c5884c53133cf855b985958e0","original_commit_id":"8daa2262b0a4186c5884c53133cf855b985958e0","user":{"login":"kspangsege","id":180705,"avatar_url":"https://avatars.githubusercontent.com/u/180705?v=3","gravatar_id":"","url":"https://api.github.com/users/kspangsege","html_url":"https://github.com/kspangsege","followers_url":"https://api.github.com/users/kspangsege/followers","following_url":"https://api.github.com/users/kspangsege/following{/other_user}","gists_url":"https://api.github.com/users/kspangsege/gists{/gist_id}","starred_url":"https://api.github.com/users/kspangsege/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kspangsege/subscriptions","organizations_url":"https://api.github.com/users/kspangsege/orgs","repos_url":"https://api.github.com/users/kspangsege/repos","events_url":"https://api.github.com/users/kspangsege/events{/privacy}","received_events_url":"https://api.github.com/users/kspangsege/received_events","type":"User","site_admin":false},"body":"> `out != out_end` does not always mean bad input.\r\n\r\n@zaki50, you are right. That is a bug.\r\n\r\nI am actually working on an improved version of `Xcode::to_utf16()`; one that lets you whether it stopped because of lack of output buffer space. That will make it easier to fix this bug. It will probably be another week before I get time to finish it though.","created_at":"2015-09-22T13:40:03Z","updated_at":"2015-09-22T13:40:03Z","html_url":"https://github.com/realm/realm-java/pull/1410#discussion_r40086619","pull_request_url":"https://api.github.com/repos/realm/realm-java/pulls/1410","_links":{"self":{"href":"https://api.github.com/repos/realm/realm-java/pulls/comments/40086619"},"html":{"href":"https://github.com/realm/realm-java/pull/1410#discussion_r40086619"},"pull_request":{"href":"https://api.github.com/repos/realm/realm-java/pulls/1410"}}},"pull_request":{"url":"https://api.github.com/repos/realm/realm-java/pulls/1410","id":43027342,"html_url":"https://github.com/realm/realm-java/pull/1410","diff_url":"https://github.com/realm/realm-java/pull/1410.diff","patch_url":"https://github.com/realm/realm-java/pull/1410.patch","issue_url":"https://api.github.com/repos/realm/realm-java/issues/1410","number":1410,"state":"open","locked":false,"title":"Reintroducing lenient UTF8 -> UTF16 transcoding ","user":{"login":"kneth","id":1225363,"avatar_url":"https://avatars.githubusercontent.com/u/1225363?v=3","gravatar_id":"","url":"https://api.github.com/users/kneth","html_url":"https://github.com/kneth","followers_url":"https://api.github.com/users/kneth/followers","following_url":"https://api.github.com/users/kneth/following{/other_user}","gists_url":"https://api.github.com/users/kneth/gists{/gist_id}","starred_url":"https://api.github.com/users/kneth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kneth/subscriptions","organizations_url":"https://api.github.com/users/kneth/orgs","repos_url":"https://api.github.com/users/kneth/repos","events_url":"https://api.github.com/users/kneth/events{/privacy}","received_events_url":"https://api.github.com/users/kneth/received_events","type":"User","site_admin":false},"body":"This has been merged before but was removed when we saw many native crashes which were believe to be related to strings back then. The crashes often came from developers ignoring Out-of-Memory, which lead to a Realm in an inconsistent state. In order to track improvements of Realm's underlaying storage engine, I wish to go back to use the methods providing by the storage engine instead of the under-maintained version here.\r\n\r\nThis is based on the work done in #647.\r\n\r\n@realm/java @kspangsege \r\n\r\nTODO:\r\n\r\n- [x] Remove `string_to_hex()` functions","created_at":"2015-08-21T11:14:14Z","updated_at":"2015-09-22T13:40:03Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2fd84112916ace449a70e02bf4e2b425f9b04bd8","assignee":{"login":"kneth","id":1225363,"avatar_url":"https://avatars.githubusercontent.com/u/1225363?v=3","gravatar_id":"","url":"https://api.github.com/users/kneth","html_url":"https://github.com/kneth","followers_url":"https://api.github.com/users/kneth/followers","following_url":"https://api.github.com/users/kneth/following{/other_user}","gists_url":"https://api.github.com/users/kneth/gists{/gist_id}","starred_url":"https://api.github.com/users/kneth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kneth/subscriptions","organizations_url":"https://api.github.com/users/kneth/orgs","repos_url":"https://api.github.com/users/kneth/repos","events_url":"https://api.github.com/users/kneth/events{/privacy}","received_events_url":"https://api.github.com/users/kneth/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/realm/realm-java/pulls/1410/commits","review_comments_url":"https://api.github.com/repos/realm/realm-java/pulls/1410/comments","review_comment_url":"https://api.github.com/repos/realm/realm-java/pulls/comments{/number}","comments_url":"https://api.github.com/repos/realm/realm-java/issues/1410/comments","statuses_url":"https://api.github.com/repos/realm/realm-java/statuses/8daa2262b0a4186c5884c53133cf855b985958e0","head":{"label":"realm:kg-revert-to-core-utf-handling","ref":"kg-revert-to-core-utf-handling","sha":"8daa2262b0a4186c5884c53133cf855b985958e0","user":{"login":"realm","id":7575099,"avatar_url":"https://avatars.githubusercontent.com/u/7575099?v=3","gravatar_id":"","url":"https://api.github.com/users/realm","html_url":"https://github.com/realm","followers_url":"https://api.github.com/users/realm/followers","following_url":"https://api.github.com/users/realm/following{/other_user}","gists_url":"https://api.github.com/users/realm/gists{/gist_id}","starred_url":"https://api.github.com/users/realm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/realm/subscriptions","organizations_url":"https://api.github.com/users/realm/orgs","repos_url":"https://api.github.com/users/realm/repos","events_url":"https://api.github.com/users/realm/events{/privacy}","received_events_url":"https://api.github.com/users/realm/received_events","type":"Organization","site_admin":false},"repo":{"id":4084908,"name":"realm-java","full_name":"realm/realm-java","owner":{"login":"realm","id":7575099,"avatar_url":"https://avatars.githubusercontent.com/u/7575099?v=3","gravatar_id":"","url":"https://api.github.com/users/realm","html_url":"https://github.com/realm","followers_url":"https://api.github.com/users/realm/followers","following_url":"https://api.github.com/users/realm/following{/other_user}","gists_url":"https://api.github.com/users/realm/gists{/gist_id}","starred_url":"https://api.github.com/users/realm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/realm/subscriptions","organizations_url":"https://api.github.com/users/realm/orgs","repos_url":"https://api.github.com/users/realm/repos","events_url":"https://api.github.com/users/realm/events{/privacy}","received_events_url":"https://api.github.com/users/realm/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/realm/realm-java","description":"Realm is a mobile database: a replacement for SQLite & ORMs","fork":false,"url":"https://api.github.com/repos/realm/realm-java","forks_url":"https://api.github.com/repos/realm/realm-java/forks","keys_url":"https://api.github.com/repos/realm/realm-java/keys{/key_id}","collaborators_url":"https://api.github.com/repos/realm/realm-java/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/realm/realm-java/teams","hooks_url":"https://api.github.com/repos/realm/realm-java/hooks","issue_events_url":"https://api.github.com/repos/realm/realm-java/issues/events{/number}","events_url":"https://api.github.com/repos/realm/realm-java/events","assignees_url":"https://api.github.com/repos/realm/realm-java/assignees{/user}","branches_url":"https://api.github.com/repos/realm/realm-java/branches{/branch}","tags_url":"https://api.github.com/repos/realm/realm-java/tags","blobs_url":"https://api.github.com/repos/realm/realm-java/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/realm/realm-java/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/realm/realm-java/git/refs{/sha}","trees_url":"https://api.github.com/repos/realm/realm-java/git/trees{/sha}","statuses_url":"https://api.github.com/repos/realm/realm-java/statuses/{sha}","languages_url":"https://api.github.com/repos/realm/realm-java/languages","stargazers_url":"https://api.github.com/repos/realm/realm-java/stargazers","contributors_url":"https://api.github.com/repos/realm/realm-java/contributors","subscribers_url":"https://api.github.com/repos/realm/realm-java/subscribers","subscription_url":"https://api.github.com/repos/realm/realm-java/subscription","commits_url":"https://api.github.com/repos/realm/realm-java/commits{/sha}","git_commits_url":"https://api.github.com/repos/realm/realm-java/git/commits{/sha}","comments_url":"https://api.github.com/repos/realm/realm-java/comments{/number}","issue_comment_url":"https://api.github.com/repos/realm/realm-java/issues/comments{/number}","contents_url":"https://api.github.com/repos/realm/realm-java/contents/{+path}","compare_url":"https://api.github.com/repos/realm/realm-java/compare/{base}...{head}","merges_url":"https://api.github.com/repos/realm/realm-java/merges","archive_url":"https://api.github.com/repos/realm/realm-java/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/realm/realm-java/downloads","issues_url":"https://api.github.com/repos/realm/realm-java/issues{/number}","pulls_url":"https://api.github.com/repos/realm/realm-java/pulls{/number}","milestones_url":"https://api.github.com/repos/realm/realm-java/milestones{/number}","notifications_url":"https://api.github.com/repos/realm/realm-java/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/realm/realm-java/labels{/name}","releases_url":"https://api.github.com/repos/realm/realm-java/releases{/id}","created_at":"2012-04-20T09:18:42Z","updated_at":"2015-09-22T13:35:05Z","pushed_at":"2015-09-22T13:33:38Z","git_url":"git://github.com/realm/realm-java.git","ssh_url":"git@github.com:realm/realm-java.git","clone_url":"https://github.com/realm/realm-java.git","svn_url":"https://github.com/realm/realm-java","homepage":"http://realm.io","size":57495,"stargazers_count":2506,"watchers_count":2506,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":253,"mirror_url":null,"open_issues_count":157,"forks":253,"open_issues":157,"watchers":2506,"default_branch":"master"}},"base":{"label":"realm:master","ref":"master","sha":"b581855e2033709797852e4058c9373b1e7bbb47","user":{"login":"realm","id":7575099,"avatar_url":"https://avatars.githubusercontent.com/u/7575099?v=3","gravatar_id":"","url":"https://api.github.com/users/realm","html_url":"https://github.com/realm","followers_url":"https://api.github.com/users/realm/followers","following_url":"https://api.github.com/users/realm/following{/other_user}","gists_url":"https://api.github.com/users/realm/gists{/gist_id}","starred_url":"https://api.github.com/users/realm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/realm/subscriptions","organizations_url":"https://api.github.com/users/realm/orgs","repos_url":"https://api.github.com/users/realm/repos","events_url":"https://api.github.com/users/realm/events{/privacy}","received_events_url":"https://api.github.com/users/realm/received_events","type":"Organization","site_admin":false},"repo":{"id":4084908,"name":"realm-java","full_name":"realm/realm-java","owner":{"login":"realm","id":7575099,"avatar_url":"https://avatars.githubusercontent.com/u/7575099?v=3","gravatar_id":"","url":"https://api.github.com/users/realm","html_url":"https://github.com/realm","followers_url":"https://api.github.com/users/realm/followers","following_url":"https://api.github.com/users/realm/following{/other_user}","gists_url":"https://api.github.com/users/realm/gists{/gist_id}","starred_url":"https://api.github.com/users/realm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/realm/subscriptions","organizations_url":"https://api.github.com/users/realm/orgs","repos_url":"https://api.github.com/users/realm/repos","events_url":"https://api.github.com/users/realm/events{/privacy}","received_events_url":"https://api.github.com/users/realm/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/realm/realm-java","description":"Realm is a mobile database: a replacement for SQLite & ORMs","fork":false,"url":"https://api.github.com/repos/realm/realm-java","forks_url":"https://api.github.com/repos/realm/realm-java/forks","keys_url":"https://api.github.com/repos/realm/realm-java/keys{/key_id}","collaborators_url":"https://api.github.com/repos/realm/realm-java/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/realm/realm-java/teams","hooks_url":"https://api.github.com/repos/realm/realm-java/hooks","issue_events_url":"https://api.github.com/repos/realm/realm-java/issues/events{/number}","events_url":"https://api.github.com/repos/realm/realm-java/events","assignees_url":"https://api.github.com/repos/realm/realm-java/assignees{/user}","branches_url":"https://api.github.com/repos/realm/realm-java/branches{/branch}","tags_url":"https://api.github.com/repos/realm/realm-java/tags","blobs_url":"https://api.github.com/repos/realm/realm-java/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/realm/realm-java/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/realm/realm-java/git/refs{/sha}","trees_url":"https://api.github.com/repos/realm/realm-java/git/trees{/sha}","statuses_url":"https://api.github.com/repos/realm/realm-java/statuses/{sha}","languages_url":"https://api.github.com/repos/realm/realm-java/languages","stargazers_url":"https://api.github.com/repos/realm/realm-java/stargazers","contributors_url":"https://api.github.com/repos/realm/realm-java/contributors","subscribers_url":"https://api.github.com/repos/realm/realm-java/subscribers","subscription_url":"https://api.github.com/repos/realm/realm-java/subscription","commits_url":"https://api.github.com/repos/realm/realm-java/commits{/sha}","git_commits_url":"https://api.github.com/repos/realm/realm-java/git/commits{/sha}","comments_url":"https://api.github.com/repos/realm/realm-java/comments{/number}","issue_comment_url":"https://api.github.com/repos/realm/realm-java/issues/comments{/number}","contents_url":"https://api.github.com/repos/realm/realm-java/contents/{+path}","compare_url":"https://api.github.com/repos/realm/realm-java/compare/{base}...{head}","merges_url":"https://api.github.com/repos/realm/realm-java/merges","archive_url":"https://api.github.com/repos/realm/realm-java/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/realm/realm-java/downloads","issues_url":"https://api.github.com/repos/realm/realm-java/issues{/number}","pulls_url":"https://api.github.com/repos/realm/realm-java/pulls{/number}","milestones_url":"https://api.github.com/repos/realm/realm-java/milestones{/number}","notifications_url":"https://api.github.com/repos/realm/realm-java/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/realm/realm-java/labels{/name}","releases_url":"https://api.github.com/repos/realm/realm-java/releases{/id}","created_at":"2012-04-20T09:18:42Z","updated_at":"2015-09-22T13:35:05Z","pushed_at":"2015-09-22T13:33:38Z","git_url":"git://github.com/realm/realm-java.git","ssh_url":"git@github.com:realm/realm-java.git","clone_url":"https://github.com/realm/realm-java.git","svn_url":"https://github.com/realm/realm-java","homepage":"http://realm.io","size":57495,"stargazers_count":2506,"watchers_count":2506,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":253,"mirror_url":null,"open_issues_count":157,"forks":253,"open_issues":157,"watchers":2506,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/realm/realm-java/pulls/1410"},"html":{"href":"https://github.com/realm/realm-java/pull/1410"},"issue":{"href":"https://api.github.com/repos/realm/realm-java/issues/1410"},"comments":{"href":"https://api.github.com/repos/realm/realm-java/issues/1410/comments"},"review_comments":{"href":"https://api.github.com/repos/realm/realm-java/pulls/1410/comments"},"review_comment":{"href":"https://api.github.com/repos/realm/realm-java/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/realm/realm-java/pulls/1410/commits"},"statuses":{"href":"https://api.github.com/repos/realm/realm-java/statuses/8daa2262b0a4186c5884c53133cf855b985958e0"}}}}