{"action":"created","comment":{"url":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/comments/29752475","id":29752475,"diff_hunk":"@@ -121,40 +138,143 @@ lz4_decompress(WT_COMPRESSOR *compressor, WT_SESSION *session,\n     size_t *result_lenp)\n {\n \tWT_EXTENSION_API *wt_api;\n-\tchar *compressed_data;\n+\tLZ4_PREFIX prefix;\n \tint decoded;\n-\tsize_t src_data_len;\n+\tuint8_t *dst_tmp;\n+\n+\t(void)src_len;\t\t\t\t\t/* Unused parameters */\n \n \twt_api = ((LZ4_COMPRESSOR *)compressor)->wt_api;\n \n-\t/* Retrieve compressed length from start of the data buffer. */\n-\tsrc_data_len = *(size_t *)src;\n-\tif (src_data_len + sizeof(size_t) > src_len) {\n-\t\t(void)wt_api->err_printf(wt_api,\n-\t\t    session,\n-\t\t    \"lz4_decompress: stored size exceeds buffer size\");\n-\t\treturn (WT_ERROR);\n+\t/*\n+\t * Retrieve the true length of the compressed block and source and the\n+\t * decompressed bytes to return from the start of the source buffer.\n+\t */\n+\tmemcpy(&prefix, src, sizeof(LZ4_PREFIX));\n+\n+\t/*\n+\t * Decompress, starting after the prefix bytes. Use safe decompression:\n+\t * we rely on decompression to detect corruption.\n+\t *\n+\t * Two code paths, one with and one without a bounce buffer. When doing\n+\t * raw compression, we compress to a target size irrespective of row\n+\t * boundaries, and return to our caller a \"useful\" compression length\n+\t * based on the last complete row that was compressed. Our caller stores\n+\t * that length, not the length of bytes actually compressed by LZ4. In\n+\t * other words, our caller doesn't know how many bytes will result from\n+\t * decompression, likely hasn't provided us a large enough buffer, and\n+\t * we have to allocate a scratch buffer.\n+\t */","path":"ext/compressors/lz4/lz4_compress.c","position":143,"original_position":143,"commit_id":"4f9c5ec385580c1a7d8762d240672e27dda1baf9","original_commit_id":"4f9c5ec385580c1a7d8762d240672e27dda1baf9","user":{"login":"keithbostic","id":1393186,"avatar_url":"https://avatars.githubusercontent.com/u/1393186?v=3","gravatar_id":"","url":"https://api.github.com/users/keithbostic","html_url":"https://github.com/keithbostic","followers_url":"https://api.github.com/users/keithbostic/followers","following_url":"https://api.github.com/users/keithbostic/following{/other_user}","gists_url":"https://api.github.com/users/keithbostic/gists{/gist_id}","starred_url":"https://api.github.com/users/keithbostic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keithbostic/subscriptions","organizations_url":"https://api.github.com/users/keithbostic/orgs","repos_url":"https://api.github.com/users/keithbostic/repos","events_url":"https://api.github.com/users/keithbostic/events{/privacy}","received_events_url":"https://api.github.com/users/keithbostic/received_events","type":"User","site_admin":false},"body":"@Cyan4973, an additional issue I wanted to ask you about.\r\n\r\nAs you know, we currently have 3 values we're maintaining:\r\n* the length of \"useful\" uncompressed bytes we want from the compressed chunk,\r\n* the length of the compressed chunk,\r\n* the length of the uncompressed chunk.\r\n\r\nBecause we don't know the number of compressed bytes associated with the useful uncompressed bytes we want, we have to give `LZ4_decompress_safe` a buffer big enough to hold the complete uncompressed chunk, which, for layering reasons, is difficult for us: we're discussing alternatives, but at the moment we're allocating a scratch buffer and doing a memory copy, which will slow things down.\r\n\r\n@michaelcahill, raised a point you and I briefly touched on: right now, `LZ4_decompress_safe ` returns an error if the number of compressed bytes is non-zero when we've run out of buffer into which we're decompressing.\r\n\r\nWould it be possible for `LZ4_decompress_safe` to allow that usage?\r\n\r\nIn other words, is there some way to tell `LZ4_decompress_safe` we only want some number of uncompressed bytes from the compressed source, and if we fill the buffer before the number of compressed source bytes goes to 0, that's OK?\r\n\r\nMight this be useful to other applications, too? I could imagine an application that only needs the start of a compressed block.\r\n\r\nAlternatively, if there's a specific error guaranteed to be returned in this case, we could treat that error as success?","created_at":"2015-05-06T12:21:47Z","updated_at":"2015-05-06T12:21:47Z","html_url":"https://github.com/wiredtiger/wiredtiger/pull/1928#discussion_r29752475","pull_request_url":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/1928","_links":{"self":{"href":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/comments/29752475"},"html":{"href":"https://github.com/wiredtiger/wiredtiger/pull/1928#discussion_r29752475"},"pull_request":{"href":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/1928"}}},"pull_request":{"url":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/1928","id":34603627,"html_url":"https://github.com/wiredtiger/wiredtiger/pull/1928","diff_url":"https://github.com/wiredtiger/wiredtiger/pull/1928.diff","patch_url":"https://github.com/wiredtiger/wiredtiger/pull/1928.patch","issue_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/1928","number":1928,"state":"open","locked":false,"title":"LZ4 raw compression","user":{"login":"keithbostic","id":1393186,"avatar_url":"https://avatars.githubusercontent.com/u/1393186?v=3","gravatar_id":"","url":"https://api.github.com/users/keithbostic","html_url":"https://github.com/keithbostic","followers_url":"https://api.github.com/users/keithbostic/followers","following_url":"https://api.github.com/users/keithbostic/following{/other_user}","gists_url":"https://api.github.com/users/keithbostic/gists{/gist_id}","starred_url":"https://api.github.com/users/keithbostic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keithbostic/subscriptions","organizations_url":"https://api.github.com/users/keithbostic/orgs","repos_url":"https://api.github.com/users/keithbostic/repos","events_url":"https://api.github.com/users/keithbostic/events{/privacy}","received_events_url":"https://api.github.com/users/keithbostic/received_events","type":"User","site_admin":false},"body":"Reference WT-1773.","created_at":"2015-05-03T16:28:20Z","updated_at":"2015-05-06T12:21:47Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a0e5f005d86d484c4d024a5831cdf4ddd1d059cc","assignee":{"login":"keithbostic","id":1393186,"avatar_url":"https://avatars.githubusercontent.com/u/1393186?v=3","gravatar_id":"","url":"https://api.github.com/users/keithbostic","html_url":"https://github.com/keithbostic","followers_url":"https://api.github.com/users/keithbostic/followers","following_url":"https://api.github.com/users/keithbostic/following{/other_user}","gists_url":"https://api.github.com/users/keithbostic/gists{/gist_id}","starred_url":"https://api.github.com/users/keithbostic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keithbostic/subscriptions","organizations_url":"https://api.github.com/users/keithbostic/orgs","repos_url":"https://api.github.com/users/keithbostic/repos","events_url":"https://api.github.com/users/keithbostic/events{/privacy}","received_events_url":"https://api.github.com/users/keithbostic/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/1928/commits","review_comments_url":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/1928/comments","review_comment_url":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/comments{/number}","comments_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/1928/comments","statuses_url":"https://api.github.com/repos/wiredtiger/wiredtiger/statuses/4f9c5ec385580c1a7d8762d240672e27dda1baf9","head":{"label":"wiredtiger:lz4-compress-destsize","ref":"lz4-compress-destsize","sha":"4f9c5ec385580c1a7d8762d240672e27dda1baf9","user":{"login":"wiredtiger","id":1251095,"avatar_url":"https://avatars.githubusercontent.com/u/1251095?v=3","gravatar_id":"","url":"https://api.github.com/users/wiredtiger","html_url":"https://github.com/wiredtiger","followers_url":"https://api.github.com/users/wiredtiger/followers","following_url":"https://api.github.com/users/wiredtiger/following{/other_user}","gists_url":"https://api.github.com/users/wiredtiger/gists{/gist_id}","starred_url":"https://api.github.com/users/wiredtiger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wiredtiger/subscriptions","organizations_url":"https://api.github.com/users/wiredtiger/orgs","repos_url":"https://api.github.com/users/wiredtiger/repos","events_url":"https://api.github.com/users/wiredtiger/events{/privacy}","received_events_url":"https://api.github.com/users/wiredtiger/received_events","type":"Organization","site_admin":false},"repo":{"id":2944302,"name":"wiredtiger","full_name":"wiredtiger/wiredtiger","owner":{"login":"wiredtiger","id":1251095,"avatar_url":"https://avatars.githubusercontent.com/u/1251095?v=3","gravatar_id":"","url":"https://api.github.com/users/wiredtiger","html_url":"https://github.com/wiredtiger","followers_url":"https://api.github.com/users/wiredtiger/followers","following_url":"https://api.github.com/users/wiredtiger/following{/other_user}","gists_url":"https://api.github.com/users/wiredtiger/gists{/gist_id}","starred_url":"https://api.github.com/users/wiredtiger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wiredtiger/subscriptions","organizations_url":"https://api.github.com/users/wiredtiger/orgs","repos_url":"https://api.github.com/users/wiredtiger/repos","events_url":"https://api.github.com/users/wiredtiger/events{/privacy}","received_events_url":"https://api.github.com/users/wiredtiger/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/wiredtiger/wiredtiger","description":"WiredTiger's source tree","fork":false,"url":"https://api.github.com/repos/wiredtiger/wiredtiger","forks_url":"https://api.github.com/repos/wiredtiger/wiredtiger/forks","keys_url":"https://api.github.com/repos/wiredtiger/wiredtiger/keys{/key_id}","collaborators_url":"https://api.github.com/repos/wiredtiger/wiredtiger/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/wiredtiger/wiredtiger/teams","hooks_url":"https://api.github.com/repos/wiredtiger/wiredtiger/hooks","issue_events_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/events{/number}","events_url":"https://api.github.com/repos/wiredtiger/wiredtiger/events","assignees_url":"https://api.github.com/repos/wiredtiger/wiredtiger/assignees{/user}","branches_url":"https://api.github.com/repos/wiredtiger/wiredtiger/branches{/branch}","tags_url":"https://api.github.com/repos/wiredtiger/wiredtiger/tags","blobs_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/refs{/sha}","trees_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/trees{/sha}","statuses_url":"https://api.github.com/repos/wiredtiger/wiredtiger/statuses/{sha}","languages_url":"https://api.github.com/repos/wiredtiger/wiredtiger/languages","stargazers_url":"https://api.github.com/repos/wiredtiger/wiredtiger/stargazers","contributors_url":"https://api.github.com/repos/wiredtiger/wiredtiger/contributors","subscribers_url":"https://api.github.com/repos/wiredtiger/wiredtiger/subscribers","subscription_url":"https://api.github.com/repos/wiredtiger/wiredtiger/subscription","commits_url":"https://api.github.com/repos/wiredtiger/wiredtiger/commits{/sha}","git_commits_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/commits{/sha}","comments_url":"https://api.github.com/repos/wiredtiger/wiredtiger/comments{/number}","issue_comment_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/comments{/number}","contents_url":"https://api.github.com/repos/wiredtiger/wiredtiger/contents/{+path}","compare_url":"https://api.github.com/repos/wiredtiger/wiredtiger/compare/{base}...{head}","merges_url":"https://api.github.com/repos/wiredtiger/wiredtiger/merges","archive_url":"https://api.github.com/repos/wiredtiger/wiredtiger/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/wiredtiger/wiredtiger/downloads","issues_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues{/number}","pulls_url":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls{/number}","milestones_url":"https://api.github.com/repos/wiredtiger/wiredtiger/milestones{/number}","notifications_url":"https://api.github.com/repos/wiredtiger/wiredtiger/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/wiredtiger/wiredtiger/labels{/name}","releases_url":"https://api.github.com/repos/wiredtiger/wiredtiger/releases{/id}","created_at":"2011-12-09T00:43:19Z","updated_at":"2015-05-06T12:19:59Z","pushed_at":"2015-05-06T11:13:47Z","git_url":"git://github.com/wiredtiger/wiredtiger.git","ssh_url":"git@github.com:wiredtiger/wiredtiger.git","clone_url":"https://github.com/wiredtiger/wiredtiger.git","svn_url":"https://github.com/wiredtiger/wiredtiger","homepage":"http://source.wiredtiger.com/","size":107491,"stargazers_count":476,"watchers_count":476,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":77,"mirror_url":null,"open_issues_count":10,"forks":77,"open_issues":10,"watchers":476,"default_branch":"master"}},"base":{"label":"wiredtiger:develop","ref":"develop","sha":"d85cefe03057e253c67444eb56e9725c3bf7a685","user":{"login":"wiredtiger","id":1251095,"avatar_url":"https://avatars.githubusercontent.com/u/1251095?v=3","gravatar_id":"","url":"https://api.github.com/users/wiredtiger","html_url":"https://github.com/wiredtiger","followers_url":"https://api.github.com/users/wiredtiger/followers","following_url":"https://api.github.com/users/wiredtiger/following{/other_user}","gists_url":"https://api.github.com/users/wiredtiger/gists{/gist_id}","starred_url":"https://api.github.com/users/wiredtiger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wiredtiger/subscriptions","organizations_url":"https://api.github.com/users/wiredtiger/orgs","repos_url":"https://api.github.com/users/wiredtiger/repos","events_url":"https://api.github.com/users/wiredtiger/events{/privacy}","received_events_url":"https://api.github.com/users/wiredtiger/received_events","type":"Organization","site_admin":false},"repo":{"id":2944302,"name":"wiredtiger","full_name":"wiredtiger/wiredtiger","owner":{"login":"wiredtiger","id":1251095,"avatar_url":"https://avatars.githubusercontent.com/u/1251095?v=3","gravatar_id":"","url":"https://api.github.com/users/wiredtiger","html_url":"https://github.com/wiredtiger","followers_url":"https://api.github.com/users/wiredtiger/followers","following_url":"https://api.github.com/users/wiredtiger/following{/other_user}","gists_url":"https://api.github.com/users/wiredtiger/gists{/gist_id}","starred_url":"https://api.github.com/users/wiredtiger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wiredtiger/subscriptions","organizations_url":"https://api.github.com/users/wiredtiger/orgs","repos_url":"https://api.github.com/users/wiredtiger/repos","events_url":"https://api.github.com/users/wiredtiger/events{/privacy}","received_events_url":"https://api.github.com/users/wiredtiger/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/wiredtiger/wiredtiger","description":"WiredTiger's source tree","fork":false,"url":"https://api.github.com/repos/wiredtiger/wiredtiger","forks_url":"https://api.github.com/repos/wiredtiger/wiredtiger/forks","keys_url":"https://api.github.com/repos/wiredtiger/wiredtiger/keys{/key_id}","collaborators_url":"https://api.github.com/repos/wiredtiger/wiredtiger/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/wiredtiger/wiredtiger/teams","hooks_url":"https://api.github.com/repos/wiredtiger/wiredtiger/hooks","issue_events_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/events{/number}","events_url":"https://api.github.com/repos/wiredtiger/wiredtiger/events","assignees_url":"https://api.github.com/repos/wiredtiger/wiredtiger/assignees{/user}","branches_url":"https://api.github.com/repos/wiredtiger/wiredtiger/branches{/branch}","tags_url":"https://api.github.com/repos/wiredtiger/wiredtiger/tags","blobs_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/refs{/sha}","trees_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/trees{/sha}","statuses_url":"https://api.github.com/repos/wiredtiger/wiredtiger/statuses/{sha}","languages_url":"https://api.github.com/repos/wiredtiger/wiredtiger/languages","stargazers_url":"https://api.github.com/repos/wiredtiger/wiredtiger/stargazers","contributors_url":"https://api.github.com/repos/wiredtiger/wiredtiger/contributors","subscribers_url":"https://api.github.com/repos/wiredtiger/wiredtiger/subscribers","subscription_url":"https://api.github.com/repos/wiredtiger/wiredtiger/subscription","commits_url":"https://api.github.com/repos/wiredtiger/wiredtiger/commits{/sha}","git_commits_url":"https://api.github.com/repos/wiredtiger/wiredtiger/git/commits{/sha}","comments_url":"https://api.github.com/repos/wiredtiger/wiredtiger/comments{/number}","issue_comment_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/comments{/number}","contents_url":"https://api.github.com/repos/wiredtiger/wiredtiger/contents/{+path}","compare_url":"https://api.github.com/repos/wiredtiger/wiredtiger/compare/{base}...{head}","merges_url":"https://api.github.com/repos/wiredtiger/wiredtiger/merges","archive_url":"https://api.github.com/repos/wiredtiger/wiredtiger/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/wiredtiger/wiredtiger/downloads","issues_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues{/number}","pulls_url":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls{/number}","milestones_url":"https://api.github.com/repos/wiredtiger/wiredtiger/milestones{/number}","notifications_url":"https://api.github.com/repos/wiredtiger/wiredtiger/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/wiredtiger/wiredtiger/labels{/name}","releases_url":"https://api.github.com/repos/wiredtiger/wiredtiger/releases{/id}","created_at":"2011-12-09T00:43:19Z","updated_at":"2015-05-06T12:19:59Z","pushed_at":"2015-05-06T11:13:47Z","git_url":"git://github.com/wiredtiger/wiredtiger.git","ssh_url":"git@github.com:wiredtiger/wiredtiger.git","clone_url":"https://github.com/wiredtiger/wiredtiger.git","svn_url":"https://github.com/wiredtiger/wiredtiger","homepage":"http://source.wiredtiger.com/","size":107491,"stargazers_count":476,"watchers_count":476,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":77,"mirror_url":null,"open_issues_count":10,"forks":77,"open_issues":10,"watchers":476,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/1928"},"html":{"href":"https://github.com/wiredtiger/wiredtiger/pull/1928"},"issue":{"href":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/1928"},"comments":{"href":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/1928/comments"},"review_comments":{"href":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/1928/comments"},"review_comment":{"href":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/wiredtiger/wiredtiger/pulls/1928/commits"},"statuses":{"href":"https://api.github.com/repos/wiredtiger/wiredtiger/statuses/4f9c5ec385580c1a7d8762d240672e27dda1baf9"}}}}