{"action":"opened","number":19,"pull_request":{"url":"https://api.github.com/repos/golang/snappy/pulls/19","id":49690909,"html_url":"https://github.com/golang/snappy/pull/19","diff_url":"https://github.com/golang/snappy/pull/19.diff","patch_url":"https://github.com/golang/snappy/pull/19.patch","issue_url":"https://api.github.com/repos/golang/snappy/issues/19","number":19,"state":"open","locked":false,"title":"Fast skipping on uncompressible data","user":{"login":"klauspost","id":5663952,"avatar_url":"https://avatars.githubusercontent.com/u/5663952?v=3","gravatar_id":"","url":"https://api.github.com/users/klauspost","html_url":"https://github.com/klauspost","followers_url":"https://api.github.com/users/klauspost/followers","following_url":"https://api.github.com/users/klauspost/following{/other_user}","gists_url":"https://api.github.com/users/klauspost/gists{/gist_id}","starred_url":"https://api.github.com/users/klauspost/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/klauspost/subscriptions","organizations_url":"https://api.github.com/users/klauspost/orgs","repos_url":"https://api.github.com/users/klauspost/repos","events_url":"https://api.github.com/users/klauspost/events{/privacy}","received_events_url":"https://api.github.com/users/klauspost/received_events","type":"User","site_admin":false},"body":"This simple change will allow the encoder to skip checks if there are many consecutive match misses. This gives about 3% performance decrease on ordinary, and in some cases up to 1% compression loss. However, the speedup on uncompressible data is >20x.\r\n\r\nConsidering the typical use case of Snappy, I think this is a reasonable tradeoff.\r\n\r\nI have added an additional \"random data\" test, as well as a benchmark that print the compression ratio.\r\n\r\nBenchmark and size comparison:\r\n\r\nbenchmark               old ns/op     new ns/op     delta\r\nBenchmark_ZFlat0-4      386453        395917        +2.45%\r\nBenchmark_ZFlat1-4      5243380       5430495       +3.57%\r\nBenchmark_ZFlat2-4      1219780       41927         -96.56%\r\nBenchmark_ZFlat3-4      1219781       41859         -96.57%\r\nBenchmark_ZFlat4-4      876567        323408        -63.11%\r\nBenchmark_ZFlat5-4      1511665       1558024       +3.07%\r\nBenchmark_ZFlat6-4      1297856       1317876       +1.54%\r\nBenchmark_ZFlat7-4      1134592       1161773       +2.40%\r\nBenchmark_ZFlat8-4      3508325       3570376       +1.77%\r\nBenchmark_ZFlat9-4      4529655       4626386       +2.14%\r\nBenchmark_ZFlat10-4     392659        403983        +2.88%\r\nBenchmark_ZFlat11-4     1060197       1071549       +1.07%\r\nBenchmark_ZFlat12-4     10637050      226674        -97.87%\r\n\r\nbenchmark               old MB/s     new MB/s     speedup\r\nBenchmark_ZFlat0-4      264.97       258.64       0.98x\r\nBenchmark_ZFlat1-4      133.90       129.29       0.97x\r\nBenchmark_ZFlat2-4      100.91       2935.84      29.09x\r\nBenchmark_ZFlat3-4      100.91       2940.62      29.14x\r\nBenchmark_ZFlat4-4      116.82       316.63       2.71x\r\nBenchmark_ZFlat5-4      270.96       262.90       0.97x\r\nBenchmark_ZFlat6-4      117.18       115.40       0.98x\r\nBenchmark_ZFlat7-4      110.33       107.75       0.98x\r\nBenchmark_ZFlat8-4      121.64       119.53       0.98x\r\nBenchmark_ZFlat9-4      106.38       104.15       0.98x\r\nBenchmark_ZFlat10-4     302.01       293.55       0.97x\r\nBenchmark_ZFlat11-4     173.85       172.01       0.99x\r\nBenchmark_ZFlat12-4     98.58        4625.91      46.93x\r\n\r\nThis is the compression loss - percentage added to compressed size:\r\nDataset\tLoss %\r\nhtml\t0.04%\r\nurls\t0.07%\r\njpg\t0.00%\r\njpg_200\t0.00%\r\npdf\t0.74%\r\nhtml4\t0.02%\r\ntxt1\t-0.06%\r\ntxt2\t0.16%\r\ntxt3\t0.04%\r\ntxt4\t0.15%\r\npb\t0.13%\r\ngaviota\t0.00%\r\nrandom\t0.00%\r\n\r\n[Sheet of compression loss data](https://docs.google.com/spreadsheets/d/1DUhTU2Pbf-RZCkvz7JLghQ5o4qwKqpHNj0uhPld2I1Y/edit?usp=sharing)","created_at":"2015-11-04T13:56:56Z","updated_at":"2015-11-04T13:56:56Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/golang/snappy/pulls/19/commits","review_comments_url":"https://api.github.com/repos/golang/snappy/pulls/19/comments","review_comment_url":"https://api.github.com/repos/golang/snappy/pulls/comments{/number}","comments_url":"https://api.github.com/repos/golang/snappy/issues/19/comments","statuses_url":"https://api.github.com/repos/golang/snappy/statuses/9353357959f43c3d0dc9bc3120d50d3ef5605ee1","head":{"label":"klauspost:fast-skip-misses","ref":"fast-skip-misses","sha":"9353357959f43c3d0dc9bc3120d50d3ef5605ee1","user":{"login":"klauspost","id":5663952,"avatar_url":"https://avatars.githubusercontent.com/u/5663952?v=3","gravatar_id":"","url":"https://api.github.com/users/klauspost","html_url":"https://github.com/klauspost","followers_url":"https://api.github.com/users/klauspost/followers","following_url":"https://api.github.com/users/klauspost/following{/other_user}","gists_url":"https://api.github.com/users/klauspost/gists{/gist_id}","starred_url":"https://api.github.com/users/klauspost/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/klauspost/subscriptions","organizations_url":"https://api.github.com/users/klauspost/orgs","repos_url":"https://api.github.com/users/klauspost/repos","events_url":"https://api.github.com/users/klauspost/events{/privacy}","received_events_url":"https://api.github.com/users/klauspost/received_events","type":"User","site_admin":false},"repo":{"id":45537599,"name":"snappy","full_name":"klauspost/snappy","owner":{"login":"klauspost","id":5663952,"avatar_url":"https://avatars.githubusercontent.com/u/5663952?v=3","gravatar_id":"","url":"https://api.github.com/users/klauspost","html_url":"https://github.com/klauspost","followers_url":"https://api.github.com/users/klauspost/followers","following_url":"https://api.github.com/users/klauspost/following{/other_user}","gists_url":"https://api.github.com/users/klauspost/gists{/gist_id}","starred_url":"https://api.github.com/users/klauspost/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/klauspost/subscriptions","organizations_url":"https://api.github.com/users/klauspost/orgs","repos_url":"https://api.github.com/users/klauspost/repos","events_url":"https://api.github.com/users/klauspost/events{/privacy}","received_events_url":"https://api.github.com/users/klauspost/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/klauspost/snappy","description":"The Snappy compression format in the Go programming language.","fork":true,"url":"https://api.github.com/repos/klauspost/snappy","forks_url":"https://api.github.com/repos/klauspost/snappy/forks","keys_url":"https://api.github.com/repos/klauspost/snappy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/klauspost/snappy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/klauspost/snappy/teams","hooks_url":"https://api.github.com/repos/klauspost/snappy/hooks","issue_events_url":"https://api.github.com/repos/klauspost/snappy/issues/events{/number}","events_url":"https://api.github.com/repos/klauspost/snappy/events","assignees_url":"https://api.github.com/repos/klauspost/snappy/assignees{/user}","branches_url":"https://api.github.com/repos/klauspost/snappy/branches{/branch}","tags_url":"https://api.github.com/repos/klauspost/snappy/tags","blobs_url":"https://api.github.com/repos/klauspost/snappy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/klauspost/snappy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/klauspost/snappy/git/refs{/sha}","trees_url":"https://api.github.com/repos/klauspost/snappy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/klauspost/snappy/statuses/{sha}","languages_url":"https://api.github.com/repos/klauspost/snappy/languages","stargazers_url":"https://api.github.com/repos/klauspost/snappy/stargazers","contributors_url":"https://api.github.com/repos/klauspost/snappy/contributors","subscribers_url":"https://api.github.com/repos/klauspost/snappy/subscribers","subscription_url":"https://api.github.com/repos/klauspost/snappy/subscription","commits_url":"https://api.github.com/repos/klauspost/snappy/commits{/sha}","git_commits_url":"https://api.github.com/repos/klauspost/snappy/git/commits{/sha}","comments_url":"https://api.github.com/repos/klauspost/snappy/comments{/number}","issue_comment_url":"https://api.github.com/repos/klauspost/snappy/issues/comments{/number}","contents_url":"https://api.github.com/repos/klauspost/snappy/contents/{+path}","compare_url":"https://api.github.com/repos/klauspost/snappy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/klauspost/snappy/merges","archive_url":"https://api.github.com/repos/klauspost/snappy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/klauspost/snappy/downloads","issues_url":"https://api.github.com/repos/klauspost/snappy/issues{/number}","pulls_url":"https://api.github.com/repos/klauspost/snappy/pulls{/number}","milestones_url":"https://api.github.com/repos/klauspost/snappy/milestones{/number}","notifications_url":"https://api.github.com/repos/klauspost/snappy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/klauspost/snappy/labels{/name}","releases_url":"https://api.github.com/repos/klauspost/snappy/releases{/id}","created_at":"2015-11-04T12:29:40Z","updated_at":"2015-11-04T12:29:41Z","pushed_at":"2015-11-04T13:54:09Z","git_url":"git://github.com/klauspost/snappy.git","ssh_url":"git@github.com:klauspost/snappy.git","clone_url":"https://github.com/klauspost/snappy.git","svn_url":"https://github.com/klauspost/snappy","homepage":"","size":119,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":false,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"golang:master","ref":"master","sha":"723cc1e459b8eea2dea4583200fd60757d40097a","user":{"login":"golang","id":4314092,"avatar_url":"https://avatars.githubusercontent.com/u/4314092?v=3","gravatar_id":"","url":"https://api.github.com/users/golang","html_url":"https://github.com/golang","followers_url":"https://api.github.com/users/golang/followers","following_url":"https://api.github.com/users/golang/following{/other_user}","gists_url":"https://api.github.com/users/golang/gists{/gist_id}","starred_url":"https://api.github.com/users/golang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/golang/subscriptions","organizations_url":"https://api.github.com/users/golang/orgs","repos_url":"https://api.github.com/users/golang/repos","events_url":"https://api.github.com/users/golang/events{/privacy}","received_events_url":"https://api.github.com/users/golang/received_events","type":"Organization","site_admin":false},"repo":{"id":35516397,"name":"snappy","full_name":"golang/snappy","owner":{"login":"golang","id":4314092,"avatar_url":"https://avatars.githubusercontent.com/u/4314092?v=3","gravatar_id":"","url":"https://api.github.com/users/golang","html_url":"https://github.com/golang","followers_url":"https://api.github.com/users/golang/followers","following_url":"https://api.github.com/users/golang/following{/other_user}","gists_url":"https://api.github.com/users/golang/gists{/gist_id}","starred_url":"https://api.github.com/users/golang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/golang/subscriptions","organizations_url":"https://api.github.com/users/golang/orgs","repos_url":"https://api.github.com/users/golang/repos","events_url":"https://api.github.com/users/golang/events{/privacy}","received_events_url":"https://api.github.com/users/golang/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/golang/snappy","description":"The Snappy compression format in the Go programming language.","fork":false,"url":"https://api.github.com/repos/golang/snappy","forks_url":"https://api.github.com/repos/golang/snappy/forks","keys_url":"https://api.github.com/repos/golang/snappy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/golang/snappy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/golang/snappy/teams","hooks_url":"https://api.github.com/repos/golang/snappy/hooks","issue_events_url":"https://api.github.com/repos/golang/snappy/issues/events{/number}","events_url":"https://api.github.com/repos/golang/snappy/events","assignees_url":"https://api.github.com/repos/golang/snappy/assignees{/user}","branches_url":"https://api.github.com/repos/golang/snappy/branches{/branch}","tags_url":"https://api.github.com/repos/golang/snappy/tags","blobs_url":"https://api.github.com/repos/golang/snappy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/golang/snappy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/golang/snappy/git/refs{/sha}","trees_url":"https://api.github.com/repos/golang/snappy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/golang/snappy/statuses/{sha}","languages_url":"https://api.github.com/repos/golang/snappy/languages","stargazers_url":"https://api.github.com/repos/golang/snappy/stargazers","contributors_url":"https://api.github.com/repos/golang/snappy/contributors","subscribers_url":"https://api.github.com/repos/golang/snappy/subscribers","subscription_url":"https://api.github.com/repos/golang/snappy/subscription","commits_url":"https://api.github.com/repos/golang/snappy/commits{/sha}","git_commits_url":"https://api.github.com/repos/golang/snappy/git/commits{/sha}","comments_url":"https://api.github.com/repos/golang/snappy/comments{/number}","issue_comment_url":"https://api.github.com/repos/golang/snappy/issues/comments{/number}","contents_url":"https://api.github.com/repos/golang/snappy/contents/{+path}","compare_url":"https://api.github.com/repos/golang/snappy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/golang/snappy/merges","archive_url":"https://api.github.com/repos/golang/snappy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/golang/snappy/downloads","issues_url":"https://api.github.com/repos/golang/snappy/issues{/number}","pulls_url":"https://api.github.com/repos/golang/snappy/pulls{/number}","milestones_url":"https://api.github.com/repos/golang/snappy/milestones{/number}","notifications_url":"https://api.github.com/repos/golang/snappy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/golang/snappy/labels{/name}","releases_url":"https://api.github.com/repos/golang/snappy/releases{/id}","created_at":"2015-05-12T22:48:31Z","updated_at":"2015-11-04T12:40:36Z","pushed_at":"2015-08-31T19:53:04Z","git_url":"git://github.com/golang/snappy.git","ssh_url":"git@github.com:golang/snappy.git","clone_url":"https://github.com/golang/snappy.git","svn_url":"https://github.com/golang/snappy","homepage":"","size":209,"stargazers_count":98,"watchers_count":98,"language":"Go","has_issues":true,"has_downloads":false,"has_wiki":false,"has_pages":false,"forks_count":14,"mirror_url":null,"open_issues_count":1,"forks":14,"open_issues":1,"watchers":98,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/golang/snappy/pulls/19"},"html":{"href":"https://github.com/golang/snappy/pull/19"},"issue":{"href":"https://api.github.com/repos/golang/snappy/issues/19"},"comments":{"href":"https://api.github.com/repos/golang/snappy/issues/19/comments"},"review_comments":{"href":"https://api.github.com/repos/golang/snappy/pulls/19/comments"},"review_comment":{"href":"https://api.github.com/repos/golang/snappy/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/golang/snappy/pulls/19/commits"},"statuses":{"href":"https://api.github.com/repos/golang/snappy/statuses/9353357959f43c3d0dc9bc3120d50d3ef5605ee1"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":39,"deletions":1,"changed_files":2}}