{"action":"created","comment":{"url":"https://api.github.com/repos/docker/docker/pulls/comments/47587417","id":47587417,"diff_hunk":"@@ -97,16 +97,20 @@ func (fm *fileMetadataTransaction) SetCacheID(cacheID string) error {\n \treturn ioutil.WriteFile(filepath.Join(fm.root, \"cache-id\"), []byte(cacheID), 0644)\n }\n \n-func (fm *fileMetadataTransaction) TarSplitWriter() (io.WriteCloser, error) {\n+func (fm *fileMetadataTransaction) TarSplitWriter(compressInput bool) (io.WriteCloser, error) {\n \tf, err := os.OpenFile(filepath.Join(fm.root, \"tar-split.json.gz\"), os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0644)\n \tif err != nil {\n \t\treturn nil, err\n \t}\n+\tvar wc io.WriteCloser","path":"layer/filestore.go","position":10,"original_position":10,"commit_id":"ee5d9601788d265a28754b8d15a6c3b7e791c0f6","original_commit_id":"ee5d9601788d265a28754b8d15a6c3b7e791c0f6","user":{"login":"tonistiigi","id":585223,"avatar_url":"https://avatars.githubusercontent.com/u/585223?v=3","gravatar_id":"","url":"https://api.github.com/users/tonistiigi","html_url":"https://github.com/tonistiigi","followers_url":"https://api.github.com/users/tonistiigi/followers","following_url":"https://api.github.com/users/tonistiigi/following{/other_user}","gists_url":"https://api.github.com/users/tonistiigi/gists{/gist_id}","starred_url":"https://api.github.com/users/tonistiigi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonistiigi/subscriptions","organizations_url":"https://api.github.com/users/tonistiigi/orgs","repos_url":"https://api.github.com/users/tonistiigi/repos","events_url":"https://api.github.com/users/tonistiigi/events{/privacy}","received_events_url":"https://api.github.com/users/tonistiigi/received_events","type":"User","site_admin":false},"body":"Sorry, I mixed it up. `bool=false` is the exception for the migration.","created_at":"2015-12-15T01:45:20Z","updated_at":"2015-12-15T01:45:20Z","html_url":"https://github.com/docker/docker/pull/18399#discussion_r47587417","pull_request_url":"https://api.github.com/repos/docker/docker/pulls/18399","_links":{"self":{"href":"https://api.github.com/repos/docker/docker/pulls/comments/47587417"},"html":{"href":"https://github.com/docker/docker/pull/18399#discussion_r47587417"},"pull_request":{"href":"https://api.github.com/repos/docker/docker/pulls/18399"}}},"pull_request":{"url":"https://api.github.com/repos/docker/docker/pulls/18399","id":52571849,"html_url":"https://github.com/docker/docker/pull/18399","diff_url":"https://github.com/docker/docker/pull/18399.diff","patch_url":"https://github.com/docker/docker/pull/18399.patch","issue_url":"https://api.github.com/repos/docker/docker/issues/18399","number":18399,"state":"open","locked":false,"title":"Migration optimizations","user":{"login":"tonistiigi","id":585223,"avatar_url":"https://avatars.githubusercontent.com/u/585223?v=3","gravatar_id":"","url":"https://api.github.com/users/tonistiigi","html_url":"https://github.com/tonistiigi","followers_url":"https://api.github.com/users/tonistiigi/followers","following_url":"https://api.github.com/users/tonistiigi/following{/other_user}","gists_url":"https://api.github.com/users/tonistiigi/gists{/gist_id}","starred_url":"https://api.github.com/users/tonistiigi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonistiigi/subscriptions","organizations_url":"https://api.github.com/users/tonistiigi/orgs","repos_url":"https://api.github.com/users/tonistiigi/repos","events_url":"https://api.github.com/users/tonistiigi/events{/privacy}","received_events_url":"https://api.github.com/users/tonistiigi/received_events","type":"User","site_admin":false},"body":"This PR contains efforts to speed up migration to content addressability. In my testcase the improvement is ~55%.\r\n\r\nLayers checksum calculation is separated from migration process so it can be ran on parallel(and in a separate tool). Also changes it so that cancelled migration doesn't start from the beginning again.\r\n\r\nOther optimizations include:\r\n - avoiding recompressing tar-data file\r\n - avoiding unnecessary blocking in `io.Pipe`\r\n - taking advantage of the tar-split optimizations merged with #18375\r\n\r\nMost of the improvement is for the tar-split migration. If you have lots of self-built images the improvement is mostly only from parallel processing.\r\n\r\nCurrent state of the external migration tool can be seen in https://github.com/tonistiigi/docker-v1.10-migrator . Please note that I have only manually tested this tool atm.\r\n\r\n@aaronlehmann @LK4D4 ","created_at":"2015-12-03T18:21:05Z","updated_at":"2015-12-15T01:45:21Z","closed_at":null,"merged_at":null,"merge_commit_sha":"690265bceeb1cf822408e08ded241a07f8c732a9","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/docker/docker/pulls/18399/commits","review_comments_url":"https://api.github.com/repos/docker/docker/pulls/18399/comments","review_comment_url":"https://api.github.com/repos/docker/docker/pulls/comments{/number}","comments_url":"https://api.github.com/repos/docker/docker/issues/18399/comments","statuses_url":"https://api.github.com/repos/docker/docker/statuses/ee5d9601788d265a28754b8d15a6c3b7e791c0f6","head":{"label":"tonistiigi:migration-optimization","ref":"migration-optimization","sha":"ee5d9601788d265a28754b8d15a6c3b7e791c0f6","user":{"login":"tonistiigi","id":585223,"avatar_url":"https://avatars.githubusercontent.com/u/585223?v=3","gravatar_id":"","url":"https://api.github.com/users/tonistiigi","html_url":"https://github.com/tonistiigi","followers_url":"https://api.github.com/users/tonistiigi/followers","following_url":"https://api.github.com/users/tonistiigi/following{/other_user}","gists_url":"https://api.github.com/users/tonistiigi/gists{/gist_id}","starred_url":"https://api.github.com/users/tonistiigi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonistiigi/subscriptions","organizations_url":"https://api.github.com/users/tonistiigi/orgs","repos_url":"https://api.github.com/users/tonistiigi/repos","events_url":"https://api.github.com/users/tonistiigi/events{/privacy}","received_events_url":"https://api.github.com/users/tonistiigi/received_events","type":"User","site_admin":false},"repo":{"id":19074395,"name":"docker","full_name":"tonistiigi/docker","owner":{"login":"tonistiigi","id":585223,"avatar_url":"https://avatars.githubusercontent.com/u/585223?v=3","gravatar_id":"","url":"https://api.github.com/users/tonistiigi","html_url":"https://github.com/tonistiigi","followers_url":"https://api.github.com/users/tonistiigi/followers","following_url":"https://api.github.com/users/tonistiigi/following{/other_user}","gists_url":"https://api.github.com/users/tonistiigi/gists{/gist_id}","starred_url":"https://api.github.com/users/tonistiigi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonistiigi/subscriptions","organizations_url":"https://api.github.com/users/tonistiigi/orgs","repos_url":"https://api.github.com/users/tonistiigi/repos","events_url":"https://api.github.com/users/tonistiigi/events{/privacy}","received_events_url":"https://api.github.com/users/tonistiigi/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tonistiigi/docker","description":"Docker - the open-source application container engine","fork":true,"url":"https://api.github.com/repos/tonistiigi/docker","forks_url":"https://api.github.com/repos/tonistiigi/docker/forks","keys_url":"https://api.github.com/repos/tonistiigi/docker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tonistiigi/docker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tonistiigi/docker/teams","hooks_url":"https://api.github.com/repos/tonistiigi/docker/hooks","issue_events_url":"https://api.github.com/repos/tonistiigi/docker/issues/events{/number}","events_url":"https://api.github.com/repos/tonistiigi/docker/events","assignees_url":"https://api.github.com/repos/tonistiigi/docker/assignees{/user}","branches_url":"https://api.github.com/repos/tonistiigi/docker/branches{/branch}","tags_url":"https://api.github.com/repos/tonistiigi/docker/tags","blobs_url":"https://api.github.com/repos/tonistiigi/docker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tonistiigi/docker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tonistiigi/docker/git/refs{/sha}","trees_url":"https://api.github.com/repos/tonistiigi/docker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tonistiigi/docker/statuses/{sha}","languages_url":"https://api.github.com/repos/tonistiigi/docker/languages","stargazers_url":"https://api.github.com/repos/tonistiigi/docker/stargazers","contributors_url":"https://api.github.com/repos/tonistiigi/docker/contributors","subscribers_url":"https://api.github.com/repos/tonistiigi/docker/subscribers","subscription_url":"https://api.github.com/repos/tonistiigi/docker/subscription","commits_url":"https://api.github.com/repos/tonistiigi/docker/commits{/sha}","git_commits_url":"https://api.github.com/repos/tonistiigi/docker/git/commits{/sha}","comments_url":"https://api.github.com/repos/tonistiigi/docker/comments{/number}","issue_comment_url":"https://api.github.com/repos/tonistiigi/docker/issues/comments{/number}","contents_url":"https://api.github.com/repos/tonistiigi/docker/contents/{+path}","compare_url":"https://api.github.com/repos/tonistiigi/docker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tonistiigi/docker/merges","archive_url":"https://api.github.com/repos/tonistiigi/docker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tonistiigi/docker/downloads","issues_url":"https://api.github.com/repos/tonistiigi/docker/issues{/number}","pulls_url":"https://api.github.com/repos/tonistiigi/docker/pulls{/number}","milestones_url":"https://api.github.com/repos/tonistiigi/docker/milestones{/number}","notifications_url":"https://api.github.com/repos/tonistiigi/docker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tonistiigi/docker/labels{/name}","releases_url":"https://api.github.com/repos/tonistiigi/docker/releases{/id}","created_at":"2014-04-23T15:15:03Z","updated_at":"2014-07-24T21:22:45Z","pushed_at":"2015-12-15T01:28:07Z","git_url":"git://github.com/tonistiigi/docker.git","ssh_url":"git@github.com:tonistiigi/docker.git","clone_url":"https://github.com/tonistiigi/docker.git","svn_url":"https://github.com/tonistiigi/docker","homepage":"http://www.docker.io","size":76434,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"docker:master","ref":"master","sha":"a16a1d75d928000c8e8e9995d8843a685aa5df65","user":{"login":"docker","id":5429470,"avatar_url":"https://avatars.githubusercontent.com/u/5429470?v=3","gravatar_id":"","url":"https://api.github.com/users/docker","html_url":"https://github.com/docker","followers_url":"https://api.github.com/users/docker/followers","following_url":"https://api.github.com/users/docker/following{/other_user}","gists_url":"https://api.github.com/users/docker/gists{/gist_id}","starred_url":"https://api.github.com/users/docker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/docker/subscriptions","organizations_url":"https://api.github.com/users/docker/orgs","repos_url":"https://api.github.com/users/docker/repos","events_url":"https://api.github.com/users/docker/events{/privacy}","received_events_url":"https://api.github.com/users/docker/received_events","type":"Organization","site_admin":false},"repo":{"id":7691631,"name":"docker","full_name":"docker/docker","owner":{"login":"docker","id":5429470,"avatar_url":"https://avatars.githubusercontent.com/u/5429470?v=3","gravatar_id":"","url":"https://api.github.com/users/docker","html_url":"https://github.com/docker","followers_url":"https://api.github.com/users/docker/followers","following_url":"https://api.github.com/users/docker/following{/other_user}","gists_url":"https://api.github.com/users/docker/gists{/gist_id}","starred_url":"https://api.github.com/users/docker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/docker/subscriptions","organizations_url":"https://api.github.com/users/docker/orgs","repos_url":"https://api.github.com/users/docker/repos","events_url":"https://api.github.com/users/docker/events{/privacy}","received_events_url":"https://api.github.com/users/docker/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/docker/docker","description":"Docker - the open-source application container engine","fork":false,"url":"https://api.github.com/repos/docker/docker","forks_url":"https://api.github.com/repos/docker/docker/forks","keys_url":"https://api.github.com/repos/docker/docker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/docker/docker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/docker/docker/teams","hooks_url":"https://api.github.com/repos/docker/docker/hooks","issue_events_url":"https://api.github.com/repos/docker/docker/issues/events{/number}","events_url":"https://api.github.com/repos/docker/docker/events","assignees_url":"https://api.github.com/repos/docker/docker/assignees{/user}","branches_url":"https://api.github.com/repos/docker/docker/branches{/branch}","tags_url":"https://api.github.com/repos/docker/docker/tags","blobs_url":"https://api.github.com/repos/docker/docker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/docker/docker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/docker/docker/git/refs{/sha}","trees_url":"https://api.github.com/repos/docker/docker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/docker/docker/statuses/{sha}","languages_url":"https://api.github.com/repos/docker/docker/languages","stargazers_url":"https://api.github.com/repos/docker/docker/stargazers","contributors_url":"https://api.github.com/repos/docker/docker/contributors","subscribers_url":"https://api.github.com/repos/docker/docker/subscribers","subscription_url":"https://api.github.com/repos/docker/docker/subscription","commits_url":"https://api.github.com/repos/docker/docker/commits{/sha}","git_commits_url":"https://api.github.com/repos/docker/docker/git/commits{/sha}","comments_url":"https://api.github.com/repos/docker/docker/comments{/number}","issue_comment_url":"https://api.github.com/repos/docker/docker/issues/comments{/number}","contents_url":"https://api.github.com/repos/docker/docker/contents/{+path}","compare_url":"https://api.github.com/repos/docker/docker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/docker/docker/merges","archive_url":"https://api.github.com/repos/docker/docker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/docker/docker/downloads","issues_url":"https://api.github.com/repos/docker/docker/issues{/number}","pulls_url":"https://api.github.com/repos/docker/docker/pulls{/number}","milestones_url":"https://api.github.com/repos/docker/docker/milestones{/number}","notifications_url":"https://api.github.com/repos/docker/docker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/docker/docker/labels{/name}","releases_url":"https://api.github.com/repos/docker/docker/releases{/id}","created_at":"2013-01-18T18:10:57Z","updated_at":"2015-12-15T01:15:22Z","pushed_at":"2015-12-15T01:41:37Z","git_url":"git://github.com/docker/docker.git","ssh_url":"git@github.com:docker/docker.git","clone_url":"https://github.com/docker/docker.git","svn_url":"https://github.com/docker/docker","homepage":"http://www.docker.com","size":118213,"stargazers_count":27016,"watchers_count":27016,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":7361,"mirror_url":null,"open_issues_count":1228,"forks":7361,"open_issues":1228,"watchers":27016,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/docker/docker/pulls/18399"},"html":{"href":"https://github.com/docker/docker/pull/18399"},"issue":{"href":"https://api.github.com/repos/docker/docker/issues/18399"},"comments":{"href":"https://api.github.com/repos/docker/docker/issues/18399/comments"},"review_comments":{"href":"https://api.github.com/repos/docker/docker/pulls/18399/comments"},"review_comment":{"href":"https://api.github.com/repos/docker/docker/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/docker/docker/pulls/18399/commits"},"statuses":{"href":"https://api.github.com/repos/docker/docker/statuses/ee5d9601788d265a28754b8d15a6c3b7e791c0f6"}}}}