{"action":"created","issue":{"url":"https://api.github.com/repos/docker/distribution/issues/756","labels_url":"https://api.github.com/repos/docker/distribution/issues/756/labels{/name}","comments_url":"https://api.github.com/repos/docker/distribution/issues/756/comments","events_url":"https://api.github.com/repos/docker/distribution/issues/756/events","html_url":"https://github.com/docker/distribution/pull/756","id":97348577,"number":756,"title":"Storage driver for Google Cloud Storage","user":{"login":"aibaars","id":10979737,"avatar_url":"https://avatars.githubusercontent.com/u/10979737?v=3","gravatar_id":"","url":"https://api.github.com/users/aibaars","html_url":"https://github.com/aibaars","followers_url":"https://api.github.com/users/aibaars/followers","following_url":"https://api.github.com/users/aibaars/following{/other_user}","gists_url":"https://api.github.com/users/aibaars/gists{/gist_id}","starred_url":"https://api.github.com/users/aibaars/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aibaars/subscriptions","organizations_url":"https://api.github.com/users/aibaars/orgs","repos_url":"https://api.github.com/users/aibaars/repos","events_url":"https://api.github.com/users/aibaars/events{/privacy}","received_events_url":"https://api.github.com/users/aibaars/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/distribution/labels/status/0-triage","name":"status/0-triage","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"RichardScothern","id":922018,"avatar_url":"https://avatars.githubusercontent.com/u/922018?v=3","gravatar_id":"","url":"https://api.github.com/users/RichardScothern","html_url":"https://github.com/RichardScothern","followers_url":"https://api.github.com/users/RichardScothern/followers","following_url":"https://api.github.com/users/RichardScothern/following{/other_user}","gists_url":"https://api.github.com/users/RichardScothern/gists{/gist_id}","starred_url":"https://api.github.com/users/RichardScothern/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RichardScothern/subscriptions","organizations_url":"https://api.github.com/users/RichardScothern/orgs","repos_url":"https://api.github.com/users/RichardScothern/repos","events_url":"https://api.github.com/users/RichardScothern/events{/privacy}","received_events_url":"https://api.github.com/users/RichardScothern/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/docker/distribution/milestones/9","html_url":"https://github.com/docker/distribution/milestones/Registry/2.2","labels_url":"https://api.github.com/repos/docker/distribution/milestones/9/labels","id":1026605,"number":9,"title":"Registry/2.2","description":"","creator":{"login":"stevvooe","id":120601,"avatar_url":"https://avatars.githubusercontent.com/u/120601?v=3","gravatar_id":"","url":"https://api.github.com/users/stevvooe","html_url":"https://github.com/stevvooe","followers_url":"https://api.github.com/users/stevvooe/followers","following_url":"https://api.github.com/users/stevvooe/following{/other_user}","gists_url":"https://api.github.com/users/stevvooe/gists{/gist_id}","starred_url":"https://api.github.com/users/stevvooe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevvooe/subscriptions","organizations_url":"https://api.github.com/users/stevvooe/orgs","repos_url":"https://api.github.com/users/stevvooe/repos","events_url":"https://api.github.com/users/stevvooe/events{/privacy}","received_events_url":"https://api.github.com/users/stevvooe/received_events","type":"User","site_admin":false},"open_issues":28,"closed_issues":64,"state":"open","created_at":"2015-03-18T04:28:58Z","updated_at":"2015-10-17T00:13:48Z","due_on":"2015-10-19T07:00:00Z","closed_at":null},"comments":16,"created_at":"2015-07-26T20:46:57Z","updated_at":"2015-10-17T10:29:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/docker/distribution/pulls/756","html_url":"https://github.com/docker/distribution/pull/756","diff_url":"https://github.com/docker/distribution/pull/756.diff","patch_url":"https://github.com/docker/distribution/pull/756.patch"},"body":"This driver implements the storagedriver.StorageDriver interface and uses Google Cloud Storage as storage backend.\r\n\r\nGoogle Cloud Storage is a key/value  object storage (much like Amazon S3). The following convention is used to keep the filesystem notions stored in this backend: \r\n\r\nAll the objects data are stored with their full paths as names; forward slashes are used to separate path components. Names have no leading or trailing slashes. Folders are not explicitly stored, and are derived by \"prefix queries\" on the cloud storage. As a result empty directories cannot be represented, but this should not be a problem.\r\n\r\nCurrently the driver is fully functional and passes all tests. The implementation of resuming of WriteStream is very inefficient at the moment. Objects on Google Cloud Storage are immutable and therefore it is not possible to append to them. The current implementation simply reads the existing object until the provided offset, and writes that contents plus the content to be appended to a new object. \r\n\r\nGoogle Cloud Storage has a resumable upload feature, however, this cannot be used without making changes to the storagedriver.StorageDriver interface. Google's resumable uploads work as follows:\r\n* Create new resumable session and obtain session URL\r\n* Use url to perform PUT operations\r\n* When the upload session is finished the new object will become visible.\r\n\r\nThe size of the object must be specified when creating the session or when performing the upload of the last chunk. To be able to use this mechanism the storagedriver.StorageDriver interface should either specify the length of the complete upload to WriteStream, or pass a value marking whether a chunk is the last of the upload. Additionally the test cases would need to be updated, to no longer assume that incomplete uploads can be inspected by the Stat method, since the objects are not visible until the resumable upload has finished. Ideally, WriteStream would also have a uuid or some other identifier to make sure we know which sequences of WriteStream invocations belong to the same upload.\r\n\r\nAs an alternative to Google's resumable uploads, we could use composite objects. A composite object is a multi-part object. The different chunks could be written to separate objects and then stitched together into a composite. This is a bit of a hassle as composites can have only 1024 pieces. \r\n\r\nMy preference would be to use resumable uploads. However, it the required changes to the storagedriver.StorageDriver interface are not possible, using composite objects would be a viable alternative.\r\n\r\nBy default the tests of this driver are skipped, as they rely on having a storage bucket hosted on Google AppEngine. To enable the test, the following environment variables need to be set:\r\n* REGISTRY_STORAGE_GCS_PROJECT : the name of a Google AppEngine project\r\n* REGISTRY_STORAGE_GCS_BUCKET : the name of a storage bucket of that project\r\n\r\nAuthentication, specify either of the following\r\n* REGISTRY_STORAGE_GCS_KEYFILE : the private key file of a service account (in JSON format)\r\n* GOOGLE_APPLICATION_CREDENTIALS : the name of a google application credentials file\r\n\r\nNote that the URLFor method requires REGISTRY_STORAGE_GCS_KEYFILE to be set, as the private key is used to sign the generated URLs\r\n\r\nTodo for this driver:\r\nImprove resume writing\r\n\r\nRelated: #221\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/docker/distribution/issues/comments/148905618","html_url":"https://github.com/docker/distribution/pull/756#issuecomment-148905618","issue_url":"https://api.github.com/repos/docker/distribution/issues/756","id":148905618,"user":{"login":"aibaars","id":10979737,"avatar_url":"https://avatars.githubusercontent.com/u/10979737?v=3","gravatar_id":"","url":"https://api.github.com/users/aibaars","html_url":"https://github.com/aibaars","followers_url":"https://api.github.com/users/aibaars/followers","following_url":"https://api.github.com/users/aibaars/following{/other_user}","gists_url":"https://api.github.com/users/aibaars/gists{/gist_id}","starred_url":"https://api.github.com/users/aibaars/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aibaars/subscriptions","organizations_url":"https://api.github.com/users/aibaars/orgs","repos_url":"https://api.github.com/users/aibaars/repos","events_url":"https://api.github.com/users/aibaars/events{/privacy}","received_events_url":"https://api.github.com/users/aibaars/received_events","type":"User","site_admin":false},"created_at":"2015-10-17T10:29:50Z","updated_at":"2015-10-17T10:29:50Z","body":"@skelterjohn I removed the project ID from the configuration parameters. I tried using empty string as project ID in the call to WithContext. Unfortunately WithContext panics on this, however, any arbitrary non-empty string seems to work (not surprising as the google cloud storage api does not use the project ID). Therefore I removed the project ID from the configuration and pass in <unknown> as dummy value. Not the prettiest solution, so if you have a better idea, let me know."}}