{"action":"opened","issue":{"url":"https://api.github.com/repos/codalab/codalab-cli/issues/297","labels_url":"https://api.github.com/repos/codalab/codalab-cli/issues/297/labels{/name}","comments_url":"https://api.github.com/repos/codalab/codalab-cli/issues/297/comments","events_url":"https://api.github.com/repos/codalab/codalab-cli/issues/297/events","html_url":"https://github.com/codalab/codalab-cli/issues/297","id":124377097,"number":297,"title":"compress bundles for long-term storage","user":{"login":"percyliang","id":435384,"avatar_url":"https://avatars.githubusercontent.com/u/435384?v=3","gravatar_id":"","url":"https://api.github.com/users/percyliang","html_url":"https://github.com/percyliang","followers_url":"https://api.github.com/users/percyliang/followers","following_url":"https://api.github.com/users/percyliang/following{/other_user}","gists_url":"https://api.github.com/users/percyliang/gists{/gist_id}","starred_url":"https://api.github.com/users/percyliang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/percyliang/subscriptions","organizations_url":"https://api.github.com/users/percyliang/orgs","repos_url":"https://api.github.com/users/percyliang/repos","events_url":"https://api.github.com/users/percyliang/events{/privacy}","received_events_url":"https://api.github.com/users/percyliang/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/codalab/codalab-cli/labels/P1","name":"P1","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-12-30T17:29:47Z","updated_at":"2015-12-30T17:29:47Z","closed_at":null,"body":"Compress bundle directories to save space (use tar.gz).\r\n\r\nUnfortunately, tar.gz doesn't allow for random access.  Should keep an index into the tar file so we can see quickly what's in the tar.  Can we get random access?  One option is to go to zip, but the rest of the system uses tar, so it'd be better to stick with that.\r\n\r\nConsideration: directives need to pull out parts of the bundle (not necessarily known in advance).  Heuristic: leave the first 1K or so each file uncompressed?  This might be useful even if we have random access because we'll need it if files are stored in S3.\r\n\r\nPolicy for compressing: some daemon in the back should be doing this to bundles that don't seem like they are used that much, or just after a certain period of time.  Or we could even just compress everything.  (When files are transferred, they are compressed anyway.)\r\n\r\nIn summary, we provide only two ways to store a bundle:\r\n1) tar.gz (.gz for uploaded files)\r\n2) summary: first N and last N lines of each file (max # files = 1000), and a full file listing\r\n3) untarred version - this should only be on the worker machine\r\n\r\nWorker should be updating 2) all along, and then should transfer 1) when everything is said and done.  When transferring 2), should transfer diff?"}}