{"action":"created","issue":{"url":"https://api.github.com/repos/docker/libpack/issues/14","labels_url":"https://api.github.com/repos/docker/libpack/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/docker/libpack/issues/14/comments","events_url":"https://api.github.com/repos/docker/libpack/issues/14/events","html_url":"https://github.com/docker/libpack/issues/14","id":53906328,"number":14,"title":"My thoughts on git as a binary storage format","user":{"login":"cgwalters","id":244096,"avatar_url":"https://avatars.githubusercontent.com/u/244096?v=3","gravatar_id":"","url":"https://api.github.com/users/cgwalters","html_url":"https://github.com/cgwalters","followers_url":"https://api.github.com/users/cgwalters/followers","following_url":"https://api.github.com/users/cgwalters/following{/other_user}","gists_url":"https://api.github.com/users/cgwalters/gists{/gist_id}","starred_url":"https://api.github.com/users/cgwalters/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgwalters/subscriptions","organizations_url":"https://api.github.com/users/cgwalters/orgs","repos_url":"https://api.github.com/users/cgwalters/repos","events_url":"https://api.github.com/users/cgwalters/events{/privacy}","received_events_url":"https://api.github.com/users/cgwalters/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-01-09T19:40:38Z","updated_at":"2015-01-12T16:27:44Z","closed_at":null,"body":"I use git myself to store binary data such as my http://keepass.info/ database.  It works just fine for small files like that.\r\n\r\nBeyond that though, whether this makes sense comes down to a several considerations:\r\n - How big is your data?\r\n - How often does it change?\r\n - When it does change, does it change in a way that a textual diff or xdelta are useful?\r\n - Do you need extended metadata such as uid/gid/xattrs?\r\n - Do you need the full history forever?\r\n - If serving on the network, are you ok with a \"smart server\"?\r\n\r\nThe big/often/change concerns all come together with x86_64 ELF, particularly things like 20MB+ statically linked golang binaries.  xdelta is just going to be a waste of CPU time on it (for OSTree, we're investigating Courgette: https://bugzilla.gnome.org/show_bug.cgi?id=721799#c27).\r\n\r\nIt looks like you're solving the metadata issue by having separate files, which I think should work just fine.  It does mean that many operations like `git checkout` and `git diff` aren't useful standalone, and will need to be wrapped.  Which isn't a big deal.\r\n\r\nBTW, see also https://github.com/bup/bup\r\n\r\nOn the history topic, while some people may be OK keeping all of their binaries forever, conceptually they're a cache of the source code, and in a continuous integration scenario you do end up wanting garbage collection of unnecessary images.  Sometimes the simplest way to do this is to have separate dev/prod data sets.  In the dev case though, it's nice to be able to filter the data without tossing everything, something like a backup style rolling history.  Implementing this with git means you're going much lower level and against its core design.\r\n\r\nFinally, the \"smart server\" consideration is a very big deal.  It locks you out of using things like CDNs/S3. And also even if you do set one up, the pack files it synthesizes are basically just going to be \"tar of new objects\" due to the above issue of diff/xdelta on ELF."},"comment":{"url":"https://api.github.com/repos/docker/libpack/issues/comments/69597455","html_url":"https://github.com/docker/libpack/issues/14#issuecomment-69597455","issue_url":"https://api.github.com/repos/docker/libpack/issues/14","id":69597455,"user":{"login":"icecrime","id":1564054,"avatar_url":"https://avatars.githubusercontent.com/u/1564054?v=3","gravatar_id":"","url":"https://api.github.com/users/icecrime","html_url":"https://github.com/icecrime","followers_url":"https://api.github.com/users/icecrime/followers","following_url":"https://api.github.com/users/icecrime/following{/other_user}","gists_url":"https://api.github.com/users/icecrime/gists{/gist_id}","starred_url":"https://api.github.com/users/icecrime/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/icecrime/subscriptions","organizations_url":"https://api.github.com/users/icecrime/orgs","repos_url":"https://api.github.com/users/icecrime/repos","events_url":"https://api.github.com/users/icecrime/events{/privacy}","received_events_url":"https://api.github.com/users/icecrime/received_events","type":"User","site_admin":false},"created_at":"2015-01-12T16:27:44Z","updated_at":"2015-01-12T16:27:44Z","body":"@cgwalters Sorry, [this is the correct link](https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution)"}}