{"action":"created","issue":{"url":"https://api.github.com/repos/cyaninc/git-fat/issues/41","labels_url":"https://api.github.com/repos/cyaninc/git-fat/issues/41/labels{/name}","comments_url":"https://api.github.com/repos/cyaninc/git-fat/issues/41/comments","events_url":"https://api.github.com/repos/cyaninc/git-fat/issues/41/events","html_url":"https://github.com/cyaninc/git-fat/issues/41","id":65500393,"number":41,"title":"Add symbolic-link backend","user":{"login":"KeithHanlan","id":7150447,"avatar_url":"https://avatars.githubusercontent.com/u/7150447?v=3","gravatar_id":"","url":"https://api.github.com/users/KeithHanlan","html_url":"https://github.com/KeithHanlan","followers_url":"https://api.github.com/users/KeithHanlan/followers","following_url":"https://api.github.com/users/KeithHanlan/following{/other_user}","gists_url":"https://api.github.com/users/KeithHanlan/gists{/gist_id}","starred_url":"https://api.github.com/users/KeithHanlan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KeithHanlan/subscriptions","organizations_url":"https://api.github.com/users/KeithHanlan/orgs","repos_url":"https://api.github.com/users/KeithHanlan/repos","events_url":"https://api.github.com/users/KeithHanlan/events{/privacy}","received_events_url":"https://api.github.com/users/KeithHanlan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-03-31T17:07:26Z","updated_at":"2015-04-02T02:07:37Z","closed_at":null,"body":"For performance reasons, the fastest way to make big fat binary files available in a repo would be to simply update symbolic links. This can already be done with native git; one can create a symbolic link that points outside the repo. Checking out new baselines remains very fast regardless of the size of the binary.\r\n\r\nHowever, git's support for symbolic links does nothing to guarantee the integrity of the file being referenced. Since git-fat records the SHA-1 of the fat file, this could be adapted to enhance the use of symbolic links for external files (fat or otherwise).\r\n\r\nOther back-ends could still be configured as fallbacks for remote users."},"comment":{"url":"https://api.github.com/repos/cyaninc/git-fat/issues/comments/88695218","html_url":"https://github.com/cyaninc/git-fat/issues/41#issuecomment-88695218","issue_url":"https://api.github.com/repos/cyaninc/git-fat/issues/41","id":88695218,"user":{"login":"abraithwaite","id":3791413,"avatar_url":"https://avatars.githubusercontent.com/u/3791413?v=3","gravatar_id":"","url":"https://api.github.com/users/abraithwaite","html_url":"https://github.com/abraithwaite","followers_url":"https://api.github.com/users/abraithwaite/followers","following_url":"https://api.github.com/users/abraithwaite/following{/other_user}","gists_url":"https://api.github.com/users/abraithwaite/gists{/gist_id}","starred_url":"https://api.github.com/users/abraithwaite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abraithwaite/subscriptions","organizations_url":"https://api.github.com/users/abraithwaite/orgs","repos_url":"https://api.github.com/users/abraithwaite/repos","events_url":"https://api.github.com/users/abraithwaite/events{/privacy}","received_events_url":"https://api.github.com/users/abraithwaite/received_events","type":"User","site_admin":false},"created_at":"2015-04-02T02:07:37Z","updated_at":"2015-04-02T02:07:37Z","body":"Indeed.  I'll let you in on a little secret in that I don't really think `git-fat` is a great solution to this problem.  Facebook has in fact worked to [solve this problem](https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/) themselves, only with mercurial instead of git.\r\n\r\nThe strategy they use is lazy loading:\r\n> This extension changes the clone and pull commands to download only the commit metadata, while omitting all file changes that account for the bulk of the download. When a user performs an operation that needs the contents of files (such as checkout), we download the file contents on demand using Facebook's existing memcache infrastructure. This allows clone and pull to be fast no matter how much history has changed, while only adding a slight overhead to checkout.\r\n\r\nWhich is the same strategy I would use to patch git, if I were to.  The major difference I would make is instead of downloading the commit metadata, I'd make it so git downloads all objects under a certain size threshold, and then fetch them on demand when you run a checkout.  The problem of course with both strategies is what happens when you're offline.  Handling cases where you're missing files becomes an issue for all the various commands.  Without handling those cases, it would just look like you're always missing those files if you try to change branches when you're offline (which might be okay, as long as it's clear to the user that that's the case).\r\n\r\nSome of the work has already been done, with [git 1.9](http://stackoverflow.com/questions/6900103/why-cant-i-push-from-a-shallow-clone) you can push from a shallow clone but there's still more to be done if you want the full history and selectively exclude the large files in some cases.\r\n"}}