{"action":"created","issue":{"url":"https://api.github.com/repos/datalad/datalad/issues/17","labels_url":"https://api.github.com/repos/datalad/datalad/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/datalad/datalad/issues/17/comments","events_url":"https://api.github.com/repos/datalad/datalad/issues/17/events","html_url":"https://github.com/datalad/datalad/issues/17","id":47322555,"number":17,"title":"git-annex add (git ls-files) etc performance on laaarge datasets","user":{"login":"yarikoptic","id":39889,"avatar_url":"https://avatars.githubusercontent.com/u/39889?v=3","gravatar_id":"","url":"https://api.github.com/users/yarikoptic","html_url":"https://github.com/yarikoptic","followers_url":"https://api.github.com/users/yarikoptic/followers","following_url":"https://api.github.com/users/yarikoptic/following{/other_user}","gists_url":"https://api.github.com/users/yarikoptic/gists{/gist_id}","starred_url":"https://api.github.com/users/yarikoptic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yarikoptic/subscriptions","organizations_url":"https://api.github.com/users/yarikoptic/orgs","repos_url":"https://api.github.com/users/yarikoptic/repos","events_url":"https://api.github.com/users/yarikoptic/events{/privacy}","received_events_url":"https://api.github.com/users/yarikoptic/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/datalad/datalad/labels/git-annex","name":"git-annex","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2014-10-30T20:27:51Z","updated_at":"2015-01-02T22:42:38Z","closed_at":null,"body":"Just a topic for possible discussion etc\r\n\r\nJust for the sake of seeing how big would get a really big dataset, I am \"simulating\" an annex repository with all the files distributed by HCP 500 subjects release, which, as deposited to S3, has >5,600,000 files.  It seems to be running for a while now ;) \r\nSome of the initial bottlenecks I have detected:\r\n\r\n- git-annex add invokes 'git ls-files --others --exclude-standard -z' which is apparently CPU-hungry (not bound by IO) and does quite a bit of exploration (e.g. for .git in every subdirectory).  Might be worth profiling and getting some stats on what it spends time on. or if 'annex add' had option '--initial'  or '--plain-ls' which would avoid using ls-files (but might have side-effects etc)\r\n\r\n- further in the process, when actually files get annexed, git-annex itself uses some locking mechanism so is quite busy opening/closing(?, just looking at fds) journal.lck .  I wonder if the process of 'annex add'ing could somehow be parallelized if not made more efficient for such obnoxious cases\r\n\r\nThe script I am using is here: https://github.com/datalad/datalad/blob/master/tools/mimic_repo (really not sure why I did it in python instead of a simple bash script). and here are the corresponding 's3cmd ls' outputs I am running on:\r\nhttp://www.onerussian.com/tmp/hcp-ls.20141020-500subject-only.txt.gz\r\nor only just a 1000 (to give a try)\r\nhttp://www.onerussian.com/tmp/hcp-ls.20141020-500subject-only1000.txt.gz\r\n\r\nso you could run e.g. with\r\n```\r\n./mimic_repo hcp-ls.20141020-500subject-only1000.txt.gz /tmp/testtt\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/datalad/datalad/issues/comments/68569727","html_url":"https://github.com/datalad/datalad/issues/17#issuecomment-68569727","issue_url":"https://api.github.com/repos/datalad/datalad/issues/17","id":68569727,"user":{"login":"yarikoptic","id":39889,"avatar_url":"https://avatars.githubusercontent.com/u/39889?v=3","gravatar_id":"","url":"https://api.github.com/users/yarikoptic","html_url":"https://github.com/yarikoptic","followers_url":"https://api.github.com/users/yarikoptic/followers","following_url":"https://api.github.com/users/yarikoptic/following{/other_user}","gists_url":"https://api.github.com/users/yarikoptic/gists{/gist_id}","starred_url":"https://api.github.com/users/yarikoptic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yarikoptic/subscriptions","organizations_url":"https://api.github.com/users/yarikoptic/orgs","repos_url":"https://api.github.com/users/yarikoptic/repos","events_url":"https://api.github.com/users/yarikoptic/events{/privacy}","received_events_url":"https://api.github.com/users/yarikoptic/received_events","type":"User","site_admin":false},"created_at":"2015-01-02T22:42:38Z","updated_at":"2015-01-02T22:42:38Z","body":"lovely exploration!!! thanks, even though I am yet to fully grasp it all and please feel free to tell me to stop ;-)...\r\n\r\n```\r\nSo, eliminating the second-level cache directories would keep the size\r\nthe same, while probably improving time efficiency by some amount. It\r\nseems like a win. However, I don't know how to realize these benefits\r\nwithout sacrificing backwards compatability. And, I'm supposed to be\r\nlooking at size, not speed...\r\n```\r\nif we are to look at the size... I wondered:  if instead of abc/def/SHA256-123456.log  it was a abc/def.log with all the keys logged in the same file, which theoretically I see of advantage for tree size/lookups (per your analysis, and overall tree would be smaller since less files), and disadvantage for storage (blobs) since additional column with with a key would need to be added to .log (unless some header with line#s is introduced), and for performance since lookup within the file would be necessary, and for implementation (custom merge facility adjusted). But given that collisions while using 6 hex digits are not very likely (I saw up to 6 in my experiment) I expect performance, even with lookups, being not that bad while we would make even more use of git's ability to get nice deltas on files modifications.  or am I missing some cornerstone(s)?\r\n\r\n```.git/objects is \"only\" 2.6 gb```\r\n\r\nis it possible to guestimate somehow what is the breakdown there between how much is used by tree objects, blobs, meta-information on commits/merges...?   may be it would become feasible to assess then effects of my silly suggestion above (or may be I should just simulate it as well...)\r\n\r\nBTW -- I thought to quickly check how much is used by trees/blobs in master -vs- git-annex branches but got confused -- any idea why .git/objects actually grew up after I removed master branch, left only git-annex and ran gc?\r\n\r\n```\r\n(git)smaug:/tmp/100307.cloned-addurl/100307.cloned-nomaster[git-annex]git\r\n\r\n$> git reflog  # is empty\r\n\r\n$> git br -a\r\n* git-annex\r\n\r\n$> git gc --aggressive\r\nCounting objects: 131283, done.\r\nDelta compression using up to 16 threads.\r\nCompressing objects: 100% (123662/123662), done.\r\nWriting objects: 100% (131283/131283), done.\r\nTotal 131283 (delta 47368), reused 83898 (delta 0)\r\nChecking connectivity: 131283, done.\r\n\r\n$> du -sck .git/objects                \r\n15856\t.git/objects\r\n15856\ttotal\r\n\r\n$> git br -D master\r\nDeleted branch master (was da447c1).\r\n\r\n$> git gc --aggressive \r\nCounting objects: 119652, done.\r\nDelta compression using up to 16 threads.\r\nCompressing objects: 100% (112058/112058), done.\r\nWriting objects: 100% (119652/119652), done.\r\nTotal 119652 (delta 47366), reused 72265 (delta 0)\r\nChecking connectivity: 119652, done.\r\n\r\n$> du -sck .git/objects\r\n61568\t.git/objects\r\n61568\ttotal\r\n```\r\n\r\nN.B. I might have underused ```git gc --aggressive``` and just relied too much on ```git repack -a -d -f --window=100```... so those 2.6gb sizes might shrink even more I guess"}}