{"action":"created","issue":{"url":"https://api.github.com/repos/dataprotocols/dataprotocols/issues/166","labels_url":"https://api.github.com/repos/dataprotocols/dataprotocols/issues/166/labels{/name}","comments_url":"https://api.github.com/repos/dataprotocols/dataprotocols/issues/166/comments","events_url":"https://api.github.com/repos/dataprotocols/dataprotocols/issues/166/events","html_url":"https://github.com/dataprotocols/dataprotocols/issues/166","id":59145718,"number":166,"title":"Comment on article Git for Data","user":{"login":"omidkrad","id":767147,"avatar_url":"https://avatars.githubusercontent.com/u/767147?v=3","gravatar_id":"","url":"https://api.github.com/users/omidkrad","html_url":"https://github.com/omidkrad","followers_url":"https://api.github.com/users/omidkrad/followers","following_url":"https://api.github.com/users/omidkrad/following{/other_user}","gists_url":"https://api.github.com/users/omidkrad/gists{/gist_id}","starred_url":"https://api.github.com/users/omidkrad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/omidkrad/subscriptions","organizations_url":"https://api.github.com/users/omidkrad/orgs","repos_url":"https://api.github.com/users/omidkrad/repos","events_url":"https://api.github.com/users/omidkrad/events{/privacy}","received_events_url":"https://api.github.com/users/omidkrad/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-26T21:23:01Z","updated_at":"2015-04-18T16:44:31Z","closed_at":null,"body":"Comment section was closed so I thought I'd post here.\r\n\r\nHi @rgrp, in your article on [Git for Data](http://blog.okfn.org/2013/07/02/git-and-github-for-data/) you say in Git \"a simple change to a table is treated as a change to every line of the file\". But reading about [Packfiles](http://git-scm.com/book/en/v2/Git-Internals-Packfiles) looks like git will do the optimizations for this and will pack all those diffs for you. I ran `git gc` on a repository with around 80,000 git objects and after packing I had only 270 objects under `.git/objects`. Does this built-in optimization/packaging in Git make it possible to use it to store lightweight streaming data such as transactions?\r\n\r\nGreat article BTW, thank you!"},"comment":{"url":"https://api.github.com/repos/dataprotocols/dataprotocols/issues/comments/94179910","html_url":"https://github.com/dataprotocols/dataprotocols/issues/166#issuecomment-94179910","issue_url":"https://api.github.com/repos/dataprotocols/dataprotocols/issues/166","id":94179910,"user":{"login":"omidkrad","id":767147,"avatar_url":"https://avatars.githubusercontent.com/u/767147?v=3","gravatar_id":"","url":"https://api.github.com/users/omidkrad","html_url":"https://github.com/omidkrad","followers_url":"https://api.github.com/users/omidkrad/followers","following_url":"https://api.github.com/users/omidkrad/following{/other_user}","gists_url":"https://api.github.com/users/omidkrad/gists{/gist_id}","starred_url":"https://api.github.com/users/omidkrad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/omidkrad/subscriptions","organizations_url":"https://api.github.com/users/omidkrad/orgs","repos_url":"https://api.github.com/users/omidkrad/repos","events_url":"https://api.github.com/users/omidkrad/events{/privacy}","received_events_url":"https://api.github.com/users/omidkrad/received_events","type":"User","site_admin":false},"created_at":"2015-04-18T16:44:31Z","updated_at":"2015-04-18T16:44:31Z","body":"I did some more experiments on this. I created two scripts:\r\n\r\nFirst Script\r\n\r\n\r\nRecordsRepo: One record per line\r\n\tTook 20 min for 10,000 records\r\n\t1.97 GB\r\n\t1 file, 30,003 Objects\r\n  After git GC:\r\n\tCounting objects: 30003, done.\r\n\tDelta compression using up to 8 threads.\r\n\tCompressing objects: 100% (20001/20001), done.\r\n\tWriting objects: 100% (30003/30003), done.\r\n\tTotal 30003 (delta 9993), reused 0 (delta 0)\r\n\tRemoving duplicate objects: 100% (256/256), done.\r\n\t-> 9.02 MB, 1 file, 1 pack, 0 objects, took more to GC\r\n\r\nRecordsRepo2: One record per file\r\n\tTook 1 hour 2 min for 10,000 records\r\n\t1.50 GB\r\n\t10,000 files, 29,997 objects\r\n  After git GC:\r\n\tCounting objects: 29997, done.\r\n\tDelta compression using up to 8 threads.\r\n\tCompressing objects: 100% (29997/29997), done.\r\n\tWriting objects: 100% (29997/29997), done.\r\n\tTotal 29997 (delta 9993), reused 0 (delta 0)\r\n\tRemoving duplicate objects: 100% (256/256), done.\r\n\tChecking connectivity: 29997, done.\r\n\t-> 10.4 MB, 10,000 files, 1 pack, 1 object, took less time to GC\r\n"}}