{"action":"created","issue":{"url":"https://api.github.com/repos/tdhock/animint/issues/76","labels_url":"https://api.github.com/repos/tdhock/animint/issues/76/labels{/name}","comments_url":"https://api.github.com/repos/tdhock/animint/issues/76/comments","events_url":"https://api.github.com/repos/tdhock/animint/issues/76/events","html_url":"https://github.com/tdhock/animint/pull/76","id":79630010,"number":76,"title":"optimizations for many selection variables","user":{"login":"tdhock","id":932850,"avatar_url":"https://avatars.githubusercontent.com/u/932850?v=3","gravatar_id":"","url":"https://api.github.com/users/tdhock","html_url":"https://github.com/tdhock","followers_url":"https://api.github.com/users/tdhock/followers","following_url":"https://api.github.com/users/tdhock/following{/other_user}","gists_url":"https://api.github.com/users/tdhock/gists{/gist_id}","starred_url":"https://api.github.com/users/tdhock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdhock/subscriptions","organizations_url":"https://api.github.com/users/tdhock/orgs","repos_url":"https://api.github.com/users/tdhock/repos","events_url":"https://api.github.com/users/tdhock/events{/privacy}","received_events_url":"https://api.github.com/users/tdhock/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-05-22T22:09:24Z","updated_at":"2015-05-26T18:53:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/tdhock/animint/pulls/76","html_url":"https://github.com/tdhock/animint/pull/76","diff_url":"https://github.com/tdhock/animint/pull/76.diff","patch_url":"https://github.com/tdhock/animint/pull/76.patch"},"body":"I ran into some problems while implementing an animint with 206 selection variables http://bl.ocks.org/tdhock/raw/36dc08ace6bd39f8693e/ source: https://github.com/tdhock/animint-examples/blob/master/examples/PSJ.R\r\n\r\nI used a for loop to define a bunch of layers with small data sets, each with a different `aes_string` mapping. This results in a lot of wasted disk space: 1180 geoms split into 1767 files for a total of 8.3M on my ext4 filesystem.\r\n\r\nJust looking at the tsv files there are many which are under 1KB each, for a rather reasonable total of about 1MB:\r\n```\r\n~/R/animint-examples/examples $ du --block-size=1 -cs --apparent-size PSJ-for/*.tsv|tail\r\n480\tPSJ-for/geom98_segment_coverage_chunk1.tsv\r\n154\tPSJ-for/geom99_segment_coverage_chunk1.tsv\r\n261\tPSJ-for/geom99_segment_coverage_chunk2.tsv\r\n368\tPSJ-for/geom99_segment_coverage_chunk3.tsv\r\n475\tPSJ-for/geom99_segment_coverage_chunk4.tsv\r\n153\tPSJ-for/geom9_segment_coverage_chunk1.tsv\r\n259\tPSJ-for/geom9_segment_coverage_chunk2.tsv\r\n365\tPSJ-for/geom9_segment_coverage_chunk3.tsv\r\n471\tPSJ-for/geom9_segment_coverage_chunk4.tsv\r\n961044\ttotal\r\n```\r\n\r\nbut actually if you count the filesystem meta-data the data files take up a lot more space:\r\n```\r\n~/R/animint-examples/examples $ du --block-size=1 -cs PSJ-for/*.tsv|tail\r\n4096\tPSJ-for/geom98_segment_coverage_chunk1.tsv\r\n4096\tPSJ-for/geom99_segment_coverage_chunk1.tsv\r\n4096\tPSJ-for/geom99_segment_coverage_chunk2.tsv\r\n4096\tPSJ-for/geom99_segment_coverage_chunk3.tsv\r\n4096\tPSJ-for/geom99_segment_coverage_chunk4.tsv\r\n4096\tPSJ-for/geom9_segment_coverage_chunk1.tsv\r\n4096\tPSJ-for/geom9_segment_coverage_chunk2.tsv\r\n4096\tPSJ-for/geom9_segment_coverage_chunk3.tsv\r\n4096\tPSJ-for/geom9_segment_coverage_chunk4.tsv\r\n7471104\ttotal\r\n```\r\nSo the data files take up almost 8MB of disk space, meaning 7MB just of meta-data.\r\n\r\nI propose that by default the compiler should not split any geoms into multiple chunks of less than 4KB. In the example above, geom9 was split into 4 chunks but it should not be split (just 1 chunk).\r\n\r\nI have already implemented a test using the breakpoints data."},"comment":{"url":"https://api.github.com/repos/tdhock/animint/issues/comments/105633353","html_url":"https://github.com/tdhock/animint/pull/76#issuecomment-105633353","issue_url":"https://api.github.com/repos/tdhock/animint/issues/76","id":105633353,"user":{"login":"cpsievert","id":1365941,"avatar_url":"https://avatars.githubusercontent.com/u/1365941?v=3","gravatar_id":"","url":"https://api.github.com/users/cpsievert","html_url":"https://github.com/cpsievert","followers_url":"https://api.github.com/users/cpsievert/followers","following_url":"https://api.github.com/users/cpsievert/following{/other_user}","gists_url":"https://api.github.com/users/cpsievert/gists{/gist_id}","starred_url":"https://api.github.com/users/cpsievert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpsievert/subscriptions","organizations_url":"https://api.github.com/users/cpsievert/orgs","repos_url":"https://api.github.com/users/cpsievert/repos","events_url":"https://api.github.com/users/cpsievert/events{/privacy}","received_events_url":"https://api.github.com/users/cpsievert/received_events","type":"User","site_admin":false},"created_at":"2015-05-26T18:53:13Z","updated_at":"2015-05-26T18:53:13Z","body":"Looking good @tdhock!\r\n\r\n> I propose that by default the compiler should not split any geoms into multiple chunks of less than 4KB. \r\n\r\nIs there anything special about that 4KB number? Maybe the filesystem attaches that much meta-data to *any* tsv file? Any idea if that is a common thing for other file systems to do?"}}