{"action":"created","comment":{"url":"https://api.github.com/repos/tdhock/animint/pulls/comments/32129167","id":32129167,"diff_hunk":"@@ -649,12 +694,26 @@ saveLayer <- function(l, d, meta){\n saveChunks <- function(x, vars, meta){\n   if(is.data.frame(x)){\n     if(length(vars) == 0){\n+      if(meta$geom %in% c(\"line\", \"path\", \"polygon\", \"ribbon\")) {\n+        if(\"group\" %in% names(x)) {\n+          # common info for groups, save only once\n+          csv.name <- sprintf(\"%s_chunk_shape.tsv\", meta$classed)\n+          chunk.shape <- file.path(meta$out.dir, csv.name)\n+          if(!file.exists(chunk.shape)){\n+            x1 <- x[c(\"x\", \"y\", \"group\")]\n+            write.table(x1, chunk.shape, quote = FALSE, row.names = FALSE, \n+                        sep = \"\\t\")\n+          }\n+          # info for each group\n+          x2 <- x[, !names(x) %in% c(\"x\", \"y\")]\n+          x <- x2[!duplicated(x2), ]\n+          }","path":"R/animint.R","position":null,"original_position":127,"commit_id":"728d97febaa743aed71ba72e22ab1a818640ef74","original_commit_id":"2b74e77474bdb329aea6166e87ad5b5f0f43c6b2","user":{"login":"caijun","id":2522593,"avatar_url":"https://avatars.githubusercontent.com/u/2522593?v=3","gravatar_id":"","url":"https://api.github.com/users/caijun","html_url":"https://github.com/caijun","followers_url":"https://api.github.com/users/caijun/followers","following_url":"https://api.github.com/users/caijun/following{/other_user}","gists_url":"https://api.github.com/users/caijun/gists{/gist_id}","starred_url":"https://api.github.com/users/caijun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caijun/subscriptions","organizations_url":"https://api.github.com/users/caijun/orgs","repos_url":"https://api.github.com/users/caijun/repos","events_url":"https://api.github.com/users/caijun/events{/privacy}","received_events_url":"https://api.github.com/users/caijun/received_events","type":"User","site_admin":false},"body":"I think the redundancy mostly occurs when drawing spatial objects (excluding point). For non-spatial objects, it may occur the join relation of n:1 for constant fields and updating fields. If I split variable `x` based on whether repeated data present, how do I deal with the case that the updating field indeed has duplicated values, for example in my `geom_polygon` example, some states are of the same fill colors.\r\n\r\nBesides, I am not clear that what should I test if write a test for this function? Could you give a hint?","created_at":"2015-06-10T15:22:06Z","updated_at":"2015-06-10T15:22:06Z","html_url":"https://github.com/tdhock/animint/pull/76#discussion_r32129167","pull_request_url":"https://api.github.com/repos/tdhock/animint/pulls/76","_links":{"self":{"href":"https://api.github.com/repos/tdhock/animint/pulls/comments/32129167"},"html":{"href":"https://github.com/tdhock/animint/pull/76#discussion_r32129167"},"pull_request":{"href":"https://api.github.com/repos/tdhock/animint/pulls/76"}}},"pull_request":{"url":"https://api.github.com/repos/tdhock/animint/pulls/76","id":36072950,"html_url":"https://github.com/tdhock/animint/pull/76","diff_url":"https://github.com/tdhock/animint/pull/76.diff","patch_url":"https://github.com/tdhock/animint/pull/76.patch","issue_url":"https://api.github.com/repos/tdhock/animint/issues/76","number":76,"state":"open","locked":false,"title":"optimizations for many selection variables","user":{"login":"tdhock","id":932850,"avatar_url":"https://avatars.githubusercontent.com/u/932850?v=3","gravatar_id":"","url":"https://api.github.com/users/tdhock","html_url":"https://github.com/tdhock","followers_url":"https://api.github.com/users/tdhock/followers","following_url":"https://api.github.com/users/tdhock/following{/other_user}","gists_url":"https://api.github.com/users/tdhock/gists{/gist_id}","starred_url":"https://api.github.com/users/tdhock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdhock/subscriptions","organizations_url":"https://api.github.com/users/tdhock/orgs","repos_url":"https://api.github.com/users/tdhock/repos","events_url":"https://api.github.com/users/tdhock/events{/privacy}","received_events_url":"https://api.github.com/users/tdhock/received_events","type":"User","site_admin":false},"body":"I ran into some problems while implementing an animint with 206 selection variables http://bl.ocks.org/tdhock/raw/36dc08ace6bd39f8693e/ source: https://github.com/tdhock/animint-examples/blob/master/examples/PSJ.R\r\n\r\nI used a for loop to define a bunch of layers with small data sets, each with a different `aes_string` mapping. This results in a lot of wasted disk space: 1180 geoms split into 1767 files for a total of 8.3M on my ext4 filesystem.\r\n\r\nJust looking at the tsv files there are many which are under 1KB each, for a rather reasonable total of about 1MB:\r\n```\r\n~/R/animint-examples/examples $ du --block-size=1 -cs --apparent-size PSJ-for/*.tsv|tail\r\n480\tPSJ-for/geom98_segment_coverage_chunk1.tsv\r\n154\tPSJ-for/geom99_segment_coverage_chunk1.tsv\r\n261\tPSJ-for/geom99_segment_coverage_chunk2.tsv\r\n368\tPSJ-for/geom99_segment_coverage_chunk3.tsv\r\n475\tPSJ-for/geom99_segment_coverage_chunk4.tsv\r\n153\tPSJ-for/geom9_segment_coverage_chunk1.tsv\r\n259\tPSJ-for/geom9_segment_coverage_chunk2.tsv\r\n365\tPSJ-for/geom9_segment_coverage_chunk3.tsv\r\n471\tPSJ-for/geom9_segment_coverage_chunk4.tsv\r\n961044\ttotal\r\n```\r\n\r\nbut actually if you count the filesystem meta-data the data files take up a lot more space:\r\n```\r\n~/R/animint-examples/examples $ du --block-size=1 -cs PSJ-for/*.tsv|tail\r\n4096\tPSJ-for/geom98_segment_coverage_chunk1.tsv\r\n4096\tPSJ-for/geom99_segment_coverage_chunk1.tsv\r\n4096\tPSJ-for/geom99_segment_coverage_chunk2.tsv\r\n4096\tPSJ-for/geom99_segment_coverage_chunk3.tsv\r\n4096\tPSJ-for/geom99_segment_coverage_chunk4.tsv\r\n4096\tPSJ-for/geom9_segment_coverage_chunk1.tsv\r\n4096\tPSJ-for/geom9_segment_coverage_chunk2.tsv\r\n4096\tPSJ-for/geom9_segment_coverage_chunk3.tsv\r\n4096\tPSJ-for/geom9_segment_coverage_chunk4.tsv\r\n7471104\ttotal\r\n```\r\nSo the data files take up almost 8MB of disk space, meaning 7MB just of meta-data.\r\n\r\nI propose that by default the compiler should not split any geoms into multiple chunks of less than 4KB. In the example above, geom9 was split into 4 chunks but it should not be split (just 1 chunk).\r\n\r\nI have already implemented a test using the breakpoints data.","created_at":"2015-05-22T22:09:24Z","updated_at":"2015-06-10T15:22:06Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b71f359c3a59726bdbf9064cc2460270926e840d","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/tdhock/animint/pulls/76/commits","review_comments_url":"https://api.github.com/repos/tdhock/animint/pulls/76/comments","review_comment_url":"https://api.github.com/repos/tdhock/animint/pulls/comments{/number}","comments_url":"https://api.github.com/repos/tdhock/animint/issues/76/comments","statuses_url":"https://api.github.com/repos/tdhock/animint/statuses/728d97febaa743aed71ba72e22ab1a818640ef74","head":{"label":"tdhock:big-chunks","ref":"big-chunks","sha":"728d97febaa743aed71ba72e22ab1a818640ef74","user":{"login":"tdhock","id":932850,"avatar_url":"https://avatars.githubusercontent.com/u/932850?v=3","gravatar_id":"","url":"https://api.github.com/users/tdhock","html_url":"https://github.com/tdhock","followers_url":"https://api.github.com/users/tdhock/followers","following_url":"https://api.github.com/users/tdhock/following{/other_user}","gists_url":"https://api.github.com/users/tdhock/gists{/gist_id}","starred_url":"https://api.github.com/users/tdhock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdhock/subscriptions","organizations_url":"https://api.github.com/users/tdhock/orgs","repos_url":"https://api.github.com/users/tdhock/repos","events_url":"https://api.github.com/users/tdhock/events{/privacy}","received_events_url":"https://api.github.com/users/tdhock/received_events","type":"User","site_admin":false},"repo":{"id":8399379,"name":"animint","full_name":"tdhock/animint","owner":{"login":"tdhock","id":932850,"avatar_url":"https://avatars.githubusercontent.com/u/932850?v=3","gravatar_id":"","url":"https://api.github.com/users/tdhock","html_url":"https://github.com/tdhock","followers_url":"https://api.github.com/users/tdhock/followers","following_url":"https://api.github.com/users/tdhock/following{/other_user}","gists_url":"https://api.github.com/users/tdhock/gists{/gist_id}","starred_url":"https://api.github.com/users/tdhock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdhock/subscriptions","organizations_url":"https://api.github.com/users/tdhock/orgs","repos_url":"https://api.github.com/users/tdhock/repos","events_url":"https://api.github.com/users/tdhock/events{/privacy}","received_events_url":"https://api.github.com/users/tdhock/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tdhock/animint","description":"animated and interactive web graphics","fork":false,"url":"https://api.github.com/repos/tdhock/animint","forks_url":"https://api.github.com/repos/tdhock/animint/forks","keys_url":"https://api.github.com/repos/tdhock/animint/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tdhock/animint/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tdhock/animint/teams","hooks_url":"https://api.github.com/repos/tdhock/animint/hooks","issue_events_url":"https://api.github.com/repos/tdhock/animint/issues/events{/number}","events_url":"https://api.github.com/repos/tdhock/animint/events","assignees_url":"https://api.github.com/repos/tdhock/animint/assignees{/user}","branches_url":"https://api.github.com/repos/tdhock/animint/branches{/branch}","tags_url":"https://api.github.com/repos/tdhock/animint/tags","blobs_url":"https://api.github.com/repos/tdhock/animint/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tdhock/animint/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tdhock/animint/git/refs{/sha}","trees_url":"https://api.github.com/repos/tdhock/animint/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tdhock/animint/statuses/{sha}","languages_url":"https://api.github.com/repos/tdhock/animint/languages","stargazers_url":"https://api.github.com/repos/tdhock/animint/stargazers","contributors_url":"https://api.github.com/repos/tdhock/animint/contributors","subscribers_url":"https://api.github.com/repos/tdhock/animint/subscribers","subscription_url":"https://api.github.com/repos/tdhock/animint/subscription","commits_url":"https://api.github.com/repos/tdhock/animint/commits{/sha}","git_commits_url":"https://api.github.com/repos/tdhock/animint/git/commits{/sha}","comments_url":"https://api.github.com/repos/tdhock/animint/comments{/number}","issue_comment_url":"https://api.github.com/repos/tdhock/animint/issues/comments{/number}","contents_url":"https://api.github.com/repos/tdhock/animint/contents/{+path}","compare_url":"https://api.github.com/repos/tdhock/animint/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tdhock/animint/merges","archive_url":"https://api.github.com/repos/tdhock/animint/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tdhock/animint/downloads","issues_url":"https://api.github.com/repos/tdhock/animint/issues{/number}","pulls_url":"https://api.github.com/repos/tdhock/animint/pulls{/number}","milestones_url":"https://api.github.com/repos/tdhock/animint/milestones{/number}","notifications_url":"https://api.github.com/repos/tdhock/animint/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tdhock/animint/labels{/name}","releases_url":"https://api.github.com/repos/tdhock/animint/releases{/id}","created_at":"2013-02-25T00:27:47Z","updated_at":"2015-06-02T19:36:52Z","pushed_at":"2015-06-10T15:08:51Z","git_url":"git://github.com/tdhock/animint.git","ssh_url":"git@github.com:tdhock/animint.git","clone_url":"https://github.com/tdhock/animint.git","svn_url":"https://github.com/tdhock/animint","homepage":"","size":137765,"stargazers_count":58,"watchers_count":58,"language":"R","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":23,"mirror_url":null,"open_issues_count":11,"forks":23,"open_issues":11,"watchers":58,"default_branch":"master"}},"base":{"label":"tdhock:master","ref":"master","sha":"a0677f2062b455da97138b43ede1f158e4bd36ae","user":{"login":"tdhock","id":932850,"avatar_url":"https://avatars.githubusercontent.com/u/932850?v=3","gravatar_id":"","url":"https://api.github.com/users/tdhock","html_url":"https://github.com/tdhock","followers_url":"https://api.github.com/users/tdhock/followers","following_url":"https://api.github.com/users/tdhock/following{/other_user}","gists_url":"https://api.github.com/users/tdhock/gists{/gist_id}","starred_url":"https://api.github.com/users/tdhock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdhock/subscriptions","organizations_url":"https://api.github.com/users/tdhock/orgs","repos_url":"https://api.github.com/users/tdhock/repos","events_url":"https://api.github.com/users/tdhock/events{/privacy}","received_events_url":"https://api.github.com/users/tdhock/received_events","type":"User","site_admin":false},"repo":{"id":8399379,"name":"animint","full_name":"tdhock/animint","owner":{"login":"tdhock","id":932850,"avatar_url":"https://avatars.githubusercontent.com/u/932850?v=3","gravatar_id":"","url":"https://api.github.com/users/tdhock","html_url":"https://github.com/tdhock","followers_url":"https://api.github.com/users/tdhock/followers","following_url":"https://api.github.com/users/tdhock/following{/other_user}","gists_url":"https://api.github.com/users/tdhock/gists{/gist_id}","starred_url":"https://api.github.com/users/tdhock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdhock/subscriptions","organizations_url":"https://api.github.com/users/tdhock/orgs","repos_url":"https://api.github.com/users/tdhock/repos","events_url":"https://api.github.com/users/tdhock/events{/privacy}","received_events_url":"https://api.github.com/users/tdhock/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tdhock/animint","description":"animated and interactive web graphics","fork":false,"url":"https://api.github.com/repos/tdhock/animint","forks_url":"https://api.github.com/repos/tdhock/animint/forks","keys_url":"https://api.github.com/repos/tdhock/animint/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tdhock/animint/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tdhock/animint/teams","hooks_url":"https://api.github.com/repos/tdhock/animint/hooks","issue_events_url":"https://api.github.com/repos/tdhock/animint/issues/events{/number}","events_url":"https://api.github.com/repos/tdhock/animint/events","assignees_url":"https://api.github.com/repos/tdhock/animint/assignees{/user}","branches_url":"https://api.github.com/repos/tdhock/animint/branches{/branch}","tags_url":"https://api.github.com/repos/tdhock/animint/tags","blobs_url":"https://api.github.com/repos/tdhock/animint/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tdhock/animint/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tdhock/animint/git/refs{/sha}","trees_url":"https://api.github.com/repos/tdhock/animint/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tdhock/animint/statuses/{sha}","languages_url":"https://api.github.com/repos/tdhock/animint/languages","stargazers_url":"https://api.github.com/repos/tdhock/animint/stargazers","contributors_url":"https://api.github.com/repos/tdhock/animint/contributors","subscribers_url":"https://api.github.com/repos/tdhock/animint/subscribers","subscription_url":"https://api.github.com/repos/tdhock/animint/subscription","commits_url":"https://api.github.com/repos/tdhock/animint/commits{/sha}","git_commits_url":"https://api.github.com/repos/tdhock/animint/git/commits{/sha}","comments_url":"https://api.github.com/repos/tdhock/animint/comments{/number}","issue_comment_url":"https://api.github.com/repos/tdhock/animint/issues/comments{/number}","contents_url":"https://api.github.com/repos/tdhock/animint/contents/{+path}","compare_url":"https://api.github.com/repos/tdhock/animint/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tdhock/animint/merges","archive_url":"https://api.github.com/repos/tdhock/animint/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tdhock/animint/downloads","issues_url":"https://api.github.com/repos/tdhock/animint/issues{/number}","pulls_url":"https://api.github.com/repos/tdhock/animint/pulls{/number}","milestones_url":"https://api.github.com/repos/tdhock/animint/milestones{/number}","notifications_url":"https://api.github.com/repos/tdhock/animint/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tdhock/animint/labels{/name}","releases_url":"https://api.github.com/repos/tdhock/animint/releases{/id}","created_at":"2013-02-25T00:27:47Z","updated_at":"2015-06-02T19:36:52Z","pushed_at":"2015-06-10T15:08:51Z","git_url":"git://github.com/tdhock/animint.git","ssh_url":"git@github.com:tdhock/animint.git","clone_url":"https://github.com/tdhock/animint.git","svn_url":"https://github.com/tdhock/animint","homepage":"","size":137765,"stargazers_count":58,"watchers_count":58,"language":"R","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":23,"mirror_url":null,"open_issues_count":11,"forks":23,"open_issues":11,"watchers":58,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/tdhock/animint/pulls/76"},"html":{"href":"https://github.com/tdhock/animint/pull/76"},"issue":{"href":"https://api.github.com/repos/tdhock/animint/issues/76"},"comments":{"href":"https://api.github.com/repos/tdhock/animint/issues/76/comments"},"review_comments":{"href":"https://api.github.com/repos/tdhock/animint/pulls/76/comments"},"review_comment":{"href":"https://api.github.com/repos/tdhock/animint/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/tdhock/animint/pulls/76/commits"},"statuses":{"href":"https://api.github.com/repos/tdhock/animint/statuses/728d97febaa743aed71ba72e22ab1a818640ef74"}}}}