{"action":"created","issue":{"url":"https://api.github.com/repos/apache/drill/issues/254","labels_url":"https://api.github.com/repos/apache/drill/issues/254/labels{/name}","comments_url":"https://api.github.com/repos/apache/drill/issues/254/comments","events_url":"https://api.github.com/repos/apache/drill/issues/254/events","html_url":"https://github.com/apache/drill/pull/254","id":116854126,"number":254,"title":"DRILL-4053: Reduce metadata cache file size. Save a merged schema ins…","user":{"login":"parthchandra","id":6529136,"avatar_url":"https://avatars.githubusercontent.com/u/6529136?v=3","gravatar_id":"","url":"https://api.github.com/users/parthchandra","html_url":"https://github.com/parthchandra","followers_url":"https://api.github.com/users/parthchandra/followers","following_url":"https://api.github.com/users/parthchandra/following{/other_user}","gists_url":"https://api.github.com/users/parthchandra/gists{/gist_id}","starred_url":"https://api.github.com/users/parthchandra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/parthchandra/subscriptions","organizations_url":"https://api.github.com/users/parthchandra/orgs","repos_url":"https://api.github.com/users/parthchandra/repos","events_url":"https://api.github.com/users/parthchandra/events{/privacy}","received_events_url":"https://api.github.com/users/parthchandra/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-13T21:19:50Z","updated_at":"2015-11-13T23:25:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/drill/pulls/254","html_url":"https://github.com/apache/drill/pull/254","diff_url":"https://github.com/apache/drill/pull/254.diff","patch_url":"https://github.com/apache/drill/pull/254.patch"},"body":"…tead of\r\n\r\nrepeating for every row group. Save maxValue in a row group iff minVal\r\nequals maxVal and null otherwise.\r\n\r\nThis patch has the following changes to reduce the size of a metadata cache file. -\r\n1) Do not write the schema for every row group. Instead merge the schemas and keep only one copy. This change currently does not handle the case where the type of a field mutates.\r\n2) Instead  of min and max values for every field, we keep only the max value and that too only if the min and max values are equal. This allows partition pruning to occur correctly.\r\n3) Use the Jackson afterburner module to deserialize faster\r\n4) Use a string array instead of SchemaPath for simpler deserialization.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/drill/issues/comments/156589078","html_url":"https://github.com/apache/drill/pull/254#issuecomment-156589078","issue_url":"https://api.github.com/repos/apache/drill/issues/254","id":156589078,"user":{"login":"parthchandra","id":6529136,"avatar_url":"https://avatars.githubusercontent.com/u/6529136?v=3","gravatar_id":"","url":"https://api.github.com/users/parthchandra","html_url":"https://github.com/parthchandra","followers_url":"https://api.github.com/users/parthchandra/followers","following_url":"https://api.github.com/users/parthchandra/following{/other_user}","gists_url":"https://api.github.com/users/parthchandra/gists{/gist_id}","starred_url":"https://api.github.com/users/parthchandra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/parthchandra/subscriptions","organizations_url":"https://api.github.com/users/parthchandra/orgs","repos_url":"https://api.github.com/users/parthchandra/repos","events_url":"https://api.github.com/users/parthchandra/events{/privacy}","received_events_url":"https://api.github.com/users/parthchandra/received_events","type":"User","site_admin":false},"created_at":"2015-11-13T23:25:26Z","updated_at":"2015-11-13T23:25:26Z","body":"The last perf run was on 400K parquet files in a 2-layer directory hierarchy:\r\nroot_dir/ \r\n  2015-10-01/\r\n     d_01/\r\n .\r\n .\r\n .\r\n   2015-10-31/\r\n     d_25/\r\n\r\nThat gave us 31x25 directories with 511 parquet files in each dir.\r\n\r\nThe numbers are with a warm file system cache.\r\n\r\nBuild                            Explain Plan     Refresh metadata      Metadata size\r\nMASTER (e275dc5)    145.854 s               225.083 s                  2.9GB\t\t\r\nwith DRILL-4053           35.430 s               202.133\ts                   1.3GB\r\n\r\n\r\nThe Explain Plan query - \"explain plan for select * from root_dir\"\r\nRefresh metadata query - \"refresh table metadata dfs.root_dir \" \r\n\r\n"}}