{"action":"created","comment":{"url":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/comments/24131327","id":24131327,"diff_hunk":"@@ -39,16 +40,19 @@\n public class MemoryManager {\n   private static final Log LOG = Log.getLog(MemoryManager.class);\n   static final float DEFAULT_MEMORY_POOL_RATIO = 0.95f;\n+  static final long DEFAULT_MIN_MEMORY_ALLOCATION = 0;","path":"parquet-hadoop/src/main/java/parquet/hadoop/MemoryManager.java","position":12,"original_position":12,"commit_id":"8f6cff1515599a94e49cde978a21c57849704cbc","original_commit_id":"8f6cff1515599a94e49cde978a21c57849704cbc","user":{"login":"danielcweeks","id":4925077,"avatar_url":"https://avatars.githubusercontent.com/u/4925077?v=3","gravatar_id":"","url":"https://api.github.com/users/danielcweeks","html_url":"https://github.com/danielcweeks","followers_url":"https://api.github.com/users/danielcweeks/followers","following_url":"https://api.github.com/users/danielcweeks/following{/other_user}","gists_url":"https://api.github.com/users/danielcweeks/gists{/gist_id}","starred_url":"https://api.github.com/users/danielcweeks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielcweeks/subscriptions","organizations_url":"https://api.github.com/users/danielcweeks/orgs","repos_url":"https://api.github.com/users/danielcweeks/repos","events_url":"https://api.github.com/users/danielcweeks/events{/privacy}","received_events_url":"https://api.github.com/users/danielcweeks/received_events","type":"User","site_admin":false},"body":"We could set it to something like 10MB, which is my best guess (without any empirical evidence).  We give tasks 1.5 GB which only safely allows for about 10 writers per task with the default 128MB Row Group setting.  A 10MB lower bound would probably push us to 100 writers, which would likely account for most situations.\r\n\r\nAny other suggestions are welcome.","created_at":"2015-02-04T23:43:07Z","updated_at":"2015-02-04T23:43:07Z","html_url":"https://github.com/apache/incubator-parquet-mr/pull/115#discussion_r24131327","pull_request_url":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/115","_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/comments/24131327"},"html":{"href":"https://github.com/apache/incubator-parquet-mr/pull/115#discussion_r24131327"},"pull_request":{"href":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/115"}}},"pull_request":{"url":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/115","id":28605054,"html_url":"https://github.com/apache/incubator-parquet-mr/pull/115","diff_url":"https://github.com/apache/incubator-parquet-mr/pull/115.diff","patch_url":"https://github.com/apache/incubator-parquet-mr/pull/115.patch","issue_url":"https://api.github.com/repos/apache/incubator-parquet-mr/issues/115","number":115,"state":"open","locked":false,"title":"Added lower bound to memory manager resize","user":{"login":"danielcweeks","id":4925077,"avatar_url":"https://avatars.githubusercontent.com/u/4925077?v=3","gravatar_id":"","url":"https://api.github.com/users/danielcweeks","html_url":"https://github.com/danielcweeks","followers_url":"https://api.github.com/users/danielcweeks/followers","following_url":"https://api.github.com/users/danielcweeks/following{/other_user}","gists_url":"https://api.github.com/users/danielcweeks/gists{/gist_id}","starred_url":"https://api.github.com/users/danielcweeks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielcweeks/subscriptions","organizations_url":"https://api.github.com/users/danielcweeks/orgs","repos_url":"https://api.github.com/users/danielcweeks/repos","events_url":"https://api.github.com/users/danielcweeks/events{/privacy}","received_events_url":"https://api.github.com/users/danielcweeks/received_events","type":"User","site_admin":false},"body":"PARQUET-177","created_at":"2015-02-04T01:05:43Z","updated_at":"2015-02-04T23:43:07Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ca8e35e156fcb071fdf34fd4e6e9ea93914e3449","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/115/commits","review_comments_url":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/115/comments","review_comment_url":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/comments/{number}","comments_url":"https://api.github.com/repos/apache/incubator-parquet-mr/issues/115/comments","statuses_url":"https://api.github.com/repos/apache/incubator-parquet-mr/statuses/8f6cff1515599a94e49cde978a21c57849704cbc","head":{"label":"danielcweeks:memory-manager-limit","ref":"memory-manager-limit","sha":"8f6cff1515599a94e49cde978a21c57849704cbc","user":{"login":"danielcweeks","id":4925077,"avatar_url":"https://avatars.githubusercontent.com/u/4925077?v=3","gravatar_id":"","url":"https://api.github.com/users/danielcweeks","html_url":"https://github.com/danielcweeks","followers_url":"https://api.github.com/users/danielcweeks/followers","following_url":"https://api.github.com/users/danielcweeks/following{/other_user}","gists_url":"https://api.github.com/users/danielcweeks/gists{/gist_id}","starred_url":"https://api.github.com/users/danielcweeks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielcweeks/subscriptions","organizations_url":"https://api.github.com/users/danielcweeks/orgs","repos_url":"https://api.github.com/users/danielcweeks/repos","events_url":"https://api.github.com/users/danielcweeks/events{/privacy}","received_events_url":"https://api.github.com/users/danielcweeks/received_events","type":"User","site_admin":false},"repo":{"id":21009481,"name":"incubator-parquet-mr","full_name":"danielcweeks/incubator-parquet-mr","owner":{"login":"danielcweeks","id":4925077,"avatar_url":"https://avatars.githubusercontent.com/u/4925077?v=3","gravatar_id":"","url":"https://api.github.com/users/danielcweeks","html_url":"https://github.com/danielcweeks","followers_url":"https://api.github.com/users/danielcweeks/followers","following_url":"https://api.github.com/users/danielcweeks/following{/other_user}","gists_url":"https://api.github.com/users/danielcweeks/gists{/gist_id}","starred_url":"https://api.github.com/users/danielcweeks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielcweeks/subscriptions","organizations_url":"https://api.github.com/users/danielcweeks/orgs","repos_url":"https://api.github.com/users/danielcweeks/repos","events_url":"https://api.github.com/users/danielcweeks/events{/privacy}","received_events_url":"https://api.github.com/users/danielcweeks/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/danielcweeks/incubator-parquet-mr","description":"Mirror of Apache Parquet","fork":true,"url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr","forks_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/forks","keys_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/keys{/key_id}","collaborators_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/teams","hooks_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/hooks","issue_events_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/issues/events{/number}","events_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/events","assignees_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/assignees{/user}","branches_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/branches{/branch}","tags_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/tags","blobs_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/git/refs{/sha}","trees_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/git/trees{/sha}","statuses_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/statuses/{sha}","languages_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/languages","stargazers_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/stargazers","contributors_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/contributors","subscribers_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/subscribers","subscription_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/subscription","commits_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/commits{/sha}","git_commits_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/git/commits{/sha}","comments_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/comments{/number}","issue_comment_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/issues/comments/{number}","contents_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/contents/{+path}","compare_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/compare/{base}...{head}","merges_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/merges","archive_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/downloads","issues_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/issues{/number}","pulls_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/pulls{/number}","milestones_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/milestones{/number}","notifications_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/labels{/name}","releases_url":"https://api.github.com/repos/danielcweeks/incubator-parquet-mr/releases{/id}","created_at":"2014-06-19T17:34:54Z","updated_at":"2014-08-11T15:58:32Z","pushed_at":"2015-02-04T01:04:16Z","git_url":"git://github.com/danielcweeks/incubator-parquet-mr.git","ssh_url":"git@github.com:danielcweeks/incubator-parquet-mr.git","clone_url":"https://github.com/danielcweeks/incubator-parquet-mr.git","svn_url":"https://github.com/danielcweeks/incubator-parquet-mr","homepage":null,"size":11852,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"80417356f04c5ee1cd6f636e9b043db3f2de24f2","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":20675636,"name":"incubator-parquet-mr","full_name":"apache/incubator-parquet-mr","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/incubator-parquet-mr","description":"Mirror of Apache Parquet","fork":false,"url":"https://api.github.com/repos/apache/incubator-parquet-mr","forks_url":"https://api.github.com/repos/apache/incubator-parquet-mr/forks","keys_url":"https://api.github.com/repos/apache/incubator-parquet-mr/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/incubator-parquet-mr/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/incubator-parquet-mr/teams","hooks_url":"https://api.github.com/repos/apache/incubator-parquet-mr/hooks","issue_events_url":"https://api.github.com/repos/apache/incubator-parquet-mr/issues/events{/number}","events_url":"https://api.github.com/repos/apache/incubator-parquet-mr/events","assignees_url":"https://api.github.com/repos/apache/incubator-parquet-mr/assignees{/user}","branches_url":"https://api.github.com/repos/apache/incubator-parquet-mr/branches{/branch}","tags_url":"https://api.github.com/repos/apache/incubator-parquet-mr/tags","blobs_url":"https://api.github.com/repos/apache/incubator-parquet-mr/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/incubator-parquet-mr/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/incubator-parquet-mr/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/incubator-parquet-mr/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/incubator-parquet-mr/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/incubator-parquet-mr/languages","stargazers_url":"https://api.github.com/repos/apache/incubator-parquet-mr/stargazers","contributors_url":"https://api.github.com/repos/apache/incubator-parquet-mr/contributors","subscribers_url":"https://api.github.com/repos/apache/incubator-parquet-mr/subscribers","subscription_url":"https://api.github.com/repos/apache/incubator-parquet-mr/subscription","commits_url":"https://api.github.com/repos/apache/incubator-parquet-mr/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/incubator-parquet-mr/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/incubator-parquet-mr/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/incubator-parquet-mr/issues/comments/{number}","contents_url":"https://api.github.com/repos/apache/incubator-parquet-mr/contents/{+path}","compare_url":"https://api.github.com/repos/apache/incubator-parquet-mr/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/incubator-parquet-mr/merges","archive_url":"https://api.github.com/repos/apache/incubator-parquet-mr/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/incubator-parquet-mr/downloads","issues_url":"https://api.github.com/repos/apache/incubator-parquet-mr/issues{/number}","pulls_url":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/incubator-parquet-mr/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/incubator-parquet-mr/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/incubator-parquet-mr/labels{/name}","releases_url":"https://api.github.com/repos/apache/incubator-parquet-mr/releases{/id}","created_at":"2014-06-10T07:00:07Z","updated_at":"2015-02-03T20:53:54Z","pushed_at":"2015-02-03T20:53:54Z","git_url":"git://github.com/apache/incubator-parquet-mr.git","ssh_url":"git@github.com:apache/incubator-parquet-mr.git","clone_url":"https://github.com/apache/incubator-parquet-mr.git","svn_url":"https://github.com/apache/incubator-parquet-mr","homepage":null,"size":79192,"stargazers_count":59,"watchers_count":59,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":84,"mirror_url":"git://git.apache.org/incubator-parquet-mr.git","open_issues_count":32,"forks":84,"open_issues":32,"watchers":59,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/115"},"html":{"href":"https://github.com/apache/incubator-parquet-mr/pull/115"},"issue":{"href":"https://api.github.com/repos/apache/incubator-parquet-mr/issues/115"},"comments":{"href":"https://api.github.com/repos/apache/incubator-parquet-mr/issues/115/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/115/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/apache/incubator-parquet-mr/pulls/115/commits"},"statuses":{"href":"https://api.github.com/repos/apache/incubator-parquet-mr/statuses/8f6cff1515599a94e49cde978a21c57849704cbc"}}}}