{"action":"opened","issue":{"url":"https://api.github.com/repos/linkedin/Cubert/issues/8","labels_url":"https://api.github.com/repos/linkedin/Cubert/issues/8/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/Cubert/issues/8/comments","events_url":"https://api.github.com/repos/linkedin/Cubert/issues/8/events","html_url":"https://github.com/linkedin/Cubert/issues/8","id":91233362,"number":8,"title":"BLOCKGEN causes Java Heap Size","user":{"login":"OopsOutOfMemory","id":6489401,"avatar_url":"https://avatars.githubusercontent.com/u/6489401?v=3","gravatar_id":"","url":"https://api.github.com/users/OopsOutOfMemory","html_url":"https://github.com/OopsOutOfMemory","followers_url":"https://api.github.com/users/OopsOutOfMemory/followers","following_url":"https://api.github.com/users/OopsOutOfMemory/following{/other_user}","gists_url":"https://api.github.com/users/OopsOutOfMemory/gists{/gist_id}","starred_url":"https://api.github.com/users/OopsOutOfMemory/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OopsOutOfMemory/subscriptions","organizations_url":"https://api.github.com/users/OopsOutOfMemory/orgs","repos_url":"https://api.github.com/users/OopsOutOfMemory/repos","events_url":"https://api.github.com/users/OopsOutOfMemory/events{/privacy}","received_events_url":"https://api.github.com/users/OopsOutOfMemory/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-06-26T11:59:48Z","updated_at":"2015-06-26T11:59:48Z","closed_at":null,"body":"Hi, @suvodeep-pyne @mparkhe \r\nWhen I perform a BLOCKGEN operation, at final reduce, the Java Heap Size exception throws.\r\nI increased the REDUCE NUMBER seems not work.\r\n\r\n```\r\n2015-06-26 16:14:56,215 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 4 in-memory map-outputs and 5 on-disk map-outputs\r\n2015-06-26 16:14:56,217 INFO [main] org.apache.hadoop.mapred.Merger: Merging 4 sorted segments\r\n2015-06-26 16:14:56,217 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 142558662 bytes\r\n2015-06-26 16:14:57,234 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 4 segments, 142558706 bytes to disk to satisfy reduce memory limit\r\n2015-06-26 16:14:57,235 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 6 files, 789110010 bytes from disk\r\n2015-06-26 16:14:57,236 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce\r\n2015-06-26 16:14:57,236 INFO [main] org.apache.hadoop.mapred.Merger: Merging 6 sorted segments\r\n2015-06-26 16:14:57,243 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy]\r\n2015-06-26 16:14:57,243 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 6 segments left of total size: 3293450894 bytes\r\n2015-06-26 16:14:57,605 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords\r\n2015-06-26 16:15:44,841 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space\r\n\tat com.linkedin.cubert.memory.PagedByteArray.ensureCapacity(PagedByteArray.java:192)\r\n\tat com.linkedin.cubert.memory.PagedByteArray.write(PagedByteArray.java:141)\r\n\tat com.linkedin.cubert.memory.PagedByteArrayOutputStream.write(PagedByteArrayOutputStream.java:67)\r\n\tat java.io.DataOutputStream.write(DataOutputStream.java:107)\r\n\tat java.io.DataOutputStream.writeUTF(DataOutputStream.java:401)\r\n\tat java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)\r\n\tat org.apache.pig.data.utils.SedesHelper.writeChararray(SedesHelper.java:66)\r\n\tat org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:580)\r\n\tat org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:462)\r\n\tat org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)\r\n\tat org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:650)\r\n\tat org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:470)\r\n\tat org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:40)\r\n\tat com.linkedin.cubert.io.DefaultTupleSerializer.serialize(DefaultTupleSerializer.java:41)\r\n\tat com.linkedin.cubert.io.DefaultTupleSerializer.serialize(DefaultTupleSerializer.java:28)\r\n\tat com.linkedin.cubert.utils.SerializedTupleStore.addToStore(SerializedTupleStore.java:118)\r\n\tat com.linkedin.cubert.block.CreateBlockOperator$StoredBlock.<init>(CreateBlockOperator.java:145)\r\n\tat com.linkedin.cubert.block.CreateBlockOperator.createBlock(CreateBlockOperator.java:536)\r\n\tat com.linkedin.cubert.block.CreateBlockOperator.next(CreateBlockOperator.java:488)\r\n\tat com.linkedin.cubert.plan.physical.PhaseExecutor.prepareOperatorChain(PhaseExecutor.java:261)\r\n\tat com.linkedin.cubert.plan.physical.PhaseExecutor.<init>(PhaseExecutor.java:111)\r\n\tat com.linkedin.cubert.plan.physical.CubertReducer.run(CubertReducer.java:68)\r\n\tat org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)\r\n\tat org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)\r\n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\r\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)\r\n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)\r\n```"}}