{"action":"created","issue":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/2267","labels_url":"https://api.github.com/repos/snowplow/snowplow/issues/2267/labels{/name}","comments_url":"https://api.github.com/repos/snowplow/snowplow/issues/2267/comments","events_url":"https://api.github.com/repos/snowplow/snowplow/issues/2267/events","html_url":"https://github.com/snowplow/snowplow/issues/2267","id":123799208,"number":2267,"title":"OOM \"Java heap space\" error while snowplow-storage-loader download phase","user":{"login":"snowindy","id":614348,"avatar_url":"https://avatars.githubusercontent.com/u/614348?v=3","gravatar_id":"","url":"https://api.github.com/users/snowindy","html_url":"https://github.com/snowindy","followers_url":"https://api.github.com/users/snowindy/followers","following_url":"https://api.github.com/users/snowindy/following{/other_user}","gists_url":"https://api.github.com/users/snowindy/gists{/gist_id}","starred_url":"https://api.github.com/users/snowindy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snowindy/subscriptions","organizations_url":"https://api.github.com/users/snowindy/orgs","repos_url":"https://api.github.com/users/snowindy/repos","events_url":"https://api.github.com/users/snowindy/events{/privacy}","received_events_url":"https://api.github.com/users/snowindy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-24T12:31:22Z","updated_at":"2015-12-24T13:43:19Z","closed_at":null,"body":"I use Snowplow r73.\r\nThere are around 6Gb (462 part files max up to 75 Mb) of shredded events in shredded/good s3 folder.\r\nI try to load them to PG database with storage loaded. It failed with OOM Java heap space.\r\nI skipped all steps except download and see problem persists, this means OOM happens in download phase.\r\nI tried to tweak java heap opts to extend heap up to 1.5 GB which did not solve this problem, OOM still happens.\r\n\r\n```\r\nexport java_args=\"-Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=memdump.hprof\"\r\n./snowplow-storage-loader -c config.yml -s delete,load,shred,analyze,archive_enriched\r\n```\r\n\r\nI see huge binary objects in java heap. Feels like something gets into memory for no reason.\r\nAny way to overcome this?\r\n\r\nMemory analyzer leak suspect output:\r\n\r\n```\r\nOne instance of \"org.jruby.RubyHash\" loaded by \"java.net.URLClassLoader @ 0xe06ce4e0\" occupies 40,901,464 (10.92%) bytes. The memory is accumulated in one instance of \"byte[]\" loaded by \"<system class loader>\".\r\n\r\nKeywords\r\njava.net.URLClassLoader @ 0xe06ce4e0\r\norg.jruby.RubyHash\r\nbyte[]\r\n\r\n\r\nClass Name\r\n\r\nShallow Heap\r\n\r\nRetained Heap\r\n\r\n\r\nbyte[40894464] @ 0xf0b38630 e100b94f-86df-4611-bf6e-f4a35a453c60.web.2015-12-23 14:50:00.112.2015-12-15 18:01:01.000..page_view.d64452c8-8eff-4cd6-9b62-ca2fa2df97c4..Studio-Common-Http.Studio-0.9.5.2.clj-1.1.0-tom-0.2.0.hadoop-1.3.0-common-0.18.0.f718a4ed-5a48-4aaa-bfc0-6a05bb26df06....\r\n 40,894,480 40,894,480 \r\n\\bytes org.jruby.util.ByteList @ 0xe3f35810\r\n 40 40,894,520 \r\n.\\value org.jruby.RubyString @ 0xe3ebc078\r\n 40 40,894,560 \r\n..\\value org.jruby.RubyHash$RubyHashEntry @ 0xe3ebc050\r\n 40 40,894,600 \r\n...\\[5] org.jruby.RubyHash$RubyHashEntry[11] @ 0xe3ebac68\r\n 64 64 \r\n....\\table org.jruby.RubyHash @ 0xe3ebac30\r\n 56 40,899,488 \r\n.....\\value org.jruby.RubyHash$RubyHashEntry @ 0xe3ebac08\r\n 40 40,899,528 \r\n......\\[4] org.jruby.RubyHash$RubyHashEntry[11] @ 0xe3bff368\r\n 64 64 \r\n.......\\table org.jruby.RubyHash @ 0xe3bff330\r\n 56 40,901,464 \r\n........+<Java Local> java.lang.Thread @ 0xe37db878 Ruby-0-Thread-3: /opt/snowplow-r73/snowplow-storage-loader!/gems/sluice-0.2.2/lib/sluice/storage/s3/s3.rb:426 Busy Monitor, Thread\r\n 104 6,864 \r\n........+[0] java.lang.Object[7] @ 0xe3bf1498 »\r\n 48 127,104 \r\n........+variableValueZero org.jruby.runtime.scope.OneVarDynamicScope @ 0xe3f11c80 »\r\n 32 32 \r\n\r\n\r\n\r\n Accumulated Objects in Dominator Tree \r\n\r\n\r\n\r\nClass Name\r\n\r\nShallow Heap\r\n\r\nRetained Heap\r\n\r\nPercentage\r\n\r\n\r\norg.jruby.RubyHash @ 0xe3bff330\r\n 56 40,901,464 10.92% \r\n\\org.jruby.RubyHash$RubyHashEntry @ 0xe3ebac08\r\n 40 40,899,528 10.92% \r\n.\\org.jruby.RubyHash @ 0xe3ebac30\r\n 56 40,899,488 10.92% \r\n..\\org.jruby.RubyHash$RubyHashEntry @ 0xe3ebc050\r\n 40 40,894,600 10.92% \r\n...\\org.jruby.RubyString @ 0xe3ebc078\r\n 40 40,894,560 10.92% \r\n....\\org.jruby.util.ByteList @ 0xe3f35810\r\n 40 40,894,520 10.92% \r\n.....\\byte[40894464] @ 0xf0b38630 e100b94f-86df-4611-bf6e-f4a35a453c60.web.2015-12-23 14:50:00.112.2015-12-15 18:01:01.000..page_view.d64452c8-8eff-4cd6-9b62-ca2fa2df97c4..Studio-Common-Http.Studio-0.9.5.2.clj-1.1.0-tom-0.2.0.hadoop-1.3.0-common-0.18.0.f718a4ed-5a48-4aaa-bfc0-6a05bb26df06....\r\n 40,894,480 40,894,480 10.92% \r\n\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/comments/167114167","html_url":"https://github.com/snowplow/snowplow/issues/2267#issuecomment-167114167","issue_url":"https://api.github.com/repos/snowplow/snowplow/issues/2267","id":167114167,"user":{"login":"alexanderdean","id":200511,"avatar_url":"https://avatars.githubusercontent.com/u/200511?v=3","gravatar_id":"","url":"https://api.github.com/users/alexanderdean","html_url":"https://github.com/alexanderdean","followers_url":"https://api.github.com/users/alexanderdean/followers","following_url":"https://api.github.com/users/alexanderdean/following{/other_user}","gists_url":"https://api.github.com/users/alexanderdean/gists{/gist_id}","starred_url":"https://api.github.com/users/alexanderdean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexanderdean/subscriptions","organizations_url":"https://api.github.com/users/alexanderdean/orgs","repos_url":"https://api.github.com/users/alexanderdean/repos","events_url":"https://api.github.com/users/alexanderdean/events{/privacy}","received_events_url":"https://api.github.com/users/alexanderdean/received_events","type":"User","site_admin":false},"created_at":"2015-12-24T13:43:19Z","updated_at":"2015-12-24T13:43:19Z","body":"Hi Eugene,\n\nFor a quick fix:\n\n   1. Precede your StorageLoader run with an $ aws s3 cp command to pull\n   down the files locally\n   2. Run your StorageLoader with --skip download\n\nIn terms of the underlying problem: StorageLoader uses a library we wrote\ncalled Sluice, which is a multi-threaded wrapper around the Fog library for\nworking with S3. The bug could be in either Sluice or Fog.\n\nNote that the Postgres load architecture for Snowplow is pretty archaic at\nthis point: all enriched events are downloaded from S3 to the orchestration\nbox running StorageLoader, and then those event files are streamed into a\nPostgres instance. It's the only place where there is actual dataflow\nthrough the orchestration box (the Redshift load process is fully remote).\n\nA much cleaner architecture would be to load the events straight from HDFS\ninto Postgres from the EMR cluster itself. We have a lot of refactoring\nplanned on our database load architectures for next year.\n\nCheers,\n\nAlex\n\nOn Thu, Dec 24, 2015 at 2:31 PM, Eugene <notifications@github.com> wrote:\n\n> I use Snowplow r73\n> There are around 6Gb (462 part files max up to 75 Mb) of shredded events\n> in shredded/good s3 folder\n> I try to load them to PG database with storage loaded It failed with OOM\n> Java heap space\n> I skipped all steps except download and see problem persists, this means\n> OOM happens in download phase\n> I tried to tweak java heap opts to extend heap up to 15 GB which did not\n> solve this problem, OOM still happens\n>\n> export java_args=\"-Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=memdumphprof\"\n> /snowplow-storage-loader -c configyml -s delete,load,shred,analyze,archive_enriched\n>\n> I see huge binary objects in java heap Feels like something gets into\n> memory for no reason\n> Any way to overcome this?\n>\n> Memory analyzer leak suspect output:\n>\n> One instance of \"orgjrubyRubyHash\" loaded by \"javanetURLClassLoader @ 0xe06ce4e0\" occupies 40,901,464 (1092%) bytes The memory is accumulated in one instance of \"byte[]\" loaded by \"<system class loader>\"\n>\n> Keywords\n> javanetURLClassLoader @ 0xe06ce4e0\n> orgjrubyRubyHash\n> byte[]\n>\n>\n> Class Name\n>\n> Shallow Heap\n>\n> Retained Heap\n>\n>\n> byte[40894464] @ 0xf0b38630 e100b94f-86df-4611-bf6e-f4a35a453c60web2015-12-23 14:50:001122015-12-15 18:01:01000page_viewd64452c8-8eff-4cd6-9b62-ca2fa2df97c4Studio-Common-HttpStudio-0952clj-110-tom-020hadoop-130-common-0180f718a4ed-5a48-4aaa-bfc0-6a05bb26df06\n>  40,894,480 40,894,480\n> \\bytes orgjrubyutilByteList @ 0xe3f35810\n>  40 40,894,520\n> \\value orgjrubyRubyString @ 0xe3ebc078\n>  40 40,894,560\n> \\value orgjrubyRubyHash$RubyHashEntry @ 0xe3ebc050\n>  40 40,894,600\n> \\[5] orgjrubyRubyHash$RubyHashEntry[11] @ 0xe3ebac68\n>  64 64\n> \\table orgjrubyRubyHash @ 0xe3ebac30\n>  56 40,899,488\n> \\value orgjrubyRubyHash$RubyHashEntry @ 0xe3ebac08\n>  40 40,899,528\n> \\[4] orgjrubyRubyHash$RubyHashEntry[11] @ 0xe3bff368\n>  64 64\n> \\table orgjrubyRubyHash @ 0xe3bff330\n>  56 40,901,464\n> +<Java Local> javalangThread @ 0xe37db878 Ruby-0-Thread-3: /opt/snowplow-r73/snowplow-storage-loader!/gems/sluice-022/lib/sluice/storage/s3/s3rb:426 Busy Monitor, Thread\n>  104 6,864\n> +[0] javalangObject[7] @ 0xe3bf1498 »\n>  48 127,104\n> +variableValueZero orgjrubyruntimescopeOneVarDynamicScope @ 0xe3f11c80 »\n>  32 32\n>\n>\n>\n>  Accumulated Objects in Dominator Tree\n>\n>\n>\n> Class Name\n>\n> Shallow Heap\n>\n> Retained Heap\n>\n> Percentage\n>\n>\n> orgjrubyRubyHash @ 0xe3bff330\n>  56 40,901,464 1092%\n> \\orgjrubyRubyHash$RubyHashEntry @ 0xe3ebac08\n>  40 40,899,528 1092%\n> \\orgjrubyRubyHash @ 0xe3ebac30\n>  56 40,899,488 1092%\n> \\orgjrubyRubyHash$RubyHashEntry @ 0xe3ebc050\n>  40 40,894,600 1092%\n> \\orgjrubyRubyString @ 0xe3ebc078\n>  40 40,894,560 1092%\n> \\orgjrubyutilByteList @ 0xe3f35810\n>  40 40,894,520 1092%\n> \\byte[40894464] @ 0xf0b38630 e100b94f-86df-4611-bf6e-f4a35a453c60web2015-12-23 14:50:001122015-12-15 18:01:01000page_viewd64452c8-8eff-4cd6-9b62-ca2fa2df97c4Studio-Common-HttpStudio-0952clj-110-tom-020hadoop-130-common-0180f718a4ed-5a48-4aaa-bfc0-6a05bb26df06\n>  40,894,480 40,894,480 1092%\n>\n>\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/snowplow/snowplow/issues/2267>.\n>\n\n\n\n-- \nCo-founder\nSnowplow Analytics <http://snowplowanalytics.com/>\nThe Roma Building, 32-38 Scrutton Street, London EC2A 4RQ, United Kingdom\n+44 (0)203 589 6116\n+44 7881 622 925\n@alexcrdean <https://twitter.com/alexcrdean>\n"}}