{"action":"opened","issue":{"url":"https://api.github.com/repos/Genomon-Project/Genomon/issues/58","labels_url":"https://api.github.com/repos/Genomon-Project/Genomon/issues/58/labels{/name}","comments_url":"https://api.github.com/repos/Genomon-Project/Genomon/issues/58/comments","events_url":"https://api.github.com/repos/Genomon-Project/Genomon/issues/58/events","html_url":"https://github.com/Genomon-Project/Genomon/issues/58","id":99180222,"number":58,"title":"EXITING: fatal error trying to allocate genome arrays, exception thrown: std::bad_alloc","user":{"login":"roko-t","id":13216812,"avatar_url":"https://avatars.githubusercontent.com/u/13216812?v=3","gravatar_id":"","url":"https://api.github.com/users/roko-t","html_url":"https://github.com/roko-t","followers_url":"https://api.github.com/users/roko-t/followers","following_url":"https://api.github.com/users/roko-t/following{/other_user}","gists_url":"https://api.github.com/users/roko-t/gists{/gist_id}","starred_url":"https://api.github.com/users/roko-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/roko-t/subscriptions","organizations_url":"https://api.github.com/users/roko-t/orgs","repos_url":"https://api.github.com/users/roko-t/repos","events_url":"https://api.github.com/users/roko-t/events{/privacy}","received_events_url":"https://api.github.com/users/roko-t/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-08-05T11:13:10Z","updated_at":"2015-08-05T11:13:10Z","closed_at":null,"body":"32GBでqloginして実行\r\nrun.shのgenomon.py 　のプロセス関係のオプションは以下。\r\n           --jobs 200 \\\r\n           --verbose 1 \\\r\n           --drmaa \\\r\n           --use_threads\r\n\r\nRead from SAindex: genomeSAindexNbases=14  nSAi=357913940\r\nGenome file size: 3220476720 bytes; state: good=1 eof=0 fail=0 bad=0\r\nSA file size: 24496209986 bytes; state: good=1 eof=0 fail=0 bad=0\r\nnGenome=3220476720;  nSAbyte=24496209986\r\nGstrandBit=32   SA number of indices=5938475148\r\n\r\nEXITING: fatal error trying to allocate genome arrays, exception thrown: std::bad_alloc\r\nPossible cause 1: not enough RAM. Check if you have enough RAM 31282560594 bytes\r\nPossible cause 2: not enough virtual memory allowed with ulimit. SOLUTION: run ulimit -v 31282560594\r\n\r\nAug 05 18:07:48 ...... FATAL ERROR, exiting\r\n＝＝＝\r\n必要とされるmemoryはエラーとなった全件同じです\r\nulimit -v 31282560594　　　<= 29.13 GB\r\nでstar　のqsub 時のreq_mem 32GBを超えてはいません\r\n100件以上は正常終了するものの1000件以上が同じエラーになるといった状況です。\r\n\r\n正常終了する場合はEXITINGメッセージのところに代わりに以下のログが続きます\r\nShared memory is not used for genomes. Allocated a private copy of the genome.\r\nGenome file size: 3220476720 bytes; state: good=1 eof=0 fail=0 bad=0\r\nLoading Genome ... done! state: good=1 eof=0 fail=0 bad=0; loaded 3220476720 bytes\r\nSA file size: 24496209986 bytes; state: good=1 eof=0 fail=0 bad=0\r\nLoading SA ... done! state: good=1 eof=0 fail=0 bad=0; loaded 24496209986 bytes\r\nLoading SAindex ... done: 1565873616 bytes\r\n\r\nこのファイルのLoadに必要となるメモリは\r\nGenome file size:+ SA file size:+ SAindex で\r\n3220476720 + 24496209986 + 1565873616 = 29282560322 (27.27 GB)です\r\n31282560594 - 29282560322 = 2000000272\r\n\r\n約1200件を一気に実行しようとして成功するのが 150件～200件\r\nその他はすべてallocate 失敗\r\n並行実行の指定方法または上限に問題ありと言うことだと思います\r\n\r\nrun.shの標準出力には以下のエラーが出ていました\r\nstar: Unexpected error: <type 'exceptions.Exception'>.\r\n    <type 'exceptions.Exception'>: star_pipeline.py:298\r\n \r\n＝＝＝＝＝＝＝＝＝＝＝＝\r\nTARGETデータの解析時も1件出ていましたが\r\n再実行したらうまくいったので　するーしていました\r\nTARGETの時は\r\n --jobs 150 　のみで　--drmaa 　--use_threads　指定なしで実行\r\nLoadに必要となるメモリは28753700891　で\r\nやはり\r\n30753701163　-　28753700891 = 2000000272\r\nで同じなので、STARで一定の使用分2000000272を足した値を\r\nulimitで指定するようにとエラーをだしているようです\r\n\r\n尚、\r\n実行時のプロセス数は　--jobs 150 のみの時は　150+3　でしたが\r\n--drmaa --use_threads を使うと　4 で抑制されています\r\n"}}