{"action":"created","issue":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/771","labels_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/771/labels{/name}","comments_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/771/comments","events_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/771/events","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/771","id":58448624,"number":771,"title":"freebayes joint calling memory exceeded","user":{"login":"dwaggott","id":3149944,"avatar_url":"https://avatars.githubusercontent.com/u/3149944?v=3","gravatar_id":"","url":"https://api.github.com/users/dwaggott","html_url":"https://github.com/dwaggott","followers_url":"https://api.github.com/users/dwaggott/followers","following_url":"https://api.github.com/users/dwaggott/following{/other_user}","gists_url":"https://api.github.com/users/dwaggott/gists{/gist_id}","starred_url":"https://api.github.com/users/dwaggott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwaggott/subscriptions","organizations_url":"https://api.github.com/users/dwaggott/orgs","repos_url":"https://api.github.com/users/dwaggott/repos","events_url":"https://api.github.com/users/dwaggott/events{/privacy}","received_events_url":"https://api.github.com/users/dwaggott/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-21T07:44:29Z","updated_at":"2015-02-21T19:18:48Z","closed_at":null,"body":"I thought I'd start a new issue regarding processes being killed and jobs going idle. The last run #741 ended up going to completion, thank you.\r\n\r\nI'm trying to run the freebayes joint caller using bams and vcfs from #741. The previous run actually did pooled and not joint calling via `jointcaller: freebayes-joint`.\r\n\r\nThe slurm scheduler killed some of the processes due to memory at what I think is the freebayes step. I previously increased the memory to 4g from 2g for freebayes. I'm now upping to 8g, does that seem rational?\r\n\r\n\r\n```\r\nProcessing reference #83 (GL000225.1)\r\nProcessing reference #84 (GL000192.1)\r\nslurmstepd: *** JOB 1616873 CANCELLED AT 2015-02-20T13:07:23 *** on sh-5-8\r\nslurmstepd: Exceeded step memory limit at some point. oom-killer likely killed a process.\r\nslurmstepd: Exceeded job memory limit at some point. oom-killer likely killed a process.\r\nslurmstepd: _slurm_cgroup_destroy: problem deleting step cgroup path /cgroup/freezer/slurm/uid_261851/job_1616873/step_batch: Device or resource busy\r\n```\r\n\r\nHowever in the engine logs I do see some sambamba issues. I'm using the latest dev code with updated tools.\r\n\r\n```\r\nProcessing reference #19 (19)\r\nProcessing reference #20 (20)\r\nProcessing reference #21 (21)\r\nsambamba-depth:  (Broken pipe)\r\nProcessing reference #1 (1)\r\nProcessing reference #2 (2)\r\nProcessing reference #3 (3)\r\nProcessing reference #4 (4)\r\nProcessing reference #5 (5)\r\n```\r\n\r\nThe submission script also shows a memory issue. It's using 8g and I'm now bumping to 12g.\r\n\r\n```\r\n[2015-02-20T21:27Z] sh-5-15.local: Could not find any mapped reads in target region 1:109707980..109708009\r\n[2015-02-20T21:28Z] sh-3-21.local: Could not find any mapped reads in target region 11:125313108..125313163\r\n[2015-02-20T21:27Z] sh-3-7.local: Could not find any mapped reads in target region 2:32211131..32211188\r\n/var/spool/slurmd/job1616871/slurm_script: line 50: 10499 Killed                  bcbio_nextgen.py --timeout 10000 --retries 2 ../config/project_n143-fb-joint.yaml -t ipython -n 320 -q normal -s slurm -r \"timelimit=2-00:00:00\"\r\nslurmstepd: Exceeded step memory limit at some point. Step may have been partially swapped out to disk.\r\nslurmstepd: _slurm_cgroup_destroy: problem deleting step cgroup path /cgroup/freezer/slurm/uid_261851/job_1616871/step_batch: Device or resource busy\r\n```"},"comment":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/comments/75387646","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/771#issuecomment-75387646","issue_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/771","id":75387646,"user":{"login":"chapmanb","id":39391,"avatar_url":"https://avatars.githubusercontent.com/u/39391?v=3","gravatar_id":"","url":"https://api.github.com/users/chapmanb","html_url":"https://github.com/chapmanb","followers_url":"https://api.github.com/users/chapmanb/followers","following_url":"https://api.github.com/users/chapmanb/following{/other_user}","gists_url":"https://api.github.com/users/chapmanb/gists{/gist_id}","starred_url":"https://api.github.com/users/chapmanb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chapmanb/subscriptions","organizations_url":"https://api.github.com/users/chapmanb/orgs","repos_url":"https://api.github.com/users/chapmanb/repos","events_url":"https://api.github.com/users/chapmanb/events{/privacy}","received_events_url":"https://api.github.com/users/chapmanb/received_events","type":"User","site_admin":false},"created_at":"2015-02-21T19:18:48Z","updated_at":"2015-02-21T19:18:48Z","body":"Daryl;\r\nSorry about the issues. Trying to tackle from top to bottom:\r\n\r\n- Is the worker engine being killed, or the main bcbio process? I wouldn't expect adding more samples to take more memory here, so it is possible that killed message is from the main process?\r\n\r\n- Those broken pipes are expected. It will report this if you stream a BAM file and only view the head to look up information, so I wouldn't worry about them.\r\n\r\n- If you've increased the number of samples, the main process and controller will require additional memory since we need to manage more messages for all the items to process. This is something we hope will be better/more scalable in the future using a CWL-based workflow processing pipeline that uses databases to manage message instead of memory like IPython.\r\n\r\nHope this helps."}}