{"action":"created","issue":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1101","labels_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1101/labels{/name}","comments_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1101/comments","events_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1101/events","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1101","id":115524660,"number":1101,"title":"Load loc2keys DB","user":{"login":"bwubb","id":13784261,"avatar_url":"https://avatars.githubusercontent.com/u/13784261?v=3","gravatar_id":"","url":"https://api.github.com/users/bwubb","html_url":"https://github.com/bwubb","followers_url":"https://api.github.com/users/bwubb/followers","following_url":"https://api.github.com/users/bwubb/following{/other_user}","gists_url":"https://api.github.com/users/bwubb/gists{/gist_id}","starred_url":"https://api.github.com/users/bwubb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwubb/subscriptions","organizations_url":"https://api.github.com/users/bwubb/orgs","repos_url":"https://api.github.com/users/bwubb/repos","events_url":"https://api.github.com/users/bwubb/events{/privacy}","received_events_url":"https://api.github.com/users/bwubb/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-06T15:21:38Z","updated_at":"2015-11-06T21:02:14Z","closed_at":"2015-11-06T21:02:13Z","body":"I am really not sure what has happened. These samples worked the other day prior to me upgrading bcbio to get gemini 0.17.2. At first the issue looks like this.\r\n\r\n```\r\n...\r\n[2015-11-06T02:06Z] INFO  21:06:38,923 GATKRunReport - Uploaded run statistics report to AWS S3\r\n[2015-11-06T02:06Z] bgzip 1205-CRT-21Aplin-2_202131133_243199373-raw-mutect.vcf\r\n[2015-11-06T02:06Z] tabix index 1205-CRT-21Aplin-2_202131133_243199373-raw-mutect.vcf.gz\r\n[2015-11-06T02:06Z] ** 0 just got out of the pool [PID: 23232 | exit code: 0 | exit signal: 0 | core dump: 0]\r\n[2015-11-06T02:06Z] Merge DBs...\r\n[2015-11-06T02:06Z] Load reference coverage...\r\n[2015-11-06T02:06Z] Load loc2keys DB...\r\nTraceback (most recent call last):\r\n  File \"/home/bwubb/bcbio-tools/bin/bcbio_nextgen.py\", line 226, in <module>\r\n    main(**kwargs)\r\n  File \"/home/bwubb/bcbio-tools/bin/bcbio_nextgen.py\", line 43, in main\r\n    run_main(**kwargs)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py\", line 37, in run_main\r\n    fc_dir, run_info_yaml)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py\", line 80, in _run_toplevel\r\n    for xs in pipeline.run(config, run_info_yaml, parallel, dirs, samples):\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py\", line 162, in run\r\n    samples = genotype.parallel_variantcall_region(samples, run_parallel)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/genotype.py\", line 167, in parallel_variantcall_region\r\n    \"vrn_file\", [\"region\", \"sam_ref\", \"config\"]))\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/split.py\", line 32, in grouped_parallel_split_combine\r\n    final_output = parallel_fn(parallel_name, split_args)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py\", line 28, in run_parallel\r\n    return run_multicore(fn, items, config, parallel=parallel)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py\", line 86, in run_multicore\r\n    for data in joblib.Parallel(parallel[\"num_jobs\"])(joblib.delayed(fn)(x) for x in items):\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 664, in __call__\r\n    self.retrieve()\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 547, in retrieve\r\n    raise exception_type(report)\r\njoblib.my_exceptions.JoblibAssertionError: JoblibAssertionError\r\n___________________________________________________________________________\r\nMultiprocessing exception:\r\n    ...........................................................................\r\n/home/bwubb/bcbio-tools/bin/bcbio_nextgen.py in <module>()\r\n    221             if setup_info is None:  # no automated run after setup\r\n    222                 sys.exit(0)\r\n    223             workdir, new_kwargs = setup_info\r\n    224             os.chdir(workdir)\r\n    225             kwargs.update(new_kwargs)\r\n--> 226         main(**kwargs)\r\n    227 \r\n    228 \r\n    229 \r\n    230 \r\n\r\n...........................................................................\r\n/home/bwubb/bcbio-tools/bin/bcbio_nextgen.py in main(**kwargs={'config_file': 'bcbio_system.yaml', 'fc_dir': None, 'parallel': {'cores': 16, 'module': 'bcbio.distributed', 'queue': None, 'resources': [], 'retries': 0, 'run_local': False, 'scheduler': None, 'tag': '', 'timeout': 15, 'type': 'local'}, 'run_info_yaml': '/home/bwubb/NGS/MelanomaTargeted/bcbio/data/1205-CRT-21Aplin/config/MelanomaTargeted.yaml', 'workdir': '/home/bwubb/NGS/MelanomaTargeted/bcbio/data/1205-CRT-21Aplin/work', 'workflow': None})\r\n     38 from bcbio.graph import graph\r\n     39 from bcbio.provenance import programs\r\n     40 from bcbio.pipeline import version\r\n     41 \r\n     42 def main(**kwargs):\r\n---> 43     run_main(**kwargs)\r\n        kwargs = {'config_file': 'bcbio_system.yaml', 'fc_dir': None, 'parallel': {'cores': 16, 'module': 'bcbio.distributed', 'queue': None, 'resources': [], 'retries': 0, 'run_local': False, 'scheduler': None, 'tag': '', 'timeout': 15, 'type': 'local'}, 'run_info_yaml': '/home/bwubb/NGS/MelanomaTargeted/bcbio/data/1205-CRT-21Aplin/config/MelanomaTargeted.yaml', 'workdir': '/home/bwubb/NGS/MelanomaTargeted/bcbio/data/1205-CRT-21Aplin/work', 'workflow': None}\r\n     44 \r\n     45 def parse_cl_args(in_args):\r\n     46     \"\"\"Parse input commandline arguments, handling multiple cases.\r\n     47 \r\n...\r\n```\r\nThis continues for ~500 more lines. Last lines are\r\n```\r\nAssertionError: Input file /home/bwubb/NGS/MelanomaTargeted/bcbio/data/1205-CRT-21Aplin/work/mutect/1/1205-CRT-21Aplin-1_0_22381370-raw-somaticIndels-scalpel-work/variants.3x.indel.vcf not found\r\n___________________________________________________________________________\r\nException in thread Thread-1 (most likely raised during interpreter shutdown):\r\n```\r\n\r\nJust to check I tried to run one without multiple cores (I only did -n 1 I did not edit the bcbio_system.yaml). It was a much shorted error message, but I dont know if it is any more informative.\r\n```\r\nTraceback (most recent call last):\r\n  File \"/home/bwubb/bcbio-tools/bin/bcbio_nextgen.py\", line 226, in <module>\r\n    main(**kwargs)\r\n  File \"/home/bwubb/bcbio-tools/bin/bcbio_nextgen.py\", line 43, in main\r\n    run_main(**kwargs)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py\", line 37, in run_main\r\n    fc_dir, run_info_yaml)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py\", line 80, in _run_toplevel\r\n    for xs in pipeline.run(config, run_info_yaml, parallel, dirs, samples):\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py\", line 162, in run\r\n    samples = genotype.parallel_variantcall_region(samples, run_parallel)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/genotype.py\", line 167, in parallel_variantcall_region\r\n    \"vrn_file\", [\"region\", \"sam_ref\", \"config\"]))\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/split.py\", line 32, in grouped_parallel_split_combine\r\n    final_output = parallel_fn(parallel_name, split_args)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py\", line 28, in run_parallel\r\n    return run_multicore(fn, items, config, parallel=parallel)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py\", line 86, in run_multicore\r\n    for data in joblib.Parallel(parallel[\"num_jobs\"])(joblib.delayed(fn)(x) for x in items):\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 657, in __call__\r\n    self.dispatch(function, args, kwargs)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 404, in dispatch\r\n    job = ImmediateApply(func, args, kwargs)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 142, in __init__\r\n    self.results = func(*args, **kwargs)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py\", line 50, in wrapper\r\n    return apply(f, *args, **kwargs)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py\", line 138, in variantcall_sample\r\n    return genotype.variantcall_sample(*args)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/genotype.py\", line 252, in variantcall_sample\r\n    call_file = caller_fn(align_bams, items, sam_ref, assoc_files, region, call_file)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/mutect.py\", line 141, in mutect_caller\r\n    region=region, out_file=out_file_indels)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/scalpel.py\", line 114, in _run_scalpel_caller\r\n    scalpel_tmp_file = bgzip_and_index(os.path.join(tmp_path, \"variants.\" + min_cov + \"x.indel.vcf\"), config)\r\n  File \"/home/bwubb/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/vcfutils.py\", line 441, in bgzip_and_index\r\n    assert os.path.exists(in_file), \"Input file %s not found\" % in_file\r\nAssertionError: Input file /home/bwubb/NGS/MelanomaTargeted/bcbio/data/1205-CRT-20Aplin/work/mutect/1/1205-CRT-20Aplin-1_0_22385208-raw-somaticIndels-scalpel-work/variants.3x.indel.vcf not found\r\n```\r\n\r\nCould any of this be related to recent update? I can not think of any other changes I made since 3 days ago. Sorry for dumping all these error messages. Thanks.\r\n\r\n-Brad W"},"comment":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/comments/154537334","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1101#issuecomment-154537334","issue_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1101","id":154537334,"user":{"login":"chapmanb","id":39391,"avatar_url":"https://avatars.githubusercontent.com/u/39391?v=3","gravatar_id":"","url":"https://api.github.com/users/chapmanb","html_url":"https://github.com/chapmanb","followers_url":"https://api.github.com/users/chapmanb/followers","following_url":"https://api.github.com/users/chapmanb/following{/other_user}","gists_url":"https://api.github.com/users/chapmanb/gists{/gist_id}","starred_url":"https://api.github.com/users/chapmanb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chapmanb/subscriptions","organizations_url":"https://api.github.com/users/chapmanb/orgs","repos_url":"https://api.github.com/users/chapmanb/repos","events_url":"https://api.github.com/users/chapmanb/events{/privacy}","received_events_url":"https://api.github.com/users/chapmanb/received_events","type":"User","site_admin":false},"created_at":"2015-11-06T21:02:13Z","updated_at":"2015-11-06T21:02:13Z","body":"Brad;\r\nSorry about the problems. This looks like it died in the middle of running scalpel for some reason on the first run, and then left the output scalpel directory in an unhappy state. It would be worth removing the work directory:\r\n```\r\nrm -rf /home/bwubb/NGS/MelanomaTargeted/bcbio/data/1205-CRT-21Aplin/work/mutect/1/1205-CRT-21Aplin-1_0_22381370-raw-somaticIndels-scalpel-work/\r\n```\r\nand then re-running with `-n 1` to see if it fails again or can continue cleanly. If you could send along a `tree` of the failed directory I can also work on trying to detect this half-finished scalpel case and correctly restart in bcbio. Thanks much."}}