{"action":"created","issue":{"url":"https://api.github.com/repos/pcingola/BigDataScript/issues/60","labels_url":"https://api.github.com/repos/pcingola/BigDataScript/issues/60/labels{/name}","comments_url":"https://api.github.com/repos/pcingola/BigDataScript/issues/60/comments","events_url":"https://api.github.com/repos/pcingola/BigDataScript/issues/60/events","html_url":"https://github.com/pcingola/BigDataScript/issues/60","id":54242712,"number":60,"title":"CheckTasksRunning.incMissingCount(98):\tWARNING: Task PID '' not found. Incrementing count as 'missing': 1(max. allowed 3)","user":{"login":"golharam","id":671079,"avatar_url":"https://avatars.githubusercontent.com/u/671079?v=3","gravatar_id":"","url":"https://api.github.com/users/golharam","html_url":"https://github.com/golharam","followers_url":"https://api.github.com/users/golharam/followers","following_url":"https://api.github.com/users/golharam/following{/other_user}","gists_url":"https://api.github.com/users/golharam/gists{/gist_id}","starred_url":"https://api.github.com/users/golharam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/golharam/subscriptions","organizations_url":"https://api.github.com/users/golharam/orgs","repos_url":"https://api.github.com/users/golharam/repos","events_url":"https://api.github.com/users/golharam/events{/privacy}","received_events_url":"https://api.github.com/users/golharam/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-13T20:20:02Z","updated_at":"2015-01-15T16:19:06Z","closed_at":null,"body":"BDS running with SGE doesn't seem to see some jobs finish.  This doesn't happen all the time.  Here's an example.  I have 27 samples consisting of 2 files each (fq1, fq2).  For each sample, I start a task to run fastqValidator on the fq file:\r\n\r\nfor (string s : samples) {\r\n\tstring fastq1 = fq1{s}\r\n\tstring fastq2 = fq2{s}\r\n\r\n\ttask fastQValidator --file $fastq1\r\n\ttask fastQValidator --file $fastq2\r\n}\r\n\r\nSometimes the script will run without any issues.  Other times it will error out with:\r\nCheckTasksRunning.incMissingCount(98):\tWARNING: Task PID '19025' not found. Incrementing count as 'missing': 1(max. allowed 3)\r\nCheckTasksRunning.incMissingCount(98):\tWARNING: Task PID '19007' not found. Incrementing count as 'missing': 1(max. allowed 3)\r\nCheckTasksRunning.incMissingCount(98):\tWARNING: Task PID '19026' not found. Incrementing count as 'missing': 1(max. allowed 3)\r\nCheckTasksRunning.incMissingCount(98):\tWARNING: Task PID '19004' not found. Incrementing count as 'missing': 1(max. allowed 3)\r\nCheckTasksRunning.incMissingCount(98):\tWARNING: Task PID '19028' not found. Incrementing count as 'missing': 1(max. allowed 3)\r\nCheckTasksRunning.incMissingCount(98):\tWARNING: Task PID '19009' not found. Incrementing count as 'missing': 1(max. allowed 3)\r\nCheckTasksRunning.incMissingCount(98):\tWARNING: Task PID '19030' not found. Incrementing count as 'missing': 1(max. allowed 3)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/pcingola/BigDataScript/issues/comments/70111595","html_url":"https://github.com/pcingola/BigDataScript/issues/60#issuecomment-70111595","issue_url":"https://api.github.com/repos/pcingola/BigDataScript/issues/60","id":70111595,"user":{"login":"golharam","id":671079,"avatar_url":"https://avatars.githubusercontent.com/u/671079?v=3","gravatar_id":"","url":"https://api.github.com/users/golharam","html_url":"https://github.com/golharam","followers_url":"https://api.github.com/users/golharam/followers","following_url":"https://api.github.com/users/golharam/following{/other_user}","gists_url":"https://api.github.com/users/golharam/gists{/gist_id}","starred_url":"https://api.github.com/users/golharam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/golharam/subscriptions","organizations_url":"https://api.github.com/users/golharam/orgs","repos_url":"https://api.github.com/users/golharam/repos","events_url":"https://api.github.com/users/golharam/events{/privacy}","received_events_url":"https://api.github.com/users/golharam/received_events","type":"User","site_admin":false},"created_at":"2015-01-15T16:19:06Z","updated_at":"2015-01-15T16:19:06Z","body":"I had to run the pipeline a few times to get the error to happen, but I\r\nhave it.  I ran bds with '-v -d -l -s sge'.  I'm running build 0.999d.\r\n\r\nTask failed:\r\n        Program & line     : 'ExomePipeline.bds', line 56\r\n        Task ID            :\r\n'ExomePipeline.bds.20150115_110710_289/task.line_56.id_13'\r\n        Task PID           : '22335'\r\n        Task hint          : ' fastQValidator --file NTC_S5_L001_R1_001'\r\n        Task resources     : 'cpus: 1   mem: -1.0 B     timeout: 86400\r\nwall-timeout: 86400'\r\n        State              : 'FINISHED'\r\n        Dependency state   : 'ERROR'\r\n        Retries available  : '1'\r\n        Input files        : '[NTC_S5_L001_R1_001.fastq.gz]'\r\n        Output files       : '[analysis/NTC.1.fastqValidator.txt]'\r\n        Script file        :\r\n'PD1_FFPE_CancerPanel/ExomePipeline.bds.20150115_110710_289/\r\ntask.line_56.id_13.sh'\r\n        Exit status        : '0'\r\n        Output file checks : 'Error: Output file 'NTC.1.fastqValidator.txt'\r\ndoes not exist.'   Program            :\r\n\r\n                # SYS command. line 57\r\n\r\n                 fastQValidator --file NTC_S5_L001_R1_001.fastq.gz >\r\nNTC.1.fastqValidator.txt\r\n\r\n\r\nWhen I look in the bds temp directory, I see task.link_56.id_13.exitCode\r\nwith a value of 0 inside it.\r\n\r\nThe pipeline runs but this error occurs inconsistently so I suspect a\r\ntiming issue.  The directory is a local block storage directory on the head\r\nnode where bds is running.\r\n\r\n\r\nOn Wed, Jan 14, 2015 at 4:34 PM, Pablo Cingolani <notifications@github.com>\r\nwrote:\r\n\r\n> Are the \"exitCode\" files created and readable from the head node? Try\r\n> running with \"-log\" and check if the files are there when the task finishes.\r\n> Example:\r\n>\r\n> $ bds -log -d z.bds\r\n> ....\r\n>\r\n> $ ls -al z.bds.20150114_155340_078/task.line_4.id_1.exitCode\r\n> -rw-rw-r-- 1 pcingola pcingola 13 Jan 14 16:21 z.bds.20150114_155340_078/task.line_4.id_1.exitCode\r\n>\r\n> This exitCode file is the way the slave node informs bds that the task has\r\n> finished. So problem might be that the exitCode file cannot be read from\r\n> the head node...\r\n>\r\n> The incMissing message is produces because a \"supposedly running\" task\r\n> disappeared from the queue (qstat does not show it). So if bds cannot\r\n> detect the exitCode file, is assumes that the task is still running, then\r\n> it checks using 'qstat' and finds out that the task is no longer there (\"it\r\n> disappeared from the cluster\") so if it cannot be found for three times in\r\n> a row, it marks the task as failed.\r\n>\r\n> —\r\n> Reply to this email directly or view it on GitHub\r\n> <https://github.com/pcingola/BigDataScript/issues/60#issuecomment-69996349>\r\n> .\r\n>"}}