{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/6125","labels_url":"https://api.github.com/repos/dmwm/WMCore/issues/6125/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/WMCore/issues/6125/comments","events_url":"https://api.github.com/repos/dmwm/WMCore/issues/6125/events","html_url":"https://github.com/dmwm/WMCore/pull/6125","id":100422233,"number":6125,"title":"making logcollect jobs run better","user":{"login":"hufnagel","id":1798355,"avatar_url":"https://avatars.githubusercontent.com/u/1798355?v=3","gravatar_id":"","url":"https://api.github.com/users/hufnagel","html_url":"https://github.com/hufnagel","followers_url":"https://api.github.com/users/hufnagel/followers","following_url":"https://api.github.com/users/hufnagel/following{/other_user}","gists_url":"https://api.github.com/users/hufnagel/gists{/gist_id}","starred_url":"https://api.github.com/users/hufnagel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hufnagel/subscriptions","organizations_url":"https://api.github.com/users/hufnagel/orgs","repos_url":"https://api.github.com/users/hufnagel/repos","events_url":"https://api.github.com/users/hufnagel/events{/privacy}","received_events_url":"https://api.github.com/users/hufnagel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-08-11T22:06:43Z","updated_at":"2015-08-24T03:07:02Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/dmwm/WMCore/pulls/6125","html_url":"https://github.com/dmwm/WMCore/pull/6125","diff_url":"https://github.com/dmwm/WMCore/pull/6125.diff","patch_url":"https://github.com/dmwm/WMCore/pull/6125.patch"},"body":"LogCollect jobs have a gazillion problems. This is fixing some of them.\r\n\r\n - try to stage the logArchive in once, with a timeout at 10 minutes, then give up\r\n - do not actually delete them unless they were staged in and added to the tarball\r\n - stage in one file, add it to the tarball and then delete it from local disk and mass storage\r\n - deletion from mass storage is optional, we try it for 5 minutes and then give up (and not fail the job)\r\n\r\nThis version seems to work, but probabaly needs some work, open questions:\r\n - should I delete all logArchive if any of them can be processed and therefore the LogCollect job succeeds ?\r\n - what about the fixed timeouts for stagein, is it ok to not retry ?\r\n - what about the fixed timeout for deletions, should it be longer (we should never retry deletions, its not worth waiting for it considering this is just unmerged cleanup at that point) ?\r\n\r\nThis is fixing two immediate problems we have observed in the Tier0:\r\n - LogCollect jobs ran out of local disk because it staged in all files and then build the tarball instead of doing it file by file\r\n - LogCollect job took hours to retry stagein of a non-existent file, for each input file, leading to 10+hour LogCollect jobs that ultimately failed anyways \r\n"},"comment":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/comments/134001869","html_url":"https://github.com/dmwm/WMCore/pull/6125#issuecomment-134001869","issue_url":"https://api.github.com/repos/dmwm/WMCore/issues/6125","id":134001869,"user":{"login":"ticoann","id":1247993,"avatar_url":"https://avatars.githubusercontent.com/u/1247993?v=3","gravatar_id":"","url":"https://api.github.com/users/ticoann","html_url":"https://github.com/ticoann","followers_url":"https://api.github.com/users/ticoann/followers","following_url":"https://api.github.com/users/ticoann/following{/other_user}","gists_url":"https://api.github.com/users/ticoann/gists{/gist_id}","starred_url":"https://api.github.com/users/ticoann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ticoann/subscriptions","organizations_url":"https://api.github.com/users/ticoann/orgs","repos_url":"https://api.github.com/users/ticoann/repos","events_url":"https://api.github.com/users/ticoann/events{/privacy}","received_events_url":"https://api.github.com/users/ticoann/received_events","type":"User","site_admin":false},"created_at":"2015-08-24T03:07:02Z","updated_at":"2015-08-24T03:07:02Z","body":"HI Dirk, \r\nStageOutMgr is a module not class in the following statement (so it is not callable)\r\n\r\n``` python\r\nimport WMCore.Storage.StageOutMgr as StageOutMgr\r\n```\r\n\r\nYou can change that\r\n\r\n``` python\r\nfrom WMCore.Storage.StageOutMgr import StageOutMgr\r\n``` \r\nSame for \r\n\r\nStageInMgr\r\nDeleteMgr\r\n\r\nSeangchan"}}