{"action":"closed","issue":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/218","labels_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/218/labels{/name}","comments_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/218/comments","events_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/218/events","html_url":"https://github.com/cBio/cbio-cluster/issues/218","id":59648897,"number":218,"title":"Job Preemption exploration discussion","user":{"login":"tatarsky","id":6439722,"avatar_url":"https://avatars.githubusercontent.com/u/6439722?v=3","gravatar_id":"","url":"https://api.github.com/users/tatarsky","html_url":"https://github.com/tatarsky","followers_url":"https://api.github.com/users/tatarsky/followers","following_url":"https://api.github.com/users/tatarsky/following{/other_user}","gists_url":"https://api.github.com/users/tatarsky/gists{/gist_id}","starred_url":"https://api.github.com/users/tatarsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tatarsky/subscriptions","organizations_url":"https://api.github.com/users/tatarsky/orgs","repos_url":"https://api.github.com/users/tatarsky/repos","events_url":"https://api.github.com/users/tatarsky/events{/privacy}","received_events_url":"https://api.github.com/users/tatarsky/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cBio/cbio-cluster/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/cBio/cbio-cluster/labels/Torque/Moab%20scheduler","name":"Torque/Moab scheduler","color":"009800"}],"state":"closed","locked":false,"assignee":{"login":"tatarsky","id":6439722,"avatar_url":"https://avatars.githubusercontent.com/u/6439722?v=3","gravatar_id":"","url":"https://api.github.com/users/tatarsky","html_url":"https://github.com/tatarsky","followers_url":"https://api.github.com/users/tatarsky/followers","following_url":"https://api.github.com/users/tatarsky/following{/other_user}","gists_url":"https://api.github.com/users/tatarsky/gists{/gist_id}","starred_url":"https://api.github.com/users/tatarsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tatarsky/subscriptions","organizations_url":"https://api.github.com/users/tatarsky/orgs","repos_url":"https://api.github.com/users/tatarsky/repos","events_url":"https://api.github.com/users/tatarsky/events{/privacy}","received_events_url":"https://api.github.com/users/tatarsky/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2015-03-03T14:29:51Z","updated_at":"2015-04-01T14:37:54Z","closed_at":"2015-04-01T14:37:54Z","body":"Per Git #64 at a point in the past job Preemption was disabled as it appeared to kill some jobs under some conditions.\r\n\r\nIt has been requested I investigate turning it back on. I will only do so carefully so this is a log of my investigation. \r\n\r\nMy reading so far is this is a feature (killing preempted jobs under some cases), not a bug. \r\n\r\nIf the setting:\r\n```\r\nPREEMPTPOLICY  SUSPEND\r\n```\r\nis set, under some conditions it appears to state some jobs could be killed:\r\n\r\n\"Moab uses preemption escalation to free up resources. This means if the PREEMPTPOLICY is set to suspend, then Moab will use this method if available but will escalate to something potentially more disruptive if necessary to preempt and free up resources\"\r\n\r\nAnd\r\n\r\n\"For a job to be suspended, it must be marked as suspendable. If not, it will be requeued or canceled. \"\r\n\r\nWe have SUSPENDABLE as a default flag for a job submit however for the batch queues but not active and gpu. My guess is that previously its possible the jobs that were killed were in those queues. I have no proof of this. \r\n\r\nSo I would like to for starters add the SUSPENDABLE flag to those other queues. Per other efforts to allow job suspension via mjobctl. \r\n\r\nThe precise definition of \"escalate to something potentially more distruptive\" appears to imply cancelling. I believe requeuing is possible if RESTARTABLE is set. \r\n\r\nI am concerned that suspending jobs doesn't really clear the resources they had used. And that using a preemption policy of REQUEUE may result in better overall results. \r\n\r\nWe would then want to make sure all jobs by default have RESTARTABLE set or they would also \"escalate\" to being cancelled if preempted.\r\n\r\nI may not fully understand all the options at this time but as noted, this is a log of my investigation. "}}