{"action":"opened","issue":{"url":"https://api.github.com/repos/spotify/luigi/issues/683","labels_url":"https://api.github.com/repos/spotify/luigi/issues/683/labels{/name}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/683/comments","events_url":"https://api.github.com/repos/spotify/luigi/issues/683/events","html_url":"https://github.com/spotify/luigi/issues/683","id":55749777,"number":683,"title":"Proposal for re-evaluating completion of ExternalTasks","user":{"login":"boosh","id":362148,"avatar_url":"https://avatars.githubusercontent.com/u/362148?v=3","gravatar_id":"","url":"https://api.github.com/users/boosh","html_url":"https://github.com/boosh","followers_url":"https://api.github.com/users/boosh/followers","following_url":"https://api.github.com/users/boosh/following{/other_user}","gists_url":"https://api.github.com/users/boosh/gists{/gist_id}","starred_url":"https://api.github.com/users/boosh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/boosh/subscriptions","organizations_url":"https://api.github.com/users/boosh/orgs","repos_url":"https://api.github.com/users/boosh/repos","events_url":"https://api.github.com/users/boosh/events{/privacy}","received_events_url":"https://api.github.com/users/boosh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-01-28T12:21:56Z","updated_at":"2015-01-28T12:21:56Z","closed_at":null,"body":"Currently because ExternalTasks don't implement `run()`, they are marked as not runnable when the task is added. That means that they aren't assigned to a worker. As a result, the scheduler will never return a PENDING ExternalTask to a worker in `get_work`. So if an ExternalTask completes while luigi is running, it will never be detected and so dependent Tasks will not run. I want to change that.\r\n\r\nIt looks like tasks are distributed to workers before any tasks are run. So that means that multiple long running tasks could be queued to the same worker, with multiple short running tasks assigned to another worker. Therefore, we could end up with one worker doing all the long running tasks, and other workers remaining idle. This isn't ideal, so I don't want to hack the logic that queues up tasks to workers.\r\n\r\nI suggest implementing logic to retry ExternalTasks as follows:\r\n\r\n1. Returning the highest priority `PENDING` task from the scheduler if there is no `best_task` (subject to retry logic in 3. below).\r\n2. Make workers call `task.complete()` if `run()` is not implemented\r\n3. Add backoff logic with a new state (BACKED_OFF) & task attribute indicating when it entered that state. The scheduler will look at these in 1. above. There's no point hammering away at missing external tasks when there are others that could be evaluated.\r\n\r\nIf `task.complete()` returns `False`, workers will remove themselves from the task.workers set so that the task can be assigned to a different worker later. \r\n\r\nWhat are your thoughts? I don't want to do this work if it won't be merged in since we don't want to maintain our own fork."}}