{"action":"created","issue":{"url":"https://api.github.com/repos/spotify/luigi/issues/683","labels_url":"https://api.github.com/repos/spotify/luigi/issues/683/labels{/name}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/683/comments","events_url":"https://api.github.com/repos/spotify/luigi/issues/683/events","html_url":"https://github.com/spotify/luigi/issues/683","id":55749777,"number":683,"title":"Proposal for re-evaluating completion of ExternalTasks","user":{"login":"boosh","id":362148,"avatar_url":"https://avatars.githubusercontent.com/u/362148?v=3","gravatar_id":"","url":"https://api.github.com/users/boosh","html_url":"https://github.com/boosh","followers_url":"https://api.github.com/users/boosh/followers","following_url":"https://api.github.com/users/boosh/following{/other_user}","gists_url":"https://api.github.com/users/boosh/gists{/gist_id}","starred_url":"https://api.github.com/users/boosh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/boosh/subscriptions","organizations_url":"https://api.github.com/users/boosh/orgs","repos_url":"https://api.github.com/users/boosh/repos","events_url":"https://api.github.com/users/boosh/events{/privacy}","received_events_url":"https://api.github.com/users/boosh/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-01-28T12:21:56Z","updated_at":"2015-02-05T12:14:50Z","closed_at":"2015-02-05T12:14:50Z","body":"Currently because ExternalTasks don't implement `run()`, they are marked as not runnable when the task is added. That means that they aren't assigned to a worker if they haven't completed. As a result, the scheduler will never return a PENDING ExternalTask to a worker in `get_work`. So if an ExternalTask completes while luigi is running, it will never be detected and so dependent Tasks will not run on that invocation. I want to change that.\r\n\r\nI suggest implementing logic to retry ExternalTasks as follows:\r\n\r\n1. Set `runnable=True` for tasks that don't implement `run()`.\r\n2. Create a new multiprocessing.Process subclass that calls `task.complete()` if `run()` is not implemented.\r\n\r\nWhat are your thoughts? I don't want to do this work if it won't be merged in since we don't want to maintain our own fork."},"comment":{"url":"https://api.github.com/repos/spotify/luigi/issues/comments/73037054","html_url":"https://github.com/spotify/luigi/issues/683#issuecomment-73037054","issue_url":"https://api.github.com/repos/spotify/luigi/issues/683","id":73037054,"user":{"login":"boosh","id":362148,"avatar_url":"https://avatars.githubusercontent.com/u/362148?v=3","gravatar_id":"","url":"https://api.github.com/users/boosh","html_url":"https://github.com/boosh","followers_url":"https://api.github.com/users/boosh/followers","following_url":"https://api.github.com/users/boosh/following{/other_user}","gists_url":"https://api.github.com/users/boosh/gists{/gist_id}","starred_url":"https://api.github.com/users/boosh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/boosh/subscriptions","organizations_url":"https://api.github.com/users/boosh/orgs","repos_url":"https://api.github.com/users/boosh/repos","events_url":"https://api.github.com/users/boosh/events{/privacy}","received_events_url":"https://api.github.com/users/boosh/received_events","type":"User","site_admin":false},"created_at":"2015-02-05T12:14:50Z","updated_at":"2015-02-05T12:14:50Z","body":"Resolved in #684."}}