{"action":"opened","issue":{"url":"https://api.github.com/repos/inveniosoftware/invenio-checker/issues/3","labels_url":"https://api.github.com/repos/inveniosoftware/invenio-checker/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/inveniosoftware/invenio-checker/issues/3/comments","events_url":"https://api.github.com/repos/inveniosoftware/invenio-checker/issues/3/events","html_url":"https://github.com/inveniosoftware/invenio-checker/issues/3","id":119528235,"number":3,"title":"Decide on a way to clean up stale redis data.","user":{"login":"dset0x","id":8993055,"avatar_url":"https://avatars.githubusercontent.com/u/8993055?v=3","gravatar_id":"","url":"https://api.github.com/users/dset0x","html_url":"https://github.com/dset0x","followers_url":"https://api.github.com/users/dset0x/followers","following_url":"https://api.github.com/users/dset0x/following{/other_user}","gists_url":"https://api.github.com/users/dset0x/gists{/gist_id}","starred_url":"https://api.github.com/users/dset0x/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dset0x/subscriptions","organizations_url":"https://api.github.com/users/dset0x/orgs","repos_url":"https://api.github.com/users/dset0x/repos","events_url":"https://api.github.com/users/dset0x/events{/privacy}","received_events_url":"https://api.github.com/users/dset0x/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-11-30T16:31:01Z","updated_at":"2015-11-30T16:31:01Z","closed_at":null,"body":"If redis or celery crash badly, future workers may end up spinning for ever. This happens because workers use redis to check whether other workers are running. This is done this way because `celery`'s introspection has no atomic way that I know of which returns both scheduled and running tasks (a worker may be retrying and be in \"scheduled\" state a lot).\r\n\r\nTimeouts, on the other hand, are problematic due to the fact that some tasks may be expected to run for very long periods of time.\r\n\r\nI have written the `clients.redis_helpers.cleanup_failed_runs` function, but it needs to be tested and integrated in a way that ensures that it does not disturb running processes. It also needs to be decided what will trigger it and how often. As a sidenote, the `.zap()` methods that it uses have not been used in a while and perhaps do more than intended (eg they kill the celery task)."}}