{"action":"created","issue":{"url":"https://api.github.com/repos/NeuroVault/NeuroVault/issues/99","labels_url":"https://api.github.com/repos/NeuroVault/NeuroVault/issues/99/labels{/name}","comments_url":"https://api.github.com/repos/NeuroVault/NeuroVault/issues/99/comments","events_url":"https://api.github.com/repos/NeuroVault/NeuroVault/issues/99/events","html_url":"https://github.com/NeuroVault/NeuroVault/issues/99","id":53581198,"number":99,"title":"task queueing system for subprocess calls","user":{"login":"infocortex","id":7991256,"avatar_url":"https://avatars.githubusercontent.com/u/7991256?v=3","gravatar_id":"","url":"https://api.github.com/users/infocortex","html_url":"https://github.com/infocortex","followers_url":"https://api.github.com/users/infocortex/followers","following_url":"https://api.github.com/users/infocortex/following{/other_user}","gists_url":"https://api.github.com/users/infocortex/gists{/gist_id}","starred_url":"https://api.github.com/users/infocortex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/infocortex/subscriptions","organizations_url":"https://api.github.com/users/infocortex/orgs","repos_url":"https://api.github.com/users/infocortex/repos","events_url":"https://api.github.com/users/infocortex/events{/privacy}","received_events_url":"https://api.github.com/users/infocortex/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/NeuroVault/NeuroVault/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"vsoch","id":814322,"avatar_url":"https://avatars.githubusercontent.com/u/814322?v=3","gravatar_id":"","url":"https://api.github.com/users/vsoch","html_url":"https://github.com/vsoch","followers_url":"https://api.github.com/users/vsoch/followers","following_url":"https://api.github.com/users/vsoch/following{/other_user}","gists_url":"https://api.github.com/users/vsoch/gists{/gist_id}","starred_url":"https://api.github.com/users/vsoch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vsoch/subscriptions","organizations_url":"https://api.github.com/users/vsoch/orgs","repos_url":"https://api.github.com/users/vsoch/repos","events_url":"https://api.github.com/users/vsoch/events{/privacy}","received_events_url":"https://api.github.com/users/vsoch/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2015-01-07T00:15:53Z","updated_at":"2015-01-21T22:59:41Z","closed_at":null,"body":"Several features require calling an external binary with possibly long running times, which should ideally be run in a separate process.  It would be good to add a asynchronous task queue for managing subprocesses so they can be decoupled from the request lifecycle.  This would be used for pycortex generation, upload_zip, upload_folder, FEAT parsing, and NIDM parsing.\r\n\r\nI often use Celery with Rabbitmq (amqp) as the message broker, but amqp is a bit complex and probably overkill.  For a more simple, lightweight addition to the stack, I think we'd be better off using Redis over Rabbitmq as the messaging backend for Celery.\r\n\r\nhttp://celery.readthedocs.org/en/latest/django/\r\nhttp://celery.readthedocs.org/en/latest/getting-started/brokers/redis.html\r\nhttp://celery.readthedocs.org/en/latest/tutorials/daemonizing.html\r\n\r\nThe celery django app can be daemonized with a sysv init script, or managed with systemd.  The prerequisites and configuration will be added to puppet, along with the necessary configuration for Redis.\r\n"},"comment":{"url":"https://api.github.com/repos/NeuroVault/NeuroVault/issues/comments/70939668","html_url":"https://github.com/NeuroVault/NeuroVault/issues/99#issuecomment-70939668","issue_url":"https://api.github.com/repos/NeuroVault/NeuroVault/issues/99","id":70939668,"user":{"login":"vsoch","id":814322,"avatar_url":"https://avatars.githubusercontent.com/u/814322?v=3","gravatar_id":"","url":"https://api.github.com/users/vsoch","html_url":"https://github.com/vsoch","followers_url":"https://api.github.com/users/vsoch/followers","following_url":"https://api.github.com/users/vsoch/following{/other_user}","gists_url":"https://api.github.com/users/vsoch/gists{/gist_id}","starred_url":"https://api.github.com/users/vsoch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vsoch/subscriptions","organizations_url":"https://api.github.com/users/vsoch/orgs","repos_url":"https://api.github.com/users/vsoch/repos","events_url":"https://api.github.com/users/vsoch/events{/privacy}","received_events_url":"https://api.github.com/users/vsoch/received_events","type":"User","site_admin":false},"created_at":"2015-01-21T22:59:41Z","updated_at":"2015-01-21T22:59:41Z","body":"I gave this a first go, and on my virtual machine I have it working (so far!)\r\n\r\nhttps://github.com/vsoch/NeuroVault/blob/enh/image_comparison/neurovault/CELERY_README.md\r\n\r\n- I'm using celery with redis backend, saving results, and configuration options are integrated into django settings.\r\n- The choice of how to start the server seems to have many options, and I tested from command line, and then found a django module that will do it for me.  This of course could be however you like (Gabriel mentioned something about systemd?)\r\n- I had permissions errors with the log files for celery. The default goes to PWD, so I just let them be there :) \r\n- My dinky test tasks work, I'm next going to try making some \"real\" tasks (generating glass brain images and the correlation matrices).\r\n- I don't know why it's called celery. It seems like the C looks like a section of celery, but now we have a chicken and egg problem!"}}