{"action":"created","issue":{"url":"https://api.github.com/repos/sul-dlss/argo/issues/231","labels_url":"https://api.github.com/repos/sul-dlss/argo/issues/231/labels{/name}","comments_url":"https://api.github.com/repos/sul-dlss/argo/issues/231/comments","events_url":"https://api.github.com/repos/sul-dlss/argo/issues/231/events","html_url":"https://github.com/sul-dlss/argo/issues/231","id":117911064,"number":231,"title":"Outline tasks for implementing Argo indexing in delayed_jobs","user":{"login":"mejackreed","id":1656824,"avatar_url":"https://avatars.githubusercontent.com/u/1656824?v=3","gravatar_id":"","url":"https://api.github.com/users/mejackreed","html_url":"https://github.com/mejackreed","followers_url":"https://api.github.com/users/mejackreed/followers","following_url":"https://api.github.com/users/mejackreed/following{/other_user}","gists_url":"https://api.github.com/users/mejackreed/gists{/gist_id}","starred_url":"https://api.github.com/users/mejackreed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mejackreed/subscriptions","organizations_url":"https://api.github.com/users/mejackreed/orgs","repos_url":"https://api.github.com/users/mejackreed/repos","events_url":"https://api.github.com/users/mejackreed/events{/privacy}","received_events_url":"https://api.github.com/users/mejackreed/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/sul-dlss/argo/milestones/5","html_url":"https://github.com/sul-dlss/argo/milestones/Sprint%204","labels_url":"https://api.github.com/repos/sul-dlss/argo/milestones/5/labels","id":1425816,"number":5,"title":"Sprint 4","description":"","creator":{"login":"mejackreed","id":1656824,"avatar_url":"https://avatars.githubusercontent.com/u/1656824?v=3","gravatar_id":"","url":"https://api.github.com/users/mejackreed","html_url":"https://github.com/mejackreed","followers_url":"https://api.github.com/users/mejackreed/followers","following_url":"https://api.github.com/users/mejackreed/following{/other_user}","gists_url":"https://api.github.com/users/mejackreed/gists{/gist_id}","starred_url":"https://api.github.com/users/mejackreed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mejackreed/subscriptions","organizations_url":"https://api.github.com/users/mejackreed/orgs","repos_url":"https://api.github.com/users/mejackreed/repos","events_url":"https://api.github.com/users/mejackreed/events{/privacy}","received_events_url":"https://api.github.com/users/mejackreed/received_events","type":"User","site_admin":false},"open_issues":15,"closed_issues":9,"state":"open","created_at":"2015-11-23T18:58:52Z","updated_at":"2015-11-25T21:01:31Z","due_on":"2015-12-07T05:00:00Z","closed_at":null},"comments":6,"created_at":"2015-11-19T21:24:46Z","updated_at":"2015-11-26T02:17:50Z","closed_at":null,"body":"This feels possible, and I just wanted to outline the work as I see it:\r\n\r\n- [ ] move `ModsulatorJob` to its own delayed_jobs queue\r\n- [x] create a queue for indexing jobs \r\n    - **Question: should we also create individual queues for the indexing groups? Or just use priorities?**\r\n- [x] Setup indexing jobs to follow the priorities outlined by @rosy1280 in https://github.com/sul-dlss/argo/issues/194#issuecomment-157513521  . Delayed jobs priority 0 is highest priority\r\n   - 0 - Uber API\r\n   - 1 - workflows\r\n   - 2 - agreements\r\n   - 3 - Hydrus Uber APO\r\n   - 4 - All APOs (hydrus and non-hydrus)\r\n   - 5 - collections/sets \r\n   - 6 - items\r\n- [ ] Scale workers for processing?\r\n\r\n**Question: How do we query the different priority objects?**\r\n"},"comment":{"url":"https://api.github.com/repos/sul-dlss/argo/issues/comments/159783769","html_url":"https://github.com/sul-dlss/argo/issues/231#issuecomment-159783769","issue_url":"https://api.github.com/repos/sul-dlss/argo/issues/231","id":159783769,"user":{"login":"jmartin-sul","id":7741604,"avatar_url":"https://avatars.githubusercontent.com/u/7741604?v=3","gravatar_id":"","url":"https://api.github.com/users/jmartin-sul","html_url":"https://github.com/jmartin-sul","followers_url":"https://api.github.com/users/jmartin-sul/followers","following_url":"https://api.github.com/users/jmartin-sul/following{/other_user}","gists_url":"https://api.github.com/users/jmartin-sul/gists{/gist_id}","starred_url":"https://api.github.com/users/jmartin-sul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmartin-sul/subscriptions","organizations_url":"https://api.github.com/users/jmartin-sul/orgs","repos_url":"https://api.github.com/users/jmartin-sul/repos","events_url":"https://api.github.com/users/jmartin-sul/events{/privacy}","received_events_url":"https://api.github.com/users/jmartin-sul/received_events","type":"User","site_admin":false},"created_at":"2015-11-26T02:17:50Z","updated_at":"2015-11-26T02:17:50Z","body":"Besides consistency with existing Argo code and Rails in general, what benefits does ActiveJob provide vs delayed_job?  Abstraction and the ability to slot in other job queuing mechanisms if delayed_job fails to meet our needs later?\r\n\r\nI think respecting indexing order in some shape or form is a must, and so would be a strong reason to consider using delayed_job directly (assuming delayed_job actually provides easier access to the job priority info).  Since strict priority adherence in the queuing mechanism likely would be easier all around than something like detection and manual remediation of things indexed out of order, I think that could also be an argument that the ActiveJob abstraction doesn't provide us with the level of control we need, and that if we switched off of delayed_job as a queuing mechanism, we'd likely have to re-implement something to that effect anyway (or turn it on in the new queuing mechanism's configuration), in which case ActiveJob would be providing less abstracting-details-away benefit than in other cases.\r\n\r\nAnother benefit of not using ActiveJob is that it's part of rails, and if we could keep from loading all of Rails when running delayed_job workers, we might be able to save a lot of memory and run more workers with the same computing resources (see issue #250).\r\n\r\nSo I guess I'm interested in exploring using delayed_job directly, or seeing whether we can come up with sane workarounds in ActiveJob for the limitations we've run into so far.  But its memory usage and its lack of access to priority info are significant strikes against it."}}