Running large amount of long running background jobs in Rails
<p>We're building a web-app where users will be uploading potentially large files that will need to be processed in the background.  The task involves calling 3rd-party APIs so each job can take several hours to complete.  We're using DelayedJob to run the background jobs.  With every user kicking off a background job, each of which will take a few hours to finish, that will add up to a lot of background jobs every quickly.  I am wondering what would be the best way to setup the deployment for this?  We're currently hosted on DigitalOcean.  I've kicked off 10 DelayedJob workers.  Each one (when ideal) takes up 157MB. When actively running it utilizes around 900 MB.  Our user-base right now is pretty small so it's not an issue but will be one soon.  So on a 4GB droplet, I can probably run like 2 or 3 workers at a time.  How should we approach this issue?  Should we be looking at using DigitalOcean's API to auto-spin cheap droplets on demand?  Should we subscribe to high-memory droplets on a monthly basis instead? If we go with auto-spinning droplets, should we stick with DigitalOcean or would Heroku make more sense?  Or is the entire approach wrong and should we be approaching it from an entire different direction?  Any help/advice would be very much appreciated.</p>

<p>Thanks!</p>