How to return data from a CherryPy BackgroundTask running as fast as possible
<p>I'm building a web service for iterative batch processing of data using CherryPy. The ideal workflow is as follows:</p>

<ol>
<li>Users POST data to the service for processing</li>
<li>When the processing job is free, it collects the queued data and starts another iteration</li>
<li>While the job is processing, users are POSTing more data to the queue for the next iteration</li>
<li>Once the current iteration is finished, the results are passed back so that users can GET them using the same API.</li>
<li>The job starts again with the next batch of queued data.</li>
</ol>

<p>The key consideration here is that the processing should run as fast as possible with each iteration starting as soon as the previous one finishes, regardless of the amount of data in the queue. There's no upper bound on how long each iteration can take so I can't create a fixed schedule for it to run on.</p>

<p>There are a few examples of using <code>BackgroundTask</code> (<a href="http://tools.cherrypy.org/wiki/BackgroundTaskQueue" rel="nofollow">like this one</a>) but I've yet to find one that deals with returning data, or one that deals with tasks running as fast as possible as opposed to on a fixed schedule.</p>

<p>I'm not wedded to the <code>BackgroundTask</code> solution so if anyone can offer an alternative one I'd be more than happy. It feels like there's a solution within the framework though.</p>