Java How to achieve parallelism, dividing up work, dynamically?
<p>I am new to multi-threading in Java. </p>

<p>My goal is to have one thread reading a file and then passing work chunks off to worker threads to process in parallel.</p>

<p>There is a really good example here.
<a href="http://www.javapractices.com/topic/TopicAction.do?Id=247" rel="nofollow">Concurrency Tutorial</a></p>

<p>This code snippet takes a list of work (<code>ArrayList&lt;String&gt; URLs</code>) and dumps it onto a block of worker threads with function designated in the Task.call() method.</p>

<pre><code>void pingAndReportEachWhenKnown()  throws InterruptedException, ExecutionException  {
  int numThreads = URLs.size() &gt; 4 ? 4 : URLs.size(); //max 4 threads
  ExecutorService executor = Executors.newFixedThreadPool(numThreads);
  CompletionService&lt;PingResult&gt; compService = new ExecutorCompletionService&lt;&gt;(executor);
  for(String url : URLs){
    Task task = new Task(url);
    compService.submit(task);
  }
  for(String url : URLs){
    Future&lt;PingResult&gt; future = compService.take();
    log(future.get());
  }
  executor.shutdown(); //always reclaim resources
}
</code></pre>

<p>This is exactly what I want to do, but I need a change.
The size of my work queue will not fit into working memory (HUGE file), so I need to buffer the read lines.  I can achieve Blocking I need with an ArrayBlockingQueue.  But then I need to also buffer the assignment of tasks to the CompletionService.  Work chunk size will vary, so completion time will vary as well.</p>

<p>How do I not put too much on the compService work queue? The following code would put the items on one at a time, because it would wait for completion before trying to get another task from the queue.  So this is insufficient.  What is the right or best way to handle this?  </p>

<pre><code>for(;;){
  Task task = arrayBlockingQueue.take();  //Blocking operation 
  compService.submit(task);
  Future&lt;PingResult&gt; future = compService.take();  //Blocking operation
  log(future.get());
}
</code></pre>