Bash Pipelined Shell scripts I wrote are Starved/Deadlocked/Performing poorly
<p>I have used <code>duff</code> to create a report of duplicate files in my file system:</p>

<pre><code>duff ~/Photos &gt; ~/Photos/duplicates.txt
</code></pre>

<p>I have written some groovy scripts to transform this report into an HTML page where I can view the duplicate photos in my browser:</p>

<pre><code>cat duplicates.txt | filter_non_existent.groovy | duff_to_json.groovy | json_to_html.groovy &gt; duplicates.html
</code></pre>

<p>About 50% of the time, it works beautifully without any problems. </p>

<p>Other times some of my scripts don't start executing, or run very slowly. Why is this, and what can I do to prevent it? (and why don't "professionally written" command line programs like grep suffer from this problem?)</p>

<p><strong>More information</strong></p>

<ul>
<li>I delay the start of each script until it receives std input (while
no data, sleep for 2.5 seconds). The starvation was more frequent
when the sleep interval was shorter</li>
<li>The more pipes I connect together, the more frequently it happens (if I just execute one of my scripts after <code>cat</code>, I never get the problem)</li>
<li><p>Here's the template each of my scripts are using (I use the groovy runtime but stick to plain java syntax):</p>

<p>public static void main(String[] args) throws IOException {</p>

<pre><code>BufferedReader br = new BufferedReader(new InputStreamReader(System.in));

while (!br.ready()) {
    Thread.sleep(2500L);
}

while (br.ready()) {
    String inputLine = br.readLine();
    // Do stuff
}
</code></pre>

<p>}</p></li>
</ul>