Failure to write output of shell command to file in Go
<p>I have written the following function for executing the <code>snt2cooc</code> command (one of the preprocessing steps for running <a href="http://www.statmt.org/moses/giza/GIZA++.html" rel="nofollow">GIZA++</a>. For our purposes I think we can consider the <code>snt2cooc</code> script to be a black box):</p>

<pre><code>func SNTToCOOC(srcVocab, tgtVocab, sntPath, outpath string) error {
    // open the out file for writing
    outfile, err := os.Create(outpath)
    if err != nil {
        return err
    }
    defer outfile.Close()

    cmdStr := "snt2cooc"
    args := []string{srcVocab, tgtVocab, sntPath}
    cmd := exec.Command(cmdStr, args...)
    cmd.Stdout = outfile
    if err = cmd.Run(); err != nil {
        return err
    }
    cmd.Wait()
    return err
}
</code></pre>

<p>When running, the function executes without an error, but the output file is empty. This same code works for other similar commands, but not for this specific <code>snt2cooc</code> command, and I noticed that when I run this command in the shell directly: </p>

<p>snt2cooc file1.vcb file2.vcb file3.snt</p>

<p>I get the following output:</p>

<pre><code>END.
0 2
0 3
0 4
0 5
0 6
</code></pre>

<p>(truncated for brevity)</p>

<p>And if I send the output of the command to a file directly from the shell:</p>

<pre><code>snt2cooc file1.vcb file2.vcb file3.snt &gt; out.txt
</code></pre>

<p>The contents of <code>out.txt</code> are as expected:</p>

<pre><code>0 2
0 3
0 4
0 5
0 6
</code></pre>

<p>Notice how in the first case, the line <code>END.</code> is output to stdout first, and only then is the real output of the command sent to stdout. I therefore think there is a race condition going on, where the Go code finishes executing before the command's final output is written to file. This despite calling <code>cmd.Wait()</code>. I'm not too sure what exactly the <code>snt2cooc</code> command is doing internally. Could someone provide a hint on how to solve this?</p>

<p><strong>Edit 1:</strong></p>

<p>It seems like the following code, with the sleep of 500ms included, consistently writes output to the file for the <code>snt2cooc</code> command:</p>

<pre><code>cmdStr := "snt2cooc"
args := []string{srcVocab, tgtVocab, sntPath}
cmd := exec.Command(cmdStr, args...)
stdout, err := cmd.StdoutPipe()
time.Sleep(500 * time.Millisecond)
if err != nil {
    return err
}
err = cmd.Start()
if err != nil {
    return err
}

out := bufio.NewScanner(stdout)
for out.Scan() {
    outfile.Write(out.Bytes())
    outfile.WriteString("\n")
}
if err := out.Err(); err != nil {
    return err
}
</code></pre>

<p>This proves to me that there is some race condition going on, with the Go program exiting before all output is written to file. I added a bounty to this question, with the hope that someone can 1) explain why this is happening and 2) provide a non-hacky way (i.e. 500ms sleep) to fix it. </p>