real time logging to file with python subprocess
<p>I expect this is really simple but I can't work this out.</p>

<p>I am trying to write to a log file in real time the output from a <code>DD</code> imaging subprocess - I'm using <code>DD v 8.25</code> from which you can get regular progress updates using the <code>'status=progress'</code> option which writes to <code>stderr</code>.</p>

<p>I can get it to log the full output real time by passing the file object to the <code>stderr</code> i.e</p>

<pre><code>log_file = open('mylog.log', 'a')
p = subprocess.Popen['dd command...'], stdout=None, stderr=log_file)
</code></pre>

<p>...but I would prefer to intercept the string from <code>stderr</code> first so I can parse it before writing to file.</p>

<p>I have tried threading but I can't seem to get it to write, or if it does, it only does it at the end of the process and not during.</p>

<p>I'm a python noob so example code would be appreciated. Thanks!</p>

<p><strong>UPDATE - NOW WORKING (ISH)</strong></p>

<p>I had a look at the link J.F. Sebastian suggested and found posts about using threads, so after that I used the "kill -USR1" trick to get DD to post progress to stderr which I could then pick up:</p>

<pre><code>#! /usr/bin/env python
from subprocess import PIPE, Popen
from threading import Thread
from queue import Queue, Empty
import time

q = Queue()

def parsestring(mystr):
    newstring = mystr[0:mystr.find('bytes')]
    return newstring

def enqueue(out, q):
    for line in proc1.stderr:
        q.put(line)
    out.close()

def getstatus():
    while proc1.poll() == None:
        proc2 = Popen(["kill -USR1 $(pgrep ^dd)"], bufsize=1, shell=True)
        time.sleep(2)

with open("log_file.log", mode="a") as log_fh:
    start_time = time.time()

    #start the imaging
    proc1 = Popen(["dd if=/dev/sda1 of=image.dd bs=524288 count=3000"], bufsize=1, stderr=PIPE, shell=True)

    #define and start the queue function thread
    t = Thread(target=enqueue, args=(proc1.stderr, q))
    t.daemon = True
    t.start()

    #define and start the getstatus function thread
    t_getstatus = Thread(target=getstatus, args=())
    t_getstatus.daemon
    t_getstatus.start()

    #get the string from the queue

    while proc1.poll() == None:
        try: nline = q.get_nowait()
        except Empty:
            continue
        else:
            mystr = nline.decode('utf-8')           
            if mystr.find('bytes') &gt; 0:
                log_fh.write(str(time.time()) + ' - ' + parsestring(mystr))
                log_fh.flush()

        #put in a delay
        #time.sleep(2)

    #print duration
    end_time=time.time()
    duration=end_time-start_time
    print('Took ' + str(duration) + ' seconds')     
</code></pre>

<p>The only issue is I can't work out how to improve performance. I only need it to report status every 2 seconds or so but increasing the time delay increases the time of the imaging, which I don't want. That's a question for another post though...</p>

<p>Thanks to both J.F. Sebastian and Ali.</p>