When is a Python program not responding to interrupts?
<p>I have a Python3 daemon process running on Linux. It is a normal single thread process running in the background and doing <code>select.select()</code> in the main loop and then handling I/O. Sometimes (approx. 1 or 2 times in a month) it stops responding. When it happens, I'd like to debug the problem.</p>

<p>I have tried the <code>pyrasite</code>, but was not succesfull, because stdin/stdout of the daemon is redirected to the <code>/dev/null</code> device and pyrasite uses this stdin/stdout, not the console where it was started from.</p>

<p>So I have added a <code>SIGUSR1</code> signal handler which logs the stack trace. Works fine normally.</p>

<p>Today I got a freeze. <code>ps</code> shows, the daemon is in "S" (interruptible sleep) state. A busy loop is ruled out. </p>

<p>The server does not respond to <code>SIGUSR</code> or <code>SIGINT</code> (used for shutdown).</p>

<p>I'd like to have at least some hint what is going on there.</p>

<p><strong>Under what conditions is a sleeping Python3 Linux process not handling interrupts that it is supposed to handle?</strong></p>

<hr>

<p><strong>UPDATE:</strong></p>

<p>I could reproduce the issue finally. After adding a lot of debug messages, I have found a race condition that I'm going to fix soon.</p>

<p>When the daemon is not responding, it is sleeping at <code>os.read(p)</code> where <code>p</code> is a read-end of a new pipe (see: <code>os.pipe</code>) where nobody writes to.</p>

<p>However, all my attempts to write a simple demonstration program failed. When I tried to read from an empty pipe, the program blocked as expected, but could be interrupted (killed from other terminal with SIGINT) as usual. The mystery remains unsolved.</p>

<hr>

<p><strong>UPDATE2:</strong></p>

<p>Finally some code! I have deliberately chosen low level system calls.</p>

<pre><code>import os
import time
import signal
import sys 

def sighandler(*unused):
    print("got signal", file=sys.stderr)

print("==========")
signal.signal(signal.SIGUSR1, sighandler)

pid = os.getpid()
rfd, wfd = os.pipe()
if os.fork():
    os.close(wfd)
    print("parent: read() start")
    os.read(rfd, 4096)
    print("parent: read() stop")
else:
    os.close(rfd)
    os.kill(pid, signal.SIGUSR1)
    print("child: wait start")
    time.sleep(3)
    print("child: wait end")
</code></pre>

<p>If you run this many time, you'll get this:</p>

<pre><code>parent: read() start
got signal
child: wait start
child: wait end
parent: read() stop
</code></pre>

<p>which is fine, but sometimes you'll see this:</p>

<pre><code>parent: read() start
child: wait start
child: wait end
got signal
parent: read() stop
</code></pre>

<p>What is happening here:</p>

<ol>
<li>parent starts a read from the pipe</li>
<li>child sends a signal to the parent. The parent must have received this signal, but it seems to be "somehow postponed"</li>
<li>child waits</li>
<li>child exits, pipe is closed automatically</li>
<li>parent's read operation ends with an EOF</li>
<li>the signal is handled now</li>
</ol>

<p>Now, due to a bug in my program, the signal was received in step 2, but the EOF was not delivered, so the read did not finish and step 6 (signal handling) was never reached.</p>

<p>That is all information I am able to provide.</p>