Mono - attempt to close TCP connection, thread hangs, connection gets stuck in CLOSE_WAIT
<p>We have a situation where a server running mono <s>4.2.2</s> 4.4 on Linux receives multiple incoming TCP connections from clients. It also maintains a single outgoing TCP connection to another system, and regularly makes HTTP REST requests to yet another system. After an arbitrary length of time, the server grinds to a halt - the number of connections shoots up and then drops off (see graph below), thread dumps show large numbers of connections, predominantly the incoming connections from clients, stuck trying to shut down (see stack trace below). Running <code>netstat</code> on the server shows a large number of connections stuck in CLOSE_WAIT.</p>

<p>As far as we can tell, the sudden increase in connections doesn't correspond to an actual sudden surge in load - new connections are actually balanced across 2 instances of the server on two different machines, and the other machine carries on happily. The other server does see quite a rise in connections in the few minutes after the first fails, but this is to be expected after one server has become unresponsive, and the rise is much more gradual, over the course of minutes rather than a sudden spike.</p>

<pre><code>"Threadpool worker"  at &lt;unknown&gt; &lt;0xffffffff&gt;
  at (wrapper managed-to-native) System.Threading.WaitHandle.WaitOne_internal (System.Threading.WaitHandle,intptr,int,bool) &lt;0xffffffff&gt;
  at System.Threading.WaitHandle.WaitOne () &lt;0x0005c&gt;
  at System.Net.Sockets.Socket.EndSend (System.IAsyncResult,System.Net.Sockets.SocketError&amp;) &lt;0x0006f&gt;
  at System.Net.Sockets.Socket.EndSend (System.IAsyncResult) &lt;0x0002b&gt;
  at System.Net.Sockets.NetworkStream.EndWrite (System.IAsyncResult) &lt;0x00057&gt;
  at Mono.Security.Protocol.Tls.RecordProtocol.EndSendRecord (System.IAsyncResult) &lt;0x000dc&gt;
  at Mono.Security.Protocol.Tls.RecordProtocol.SendRecord (Mono.Security.Protocol.Tls.ContentType,byte[]) &lt;0x00037&gt;
  at Mono.Security.Protocol.Tls.RecordProtocol.SendAlert (Mono.Security.Protocol.Tls.Alert) &lt;0x0009f&gt;
  at Mono.Security.Protocol.Tls.RecordProtocol.SendAlert (Mono.Security.Protocol.Tls.AlertDescription) &lt;0x0004b&gt;
  at Mono.Security.Protocol.Tls.SslStreamBase.Dispose (bool) &lt;0x0008b&gt;
  at Mono.Security.Protocol.Tls.SslServerStream.Dispose (bool) &lt;0x00017&gt;
  at System.IO.Stream.Close () &lt;0x00019&gt;
  at Mono.Security.Protocol.Tls.SslStreamBase.Close () &lt;0x00014&gt;
  at System.IO.Stream.Dispose () &lt;0x00013&gt;
  at (wrapper remoting-invoke-with-check) System.IO.Stream.Dispose () &lt;0xffffffff&gt;
  at System.Net.Security.SslStream.Dispose (bool) &lt;0x00037&gt;
  at System.IO.Stream.Close () &lt;0x00019&gt;
  at CompanyName.Networking.Connection.Close () &lt;0x00057&gt;
  at CompanyName.Networking.Connection.ProcessBytes (int) &lt;0x00202&gt;
  at CompanyName.Networking.Connection.DataReceivedAsync (System.IAsyncResult) &lt;0x0006f&gt;
  at (wrapper runtime-invoke) &lt;Module&gt;.runtime_invoke_void__this___object (object,intptr,intptr,intptr) &lt;0xffffffff&gt;
  at &lt;unknown&gt; &lt;0xffffffff&gt;
  at (wrapper managed-to-native) System.Runtime.Remoting.Messaging.AsyncResult.Invoke (System.Runtime.Remoting.Messaging.AsyncResult) &lt;0xffffffff&gt;
  at System.Runtime.Remoting.Messaging.AsyncResult.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () &lt;0x0000c&gt;
  at System.Threading.ThreadPoolWorkQueue.Dispatch () &lt;0x001d6&gt;
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () &lt;0x00008&gt;
  at (wrapper runtime-invoke) &lt;Module&gt;.runtime_invoke_bool (object,intptr,intptr,intptr) &lt;0xffffffff&gt;
</code></pre>

<p>Only three lines of the stacktrace are in our code. Given that they are <code>DataReceivedAsync -&gt; ProcessBytes -&gt; Close</code>, this means that the earlier <code>stream.BeginRead</code> has triggered the callback we give it (<code>DataReceivedAsync</code>), which calls <code>EndRead</code> and finds that 0 bytes have been read. This causes <code>ProcessBytes</code> to conclude that the TCP connection has been closed by the other end, and it calls our <code>Close()</code> method, which calls <code>Close()</code> on the <code>SslStream</code> and the <code>TcpClient</code>. Looking at the stacktrace, the <code>Close()</code> call on the <code>SslStream</code> is where we're getting stuck.</p>

<p><a href="https://i.stack.imgur.com/Hw3ah.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Hw3ah.png" alt="Concurrent connections graph"></a></p>

<p>The usual response to "why is my connection stuck in CLOSE_WAIT?" is "you need to close the connection". However, we're attempting to close the connection and finding that the framework code hangs on a <code>WaitHandle</code>. How can we resolve this issue?</p>