Windows TCP/socket/thread/buffer exhaustion (c#)
<p><strong>Foreword</strong>:
In a VMware environment we are running a windows 2012 server. On the server we have several application running in windows processes.
One process are opening a socket (tcp server) and listen for clients. There are 3000 clients connecting and sending data to the server process.
A scenario could be that each client sends 50 messages to the server process (3000 *50 = 150000 msg to process)
A message is decrypted with a key that is requested from a shared keystore(served from another server eg. a network call) </p>

<p><a href="http://i.stack.imgur.com/7cV5h.png" rel="nofollow">See diagram</a></p>

<p><strong>Problem</strong>
The tcp server process cannot handle the messages in time because the network call to the keyrepository is “slow”. The threadcount increases as the “async receiving” buffer up threads(C# socket.ReceiveAsync()). That problem I understand. 
BUT when other threads in that process, unrelated to socket receiving, needs to make a network connection to eg. a datastore: An connection exception raises:"An exception occurred while opening a connection to the server" - this is only periodic when the tcpserver is under full load. </p>

<p>Can anybody explain how this is related? Or how I can troubleshoot or investigate such issue?</p>