Large number of threads under unicorn
<p>I am in the process of debugging some Posgtres connection leaks in our application. Few days back we suddenly crossed 100 connections when we should not be - coz we just have 8 unicorn workers and a sidekiq process (25 threads).</p>

<p>I was looking at htop today and saw that a ton of threads were being spawned from my unicorn workers. Eg:</p>

<p><a href="https://i.stack.imgur.com/Mkzc8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Mkzc8.png" alt="enter image description here"></a>
Am I reading this correctly? This should not be happening right? If these are threads being spawned, any idea how to debug this?</p>

<p>Thanks! Btw, my other problem - (Postgres connections) <a href="https://stackoverflow.com/questions/36484097/debugging-unicorn-postgres-connection-leak">Debugging unicorn postgres connection leak</a></p>

<p><strong><em>EDIT</em></strong></p>

<p>I just followed some tips here - <a href="http://varaneckas.com/blog/ruby-tracing-threads-unicorn/" rel="nofollow noreferrer">http://varaneckas.com/blog/ruby-tracing-threads-unicorn/</a> - and when I printed the stack trace from the workers' threads, here's what I got when there are many threads..</p>

<pre><code>[17176] ---
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `pop'
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `block in spawn_threadpool'
[17176] ---
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `pop'
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `block in spawn_threadpool'
[17176] ---
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `pop'
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `block in spawn_threadpool'
[17176] ---
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `pop'
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `block in spawn_threadpool'
[17176] ---
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `pop'
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `block in spawn_threadpool'
[17176] ---
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `pop'
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `block in spawn_threadpool'
[17176] ---
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `pop'
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `block in spawn_threadpool'
[17176] ---
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `pop'
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `block in spawn_threadpool'
[17176] ---
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `pop'
[17176] /u/apps/eventstream_production/shared/bundle/ruby/2.2.0/gems/eventmachine-1.0.8/lib/eventmachine.rb:1057:in `block in spawn_threadpool'
[17176] ---
[17176] -------------------
</code></pre>

<p>This is my unicorn.rb <a href="https://gist.github.com/steverob/b83e41bb49d78f9aa32f79136df5af5f" rel="nofollow noreferrer">https://gist.github.com/steverob/b83e41bb49d78f9aa32f79136df5af5f</a> and it spawns a thread for EventMachine in after_fork. </p>

<p>The reason for EventMachine is this --> <a href="https://github.com/keenlabs/keen-gem#asynchronous-publishing" rel="nofollow noreferrer">https://github.com/keenlabs/keen-gem#asynchronous-publishing</a> </p>

<p>Is this normal? Shouldn't the threads be killed? Could this be also causing unncessary db connections to be open?
Thanks</p>

<p><strong>UPDATE:</strong>
I just found out that I am using an older version of the PubNub gem which uses EM and I ran into these lines in the pubnub.log file - </p>

<pre><code>D, [2016-04-06T21:31:12.130123 #1573] DEBUG -- pubnub: Created event Pubnub::Publish
D, [2016-04-06T21:31:12.130144 #1573] DEBUG -- pubnub: Pubnub::SingleEvent#fire
D, [2016-04-06T21:31:12.130162 #1573] DEBUG -- pubnub: Pubnub::SingleEvent#fire | Adding event to async_events
D, [2016-04-06T21:31:12.130178 #1573] DEBUG -- pubnub: Pubnub::SingleEvent#fire | Starting railgun
D, [2016-04-06T21:31:12.130194 #1573] DEBUG -- pubnub: Pubnub::Client#start_event_machine | starting EM in new thread
D, [2016-04-06T21:31:12.130243 #1573] DEBUG -- pubnub: Pubnub::Client#start_event_machine | We aren't running on thin
D, [2016-04-06T21:31:12.130264 #1573] DEBUG -- pubnub: Pubnub::Client#start_event_machine | EM already running
</code></pre>