{"action":"created","issue":{"url":"https://api.github.com/repos/phusion/passenger/issues/1621","labels_url":"https://api.github.com/repos/phusion/passenger/issues/1621/labels{/name}","comments_url":"https://api.github.com/repos/phusion/passenger/issues/1621/comments","events_url":"https://api.github.com/repos/phusion/passenger/issues/1621/events","html_url":"https://github.com/phusion/passenger/issues/1621","id":108569248,"number":1621,"title":"Stuck with Sessions: 1","user":{"login":"JasonBarnabe","id":583995,"avatar_url":"https://avatars.githubusercontent.com/u/583995?v=3","gravatar_id":"","url":"https://api.github.com/users/JasonBarnabe","html_url":"https://github.com/JasonBarnabe","followers_url":"https://api.github.com/users/JasonBarnabe/followers","following_url":"https://api.github.com/users/JasonBarnabe/following{/other_user}","gists_url":"https://api.github.com/users/JasonBarnabe/gists{/gist_id}","starred_url":"https://api.github.com/users/JasonBarnabe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JasonBarnabe/subscriptions","organizations_url":"https://api.github.com/users/JasonBarnabe/orgs","repos_url":"https://api.github.com/users/JasonBarnabe/repos","events_url":"https://api.github.com/users/JasonBarnabe/events{/privacy}","received_events_url":"https://api.github.com/users/JasonBarnabe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/phusion/passenger/labels/SupportCentral","name":"SupportCentral","color":"fef2c0"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-09-28T00:04:04Z","updated_at":"2015-12-29T11:05:12Z","closed_at":null,"body":"Since upgrading to Passenger 5.0.18 (CentOS 6.6, Rails), my site has been going down every few days. In the nginx log, I see:\r\n\r\n```\r\n[ 2015-09-27 23:32:36.4234 17309/7f5e2abfd700 age/Cor/Req/CheckoutSession.cpp:236 ]: [Client 10-52676] Returning HTTP 503 due to: Request queue full (configured max. size: 100)\r\n```\r\nHowever, `passenger-status` reports nothing in the queue:\r\n```\r\n[www@mysite ~]$ passenger-status\r\nVersion : 5.0.18\r\nDate    : 2015-09-27 23:32:48 +0000\r\nInstance: 8MvPlgUW (nginx/1.8.0 Phusion_Passenger/5.0.18)\r\n\r\n----------- General information -----------\r\nMax pool size : 8\r\nApp groups    : 1\r\nProcesses     : 8\r\nRequests in top-level queue : 0\r\n\r\n----------- Application groups -----------\r\n/home/www/mysite/public:\r\n  App root: /home/www/mysite\r\n  Requests in queue: 100\r\n  * PID: 13478   Sessions: 1       Processed: 7499    Uptime: 8h 19m 20s\r\n    CPU: 0%      Memory  : 327M    Last used: 7h 6m 17\r\n  * PID: 20515   Sessions: 1       Processed: 2240    Uptime: 8h 2m 9s\r\n    CPU: 0%      Memory  : 295M    Last used: 7h 6m 6s a\r\n  * PID: 28619   Sessions: 1       Processed: 488     Uptime: 7h 39m 47s\r\n    CPU: 0%      Memory  : 200M    Last used: 7h 6m 3s\r\n  * PID: 29059   Sessions: 1       Processed: 83      Uptime: 7h 38m 19s\r\n    CPU: 0%      Memory  : 187M    Last used: 7h 6m 2s\r\n  * PID: 5537    Sessions: 1       Processed: 26      Uptime: 7h 15m 16s\r\n    CPU: 0%      Memory  : 224M    Last used: 7h 6m 0s\r\n  * PID: 9359    Sessions: 1       Processed: 194     Uptime: 7h 5m 58s\r\n    CPU: 0%      Memory  : 208M    Last used: 7h 5m 29s\r\n  * PID: 2652    Sessions: 1       Processed: 7329    Uptime: 43m 20s\r\n    CPU: 9%      Memory  : 259M    Last used: 9m 56s ago\r\n  * PID: 11210   Sessions: 1       Processed: 2172    Uptime: 23m 20s\r\n    CPU: 5%      Memory  : 207M    Last used: 8m 44s ago\r\n```\r\nIt also seems odd to have processes that haven't done anything in 7 hours.\r\n\r\nLess than a minute before the \"Request queue full\" entries started, this was in the log, unsure if related:\r\n```\r\nApp 11102 stderr: [ 2015-09-27 23:23:44.8973 11210/0x0000000775d878(Worker 1) utils.rb:86 ]: *** Exception Errno::ECONNRESET in Rack response body object (Connection \r\nreset by peer) (process 11210, thread 0x0000000775d878(Worker 1)):\r\nApp 11102 stderr:       from /home/www/.rvm/gems/ruby-2.2.2/gems/passenger-5.0.18/lib/phusion_passenger/utils/unseekable_socket.rb:114:in `write'\r\nApp 11102 stderr:       from /home/www/.rvm/gems/ruby-2.2.2/gems/passenger-5.0.18/lib/phusion_passenger/utils/unseekable_socket.rb:114:in `write'\r\nApp 11102 stderr:       from /home/www/.rvm/gems/ruby-2.2.2/gems/passenger-5.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:303:in `process_body'\r\nApp 11102 stderr:       from /home/www/.rvm/gems/ruby-2.2.2/gems/passenger-5.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:143:in `process_request'\r\nApp 11102 stderr:       from /home/www/.rvm/gems/ruby-2.2.2/gems/passenger-5.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_n\r\next_request'\r\nApp 11102 stderr:       from /home/www/.rvm/gems/ruby-2.2.2/gems/passenger-5.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'\r\nApp 11102 stderr:       from /home/www/.rvm/gems/ruby-2.2.2/gems/passenger-5.0.18/lib/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads'\r\nApp 11102 stderr:       from /home/www/.rvm/gems/ruby-2.2.2/gems/passenger-5.0.18/lib/phusion_passenger/utils.rb:112:in `block in create_thread_and_abort_on_exception\r\n'\r\n[ 2015-09-27 23:24:23.6833 17309/7f5e5a1fc700 age/Cor/Req/CheckoutSession.cpp:236 ]: [Client 5-52522] Returning HTTP 503 due to: Request queue full (configured max. s\r\nize: 100)\r\n```\r\nResults of `--show=` `backtraces`, `server`, and `union-station` at https://gist.github.com/JasonBarnabe/86128084d3a46efb2e2c"},"comment":{"url":"https://api.github.com/repos/phusion/passenger/issues/comments/167765922","html_url":"https://github.com/phusion/passenger/issues/1621#issuecomment-167765922","issue_url":"https://api.github.com/repos/phusion/passenger/issues/1621","id":167765922,"user":{"login":"OnixGH","id":10709325,"avatar_url":"https://avatars.githubusercontent.com/u/10709325?v=3","gravatar_id":"","url":"https://api.github.com/users/OnixGH","html_url":"https://github.com/OnixGH","followers_url":"https://api.github.com/users/OnixGH/followers","following_url":"https://api.github.com/users/OnixGH/following{/other_user}","gists_url":"https://api.github.com/users/OnixGH/gists{/gist_id}","starred_url":"https://api.github.com/users/OnixGH/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OnixGH/subscriptions","organizations_url":"https://api.github.com/users/OnixGH/orgs","repos_url":"https://api.github.com/users/OnixGH/repos","events_url":"https://api.github.com/users/OnixGH/events{/privacy}","received_events_url":"https://api.github.com/users/OnixGH/received_events","type":"User","site_admin":false},"created_at":"2015-12-29T11:05:12Z","updated_at":"2015-12-29T11:05:12Z","body":"@antoineco the output you posted doesn't necessarily mean there is a problem with Passenger. For example, if your application stops returning responses, this will also happen.\r\n\r\nPlease make sure that it's not a problem in your application by following the steps outlined in: http://old.blog.phusion.nl/2012/09/21/the-right-way-to-deal-with-frozen-processes-on-unix/\r\n\r\nIf you do think it's a Passenger problem, please post information about your setup (platform, what language is your app, ruby/rails/nodejs/etc. versions, how often does it happen, anything in particular worth mentioning), and:\r\n- the output of `passenger-status --show=requests` (on a stuck server)\r\n- any errors from the logfile\r\n- any other helpful output like backtraces from crash-watch"}}