{"action":"opened","issue":{"url":"https://api.github.com/repos/phusion/passenger/issues/1414","labels_url":"https://api.github.com/repos/phusion/passenger/issues/1414/labels{/name}","comments_url":"https://api.github.com/repos/phusion/passenger/issues/1414/comments","events_url":"https://api.github.com/repos/phusion/passenger/issues/1414/events","html_url":"https://github.com/phusion/passenger/issues/1414","id":60332187,"number":1414,"title":"[5.0.2] All workers stuck in Rack Exception","user":{"login":"felixbuenemann","id":909587,"avatar_url":"https://avatars.githubusercontent.com/u/909587?v=3","gravatar_id":"","url":"https://api.github.com/users/felixbuenemann","html_url":"https://github.com/felixbuenemann","followers_url":"https://api.github.com/users/felixbuenemann/followers","following_url":"https://api.github.com/users/felixbuenemann/following{/other_user}","gists_url":"https://api.github.com/users/felixbuenemann/gists{/gist_id}","starred_url":"https://api.github.com/users/felixbuenemann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/felixbuenemann/subscriptions","organizations_url":"https://api.github.com/users/felixbuenemann/orgs","repos_url":"https://api.github.com/users/felixbuenemann/repos","events_url":"https://api.github.com/users/felixbuenemann/events{/privacy}","received_events_url":"https://api.github.com/users/felixbuenemann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-03-09T11:50:59Z","updated_at":"2015-03-09T11:50:59Z","closed_at":null,"body":"After upgrading one of our servers to 5.0.2 we are seeing the following errors:\r\n```\r\nApp 15686 stderr: [ 2015-03-09 11:23:55.6927 16170/0x000000027c5400(Worker 1) utils.rb:85 ]: *** Exception RuntimeError in Rack body object #each method (can't modify frozen String) (process 16170, thread 0x000000027c5400\r\n(Worker 1)):\r\nApp 15686 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:264:in `force_encoding'\r\nApp 15686 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:264:in `force_binary'\r\nApp 15686 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:178:in `block in process_body'\r\nApp 15686 stderr:       from /srv/www/schomburg-gmbh-frontend/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/body_proxy.rb:31:in `each'\r\nApp 15686 stderr:       from /srv/www/schomburg-gmbh-frontend/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/body_proxy.rb:31:in `each'\r\nApp 15686 stderr:       from /srv/www/schomburg-gmbh-frontend/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/body_proxy.rb:31:in `each'\r\nApp 15686 stderr:       from /srv/www/schomburg-gmbh-frontend/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/body_proxy.rb:31:in `each'\r\nApp 15686 stderr:       from /srv/www/schomburg-gmbh-frontend/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/body_proxy.rb:31:in `each'\r\nApp 15686 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:177:in `process_body'\r\nApp 15686 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:103:in `process_request'\r\nApp 15686 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'\r\nApp 15686 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'\r\nApp 15686 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:414:in `block (3 levels) in start_threads'\r\nApp 15686 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:111:in `block in create_thread_and_abort_on_exception'\r\n```\r\n\r\nAfter some time this blocks up all workers and the server starts throwing 503 errors:\r\n```\r\n[ 2015-03-09 11:33:52.1094 11159/7fc999d83700 agents/HelperAgent/RequestHandler/CheckoutSession.cpp:236 ]: [Client 1-552] Returning HTTP 503 due to: Request queue full (configured max. size: 100)\r\n[ 2015-03-09 11:33:53.9685 11159/7fc999582700 agents/HelperAgent/RequestHandler/CheckoutSession.cpp:236 ]: [Client 2-552] Returning HTTP 503 due to: Request queue full (configured max. size: 100)\r\n[ 2015-03-09 11:33:57.1930 11159/7fc998d81700 agents/HelperAgent/RequestHandler/CheckoutSession.cpp:236 ]: [Client 3-552] Returning HTTP 503 due to: Request queue full (configured max. size: 100)\r\n```\r\n\r\nPool status in stuck state:\r\n```\r\nVersion : 5.0.2\r\nDate    : 2015-03-09 11:51:35 +0100\r\nInstance: W3sT24zH (nginx/1.6.2 Phusion_Passenger/5.0.2)\r\n\r\n----------- General information -----------\r\nMax pool size : 6\r\nProcesses     : 6\r\nRequests in top-level queue : 0\r\n\r\n----------- Application groups -----------\r\n/srv/www/appname/web/staging/current/public (staging)#default:\r\n  App root: /srv/www/appname/web/staging/current\r\n  Requests in queue: 0\r\n  * PID: 15657   Sessions: 0       Processed: 16      Uptime: 34m 51s\r\n    CPU: 0%      Memory  : 242M    Last used: 29m 16s ago\r\n\r\n/srv/www/appname/web/production/current/public (production)#default:\r\n  App root: /srv/www/appname/web/production/current\r\n  Requests in queue: 100\r\n  * PID: 11217   Sessions: 1       Processed: 1416    Uptime: 1h 45m 9s\r\n    CPU: 2%      Memory  : 309M    Last used: 27m 41s a\r\n  * PID: 15709   Sessions: 1       Processed: 9       Uptime: 34m 29s\r\n    CPU: 0%      Memory  : 145M    Last used: 27m 41s ago\r\n  * PID: 15850   Sessions: 1       Processed: 1       Uptime: 31m 28s\r\n    CPU: 0%      Memory  : 107M    Last used: 27m 41s ago\r\n  * PID: 16158   Sessions: 1       Processed: 0       Uptime: 27m 44s\r\n    CPU: 0%      Memory  : 94M     Last used: 27m 41s ago\r\n  * PID: 16170   Sessions: 1       Processed: 0       Uptime: 27m 40s\r\n    CPU: 0%      Memory  : 94M     Last used: 27m 40s ago\r\n```\r\n\r\nFor now I've switched to the v4 PPA and downgraded to 4.0.59 which is not affected."}}