{"action":"created","issue":{"url":"https://api.github.com/repos/warmshowers/Warmshowers.org/issues/562","labels_url":"https://api.github.com/repos/warmshowers/Warmshowers.org/issues/562/labels{/name}","comments_url":"https://api.github.com/repos/warmshowers/Warmshowers.org/issues/562/comments","events_url":"https://api.github.com/repos/warmshowers/Warmshowers.org/issues/562/events","html_url":"https://github.com/warmshowers/Warmshowers.org/issues/562","id":84574229,"number":562,"title":"php5-fpm lockup daily - serious system problem","user":{"login":"rfay","id":112444,"avatar_url":"https://avatars.githubusercontent.com/u/112444?v=3","gravatar_id":"","url":"https://api.github.com/users/rfay","html_url":"https://github.com/rfay","followers_url":"https://api.github.com/users/rfay/followers","following_url":"https://api.github.com/users/rfay/following{/other_user}","gists_url":"https://api.github.com/users/rfay/gists{/gist_id}","starred_url":"https://api.github.com/users/rfay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rfay/subscriptions","organizations_url":"https://api.github.com/users/rfay/orgs","repos_url":"https://api.github.com/users/rfay/repos","events_url":"https://api.github.com/users/rfay/events{/privacy}","received_events_url":"https://api.github.com/users/rfay/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-06-03T13:48:22Z","updated_at":"2015-06-12T19:15:38Z","closed_at":null,"body":"In the last several days we've been having many times where php5-fpm locks up and stops responding, taking the site offline. Log says `Jun 02 16:29:54 warmshowers php-fpm:  [WARNING] [pool www] server reached pm.max_children setting (75), consider raising it`. In the moments leading up to that we see lots of children being spawned in just a few seconds. After it happens it seems that all children are unresponsive.\r\n\r\nThe quick fix is\r\n\r\n`sudo service php5-fpm restart`\r\n\r\n### Diagnostics\r\n\r\n* To see the php5-fpm status report, use /usr/local/bin/check_fpm - if it responds, note the status\r\n* To see the php5-fpm status report via nginx, use curl localhost/status. If that doesn't work but the direct report does, it probably means something about nginx/php-fpm cooperation.\r\n\r\n### Resources:\r\n* [nginx forum on seeming exact problem](http://forum.nginx.org/read.php?3,224236,224236#msg-224236)\r\n\r\n### Todo:\r\n* Turn on php5-fpm slow log\r\n* Make sure that mysql slow log is actually going to papertrail\r\n\r\n### Things that have been done\r\n* Set up a much larger set of children/workers\r\n* Added monitoring on newrelic\r\n* Changed config to recycle php5-fpm workers more often\r\n* Added tracking of mysql slow query log to papertrail\r\n* Emit a watchdog when the country_count page is being updated.\r\n* Check that all tables are innodb\r\n* Turned on `catch_workers_output = yes` in fpm config so we can see when they emit issues.\r\n"},"comment":{"url":"https://api.github.com/repos/warmshowers/Warmshowers.org/issues/comments/111592447","html_url":"https://github.com/warmshowers/Warmshowers.org/issues/562#issuecomment-111592447","issue_url":"https://api.github.com/repos/warmshowers/Warmshowers.org/issues/562","id":111592447,"user":{"login":"rfay","id":112444,"avatar_url":"https://avatars.githubusercontent.com/u/112444?v=3","gravatar_id":"","url":"https://api.github.com/users/rfay","html_url":"https://github.com/rfay","followers_url":"https://api.github.com/users/rfay/followers","following_url":"https://api.github.com/users/rfay/following{/other_user}","gists_url":"https://api.github.com/users/rfay/gists{/gist_id}","starred_url":"https://api.github.com/users/rfay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rfay/subscriptions","organizations_url":"https://api.github.com/users/rfay/orgs","repos_url":"https://api.github.com/users/rfay/repos","events_url":"https://api.github.com/users/rfay/events{/privacy}","received_events_url":"https://api.github.com/users/rfay/received_events","type":"User","site_admin":false},"created_at":"2015-06-12T19:15:38Z","updated_at":"2015-06-12T19:15:38Z","body":"No more SIGTERMs. Active processes most of the time at 1-3. Max was 20, before that was at 10.\r\n\r\nEvery 3 hours or so all 75 of the processes restart (after exiting with a 0). I imagine this is the result of the higher max active. They're all set to restart after handling 500 requests, and it seems that all are being treated equally, so they end up restarting around the same time.\r\n\r\nHoping this weekend goes well. If it does we'll:\r\n\r\n- Trim the number of processes in the pool down to 40 or so\r\n- Turn on the php slow log again to see what we see.\r\n\r\nOverall, my take is that switching to static pool was the key win so far, then that turning off fastcgi_keep_conn made for much more appropriate use. \r\n\r\nI think I understand why the serialization warnings happen, see https://github.com/warmshowers/Warmshowers.org/issues/564#issuecomment-111301600, and IMO we should just ignore them right now.\r\n\r\nCrossing fingers for the weekend."}}