{"action":"created","issue":{"url":"https://api.github.com/repos/GrahamDumpleton/mod_wsgi/issues/81","labels_url":"https://api.github.com/repos/GrahamDumpleton/mod_wsgi/issues/81/labels{/name}","comments_url":"https://api.github.com/repos/GrahamDumpleton/mod_wsgi/issues/81/comments","events_url":"https://api.github.com/repos/GrahamDumpleton/mod_wsgi/issues/81/events","html_url":"https://github.com/GrahamDumpleton/mod_wsgi/issues/81","id":94403134,"number":81,"title":"Daemon segfault after a few graceful reloads with multiple threads","user":{"login":"linuxflower","id":12970280,"avatar_url":"https://avatars.githubusercontent.com/u/12970280?v=3","gravatar_id":"","url":"https://api.github.com/users/linuxflower","html_url":"https://github.com/linuxflower","followers_url":"https://api.github.com/users/linuxflower/followers","following_url":"https://api.github.com/users/linuxflower/following{/other_user}","gists_url":"https://api.github.com/users/linuxflower/gists{/gist_id}","starred_url":"https://api.github.com/users/linuxflower/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linuxflower/subscriptions","organizations_url":"https://api.github.com/users/linuxflower/orgs","repos_url":"https://api.github.com/users/linuxflower/repos","events_url":"https://api.github.com/users/linuxflower/events{/privacy}","received_events_url":"https://api.github.com/users/linuxflower/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-07-10T22:08:19Z","updated_at":"2015-10-04T11:08:45Z","closed_at":null,"body":"We are running on our own Debian Jessie derivative with apache 2.4.10 and mod-wsgi (4.4.13) and experience a segfault upon multiple reloads of apache. The segfault also seems to be dependent on amount of time between reloads. For example, if we do `for i in `seq 1 10`; do echo $i; sudo service apache2 reload; sleep 2; done;` it will fail on the 7th reload every time. If we increase the sleep in between reloads to sleep 3, all 10 reloads are issued successfully every time.\r\n\r\nOur Jessie derivative is the base OS for openstack so we have several openstack processes running:\r\n$ ps -ef | grep apache2\r\nroot     27505 33129  0 14:52 pts/1    00:00:00 sudo vi /var/log/apache2/error.log\r\nroot     27506 27505  0 14:52 pts/1    00:00:00 vi /var/log/apache2/error.log\r\nroot     31470     1  0 15:07 ?        00:00:00 /usr/sbin/apache2 -k start\r\nceilome+ 31474 31470  0 15:07 ?        00:00:39 /usr/sbin/apache2 -k start\r\nceilome+ 31475 31470  0 15:07 ?        00:00:40 /usr/sbin/apache2 -k start\r\nceilome+ 31476 31470  0 15:07 ?        00:00:40 /usr/sbin/apache2 -k start\r\nceilome+ 31477 31470  0 15:07 ?        00:00:39 /usr/sbin/apache2 -k start\r\nhorizon  31478 31470  1 15:07 ?        00:03:27 /usr/sbin/apache2 -k start\r\nhorizon  31479 31470  1 15:07 ?        00:03:28 /usr/sbin/apache2 -k start\r\nhorizon  31480 31470  1 15:07 ?        00:03:24 /usr/sbin/apache2 -k start\r\nkeystone 31481 31470  0 15:07 ?        00:01:53 /usr/sbin/apache2 -k start\r\nkeystone 31482 31470  0 15:07 ?        00:01:50 /usr/sbin/apache2 -k start\r\nkeystone 31483 31470  1 15:07 ?        00:02:16 /usr/sbin/apache2 -k start\r\nkeystone 31484 31470  1 15:07 ?        00:02:15 /usr/sbin/apache2 -k start\r\nwww-data 31485 31470  0 15:07 ?        00:00:37 /usr/sbin/apache2 -k start\r\nwww-data 31486 31470  0 15:07 ?        00:01:06 /usr/sbin/apache2 -k start\r\nstack    32639 25024  0 18:38 pts/3    00:00:00 grep apache2\r\n\r\nIf i do a `sudo service apache2 reload` anywhere from 3-10 times, at some point it will fail. Note that this happens if there is less than 1 second in between reloads:\r\n$ sudo service apache2 reload\r\nJob for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.\r\n$ systemctl status apache2.service\r\n● apache2.service - LSB: Apache2 web server\r\n   Loaded: loaded (/etc/init.d/apache2)\r\n   Active: active (running) (Result: exit-code) since Fri 2015-07-10 15:07:19 EDT; 5h 35min ago\r\n  Process: 31436 ExecStop=/etc/init.d/apache2 stop (code=exited, status=1/FAILURE)\r\n  Process: 13873 ExecReload=/etc/init.d/apache2 reload (code=exited, status=1/FAILURE)\r\n  Process: 31449 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)\r\n   CGroup: /system.slice/apache2.service\r\n           └─13413 /usr/sbin/apache2 -k start\r\n$ journalctl -xn\r\nNo journal files were found.\r\n\r\nAnalyzing the core dump file with gdb yields:\r\n$ sudo gdb /usr/sbin/apache2 /tmp/mycoredump/core \r\nGNU gdb (Debian 7.7.1+dfsg-5) 7.7.1\r\nCopyright (C) 2014 Free Software Foundation, Inc.\r\nLicense GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\r\nThis is free software: you are free to change and redistribute it.\r\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\r\nand \"show warranty\" for details.\r\nThis GDB was configured as \"x86_64-linux-gnu\".\r\nType \"show configuration\" for configuration details.\r\nFor bug reporting instructions, please see:\r\n<http://www.gnu.org/software/gdb/bugs/>.\r\nFind the GDB manual and other documentation resources online at:\r\n<http://www.gnu.org/software/gdb/documentation/>.\r\nFor help, type \"help\".\r\nType \"apropos word\" to search for commands related to \"word\"...\r\nReading symbols from /usr/sbin/apache2...done.\r\n[New LWP 31470]\r\n[Thread debugging using libthread_db enabled]\r\nUsing host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\r\nCore was generated by `/usr/sbin/apache2 -k start'.\r\nProgram terminated with signal SIGSEGV, Segmentation fault.\r\n#0  0x00007f1526f92ad6 in ?? ()\r\n\r\nIn the /var/log/apache2/error.log, each successful reload causes this sequence:\r\n*  SIGUSR1 received\r\n* services are deregistered\r\n* destroy/cleanup interpreters/terminate python for each pid\r\n* attach interpreter\r\n\r\nFor the situation where we use a sleep 2 (or less) in between reloads, we see in the error log:\r\n*  SIGUSR1 received\r\n* services are deregistered\r\n* destroy/cleanup interpreters/terminate python for each pid\r\n*[core:notice] [pid 48512] AH00060: seg fault or similar nasty error detected in the parent process\r\nAnd the attach interpreter never happens.\r\n\r\nThen apache2 has to just be restarted (sudo service apache2 restart) before we can do any further reloads. Also, searching around online, I found https://bugs.launchpad.net/ubuntu/+source/redland-bindings/+bug/1416875 which may be related.\r\n\r\nLooking at one of the virtual hosts conf file, it's running mod_wsgi in daemon mode and everything looks fine to me:\r\n\r\nListen 127.0.0.1:8777\r\nWSGIPythonHome /opt/stack/service/ceilometer-common/venv/bin/../\r\n<VirtualHost *:8777>\r\n  WSGIScriptAlias / /opt/stack/service/ceilometer-common/venv/bin/../lib/python2.7/site-packages/ceilometer/api/app.wsgi\r\n  WSGIDaemonProcess ceilometer user=ceilometer group=ceilometer processes=4 threads=5 python-path=/opt/stack/service/ceilometer-common/venv/bin/../lib/python2.7/site-packages\r\n  WSGIApplicationGroup ceilometer\r\n  WSGIProcessGroup ceilometer\r\n\r\n  ErrorLog /var/log/apache2/ceilometer_modwsgi.log\r\n  LogLevel info\r\n  CustomLog /var/log/apache2/ceilometer_access.log combined\r\n\r\n  <Directory /opt/stack/service/ceilometer-common/venv/bin/../lib/python2.7/site-packages/ceilometer>\r\n    Options Indexes FollowSymLinks MultiViews\r\n    Require all granted\r\n    AllowOverride None\r\n    Order allow,deny\r\n    allow from all\r\n    LimitRequestBody 102400\r\n  </Directory>\r\n</VirtualHost>\r\n\r\nDo you have any ideas of what could be causing the reloads to fail? Any advice or thoughts would be very useful. \r\n\r\nThank you for looking at this issue,\r\nHeather Brown\r\nhbrown@hp.com"},"comment":{"url":"https://api.github.com/repos/GrahamDumpleton/mod_wsgi/issues/comments/145336952","html_url":"https://github.com/GrahamDumpleton/mod_wsgi/issues/81#issuecomment-145336952","issue_url":"https://api.github.com/repos/GrahamDumpleton/mod_wsgi/issues/81","id":145336952,"user":{"login":"GrahamDumpleton","id":507637,"avatar_url":"https://avatars.githubusercontent.com/u/507637?v=3","gravatar_id":"","url":"https://api.github.com/users/GrahamDumpleton","html_url":"https://github.com/GrahamDumpleton","followers_url":"https://api.github.com/users/GrahamDumpleton/followers","following_url":"https://api.github.com/users/GrahamDumpleton/following{/other_user}","gists_url":"https://api.github.com/users/GrahamDumpleton/gists{/gist_id}","starred_url":"https://api.github.com/users/GrahamDumpleton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GrahamDumpleton/subscriptions","organizations_url":"https://api.github.com/users/GrahamDumpleton/orgs","repos_url":"https://api.github.com/users/GrahamDumpleton/repos","events_url":"https://api.github.com/users/GrahamDumpleton/events{/privacy}","received_events_url":"https://api.github.com/users/GrahamDumpleton/received_events","type":"User","site_admin":false},"created_at":"2015-10-04T11:08:45Z","updated_at":"2015-10-04T11:08:45Z","body":"Try 4.4.15 and see if issue goes away. That includes a fix for core dumps when doing graceful restarts and mod_wsgi was being loaded the first time. The fix was for a permanent failure and not a transient one though, so could well be different."}}