{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/hhvm/issues/4165","labels_url":"https://api.github.com/repos/facebook/hhvm/issues/4165/labels{/name}","comments_url":"https://api.github.com/repos/facebook/hhvm/issues/4165/comments","events_url":"https://api.github.com/repos/facebook/hhvm/issues/4165/events","html_url":"https://github.com/facebook/hhvm/issues/4165","id":48031019,"number":4165,"title":"HHVM in FastCGI mode hangs randomly in an apparent deadlock between the OpenSSL extension and PostgreSQL which also uses OpenSSL","user":{"login":"isegal","id":4561505,"avatar_url":"https://avatars.githubusercontent.com/u/4561505?v=3","gravatar_id":"","url":"https://api.github.com/users/isegal","html_url":"https://github.com/isegal","followers_url":"https://api.github.com/users/isegal/followers","following_url":"https://api.github.com/users/isegal/following{/other_user}","gists_url":"https://api.github.com/users/isegal/gists{/gist_id}","starred_url":"https://api.github.com/users/isegal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isegal/subscriptions","organizations_url":"https://api.github.com/users/isegal/orgs","repos_url":"https://api.github.com/users/isegal/repos","events_url":"https://api.github.com/users/isegal/events{/privacy}","received_events_url":"https://api.github.com/users/isegal/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/facebook/hhvm/labels/crash","name":"crash","color":"e63737"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2014-11-06T23:53:42Z","updated_at":"2015-02-06T22:32:33Z","closed_at":null,"body":"We are running in production using the nginx-FastCGI-hhvm setup and are experiencing occasional random lockups of hhvm process. It appears to be due to a deadlock from the interaction with openssl. We have collected GDB traces of the hanging threads below. Please let me know if I can help in any way. Unfortunately we do not have exact steps to reproduce due to the random nature, but we are able to collect more GDB information when it does happen, if needed.\r\n\r\n\r\nAreas of interest and possible theory:\r\n\r\nIt appears that openssl_random_pseudo_bytes(), which calls openssl, causes the underlying libcrypto to invoke a callback that was established previously by PostgreSQL library as part of the lock portability callbacks for multi-threading of openssl.\r\n\r\nSome information on the topic can be found here http://wiki.openssl.org/index.php/Manual:Threads(3)\r\n\r\nIf HHVM openssl extension does not establish these same callbacks, it may be causing the wrong callbacks to be called.\r\n\r\nAll worker threads except for one, hang in the following state:\r\n```\r\nThread 65 (Thread 0x7faa047ff700 (LWP 453)):\r\n #0  0x00007faa1c1dbf2c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0\r\n #1  0x00007faa1c1d7657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0\r\n #2  0x00007faa1c1d7480 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0\r\n\r\n #3  0x00007faa0ffd4510 in ?? () from /usr/lib/libpq.so.5 <-- **appears to be a callback from openssl to postgres**\r\n\r\n #4  0x00007faa1ef9c9aa in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0\r\n #5  0x00000000010d54e8 in HPHP::f_openssl_random_pseudo_bytes(int, HPHP::VRefParamValue const&) ()\r\n #6  0x0000000003ebf8b0 in ?? ()\r\n #7  0x0000000000ebe01e in enterTCHelper ()\r\n #8  0x0000000000d34a59 in HPHP::jit::x64::BackEnd::enterTCHelper(unsigned char*, HPHP::jit::TReqInfo&) ()\r\n #9  0x0000000000cdaba5 in HPHP::jit::MCGenerator::enterTC(unsigned char*, void*) ()\r\n #10 0x0000000000c3de5e in HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) ()\r\n #11 0x0000000000c3e0cc in HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) ()\r\n #12 0x0000000000c3e31a in HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) ()\r\n #13 0x0000000000972491 in HPHP::invoke_file(HPHP::String const&, bool, char const*) ()\r\n #14 0x0000000000974b2a in HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) ()\r\n #15 0x0000000000a78fc9 in HPHP::hphp_invoke(HPHP::ExecutionContext*, std::string const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::string const&, std::string const&, bool&, std::string&, bool, bool, bool) ()\r\n #16 0x0000000000b55e74 in HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) ()\r\n #17 0x0000000000b56fb0 in HPHP::HttpRequestHandler::handleRequestImpl(HPHP::Transport*) ()\r\n #18 0x0000000000b57ad9 in HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) ()\r\n #19 0x0000000000b73bf9 in HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) ()\r\n #20 0x0000000000b73e28 in HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) ()\r\n #21 0x0000000000b72d98 in HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() ()\r\n #22 0x00000000015f9687 in HPHP::AsyncFuncImpl::ThreadFunc(void*) ()\r\n #23 0x00007faa1c1d5182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0\r\n #24 0x00007faa1b6e1fbd in clone () from /lib/x86_64-linux-gnu/libc.so.6\r\n```\r\n\r\n\r\nOne worker hangs inside this state:\r\n```\r\nThread 39 (Thread 0x7fa9813ff700 (LWP 539)):\r\n #0  0x00007faa1c1dbf2c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0\r\n #1  0x00007faa1c1d7657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0\r\n #2  0x00007faa1c1d7480 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0\r\n\r\n #3  0x00007faa0ffd4510 in ?? () from /usr/lib/libpq.so.5 <-- **the same callback as above. This appears to be by-design.**\r\n\r\n #4  0x00007faa1ef9c511 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0\r\n #5  0x00007faa1f2ac79c in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0\r\n #6  0x00007faa0ffd53bb in ?? () from /usr/lib/libpq.so.5\r\n #7  0x00007faa0ffc5b76 in PQconnectPoll () from /usr/lib/libpq.so.5\r\n #8  0x00007faa0ffc6b8e in ?? () from /usr/lib/libpq.so.5\r\n #9  0x00007faa0ffc7497 in PQconnectdb () from /usr/lib/libpq.so.5\r\n #10 0x00007faa101f0da2 in HPHP::f_pg_connect(HPHP::String const&, int) () from /etc/hhvm/ext/pgsql.so\r\n #11 0x0000000000bd8135 in void HPHP::Native::callFunc<false, false>(HPHP::Func const*, void*, HPHP::TypedValue*, HPHP::TypedValue&) ()\r\n #12 0x0000000000bd81dc in HPHP::TypedValue* HPHP::Native::functionWrapper<false, false>(HPHP::ActRec*) ()\r\n #13 0x00000000047f6e94 in ?? ()\r\n #14 0x0000000000ebe01e in enterTCHelper ()\r\n #15 0x0000000000d34a59 in HPHP::jit::x64::BackEnd::enterTCHelper(unsigned char*, HPHP::jit::TReqInfo&) ()\r\n #16 0x0000000000cdaba5 in HPHP::jit::MCGenerator::enterTC(unsigned char*, void*) ()\r\n #17 0x0000000000c3de5e in HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) ()\r\n #18 0x0000000000c3e0cc in HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) ()\r\n #19 0x0000000000c3e31a in HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) ()\r\n #20 0x0000000000972491 in HPHP::invoke_file(HPHP::String const&, bool, char const*) ()\r\n #21 0x0000000000974b2a in HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) ()\r\n #22 0x0000000000a78fc9 in HPHP::hphp_invoke(HPHP::ExecutionContext*, std::string const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::string const&, std::string const&, bool&, std::string&, bool, bool, bool) ()\r\n #23 0x0000000000b55e74 in HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) ()\r\n #24 0x0000000000b56fb0 in HPHP::HttpRequestHandler::handleRequestImpl(HPHP::Transport*) ()\r\n #25 0x0000000000b57ad9 in HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) ()\r\n #26 0x0000000000b73bf9 in HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) ()\r\n #27 0x0000000000b73e28 in HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) ()\r\n #28 0x0000000000b72d98 in HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() ()\r\n #29 0x00000000015f9687 in HPHP::AsyncFuncImpl::ThreadFunc(void*) ()\r\n #30 0x00007faa1c1d5182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0\r\n #31 0x00007faa1b6e1fbd in clone () from /lib/x86_64-linux-gnu/libc.so.6\r\n```\r\n\r\nSystem info:\r\n\r\nHipHop VM 3.3.0 (rel)\r\nCompiler: tags/HHVM-3.3.0-0-g0a3cfb87b8a353fc7e1d15374f4adc413e37aba9\r\nRepo schema: 9a391d9a03e15fccba1cde6d35c05b7cdd380238\r\nExtension API: 20140829\r\n\r\n```\r\nUbuntu 14.04 LTS (Trusty Tahr)\r\n\r\nAmazon EC2 instance information:\r\ninstance-type: c3.8xlarge\r\nkernel-id: aki-880531cd\r\n\r\n\r\nArchitecture:          x86_64\r\nCPU op-mode(s):        32-bit, 64-bit\r\nByte Order:            Little Endian\r\nCPU(s):                32\r\nOn-line CPU(s) list:   0-31\r\nThread(s) per core:    32\r\nCore(s) per socket:    1\r\nSocket(s):             1\r\nNUMA node(s):          1\r\nVendor ID:             GenuineIntel\r\nCPU family:            6\r\nModel:                 62\r\nStepping:              4\r\nCPU MHz:               2800.050\r\nBogoMIPS:              5600.10\r\nHypervisor vendor:     Xen\r\nVirtualization type:   full\r\nL1d cache:             32K\r\nL1i cache:             32K\r\nL2 cache:              256K\r\nL3 cache:              25600K\r\nNUMA node0 CPU(s):     0-31\r\n\r\n\r\nLinux web4c 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\nPackages\r\n\r\nii  libpq-dev                              9.3.5-0ubuntu0.14.04.1              amd64        header files for libpq5 (PostgreSQL library)\r\nii  libpq5                                 9.3.5-0ubuntu0.14.04.1              amd64        PostgreSQL C client library\r\n\r\nii  openssl                                1.0.1f-1ubuntu2.5                   amd64        Secure Sockets Layer toolkit - cryptographic utility\r\n\r\nii  hhvm                                   3.3.1~trusty                        amd64        HHVM virtual machine, runtime, and JIT for the PHP language\r\n\r\n\r\nii  nginx                                  1.6.2-2+trusty0                     all          small, powerful, scalable web/proxy server\r\nii  nginx-common                           1.6.2-2+trusty0                     all          small, powerful, scalable web/proxy server - common files\r\nii  nginx-full                             1.6.2-2+trusty0                     amd64        nginx web/proxy server (standard version)\r\n```"},"comment":{"url":"https://api.github.com/repos/facebook/hhvm/issues/comments/73325354","html_url":"https://github.com/facebook/hhvm/issues/4165#issuecomment-73325354","issue_url":"https://api.github.com/repos/facebook/hhvm/issues/4165","id":73325354,"user":{"login":"fredemmott","id":360927,"avatar_url":"https://avatars.githubusercontent.com/u/360927?v=3","gravatar_id":"","url":"https://api.github.com/users/fredemmott","html_url":"https://github.com/fredemmott","followers_url":"https://api.github.com/users/fredemmott/followers","following_url":"https://api.github.com/users/fredemmott/following{/other_user}","gists_url":"https://api.github.com/users/fredemmott/gists{/gist_id}","starred_url":"https://api.github.com/users/fredemmott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredemmott/subscriptions","organizations_url":"https://api.github.com/users/fredemmott/orgs","repos_url":"https://api.github.com/users/fredemmott/repos","events_url":"https://api.github.com/users/fredemmott/events{/privacy}","received_events_url":"https://api.github.com/users/fredemmott/received_events","type":"User","site_admin":false},"created_at":"2015-02-06T22:32:33Z","updated_at":"2015-02-06T22:32:33Z","body":"cc @simonwelsh - any thoughts on the best way to aaddress this?"}}