{"push_id":826022141,"size":1,"distinct_size":1,"ref":"refs/heads/3.0.x","head":"5305a71e8b190bc6cf91c737fe800a2e8bb458d6","before":"b1553094a8ba54966e92cab2aa7b6b17770f99c7","commits":[{"sha":"5305a71e8b190bc6cf91c737fe800a2e8bb458d6","author":{"email":"a09264da4832c7ff1d3bf1608a19f4b870f93750@couchbase.com","name":"Dave Rigby"},"message":"MB-16500 [BP]: Fix potenial deadlock around Connmap::relaseLock / connLock\n\nAs reported by ThreadSanitizer (see below), we have a lock inversion\ncreating a potential deadlock in Connmap, related to how we shutdown\nconnections:\n\nThere exists a cycle in lock order graph:\n\n M2176 (Connmap::releaseLock in ConnMap::notifyPausedConnection() connmap.cc:235) =>\n   M128093 (mock_server::conn_struct::mutex) =>\n     M2177 (Connmap::connsLock in TapConnMap::shutdownAllConnections(), connmap.cc:770) =>\n       M2176 (Connmap::releaseLock in TapConnMap::shutdownAllConnections(), connmap.cc:777) DEADLOCK!\n\nThe problem appears to be that in TapConnMap::shutdownAllConnections()\nwe first acquire {connsLock}, then acquire {releaseLock}; all while\nholding the cookie lock.\n\nFix is to drop releaseLock in shutdownAllConnections() once we've\nreleased any references, *then* acquire the connLock to actually clear\nout the connection map.\n\nThreadSanitizer report follows (irrelevent parts removed):\n\nWARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=1087)\n\n  Cycle in lock order graph: M2176 (0x7d840001b810) => M128093 (0x7d280000efa0) => M2177 (0x7d840001b850) => M2176\n\n  Mutex M128093 acquired here while holding mutex M2176 in thread T10:\n    <cut>\n\n  Mutex M2176 previously acquired by the same thread here:\n    #0 pthread_mutex_lock <null> ()\n    ...\n    #5 ConnMap::notifyPausedConnection() ep-engine/src/connmap.cc:235 ()\n    <cut>\n\n  Mutex M2177 acquired here while holding mutex M128093 in main thread:\n    #0 pthread_mutex_lock <null> ()\n    ...\n    #5 TapConnMap::newProducer() ep-engine/src/connmap.cc:378 ()\n    #6 EventuallyPersistentEngine::createTapQueue() ep-engine/src/ep_engine.cc:2663:23 ()\n    <cut>\n\n  Mutex M128093 previously acquired by the same thread here:\n    #0 pthread_mutex_lock <null> ()\n    #1 cb_mutex_enter platform/src/cb_pthreads.c:115:14 ()\n    #2 lock_mock_cookie memcached/programs/engine_testapp/mock_server.c:436:4 ()\n    #3 test_tap_stream() ep-engine/tests/ep_testsuite.cc:6751:5 ()\n    #4 execute_test() memcached/programs/engine_testapp/engine_testapp.cc:1090:19 ()\n    #5 main memcached/programs/engine_testapp/engine_testapp.cc:1439 ()\n\n  Mutex M2176 acquired here while holding mutex M2177 in main thread:\n    #0 pthread_mutex_lock <null> ()\n    ...\n    #5 TapConnMap::shutdownAllConnections() ep-engine/src/connmap.cc:777 ()\n    #6 EventuallyPersistentEngine::destroy() ep-engine/src/ep_engine.cc:2190:9 ()\n    <cut>\n\n  Mutex M2177 previously acquired by the same thread here:\n    #0 pthread_mutex_lock <null> ()\n    ...\n    #5 TapConnMap::shutdownAllConnections() ep-engine/src/connmap.cc:770 ()\n    #6 EventuallyPersistentEngine::destroy() ep-engine/src/ep_engine.cc:2190:9 ()\n    <cut>\n\nChange-Id: Ic9a4f028b202277729df025333ce630be056903d\nReviewed-on: http://review.couchbase.org/55863\nTested-by: buildbot <build@couchbase.com>\nReviewed-by: Chiyoung Seo <chiyoung@couchbase.com>\nReviewed-on: http://review.couchbase.org/56064","distinct":true,"url":"https://api.github.com/repos/couchbase/ep-engine/commits/5305a71e8b190bc6cf91c737fe800a2e8bb458d6"}]}