{"action":"created","issue":{"url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/1575","labels_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/1575/labels{/name}","comments_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/1575/comments","events_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/1575/events","html_url":"https://github.com/wiredtiger/wiredtiger/issues/1575","id":54677552,"number":1575,"title":"Deadlock opening multiple cursors in parallel","user":{"login":"agorrod","id":1930121,"avatar_url":"https://avatars.githubusercontent.com/u/1930121?v=3","gravatar_id":"","url":"https://api.github.com/users/agorrod","html_url":"https://github.com/agorrod","followers_url":"https://api.github.com/users/agorrod/followers","following_url":"https://api.github.com/users/agorrod/following{/other_user}","gists_url":"https://api.github.com/users/agorrod/gists{/gist_id}","starred_url":"https://api.github.com/users/agorrod/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/agorrod/subscriptions","organizations_url":"https://api.github.com/users/agorrod/orgs","repos_url":"https://api.github.com/users/agorrod/repos","events_url":"https://api.github.com/users/agorrod/events{/privacy}","received_events_url":"https://api.github.com/users/agorrod/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-17T22:19:59Z","updated_at":"2015-01-17T22:45:37Z","closed_at":null,"body":"If an application opens multiple statistics cursors in parallel WiredTiger can end up deadlocked between the DHANDLE and SCHEMA locks.\r\n\r\nExample stack traces - full stack traces are available in JIRA SERVER-16738 ticket.\r\n\r\nThread 1:\r\n```\r\n#2  0x00007ffff7bc6480 in __GI___pthread_mutex_lock (mutex=0x3420400) at ../nptl/pthread_mutex_lock.c:79\r\n#3  0x0000000001ef5262 in __wt_spin_lock (session=0x3547280, t=0x3420400) at src/third_party/wiredtiger/src/include/mutex.i:175\r\n#4  0x0000000001ef6000 in __wt_session_get_btree (session=0x3547280, uri=0x1593d000b \"file:index-491659-4306617738107441063.wt\", checkpoint=0x0, cfg=0x7ffff0bacec0, flags=8)\r\n    at src/third_party/wiredtiger/src/session/session_dhandle.c:397\r\n#5  0x0000000001ef58c0 in __wt_session_get_btree_ckpt (session=0x3547280, uri=0x1593d000b \"file:index-491659-4306617738107441063.wt\", cfg=0x7ffff0bacec0, flags=0)\r\n    at src/third_party/wiredtiger/src/session/session_dhandle.c:229\r\n#6  0x0000000001e9a2da in __curstat_file_init (session=0x3547280, uri=0x1593d000b \"file:index-491659-4306617738107441063.wt\", cfg=0x7ffff0bacec0, cst=0xf441be00)\r\n    at src/third_party/wiredtiger/src/cursor/cur_stat.c:379\r\n```\r\n\r\nThread 2:\r\n```\r\n#3  0x0000000001ef5262 in __wt_spin_lock (session=0x35459c0, t=0x34204c0) at src/third_party/wiredtiger/src/include/mutex.i:175\r\n#4  0x0000000001ef5f93 in __wt_session_get_btree (session=0x35459c0, uri=0x141baf9f0 \"file:collection-491658-4306617738107441063.wt\", checkpoint=0x0, cfg=0x0, flags=8)\r\n    at src/third_party/wiredtiger/src/session/session_dhandle.c:397\r\n#5  0x0000000001e7cb70 in __conn_btree_apply_internal (session=0x35459c0, dhandle=0x1041edc00, func=0x1e9a1fb <__curstat_checkpoint>, cfg=0x7ffff13b4b20)\r\n    at src/third_party/wiredtiger/src/conn/conn_dhandle.c:484\r\n#6  0x0000000001e7cd26 in __wt_conn_btree_apply (session=0x35459c0, apply_checkpoints=1, uri=0x141baf9f0 \"file:collection-491658-4306617738107441063.wt\", func=0x1e9a1fb <__curstat_checkpoint>,\r\n    cfg=0x7ffff13b4b20) at src/third_party/wiredtiger/src/conn/conn_dhandle.c:526\r\n#7  0x0000000001e9a4a1 in __curstat_file_init (session=0x35459c0, uri=0x172e0c00b \"file:collection-491658-4306617738107441063.wt\", cfg=0x7ffff13b4ef0, cst=0xf62ac000)\r\n    at src/third_party/wiredtiger/src/cursor/cur_stat.c:413\r\n```\r\n\r\nThe sequence of events leading up to the deadlock is:\r\n\r\n* Thread 2 grabs the DHANDLE lock in __curstat_file_init before calling __wt_conn_btree_apply.\r\n* Thread 1 doesn't hold the DHANDLE lock and calls session_get_btree, which grabs the SCHEMA lock, then waits on the DHANDLE lock that is held by thread 2.\r\n* Thread 2 ends up in __wt_session_get_btree while already holding the DHANDLE lock. __wt_session_get_btree attempts to get the SCHEMA lock which is held by thread 1.\r\n\r\nThread 2 is the \"problem\" thread - all other operations that need both the SCHEMA and DHANDLE locks get the SCHEMA lock first. It's probably enough to change __curstat_file_init so that it takes both the SCHEMA and DHANDLE locks at the start."},"comment":{"url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/comments/70387847","html_url":"https://github.com/wiredtiger/wiredtiger/issues/1575#issuecomment-70387847","issue_url":"https://api.github.com/repos/wiredtiger/wiredtiger/issues/1575","id":70387847,"user":{"login":"agorrod","id":1930121,"avatar_url":"https://avatars.githubusercontent.com/u/1930121?v=3","gravatar_id":"","url":"https://api.github.com/users/agorrod","html_url":"https://github.com/agorrod","followers_url":"https://api.github.com/users/agorrod/followers","following_url":"https://api.github.com/users/agorrod/following{/other_user}","gists_url":"https://api.github.com/users/agorrod/gists{/gist_id}","starred_url":"https://api.github.com/users/agorrod/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/agorrod/subscriptions","organizations_url":"https://api.github.com/users/agorrod/orgs","repos_url":"https://api.github.com/users/agorrod/repos","events_url":"https://api.github.com/users/agorrod/events{/privacy}","received_events_url":"https://api.github.com/users/agorrod/received_events","type":"User","site_admin":false},"created_at":"2015-01-17T22:45:37Z","updated_at":"2015-01-17T22:45:37Z","body":"> It is possible that this is a reproducible case of SERVER-16738.\r\n\r\nThat's where the stack traces came from (sorry I didn't make that clear). I haven't reproduced this, but I'm pretty confident in my analysis (apart from the question I added to #1576)."}}