{"action":"created","issue":{"url":"https://api.github.com/repos/mongoid/moped/issues/345","labels_url":"https://api.github.com/repos/mongoid/moped/issues/345/labels{/name}","comments_url":"https://api.github.com/repos/mongoid/moped/issues/345/comments","events_url":"https://api.github.com/repos/mongoid/moped/issues/345/events","html_url":"https://github.com/mongoid/moped/issues/345","id":56290965,"number":345,"title":"Writes fail with ConnectionPool::PoolShuttingDownError","user":{"login":"dblock","id":542335,"avatar_url":"https://avatars.githubusercontent.com/u/542335?v=3","gravatar_id":"","url":"https://api.github.com/users/dblock","html_url":"https://github.com/dblock","followers_url":"https://api.github.com/users/dblock/followers","following_url":"https://api.github.com/users/dblock/following{/other_user}","gists_url":"https://api.github.com/users/dblock/gists{/gist_id}","starred_url":"https://api.github.com/users/dblock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dblock/subscriptions","organizations_url":"https://api.github.com/users/dblock/orgs","repos_url":"https://api.github.com/users/dblock/repos","events_url":"https://api.github.com/users/dblock/events{/privacy}","received_events_url":"https://api.github.com/users/dblock/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":29,"created_at":"2015-02-02T20:48:50Z","updated_at":"2015-06-12T21:08:51Z","closed_at":"2015-02-12T19:42:26Z","body":"Something happens in our MongoDB configuration (likely a step down of a master or some kind of failure) that causes all servers to start reporting the following error: `ConnectionPool::PoolShuttingDownError` on database writes to primary.\r\n\r\nThis is recent, we've seen it take out all our writes twice.\r\n\r\nMoped 2.0.3\r\nMongoid 4.0.0\r\n\r\n```\r\n…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  78:in `block (2 levels) in pop'\r\n…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  77:in `loop'\r\n…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  77:in `block in pop'\r\n…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  76:in `synchronize'\r\n…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  76:in `pop'\r\n….0.0/gems/connection_pool-2.1.1/lib/connection_pool.rb:  78:in `checkout'\r\n….0.0/gems/connection_pool-2.1.1/lib/connection_pool.rb:  60:in `with'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 114:in `connection'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 140:in `disconnect'\r\n….0.0/gems/moped-2.0.3/lib/moped/failover/disconnect.rb:  26:in `execute'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 185:in `rescue in ensure_connected'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 188:in `ensure_connected'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 589:in `block in flush'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 617:in `block in logging'\r\n…ctivesupport-4.1.8/lib/active_support/notifications.rb: 159:in `block in instrument'\r\n…4.1.8/lib/active_support/notifications/instrumenter.rb:  20:in `instrument'\r\n…ctivesupport-4.1.8/lib/active_support/notifications.rb: 159:in `instrument'\r\n…uby/2.0.0/gems/moped-2.0.3/lib/moped/instrumentable.rb:  31:in `instrument'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 616:in `logging'\r\n…lic_moped-0.0.12/lib/newrelic_moped/instrumentation.rb:  40:in `block in logging_with_newrelic_trace'\r\n…lic_moped-0.0.12/lib/newrelic_moped/instrumentation.rb:  49:in `call'\r\n…lic_moped-0.0.12/lib/newrelic_moped/instrumentation.rb:  49:in `block in logging_with_newrelic_trace'\r\n…lic_moped-0.0.12/lib/newrelic_moped/instrumentation.rb:  46:in `logging_with_newrelic_trace'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 587:in `flush'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 358:in `pipeline'\r\n…by/2.0.0/gems/moped-2.0.3/lib/moped/operation/write.rb:  47:in `execute'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 665:in `write'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 502:in `update'\r\n…gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/query.rb: 427:in `block in update'\r\n…avity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/cluster.rb: 249:in `block in with_primary'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 204:in `block in ensure_primary'\r\n…ty/ruby/2.0.0/gems/moped-2.0.3/lib/moped/executable.rb:  25:in `execute'\r\n…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 203:in `ensure_primary'\r\n…avity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/cluster.rb: 248:in `with_primary'\r\n…gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/query.rb: 426:in `update'\r\n…by/2.0.0/gems/mongoid-4.0.0/lib/mongoid/query_cache.rb: 117:in `update_with_clear_cache'\r\n…gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/query.rb: 449:in `update_all'\r\n…by/2.0.0/gems/mongoid-4.0.0/lib/mongoid/query_cache.rb: 117:in `update_all_with_clear_cache'\r\n….0/gems/mongoid-4.0.0/lib/mongoid/contextual/atomic.rb: 152:in `set'\r\n…uby/2.0.0/gems/mongoid-4.0.0/lib/mongoid/contextual.rb:  16:in `set'\r\n```"},"comment":{"url":"https://api.github.com/repos/mongoid/moped/issues/comments/111619821","html_url":"https://github.com/mongoid/moped/issues/345#issuecomment-111619821","issue_url":"https://api.github.com/repos/mongoid/moped/issues/345","id":111619821,"user":{"login":"jonhyman","id":832655,"avatar_url":"https://avatars.githubusercontent.com/u/832655?v=3","gravatar_id":"","url":"https://api.github.com/users/jonhyman","html_url":"https://github.com/jonhyman","followers_url":"https://api.github.com/users/jonhyman/followers","following_url":"https://api.github.com/users/jonhyman/following{/other_user}","gists_url":"https://api.github.com/users/jonhyman/gists{/gist_id}","starred_url":"https://api.github.com/users/jonhyman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonhyman/subscriptions","organizations_url":"https://api.github.com/users/jonhyman/orgs","repos_url":"https://api.github.com/users/jonhyman/repos","events_url":"https://api.github.com/users/jonhyman/events{/privacy}","received_events_url":"https://api.github.com/users/jonhyman/received_events","type":"User","site_admin":false},"created_at":"2015-06-12T21:08:51Z","updated_at":"2015-06-12T21:08:51Z","body":"Thanks. Yeah, we've used my branch through multiple stepdowns and incidents (like a mongos running out of memory, a physical host failing, etc.) and it holds up great now. I don't think there's any support for Moped anymore. @durran are you interested in me cleaning up my branch (basically just removing the log statements I added) and letting me submit a pull?"}}