{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry/bosh/issues/748","labels_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/748/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/748/comments","events_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/748/events","html_url":"https://github.com/cloudfoundry/bosh/pull/748","id":56629235,"number":748,"title":"validate_db_connection","user":{"login":"ronakbanka","id":2391105,"avatar_url":"https://avatars.githubusercontent.com/u/2391105?v=3","gravatar_id":"","url":"https://api.github.com/users/ronakbanka","html_url":"https://github.com/ronakbanka","followers_url":"https://api.github.com/users/ronakbanka/followers","following_url":"https://api.github.com/users/ronakbanka/following{/other_user}","gists_url":"https://api.github.com/users/ronakbanka/gists{/gist_id}","starred_url":"https://api.github.com/users/ronakbanka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronakbanka/subscriptions","organizations_url":"https://api.github.com/users/ronakbanka/orgs","repos_url":"https://api.github.com/users/ronakbanka/repos","events_url":"https://api.github.com/users/ronakbanka/events{/privacy}","received_events_url":"https://api.github.com/users/ronakbanka/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cloudfoundry/bosh/labels/ready-to-pull-in","name":"ready-to-pull-in","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-02-05T05:28:38Z","updated_at":"2015-02-06T01:53:55Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/cloudfoundry/bosh/pulls/748","html_url":"https://github.com/cloudfoundry/bosh/pull/748","diff_url":"https://github.com/cloudfoundry/bosh/pull/748.diff","patch_url":"https://github.com/cloudfoundry/bosh/pull/748.patch"},"body":"Abstract : To validate the db connection because once the director-worker is spawned by the monit , if there is a connection break on postgres DB side then this thread gives a Error i.e not able to connect PG error and no worker Job is processed (create release , delete release).\r\n\r\nSolution: Using connection validator extension from sequel , if there is a persistent connection then all things are fine but if connection breaks this extension will validate the connection and worker will process the job.\r\n\r\nBy default the timeout is 3600 ( 1 hour ) but can reduce in cases where either firewall or other network issues we are loosing connections to database.\r\n\r\nBelow is error trace:\r\n\r\n INFO  DirectorWorker : got: (Job{normal} | Bosh::Director::Jobs::DeleteRelease | [10, \"pgpool\", {\"version\"=>nil}])\r\n INFO  DirectorWorker : Running before_fork hooks with [(Job{normal} | Bosh::Director::Jobs::DeleteRelease | [10, \"pgpool\", {\"version\"=>nil}])]\r\n INFO  DirectorWorker : Running after_fork hooks with [(Job{normal} | Bosh::Director::Jobs::DeleteRelease | [10, \"pgpool\", {\"version\"=>nil}])]\r\n INFO  DirectorWorker : (Job{normal} | Bosh::Director::Jobs::DeleteRelease | [10, \"pgpool\", {\"version\"=>nil}]) failed: #<Sequel::DatabaseDisconnectError: PG::Error: server closed the connection unexpectedly\r\n\tThis probably means the server terminated abnormally\r\n\tbefore or while processing the request.\r\n>\r\n INFO  DirectorWorker : /var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:145:in `exec'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:145:in `block in execute_query'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/database/logging.rb:37:in `log_yield'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:145:in `execute_query'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:132:in `block in execute'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:111:in `check_disconnect_errors'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:132:in `execute'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:413:in `_execute'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:242:in `block (2 levels) in execute'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:425:in `check_database_errors'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:242:in `block in execute'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/database/connecting.rb:236:in `block in synchronize'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/connection_pool/threaded.rb:104:in `hold'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/database/connecting.rb:236:in `synchronize'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:242:in `execute'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/dataset/actions.rb:801:in `execute'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:525:in `fetch_rows'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/model/base.rb:798:in `primary_key_lookup'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/model/base.rb:117:in `[]'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2780.0/lib/bosh/director/api/task_manager.rb:11:in `find_task'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2780.0/lib/bosh/director/job_runner.rb:21:in `initialize'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2780.0/lib/bosh/director/jobs/base_job.rb:10:in `new'\r\n/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2780.0/lib/bosh/director/jobs/base_job.rb:10:in `perform'"},"comment":{"url":"https://api.github.com/repos/cloudfoundry/bosh/issues/comments/73167854","html_url":"https://github.com/cloudfoundry/bosh/pull/748#issuecomment-73167854","issue_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/748","id":73167854,"user":{"login":"shinji62","id":4757816,"avatar_url":"https://avatars.githubusercontent.com/u/4757816?v=3","gravatar_id":"","url":"https://api.github.com/users/shinji62","html_url":"https://github.com/shinji62","followers_url":"https://api.github.com/users/shinji62/followers","following_url":"https://api.github.com/users/shinji62/following{/other_user}","gists_url":"https://api.github.com/users/shinji62/gists{/gist_id}","starred_url":"https://api.github.com/users/shinji62/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shinji62/subscriptions","organizations_url":"https://api.github.com/users/shinji62/orgs","repos_url":"https://api.github.com/users/shinji62/repos","events_url":"https://api.github.com/users/shinji62/events{/privacy}","received_events_url":"https://api.github.com/users/shinji62/received_events","type":"User","site_admin":false},"created_at":"2015-02-06T01:53:55Z","updated_at":"2015-02-06T01:53:55Z","body":"Yes basicaly we created a pgpool release, to make postgres HA and so on."}}