{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry/dea_ng/issues/165","labels_url":"https://api.github.com/repos/cloudfoundry/dea_ng/issues/165/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry/dea_ng/issues/165/comments","events_url":"https://api.github.com/repos/cloudfoundry/dea_ng/issues/165/events","html_url":"https://github.com/cloudfoundry/dea_ng/issues/165","id":81991255,"number":165,"title":"Staging tasks are failing with \"Disconnected\" messages to Warden","user":{"login":"fraenkel","id":929377,"avatar_url":"https://avatars.githubusercontent.com/u/929377?v=3","gravatar_id":"","url":"https://api.github.com/users/fraenkel","html_url":"https://github.com/fraenkel","followers_url":"https://api.github.com/users/fraenkel/followers","following_url":"https://api.github.com/users/fraenkel/following{/other_user}","gists_url":"https://api.github.com/users/fraenkel/gists{/gist_id}","starred_url":"https://api.github.com/users/fraenkel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fraenkel/subscriptions","organizations_url":"https://api.github.com/users/fraenkel/orgs","repos_url":"https://api.github.com/users/fraenkel/repos","events_url":"https://api.github.com/users/fraenkel/events{/privacy}","received_events_url":"https://api.github.com/users/fraenkel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-05-28T15:42:06Z","updated_at":"2015-07-21T20:43:07Z","closed_at":null,"body":"An idle timer is set on the connection to Warden when there are no outstanding requests. However, the way the DEA acquires a warden connection does not prevent it from retrieving a Disconnected connection or being disconnected while using it.\r\n\r\nThe best solution is to remove the idle timer since it serves no real purpose. We don't have many connections to Warden, 3 or so. We could increase the timeout but this problem will still occur.\r\n\r\nThe problem does occur most frequently with the Java buildpack/Tomcat where it has a long download for artifacts. Below is a snippet of the logs.\r\n\r\n```\r\n{\"timestamp\":1431225565.530343,\"message\":\"staging.task.execute-staging\",\"log_level\":\"debug\",\"source\":\"Staging\",\"data\":{\"app_guid\":\"25b6b2e0-e01d-4714-972d-996a1789fcc1\",\"task_id\":\"df72259ac08f4bf99f05dba87a9d5fa0\",\"script\":\"set -o pipefail; export VCAP_APPLICATION=\\\\{\\\\\\\"limits\\\\\\\":\\\\{\\\\\\\"mem\\\\\\\":512,\\\\\\\"disk\\\\\\\":1024,\\\\\\\"fds\\\\\\\":16384\\\\},\\\\\\\"application_version\\\\\\\":\\\\\\\"4bc46693-7d6e-4824-886a-19c07c627f2e\\\\\\\",\\\\\\\"application_name\\\\\\\":\\\\\\\"A1-StatusTestApp1431225546492\\\\\\\",\\\\\\\"application_uris\\\\\\\":\\\\[\\\\\\\"A1-StatusTestApp1431225546492.stage1.ng.bluemix.net\\\\\\\"\\\\],\\\\\\\"version\\\\\\\":\\\\\\\"4bc46693-7d6e-4824-886a-19c07c627f2e\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"A1-StatusTestApp1431225546492\\\\\\\",\\\\\\\"space_name\\\\\\\":\\\\\\\"uptime\\\\\\\",\\\\\\\"space_id\\\\\\\":\\\\\\\"900f5c68-0910-4faf-84ed-24c0b805a9dd\\\\\\\",\\\\\\\"uris\\\\\\\":\\\\[\\\\\\\"A1-StatusTestApp1431225546492.stage1.ng.bluemix.net\\\\\\\"\\\\],\\\\\\\"users\\\\\\\":null\\\\};\\nexport VCAP_SERVICES=\\\\{\\\\};\\nexport MEMORY_LIMIT=\\\"512m\\\";\\nexport BUILDPACK_CACHE=\\\"/var/vcap/packages/buildpack_cache\\\";\\nexport STAGING_TIMEOUT=\\\"900.0\\\";\\nexport MEMORY_LIMIT=\\\"512m\\\";\\nexport CF_STACK=\\\"lucid64\\\";\\n /usr/bin/ruby /var/vcap/packages/dea_next/buildpacks/bin/run /var/vcap/data/dea_next/staging/d20150510-13604-ta5qyn/plugin_config | tee -a /tmp/staged/logs/staging_task.log\"},\"thread_id\":70230060757760,\"fiber_id\":70230188194260,\"process_id\":13604,\"file\":\"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb\",\"lineno\":222,\"method\":\"block in promise_stage\"}\r\n{\"timestamp\":1431225565.542062,\"message\":\"snapshot_attributes\",\"log_level\":\"debug\",\"source\":\"Staging\",\"data\":{\"app_guid\":\"25b6b2e0-e01d-4714-972d-996a1789fcc1\",\"task_id\":\"df72259ac08f4bf99f05dba87a9d5fa0\",\"properties\":{\"services\":[],\"resources\":{\"memory\":512,\"disk\":1024,\"fds\":16384},\"environment\":[\"CF_STACK=lucid64\"],\"meta\":{\"console\":false},\"buildpack_key\":\"0041b66c-cc13-4b5e-91b8-45472142037b_cf07d48f2d45a0eeb02b56c56e9a39cee81044b3\"}},\"thread_id\":70230060757760,\"fiber_id\":70230188194260,\"process_id\":13604,\"file\":\"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb\",\"lineno\":502,\"method\":\"snapshot_attributes\"}\r\n{\"timestamp\":1431225597.5597432,\"message\":\"staging.task.packing-droplet\",\"log_level\":\"info\",\"source\":\"Staging\",\"data\":{\"app_guid\":\"25b6b2e0-e01d-4714-972d-996a1789fcc1\",\"task_id\":\"df72259ac08f4bf99f05dba87a9d5fa0\"},\"thread_id\":70230060757760,\"fiber_id\":70230080860460,\"process_id\":13604,\"file\":\"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb\",\"lineno\":287,\"method\":\"block in promise_pack_app\"}\r\n{\"timestamp\":1431225597.5608807,\"message\":\"staging.task-info.copying-out\",\"log_level\":\"info\",\"source\":\"Staging\",\"data\":{\"app_guid\":\"25b6b2e0-e01d-4714-972d-996a1789fcc1\",\"task_id\":\"df72259ac08f4bf99f05dba87a9d5fa0\",\"source\":\"/tmp/staged/staging_info.yml\",\"destination\":\"/var/vcap/data/dea_next/staging/d20150510-13604-ta5qyn/staging_info.yml\"},\"thread_id\":70230060757760,\"fiber_id\":70230079329860,\"process_id\":13604,\"file\":\"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb\",\"lineno\":263,\"method\":\"block in promise_staging_info\"}\r\n{\"timestamp\":1431225599.243332,\"message\":\"staging.task-log.copying-out\",\"log_level\":\"info\",\"source\":\"Staging\",\"data\":{\"app_guid\":\"25b6b2e0-e01d-4714-972d-996a1789fcc1\",\"task_id\":\"df72259ac08f4bf99f05dba87a9d5fa0\",\"source\":\"/tmp/staged/logs/staging_task.log\",\"destination\":\"/var/vcap/data/dea_next/staging/d20150510-13604-ta5qyn/staging_task.log\"},\"thread_id\":70230060757760,\"fiber_id\":70230120371960,\"process_id\":13604,\"file\":\"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb\",\"lineno\":254,\"method\":\"block in promise_task_log\"}\r\n{\"timestamp\":1431225599.4826744,\"message\":\"staging.task.failed\",\"log_level\":\"info\",\"source\":\"Staging\",\"data\":{\"app_guid\":\"25b6b2e0-e01d-4714-972d-996a1789fcc1\",\"task_id\":\"df72259ac08f4bf99f05dba87a9d5fa0\",\"error\":\"Disconnected\",\"backtrace\":[\"/var/vcap/packages/dea_next/vendor/cache/warden-ad18bff7dc56/em-warden-client/lib/em/warden/client/connection.rb:27:in `get'\",\"/var/vcap/packages/dea_next/vendor/cache/warden-ad18bff7dc56/em-warden-client/lib/em/warden/client.rb:46:in `call'\",\"/var/vcap/packages/dea_next/lib/container/container.rb:192:in `call'\",\"/var/vcap/packages/dea_next/lib/container/container.rb:72:in `run_script'\",\"/var/vcap/packages/dea_next/lib/dea/staging/staging_task.rb:289:in `block in promise_pack_app'\",\"/var/vcap/packages/dea_next/lib/dea/promise.rb:92:in `call'\",\"/var/vcap/packages/dea_next/lib/dea/promise.rb:92:in `block in\r\n```"},"comment":{"url":"https://api.github.com/repos/cloudfoundry/dea_ng/issues/comments/123471663","html_url":"https://github.com/cloudfoundry/dea_ng/issues/165#issuecomment-123471663","issue_url":"https://api.github.com/repos/cloudfoundry/dea_ng/issues/165","id":123471663,"user":{"login":"zaksoup","id":296065,"avatar_url":"https://avatars.githubusercontent.com/u/296065?v=3","gravatar_id":"","url":"https://api.github.com/users/zaksoup","html_url":"https://github.com/zaksoup","followers_url":"https://api.github.com/users/zaksoup/followers","following_url":"https://api.github.com/users/zaksoup/following{/other_user}","gists_url":"https://api.github.com/users/zaksoup/gists{/gist_id}","starred_url":"https://api.github.com/users/zaksoup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zaksoup/subscriptions","organizations_url":"https://api.github.com/users/zaksoup/orgs","repos_url":"https://api.github.com/users/zaksoup/repos","events_url":"https://api.github.com/users/zaksoup/events{/privacy}","received_events_url":"https://api.github.com/users/zaksoup/received_events","type":"User","site_admin":false},"created_at":"2015-07-21T20:43:07Z","updated_at":"2015-07-21T20:43:07Z","body":"what's happening here. Who knows what's going on? I have no context."}}