{"action":"closed","issue":{"url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/166","labels_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/166/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/166/comments","events_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/166/events","html_url":"https://github.com/cloudfoundry-incubator/cf-abacus/issues/166","id":116503860,"number":166,"title":"CF Bridge breaks Abacus pipeline","user":{"login":"hsiliev","id":732222,"avatar_url":"https://avatars.githubusercontent.com/u/732222?v=3","gravatar_id":"","url":"https://api.github.com/users/hsiliev","html_url":"https://github.com/hsiliev","followers_url":"https://api.github.com/users/hsiliev/followers","following_url":"https://api.github.com/users/hsiliev/following{/other_user}","gists_url":"https://api.github.com/users/hsiliev/gists{/gist_id}","starred_url":"https://api.github.com/users/hsiliev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hsiliev/subscriptions","organizations_url":"https://api.github.com/users/hsiliev/orgs","repos_url":"https://api.github.com/users/hsiliev/repos","events_url":"https://api.github.com/users/hsiliev/events{/privacy}","received_events_url":"https://api.github.com/users/hsiliev/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-12T08:45:39Z","updated_at":"2015-11-12T13:28:49Z","closed_at":"2015-11-12T13:28:49Z","body":"```\r\nGetting events for app abacus-eureka-stub in org abacus / space abacus as i024099...\r\n\r\ntime                          event                 actor                description\r\n2015-11-12T09:46:26.00+0200   app.crash             abacus-eureka-stub   index: 1, reason: CRASHED, exit_description: failed to start, exit_status: -1\r\n2015-11-12T09:43:52.00+0200   audit.app.update      i024099              state: STARTED\r\n2015-11-12T09:43:30.00+0200   audit.app.update      i024099\r\n2015-11-12T09:43:30.00+0200   audit.app.map-route   i024099\r\n2015-11-12T09:43:28.00+0200   audit.app.create      i024099              disk_quota: 512, instances: 2, memory: 512, state: STOPPED, environment_json: PRIVATE DATA HIDDEN\r\n```\r\n\r\nThe logs of the eureka stub show this:\r\n```\r\n2015-11-12T09:46:14.61+0200 [STG/1]      OUT -----> Uploading droplet (20M)\r\n2015-11-12T09:46:26.19+0200 [DEA/1]      OUT Starting app instance (index 0) with guid 3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:46:26.39+0200 [DEA/11]     OUT Starting app instance (index 1) with guid 3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:46:26.68+0200 [API/0]      OUT App instance exited with guid 3ddaa1f2-6665-4d64-b8be-36f5ba024b07 payload: {\"cc_partition\"=>\"default\", \"droplet\"=>\"3ddaa1f2-6665-4d64-b8be-36f5ba024b07\", \"version\"=>\"10807f48-1a35-401f-aaf0-932d4f3757df\", \"instance\"=>\"faca1236eb704843a2b97ba11837c501\", \"index\"=>1, \"reason\"=>\"CRASHED\", \"exit_status\"=>-1, \"exit_description\"=>\"failed to start\", \"crash_timestamp\"=>1447314386}\r\n2015-11-12T09:46:55.38+0200 [App/0]      OUT\r\n2015-11-12T09:46:55.38+0200 [App/0]      OUT > abacus-eureka-stub@0.0.3 start /home/vcap/app\r\n2015-11-12T09:46:55.38+0200 [App/0]      OUT > launch webapp start\r\n2015-11-12T09:46:59.19+0200 [DEA/11]     OUT Starting app instance (index 1) with guid 3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:03.47+0200 [App/0]      OUT 2015-11-12T07:47:03.468Z e-abacus-breaker 49 Function dbclient.wrapper calling back with error, { message: 'Database encountered an unknown error',\r\n2015-11-12T09:47:03.47+0200 [App/0]      OUT   status: 404,\r\n2015-11-12T09:47:03.47+0200 [App/0]      OUT   name: 'unknown_error',\r\n2015-11-12T09:47:03.47+0200 [App/0]      OUT   error: true,\r\n2015-11-12T09:47:03.47+0200 [App/0]      OUT   noretry: true,\r\n2015-11-12T09:47:03.47+0200 [App/0]      OUT   nobreaker: true,\r\n2015-11-12T09:47:03.47+0200 [App/0]      OUT   toString: [Function] }\r\n2015-11-12T09:47:03.48+0200 [App/0]      OUT 2015-11-12T07:47:03.485Z e-abacus-retry 49 Function dbclient.wrapper calling back with error { message: 'Database encountered an unknown error',\r\n2015-11-12T09:47:03.48+0200 [App/0]      OUT   status: 404,\r\n2015-11-12T09:47:03.48+0200 [App/0]      OUT   name: 'unknown_error',\r\n2015-11-12T09:47:03.48+0200 [App/0]      OUT   error: true,\r\n2015-11-12T09:47:03.48+0200 [App/0]      OUT   noretry: true,\r\n2015-11-12T09:47:03.48+0200 [App/0]      OUT   nobreaker: true,\r\n2015-11-12T09:47:03.48+0200 [App/0]      OUT   toString: [Function] }\r\n2015-11-12T09:47:03.49+0200 [App/0]      OUT 2015-11-12T07:47:03.492Z e-abacus-router 49 Route error - generator error - { message: 'Database encountered an unknown error',\r\n2015-11-12T09:47:03.49+0200 [App/0]      OUT   status: 404,\r\n2015-11-12T09:47:03.49+0200 [App/0]      OUT   name: 'unknown_error',\r\n2015-11-12T09:47:03.49+0200 [App/0]      OUT   error: true,\r\n2015-11-12T09:47:03.49+0200 [App/0]      OUT   noretry: true,\r\n2015-11-12T09:47:03.49+0200 [App/0]      OUT   nobreaker: true,\r\n2015-11-12T09:47:03.49+0200 [App/0]      OUT   toString: [Function] }\r\n2015-11-12T09:47:03.50+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:02 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-ACCUMULATOR HTTP/1.1\" 404 204 49 \"-\" \"-\" 10.78.148.6:19259 x_forwarded_for:\"155.56.196.2\" vcap_request_id:48ac764b-f008-4cb0-6bce-c838366040f6 response_time:0.553714356 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT 2015-11-12T07:47:03.519Z e-abacus-breaker 49 Function dbclient.wrapper calling back with error, { message: 'Database encountered an unknown error',\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   status: 404,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   name: 'unknown_error',\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   error: true,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   noretry: true,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   nobreaker: true,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   toString: [Function] }\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT 2015-11-12T07:47:03.521Z e-abacus-retry 49 Function dbclient.wrapper calling back with error { message: 'Database encountered an unknown error',\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   status: 404,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   name: 'unknown_error',\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   error: true,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   noretry: true,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   nobreaker: true,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   toString: [Function] }\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT 2015-11-12T07:47:03.523Z e-abacus-router 49 Route error - generator error - { message: 'Database encountered an unknown error',\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   status: 404,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   name: 'unknown_error',\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   error: true,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   noretry: true,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   nobreaker: true,\r\n2015-11-12T09:47:03.52+0200 [App/0]      OUT   toString: [Function] }\r\n2015-11-12T09:47:03.52+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:02 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-AGGREGATOR HTTP/1.1\" 404 202 49 \"-\" \"-\" 10.78.148.6:63477 x_forwarded_for:\"155.56.196.2\" vcap_request_id:a0344f9b-03eb-48d1-7006-2a820f876c34 response_time:0.977704250 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:03.70+0200 [App/0]      OUT 2015-11-12T07:47:03.700Z e-abacus-breaker 49 Function dbclient.wrapper calling back with error, { message: 'Database encountered an unknown error',\r\n2015-11-12T09:47:03.70+0200 [App/0]      OUT   status: 404,\r\n2015-11-12T09:47:03.70+0200 [App/0]      OUT   name: 'unknown_error',\r\n2015-11-12T09:47:03.70+0200 [App/0]      OUT   error: true,\r\n2015-11-12T09:47:03.70+0200 [App/0]      OUT   noretry: true,\r\n2015-11-12T09:47:03.70+0200 [App/0]      OUT   nobreaker: true,\r\n2015-11-12T09:47:03.70+0200 [App/0]      OUT   toString: [Function] }\r\n2015-11-12T09:47:03.71+0200 [App/0]      OUT 2015-11-12T07:47:03.714Z e-abacus-retry 49 Function dbclient.wrapper calling back with error { message: 'Database encountered an unknown error',\r\n2015-11-12T09:47:03.71+0200 [App/0]      OUT   status: 404,\r\n2015-11-12T09:47:03.71+0200 [App/0]      OUT   name: 'unknown_error',\r\n2015-11-12T09:47:03.71+0200 [App/0]      OUT   error: true,\r\n2015-11-12T09:47:03.71+0200 [App/0]      OUT   noretry: true,\r\n2015-11-12T09:47:03.71+0200 [App/0]      OUT   nobreaker: true,\r\n2015-11-12T09:47:03.71+0200 [App/0]      OUT   toString: [Function] }\r\n2015-11-12T09:47:03.72+0200 [App/0]      OUT 2015-11-12T07:47:03.715Z e-abacus-router 49 Route error - generator error - { message: 'Database encountered an unknown error',\r\n2015-11-12T09:47:03.72+0200 [App/0]      OUT   status: 404,\r\n2015-11-12T09:47:03.72+0200 [App/0]      OUT   name: 'unknown_error',\r\n2015-11-12T09:47:03.72+0200 [App/0]      OUT   error: true,\r\n2015-11-12T09:47:03.72+0200 [App/0]      OUT   noretry: true,\r\n2015-11-12T09:47:03.72+0200 [App/0]      OUT   nobreaker: true,\r\n2015-11-12T09:47:03.72+0200 [App/0]      OUT   toString: [Function] }\r\n2015-11-12T09:47:03.72+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:03 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-REPORTING HTTP/1.1\" 404 200 49 \"-\" \"-\" 10.78.148.6:4512 x_forwarded_for:\"155.56.196.2\" vcap_request_id:54f37098-dc38-4b3f-4906-a91b215747b0 response_time:0.486497418 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:17.69+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:17 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-COLLECTOR HTTP/1.1\" 201 198 0 \"-\" \"-\" 10.78.148.6:39261 x_forwarded_for:\"155.56.196.2\" vcap_request_id:5bb117e5-f229-4d5a-611c-e4705185df2b response_time:0.452706795 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:27.45+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:27 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-AGGREGATOR HTTP/1.1\" 201 202 0 \"-\" \"-\" 10.78.148.6:42261 x_forwarded_for:\"155.56.196.2\" vcap_request_id:96da5caa-641f-45ee-7c07-fb8b3428a8ed response_time:0.212892774 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:32.83+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:32 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-AGGREGATOR HTTP/1.1\" 201 200 0 \"-\" \"-\" 10.78.148.6:43898 x_forwarded_for:\"155.56.196.2\" vcap_request_id:8f92b068-dbac-4aab-5be0-eeaa5e0bb867 response_time:0.134443424 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:36.95+0200 [App/1]      OUT\r\n2015-11-12T09:47:36.95+0200 [App/1]      OUT > abacus-eureka-stub@0.0.3 start /home/vcap/app\r\n2015-11-12T09:47:36.95+0200 [App/1]      OUT > launch webapp start\r\n2015-11-12T09:47:39.17+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:38 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-METER HTTP/1.1\" 201 190 0 \"-\" \"-\" 10.78.148.6:23126 x_forwarded_for:\"155.56.196.2\" vcap_request_id:63faac0e-583f-419d-4504-14e6e60faf4c response_time:0.228096914 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:41.67+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:41 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-ACCUMULATOR HTTP/1.1\" 201 204 0 \"-\" \"-\" 10.78.148.6:6347 x_forwarded_for:\"155.56.196.2\" vcap_request_id:13973dd5-1f7e-493f-6d40-974829665d84 response_time:0.247007467 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:46.42+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:46 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-METER HTTP/1.1\" 201 192 0 \"-\" \"-\" 10.78.148.6:31814 x_forwarded_for:\"155.56.196.2\" vcap_request_id:59f155b5-4b2e-4ef4-4a7d-81c29b29b0e1 response_time:0.268165681 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:47.24+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:46 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-ACCUMULATOR HTTP/1.1\" 201 204 0 \"-\" \"-\" 10.78.148.6:15331 x_forwarded_for:\"155.56.196.2\" vcap_request_id:2a2f59f2-449e-4168-5871-65d80055141d response_time:0.250260114 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:47.66+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:47 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-RATE HTTP/1.1\" 201 188 0 \"-\" \"-\" 10.78.148.6:30295 x_forwarded_for:\"155.56.196.2\" vcap_request_id:a1288542-fec9-4d33-5b10-030aafe8c551 response_time:0.501123649 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:47.85+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:47 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-METER HTTP/1.1\" 201 192 0 \"-\" \"-\" 10.78.148.6:19135 x_forwarded_for:\"155.56.196.2\" vcap_request_id:69c98571-5805-4110-76d4-ede32e9cf177 response_time:0.324512517 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:47:49.55+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:47:49 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-RATE HTTP/1.1\" 201 188 0 \"-\" \"-\" 10.78.148.6:25719 x_forwarded_for:\"155.56.196.2\" vcap_request_id:3ddee395-c5af-40aa-5695-bb8963d96481 response_time:0.163349626 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:52:33.07+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:52:32 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-RATE HTTP/1.1\" 201 190 0 \"-\" \"-\" 10.78.148.6:52850 x_forwarded_for:\"155.56.196.2\" vcap_request_id:3ccbd86f-9502-4953-60e4-7976fe1028bc response_time:0.326292628 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:55:39.98+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:55:39 +0000] \"POST /eureka/v2/apps/ABACUS-USAGE-METER HTTP/1.1\" 201 190 0 \"-\" \"-\" 10.78.148.6:56415 x_forwarded_for:\"155.56.196.2\" vcap_request_id:ee08e85b-65f8-401f-4580-2635104309d6 response_time:0.140981374 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n2015-11-12T09:58:17.94+0200 [RTR/0]      OUT abacus-eureka-stub.cfapps.neo.ondemand.com - [12/11/2015:07:58:17 +0000] \"POST /eureka/v2/apps/ABACUS-CF-BRIDGE HTTP/1.1\" 201 188 0 \"-\" \"-\" 10.78.148.6:65375 x_forwarded_for:\"155.56.196.2\" vcap_request_id:a4566515-9a57-4b32-73d0-0188474f90a3 response_time:0.122350692 app_id:3ddaa1f2-6665-4d64-b8be-36f5ba024b07\r\n```\r\n\r\nThe DB logs show no errors or error responses event.\r\n\r\nThe CF Bridge logs contains 405 and later 409 responses:\r\n```\r\n2015-11-12T12:08:22.55+0200 [App/0]      OUT \tresponse {\"statusCode\":405,\"body\":{\"error\":\"Not found\"},\"headers\":{\"content-length\":\"21\",\"content-type\":\"application/json; charset=utf-8\",\"date\":\"Thu, 12 Nov 2015 10:08:22 GMT\",\"etag\":\"W/\\\"15-Lvrg4Gz9w+rMdDVG/1lddA\\\"\",\"vary\":\"X-HTTP-Method-Override\",\"x-app-index\":\"0\",\"x-app-name\":\"abacus-usage-collector\",\"x-cf-requestid\":\"0227edb7-5d77-4b28-4b17-0e85750f1514\",\"x-heap-used\":\"86935184\",\"x-instance-id\":\"1a22d5b3eeae4a7a8f22c15500108aef\",\"x-instance-index\":\"3\",\"x-node-version\":\"v0.12.2\",\"x-powered-by\":\"Express\",\"x-process-id\":\"49\",\"x-response-time\":\"11595.199ms\",\"x-uptime\":\"1337.808\"},\"request\":{\"uri\":{\"protocol\":\"https:\",\"slashes\":true,\"auth\":null,\"host\":\"abacus-usage-collector.cfapps.neo.ondemand.com\",\"port\":443,\"hostname\":\"abacus-usage-collector.cfapps.neo.ondemand.com\",\"hash\":null,\"search\":null,\"query\":null,\"pathname\":\"/batch\",\"path\":\"/batch\",\"href\":\"https://abacus-usage-collector.cfapps.neo.ondemand.com/batch\"},\"method\":\"POST\",\"headers\":{\"accept\":\"application/json\",\"content-type\":\"application/json\",\"content-length\":898}}}\r\n2015-11-12T12:08:24.31+0200 [App/0]      OUT \tresponse {\"statusCode\":409,\"headers\":{\"date\":\"Thu, 12 Nov 2015 10:08:24 GMT\",\"etag\":\"W/\\\"24a-BPAYU9gNCi/LUIF2516ZYw\\\"\",\"vary\":\"X-HTTP-Method-Override\",\"x-app-index\":\"0\",\"x-app-name\":\"abacus-usage-collector\",\"x-cf-requestid\":\"43ac6cfe-7e6f-4169-6d2b-d6580f89044a\",\"x-heap-used\":\"127418528\",\"x-instance-id\":\"ad5f054358174b72bf74718f28ff9bff\",\"x-instance-index\":\"1\",\"x-node-version\":\"v0.12.2\",\"x-powered-by\":\"Express\",\"x-process-id\":\"49\",\"x-response-time\":\"13467.725ms\",\"x-uptime\":\"1338.046\"},\"body\":{\"error\":\"Document update conflict\"}}\r\n```\r\n\r\nNo further submit of metrics is possible, so I suspect this is caused by an open circuit."}}