{"action":"created","issue":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/271","labels_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/271/labels{/name}","comments_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/271/comments","events_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/271/events","html_url":"https://github.com/aws/amazon-ecs-agent/issues/271","id":122791006,"number":271,"title":"ECS takes hours to synchronize with requested service update","user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-12-17T18:07:47Z","updated_at":"2015-12-18T05:32:53Z","closed_at":null,"body":"Hello,\r\n\r\nIs anyone aware of any issues with ECS not synchronizing with the service definition until hours later?\r\n\r\nWe had a service definition updated to use a new task def and desired count, but did not see those actual changes take affect until 2-3 hours later according to the event history in ECS console. \r\n\r\nI say 2-3 hours because we have the same service in us-west-2, us-east-1, and eu-west-1. In us-east-1, the change took affect about 2 hours later. Us-west-2 took 1 hour and 45 minutes, eu-west-1 took 1 hour 30 minutes. When the changes started, some took even more hours to finalize.\r\n\r\nI was confident that re-updating the service definition would have \"kicked\" ecs to update, but instead I figured I'd wait and see how long it would take for it to do it itself. Surprisingly, it was hours later.\r\n\r\nService update was made at ~11:10PM.\r\n\r\nUS-West-2 (almost 5 hours until desired state was reached)\r\n```\r\n17239049-f7ce-425c-8ba2-8b53cf103550\r\n2015-12-17 03:58:42 -0800\r\nservice zoo-quiz has reached a steady state.\r\n04274f82-2f2e-47b2-9f59-1df2e90a9550\r\n2015-12-17 03:58:30 -0800\r\nservice zoo-quiz has started 1 tasks: task 8af97763-f280-4633-8edd-abd3b363a4b8.\r\n87b6f19f-7754-4905-8c5e-b1c52d94c979\r\n2015-12-17 01:47:28 -0800\r\nservice zoo-quiz has reached a steady state.\r\nbd3d3fc8-9e83-4722-803f-b64bd4bf5334\r\n2015-12-17 01:46:45 -0800\r\nservice zoo-quiz stopped 1 running tasks.\r\nc061bbe4-d738-4545-b5aa-1cb41b682b37\r\n2015-12-17 01:46:04 -0800\r\nservice zoo-quiz has started 2 tasks: task 60128b18-caed-4595-9b90-291dcefb857b task 43a3e6e2-c468-4425-b746-581a63554444.\r\n2695ce79-51d6-4c15-b438-bdc31bfbff93\r\n2015-12-17 01:45:51 -0800\r\nservice zoo-quiz stopped 2 running tasks.\r\nfaa53a97-074f-421f-8c6b-fbb8e8cfc1bc\r\n2015-12-17 01:45:10 -0800\r\nservice zoo-quiz has started 4 tasks: task 6ff11491-b285-4fb7-b6cb-a110ffb4fb3a task 74856b65-dfd5-41d4-ab08-4eb8feac3f18 task 501b8821-1e51-49b9-922f-825fe0ffd21b task 51bc08b3-63dd-475b-bbec-380d630bcb0b.\r\n```\r\n\r\nUS-East-1 (2 hours)\r\n```\r\n768be04d-b4f0-4f31-b0be-7a2832f775ad\r\n2015-12-17 01:02:40 -0800\r\nservice zoo-quiz has reached a steady state.\r\n9b401534-2605-49fa-8ee3-4bbae4e50752\r\n2015-12-17 01:01:56 -0800\r\nservice zoo-quiz stopped 1 running tasks.\r\n16cb40aa-547d-407d-9c7b-103fe9c085bf\r\n2015-12-17 01:01:14 -0800\r\nservice zoo-quiz has started 1 tasks: task 5b84474f-90f1-49cd-8947-db5e292df1ab.\r\nb36032be-f072-464e-b17c-9355ade0acca\r\n2015-12-17 01:01:04 -0800\r\nservice zoo-quiz stopped 3 running tasks.\r\n7114d70d-3707-4be8-be89-c95b93bbd54d\r\n2015-12-17 01:00:51 -0800\r\nservice zoo-quiz has started 3 tasks: task 78da88da-f6b3-4729-95a4-4cb2d510c57c task 480db5b5-99a7-4058-9334-8e5f51d22ba7 task 9892be4a-3dd3-4797-a140-1c631262c84b.\r\n```\r\n\r\nEU-West-1 (3 1/2 hours)\r\n```\r\ne3e43902-7c66-4d0d-b1c9-4050afcd76e0\r\n2015-12-17 02:33:35 -0800\r\nservice zoo-quiz has reached a steady state.\r\n9b8fd84a-2517-4f1e-b6ef-73dbbdd5d2ce\r\n2015-12-17 02:33:24 -0800\r\nservice zoo-quiz stopped 1 running tasks.\r\nb82e1a81-1cf8-4fae-89b7-77ac5688876c\r\n2015-12-17 02:33:13 -0800\r\nservice zoo-quiz has started 1 tasks: task 485669b0-fbba-40b6-9c47-117740a6f731.\r\nd1afc18e-7f07-48ec-b305-b797c2e8a424\r\n2015-12-17 01:33:45 -0800\r\nservice zoo-quiz has started 1 tasks: task 6660fdbd-e070-4cbf-a57b-1c72b93608a4.\r\nadb1844f-9d77-4fb4-8fe7-8520c0805d89\r\n2015-12-17 01:33:02 -0800\r\nservice zoo-quiz has started 4 tasks: task 74aad805-2f15-4f28-a87d-21a75c06314d task 725e3dfc-2f38-41c2-9e3e-c2ce8986c0d2 task e98f00a0-04cf-4b37-87c0-1fe75d4c5985 task fa92251b-cacc-4309-8d24-db102807f6f9.\r\n```\r\n\r\nAny ideas?\r\n\r\nThanks."},"comment":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/comments/165677588","html_url":"https://github.com/aws/amazon-ecs-agent/issues/271#issuecomment-165677588","issue_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/271","id":165677588,"user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"created_at":"2015-12-18T05:32:53Z","updated_at":"2015-12-18T05:32:53Z","body":"@samuelkarp, could i send it to you in an email?"}}