{"action":"closed","issue":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/242","labels_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/242/labels{/name}","comments_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/242/comments","events_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/242/events","html_url":"https://github.com/aws/amazon-ecs-agent/issues/242","id":114292866,"number":242,"title":"Agent in bad state after crash","user":{"login":"jbergknoff","id":2678832,"avatar_url":"https://avatars.githubusercontent.com/u/2678832?v=3","gravatar_id":"","url":"https://api.github.com/users/jbergknoff","html_url":"https://github.com/jbergknoff","followers_url":"https://api.github.com/users/jbergknoff/followers","following_url":"https://api.github.com/users/jbergknoff/following{/other_user}","gists_url":"https://api.github.com/users/jbergknoff/gists{/gist_id}","starred_url":"https://api.github.com/users/jbergknoff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbergknoff/subscriptions","organizations_url":"https://api.github.com/users/jbergknoff/orgs","repos_url":"https://api.github.com/users/jbergknoff/repos","events_url":"https://api.github.com/users/jbergknoff/events{/privacy}","received_events_url":"https://api.github.com/users/jbergknoff/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/labels/pending%20release","name":"pending release","color":"0052cc"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-30T15:32:01Z","updated_at":"2015-12-14T18:03:24Z","closed_at":"2015-12-14T18:03:24Z","body":"I have an ECS task that ran and then stopped, but ecs-agent crashed (or at least stopped) in the meantime. When the agent came back up, it considered the task to be running, although the container was not running. The web UI stated that the task was running and that the service was in a steady state. It stayed in that state until now (5 hours later), when I updated the service to 0 desired count and back to 1, which resolved the issue.\r\n\r\n##### ecs-agent.log\r\n\r\n```\r\n2015-10-30T10:51:11Z [INFO] Pulling container module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f, Status: (NONE->RUNNING) Containers: [mytaskdefinition (NONE->RUNNING),]\" container=\"mytaskdefinition/mytaskdefinition:0c6ba2202fb267d75a4f091e077c5966147b7efd (NONE->RUNNING)\"\r\n2015-10-30T10:51:11Z [INFO] Creating container module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f, Status: (NONE->RUNNING) Containers: [mytaskdefinition (PULLED->RUNNING),]\" container=\"mytaskdefinition/mytaskdefinition:0c6ba2202fb267d75a4f091e077c5966147b7efd (PULLED->RUNNING)\"\r\n2015-10-30T10:51:11Z [INFO] Created container successfully module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f, Status: (NONE->RUNNING) Containers: [mytaskdefinition (PULLED->RUNNING),]\" container=\"mytaskdefinition/mytaskdefinition:0c6ba2202fb267d75a4f091e077c5966147b7efd (PULLED->RUNNING)\"\r\n2015-10-30T10:51:11Z [INFO] Starting container module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f, Status: (CREATED->RUNNING) Containers: [mytaskdefinition (CREATED->RUNNING),]\" container=\"mytaskdefinition/mytaskdefinition:0c6ba2202fb267d75a4f091e077c5966147b7efd (CREATED->RUNNING) - Exit: 0\"\r\n2015-10-30T10:51:11Z [INFO] Redundant status change; ignoring module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f, Status: (CREATED->RUNNING) Containers: [mytaskdefinition (CREATED->RUNNING),]\" current=\"CREATED\" change=\"CREATED\"\r\n2015-10-30T10:51:11Z [INFO] Task change event module=\"TaskEngine\" event=\"{TaskArn:arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f Status:3 Reason: SentStatus:NONE}\"\r\n2015-10-30T10:51:11Z [INFO] Adding event module=\"eventhandler\" change=\"ContainerChange: arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f mytaskdefinition -> RUNNING, Exit 0, , Known Sent: NONE\"\r\n2015-10-30T10:51:11Z [INFO] Adding event module=\"eventhandler\" change=\"TaskChange: arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f -> RUNNING, Known Sent: NONE\"\r\n2015-10-30T10:51:11Z [INFO] Sending container change module=\"eventhandler\" event=\"ContainerChange: arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f mytaskdefinition -> RUNNING, Exit 0, , Known Sent: NONE\" change=\"ContainerChange: arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f mytaskdefinition -> RUNNING, Exit 0, , Known Sent: NONE\"\r\n2015-10-30T10:51:11Z [INFO] Redundant status change; ignoring module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f, Status: (RUNNING->RUNNING) Containers: [mytaskdefinition (RUNNING->RUNNING),]\" current=\"RUNNING\" change=\"RUNNING\"\r\n2015-10-30T10:51:12Z [INFO] Sending task change module=\"eventhandler\" event=\"TaskChange: arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f -> RUNNING, Known Sent: NONE\" change=\"TaskChange: arn:aws:ecs:us-east-1:...:task/cb57da9c-a487-4fcd-9fab-495e9f0e5b1f -> RUNNING, Known Sent: NONE\"\r\n2015-10-30T10:51:15Z [INFO] Starting Agent: Amazon ECS Agent - v1.4.0 (4ab1051)\r\n2015-10-30T10:51:15Z [INFO] Loading configuration\r\n2015-10-30T10:51:15Z [INFO] Checkpointing is enabled. Attempting to load state\r\n2015-10-30T10:51:15Z [INFO] Loading state! module=\"statemanager\"\r\n2015-10-30T10:51:16Z [INFO] Restored cluster 'ecs-cluster'\r\n2015-10-30T10:51:16Z [INFO] Restored from checkpoint file. I am running as 'arn:aws:ecs:us-east-1:...:container-instance/03aaad90-589c-4097-b6d8-07c2f2aa81ff' in cluster 'ecs-cluster'\r\n2015-10-30T10:51:16Z [INFO] Registered! module=\"api client\"\r\n2015-10-30T10:51:16Z [INFO] Saving state! module=\"statemanager\"\r\n2015-10-30T10:51:16Z [INFO] Removing container module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/b81e7252-e2a8-4742-8aac-fbcdb7a2fcb3, Status: (STOPPED->STOPPED) Containers: [mytaskdefinition (STOPPED->STOPPED),]\" container=\"mytaskdefinition/mytaskdefinition:0c6ba2202fb267d75a4f091e077c5966147b7efd (STOPPED->STOPPED) - Exit: 0\"\r\n2015-10-30T10:51:16Z [INFO] Beginning Polling for updates\r\n2015-10-30T10:51:16Z [INFO] Initializing stats engine module=\"stats\"\r\n2015-10-30T10:51:16Z [INFO] Creating poll dialer module=\"ws client\" host=\"ecs-t-1.us-east-1.amazonaws.com\"\r\n2015-10-30T10:51:16Z [INFO] Creating poll dialer module=\"ws client\" host=\"ecs-a-1.us-east-1.amazonaws.com\"\r\n2015-10-30T10:51:16Z [WARN] Error getting cpu stats module=\"stats\" err=\"No data in the queue\" container=\"&{DockerID:89399d98d2940569a264ce9092b3e0e435f19ffff5735145f271a80a9d0ed93d}\"\r\n2015-10-30T10:51:16Z [WARN] Error getting instance metrics module=\"tcs client\" err=\"No task metrics to report\"\r\n2015-10-30T10:51:17Z [INFO] Removing container module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/b0cb5b5d-e4dc-4378-8999-b1c59c928fc3, Status: (STOPPED->STOPPED) Containers: [mytaskdefinition (STOPPED->STOPPED),]\" container=\"mytaskdefinition/mytaskdefinition:0c6ba2202fb267d75a4f091e077c5966147b7efd (STOPPED->STOPPED) - Exit: 0\"\r\n2015-10-30T10:51:26Z [INFO] Saving state! module=\"statemanager\"\r\n2015-10-30T10:51:37Z [INFO] Removing container module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/117ad5b3-ee07-4d3f-99e2-b704c96bc573, Status: (STOPPED->STOPPED) Containers: [mytaskdefinition (STOPPED->STOPPED),]\" container=\"mytaskdefinition/mytaskdefinition:0c6ba2202fb267d75a4f091e077c5966147b7efd (STOPPED->STOPPED) - Exit: 0\"\r\n2015-10-30T10:51:37Z [INFO] Saving state! module=\"statemanager\"\r\n2015-10-30T10:51:48Z [INFO] Removing container module=\"TaskEngine\" task=\"mytaskdefinition-10 arn:aws:ecs:us-east-1:...:task/7f119b00-1ab7-461a-9a8f-288025f5ce4e, Status: (STOPPED->STOPPED) Containers: [mytaskdefinition (STOPPED->STOPPED),]\" container=\"mytaskdefinition/mytaskdefinition:0c6ba2202fb267d75a4f091e077c5966147b7efd (STOPPED->STOPPED) - Exit: 0\"\r\n2015-10-30T10:51:49Z [INFO] Saving state! module=\"statemanager\"\r\n...\r\n```\r\n\r\n##### ecs-init.log\r\n\r\n```\r\n2015-10-30T10:51:14Z [INFO] Agent exited with code 2\r\n2015-10-30T10:51:14Z [INFO] Container name: ...\r\n2015-10-30T10:51:14Z [INFO] Container name: /ecs-agent\r\n2015-10-30T10:51:14Z [INFO] Removing existing agent container ID: bf888c7f192b835803848582785ef26dabc96d2fa4c885a26e2664772ee8cc3a\r\n2015-10-30T10:51:15Z [INFO] Starting Amazon EC2 Container Service Agent\r\n```\r\n\r\n`curl http://localhost:51678/v1/tasks` did not list the misreported task.\r\n\r\nCan you give any insight into this behavior? I have since updated the agent from 1.4.0 to 1.6.0. Is there any chance that the issue will be solved by that upgrade?"}}