Why are my Amazon ECS container instances with Amazon Linux 1 AMIs disconnected?
My container instances for Amazon Elastic Container Service (Amazon ECS) are disconnected.
"My container instances for Amazon Elastic Container Service (Amazon ECS) are disconnected.Short descriptionYour Amazon ECS container agent might connect and reconnect several times an hour. These change events are normal and aren't a cause for concern.However, if your container agent remains in a disconnected state, then the container instance can't operate as part of your ECS cluster. Your agent is disconnected when agentConnected returns false. The issue can be caused by the following:Networking issues prevent communication between the instance and Amazon ECS.The container agent doesn't have the required AWS Identity and Access Management (IAM) permissions to communicate with Amazon ECS endpoints.There are problems with the host or Docker service inside the container instance.To identify the cause of the disconnection, complete the following steps.ResolutionNote: The following resolution applies to Amazon ECS-optimized Amazon Linux 1 AMIs. For a resolution that applies to Amazon ECS-optimized Amazon Linux 2 AMIs, see How do I troubleshoot a disconnected Amazon ECS agent?Verify that the Docker service is running on the container instance1.    To verify that the Docker service is running on the affected container instance, run the following command:sudo service docker statusThe command output is similar to the following:docker (pid 23013) is running...If the Docker service isn't running, or if you need to restart the service, run the following command:sudo service docker restartNote: Don't enter this command while the service is already running. First, make sure to set the container instance to the draining state. Then, restart the Docker service for existing tasks to be scheduled on another container instance.The command output must include the following lines:Stopping docker: [  OK  ]Starting docker: [  OK  ]Note: To verify that the Docker service is running after the restart command, run the sudo service docker status command.2.    To start the ECS agent, run the following command:sudo start ecsVerify that the container agent is running on the container instanceTo verify that the container agent is running on the affected container instance, run the following command:sudo status ecsIf the container agent isn't running on your container instance, then run the following command to start the agent:sudo start ecsThe command output is similar to the following:ecs start/running, process 23403Review log files for the container agent and DockerIf your container instances are still disconnected, review the log files on the container host for the container agent and Docker.To output the log files for the container agent and Docker, run the following commands:sudo cat /var/log/ecs/ecs-agent.log.YYYY-MM-DD-**sudo cat /var/log/dockerNote: To collect log information from the container instance, run the Amazon ECS logs collector.Verify that the IAM instance profile has the necessary permissionsIf the container agent is still disconnected, verify that the IAM instance profile associated with the container instance has the necessary IAM permissions.1.    Connect to the instance using SSH.2.    To view the instance metadata on the instance profile associated with the instance, run the following command:curl http://169.254.169.254/latest/meta-data/iam/infoThe command output is similar to the following:{  "Code" : "Success",  "LastUpdated" : "2019-06-29T15:47:03Z",  "InstanceProfileArn" : "arn:aws:iam::1122334455:instance-profile/ecsInstanceRole",  "InstanceProfileId" : "AIPAJ5WF3LZVY7PLUHV72"}3.    Verify that the IAM role contains the correct permissions for your container instances.4.    To verify specific credential errors with the container agent, run the following command to check the container agent log for a list of ECS logs:cat /var/log/ecs/ecs-agent.log.YYYY-MM-DD-**Note: The container agent log is rotated every hour, and the suffix automatically changes to reflect the current date and time. Update the command to include the date range and log ID for when the issue occurred.If the container agent doesn't have the necessary credentials, you receive an error similar to the following in the logs:2019-06-29T16:10:09Z [ERROR] Unable to register as a container instance with ECS: AccessDeniedException: User: arn:aws:sts::1122334455:assumed-role/ecsInstanceRole/i-0052b2e858b1891ef is not authorized to perform: ecs:RegisterContainerInstance on resource: arn:aws:ecs:us-east-1:1122334455:cluster/exampleCluster    status code: 400, request id: 0b73e260-5088-4688-a425-6f35f1ef440f2019-06-29T16:10:09Z [ERROR] Error re-registering: AccessDeniedException: User: arn:aws:sts::1122334455:assumed-role/ecsInstanceRole/i-0052b2e858b1891ef is not authorized to perform: ecs:RegisterContainerInstance on resource: arn:aws:ecs:us-east-1:1122334455:cluster/exampleCluster    status code: 400, request id: 0b73e260-5088-4688-a425-6f35f1ef440fAdditional helpIf you can't identify the issue with your ECS container instance from this resolution, contact Premium Support for help. First, use Amazon ECS logs collector to create an archive of your instance's logs. Then, attach the logs to a support ticket to help the support engineer troubleshoot the issue.Related informationAmazon ECS troubleshootingAmazon ECS container agentAmazon ECS container instance IAM roleAmazon ECS log file locationsFollow"