Why isn't the unified CloudWatch agent pushing my metrics or log events to CloudWatch?
I have configured the unified CloudWatch agent on my Amazon Elastic Compute Cloud (Amazon EC2) instance to post metrics and logs to Amazon CloudWatch. But I can't see my metrics or logs in the CloudWatch console. Why isn't the unified CloudWatch agent pushing my metrics and logs to CloudWatch?
"I have configured the unified CloudWatch agent on my Amazon Elastic Compute Cloud (Amazon EC2) instance to post metrics and logs to Amazon CloudWatch. But I can't see my metrics or logs in the CloudWatch console. Why isn't the unified CloudWatch agent pushing my metrics and logs to CloudWatch?Short descriptionThere are a number of reasons why the unified CloudWatch agent might not be pushing your metrics or logs to CloudWatch. For example, you might have a permissions or connectivity error that is preventing the agent from publishing your metrics. When you review the unified CloudWatch agent logs, you might see an error like this:Agent Log Error: No Connectivity to EndpointAgent Log Error: Insufficient PermissionsResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Review the unified CloudWatch agent logsUse the agent log file to help you troubleshoot problems you encounter with the unified CloudWatch agent package. You might be experiencing one of these common issues:You're experiencing connectivity issues to the required AWS service endpoints or VPC endpoints.You don't have the correct permissions to make support API calls to CloudWatch.The log file isn't present on your local file system.You might see one of these errors in the logs:Agent Log Error: No Connectivity to Endpoint2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout2021-08-30T04:07:46Z W! 210 retries, going to sleep 1m0s before retrying.2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout2021-08-30T04:07:46Z W! 211 retries, going to sleep 1m0s before retrying.Agent Log Error: Insufficient Permissions2021-08-30T02:15:45Z E! cloudwatch: code: AccessDenied, message: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData, original error: 2021-08-30T02:15:45Z W! 1 retries, going to sleep 400ms before retrying.2021-08-30T02:15:46Z E! WriteToCloudWatch failure, err:  AccessDenied: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData    status code: 403, request id: f1171fd0-05b6-4f7d-bac2-629c8594c46eConfirm connectivity to the CloudWatch endpointsWhen traffic to CloudWatch should not transit the public internet, you can use VPC endpoints instead. If you are using VPC endpoints, check the following:If you are using private nameservers, confirm that DNS resolution provided accurate responses.Confirm that the CloudWatch endpoints resolve to private IP addresses.Confirm the security group associated with the VPC endpoint allows inbound traffic from the host.1.    Check connectivity to the metrics endpoint:$ telnet monitoring.us-east-1.amazonaws.com 443Trying 52.46.138.115...Connected to monitoring.amazonaws.com.Escape character is '^]'.^]telnet> quitConnection closed.2.    Check connectivity to the logs endpoint:$ telnet logs.us-east-1.amazonaws.com 443Trying 3.236.94.218...Connected to logs.us-east-1.amazonaws.com.Escape character is '^]'.^]telnet> quitConnection closed3.    Check that the VPC endpoint resolves to private IP addresses:$ dig monitoring.us-east-1.amazonaws.com +short172.31.11.121172.31.0.13Review the unified CloudWatch agent configurationThe agent configuration file details the metrics and logs that are published to CloudWatch. Review the agent configuration file to confirm that the logs and metrics you want to publish are included.Confirm that the host has permissions to publish metrics and logsThe AWS managed policies CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy can help you deploy the unified CloudWatch agent and check that you have the correct permissions. Use these policies as a reference to make sure that your host has the correct permissions.The AWS CLI outputs in these examples show insufficient permissions.This agent start command output shows that there is no IAM role attached to EC2 instance:$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s****** processing amazon-cloudwatch-agent ******/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config defaultRegion: us-east-1credsConfig: map[]Error in retrieving parameter store content: NoCredentialProviders: no valid providers in chain. Deprecated.    For verbose messaging see aws.Config.CredentialsChainVerboseErrorsFail to fetch/remove json config: NoCredentialProviders: no valid providers in chain. Deprecated.    For verbose messaging see aws.Config.CredentialsChainVerboseErrorsFail to fetch the config!This agent start command output shows that the incorrect IAM role is attached to the EC2 instance:$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s****** processing amazon-cloudwatch-agent ******/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config defaultRegion: us-east-1credsConfig: map[]Error in retrieving parameter store content: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server    status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4dFail to fetch/remove json config: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server    status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4dFail to fetch the config!In some cases, an IAM user might be at the command line. The obtain user/role command returns the IAM user or role associated with the instance:$ aws sts get-caller-identity{    "UserId": "AROA123456789012ABCDE:i-0744de7c842d2c2ba",    "Account": "123456789012",    "Arn": "arn:aws:sts::123456789012:assumed-role/CloudWatchAgentServerRole/i-0744de7c842d2c2ba"}Confirm that the agent starts correctlyThe agent is designed to be started using the AWS CLI with the configuration file passed as an argument. Use these valid start commands.Linux Commands:- `$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path`- `$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name`Windows Commands:- `& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json"`- `& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name`Important: Don't start the agent from the Windows Control Panel.Confirm that the agent is runningTo publish metrics and logs, the agent must be running. Run this command to confirm that the agent is active.$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status{    "status": "running",    "starttime": "2021-08-30T02:13:44+00:00",    "configstatus": "configured",    "cwoc_status": "stopped",    "cwoc_starttime": "",    "cwoc_configstatus": "not configured",    "version": "1.247349.0b251399"}Restart the agent after updating the agent configurationThe agent doesn't automatically register changes to the configuration file. If the agent configuration is updated to include new or different metrics and logs, then restart the agent using this command:$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop****** processing cwagent-otel-collector ******cwagent-otel-collector has already been stopped****** processing amazon-cloudwatch-agent ******Redirecting to /bin/systemctl stop amazon-cloudwatch-agent.service$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:config.json****** processing amazon-cloudwatch-agent ******/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config defaultSuccessfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmpStart configuration validation.../opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default2021/08/31 02:45:37 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...Valid Json input schema.I! Detecting run_as_user...Configuration validation first phase succeeded/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.tomlConfiguration validation second phase succeededConfiguration validation succeededamazon-cloudwatch-agent has already been stoppedRedirecting to /bin/systemctl restart amazon-cloudwatch-agent.service$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status{  "status": "running",  "starttime": "2021-08-31T02:45:37+0000",  "configstatus": "configured",  "cwoc_status": "stopped",  "cwoc_starttime": "",  "cwoc_configstatus": "not configured",  "version": "1.247349.0b251399"}Related informationHow do I install and configure the unified CloudWatch agent to push metrics and logs from my EC2 instance to CloudWatch?Troubleshooting the CloudWatch AgentFollow"