How do I best manage the logging levels of my AWS IoT logs in AWS IoT Core?
I want to follow best practices when configuring my AWS IoT logs in AWS IoT Core to reduce costs and data traffic.
"I want to follow best practices when configuring my AWS IoT logs in AWS IoT Core to reduce costs and data traffic.Short descriptionNote: This article relates to V2 of AWS IoT Core logs only.It's a best practice to know what AWS IoT Core resources must be monitored and at what level. You can segment your AWS IoT Core resources and assign each segment a logging level that suits your monitoring needs. AWS IoT Core log allows you to set default logging levels and specific logging levels for your resources.It's a best practice to configure:Default logging to have a low verbosity logging level, such as ERROR or WARN levels. If default logging is configured, then it's applied to all resources unless they have resource-specific logging configured.Resource-specific logging (either manual or dynamic) to have a more verbose level, such as INFO or DEBUG. This can be configured for different target types, such as THING_GROUP, CLIENT_ID, SOURCE_IP, or PRINCIPAL_ID.Default logging and resource-specific logging can be used simultaneously. Verbosity levels include DISABLED (lowest), ERROR, WARN, INFO, and DEBUG (highest).Important: Depending on your AWS IoT Core fleet size, turning on more verbose log levels can incur high costs and make troubleshooting more difficult. More verbose logging levels also create higher data traffic. It's a best practice to use INFO or DEBUG only as a temporary measure when troubleshooting. After troubleshooting complete, it's a best practice to reset logging levels to a less verbose setting.ResolutionPrerequisiteMake sure that you have the AWS Command Line Interface (AWS CLI) installed locally with IoT administrator permission credentials. The default AWS Region for AWS CLI must point towards the targeted AWS Region. You must have clients connected to your AWS IoT Core, either as registered or non-registered IoT things.Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.Default loggingBy setting the default logging level for your AWS IoT logs to ERROR or WARN, you can review log errors or warnings. Setting the default to less verbose levels lowers your costs and your data traffic. For more information, see How do I configure the default logging settings for AWS IoT Core?Note: You must provide AWS IoT Core logs to AWS Support when creating a case. AWS Support doesn't have access to this information because it's considered customer data.Logging specific resources manuallyAWS IoT Core logs allow you to set resource-specific logging levels for both:Clients registered as thingsClients not registered as thingsFor more information, see How do I configure logging levels manually for specific resources in AWS IoT Core?For clients registered as things, you can manage resource-specific logging for multiple things at a defined logging level. You can also add or remove things from the thing group manually. Your clients must be registered as IoT things in AWS IoT Core and must connect using the same client ID associated with its thing name. You can then use a static thing group with a target type of THING_GROUP to manage the thing group.Note: You must consider the quota limits when using static thing groups. For more information, see AWS IoT Core thing group resource limits and quotas.For clients not registered as things, you can use client attributes as the target type to manage resources. These attributes include CLIENT_ID, SOURCE_IP, or PRINCIPAL_ID.Logging specific resources automaticallyYou can use a dynamic thing group to set resource-specific logging levels for your things automatically. Your clients must be registered as IoT things and connect with the same Client ID as its thing name. The dynamic thing group uses a fleet indexing query that continuously finds and adds things automatically to the group that meet the query conditions. For more information, see How do I configure logging levels dynamically for specific resources in AWS IoT Core?Note: When creating or updating a dynamic thing group, things can be eligible to be in a dynamic thing group and not be added to it automatically. For more information, see Successful commands can log errors.Monitoring log entriesYou can monitor your AWS IoT log entries with either the Amazon CloudWatch console or the AWS CLI:CloudWatch console1.    Open the CloudWatch console.2.    In the navigation pane, under Logs, choose Log groups.3.    In the filters text box, enter AWSIoTLogsV2, and then press enter.Note: The AWSIoTLogsV2 log group isn't visible in the CloudWatch console until you have successfully turned on logging in AWS IoT Core. Log entries also must be written by AWS IoT operations for the log group to be visible.4.    Choose AWSIoTLogsV2.5.    Choose Search all. A complete list of your AWS IoT logs generated for your account appears.AWS CLIRun the following command to get your AWS IoT logs in AWS CLI:aws logs filter-log-events --log-group-name AWSIotLogsV2 \ --start-time YourStartTimeInMillisecondsNote: You can use the --filter-pattern to search for specific log attributes. For more information, see Filter and pattern syntax.The value of "logLevel" field reflects the logging level that is configured for the target. In the following example, the logging level for the target is configured to INFO:{ "timestamp": "2017-08-10 15:37:23.476", "logLevel": "INFO", "traceId": "20b23f3f-d7f1-feae-169f-82263394fbdb", "accountId": "123456789012", "status": "Success", "eventType": "Connect", "protocol": "MQTT", "clientId": "abf27092886e49a8a5c1922749736453", "principalId": "145179c40e2219e18a909d896a5340b74cf97a39641beec2fc3eeafc5a123456", "sourceIp": "123.123.123.123", "sourcePort": 13490}Note: CLIENT_ID is available only when connecting to AWS IoT Core using MQTT or MQTT over WSS protocols. HTTPS protocol doesn't support client IDs. Use SOURCE_IP or PRINCIPAL_ID target types instead.Editing or deleting logging levelsYou can change the logging level for both the default level and for specific resources at any time. It can take up to ten minutes for the changes to be reflected.In AWS CLI, run the delete-v2-logging-level command with the following edits made to the --target-type and --target-name fields:aws iot delete-v2-logging-level \ --target-type "THING_GROUP | CLIENT_ID | SOURCE_IP | PRINCIPAL_ID" \ --target-name YourTargetNameRelated informationAWS IoT Device Management endpoints and quotasMonitoring AWS IoTAWS IoT Device Management pricingFollow"