"Why do I see "audit: backlog limit exceeded" errors in my EC2 Linux instance's screenshot and system logs, and what can I do to avoid this?"
"I see "audit callbacks suppressed" and "audit: backlog limit exceeded" error messages in my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance's screenshot and system logs. Why am I receiving these messages, and how can I prevent them from reoccurring?"
"I see "audit callbacks suppressed" and "audit: backlog limit exceeded" error messages in my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance's screenshot and system logs. Why am I receiving these messages, and how can I prevent them from reoccurring?Short descriptionThe audit backlog buffer in a Linux system is a kernel level socket buffer queue that the operating system uses to maintain or log audit events. When a new audit event triggers, the system logs the event and adds it to the audit backlog buffer queue.The backlog_limit parameter value is the number of audit backlog buffers. The parameter is set to 320 by default, as shown in the following example:# auditctl -senabled 1failure 1pid 2264rate_limit 0backlog_limit 320lost 0backlog 0Audit events logged beyond the default number of 320 cause the following errors on the instance:audit: audit_backlog=321 > audit_backlog_limit=320 audit: audit_lost=44393 audit_rate_limit=0 audit_backlog_limit=320 audit: backlog limit exceeded-or-audit_printk_skb: 153 callbacks suppressedaudit_printk_skb: 114 callbacks suppressedAn audit buffer queue at or exceeding capacity might also cause the instance to freeze or remain in an unresponsive state.To avoid backlog limit exceeded errors, increase the backlog_limit parameter value. Large servers have a larger number of audit logs triggered, so increasing buffer space helps avoid error messages.Note: Increasing the audit buffer consumes more of the instance's memory. How large you make the backlog_limit parameter depends on the total memory of the instance. If the system has enough memory, you can try doubling the existing backlog_limit parameter value.The following is a calculation of the memory required for the auditd backlog. Use this calculation to determine how large you can make the backlog queue without causing memory stress on your instance.One audit buffer = 8970 BytesDefault number of audit buffers (backlog_limit parameter) = 320320 * 8970 = 2870400 Bytes, or 2.7 MiBThe size of the audit buffer is defined by the MAX_AUDIT_MESSAGE_LENGTH parameter. For more information, see MAX_AUDIT_MESSAGE_LENGTH in the Linux audit library on github.com.Note: If your instance is inaccessible and you see backlog limit exceeded messages in the system log, stop and start the instance. Then, perform the following steps to change the audit buffer value.ResolutionNote: In this example, we're changing the backlog_limit parameter value to 8192 buffers. 8192 buffers equals 70 MiB of memory based on the preceding calculation. You can use any value based on your memory calculation.Access the instance using SSH.Verify the current audit buffer size.Note: The backlog_limit parameter is listed as -b. For more information, see auditctl(8) on the auditctl-man-pageAmazon Linux 1 and other operating systems that don't have systemd:$ sudo cat /etc/audit/audit.rules# This file contains the auditctl rules that are loaded# whenever the audit daemon is started via the initscripts.# The rules are simply the parameters that would be passed# to auditctl.# First rule - delete all-D# Increase the buffers to survive stress events.# Make this bigger for busy systems-b 320# Disable system call auditing.# Remove the following line if you need the auditing.-a never,task# Feel free to add below this line. See auditctl man pageAmazon Linux 2 and other operating systems that use systemd:$ sudo cat /etc/audit/audit.rules# This file is automatically generated from /etc/audit/rules.d-D-b 320-f 1Access the audit.rules file using an editor, such as the vi editor:Amazon Linux 1 and other operating systems that don't use systemd:$ sudo vi /etc/audit/audit.rulesAmazon Linux 2 and other operating systems that use systemd:$ sudo vi /etc/audit/rules.d/audit.rulesEdit the -b parameter to a larger value. The following example changes the -b value to 8192.$ sudo cat /etc/audit/audit.rules# This file contains the auditctl rules that are loaded# whenever the audit daemon is started via the initscripts.# The rules are simply the parameters that would be passed# to auditctl.# First rule - delete all-D# Increase the buffers to survive stress events.# Make this bigger for busy systems-b 8192# Disable system call auditing.# Remove the following line if you need the auditing.-a never,task# Feel free to add below this line. See auditctl man page$ sudo auditctl -senabled 1failure 1pid 2264rate_limit 0backlog_limit 320lost 0backlog 0Restart the auditd service. The new backlog_limit value takes effect. The value also updates in auditctl -s, as shown in the following example:# sudo service auditd stopStopping auditd:                                           [  OK  ]# sudo service auditd startStarting auditd:                                           [  OK  ]# auditctl -senabled 1failure 1pid 26823rate_limit 0backlog_limit 8192lost 0backlog 0Note: If your instance is inaccessible and you see backlog limit exceeded messages in the system log, stop and start the instance. Then, perform the preceding steps to change the audit buffer value.Follow"