How can I prevent an increasing backlog of messages in my Amazon SQS queue?
"The backlog of messages in my Amazon Simple Queue Service (Amazon SQS) queue is increasing. Why does this happen, and how can I prevent the backlog from increasing?"
"The backlog of messages in my Amazon Simple Queue Service (Amazon SQS) queue is increasing. Why does this happen, and how can I prevent the backlog from increasing?Short descriptionStandard and FIFO SQS queuesThe backlog of messages increases under the following circumstances:Producers send messages at a faster rate than the messages are consumed.Consumers dont delete messages within the visibility timeout period. When the SQS queue is polled, messages reappear in the queue.FIFO SQS queuesFor FIFO (First-In-First-Out) SQS queues, an increase in the backlog of messages can also result from either of the following:The FIFO queue 20,000 message buffer limit.A message that belongs to a message group is stuck with a consumer, preventing other messages from the same message group from processing.ResolutionFollow these best practices to prevent the backlog of messages from increasing.Standard and FIFO SQS queuesSet the SQS queue optimum visibility timeout to allow the consumer to delete messages after processing them within the visibility timeout period. If you dont know how long it takes to process a message, then create a heartbeat for your consumer process. Specify the initial visibility timeout (for example, 2 minutes). Then, if the consumer needs more time to process the message, continue to increase the visibility timeout using the ChangeMessageVisibility API call.Increase the batch size when you make ReceiveMessage API calls. Set the MaxNumberOfMessages parameter value to more than 1 and up to a maximum of 10.Monitor the SQS queue metric Approximate Number of Messages Visible. This metric allows you to see if producers start to produce messages at a higher rate than consumers can consume the messages. To scale horizontally, increase the number of consumers or clients that consume the SQS queue, or increase the number of threads that poll the queue.FIFO SQS queues20,000 message bufferFIFO queues allow a maximum of 20,000 inflight messages. Inflight messages include those received from a queue by a consumer, but not yet deleted from the queue. If you reach the 20,000 quota, Amazon SQS doesnt return error messages. A FIFO queue looks through the first 20,000 messages to determine available message groups. If you have a backlog of messages in a single message group, you cant consume messages from other message groups until you successfully consume messages from the backlog.Scaling over message groupsMessages that belong to the same message group are processed one by one, in the order relative to the message group. When receiving messages with multiple message group IDs, Amazon SQS tries first to return as many messages with the same message group ID as possible. This allows other consumers to process messages with a different message group ID. When messages that belong to a specific message group ID are invisible, no other consumer can process messages with the same message group ID. However, consumers can process messages from other message groups. Try increasing the number of message groups where the order isnt important.Follow"