How do I prevent "rate exceeded" ThrottlingException errors when I run monitoring scripts in Amazon EMR?
"To monitor my Amazon EMR clusters, I run scripts that make API calls. The scripts return errors similar to the following:"Rate exceeded (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ThrottlingException; Request ID: e2b6191c-gkl5-269r-u735-cryyz251a837)"How do I prevent "rate exceeded" errors?"
"To monitor my Amazon EMR clusters, I run scripts that make API calls. The scripts return errors similar to the following:"Rate exceeded (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ThrottlingException; Request ID: e2b6191c-gkl5-269r-u735-cryyz251a837)"How do I prevent "rate exceeded" errors?Short descriptionAmazon EMR throttles API calls to maintain system stability. Throttling exceptions usually occur when you run monitoring scripts at regular intervals to check clusters for a parameter. Here's an example: calling DescribeCluster every 60 seconds to check if the cluster has reached the WAITING state. The more clusters that you have and the more monitoring scripts that you run, the more likely you are to get throttling errors.ResolutionTo prevent throttling errors:Reduce the frequency of the API calls.Stagger the intervals of the API calls so that they don't all run at the same time.Implement exponential backoff when making API calls.Consider moving to an event-based architecture.To understand the source of throttling errors, use AWS CloudTrail to track Event History. CloudTrail can help identify event details such as the following:Frequent API callsRate exceeded errors and their related API callsWhether API calls are triggered by users or automationRelated informationCommon errorsManaging and monitoring API throttling in your workloadsHow CloudTrail worksFollow"