How can I resolve SageMaker Python SDK rate exceeded and throttling exceptions?
How can I resolve throttling errors such as "botocore.exceptions.ClientError: An error occurred (ThrottlingException)" when using the Amazon SageMaker Python SDK?
"How can I resolve throttling errors such as "botocore.exceptions.ClientError: An error occurred (ThrottlingException)" when using the Amazon SageMaker Python SDK?Short descriptionAPI calls to any AWS service can't exceed the maximum allowed API request rate per account and per AWS Region. These API calls might be from an application, the AWS Command Line Interface (AWS CLI), or the AWS Management Console. If the API requests exceed the maximum rate, then you receive the “Rate Exceeded” error, and the API calls are throttled.You might get this error when calling the SageMaker APIs because of the default retry configuration in Boto3. You can override this configuration to increase the number of retry attempts and the timeouts for connecting and reading a response.You can resolve this error by adding a SageMaker boto3 client with a custom retry configuration to the SageMaker Python SDK client.Resolution1.    Create a SageMaker boto3 client with a custom retry configuration. Example:import boto3 from botocore.config import Configsm_boto = boto3.client('sagemaker', config=Config(connect_timeout=5, read_timeout=60, retries={'max_attempts': 20}))print(sm_boto.meta.config.retries)2.    Create a SageMaker Python SDK client using the boto3 client from the previous step. Example:import sagemakersagemaker_session = sagemaker.Session(sagemaker_client = sm_boto)region = sagemaker_session.boto_session.region_nameprint(sagemaker_session.sagemaker_client.meta.config.retries)3.    Test a SageMaker API with multiple requests from the SageMaker Python SDK. Example:import multiprocessingdef worker(TrainingJobName):    print(sagemaker_session.sagemaker_client          .describe_training_job(TrainingJobName=TrainingJobName)          ['TrainingJobName'])    returnif __name__ == '__main__':    jobs = []    TrainingJobName = 'your-job-name'    for i in range(10):        p = multiprocessing.Process(target=worker, args=(TrainingJobName,))        jobs.append(p)        p.start()4.    Create an instance of the sagemaker.estimator.Estimator class with the sagemaker_session parameter. Example:estimator = sagemaker.estimator.Estimator(container,                                             role,                                              train_instance_count=1,                                              train_instance_type='ml.c4.4xlarge',                                             train_volume_size = 30,                                             train_max_run = 360000,                                             input_mode= 'File',                                             output_path=s3_output_location,                                             sagemaker_session=sagemaker_session )5.    To confirm that the retry configuration resolves the throttling exceptions, launch a training job from the estimator that you created in the previous step:estimator.fit()Related informationBoto3 documentationFollow"