How do I troubleshoot Lambda function invocation timeout errors?
"My AWS Lambda function is timing out intermittently, even though I haven't deployed any code changes. How do I troubleshoot and prevent Lambda function invocation timeout issues?"
"My AWS Lambda function is timing out intermittently, even though I haven't deployed any code changes. How do I troubleshoot and prevent Lambda function invocation timeout issues?ResolutionLambda functions can time out for a variety of reasons. To troubleshoot Lambda function timeouts, first determine what's causing the issue by using one of the AWS services and features listed in this article. Then, remediate the problem based on your use case.To help prevent your Lambda function from timing out, see the Best practices for preventing Lambda function timeouts section of this article.Verify that your Lambda function is timing outRetrieve the request IDs of any timed-out invocations by searching the function's Amazon CloudWatch log group for the phrase Task timed out. Then, use the request IDs of the associated timed-out invocations to retrieve the full logs for each invocation timeout.For instructions, see How do I determine if my Lambda function is timing out?Identify what's causing your Lambda function to time outUse one or more of the following methods to identify the point of failure that's causing your function to time out:Review your CloudWatch Logs for LambdaYou can use Amazon CloudWatch to view all logs generated by your function's code and identify potential issues. For instructions, see Accessing CloudWatch Logs for Lambda.If your function is returning a stack trace, then the error message in the stack trace specifies what's causing the error.Important: Lambda generates three log lines for each invocation automatically (START, END, and REPORT). These three lines are the only lines that appear in your function's CloudWatch logs if either of the following is true:There's no other explicit logging configured in the Lambda function's custom code.The function's duration limit is reached before Lambda can run the function's code that outputs logs.If you can't determine what's causing the timeouts after reviewing the logs, try one or more of the following solutions:Verify that the retry count and timeout settings on the AWS SDK that you're using allow enough time for your function to initialize.Increase the Lambda function's timeout setting temporarily to allow enough time for the function code to generate log data.Increase the function's configured memory to help reduce invocation duration latency and increase computing power.To add more logging output to your function's code, see the following documentation for the Lambda runtime that you're using:AWS Lambda function logging in Node.jsAWS Lambda function logging in PythonAWS Lambda function logging in RubyAWS Lambda function logging in JavaAWS Lambda function logging in GoAWS Lambda function logging in C#AWS Lambda function logging in PowerShellUse AWS X-Ray to identify any code performance bottlenecksIf your Lambda function uses downstream AWS resources, microservices, databases, or HTTP web APIs, then you can use AWS X-Ray to help troubleshoot code performance issues.For more information, see Using AWS Lambda with AWS X-Ray.Use Lambda Insights to collect system-level metrics for your functionLambda Insights collects system-level metrics, including CPU time, memory, disk, and network metrics. It also collects diagnostic information, including cold starts and Lambda worker shutdowns to help you isolate issues with your Lambda functions.For more information, see Using Lambda Insights.Note: Using Lambda Insights incurs charges to your AWS account. You're charged for the invocation time consumed by the Lambda extension in 1 ms increments.Use VPC Flow Logs to determine why a specific invocation request was denied or didn't routeVPC Flow Logs allow you to see all the network traffic flowing to and from an Amazon Virtual Private Cloud (Amazon VPC).For more information, see Troubleshoot networking issues in Lambda.Note: Keep in mind the following if you choose to set up VPC Flow Logs:Data ingestion and archival charges for vended logs apply when you publish flow logs to either of the following:CloudWatch LogsAmazon Simple Storage Service (Amazon S3)When you configure a Lambda function to access resources in an Amazon VPC, Lambda assigns the function to an elastic network interface. To identify the network traffic associated with your Lambda function, you must locate the function's network interface. For instructions, see Why can't I detach or delete an elastic network interface that Lambda created?Use HTTP wire traces for detailed logging of network requests that are generated by the function's code during an invocationFor more information, see Logging HTTP wire traces.Best practices for preventing Lambda function timeoutsMake sure that your Lambda function is idempotentAPI calls can take longer than expected due to transient network issues. Network issues can also cause retries and duplicated API requests. To prepare for these occurrences, make sure that your Lambda function is idempotent.For more information, see How do I make my Lambda function idempotent?Initialize your function's static logic outside of the function handlerWhen you initialize a Lambda function, Lambda allocates up to 10 seconds for the Init phase of the invocation to complete. Because of this time constraint, it's a best practice to do the following outside of the function handler in initialization code:Import libraries and dependenciesSet up initial configurationInitialize connections to other downstream services and resourcesThis static initialization allows these resources to be initialized once per sandbox and then reused for all future invocations in the same runtime environment.For more information, see Optimizing static initialization. Also, Downstream unavailability in the Lambda Operator Guide and Function code in the Lambda Developer Guide.Note: Lambda removes idle connections to downstream resources. To allow your function to maintain a persistent connection, use the tcp_keepalive variable that's associated with the Lambda runtime that you're using.Verify that the retry count and timeout settings on the AWS SDK that you're using allow enough time for your function to initializeIf you make an API call using an AWS SDK and the call fails, then the AWS SDK automatically retries the call. How many times the AWS SDK retries and for how long is determined by settings that vary between each AWS SDK. Your function might need more time to initialize than the default AWS SDK settings allow.For more information, see How do I troubleshoot retry and timeout issues when invoking a Lambda function using an AWS SDK?(Optional) Configure provisioned concurrency for your Lambda functionProvisioned concurrency initializes a requested number of runtime environments so that they're prepared to respond immediately to your function's invocations. To set up provisioned concurrency for your function, follow the instructions in Configuring provisioned concurrency.Note: Configuring provisioned concurrency incurs charges to your AWS account. You can configure provisioned concurrency on a version of a function or on a Lambda function alias.Verify that your Lambda function has enough system resourcesThe amount of network bandwidth and CPU allocated to a Lambda function invocation is determined by the function's memory configuration.For more information, see Memory and computing power in the Lambda Operator Guide.Make sure that any background processes your Lambda function uses complete before the function handler returns a stringFor more information, see Understanding container reuse in AWS Lambda.Verify that your Lambda function is configured to work within the maximum timeout settings of any integrated AWS servicesEven though a Lambda function's maximum invocation timeout limit is 15 minutes, other AWS services may have different timeout limits.For example, Amazon API Gateway waits a maximum of 29 seconds for a Lambda function proxy invocation to complete. For more information, see How can I resolve the errors that I receive when I integrate API Gateway with a Lambda function? Also, see Using AWS Lambda with other services.Confirm that there's a valid network path to the endpoint that your function is trying to reachTo review your network settings, follow the instructions in How do I troubleshoot timeout issues with a Lambda function that's in an Amazon VPC?Follow"