How do I troubleshoot HTTP 504 errors from an API Gateway REST API with a Lambda backend?
I receive an HTTP 504 error status code when I invoke a REST API using Amazon API Gateway with an AWS Lambda backend. How do I troubleshoot this error?
"I receive an HTTP 504 error status code when I invoke a REST API using Amazon API Gateway with an AWS Lambda backend. How do I troubleshoot this error?ResolutionWhen your REST API requests return an HTTP 504 error status code, you must run various checks.Check for IntegrationLatency spikesCheck if there's an IntegrationLatency spike in API Gateway by reviewing the integration latency duration time. To see the IntegrationLatency duration time, configure the access logging variable $context.integration.latency for HTTP API logging.For more information, see Set up Amazon CloudWatch API logging using the API Gateway console.An IntegrationLatency spike in API Gateway indicates that the request spent most of its time in Lambda. Check the Lambda function's Duration performance metric to confirm this.For more information, see Working with Lambda function metrics.Review requests with CloudWatch Logs InsightsUse Amazon CloudWatch Logs Insights to review requests that resulted in 504 errors. To review the requests, on the CloudWatch console, in the navigation pane, choose Logs, Log Insights. Select your API Gateway log group. Then, set the relative time with one of the following queries:parse @message '(*) *' as reqId, message| filter message like /Method completed with status: \d\d\d/| parse message 'Method completed with status: *' as status| filter status = 504| sort @timestamp desc| limit 20-or-fields @timestamp, @message| filter message like /Method completed with status: 504/| sort @timestamp desc| limit 20Instrument X-Ray tracesIf 504 errors keep happening continuously, determine where the Lambda function is spending its time. Instrument AWS X-Ray traces for Lambda functions based on the Lambda function runtime.For Python:from aws_xray_sdk.core import xray_recorderfrom aws_xray_sdk.core import patch_allpatch_all()For Node.js:const AWSXRay = require('aws-xray-sdk-core')const AWS = AWSXRay.captureAWS(require('aws-sdk'))Note: You need to build a new deployment package after instrumenting X-Ray traces on a Lambda function.Implement API retriesWhen a 504 error occurs and the request isn't found in Lambda, implement API retries on the client. The error might have resulted from a temporary network failure in API Gateway.Check the Lambda function configurationMake sure that your Lambda function has only API Gateway event-specific processing logic. This way, the Lambda function takes less time to run and can keep up with incoming events.REST APIs have a default maximum integration timeout of 29 seconds. Therefore, make sure that the Lambda function's running duration is under 29 seconds.If you have a use case where the application is time sensitive, see Set up asynchronous invocation of the backend Lambda function.Related informationAPI Gateway dimensions and metricsSetting up CloudWatch logging for a REST API in API GatewayView API Gateway log events in the CloudWatch consoleTracing user requests to REST APIs using X-RayUsing Lambda with X-RayFollow"