What are my options to analyze AWS WAF logs stored in CloudWatch or Amazon S3?
"I am storing my AWS WAF logs in Amazon CloudWatch, Amazon Simple Storage Solution (Amazon S3), or Amazon S3 as the destination for my Amazon Kinesis Data Firehose delivery stream. What options do I have to analyze my AWS WAF access logs."
"I am storing my AWS WAF logs in Amazon CloudWatch, Amazon Simple Storage Solution (Amazon S3), or Amazon S3 as the destination for my Amazon Kinesis Data Firehose delivery stream. What options do I have to analyze my AWS WAF access logs.ResolutionTo analyze and filter specific log requests, use Amazon CloudWatch Logs insights for CloudWatch logs or Amazon Athena for Amazon S3 logs.Analyzing AWS WAF access logs with CloudWatch Logs insightsOpen the Amazon CloudWatch console.In the navigation pane, choose Logs, and then choose Log Insights.For Select log group(s), choose one or more log groups to query that consist of AWS WAF access logs.(Optional) Choose a time range for the period that you want to query.Use query syntax to design queries.Choose Run to view the results for the log group.These are examples queries that you can use to filter out specific information for CloudWatch Logs Insights:Filter on a specific stringRun this query to filter the log based on a specific string:Note: Replace string {jndi:ldap. with the string you want to search.fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.httpMethod as Method,httpRequest.uri as URI| parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/| parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/| filter @message like "{jndi:ldap"| sort action, URI descFilter by hostRun this query to filter by host:Note: Replace Host value www.example.com with the Host you want to search.parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/| filter Host = "www.example.com"| fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.uri as URIFilter on POST requestsRun this query to isolate any POST requests:parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/| parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/| fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.httpMethod as Method, httpRequest.uri as URI, httpRequest.requestId as RequestID| filter httpRequest.httpMethod ="POST"| display Rule, action, Country, ClientIP, Method, URI, Host, UserAgent, RequestID| sort Rule, action descFilter on UserAgentRun this query to filter by UserAgent:Note: Replace User-Agent-Value with your UserAgent value**.**parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/| filter UserAgent like "<User-Agent-Value>"| fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.uri as URIFilter requests not originating from a countryRun this query to filter requests that don't originate from a specific country:fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.uri as URI| parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/| parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/| filter Country != "US"| sort Country, action descFilter for cross-site scripting or SQL injectionRun this query to filter for cross-site scripting or SQL injection:fields @timestamp, terminatingRuleId, action, httpRequest.clientIp as ClientIP, httpRequest.country as Country, terminatingRuleMatchDetails.0.conditionType as ConditionType, terminatingRuleMatchDetails.0.location as Location, terminatingRuleMatchDetails.0.matchedData.0 as MatchedData| filter ConditionType in["XSS","SQL_INJECTION"]Time series based on a terminating ruleRun this query to filter a time series based on a terminating rule:#Time Series by Terminating Rulefilter terminatingRuleId = "AWS-AWSManagedRulesCommonRuleSet"| stats count(*) as requestCount by bin(30m)Summarize blocked requests by ClientIP, country, URI, and ruleRun this query to summarize blocked requests by ClientIP, country, URI, and rule:fields httpRequest.clientIp as ClientIP, httpRequest.country as Country, httpRequest.uri as URI, terminatingRuleId as Rule| filter action = "BLOCK"| stats count(*) as RequestCount by Country, ClientIP, URI, Rule| sort RequestCount descTop client IPsRun this query to count the top client IPs:stats count(*) as RequestCount by httpRequest.clientIp as ClientIP| sort RequestCount descTop countriesRun this query to count the top countries:stats count(*) as RequestCount by httpRequest.country as Country| sort RequestCount descTop hostsRun this query to count the top hosts:parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/| stats count(*) as RequestCount by Host| sort RequestCount descTop methodsRun this query to count the top methods:stats count(*)as RequestCount by httpRequest.httpMethod as Method| sort RequestCount descTop terminating rulesRun this query to count the top terminating rules:stats count(*) as RequestCount by terminatingRuleId| sort RequestCount descTop UserAgentsRun this query to count the top UserAgents:parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/| stats count(*) as RequestCount by UserAgent| sort RequestCount descRequest not terminated by Default_Action or rules with action ALLOWRun this query to filter by requests not terminated by a Default_Action or rules with an ALLOW action:fields @timestamp, terminatingRuleId, action, @message| filter terminatingRuleId != 'Default_Action' and action != 'ALLOW'| sort @timestamp descRequest with invalid Captcha tokenRun this query to filter by requests with an invalid Captcha token:fields @timestamp, httpRequest.clientIp, httpRequest.requestId, captchaResponse.failureReason, @message|filter captchaResponse.failureReason ='TOKEN_MISSING'| sort @timestamp descRequest block by rate-based ruleRun this query to filter by requests blocked by a rate-based rule:fields @timestamp, httpRequest.clientIp, terminatingRuleId, httpRequest.country,@message| filter terminatingRuleType ="RATE_BASED" ## and webaclId = "arn:aws:wafv2:us-east-1:xxxxxxxx:regional/webacl/waf-test/abcdefghijkl" ## uncomment to filter for specific WebACL| sort requestCount descFilter all request detected by AWS Bot Control (ABC)Run this query to filter all requests detected by ABC:fields @timestamp, @message|filter @message like 'awswaf:managed:aws:bot-control'| parse @message '"labels":[*]' as Labels| sort @timestamp descAnalyzing AWS WAF access logs with Amazon AthenaYou can turn on AWS WAF access logging directly to an Amazon S3 bucket. Or, you can use Amazon Kinesis Data Firehose delivery stream to deliver your AWS WAF access logs to an Amazon S3 bucket. To store logs in Amazon S3, see How do I configure AWS WAF comprehensive logging to store logs in Amazon S3?When your access logs are in the Amazon S3 bucket, create the AWS WAF table to use Amazon Athena to query logs and filter various details.These queries are examples that you can use to query AWS WAF logs with Athena:Blocked requests with AWS WAF rule informationRun this Athena query to list all the blocked requests with AWS WAF rule:SELECT timestamp,    action,    httpsourcename,    httpsourceid,    httprequest.requestID,    httprequest.clientip,    webaclid,    terminatingruleid,    terminatingruletype,    rulegrouplist,    terminatingrulematchdetailsFROM "wafv2"."waf_logs"WHERE ("action" LIKE 'BLOCK')Request User AgentRun this Athena query to request the User Agent:Note: Replace User-Agent with your UserAgent value.select n.value, count(n.value) as countfrom waf_logscross joinunnest(  cast(    httprequest.headers as ARRAY(ROW(name VARCHAR, value VARCHAR))    )  ) as x(n)where n.name = 'User-Agent'group by n.valueORDER BY count(n.value) DESCRequest URIRun this Athena query to check the request URI:SELECT"httprequest"."uri", "count"(*) "count"FROM  waf_logsWHERE ("action" LIKE 'BLOCK')GROUP BY "httprequest"."uri"ORDER BY "count" DESCCount blocked requests based on ClientIPRun this Athena query to view the count of blocked requests based on ClientIP and country:SELECT  "httprequest"."clientip", "count"(*) "count", "httprequest"."country"FROMwaf_logsWHERE ("action" LIKE 'BLOCK')GROUP BY "httprequest"."clientip", "httprequest"."country"ORDER BY "count" DESCView request countRun this Athena query to view the request count:SELECT   "httprequest"."clientip", "count"(*) "count","httprequest"."country"FROM waf_logsWHERE ("action" LIKE'BLOCK')GROUP BY"httprequest"."clientip", "httprequest"."country"ORDER BY "count" DESCFor additional Athena query examples, see Example queries for AWS WAF logs.Follow"