How can I analyze custom VPC Flow Logs using CloudWatch Logs Insights?
I have configured custom VPC Flow Logs. How can I discover patterns and trends with Amazon CloudWatch Logs Insights?
"I have configured custom VPC Flow Logs. How can I discover patterns and trends with Amazon CloudWatch Logs Insights?Short descriptionYou can use CloudWatch Logs Insights to analyze VPC Flow Logs. CloudWatch Log Insights automatically discovers fields in many Amazon provided logs, as well as JSON formatted log events, to allow for easy query construction and log exploration. VPC Flow Logs that are in the default format are automatically discovered by CloudWatch Logs Insights.But, VPC Flow Logs are deployed in a custom format. Because of this, they aren't automatically discovered, so you must modify the queries. This article gives several examples of queries that you can customize and extend to match your use cases.This custom VPC Flow Logs format is used:${account-id} ${vpc-id} ${subnet-id} ${interface-id} ${instance-id} ${srcaddr} ${srcport} ${dstaddr} ${dstport} ${protocol} ${packets} ${bytes} ${action} ${log-status} ${start} ${end} ${flow-direction} ${traffic-path} ${tcp-flags} ${pkt-srcaddr} ${pkt-src-aws-service} ${pkt-dstaddr} ${pkt-dst-aws-service} ${region} ${az-id} ${sublocation-type} ${sublocation-id}ResolutionRetrieve latest VPC Flow LogsBecause log fields are not automatically discovered by CloudWatch Logs Insights, you must use the parse keyword to isolate desired fields. In this query, the results are sorted by the flow log event start time, and restricted to the two most recent log entries.Query#Retrieve latest custom VPC Flow Logsparse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * *" as account_id, vpc_id, subnet_id, interface_id,instance_id, srcaddr, srcport, dstaddr, dstport, protocol, packets, bytes, action, log_status, start, end, flow_direction, traffic_path, tcp_flags, pkt_srcaddr, pkt_src_aws_service, pkt_dstaddr, pkt_dst_aws_service, region, az_id, sublocation_type, sublocation_id| sort start desc| limit 2Resultsaccount_id vpc_id  subnet_id interface_idinstance_idsrcaddrsrcport123456789012 vpc-0b69ce8d04278ddd subnet-002bdfe1767d0ddb0eni-0435cbb62960f230e- 172.31.0.10455125123456789012 vpc-0b69ce8d04278ddd1 subnet-002bdfe1767d0ddb0eni-0435cbb62960f230e- 91.240.118.8149422Summarize data transfers by source/destination IP address pairsNext, summarize the network traffic by source/destination IP address pairs. In this example, the sum statistic is used to perform an aggregation on the bytes field. This calculates a cumulative total of the data transferred between hosts. For more context, the flow_direction is included. The results of this aggregation are then assigned to the Data_Transferred field, temporarily. Then, the results are sorted by Data_Transferred in descending order, and the two largest pairs are returned.Queryparse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * *" as account_id, vpc_id, subnet_id, interface_id,instance_id, srcaddr, srcport, dstaddr, dstport, protocol, packets, bytes, action, log_status, start, end, flow_direction, traffic_path, tcp_flags, pkt_srcaddr, pkt_src_aws_service, pkt_dstaddr, pkt_dst_aws_service, region, az_id, sublocation_type, sublocation_id| stats sum(bytes) as Data_Transferred by srcaddr, dstaddr, flow_direction| sort by Data_Transferred desc| limit 2Resultssrcaddrdstaddrflow_directionData_Transferred172.31.1.2473.230.172.154egress346952038172.31.0.463.230.172.154egress343799447Analyze data transfers by EC2 instance IDYou can use custom VPC Flow Logs to analyze an Amazon Elastic Compute Cloud (Amazon EC2) instance ID, directly. Taking the previous query, you can now determine the most active EC2 instances by using the instance_id field.Queryparse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * *" as account_id, vpc_id, subnet_id, interface_id,instance_id, srcaddr, srcport, dstaddr, dstport, protocol, packets, bytes, action, log_status, start, end, flow_direction, traffic_path, tcp_flags, pkt_srcaddr, pkt_src_aws_service, pkt_dstaddr, pkt_dst_aws_service, region, az_id, sublocation_type, sublocation_id| stats sum(bytes) as Data_Transferred by instance_id| sort by Data_Transferred desc| limit 5Resultsinstance_idData_Transferred-1443477306i-03205758c9203c979517558754i-0ae33894105aa500c324629414i-01506ab9e9e90749d198063232i-0724007fef3cb06f354847643Filter for rejected SSH trafficTo better understand the traffic that was denied by your security group and network access control lists (ACL), filter on reject VPC Flow Logs. You can further narrow this filter down to include protocol and target port. To identify hosts that are rejected on SSH traffic, extend the filter to include TCP protocol (for example, protocol 6) and traffic with a destination port of 22.Queryparse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * *" as account_id, vpc_id, subnet_id, interface_id,instance_id, srcaddr, srcport, dstaddr, dstport, protocol, packets, bytes, action, log_status, start, end, flow_direction, traffic_path, tcp_flags, pkt_srcaddr, pkt_src_aws_service, pkt_dstaddr, pkt_dst_aws_service, region, az_id, sublocation_type, sublocation_id| filter action = "REJECT" and protocol = 6 and dstport = 22| stats sum(bytes) as SSH_Traffic_Volume by srcaddr| sort by SSH_Traffic_Volume desc| limit 2ResultssrcaddrSSH_Traffic_Volume23.95.222.129160179.43.167.7480Isolate HTTP data stream for a specific source/destination pairTo further investigate trends in your data using CloudWatch Logs Insights, isolate bidirectional traffic between two IP addresses. In this query, ["172.31.1.247","172.31.11.212"] returns flow logs using either IP address as the source or destination IP address. To isolate HTTP traffic, the filter statements match VPC Flow Log events with protocol 6 (TCP) and port 80. Use the display keyword to return a subset of all available fields.Query#HTTP Data Stream for Specific Source/Destination Pairparse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * *" as account_id, vpc_id, subnet_id, interface_id,instance_id, srcaddr, srcport, dstaddr, dstport, protocol, packets, bytes, action, log_status, start, end, flow_direction, traffic_path, tcp_flags, pkt_srcaddr, pkt_src_aws_service, pkt_dstaddr, pkt_dst_aws_service, region, az_id, sublocation_type, sublocation_id| filter srcaddr in ["172.31.1.247","172.31.11.212"] and dstaddr in ["172.31.1.247","172.31.11.212"] and protocol = 6 and (dstport = 80 or srcport=80)| display interface_id,srcaddr, srcport, dstaddr, dstport, protocol, bytes, action, log_status, start, end, flow_direction, tcp_flags| sort by start desc| limit 2Resultsinterface_idsrcaddrsrcportdstaddrdstportprotocolbytesactionlog_statuseni-0b74120275654905e172.31.11.21280172.31.1.2472937665160876ACCEPTOKeni-0b74120275654905e172.31.1.24729376172.31.11.21280697380ACCEPTOKIsolate HTTP data stream for specific source/destination pairYou can use CloudWatch Logs Insights to visualize results as a bar or pie chart. If the results include the bin() function, then query results are returned with a timestamp. This timeseries can then be visualized with a line or stacked area graph.Building on the previous query, you can use stats sum(bytes) as Data_Trasferred by bin(1m) to calculate the cumulative data transferred over one-minute intervals. To view this visualization, toggle between the Logs and Visualization tables in the CloudWatch Logs Insights console.Queryparse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * *" as account_id, vpc_id, subnet_id, interface_id,instance_id, srcaddr, srcport, dstaddr, dstport, protocol, packets, bytes, action, log_status, start, end, flow_direction, traffic_path, tcp_flags, pkt_srcaddr, pkt_src_aws_service, pkt_dstaddr, pkt_dst_aws_service, region, az_id, sublocation_type, sublocation_id| filter srcaddr in ["172.31.1.247","172.31.11.212"] and dstaddr in ["172.31.1.247","172.31.11.212"] and protocol = 6 and (dstport = 80 or srcport=80)| stats sum(bytes) as Data_Transferred by bin(1m)Resultsbin(1m)Data_Transferred2022-04-01 15:23:00.000172257872022-04-01 15:21:00.000177244992022-04-01 15:20:00.00011255002022-04-01 15:19:00.0001015252022-04-01 15:18:00.00081376Related informationSupported logs and discovered fieldsAnalyzing log data with CloudWatch Logs InsightsCloudWatch Logs Insights query commandsTutorial: Run a query that produces a time series visualizationFollow"