<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step2:  Analyzing public cloud and hybrid networks
Public cloud and hybrid networks can be hard to debug and secure. Many of the standard tools (e.g., traceroute) do not work in the cloud setting, even though the types of paths that can emerge are highly complicated, depending on whether the endpoints are in the same region, different regions, across physical and virtual infrastructure, or whether public or private IPs of cloud instances are being used. 
At same time, the fast pace of evolution of these networks, where new subnets and instances can be spun up rapidly by different groups of people, creates a significant security risk. Network engineers need tools than can provide comprehensive guaratnees that services and applications are available and secure as intended at all possible times.
In this notebook, we show how Batfish can predict and help debug network paths for cloud and hybrid networks and how it can guarantee that the network's availability and security posture is exactly as desired.
Step3:  Initializing the Network and Snapshot
SNAPSHOT_PATH below can be updated to point to a custom snapshot directory. See instructions for how to package data for analysis.
Step4:  The network snapshot that we just initialized is illustrated below. It has a datacenter network with the standard leaf-spine design on the left. Though not strictly necessary, we have included a host srv-101 in this network to enable end-to-end analysis. The exit gateway of the datacenter connects to an Internet service provider (ASN 65200) that we call isp_dc.
The AWS network is shown on the right. It is spread across two regions, us-east-2 and us-west-2. Each region has two VPCs, one of which is meant to host Internet-facing services and the other is meant to host only private services. Subnets in the public-facing VPCs use an Internet gateway to send and receive traffic outside of AWS. The two VPCs in a region peer via a transit gateway. Each VPC has two subnets, and we have some instances running as well. 
The physical network connects to the AWS network using IPSec tunnels, shown in pink, between exitgw and the two transit gateways. BGP sessions run atop these tunnels to make endpoints aware of prefixes on the other side.
You can view configuration files that we used here. The AWS portion of the configuration is in the aws_configs subfolder. It has JSON files obtained via AWS APIs. An example script that packages AWS data into a Batfish snapshot is here.
Analyzing network paths
Batfish can help analyze cloud and hybrid networks by showing how exactly traffic flows (or not) in the network, which can help debug and fix configuration errors. Batfish can also help ensure that the network is configured exactly as desired, with respect to reachability and security policies. We illustrate these types of analysis below.
First, lets define a couple of maps to help with the analysis.
Step5:  Paths across VPCs within an AWS region
To see how traffic flows between two instances in the same region but across different VPCs, say, from hosts["east2_private"] to hosts["east2_public"], we can run a traceroute query across them as follows. 
In the query below, we use the name of the instance as the destination for the traceroute. This makes Batfish pick the instance's private (i.e., non-Elastic) IP (10.20.1.207). It does not pick the public IP because that those IPs do not reside on instances but are used by the Internet gateway to NAT instance's traffic in and out of AWS (see documentation). If an instance has multiple private IPs, Batfish will pick one at random. To make Batfish use a specific IP, supply that IP as the argument to the dstIps parameter.
Step6:  The trace above shows how traffic goes from host["east2_private"] to host["east2_public"] -- via the source subnet and VPC, then to the transit gateway, and finally to the destination VPC and subnet. Along the way, it also shows where the flow encounters security groups (at both instances) and network ACLs (at subnets). In this instance, all security groups and network ACLs permit this particular flow.
This type of insight into traffic paths, which helps understand and debug network configuration, is difficult to obtain otherwise. Traceroutes on the live AWS network do not yield any information if the flow does not make it through, and do not show why or where a packet is dropped.
Paths across AWS regions
The traceroute query below shows paths across instances in two different regions.
Step7:  We see that such traffic does not reach the destination but instead is dropped by the AWS backbone (ASN 16509). This happens because, in our network, there is no (transit gateway or VPC) peering between VPCs in different regions. So, the source subnet is unaware of the address space of the destination subnet, which makes it use the default route that points to the Internet gateway (igw-02fd68f94367a67c7). The Internet gateway forwards the packet to aws-backbone, after NAT'ing its source IP. The packet is eventually dropped as it is using a private address as destination. Recall that using the instance name as destination amounts to using its private IP.
The behavior is different if we use the public IP instead, as shown below.
Step8:  This traceroute starts out like the previous one, up until the AWS backbone (isp_16509) -- from source subnet to the Internet gateway which forwards it to the backbone, after source NAT'ing the packet. The backbone carries it to the internet gateway in the destination region (igw-0a8309f3192e7cea3), and this gateway NATs the packet's destination from the public IP to the instance's private IP. 
Connectivity between DC and AWS
A common mode to connect to AWS is using VPNs and BGP, that is, establish IPSec tunnels between exit gateways on the physical side and AWS gateways and run BGP on top of these tunnels to exchange prefixes. Incompatibility in either IPSec or BGP settings on the two sides means that connectivity between the DC and AWS will not work. 
Batfish can determine if the two sides are compatibly configured with respect to IPSec and BGP settings and if those sessions will come up. 
The query below lists the status of all IPSec sessions between the exitgw and AWS transit gateways (specified using the regular expression ^tgw- that matches those node names). This filtering lets us ignore any other IPSec sessions that may exist in our network and focus on DC-AWS connectivity.
Step9:  In the output above, we see all expected tunnels. Each transit gateways has two established sessions to exitgw. The default AWS behavior is to have two IPSec tunnels between gateways and physical nodes.
Now that we know IPSec tunnels are working, we can check BGP sessions. The query below lists the status of all BGP sessions where one end is an AWS transit gateway.
Step10:  The output above shows that all BGP sessions are established as expected. 
Paths from the DC to AWS
Finally, lets look at paths from the datacenter to AWS. The query below does that using the private IP of the public instance in us-east-2 region.
Step11:  We see that this traffic travels on the IPSec links between the datacenter's exitgw and the transit gateway in the destination region (tgw-06b348adabd13452d), and then makes it to the destination instance after making it successfully past the network ACL on the subnet node and the security group on the instance.
A different path emerges if we use the public IP of the same instance, as shown below.
Step12:  We now see that the traffic traverses the Internet via isp_65200 and the Internet gateway (igw-02fd68f94367a67c7), which NATs the destination address of the packet from the public to the private IP.
Evaluating the network's availability and security
In addition to helping you understand and debug network paths, Batfish can also help ensure that the network is correctly configured with respect to its availability and security policies.
As examples, the queries below evaluate which instances are or are not reachable from the Internet.
Step13:  We see that Batfish correctly computes that the two instances in the public subnets are accessible from the Internet, and the other two are not. 
We can compare the answers produced by Batfish to what is expected based on network policy. This comparison can ensure that all instances that are expected to host public-facing services are indeed reachable from the Internet, and all instances that are expecpted to host private services are indeed not accessible from the Internet.
We can similarly compute which instances are reachable from hosts in the datacenter, using the query like the following.
Step14:  We see that all four instances are accessible from the datacenter host.
Batfish allows a finer-grained evaluation of security policy as well. In our network, our intent is that the public instances should only allow SSH traffic. Let us see if this invariant actually holds.
Step15:  We see that, against our policy, the public-facing instance allows non-SSH traffic. To see examples of such traffic, we can run the following query.
Step16:  We thus see that our misconfigured public instance allows TCP traffic to port 3306 (MySQL).
In this and earlier reachability queries, we are not specifying anything about the flow to Batfish. It automatically figures out that the flow from the Internet that can reach hosts["east2_public"] must have 13.59.144.125 as its destination address, which after NAT'ing becomes the private IP of the instance. Such exhaustive analysis over all possible header spaces is unique to Batfish, which makes it an ideal tool for comprehensive availability and security analysis.
Batfish can also diagnose why certain traffic makes it past security groups and network ACLs. For example, we can run the testFilters question as below to reveal why the flow above made it past the security group on hosts["east2_public"].
<ASSISTANT_TASK:>
Python Code:
# Import packages
%run startup.py
bf = Session(host="localhost")
def show_first_trace(trace_answer_frame):
    
    Prints the first trace in the answer frame.
    
    In the presence of multipath routing, Batfish outputs all traces 
    from the source to destination. This function picks the first one.
    
    if len(trace_answer_frame) == 0:
        print("No flows found")
    else:
        show("Flow: {}".format(trace_answer_frame.iloc[0]['Flow']))
        show(trace_answer_frame.iloc[0]['Traces'][0])
    
def is_reachable(start_location, end_location, headers=None):
    
    Checks if the start_location can reach the end_location using specified packet headers.
    
    All possible headers are considered if headers is None. 
    
    ans = bf.q.reachability(pathConstraints=PathConstraints(startLocation=start_location, 
                                                           endLocation=end_location),
                          headers=headers).answer()
    return len(ans.frame()) > 0
Explanation: Analyzing public cloud and hybrid networks
Public cloud and hybrid networks can be hard to debug and secure. Many of the standard tools (e.g., traceroute) do not work in the cloud setting, even though the types of paths that can emerge are highly complicated, depending on whether the endpoints are in the same region, different regions, across physical and virtual infrastructure, or whether public or private IPs of cloud instances are being used. 
At same time, the fast pace of evolution of these networks, where new subnets and instances can be spun up rapidly by different groups of people, creates a significant security risk. Network engineers need tools than can provide comprehensive guaratnees that services and applications are available and secure as intended at all possible times.
In this notebook, we show how Batfish can predict and help debug network paths for cloud and hybrid networks and how it can guarantee that the network's availability and security posture is exactly as desired.
End of explanation
# Initialize a network and snapshot
NETWORK_NAME = "hybrid-cloud"
SNAPSHOT_NAME = "snapshot"
SNAPSHOT_PATH = "networks/hybrid-cloud"
bf.set_network(NETWORK_NAME)
bf.init_snapshot(SNAPSHOT_PATH, name=SNAPSHOT_NAME, overwrite=True)
Explanation: Initializing the Network and Snapshot
SNAPSHOT_PATH below can be updated to point to a custom snapshot directory. See instructions for how to package data for analysis.
End of explanation
#Instances in AWS in each region and VPC type (public, private)
hosts = {}
hosts["east2_private"] = "i-04cd3db5124a05ee6"
hosts["east2_public"] = "i-01602d9efaed4409a"
hosts["west2_private"] = "i-0a5d64b8b58c6dd09"
hosts["west2_public"] = "i-02cae6eaa9edeed70"
#Public IPs of instances in AWS
public_ips = {}
public_ips["east2_public"] = "13.59.144.125" # of i-01602d9efaed4409a
public_ips["west2_public"] = "54.191.42.182" # of i-02cae6eaa9edeed70
Explanation: The network snapshot that we just initialized is illustrated below. It has a datacenter network with the standard leaf-spine design on the left. Though not strictly necessary, we have included a host srv-101 in this network to enable end-to-end analysis. The exit gateway of the datacenter connects to an Internet service provider (ASN 65200) that we call isp_dc.
The AWS network is shown on the right. It is spread across two regions, us-east-2 and us-west-2. Each region has two VPCs, one of which is meant to host Internet-facing services and the other is meant to host only private services. Subnets in the public-facing VPCs use an Internet gateway to send and receive traffic outside of AWS. The two VPCs in a region peer via a transit gateway. Each VPC has two subnets, and we have some instances running as well. 
The physical network connects to the AWS network using IPSec tunnels, shown in pink, between exitgw and the two transit gateways. BGP sessions run atop these tunnels to make endpoints aware of prefixes on the other side.
You can view configuration files that we used here. The AWS portion of the configuration is in the aws_configs subfolder. It has JSON files obtained via AWS APIs. An example script that packages AWS data into a Batfish snapshot is here.
Analyzing network paths
Batfish can help analyze cloud and hybrid networks by showing how exactly traffic flows (or not) in the network, which can help debug and fix configuration errors. Batfish can also help ensure that the network is configured exactly as desired, with respect to reachability and security policies. We illustrate these types of analysis below.
First, lets define a couple of maps to help with the analysis.
End of explanation
# traceroute between instances in the same region, using SSH
ans = bf.q.traceroute(startLocation=hosts["east2_private"], 
                     headers=HeaderConstraints(dstIps=hosts["east2_public"], 
                                               applications="ssh")).answer()
show_first_trace(ans.frame())
Explanation: Paths across VPCs within an AWS region
To see how traffic flows between two instances in the same region but across different VPCs, say, from hosts["east2_private"] to hosts["east2_public"], we can run a traceroute query across them as follows. 
In the query below, we use the name of the instance as the destination for the traceroute. This makes Batfish pick the instance's private (i.e., non-Elastic) IP (10.20.1.207). It does not pick the public IP because that those IPs do not reside on instances but are used by the Internet gateway to NAT instance's traffic in and out of AWS (see documentation). If an instance has multiple private IPs, Batfish will pick one at random. To make Batfish use a specific IP, supply that IP as the argument to the dstIps parameter.
End of explanation
# traceroute between instances across region using the destination's private IP
ans = bf.q.traceroute(startLocation=hosts["east2_public"], 
                     headers=HeaderConstraints(dstIps=hosts["west2_public"],
                                              applications="ssh")).answer()
show_first_trace(ans.frame())
Explanation: The trace above shows how traffic goes from host["east2_private"] to host["east2_public"] -- via the source subnet and VPC, then to the transit gateway, and finally to the destination VPC and subnet. Along the way, it also shows where the flow encounters security groups (at both instances) and network ACLs (at subnets). In this instance, all security groups and network ACLs permit this particular flow.
This type of insight into traffic paths, which helps understand and debug network configuration, is difficult to obtain otherwise. Traceroutes on the live AWS network do not yield any information if the flow does not make it through, and do not show why or where a packet is dropped.
Paths across AWS regions
The traceroute query below shows paths across instances in two different regions.
End of explanation
# traceroute betwee instances across region using the destination's public IP
ans = bf.q.traceroute(startLocation=hosts["east2_public"], 
                     headers=HeaderConstraints(dstIps=public_ips["west2_public"],
                                              applications="ssh")).answer()
show_first_trace(ans.frame())
Explanation: We see that such traffic does not reach the destination but instead is dropped by the AWS backbone (ASN 16509). This happens because, in our network, there is no (transit gateway or VPC) peering between VPCs in different regions. So, the source subnet is unaware of the address space of the destination subnet, which makes it use the default route that points to the Internet gateway (igw-02fd68f94367a67c7). The Internet gateway forwards the packet to aws-backbone, after NAT'ing its source IP. The packet is eventually dropped as it is using a private address as destination. Recall that using the instance name as destination amounts to using its private IP.
The behavior is different if we use the public IP instead, as shown below.
End of explanation
# show the status of all IPSec tunnels between exitgw and AWS transit gateways
ans = bf.q.ipsecSessionStatus(nodes="exitgw", remoteNodes="/^tgw-/").answer()
show(ans.frame())
Explanation: This traceroute starts out like the previous one, up until the AWS backbone (isp_16509) -- from source subnet to the Internet gateway which forwards it to the backbone, after source NAT'ing the packet. The backbone carries it to the internet gateway in the destination region (igw-0a8309f3192e7cea3), and this gateway NATs the packet's destination from the public IP to the instance's private IP. 
Connectivity between DC and AWS
A common mode to connect to AWS is using VPNs and BGP, that is, establish IPSec tunnels between exit gateways on the physical side and AWS gateways and run BGP on top of these tunnels to exchange prefixes. Incompatibility in either IPSec or BGP settings on the two sides means that connectivity between the DC and AWS will not work. 
Batfish can determine if the two sides are compatibly configured with respect to IPSec and BGP settings and if those sessions will come up. 
The query below lists the status of all IPSec sessions between the exitgw and AWS transit gateways (specified using the regular expression ^tgw- that matches those node names). This filtering lets us ignore any other IPSec sessions that may exist in our network and focus on DC-AWS connectivity.
End of explanation
# show the status of all BGP sessions between exitgw and AWS transit gateways
ans = bf.q.bgpSessionStatus(nodes="exitgw", remoteNodes="/^tgw-/").answer()
show(ans.frame())
Explanation: In the output above, we see all expected tunnels. Each transit gateways has two established sessions to exitgw. The default AWS behavior is to have two IPSec tunnels between gateways and physical nodes.
Now that we know IPSec tunnels are working, we can check BGP sessions. The query below lists the status of all BGP sessions where one end is an AWS transit gateway.
End of explanation
# traceroute from DC host to an instances using private IP
ans = bf.q.traceroute(startLocation="srv-101", 
                     headers=HeaderConstraints(dstIps=hosts["east2_public"],
                                              applications="ssh")).answer()
show_first_trace(ans.frame())
Explanation: The output above shows that all BGP sessions are established as expected. 
Paths from the DC to AWS
Finally, lets look at paths from the datacenter to AWS. The query below does that using the private IP of the public instance in us-east-2 region.
End of explanation
# traceroute from DC host to an instances using public IP
ans = bf.q.traceroute(startLocation="srv-101", 
                     headers=HeaderConstraints(dstIps=public_ips["east2_public"],
                                              applications="ssh")).answer()
show_first_trace(ans.frame())
Explanation: We see that this traffic travels on the IPSec links between the datacenter's exitgw and the transit gateway in the destination region (tgw-06b348adabd13452d), and then makes it to the destination instance after making it successfully past the network ACL on the subnet node and the security group on the instance.
A different path emerges if we use the public IP of the same instance, as shown below.
End of explanation
# compute which instances are open to the Internet
reachable_from_internet = [key for (key, value) in hosts.items() if is_reachable("internet", value)]
print("\nInstances reachable from the Internet: {}".format(sorted(reachable_from_internet)))
# compute which instances are NOT open to the Internet
unreachable_from_internet = [key for (key, value) in hosts.items() if not is_reachable("internet", value)]
print("\nInstances NOT reachable from the Internet: {}".format(sorted(unreachable_from_internet)))
Explanation: We now see that the traffic traverses the Internet via isp_65200 and the Internet gateway (igw-02fd68f94367a67c7), which NATs the destination address of the packet from the public to the private IP.
Evaluating the network's availability and security
In addition to helping you understand and debug network paths, Batfish can also help ensure that the network is correctly configured with respect to its availability and security policies.
As examples, the queries below evaluate which instances are or are not reachable from the Internet.
End of explanation
# compute which instances are reachable from data center
reachable_from_dc = [key for (key,value) in hosts.items() if is_reachable("srv-101", value)]
print("\nInstances reachable from the DC: {}".format(sorted(reachable_from_dc)))
Explanation: We see that Batfish correctly computes that the two instances in the public subnets are accessible from the Internet, and the other two are not. 
We can compare the answers produced by Batfish to what is expected based on network policy. This comparison can ensure that all instances that are expected to host public-facing services are indeed reachable from the Internet, and all instances that are expecpted to host private services are indeed not accessible from the Internet.
We can similarly compute which instances are reachable from hosts in the datacenter, using the query like the following.
End of explanation
tcp_non_ssh = HeaderConstraints(ipProtocols="tcp", dstPorts="!22")
reachable_from_internet_non_ssh = [key for (key, value) in hosts.items() 
                                   if is_reachable("internet", value, tcp_non_ssh)]
print("\nInstances reachable from the Internet with non-SSH traffic: {}".format(
    sorted(reachable_from_internet_non_ssh)))
Explanation: We see that all four instances are accessible from the datacenter host.
Batfish allows a finer-grained evaluation of security policy as well. In our network, our intent is that the public instances should only allow SSH traffic. Let us see if this invariant actually holds.
End of explanation
ans = bf.q.reachability(pathConstraints=PathConstraints(startLocation="internet", 
                                                       endLocation=hosts["east2_public"]),
                      headers=tcp_non_ssh).answer()
show_first_trace(ans.frame())
Explanation: We see that, against our policy, the public-facing instance allows non-SSH traffic. To see examples of such traffic, we can run the following query.
End of explanation
flow=ans.frame().iloc[0]['Flow']  # the rogue flow uncovered by Batfish above
ans = bf.q.testFilters(nodes=hosts["east2_public"], 
                      filters="~INGRESS_ACL~eni-01997085076a9b98a",
                      headers=HeaderConstraints(srcIps=flow.srcIp, 
                                                dstIps="10.20.1.207", # destination IP after the NAT at Step 3 above
                                                srcPorts=flow.srcPort, 
                                                dstPorts=flow.dstPort, 
                                                ipProtocols=flow.ipProtocol)).answer()
show(ans.frame())
Explanation: We thus see that our misconfigured public instance allows TCP traffic to port 3306 (MySQL).
In this and earlier reachability queries, we are not specifying anything about the flow to Batfish. It automatically figures out that the flow from the Internet that can reach hosts["east2_public"] must have 13.59.144.125 as its destination address, which after NAT'ing becomes the private IP of the instance. Such exhaustive analysis over all possible header spaces is unique to Batfish, which makes it an ideal tool for comprehensive availability and security analysis.
Batfish can also diagnose why certain traffic makes it past security groups and network ACLs. For example, we can run the testFilters question as below to reveal why the flow above made it past the security group on hosts["east2_public"].
End of explanation
<END_TASK>