<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step1:  Safely refactoring ACLs and firewall rules
Changing ACLs or firewall rules (or filters) is one of the riskiest updates to a network. Even a small error can block connectivity for a large set of critical services or open up sensitive resources to the world at large. Earlier notebooks showed how to analyze filters for what they do and do not allow and how to make specific changes in a provably safe manner.
This notebook shows how to refactor complex filters in a way that the full impact of refactoring can be understood and analyzed for correctness before refactored filters are pushed to the network. 
Original ACL
We will use the following ACL as a running example in this notebook. The ACL can be read as a few separate sections
Step3:  Compressed ACL
Now, assume that we want to compress this ACL to make it more manageable. We do the following operations
Step4:  The challenge for us is to find out if and how this compressed ACL differs from the original. That is, is there is traffic that is treated differently by the two ACLs, and if so, which lines are responsible for the difference.
This task is difficult to get right through manual reasoning alone, which is why we developed the compareFilters question in Batfish.
Comparing filters
We can compare the two ACLs above as follows. To initialize snapshots, we will use Batfish's init_snapshot_from_text function which creates a snapshot with a single device who configuration is the provided text. The analysis shown below can be done even when the filters are embedded within bigger device configurations.
Step6:  The compareFilters question compares two filters and returns pairs of lines, one from each filter, that match the same flow(s) but treat them differently. If it reports no output, the filters are guaranteed to be identical. The analysis is exhaustive and considers all possible flows.
As we can see from the output above, our compressed ACL is not the same as the original one. In particular, line 210 of the compressed ACL will deny some flows that were being permitted by line 510 of the original; and line 510 of the compressed ACL will permit some flows that were being denied by line 220 of the original ACL. Because the permit statements correspond to ICMP traffic, we can tell that the traffic treated by the two filters is ICMP. To narrow learn specific source and destination IPs that are impacted, one may run the searchFilters question, as shown here. 
By looking at the output above, we can immediately understand the difference
Step7:  Given the split ACLs above, one analysis may be to figure out if each untrusted source subnet was included in a smaller ACL. Otherwise, we have lost protection that was present in the original ACL. We can accomplish this analysis via the findMatchingFilterLines question, as shown below. 
Once we are satisfied with analysis of filters, for an end-to-end safety guarantee, we should also analyze if there are new flows that the network will allow (or disallow) after the change. Such an analysis can be done via the differentialReachability question, as shown here.
<ASSISTANT_TASK:>
Python Code:
# The ACL before refactoring
original_acl = 
ip access-list acl
  10 deny icmp any any redirect
  20 permit udp 117.186.185.0/24 range 49152 65535 117.186.185.0/24 eq 3784
  30 permit udp 117.186.185.0/24 range 49152 65535 117.186.185.0/24 eq 3785
  40 permit tcp 11.36.216.170/32 11.36.216.169/32 eq bgp
  50 permit tcp 11.36.216.176/32 11.36.216.179/32 eq bgp
  60 permit tcp 204.150.33.175/32 204.150.33.83/32 eq bgp
  70 permit tcp 205.248.59.64/32 205.248.59.67/32 eq bgp
  80 permit tcp 205.248.58.190/32 205.248.58.188/32 eq bgp
  90 deny udp 10.10.10.42/32 218.8.104.58/32 eq domain
  100 permit udp 10.10.10.0/24 218.8.104.58/32 eq domain
  110 deny ip 54.0.0.0/8 any
  120 deny ip 163.157.0.0/16 any
  130 deny ip 166.144.0.0/12 any
  140 deny ip 198.170.50.0/24 any
  150 deny ip 198.120.0.0/16 any
  160 deny ip 11.36.192.0/19 any
  170 deny ip 11.125.64.0/19 any
  180 permit ip 166.146.58.184/32 any
  190 deny ip 218.66.57.0/24 any
  200 deny ip 218.66.56.0/24 any
  210 deny ip 218.67.71.0/24 any
  220 deny ip 218.67.72.0/24 any
  230 deny ip 218.67.96.0/22 any
  240 deny ip 8.89.120.0/22 any
  250 deny ip 54.203.159.1/32 any
  260 permit ip 218.8.104.0/25 any
  270 permit ip 218.8.104.128/25 any
  280 permit ip 218.8.103.0/24 any
  290 deny ip 144.49.45.40/32 any
  300 deny ip 163.255.18.63/32 any
  310 deny ip 202.45.130.141/32 any
  320 deny ip 212.26.132.18/32 any
  330 deny ip 218.111.16.132/32 any
  340 deny ip 218.246.165.90/32 any
  350 deny ip 29.228.179.210/32 any
  360 deny ip 194.181.135.214/32 any
  370 deny ip 10.64.90.249/32 any
  380 deny ip 207.70.46.217/32 any
  390 deny ip 219.185.241.117/32 any
  400 deny ip 2.80.3.219/32 any
  410 deny ip 27.212.145.150/32 any
  420 deny ip 131.159.53.215/32 any
  430 deny ip 214.220.213.107/32 any
  440 deny ip 196.64.84.239/32 any
  450 deny ip 28.69.250.136/32 any
  460 deny ip 200.45.87.238/32 any
  470 deny ip any 11.125.89.32/30
  480 deny ip any 11.125.89.36/30
  490 deny ip any 11.125.89.40/30
  500 deny ip any 11.125.89.44/30
  510 permit icmp any any echo-reply
  520 deny ip any 11.36.199.216/30
  530 deny ip any 11.36.199.36/30
  540 deny ip any 11.36.199.2/30
  550 deny ip any 11.36.199.52/30
  560 deny ip any 11.36.199.20/30
  570 deny ip any 11.125.82.216/30
  580 deny ip any 11.125.82.220/32
  590 deny ip any 11.125.82.36/30
  600 deny ip any 11.125.82.12/30
  610 deny ip any 11.125.80.136/30
  620 deny ip any 11.125.80.141/32
  630 deny ip any 11.125.87.48/30
  640 deny ip any 11.125.87.168/30
  650 deny ip any 11.125.87.173/32
  660 deny ip any 11.125.90.56/30
  670 deny ip any 11.125.90.240/30
  680 deny ip any 11.125.74.224/30
  690 deny ip any 11.125.91.132/30
  700 deny ip any 11.125.89.132/30
  710 deny ip any 11.125.89.12/30
  720 deny ip any 11.125.92.108/30
  730 deny ip any 11.125.92.104/32
  740 deny ip any 11.125.92.28/30
  750 deny ip any 11.125.92.27/32
  760 deny ip any 11.125.92.160/30
  770 deny ip any 11.125.92.164/32
  780 deny ip any 11.125.92.204/30
  790 deny ip any 11.125.92.202/32
  800 deny ip any 11.125.93.192/29
  810 deny ip any 11.125.95.204/30
  820 deny ip any 11.125.95.224/30
  830 deny ip any 11.125.95.180/30
  840 deny ip any 11.125.95.156/30
  850 deny tcp any any
  860 deny icmp any any
  870 deny udp any any
  880 deny ip any any
Explanation: Safely refactoring ACLs and firewall rules
Changing ACLs or firewall rules (or filters) is one of the riskiest updates to a network. Even a small error can block connectivity for a large set of critical services or open up sensitive resources to the world at large. Earlier notebooks showed how to analyze filters for what they do and do not allow and how to make specific changes in a provably safe manner.
This notebook shows how to refactor complex filters in a way that the full impact of refactoring can be understood and analyzed for correctness before refactored filters are pushed to the network. 
Original ACL
We will use the following ACL as a running example in this notebook. The ACL can be read as a few separate sections:
Line 10: Deny ICMP redirects
Lines 20, 23: Permit BFD traffic on certain blocks
Lines 40-80: Permit BGP traffic
Lines 90-100: Permit DNS traffic a /24 subnet while denying it from a /32 within that
Lines 110-500: Permit or deny IP traffic from certain subnets
Line 510: Permit ICMP echo reply
Lines 520-840: Deny IP traffic to certain subnets
Lines 850-880: Deny all other types of traffic
(The IP address space in the ACL appears all over the place because it has been anonymized via Netconan. Netconan preserves the super- and sub-prefix relationships when anonymizing IP addresses and prefixes.)
End of explanation
compressed_acl = 
ip access-list acl
  10 deny icmp any any redirect
  20 permit udp 117.186.185.0/24 range 49152 65535 117.186.185.0/24 range 3784 3785
! 30 MERGED WITH LINE ABOVE 
  40 permit tcp 11.36.216.170/32 11.36.216.169/32 eq bgp
  50 permit tcp 11.36.216.176/32 11.36.216.179/32 eq bgp
  60 permit tcp 204.150.33.175/32 204.150.33.83/32 eq bgp
  70 permit tcp 205.248.59.64/32 205.248.59.67/32 eq bgp
! 80 DECOMMISSIONED BGP SESSION
  90 deny udp 10.10.10.42/32 218.8.104.58/32 eq domain
  100 permit udp 10.10.10.0/24 218.8.104.58/32 eq domain
  110 deny ip 54.0.0.0/8 any
  120 deny ip 163.157.0.0/16 any
  130 deny ip 166.144.0.0/12 any
  140 deny ip 198.170.50.0/24 any
  150 deny ip 198.120.0.0/16 any
  160 deny ip 11.36.192.0/19 any
  170 deny ip 11.125.64.0/19 any
! 180 REMOVED UNREACHABLE LINE
  190 deny ip 218.66.56.0/23 any
! 200 MERGED WITH LINE ABOVE 
  210 deny ip 218.67.71.0/23 any
! 220 MERGED WITH LINE ABOVE  
  230 deny ip 218.67.96.0/22 any
  240 deny ip 8.89.120.0/22 any
! 250 REMOVED UNREACHABLE LINE
  260 permit ip 218.8.104.0/24 any
! 270 MERGED WITH LINE ABOVE
  280 permit ip 218.8.103.0/24 any
  290 deny ip 144.49.45.40/32 any
  300 deny ip 163.255.18.63/32 any
  310 deny ip 202.45.130.141/32 any
  320 deny ip 212.26.132.18/32 any
  330 deny ip 218.111.16.132/32 any
  340 deny ip 218.246.165.90/32 any
  350 deny ip 29.228.179.210/32 any
  360 deny ip 194.181.135.214/32 any
  370 deny ip 10.64.90.249/32 any
  380 deny ip 207.70.46.217/32 any
  390 deny ip 219.185.241.117/32 any
  400 deny ip 2.80.3.219/32 any
  410 deny ip 27.212.145.150/32 any
  420 deny ip 131.159.53.215/32 any
  430 deny ip 214.220.213.107/32 any
  440 deny ip 196.64.84.239/32 any
  450 deny ip 28.69.250.136/32 any
  460 deny ip 200.45.87.238/32 any
  470 deny ip any 11.125.89.32/28
  510 permit icmp any any echo-reply
! 520-870 REMOVED UNNECESSARY DENIES
  880 deny ip any any
Explanation: Compressed ACL
Now, assume that we want to compress this ACL to make it more manageable. We do the following operations:
Merge the two BFD permit statements on lines 20-30 into one statement using the range directive.
Remove the BGP session on line 80 because it has been decommissioned
Remove lines 180 and 250 because they are shadowed by earlier lines and will never match a packet. Such lines can be found via the filterLineReachability question, as shown here.
Merge pairs of lines (190, 200), (210, 220), and (260, 270) by combining their prefixes into a less specific prefix.
Remove all deny statements on lines 520-870. They are not needed given the final deny on line 880.
The result of these actions, which halve the ACL size, is shown below. To enable easy observation of changes, we have preserved the line numbers.
End of explanation
# Import packages 
%run startup.py
bf = Session(host="localhost")
# Initialize a snapshot with the original ACL
original_snapshot = bf.init_snapshot_from_text(
    original_acl, 
    platform="cisco-nx", 
    snapshot_name="original", 
    overwrite=True)
# Initialize a snapshot with the compressed ACL
compressed_snapshot = bf.init_snapshot_from_text(
    compressed_acl, 
    platform="cisco-nx", 
    snapshot_name="compressed", 
    overwrite=True)
# Now, compare the two ACLs in the two snapshots
answer = bf.q.compareFilters().answer(snapshot=compressed_snapshot, reference_snapshot=original_snapshot)
show(answer.frame())
Explanation: The challenge for us is to find out if and how this compressed ACL differs from the original. That is, is there is traffic that is treated differently by the two ACLs, and if so, which lines are responsible for the difference.
This task is difficult to get right through manual reasoning alone, which is why we developed the compareFilters question in Batfish.
Comparing filters
We can compare the two ACLs above as follows. To initialize snapshots, we will use Batfish's init_snapshot_from_text function which creates a snapshot with a single device who configuration is the provided text. The analysis shown below can be done even when the filters are embedded within bigger device configurations.
End of explanation
smaller_acls = 
ip access-list deny-icmp-redirect
  10 deny icmp any any redirect
ip access-list permit-bfd
  20 permit udp 117.186.185.0/24 range 49152 65535 117.186.185.0/24 range 3784 3785
ip access-list permit-bgp-session
  40 permit tcp 11.36.216.170/32 11.36.216.169/32 eq bgp
  50 permit tcp 11.36.216.176/32 11.36.216.179/32 eq bgp
  60 permit tcp 204.150.33.175/32 204.150.33.83/32 eq bgp
  70 permit tcp 205.248.59.64/32 205.248.59.67/32 eq bgp
ip access-list acl-dns
  90 deny udp 10.10.10.42/32 218.8.104.58/32 eq domain
  100 permit udp 10.10.10.0/24 218.8.104.58/32 eq domain
ip access-list deny-untrusted-sources-group1
  110 deny ip 54.0.0.0/8 any
  120 deny ip 163.157.0.0/16 any
  130 deny ip 166.144.0.0/12 any
  140 deny ip 198.170.50.0/24 any
  150 deny ip 198.120.0.0/16 any
  160 deny ip 11.36.192.0/19 any
ip access-list deny-untrusted-sources-group2
  160 deny ip 11.36.192.0/20 any
  190 deny ip 218.66.56.0/23 any
  210 deny ip 218.67.71.0/23 any
  230 deny ip 218.67.96.0/22 any
  240 deny ip 8.89.120.0/22 any
  
ip access-list permit-trusted-sources
  260 permit ip 218.8.104.0/24 any
  280 permit ip 218.8.103.0/24 any
ip access-list deny-untrusted-sources-group3
  290 deny ip 144.49.45.40/32 any
  300 deny ip 163.255.18.63/32 any
  310 deny ip 202.45.130.141/32 any
  320 deny ip 212.26.132.18/32 any
  300 deny ip 218.111.16.132/32 any
  340 deny ip 218.246.165.90/32 any
  350 deny ip 29.228.179.210/32 any
  360 deny ip 194.181.135.214/32 any
  370 deny ip 10.64.90.249/32 any
  380 deny ip 207.70.46.217/32 any
  390 deny ip 219.185.241.117/32 any
  
ip access-list deny-untrusted-sources-group4
  400 deny ip 2.80.3.219/32 any
  410 deny ip 27.212.145.150/32 any
  420 deny ip 131.159.53.215/32 any
  430 deny ip 214.220.213.107/32 any
  440 deny ip 196.64.84.239/32 any
  450 deny ip 28.69.250.136/32 any
  460 deny ip 200.45.87.238/32 any
ip access-list acl-tail
  470 deny ip any 11.125.89.32/28
  510 permit icmp any any echo-reply
  880 deny ip any any
Explanation: The compareFilters question compares two filters and returns pairs of lines, one from each filter, that match the same flow(s) but treat them differently. If it reports no output, the filters are guaranteed to be identical. The analysis is exhaustive and considers all possible flows.
As we can see from the output above, our compressed ACL is not the same as the original one. In particular, line 210 of the compressed ACL will deny some flows that were being permitted by line 510 of the original; and line 510 of the compressed ACL will permit some flows that were being denied by line 220 of the original ACL. Because the permit statements correspond to ICMP traffic, we can tell that the traffic treated by the two filters is ICMP. To narrow learn specific source and destination IPs that are impacted, one may run the searchFilters question, as shown here. 
By looking at the output above, we can immediately understand the difference: 
The first line is showing that the compressed ACL is denying some traffic on line 210 (with index 16) that the original ACL was permitting via line 510, and the compressed ACL is permitting some traffic on line 510 that the original ACL was denying via line 220. 
It turns out that the address space merger we did for lines 210 and 220 in the original ACL, where we combined 218.67.72.0/24 and 218.67.71.0/24 into  218.67.71.0/23, was not correct. The other similar mergers of 218.66.57.0/24 and 218.66.56.0/24 into 218.66.56.0/23 and of 218.8.104.0/25 and 218.8.104.128/25 into 218.8.104.0/24 were correct.
The third line is showing that the compressed ACL is denying some traffic at the end of the ACL that the original ACL was permitting via line 80. This is an expected change of decommissioning the BGP session on line 80. 
It is not always the case that refactoring is semantics preserving. Where compareFilters helps is succinctly enumerating all differences. Engineers can look at the differences and decide if the refactored filter meets their intent.
Splitting ACLs
Compressing large ACLs is one type of refactoring engineers do; another one is splitting a large ACL into multiple smaller ACLs and composing them on the same device or spreading across multiple devices in the network. Smaller ACLs are easier to maintain and evolve. However, the split operation is risky. We may forget to include in the smaller ACLs some protections that exist in the original ACL. We show how such splits can be safely done using Batfish.
Suppose we want to split the compressed ACL above into multiple smaller ACLs that handle different concerns. So, we should have different ACLs for different types of traffic and different ACLs for different logical groups of nodes in the network. The result of such splitting is shown below. For ease of exposition, we have retained the line numbers from the original ACL and mimic a scenario in which all ACLs live on the same device.
End of explanation
# Initialize a snapshot with the smaller ACLs
smaller_snapshot = bf.init_snapshot_from_text(
    smaller_acls, 
    platform="cisco-nx", 
    snapshot_name="smaller", 
    overwrite=True)
# All untrusted subnets
untrusted_source_subnets = ["54.0.0.0/8", 
                            "163.157.0.0/16", 
                            "166.144.0.0/12", 
                            "198.170.50.0/24", 
                            "198.120.0.0/16", 
                            "11.36.192.0/19", 
                            "11.125.64.0/19", 
                            "218.66.56.0/24", 
                            "218.66.57.0/24", 
                            "218.67.71.0/23", 
                            "218.67.96.0/22", 
                            "8.89.120.0/22"
                           ]
for subnet in untrusted_source_subnets:
    # Find which ACLs match traffic from this source subnet
    answer = bf.q.findMatchingFilterLines(
        headers=HeaderConstraints(srcIps=subnet),
        filters="/deny-untrusted/").answer(snapshot=smaller_snapshot)
    # Each source subnet should match exactly one ACL
    af = answer.frame()
    if len(af) == 1:
        print("{} .... OK".format(subnet))
    elif len(af) == 0:
        print("{} .... ABSENT".format(subnet))
    else:
        print("{} .... Multiply present".format(subnet))
        show(af)
Explanation: Given the split ACLs above, one analysis may be to figure out if each untrusted source subnet was included in a smaller ACL. Otherwise, we have lost protection that was present in the original ACL. We can accomplish this analysis via the findMatchingFilterLines question, as shown below. 
Once we are satisfied with analysis of filters, for an end-to-end safety guarantee, we should also analyze if there are new flows that the network will allow (or disallow) after the change. Such an analysis can be done via the differentialReachability question, as shown here.
End of explanation
<END_TASK>