Okay, can you guys hear me okay? If you can, please let me know in the chat and let me know if the video quality is okay and it's all readable, etc. But we should be live. Okay, guys, can somebody say something in the channel if they can hear me? By the way, for those of you who are talking to me on WhatsApp if you can use the YouTube chat channel there it just makes it easier for everybody to see the questions, etc. Okay, great stuff. So today we are going to be talking about creating a BGP demon from scratch. So before I start writing code I want to give you guys a little bit of a feel for what BGP is. And BGP is essentially a rooting demon. It tells routers how to get to places on the internet so that traffic can flow correctly. So when I send data from myself to YouTube in this case my PC has to know how to get to YouTube. It will talk to a router. That router will then say, okay, I need to send traffic to the next point. That router will in turn go to the next point to the next point. And that's all in internet terms generally constructed using this protocol called BGP which exchange routes over the internet so that everything knows how to get wherever they're going. BGP is essentially, and I'm going to see if I can add something into the stream here just to see if I can share some slides I did earlier. And then I'll be able to give you a little bit more of a rundown. So let's see. Okay, this is a little bit about BGP and I'm only going to cover the very basics in these slides as I start because it will give you an introduction as to how I'm looking at this as I start to write code. So firstly there are certain things that we need to know about BGP. Number one, it's TCP based. Now the moment that you're writing something that is TCP based it means that you're going to have to hold state. Effectively when you're working with TCP a session is established and that session has to then be polled for new packets. Packets have to be read. You've got to know when that session goes down. Whereas if I was dealing with a UDP stream a UDP stream would be stateless and I just get packets as they come in and if they don't come in they don't come in. So that's very, very critical because when you start writing your code for BGP the difference between writing a stateful demon versus writing a non-stateful demon is there is quite a bit to it. The next thing to know is that BGP is essentially a pretty simple protocol in many ways and at the start of it you have four basic packet types that you're going to get with BGP. You've got open packet that's used for session establishment. You've got update packets. Update packets are used to basically send actual data over the network about routing information. Keep alive packets that basically is to ensure that you know that the session is going to stay up and it is up and things are still communicating so that you don't have a situation where your TCP session itself is up but there's no data flowing over it and the routers can't tell that something's gone wrong. Then you've got keep alive packets. Those are basically packets that are just exchanged to say listen I'm still alive, I'm still here and you've got notification packets. Notification packets are typically used for error handling and the rest of it and all of these packets start with the same header which is basically a 16 byte marker followed by a 2 byte length and a 1 byte packet type. This slide here that I took was taken out of the RFC itself. If you're looking at that RFC and you want to know what that looks like, this is RFC 4271. I will put that the URL for 4271 in the YouTube chat so that you can look this up yourselves if you want to. With any of these protocols you're effectively writing something against a spec and these specs are developed within the IETF. One other thing to note is that you'll notice that the spec was produced in January 2006. It's been expanded on, it's grown so as we develop code for this I also have to ensure that I'm developing for all the supplementary add-ons for BGP and there are a lot of them but we'll get to that as we go. Going back to that header that I was talking about, this one here. So every packet is going to start with 16 bytes. It's basically 128 bit header. All of those bytes have all their bits set which means that if you were to consider this in terms of a Golang slice you would basically see 16 elements in the slice of type U and A to a byte and they would all have a value of 0x ff in hex or 255 in decimal. One thing to note is you'll notice that as I work through this we're going to be talking a fair amount in hexadecimal. I prefer to work with hex when I'm working in protocols because if you look at Wireshark or any of the other things they typically work in hex so it's easier if when I'm looking at packet capture software I'm working in the same standard so hence the hexadecimal reference there. The next thing after that 16 byte is a length field that specifies the entire length of the packet and then you've got a type which is one of the four types I referred to. Now something to note about that length field so firstly it includes the entire size of the header and it's actually quite important to have that field because typically you'll notice when we get around to actually writing the socket code that what I do is I write the socket code such that it reads until I get that length and then I read the remainder of the packet so that I'm always sure that I'm reading on an aligned packet boundary rather than potentially reading too little or too much from a particular packet. I tend to say okay I know that I'm expecting at least 18 bytes which is going to contain my length as the last two bytes. Once I've got that grab the length and then tell it read the rest of the packet add it to my buffer so that I've got a full packet and then hand that off for processing. Obviously dependent on the type is going to depend on what type of processing that is. Then what we'll probably get to tonight I don't know how much further we'll get much beyond the open packet is how to establish a session in BGP. Now when you start the session the first thing that's going to happen is either you're going to have to send an open packet or the router you're connected to is going to have to send an open packet so someone sends an open packet. That open packet contains a variety of fields after that header that I've spoken about. It'll contain a version, an autonomous system number, autonomous system numbers in BGP terms effectively refer to what I've referred to as an administrative domain. So effectively every company that is talking BGP on the net will have an ASN or an autonomous system number and that's granted to them by their regional internet registry and the information that is sent out onto the internet is associated with that number. One thing to note here is that in this particular definition because we're working on the original BGP SPAC when they designed this they designed for autonomous system numbers to be 16 bit rather than 32 bit which means they're limited to 65,000 ASNs. That's not big enough so there is an expansion to BGP which we'll talk about when we start talking about capabilities and the rest of it allows you to introduce a 32 bit autonomous system number using what's called AS4 capability but we'll get to that in a while. Then after that you've got the whole time. The whole time is basically how long between packets at an absolute maximum before the demon is supposed to go listen I need to time something out because something's gone wrong with a session. Then you've got your BGP identifier. The identifier is basically a 32 bit number that identifies the router that is sending the information into the BGP hierarchy on the internet followed by an optional parameter length followed by optional parameters. Just one note there is only one optional parameter defined. It's parameter 2 and that refers to capabilities. Following that you've got your capabilities. One of the primary reasons why I wanted to talk about this before we start writing any code at all is that if you look at how capabilities are structured this is very typical to how a lot of protocol code is structured where you have what's effectively what's called a TLV type length value. The capability code in this case would represent type. The length is the length of the value and then you've got the value itself. So effectively every protocol that you'll see is comprised in their packets of a series of these TLVs which are then decoded and acted upon in each packet and we'll talk a little bit further about that in a second. Now one thing to note when we start in the actual coding is that in writing any protocol code you need to kind of start and look at it and go where am I going? What am I going to have to do here? How much state? How many elements of this? One of the easiest ways to kind of get to that point where you're planning for this is to start looking at how am I going to configure this demon to talk to things because that configuration can often give you a fair amount of information. So what we're going to start with in the code itself is we're going to start by looking at the configuration section of this. So the first thing that I'm going to do in my Golang window here is I'm going to create certain directories which are going to make up my sub-packages which I'm going to look at in the various sections of the code. The first one I'm going to make is a config directory. I also know that I'm going to have to have a session manager directory but it's actually going to be multiple managers so we'll call it multiple managers and I'm going to have to have something like capabilities. Capabilities is handling my open capabilities and we're also going to have a packet structure for handling packets. Config is where I want to focus first. So I'm going to make a file here called configdef.go. Now configdef.go is going to effectively contain how I'm going to define what my configuration file is going to look like. Now typically you could do configs in databases and the rest of it. For a lot of the stuff I prefer to effectively stick to a bunch of JSON. It makes it easiest to decode. In saying that, what do I need when I establish a BGP demon? So a BGP demon is typically works on talking to a series of neighbors. Neighbors are what you're connected to. So we're going to make something here. We're going to call it neighbor config struct. What do I need to know here? Firstly, my local IP. This local IP is what I'm going to be using as the source of the session. However, it's also an optional parameter. Now you'll notice when I'm creating these structures, I'm going to tag all of them with JSON tags to make it easier to un-martial them. Because local IP, technically I could use any IP on the box. It's an optional parameter and I can use a default to handle that as well. Then I've got my neighbor IP and we're going to call that neighbor. That is not an optional parameter, so I don't tag it with a mit-empty. Then we've got our pure ASN. Remember I spoke about autonomous system numbers. When a BGP session is established, systems effectively have autonomous systems that have to match in the config. Your ASN, the other router has to know about it. At the same time, you've got to know what their ASN is. If in the session establishment these don't match, things aren't going to work. I'm going to define this as a UN32. Keep in mind what I said that in the initial spec they used 16-bit, but because of AS for expansion they can be 32 rather use the larger value. This is a pure ASN. We've got the local ASN, which is what our ASN is. The difference between those two is effectively that your ASN versus their ASN and it's got to be configured to match. We're going to create a boolean here called shut. Basically I'm creating something so that I can have something in the config to have a neighbor that's configured but effectively shut down. We've got a keep alive time and the keep alive time is a UN16. We've got to know that from the spec as well. I'm just thinking other fields here that I'm going to need before we get to the more complicated one. For later purpose, and we probably won't cover this tonight, but I'm going to add another boolean in here called recursive resolve. Basically what this means is that when you're doing rooting your next hop has to be reachable to know where to send the traffic to when you're sending towards a particular prefix. With a recursive resolution, I'm basically saying the next hop that is coming in from my neighbor router may not be directly connected so I may need to look this up from elsewhere and we'll set that up. Then we're going to get to this next section that I'm going to specify is an array and we're going to call this. Now on the internet, you've got IPv4 and you've got IPv6. Those are address families. You've also got a whole bunch of other data types. Well, a whole bunch of other families that are used for different types of rooting a network interaction for things like L2 VPNs there's something called link state data which effectively gives you other types of rooting information encapsulated in BGP and in BGP terms when the information for these things are exchanged the base BGP spec deals only with the IPv4 address family. However, using something called multi protocol BGP you can have multiple address families in the same session. So one TCP session that's effectively exchanging information about multiple different address family types on the internet. By the way, if anybody does have any questions I am keeping an eye on the chat and interaction and questions are always appreciated because it knows that I haven't completely lost everybody. Families as you'll also see there is an array of struct pointers that I've defined. So I now need to create that family structure and then we have below this something called a sub-address family because address families can have sub-address families. Now this is really, really, really important to note because when you're constructing all of this and you have to deal with packets and you have to manage these packets coming in what you'll find is that you've effectively got to manage to send packets to the correct address families as you're going and they need to be managed separately. So your managers need to kind of almost handle state for each address family separately. It's not really state but it's kind of where do I send the stuff and how do I distribute this data. So effectively you're creating something that is fairly hierarchical in nature here. We just finished defining this. So we've now got effectively our config that we're going to define and we'll then because these are multiple neighbors we'll create another one called bgpconfig and we'll go neighbors neighbor config. Now the thing is I'm defining a default source AS in here so that I've got defaults if somebody doesn't specify a local AS in. I can have one AS in for the entire router and I can just pull that in. So I now know from this kind of what my config is going to look like. So the next thing that I'm going to do here is I'm going to go and create a JSON file and I'm going to split this right and effectively I'm going to construct the config file. So you'll notice what I'm doing here is effectively constructing a config before I write any serious code because when I start this demon how I interact etc. is going to be fairly related to this config file that I'm going to be using and I can structure my code around some of this. So effectively we then go and we know that from our top level which is the BGP config this will be neighbors and sorry wrong way around. Writing manual JSON is not something that I've always been particularly great at but let's give it a go. And here we're going to define one neighbor and we're going to say that the neighbors got a local IP and in this particular case we're going to make this 10-20-21 as just as a random example we're going to have a neighbor IP and we'll make our neighbor 10-30-31 these are just random IP addresses just for the purposes of examples. Excuse my chronic typing tonight. We're going to say that we've got a now you'll notice that the field names here that I'm using over here are matching what is in my struct tags because that's how it's going to un-mortul it, it's going to match those up and we'll give it a pure ASN here of 1, 2, 3, 4, 5 and we'll say that our local ASN is going to be defined as 5, 4, 3, 2, 1 and we're going to set neighbor shutdown to false and we'll set a keep alive time of 30 seconds BGPSpec says that minimum keep alive time has to be 3 seconds I think, we'll go with 30 it's that's fine and then we'll set recursive resolution to true and then we get to the families now families is obviously going to form another array of structures and that will look like this before this family now one of the things you'll see with what I'm about to do is some of the stuff is going to look almost a little bit some of it's going to look a little bit strange but I'll explain it as we go here I just want to fix my JSON here this will be going up here let's just see have I missed anything here actually prefer to kind of collapse some of these brackets a little bit so that I can make it more easy to see kind of what I'm doing what's it wanting about my syntax here one of the things about manually editing JSON is that IDEs for this are wonderful because you get to see all your errors there and close that off and effectively I have a config file those of you who are currently watching and I see that there are 15 people on the stream anybody got any questions about anything that I've said so far that I can address please drop them in the YouTube chat and I can address them before we kind of continue so Fred asked about the daemon that I hard coded into the configs JSON so Fred effectively what the config does is simply set up what this daemon is going to be connecting to and the parameters it's going to be using once it connects to the router use everything that I've coded into that config file to then take that information negotiate a session and then get the dynamic information from the session itself but every BGP session does have certain configured parameters which is what's hard coded in this JSON file it defines what it's connecting to when it's connected how it's going to connect and how it's going to interact and the parameters for the session so if you were to log into a router and configure a BGP session these are kind of some of the parameters you would have to tell the router about hope that answers your question let me know if it doesn't or you need further clarification then so I've now got this config that I know about now keep in mind remember I said to you that because BGP is TCP based there's going to be a lot of state involved the easiest way to handle state now in BGP they have what they refer to as a finite state machine and the state machine basically handles state on a session what I typically like to do here is create a manager effectively to handle all of this so we're going to make a file we're going to call it manager.go and it's in our package managers and we're going to create a structure that we're going to start tying everything to and number neighbor how many neighbors have we got you could probably pick this up dynamically later when you need to have that information simply by doing a length on the neighbors or etc but one of the reasons why I tend to actually store this information in a particular variable here is because if I ever need to take a slice from go and hand it off to CGO I'm going to be handing off to CGO and the C content simply a pointer to the start of an array and I'm going to have to be able to tell see what the length of that is and it's nice and easy just to have it sitting there in a variable rather than having to run length commands etc then I also want to know how many of these neighbors are active I want to know how many are inactive I want to know the neighbors themselves and this is going to be a separate structure here which we'll create just now I also want to put a mutex on this because the thing is a lot of what I'm going to be doing here has got a fair amount of threading involved in it I don't want one go routine or thread to be screwing with my manager, slices etc while another routine is so I'm going to need to lock those threads to make sure that I don't have cross talk and race conditions then I also have a listener which is a net and the listener is basically what's going to listen for new connections now one of the things here is that in creating this I have this structure here called neighbor now my neighbor structure is not the same thing as my config definition structure which is over here because the actual processing of neighbors is slightly different to how I'm going to handle the neighbor in code terms so this is just a config structure but in neighbor here we're going to have a completely separate neighbor structure that we're going to work with so we're going to go and create another file here and the other thing about the neighbors is you'll see that one of the other reasons why I start tying this to a completely separate structure is that I'm going to be creating a bunch of methods to that to handle each neighbor almost separately so the first thing is create this structure now I'm going to split and move right so that you can see it's coming from so that's the structure that I'm going to be creating I've got my peer which is my neighbor effectively I've got my local IP which is going to be a string my neighbor ASN my source ASN keep alive time so a lot of this we'll see kind of matches what's in that config but at the same time there's a bunch of other stuff which I either didn't define in that config which I'll set by default maybe add it to the config later and then there's also other things here which are more related to the internal functioning of the demon state is what we're going to use to do what's called collision avoidance where effectively if I've tried to establish a session with a router and it's tried to establish a session with me at the same time I'm going to use this state to go listen wait a sec there's a problem here because I'm effectively going to end up creating two sessions if I don't and effectively you create almost a network race condition if you don't watch that state I also want to know what the established time of the session is to know the last keep alive that was sent that's useful for when I start writing my keep alive sender routines then we've got the received capabilities now I'll talk a bit about this in a second add some additional things here I'll use a net.con I may change that and again this thing has got its own mutex for locking when I'm playing with the neighbor and now one of the things you'll notice here is that the manager has an array of neighbors structures two neighbors and those are effectively going to be the list of all of our neighbors the thing is because I'm going to be tying so many methods to the neighbor and I may need to access information in the manager itself one of the elements that I include in every neighbor is a pointer back to the original manager so there's a pointer to the manager and that effectively creates my capability my structure that I'm going to be using for neighbors the next thing that I need to talk about is these capabilities because if you look at originally in the config that we defined you've got these address families for example and in BGP terms these things are exchanged as capabilities when the session is established now if a router says to me I support IPv4 and IPv6 and I say to it I only support IPv4 you end up with a negotiated capability because you can only support v4 because both sides don't support v6 so what I do here is I've got a received capabilities and this is what I'm going to be using to store what my peer is sending me you've got a configured capabilities that's what I'm going to be sending out and negotiated is when I compare these two and figure out what I can actually do so any other questions so far on the chat before I start talking about how we're going to handle these capabilities let's talk about capabilities now remember something in a packet if we look at and I will pull this up again quickly the power point so in this power point every packet is simply a series of bytes imagine a normal Golang slice where you've effectively if I have to define a slice something to this effect which I'll take out now var, blah urn 8 0, 1, 2, 3, 4, 5 effectively it's a slice of bytes and that's effectively how packets are structured which will look something like let me just pull that up something like this var, blah urn 8 so it's a bunch of 8 bit numbers in a slice and every packet can be viewed like that and those packets are given to you in a particular order they give them and there's actually structure to them that you can apply so if we look at what I had in this power point we know that the capabilities that are going to be received are going to be in the form of code length and value like that what this means is that while the capability value is variable the way that I process each capability to read it and because I've got to be able to send capabilities out on my own packets can be done in a pretty uniform manner so what I'll do here is in my code that front is I'm going to create a file here called capabilities.go which is going to handle my capabilities now I haven't defined anything to do with the open packet but what I'm working from is what have I got in my other code so I'm effectively working on this thing so that I'm filling in my structures as I go with these various parts and then I'll get to the upper level packets so at each point I'm making sure that I don't get too far ahead of myself and I know that there are certain capabilities that I'm going to have to deal with when I'm creating these neighbors so with regards to capabilities the capabilities themselves I just want to open something here every capability is going to be different because again if you look at that PowerPoint that I had up a little bit earlier what you'll have noticed let me see if I can bring it up again here is that the capability value is variable and it's also got a variable length that's because different capabilities do different things they've got different properties etc but that being said with the capabilities as I said you can kind of treat these in a fairly standard way so what we're going to do in Goland here is we're going to create an interface type and effectively we're going to say that every capability structure that we create is going to have to have certain methods that are bound to it in every case so we're going to make a type method which is mandatory it's a function and it's going to return urn8 we're going to have a length message which is a urn8 we'll have a value and because that value is kind of dynamic we're going to return it as an interface type so that we can use dynamic structures here different values for different capabilities we're going to put a Marshall in here and that's going to return a byte or an error Marshall is basically what we use when we need to take a capability and turn it back into a byte array then we're going to put a resolve name in there and that's simply so that if we want to know what this capability is in human words instead of just a capability code there's a function that's going to give me back the name of it and we're going to put an original in here which gives me back a byte now the original is so that if I've got a capability and I'm just adding it to another packet and I have to actually process anything in there because it doesn't really matter I just want to get the original that's a quick way to do that so that creates a capability interface and I just want to make sure that I make sure that this thing is actually I have, okay there we go so I just go in here to fix my typo here and it clears now because that's an interface in that way these structures can actually be pretty much anything and I know that there are certain capabilities that I'm going to need right from the start to handle my own open packets in particular as I said earlier the original spec referred to 16 bit autonomous system numbers a lot of the internet nowadays uses 32 bit autonomous numbers so I'm going to need to be able to tell routers that I support 32 bit ASNs and that they can send them to me so I'm going to need that capability so what I typically do here is for each capability I actually make a completely separate file and the first thing that we're going to do is define what all of these capabilities I'm going to support are and for this effectively we're going to use a bunch of constants which I pre-define even if I'm not going to use them right away so that I've got the full list the capability codes that I'm about to put in here are defined and allocated by a group called IANA and if you search on BGP capability codes on Google you'll find a lot of this I'm going to create constants for the ones that I'm planning on actually supporting within my BGP daemon so capability codes are always 0 to 255 so it's a UND8 so we'll make the constants UND8 as well and we've got the naming style here is a little bit this is kind of how it comes down to personal taste I prefer to have kind of longer variable names for my constants that actually explain what the damn constant does so that when you read the code you know what you're looking at and that's why each of these is ended in cap code descriptive name to it BGP security we've got multi label cap code this is used for MPLS based labeling on BGP packets we've got a roll a roll code that is code 9 we've got a graceful restart cap code and that is type 64 we've got an AS4 cap code which is type 65 ability cap code that is type 67 and that's type 68 almost done you'll find that some of the time when you're typing out files like this it does get a little bit painful and you kind of hope that you type fast because you can have some of these capability lists and other things can get to really long lists of constants the other thing that I will say is kind of writing protocol code it's very easy to fall into the trap of just using a number in the code I would say always use a constant if it's a static number because it makes your code so much easier to read when you have to revisit it and come back and go what is this thing doing I've got a whole bunch of capability codes there those are the capabilities that I know that I'm going to support the next thing is what are the capabilities that I know for a fact that I'm going to need at the start I'm going to start with that AS4 capability because I've referred to it multiple times I'm going to define a file for it and the first thing that I'm going to do is define a structure for the AS4 capability call it AS4 cap and this thing has a type remember I said all capabilities are defined on type length value so it's got a length, it's got a value in this particular case I know that AS4 capabilities value if I go and dig up the AS4 RFC is going to be a 32 bit number which is going to be that 32 bit ASN that we're going to insert into the packet and we've got the original for when we need to actually grab this from an inbound packet so I've got this nice structure to hold these capability codes next thing I want to do is create a function to create a new capability code and this will just return a pointer to that with everything filled in here type is AS4 now this type is going to cross reference back to that capability codes file which I created earlier so you'll see that that constant is now actively used I also know that in this particular case it's nice and easy because I'm working with a UN32 I know that that's 4 bytes long so my length here is nice and static there are certain capabilities which you'll see just now where the length value is far less static then we've got the value and that's set to ASN which is the input parameter when I'm creating this and the ORIG slice because I'm creating a new one while there is no original so it doesn't really matter you can just leave that out so that'll create my new capability one of the things that I did when I set up the capabilities file is that there were a whole ton of methods that were associated to every capability struct these are mandatory functions that I have to attach to the structure to make it work so we'll run through them in order so the first one would be pipe and this will simply return c.tip and then we've got our length which is what I find useful to do in creating stuff like this is very often I'll create a template file so that every capability I can kind of copy the template and just modify the template in this case just to demonstrate what I'm doing and doing this manually I've got a value and that now this is where things get a little interesting this martial function which has got to turn this whole thing back into a byte array so if I look at this when I'm going to take this thing back into a byte array there are a couple of things to know about network protocols number one in what's called big indian format so big indian format effectively means that the way the computer thinks about numbers that are larger than one byte is almost in reverse so I'm trying to think of the best way to explain this but effectively if you've got a number of let's say 65534 I'm just opening a calculator to check this quickly that I'm not talking complete crap so 65534 in byte terms would normally be FFFE in hexadecimal so two bytes made up of fully set and then unset however what the computer will do in little indian format which is what most processors handle is that that byte order of the number is reversed so that means that if I've got a value for my ASN that represents a particular number I've got to reverse the byte order on that to turn it into big indian because network data is in big indian format and I need to actually change it so that the value that my CPU on my normal PC handles versus what's on the network are in effect reversed in terms of byte order luckily Go has some functions for that there are also a couple of other things when you think about marshalling a struct into a byte you can use a bunch of Go's helper functions so what I'm going to do to this is make a new directory and we're going to call this helpers helpers is where I'm going to put a bunch of functions that I'm going to use throughout my processing to kind of handle bits and pieces of things that aren't related to BGP itself and we're going to call this I don't know encode.go will probably put our decode functions in the same helper in fact let's rename this and we're going to call this data helpers.go just so that it's cleaner now in data helpers basically what I want to do is create a way that I can effectively stick a bunch of data into a byte array using a single method to do it in the same way so what am I going to do here so the first thing that I'm going to do is I'm going to create something that I'm going to call right field and this has got a rather strange function definition and I'll explain why in a second let me close this on the other window so that you get a better view of it so what right field is going to do is it's going to take data and it's going to push it back into an output byte slice now here's the thing I want to be able to return an error I'm not returning a new byte slice I want to modify a byte slice in place that means that I've got to pass that byte slice in as a pointer that's going to get a little tricky in a second but it is what it is and I'll explain it as I go the other thing is that when I write a field to a byte slice I want to know now that I've expanded that byte slice where am I sitting after that field so that I know where to write the next one again I don't want to have to start returning things so what I'm going to do is I'm going to take an offset into this function that is a pointer that I can modify so that as I encode stuff and add it to this byte slice I can change offset which is a pointer and hence will be passed in and out and keep track of the offset of where I am in that byte slice for those of you on the chat any questions does this make sense so far I'm getting amazing silence on the chat so either I've lost everybody or everybody is quite happy with everything that I've said I'm hoping that it's kind of the latter but please guys if you've got questions, interactions, comments need me to slow down anything please let me know because it's always easier if you feel like you've got an interactive audience rather than dead silence so please come with the comments anyway this right field function so I'm going to use Golang structure here call it byte buffer I'm going to define an error so that I've got something to return there and then remember I referred to bigendian and littleendian and for this go is quite nice because it has this function called binary write and I can use the buffer as an output for that bytes buffer I can specify the endianness that's the byte ordering Golang will use that to kind of go and reverse the byte order as is necessary for the input field and it also takes binary write takes an interface type so it'll figure out what type of variable that was on the input now errors.wrap I typically for this use a package here github.com package errors it's a decent kind of error handling package just sync that quickly so it'll go and pull that package and that just basically kind of wraps my error nicely so that I can add a message etc without using format.errorf etc so this writes my value for my input variable into this byte buffer array but now I need to put it into my output array so if the length of the output array now notice because I passed it in as a pointer to get the length I've got to dereference that pointer so hence the the asterix there and I'm effectively now checking is this array here going to be long enough to handle what I'm about to put into it we're going to create a new byte array if it's not big enough copy the data into that and put the data back in so I just want to check something here because I've got an error here somewhere sorry this should be if len so effectively say is my output buffer big enough if not I'm going to make a new one and I'm effectively going to create copy my original data into this new one effectively reconstruct the new one and you'll notice that this star out equals temp effectively what this is going to do is effectively reset this to an entirely new slice if I need to but with the original data because I've copied the data in there then now that I know that that slice is going to be long enough I can then at the offset which I've specified copy the encoded data in there and increase my offset so that my offset always returns incremented by how much data I've added to the buffer and I then want to return sorry that's got to be a slice so this function is basically going to allow me to pretty much encode any value just close that into a bite slice but I don't want to have to do this kind of one value at a time so what I'm going to do is I'm going to add another function into my helpers and we're going to call this one encode and again this is going to take a bite slice it's going to take the offset that we're going to start at and this time I'm going to take an array of interfaces and it's going to return an error as well we'll define that error over here and we'll go full field range fields so effectively I'm going to range through the list of fields that I'm getting given and I'm going to say right field so we're using the field that I just used we'll pause the same pointer from here into that function we'll pause that offset which is again it was already a pointer coming into my encode function so it'll be a pointer when I pause it on and we'll take that if error is not equal to null error is dot wrap error unable to write target field to buffer and return null so effectively encode just wraps this right field so that I can handle multiple fields at the same time now with these helper functions written go back to my as4 capability here and in this Marshall function that I'm writing to turn an as4 capability this is a particularly simple kind of capability that I'm going to Marshall here they do get a lot more complicated but effectively I can I know in this particular case that my capability is effectively a 6 byte byte array it's got a type it's got a code and it's got a 4 byte value so I can make this byte array with 6 bytes if I'd made this as a 0 byte empty array then my encode helper function would have expanded that array as required but don't need to do that because if I do that there's all sorts of copying going on etc similar to a normal append and you don't really want to have to do that so rather make the slice the correct length as you need to from the start then I want an error variable just to kind of populate coming back errors and we know that we're going to be starting the offset of our byte array at byte 0 because I'm processing the whole capability now here's where the helper function comes in because effectively all that I have to do now is put the various fields in my array into a slice of interface types and call if error equals helpers.encode so pointer to my byte slice over here a pointer to my offset which is going to be incremented by this thing and my field of interfaces test it to make sure that it's null and return the resulting byte array with a null value if that doesn't succeed I can return an error actually that will now take these fields encode them and shove them into the byte array any questions before I add the rest of the kind of functions from my capabilities just a code infinity I see you on whatsapp saying you're trying to keep up that implies to me that you have questions let this so I'm not seeing any questions but so let me continue for now but effectively I've now got my Marshall over here and I've got two more functions that I'm going to need now what you may have noticed is that the moment that I added this original function here in my IDE you see that I've suddenly got self-marked what this is effectively telling me is that this structure defined over here now has all the methods needed to fit into this interface type so AS4CAP can now be used as a capability interface because it's got all of those various functions kind of tied to it and that'll create my AS4 capability code which I really do need when I'm going to be reading my config the other capability because I'm dealing in multiple address families which you'll remember from the config that I created that I'm going to need is to be able to say that I'm using multi-protocol BGP so I'm going to need a multi-protocol BGP capability here as well we'll call this mppgpcapability.go now if you remember what I said earlier about using a kind of cut and paste template what I'm going to do is take this whole AS4 capability and put all the kind of sub-functions in it and I'm going to copy it in here and I'm going to start kind of refactoring bits and well actually just renaming stuff I don't want to actual refactor because I'll break things in the other go file but we'll call this mppgpcap now there's a couple of things about the mppgp capability and I said to you earlier with capabilities they've got different values so they're fairly different structures in some case in the case of a multi-protocol BGP capability I've got a type and a length the same as the others I've also got what's called my address family number which is a UN16 I've got a reserved byte which is a UN8 I've got a sub-address family which is a UN8 as well and so I know looking at this capability for mppgp that I've got a couple of things going on there that will eventually end up making this also a four byte length because it's got an AFI and it's got which is a UN16 plus a reserved plus a SAFI that tells me my length automatically but different variables then I can simply modify this to call this change the input parameters here change the return type my type becomes capability I know again that my length is four because I've got AFI reserved and SAFI and I do not have a vowel here because that's not part of the structure at the moment so I fill this in reserved byte is always zero in this particular case and fill that in so that will return a new capability structure all filled in with my parameters then this I just have to change these over here which means I'm not having to rewrite my functions I'm just basically using the pattern paste then we get to this value particular proposition now value here is going to return an interface type when I call a value and I want to know what's in this what do I actually want to return to know the value and in reality I want to know the AFI and the SAFI from the capability that I've effectively un-martialed or read so what I can do here is actually just return the whole thing because I'll already be decoded but it's probably the easiest so I can just return C which is fine now again when I want to marshal this thing to turn it into a byte array type length all I have to do here is go and add the various fields this particular case make sure that I change this C reserved C dot SAFI change the resolved function change this over here and effectively what you'll have noticed is that I created the MPBGP capability that I've just created here in an absolute fraction of the time that I did the first one because all that I'm now doing in this is applying the same methods that I've used for another capability and adjusting as necessary and I have that helper function that I created that allows me to easily encode this so I can go and add capabilities until I'm blue in the face so long as I know the format and what is in them nice and easy there is one other capability that we need before we can start kind of building conflicts and this one is slightly more complicated this is called your ADPATH capability now in BGP terms effectively if you've got multiple BGP neighbors you can get routes from multiple neighbors and with ADPATH instead of sending just the most preferred route you'll send them all on so everything that you've received it also means that you can get multiple routes from one neighbor this is a special capability it's called the ADPATH capability and this one is slightly more complicated in terms of what we're going to create here as you'll see in a second so an ADPATH capability has an AFI it has a staffee it has a send and receive it has a single byte and exactly as per the other ones we're going to put a type, we're going to put a length we're going to put a value except in this particular case the value is going to be an array of these things because you can have ADPATH for different address family combinations in some cases it's there in some cases it's not and an array just to store the original in case we need it we've got first where we're going to return a in this particular case what I'm going to do here is I'm going to return an empty ADPATH capability so effectively all that I'm going to put in this particular function is to return the capability with a length of 0 and I'm going to add a function just now that will effectively allow me to go and modify that but the first thing that I want to do is go and restore the functions that are absolutely mandatory because of my interface typing so I'll paste those from the AS4 and just modify this as I said normally I have a kind of predefined template file that's got things that I can just search and replace on when I'm doing some of these functions because it makes it a lot kind of easier so the next function that I want to make here is the Marshall for this and again because we've got multiple parts to this where you've got kind of embedded almost embedded arrays this one is slightly different from what we've done in the other Marshalls which were relatively short functions again create an error and offset which we're going to need for the functions and in this case what we know about when we're marshalling this is that for each structure that I'm kind of each combination of affi and safi which is your address type and your sub-adress type I know that I'm going to have effectively a number of static lengths that I can multiply out but because I'm just marshalling this I've got the length value already so I can make this as a vorage colon equal sorry equals I know that when I'm marshalling this thing together into a byte structure it's going to be my length plus 2 the plus 2 being the actual length byte and the type byte so nice and simple there then the first thing that I do is create a fields over here so fields colon equals now the thing about the fields part of this is that the top of my capability is going to contain the type and the length after that things get a little bit more complicated so what I'm going to do here is I'm going to encode these 2 fields first separately then once I've encoded those I'm going to step through the value on this thing which in this particular case it's called VAL and I'm going to reset this fields array to say aphisafi.afi aphisafi.safi aphisafi.send receive and then I simply call the encode again and the thing is because of what I said earlier about these pointers and the way that this encode function increments that offset every time I do this what you're looking at here is effectively I'm always going to be pushing to the end of this byte slice because offset is going to increment every time I call this encode function because of the way that it's set up as a pointer and if it's not null I can return null or my error then the only other thing I need to do here is add my again remember I've got to add all of those functions that are specified in the interface type definitions the good thing about that is that you don't end up taking shortcuts and skipping things it actually forces you to do what you're supposed to do and actually add everything and not kind of skip out on things now if you'll remember I've actually kind of set this whole thing up my new capability function which I created here the new add path capability returned a blank capability the thing is I actually need more than that because I've now got to actually start adding the address families that I need so I'm going to add an extra function to this an extra method to this particular structure to handle that and we're going to call this add affi-saffi and the first thing that we're going to do with this is make sure that we're not adding something that's already added just kind of as a safety check because I don't want a capability that says this is unicost IPv4 three times over it but I need to do that once so we'll just put a quick check in here basically says if this thing already exists in here bail otherwise I can just append a man sorry just stick in a new one at the end of it now I reset the length over here and if you look at the length I know that this structure that I created up here contains a 2 byte affi, 1 byte safi, 1 byte send receive that's 4 bytes so every affi-saffi that I add here is going to be 4 bytes so I can simply take this and calculate that length by multiplying it out the number of affi-saffi combinations plant 4 bytes gives me my length done and dusted and so that finalizes my ad-path capabilities now why did I need to kind of create all of that so the first thing that I'm going to show you the next section is when I want to create a new neighbor right from the conflicts over here under my managers in neighbors.go I've got this whole neighbor structure and what I'm going to do is I'm going to tie something to this whole config neighbor and this is going to take in my config structure which I defined earlier earlier on in the evening and this is now going to actually create a full neighbor configuration from the config that we read out of that JSON file so because I'm going to be screwing around with a particular neighbor that may have other things going on in other threads the first thing that I'm going to do is effectively lock this thing to say listen nobody else can touch this until I'm done I'm going to also put a defer in there to unlock this the moment this function exits. Then we've got a whole bunch of stock standard sets to copy bits and pieces of data over which will be local IP JSON and this is just copying some of the more basic functionalities over then we need to start creating capabilities because remember the capabilities I'm going to be using in protocol terms versus what's in my config are they're the same thing but they're represented very differently so what I'm going to do here is I'm going to handle purely on my configured capabilities so my configured capabilities we know that we're going to have an AS4 capabilities and we put our local ASN in there so that's going to add a capability to say listen we do AS4 here's our ASN and it'll make that capability for us we'll also do an if-inconf shutdown end or state is shutdown state now I haven't defined that capability yet that constant yet but with neighbors as I said there are multiple states a neighbor can be in so it's defined a bunch of constants we'll have an established state so in my typing typing is going to hell as I get tired we'll have an establishing state which is not the same as established it means it's in the process of creating we've got an active state which basically means listen I'm attempting to connect but I haven't even got a socket yet we've got an idle state so this thing is basically waiting for something to happen we've got a teardown state which basically means it's in the process of being removed and we've got a shutdown state and these numbers that I've used here are entirely whatever you can use whatever number etc that you want there they're just the lineators in the code for the various states so after I've got that constant if it's configured a shutdown we're going to set that as a shutdown state then just find something that I just want to make 100% sure if it's not shut down the other thing is we want to set this thing so that it's idle because I effectively until I actually tell this thing to start when I've configured the whole theme and as it comes up I don't want everything to start trying to kind of establish before I've initiated the rest of the code then we start getting into the whole issue of those families now remember that when I configured this thing in my config desks let me split that right the configs have got a series of kind of arrays of families and because in my config I'm using strings to define these families whereas in protocol code it's all numeric so I need to be able to resolve what's in my config to a bunch of numbers effectively so the way that I always find fairly useful to do something like this is that I create a new package here which will have other things in it later but for now we'll call it families.go sorry families and families.go and the first thing that we're going to do in those kind of families is define a text that is going to in effect map from the config to real values and this is going to be a bunch of constants so we've got ipv4, afi we've got ipv6, afi l2vpn, afi linkstate, afi these are all the address families our sub-adress families I'm just trying to remember if I'm forgetting any here one of the things just as a note when you're kind of typing this stuff make sure that you don't make typos in your numbers because if you do you can be chasing that error for ever in a day because it's an easy thing once you've got a constant and you've kind of mistyped a number to not know kind of where you screwed up so those are a bunch of constants then what I'm going to do here is I'm going to create a simple function here that I'm going to call resolve afi and I'm going to take in the string that we get from the config and I'm going to return a un16 and I just want to figure out there we go I knew I had a typo in there somewhere and now basically what is going to be in my config file and I'll return the correct number this will also allow me by the way to detect whether or not I've got a config issue because I've got a config issue this is going to return 0 and there is no such protocol code as 0 so if I'm getting a 0 back here I'm going to know straight away that something's wrong I'm also going to have a function here called valid afi safi and I'm just going to take both into it and this is going to have a dual return of a un16 and a un8 that's because your address family is always a un16 your sub address family is always a un8 let me just bring this across here quickly and this is effectively going to be a double switch again we've got pretty much what we've got above and in fact I can actually just copy this down to make it easier then I can just modify it so paste that and this time I also want to switch on the sub family names this part of the coding is always kind of a little tedious but once it's done having stuff like this will save you such a lot of time later so you kind of set this stuff up and sure it might take you a couple of minutes to type out the switch statements but save you time later the nice thing here is that because my IPv6 and my IPv4 can match each other pretty damn closely I can simply paste that in and modify this and with the link state stuff and keep in mind that at this point what I'm effectively doing is setting up to support things that I may not be supporting right away but my config will support this stuff so I'm building code that is very very expandable here long before I even touch a single network packet everything here is set up so that I can process well every packet that BGP could throw at me and that's going to make my life a lot easier as we go down the road so rather than kind of saying I want to take this just as it comes I'm working to overwatching holistic structure effectively as you can see there is rather a lot to BGP that people don't even kind of think about but I think that's almost all of it and so now I've got this function which is going to give me my saffi and affi numbers from my config and that's all done going back to my config though the reason that I just created those functions is because again keep in mind that what's in my config now needs to be turned into what is going to actually end up in my configured neighbor structure so now with that I can do something like this this is another function I'm going to need to create which will basically make sure that whether or not I've actually got a capability already am I adding to it, what am I doing the find configured capability function is just a really simple function that I'm going to throw in the bottom here quickly just to clear that error and again because these are all methods of a structure it becomes really easy because you're accessing effectively the data for that particular neighbor and you want to know if these things are already configured so we'll just go through what's already configured there and if c.type matter what the capability structure was because it said interface it will always have a type method so I can simply call the type methods to get its code and if it's the matching one that I'm looking for shove it into the array that I'm going to return and if I have any results I'll and in fact you don't even need that because I can return an array which is fine so that'll just go and find my configured capability to make sure that I've actually got such capability now I can simply do so effectively here what I'm doing is I'm iterating through the config basically looking to see do I have a multi protocol capability code and give me the array of the multi protocol BGP codes that I do have and then I'm going to iterate through the subfamilies and I can now do an affi number remember I did that whole validate to kind of the config names against the numeric value so if I call it like this subfamily name I can then say basically if that's not a valid name in the config run away skip it out don't care anymore I check whether or not I actually have any capabilities already just as a safety sake and if I don't I can then go and add a new one capabilities.new and affi number and that'll throw in a new capability if that's not null but basically at this point what I'm saying is if I don't have any capabilities I can just go and add I don't need to check whether or not it was already there if that wasn't null or there was something there I now need to just check that make sure that I'm not going to go and kind of re-add something that was already there now this is where things get a little bit interesting because I'm ranging through this array of configured capabilities and those are interface types remember even though they have a type but I now need to do a comparison against a particular element in that interface structure so I have to assert it to the correct interface type which is what I've just typed up there and the same thing for the sub-address family capability.safi safi number located equals true basically tell it that it's been located so I have something and if I didn't find it because keep in mind if I did find it I don't care I'm just basically going to continue as if nothing I've already got this so I don't really care about re-adding it then I can throw it in there if it wasn't located and then to make things even more complicated because when I'm creating this the add path and the send path that I researched to earlier is configured such that it's different for each address and sub-family configuration so when I create this neighbor config staff I need to add that in the same place that I'm dealing with creating this configuration for the different address families so what I'm going to do here is add another loop here I just want to check I think I've actually left something out here in my neighbor config that's why I hadn't actually defined my send and add path stuff in my config the way that I should have so I'm just going to throw those in here quickly there we go space is out nicely and make it pretty go back here so basically check to see whether or not these things are set and if they are we're going to need to add the correct add path capabilities so we'll add that create a variable for it and we can then do a find to check whether or not it's ill basically I'm going to go and check whether or not I already have this capability again if I've got the capability now the thing about capability codes in particular this one is that while in certain cases you can have multiple of the same capability you can only have one kind of add path capability because it contains an array under it so in this particular case because my find configured capability returns effectively an array what I can then do is to simply say apc0 and do an assertion all in one go capabilities.newadd pathcap and we'll add this capability and then once I've effectively added that capability I need to set up those internal arrays into this and that is done by simply running the functions that I originally set up to do this I'm just trying to check why I had a bit of an error here that variable was basically within the wrong scope so that's a relatively easy one to fix then this is simply add path and that's affi number safi number 3 this is actually quite interesting because what it's saying to me is it's effectively possible because of where I created the add path over here so I've got this for add path and it's not in scope of anything now what that means is potentially my IDE is saying to me this structure may end up null so it's throwing me a warning over here it'll let me compile this but it's claiming it could be new it's actually wrong because there is no there is actually sorry but it's because this over here is in the wrong scope so now you'll see that warning is cleared because I know that add path is going to be set there or it's going to be set there if it gets to this point in the code so I'm not going to potentially end up with a a null variable that I'm going to call that's going to effectively when I call the method crash the whole code because that would well suck add path we can then add path affi number, safi number and you'll notice that what I'm doing here is changing the send and receive such that if it's set in one direction for add or receive if it's set in both add and receive then I use code 3 if I'm sending it such that I can effectively only send then it's 2 if I'm only receiving it'll be 1 and that sets it up as 1 and from there I can say continue and this next piece of code will simply say if I have a and I'll actually set that up in a second because let me just check something you'll see that effectively when I've gone through this whole affi-safi process I've looked to see that I'm creating an mpbgp here now effectively what I want to say is so over here I've got this located variable that says I located mpbgp and once I've I'm just occasionally I've got to kind of stop to think for a second okay so new affi equals 2 basically if I'm going to create a new address family in fact I don't even need that because I've already appended it over here I don't actually need that at all and then I just want to make sure that I've gone through everything over here and I could return all at that point and effectively that will set up my whole config so I've set up my families, I've set up my address path, I've set up my AS4 capabilities and I've created this neighbor that's currently in idle and shut down state this now will leave me in a situation that when I start I can read all my config files in and oh hello Isaac I just saw someone on the YouTube chat going hey so how's it, if you've got any questions feel free to ask once I've got all of this config read in I can now and only now am I actually going to start looking at what am I doing on the network itself now with regards to the network side what we're going to do here is create a socket, now keep in mind of what I said about TCP and it being kind of state-based etc, so the first thing that I'm going to need to do I'm just wondering if I should put this into, so we'll leave that socket file, I'll come back and use that, we're going to tie some stuff into our manager struct first so remember this is the first manager that we created the overarching manager right at the start of when I started out doing this we created this manager that is going to have all of the neighbors under it and basically I need to set this up such that when I start the manager it's going to go and create everything that it needs effectively so we're going to read in the config and we're going to use a function there called readBGPconfig now that readBGPconfig I haven't actually created yet typically when I'm creating a config and I'm doing reads I normally have a function that'll just make things nice and easy to go and kind of read the thing and that function a really really simple function it'll basically set up to read the file and in this particular case because typically when I'm writing this I'll have a static config file the name is not going to change I'll just make a constant at the top here rather than pausing in a file name gbconfig.json so read the file into the byte array if it's not equal to null keep in mind that without a config I can't keep operating so at this point I'm just going to be effectively panic out of this but effectively listen couldn't read a config file I can go no further I am out of here and the same thing effectively when I un-marshall the JSON here if I can't get past this point I'm going to panic because I can't do anything without a config file and just panic so if that all works I can then return the config and so when I start my manager the first thing I'm going to do here read my configuration file and then I'm going to do something to the effect of one other thing about that config file I don't know if you guys remember but when I defined the config I had a kind of default source ASN here in the BGP config that's to set a system-wide ASN which will be set if you leave out a local ASN because multiple neighbors you have the same source ASN because it's from the same administrative domain so that effectively is just going to set up a default so once I've kind of un-marshalled this I can also then just go and set this up such that if you've left out the local ASN it'll set the local ASN from the global default otherwise your one will override and with the Keep Alive you have to have a Keep Alive time so what we do is we say that if somebody was stupid enough to leave that out of a config we can set that to a default Keep Alive time and we'll put a constant for that as well which will make this all happy so going back to the manager now that I can read that whole config file once I've read that what we're going to do is we're going to range through that config for each neighbor and we're going to set up a new neighbor config now we don't want to end up in a situation where you're effectively creating multiple of the same neighbor because somebody did something stupid because that's going to be a mess so we're going to write a function there called find neighbor which will validate whether or not this neighbor actually existed I just want to make sure this will be neighbor IP there we go so there's a couple of functions here that I'm going to need to add to add the neighbor etc so effectively we've got two functions here that we're going to need we're going to need a find neighbor and we're going to need an add neighbor so we're going to go back to our neighbor's file and in fact, sorry I'm wrong if you look those are both manager bound because we're acting on the overarching manager when we're finding here so these are methods of this manager structure they're fairly simple functions and again because I'm acting on the manager and I don't want to have a potential race I will effectively lock the thread and set it deferral so that when I'm done I can unlock it neighbor now keep in mind that because my mutex is at each layer in this kind of manager, neighbor, etc hierarchy of pointers when I create the new neighbor I need to also create a new mutex for that neighbor otherwise I'm going to have a null pointer there and I'm going to be in trouble so that's why that creates remember every neighbor has a pointer back to its own manager so that I can access stuff under it if I need to that's what I just set up there and then I can just call the config for the neighbor and it'll happily set up to add neighbors to my manager from the config the other thing that I wanted to do is just make sure that I wasn't adding a duplicate neighbor though I can add a simple method here now one thing to note about mutexes and locking you do not want to end up in a situation where you end up with effectively almost a lock race and as soon as I've finished adding this I'll show you what I mean so this add neighbor and this find neighbor these are separate functions and you'll notice that I lock my mutex in my manager inside those functions and I always unlock it as I exit those functions if however I'd gone I'm modifying the manager and I'm going to be running in a loop here anyway and I would have locked the mutex outside of the loop with the intention of unlocking it as I exited the loop the moment I called this function I'd be locking something that was already locked code would hang I'd have a problem that's not going to work I hope that makes sense to everybody so anyone got any questions because I think what I'm going to do from here is probably take a bit of a break and either later tonight or tomorrow I'll schedule another event and we'll start talking about actual socket coding now that we've gone as far as to set up all of our messages so questions guys any thoughts before we take a bit of a break thoughts, questions, comments speak now or forever hold your peace okay I am seeing no comments so guys as always drop comments in the stuff and I will add stuff comments and check and respond to them as I find time I'll probably also schedule a follow up session and we shall continue in the next coding session but I do need a bit of a break after writing that in two and a half hours so yeah thanks guys and I shall chat to you in a while cheers