Greetings, processor friends. This is video 10 in the series of creating a 6800 processor on an FPGA using nMyGen. So we've only got 3 instructions left to implement, and that is the wait for interrupt, return from interrupt, and software interrupt instructions. So before I get into implementing these, I just did a little bit of housekeeping with respect to formal verification. Now, one of the things that I've done is I've reworked the formal verification framework a little bit. So I have this record, which is an nMyGen construct. It's basically a bundle of signals. So for example, it's a bundle of an address, which is an unsigned 16-bit signal, data in, data out, read, write, VMA, and BA, and you probably see where I'm going, is that I want to make sure that on a cycle by cycle basis, the read, write, VMA, and bus available signals are as they should be, as well as the address line, if VMA is showing that it's a valid address. So verification has changed so that you set up what you want to see on each cycle. In addition, you set up what registers you want to see at the end of the instruction, and then you call verify instead of check. What verify does is it actually calls check on the inside, but it also sets up data and instruction as instance variables so that we can access it without having to pass around instruction and data. And then you get something like this, which is assert cycles. So what you do is you say how many cycles the instruction was supposed to last, and what this does is it checks that the number of cycles was indeed the number of cycles that you wanted. And on every cycle, what you do is you tell it, let's see where is it, what you do is you tell it to snapshot the signals. And what that does is it stores the address data in and data outlines, read, write, VMA, and BA, as well as the cycle number. And it stores it in an array of those records. So then at the end of the instruction, you can check every single cycle to make sure that things happen the way you think they did. So we have assert cycle signals, and you pass it the cycle number, and then you pass it the VMA, the BA, and the read, write, and optionally, you can pass it an address. You don't have to if VMA is zero, because of course, the address is invalid. So it doesn't really matter what's on the address lines. And then this is just a shortcut to say the signals that we want, the signals that we actually got, and then we just compare bus available and valid memory address to that. And if valid memory address was one, then we also check the read, write line and the address line. And then we're going to return either the data in or data out, depending on whether we did a read or a write. And this is one of the keys we no longer have to say during the instructions, oh, I just did a read, let me do a snapshot of the data that I read and the address that I read it from. Because we're doing this every single cycle, if the read, write line is high, and the valid memory address line is high, then we have done a read. So we will store that information, same thing with writes. So we no longer have to explicitly say, you know, formal read or formal write, which is nice. I also have assert registers, which pretty much does the same thing. Except in this case, you can specify any of ABX or the stack pointer or the PC. And if you don't specify one of those, then it doesn't change. It's not supposed to change. But if you do specify it, then it should be equal to what you wanted it to be. In addition, I've done the truncation for you. So you actually, that should be h, shouldn't it? I've done the truncation for you. So that, for example, when you're adding one to the PC, you don't have to do the truncation yourself. It's truncated right here. So no more of that. And then assert flags is pretty much the same API, except you don't have to pass it pre-CCS or post-CCS because now data is part of the instance. So we don't have to pass a reference to it, which is kind of convenient. So what does it look like? So here's a formal branch. It hasn't changed all that much. So one of the things that changed is at the top of the check, I checked that the number of cycles is four because that's how many cycles a branch instruction is. For cycle one, I'm checking to make sure that I have read, here's RW equals one, VMA is one and bus available is zero, that I've read from this address, which is the byte that comes after the opcode. Now in the cycle by cycle breakdown, cycles two and three are unused. The valid memory address is set to zero. So presumably it's being used for some sort of calculation. So and what this actually says is that you have set VMA to zero. So I don't need to check the address lines or the data lines. The rest is pretty much exactly the same as it was with the exception of the very end, where I assert registers and all the registers should remain the same, except for the program counter, which should be equal to the target. What's the target? Well, the target is the offset plus the PC, if you are taking the branch. And what's the offset? Well, the offset is actually, let's see, where do I store it? Right here. Offset is set from data. And data is actually returned from assert cycle signals. Again, if you're doing a read, it will return the data read. If you're doing a write, it will return the data written. And finally, we just assert that the flags have not changed. And that's really it. Now the interesting thing is, is that if we look at the formal verification, we can see that it failed. So if we look at the log file, and this is from cover, if we look at the log file down here, we can see that it failed at verification dot pi line 110. So we take a look at 110. And we see that, aha, it's a VMA line. So at some in some cycle, we said we wanted this VMA, but we got a different VMA. So immediately, I would look at the implementation of the instruction and say, well, did I actually set VMA properly? Because remember, we wanted to set VMA to zero during cycles two and three. And did we? Here's BR. Here's cycle two and here's cycle three. And I don't see any setting of VMA set to zero. So that's one way of checking it. The other way of checking it is is just look at the trace. So let's look at the trace. Right. So here's the trace. And I've opened up here, let me zoom in a little bit. So I've opened up, I don't actually need this. Whether the snapshot has been taken, which of course it has. Here are the cycles. Here are the recorded cycles. And we have the instruction. And we have record record one address. So this is the address during cycle one. You don't bother to record cycle zero because that's just reading the op code. So there's nothing really special going on there. So this is the recorded address from branch at C821, which makes sense because right over here is the op code address and C821 is the address next to it, which is the offset. And you can see that record one data in is two zero. So we know what the offset is. In addition, we can see that record one read write is high because we're doing a read. And record one VMA is high because the memory address is valid. Now let's look at record two address zero branch up branch bus available zero as it should be data in zero, okay, whatever, read, write zero, fine, whatever VMA though is high. That's wrong. Same thing with record three VMA. Now we can also see what we wanted by opening up the signals and scrolling all the way down to all the wants. So we can just take this and we could just open them all up. And we can see what we wanted. So we can see that the want address for cycle one is C821, which was correct. And then we can just compare. And we can see that want two and want three VMA are zero. So that would not compare. We can see that want cycles is set to four because it's a four cycle instruction. And want PC is C822, which is the target of the branch. So so by looking at that, we can see Oh, VMA was set wrong. So we can just fix that. That's really all you wanted to do. And then we can rerun formal verification and see what happens. So my mistake, the problem was that I was setting VMA at the end of cycle two and at the end of cycle three. And that's not what was supposed to happen. It should have been at the end of cycle one and at the end of cycle two, which means that it would be zero through cycle two and cycle three. So okay, and now that that's been fixed, we can see that verification has succeeded for the branch instructions, which means now that the branch instructions are cycle for cycle accurate, at least with respect to the address lines, the data lines, read, write VMA and be a now again, when VMA is zero, we don't output any particular address or read, write signal or data signal, because it's irrelevant. If there is no valid memory address on the bus, then you should not be paying attention to that memory address. And you should also not be paying attention to read, write or anything like that. And if you really, really cared about that sort of thing, you could just move the read, write and address checks up to here so that they take place regardless of whether you have a valid memory address or not. But you know, in looking at the cycle by cycle breakdowns, I've seen that the address when VMA is zero is really just used for like intermediate calculations. So again, it's not something that the external circuitry should be paying attention to. But if you really wanted to, you could include that in your implementation and your verification. Okay, so let's talk a little bit about interrupts. So the idea behind interrupts is that your processor is booking along executing its program. And then you want it to do something, you know, maybe you want it to do something periodically, or maybe a signal comes in from the outside world, and it has to do something. So in that case, an interrupt happens. And when that happens, you push the entire processor state onto the stack. So in this case, the entire processor state is the stack pointer, the flags, the accumulators, the index register and the program counter.  And then you pop the processor state off of the stack. And now your processor is continuing where it left off. Because of course, it's stored the PC. Now, you can't just store the processor state and start another routine in any old place. You have to wait until the instruction that you're currently executing is complete. So you can see this here in the data sheet that the IRQ and NMI interrupts are sampled during phase two.  So how do you know which interrupt routine to execute? There are actually three different interrupt routines. The first is the non-maskable interrupt routine, which is stored at addresses FFFC and FFFD. So that's the address of your non-maskable interrupt routine. So non-maskable basically means that there are actually two types of interrupts. There's a maskable interrupt and a non-maskable interrupt. And a non-maskable interrupt must be serviced. A maskable interrupt, there's a mask bit in the flags register called I that you can set or clear. And that tells you whether if a maskable interrupt comes in, whether you have to service it or not. So the maskable interrupt is stored at FFF8 and FFF9. And then you've got a software interrupt, which is what happens when the program itself wants to perform an interrupt. And that is stored at FFFA and FFFB. So technically speaking, I guess you could consider reset, kind of like an interrupt in that it has an interrupt vector at FFFE and FFFF. The only difference, of course, is that when you do a reset, it will immediately take place, regardless of where you are in an instruction. In other words, if you hit the reset button, it's not going to wait until the end of the instruction. Essentially, it's like a power cycle, kind of. So let's take a look at the actual instructions that make use of these interrupts. So I guess the simplest one is the software interrupt. So in this case, we can see it's a whopping 12 cycles. Of course, it's so long because the entire processor state has to be stored onto the stack. And that's seven bytes, it looks like. So there's two for the return address, two for the index, two for the accumulators, and then the flags register. So that's seven bytes in total that need to be stored on the stack. Then you have to go ahead and read the software interrupt vector at FFFA and FFFB, and then go there. So that takes 12 cycles. So when you are ready to return from the interrupt routine, basically you pop the processor state off of the stack. Again, that requires reading seven bytes of data from the stack. And then that's pretty much it. So it's kind of like an RTS in that the very last two bytes are the address that you have to return to. But it's a return from interrupt. So you've got some extra stuff on the stack that you need to deal with. That's why, of course, in an interrupt routine, you use RTI and not RTS. So here's the software description, the programming reference manual's description of software interrupt, which basically says what happens when. So we can see that we're pushing a whole bunch of stuff over here. And then we set the interrupt mask flag. And basically what that means is that you're not allowed to do a maskable interrupt request at this point in time.  Now, one of the interesting ways that the programming reference manual talks about the vector is this sentence over here, which is kind of a weird way of talking about addresses. It basically says memory locations n minus five and n minus four, where n is the address corresponding to a high state on all lines of the address bus, which is kind of a funny way of saying FFFA and FFFB, right? Because the address corresponding to a high state on all lines of the address bus is basically just FFFF. So why wouldn't you have just used the hexadecimal notation? I really don't know. But in any case, the implementation of software interrupt is relatively straightforward. You basically do what it says, keeping in mind that you do have to set the interrupt mask bit just before you read the interrupt vector. So that's pretty straightforward. Likewise, return from interrupt is equally, fairly trivial to implement. Wait for interrupt is a different story now. If we look at the cycle by cycle breakdown for wait for interrupt, you can see that it's nine cycles, and basically it pushes the processor state onto the stack, and then basically it just sits there doing nothing, waiting for an external interrupt signal to come in, which could be an IRQ, a maskable interrupt, or it could be a non-maskable interrupt. So basically in wait for interrupt, once we push the processor state onto the stack, we're waiting for an interrupt, and it could be an NMI or an IRQ, whichever one comes first. Now, interestingly, the description in the programming reference manual of wait for interrupt is actually incorrect. So it says down here that when an interrupt is signaled on the interrupt request line, and the iBit is clear, so in other words, you're not masking things off, execution proceeds as follows, and then it's basically the same as software interrupt, except that the interrupt vector you're using is for the not, for the maskable interrupt, and it basically says N minus 7 and N minus 6, which of course is FFFF minus 7 and FFFF minus 6. What it doesn't talk about is what happens when there's a non-maskable interrupt. So in fact, wait for interrupt waits for both cases, either a non-maskable interrupt or an interrupt. Now there's an interesting flowchart here in the data sheet which shows what happens when there's a non-maskable interrupt and a maskable interrupt. So if you get a non-maskable interrupt, then you just execute the interrupt routine that's stored at FFFC and FFFD. If you get an interrupt request, though, and provided that the iBit is not set, then you execute the interrupt routine stored at FFF8 and FFF9, so far so good. The interesting thing is that NMI and IRQ are sampled on Phase 2, which means that they're sort of, you know, latched, and then the processor takes a look at what got latched. What happens if both of them are latched at the same time? Well, NMI takes precedence according to this flowchart, so that's kind of interesting. So the implementation of SWI is not exactly straightforward because we can see that the instruction technically ends once you've pushed the state of the processor onto the stack, but it doesn't really end because you're just sitting there and not executing the next instruction. So what exactly happens? Well, I suspect that what happens is that the instruction does end, but the processor enters a special state where it just spins and does nothing. And then it handles interrupts and NMIs normally, which means that at the end of any instruction, we check the NMI and IRQ lines, and if any of them are set, we do the interrupt. So the first thing I've done here is I've added an IRQ and an NMI signal. Now, on the processor pins, these are active low signals. So in other words, if the negative IRQ goes low, then that means you're doing an interrupt. So we don't really want to do that in software because just handling negative signals is just confusing enough. So in the CPU implementation, we will just invert the signal and pass that to the core. So that's why we're just calling it IRQ and NMI and not NIRQ and NNMI. So here's the implementation for software interrupt. So basically, it just, you know, does what it needs to do. It stuffs everything onto the stack. And then if you look at the thing starting from, so here's cycle eight, where during cycle nine, we're going to set VMA to zero because that's what you do. And then during cycle nine, you can see that we're setting the interrupt flag, and we're reading the address at FFFA. And then on the next cycle, we're incrementing the address and reading that. And then on the final, at the end of the instruction, what we do is we jump to that address. With wait for interrupt though, it's not as straightforward as that. So this was my original implementation, which is not, which is not correct. So we do basically exactly the same thing as SWY. We just push things onto the stack. However, in the setup for cycle nine, what we're doing is we are setting VMA to zero and bus available to one. And that basically means that the processor is going to be disengaged and, you know, some external hardware can take control of the address and data lines. That's really all that means. And then here in cycle nine, my implementation basically just said, okay, well, if there's an IRQ and it's not masked off, then set the interrupt flag and go to FFFA. Go to the address stored at FFFA. If it's an NMI, then set the interrupt flag. I'm not sure that I had to do that, but set the interrupt flag and go to the interrupt routine stored at FFFC. But if neither of them are true, then again, just keep the signals as they are and don't increment the cycle. So in other words, we're stuck here in cycle nine, just spinning round and round and round until one of the IRQs and NMIs goes off. And then we continue the instruction. So this is the wrong thing because the instruction technically ends right over here, cycle eight. So what I'd like to do instead is I'd like to take essentially this code out and put it somewhere else and then have the processor enter some special state where it's just waiting for this, waiting for one of the interrupts to happen. As a consequence of that, we'll also be able to handle interrupts during any instruction. So we sort of get that for free. So in order to implement interrupts, what I've done is I've kind of modeled an interrupt as just another instruction. So if you look at the end of instruction handler, this is the stuff that happens at the end of an instruction. So as long as we're not in a wait for an interrupt, right? So we're just, we've just ended an instruction. So ordinarily, we take whatever the end of the instruction address was. So in the end, normally, we would just go to the instruction that you wanted to at the end of the instruction, and you would set the cycle to zero. And again, normally, you wouldn't be handling an interrupt. That is not supposed to be there. And you would also set the address lines in preparation for a read. And then cycle zero would come around and you would actually do the read. That's your fetch cycle. Now, if on the other hand, at the end of an instruction, you were hit with an interrupt. So for example, in NMI, then what we do is we set a special interrupt flag, we set a special interrupt vector register. So in this case, the vector is going to be number two. We set the address lines to the stack pointer, we get ready to do a write. And what we're going to do is we're going to write the PC that you're going to return to, which of course should not be PC, but it should actually be where you wanted to go to at the end of the instruction. So obviously, I haven't tested this thoroughly. And I'm still thinking about how to formally verify that interrupts actually work. But what I have done is I've simulated at least a non-maskable interrupt. So let's take a look at that. So here what we can see is here is the end of the instruction. And you can see that right here. There's the end of the instruction. And there should be an NMI here somewhere. Okay. And there is non-maskable interrupt. So we're setting the non-maskable interrupt. So what should really happen right here is that we start writing the state to the stack pointer. Now in this case, the stack pointer is zero. So ignore the fact that in simulation, we don't actually do any writes because this is clearly overwriting all the vectors. But anyway, so we spend one, two, three, four, five, six, seven cycles writing the state of the register, writing the state of the CPU into the stack. After that, we do nothing. You can see that the VMA is low. And I'm not quite sure if that's correct. But let's continue. And the NMI vector is stored at FFFC. So here we're doing a read of FFFC, a read of FFFD. And I've programmed the simulated memory so that it should return F0, F0. And then right after we do that read, we go into cycle zero, which is the fetch cycle. And the address is F0, F0. And at F0, F0, I set up an instruction of a knob. So that's zero, one. So there's the next cycle of executing the knob. And then we continue. In this case, we ran off the end of memory. So I'm putting FFs in here. And right now, FFs are actually valid, valid instructions. So it just continues to go. So I don't have the RTI yet. It just continues to execute FF instructions. RTI would require reading back the state of the the state of the processor. And of course, I don't have a memory that I'm writing to. So I can't actually simulate that. However, we have formally verified that that worked. Okay, so in this next simulation, I've simulated a wait for interrupt. And I've written the code if we look over here. So there's the wait for interrupt after just doing a branch to the next instruction, we do a wait for interrupt. And then after that comes a knob. And in the process, what I've done is I've basically said, okay, go for 20 cycles, clock cycles, and then set NMI to one, go for one clock cycle, set NMI back to zero, and then go for another 20 clock cycles. So in the simulation, we can see, and I've gotten kind of gotten rid of a lot of the crud here, we can see that the instruction is a three, which is a wait for interrupt. And we can see that we're going from cycle zero all the way up to cycle seven. And the read write line is low for two, three, four, five, six, seven, and eight. So that basically means that we are doing a write. And we're writing the state of the processor. So there's the address zero, because we're starting with the stack pointer of zero. So that's 1234567. Okay, and then you can see that we enter the wait for interrupt state. So we're just going to basically sit here doing nothing. VMA goes low, so we're not doing any writes anymore.  Now where's NMI? I don't see it. Okay, well, in any case, NMI goes off, and we immediately go and read FFFC and FFFD. And we end up with F zero F zero as the destination. So that worked as well. And the way that I made that work is that if we are waiting for an interrupt, then basically we just keep the same cycle, and we're just waiting for NMI to go off or IRQ to go off. And if none of them go off, then we just set valid memory address to zero and bus available to one. If one of these goes off, however, then again, we set interrupt to one. We do a set interrupt mask. We set the appropriate address, because what we're going to do is we're going to go into the middle of the interrupt pseudo instruction, we're going to go straight into cycle eight. So first we need to set up the address lines to do a read of the first byte of the interrupt vector, which we do. And then we take the data in and put it into temp eight. And then we increment the address and we do another read. And we go ahead and end the pseudo instruction right there. So that's how that works. I haven't fully thought about how to actually formally verify this. What I do know is that yosis is going to feel free to get rid of that. Yosis is going to feel free to modify the IRQ and the NMI signals whenever it wants. So it'll be interesting to just try a formal verification of any old instruction and see what happens. Okay, so what I've chosen to do is attempt to formally verify the pull instruction, just because it's a short instruction and it only takes four cycles and it doesn't really do much of anything. And we can see that yosis has actually changed the NMI signal. So let's take a look at some of the other signals that are happening. So we will look at the core signals and we're going to pull up the address and maybe zoom in a little bit. Okay, and we're going to take a look at the instruction that we're executing. Okay, and the pull instruction and the pull instruction is this one right here, three, two. Now it does take four cycles. So we can see that in fact the NMI line is not high by the end of the instruction. So that's just perfectly fine. However, so this is the cover statement. So we found one instance of a pull instruction that works. Now formal verification did fail. So let's take a look at y. So here's BMC failed and we can see that it pretty much immediately fails and there was an assert in the core. Wow, that's not great. What's 1577? Oh, this is interesting. Apparently, apparently NMI and reset don't play well together. So let's take a look at that and see what exactly it came up with. So first let's pull up the clock. Why not? Let's pull up the reset line. And let's pull up the NMI line. All right, clearly something's happening. So the formal verification is supposed to say, okay, if the past of reset was zero, four cycles ago, and three cycles ago, two cycles ago, and one cycle ago, I guess that would be here, here, and here, if reset was low, then we want to make sure that we actually did jump to the reset factor. So let's take a look at the address lines and the data inlines. Okay, so here's the reset factor, the reset vector that you just chose was 0001. And clearly, we did not jump there because the NMI line just went high. So we jumped to 00. Actually, we didn't jump to 00 where the stack pointer is 00. So as you can see, we're actually doing a right. So in other words, we just loaded the interrupt vector, and we just loaded the reset vector, and we got a non-maskable interrupt. So of course, we're going to start saving the state of the CPU. Now, I think this is correct behavior. So what's actually wrong is the formal verification what I need to make sure. So this is what I've done. I've just added this requirement. I'm not sure that that's going to satisfy. I'm not sure that that's actually going to do what I think it does. But I'm just going to run formal verification again on pull and just see what happens. So here's the problem. And I believe it's a problem with our formal verification framework, which is basically how we know when to start formal verification. So you can see here is a three two, that's the pull instruction, we can see that we're at cycle zero. However, notice that this was due to an NMI. So we're actually not executing the three two instruction, we're executing the interrupt pseudo instruction, because we're going from cycle zero all the way up to cycle nine. Now, when that pseudo instruction ends, formal verification thinks it has to kick off. You can see here that it took a snapshot of the instruction. And if we look at the code, we can see exactly why that's happening. It's right over here, because we've assumed that when we're at cycle zero, then whatever's in the data in is the instruction that we need to look at. And, you know, prior to interrupts, that was actually true. However, this is no longer true. So we have to gate this on whether we're in an interrupt situation or not. Now I've just added the interrupt line so that you can actually see what we could possibly do, which is that we could gate formal verification snapshotting on whether interrupt is high or low. So here I've just added that clause at the end. So we just want to make sure that interrupt is low. That means that we're not in the pseudo instruction for interrupting. So we really should be looking at data in. So let's see if that actually works. Okay, we got another failure this time during cover. So let's take a look at cover and see what it found. Okay, this time it's a slightly different problem. So here we are genuinely executing a three to pull instruction. We get to the end of the instruction right over here. And oops, we've got an NMI. So that immediately starts the next pseudo instruction. But the next pseudo instruction has interrupt sat high. And according to the code that we just wrote, we're not actually going to do formal verification until we get to the end and interrupt is low and cycle is zero. And then we start doing formal verification, which is incorrect, we need to do formal verification right here. So let's modify the code again. So this is the next iteration. Basically what I've done is I've moved this up one level. So if the cycle is zero and the reset state is three, it doesn't matter whether interrupt is set or not. But if we've taken a snapshot, then we go ahead and take a post snapshot and then do the verification. So let's see if that works. So what I've done is I've restricted taking the snapshot to only those situations where the interrupt line is low. And if the interrupt line is not low, then we're not taking a snapshot. So with that, finally, formal verification does actually work. So that's pretty good. So what I should do is I should run all of the instructions through formal verification, again, remembering that the only formal verification that I don't have yet is wait for interrupt. Oh, and I also don't have not. I really should do that. Taking all of the instructions through formal verification takes just 15 minutes with six threads. So I'm going to do that. And we're going to get back to so I'm going to do that. And we will get back to it as soon as that finishes. So with formally verifying wait for interrupt, we basically just go all the way up to the end of the instruction, you know, up until we're actually waiting for the interrupt. And at that point, we could immediately get an interrupt, or we could be waiting for a while. So, you know, the the interrupt flag could be set because we got an interrupt or, you know, maybe we're just still waiting for an interrupt and the interrupt flag could be zero or one depending on what it was before. So so the only thing that I could do in this case is just make sure that the I flag is set to whatever it is actually set to. So that's why I put in this comment saying that this is not a mistake because this may look really weird and it really look like it should be pre CCS, but it isn't. Okay. And finally, all instructions have verified including wait for interrupt. Eventually, I had to end up adding this line, because in the middle of a wait for interrupt, you're waiting for interrupt and your cycle number is not zero. So this so this part doesn't apply. So I had to add this little section over here. And all the instructions have verified. So it took about 20 minutes to do from beginning to end. So I also compiled the lattice version. And if we take a look at it, we can see that I've added a few more pins, I've added the bus available pin, negative IRQ negative NMI, I took away the reset state, I don't really need that. I did change the address lines to OE, which means that they can be tri-stated. Data of course always could be tri-stated because you could just turn them into an input. And the read write line is also now tri-statable. Because of course, when you assert bus available, the read write lines and the address lines have to be tri-stated and the data lines have to turn into inputs. They can't output anything. So I added the logic for that. Obviously, you can see here that we're getting the NIRQ and NMI pins. And we are sending them to the CPU in their negated form as we wanted. For the bus available pin, we just copy it over. But based on the state of the bus available pin, we can tri-state read write, the address lines, and the data lines turn into inputs if we are doing a read or the bus is not available. In other words, the data lines are outputs if the bus is not available and we are doing a write. So let's take a look at the number of cells we've used. It is a huge 3,895 cells, 3,500 LUTs. So that's pretty huge, which is okay. So I think the next thing that I may work on is a couple of small optimizations that may help knock that down. And now the moment that you've been waiting for. Doop! How's that, Cat?


She told me she could sort of sneak me in as like a caterer if I wore a suit and tie So she's like look it's in a month and if I were you I would do your research And I did just that I watched all his movies I read his eight, you know 800 page biography and I finally get there and sure enough Spielberg You know in the beginning of the party is like 20 feet away from me But he's talking to like George Lucas and Jeffrey Katzenberg and Jack Black and the flinches in full effect and finally When it was my time to like try to make my move. I was completely frozen mouth-wired shut feet turned to stone and I lost my chance And I go to the bathroom and I told myself and again this goes back to what you had mentioned if I don't push through this my entire journey will be over before it started and And I go back to the party and I just you know Spielberg's in one set of the room I'm I'm the other Spielberg's on one side. I go to the other and this is like really awkward dance and Finally, I see him walking towards the exit and again my fear of failing Was now even more powerful than my fear of getting rejected. So I just start sprinting through the party now I'm dodging waiters.  Spielberg And he turns around and I remember like his eyes eyebrows shooting up and he puts his arms in the air and I like coil back And he gives me this big hug And he's like I've been on a college campus all day and you're the first student I've met And I'm like, can I ask you a question?  They're not gonna say no So I'm walking with him to his car and he asked me what I'm working on and I just I didn't have a pitch I just started pouring my heart out to him telling me about this dream and Finally, I'm like mr. Spielberg. Would you? Would you do an interview for the book?   I don't know what it is, but I believe you can make this possible and From there. He gave me one of the most important things anyone can give you Which is that possibility? Yeah, chance. Yeah, and What's so interesting to me about that is You have this idea you sit down with your friends and now it's happening Yeah, and the first crucial moment. You're like, I don't have a pitch I just got to go with it. I just got to go with my gut and I think a lot of times we had Warren Klopp on Last month talking about pitching I think a lot of times we feel like there's a magic phrase or a magic key that we just seem to say the Right words and it's gonna happen and we lose out on that heartfelt genuine moment where you know what? I'm just gonna be me. Yeah, and this is my chance and it resonated and he gave you the maybe So you ran home and told everyone I got a baby from Steven Spielberg this book is happening


The World Economic Forum on India is your window to the world from the 6th to 8th of November in Gurdong. Global leaders are meeting to discuss the most important topic for the country. How can we best tackle corruption and improve government transparency in India? How can India better address the low value placed on women and their role in society? How can India protect its national security without compromising personal freedom and censoring conversations on the internet? What do you want to ask the leaders on these topics? Send us your questions now. You can post your questions on these topics using the World Economic Forum's Facebook page www.weft.ch. Ask on Facebook or tweet us using the hashtag web ask or email us at ask at www.weft.ch. The World Economic Forum will put the best questions to some of the most powerful and influential people in the world at the meeting which you can also watch live at www.weft.ch. Slash Live!


Welcome to Tokyo crowd in crowd in all right good morning thank you this is the open v-switch with open virtual networking session by people that don't work on open v-switch yet that that's a subtitle that you can't read but that's that's that's basically us I'm Sean Lim I'm a principal engineer at Time Warner Cable work with Dave and a whole bunch of our other team members on the open-stack team my focus is generally Neutron and Nova and I'm Dave and I work at Time Warner Cable as well we've had an open-stack cloud up for a year and a quarter now and have brought it up to Kilo and Partial Liberty and I'm the lead engineer for compute but also work on a lot of other things including the networking I'd like to give a shout out to the Denver open-stack meetup leaders that also joined our Colorado cloud we're both from Colorado and I also leave the Colorado open-stack meetup I think I'd like to open up before Dave jumps into the the heart of everything to say that this is probably gonna be a little more of an overview than some of you were expecting I see a smattering of faces here who probably know way more about some of these subjects than we do but we are operators of it and we're started spiking OVN so we're kind of delving into a few things for the first time and what we want to do is cover some of the problems that we've seen with OBS some of the things that we've seen with OBS and neutron integration and then go over some of our interesting material with the OVN yeah and he's stole a little bit of my thunder there I've been I've had West Nile virus for the last six weeks I just got better in time to get here but really not to prep so Sean has none of the none of the responsibilities for our lack of a preparation and I bear all those so all the mistakes are mine and all the omissions are mine we will also give a huge shout out to the open V switch and neutron team they did a stellar presentation on Tuesday if you have not had a chance to see it I'll show you a link at the end it'll go into a lot more depth on OVN than we will be able to go to all right so OBS 2.4.0 is basically the release that is out with Liberty I think all of the distributions are also providing that version of OBS now it includes open virtual networkings toaster oven I love their names the toaster oven release of oven sometimes OVN is called oven and that's the ready for test release but open V switch itself also includes a lot of improvements DP DK which is basically fast hardware enabled networking Geneva support for additional tunneling Docker support some default port changes and bash completion that's my favorite one that's your favorite one all right and line yeah there also obviously some open-stack improvements that are utilizing this in in key so when when we proposed this talk we were going to talk about kilo in Liberty I know Liberty's release but there are a lot of people still running Juno or get or not quite running kilo yet so we'll talk a little bit about that so yeah I was gonna say so if you aren't going the OVN route and spiking that and you're using the native are using standard everyday kilo Liberty layer 2 agent these are two things that you want to actually look at and start turning on we haven't rolled these out in production but I have spiked it all the way up to that essentially up to kilo most of these calls were native calls that were route wrapped and when you end up on the control node side the network node side of the world and you have a huge number of routers and you have to rebuild those it takes four over 10 to 15 minutes for 50 to 60 routers and there's a lot of causes of that but a big chunk of that is doing this route wrap pseudo inefficient call so as of kilo rolling out the VS CTL the VS Kettle calls all have a native Python interface on that I'm not sure I saw six to seven times speed up as the blueprint said but definitely noticeable in that and I haven't experienced any issues with with switching that over it's a one-line switch in the config file and we're getting close to running rolling it out in production as well and probably that last bullet is probably the most important if you're an operator this has been a long-standing issue with us if you had to restart OBS you took a significant outage on your control plane it wasn't so bad on a compute node because there just isn't as much running there but on a control plane an OBS L2 agent outage was painful and it was on the startup side you could actually shut down the agent and everything ran fine until about five minutes when the Mac learning ran out but on startup the decision-making was I don't know what to do if it's just flush everything so on our control nodes that amounted to about 2,500 flows 3,000 flows and that just takes a lot of time to rebuild and during that time we have customer outages so this is really really important to us absolutely yeah this is one that I ran into sideways it's since been fixed in kilo essentially when you create a port in the OBS if you've looked underneath the hood in the Linux namespaces and then that's added to a namespace that's kind of a semi illegal operation or has been recently and it just works and nobody really notices it during their gate test for DVR this cause started causing kernel faults and as I kind of stumbled on to this I realized that some of some of the cleanup we were doing for upgrade to kilo and some of the other operations we were doing in depth there were kernel faults that trace back to this it is a fixing kilo but it also required OBS kernel support kernel support yes so it was out in 4.0 but I think I think everybody's back ported it it's got a supported stream and the outcome of it if you ever see it is really that first of all kernel faults and the logs are not cool but and a little bit scary but your functionality of those ports and OBS just doesn't work anymore and you have to end up by hand removing them and getting them re-created from scratch so it creates operational problems so this is what we do at Time Warner Cable with open v-switch Sean actually wrote this slide but when did you go ahead sure okay we have pretty simple use case up front I guess at the time when two years ago when we put things in place VX LAN was not not simple that's a little bit why we decided to go OBS versus Linux bridge is better be excellent support and behind the scenes I knew that because we're a cable company and we have our cloud integrating with lots of other entities in the company that that we were going to have to do some fun networking at the edge of open stack that has actually come true and and OBS is providing us an easier means to connect to things on the edge neutron may not support it but we can at least put that in place the rest of our use case is pretty pretty normal traffic we aren't trying to do anything with video streaming or non TCP UDP protocols so it's pretty average and we aren't pushing crazy bandwidth so we don't use DBDK yeah yeah yeah and a lot of our business use is actually it's more like business usage than it is video streaming usage we'll get there but right now we've got plenty of uptake inside of our company with just kind of your web page type type access not we're not video streaming at the current time if you ended up seeing the incredible rack space demo that went viral and Paris that was the ludicrous speed demo we were caught by that and so it was exciting for us to see that other people were having a horrid problems as our as we went and started scaling up the cloud we started seeing a ton of crashes in OBS one for operational production level what's going on and it didn't trace back to anything so basically as soon as we got back from Paris like that week we used we used Ansible and we upgraded our all of our production like minimal testing I would say because we were super confident well we had we built our own packages that was one of the first times we had to build our own packages the distro wasn't ready to do this yet and they weren't sure that they had a root cause so they couldn't convince us that they had a root cause so that we could use a back port turns out they didn't actually have it at that point we found out later so so we had to we basically had to move forward and and I think with OBS that's probably the right answer move forward when you can with OBS everything gets better yeah the upgrade was actually relatively painless we we have pain points on our network nodes because of all our we used legacy routers at this point and and that requires those flows flushing and so we had to take a little bit of an outage or at least set my work around that but but the upgrade itself was pretty painless and since then we realized that we've come to blame everything on OBS and now we can't so we actually have to solve actually we've actually gotten better at troubleshooting because we didn't have an obvious smoking gun anywhere right we couldn't just always point at OBS we did come up with we pointed at Sean actually we didn't point at OBS I got to sleep finally but we did end up with a massive number of automation scripts to do crazy cleanup things that now just sit there gathering dust which is incredible for us to see and so we've had solid performance since then and let's go on all right so the two four oh release notes include some deprecations that you need to be aware of mostly for going beyond that I can't say that I'm intimately familiar with these but I did see them and I want to point them out something about Jerry 64 tunnel is deprecated and we'll be removed in two five so just be aware of that if you're using it we're not using that let's see external network bridge option for the L3 agent has been deprecated in favor of a bridge mapping yeah I'm reading this slide to you but that's what I do I have more to say on the external okay go ahead with the external one of our customer requests is you know we've got a huge subnet of our own and we want to put that into OpenStack we call this service bring your own IP BYOIT exactly and it's gone viral on the company kind of silly but this external bridge networking option allows it was the way you mapped your your connection to the external world this change here allows us to and related allows us to plumb out multiple even L2 subnets and L3 out of OBS and it actually helps us to realize BYOIP BYOIP I can't say it okay for our customers cool okay so open virtual networking again big shout out to Justin, Ben, Russell and Kyle for their talk on Tuesday morning it was up on YouTube by Wednesday morning so you can you can go watch it and you can just leave right now if you want because that'll be a better use at your time but it is in the master tree so if you do a get clone of OpenVSwitch you get open virtual networking for free it does it does have these goals L2 L3 network virtualization logical routers multiple tonneau overlays that added to Geneva works anywhere that OpenVSwitch works that's really important OpenVSwitch is a lot of the reference architectures for OpenStack rely on OpenVSwitch and so it'll work anywhere that OpenVSwitch is working I can't say that that strongly enough so that that's one of the big gets that's that's an easy get out of OVM it works if OVS works here's the architecture and this is this is a big change I'm gonna I'm gonna not do this justice but I'm gonna talk through that this particular diagram comes out of the build when you build OVM there's an architecture man page and this comes straight out of it so at the top is your cloud management system so you can you can just substitute CMS with OpenStack because OpenStack is the only one that's ported so far so this was basically built out of the box for OpenStack so there's a there's an OVM plug-in that goes into OpenStack and then there's a northbound database and that's what OpenStack is going to talk to and that OVM NorthD Demon will then talk to this outbound database and then inside of each of your hypervisors whatever that hypervisor is and or Libvert KBM etc you'll have an OVM controller and inside of there you'll have your normal vSwitch and your normal OVS DB server and so you're writing kind of at a high level language at the very top for the northbound database and then the actual flows get pushed in at the OVM controller level inside of your hypervisor so you're kind of writing a meta at the OpenStack plug-in is writing kind of a metaflow and then the actual open flow is written in the hypervisor what I didn't have and probably should have up here also is a view of how things were before OVM or if you're not yet using OVM there's actually some additional complexity that were basically that basically gets abstracted away and that's probably the biggest value I've seen out of OVM so far is it simplifies things it gets rid of a Linux bridge that handles the IP tables today and you basically just get closer to the metal so you get faster and one of our first slides I was saying you know neutron integration with native Python for OVS VS Kettle and OF CTL and Liberty that all goes away and gets replaced just do this yeah right so all right no no we can't no no sorry it's just very bright yeah I'm just blocking the light I'm not blocking you I'm blocking the light no no no OVM relies on OVS OVS is still there and OVS will continue working the way it was always worked 2.4.0 if you're running OVS today you should upgrade to 2.4.0 it'll just keep working the same way so your your L2 and L3 agents will work the same way they do today maybe better in Liberty but if you want to move away from that architecture to a different architecture this this is this is what you will find opportunistically available inside of the OVS Git tree there is an OVM directory and there's some readme's in there that tell you how to build it and how to play with it you don't have to actually deploy it you can just go play with it and you can learn about you can learn about this architecture very simply by just doing a Git pull doing a dot let's see the auto tool stuff configure make and then and then you can make a sandbox where you can start playing with OVM we'll talk a little bit about that on the last slide and and one of the big improvements on that is if you've ever trolled through the neutron code open L2 agent L3 agent you'll come down at some point where all all those OVS commands are wrapped right now so like oh add a port and delete a port and add this flow and it's just hugely enumerated and messy to follow and if you've ever tried to troubleshoot it backwards up the stack it's super painful those use cases were encapsulated so 10 10 commands might create a port and plummet out into neutron those are all encapsulated rolled back up into OVM and taken out of the neutron code so OVM also is utilizing some of those OVS improvements so in an upcoming release I think I think there is a branches for this already and again this came out of the release notes so my apologies and also all the credit belongs to Russell Bryant it does require new some new or kernel stuff in order to use that but this again gets rid of the IP tables type rules security groups are implemented as well it just it just cuts out several stages of the pipeline so everything should flow faster in neutron as a result Russell Bryant put up a blog entry last week that talks about these new security groups the security groups from a from a user perspective work the same way it's just how they're implemented via ACLs that changes if you've ever followed where security groups are placed and how they're placed in the 27 different hops that has to do I don't know I'm super exaggerating yeah you want to wash your hands afterwards you want to wash your hands after trace trace traffic through there it kind of collapses it down to a much more manageable troubleshooting scheme that's a good question yes all right so I told you there was a more info slide coming so there is more info so this is you just want to check out check out OVS do do the regular build and but inside of inside of there there'll be an OVN directory where you can actually go build play with OVN they it comes with a sandbox area so that you can create create ports create flows and watch it watch the kind of the meta flows flow that fall down into the open flow world and get implemented but more importantly you really want to watch this OVN talk from two days ago it's right here on YouTube they did an excellent job of covering this in much more detail they are the developers they know it way better than anybody externally could ever know it it is a good use of 38 minutes of your time let's see what else play with it in dev stack so this is something I haven't gotten around to yet but I would highly recommend it this link will take you to a page where you can set up dev stack in a multi-note environment and actually run OVN and see how it how it's supposed to work before you go to the work of actually deploying it and changing all your puppet and ansible and juju and whatever else you use but but but you do want to start using this there are also some other competing technologies I'm not going to say that OVN is going to replace sliced bread it's world peace butter but we're world peace I think I think there are a number of other solutions that are also helping to flatten that stack I think the Astara the new the new project from a conda will also help in that environment so be aware that there are some other technologies and oh I had one more thing yeah so in addition to the devs at Time Warner we don't really run dev stack we have a really everything's automated we can actually put a cloud of our cloud in our cloud so nested clouds throughout the stack so we can bring up a subset or our entire cloud inside our cloud and do all our testing it's the way we do all our development all testing we so we use vagrant with the open stack provider I think it's called so that we can actually deploy any of our architecture on top of our cloud so that we can do next next level develop I don't know if we did a did we do a lightning talk on that this week nobody nobody thinks we did a lightning talk on that this week but if you've got questions about what we call vdev virtual development of our event feel free to get any of us up the advantage there was in you know one of the long flights over here I was able to actually start hacking together and I say hacking because it was mostly on our automation side OVN and a place in our cloud and I got I got quite a bit of progress I'm pretty pleased with the way things are coming together but there's still some upstream patches that I had to do and most of the work to be honest was on our automation side to get that to work so I wasn't just manually hacking on 27 virtual machines but I'm pretty pleased with the way things are going it's worked worked pretty well okay that's all the material we have present prepared we can take questions we may not be able to answer them but we can start go ahead flooring now just your way too far back there for me and Mike you did okay I'm going to repeat that for the audience for the video as well how do you in a virtual development environment how do you basically nest floating IP type information and it is a problem so a lot of the work that we do the short answers we just don't run into the problem because a lot of the time we're actually trying to set up our automation and stuff that doesn't actually rely on a nested virtualation floating IP so we can set up the networks you just can't get out so that that's the main solution when we ignored the problem yeah we ignored the problem we do have a fix but we do do we do do live migration and everything inside of our virtual development environment but we're just not able to drill into it from outside they can get out but you can't get it so the recent solution that we've done since we upgraded the kilo is we yep yep what's that turn up for support security and then allow it to plum out yep you know properly you still have to have some external the other virtual machine and a device on there to test it but the other way we were doing it is the neutron debug port create for a long time at least allows you to do some funky magic to get down to that but and Florian see me any time after lunch or after 10 o'clock today I have a frisbee for you for being the first one to ask a question and this gentleman over here that asked a question also you can find me any time after this meeting and I've got a frisbee what's that okay no no you're not ringing any bells up here so short answer is no no no any another question go ahead so okay so my my take on it is that there will always be a kind of the dev stack defaults and that's what is going to be the best tested but anybody that gets gate requirements in and I think all three of those that you mentioned have gate gating requirements should work it's it's really kind of giving you a broader menu to pick from they have they have different resources behind them obviously if you you know if you need commercial support you might have to go a different way say from one than the other there are it's great that there are a lot of people solving this problem because there are lots of ways to skin this cat so if you need if you need switch level hardware support like if you need a metal iron support old-school old-school network metal iron support is going to favor one of those solutions virtual metal I mean an OS running on metal like a broadcast chipset is going to favor another one of these solutions or maybe a couple of these solutions and if you want to pure software implementation with no metal at the top you might want to go this way or the other ways but that that's going to skew your perspective right we are a large IP shop basically well I mean that's anybody that's in cable these days is basically an IP shop and so we've got a lot of a lot of custom or expensive networking gear that we would like to be able to plum into so that kind of skews our thinking but we also want to be able to go forward and do a pure software deploy on that same kind of gear and we're doing both so that that that definitely flavors are thinking I think I think the first step that I always think about is since since I take the calls as well our whole team I mean we're a real DevOps team so you take a few weeks every year getting yelled at which really teaches you quick that operational efficiency and ease of troubleshooting are you know two things first and foremost anything for us that collapses you know collapses the problem set down and allows us to troubleshoot faster is always more of a shoe in now we we have a usual at big companies pretty well-defined silos in the company these are these are breaking down pretty quick for us but but there's still a there's still a large networking silo or networking expertise so any solution that crosses those bounds outside of OpenStack or outside of what we should be controlling is is a danger area I guess so we would you know we've actually started going down the contrail open contrail route I think it's got a lot of benefits we're a huge juniper shop and there's a lot of integration with with neutron as well although it rips out most of it but there's some compelling arguments to that especially on those edge conditions where where customers want to plum things in and out of OpenStack it would make it a lot easier and it also creates a single dashboard for both for multiple teams to look at it and be able to control so that's a good thing but there's a ton of complexity behind that and a ton of extra operational and operational cost as far as manpower and training and getting used to that and I would say that we just aren't quite there yet with respect to that so we're always looking for solutions that we can deploy tomorrow or three months later or in the next release of OpenStack that really help fundamentally reduce the problems that we have with an OpenStack. One more follow-on comment on that Kyle Mastery at the PTL during the Liberty cycle has reiterated and re-reiterated and re-reiterated that neutron is now a platform it's not a solution it's a place to make solutions so be aware that that you know there is a lot of valid solutions and there will be more valid solutions as time goes on so so don't think of neutron as a solution think of neutron as a platform for networking and people are bringing solutions to it. I actually one more thing on that I actually really like some of the things that ODL is doing and I was trying to set up a project where all using ODL to map our entire network infrastructure in OpenStack which actually I just didn't have time to complete my my feedback on that is it's you have to be a technical expert at a certain point you need to dig in have time to maintain error to spend on that and become the expert because it's not out of the box flip a few switches and go there's a ton of setup time and so that goes back to the simplicity and ease of troubleshooting when something went wrong with ODL or ODL caused some impact on my my OpenStack instance which did happen up front until I learned the bits to flip there's nothing wrong with a lot of these tools it's what your organization is trained for and how much technical ability and time you have to introduce those maybe that's way too practical but another question and then you distributed virtual router DVR DVR is a very overloaded term it turns out so we have we have internal projects that's DVR digital video recording because we're and it's not only that it's cloud DVR it's very overloaded sure so we have to contact switch the short answer is I keep going down that path every release and following that and then I feel like there's not enough operational tooling and neutron to help troubleshoot as it is and now we've just distributed that problem around so I spend enough at least I have one I have I have three network nodes and I know where to go for most of those problems I don't want to have 200 problems faces to go across there are a few bugs in there that I'm following that I just at this point I for us it's not ready there's like two blueprints to to make that even more distributed I don't know if you saw those there were sessions this week on that as well so it's definitely something we're keeping in touch with but I think too if I had to you know put my finger to the wind and in guess yeah I think it'll be two releases for us before we probably are ready to take that plunge it's I think it's going the right direction slower than everybody wants obviously me too but it's the way it is gentlemen in the middle easy OS I don't there's some reference information in the documentation and in the blog post that will talk about the speed ups but we have not done any we have not done any performance evaluations I was happy to get it running so on the plane flight over I don't have anything yet but if I can get it put together in in neutron in our environment in 12 hours that's pretty good go there but another question yes ma'am this or that of the kernel that the Ford auto okay where are we here there we go I would so the best resource that I have is rack space gave a talk at the Paris summit that's called you know taking open v-switch to the ludicrous be different keyword is ludicrous they outline everything in there that that was an amazing so search for this term that's why I put ludicrous in there let's see yeah if you search for this term ludicrous not the pre but just the ludicrous and rack space and open stack you'll probably find the right talk and that's everything that they presented there I found true when we upgraded there was a quick we haven't we haven't tested that or even exercise I can't say we haven't even brought that up certainly haven't tested it we're certainly not I was unfamiliar with the term dpdk until last week well versus versus which versus I think because it's simple to find the pipeline it probably improves it I don't know if it still falls falls apart at scale or not we haven't done that test well got two things on that one is when I this is the reason I put normal use cases on here we're not trying to press for traffic we have where we gate right now on standard neutron is we're using you know to 10 gig bonds so we've got 20 good gig pipe for tenant traffic once we we have some weird interaction that I'm troubleshooting right now between security groups and VX LAN don't ask me why because I don't know yet if you take the straight straight tunnel I can push about eight eight gigs six to eight gig through a pipe which is pretty pretty good I thought it was pretty good I figure I can tweak that a little bit as soon as I add security groups to the mix I'm dropping down to about six gig and so with a reworked security group solution we may get some of that back so that I want to understand how that works I suspect although you know I my plane flight hacking aside over here that OVN will will increase or decrease the overhead of making changes to the system I think fundamentally if you're seeing some problems in OBS that's gonna remain there thank you for your participation and I have one more talk later today I'll give a little shout-out to it's about open-stack trivia I'm slightly better prepared for it and I promise Frisbees so and again if you had a question today and you would like a Frisbee come and find me in an hour we'll take questions up here and thank you guys very much for coming also a time-order cable has another talk coming up I think right right next upstairs I think or no downstairs downstairs using bailing wire to fix your production open-stack