Okay, happy Monday everyone. First off, apologies to everyone who's watching online. I did not bring, forgot the mic. So you have to deal with the microphone that's right here. So that'll be fun when you watch this lecture later. Oh, let's see, midterm next Monday. Practice, you did not require a response, but thanks. Let's see, you'll get your midterms back before then. Sometime this week. And then there'll be a practice midterm out later this week, but we need to cover Lambda calculus stuff so I can give you a practice midterm that actually makes sense. Any questions? Project fives up. Go, go, go. Cool. Okay, so we are going over Lambda calculus. So what we're trying to do right now is to build up some intuition on what does a Lambda expression mean? What does something written in Lambda calculus actually mean? So we saw that what we have here is Lambda x dot plus x one. So if you were to disambiguate this, where would you put parentheses? So let me give me one set of parentheses. Okay, one after the dot and where would you put its corresponding pair? After the one, right? So our first rule is anything from a dot tries to extend as far to the right as possible. Great, what about the next one? Plus and x. Yeah, there should be parentheses after plus and x if we wanna be completely unambiguous. Cool. But the whole beauty of having these disambiguation rules is we don't have to do that explicitly, right? When we see a Lambda expression that is Lambda x dot plus x one, we know exactly how that goes. And so as we talked about on Wednesday, this represents a function that adds one to its argument, right? The way to think about the Lambda expressions, right? Is there an abstraction? You can think of them as functions over some variable x as a parameter and it's returning plus x one. And so if we applied this abstraction and basically added a two after it, but why do we have to add parentheses here? What happened if we did not have these parentheses? Yeah, it would include the two, our disambiguation rules would mean that the two is included as the body, right? So this is not a function application, but here we have Lambda expression, Lambda expression separated by a space, which means this is a function application. So we're actually trying to call a function. And when we do that, we're essentially gonna replace in the body the x with two. And so it would eventually reduce to three. But as we've seen, okay, if we're trying to think about Lambda abstractions, if we're trying to think about abstractions as functions, right, this Lambda x dot plus x one, we don't have a way where we can say Lambda x comma y to say we want a function that takes in two parameters. So why can we define plus if abstractions only accept one parameter? How many parameters should the plus operator or function take in? Two, right? Should take in two parameters and return one thing. So it turns out the result is we're not gonna go in, we're not gonna prove this or anything. It turns out the result that Curing is a technique to translate the evaluation of a function that takes in multiple arguments into a sequence of functions that each take in one argument. And the really cool thing about this is this basically means that a language that can take in multiple parameters for a function is equivalent to a language that can just take in one parameter for a function. And so we can define a function like this. So this is a Lambda expression that basically does addition, will add two parameters. So the way to think about this is so we have Lambda x, so we have one abstraction and the body of this abstraction is this whole body, right? This whole thing here. But the body of that abstraction is Lambda y dot plus xy. So when I call this and I pass in, let's say, the parameter one, what's it going to return when I evaluate this? So first off, what do I do? I mean, we're gonna go into the exact precise steps later. But at a high level, what is this going to return in some sense? What is it gonna return? It's gonna return Lambda y plus one y. So what we're gonna do is replace inside the body here, we're gonna replace all the x's with one. And so that's gonna return Lambda y plus one y. So you can think about this is a function that returned a function, right? So it's a big function. We evaluated it the outside function and it returned us a function and it specified, hey, in my body now, instead of x being some random thing, I know x is one. And so now if, and that returns a function. So this returns a function that adds ones to its argument. So if we apply this again to something like two, it'll finally return three. So we can do this here. We can say, okay, now what happens if I do this? So is this, so first what's gonna happen, right? We know with our disambiguation rules, application is gonna be left associative. So this is going to happen first before the outer one. Everyone agree? Cool, so we do exactly what we did here. We first are gonna replace all x in here with 10. So this is gonna return Lambda y dot plus 10 y. Now we have another application, right? We have a new abstraction, a new function. Now we can replace y with 20. And now we replace y with 20 and we get plus 10 y, which we're adding 10 and 20 together. And now we're going to eventually get 30 if we were to expand that out and do all of that. So super cool way to think about it. So instead of thinking about one function or one abstraction that takes in multiple parameters, we're essentially gonna have nested functions like this. So if anybody's done any Haskell programming, maybe F sharp, I'm not super into F sharp, but you can do this with any kind of function. So even though the functions take in multiple parameters, if you just pass in one argument to a function, you get a new function that takes in one less parameter.  Questions on this? So this will really help you to read Lambda expressions. Right, when we see Lambda x dot Lambda y, you can think of this as a function that takes in two parameters, x and then y. So it'll become more clear as we go through more examples and start cranking through how to actually do these. Questions on this? Cool, Korean, it's awesome. Okay, so now we need to talk about different types. Actually forget, did we talk about bound and free variables yet? I don't think so, no, okay, that's what we're gonna do here. Cool, so we need to make two definitions of when we see a variable, when we see an ID, what kind of variable is it? And the way to think about it is this. It's gonna depend on the exact scope that we think about. So given a Lambda expression, we wanna know if a variable is free, that essentially means it's a global variable. If it's not free, we'll say it's bound and that means it's basically, it maps to an argument of a function. So this gives us a nice way as we'll see when we start computing these things formally about how to actually do these reductions in this computation. But for right now, we're saying a variable is free if it does not appear within the body of an abstraction, so it's not within the body of an abstraction with the meta, that has a meta variable of the same name. So is X free in the Lambda expression Lambda X dot XYZ? Why not? Yeah, so it appears within the body of an abstraction that has the meta variable of the same name. So if you think of this as a function, you can think of X as essentially a local variable. It is bound to the parameter of a function. I mean, local variable maybe is not the right way, but it is the parameter of this function. What about Y? Yeah, so Y is free. Y could be anything. X always, in this thing, X always has to refer to that parameter. It's not gonna change. X always refers to that parameter, but Y is what they're called free. What about this? Is X free there? What does the definition of free variable say? Yeah, so the part of the problem is there's two Xs, right? There's an inner X inside here. Is this X free? No, this X is a bound X. Is this X free? Yes. Right, so this X is inside the body of an abstraction that has a meta variable of the same name. This X is not inside that body, and it's not inside the body of any abstraction, and so therefore that X is free. So if you had to do something like circle or identify free variables, you would circle this X on the right, but not this X on the left. And depending on what one is, right? We don't actually know what one is, so maybe there's some free variables in there. What about Z? Is Z free? No? So how would you read this as a function that does what? That takes in three parameters, and what does it do with those parameters? Yeah, kind of, so you think of X is the leftmost, so that's gonna be the first parameter. The second parameter is going to be Y, and the third parameter is going to be Z. So you can think of, all it does is rearrange the order of those three, right? It was X, Y, Z, and that's gonna flip it and do Z, Y, then X. So it's actually just a cool way of kind of moving things around. Cool, what about this? Is X free in here? I mean, here, let's look at this. Sorry, and I also don't have my handy-dandy drawing thing, so I'll have to go with this. What I mean is if we have this as a lambda expression, and let's say we also have after this, let's say one, two, and three, well, it's kind of tricky, but let's say we first apply it to one, and that'll return, let's say this with a one here, and then I apply it to two, and that'll return the middle with a two here, and then I apply this to three, then it'll do three, two, one. So even though the parameters were in order one, two, three, the end result is three, two, one, what it's going to return. Now, whatever those things do, we don't know. They could be any arbitrary functions or nothing, they could just be variables, but we know that they're gonna happen in that order. So if you think about it, if you wrote it like this, one, two, three, right, the one and the two are grouped together first, well, actually technically it'd be like this, like this, and then this, and then this, so when we return, we return something that looks like this, but I've kind of shuffled those orders around. Anyways, yes? Yes, good question. So if I had done, which why I didn't do that at first, but if I had done like this, one, two, three, which we know is actually like this, right? So if I had done this, now I'm replacing in this body X with my first parameter, which is one, two, three, cool. Okay, so back to our original question, is X free in this lambda expression? And we can see that the light's working, it's no longer blue, that's nice. Was that for yes, the thing is no longer blue or yes, that X is free? It's free, right? So inside this body, remember this body only extends all the way to the end of this right parenthesis, so this is the body of this abstraction, right? So if there was an X to occur in this body, it would not be free, it would be bound here. In this other abstraction, here we have X, but the meta variable is Y, and so this X is a free variable. Cool, questions on this? Free, free variables? Yes, yes, so if we were to reduce this, right? So the question is, A, is this a valid lambda expression? Does this correspond to our syntax for lambda expressions? Yes, right? Then the next question is, okay, what type is it? It's an, as we'll see, it's an application, and the thing on the left of the application is an abstraction, which means we can reduce it as we saw, we haven't gone through and gone through what exactly those steps are, but yeah, what we would do is we would return Z foo. So this X, because it's not used anywhere, basically consumes this argument, gets rid of it, and then we have Z foo as the return here. All right, so the rules, so what we're gonna do is we're gonna go through, and just like we had rules for syntax of the different types of cases, right? We wanna be exhaustive when we have a definition of what does it actually mean to be a free variable, and as we'll see, these are gonna mirror very similarly to our rules of our syntax. So X is free in some lambda expression E if X is equal to E. So if we just have the expression X, we know that X is free, right? It's not in any abstraction. We know that X by itself is free, agree? So what about the case where the expression is a lambda is an abstraction, but the abstraction variable is not Y, or is not the same thing as the variable we're interested in. So let's say this expression is in the form lambdaY dot E1, or E1 is the body of this abstraction. So X will be free if X is free in E1. So what does this in essence say that we can do? Yeah, if the meta variable isn't the same name, we just ignore and return whatever the body says. So this would be if E1 is X, then we know from here E1, X is free, and so we know that because this is lambdaY that doesn't change anything, right? Because we're not bound, X is still free no matter, well, if this meta variable is not the same thing as X. But it could be anything, right? Any meta variable name, just as long as it's not the same thing as X. Cool? So then what's the other case in this? If it's equal to X, well, okay. Damn. All right, but if it is equal to X, then what should the answer be? Not free, right? It can't be free because there's a lambdaX. But here we're defining when X is free, right? So, oh yeah, so we have the different cases. We have application, right? So a lambda expression could be E1 space E2. Does that affect our free variable decisions? So if we have an application, we say if X is free in E1, does that mean X is free in the entire expression? No? Yeah, we've already found that X is free in one side of the application, right? So if the left side is X and the right side is Y, right? This is saying that X is free in the left side. Therefore, shouldn't X be free in the entire expression? Right, just like we saw on this last one here, right? X is free on the right of the application, so that means overall X is free, right? We just need to find one instance. That's the way to think about that. Cool. So if it's free on the left, then X is free. If it's free on the right, X is free, right? This is pretty straightforward. So basically what this means is an abstraction doesn't change anything in terms of free variables, right? Because free variables all we care about are abstractions. And that's all we have. So we could do parentheses, but parentheses don't change things either. So basically what this is saying is, right, we can only say yes if inside of an abstraction, the meta variable is not the same thing as the variable that we're interested in. Otherwise, we just say, hey, then it's not free, right? So this is just formal definitions of our intuition of what it means to be a free variable. Any questions on this? Yes. Does that match any of our cases? It does, which case? If it's lambda X dot something, yes. So if X is equal to Y, if it's not equal to Y, as long as X is free in E1, then it's free overall, right? What this line says is, hey, if the meta variable is not equal to X, if it's free in E1 in the body of the abstraction, then it's free overall. If Y equals X, does that match any of these cases? No, so it's not. So what we're defining here is when X is free. So we'd go through all these things and say, okay, the expression is not X, the expression is not lambda X dot E1. The expression is not an application, and so it'd be false. In this case, if it's exactly X, so the question is, right? So is this a lambda expression? Yeah. And so it matches this case, right? Of just X. So we'd say, is X free in this expression? Yeah. Is X free in this expression? No, it doesn't exist, right? It's not free, there is no free X. There's no free X, there's no bound X, right? And so that's because it doesn't match any of these conditions. We'd say no, it doesn't match any of these conditions, right? This is the case where it's just an ID and it's exactly what we're looking for. I don't care, I don't know. So what the second rule basically says is hey, if there's a lambda X in here, then I don't care X, Y, Z, A, B, C. I don't care about what's in this body. I know this is not, X cannot be free in here, right? Because there's a meta variable of the name X. It's not possible for that to be in there. But if this is part of an abstraction, or if this is, sorry, if this is part of an application, then yeah, I have a free X here. This X here is free, okay. So using this as X free here, yeah. And we could break this down by each of the rules and eventually we get to an X that is not inside of an abstraction. And we'd say yes. What about in this? And this, oh, no, right? This is in the body of the abstraction. So we never go into that abstraction, cool. So combinators, basically this is a way for us to talk about essentially functions. So combinators are, of course, another term, well, like in many things in Lambda-Cogulus, there's like a direct analogy to programming terms the way I think about it. So a combinator is a function, or is, okay, the technical definition, it's an, is that right, yeah. Expression is a combinator if it does not have any free variables. That means no variables are free. So you can think of it as a function definition that does what? So what is the free variable analogy to regular functions in programming languages? Or not programs, regular programs. Global variables. So if you define a function with no global variables, then what is it? I mean, there's not a specific term for it, but how would you, you know, it's actually more of a mathematical function, right? Its output is only the, is only ever based on its input, right? And actually, hopefully some of you have learned maybe that lesson in a painful way. The more you can minimize your usage of global variables, the easier it is to reason about your own programs. Right? Because you don't have to worry about some function over here modifying some global variable in a weird way that over here, then breaks your code into completely unrelated areas. If every function has just returned something based on its input, then it becomes a lot easier to test each function in isolation. So you can be sure that each function actually works. This is something you should think about a lot more and try to apply your own programming. So in Lambda Cagulus, we have a way to talk about this because we defined free variables. So we'll say that any expression is a combinator if it does not have any free variables. So we can ask questions like, is this expression a combinator? Yeah, no free variables, right? All variables inside here, all IDs are not free. What about x dot x? Is it an interesting function? It's the identity function, it just returns whatever it's passed in basically. What about Lambda z, Lambda x dot x y z? Why not? Right, y is free, cool. This is gonna come up, this is gonna be actually really interesting. There's a whole, we'll see that most of the interesting functions we wanna write, like when we start doing Boolean arithmetic and start doing ands and ors and true and false, those are all gonna be defined in terms of combinators. Similarly with math, we're gonna find plus and minus in terms of combinators and also all of the integers in terms of combinators. And there's a super interesting combinator that we'll study later, but I'll leave some suspense. So the opposite of a free variable is then as we've been alluding to a bound variable, right? So you can think of this as the parameter to a function. That variable when you use it, you know exactly where it comes from, right? It comes from that function definition. So the easy way to think about this, if it's not free, it must be bound. If it does not exist in your expression, then you don't care. The question though is just like scoping rules, right? When we talked about scoping rules, we always wanted to say, okay, we see that this variable is used. What declaration does it map to? And so that's the question we also wanna ask here is, it's bound by what abstraction? So we know that this variable is bound. It means it must be the parameter to some function, but which function is it? So what would be, so let's say we have a case of, man this copy paste is awesome, okay. So where would this x do you think would be bound by? Which one? Yeah, the second function definition, y. Why is that natural? Maybe, so where does this x bound to? The outer x, and this x is bound where? Here, right? So what about scoping rules? Does this have any analog to scoping rules? What about scoping rules? Yeah, we look closely, right? We look and see, when we're trying to map a variable usage to its declaration, we first look in the local scope, and then we look one scope out, and then we look one scope out, and then we look one scope out, and then finally we look in the global scope, right? That's exactly what we're doing here. So this is something that you've been doing forever. You just have to try to map your ideas of programming into this weird-looking parentheses and lambda language, right? So here I see this x, I need to look and see what abstraction is this bound to. Well, I look at the innermost abstraction, and I say it doesn't have the name x. If it does, then that's where it's bound to, and if not, then look out in its abstraction. Is that named after? So if not, keep looking out until, if I find one, then that's the abstraction, otherwise I say it's not bound variable, it's a free variable. So if I change this to y, right, now which x is this bound to? Is it a free variable or a, right? This x is bound here, cool. So now going back to our, building up our intuition about what these things actually mean. If I put one in here and I wanted to try to reduce this, what happens? What does this return? What should I do, like this? Should I change this to one, right? So remember that what we talked about informally was, hey, let's replace every x inside the body with this parameter, right? But really that's why we were talking about it imprecisely, right? Because we haven't developed the language to talk about it very precisely. We don't wanna replace every instance of x in here. What do we wanna replace? Every x that's bound to this x, right? We wanna replace all of these essentially uses of this function parameter with the parameter that we passed in, right? This x is bound to this x. It basically has nothing to do with the outer x, right? It completely shadows it. Just like when you have a variable that you've used inside of a smaller scope, if it has the same name as something in the outer scope, it shadows it and you don't care what that is or what it refers to, you can never refer to it. Yes, I wouldn't phrase it like that. We'll see exactly the way to think about it is if you take off one layer of this abstraction of this outer x, so if I look at just the body by itself, are there any free x's in this? Yeah, there's a free x, so we replace all the free x's with the parameter, right? You gotta think, it's actually a very cool way to define it because all the free x, if you essentially strip off, so now are there any free x's? No, so it's kinda cool. If I strip off this function definition by one, now any free x's in here must have been bound to the outermost x, and so I'm gonna replace all of those free x's with one, with whatever the parameter is. Does that make sense? Cool, we're getting a little bit ahead of ourselves, but I think it's good. So, this is what we're looking at. We wanted to find, remember free variables are really easy, you just say yes. They don't refer to anything, they're free, right? But once you're bound to an abstraction, you wanna know which function are you bound to, right? Which function meta variable are you bound to? So, just like before, we now have rules to do that. And this is exactly what this says, so this is gonna be just what we talked about right now with essentially stripping out the body of the abstraction and actually seeing who's free. This is the same logic here, so we say, if an occurrence of x is free in some lambda expression, so there's an expression, you have a free x in there, then if you throw a lambda x around it, it's gonna be bound to that lambda x, right? So, this is the way to think about this, is exactly like this. So, if you, I don't have a good way to do this, maybe, well I can't jump up there, maybe I can. But basically the idea is, if you have an abstraction, if you take away this outermost, what was that so much? If you take away the outermost abstraction, any free x's are bound to that lambda expression. Just like here, if I take any x, y, z, right? X, y, and z are both free. But if I put a lambda y around this, then now y is bound and it's bound to this abstraction. Now, is there a lambda y that is free in this whole expression? No, therefore if I add a lambda y dot y here, this y is not bound to the outer y, it's bound to the inner y, because it was not free. Does that make sense? What is free inside here? X and z, so let's say, now if I take this, and I throw a lambda z around it, is z bound? Yeah, and what abstraction is it bound to? This one, z is no longer bound, so if I throw now a lambda z around there, that z is not bound here, it's bound to the inner z. Does that make sense? It's kind of weird defining it in terms of throwing a function around a body. Yes, what example? Which example? This one right here? This? Yeah, so the idea is inside here, so if we take this by itself, right? We say, are there any free x's? Yes. Then if we put a lambda x around that expression, then where is that free x bound? To this abstraction that we just made. The other way to think about that is if you have a lambda abstraction and you're trying to say, what is bound to this x? Well, it's easy to figure out. You just get rid of it and say, what's free? What, sorry, not what is free. What x's are free in here? Those must be bound to that abstraction. So that's the way to think about it. You can think about covering it up with your hands if I can't really do it digitally. Okay, so this is saying that if we have an x, this just means whatever we throw after it doesn't matter, right? So this is saying if an occurrence of x is bound by a particular lambda x in E, so inside our expression there's a lambda x that binds this x, then it doesn't matter what lambda z dot expression, lambda x, lambda a, lambda beta, just like we saw, we can keep adding abstractions. It doesn't change that binding, right? This basically says the binding happens the closest it can. So even if z is equal to x, just as we saw, we can keep adding lambda x's and the innermost lambda x is going to bind that x. So by this we can see that this x is bound to this lambda x, and it doesn't matter if we add more lambda x's, it does not change that binding. And this just says that if we have an abstract, if we have function application, E1, E2, E2, E1, it doesn't change things, right? All we care about is function abstractions. Cool, let's go through some examples. So what's bound to what? So what is this x bound to? This x, what about this x? Hmm, this y, here? This x is bound here? Here? What about this y? The lambda y, what about this z? Free, what about this y? Lambda y, what about this x? So this x, what about this x? And the y? Also free, cool. So let's see if this is right. All the free variables should be underlined, all the bold variables refer to the same x and all the italicized variables refer to the same y. I think that's what you said. Let's say it is. x is a free variable in this expression, y is a free variable, and z is a free variable, exactly. Question is which one, right, I mean? That's another thing. Yes, as long as it's free anywhere, when asking is x free in this lambda expression, you just need one. What if I said is this x free? The answer would clearly be no. Okay, so this x is bound where? This outer x, the y is bound here, what about this x? Free, and this z? Lambda z, cool. I think, what do we have here? So this x? Outer this x, that's the meta variable this x? Inner, what about the z? Free. Cool, sweet. Okay, so now we're starting to get a feel for how to call functions, but there'll be some, as we're doing these substitutions, we haven't got there, but there are definite ways where we can mess up substitution. So actually, let's start to think about that, because I think that'll make things easier. So let's say I have, okay, is it gonna be confusing to have all these? Let me say we have lambda y in here, and then we have x, y, something like this, cool. So pretty easy function, right? So let's say my parameters are one, no, let's say they are, so my parameters are, if my parameters are one and two, what is this going to return? One and two, right? So the x is gonna be substituted for, this x is gonna be substituted for one, this y will then be substituted for two once we reduce that, right? And so that will result in one and two. So by that same logic, what should this give us? Y, x, what does it give us? So let's substitute, just like we've been talking about, we're gonna substitute this x for what? The y, and so we're gonna return this, substituting that x for y, right? And the important thing to remember is this is its own thing, and then we have the x. So what is this return? X, x, so what was the problem? Yeah, so with this y, what is this y? Is it free, is it bound? What about this y? And then when I substitute and change this x for here, what happens now to this y? It's bound, it's no longer free, right? It used to be free, but now we stuck it in there and because there's this abstraction around y and it's just that we happen to be using the same names, now we've messed everything up, right? So this is why we start off with that really simple intuition about, yeah, yeah, you just substitute things in there, but the reality is a bit more complicated. And so to help us deal with this, I want you to think about what's the difference between these two things? So this is a function that does what? It returns itself, so if I call this with one, what is it gonna return?  If I call it with, I don't know, ABC, it'll return ABC, right? So what is this function return? It's first parameter, exactly. What is the difference between this function and this is kind of a philosophical question? And this function, are they the same? Let me tell you more, they're not the same. Yeah, they're not texturally the same, right? If you take this one string and then you look at this string, are they the same? No, they're clearly different. But if you give them the same input, will they give you the same output? And is that true for every single input that you could possibly give? Are you sure? That'd make you doubt your reality? Then we'll give what? Will it ever give Y or X as output? So if I pass in X for both of them, what are they each gonna return? What about Y? What's the only difference here? We've renamed the meta variable from X to Y. And so this is what we need to talk about now before we really get into how we do these substitutions and actually perform this calculation, is because we wanna talk about, well, what does it mean for two functions to be equivalent? Right? Just like we saw, is lambda Y dot Y, is that the same thing as lambda X dot X? I mean, it's clearly not exactly the same thing. But conceptually, you could think of the input spaces for all these functions will map to the same outputs. What about this? Lambda X dot XY and lambda Y dot Y X. So there's still not the exact same string. So what was changed? So was the function parameters renamed? The meta variable, like in the first example? Yes. But are they the same function? So you're shaking your head, why? They have different free variables, right? They each refer to different global variables, right? So if you passed in, let's say, right, we have this function here. We pass one in, what is it gonna return? One Y. And then I have this function here. And I pass in one, what is it gonna return? Is one Y and one X the same thing? No, they could be, but are they always? The problem is it depends on what X and Y is, right? So here we have two functions and even though we've changed the parameter names, like we did in the first case, we've changed the global variables that they rely on. So then can we say that those are equivalent? Why no? They return different things. They'd only be equivalent if X and Y were always the same thing. All right, what about lambda X dot X and lambda X dot X? Very clearly they're the same. They are literally the exact same string. They're the same function. If they're not the same function, something's horribly wrong. So we're gonna define a notion to capture, hey, in this case, these two functions are what we're gonna call alpha equivalent. They're basically the same function just with renamed parameter names, but these functions are different functions, right? These functions are not the same function because the free variables have changed. So the way we're gonna do this is we're gonna, I'm gonna just go through this real quick since we've already gone through all the intuition. We're gonna use this equal and the alpha operators. We're gonna say these two lambda expressions are alpha equivalent. If we can replace all the meta variables of one and get the other, but we need a way to say that, to replace all the meta variables of one and get another. So we need a way to rename variables in an expression. So we're gonna see that we have to do that and we'll see that on Wednesday.


Today is a fun day. We are surprising my uncle for his 50th birthday, so I'm with my aunt Hi And I'm with my cousin Joshua And we are having to park like kind of far away from his house because he got sent out to pick up the food and so We're gonna who's that just I think that was on july So we're parking away from this house.  It's like an ant leader thing. Oh zaza zot zot zot.  Yeah This is the lady that introduced me to sushi and now I'm have a very expensive addiction So What kind of a champagne thing is that? That's weird I understand why you're having trouble now It's just a weird champagne bottle like top Like i'm confused by it Does it say anything about how to You probably just have to like keel it Oh, yeah, what does it say any strength anymore? I have to get some straw bars Yeah, I don't know if that that would work You've never seen this classic thing before Who's turning 50? It doesn't have a that bite fit Yeah, like that alpha j scan Already getting dirty I just got my nails done And I chose white And I do not think about how we're going to hire Deep digging hands You've got to do what you got to do. So we're going to see how this thing got they had gloves like I really didn't think they would I know very prepared Okay, you cannot see your face landed You can Landon come on such a smart girl who's ready for cake


Hello, KubeCon, for this last day on Friday. My name is Thomas. I'm one of the creators and maintainers of Cilium. And Cilium is one of the popular Kubernetes networking implementations. And based on what we have learned creating Cilium and working with many of you, I would like to share how to survive day two and how to troubleshoot Kubernetes networking. So a bit of context, I'm one of the maintainers of Cilium. So what I'm sharing today, of course, has a bit of Cilium context because I've been mostly working with Cilium the past few years. But the exercises or the best practices and how to approach this is, of course, not at all specific to Cilium and can be applied regardless of what Kubernetes networking implementation you are using. For those who have never heard about Cilium, Cilium is a CNI plug-in among other things. It is a CNCF project at incubation stage. But Cilium does not only provide networking or CNI functionality, which is what we will focus on today. It also provides service mesh and Hubble. And a lot of what we will hear about today is made possible by Hubble or tools similar to Hubble. Hubble is the observability layer of Cilium. Then we also have in the Cilium family Tetragon, which is providing runtime security and security observability. All aspects of Cilium are being done using a technology called EBPF. And that's actually what's enabling a lot of the observability that we will see today, which will assist you in troubleshooting. For some, in particular, in the last part of the talk, we'll also look at layer 7 latency or HDP latency. Some of that work is done through Envoy, the Envoy proxy, which is also a CNCF project. So let's jump into Kubernetes networking or how some people call it the dark side of Kubernetes. How many of you are familiar with the core concepts of Kubernetes networking? Excellent. Almost everybody. So this will probably be a repetition for most of you. But if you've never heard about or have never seen how Kubernetes networking works, this may help you understand the rest of this talk. First of all, Kubernetes networking is very simple and basic. All pods have an IP address, which means every individual pod has an IP address. They can directly talk to each other. There is no specific network topology required. Kubernetes makes a simple assumption that we have a flat, so-called flat layer 3 network. Pods have IPs, and thus pods can talk to each other. This is sometimes implemented differently depending on whether you're running in the cloud or whether you're running on-prem. You may be running BGP on-prem. You might be using the Cloud Provider SDN as you're running in the cloud, or you might be running in overlay. Doesn't matter, all of these implementation principles implement this basic foundation. Typically, this is not true for all Kubernetes implementations, but typically, every node has a so-called pod-sider range, which means pods on a particular node will have IPs assigned to that node. So that's a particular set of IPs, and all pods on that node will have an IP out of that range. Some implementations do this differently and allocate a unique IP. But there is a concept, you may see this, in Kubernetes called the pod-sider range or per-node pod-sider range. Kubernetes uses services for load balancing. You're probably using this all the time. This is the load balancing layer of Kubernetes. This is what's allowing us to have multiple replicas of a pod, address a single service name or single cluster IP, and Kubernetes will load balance to any of those replicas. We'll look into some troubleshooting there as well. And then, of course, Kubernetes uses DNS for service discovery. So you can address a service by its name using DNS instead of hard coding IP addresses into the application. And then as the last principle of Kubernetes networking, there is network policy. Network policy is what implements the segmentation. So network policy is what allows you to define which pods can talk to which other pods. You could be doing segmentation maybe on namespace level. So maybe you want to allow within a namespace, you want to allow communication, but the cross namespaces communication should not be allowed, or you can even do this at the pod level and essentially say that only certain pods can talk to certain other pods. This is called Kubernetes network policy. And as we will see in the demo later on, plays a huge role in some of the troubleshooting. This slide looked very simple, right? It looks beautiful. Kubernetes networking is very, very basic. As you learn Kubernetes, as you run Kubernetes, I did this slide back in 2021. That's a little bit more how it looks like in practice, right? So you have an overall goal, which is to forward together, to do network forwarding. You have IP tables somewhere. You have Q-proxy using IP tables. You have an application team trying to schedule 5,000 services, really putting stress on Q-proxy. You have the lifeness probes succeeding. So the applications seem like they're up and running because they're not aware of the actual network underneath, or the applications are just reporting, hey, I'm healthy, I'm healthy, I'm healthy, but they're not at all because they're actually not even reachable. And you may have a platform team that is completely ignoring the crash loop backoffing coordinate spots. That may be closer to the reality sometimes. So this talk is giving you tools on how you can look into this and figure out where the problem actually is. So let's dive one step deeper into Kubernetes and see how this is actually implemented. Because as we troubleshoot networking, it helps to understand the base concepts of how the CNI actually works in the Kubernetes cluster. And this is true for all CNIs. It's a core concept of Kubernetes. So we have multiple nodes, and we have parts running on the nodes. And of course, containers running inside them. We then have a network CNI level, or layer, which typically runs as an agent or as a demon set on all the nodes. And the CNI then essentially spans the network plane, which allows the parts to talk to each other. Kubernetes itself does not have a built-in networking layer. It requires there's a default CNI, KubeNet, but it requires the CNI plugin to actually allow parts to talk, not only across nodes, but even inside of the node itself. And then we have KubeProxy, which is the default implementation to implement Kubernetes services and as well, Coordiners, which is not in the picture here. When we implement this with Cilium, this looks very similar. We have Cilium that provides the networking data path. And then we have EBPF, which is doing the implementation of that and is actually causing the package to be forward or net or policy rules to be implemented. The last concept we need to understand before we can get a little bit more hands on is Hubble. Hubble is based on top of Cilium and provides observability. There is no demand. There's no requirement for a CNI to actually provide observability to stand that only demands for parts to be able to connect to each other. And everything else is optional. So even net or policy implementation is actually optional. So a CNI can provide as little as just pure network connectivity. With Hubble, we are essentially providing additional observability tooling in the form of Flowlocks, who is talking to whom, and metrics. It's essentially a TCP dump for Kubernetes. Because in the early days of Kubernetes, many of us have been executing into a node or into a pod and literally running TCP dump to somehow figure out what is going on on the network. And for those of you who have never heard about TCP dump, it's a 25-year plus-year-old tool, which has obviously never been intended to be used for an environment like Kubernetes. Hubble has native integration with Prometheus and Grafana. So you are not looking at you can, but you don't have to look at the terminal. You can look at dashboards and at Prometheus metrics. Let's look at a couple of examples, and we'll jump into a live demo afterwards. So we can, for example, look at Grafana-based panels. How many packets are we forwarding? What is the drop rate? So how many packets? How many network packets are getting dropped by the network layer? What is the total amount of traffic being forwarded? Or we can even create dashboards to display how much cross-region, cross-AC network traffic do I have in my cloud? Because that's usually what your cloud provider charges a little bit extra for. That's the Grafana view. We also have a Hubble UI service map where you can see all the services that are running and how they are talking to each other. So here we actually see the individual network connections and the API calls for application protocols that we understand. These are HTTP, GRPC, Kafka, Cassandra. So we're not only showing you who is talking to whom, we can even show you with Hubble what type of API calls they are making or what is the request response latency for a GRPC call. And on the lower part, you actually see the live flow data. This service map is being based on because what you're seeing here on the screen is derived completely transparently. This is not requiring changes in the application. It's essentially Hubble running transparently on the nodes and extracting all the connectivity data transparently. And then we can calculate what we call the service map from that data. And the raw data is what you see in the lower part of the screen. As a last concept, the Kubernetes service implementation and this is if we would go into a lot of details, several types of services for the purpose of this talk, I will keep it simple and talk about cluster IP. So this is the ability to expose multiple pod replicas via a single IP, a single cluster IP, which means that instead of being aware of the potentially hundreds or even thousands of pod replicas to talk to, I can talk to a single cluster IP. And this cluster IP will then get low balance to any of the hundreds or thousands of pod replicas. And of course, you do not want to talk specifically to an IP. So you will, Kubernetes allocates a service name for you and makes that available as a DNS name via coordinates. So essentially your application talks to the service app name and Kubernetes takes care of the rest for you. Let's jump in into the first demo and actually show you how to troubleshoot some problems. And I figured let's start simple and use a very basic network policy example because what could possibly go wrong there, right? In this simple example, we will have a frontend and a backend pod and we'll do a network policy that will look something like this. So this is a Kubernetes network policy. And this policy, for those of you have used this before, this will look very, very simple. We're allowing from the frontend pod to talk egress, so outgoing egress, when we are networking people we'll call egress the outgoing side and ingress the incoming side. So we are creating a policy that the frontend pod is allowed to talk to the backend pod. And does somebody already see the problem with this? If so, yes, I see one hand, excellent, right? But it looks very simple like what could go wrong. So let's actually try this out. So I have this running right here. So the pods are up and running and they seem to be pretty happy. Is it big enough? I think so. They're up and running, right? So this is probably good. But actually under the hood, this is not good at all. And it's pretty, pretty hard to actually even see this because as I mentioned, the health check here, the application is reporting just fine, the application is up and running. In this case, the application, they may actually be complaining, hey, my app team, my app is not really working even though it's up and running, what is going on? So let's actually start looking below the hood what we can see on the network observability side. So I'm swapping over to the Grafana dashboard of Hubble UI and we'll start out with the overall view. This is the cluster wide view of everything that is going on in this cluster. Here we see the total amount of traffic being forward. We see the total of endpoints. That's the total of parts running in these clusters. We see the number of nodes. We see that we have no unreachable psyllium nodes. We see that we have no warnings or errors being reported by any of the agents. We do see a bunch of DNS errors though. That's probably, so if we zoom in here. That's probably something we should be looking into. So we have quite a few DNS errors that are ongoing. We also see that we have policies loaded. So we have 16 policy loaded. We can look at the enforcement status and we see we have 53 parts which have network policy enforcement enabled and we have 61 parts which have no network policy enforcement enabled at all. We can go forward and actually see we have some missing DNS responses. We can look into the connection tracking table and so on, a lot of information. In this case, I actually know that something is wrong potentially with this application, front end and back end and this is exposed into a namespace that I'm known. So I'll go over here and actually look into this KubeCon simple namespace. This is where front end and back end is running in. And it's very interesting. We can see there's some flows being forwarded. We can scroll down. And here we go, we see network policy drops, right? We see a constant rate of network policy drops from the front end. We can go deeper and actually look at well, where are these network policy drops? Where are these packets attempting to go? We can zoom in here and we see, huh, this is going to the Kube DNS part. So what is wrong? At this point, we know that something is being dropped. So let's figure out what's actually being dropped. So I will go back into my terminal here and I will use the Hubble Observe CLI command. So this is the CLI that will allow me to query Hubble. I will run that and it will show me all the network drops and all the forwarded flows that is happening in this namespace because I run it with the dash and KubeCon simple namespace filter. And yes, indeed, we are seeing drops from the front end part to the Kube DNS part. So we've troubleshooted it down from the high level overview where we saw we have some policy drops to the namespace view where we actually saw this was going to Kube DNS. And then with Hubble CLI, we were able to see the actual real drops. So we go back to the slides. It was indeed DNS as often the case for many incidents, right? Even though it was actually not obvious at all. I think a lot of application teams will inject in our policy, they will not understand that or may not understand that I also need to allow to Kube DNS. I need DNS for my service. So let's go to the Hubble UI. This is the service map with the view on just that namespace. And we see in fact that we only have communication from the front end to Kube DNS. And we actually see in the lower part, it's a bit small maybe, but these are all dropped flows. So this is all been dropped. Could even click on one and it would tell me the drop reason is policy denied. So let's actually fix this. So let's allow core DNS. So I have a policy, allow DNS. See if that works. So I created this policy, allow DNS. So this is the policy which on top of the policy that already have now allows to Kube DNS. And bam, we can go back into the service map and we now see not only connectivity to Kube DNS but also to the back end service. And we now see new forwarded flows in green. Those are allowed flows. So we fixed our problem. Now, how did I get to this policy to this allow Kube DNS policy? Of course, you can ask chat GPT today, right? If you don't know how to use that yet, you can also use what we have, what we call the network policy editor. This is available for everybody, editor.networkpolicy.io, it's free. You can use it and it will visualize network policies for you. And right now I've loaded the original policy and based on the graphic you see in the top, you can actually see that, yes indeed we are allowing to the back end what we are not allowing to Kube DNS. You can see the arrow is actually red. I can now quickly allow this, go in here, allow rule, and it has now extended our existing policy to allow DNS. And you can see the new YAML that was added below which will actually allow Kube DNS. So if you don't know how to get to the YAML yourself, network policy editor can do this for you or even in the visualization you can see the problem straight away. If you do not know the namespace specifically, where the application is running, we have what's called a policy verdict dashboard. This is this one, cluster wide and we see a whole bunch of drops going on. So you'll probably need to look into, well, what are the drops I care about? So there's actually a namespace level view down here where we can indeed see that we have drops from the front end, Kube simple part into the Kube system, Kube DNS part. So this is another way if you're not sure which application is even affected, you can go in here and see all the drops by name, filtered by namespace across the cluster. So this is how the policy looks like fixed. We not only need to allow to the back end, we also need to allow to the Kube DNS or Kube DNS part. So more DNS because DNS is really often the problem. We looked at part to part. Next example, we're looking at something that's a little bit more advanced. So as we have learned, Kube DNS DNS is used for service discovery. It's usually implemented by core DNS, but that's actually not mandatory. You can of course use a different DNS implementation and it looks simple. It is not always simple. So let's look at how to troubleshoot something like this. And this is a simple part in another namespace that simply tries to reach out or intends to reach out to google.com. So let me go over and switch tab. This is same cluster, but I have just one pod in one namespace running. Then is DNS who is doing curl to google.com. And let's say the application team is actually complaining, hey, this is not working. What is going on? Let's go back into the dashboard and I have the demo DNS. So we've seen this view before and we can see that the dashboard view of how the DNS layer of Kubernetes is doing in general. So we see about 30 DNS requests per second and we see the top 10 DNS. I can make this a bit bigger. We can see the top 10 DNS queries. We can see we're looking up elastic search, core API and so on. Then I also see a bunch of DNS errors, right? So we're seeing quite a few of them and one of them is from my Dennis DNS pod. So these are like the rate of DNS errors that this DNS or this pod is actually experiencing. We can then go in and actually look at what are the specific queries that are making that are not successful and we see down here that yes, we're trying to look up or curl Google but the app has spelled Google wrong and was using zeros instead of O's. Of course, that's the dashboard view. We can go back and actually look at the observe view as well. So I go back and I run Hubble observe. This time I'm specifying the namespace debug DNS which is where my application is running in and here we actually see, yes, we see the communication to Qtns that this is UDP and then we see the actual DNS requests and responses. We see here that the pod is actually attempting to resolve Google with zeros for both IPv6 and IPv4. I can then even dive deeper and say I actually only want the layer seven information like this and now it will only show me the resolution paths. Like what is the DNS resolution that is going on inside of this namespace? So in here I can quickly with the dashboard here look at is my cluster are there pods which are experiencing DNS resolutions and as I've identified those pods I can use Hubble CLI to go in and actually find out what is specifically going on, what are they trying to look up and so on. Last example, debugging service latency. Service latency is let's say you have deployed an application and the application is not performing as fast as it should be. How are you troubleshooting this? How are you even identifying this? And there is a standard for this or a best practice for this called golden signal dashboard that's been invented or written down by the Google SRE team. You can see the information at the bottom. It's actually documented pretty well. It's a standard way of monitoring your infrastructure specifically for cases where you're running a service that's publicly available and the famous four golden signals that matter in this case are latency, traffic or throughput, errors and saturation and we'll look at what that actually means and why that's useful. So I go over here and I open up the third demo. This is the Hubble dashboard for golden signal dash for the four golden signals. We see at the top the request rate, how many requests per second are actually coming in. We're then seeing the request duration and in this case this is HTTP. So this is actually the HTTP request to response latency. We can see P50, P95 and P99. These are essentially the tails. So P50 is the worst half of the latency number. P95 is the worst 5%.  And what really matters is P99 or P95 because on average it often looks good but then for some customers the experience could be really, really bad. So you really want to not only monitor an average, you want to monitor P99 as well. And it's interesting that we can actually see significant peaks here. So the average is actually pretty good. If you would look at P50, that's the green line. It's all the way at the bottom. So if you only look at the average even of the worst half, everybody's like, yay, happy, like it's almost zero. That's probably really good. But if you look at P99, it's sometimes up to two seconds. So some requests going into this service have actually experienced a latency of two seconds before they got back a response. That's great. Now we understand that there is a problem in terms of latency. We can also understand that there's actually some problems in terms of errors being returned which is the second column. We can see this, this is the error rate. So this is the rate of HTTP errors that are being returned by the application. So any sort of HTTP 500 type code. So one source of problem can be the request taking very long so the service being slow. And another common problem is that the service is crashing and returning like an HTTP 500 error code. Both are very meaningful and important to monitor. That's great. Now we know that there are some problems. In order to now actually debug this, we need to correlate this with saturation because we need to understand what a latency is because there is limited availability of resources and that's often CPU. So we actually have the HTTP request duration and the CPU usage right next to each other. So I can actually go in here and see whether a spike, I can go in and actually zoom in here and just let's just look at this spike. I can look whether this spike in latency is caused because there was a spike in CPU usage on that note. And I can do that for both the source and the destination. So if I have two parts running in the cluster, I can measure the latency and I will see the CPU consumption on the source node and on the destination node. So I can even understand if the latency is bad on the destination side, whether that was caused by limited availability of CPU on the source node side. This is allowing to monitor applications broadly, cluster-wide to figure out is my service up and running, is it returning errors and how fast is my service actually running? Like how quick is the app functioning and so on. And again, all of this is done using transparent extraction of observability data. So this is not using application instrumentation. We're now looking at the Grafano dashboard of this, but all of this data is also available as open telemetry metrics and traces as well. So you can also visualize this in other tooling that is open telemetry capable. And with that, I think we have a couple of minutes left for questions. This was like a quick overview. Well, please, thank you. If you want to learn more, cilium.io, where you find information about cilium and hobble, or if you want to ask me questions afterwards, feel free to reach out on Twitter. Now I think there are mics left and right in case you want to ask questions, but I also know that it is lunchtime. This side. Hello, just one simple question and fast, I hope. So, okay, let's say we would like to use cilium, but we use OpenShift now. And OpenShift has, now we don't use OpenShift CNI, but OpenShift OVN. Do you know if they are compatible? I mean, can we add cilium over OpenShift and don't break everything? Yes, you can run CNI chaining modes where you can run cilium, just a hobble portion on top of OpenShift as CN or OVN and so on, absolutely. Okay. But cilium is also available on OpenShift if you want to replace it, both or both. I see it. Thank you. Hi, I have a question regarding as a network plugin. What does a network plugin like Calco or cilium provide, not provide by EKS VPC CNI? What will make me choose the network plugin and not the cloud plugin? Especially, I feel that VPC CNI makes the life easy. So it makes the pod reachable or accessible inside the VPC. Also, if I will use a low-balancer controller and I use the IP target, so the request can reach to the pod directly without it go to the API server. So, yeah, that's it. Yeah, so I think a good, often a motivation to run Cilium on top of the VPC CNI in EKS would be to install Hubble on top or the network policy implementation. So just like with OpenShift, you can run on top of the VPC CNI. You can also run Cilium natively in what's called ENI mode, in which case it can, just like the VPC CNI plugging itself natively route all of the pod traffic. So you don't need to run Cilium in what's called overlay mode. Was that your question? I'm not 100% sure whether I got it right. Yeah, so what we will provide for me if I put it on top of the CNI. So CNI now provide me the network or implement the network and make the pod already accessible. So, and also increase that latency. So what this will add to my configuration or my implementation? What's the main feature for that? Can you repeat the last part? I didn't fully understand it. Yeah, so I mean, if I have CNI and after that I installed Cilium in top of the CNI, I mean VPC CNI, what is the feature for that? What we will add more to my implementation? I see, okay. So the VPC CNI, at least as my understanding right now those connectivity, and there is of course Q-proxy already running. So the service implementation. And I think there already is or there is an upcoming network policy implementation. What does not existing there is all the observability that we've seen today, which is needed for the troubleshooting. Then Cilium also has functionality such as cluster mesh, service mesh, Tetragon, all of these are not available in the VPC CNI plugin. So the extended functionality. Does that make sense? Yeah, yeah, thank you. Yeah, a good question as well. We are currently using Istio with Kiali on top, which provides a very similar set of capabilities with what you were just describing for Hubble. So what would be the benefit of switching from one to the other? Because there is of course the service mesh level and the CNI level and like what's the difference that that gives you? So I think the different for most users will be that if you don't have a service mesh yet, you can get this easily just natively from Cilium. You don't need to actually deploy a full service mesh. If you are already running Istio, then Cilium provides very similar observability data. We even implement the same open standards. There are, there can be some performance differences depending on how you run Istio, ambient mesh versus the traditional Istio and so on. That you would have to look into in detail. So the difference is less in capability but more on the architectural model, potentially the performance of it and so on. Let's go ahead. I was trying to think for a great presentation first of all and I was trying to come up with a question. We, I'm trying to troubleshoot the problems from AWS ALBs talking to the Q cluster itself. And a lot of times I see that there is like connection to target errors or so. So like based on the observer like investigation and I call it that is somewhere we are, we're suspecting that the target, the pod is basically still not ready but the traffic is still trying like trying, there are attempts to route it there. If there is something like, is it anyhow the use case of Hubble that can help us with troubleshooting all those things? Yes, absolutely. Because if that is happening on the network you typically receive what's called a reset, TCP reset because the port is not yet open. There's even a Hubble filter or a dashboard that shows you those parts, right? Where you're reaching out to a service, maybe the back end, the replicas are still scaling up or they're already scaling down. And this is actually recognizable on the network because typically you don't have a lot of reset, TCP resets in your Kubernetes network. So it's often a pattern that you can see. If you know the specific service then with the observer, with this view with the Hubble Observe view here with the CLI you could actually see each reset. So you could see the service trying out and then instead of just not receiving anything back you will get back to reset. I can show you an example on how that looks like. So that's how many Scyllium users actually troubleshoot this. Because it's more like not on the service itself level but maybe when the node itself spins up. So at that time, so it's somehow show that the node is already ready and even the slowdown side, so like ALBs should wait. So if you're running Scyllium in those scenario that's a motivation. I know that we have seen the Hubble dashboard for the new cluster we just launched and we see the same, exactly the same problems and I think at that time we were using EKS because before it was for the self-hosted cluster. So there is Hubble somehow. There is Hubble, yeah, okay. And if you already run Scyllium in that when the node is still coming out there's actually a specific error, a specific drop reason that you will see that this went to a node that does either not exist or is not available yet. So if it even gets to the nodes but Scyllium is not ready there or the node is not ready yet, Scyllium will actually report you the specific reason why it could not deliver the packet. So that could be a motivation to run the Scyllium CNI part. You said I can see a demo, how can I? Yeah. Can I just come up or? Yeah, absolutely. Absolutely. Okay, cool, thank you. So I have a couple of questions. Great presentation firstly. So we wanna migrate away from Calico to Scyllium. We run on-prem, so all the fun stuff that goes with on-prem clusters. Is there a migration document that we can follow? I asked the booth and apparently there isn't one. Is there anything in the works? There is, even better, there's actually a video recording of Duffy Cooley migrating I think from Calico but I'm not quite sure. But yeah, so migration paths exist and there is on the Echo newsletter which is the EVPF and Scyllium newsletter and show. There's actually a migration episode on how to migrate to Scyllium. That can be a great starting point. And then the next step, if that's not enough, ping us on Slack, we're happy to help. We may have a more formal, more official migration path or documentation path soon. The trouble with that is often there's not just one way to run other CNI plugins. So it's often really depends on are you running, let's say Calico in BGP mode? Or are you running that in the cloud? Are you using that or policy? Are you using Calico or policy and so on? That's why it's not just a flip of a switch. Okay, and the next question was, so we run vanilla Kubernetes across a couple of sites and Azure too. So if we were to use Scyllium and Hubble, Hubble UI, how will traffic show when it goes through several other layers of infrastructure like firewalls, gateways, et cetera? So we obviously have access to the virtual machines but the infrastructure teams manage the firewalls. So how will that show on Hubble? Would it actually show as next hops or what will Hubble do with that traffic? Unless you do something and run something on those middle boxes, you will not see anything, but you can actually, if you want, install Hubble on them, if you want the observability data. Often Scyllium users will then replace, for example, the external local answer, with Scyllium, the Scyllium local answer and then you get the observability data. But if you run Scyllium in overlay mode, it will not actually even care about what the underlying infrastructure looks like. And as such, you will not see any details, but you will see, so Scyllium will warn you if it transmitted on one node and it did not receive on the other node, which will stand usually the indication that the underlying network dropped it somewhere. Right? Cool, thanks. Oh, by the way, you can see, typically if it's L3 routers, you can compare the TCP TTL on the sending and the receiving node, then you at least know how many routers you had in between. Cool, thank you. I have a question regarding the L7 detection. I mean, you showed with the command line tool in the UI that there was some information about the L7. And I wanted to ask, in case we have a TLS, do we still have some information about the L7 protocol used? The answer is it depends. So if you are using Scyllium to do the TLS, then yes. If the application is doing the TLS itself, then the answer is by default, no. But with KTLS, we can optionally if the application allows for this. And then if you are running a service mesh to do MTLS, then Hubble can also see it because it's unencrypted up until to the proxy. But if the application itself is TLS, and if the application does not want to be eavesdrops on, then we cannot see it. That would break the security mode. There are some ways to do that with Uprobes in the app, but that's not really reliable. So we can demo that, like many others as well, but it's not something we recommend to run in production because it's not 100% guaranteed. Okay, thank you. Hi, you mentioned that there are traces from Hubble and Scyllium companies. What are they or what kind of information I can get from them? So traces, we do open telemetry traces. We can do HTTP traces with request, response, spans, with the latency, the return code, all of this. We can also do traces for the DNS resolution, which is interesting. You can see the whole trace and then the spans including the DNS resolution so you actually know how long the DNS resolution took. Then we also have TCP traces support as well. So for whatever HTTP or layer seven protocol we support, as well as for the underlying protocol, we can emit open telemetry traces with spans. And then if you want to add additional instrumentation via open telemetry, application instrumentation, you can combine that together. So if you want to have spans inside of the app or including trace ID support. Okay, cool. And second question gonna be about routing of L7 level layer seven protocols like HTTP one, like in case of service mesh, like is it really possible with Scyllium? I mean, like the build some custom logic with some specific URLs, et cetera, like the things that like easier can do, right? So Scyllium implements the ingress standard and the gateway API standard. So whatever you can do with either of them, you can fully perform URL rewrites, path-based routing, header-based routing, carrier rollouts, then we have some annotations on both of them. And then as the most sophisticated, most powerful API, we have what's called an Envoy CRD, which allows you to configure direct Envoy configuration. So whatever is possible to configure in an Envoy listener, you can use. So the full Envoy feature configuration that's available in a so-called listener, you can actually configure for retries or for specific TLS termination that is going beyond what's possible in ingress and gateway API. So is it correct that it's not happening in the inside of the kernel, right? I mean, it's- Correct. So all of the layer seven low balancing is being done with Envoy. We have some observability into layer seven using EBPF, but the low balancing retries rate limiting on layer seven is all done through Envoy. We do have an MTLS model that does not rely on a proxy, but that has nothing to do with HTTP or any layer seven processing that's purely on the layer four, right? Okay, cool, thanks. Hi, so we are also running Calico on prem. And I was wondering whether it's possible, assuming the worst case, so we cannot migrate to Selium for some reason. So whether we can run Apple on top of Calico and have the same observability metrics and whatsoever. So yes, you can chain Selium on top of Calico and run Hubble. It does have some limitations because the sum of the layer seven observability into HTTP, the Envoy injection we're doing that relies on using a particular feature that Calico also uses right now. So that specific aspect is currently not compatible, but everything else, I would say roughly 95% what you have seen today is possible to run on top of Calico, absolutely. And does it require any change from an application level or is it completely transparent? No, it's completely, you are changing your CNI configuration to also launch Selium next to Calico on top of it. And then it will not actually do the routing. You can even still have Calico do the network policy if you want, or you can say, I'm migrating away from Calico network policy and I'm using Selium network policy. And then in either way, do the observability on top, that is possible. Okay, so the traffic is going to Selium and then down to Calico and then it's going. Okay, please. Calico actually only uses the standard. So if you're running Calico in the default configuration, then it's just using Linux networking. There's nothing too special about it. If you're using Calico in EBPF mode, then it would use that EBPF. But both are compatible, I think. Would there be maybe performance issues by running two CNI on one on top of each other? Like only running one is definitely more efficient than running two, we would have to run it. If you are not doing anything with Selium, right, the overhead is very small, I'd say overall, right? Like as you configure all of things in Selium, there is overhead.  But in general, that is definitely 100% feasible and we have many Selium users that are doing this. Okay, thank you. Hello. Hello. We're thinking about shifting from AWS to GCP, but we know that Hubble isn't available in GKE. So do you know if it will change someday or if there is an alternative to that? You can run Selium on all the cloud providers natively. I can obviously not speak to when the cloud providers would enable specific functionality in their version of Selium. I know that some of them are considering to bring Hubble, but you can run Selium, to my understanding, on all the cloud provider as a replacement CNI or in chaining mode on top. In particular, of course, on AWS, that's common. And that's actually what a lot of users and ISOvalent customers are running. Okay. I think we got all the questions. One more. One more. Sorry, I'm sorry. Two more. Two more. We are also an OpenShift customer and we are using OpenShift with default OpenShift SDN. They use IP tables. Currently, we have a cluster IP service and the clients are calling this service in a high rate. And we are creating lots of new TCP connections and we are close to consuming all the available TCP source ports. In the Selium implementation, I think you are replacing the cluster IP address with the real endpoint IP address during the TCP connect times. Does this implementation increase the number of tuples? So it increases the number of the usable TCP source ports. Very good point. We have a good answer for you because when Selium implements the load balancing with Qproxy replacement, we actually use our own connection tracking table which you can set the limit for. So that solves limit number one. And then if you're really running out of tuples overall, I would recommend that you are using the socket-based load balancing, which does the load balancing at the socket connect system time, which is incredibly more scalable. If you ping me on Slack, I can give you pointers on that to get you started. Thanks for the demo. So since GKE DataPlane v2 uses Selium, I was wondering if that's compatible with Hubbell because I did a quick search and said that the Hubbell UI wasn't available because the agents don't expose the ports but I was just interested whether the CLI Observe tool would work. I know that there is work to enable Hubbell on GKE. I'm not aware of the latest status as of now. In the initial version, it was not enabled but I think the Google team is working on enabling Hubbell. As soon as the Hubbell API is exposed, then the dashboards and the Hubbell CLI will work to some extent because they may actually not immediately enable all of the Hubbell functionality, for example, the DNS observability or the Layer 7 observability. That's a question to ask for Google what they will actually specifically enable there. Okay, I follow the project. Thanks. Awesome. Thank you very much again.