You keep it clean. It's time to finally get it started. But what exactly are we getting started? Well, unfortunately, it's the focus on the 2024 NFL season because the 2023 season is officially done. The Chiefs are the Super Bowl champions. We all had high expectations for the Baltimore Ravens. And then a lot of our expectations were not only met but surpassed in the regular season. And then playoffs came around. And for the first game, it was great. But the second game was obviously a much different story. But now we turn the page to the 2024 season prematurely. But it is what it is. But with Team Keeping Clean, I asked the Team Keeping Clean patrons to shout out to all the Team Keeping Clean patrons. I asked them the other day, like, how are we feeling? What are we doing in the offseason? What are our plans for the offseason? And let's hear what some of them had to say. Oh, my God, Dominique, he said, sit back and relax until the schedule release comes out and start planning that trip to Dallas or trip to LA. So he ready to go to some games next year. Joshua said, watch some Team Keeping Clean videos on some NBA now. Ain't much to do till summer.  It sure will. Harrison said, I'm too sad to think about it. I couldn't even watch the Super Bowl. Definitely not watching ESPN. Just had a new baby boy. My wife joked with me and said that he would be a Steelers fan. Boy wouldn't have a home. LOL will keep grinding, man. Appreciate that, Harrison. John said, he's just going to chill and watch the videos and see who the Ravens pick up. And let's hopefully make it back to the ship and win it this time. My God, Martin said, he's just chilling till football season starts again. He said he's grinding on himself. And he hit a big milestone. So shout out to my guys. Martin said he lost 40 pounds so far and playing loose 75 pounds more. I need to get like you, my friend. That's great for real, man. I got Kevin said, congrats on the baby girl refocus. And we put ourselves right back in the situation next season. Don't forget who we are next time and we should win. And the nail said, too hurt to even think about football. But I know as soon as the league gets starts, I'll be there. And yeah, I get it. I get it. Funny with my guy, Kevin, what he said, refocus. And we put ourselves right back in the situation next season. Just don't forget who we are next time. So that's something that's been ringing a lot of bell. Ringing a lot of bells with Ravens fans because the Ravens just keep on doing the same thing in the big moments. They keep forgetting who they are. And that's an issue. That's a really, really big issue. But we got a lot of issues to talk about in this episode of questions from subscribers, questions from Team Keep It Clean. And this special episode is going to focus on the Team Keep It Clean patrons question. And if you would like to become a Team Keep It Clean patron, you can go to patreon.com slash engraving viz.  We got some fire questions to get into to get us started. We are going to start with my guy, Harry. He said, what's good in graving in the Team Keep It Clean family? I had to take some time away and calm myself down after the FC Championship game. I waited to the end of the year, presser from the Ravens hoping that watching that would give me as a fan some answers and hope it did not. You know what? Real quick, I had meant to watch. I never watched. I didn't even watch it this year. Didn't watch it. I didn't like protest or not watch it out of being angry or anything like that. I just didn't watch it. I had meant to, but got caught up, got busy, forgot about it. And it never really just never ended up being important to me to watch. So I feel like I didn't really miss anything. But anyway, let's continue. He said, EDC for the most part gave me hope that he would do his best to get the pieces for us to have a team good enough to make it to the Super Bowl. Now, and hey, he did that this year. EDC, we had some questions about EDC, especially going in to last season. He exceeded expectations, he did his thing. He did his thing. So EDC, great job last year. And he said, my problem, which it is most of the time is with John Harbaugh. The reporter asks, why every time we have a team that makes it to the playoffs, do we go away from what got us to the playoffs? That's, yeah, that's that. Now I do remember seeing that sound clip. Oh, that's just how the game went. But let me, let me continue. He said, Harbaugh answered, that's a fair criticism and a fair assessment. Then he proceeded to state that the other teams coaches just had better game plans and they dictated to the Ravens to change their game. My question is simple. After those statements, why is John Harbaugh still the head coach of the Baltimore Ravens? I honestly feel whatever players EDC gets, no matter how well Lamar and others play, no matter how good the team is in the regular season, John Harbaugh will go away from what we do best and get outcoached in the postseason. I don't believe John Harbaugh is a good coach. And with statements that he said, I don't see why Steve Bishati should feel he is a good coach either. Harbaugh is the reason we have two Lombardi trophies instead of four, or possibly five. He's wasting the team's talents. And I think that's part of the reason the staff was willing to go to other teams. Let me know your thoughts. Well, about that last part, why the staff is willing to go to other teams, I don't think that's an indictment on John Harbaugh right there. Because if you get an opportunity to get a promotion, if you're at your current job right now and you want a promotion, you want to move up, but there's no availability for you to move up and you see that availability somewhere else, then you're gonna take it. You're gonna take it. A new position, a raise, a new job position that's higher than what yours was before.  So I don't think that's an indictment on John Harbaugh, but however, the Baltimore Ravens completely losing their identity in the biggest moments that is. And that's an issue. And it's continued to be an issue. And it's something that's very, very scary to think about when you think about the long term of these Baltimore Ravens, you think about, man, how are they gonna be when they get there? How are they gonna be? Like regular season, everybody cool with regular season. Like regular season, oh yeah, Ravens, as long as they healthy for the most part, they got regular season taking care of. But playoffs is where everybody is, well, and if you're not, you should be scared. You should be worried because this has been a pattern. With a healthy Baltimore Ravens team, this has been a pattern. Because I can't count last year because Tyler Huntley, he was playing in a playoff game last year. But this has been, and even in that game, like for them to put the ball on Tyler Huntley, and J.K. Dobbins, he talked about it too, but anyway, it's been a pattern for Ravens to just go away from stuff that's worked to get away from themselves. And that's a big issue. And you can think, oh, all right, we'll just get back next year and we'll fix it next year. This has been going on since, this happened in 2019. Because we're just gonna talk about this Baltimore Ravens era right here. You don't know about the Joe Flack or this era right here. This same thing happened in 2019. They literally had a record-breaking rushing team, record-breaking feature. Obviously Lamar Jackson, Mark Ingram, Gus Edwards, but they went into the playoffs and Mark Ingram was hurt. He was hurt. I forgot what injury he had, but he was hurt. Maybe it was a high-angle sprain, I think. But you still, you had a healthy Gus Edwards. They literally ran the ball with their running backs. Six times. Six times. Now Lamar, he ended up running a lot. He had a run for his life, but they ran the ball six times. Six with their running backs. Remember this game a couple of weeks ago? It's a Kansas City Chiefs and a bad run defense Kansas City Chiefs. It's like, the Ravens is just weird. Why are we not doing the obvious? Why are we going away from the obvious? Oh, bad run defense. Okay, attack them. Why are we not attacking them? And it's like, it's been under different offensive coordinators. And even when we go back to Lamar's rookie year. Like, it's crazy because in the biggest moments, it seems like the coaching staff does not put these players in positions to succeed. It's not all on the coaching staff players. They've got to execute too now. But if we even go back to 2018, since we're still talking about this era, that was Lamar Jackson's first playoff start against the Chargers. They had just played the Chargers a couple of weeks ago and they took care of business against the Chargers in a big way. Play them again. First quarter, all the RPO stuff's not working. They still keep doing it. Second quarter, all the RPO stuff's not working. They still keep doing it. So all right, they went in the way till half-time makes them adjust. Cool, nope, no adjustment. Third quarter, still doing the same stuff. It's not working. They still keep doing it. Still keep doing it. Fourth quarter, guess what? They decided to open up the passing game. Guess what? It worked. This has been an issue for the longest. For the longest. They keep going away from what works and they don't make adjustments at the right time, in time, in real time. And then they look back at it. Oh, we should have done this. Oh, we should have done this. This has been happening since Lamar's rookie year. Rookie year. Rookie year. First offensive coordinator is Marty Morniwak. Second offensive coordinator was Greg Roman. That offensive coordinator's Todd Monkey. And initially we would think, oh man, these offensive coordinator, why they call them play like that? But then it's like, hold up. The same issue has been happening with different offensive coordinators. But we still got the same head coach. Lamar needs to stop. Next question came from AW Juice, man. He said, I hope all is well with you on that Super Bowl afternoon. I wish it could have been us, but it is what it is. Anyone who will allow me to go into details about Lamar Jackson, here's an analogy. As a kid, I had a tendency for wanting to be friends with everyone going out of my way to please everyone. I did things out of my ordinary to try to impress them for them to think I was cool. Mainly for the ones who didn't like me no matter what I did, how nice I treated them. Some days they were okay to me, but usually it was always back to bad mouthing me and putting me down. My mom used to tell me, if someone doesn't like you no matter how good you treat them, there's nothing you can do to change their perception of you. I realized I had true friends that liked me no matter how different I was or how unorthodox my social skills were, but they loved like cherish and accepted me. That was in the late 90s. Fast forward to 20 plus years later and put Lamar in that scenario. But not with me making friends, but with his football career and playing style. But we know Lamar wants to put on a show and wow and impress all the Ravens nation. However, it seems as though when the lights are the brightest and the stage is at its biggest, Lamar Jackson's listens to his booze more than he does his cheers and he turns into that kid. That I was minus making the friends of course, but he has to go out of his way to show everyone, mainly his haters and critics. He's capable of being a passing quarterback in this league. Now we know that he is and that he can be that, but his haters will always say otherwise no matter what he does. And it seems as though Lamar's ego of showing he's a pocket passer is getting in the way of his success to reach the promised land. Lamar needs to realize that no matter what, passing, rushing, stat records, he breaks or does both in a game example, he can pass for 385 plus yards, five touchdowns, 30 out of 37 completions and 190 rushing yards plus a two rushing touchdown and his haters are still going to bash him and bad mouth him because they simply do not like him at all the way his Louisville and Ravens fans do. More of the story, go out there and be yourself is what I'm trying to say and LJ needs to stop trying to impress his critics and haters, that was a good one. And yeah, that's a big, big part of it too. Ravens just not playing their game Lamar, not playing his game too. That has been an issue we saw it in this last game. It's like, and I like how you said it about the stats and the records and this and that breaking all these records and it's cool, those are great. But especially come playoff time, really regular season, but especially come playoff time. You got to let it all out there. You got to put it all out there and you got to do everything that you possibly can do not for a certain record, not for a certain stat but to win because this is it. Like we talked about it all year that Lamar Jackson was clearly holding back. He was clearly holding back. We could all tell that he was holding back. Then in that playoff game, he said, oh, in the Texans game, he showed us like, oh, okay, there we go. But in the chief's game, it's like he went to hold him back again. He went to hold him back and it was like, whoa, what's going on? A lot of people highlight that playoff, that fourth down play that, I think the Ravens were down, I want to say 14-0 and they were going for it on fourth down. I forget what the score was, but the Ravens were down. They were going for it on fourth down. Maybe it was seven-0, I don't remember. But anyway, Lamar Jackson, he broke through, broke through the offensive line and it was just him. Maybe it was Lageria Snead. I forget which cornerback it was, but he could have outran him. He had an angle, but he slowed down. He slowed down. He was looking at the cornerback and whatnot. Maybe you're trying to make him, but it's like, hey, what's going on, man? What's happening? And so that was that. But yeah, that's a big part of it too. He got to just, you cannot play to try to prove, so don't try, don't play prove it, ball. Play winning football. Next question came from my guy, Maura, and next couple of questions, actually. He said, now that the season is over, I have to issue an apology and my apology goes out to EDC. This time last year, I was saying, man, we need to move on from EDC. He's not getting the job done. My biggest gripe was his drafts were bad, but boy, did those draft picks show up this year. Keith Mitchell, well, Keith Mitchell was an undrafted rookie free agent, so he don't count as a draft pick, but he was a good signing though. Geno Stone met a BKPQ, Brandon Stevens, Tyler Lindebaum, Zay Flowers, Isaiah Likely, and I still think Adolfi and Bateman have potential. Those guys really showed out in a big way. I didn't say anything earlier because usually when I start praising the team, they fall apart, LOL. Jinx may not, maybe not really, but I didn't want to hurt our chances just in case, but not only those draft picks for EDC kills at the trade deal line with Marcus Peters, Kaleis Campbell, and Roquan Smith being absolute stars and steals a big free agent signing in hashtag JC24 and Kyle Vanoy, whom I really hope the Ravens bring back. Oh, and he got the Lamar deal done. That was huge. It was. EDC did a phenomenal job last year, phenomenal job last season. They're just really setting the Ravens up for success and you could tell that he did a phenomenal job because the Ravens lost a lot of people throughout the season for different amounts of time throughout the season. Some of those guys came back, some of those guys didn't, but the Ravens still kept winning. They still kept playing amazing football. So that showed that they had a lot of quality depth on the team and how's that debt built? Well, that's from your general manager making the deals. He also said, man, people got to chill with this Lamar is a fraud talk. Patrick Mahomes is the only current quarterback to win a Super Bowl in the AFC besides Flacco and Aaron Rodgers. But so I don't see how anyone can say Lamar is a fraud when literally nobody else has won, has won one. He played a bad game short. If Zay doesn't fumble on a one yard line, no disrespect towards Zay. I know he was just trying to make a play for the team, but I'm saying Lamar is one fumble away from a potential Super Bowl in two Lamar. I say this, don't let this get you down after all the guys before we had a Y receiver that had the ball knocked out of his hands in the end zone in the AFC championship game. And then the next year we won a Super Bowl, but let's stay calm and run it back. I like it. And yeah, that's something that I don't think about enough when it comes to this, that AFC championship game. Lamar technically threw a touchdown, but Zay Flowers just fumbled it. And I get what Zay Flowers was doing. He was really trying to make a play. He was pressing because the Ravens offense just, they hadn't been doing anything. And then they got that big play a couple of plays earlier and they were right there. It was like, oh yeah, let me just dive in, get this touchdown and then boom, got knocked out. Next question came from my guy, Nick Brick. He said, in Raven shaking my head, is it better to have loved and lost or never to have loved at all? I'm glad we got this far, but I hate to see it all in an embarrassing, pitiful fashion, but we move on. I'll keep my question short. Do you think the Ravens offense would be better if we brought back Hollywood? Reason I think so is because we know Lamar is picky. It's unexplainable, but there are guys that he can get the ball to with easing guys that he just can't. With Zay and Hollywood, I feel like Lamar could throw for 4,000 yards easy. I don't know what it is about Holly and Zay, but Lamar doesn't miss them. We've been talking about Lamar's deep ball all year, but when it was him and Hollywood, Lamar was on the money, assuming Hollywood caught it, LOL.  It's not about who's the best, but who fits Lamar the best. Plus, I'm not afraid to say I love Hollywood. He belongs here. Oh, I'm not afraid to say it either. We love Hollywood. Shout out to Holly. I wouldn't be mad if they did that at all. And like you talked about the deep ball, I think with Hollywood, one of the reasons that his deep ball, Lamar Jackson's deep ball was so good with Hollywood because Hollywood had that speed, that real speed. And Zay flowers is no slouch when it comes to speed. Neither Israel Shaw or Bateman, but Hollywood's speed was on another level faster than those guys. So that's why him and Lamar, they just, they were able to get it like that, man. So it was just, it was a beautiful thing seeing. So imagine if you had a Hollywood, I don't think it's gonna happen. I would love if it did happen, but I don't think it will. But imagine if you had a Hollywood brand with that speed and he has Zay flowers with that speed and that agility. And then Rashal Bateman too. And it's just, it'd be a nice scenario too. And then Nelson Aguilore too. Like though that would not be a slow rock, wide receiver group at all. They'll, that'd be some real nice speed out there. But I just, I don't think it's gonna happen. Next question. Well, more so a comment came from my guy, Jason EG, said this was about the Chiefs and Ravens game. He said horribly called offensive game, two end zone turnovers, one blatant defensive pass interference. Chiefs were gifted, especially with the no calls and not running the ball on a defense that Buffalo just ran all over. That is the game summary. And he is 1,000% spot on. Next question came from my guy, Dominique. He said, hey man, hope you're doing well. I'm still grieving over that loss. It was a weird game playing. I felt the Ravens had yesterday. Well, we won the game. He's the number two Russian team in the league knowing the Chiefs defense could not stop what we had on the ground. I feel like when we play the Chiefs, we always revert from what it's actually working. It's not like we were down 20 with five minutes left. I'm disappointed in the monkey because this was his worst call game all year and he chose the wrong time to do. So he isn't the only one to blame though. Yeah, it's frustrating. Very, very, very, very frustrating. Ravens had everything in their favor. They had, it was a perfect setup but they let it all fall apart. Next question came from my guy Travis B. He said, hey man, so with Odell Beckham, Julian Nelson Aguilore hitting the free agency this off season. Well, maybe Odell Beckham, Julian will see if they ended up releasing him or they ended up signing him to a contract extension we'll see. But anyway, with those two hitting free agency this off season, how do you think the Ravens will approach getting a little more new weapons and hope you and the family are doing great and keep up the great work? Appreciate it Travis. Same way, they continue to approach it. A mix of free agency and the draft because Devon Duvenay is a free agent too then. Don't forget about him. Gus Edwards, J.K. Dobbins. But yeah, as far as the wide receivers, the pass catchers, I think the same way, free agency and through the drafts so a healthy mix of both. The next question came from my guy, Plex. He said, it's a tough loss. It's like when we play the Chiefs, it's Coach Klein versus Coach Red from the water boy and Coach Red has the green playbook and we had a chance. The first half was rough. Defense showed up in the second half and I appreciate them for that. All fans had the opportunities. They fumbling was critical. I hate it for him. A young guy trying to make a play. Lamar was pressing on that interception. There was plenty of time left. Bateman was non-existent. It's not his fault. He was put into positions in this game. Really the season to make plays. I know we're not picking up the fifth year option. I do want Odell Beckham, Jr. Beck. I don't know what the contract will be like but the leadership I saw from him is something I want to keep. Monkey let me down a bit. It wasn't an enrollment performance but some of the situational play calling brought back memories. It's gonna be a completely different team next season. With the roster we had and the injuries to other teams, we came up short when we really couldn't afford to. This should have been the year. I agree 1,000%. Gus only having three carries is inexcusable. We fell behind early but the game was never out of reach to where we should have abandoned the run, right? He said, I don't know what next season's gonna look like. We really needed something to shake all the way. It didn't happen. At least nothing major. And yeah, just to go piece by piece. Zay Fumlin was critical. Yeah, it was. He said the young guy tried to make a play. Lamar was pressing on it, an exception to want to likely. But yeah, that should have been passing interference too though. So yeah, it's a double win. Bateman was non-existent. It's tough, man. He said it's not his fault. He wasn't put into position in this game. Really the season to make plays. And there was someone they just missed but it's tough. Now as far as O'Dell, he said he wants the leadership back. O'Dell is tricky. Great leadership, amazing leadership. But leadership is great but on the field, the contribution's on the field. In my opinion, they gotta be there too. That's why it's tricky with O'Dell. So with O'Dell, I wouldn't mind him back. Not at the number that he was in. I get it this year Ravens were desperate because you had to get Lamar. Lamar wanted O'Dell back on Junior. And DeAndre Hopkins, Ravens got O'Dell and they really had to press because Jets were getting ready to get O'Dell for way cheaper. Ravens say, here, we'll give you about five times more than what they gonna pay you. And they paid him. We got it. He made a lot of money this year. But now I just, for that price, yeah, that would be a no-go for me. But hey, I ain't the GM. But yeah. So anyway, and he said to add on, we treated this game like a four by one when it was really a marathon. It started off fast. My homes had maybe one incompletion in the first. After that, the defense settled down but the offense was still in panic mode. It's like we were playing to keep up with them when we really didn't have to. It was 17-7 at halftime. We got the first stop and should have got to playing our game. We beat ourselves for the umpteenth time. And I didn't want to do it, but it leads me back to the head man in charge, John Doc Rivers Harbaugh. He's been living off that one championship. It's been a decade. There needs to be a coaching change if we don't win it all next season, making it to the bowl isn't enough. That's some powerful words to end off his question right there. John Harbaugh, that is a big common denominator with this thing because it's like the same thing. It keeps happening in so many different ways, but it all leads back to the same issue.


to reduce your daily routine stress you should focus on you mean making a plan for a day and stick with that plan or if you're not making any plan so be in present don't go in future and don't travel in the past and have the fluid good amount of fluid in between your day or in between your tasks have in fruits and enjoy your day with some nuts and seeds to reduce your stress or I will say to manage your stress thank you


Hello everybody, and welcome here to the Neutron control plane performance improvements. So let's go through the agenda. So I'm Aracela, and I will be talking about L2H and performance. Then we have the distributed virtual router and security group, this will be Brian. And then we have measuring RPC performance and some very interesting data by Kevin. So let's start introducing a little bit of the L2 agent. So the L2 agent runs on compute nodes, and he's in charge of configuring the virtual bridges. So in the obvious implementation, we are talking about the brint and brtun. Brint is the bridge that has the flow to tag and untag the packets coming and going to the VM. And brtun, as the name says, it's the bridge in charge of the tunneling. So it's the one that has the flow to translate the VLAN ID assigned to the network into the segmentation ID. So if you're using, for example, grid tunnels, it would be the grid tunnel ID. So its main task is to wire new devices. It basically checks every time if a new device was added. And by new device, I mean a tab interface that it's created by Nova to connect the VM. So if it attacks a new device, it will communicate with the neutron server to get the device details, and it will wire the device. The L2 agent, it's also in charge of applying security group rules that are firewall rules, and in neutron are implemented using IP tables and IPsec. So if you want to improve the performance of the L2 agent, so what's the area that you're going to look at? So in my mind, we have two different categories of improvements. We have the one related to what I call internal processing. So it's the processing that the agent does without involving any other process. So for example, when the way it detects new device or the way it computes which flow needs to be applied and the way it handles failures. So in this category, to improve the performance, the best thing you can do is to use better algorithms. In the other category, we have some external process involved. For example, the obvious agent uses RPC a lot to get port notification, like when a port is updated, for example, to get security groups updates, and also to require the device information to the neutron server. And to notify the neutron server that our device is up or down. As I was saying before, the obvious agent also interacts with IP tables and IPsec to apply the firewall rules. And of course, it communicates a lot with OpenVid switch to apply the flows and also to get information regarding the interfaces. So if an external process is involved, an easy way to improve the performance is to reduce the overhead in the communication. And that's actually what we did. So regarding the L2 agent, we can divide the performance improvements in three categories. So improve the RPC call, reduce the overhead in the communication with OpenVid switch and provide a better way to handle agent restarts. So regarding RPC call, the agent, when a device is up or down, will send a message to notify the neutron server. And liberty was one message for every device. And what we did was to create a new call that it's a bar call so that you can update many devices at once. And we also added a parameter in this call, fail devices. This will be used in future to improve the way the agent handles a failure. We also improved another call that it's the security groups provider updated. This is the call that it sent when the provider rule needs to be refreshed. The provider rule is the rule that allowed traffic coming from the HCP server. And so you need to update it when the IP or the MAC address of the board changes. Before this message was triggering a full refresh of the firewall, so all the devices were updated. And what we did is to modify it to add the list of devices that needs to be updated. So basically, they are the devices that are on the network where the HCP port changes IP or MAC. So to reduce the overhead specifically in the communication with OpenVswitch, we have... So the OVS agent was issuing a call for every interface to get the details from OVSDB. And what we did is instead of issuing one call per device, just have a bulk call to reduce again the overhead. And the last one, it's not only a performance improvement, it's also a kind of correctness improvement. Like before a liberty, the OVS agent were deleting all the flows at start. So if you had a connection, basically the connection dropped. What was done in liberty is that when the agent adds a flow, it uses a cookie, so the flow is associated with a cookie, that it's basically a UUID that it's created when the agent starts. And so the agent wants to delete any flow at start-up. It will create the new flows associating the new cookie. And then where the new flows are in place, it will clean up the stale flows. And it will be able to recognize them because the cookie is different. And so we'll delete the stale flows, but the connection will stay in place because the new flows are already installed. So I wanted to check if that really improved the situation. So I created a very small test, especially compared to what Kevin will show later. So it's just 20 VMs that are booted. And these are the results before these improvements, and these are the results after. So just to show you that it worked, actually, so the minimal time is 0.6% better, the average time is 4% better, and the 95% is 5.9% better. But there's still a lot of work to do, so if anybody's interested, please gram me later, and really you can help a lot. So some of the things that can be improved, like we still use the common line for OVS DB Monitor, and it would be nice to use the OVS Python library. We also should create a queue of events so that we can have multiple workers. We should also add a priority so that higher priority events can be processed first. And generally, we should improve the state convergence so that what's in the neutron DB reflects what's actually on the compute host. And we can reduce more the overhead in the communication with OpenVswitch. For example, right now, when we get the interface details, we need another call to know to which bridge the interface is connected, and this can be easily avoided adding the bridge ID in the external IDs. And also one last point that it's not in the slide, it was just mentioned in the design summits. Probably having all these bulk calls for the RPC is not really an optimal solution because especially when the agent starts up, if you have many, many devices, you can incur in timeouts. So we also need to find a better way to address that. And now, Brian, we'll go on. Thank you, Ursela. All right, I wanted to talk a little bit about distributed virtual router, but in order to set the context, I wanted to first show what things were like before distributed virtual router. So if you look at this diagram, there was typically a network node where it was centralized L3 agent would run. You have all these compute nodes. And any time they wanted to access the external network via their floating IPs, if you look at the blue network, they would all go out over the blue network. Eventually, they would all converge on the network node and they would basically saturate that wire, saturate the network node such that it couldn't perform as well because it's doing SNAT traffic. So with DVR, we changed that by adding these red lines here so that every compute node was also connected to the external network. So this took a lot of the load off the network node. It let all the floating IP traffic go in and out of the compute nodes to all the local VMs on those compute nodes. So what happened was inside of a DVR compute node, in addition to the OBS agent, there's now also an L3 agent that is instantiating the neutron routers and there's a metadata agent to handle Nova metadata. So this was great for the data plane. But now if we look at the control plane, now that every one of these compute nodes has an L3 agent running, the neutron server was initially sending out, or was sending out RPC messages on an L3 fan out queue, but initially it was only sending it out to a small number of networking nodes, one to some small number. But what happened was with the distributed nature of it, since every one of these had an L3 agent, every one of them started getting RPC messages. So this started to kill the message queue. It started to kill the network. So if you look at this purple line, and we've noticed this a lot in the Liberty time frame, was there any time a floating IP create, update, delete, or other such operations were done, we could basically kill the server. So we had to do something about this. So obviously the DVR code needs to be a little bit smarter. If it knew where the port was for the floating IP, why didn't it just send the RPC message to that one compute node? Why did it fan it out? So there was a change made actually just a couple of weeks ago in early Metaka that changed the floating IP create code to do that. Instead of putting it on the L3 fan out queue, it sends it directly to that one compute node to that one L3 agent. We also have patches in flight for the update and delete paths that hopefully will be done in the next couple of weeks. And then once we knew about this issue, we need to start looking at other places where we can apply this same methodology, for example, in the port update path. The other change with DVR is there were some suboptimal DB queries being done. For example, when there was a floating IP query, it was querying for all the ports, not just the ports that were floating IP ports. And Kevin actually has this on one of his later slides, so I won't go into this in too much detail. So security groups, to kind of set the background on what security groups are and how they work in Neutron, security groups are all instantiated in IP tables. And the way that the code works is it first calls IP table save, pulls all of the security groups out of the kernel. It then takes what its view of the world is for rules and chains, and it merges them together such that the table is updated, and then it pushes them all back.  So while we always thought that increasing capacity in compute nodes was great, it actually turns out to be a bad thing in cases like this. Because as counts increase from, say, 50 to 200 VMs on a compute node, the number of IP tables rules grows very quickly. So luckily, we noticed this early enough in the Liberty cycle that Kevin here, who does a lot of the IP tables patches, was able to change the code from doing serial searching of this entire IP tables rule set to doing a hashed lookup such that we got this performance improvement down in a manageable space. Additionally, we have some other changes in flight because we still have a few issues where, again, with large number of rule sets, the worst case scenario can still be 30 second delays in getting rules configured. And while this might not seem like a big issue, a lot of times it impacts booting of VMs because DHCP packets can't get out until the security group rules are added. And then that way, DHCP could fail. And it also causes the agent to be stuck doing IP tables work when it could be doing a lot of other things and it can get backlogged. So we have a change in flight at the moment. So instead of building into the entire table, IP tables table, and merging in the rules we want, we're just going to start doing deltas of those rules and then push just small deltas back into the kernel so that it can apply them. So the delta could be a deleter, an add of a rule or a chain. And Kevin has done some work on this. And the startup time, if you restart the OBS agent, he's seen the time actually cut in half of doing a restart. So it's a pretty good improvement. And then the last bullet here, some people might wonder why I'm mentioning Nova Network in a neutron talk. But at the time the quantum project was started, that IP tables manager code was the same in both projects. It was basically taken line for line. And over time we've diverged in how we work. But Nova Network continues to make performance improvements. And periodically I do take looks at that code and do diffs with neutron and find some little nuggets there that we can possibly apply. And the latest one that Nova Network added was the concept of a dirty table. For example, there's multiple IP tables, tables. And if we knew that we only need to touch one of them, or two of them, then we would only operate on those and we could leave the other ones alone. And I talked with Kevin about this the other day. And we're thinking that if we could apply that concept to chains in neutron, that it could also reduce the amount of work we're doing from right now, which is a zero N down to below that. Because if we know we have just a very small thing to work on, then we don't have to look at the rest. We can just ignore it. So there is some future work here. The first obvious one is that the OpenV-Switch agent is gonna have security group and contract support. And when we're able to get that work in and able to run later kernels in the gate, that should improve the OVS security group performance significantly. But we still have to worry about IP tables because the Linux bridge is still gonna be using IP tables. So it might be time to start looking at alternatives like NF tables, which is the follow-on to IP tables and is gonna eventually replace it. We don't know yet. I think it's something Bill needs to talk about. One of the other things that we're starting to do is profile the code. IBM has started profiling the L3 agent to try to find places where we're spending much too much time in certain areas of the code. So they found a possible issue in the plug code in the drivers. And they've also started noticing calls to SBIN IP are taking much too long. So there have been a couple of patches proposed in the past regarding privsep privilege separation that started using a built-in Python library instead of using a call out to shell to list the interfaces. So using the Netlink library basically right from Python. And I think that found about a 4x improvement in some of these commands, even though it's maybe a second, but you're getting that second back. So I think trying to revive those patches and do some more performance testing on those would be great. The other thing, since I had mentioned DVR at the beginning, we've noticed a lot of both failures and performance issues with DVR through the Liberty Cycle. And we decided yesterday that we're gonna start doing a weekly meeting just to cover the DVR performance scaling and bugs starting next week. So if any of you are interested, I'll be sending an email to the list so that we can start doing that. I think the meeting time is 1500 UTC. And with that, I hand baton to Kevin. Okay. So when we started talking about giving this presentation, what I wanted to do was measure how the performance of our RPC APIs have been changing since we've been making changes throughout the cycles. The problem we have is we have Rally, which is really good at measuring HTTP API performance, but compared to how many calls are made to Neutron from an API perspective, the agents make way more many calls than any kind of HTTP client like Nova or tooling that users are directly making calls to the server with. Because like one security group update can result in 50 agents asking the server for information. So that's 50 calls to the RPC API versus the one that would go onto the HTTP API. And so we have this issue where the performance of our RPC APIs while we're developing isn't really visible to upstream developers because if there's an operation that's slow for only a few ports, which is what developers are usually working with, it will still return and start working fast enough that we'll have a performance degradation go unnoticed. So what I wanted to do was see if I can get Rally to measure the RPC performance that we have. So what I initially started to do was try to make a Rally scenario that embedded Neutron client bits in it directly. But this led to a lot of brittle stuff because Neutron, or not Neutron client, the Neutron agent components in it directly. But we would refactor, those would move around and that would break. And they would also have different Oslo configurations. So Rally has its whole configuration set and the agent has hit its configuration set that would conflict. So I added this extra layer in between where we just have a small process running that runs the Neutron agent components, the RPC components that make calls to the Neutron server and then exposes those via a little HTTP using the bottle library. Then I can point the Rally scenarios at it and just do single HTTP requests that end up measuring the performance of our Neutron server RPC APIs. So for the test scenario data, because we noticed some of the issues that we have weren't starting to show up until we'd have a lot of scale, I created a whole bunch of stuff in the DB to begin with. So 1,000 networks, almost 2,000 subnets, 1,000 security groups, 14,000 ports, 27,000 security group rules. A lot of these numbers weren't too specific. I just wanted to create big numbers and stopped when I got bored waiting for it. 950 routers, 4,000 floating IPs, and then make an RPC call, retrieve information for 400 of the ports and 50 of the routers. This would be kind of a heavily loaded agent in a deployment you might see now. So I wanted to measure how this changed from the start of key load development all the way to the end of the Liberty release now. So I have had a script that would check out a commit, load that DB test data, run the DB migrations to get it to the right position for that state of neutron code at the time, start up the neutron server, run the rally scenarios against it, and then start over the process again with the next commit in the line so I can get performance data for every single commit throughout the neutron development cycle. So the first one I'm looking at is the L3 agents get routers call. This is the main call the L3 agent uses to retrieve all the information it needs from the neutron server. This includes router information like extra routes, the gateway interface, the interfaces to its local subnets, floating IPs, and internal interfaces for DVR. So here's the get routers performance from the start of Kilo all the way down to the Liberty release. I don't know if you can see the small dotted line right in the middle, that was our Kilo release and then the small dotted line right at the end was the Liberty release and that'll be the same on the next graphs. So this first drop, as this goes down, it's better because this is the time it takes to fill one RPC call. So we went from 12 seconds down to, I think this was about nine seconds. This was a patch that added a DV relationship directly between a neutron router and its interfaces. Before this had to be inferred by looking up all the ports that had the same device ID as the router that used to exist. So this resulted in a performance improvement. Hierarchical port binding, which didn't really have anything to do with routers at all, for some reason gained us some performance here. This changed some of the internals of the get port retrieval on the ML2 side. So we got a performance bump there sort of for free without even trying. Here, this little drop down, down below eight seconds was when we changed the query to get the SNAT interfaces for a router into one bulk query. So before it used to be, if you have 50 routers, we were doing 50 database lookups, this bulked up into one lookup. And well, before I talk about this one, this little bouncy thing here you see is on all the graphs and that was the HP cloud being unstable. It wasn't anything to do with neutron. So here we had a really big performance regression, this add auto address to subnet lookup. It was a part of a DNS feature to match a funny thing we were doing on the DNS mask side. But what happened was it was causing a get subnet call for every port when a port lookup was being done. So a router's asking for, or this is asking for 50 routers. It doesn't care particularly about the subnet information. It just needs all the port information and it was resulting in a ton of subnet lookups. And this combined with another bug actually resulted in a really big performance drop. I'll show on the next slide how bad this one was. And then here it dropped down to about six seconds. You can't see it. There's actually a distinct point by only fetching the floating IP ports that were relevant to the host. And that was the query that Brian mentioned. So that brought it back down to six seconds. And then this last thing that brought it down to, I think it's just under two seconds now, is removing that auto address subnet lookup that was added in to mimic what was actually a bug. So this is how bad that add auto address subnet lookup regression was. It caused us to go from eight seconds to about 270 seconds to fill one get routers call that was going to the server. And part of that was because of this query we have here. It was hard. We had a whole bunch of people staring at it and didn't notice a difference. But the actual problem was that when we were trying to find all the router reports that were, all the router reports that were related to a router that was passed in, we were issuing this query, but the join condition was wrong. So what was happening is it was returning every router interface back from the DV. So with that initial data I had 950 routers that would be attached to two subnet seats or something like that, it was returning some 3,000 ports. And then when you combine that with a subnet lookup for every port because of that other bug we resulted in that huge spike going up. So the next one I wanna talk about is L2 agents get device details list. This one is, from the L2 agents perspective, it gets all the information that's necessary to wire it up, connect it to the network. It has all the encapsulation info, so if it's VXLAN, knowing what VXLAN tunnels are set up, or VLAN tags, that kind of stuff. It also has extension info for quality of service, the port security extension to see if it should disable port security. And the network obviously, so it can put all the ports on the same network if they're from the same, from VMs on the same network. So this one unfortunately is one that got worse over time and kind of illustrates why we need to add tooling in that automates this process so we prevent these regressions. The funny thing is this first jump was from a commit that was refactoring a call to avoid issuing unnecessary database queries. So this was a case where this was submitted to improve performance of a completely unrelated call that was going to get subnet so it resulted in a preemptive database join that ended up slowing down this code path while it was improving another one somewhere else. This time, hierarchical port binding, the driver changes resulted in a jump from 12 seconds up to 13 seconds and the database changes for it too also resulted in a jump. So with hierarchical port binding, now with every port lookup in ML2 you also have lookups of network segment information and subnet information that are implicit and the L2 agent is getting all this back even though it doesn't necessarily need it. So here we can see someone came through and did reduce DB calls and get device details list and that kind of brought it back down to where it was before the hierarchical port binding DB changes and then one more change to avoid eagerly loading subnet range or the eagerly loading the IP allocation range from the allocation pools but that still didn't get us back down to where we were way back at the start of Kilo and this is when Kilo was released. So Kilo released with like a 20% performance regression on this particular RPC call. And these two jumps here or this jump here, this one was my fault from the network RBAC work that I was doing resulted in a new lookup to another table so that resulted in a two second jump. This is something we'll have to come back and optimize because from the agent's perspective it doesn't care about role-based access control at all. Quality of service because it added more information that the agents need resulted in another bump up, another second that it takes to fulfill. This call is to retrieve 400 devices so. And the last one was another one to improve the performance of an RBAC query that ended up impacting this one. That one went in right after the Liberty release so Liberty release with another 20% regression on this particular RPC call. The L2 agents security group info for devices call is pretty easy, it's just get all the security group information for a given list of port IDs. This includes all the rules for each security group that the port is a member of and then all the member IPs of each security group so they can know if someone has a rule that says allow everybody from this security group they get a list of IPs that need to be converted into IP tables rules. This one's pretty easy. We started optimizing this back in Juneau quite a bit so a lot of the optimizations were almost completely done by the time Kilo opened for development. So we just had one patch that went in that improved the performance of this quite a bit. Just batch DB lookup of security group info so it turned a whole bunch of database queries into one big one and then it's pretty much stayed the same sense. So to prevent regressions in the future what we wanna do is figure out how to add the native RPC support to rally or launch the little translator tool I had in the rally gate job and then add rally SLAs so just say things like oh 50 ports should always come back in less than two seconds and without consistent hardware we can't have them the SLAs be too strict so we might not be able to catch 10% 10% bumps but we can easily block out the ones that double or triple the amount of time that is supposed to happen. And then what we can do for these smaller changes is have a periodic task that's running on the same type of hardware all the time and just collect the historical data and watch the trends look at it every couple of weeks hey is our performance of these calls been getting worse over time. So that's it, any questions? Okay.  The work you've done with the performance I think that's really great and it's just wondering is there any plan to basically capture that in Neutron and use it to monitor the fixed coming use it as a guideline that if you pass certain threshold that make the performance going down then. Yeah that's what I meant on this last thing with the rally SLAs. Rally actually makes it very easy to do once you get something into a rally job you can just say this should take no longer than this amount of time to complete and then if it does take that long fail the job so it would just be like any other failure like a unit test failure or functional failure Jenkins would come back and say hey this rally job failed and then you could click on it and see hey there was a big regression in this particular. So that's a requirement for any fixes coming into quantum. That's what I wanna add it's not in there yet but I have passed it up to start. Okay great thank you. Yeah a lot of improvements have been shown in from all the speakers there was no explicit like time reference. So for example as an operator I have experienced that a lot of people are running open stock still on Juno and the still they have to upgrade to kilo but all these improvements that you have shown they refer like if you do an upgrade path from kilo to liberty right? Yeah I see no these so what I was how these graphs were generated was just from the master branch checking out way back and then just replaying history basically as we merge changes so this isn't representative of what has been back ported so that won't be the exact performance whenever we have a big performance improvement usually we'll back port it as well so someone upgrading from Juno to kilo you might actually have better performance than what I show on here that we release kilo this is just how the performance was right as kilo was released it doesn't include any back port information. Okay thank you. Yeah I know a lot of the IP tables changes I think were back ported to kilo. Right late in the state late in the cycle because we noticed them on some of our performance runs. It could be nice for example we also do rally tests to see the shape of our infrastructure. So it would be nice like to have some reference numbers so that people that run rally consistently on their infrastructure they can see if they get values that are more or less in shape. That are what they're supposed to. Yeah exactly so that would be nice like to advertise to operators. Right and that's another reason I kind of want to get this stuff upstream so then other people can test this stuff on their infrastructure and see if there's deployment cases that we're not considering correctly in our performance. Thank you. Thank you. Thank you.


Coming to you from our GIS Studios, I am Anicia Antoine and this is your Midday News Brief. The Souffre Original Development Foundation, SRDF, has announced the opening of the long-anticipated Souffre Beach Park. The project was conceptualized 10 years ago when the Souffre Original Development Foundation, together with the hotel sector, recognized the need for recreational attractions within the area. With assistance from the Taiwanese government, development plans were drawn up. The facility will employ one administrator, two janitors, seven wardens, including one beach ranger, two lifeguards, two hostesses, and one maintenance person. In addition, 11 local entrepreneurs are conducting business within the park. The official opening ceremony for the park has been scheduled to take place on the 5th of April 2019. The United Nations Educational Scientific and Cultural Organization, UNESCO, is assisting St. Lucia's Cultural Development Foundation, CDF, in identifying and preserving the island's intangible heritage. A five-day capacity development workshop held recently focused on the implementation of the UNESCO Convention for the safeguarding of the intangible cultural heritage at the national level. Cultural heritage encompasses the practices, representations, expressions, knowledge, skills, as well as the instruments, objects, artifacts, and cultural spaces. Residents of Canaries teamed up with the National Volunteer Programme to conduct a major cleanup exercise as part of efforts to ensure the community's resilience to natural disasters and climate change. In the spirit of Kudme, the residents of the community joined in the activity along with 50 volunteers. President of the Canaries Volunteer Chapter, David Henry, hopes that this form of volunteerism can stay residents towards creating a healthier and sustainable environment. This has been your Midday News Brief. Thank you for watching.


Live from the Mandalay Convention Center in Las Vegas, Nevada, it's theCUBE at IBM Insight 2014. Here are your hosts, John Furrier and Dave Vellante. Okay, welcome back everyone. We are here live in Las Vegas for IBM Insight. This is theCUBE. I'm John Furrier with Dave Vellante, and we'd love to talk to all the influencers, thought leaders, experts, but we'd love to bring in also the IBM tech athletes, Bob Pitchie, John Furrier, senior vice president of the analytics group here with IBM. I know he's got a short schedule, but it's great to have you in, swing by theCUBE and share your color and what's going on on the keynote. So give us some highlights. What was the key things, a couple of things that we would like to talk about, but give us the quick highlights of the keynote. Sure, look, John, I'm happy to be here. I'm happy you guys are here again. I think this is a great venue where people can actually hear a diverse set of opinions about what's going on here at the conference from wherever they are. You guys reach out very wide in the marketplace, so we're grateful that you came here so that the story about all the innovation that's happening here is going to get out into the marketplace. So the keynote, I thought it was great today. It allowed me to talk about something I'm very passionate about, which is I see that we're at a new inflection point on an IT value creation basis. And that inflection point is really an inflection point that moves from process-based value creation to insight-based value creation. And it is right in line with our view that we're moving out of the programmatic era of computing and into the cognitive era of computing. And when you think about the programmatic era, it was all about codification of business process and logic and driving that into applications and industrializing business process. But when you think about the insight-based economy, it really has to do with the deployment of rich analytics, analytics-empowered individuals that are really going to get the discipline of predictive analytics employed for them in the business process that they're trying to serve and optimize, and that will allow the organizations to scale. And that insight that they're going to derive from that data in context of the business process, in context to the person and systems of engagement, in context to the business areas, that is going to be where the value curve is going to hit. So process improvement, you know, go back to the Drucker books and all the management consulting, things we've read and we were getting our MBAs and trying to study up, that's old school, right? I mean, that's client-server, that's computing, it adds value there. Certainly process improvement is great, but now you mentioned this new insight. Just drill down on that a piece, be specific. I mean, there's still process improvement happening, there's some sort of transformation going on in parallel to this new insight. What is that dynamic? Can you just unpack that a little bit? I will, and I'm careful to call this an inflection point, not a shift, right? And there's a distinction between an inflection point and a shift. You know, when a shift, everybody moves into that new space. And an inflection point, in fact, both continue to be important, but that new value creation that gets additive on top of the improvements that we're continuing to make in a process-based economy continue to exist. And one of the things that fuels more efficiency in that process-based space is really cloud computing. And all of the economies that are available in cloud computing, and importantly, the integration of hybrid cloud or hybrid IT between using the efficiencies of systems of record that clients have already invested into today in collaboration with new services that they're going to bring online, either in dedicated private clouds or public clouds. And more importantly, I think really, as it relates to the data, I think it's about data and cloud and engagement. So I see more companies now concerned probably more so with the data that's available to them outside their firewall and how to use that in a collaborative manner with the data that's inside their firewall. So they need a fluid data layer that matches the fluidity of the cloud as it has done for business processes. So what you're saying is the process improvement external is a process that has no process. That's unstructured data. You don't know what's going to come at you, whether it's Internet of Things, consumer experience, live events. Well, I do think all of those are still in the context of a business problem in process. Even Internet of Things, right? Maybe we're looking at asset optimization or predictive maintenance and quality. So the business process there is about how I ensure that my capital investments are going to continue to stay online and introduce the least amount of disruption to the clients I'm serving and the clients they're serving. So I still think that's the business process. Unstructured data has an enormous role in the context of being able to do it. Dave wants to jump in, but I want to add one more thing. One problem that we see in programming right now, the old namespace console, here's my namespace, set it up and do all that stuff, is a data space. Observation space, as Jeff Jonas talks about it. The big problem we're seeing is customers set up their data space and they think we're good. And then new data comes in, they've got to re-engineer their infrastructure. How are you guys? Is that software, the magic bullet there? What is the key aspect that's going to change that to? I set up some base infrastructure. I can stand up stuff, software. And then how can I be agile? Look, I think it is not just software. I think it's very specific discipline of software. And I think it's around context computing. And so, you know, data changes data. And sometimes that data is a temporal element that's going to allow me to go back and reassess other decisions that I've made up to this point. Because it could significantly change that next decision I make. So this aspect of context, especially as it relates to all the ambitions around omnichannel interactions, richer systems of engagement. When you go out into any of those spaces, the notion of a master data object, whether it's a person, whether it's a product, whether it's a company, it gets more sparse. It's less clear, there's less focus. So you have to use more of the context computing fabric to be able to bring that data alive, so to speak, and utilize more real-time influences to understand what's going on in the business moment and how I should assess the best decision I can make at that time. And many times, the thing that I learn has to go back and update the past to be able to give me the best insight into the future. I'm intrigued by this notion of a fluid data layer, Bob. And if you look at historically your clients within any industry, whether it's finance, manufacturing, healthcare, media, they've built their stacks. And those stacks are hardened. Production, manufacturing, partnerships, the ecosystem cement hardened stacks. And this fluid data layer seems to be changing that. We heard from Pratt and Whitney this morning earlier on how they're taking advantage of this. But he's stressed, look, it's early days. So can your clients, how can you help your clients take advantage of this fluid data layer? And not just the data layer, it's obviously cloud. It's systems of engagement. Fluid analytic layer. So you've got this digital matrix now emerging. And it seems to be news business models are riding on top of that. I wonder if you could talk about what you're seeing in that regard. Well, look, I think it's a good point that you raise about the rigidity of some of the client environments as it relates to data processing or analytics processing. And clients that have gone down a route of being proprietary locked in. And I even see some vendors make an announcement recently about, hey, we're in the cloud now, we're great in the cloud, and then you look at what they're doing. And they're just trying to create new contemporary interfaces into that same proprietary lock-in. So we step back from this and we said, look, there's a new reference architecture that really should be laid into the foundation of companies as they think about the role of the chief data officer or chief analytics officer or as they think about these problem-solving arenas. IBM does not approach this as a technology problem. It's a business outcome problem. And if you map that reference architecture to the capabilities you already have, it becomes clear what I can use, both in terms of the data that's available and the processing zones or analytics zones that are available and how that needs to be augmented with new capabilities like context computing or stream computing or new front-end capabilities like Watson analytics or information privacy, security and governance or unstructured data management. All those things sort of light up, if you will, against the contrast of a reference architecture to say, all right, if I want to solve that problem, I really know how these other zones of capability, these other analytic disciplines, and they don't have them in that existing rigid architecture. So since we're based on all open interfaces with a reference architecture, I think we can come in with agility and help our clients make that difficult traverse. So and not all customers are going to get this to the same degree of each other, right? So you have a spectrum. Sure. And so can you discern, and what are the characteristics of those organizations? Let's keep it a traditional organization, Fortune 1000, that are actually leveraging whether it's that data fabric or the cloud fabric, the engagement fabric. Can you discern the ones that sort of get it? I hate to use that term. Yes, the Kool-Aid. Because we all love to talk about the sort of mean, but there are 10% that are really, really good at it. Are you starting to see those guys emerge? We are definitely seeing the standard deviations emerge. And we've released this research paper around the generation D enterprises, and those generation D enterprises are really data savvy, analytics savvy, data centric in the way that they think about things. Now, I see great organizations. I mean, Larry is a good example of this, sort of understanding that in order to really get the value of this opportunity, it has to be a marriage of line of business outcomes with what he can do in an IT fabric. So, you know, a very interesting thing that Larry did when he engaged with us is he said, look, I have two years of data. I'm going to give you guys 18 months, and guess what? It's the first 18 months. If you guys are so good, I want you to use your predictive analytics to tell me what happened in that last six months. And we were able to nail, with a 98% accuracy, what happened in that last six months. Okay, who's going to win the World Series? I'm in Vegas. I'm going to put some bets down. So, you know, we track a lot of data points. You're holding that secret weapon in the table. It's like, you've got nukes, and you're not bringing them out yet. There's some things I just can't talk about. So, we've got the World Series. I've got to ask you. I know we've got a couple of minutes left. I asked it earlier. Kansas City Royals obviously got dominant pitching, and the Giants are scrappy, and they always find a way to win. And we were talking just earlier about some of the sports radio commentary, and they say the Giants win because they have a mindset. Sometimes they go up against good pitching and can hit, but they come in, and how they approach the game is what makes them unique. So, I've got to ask you. The practitioners that are really pushing this scrappy trying to get the inflection point rolling has got that serious heat coming at them, pitching from, no, stop. We shouldn't be changing. It's too expensive. How do you approach that? What's your advice to customers saying, here's an approach to take? Well, look, it's actually a great set-up question because it relates to your question as well. It's not about going in and trying to hit the Grand Slam to win the game in the ninth inning. It's about production.  Everybody took the opportunity at the plate to contribute something in the form of a single to get the runners around the bases. Great approach. And that's a great approach. So, when you think about the quick wins that we engage in with our clients, it's about the line of business and IT agreeing there's a quick win here. Let's not try to boil the ocean. Let's not try to deploy a full reference architecture or try to drive forklifts around in the data center. Let's identify that single, the one that's going to move the ducks around the pond and let's start there and we'll get the buy-in. Small ball. Small IT. Small IT. It's because it gets infectious, right? Culturally it becomes infectious and you get this domino effect. So, I think it's hard to bet against the Giants right now with the way they're producing. Let's see, Kansas City has got tough pitching. The fans are crazy and Kansas City are very loyal and I wouldn't be surprised if they could run the table. I'm a Red Sox fan in general. Well, thanks a lot. I really appreciate you coming on. I'll give you the final word here. What's the vibe? And I'll see the insights now. The new show was called IOD, which is information on demand. I'll see the insight. You nailed the inflection point. Love that concept. What's this going to morph into? What's your vision for the show? It is really bad insight and it really is about client outcomes. One of the things I'm very excited about is the speed and the completeness that we're introducing our portfolio as composable services in Blue Mix and the cloud and also as software services when we think we have a differentiated analytics platform. So when you really look around, we've gone and produced, I think, like nobody else has with the range of data services and the range of analytically powered software services that our clients could take advantage. And when you talk about agility and doing those quick wins, sometimes that's the best place to start. Developers are the new king makers, so we want to put the tools and the jewels in their hands to take this thing to the next level. So comment on the economics of the cloud as a final word. Obviously, there's always the calm before the storm in terms of leverage. The cloud is the ultimate leverage. You look at the economics of what the cloud is in here. He was talking about some of the new services data works. It's all self-service. I mean, when the flywheel gets going, the economics are pretty significant. You guys have great software presence in there. Just share with the folks some of the executive conversation around the economic leverage that's going to come out of this engine of innovation. Yeah, and see, I thought you said I had the last word. No, I want you to get that out there one more time. Well, I do think it's undeniable. It's about data and cloud and engagement. So I even think, John, if you stop at just data and cloud, it's not enough. I do think you have to think about what system of engagement, what new analytically powered outcome are you going to drive? How is that going to change the course of me seizing that opportunity or creating a new business model? So I think it really has to be all three, and that cloud is an ether, not just for the productivity of the developer and what they can do in terms of composable services, but also the fluidity of the data access that it gives you, both things that are inside the enterprise and the things that are outside the enterprise. Generation D is here. Bapachiano, Senior Vice President here at IBM inside theCUBE, sharing his insight and a drill down on some of these important topics. Data changes, data, cognitive computing all happening. This is theCUBE here, live in Las Vegas for IBM Insight in the social lounge. We'll be right back after this short break.


Hey there, happy new year. Happy new year. Got Harvey here as well. You should have brought them flowers down here. Yeah, yeah. Not for this meeting. Hold update. Oh, you already put the one up there. Yeah, but the drone's back here. There's an emergency conflict. Yeah. I want to, yes. I'm going to show you the get to it. Works for me? Since we're starting with V3API update, I think that's up to you. Okay. So V3 Alpha, many of the major PRs were merged last week. So anyone who's working with the APIs internally, if there's a V3 Alpha alternative, you should be using that. Most of onboard internally is updated. Now, in theory, they should make absolutely zero difference to configurations between just like bootstrap, which comes from the wire. If you notice issues, please speak up and say something because we are planning on freezing V3 after V3 in the coming days. And also cutting the 1.13 release. Ideally before end of week. So a lot of this lies in us getting a sort of useful signal from folks who are actually, you know, working on master its head and who might notice things, issues that arise as, you know, they actually do rollouts and they, they bump to pass where we've cut V3. We've noticed in like the recent, a couple of recent PRs, there has been some memory increase in like per endpoint and per cluster. The per cluster one is not significant. Per endpoint one, I'm actually about a merge one, which will increase the minimal endpoint overhead by about 20%. Which is something to do with V3. It has to do with switching away from house to cluster load assignments. I don't know why exactly, but you may notice that you may also notice some additional CPU usage. If you do a lot of configuration processing, like for example, you know, ingesting large configurations, some of this is to be expected. We haven't quite quantified what that is. We expect it not to be like, you know, a factor of two blow up, but it could be a single digits or low double digits overhead. So we would like to learn more as we progress, but I don't think any of these things are going to be show stoppers for the 1.13 release. So onto the 1.13 release, it's as you know, it's time for our quarterly release. And I'm not sure there's been a lot of stuff which has happened in the last quarter. One of the major things will be the V3 API. There's lots of other things which are going to make their way in there. I'll be looking at kind of that release later this week, if we don't hear any, you know, if there are any fires burning. Yeah, I'm not sure what else there is really to say there. Okay. I think that's a good summary. Are we, so are we shooting for the end of this week or do we think that that's not, not realistic still? I don't know. How are you going to deprecation?  That's my guess. I have everything set up. It'll take me like an hour per PR. Once you land yours. I was reminded last night. Oh jeez. I messed up. Okay. I'll do that. I'll do that today. Did you do the, the changes to the runtime thing or did you know something? Yeah. We'll sync. I should be able to get one of mine, if not both of them in today and the other one in tomorrow. Like again, 98% of the work is done. It's like merging Harvey's and doing a tiny bit of cleanup. Yeah. It's like, like spelling of config strings and tests. Okay. Cool. Yeah. I think. Yeah. Yeah. I think that on those runtime keys, my personal opinion is that we can play it a little fast and loose just in this sense that I think people generally are not setting them. And if they were setting them, they should have told us. So I think it's okay to just like change the keys and just make sure we change it. We changed it to something that made sense. Yeah. Yeah. Yeah. Yeah. Yeah. That should be. Awesome. So yeah. So I think optimistically end of week. If there could be potential slippage inch to inch next week, but definitely no later because we have a bunch of other reasons. That I can't slip further. And particularly like right now we would like folks to refrain from merging any high risk PRs. So obviously you kind of made in that state indefinitely. Yep. You should take that on maintainers because not everyone's here. I did on the channel. Oh God. I'm so. Thanks. Okay. Okay. I just wanted to briefly mention this has come up before, but and it came up again recently with the increasing number of extensions that people are trying to merge into the main repo. I think we are going to have to start thinking about either initially doing multiple Docker images. So you could think of like a, you know, like a core Envoy version with like only security vetted things or something like that. And then there's a, yeah, per that, per that tweet. There's the Envoy ultimate edition, which has all of the, all of the extensions, but I think the next logical step here, which is something that the community is really going to have to talk about is whether longer term, we keep every extension in the main repo, or we end up having a, you know, something like Envoy filter example that ends up being a little more official. Just in this, so like all of the overflow extensions would go into that repo and the bar of merging those extensions would be less. And then if they want to get them back in the main repo for inclusion in like the core images or something like that, you know, we would require fuzzing or like various other security type things and documentation and stuff like that. So there's no real policy here. I think I'm going to put something together for, for community discussion in the next couple of weeks. But if, you know, if folks have thoughts here, you know, I think that would be useful, you know, or if anyone has any thoughts on the call right now. Yeah. I think that the thing to notice is that submit to Envoy proper their extensions for a variety of reasons. Now, one of these is because, hey, part of the official Envoy distribution is going to be everybody's Envoy binary. It's going to have and so on. So the other is get CI coverage. The other is like so that it's, you know, like probably I don't know that it's considered to be one of the high quality Envoy first class extensions. I figure we can at least provide the CI coverage one by this external repo and that's essentially what we'll be shooting for, right? Let's think if you want them, you're going to have to work for them at a, probably a higher bar, right? Yeah. So if you're, if you're interested in this topic and if you're, if you're a vendor, you probably are, you know, I would say definitely reach out. I think it would be an interesting conversation to have. Does this, oh, quick question, Matt. Does this then broach the question? Bring us back to the question of loadable modules at some point. Yeah.  And there's the open issue there, which again, I don't really think it's that hard. Like I think someone could implement it. I don't know that that solves the question.  Yada, yada, yada. So I, I, I agree that loadable modules potentially makes it better. I don't know that it really fixes the underlying problem in the sense that people are still like many vendors today from what I've seen in the envoy ecosystem is they're expecting like people to bring their own envoy and have it just work. And that means that all of the extensions are already bundled in. So I think just from a community perspective, you know, there's been some ongoing work in like having config dumps, say what extensions are compiled in. It's just like, this is a very complicated topic where we have a tension between people that want to have a very limited trusting trusted computing base and vendors that want like the kitchen sink so that they don't need to worry about deployment issues. It's, it's quite complicated. Yeah. I think if we want to go to loadable modules, simply based on the way of assembly, no VM, even if it's there, C plus plus modules will be the only thing that really makes you avoid having significant breakages. Yeah. Basically on a longer basis with no internal API stability. Yeah. Anyway, if you're if you're interested, please reach out to me on Slack. I'll type something up and we can we can hash it out in a doc. The only other thing that I wanted to mention about the stable release policy is that this has also been kicking around for a long time. I think post one 13. We're probably going to start a policy where we start maintaining multiple stable branches. So that will include probably back porting security fixes to at least the last two releases to give people at least a three month window to upgrade. And it will probably also mean in conjunction at least with the Istio team of back porting bug fixes also. However, this will require community stable maintainers. So we've talked to various people who might be interested, but I would expect to see another policy update here probably in the next couple of weeks. But we will need community help. So if you're if you're interested, please also reach out. I thought Istio wanted at least a year worth of releases. I think it right. So I think if they, I think if people want more than X, they're, they're going to have to pony up the resources, but that's something that we will have to hash out here. Like I don't, I don't think it's reasonable to ask, you know, the primary envoy security team, the upstream maintainers to say back port security fixes to the Istio team. So I think that there's going to be a lot of resources for releases. It's just too time consuming. So I think that, but these are the kinds of things that we will have to hash out for sure. I mean, technically they already have resources because they're already doing. No, right. Right. So like this, this, I mean the, the quote get resources may just mean that the Istio people agree to do it. And if they don't do it, it just doesn't get done basically. But again, you know, I mean, I think it's really important that the upstream maintainers will and won't do versus what the community will and won't do. So for example, it's great if the community does X, but I think it's more important to say that the upstream maintainers will not do X, meaning like the upstream maintainers will only back port security fixes to the last two releases or something like that. But again, we'll help with the scaffolding.  But again, we'll help with the scaffolding to, you know, allow people with early access to do the work to do more, right. Yup. I'm done. I'm going to guess at Yon for anyone hasn't Yon. Welcome in Yon. So what actually now and on the. I think in. I think. Is that a lot of awesome work in the areas of. the areas of like security fixes from Netflix, they actually do issues around codecs, around bootstrap sort of like sort of node enhancements for exactly this kind of thing that Matt was talking about for extension discovery. And also in the space of like the APIs and so they made a bunch of contributions towards the V3 API. So yeah, that's what we've been super awesome. And I think, I don't know, Leon, do you want to say anything about the way that we think about this? Well, I think the plan right now is to focus on some of the data plane work and make sure we get some sanity there, maybe doing some refactoring and in general helping the reviews and other stuff. Great. And then if at all I wanted to talk about file system normalization. Yeah, just to one thing that we, most of this discussion happened on the OSX dev channel. It just turns out that C++17, standard file system, just still remains completely unusable or unpredictable between POSIX and Mac and Windows and so forth. So we finally realized it's easier to just deal with all of this in terms of POSIX or in terms of well POSIX or Windows specifics. So we're basically looking, we've almost got a patch together that gets rid of standard file system altogether from the working tree. And I'm also looking at the fact that there's a tremendous amount of overlap in the API between the way we're doing IO handle for sockets and the way that we're doing the file system API. It looks like there's got to be a super class of that or we should be able to abstract this enough that all these things look similar and are usable by everybody without a great deal of hassle. So that's our current project and why we've kind of gone off the rails again in terms of getting the core master compiling on Windows because this problem was to be dealt with. Yeah, on the standard file system side, I think that makes sense to me just because I think that will likely allow us to upgrade to C++17 since we already know that on Envoy Mobile, like standard file system is not going to work for the foreseeable future. So my advice is we already have the file system abstraction. I think as long as we just plumb that through everywhere and then even if it's not implemented yet on Envoy Mobile or in Windows or whatever, we can get that done. And then on the IO handle stuff, there's more work that needs to be done there. So if you could flow any proposal by myself and Dan, who's working on Quick, she and I can take a look. I'm happy to do that. Actually, I realize now what your point is at me because the next agenda item is mine. I just wanted to put in a plug before I release it. We've been iterating a little bit on a proposal to experiment with improving a header map performance in the worst case. And it's almost ready to be distributed. And I just wanted to see if, say, if you are interested in reviewing it and be on Slack, I'll of course put a link on the channel as well. Could you get, I mean, could you give some, like type of quick summary of what it is? Sure. So there's some order and squared, there's some order and operations around headers, which the system doesn't know about yet. And there's also a little bit of a conflation of being fast versus coalescing behavior, which I think would be better to clean up a little bit. And actually it doesn't, the doc does not specify what to do. It just gives a bunch of options and pays some performance graphs from what I had experimented with like a month ago, just to kind of get that out. And I wanted to kind of get somebody full-time to work on this. There's also like another issue, and that is like every time we show this code to security auditors, then we tell, we put them at envoy. I know Matt's looking away because he has solutions here, but historically every time we have security auditors, folks zoom in on this code. It's not that the code is incorrect. It's that it's, it burns time with the auditors because they're looking at very low level mem copy, malloc, you know, buffer resizing kind of stuff, which definitely, and I realized with the app salvers, like, you know, replacements for that, whatever we do should hopefully simplify things and make it so that this is not the focus of auditors time. Auditors can focus on like the interesting stuff. The only thing that I would say is, I think there's two separate things here. And what I would suggest is that the proposal that I put out to replace all of the low level stuff with AppSeal, I think I'm hoping that will be non-controversial because I think algorithmically, that should roughly replace the current implementation with something that is much easier to understand. So like there shouldn't be a lot of discussion around Perf, like we can do basic benchmarks, but it should be roughly the same. I think what Josh is talking about, because I think Josh, this is based on that PR that you did like a month or two ago where you did this adaptive thing that like occasionally makes a map or something like that, right? There's a couple of other options, which I never, I never prototype, but I listed out there. Yeah, but functionally, like the last time we did audits, I was doing audits for like M squared header stuff and there were a couple that were like, you know, we get these reviews where someone adds a new header lookup and it's like finding this header is going to be, you know, organizing it and like adding those up and it's like, it'd be nicer to just not have to worry about it and say the status structures are efficient enough, we don't have to stress them again. And also like one thing I said that Josh is looking at it a while, hopefully we were discussing this in a day, was like, could we just rethink everything and start with a simple map and then go from there and like, yeah, maybe that isn't performed enough, but like let's actually do some simple benchmarking and actually, you know, start off. Is that any of the simplicity here could allow us to remove a lot of the other things that have historically been complicating here.  And we should just like, get a clear understanding of what complexity is buying us where we have it and what is the essential complexity and what is actually optional. I think that's all fine. I mean, and that sounds like a great use of time to actually look at this from a holistic perspective is something that I've wanted to do for a long time. My point more is that I think we can land a functionally equivalent code cleanup relatively quickly. And that will be probably non-controversial and make auditors like much happier. So I would suggest that we just do that because I don't think it's that hard. We'll give us a little more breathing room to hopefully look at this from a, you know, like a top-down perspective. Also, Josh is cleaning up, I think links to like the four in-flight open issues we have regarding header cleanup. And it's saying like, let's do all of it. Yeah, yeah. Yeah, I think that's the most uncomfortable part. Although I think that if we make some data structure changes, I'm not sure if the other data structure swaps will still make sense. No, they probably don't. It's just, I think it's just a matter of timing again. Yeah, because we really have weeks of analysis to do before the real one lands and you could probably do the actual swap one in a day. Yeah. Yeah.  Probably not. So I'll defer to all of you to decide what, you know, what we should do there. But yeah, it would be great. Like this is one of the areas where, you know, that code is really from three and a half or four years ago. And it hasn't really had any, I would say rigor as performance analysis since then. I think that's it. We had for the agenda. Does anyone have other things they want to talk about while we're here? Yeah, this is Gary. I just had a quick question about the dynamic forward proxy. I saw that it was checked in there, Matt, back in the summertime. And according to the documentation, it's done as an alpha state. I haven't seen a document document update since then. I've been playing around a little bit with it. It seems to be stable. Didn't have any issues with it. Do you have an update as far as, is it expected to be stable or are there other things? To be, to be perfectly honest, the way in the past that we have gone from alpha to stable is when I hear about people using production and issues. Sorry, there's some, some bad echo there. So I think it's probably stable and we can probably remove that tag. Yeah, like again, we don't have any rigorous process right now moving to alpha this table. As robust, I think it's marked as robust to upstream and downstream untrusted upstream and downstream. So it's sort of surprising that it's still in the alpha state if it's marked like that from the security perspective. I think there's, I think again, this probably needs a little more rigor in terms of how we mark different things. I think the security perspective is obviously what the intention is, meaning we would fix any issues that come up there. I think the alpha status was more just typically when we put out a new filter that doesn't have a lot of production coverage, you know, I think we're just trying to say that it doesn't have a lot of production coverage. But I do know implicitly that there are people using it in prod. So I think it's probably okay to, to, to switch it now. But again, that's yet another process thing that we probably need to look at. So I will, I will make a note there. Okay. Yeah, that's great. Thank you. Has anyone done any sort of scalability testing in the sense that from the dynamic hosts that are sort of at scales to I don't, I don't know. I think the code has all of the right circuit breakers in place to avoid unlimited expansion and things like that. So I think that you could do your own measurements, you know, to, to, to see, we definitely wrote the code like with it in mind to, you know, use a limited amount of memory, but I think you'd have to do your own measurements. Okay, great. Thank you. And last question I had is currently the HB connect support is present yet. Are you aware of anyone that's planning to work on that? There's, there's someone that said that they were going to do it, but I haven't seen any progress on that. So I would assume that it's not being worked on right now. It's something that I would love to see land. I think we have a pretty good idea of roughly what needs to be done and it would just require a development resource. Thank you so much. Anyone else? Hi. Are you hear me? Yeah. Yeah. I'm working on, I'm coming and I'm now I'm working on the and I want to enable the employee city on 64 platform. But I have a few questions about it. The first question is which cloud platform is chosen by the invoice.  The invoice is a CD platform supporting the arm 64 platform does support arm 64. Right. Right now we're doing our, all of our builds through Google's RBE service. And I, I don't know, but I doubt that there are arm hosts available. So it seems like it, at least if we do it through that system, it would have to be cross compiled. I know there are, I know there are a couple of people running arm builds right now. There's an open issue for it somewhere that has people linking to their builds. But as far as I know, there's nothing official. Yeah. Okay. And Google cloud. Sorry, what? Is it on Google cloud? Yeah. So right now the RBE system runs on Google cloud instances. My, my advice would be to follow up with Lee's in who's not on the call right now. He's our, he's our build guru. I, I don't know that anyone on the call right now is going to be able to, to give you like in depth low level details on this. And, and as a heads up, Lee's on is out of country right now.  So email may work better than, than Slack. Okay. Thanks. Or Slack understanding there's going to be eight hour time lag is also fine. Hi, this is a rule off. I have a question. So you're currently using the Google Cush for quick. Would you be interested in another implementation using NGTC to make it work? I'm probably not. At least not without someone doing a ton of work on their own. It's just like getting it, getting it working at all is a absolutely herculean effort. So, you know, Yeah, I would say from, from experience, I, I tealed the quick launch over at Google. It took us like two and a half years of tuning to sort out all the weird corner cases in, you know, the pathological act patterns and stuff like I wouldn't want to go through that again, which is why we're doing the keys integration for envoy is that's what we wanted. We have, I don't think we have objections, but it's more like the, the open SSL versus boring SSL thing of like maintaining Pew is going to be uncomfortable. So if we have a good reason, like let's do it, but without it, it's easier to have one. Yeah. I mean, we, we did build the quick listener thing as an extension. So it should be technically possible to plug in a different quick listener. But, you know, I think that's the kind of thing that you would probably be mostly on your own. And I'm honestly not sure what technical reason, what, what technical problem you would be, you would be solving. Okay. Thank you. Sounds good. See you on a few weeks. Bye. See you in a couple of weeks. Thank you.