HCC 788 supports Girls of the Finest, a G.I. Joe costume club. Be sure to order their 2017 calendar. Proceeds to benefit the charity K9s for Warriors. Find them on Indiegogo.com. Hello everybody, Hoodicover Commander 788 here, and this week I am not in front of my green screen because we have something a little special. This week we are introducing a new segment to this channel called Literal Codenames. That's literal codenames. And this week we are looking at G.I. Joe's second Navy SEAL wetsuit. Wait for it. As I say, we are looking at wetsuits from 1986 and we are beginning the new segment literal codenames. So HCC 788 presents wetsuits. They have to be cold. This is wetsuit, the SEAL introduced in 1986. He was also available in 1987 and was discontinued for 1988, though he did not have a direct replacement in 1988. There was a second version of wetsuit introduced in 1986, the same year that the first version was available. And version two of wetsuit was part of the Special Mission Brazil box set. That was a five figure box set that came with a cassette tape. Wetsuit is of course the successor to Torpedo, the Navy SEAL introduced in 1983. Torpedo was the first naval personnel introduced in 1980s G.I. Joe and both wetsuit and torpedo as you can see are equipped for really just one job and that is as a frogman or a navy diver. There were two other versions of wetsuit in the vintage line in 1992 and 1993. Those later versions fixed some of the problems that the first version had, but those versions had their own problems and we'll get to that later. Seals as a reference to Navy SEALs and SEAL is an acronym for Sea, Air, and Land. Navy SEALs are trained to operate in all environments, they are the US Navy's Special Operations Force. They conduct missions that originate from water or a coastline. Unfortunately wetsuit is equipped for only one environment and that is sea diving. SEALs would of course be trained to do this, but as a SEAL, wetsuit should be able to go anywhere. Let's look at wetsuits accessories and as you can clearly see they are yellow. They have a very slight orange tint to them, but they are definitely on the yellow end of the spectrum. You can't miss them. He has what the contents of the card call an underwater sea sled, fits in his hand pretty well by this handle. This is a very unusual little accessory. This would be used to propel him through the water. It has some nice detail on the bottom. It has what looks like a headlight here to me. It has what, well I thought these were instrument gauges right here, but I think these actually are the propellers that get this thing going through the water. It doesn't have anything that looks like a weapon, at least nothing that looks like a weapon to me. There were underwater sleds similar to this in the 007 movie Thunderball. Wetsuit's next accessory is also not a weapon, it is a big flashlight. The card contents call this an underwater searchlight. This big flashlight really has very minimal detail. There's not really much going on here. It is made out of a softer plastic, so it is pliable. It has this loop on the handle, so you can loop that around Wetsuit's wrist if you want to. It can be sometimes difficult to get in and out of his hand, which accounts for the scuff marks on the handle. But what else is there to say about it? It's a big flashlight. Then he has the hose that goes to his scuba tanks, and we can pop these out and take a look at them. There's not a lot of detail on here, but it has some a little bit. There is a little bit of texture pattern on the hose itself. It has a couple knobs here to plug into the backpack, and we'll look at that in a minute. Then on the mouthpiece, it's designed to connect to the face mask on the action figure. It has these two holes, and those connect to two small nubs on the sculpted on face mask. The holes in the mouthpiece and the nubs on the face mask are quite small, so it can sometimes be difficult to get it plugged in here. But you just work with it until you get it plugged into the face mask. Once it is plugged in, it does stay in fairly well. Then we have Wetsuit's backpack, which contains his scuba tanks. But the scuba tanks must be inside the housing of the backpack, because there are no obvious scuba tanks sculpted on here. You can connect the hose to the backpack at these holes here near the top, and you just plug the ends of the hose into those holes. Then the hose would go over Wetsuit's shoulders, and the mouthpiece would plug into his face mask, as we've seen. This is pretty sophisticated for an 80s GI Joe toy. Wetsuit's backpack does have some detail, and if I were to have a criticism of it, it wouldn't be for lack of detail, is that it really doesn't look anything like scuba tanks. If you compare that to the 1983 Torpedo's backpack, Torpedo's backpack was a set of rebreather tanks, and although it's smaller, it looks more like underwater gear than Wetsuit's backpack does. Plus, Torpedo's backpack had that cool paint application, and that was pretty rare on GI Joe action figure accessories. Next we get to two accessories that are not listed on the contents of the card, his flippers. He has two black flippers. They are both the same. There's not a left and a right one. They connect to his feet by the foot pegs, just the way a figure stand would. They just connect right to his feet like that. They stay on pretty well, and you know they actually work kind of in lieu of a figure stand. Wetsuit stands pretty well with his flippers on. These flippers are not new. They are reissued from the 1985 Cobra Eels, and they are the same. Right down to that black plastic, they didn't even change the color. Normally, I'm not a big fan of reissuing accessories like this, but if you're going to have a standard accessory that gets reissued with other action figures, diving flippers would probably be a good candidate for that. I really don't see the need to sculpt new flippers for every diver you have in the toy line, so these are fine. They're just generic flippers. Compare Wetsuit's accessories with Torpedo from 1983. Both were Navy SEALs. Both were only equipped as divers. But at least Torpedo had a weapon. He came with this harpoon gun. Wetsuit had no weapon accessories. He looks more like a sport diver or an undersea explorer than a GI Joe ready to fight Cobra. Let's take a look at Wetsuit's articulation. He had the articulation that was standard by 1986, meaning he could turn his head from left to right and look up and down. He could swing his arm up at the shoulder, and he could swivel at the shoulder all the way around. He had a hinge at the elbow, and he had a swivel at the bicep. The figure was held together with a rubber O-ring that looped around the inside. That allowed him to move at the torso a bit. He could move his legs apart about so far. He could move his leg at the hip about 90 degrees, and he could bend at the knee about 90 degrees. Let's take a look at the sculpt design and color of Wetsuit, starting with his head. And that head is really something. It has that big, bright, orangey yellow helmet. It has some very interesting details sculpted on the helmet. Up here at the top, it has what looks like sculpted on lights for illumination underwater. It has goggles. It has that infamous face mask. You can connect the hose to that. Of course, this helmet is non-removable. The face mask and goggles are non-removable. He's essentially stuck in his diving gear. The US Navy does use helmets similar to this, even in a similar color. Those helmets are tethered to the surface by hoses. But I'd say this helmet is not really unrealistic.  Whether or not it's a good action figure design is another question entirely. Since the helmet has protrusions, that face mask and the knobs on the face mask, paint wear is frequently going to be a problem. So if you care about that kind of thing, take a look at the face mask. That black paint will often be worn away. On his chest, he has a diving wetsuit that is primarily in, I would call this an aquamarine color. It's a light bluish green. He has a white patch on the chest. And on that patch, he has a red detail here. It looks like a flying fish. This flying fish tampo is fine. It's a nice little splash of color on the action figure. But I think this was a missed opportunity. Instead of using the flying fish, they could have used the emblem of the Navy Seals. And that would have been a nice tie in to his real world origins, similar to the way Gung Ho has his Marine Corps tattoo. There are some other remarkable details on this chest.  And I do think those are very well done.  And his arms have short sleeves, gray straps and black pouches. On his right wrist, he has what I think looks like a sculpted on depth gauge and then on his left wrist, he has a black watch.  Those go all the way around and those look really nice. He has white stripes that continue down his legs. His legs are interesting in that they include the only weapons that wetsuit has. They are sculpted on and non-removable. On his right leg, he has a black pistol and a gray holster. And I don't know what he's gonna do with that pistol underwater, but I guess he's taking it with him. On his left leg, he has a gray knife. Those white stripes continue down his thighs. He has some tall black boots, not a lot of detail on these, but of course the flippers will connect to his feet to complete the look. Let's take a look at wetsuits file card. It has his faction here as GI Joe and it has a portrait of wetsuit here. It has his code name as wetsuit with a hyphen. It has his specialty as seals and I don't think wetsuit is a plural seals. I just think this indicates that he came from the Navy seals. His final name is Brian M. Forrest. His primary military specialty is seal. His secondary military specialty is UDT or underwater demolitions. His birthplace is Myrtle Beach, South Carolina and his grade is E5. This paragraph says seals are the guys who were too nasty to be airborne rangers or marines. The Navy keeps their seals locked up below decks until something bad becomes worse. That's when they all get dumped into the soup on a rubber raft with a knife, a gun and all the ammo and explosives they can carry. Wetsuit may just as well be the roughest one of the bunch. He's wild and unruly, but he's simply the best at what he does. This quote down at the bottom says, wetsuit may be mean to the bone, but he's also quite bright, being well read in both the classics and the standard texts of military tactics. Pretty amazing considering that the level to which he's developed his toughness would seem to indicate full-time occupation. I think there's a text variant of this file card. Some wetsuit file cards I've found online I have in place of toughness, they say orneriness and cussedness. Changing that to the word toughness does not quite convey the same sense of wetsuit being someone who's very difficult to get along with, but orneriness and cussedness are kind of hokey colloquialisms, so it's probably better that they changed it. Wetsuit's entry in G.I. Joe Order of Battle issue number two uses the toughness line, not the orneriness and cussedness line. You know what this file card doesn't do? It doesn't tell you what SEAL means. And Torpedo's file card, right up in his specialty, it says in parentheses SEAL is sea air and land. There's nothing about that on wetsuit's file card, and I guess it's just assumed by 1986 you know what a Navy SEAL is, but all the same I think the SEALs were a little bit short changed and deserved a better write-up in wetsuit's file card. Wetsuit made some appearances in the G.I. Joe animated series and the creative people on that series knew that of course they couldn't just leave wetsuit in his diving helmet all the time. So they took some artistic license in creating a face for wetsuit. Wetsuit had short brown curly hair in the G.I. Joe cartoon series. In that animated series he was often paired with Leatherneck the Marine with whom he had a contentious friendship. You could say they were frenemies. They were buddies but they traded insults. I think that's a good way to develop characters by kind of having them bounce off each other like that. In reality both these guys would be type A's so it probably would be difficult for them to get along together even if they did like each other on some level. In the G.I. Joe comic book published by Marvel Comics wetsuit first appears in issue number 47 and he's right on the cover. And it's a great cover, a great action scene featuring Hawk, Beachhead and the Devilfish. Wetsuit's yellow helmet is recolored red which is still a bright color but it looks maybe a little better than the yellow. Wetsuit appears in G.I. Joe's special missions in the very first issue and he's on the cover and again he looks great alongside torpedo. They changed his wetsuit color to green and they gave him dark green accessories and I have to say this looks good. I could imagine wetsuit in this color scheme. Looking at this figure overall wetsuit is not equipped as a combat troop or even a combat diver. He comes with no weapons other than what is sculpted on the figure and non-removable. And I just don't think wetsuit is a very creative codename. You were supposed to do it once, once. It was a dry spot. Was there ice? I felt ice. His non-removable helmet is a problem for me. He looks wrong in any other environment. He's a seal. He should be able to fight on land. But if you take away his flippers and his scuba tank he still looks awkward in any other environment. This is a well sculpted figure and the bright colors work fine as long as you don't use him in combat. He looks like a sport diver or maybe a rescue diver. Unfortunately this doesn't work well for a Navy SEAL and they did this to torpedo too. I think the Navy SEALs deserved better representation in G.I. Joe. Later versions of wetsuit fixed some of these problems. Version two from Special Missions Brazil looks nice in silver. Versions of wetsuit that were released in the 1990s had a removable helmet, nice, and they had a base black colored wetsuit. That's an improvement, but unfortunately the highlights were still in very bright colors. I call this a middle tier figure. It misses the bottom tier because of that nice sculpting and detail but I've never been a fan of that color scheme. Although later versions improved on this none of them really did justice to wetsuit. He never became the Navy SEAL we deserved. That was my review of the 1986 wetsuit. I hope you enjoyed it. Make sure you check back next week for another Vintage G.I. Joe Twitter review. Make sure you like me on Facebook, follow me on Twitter, support me on Patreon, and share this video to help this channel grow. Okay, I know you've been wanting to do it again. Dunk me. Thank you. This is our last segment of literal codenames. We're not doing this again. There will not be a literal codenames for Skidmark. I guarantee that. And remember, only G.I. Joe is G.I. Joe.


Hey guys, it's Coach Ben from Soccer Entrepreneur, and in today's video, I'm going to be talking about the difference between a soccer trainer and a business owner, and how if you can combine those two, you can have mega success with your soccer training academy. So here's how it was for me when I was younger, when I was first starting, I just focused so much on soccer training and how I could help players get better, which if I didn't have that mindset, I wouldn't be here right now. So I had to have that mindset, I wanted to help as many players as I could. But what happened was, when I'm in the thick of my sessions, when I'm trying to train a lot of players, I'm not thinking clearly about what's happening over here to the business. And yes, you can be a great trainer, you can provide a lot of value, you can give great sessions, all of that is great. But you're going to hit a certain capacity on how many sessions you can actually do per day, how many players you can take on, and how you have everything structured based on your time. And so once I started to kind of take a step back and realize, okay, I have soccer training down, I know that I can help players, I know I'm good at that, I know that players are seeing great value, players are seeing great results. I need to take a step away from that for a second and think about what is happening over here with the business, like how much actual time am I spending every single week over the phone, how many, how many hours am I spending at the field. And once I started to figure out those things, I started to like really study business and figure out, okay, I'm spending way too many hours at the field right now, I'm spending way too many hours on the phone, spending way too many hours trying to do my website. And I started to cut away all of the little distractions that take a lot of people tons of time, like maybe years of figuring out. And once I sorted that out, like my business started to thrive because I took a step away from my training and learned, okay, how can I be more efficient with this? Like, am I charging monthly or am I charging by the hour? How am I collecting my payments? Like what messages are parents seeing on Facebook or through my email newsletter? And once I started to really dive into what's happening with the business, and I still kept the same level of my training and I still kept improving there. Then once I combined those things, that's where I felt my academy became unstoppable because this is the thing. There is a difference between knowing training and knowing business. If you only know business and you're not a good trainer, you'll never succeed. If you know training well and you understand business well, then you're going to succeed at a level that no one could ever compete with you because this is, this is like the raw facts. I've done so much research. I've had so many phone calls with other trainers. I know exactly what's going on in other cities and other states and I see what the biggest problems are. And it's not the lack of training or the lack of interest of wanting to help other players like most coaches I talk to have that. Everything that they lack is over here. It's on the business side. It's how to structure your business, how to set everything up to where you can start to automate your training. You can start to bring on more trainers. You have an automated website that collects leads. You have a sales process. And this is why I started my four week business accelerator program. And here's how I have it set up. I don't want you to go check out that page or anything right now. I would rather just talk to you over the phone and see if you're the right fit for that. Because I've had a couple people recently, they've gone through it. I know they weren't the right fit to go through that program. They did not put the work in and I'm not willing to spend my time to work with those people anymore. So what I want you to do is just simply go to soccerentrepreneur.com slash call. Once you get there, I'm gonna give you more information about that program and see if you're the actual right fit to work with me over the next four weeks. So we can really start and scale your business in a way where I can help you with the business side. And you can be the person who's committed to training. And once I can help you combine those things, I'm telling you, everything is a lot easier. And that's the whole purpose here at Soccer Entrepreneur. I create all the free videos here on YouTube, that's not gonna stop. I wanna continue giving advice. But if you wanna be more serious and take it to the next level, I highly advise you go check out my website there at soccerentrepreneur.com slash call, we can get on a free call together and I can walk you through what's included in that program. So that's it for today guys, go ahead check out that website and I'll talk to you soon.


Well, hello everybody and welcome again to another OpenShift Commons briefing. This time, we have AviNetworks, his Shisha, and they're new members to the OpenShift Commons, and I ran into them at KubeCon a while back and they had a different approach to some of the application network services and working with OpenShift. So I thought it would be a good way to introduce them to the community by having them come on and talk a little bit about what they're doing, and I'm not going to do too much more of an intro because I'm pretty new to it. So, if you could introduce yourself and your colleague who's going to be doing the demos, we'll get started and there'll be Q&A in the chat and after the demonstrations is done, there'll be time to ask questions again too. So please take it away. Thank you, Diane. Thank you very much for giving us the opportunity to present at the OpenShift Commons webinar. My name is Ashish, I'm a Senior Director of Product Management at AviNetworks. With me is my colleague, Bhushan Pai, who's a Senior Solutions Architect, and he will be driving the demo about halfway through the presentation. We'll do, in terms of the agenda today, we'll do a brief introduction to AviNetworks, including the product architecture, and then we will deep dive into how AviSolution works with OpenShift. We'll do a deep technical dive there, and then we'll do a live demo where we will show you how Avi works in an OpenShift environment from beginning to the end and what all the services it provides. If you have any questions, please feel free to ask on the chat window and then we will address them as we go, as well as towards the end. So let's start with a brief introduction to AviNetworks. AviNetworks is about four and a half, five years old company based headquartered in Santa Clara, California, but with offices worldwide, including the Continental US, Europe, including UK, mainland Europe, Germany, France, and the Netherlands, and in Asia, including India and Southeast Asia. And so we have global footprint with R&D and customer engineering. As a company, what we do is software-defined application services, layer 4 to layer 7 services, that starts with load balancing or service proxy, as we call it in a container world, but also application security, including SSL, termination, DDoS, detection and mitigation, web application firewall, and also application visibility and monitoring. So all of these layer 4 to layer 7 application services are built into the same product. And unlike any other solutions out in the market, this is a software-only solution that works across all the use cases, including what's a traditional load balancing use cases, as well as private cloud with OpenStack, public cloud, and container use cases, as we'll see with OpenShift, as well as Kubernetes. We have customers worldwide, including some joint common customers. So we have customers that are using us in OpenStack environment with Red Hat. We have customers using us in OpenShift, customers such as Deutsche Bank and Danska Bank, as well as other customers, these customers running in production today with Adi and Red Hat OpenShift. And also we have customers piloting us, customers such as Intuit and others, with the OpenShift environment. And all of these customers also use Ansible as the automation tool. So as far as the ecosystem is concerned, we have customers with OpenStack, with Red Hat, with Ansible automation and OpenShift for their container fabrics. We'll talk about more of the use cases as we go along. But let me give you a little bit more background on the problem that Adi is trying to solve in general. What our customers are trying to achieve across all the use cases that they're running in on-prem or private cloud or public cloud or containers is going from the left side of this picture with a manual ticket-driven operation to a self-service, fully automated experience with CI CD, with network automation, with application automation deployment, with OpenShift, with Ansible and so on. And the use cases might be different, but the underlying reason, the underlying driving factor for these customers is automation. And that's why automation is built in fundamentally into Adi's solution. We'll talk about that in next couple of slides. So what is the problem today? Why does Adi network exist? Why did we get started? And the problem that exists today is if you look at any modern data center, you have racks and racks of servers. And these are standard x86 servers on which you deploy applications. There is no snowflake here, all servers at x86 servers. You can deploy your applications on any servers. You can start with, let's say, 10 servers and you can scale out to 15 or 20 or scale back to five or six as the load increases or decreases. You manage them as a pool of capacities, a fluid capacity pool. You don't have any specialized hardware for applications. And that allows you to do automation. That allows you to have a telemetry that's built in that drives that automation. And it makes the entire management of the fabric much simpler. And that's how all the modern data centers are built. But if we look at any of the layer for the layer of seven services and especially the traditional hardware load balancers, whether they're F5 or Citrix Netscalers or A10s, they're built on proprietary hardware. They're an appliance-centric model. Even if you look at the virtual appliances, there are appliances, which means you're going to manage them as individual devices. There is lack of automation. There's APIs at an afterthought. And you can't basically automate as an elastic pool of capacity. That is the fundamental problem that ADI solves. So let's see how it solves it and then we'll talk about how it works in an open-shift environment. If you look under the hood of a traditional appliance-based load balancer, that it's a monolithic piece of software running on an appropriate hardware. In that software, there are two components, the management slash control plane, which handles the configuration, the policy, et cetera, and the data plane, which is actually doing the load balancing or other security services. The first innovation and disruption that ADI networks did was applied software-defined principles, where we separated the control plane from data plane. And so we have a concept of ADI controller, which is the centralized control plane, that manages a distributed set of data plane entities, we call them service engines. In open-shift parlance, you can just say, this is your open-shift master and then your individual open-shift nodes. Same concept as far as the separation of control and data plane goes. That's the first thing that we did.  The second problem we want to solve was the heterogeneous compute environment. You might have applications running on a bare metal, on a virtualized environment, let's say with KVM on OpenStack, or you're running in containers with Docker containers running in Kubernetes or an open-shift environment, or you're running in a public cloud environment, whether it's Google, public cloud, AWS, Azure, or anybody else. How do we have a same solution with operational consistency, feature consistency that works across all of the above environment? Well, that's a problem Avi has solved as well. How? Because Avi is software. It can take a form factor of our server by running Avi as a software on a bare metal server. It can be a VM running with KVM or ESXi. It can be a Docker container running with Kubernetes or OpenShift integration as we show in the demo, or it can run as a VM or a container in a public cloud instance. The same software that runs across the board. And the beauty of these two pieces of architecture is that this is managed from a centralized Avi controller. And the way Avi achieves, the Avi controller achieves that automation is through integration with the ecosystem. So Avi talks to OpenShift master, or talks to Kube master, or talks to Amazon APIs or OpenStack APIs to spin up these Docker instances, for example, or VM instances automatically based on demand. So the operational flow is that you communicate to the Avi controller through REST API or Ansible Playbook that's built on REST API and say, I want to have load balancing services for these applications or I'm deploying these applications, do everything.  And the last piece of the last secret ingredient of the architecture is the built-in visibility in analytics. So these service engines, which are your proxies, are running, are sitting in line with your traffic, right? They're intercepting the TCP, the HTTP traffic. And as part of the proxy functionality, it's collecting hundreds of logs and metrics every minute, pushing it up to the Avi controller and the Avi controller runs the big data analytics engine. And not only to present a nice dashboard for you to see what's going on from performance and visibility point of view, but the controller uses that information in a closed-loop fashion to make automated policy-based changes on the underlying infrastructure. So for example, if it sees the traffic growing, it can talk to the OpenShift APIs and auto-scale your infrastructure, or auto-scale out or scale in, depending on the policies. So it's a fully elastic, automated services fabric. And it's a full-feature load balancer, a full-feature fabric. So if you want to, if the question that comes to your mind is, well, does it have the features that my F5s or Citrix have? Yes. It's not an open-source HA proxy or a coop proxy-like basic load balancer. It's a full enterprise grade with L4, L7, caching compression, content switching, auto-scaling, global server load balancing, SSL, DDoS, VAS. You name it, it's an enterprise-grade solution. Think of Avi as the best of an enterprise-grade a proxy or load balancing solution married with the elasticity of your, of your coop proxy or HA proxy or NGINX that's built-in with integration into your OpenShift environment. Plus the application visibility, the log analytics, the security analytics, the built-in multi-tenancy, built-in service discovery, centralized management are in addition to the features that we talked about. And we'll talk more about this as we go through the demo. But a common question that gets asked is, well, this is software, what about performance? Surely the hardware is required to get high performance. Well, that's a myth that the hardware vendors have propagated, that's absolutely not the case. Here is two data points. First, you can have one gig of encrypted sustained throughput or five gig of total unencrypted throughput, 2,500 SSL transactions per second with perfect forward secrecy in a single VCPO core as a proxy. And then if this Lego block of your proxy, the service engine can be one core, two core, or a full bare metal server, and that gets you more than what your hardware appliances give you in that form factor. The second part, it's horizontal scale out. With these Lego blocks, you can build lots of service engines in a scale-out fashion, centrally managed by every controller. It can reach millions of SSL TPS, terabits of throughput on commodity X86 hardware. And as a proof point, one of our customers' largest, one of the largest e-commerce and payment companies did an experiment where in a Google public cloud, they scaled out with Abhi a single application, single VIP to one million SSL TPS, zero to one million in eight minutes, running on commodity Google X86 instances. You cannot even do that in hardware appliance. So performance is a non-issue. It actually outperforms any hardware load balancers. All right, last slide on the use cases, and then we switch over to the OpenShift specific integration. So Abhi works across your traditional load balancer refresh use cases, where you work on a bare metal or a VM, taking, replacing hardware load balancers, saving over half the cost to our customers. Our private cloud use cases is OpenStack with Ansible Automation or a public cloud or an SVN environment, or the case in point today is Container and Pass Solutions. Okay, so let's switch gears here. And let's, by the way, we have common Red Hat customers with Red Hat on all of the above use cases here. Let's switch gears into how does it work in OpenShift? How does architecture that we just talked about for the last 10 minutes fit into an OpenShift architecture? So let's start with the problem statement first, right? So as the application architectures have evolved from a monolithic application to distributed microservices-based application, you have a problem. You can use your traditional appliances for North-South or as OpenShift calls it, Routes layer, right? Where you can deploy your SSL offload and load balancing for your North-South application. But what about East-West? Because your microservices-based architecture, there's a lot of East-West traffic that goes on. And your traditional appliances don't work because it's an appliance, even it's a virtual appliance. So the common solution is a coop proxy, for example, in an OpenShift environment. There are challenges with that and we'll talk about that in a minute. What AVI has done is if you look at the infrastructure stack of a container services fabric, you have your server layer, physical or virtual, doesn't matter, you have your network layer with or without SDN, and then you have your L4 to L7 layer. And on top of that, you have the Kubernetes and OpenShift for the scheduling and the resource management. AVI network fills in the gap that's in the middle. When you are running on-prem or in public cloud, doesn't matter. But from service discovery to service proxy to application visibility and performance monitoring to micro-segmentation at the application level and security at L7 to DDoS to SLA-based auto-scaling by integrating with Kubernetes or OpenShift. That's the gap that AVI provides an enterprise-grade L4 to L7 services solution. And we'll go into now or the next 10 minutes before the demo, we'll go into the detailed architecture. So what is the before and after? So if you look at the before, if you look at either a traditional hardware-based solutions or if you look at a combination with an open-source solutions, what are the challenges? First of all, you need multiple tools for your route or for the not-south load balance so typically use an HA proxy front-end by a hardware load balancer for SSL offload. That's a common example. There are challenges with that. HA proxy itself is not HA. You need a separate daemon to monitor its hard beats. It doesn't have enterprise-grade features. It's a very basic load balancer. And then when you combine that with a hardware load balancer, it's too expensive and complex. And then you separate tools for your east-west, typically use coop proxy. Coop proxy itself is a probabilistic load balancer. It doesn't do L7, it doesn't even do east-west SSL. And then if you combine that with a separate service discovery solution for DNS, centralized management, hardware load balance for global server load balancing, these multiple tools add pressure complexity and higher cost. How does AV solve this problem? AV has one solution that does all of the above. It does service discovery, service proxy, so local and global load balancing, application maps, application performance monitoring, security, micro-segmentation, auto-scaling. And so architecture-wise, how does it work? So we saw earlier in the architecture slide we have a centralized controller and a distributed data in the service engine. So the way it looks like is that is the AV controller that is deployed typically as a container, typically on the OpenShift Master, but it can be outside. And AV service engines, one pod per your OpenShift node. So AV service engine runs as a container and a pod on each of your OpenShift nodes, okay? So let's go through the light in the life of. So from day zero to day one to ongoing and then we go into the demo to show you how this works, right? So I'm gonna walk you through the steps of how the integration works and how the flow works and show you that exactly in the live demo. Day zero, the way it works is on step one, on AV controller, you configure the OpenShift credentials. So you have deployed AV controller and you as a container and you give it access to the OpenShift Master. That's it. That's the only config you ever have to do on AV. From then on, AV controller talks to OpenShift Master, figures out how many OpenShift nodes are running in the environment and automatically brings up AV service engines as rocker containers or pods in each of the nodes. Then what we're gonna show you is when you create a deployment, let's app one in OpenShift. AV automatically listens to the notifications and pulls configuration from OpenShift Masters, automatically creates these corresponding services, creates a VIP, creates a DNS record, fully automated. Okay, so on day zero, you configure the controller with the master credentials and from then on, as you do deployments, AV takes care of everything. If you're GSLB, same thing, you just do it, you do it, oops, sorry about that. You do it on each of the data centers and then you do GSLB configuration. So again, you do the deployment app one in data center one and data center two. AV automatically creates the corresponding local VIPs. AV automatically creates the global service for GSLB, covering both the VIPs. AV automatically syncs that to all the followers and starts health monitoring the local VIPs. So even for GSLB, zero touch. So how does the traffic flow work? So now that the deployment has been done and your services have been created, let's look at the traffic flow. How does AV provide the set of discovery, service proxy capabilities? So let's say an external user, let's talk about North South Traffic Management, so this is the route part of it. If a customer comes in from outside, looks for app one dot os dot acme dot com, recursively through the DNS, it comes through the DNS that AV is running and it, first of all, does the GSLB resolution because you're running in two data centers. The GSLB returns a local VIP and then the user performs the get on that local IP address and then AV automatically handles that as a service engine as a proxy, TCP, HTTP, SSL proxy. That figures out where the right part is that should handle this specific request and follows up and basically does a load balancing.  Let's talk about East-West Traffic Management. So what happens when an internal service is trying to reach another service for a through DNS? So similarly, AV is going to perform, not the AV, the service is going to perform a DNS query for the East-West application, recursively hits the GSLB, returns a VIP and East-West traffic also is managed the same way. So basically it's a client-side proxy. So the way it works locally is that the part that is doing the request is the traffic is intercepted by the local proxy that's running on AV service engine and the service engine then decides who the corresponding best part is to respond. So because we have a service engine running on every part, this is the client-side proxy, just like Coupe proxy. AV service engine basically replaces Coupe proxy. So every time the request is sent out from the requesting part that's intercepted by the local service engine proxy and then it's load balanced in a stateful fashion. So unlike Coupe proxy, which is a probabilistic packet distributor, in this case, AV is a fully stateful proxy and it does the health checks and it makes sure that the load is evenly distributed. It can handle SSL offload, it can do all L7 policies and rules like URL rewriting, rate limiting, URL-based redirects, everything that you expect in an enterprise grade load balancer is also available for your East-West traffic.  Let's say you're deploying applications version two and you want to do a policy-based switchover from a blue version to a green version or version one to version two. Well, all you have to do is you create through annotations, you create policies on AV and as what AV can do is you can configure, say, send 10% of the traffic first to the new version and if everything looks good, then automatically start sending everything to the new version. If it doesn't look good, is a fall back to the blue version. So you can do that programmatically based on policies through this solution and non-destructively. So the existing connections allow to drain the new connections switchover to the new version. Another common use case that we have with this is a security capability where let's say you have an external service whether it's a database service or active directory service that your internal services wants to access. However, you want to get that only certain services only certain internal applications can access those external service and you want to force that policy. And let's say you have a firewall in the middle that's also controlling the access to that shared resource. Well, what you can do with AV is that create this external pod called AV pod that what that does is through policies it says only blue service is allowed to access the external resource and we also do a NAP source netting to a single IP address or in the firewall you have to open up a single IP. You don't have to open up multiple IP addresses. And so it does two things. It allows you to simplify a firewall configuration at the same time. It allows you to control with service in this case a blue service can access the external resource and the red service cannot. All right, so with that, let me pause here, hand the control over to Bhushan where we'll do a detailed demo about everything that we just talked about step by step starting from scratch where you have an OpenShift cluster and now we control deployed and we'll spin up AV service engines what creates deployments and we've run some traffic. Bhushan, over to you. Thank you Ashish. Let me start sharing out here.  You see it now? Okay, yes. Give it a second. Still trying to share your screen. I think it's a bit. Let's see what we get. We're sharing the screen. I can see that you're sharing the screen but you need to click into your browser. Networks. Okay, so this is what we aim to do in this demo today. We are going to deploy services on OpenShift and this is what it's going to look like. We have a virtual service which corresponds to a service in OpenShift which has a bit on it which is placed on one of the service engines running on one of the cluster nodes. This service, virtual service can have one or multiple pool members and each pool can have multiple backend pods. So let's start with our clean setup. As you see out here, we have OpenShift system with multiple projects on it and AV controller which is right now we don't have anything on it. It's completely clean setup. No applications, no virtual services, nothing. And I also have a dashboard to run some REST APIs to either the OpenShift master or AV controller to provision some services quickly. So let's start with the first day zero configuration that is setting up the OpenShift cloud and let's go back and check out our infrastructure. So as you see earlier, we didn't have, this was none but now we have configured the OpenShift cloud out here. So behind the scenes what it does is configure the master node IP and authentication token, set the AC deployment policies, telling from where to get the image, how to deploy, where to deploy. And on the application side, it also configures the IPAM and DNS profile. So for this demo purpose we are using our internal IPAM and DNS for both east, west and north, south virtual services. In addition to this, AV also maps all the projects which are there on OpenShift as tenants which you can see out here in the tenant view. So for each project on OpenShift, we map it to a tenant on AV. So let's be on admin tenant for now and the next step is to configure the DNS virtual service which will be responding to all the DNS queries for all the applications running that AV will handle. So let's go back to application and see that the virtual service is already up and running. If you go to infrastructure, we see that the service engines are also up. The OpenShift cluster has four nodes, one master and three, three minutes node. And AV controller goes and deploys a service engine on each node. AV controller also has the intelligence to disable the service engine on nodes which are not scheduled for parts. For example, the master node out here is disabled, has the service engine disabled. And if you go back to the DNS virtual service, you see that we are listening on one of the IPs for the North, in the North South network and it will start serving for all the expedients of different services on the OpenShift, right? So let's go to admin project right now. As you see, there are no applications running. So let's go and start some applications on OpenShift side. So when I clicked that button on the demo controller page, what it did was ran a REST API to OpenShift master on deploying the applications and deployment conflicts which in turn deployed the application parts, as you see. And as these applications come up, we automatically sync those and provisions corresponding virtual services automatically and places them on all of those service engines you saw earlier. So let's wait for the applications to come up. All right, looks like all of them are already up and green. As you see out here, all of the applications are have IP in the 172 submit except for photo, which is our North South facing service. It has an IP in the North South submit. All this happens automatically using our internal IPAM. The way we provision this virtual services is through annotations on the service which you configure on OpenShift. So if you go to one of the services and see the annotations for the service, the only thing that the app developer has to do is provide a label called Aviproxy and in the value of that label specify what type of load balancing it requires. Is it regular HTTP or HTTPS or L4 service? What kind of load balancing algorithm requires is it round robin or least connections or any other option which we have. And Aviproxy automatically reads those labels or annotations and provisions corresponding to it. So it's zero touch on Avipy side. So another view which we have out here is called a map view. So let's go for it and let's focus on the photo.com app which is our North South facing app and start some traffic to it. So since you take in line with the traffic, we know what application is talking to what other application and as you see out here in a while, we monitored the traffic coming into your application, North South application and we also trace it back to all of their applications as it flows through your application stack. For example, the client is accessing photo and photo in turn is accessing other applications or each best web that likes shopping cart inventory and checkout. What this enables us to do is let's dig into one of the applications and see how it can help with the security of the virtual services in this case. We see that this application is getting traffic from three other virtual services and we automatically populate those virtual services into the white list. From this day onwards, if you just save this configuration and secure this VS, no other application will be able to talk to this. So to demonstrate this in real time, let's just move one of the application out of the white list and in some time, we will see that the application which removed the arrow will turn red, which indicating that all the traffic from that application is helping blocked while the, okay, as you see right now. In addition to this, along the edges connecting the different applications, we can also see various metrics like throughput, number of complete requests, total request errors, latency, et cetera. The user can also see the same things in a graphical manner on the analytics page for each and every virtual service. Give it some time to load. So you can see end to end timing, throughput, open connections, number of connections, request and all other metrics out here. Also under logs, you can see each and every request coming into the application. For example, out here, it says what client app is trying to reach this app, what URL it is hitting. And if you want to dig even deeper, you can expand one of those requests and see in detail what pod IP center request, which load balancer handled it and which backend pool member it went to along with hundreds of other metrics out here. If we write on this one is just a simple HTTP VS, but if you move to our photo.com, which is North-South, I have configured it as a secure HTTP or HTTPS VS. So on this, we can also see the security analytics for this virtual service where we can see how much percentage of traffic is RSA or EC. You can also see right now, we don't have any attacks happening on this service, but if any attacks happen, we can also see that it comes out here. Apart from that, we have other graphs showing that a percentage of traffic are using different TLS versions, 1.0, 1.1, 1.2, et cetera. Again, transactions per second, a number of key exchanges happening, different types of certificates the clients are using, et cetera, too. I guess from the demo point of view, that was all I'll hand back to Ashish. Yeah, Bushan, one more thing if we can show that if we can show in a North-South whip on other demo whip that we have, the end-to-end timing, the type of geography, call information, the type of top URIs, 404s, et cetera, so maybe we can spend five minutes or so on some of the other things that we can do. Sure, so moving back to the whip we saw initially at the beginning of the demo, for this demo, we have much richer traffic to show. So as you can see on the end-to-end timing side, we can show you a summary of what the client response time looked like, how the server response time looks like, what the data transfer time was, how fast is the application responding. And it's not just for a given current time, but you can view it across multiple time windows. Like you see out here, we are viewing for the past six hours and you see between 8 a.m. to 9 a.m. we saw a certain search in the end-to-end timing. Some of our clients must have seen the application slow down. In this case, usually the blame goes to the networking team, but using OVC UI, you can clearly see that there is no change, relatively no change on the client RTT side, but the app response changed drastically. So maybe it's surely the application which is behaving a bit weirdly, causing the application to slow, client experience to slow down. In addition to this, we can, like I said, we can see each and every request coming in and out of the application. We classify those requests into two types, significant and non-significant logs. The significant ones are the one which either end up in 400 or in some kind of error and or take a really long time. Basically something which needs, which can help debug problems. Non-significant ones are the one which breeze through really easy with no errors at all, like regular 200 requests. So if we open one of these requests, we can see a lot more data about where the actual client is located, what kind of OS is it using, what kind of device is using, what another L7 security metrics like what TLS version, what certificate type is using. And again, we can see from what client IP to which service engine handled the traffic on what port and also on the backend side, which web server handled the request. We can also, if you want, you can also dig down even deeper and see each and every header on the request. All these logs are captured real time but can be either enabled or disabled to whenever required. How this thing helps is, for example, out here, you see that if you go to end to end timing, you see that most of the clients got service between with a relatively low amount of latency, but some of them are seeing about a second of delay for the response. If we just click on this, we automatically filter out all the requests which ended up getting more than 800 milliseconds of delay. If we see that most of this have relatively high client RTT times. So if we start this out based on location, we see that 99% of this traffic is coming from India. So maybe the van link is the culprit out here. So within seconds we are able to quickly pinpoint where the problem is. This was from the networking point of view, but even for the application administrator, you can just, if you see a lot of 404 errors on this application, you can just click on 404 and see all the logs that ended up in 404 and then he can fit this information based on server IP addresses. And he knows which server is the application erroring out on. So even debugging the application problems is really easy when you use RV. On the security side, like I was talking earlier, we can show different percentages of client using different types of TLS versions, different types of certificates. We also give you a real time score of how your application is performing based on your security. For example, out here we are using self-signed certificates. So it's changing the health of the virtual service. I guess that was all back to you Ashish. Thank you very much for a detailed demo. Hope that gave you a glimpse of what RV can do today. And it has more things that we can show you if you are interested in a one-on-one environment, including doing things like auto-scaling capabilities, et cetera. So let us wrap up and then we can take a Q&A. But let me summarize first what we saw earlier today in the presentation. So what you saw was a fully distributed enterprise-grade load balancer that takes care of both your routes for north-south traffic and services for the east-west traffic as well as GSLB. It's a single solution that takes care of all of the above capabilities, including integrated IPAM and DNS for service discovery, fully automated service proxy with service creation, application map and analytics and troubleshooting capabilities that you saw. And what this allows you to do is to have a much simpler experience with very few moving parts of the same RV solution that does all of the above. You don't need a separate hardware load balancer for GSLB or for SSL and a software HAProxy or GroupProxy for East-West, which again, doesn't give you anywhere close to what RV offers. And that results in significant CAPEX and AFEX reduction given the simplification as well as elimination of hardware load balancers. And lower resource consumption. There was a question on the chat that what's the footprint? The footprint is as small as one BCP and one to two gig of memory. And the beauty is that it works in any environment. It works on-prem, it works in public cloud, it works with OpenShift, it works with Kubernetes, it works in AWS, Azure, GCP, et cetera. And finally, the thing that we didn't have time to cover today, but I'll just plug in here is our Ansible automation. So if you were to go to our Ansible repo in GitHub, what you'll notice is that that is, we have a full Ansible modules as you can see here for day zero deployment, so for RV controller deployment, for RV service engine deployment, for web provisioning in a non-OpenShift environment as well. Because OpenShift, as you saw, everything was automated, but in any other environment, we have a full Ansible playbook. And in fact, just today, Red Hat and Avi did an Ansible Fest in London, where one of our Ansible architects along with the Red Hat engineer presented the automation that Avi provides. So check out our GitHub repo as well. And if you're interested in beyond OpenShift, even in our OpenStack solution, reach out to us because we have joint customers using Avi in OpenStack with Red Hat. So with that, I'll leave you here with these resources and we'll take questions. So if you have any questions, reach out to me at AshishatAviNetworks.com or Edward Sharp, who's in our business development department. And then let's take some questions. So the question is, is it possible to test Avi on OpenShift Origin? The answer is yes. And Bushan, do you want to have any color on that? Yeah, it works exactly the same both on the Enterprise version as well as on the OpenShift origin, open source version. Thank you. There's very little difference between the OCP and Origin in terms of the playing and pretty much anything here. So this has been pretty, I think my mind has been blown because I've been looking at Kube Proxy and this just really blows Kube Proxy out of the water. And I'm very appreciative of you guys spending the time giving us this demo and the insights into what we can do with Avi Networks' offering. And I really, I can see now why you have so many joint customers with us because this is something that I'm actually curious to know if the OpenShift Online or the dedicated folks at OpenShift have been using this in the background unbeknownst to me. So I'm gonna have to reach out to them and that team because this just seems like something that's very, not, I wouldn't say completely lightweight, but the benefits you get from running it are just mind-blowing as opposed to just simple load balancing that I've played with a little bit. Yeah, this has been a very great presentation. So folks, if you have questions, please do reach out to the team here and we will looking to see if there's any other questions online in the chat. I think everything got answered in the chat. From a performance standpoint, it looks like this is really not anything with any serious overhead. So I definitely think I'd suggest you give it a try and give them a call. And I would love to talk further, maybe with some of your customers to get them to tell their stories as well. Because I think it's great with the demo sets, but also to get something at scale would be really wonderful. We can get to- Absolutely, Diane. Yeah, absolutely, yes. Thank you. Thank you for the opportunity. This was great. So everybody will be back on next week. Again, we have a couple of events coming up soon. The first big event will be that 2.1.7 is coming out. So we'll have a number of talks coming up on that. We're gonna continue our theme this month with monitoring of all ilks. So there'll be more monitoring talks. And they'll post this and the slide deck. So if you guys can send me a PDF version and the slide deck, we'll post this video and the slides up on the OpenShift blog within two days. It should be up there and running. So if you're listening to this virtually later on, please do reach out and join the mailing list or join our Slack channel. Find us on commons.openshift.org. And really a great presentation from Avi today. My mind is a little blown. There's a lot of details there. And I think this is something that we look to for partners to provide. And this is just really outstanding work. So thanks guys and we'll talk to you all soon. Thank you, Diane. Thank you everyone for joining. Appreciate it.


Hello everyone and welcome to the Sig Auto Scaling update for KubeCon, CloudNativeCon Europe 2023. So what does Sig Auto Scaling own? We now own a number of different hub projects. So the Balancer, which is new for the sort of past year, Piotr will be giving you an update on that and the implementation of that. We also own the cluster autoscaler for scaling your cluster in terms of nodes horizontally. Giant will be giving you an update on one of the newer bits of functionality for that to aid cluster operators in debugging what the cluster autoscaler is doing and why. We also own horizontal and vertical pod autoscaling, horizontal pod autoscaling, which is implemented as part of the Kubernetes controller manager, but also vertical pod autoscaling, a newer standalone component. And Chen and McKellie will be talking you through a demo of a proposal for a new feature called the multidimensional pod autoscaler and how that will allow users to potentially overcome some of the current challenges with using horizontal and vertical pod autoscaling together. And finally, we also own the Adornary Sizer, however, we won't really be covering that this time round. So I've mentioned some of the in-depth demos and sort of walkthroughs you're about to get. However, we also have a number of other updates that might be of interest to you as a cluster operator or a user of a Kubernetes offering. So for the horizontal pod autoscaler, container resource scaling has been available in alpha since 120. A member of the community has finally picked up the work to get that promoted to beta in time for Kubernetes 127, hopefully, and that will allow users potentially have managed services provided by cloud providers who might block off alpha features potentially to make use of it. And this has the potential to improve the durability as a service owner to improve the granularity of your scaling behaviors. And we also asked owners cluster autoscaler have committed to an improved release process going forward. And so current users may be aware that the patch release to this point has been a bit ad hoc. We are looking to provide a far more clarity for users of cluster autoscaler when patch releases might come out and therefore when cherry picks of bug fixes, etc, might make it back to supported releases. And finally, on the vertical pod autoscaling side, we're beginning to gather the work to enable taking advantage of dynamic pod resizing. This has always been intended as a feature for the vertical pod autoscaler. However, we've been dependent on work from other SIGs and that's now progressing significantly and looks like it's going to make into Kubernetes 127 release. And so we want to be ready to do all of the work that's required for the VPA to take advantage of that as soon as possible. And if any of this interests you, we as a SIG definitely need help. As mentioned, we have ownership of a number of different sub projects and we need your help as the community in all of these areas. We'd love to get your feedback where issues where some of our projects aren't working the way you think they should or progressing up that contributor ladder via issue triage, reviewing PRs or even contributing features or bug fixes that you think are important or excited about. If any of that interests you, please get in touch with us as a SIG. We hang out on the GitHub repo that hosts most of our projects. We also have weekly SIG meetings at 1600 CET. If this time zone doesn't work for you, please reach out and we'll try to arrange something. However, we're also reachable on Slack at SIG autoscaling and enjoy the demos. Hi, I'm going to cover Balancer, a new resource in the autoscaling space. It allows to control how pods are distributed across similar deployments and autoscaling together. In a second, I will explain why it may be useful. We've introduced the Balancer resource and a controller for it. To use it, you need to install an optional component from Kubernetes slash autoscaler. It's just been published, so it's an early alpha stage. So why would you want to distribute pods in a workload across multiple similar deployments? There are a couple of use cases. Perhaps you have a regional cluster and want to ensure that the workload is spread evenly across zones for availability reasons. You have multiple deployments, each responsible for a different zone. When pods are added or removed to the workload, you expect this even distribution to be maintained. Or perhaps you want to run your workload on both standard VMs and spot or preemptible VMs, which are cheaper but less reliable. You would like to always have 25% of pods running on these less reliable VMs. Again, you expect new pods to be distributed according to the defined ratio. Or perhaps you run your workload on different machine families, or there's any other slight difference in configuration. Now that we know when the Balancer feature may be useful, let's explore how to configure it. Let's say I want to balance my app between two deployments in a 3 to 1 ratio. The use case we described earlier for the 25% of pods on spot VMs. You can see the Balancer YAML to the right. API version balancer.xkate.io slash v1 alpha 1, kind balancer. This balancer defines two targets, which are deployments called my app A and my app B. We can define min and max replicas for each target. And lastly, we define the policy. In this case, this will be a proportional policy with ratios 3 and 1. With this config, if the Balancer controller detects that the number of replicas in deployments is no longer in proportions of 3 to 1, it will redistribute pods to bring back the desired ratio. Also, when scale on the Balancer object changes, so pods are to be added or removed, Balancer controller will add or remove them from the right deployments in order to maintain the 3 to 1 ratio. This combines well with horizontal pod autoscaling, so we can define an HPA that has the Balancer object as its scale target. This HPA covers the whole workload, both deployments combined, and recommends by updating the scale on the Balancer object. The Balancer controller then distributes this new scale across its targets, deployment A and B. Let's have a look at a different use case. I want to run my app generally in deployment A, maybe because it's a particular zone I prefer. However, if there's no space there, max replicas are reached, then overflow to deployment B, a different zone perhaps. As soon as there's space in deployment A, rebalance pods back there. You can see that now the policy part of the Balancer YAML at the bottom is set to type priority, and priority order is defined as my app A first and my app B second. We can add a new requirement where if pods cannot start in deployment A, even if max replicas is not reached there, we will still start pods in deployment B. Perhaps there are pending pods in deployment A because there's no space in the Notebook. We can configure this fallback behavior by adding a fallback section to the policy spec. We enable fallback and define the startup timeout, the time after which a pod is considered blocked. It will trigger starting another pod in deployment B. Once the original pods finally start, the excessive pods in deployment B are removed. In summary, the new Balancer resource allows to define how pods should be distributed across similar deployments. You can additionally define horizontal pod autoscaling on the Balancer object to autoscale the pods together. You can learn more at the GitHub repository page, Kubernetes slash autoscaler, and we're waiting for your feedback. Thanks. Hi, everyone. How are you doing? Today, I'm going to be presenting Debugging Snapshotter, which is a tool that we have in Cluster and Scalar. Before we get into what the tool is, let me try to explain what are we trying to solve here. CA logs a lot of information about the decisions taken. For example, if you scale up, you scale down, you don't scale up, but we don't log what data these decisions are based on. CA internally simulates the behavior of the impact cluster, and that's just too much to log. When something goes wrong, we usually need to understand how these decisions were taken, and not just what these decisions were. RCA of an issue usually takes time, and the internal state of the cluster changes when we mitigate the issue, thereby making it more difficult to debug the issue. Debugging Snapshotter is a tool to visualize the internal state of Cluster and Scalar at a point in time to help debug autoscaling issues. So before you go about mitigating the cluster, maybe it would be easier if you had a snapshot of what was the state of Cluster and Scalar at the time when the issue was happening. So let me quickly take you through some common use cases that are available. Scale up and scale down not working with special focus on scale from zero nodes. Maybe if you have an attached instance, for example, there's a mismatch between scheduler and cluster autoscalar decisions where cluster autoscalar decides to not scale up, but scheduler says it cannot schedule anything. There's a mismatch in resource availability on a node where there might be extra resource on the node, but cluster autoscalar assumes there's a different resource available. So what data is being captured by the Snapshotter that's available for you? First and foremost, the node list that essentially all of the nodes in the cluster. Secondly, any unschedulable pods that cluster autoscalar things are schedulable. There's also template nodes, which are simulated nodes or any attached instance group which with no nodes on the cluster. There is an error field which which is filled in case the Snapshotter fails and there's an error generating the snapshot itself. And there's a start and end timestamp. This could help you with if you decide to take multiple Snapshots and also how also because Snapshotter might sometimes take a non-trivial amount of time. So this would encapsulate the complete timeframe when this particular Snapshot was generated. How does it work? It basically captures the internal fields that we talked about when we receive a request for a Snapshot and returns a well-formed JSON. After receiving the request in the following cluster autoscalar processing loop, Snapshotter will snapshot the cluster state and limit it to a single loop, no crossing data. So it will start to capture in the following loop at the start of the loop and it will go through all of the steps that cluster autoscalar goes through and collect all of the data that is part of the Snapshot and end when that particular process loop closes. The HTTP request blocks for the duration of the Snapshot generation and returns a JSON as the HTTP response to the request. There's no files. You make a request, you wait for some time and you get the response as HTTP response. How do you make this request? You SSH onto the server running the leader cluster autoscalar in case you're running multiple cluster autoscalers and you curl to this following link. It's a local port that cluster autoscalar has a path available on. Let's quickly go through the demo. It's super simple. We have a cluster autoscalar running and we have a cluster running where we have three nodes in the cluster. We have cluster autoscalar logs here, which has an attached instance group, which is empty for now. Let's apply a pod or deployment with some resource request which cannot be accommodated on any of the existing nodes and we should see a scale up at that point. Let me go ahead and apply the deployment. We have done that. If you see here in some space, we have a pod that is unschedulable and we see a scale up for an instance group. This is GC specific, but this also simulates two different cloud providers. As soon as we see the whole operation being done, let's go ahead and make a snapshot request as we see here. If you see in the logs, we also have logs around the snapshot itself. It marks when it has received the request. Let me just take you through it really quickly. There's a bunch of other things on when the data collection has started, what data is already being connected, and snapshot being flushed back as a response. We just made a call request as soon as that happened. We see now there are four nodes, but we made this before the new node is registered. We should see the new pod as something that can be scheduled, but there's no other. Let's go through all of the different items. We're going to use a common JQ tool that's used to parse JSONs. If you see here, we have a bunch of items that are available as keys. Let's go ahead and see how many nodes there are. There are four nodes. We can also see template nodes here. Let's go through that. There will be one template because we have one attached instance group which has a simulated node available. We will also have unscheduled pods that can be scheduled. This would be the node that is available. This is a pod that could be scheduled on the upcoming node, but isn't. It's in that middle state. This could also represent a bad pod or unscheduled pod, which you might want to debug. If you see here, NGINX deployment 1, there is a bunch of information around it. This is how you can easily navigate your JSON. That's it. Let me give you a quick rundown, a quick start on how you can get yourself up and running with debugging Snapshotter. You just need to enable the Snapshotter on your ClusterArtsScaler by adding the following flag to the manifest. This is already available in production on all ClusterArtsScaler versions 124 and above. I'm hoping, even for older ClusterArtsScaler, people want more stability. This should also be available for you. Also, there are going to be detailed instructions in ClusterArtsScaler FAQ, similar to what I have gone through that you can refer to and how to operate the Snapshotter. We're looking forward to hearing back on any feedback that you have and any possible extensions to the Snapshotter that you think will make everybody's life easier. Yeah, that's all from me. Thanks, everyone, and passing it on. Hello, everyone. I'm Chen Wang from IBM Research. Today, my colleague, Mikali Orlandy, and I will introduce our new enhancement proposal for Kubernetes Autoscaler, which is called Multi-Dimensional Pod Autoscaler. I'm a research staff member from IBM Research, and my daily work involves enhancement in Kubernetes, including Autoscalers, Schedulers, and Node Resource Plugins. I'm also actively working on cloud-native AI system platform and applying AI techniques in cloud platform management. I'm an open-source advocate, Kubernetes contributor, and a regular KubeCon speaker. So today, I will briefly introduce our motivation, why we propose Multi-Dimensional Pod Autoscaler and the design of the MPA, and later, Mikali will show us a demo on how to use the MPA. Currently, there are two Autoscaling controllers available in the community, which is the Horizontal Pod Autoscaler and Vertical Pod Autoscaler. So Kubernetes Horizontal Pod Autoscaler, later we refer to HPA, is an Autoscaler that allows you to automatically scale the number of paths in a deployment or replica site based on either the CPU utilization metric or other custom performance metrics. So with HPA, you can ensure that your application is running at an optimal performance, at optimal CPU utilization rate, and by automatically scaling out and in the number paths based on the metric. And another controller is called Vertical Pod Autoscaler, we later refer to VPA. So VPA automatically adjusts the resource request and limits of a container based on its actual usage. Rather than the initial value set by the developer, it analyzes the historical resource usage patterns of the pod periodically, and then recommends an appropriate resource request or limit to be set for the pod. This ensures that the pod always has the amount of resources allocated to it based on its usage, and can help prevent over provisioning of resources. So the design of VPA actually consists of three main controllers. The recommender analyzes the historical usage patterns of the pod and recommends the resource usage and limit to be set based on the histogram of the resource usage observed in a previous time window, usually 15 minutes. And the updater is responsible for observing the difference between the recommended request and limits and the current set request and limit. And we'll evict the paths if the gap is too big. And the admission controller is responsible for updating the paths request and limit when the past evicted paths are restarting, according to the recommended values provided by the recommender. So those three controllers work together to provide automatic vertical scaling for Kubernetes paths, allowing them to dynamically adjust their resource request and limits based on their actual usage. So because right now, HPA and VPA control the scaling actions separately as independent controllers. So when they are configured to optimize certain targets, for example CPU usage, they can lead to an awkward situation where HPA tries to spin more paths based on the higher threshold CPU usage. Well, VPA tries to squeeze the size of each pod based on the lower CPU usage after scaling out by the HPA. So the final outcome could be conflicting, meaning it will lead to a large number small paths created for the workload. And none of the objectives can be guaranteed. So there's some motivations for the combined control of both horizontal and vertical scaling. Because first, we sometimes want to fine tuning the timing to do the vertical and horizontal scaling and prioritize a certain dimension auto-scaling than the other. And there might be some, there might need some synchronizations between both actions. And sometimes controlling the vertical scaling based on the usage and controlling the horizontal scaling based on the performance just doesn't guarantee either performance or resource efficiency. Because the usage observed is statistics from a time window. And sometimes a certain margin of resource over provisioning is needed to handle the fluctuation of the workload to guarantee certain performance objective. Therefore, there needs an advanced combined algorithm to find an optimal combined control of vertical and horizontal scaling actions for a certain application under a certain load. So moreover, in some cases, one objective is prioritized than the other. For example, when there is only one replica for deployment, you may only want to scale down the vertical scaling down the resources if the resource utilization is low. So therefore, here we propose a multi-dimensional part auto-scaling framework that combines the control of vertical and horizontal scaling in a single action but separates the actuation of actions completely from the controlling algorithm. Just similarly, as a design of VPA, NPA consists of three controllers, a recommender, an updater, and then a mission controller. And of course we have a new API defined in the customer resources as NPA. And that connects all the auto-scaling recommendations to its actuation. So the multi-dimensional scaling algorithm is implemented in the recommender. The scaling decisions derived from the recommender are stored in the NPA object, and the updater and a mission controller retrieves those decisions from the NPA object and actuates those vertical horizontal actions. So our proposed NPA can also support developers to replace the default recommender with their alternative customized recommender algorithm. So developers can provide their own recommendations, their own algorithms implementing some advanced control of both actions. So in details, in NPA API, developers can specify the auto-scaling configurations include whether they only want to know the recommendations from NPA or whether they want NPA to directly actuate the auto-scaling decision. And it can specify certain application performance targets such as latency output, and it allows any custom metric to be used. So other auto-scaling configurations such as what has been available in HPA and VPA are also available in NPA. So NPA API is also responsible for connecting the auto-scaling actions generated from the recommender to the mission controller and updater. And it's created based on the NPA, actually the NPA custom resources provided by the upstream community. And basically it is a CR and it keeps tracks of what's recommended size of pod and the number of replicas. So the recommender just retrieves the time indexed measurement data from the metrics API about the usage and it generates the vertebrate horizontal scaling actions and the actions from the recommender are then updated in the object and the auto-scaling behavior is based on the user defined configuration as well. And the updater will update the number of replicas of the deployment and evict eligible pods for vertical scaling and the admission controller just similar as the VPA admission controller will just update the pod request limit for those evicted pod when they are starting. So here is an example YAML defined an NPA object so it includes configurations such as update policy namely whether you want to do recommendation only or you want to actually actuate those auto-scaling actions. And then the recommender you can define the default recommender or any customized alternative recommender with the name you provided. And there are also vertical scaling related configurations such as the minimum maximum allowed resources the type of resources you want to control and what containers to resize. The horizontal scaling related configurations include the minimum maximum number of replicas and the type of magic horizontal auto-scaling is operating on. So that's all of my introduction of NPA framework and next I will hand over it to Michele to show a simple demo on how to use it. Hello everybody my name is Michele and I'm from Italy. Today I'm going to show you the multi-dimensional pod auto-scaler. We can see this as a mixture between vertical and horizontal auto-scaler. I'm running on IBM cloud IKS, Kubernetes cluster. So let me deploy the objects the YAMLs that are going to make up my auto-scaler. So as you can see it's creating some customer resource definitions, cluster role bindings, services. I'm going to mainly look at this particular deployment which is the actual recommender. So let me just get the actual name of the pod for this one. So that's the one okay sorry okay. So it has no actual NPA object to observe. So I'm going to create one right now. This object is just a deployment is the actual auto-scaler and as a target it has this deployment here which I'm going to actually apply here. So as you can see I've created my apache application text. We're going to wait until the recommender actually detects the metrics from the application. At the moment no metrics are available yet. Still not getting the metrics here. As you can see the recommended CPU is 100 millicores. Oh there there we go. Metrics have arrived so we should get the recommender to detect them. Over here we have a limit and requested CPU for the only pod running. Okay so here we have the vertical scaling. It has updated the CPU recommendation but it has decided not to scale the number of horizontally scale the number of pods because as you can see the pod is basically just sitting there. So I'm just going to add a load to this pod. I'm going to load it with the request and let's see what the recommender says. When does the recommender detect the load? As you can see the metrics are still not being updated even though the pod is fulfilling all these requests and the replicas have not been updated either.  We have 157 millicores so the recommender should realize this on the next run and should scale up both horizontally and vertically. Let's see what happens. Okay there you go. So this is the new horizontal scaling, sorry vertical and this is the actual horizontal one because now we have four replicas, four desired replicas and in fact they have already been updated here. You can see also the replica set and then deployment has been modified and you can see also the limits and the requests have been modified. This is the recommended millicore value for the CPU. We have 182 as the recommended value. Now let's see what happens on the next run. Okay 163 has changed the value slightly but the replicas have now gone up to six which is the actual maximum value that we can have as you can see. So now let's see if it scales down. We're gonna stop the loading, we're gonna interrupt it. Okay control C, there you go. So now the load pod has disappeared. We're going to wait until the the MPA realizes the text a lower load here coming from the metrics and we expect to have another date on the vertical and on the horizontal side. So let's wait for the recommender to detect this change. Okay what do we have here? Vertical 126 yeah it's definitely going down before I think it was 182 but the actual the horizontal scaling is it has still not gone down. As you can see average utilization is basically zero still not scaling. Vertical recommendation is not changing either even though metrics are all below target. So it's actually watching metrics from all pods. A vertical recommendation has gone down but horizontal is still at six. Okay I think we have something there. Okay this is the new desired replicas. Okay here's already scaled down and we're basically back to the initial situation. Yes it took a couple of minutes to actually scale down but in the end we did it. Okay I think that's it for now. Thank you for watching this video and hope to see you soon. Bye.


In this video I'm going to talk about the paradox of documentation, which may be a term that exists out there, or I may have just made it up, but either way I think it's something really important if you're someone who is writing software, or does anything software related, or honestly even if you're a teacher or something else, it's something you need to keep in mind. Now, what do I mean by the paradox of documentation? Well, I'll put it this way. Let me give you an example, in fact. Around a year ago, I put out what I call a mutt wizard. So I used to get all of these questions about the email client mutt. Now, mutt is a very nice program, very customizable, but it's somewhat difficult to set up, but a lot of people would like to have an email client in the terminal. So I made a wizard, after I figured out how to write, how to, you know, configure mutt. I wrote a wizard for auto-configuring mutt, not just for mutt, but for automatically setting up offline IMAP so you can have offline email on your own computer, don't need any kind of internet connection, and all of that is configured automatically. Otherwise it takes a lot of effort. So I wrote this wizard for people with the idea that I would get less comments and questions on my channel or by email about mutt if I put this out. Now, one thing that I realized very quickly is that this is not the case. Or I lose, there are some types of questions I get less of, but in general I get more and more questions, not mind you about specific things in mutt. But I started getting questions from more and more basic users who lacked more and more general knowledge or things that I took for as general knowledge. Meaning once I put this thing up on GitHub, I had all these questions about how do I actually download it from GitHub? How do I clone a repository? The script doesn't work, ends up they put it in the wrong directory, or they download one file instead of the entire repository, or they make other basic mistakes. So what ended up happening is that I put this thing out that actually made getting things done a whole lot easier for many, many people. But I ended up getting more questions because while I had put good documentation, in fact more than that I put out a wizard to make things easier for people. I got more questions because it was now that much more accessible to an even wider swath of people. So this is something I have to look out for and it's happened to me many times, it happens to other people all the time. That is if you are writing documentation or if you're trying to make accessing your software easy for people and you have a large audience, a lot of consumers. If your concern is making things easy for yourself, documentation is not a good investment. Now I'm not saying you shouldn't document your software, obviously you should, but if you are just swamped with emails and swamped with different concerns, it is a mistake to invest a lot of time in documentation unless you've just documented poorly and you want to fix some mistakes. But in general, if you make things more accessible to people, you will get more questions. Now you're still making the world a better place or whatever, but if you're thinking in terms of you getting stuff done, this will probably get in your way. And it's happened to me in multiple domains.  And of course I get many, many questions about that, but in fact I even, I mean as most of you I assume know, I have a script that automatically installs not just my .files, but configures a blank Arch Linux install, installing all the needed programs for them, setting up system settings, all that kind of stuff. You can get it at larbs.xyz, but I have this script that basically automatically installs all of my .files. So you would think that would make things even easier, especially considering once you've installed this, you have, you can press, you know, Super F1 to get a readme file. You can stream videos directly from my YouTube channel with another keyboard setup. I've documented everything. Does that mean that I get less questions about it? No, because now I'm at a point where, you know, someone who barely knows how to install Arch Linux, which isn't really that big of a hurdle to go over, but even someone who barely knows how to install Arch Linux doesn't even know how to install a program on Arch. They are now using my system and the more the number of basic questions I get has exploded since I've done this. So what I'm trying to say, you know, well, I will say there are some people who take the opposite approach. And honestly, I understand this now, but every once in a while I'll get some butthurt comments from people who go to Succlis's website. Now, Succlis has the opposite philosophy of documenting everything. For them, I mean, it's somewhat of an exaggeration, but the source code is the documentation. If you want to know what it is, know what this program does, read the source code. Maybe there'll be comments in there. You don't need comments, though.   And I've had many people come to my channel said, that's so mean, Succlis, it's so mean to me. But until you've been on that other side, until you have been on that side where you are actually writing documentation and getting questions, you do not understand how tempting that is. It's not, it would be nice to just be able to write something, it'd be nice to be able to write and document something that the better you write and better you document it, the worse your life doesn't get. That was a very complex sentence. But what I mean is, if you're in my position, the easier I make things, the better, more efficient I make things, the more difficulty that is going to cause for my life. The more, the better documentation I give, the more confused emails I will get. And that's on, it's honestly like a law in my life. Now it's different for, I mean, if you have a project that has a bunch of people on it, and you don't have to deal with every question, I can understand, you know, you might avoid that kind of stuff. And again, the whole point is to make things easy and accessible. But if you, one thing that I've found is that making things easy and accessible only makes things more difficult for the person who's making them easy and accessible. And that's just, you either have to deal with it or you have to get to a point where, honestly, where I've gotten, where I have to ignore most of the emails that I get, some of them are perfectly good questions, perfectly legitimate intelligent questions, but yeah, there's such a quantity to them, it's hard to even manage. So anyway, I just wanted to throw this out at you guys.  It's there for a reason. And, you know, a lot of people have put work into it. And most of the times, if you figure something out on yourself, it's much better. But I will say, if you're someone who has swamped with all the projects you're working on, and you're trying to minimize your workload, I'm not saying you shouldn't document your software, you should. But that, that is not going to make your life easier. If anything, if you are successful at documenting your software, it's probably going to make your life harder because your software is just going to be so great. It's going to be so easy to use that it's going to cause you a bunch of confused questions. So anyway, that's all I wanted to do. This would have been a Boomerance and Woods video, but as you can see, the wind is too intense and the audio wasn't recording right. So anyway, that's about it. And I will see you guys next time.