a three run lead until that seventh inning. Just how badly does this one sting? Yeah, definitely staying. It's got a chance to really salvage the trip and you know, they put some good, good at bats together to get back in the game, obviously. But we, you know, we obviously take away a lot of good from the way Pax threw the ball. And so kind of regroup in the off day and get ready for a home stand. You mentioned James Pax in what stood out to you the most prior to that seventh inning. Just, I thought just great conviction and confidence that he threw the ball with. And it just goes to show you, he can go through lineups even, you know, with where he doesn't have all his velocity back. We saw a little tick up in that velocity, but his execution was excellent. You know, he mixed in the change up as well, but his fastball played really well. You know, the curve and the cutter were there for him. I just thought he was really a sharp, really on the tack, just had that aggressive mindset we saw from Pax most of last season. And this is a huge stepping stone for him, I think. I know he had been rolling, but he threw just 62 pitches last time out. He was well into the 80s.  Do you second guess, sticking with Paxton in that situation? Well, I mean, I think you always do that a little bit. Felt like 85 or 90 with Pax today was where we're comfortable going. And then after he got the pop-up of Renfro, for the first out after the double, you know, he, I have to look it back, but it looked like he made a decent pitch that Brasso got the head to for the two-run homer. You know, and then I wanted him to face Lau then at that point, the lefty on lefty, I felt like for his last hitter and Lau got him. So, yeah, I mean, you're always going to do that when you got somebody going there, but also felt like he was sound to go get it. Thank you. Take the next one from Ken Davidoff. Aaron, your offense was up and down this past week. How much do you attribute that to going to get some pretty good pitching and how much to anything in liability on yours?  You know, they're, they do a really good job of that. They have a lot of quality arms. They, you know, they match up really well. So they make it difficult. So I think it's a little bit of that, you know, we still came in here and scored enough to be able to win some games. Obviously the first game, you know, where we lose one to nothing. So I attributed a little bit to that. I feel like we're, we should be in a good spot here moving forward with where our guys are. And just to reiterate, you'll take tomorrow to figure out whether you keep the stride or do something else with that. Yeah, well, yeah, we'll talk about that tonight. Thank you. Ramon, do you get your hands up if you want to get into the queue? Next question goes to Pete Caldera. Aaron, when you came to get packs it looked like you left them with some encouraging words, kind of pump them up there. Is that, describe what you said there? Yeah, I mean, yeah. I mean, just that this was a huge outing, you know, he's, you know, obviously been grinding hard and a couple of tough starts to start the season and that I know how that weighs on you, but as we've seen him here in his Yankee career anytime he's faced adversity, he's stared it down and answered the call. And I thought today, you know, against a really good team to go out there and pitch as well as he did. Just proud of the effort he put forward.


Well, you're not sure exactly who you can have available that maybe a receiver and even on the old line, you just kind of prepare and try to get everybody involved and whoever's available plays. We've had some good practice at that this year. It's just gotta be that prepare with the guys that are here at practice and the guys that are available now and then whoever they tell us we have on Sunday, we'll try them out there. How do you find a balance between, you know, in any given situation here's the best play for this second and three or whatever versus trying to have some kind of through line and seeing to the end of the game and I guess doing things to set up things later on. Yeah, I think there is definitely a balance of that mixture, you know, and I think that you want to set some things up. Sometimes you're excited to get something called that you've, you know, kind of set up with a couple calls earlier in the game and so you're looking for an opportunity to do that and then there are times, a lot of times, particularly where we've been personnel wise some this year where you have to consider game management and, you know, I think back to even the Jacksonville game and not doing anything to put the ball in harm's way because our defense is playing lights out. So I think that all, you know, kind of plays into it and certainly it's an ebb and flow throughout the game. There's communication with Coach Fribill, communication with the rest of the offensive staff and that's kind of how you build it. You had a, I think it was like a third and 16 on that first series and you called it a draw. Was that what you would consider game management there? Yeah, I don't, you know, with where we were at with some personnel things, didn't think it was a great idea to say, hey, let's drop back, hang on to the ball as long as we can and see how fast Joey Bosa can get in the backfield, you know. When a guy gets a hot hand, they make a concerted effort to get him the ball so he can keep shooting. Do you do that in your play calling when a guy gets in a groove like A.J. was in the second half against the 49ers?  He's a guy that, you know, you certainly want to give as many opportunities as you can. My son loves Steph Curry, so I'm all about the basketball analogies, if you can't tell. But, you know, I think with a guy like him, you know, as he gets going and kind of feels that momentum, that confidence kind of builds. And I would say that, you know, when you feel a player of A.J.'s caliber getting into a groove, it certainly gives you confidence as a play caller to find, you know, find ways to highlight some place for him for sure.  Yeah, I think, you know, anytime you have that kind of vertical threat on the outside, it's going to help, you know, some of the conversions on the inside throws to Nick and things of that nature, you know, when you're getting the defense to expand and making that umbrella open, you know, it's going to help things. And we expect that, whether it's A.J. or anybody else, you know, we've got to be a vertical threat offense and make them defend the whole field.  What do you see from them? Obviously, they like the blitz a lot and they cause a lot of turnovers as well. Yeah, they're very multiple group, you know, they have a lot of different looks. They're extremely well coached and you can tell that they're great situationally and that they're prepared for the different things that offense may go to as a counter punch, you know, and so we need to be very detailed.  And then we just got to go win some one-on-one matchups, you know, and that's, I think the name of the game is can we get open faster and their pass rush gets there? And we push piles more than they can push piles backwards. So it's going to be a fun challenge for us. We're looking forward to it. You had to really do it on the third and long last week. It would allow that to happen and how do you kind of sustain that? Yeah, it all starts with protection. You look at those third and long conversions and they all started with a great pocket, a clean pocket where Ryan could step up and deliver a ball, you know, with as much velocity as he needed to on it, you know, and so it all starts there, a big credit to those guys up front. And the tight ends who stepped in and did a nice job in the protection adjustments we asked them to do. Then, of course, it goes back to earlier, saying the vertical stretch and, you know, opening up those safeties a little bit, getting people to play with some depth in their vision drops and Ryan making good decisions with the football. So is it safe to say that Ryan has like a different level of confidence when AJ Brown is on the field with him? It seemed like, you know, some of the throws he was making, were the ones that were reminiscent of 2020 and 2019. You know, I've always operated under the assumption that confidence is built off repeated action, right? And so when you get opportunities to work with guys, as many reps as he has thrown those routes full speed to AJ, you know, although we haven't had as many lately in practice, you know, he knows where AJ is gonna be coming out of a strike route. He knows where AJ is gonna be if he, you know, gives him a chance at a jump ball or a back shoulder ball, you know, so there's a lot of logged reps and banked reps there. And so that's where I think you see that confidence coming from and I agree. I think it does look different when we're throwing on time and we're cutting it loose to a spot and we're getting open physically and with speed. You know, I agree there is a different look to it. How about for the offense overall?  Does that kind of add like a degree of juice to the whole unit? Absolutely. I think anybody that goes and makes plays, you know, a thing back to the New England game and although the result of the game wasn't what we wanted, Don Trell popped that long run and you saw our bench explode, you know, and that's what this game's all about. Guys getting the opportunity to make plays and when they go and make them, it fires everybody up. And I would think if I was playing offensive line, which I never have, but if I were protecting for the quarterback and heard the roar of the crowd because my sustained block and the good pocket we provided helped him make the play that fire me up. He played a little bit here and there, but nothing like he did on Thursday. Did a part of you wonder how he would do and how did he kind of handle it from the time he was told he's gonna be the starter and just how he handled this position all the way through? Yeah, it was a unique situation from a timing standpoint cause some of those, you know, personnel challenges happened on a short week, but I've watched Dylan prepare and I know how hard he's worked. I know how hard Keith and Sully have worked with him and, you know, kept him in tune to the game plans at multiple positions. And so I was excited for his opportunity.  Derek Henry out, has it been kind of the best case scenario with these backs that have stepped up, especially guy like Foreman who has just such grit to him when he's out on the field? I don't know that there is a best case scenario to losing Derek, but I certainly am proud of the way those guys have handled it. Yeah, so having a guy like Foreman that's able to handle some of the heavier lifting stuff and then having a counter punch with Don Trell, you know, has been great. And it's a testament to those guys, how hard they've worked and Tony Dews for getting them prepared as well as he has. But yeah, we're fortunate to have those guys. Back on Dylan for a minute. Did he play well enough Thursday night that maybe he's moved up a little bit in the pecking order and you guys are a little more confident in putting him in if he has to plug into another position with Nate still on the COVID list. Yeah, I think as much as the season ebbs and flows, it's a little dangerous to get two into pecking orders. I would say he certainly didn't hurt himself. He instilled a lot of confidence in those around him out there and guys that hadn't worked with him before. So he helped himself from a confidence standpoint and certainly the staff feels great about him. What were the guys coached to do on the play where they jump off sides? What's the receiver supposed to do there? Maybe what's Ryan supposed to do? Obviously that worked out well. And how good was it to see that maybe executed on what could have been a giveaway down? They're supposed to do exactly what they did. Exactly what they did, just how we drew it up. Without getting too into scheme in particulars, you obviously want to take advantage if you feel like you have a free play and we did. Your meetings with Ryan this week, obviously going against his former team. Have you sensed any elevated focus or any different excitement or anything going against it? Honestly, zero. Ryan's so focused on the task at hand each and every week. It's one of the most impressive things about his professionalism. I haven't even heard him reference former teammates or any of that stuff. He's just focused on what we need to do schematically to get this W. Where Julio and AJ and Firks are on the sidelines, do you worry that it keys a defense too much into the probability of that being a run? Yeah, I think you have to be in tune to all sorts of personnel tendencies. We've been in some unique scenarios from our personnel groups this year, so I would imagine those that are breaking us down are having a little bit of a tough time figuring out exactly why we were in certain groupings and all that. But we track that and we try to take advantage of any glaring tendencies that we have. I don't feel remit John Madden, but during your time with the Raiders, could you kind of sense his presence on that organization and what were your thoughts about maybe what he meant to the league? Yeah, I was very fortunate to spend a couple of different days around Coach Madden, played in a bocce ball tournament, a charity bocce ball event with him when I was out in the Bay Area, and just a wonderful, wonderful man.  And he was one of those people.  So.


There are many conspiracy theories pertaining to hidden ancient relics which can be found upon our moon. Conspiracies plague not only the known visit to it, but also the possible past visitation by a now-lost civilization. A number of studies, including some here upon our channel, have exposed not only theories but valid proofs to support the hypothesis of ancient ruins not only being present on the moon, but ancient lunar ruins having actually been visited in secret operations by NASA and other space agencies. Not only aware of said sites, but accused of studying them in secret. However, a discovery has been made which has seemingly been a lot more difficult, apparently impossible to keep concealed. Thus, it has been exposed after having been discovered by a group of intellectuals at the Baylor University, featured within a study published in the journal Geophysical Research Letters. They combine data from NASA's Gravity Recovery and Interior Laboratory, GRAIL, and the Lunar Reconnaissance Orbiter, LRO, after finding a huge blob lurking over a hundred miles beneath the South Pole it can basin. Imagine taking a pile of metal five times larger than the big island of Hawaii and burying it underground. That's roughly how much unexpected mass we detected, said Peter B. James, assistant professor of planetary geophysics at Baylor University. The basin near the bottom of our sole natural satellite is an oval-shaped crater several miles deep and up to 1,242 miles wide, the largest found upon the moon. Located upon the moon's far side, also known as the dark side, it never faces the earth, permanently concealed from the world, it's an ideal place for future investigation by secretive agencies, as the far side would make an ideal location for any base development that one wished to keep hidden from the world. However, the metal object, which is of an incredible size, is yet to be identified, and if it has been found already by secret intelligence upon earth, it has until now remained unmentioned and, as far as anyone knows, unstudied. One of the explanations of this extra mass is that the metal from the asteroid that formed the crater is still embedded in the moon's mantle, said James, adding that the crater is believed to be 4 billion years old, almost as old as earth itself. This means that if this was indeed left by a craft, the beings who made it would now, if still in existence, be unimaginably more advanced than the modern man of earth, and could, in all possibility, still be visiting our solar system. Was this crater created by a metallic craft? If so, who were these beings? Were they interdimensional or merely interstellar travelers? It is an anomaly, like many others, within our solar system, which we find highly compelling.


Right, hopefully everyone can see my screen. Yes. Yeah. Yes. Okay. Well, whatever. Right. So the agenda as Chris is not here, I'll try to do my best and hopefully just, you know, raise your hand if, you know, I miss something or if I'm not clear enough. Today we're going to have one demo around PCF from, from you guys, Karen and Ramesh. And then we'll have a quick landscape and white paper update. I'm not sure there is, there is much there, but let's, let's keep the discussion going. I think Chris was meant to do a quick recap of, of chaos conf of fully there will be someone who can, you know, you know, tell us all about the, the conference. Yeah, I was there. Great. So probably you'll feel the role. I don't mind. Right. So let's get started. I think we are rolling for the demo guys.  I think that's probably best. Sure. I'll stop sharing. Is there any button here to share? I think on zoom, do you have a green share screen at the bottom? Yeah. There you go.  It works. Okay. So you want to start? Yeah, sure. Thanks. Good morning folks. My name is Ramesh. I'm the senior engineering manager for the platform engineering team at T-Mobile. First off, thank you for giving us an opportunity to present here. Very excited to know that there's a community around this and we can tap into like extensive community network to also get help on what we're trying to do at T-Mobile. But quick background on me. I've been with T-Mobile for 10 months. My team owns the container strategy for T-Mobile. And Karun is one of the brilliant engineers we have on the team. So Karun, do you want to introduce yourself for the question? Yes. So my name is Karun Chanuri and I'm senior software engineer working at T-Mobile. I joined this team in March, 2008. And so I'm fairly new to T-Mobile. Before that, I was with Huawei as a cloud security engineer for them. I have about 13 years of experience in the field of information security and enterprise security. So yeah, that's pretty much. So here at T-Mobile, we take care of pivotal cloud foundry operations and DevOps kind of a role. And also we have Kubernetes in-house cluster. These two come under Ramesh. OK, so let's get started. I know some folks are excited to see the demo and what we're trying to talk about, right? So I'll gradually take you through the journey. Karun, do you want to go to the next slide, please? Yeah. So one of the things that I'd like to start off with is Joker's analogy on how he interprets chaos in this movie Dark Knight. And we kind of spoke about that at our conference at spring one. And Joker calls it as a fair act, which is every time you disturb the harmony of systems, good things will come out of it, right? So that analogy has started kind of like started my thinking also, which is my team builds all of these capabilities on top of like a massive infrastructure. And behind the scenes, there's compute network and storage and things will always go wrong, right? So I'll get to the actual problem statement, but I always like to start off any presentation with who we are, what we do, what are the services we provide in this pack, and kind of drive you towards the problem statement and then hand it over to Karun. So our vision today at T-Mobile is to deliver simple, secure, scalable platform services that are infrastructure and platform agnostic. And we do this with relentless focus on customer obsession because we are connected to the needs of an internal engineering community with over 800 plus active users on our platform. And on the right side, you're seeing the evolution of the access service model. You might have seen different variations of this, but we're one of the biggest partners of our infrastructure team who is also moving towards more of an automation model with one click access to compute network and storage and other resources, but then we're their biggest customers and we're extending that portfolio to even give better capabilities. One of the capabilities and the container as a service matrix is offering for Kubernetes so people can bring their own containers and then a platform as a service abstraction where people can just give us their code, we'll go run it within our abstraction layer and they don't have to worry about other capabilities. It's like a self-driving car, we give them the bells and whistles of operating their code at scale. And at the same time, they get the best experience in terms of dealing with live customer issues. So depending upon the kind of abstraction you choose, different flavors comes in. So that's really what my team does in a nutshell. Let's move on to the next slide. So one of the things that folks have asked me is, okay, so what's the big deal? Every company is doing this, you know, what's really in my portfolio, which is driving the need for chaos engineering, right? I'll focus on the fact that we're building our services on top of on-prem infrastructure today. There is compute network and storage. We've gotten the business used to agility already in the last two years, roughly 4,000 applications, 500 active users per day for 31,000 containers across development and prod. They're a production for me, even though that's development environments. Business has gotten used to agility, which is faster applications, faster meantime to respond and resolve. The last iPhone launch event saw a max peak of like 16,000 requests per second, right at the minute iPhone launch was launched, right? And then since then it has been trending around an average of 14,000, hang on there. I'll tell you when to move on to the next slide, please. So then we're moving towards the feature, which is like, you know, where we want to extend our capabilities around simplicity, security, and scalability. We are trying to deliver many new capabilities on the function of the service and also exploring new capabilities at the platform as a service layer. So a lot of work that is already being planned in terms of agility, resilience, and enhancements, all of which entails infrastructure in the background. Next slide, please. Okay, so a bit of the explanation of the problem statement a little more.  Anybody? Okay, so, let's actually go through the animation here, Karan. So basically what you see here is what's called the Death Star Diagram, and it's a representation of the kind of the ecosystem your microservices deal, you know, live in, and the kind of interactions that they have with dependent services. The snapshot on the left is from Amazon from 2010, and the Netflix Death Star Diagram is the blue version, and then ours looks a little more less chaotic, but we're getting there in terms of, you know, what the chaos is going to look like in the future. So the key message here is, you know, we as engineers, we write services, the services have backend systems they interact with, and obviously when systems fail, you know, things, your customer is going to take the impact in terms of, like, any customer impacting events. So what we're trying to do here is, like, in terms of our digital transformation initiative, we're trying to think about failure in a different way and try to embrace failure, but failure is inevitable. Next slide, please. Okay, so, and I actually started off with this problem statement with Karun, and a couple of months ago, we wanted to look at, like, two kinds of failures. Obviously there is the platform level failures that I care about because I run the platform, and what I mean here is, what are the kinds of things that go wrong with my platform? What are the assumptions engineers make when they build services on top of an infrastructure, right? Think about things like a network being homogeneous, the fact that we have infinite compute resources, all of these assumptions need to be validated because when you fail to validate it, there's a when problems start happening, and, you know, you could get into a disaster scenario, like these two guys on the left side, which is not our data center, but somebody else's data center.  We have active volcanoes in this region, so we're trying to be cognizant about the fact that, okay, if things fail, how are our systems going to react? How can T-Mobile continue its business on because a lot of the business critical applications run on this platform, right? And then there is the containers that are running within the platform. Containers have applications, and it's not just one target application, right? There's several applications, so you want to launch specific targeted attacks on containers, and just affect that one application under context, right? Or the dependent service it interacts with. So that's a fairly significant problem on its own because, you know, because of the way PCF is built and because of the way containers work on PCF. Okay, next slide please. So this is where Karun, you know, I think you should hand off and talk through your journey, Karun. Sure. So looking at the problem statement, we have two problems there. One is the platform level attacks and the other one is applications running on the platform, you know, attacking the applications that are running on the platform. So we started exploring, you know, are there any existing tools because we didn't want to reinvent the wheel. So we started with an open-source solution called Chaos Lamer, but all we could see, make it work with the Chaos Lamer is killing of the virtual machines. But in fact, at the platform level, we wanted to achieve killing virtual machines, killing a process, introducing latency into the system and introducing memory and CPU hog, right? So all these come under the infrastructure level attacks, but Chaos Lamer for us was like more like a chaos monkey. It can only go and then, you know, turn off a random virtual machine, which is definitely not something that we are looking for. We're looking for a more bigger solution. So we started looking at Gremlin as one of the commercial offerings as well. And we see here the version that we evaluated with the Gremlin. It's a pretty very powerful tool, I must say, because it comes with a very neat UI. And there was initial thinking whether Gremlin would work on the PCF environment or not, but we made it work. And it seemed to fairly work, perform the operations at the infrastructure level, like killing of virtual machines, killing of process, introducing latency and memory hog. Again, the version that we evaluated doesn't have the application knowledge, which seemed to be the case that, you know, the Gremlin is working on it. And even in the recent intro from their CEO, Mr. Colton, I think it seemed to be coming up and they have added this capability in the latest release of Gremlin, which we have never looked at yet, right? But again, Gremlin comes with the cost and we are also very conscious about the cost that is involved, you know, and running on the infrastructure. So we looked at turbulence as another alternative and it's an open source. As you can see, it performs fairly, you know, and it's very native to the cloud foundry as well, like, you know, which is pretty much we were looking for any Bosch-hosted virtual machines can be done with the chaos engineering attacks or failure injection attacks performed on the Bosch-hosted environment, performing killing virtual machines process and introducing latency and CPU memory hog. But again, it lacks application knowledge, right? So for us, as I said, like, you know, as Ramesh explained, those are the two problem statements, like infrastructure level chaos engineering attack or the platform level chaos engineering attack and the application level attack, right? So here is what we looked at chaos toolkit as a framework. It basically orchestrates multiple other solutions like gremlin and turbulence as drivers. So what we built is we built two drivers there. One is a driver for the turbulence itself and then another custom homegrown driver that's built from the scratch, which has application knowledge. So it can go and then figure out, discover where your application is running within the cluster. So if I have a cluster of 2,000 nodes, in that 2,000 nodes, this driver can go and figure out your application is running on those particular nodes. It can also figure out what service dependence is this application has. So if it is dependent on MySQL database, it can go and randomly kill the MySQL database instance and see how your application behaves. So these are the two drivers that would be demo today. And let me, before I jump into the demo, let me explain more clearly what exactly that we are talking about here on simulating failures on the platform level, right? You can see the component diagram for the PCF, the pivotal cloud foundry. There are various components here. Each component could be a virtual machine or multiple boxes here could be processes running inside a virtual machine. So failure can happen. There's a lot of interaction happening. So it is so obvious that failure can randomly happen at one point at any point that it might eventually lead to the disaster as well, right? So for example, let's take a simple example here, the rep process going down. So what happens is the rep process running inside the Diego cell is responsible for managing the lifecycle of the containers running in it. So Diego cell is like a worker node in Kubernetes. So if the rep process goes down in that node, there is no way to manage the lifecycle. So it's one good way to simulate and attack via turbulence and the driver that we spoke about. And another attack is think of applications running in your cluster. Let's say a set of apps have auto scaling enabled, which means based on the load or the CPU stress or the HTTP latency, the apps going to scale up and scale down in terms of the instances. So for that, the auto scaler as a service depends on cloud controller. So what happens when there is a huge traffic to the app and then the cloud controller goes down. And then what happens is the app going to scale up or scale down, you know, those kind of failure injection tests that can seamlessly be performed via the driver that we are talking about here. So how we perform this, this is a turbulence. Turbulence comes with API server and the agent. The agent goes and sits in each of the virtual machines in your cluster and then listening to your API server, which is a control plane. So we use CTK and initiate few attacks, which goes through the API server and then agents fulfill that request. So all the ones which are highlighted here, like pausing a process. So some of the attacks that we can perform is killing a VM, killing a process, pausing a process. So that would be one of the demo scenarios here. Introducing a stress into the system by introducing like CPU and memory hog, corrupting a disk associated to a virtual machine, right? And network delays, limiting the bandwidth, reordering of the packets. What happens if the packets are reordered? How is your system going to behave? How is your platform going to react?  So these are some of the features that come with the turbulence and highlighted ones are the ones that we have added and contributed back to the open source. So let me show you the first demo for this. I would like to run the video from my desktop. So just give me a second here. Are you able to see my screen now or are you still seeing the presentation? Presentation indeed. Oh, okay. I think I have to share this particular window then. Okay, so. Yep. It's okay now. Yeah. So this is a demo I was talking about. So what we will do here is we are going to demonstrate how chaos toolkit has been used as a driver and then a turbulence driver has been added. What it does is like, you know, it demonstrates two scenarios. The first scenario is pausing a process. So here for this, we are going to pause SSH process. What happens if an SSH process has been paused to the Diego cell, right? And then the other one is killing a particular VM itself, like killing a Diego cell. What happens to the containers running in that, right? So a very short video and it's there on the YouTube as well. The reason why I run it here is it has better quality and the zoom-in effect. So first I go in here, pause process.json. This is my experiment file in chaos toolkit with title and descriptions here with some steady state hypothesis and configuration information that I'm supposed to pass as a part of the experiment. So these can be, again, can be enhanced, like, you know, instead of putting a username and password here, it can come from the vault as well. So what I'm doing here is this is a one box environment called Bosch Light, which gives you a cloud form running in one laptop. And you can see in the turbulence deployment virtual machines, you can see the API server, turbulence API server is running there. As I said, like, you know, there's a turbulence API server and turbulence agent running in each virtual machine. So that's what the configuration we provided in the experiment. And then the method here is to attack, pause process. SSH for one minute. So we are going to pause SSH process for one minute on deployment CF and group DGOSL limit to one, which can be any DGOSL. So right now in this environment, I have only one DGOSL since it's a one box environment, but we tested this successfully on the staging environment with about hundreds of VMs there. So there's only one VM here, DGOSL. Let's perform SSH operation on the DGOSL. As you can see, it's pretty responsive, right? It's very quick. And then now I go and perform failure injection test using kiosk toolkit by running this experiment pause process.json. So these are some verbose running the experiment and steady state hypothesis has been met as well. And then experiment ended with status has completed. Now let's try to do an SSH into the same DGOSL. There's a pause like, you know, for, and it's pause. So for about 60 minutes and we can go and check the UI as well. There's an UI aspect of turbulence. It is saying the pause process is in progress and it will continue to, you know, spin up till for like about one minute. So after about a minute, you will see the lock is released. Right? So you can try again to do an SSH and it's again, you know, very responsive after one minute, right? So because there is no lock on that SSH. So since we could do it for SSH process, you can do it for anything like, you know, you can do it for the rep process or anything. Second scenario is killing a DGOSL. Any DGOSL, like again, there's an experiment file for this separate one. We go with the standard definition experiment file for now. Steady state hypothesis is actually empty for now. Like, you know, we're not doing much, but we can do, we can add some probes there. Method is to attack and kill a DGOSL, which is running in the deployment CF and limit to one.  So any DGOSL.  So as you could see, there is the DGOSL running in this. Now let's go and then print the VMs. Clearly there's no DGOSL here, right? So it's killed, right? So what happens to the applications running in that DGOSL or containers running in there, right? So that's another way of looking at things. So again, the UI shows that the attack has been successfully completed. And that's the reason it is in green color.  So after a certain interval of time, you could see that the DGOSL is back because Bosch has created it again, right? So in the same, in that scenario, Bosch also made sure the apps are scheduled to back again. So that's my first demo. Let me go back to my presentation. So this is again, so that's the first half of the problem statement. How do you perform the platform level chaos engineering attacks is via turbulence and chaos toolkit driver for the turbulence. So the next half of the problem statement, which is crucial for us is application level chaos engineering because we have about 4,000 applications running on our platform. It's not a single, we are not a single application company, right? Where in you just attack an application and all the components of that application and see what happens, right? It's not that we have different teams building different stuff every day and about 4,000 applications, not at all independent. They're pretty independent and not interdependent applications, right?  It would impact multiple teams within T-Mobile and that's a big problem for us. We wanted to make a very conscious level attacks like in a very targeted attacks in a way that specific application is targeted for the chaos engineering. What would happen? How would, you know, without disturbing other applications running on the same DGOSL? So do you want to talk about this, Ramesh? In the ops world? Or should I go ahead? So I think Ramesh... Karun, sorry, I was speaking. Karun, I was on mute. Yeah. Can you hear me now? Yeah, so in our ops world we deal with an open support model where we get a number of different questions and I want to talk about a few of our favorite questions as to how we can actually not just be enablers but also be guardians, right? Which is not necessarily just to rely on best intentions but provide tools and capabilities that will also kind of like help with the best intentions when working with such a large intelligent customer base. First to one is that my app is in picking up latest configurations and our first reaction is this is because of bad karma and your app is misbehaving, right? Second one is my app is in connecting to Cassandra and our first reaction is it's because your Cassandra cluster was potentially decommissioned or you misbehaved with the Cassandra team. And then the next instance we see is my app works locally but not on PCF. This is likely because you misbehaved with the PCF team, which is us. And then last one we see is it was working till yesterday and then it stopped working and this is because we believe that you've not paid the bills for us. All right, so with jokes aside guys, these are some serious concerns where we classify them as debugging as a service and oftentimes customers like to start with us because we're very nice to them. We try to enable them. We try to like make them some sufficient but that's not enough. So you need to like provide tools and capabilities which will also provide God rails for them to operate with them. And that's where a tool like this is going to come, you know, going to be very effective which is it's going to like, you know, enlighten them as to what they can do to actually validate some of these optional conditions, right? And help them be more sufficient. Perfect. Sounds good. So having said that, you know, again, the same extending the problem statement, I would like to touch base on the cascading effect popularly, butterfly effect as well, right? So there are two, we all know this, what is cascading effect? I just wanted to, I don't want to dig into the more details but quickly explaining this. We have a concert and web, weather microservices running in our platform. And in this case, whether it's dependent on third party, what happens if the third party application, you know, goes down. So it totally impacts whether and that's times out concert, right? And it may so happen that, you know, the database also, which is dependent on, which concert is dependent on might go down. So all this put together creates a cascading effect and gives an unfavorable behavior to the experience to the web application running on the front end of, front end to the client, right? So the client will see experience a very unfavorable thing. So zooming in a bit, what happens if these applications are running in a spring cloud kind of an ecosystem? So these are the degos. So imagine that, you know, weather and concert microservices both are scheduled and running on the same dego cell. Dego cell is virtual machine. So both these containers are scheduled to are scheduled and running on one particular node. So it means what I mean by targeted attack is what happened? How about blocking the traffic to the concert? So coming from the go out or the load balancer to the concert service only and then blocking the traffic from weather service to the MySQL database. As you can see weather and concert services both are dependent on MySQL database. So I'm doing a very targeted attack here. Both are running on the same node, but these two apps I could go and then, you know, do a fine grained attacks in a way that, okay, blocking the traffic to the concert and blocking the traffic to database from weather. Again, the failure can again happen at different levels as well. There is a lot of interaction happening here. The weather might lose the connection to the service discovery or circuit breaker concert can lose connection to the config server as well. All these attacks can be simulated. All right. So how we do that today is CTK CF blocker. That's a new driver that we wrote and then target specific CF apps application host. So what it does is it discovers where your application is running and then it discovers what services your application is bound to. So in this case, my weather and concert micro service are bound to MySQL database config server, Eureka and then a strict service. And then it can also go into service instance. Like for example, the CTK CF blocker can also go into your config server that the app is bound to and it can bring down the config server as well. So what it does is primarily blocking all traffic to app instances and blocking traffic to bound services as well. So how we do that is again, the next demo for us is let me share this screen. So again, we are going to see two scenarios here. The first one is blocking database connection. So I have two apps running in the cloud on respring music and spring music too. As you can see these two, there are two URLs routes that's pointed to it. You can go and then check. Karun. Yeah. Karun and team. Yeah. Sorry guys. Sorry, didn't mean to interrupt you. I'm going to stay on the call. I have to like dial out of the web meeting. I have to drop my daughter at school, but I'll stay on the call. Okay. So the spring music. So just to see what happened here was the app got loaded and these are the breadcrumbs. These are the album information that got loaded from a database. So you can see here which database it is. It is my SQL database called music DB. So these two apps spring music and spring music to both are bound to this music DB. So now what we're going to do is bring down application connectivity from two music DB from spring music only. Okay. So again, we have a experiment file here. With all the attack definitions. And it has got probes. We have used probes in this example. We are going to check for HTTP 200. Okay or not for this URL. And then the method here is blocking a service and unblocking a service. So first we block the service for 60 seconds. The service that we are going to block is for the app spring music and the service name is music DB. And we have to provide some information like org and space name. So which is specific to PCF again. In the on rollback unblocking the service provide org name space name and the app name and then the service to be unblocked. So let's verify again. These two apps are pretty, pretty much working good. There's no problem there. So let's perform the attack now. Again, it's a Python script chaos toolkit. And running the experiment. It's a block service. And there is a verbose. You can enable verbose to have a deep dive stack trace information, but I will not use that for this demo. So what this driver is doing is it is trying to block traffic to music DB bound to only spring music app. So now it found where the applications are running. And the app has three instances. So it figured out all the three instances of the app running in your cluster. And then it started attacking now. So you can just refresh. As you can see, there's a successful attack. There's no data now. Again, spring music too, which is running on the same VM and pointing to same database can fetch the data. There's no issue there, but spring music, there's no data. So that proves our successful attack. And it takes like 60 seconds to bring the system back because we have a rollback policy after 60 seconds. So let's go back and see what's happening. So the rollback unblocking music DB has kick started. So there you go. You see that spring music is back in action. So within that 60 seconds, you can see what happens with your application. That's what is actual goal here. And it can be any service now. It's not just music DB, but it can be anything, MySQL or Eureka service, history, all the services that app is bound to. The second scenario is blocking traffic to an app. So again, we are going to use the same apps here, spring music and spring music too. So what we are going to do is we are going to block traffic to this spring music app only. Again, these two apps are running on the same virtual machine. Blocking traffic is an experiment file. Let's look at the block traffic JSON file. And you can see, again, it's the same, you know, similar pattern of definition configuration goes in here, study state hypothesis. Let me come back to the method here. What we are doing is blocking a traffic to the app spring music. There's no service here. That's fine. We don't need to define a service here. We are going to block the traffic for 60 seconds or, you know, and then the function has unblocked traffic. So blocking all traffic to spring music. So it's still working, but let's give it a time. And there you go. You see the traffic has been blocked and you see the 502 bad gateways. So the gateways aware of the problem route, but it doesn't know how to do a TCP connection to the app. So that's just a successful attack for us. Right. So and unblocking is been initiated. So you can see that, you know, unblock happened because there is no time out for 60 seconds here. So it happened so quickly. So that's that brings down end of the second demo as well. And let me go back to my other slide. I have like two, three slides now. That's that's all I have. So I think Ramesh is not in the call. Okay, anyway, so this is upstream contribution we have made. So all the demos that you have seen here, they are part of our upstream contribution from the T-Mobile side. There are two. I don't know if you want it. No, no, no, you can speak. I'm actually going to be driving so you can speak for this. Yeah, please. So we have chaos toolkit CF blocker driver and chaos toolkit turbulence. We wanted to bring these two into the chaos toolkit umbrella, but that's the effort we are putting in as well. And these demo videos are available for you to go through again. And we have raised, again, turbulence is built by another person, you know, and we, we are not the ones who created turbulence. So we wanted to do a pull request and we did the pull request with all the new feature add-ons and it's pending approval of the PR still. We'll wait and then see if there is no action happening. Then we wanted to create a new report on top of it and start adding more features to it. So you want to talk about this, Ramesh? This is our team. What do you want to do next and all that, Ramesh? So the next is actually, we wanted to conduct some game days, game days. We wanted, this is still a slightly matured and proof of concept right now. We wanted to make it work and productize it and call out teams on team-by-team basis and perform game days in a war room and randomly attack our own infrastructure and see how their applications behave. So we wanted to build this capability and give it to the application teams to, you know, perform chaos engineering attacks on their apps. So that's all I have. Well, thank you very much, Karen and Ramesh. It was a really, really sweet demo. It was really, really interesting to see. Thank you very much for that. I'll carry on on the slides if you don't mind now. So if you share. Should I stop sharing? Yes, please. Yeah, thank you. It was really a very interesting demo. Very fun to watch. All right. I think that's one. All right, so we pass that. Nope. Yeah. I think we are back to all, you know, usual discussion around landscape and, you know, and crafting the categories. I personally didn't get any chance to actually look at it yet. So if anyone has a fantastic, I'll be paying. I see there is a prerequisite. I don't have the prerequisites as best as I can, but I think the idea, and we'll be talking about KubeCon in a minute is probably to have that wrapped up in some fashion enough. Anyway, by, by soon ish to actually get that, you know, demonstrated that KubeCon and, you know, and basically settled in some fashion. So I don't know if anyone has comment to that, I don't want to put a request specifically, but, you know, please raise your hand. Happy to try to talk about that. Right. I'll carry on. Just, just let me know. Otherwise. So yeah, KubeCon is a Windsor bonus, but certainly not far away. It's a Christmas, you know, treat almost. And we have, Chris has set up and two, two talks for that. One is the intro of Chaos Engineering and the other one is deep dive. The intro is certainly talking about the working group, but not only. It works like you presented Karen and Ramesh and, you know, all things that have happened in the field of Chaos Engineering, you know, Grayling announcement, recent announcement, all those things need to be, you know, saying basically this is where we are at now as a community with Chaos Engineering. The deep dive, I think, at least I know I and Julian, who presented two weeks ago, will be doing a demo, a joint demo during that, that one on ECO where he started and showing how we can actually automate that after that with the Chaos Toolkit. Please, Chris, if you're interested in talking or being there, you know, in some, some capacity, obviously. I'll reach out to him. Yes, please. Don't reach out to me because I don't have any control over that. I'm just passing the message. Yeah, please do talk to Chris. You'll be happy about it. All right.  So we, T-Mobile, we are in Seattle, so, and we are joining KubeCon. So hope to see you guys. We'll see you there. That's definitely fantastic. All right. This is where I'll be leaning on the people who were at ChaosConf. I've felt very good things. So I'm willing to listen to a bit more about it. So who's be, you know, would be willing to talk about it really? I think Michael. Yeah. I don't know if there was anyone else from the group. So ChaosConf was about a week and a half ago in San Francisco. We kicked off the day with Colton from Gremlin, talking about the sort of evolution of actual chaos engineering attacks and sort of concluded by talking about Gremlin's new product, Alfie, which is application level fault injection, where you can write into your application various attacks, which was pretty cool. Yeah. Adrian Cockroft from Amazon then spoke about the history of chaos, which I really enjoyed. It was really good talk. We had a number of other presentations from, I think the Bloomberg. We had a great one from Twitter. I'm just talking about, so one of the lady who was presenting was a technical diver and she was talking about chaos engineering in team building. Later in the afternoon, we had Tammy and Anna from Gremlin do a really cool demo of breaking AKS and EKS, which was pretty cool. And we finished the day with Jesse Farzell, who was talking about containers and breaking stuff in containers. So it was a really good day. Sorry. No, no, go on. I was just saying nice. Yeah. It was a really cool day. Good to meet more people in the community. And I look forward to next year. And I hear them may have a new venue, which will be cool. Very cool. Exciting. I'll be trying to watch to all the videos online. It always takes time to watch all the talks, but it's usually worth it. Yeah. All our videos are online, by the way. That's actually a really good point. I just posted the link. All right. Thank you. I was just digging it out of my inbox. I'm going to scroll online now if you want to watch. Okay, cool. I think last time when I clicked on the link, it was dead. So have you posted the link here in the chat window right now? The link to YouTube link to the Kiosk on talk. It's in the Slack. I'll post it in the chat now. Okay. All right. Awesome. Awesome. Thank you. Thank you. Carry on. That's back to what I was saying about the working group work. I need to pay attention personally to the state of the PRs and everything. Not just the one I mentioned earlier, but the various ones on the, on the white paper, just try to merge everything into one document and see where it stands right now. Probably needs a lot of polishing. All LPs, you know, obviously welcome. And that's, that's about it for this week, you know, a meetup. Anyone ask you and I think Chris usually ends up by saying that we are welcoming any demo and anyone wants to do a new demo or better one, you know, whatever. I think it's always cool to have that. So pass along as well to people that aren't aware of this working group. I think it's important to have, you know, I'm sure there are plenty of companies who do, while doing chaos engineering or resiliency engineering, whatever you want to call it. And they should come and just show it to us really. All right. Any questions? Not that I can answer them probably, but they'll be there. No, that's fine. So this working group is, so we are, we are late entrance, right? Ramisha and I and what is the expectation like, you know, what are the best now low hanging fruits that we could grab and then, you know, start contributing? Good question. Yeah. Yeah. If anyone wants to respond, I don't want to hold the, you know, I actually actually done this sharing the screen so that we can see each other. Happy to let anyone answer that one. I can go for it. I think I think the working group is not yet a working group. That's important to notice. From a CNCF point of view, it's not yet a working group. It needs to be proposed and accepted and blah, blah, blah. The point of what we're trying to do as a community is, is to bring everyone across and basically work together. And for that, we try to at least have a white pepper.  I know Chris is, is, is looking at expanding on the landing scape of the CNCF with tools like turbulence could be one for some of the tools. Seeing this is a field and, you know, bias technology that exists. So I think right now it's, it's more awareness and trying to have that step one to, you know, to get us that trampoline that get us elsewhere after that. I have a question. Did you have this discussion? Sorry, I didn't come from a while. Did you have this discussion for Chris? What is the next step is to transform this kind of to a real working group in CNCF or, or what is the, did you, I think it's important to fix date or milestone because something interesting to fix target, maybe next, maybe not next CNCF but Barcelona or something like that to, to achieve something and to deliver something and to go further for that. Yeah. And Michael, Michael did mention that last, last time as well. We need to. Sorry. I didn't. No, no, no worries, but it's, it's a good point to rate. I think the idea is, you know, I, I don't work for this CNCF. I don't know that, that proceeds. So I, all that I can do is echo what we discussed before, but I, as far as understand the idea would be to put that work, you know, that kit up repo in good shape where we, we can, you know, agree on that white paper, at least with, you know, phase one, version one, it doesn't have to be stable or anything and, and increase the landscape so that we have a good view of what's, what exists as a milestone. I think it would be before the end of the year. And the best would be if we could, you know, do that before KubeCon because during KubeCon, I think Chris is trying to raise more awareness and more, you know, promotion. Yeah. So. That would be better. Okay. That means no official deadline. Maybe Chris having better visibility, but the most, the best case is to fix the target to finish the landscape before the end of the year and to motivate to, to grow up to working, a real working group for, for the target to write specifications. I would say specifically, but writes best practice and co-engineers, guideline for co-engineers, for all engineer working in the microservices. I think, I think that's interesting because as far as I understand the CNCF tries to, to avoid specification initially. They don't want to be in us, you know, a governing body in that, in that sense. I don't understand it. So my, my, my assumption here is that we do the tidy up, you know, within like six weeks or something. So that gets ready for, you know, during November. So that at KubeCon we can meet up and probably discuss about next year. If we look at what serverless has done, it's certainly interesting. I was talking about specification. They came up with a specification on the side. And I, I don't know if we need a specification or, I don't know what we need as a community, but KubeCon and, you know, another meetups here for those who can't come to MewCon are the good place to say, this is where I would like to go. And as a community, we can basically decide that. So it's, it's probably up to us as well to say, not just where are we, but where we want to go. That's me. That's me. The next since Seattle is a good place to discuss about that. Well, every time we meet up in person is probably a good, a good, you know, it's, it's, it's faster. It's more concrete and, you know, it's, it seems to work better.  Somehow when we meet up in real person, that seems to be faster. Thank you. Thank you for your answer. All right. Any other question that I can't answer properly, probably. All right. I, you know, it's five minutes to the end. So let's just wrap it up. And thank you very much for, for everything today.  Thank you for managing the session. Thank you. Thank you very much. Okay. See you next time. Bye-bye.


more EXO Zeros content. Today's video we're going to discuss the overall tier list for March 2021 but before that if you want an updated EXO Zeros content please hit the subscribe button. Hey guys before we start I'd like to invite you guys to join our Discord server for any chat regarding EXO Zeros discussions guides as well. Please check out the description below so we start off with our tier C so more or less your tier C are if I would classify them they are the heroes that will be good as you start the game they are actually very good kind of substitutes for the existing you know the existing meta meta heroes okay so we start off with Ramji so this is another version of Ramji that has turned into an FC but never really took off because he doesn't really have that much to add aside from his you know nullify heals and his kit is actually also squishy that is why he is in tier C you would only bring him if you don't have first guardian Ramji okay so before I proceed please ignore Valentina right there on your right side she's not part of the tier list she's just there to watch over us okay to proceed Bernadette is there for tier C because usually again for those who don't have a top tier you know burst high damage heroes Bernadette is a very good substitution at tier C because of her power powerful burst for S2 and a low cost S1 okay and take note that she has flop flop ups well and summer's blessing so she is actually viable for tier C next beside her would be Othard or FC Othard so FC Othard is actually she's actually good she she can be in the upper tiers but since there are a lot you know more viable high damage high damage fire fire heroes in terms of their FC their their better most of them have actually been played in the top teams of PVP while Othard hasn't had her you know has her time hasn't had her time yet in going up to the top tier of PVP but she is actually very useful in your PvE setup or your PvE PvE runs so actually she actually deals extra high damage for those who have status effects so that is why she is here at tier C I'll point out later which are the best fire heroes in terms of her high damage category when I go to the upper tiers okay so next would be Redley so Redley is again he he when when you actually are starting the game FC Redley is basically one of the best you know picks that you have early on he gives you he's a good attacker and a buyer support provided that you have command with Garf on your team okay so he is very good at tier C again very still still very good for PvE and also for PvP in the lower tiers okay next would be FC Emma FC Emma is excellent in on debuffs and on attack and on defense she she used to be viable in terms of pairing her with Linombe heroes but now you you have FC April you have you know you have FC Schmidt out so she has slipped from probably I think she was in tier B or tier A before she's now slipped to tier C okay next would be FC Baraka FC Baraka is this is again is another you know it's it's a shame that they put out these FCs and they have duplicates and the duplicates are very very good so for this version of FC Baraka he's still good he actually is the less defensive version of the FC compared to the first Garjan version of Baraka so but he is still good if you still want to use him okay good in PvP content and PvE as well so moving on to the next would be Degas so FC Degas is good because he has shared health and freeze for his S2 the current meta of the game actually has driven him down it's as driven down his value because there we have had we have a lot of defense oriented heroes that are better stacked than him in terms of skills that is why he's actually now in tier C if you don't have any of the upper tier heroes FC Degas is a good grab at tier C okay so next to round out the tier C would be Uloom so Uloom is again the basic version of himself so he actually is still viable because if you don't have FC Uloom the original Uloom will actually still break garden stones if you have bathroom routine and if he has track well so that is why he is so useful again in your PvE and PvP content okay so this actually rounds out our tier C okay here are we now with our tier B so they're a bit you know there some of them are actually used in the mid tier of PvP and they're very good in your PvE content so we'll start off with blue Rachel so blue Rachel is similar to you know a gold fate for Rachel he just has less damage and but he has silence actually in his S1 so more or less preference would be gold Rachel because of the damage output of the gold versus this version of blue so that is why he is in tier B but if you don't have the gold fated version he is actually a good replacement once you get or before you get gold gold fated Rachel okay so next would be FC Luke so FC Luke is here again FC Luke came out after Degas basically he is actually better than Degas because he absorbs all damage from all allies he is actually a very good tank but again is you know his skills is limited to basically that only okay to his to his left is actually FC Scarlett so FC Scarlett is one of the one of the first I think one of the first FCs or sorry Summer Festa FCs that was actually useful in PvP way back together with FC Iris so she actually has a S1 and S2 as a burst scale plus she has flap flap that is why she's good because she can turn your you know your enemies into rubber duckies or probably a fish so that is why she's still here in tier B for FC Anastasia again FC Gar FC sorry first guardians actually are a bit you know they are valued very much in this game especially for FC Anastasia with her heal or her burst heal and with her S2 she is actually a very all-around you know all-around healer with her own right in terms of dealing high damage also she's a first guardian so she's tough and she actually punishes enemies who attack her that is why she's still good at tier B next would be regular batteries or regular batteries not that bad versus your FC battery she has she still has the basic skills that make her good aside from the FC form so tranquil is still her you know her her bread and butter and she actually still is very good even if she doesn't have her FC okay so try to bring her even if you don't have her FC still okay next again to her left would be FC Adams so FC Adams has climbed because he synergizes well with Astorys Republic heroes like Junai FC Shell and who else so more or less the three of them are very good he deals actually high single target damage for both his S1 and S2 that is why he's valued and more or less he has heart strike as well I forgot to mention Luke has also heart strike so mostly all of the Banga family FC have heart strike that is why they're still valued in this game and Adams is valued at tier B as well okay moving on to his older brother which is FC Jin so FC Jin the ninja is actually also good because apart from again having heart strike FC Jin is actually giving out silence as long as your enemies are charm so that's it so he can control you know control the tempo he's actually come to think of it good with FC Schmidt because if he attacks again and deal silence then you know it's kind of pretty good synergy Astorys Republic with FC Schmidt so I'm thinking about that let me think more about that okay but before before anything else let's proceed to your left so you have FC Brooke FC Brooke has been very good in PvP before probably one of the top heroes in PvP for you know past few months but she has dropped in usage and she is still viable again in both PvP and PvE content okay she is actually valued for her relentless attack once once another you know another hero attacks your enemy she actually deals a normal attack I think there's a certain buff applied to her for her to do that okay next would be regular Garf so regular Garf is not really regular he's actually pretty powerful still same shared health same you know same S1 S2 and he is actually still tough the only you know the only add-on for FC Garf would be I think his damage over time for his S1 but FC Garf you know if you don't really have you know FC Garf regular Garf would be actually a very good still very good in your PvP also in PvE as well so take note so again he is actually a very good defensive type character if you don't have the upper tiers of his version okay so now here we are with tier A so the creme de la creme more or less is rising so we have to start off on your right excluding FC Valentina actually she's there sorry in team A so but we'll go to go to her later so to start is FC Uloom so the difference between FC Uloom and Uloom would be this version the FC version actually heals himself or heals herself sorry so that is actually very good about her in terms of being tanky breaking garden stones and having provoked she actually heals herself okay next would be Deva so Deva is actually a bit she dropped in usage already because again way back when the meta was still squishy she had her way with a lot of you know a lot of teams she was priced I even had to level up by them Deva so that I could compete in PvP but again she is still viable in PvE and the lower tiers of PvP or PvP tag even so Deva doubles critical hits and also has kind of silence for her S2 so you can't go wrong with Deva she's actually still viable the problem with her is she's just that squishy in the meta right now just have to make sure that she survives and she deals enough damage okay so moving on to your left left right okay your left is FC Maggie so FC Maggie has been touted to be one of the best heroes in the dragon and holy dragon grounds she's actually also included in some of the teams for Wasted Red so she is actually very good again in PvE can't go wrong with FC Maggie again if you notice earlier Autard is at tier C because of actually one of them is FC Maggie she's actually more tankier because she's a first-guardian and she's over she overall in terms of mana in terms of generating for herself she's actually better than Autard so I'd take FC Maggie any time over FC Autard okay next to your left again would be FC Lepin so FC Lepin she used to be you know she used to dominate again there was there is just actually a lot of you know better you know better FCs or heroes that came out but again she is still usable in the lower tiers of PvP and also in PvP tag she's very good actually in PvE content because you will need eventually to to to revive your hero so she has revive heal and also a valuable turn reset and damage immunity but you know because of that guy on your on your on the left most side has his FC already well I guess Lepin will have to be contented with PvE for now but again can't go wrong with Lepin she's a very good support hero for FC okay next again to her left is FC Baraka again the first-guardian version he actually has catapulted his status towards tier 8 because mainly of the team composition for Team Linombe he's one of the vital heroes there he's actually a good hero because he's a blue fate core but he gives bonuses to wasted red so I have him at I think plus two or plus three fusion but I actually Baraka is actually very good very good in the early stages of the game very good as well in PvE and right now one of the most important team members in Team Linombe so you can't go wrong in investing with FC Baraka the first dragon version okay so next to your left again would be FC Ramji this version of Ramji is actually going to be great actually he is great ready usage wise in terms of countering no the stall meta which is going to die because of FC Schmidt again Ramji's usage might drop so but for now he's still being used very you know very good for his nullify heal very good for his first dragon passives and he is actually a good defensive character I use him actually in my wasted red team with Garf FC Garf FC Red Lee FC Annie him Ramji FC Ramji and FC Iris so again I still you know value him and I don't know if his usage would go down but again you can never go wrong with FC Ramji on your team okay to the beautiful lady on his left would be FC Valentina so I have twins there sorry for that but FC Valentina very good in terms of control the problem with her is that she's dependent so much with with her father Shufracken or FC Shufracken that she's at tier A her father's not here so let's look for her father later but but actually she is very good in terms of in terms of locking opponents limiting them to less mana not you know mana burn and not letting them gain mana so very good actually in the recent she she and her father in north one frosty are one of the most sought-after teams right now combos shut down opponents they might still be in the running even if FC Schmidt is out already so again her usage might might take up depending on what other team comps we can build to counter FC Schmidt or with FC Schmidt okay to your to your left okay a Schmidt so the regular Schmidt is actually very good again because a prey in Masters footsteps so turn resets his his FC is just crazy this version of him is actually pretty much very good already very good in PvP very good in PvE so again you can't go wrong with FC Schmidt if you have leveled him up so here we are with our tier s so these are more or less the heroes that are very good but some of them need certain conditions like for example you have their FC shell FC shell is actually very good but she has to be with Junae so again they cannot really stand alone she can you know she you can actually value her without FC Junae only for s2 but that's basically it but she is a very good character to have or a hero to have so and she has you know she has she can actually do turn resets for any of your heroes add mana and supposedly if you have FC Junae on your team she can nullify a barrier so what what's wrong with her kids she's actually good but she has to be paired with FC Junae and FC Junae has to put charm on the enemy so that is why she is here at tier s next to her is FC Annie sad to say FC Annie has dropped because I see in the future in her future that she's going to have a hard time with FC Smith because I actually had a playtest with my own team using FC Annie and once Schmidt hits the team with prey her attack and defense goes down again also once her attacks in the so once that once her attack and defense goes down because she gives damage immunity same as the whole row so including ruddy as well so some some players might think twice already and putting her in this lineup so I'm putting her at tier s even though she is that good but when faced with FC Schmidt she's really going to you know to cry her are her heart out okay so more or less she's still good in PvE as well next to her is FC Zeon FC Zeon has actually catapulted his status up because again oblinombe and it has regained part of his you know previous stature as a dominant hero but actually his kid is crazy good relentless attacks turn resets AOE attacks single target attacks you name it he's actually relentless and I sometimes hate him when I face him together with other linombe heroes so very good he's up to tier s I'm sorry FC Tatalo but you're going down you're going further down I don't know in a couple of months but right now you are actually a tier s from tier SS because again of Schmidt because of the anti nullify heal decrease attack and decrease damage for his row so again what could I say so it's actually fortunate now that he's still being used but what if in the future people don't use him anymore in terms of in terms of PvP but actually he still is very good in PvE content that's actually why he's still here his excellent in PvE content in terms of damage over time in terms of heart strike for the bosses so basically he's still good probably a he's now a 8 out of 10 before he was a 9.5 probably or a 10 so I'm sorry Tatalo he's you really have to step down okay next would be FC Shu fracken so again here's the father of FC Valentina so he's here he runs the meta he runs north one frosty and again their team is still winning because of him so he is here at tier s he has stun he has stun for the back row he actually has fear of the abyss for those who have less health so extra damage again very good kit very good in terms of defense and he is a tier s to his left is actually FC April FC April could have been tier SS if not for her dependency in in FC Rachel so she heals allies when she dies she heals them she silences them with her s1 and but she requires FC Rachel so so that your enemies would be disheartened okay proceeding now to the young lady to your left okay so it's FC Iris FC Iris actually I think we'll just hover around tier s and tier SS currently her uses has dropped because people have been packing their teams more with damage but again you can't go wrong with her nullify barrier and her mana for everyone on your team so she's still very good PvE wise or PvP wise again PvP tag will always have a place for her actually most of these guys have a place in PvP and they will still dominate last on tier s would be FC garf my garf okay my garf because I've been using garf for quite a while for my wasted red very good hero except when when when she encounters tantalo because she usually gets heart strike very good hero very tanky shared health in terms of in terms of protecting other allies she's good back row damage very good as well so very dependent in terms of your team for wasted red and she's the generals of course she's that good so for our tier SS we have here the Krem the la Krem of more or less the best heroes right now in PvP and PvE to start off we have FC Reddly FC Reddly again can't go wrong with him I really underrated him when I started now he's my favorite hero gives you defense gives you HP gives you a barrier what can you ask for that plus he gives a lot of burst damage for his S2 so no if you can't appreciate that then I don't know what you know what is good for you so moving on to his to his left left okay left would be FC Talia so FC Talia is basically a you know a healer on steroids so everything heals heals for days for her passive for S1 and S2 what can I say she's a she's a healer galore okay next to her left again is FC Jani with basically charm lowers attack speed dot increase damage decrease attack and increase attack so that is actually incorporated in her kit so why she's not used that much because again it's only her in FC Shell that's usually viable in stories apart from FC Adams or FC Jin those two are just the you know the most versatile heroes that you can use and she is very good actually in PvP and also PvE because of her damage over time in terms of breaking garden stones perfect together with Tantalot they are insane okay so FC Jani premiere you know she is a premiere hero in terms of overall overall usefulness in this game next would be FC Bathory FC Bathory again has been and has been in the has been in the top ever since because of tranquil survivability high damage burst damage you name it even if there is summer's blessing she can still survive on her own give her team give her team one more chance to survive because of tranquil is actually nuts so she will not go away as one of the top heroes in this game to her left is FC Rachel FC Rachel is actually here because of the top team for top team for PvP he's here because there won't be any top team for Enombe without him without the signature force without you know the synergy that he gives to you know to all the to all the allies that he is with so speed again speed increase hit attack speed decrease dodge for the enemy and defense as well so again very good support very fast and very efficient next would be one of my favorites all time favorites in terms of overall utility and usability sometimes she is she's in the top in terms of PvP right now she's she's very solid with North One Frosty it's Dorca so Dorca in terms of damage in terms of control in terms of heal what else can I say so basically she's very good all she needs is just an FC an FC if you can see Schmid Schmid has been you know transformed with this FC I'm just waiting for Dorcas you know either black or or gold FC not her orange FC of course but she's you know she's good to dominate more PvP and PvE so he can't go wrong with her damage over time as well for her S1 so again Dorca very good always reliable always going to be a top hero in this game and in my heart okay so next to her left would be FC Rera so basically FC Rera has been in this game for quite a long time has dominated a lot especially with Bathory in the lineup but without Bathory she can still hold her own okay take note of that I've seen team dominate without you know her pairing with Bathory so basically very good in terms of high damage skills and she also gets added benefits if she has tranquil from Bathory okay so again she also has Spears damage okay so take note of that and last but not the least our most recent addition to the family of tier SS so again the insanely insanely overpowered FC Schmidt I hate to admit it but he makes Linombe crazier than crazy okay very crazy again if you haven't seen my review on him or my you know review on him my gameplay and also my pulls for him I'll put up a video up there so that you can take a look on how I reviewed him in terms of his gameplay and also a discussion how how he fares in this current meta but actually he is really good okay so master's footsteps he still has it so basically pray as well so reset attack anti anti damage immunity again as I mentioned with tantalus slip because of this guy Annie as well so lepin and the list goes on in terms of those who have damage immunity their attack and defense will go down 80% as well as the same heroes on their row so you know it's actually a really big headache so I kind of found you know a way on how to counter him already into our list keep his mouth shut so basically silence him not use his spells but again it's really difficult to be in a matchup with FC Schmidt okay so he also has an increased attack by the way and damage reduction okay so he's very good very good kit released just few days ago but I see him being in most teams or if not all of the teams in the top 50 for Asia server very good ridiculous and you know very very impressive for FC Schmidt to debut in XC and XO heroes for his fate okay so anyway guys it has been a long tier list and anyway if you have questions comments below write them down again please subscribe to my channel it really helps my channel a lot and always guys take care stay safe this is the Warden and I'm out of here