I literally had no clue that a new season was going on until like last week because the previous season took like 12 years to finish. I don't know how I'm feeling about this one, the epic really needs to step up their game if they want people to come back because chapter 2 season 1, not the best. In fact, I think this is probably the least hyped season in a long time. Hey look, animal casting, I was watching that. Wait a second. Go from season 7. Peely. Oh my god, no way. Did I see the red ninja? I like the echo people, the ego. The agency, you come highly voice acting. Oh. Oh. How. Customization. What? Ghost or shadow, choose wisely. You'll also embark on limited time operations that will change the fate of the island forever. Your future begins on the battle course. Earn your team of agents, 1,500 fever ups and all new items. The future is in your hands. We know you'll make the right decisions. Good luck.  Oh my god. Well, you already know what we're doing. What is going on? This is certainly different from previous battle passes. Oh my god. All right, so why is your angle like this? That's kind of weird. Anyways, so you're there and we got agent Peely.  Oh my god.  I'm not seeing any map changes. What's really disappoints me. Actually, wait, we got to check out the battle pass. How do I actually do that? Here, what cringey things is being added today? What's this? Call me, of course that was on the trailer. We got that. Okay. Wow. You spin very fast. Getaway case. Nice. Banana briefcase. All right. That's kind of funny. I'm not going to lie. Got a new wrap. Okay. Spies. Can't really hear it. That's always nice. Oh, all right.  Felix. What's going on now? Clash. Okay, that's okay. Oh my god. That is the best spray there ever is. We also got kingpin.  What is this guy? Kind of skip past some of these. Oh, that's a glider. Interesting. Mountain raid. Okay, so is there a boat on the map? I'm guessing that is an interesting wallpaper. What's this? Wait, what is this? Oh, wait, is it just a glider? I think it might just be a glider. I don't know what this is. Oh, it's the first glider you can ride. That's awesome. Not gonna lie. Kind of digging that. What is this? Okay. Yeah, sure. Uh-huh. Awesome. All right, so we got some booms. We got agent peely ghost. Okay. Got this. And then dance. Interesting. Awesome. Funny bounce. So I just know the music thing. Got you. Phenomenal. Got... Jesus Christ. If he's not the tier 100 skin, then oh my god. What is this? Oh yeah, we got grenade mouses on a ball of yawn. Great. Got a meat. I don't know why it's all the way over there. That's great. What's this? Really running out of ideas? Everything's fine. Oh, wow. Great. Is that like that one meme with dogs in the house and it's on fire? I get it. Jokes. Solid jokes. Epic games is so funny. Anyway, is this another rideable one? Because I... Oh, no, it's not. That's so cool though. I like that. What's this? Some soup? Some alphabet soup? What's this? Pause it. I'm good. I'm good. Thank you. I... This is just buff helpy. Like, let's be honest, that's just buff helpy. Oh, I skipped something. Oh, that's cool. Here it is.  What's this? Beep beep. I'm good. I really like those actually. Nice. What's this? Bon bon. Go get them. Oh, wait. No, it's just chocolates. Great. Dangerous waters. That's nice. I've really... Okay, if there's not any map changes, I'm gonna be real mad.  Oh, what the heck? That's just the new default glider, but it's... All right, yeah, just reusing ideas, I see. Shadow, of course. The mighty tree. All right, I guess we're gonna be looking out for a tree. Denim destiny. All right, sure. Then we got you. You're the one that actually rides whatever this is called. Ollie. Then we've got... What's this? That's actually pretty cool. I like that. What's this? Coinflip. Sure. This. The gold touch. I heard that there was gonna be like a new material. So maybe gold's in the game now?  The banana bomber. Another reused thing. Oh. I mean... Okay, yeah, sure. I feel like we've gotten something similar to this before, but whatever. We're gonna go with it. Boss fight. Nice. Okay. Golden King. Phenomenal. And here it is. The final... I don't think...  Oh, it's literally just a night ball. Okay. Then we got another remote. All righty then. What's this? Do you have any stages that you do know? Oh, I didn't realize that I could... I would have done this for the other skins. Damn it. All right. I guess he's just playing like that. Okay, nice. I really do like this new hub, though. Dear Fortnite, I came up with a cool redesign for the Battle Bus. I bet my friends, my good drawing arm, you'd put this in the game last season. Please tell me your artist limbs are on the line on... What? Wait. Oh. Don't thank the bus driver. Okay, I guess we found his hideout. Sure. I didn't even mean to do that. What's up here? Upgrade vault. Oh, this is where you... Gotcha. Gotcha. Oh, we have to complete challenges. I really like this. This is cool. Customization. That's nice. What's up here? Agents. Oh. There's so much new things to explore. Brutus. Okay, sure. I don't know what I'm doing. There's so many new things. All right, so as the season goes on, we'll unlock more dudes. All right, daily missions. Wait, so where are the regular missions? Are there no regular missions? What's this? Coming soon. Oh, wait, challenge table. Oh, no, there are. Oh, this is interesting. I don't know how much I like this new missions menu. What's up here? Also, they did this. I can't play it for long because I don't want to get copyrighted, but that is a thing. All right, so let's jump in.  How ironic is it that we got this loading screen? Like, they're basically just doing season four over again with the Alliance. Also, was there an event last season? I don't think there was. Again, I kind of missed out on a lot of things last season. I really did not like chapter two, season one. It was just so bad. I felt like they got rid of a lot of things that made Fortnite different. Oh my God. All right, map changes. There is something up here. Okay, so there are slight changes, but they're so small, but I guess we have to go to one of the locations. So here we go. Oh, I see the boat all the way over there. It looks like there's a house over there. Oh, what is that place? I kind of want to go there. Damn it. And this looks like I made a bad choice. Yeah, we're going to die. Oh, so this is like their home base? Sure.   Oh, hello. You want to hide together? Oh, nope. Oh, no. Oh, add a place to drop. I don't know what you can crawl around in the vents. Sure. All right, the mini gun is back. Oh my God. I should have read the patch notes. I did see that they bolted the trap though. They did bring back the heavy sniper and the SMG, the suppressed SMG, I think. And a few other things I think they brought back the grappler. Launch pad? What? I mean, thank God they brought that back. Oh my God, there's freaking mobility in the game now. What's this place right here, huh? It's another base? We're going to head out over here. Excuse me. What? Is that a bot? That wasn't a bot. What's this? It's me. Damn, me looking fine. Anyways, disguise as ghost henchmen. Oh, yes, of course. But now I'm not peely, man. So these are literally just like the sneaky snowmen, but boxes. Yeah, okay. Looking good, man. Looking good. So you disguise yourself so you can get in there. I got you. I got you. Another hideout right there. I like how the challenges stay on screen. That's really helpful. Also kind of sad that there's no new vehicles. There's one thing chapter two needs, it's mobility. And it's good that they brought back the launch pad, but, you know, they could have done a bit more. So my friends online, Cam, I'm going to join his party.  All right, here's the thing. Cam, he plays on mobile, but he's really good on mobile. I don't know how he does it, but he's pretty good. Oh god, that'd be so bad. Wait, what's ShakeDown?   I don't know what that did. I literally just like jumped him. Nice. Yes, Johnny. All right, where's Cam? Oh, I don't have anything. Jesus. Oh, it's fine. We got this. How did you take note? I have killed five people. Oh my god. What? I don't know. I was playing so bad. Bro, I've been playing so bad for the past two hours. I know.  Listen, man. Whoa, you're dying. Good. Oh, not good.  Where did you go? I'm in a safe house. Oh, I know what you did. Oh, Jesus. Knocked down Shadow Hedgeman. So this is legit just under buzzer. There's snipers in here. Johnny, what I got? What did you get? I got my drum gun. What? It's like a golden drum gun. Are you kidding me? Wait, let me see. Oh my god.  Jesus Christ. Ah, I'm done.  Oh my god. Can't move. That was fun. That guy had a mini gun. I found a vault. Yeah, I'm coming. I have the key card to it. Wait, really? It's got to be like a legendary chest. Huh, interesting. Whoa, what the hell? Oh my god. This is just too much. This is overpowered. Oh my god. This is too much to understand. What's going on? Knock one? Yep. Oh, nice. He's trying to rise. I have one help. With the mini gun. With the mini gun. Let's go.  Never going to let you down. Oh, nice job, camp. I classified till no one knew about it.  Oh, it sparkles as well. Well, GG, man. That was great. And it only took four hours and 23 minutes. All right, so now that we're actually done playing the game, I'm going to talk about my thoughts on this new season. This upgraded battle pass is amazing. Okay. I did not realize just how detailed it was when I was first looking through it. For example, if we go all the way over to tier 100, I didn't know that he can turn freaking gold. That is awesome. I really love this tier 100 skin. I think it's really cool. As for the rest of the tier, it's nothing too spectacular, especially the emotes. No, this guy though. Look at those this guy is definitely something. Yeah, it's a fairly decent battle pass. I mean, it's just kind of average. I do really like the skins. This new challenge system is a bit strange, but I feel like I'm going to get used to it as the season goes on. The whole cinematic as well. That was amazing. Also, Deadpool's here. Yeah, customization. I love that fact. I feel like this was something that took a long time to get into the game. And it was something that they probably should have added a lot sooner. But the fact that it's here is just freaking amazing. Coming soon, I'm guessing this is an event. This new challenge system, I like it. I mean, it's kind of just an upgraded version of the original system, but it's nice. I also liked how organized this is the agents as well. Phenomenal. I love the idea of what are they called like ghost and shadow. I love that idea. That's great. This main menu screen is kind of nothing too special. The only thing I don't like is the map. I feel like, yeah, there are changes and they are pretty amazing changes, but I feel like there could have been much more. With chapter one, when there was a map change, it changed a whole portion of the map. But with this season, it changed just a very small thing in five places around the map. So nothing too big. The gameplay is mostly the same. Yeah, the map changes. They could have been better, but I do still appreciate them. Bots are kind of weird, but I like them in a way. I like the bunker under Pleasant Park that Cam and I found. That was crazy to find. Just being swarmed with loads of bots. That was actually really fun. I mean, overall, I think this is a major plus from last season. I think this is a change that was definitely needed. Fortnite's a good game and it's done a lot over the past like, what, two and a half years? And they're still chugging along. I feel like this was a great update to get people back into playing the game. I do really enjoy playing Fortnite. I think it's a great game and the fact that Cam was actually online and we got to play loads together on Fortnite today, that was really fun. I would love to play more, but honestly, I think this video is probably enough. I think this video has probably angered too many people because I'm a Flap Channel and I shouldn't be playing Fortnite, but you know, you can like more than one thing. Anyways, I think this was a great season.  Yeah, this was really fun. So thanks for watching and I'll see you on the flip side. Goodbye.


I'm Tim Randles, and I'm with Los Alamos National Lab. I'm going to talk today a little bit about some neutron slurm integration work that we're doing. So we'll cover quickly a little bit of the past work. It lays a little groundwork. Talk about the cluster requirements for the plugin that we've written, the new plugin, select OpenStack, and then some future work ideas. So this motivation and background slide, I stole from a talk I gave in Austin in the last summit. And I think I put it up here purposely because I think it should just sort of be a priori knowledge now when someone gives a talk like this, that these requirements, I've heard four or five times at different talks so far at the summit, that users want more flexibility from their HPC resources. We tend to build HPC resources and finally tune them for MPI jobs. But increasingly we have users that we're not increasingly. It's been a longstanding problem. That solution, you hope, meets 70, 80, whatever metric you use, percent of your user population. But users increasingly, and I do think this is true, have more complex dependencies in their software than ever before. They have build time requirements, build systems such as Ant and Maven. The Python runtime wants to run off. A lot of these things expect to have unfettered internet access. We don't typically give that from our compute nodes inside of a cluster. We've always had validated and vendor-supplied software stacks. That's not a new problem. The problem comes when you have to build a Venn diagram of requirements of what OS you can run on your cluster so that you meet all of the vendor requirements or you decide which vendors you just won't go to for support when something doesn't work. And then there's legacy code with legacy requirements. This, in particular, at Los Alamos is important for us. We've been doing heavy computational work for 70 years now. And while we're not going to try to recreate the Cray-1 environment or Maniac or something like that, we do have codes that are constantly being updated for new architectures that we need to validate the updated code results against the old code. And the old code in this case is probably 15 or 20 years old. So if we can run the environment we ran 15 or 20 years ago, it's largely x86, so it'll be a Pentium class processor we need to provide to something like a VM. Code teams can go back and rerun that in a virtual environment with the old input dex and the new input dex and see if the new models match. Increasingly, there are data-intensive frameworks being developed, Hadoop with the Yarn resource manager and Hadoop 2, Spark, which can be executed through Apache Mesos or Yarn or standalone. But you notice Slurm, Torq, PBS. These aren't resource managers that these frameworks can interface with easily today. So we need some way to let users run data-intensive frameworks at the kind of scale we have in our HPC centers. Now they rely on closet clusters. They've got a rack in a closet somewhere. They've got a bunch of workstations that should have been retired five or six years ago. Loosely cluster together so that they can run these things, but they can't run them at a scale that really means anything for them. And users also want an environment they understand and control. The OS we run is Red Hat based on Red Hat 6, so it's still supported. But users are running things much newer than that. Usually it's a bunch of 16 or something. They want Python 3. These things exist for Red Hat 6. You have to jump through some hoops to get them, or you're building them by yourself and supporting them. But these reasons are common. We see them all the time. So I think what the discussion should shift to is we understand why people want flexibility, but how are we going to meet flexibility in our environments? And where does the line of demarcation between a traditional HPC resource that runs HPC workloads, MPI workloads and such, and the newer runtime frameworks, can they coexist on the same hardware and neither one of them is the lesser for it? And I think the answer is yes. I'd be really interested in having conversations with everybody who's interested in this about how they think they want to do it. I think some of the prejudices that exist on building flexibility into an HPC system, such as IO is bad and VM. Well, that may be not true anymore. We need to break down some of these barriers and start talking about tighter integration between these newer technologies and our old way of doing things. So the past effort to provide some of these frameworks, GLOME was an HPC style machine. It was repurposed HPC hardware, but we just built it as a Hadoop 2 cluster. We built it the way the Hadoop folks told you to build it and we ran Hadoop on it. We ran Spark on it. It was right after Hadoop 2 was released in late 2013. So the Yarn support for Spark was not very good, but users play with it a little bit, didn't really like it. We're an HPC center. Users aren't used to writing anything in Java. So they felt crippled, the barrier to learning Java to be productive with Java was high enough that it never really did more than write some hello-worlds and kick the tires and wander away and say, this isn't for me. To address different Linux distributions, we have built another machine that was a mirror of GLOME actually called Charli Cloud or called Kugel, but on it we ran Charli Cloud VMs. And Charli Cloud is an internally developed orchestration engine. You might be looking at me saying, which you're at the OpenStack conference, why don't you just install Cloud and Nova and let it fire off your VMs? What we were trying to do was come up with a lightweight framework that was very simple to understand that would allow the cluster to run traditional MPI jobs on the cluster or Charli Cloud VMs and no one's interfering with each other.  There was no Neutron service running. There was no Nova service running. We didn't want to introduce a lot of system noise and jitter for the MPI jobs that would come along later. Wasn't very successful. For a user who says just give me Ubuntu and I tell them, great, well you can build a virtual machine, you can install these scripts inside your virtual machine and you can copy it into our infrastructure and then you change your job submission script to do all this Charli Cloud stuff and then you're, but trust us, just your VM's gonna run. There wasn't much uptake. We had a few brave users but it wasn't widely used. So our current direction is something we're thinking of as a converged data compute machine. It's Woodchuck, we've had it around for about a year now. And there we're developing something called Charli Cloud containers. So once again, why are you using Charli Cloud for containers? We wanted a container implementation that was once again small, simple and was unprivileged. I'm never going to give a user access to the Docker group so he can have root on my systems. So we wanted something that was small, easily auditable, started up a username space and a mount name space so the user could bind file systems into it that they had permissions to bind into it and that was it. So a user could run this, he could do an M-Alic or sorry, S-Alic, get a compute node, SSH into it, run the Charli Cloud tool and then he would land in his container environment. It's small, I have a link for a research paper we wrote on a technical review. If anybody's interested, just talk to me afterwards. I'm forgetting something, okay. So for Charli Cloud VMs we had some complex network configurations that were required to secure this thing. This is the low hanging fruit of security vulnerabilities. A user requests two compute nodes and their VMs start up on those compute nodes and the compute nodes are mounting the parallel file system. In our case it's Luster or maybe at that time it was Panassas but either way, it's an IP based access model. We don't, the large parallel file systems don't authenticate generally, they wanna be fast. So the exploit here, the obvious one is user takes the IP off compute node two, he owns the VM, he's got root, puts the IP in there, he crashes node two, he mounts your parallel file system, he owns all your data, this is really bad. We can't have this in any environment let alone where we're at.  So we get Slurm to actually use epilogue and prologue scripts to set up a firewall. So now we're gonna block packets and the firewall is pretty simple at this point because if the IP address of the VM, if the source address of the packet isn't what the VM should be, we're just gonna block it. But a single node job isn't very useful for most people. So then we had to get fancier with our firewall configuration and allow the good packets to go off to the right places and the bad packets get stopped and all the while we're introducing complexity, we're introducing failure modes. If a job crashes, that firewall disappears, the job might come back, the node might come back, the job's still running, we had all sorts of problems in this and really it was the fragility scared us. So using Charlie Cloud, user requests a Charlie Cloud container that lands on the compute node. We have a separate physical network to mount the file system over, the parallel file system. We dedicated an Ethernet interface for the Charlie Cloud container and we use Slurm to join the Charlie Cloud or to create a new network namespace and give it to the container. Then we bind mount in the parallel file system. The user has what they need, a lot cleaner. We don't have taps, we don't have bridges, we don't have the overhead that that introduces, we don't have complex firewall setups but they have unfettered access to everything right now over this dedicated Ethernet. So what we're looking for is enhanced security and resource availability in this mode of operation. We want network isolation for the user jobs. We want Charlie Cloud to have hardware level access to the network for the job without putting a bunch of barriers in its place. So we want to isolate those jobs so that they can't do evil things to each other. So we want to use VLANs instead of firewalls and bridges. The hardware supports this, our switch hardware supports it, let's take advantage of it. We're gonna get better performance all around. Even if we're gonna run a VM, we're gonna run an old Charlie Cloud VM, in this mode of operation we can give the VM access to the hardware via PCI pass through or SRIOV, the network itself is going to isolate this job and only allow it to have access to what it should have access to. And then this should open up the possibility of dynamic access to network attached resources. So file systems, databases, on-demand networks, a common use case that we encounter is a user has a database and he's running it on his closet cluster because the program manager says that database needs to be protected for some reason. Currently, when they want to scale up, they have to buy more hardware. When the hardware needs to be refreshed, they have to have a fight with the program manager to get some number of zeros of dollars to go buy more hardware. If we had the ability to cite their database on a network that our clusters have access to, or we do today, the only granularity we have is this cluster can see that database. Not that compute node can see the database. So we tell the users, well, of course it's a multi-user cluster, we're not gonna build a cluster for just you guys, but be aware that anybody on the cluster can hammer away your database and try to get into it. We wanna build a system that allows this to happen. So in this view, we have Charlie Cloud with its dedicated interface, and then let's say it's connected to 10 gigabit ethernet, in this case it actually is. I think if we had a little more money, we'd have gone with 40. Regardless, it's basically IP over IB, but with VLANs because IB partitioning isn't dynamic and there's only 16 of them that I can't run many jobs. So there's the database that lives on the network and the colors denote VLANs, so your Charlie Cloud jobs there. Slurm talks to Neutron and says compute jobs running, it's allowed to have access to the database, create my VLAN, yay, my job can see it. When the job's done, Neutron goes off to the SDN controller and says, take this node out of the VLAN, we don't need that access anymore, database disappears from the fabric. The user has multiple nodes or there are multiple users. We don't want them to talk to each other, so we dynamically create a VLAN and we put the user's jobs in the VLAN so he can't bludgeon someone else to death. Once again, we have an ethernet fabric here, so congestion gets to be a real problem. Users have inadvertently denial of service to each other by doing bad network things. So to make this all work, Neutron has to control the network that you're allowing the containers to be provisioned upon. So in our case, we're doing something simple. It's a provider network and it's just VLANs. We don't want open V switch, once again, we don't want to introduce this layer of host complexity that introduces performance penalties, complexity of configuration. I'm the one that's on call for this. The other 12 people on the team think I'm crazy. They don't want to get an on-call ticket saying, packets aren't flowing and it has to, you know, it's because OVS is best up in some way and they have no idea how to debug it. Topology or where switches? The way this works is the switch has to know which nodes on which port. So you run LLDP, it talks back to the switch. Topology gets communicated through the switches back to the SDN controller. You need an SDN controller. I'm using a Rista hardware and the cloud vision stuff's just built in, works well. I run, Rista will tell you, run a virtual machine of this. So we have the EOS running that acts as our SDN controller to talk to our physical fabric. Other vendors have similar solutions. We need the Neutron ML2 plugin to work. Vendor might have a solution. Vendor probably has an ML2 plugin available for you. And we need Nova with a fake driver. Get to that in a sec. So what we've done with this Neutron integration with Slurm is we've replaced the complexity of the Charlie Cloud VM configuration in Scripps. We don't rely on epilogue and prologue anymore. We did run into cases where IP tables would hang. So epilogue would hang, but SlurmD had already told the controller D that the job was finished and that the node was available and then nodes start landing on this misconfigured node or job start landing on this misconfigured node and dying. Users don't like it when you drain the queue by killing their jobs. This required modifications to an existing plugin. I wrote job submit open stack glance back in the spring and that was for the image management portion of this. It's now been renamed job submit open stack because we're validating not only image requests but network requests that the users can make. So if the user wants that database on his network, he's gonna have to request it somehow. We're gonna validate that in the job submit plugin and then it required a new plugin, select open stack and this is what actually interfaces with Neutron to create networks, create subnet, does the nova boot part, deletes everything when the job terminates, take the cluster back to unknown state. So how does a user actually use this? They just put an S batch option in their job submit script. In the spring it was Gres, the generic resource request UDSS which is user defined software stack and that's what we use internally to call it and then some name of an image that lives in the glance registry. It's not there, we're not gonna accept your job and you can fix it.  Now we've added network ability. So a user can ask for a network, has to match the network name that's in Neutron. Pretty straightforward, handles both cases. The job submit plugin workflow, this is also from the spring. It's kind of obvious. It looks for the requested resource, did you ask for anything? If you did, let's get data about it. Does it exist? No, then resubmit your job. Is a user allowed to grab an existing image as a public? Are they in a member tenant? It's allowed for the image, yes, no. We go ahead and pass on. What we did to expand this to handle the network request was replace image with net, replace glance with Neutron. It's pretty straightforward, but it's the same workflow. The select open stack plugin. So I talked to one of our scheduler experts and he said, why would you wanna overload a select plugin to do this? And I pointed to him at the documentation on the SCED MD site. This is the mechanism for providing, or providing a mechanism for performing any system-specific tasks for job launch and termination. He said, well, select's supposed to be very fast. It's supposed to just come back. You're supposed to get the allocation and the job should start executing. And then I said, well, Cray's already done a lot of this work for me because Cray is using this same mechanism to handle a lot of the set up and clean up of their blades in an XC architecture. And on an XC from Cray, they don't tend to run Slurm through the whole thing. They run Alps. And Alps is their own resource manager. And there's a translation layer built into their plugin that actually takes Slurm, like what nodes am I allocated and things like this, and translates it into Alps and then pushes it into the Alps resource manager. So Cray's done a lot of work for me. I use their wrapper plugin, the other select, and then I use their select Cray plugin as just a blueprint for my own. So in the select API, there's select P, select job begin, and there's select P, select job fini. Begin handles creation of the network, the subnet does a nova boot, does not do revalidation. And then job termination, we tear it all down. But is this deceptively simple? Well, yes and no. The neutron net create, and I put this in terms of the command line tools you may be familiar with. I'm not actually reaching outside of Slurm to run the CLI. This is written in C. So we're building JSON payloads and parsing JSON responses, but we're calling the RESTful API directly using curl. So net create is straightforward. The very nice thing about this is when we first started this, we thought we would just directly control the switches from Slurm, and then we started down the road of, but how do I know which VLAN belongs to which job? How do I know if it's an existing VLAN? How do I get the information? We were quickly realizing we're gonna build our own database that described the network on the cluster. The neutrons already got that. In fact, neutron, I just, you configure it properly to do provider networks in VLANs. You give it a resource pool of VLANs, you say you can use VLAN 100 through 4,000, and it just takes care of it. It knows if it's in use, it takes it back, it talks to your SDN controller to know, so that solved a huge problem for us. Subnet create is necessary for instances to use neutron networks. So if you create a network and you tell Nova to boot and use that network, it's gonna come right back and say, there's no subnet on that, I can't use it. So this is a step that is required, but does nothing, basically. It just meets a condition that Nova needs to go ahead and create the fake instance. Nova boot, you specify the networks, you know how that works, but you need to specify which nodes, and then you're gonna boot what? Because I'm not actually booting anything. And then Nova delete and neutron delete, does this leave anything behind? Nova delete, I'm pretty sure it doesn't. Neutron delete, I'm not doing a subnet delete because I can't find a subnet after I delete a network that the subnet belongs to. Anybody knows if it's filling up orphan rows in a table somewhere of subnets, let me know, because I haven't seen them, but I haven't looked real hard either. This part was a problem for us, kind of. So boot what? We're using the Nova fake driver, we don't care. It doesn't matter. You just need to tell neutron that you wanna boot something that actually exists in the glance registry and that there's a flavor that actually exists so it can resolve these things. So it doesn't think that you're asking for something it can't provide. And then which nodes? In this case, we're using availability zones. You can specify an availability zone. We're giving it the cluster name because we envision this being at the center of our backend infrastructure so that we may have multiple network or multiple clusters that we're configuring from here. And then you specify the networks you need and then you give it some name. At LANL, we use two-letter abbreviations for all of our clusters. So Woodchuck, a compute node is WC001 is the first compute node. We have an availability zone mapped out with all the compute nodes in it. The network name, if you get in and start poking, if you have Horizon running, you start looking at your networks. That's the job ID from the Slurm job. So it's Woodchuck Job1234 has this network. MyDB, that comes in from the user request if you want something. And then what did I boot? Nova needs a name for the host and it needs to be deterministic. So it's the cluster of the job ID and then the host name. So other minor details here. How are we gonna get log in node access to the containers? Because we've said that the containers are on their own physical interface, on their own fabric, and we vlan them. So we made it deliberately hard to get to them and now we need to make it easy to get to them. The first use cases, we'll just stick an access port. We'll put an interface on our log in node. We'll plug it into the fabric. We'll put it in trunking mode. It can see all the VLANs. User can just go ahead and SSH into their container if it's running SSHD. But then anyone, of course, can try to access your container. The argument right now is, well, anybody can try to access your compute node. What's the difference? The difference is that maybe the user created the container or the virtual machine. They didn't secure it. They didn't, they left a default password somewhere. They didn't put a password in it. So I think what we're gonna end up doing in the short term is using something like cloud init to insert a user's key, SSH key into the container so that only he can get into it and not allow password authentication. And if we find containers that have password authentication enabled, we're gonna tell them they can't run them in the short term. A longer term idea that we have, and it's one that I'd love to discuss if anybody's done anything this style. Our user's login to a gateway to reach our front end network or our login nodes. We would like to come up with a system where they hit the login or the gateway and then instead of SSHing to the login node, make a request. And maybe this is open stack running as just a cloud manager. What they end up with is a VM on the physical node attached to that cluster. VX land with a gateway into the VLAN. Well, it won't be into the VLAN initially because they haven't submitted a job yet, but set up an arrangement where users are sandboxing VMs on our physical hardware and then the VTAPs are set up when their jobs are submitted so they can get to the nodes that are in their job allocation. It should provide stronger, it's obviously stronger isolation between users than we have now. We get a lot of support requests of long running process on front end node, whatever, because someone's testing their code and he went to lunch and it's just running a simulation single threaded on the node and is going to do this. When it happens on Friday afternoon, the Monday ticket queue looks really ugly because no one else can do anything on that compute node because he's got it CPU bound. So we have other reasons why we want to come up with this regime, but it's definitely non-trivial to just give users VMs as their own front end nodes. And then how about dynamic network access beyond the cluster? So one of the use cases earlier in here was what if a user wants unfettered internet access? Do we allow Neutron, which is really being controlled by Slurm, which means a user's telling it what to do. Do we really want that controlling our infrastructure outside of our cluster?  So do we do static VLAN assignments outside of the cluster? Maybe it goes into the third point with QOS, open flow, traffic shaping, all these other things that SDN give you that we're not doing yet. We have Juniper firewalls sitting around. Do we want to orchestrate our infrastructure so that we can allow the users to reconfigure a larger piece of the network for on-demand stuff? But I think this is definitely the case. This is way out of scope for a Slurm plugin. But for this plugin or for this model to extend very far, we're gonna have to come up with something that does those things. What that is, I'm not sure. Questions? Reads the author of Charlie Cloud. Chuck's my networking guy and Steve Senator is our Slurm expert. Thanks Tim. How do you do deterministic IP address assignment? So in the case of containers, we're hoping no one really asked for Charlie Cloud VMs in this regime because we told them they're deprecated. In the case of containers, the users getting access to the bare metal hardware it's already configured, the network namespace should carry that in for us. We're not reassigning the IP when the container is attached to the device. No, the nicks are statically assigned. We had the same thing with Charlie Cloud VMs and that there was one of the reasons why we could do the firewalling we did which was we had a subnet setup so that your Charlie Cloud job with a VM had access to this device to this bridge but the external interface was statically assigned. We had an immediate scaling issue in that the way the scripts were written in the first place, you couldn't go beyond 255 addresses because we ran out. Luckily we had a 96 node cluster and that wasn't a problem. Any other questions? I do plan to start talking to SCEDMD and try to upstream some of this. I don't know that it's generally useful, it's just the direction we're taking right now. You know, one of our, like I said earlier, I think the prejudice is against HPC's view of non-HPC workloads and mechanisms for running those on clusters are ill-held now. They need to fall away. So we're just starting down the road of this. You know, we have image management, we have a little bit of network management now. It all needs to grow but I think we're trying to to prove the case that with the existing technologies with things like OpenStack components, not running a cloud but using the components as bolt-ons to our clusters, we can give flexibility that's just as performant as if we had a bare metal cluster built for a single purpose.


Hello, everybody. My name is Jorge Torres. And MyCB brings the entire ML ecosystem to databases. And let me walk you into that. Also, the problem that we're trying to solve is precisely what they were describing before. It's the idea of being data-centric as opposed to machinery model-centric, which is probably the way that most people do it today. And that has a lot of problems. The first problem that you have is that it carries a lot of the intuitions that we bring in from research. But when we want to apply it, it's not necessarily the case. So when you're doing modeling and you forget where the data is coming from, you start dumping a lot of files into your computer and make a lot of transformations in Python, et cetera, how you want to transform. And eventually, you have a model that works. But it has a lot of issues. The first issue is that now you're going to have to turn all of that spaghetti code that you built to transform data for your model into some ETL-ing tool. And now you have to maintain both. And if you deploy your model, usually people deploy these models behind like a RESTful API or some other web service. And every time you want to make a prediction, you need to extract data from the database, run the ETL-ing pipeline, run the model, and many times have to put the predictions back into a database of the data warehouse. So there's a lot of unnecessary ETL-ing happening. And what Miceby tries to do is to simplify the problem. Essentially, if your data is in the database, you can train the model directly from the database. And actually, you can consume the model as if it was a table. I think that is the second innovation that Miceby brought to the table. And why Miceby today is the fastest growing applied ML project in the world. And the concept is very simple. Imagine that you have a table, this simple table. Of course, tables in production don't look like this. But for the example, you have income and debt. And databases are meant to give you a response if what you query has a direct row or an exact match into a database. So say, for instance, if you want to ask, what will be the debt for when income is $80,000? And if that exists on the table, then you will get a response. But they're not designed to give you an approximation. Now, anyone that has done machine learning knows that you can build a hyperplane that can fit that data, a machinery model. And what Miceby allows you to do is to create that model in a syntax that is very similar to that of creating a table or a view. So you can say, create model, the name that you want to give the model. And then from what table you want to learn and what column you want to learn. Now, you can specify what machine learning network or what machine learning capability you want to bring into the mix. But if you don't specify anything, then Miceby will figure this out. Now, the cool thing is that now you can query that model as if it was a table. So you can say, OK, tell me what will be the income and debt and the predicted debt for when the debt model is, I don't know, $90,120. And then you get that approximation. Now, again, most problems in machine learning are not as simple as this. The concept is a concept that extrapolates all the way to very sophisticated stuff. So one example of this is NLP. You have a growing ecosystem of machine learning frameworks like Huggingface. And with Miceby, you can bring the power of Huggingface down to the data layer. You can query models that are meant to be doing NLP and outside of the database in the database itself. For example, imagine that you have thousands of reviews of products, and you want to get sentiment about what people think of your products. Now, Miceby does this in a very simple way. You can say, I want to create a model, however you want to name the model, and you want to predict sentiment, and then you use the engine Huggingface. And the reason why you have to specify the engine is that now Miceby supports more than 70 engines of machine learning and more than 70 data frameworks. And then you specify, OK, what will be the actual model that you bring from Huggingface, in this case is a Roberta classifier. And then what is the column that you want to predict from, and what are the tags or labels that you want to do? Now, one single SQL statement brings the power of Huggingface to any database. And again, that any database by that we mean Miceql, Postgres, Oracle. Now, once you have this model as a table, it becomes very easy for you to query that model like if it was a table. So you say, OK, let me ask the sentiment classifier, what will be the sentiment when I have a sentence like this one? And of course, for the sake of this example, that sentence is very stupid, but you get the concept here. It's like neutral. Now, the second thing is you can play it the same way, like what will be if the sentence changes to something positive. But the interesting thing becomes when you actually want to do bulk operations. Say, for instance, you want to get the prediction for the sentiment for a whole bunch of rows. Again, imagine all the products that you have. Let's say that you have this table. You can join the table with the predictor, and then you get a prediction of the sentiment for each one of those. Now, again, the concept that I want to bring to you is that Miceql is bringing this capability to all of the developers. At the end, only 1% of the developer population knows how to really implement effectively research type of machine learning. And what we do is we bring the state of the art of machine learning to the people that actually know how to build products. And well, I invite you guys to go check it out. Miceql.com. Thank you.


How can I add teachers and learners to my courses? When a new account is created in Moodle, the account holder is called an authenticated user. They don't have a specific role on the site. It's the administrator's job to make them a learner, or student, or a teacher in the courses where they will learn or teach. This video explores how you can add users to courses one by one and give them the roles they need for that course, such as teacher or student. As an admin, you can add several users in bulk with the text file made from a spreadsheet program. To see how to do this, watch our video on how can I add users to my Moodle, bulk uploading, and read the documentation, upload users. To add users to a course, access the course, click the participants link on the left, and then click Enroll users. As an admin, you can choose to enroll them as teachers, students, or other roles. When a regular course teacher clicks this button, they cannot enroll other teachers. If you need that, read the documentation, assign roles, for information on how you can allow teachers to enroll other teachers. Clicking show more offers the option to specify the start date and length of enrollment. When you save, the newly enrolled users will appear in the participants list with their roles. You can make changes by clicking the edit icon and remove them if needed, by clicking the unenroll icon. To summarise, admins can enroll teachers and learners in bulk into courses with a text file, admins can enroll teachers and learners into individual courses from participants, enroll users, and course teachers can enroll learners, that is students, into their own courses from participants, enroll users.


Hello people! I'm Ginny Metherill and I'm a fourth generation witch. Today's video is a little different because I haven't been here and so I thought I'd bring you on my trip to Oxford which I made with my coven. Sadly I can't show you what the coven looked like because everybody else is a little camera shy and don't want to be seen by my subscribers so sorry about that. However I can take you along for the rest of it and so this is what happened when we went to that amazing and beautiful city of Oxford. I had to drive which was embarrassing because I'm not very good at driving but we arrived in one piece. We stayed in this beautiful 16th century coach and horse's pub. Recommended I loved it. This was my bedroom which I thought was rather glamorous for a just a pub bedroom and as it's very late I went straight to bed. So welcome to my hotel room. It's rather glorious here. I'm in Oxfordshire. I'm about to go into the town because there's a couple of things that I want to do there but first I need to protect my room and I always do this with a few crystals well placed around the outside edges. I'm using quartz and hematite because these I find work well with my energy and also provide great protection. I put adventuring on my desk to help me with the work that I have to do whilst I'm here because I get easily distracted and it worked as I got loads done. I was really pleased. After work I went down and had some breakfast and this was it. It was delicious so thank you very much to the chef. Feeling quite full we set off a driving through the very pretty country villages on our way into the beautiful city of Oxford. Should we get the opportunity to visit Oxford? Do go. It is marvellous. I mean I haven't been there for 20 or 30 years and it is just as stunning as I remembered it. Sadly all the colleges were shut so we just had to stand in their doorways and look at them from the outside but it was still worth it. I did have to show you this beautiful shop which I want my kitchen to look like. It never will obviously however this cauldron really caught my attention and I want it. Maybe next month my cover member did buy a beautiful set of glasses and I'm quite jealous. To make up for it we went and had a drink at a pub on the river and then joined up with the rest of our cover but as I said that is not for your eyes. However we did do a bit of ghost busting at St Catherine's Chapel in Chisell Hampton so I shall show you this. I love a chapel and a graveyard. They are such fun to look around. The chapel has a wonderful old yutri next to it. Yutries were part of our pagan ancestry and were there to protect and help the dead move on. They were sort of bottom up rather than top down meaning that when they built the church the people went oh we better put a yutri in because you know we've always had yutries as opposed to the christian faith trying to incorporate this into their religion. We did move on a couple of spirits whilst here so that's one of the reasons why I love going to a graveyard to help those in need. So having caught up with my cover and completed what we came here for it's time to go. Don't forget the crystals though are always forgotten. I tend to leave them wherever I go but this time I've remembered. These beautiful roses were given to me so I'm going to take them with me and with that it's goodbye to the coach and horses and thank you for a lovely stay. If you're missing the Ginny Metheral witchiness you can always go and have a look at my brand new blog the only alternate witchcraft blog ginnymetheral.co.uk check it out. Don't forget my coven meeting is coming up and we will be doing a circle of healing and I'll show everyone how to use circle of healing to help everybody's complaints. Go to patreon.com forward slash ginnymetheral for further details on that and in the meantime I would love to hear your comments on my trip to Oxford. I'm sorry I didn't get very much in there but I was a bit busy. Please don't forget to like and subscribe as well because it really helps my channel and enables me to keep making these videos for you and allows me to take a couple of days off every now and then and go to Oxford. I loved Oxford and I will see you in a couple of days.


تفوته آدك الأيام فتا حسين ابن علي ابن بيطالب he reached Karbala and when حسين ابن علي reached Karbala he said to those who he was with he asked them he said to them ما هذيه what is this place they said to him this place is called Karbala حسين ابن علي he replied by saying Karbala it's a place that has distress and problems when the army of عمر ابن سعاد reached and the number of people they were was 4,000 they spoke to حسين ابن علي ابن ابيطالب and he requested from حسين ابن علي to go with him he said to حسين ابن علي come with me and we go to عبيد الله ابن زياد حسين ابن علي فأبا he refused he said I'm not going to go to عبيد الله ابن زياد but when حسين saw that the matter was very serious أن الأمر جد that the matter is serious he spoke to عمر ابن سعاد he said to him إني أخيرك I'm going to give you a choice an option بين سلاثة أمور in three matters فختن منها ما شئ تتشوز whichever one of those you want he said to him موما هيا what are the three that you're the options that you're giving me he said to him أنت دعني أرجع that you let me go and go back to where I came back from أذهب إلى سغر من صغور المسلمين or I go to from the borders of the Muslim lands where they patrol أو أذهب إلى يزيد or I go to يزيد ابن معاوية حتى أضع يدي في يديه بالشام and I place my hand in his hand in Sham and this shows you that حسين ابن عالي ابن عبيطالب رضي الله تعالى عنهم it seems that it seems that he had in his heart some respect for يزيد ابن معاوية and that's why he wanted to go to يزيد ابن معاوية to give him بيع but he didn't want to نتعميد الله من زياد he didn't want to see him عمر ابن سعاد then said okay yes no problem أرسل أنت إلى يزيد you send a message to يزيد وأرسل وأرسل أنا إلى عميد الله من زياد and I will send a message to عميد الله من زياد وننظر أن we will look at the matter ماذا يكون في الأمر what the matter is going to be but the real truth is فلم يرسل الحسين إلى يزيد حسين did not send anything to يزيد من المعاوية وأرسل عمر ابن سعاد and عمر ابن سعاد he sent a messenger إلى عميد الله من زياد عميد الله من زياد when the message it was brought to عميد الله من زياد and he was told that حسين ابن عالي has chosen one of these three options he wants these three options one of those to be fulfilled for him عميد الله من زياد said I'm pleased with any of them whichever one he wants حسين ابن عالي لهم to choose but at that time when the message reached عميد الله من زياد and he was allowing حسين ابن عالي to take whichever of the three there was a person next to him and his name was شبر ابن ذي الجوشن he was with عميد الله من زياد at that time and he was from the من المقربين من ابن زياد he was one of those people who was very close to عميد الله من زياد those people who عميد الله من زياد used to consult and he was an evil individual he said to him no والله he said to عميد الله من زياد no والله no حتى ينزل على حكمك حتى ينزل under your leadership how is how dare that عميد الله that حسين gives us the option of one of three never until he comes under your leadership when that was said فاعتر عميد الله من زياد عميد الله من زياد became conceit full of himself arrogance and then he said نعم yes حتى ينزل على حكمك until حسين ابن عالي سأعطيه لا يوجد أخر فرصة. عما يدول الله إبن وزياد ماذا يفعله هو أنه قلت شمر ابن الآل جوشن لنذهب ، لنذهب to حسين عالي ، قال لنذهب لهم وخبرهم حتى ينزل على حكم ذا until he comes to my leadership قلت لهم ، بعض الأشياء they mentioned that عما يدول الله ابن وزياد قلت شمر ابن وزياد to go to عمار ابن وزياد first to tell عمار ابن وزياد that حسين has to come under the leadership of عما يدول الله ابن وزياد if عمر disagrees that you take over the matter and go to حسين عالي and deal with it and forget عمار ابن وزياد because عما يدول الله ابن وزياد he prepared عمار ابن وزياد with 4000 men and these 4000 he wanted عمار ابن وزياد to go to ري and for him to be the leader of ري so the 4000 army that عمار ابن وزياد has he was meant to go to ري but عما يدول الله ابن وزياد said to him go to ري go to حسين عالي first deal with this matter and then after that you can go to ري because he promised him the will of ري so شمر ابن وزياد he went he conveyed the message to حسين عالي and he told him that you have to come under the leadership of عما يدول الله ابن وزياد حسين عالي refused he said his famous statement لا والله ابن وزياد in the name of Allah لا أنزل على حكم عما يدول الله ابن وزياد I will never fall under the leadership of عما يدول الله ابن وزياد when that happened حسين عالي only has 72 horsemen and the army of كوفا عما يدول الله ابن وزياد are in total all together they are 5000 so when the two armies stood in front of each other حسين عالي took one last opportunity he started to speak to the people to the army of عما يدول الله ابن وزياد the army that عما يدول الله ابن وزياد sent he said to them he said to them راجع وانفصكم revise yourselves revise the matter look into it again وحسبوها account yourselves is it befitting هل يصلح لكم قتالة مثلي is it befitting قتالوا مثلي to kill somebody like me وانبن وبنتي نبيكم and I am the son of the daughter of the Prophet ﷺ وليس على وجه الأرضي and there is no one in the face of this earth a son a son ابن وبنتي نبي غيري there is no son who is the son of the daughter of the Prophet ﷺ on the face of this earth and the Messenger ﷺ said in my affairs حسين said have any of these two حسن الحسين سيدة شبابي أهل الجنة that they are both the masters of the youngsters in Jannah he spoke to them he told them to leave the affairs of عما يدول الله ابن وزياد and he asked them to come with him but the people who came to فضنمل الحسين منهم ثلاثون أو نثاثين from them was الحر ابن يزيد التميمي who was before that who was the leader of the army of عما يدول الله ابن وزياد it was said to حر ابن يزيد التميمي it was said to him أنت جئت معنا أمير المقدمة والآن تذهب إلى الحسين you came out with us the army of عما يدول الله ابن وزياد and you were the front men you were the leader, you were the captain, you were the head of the army and now you go to the army of حسين الملعلي حر ابن يزيد التميمي replied by saying ويحكم دستخشبي to you guys والله بي الله إني أخيروا نفسي بين الجنة والنار by Allah I am going to give my choice between جنة والنار am I going to be choosing between جنة والنار والله بي الله لا أختاروا على الجنة والله I am not going to choose over جنة ولو قطعتو وأحرقتو even if I was cut into pieces and I was bent I will never choose nor over جنة and after that حسين ابن علي he prayed دهر with أسر and it was on a Thursday rather حسين ابن علي he said to all of them مينكم إمامون for you guys there's going to be an Imam ومننا إمامون and for us it's going to be an Imam they said no there's not going to be an Imam from you guys and an Imam from us not at all بالنصلي خلفك راذا we're going to pray behind you they all prayed behind حسين دهر and أسر فلما قارب وقت المغرب when the time of مغرب came close the army of عميد الله ابن وزياد they came forward towards حسين ابن علي and at that time حسين ابن علي ابن علي طالب he's all sword and everything wasn't out in the open when he saw their horses coming closer حسين ابن علي he was sleeping he said ما هذا what is this they said to him إنهم تقدموا they came forward he then said go and check and speak to them and ask them وقولوا لهم say to them ماذا يريدون what do they want 20 men 20 men with horses they went and the person who was leading them was عباس ابن علي ابن علي طالب حسين's brother حسين's brother فكلمهم فكلموهم sorry he spoke to them وسألوهم he asked them why are you guys coming forward they said إما أن يلزل على حكم عميد الله ابن وزياد إذا حسين is going to come under the leadership of عميد الله ابن وزياد وإما أن يقاتل or he has to fight with us عباس ابن علي ابن علي طالب said حتى نخبر أبا عبد الله give us a chance a time so we can go to حسين ابن علي and we can tell him فرجعوا إلى الحسين they came back to حسين وأخبروه they told him what they said فقاله he said قولوا لهم say to them أمهلون هذه الليلة let them wait for us tonight وغدا نخبرهم and tomorrow we will tell them the answer to this حتى أصلي لي ربي so I can pray to my Lord tonight فإني أحب أحب أصلي لي ربي for me to pray to my Lord I love to تبارك وتعالى فبات الليلته he slept that night that night so he spent that whole night praying for Allah asking for forgiveness supplicating to Allah he was doing that ومن معه رضي الله عنه أجمعين and those who were with him may Allah forgive them all when the year was 61 هجرية that Friday morning after they prayed that whole night the armies, the two armies they came face to face and حسين ابن علي he refused and yes that سيرا العبيد بن زياد for him to be a captive for عبيد الله بن زياد he refused and as you can all see the two sides of the army they're both not the same غير متكافئة تيني they're not equal to one another and the army that were with حسين ابن علي they all saw أنهم لا طاقة لهم بهذا الجيش they knew that they had not they didn't have enough men to fight عبيد الله بن زياد they realized they knew that but the only reason why they fought was why فصارهم مهم الوحيد to the only aspiration and the only motive for them to fight knowing that they will not stand the army of عبيد الله بن زياد was الموت بين يدي الحسين ابن علي رضي الله تعالى عنهما they just wanted to die shoulder to shoulder حسين ابن علي بن أبي طالب and they were فأصبحوا يموتون and they started to drop one after the other الواحد و تلو الآخر then the other one drops and the other one drops each individual dying حتى فنو جميعا until everyone died و لم يبقى منهم أحد أن not one of them was left إلا الحسين ابن علي رضي الله عنه except حسين ابن علي رضي الله تعالى عنهما was left and his son علي بن الحسين who was sick زياد عبيدين they were the only two who were left و بقى الحسين وبعد ذلك كان هارا طويلا and his son spent that whole day that whole day by himself walking around no one was coming forward to him no one wants to kill him لا يريد أن يبتلاب قتله رضي الله عنه all of them are aware they don't want to spill the blood of the grandson of the messenger the matter stayed for that long no one was coming forward to him he's standing there by himself with his sword until شمر ابن ذي الجوشن he screamed فصاحة he screamed at the people and he said to them واي حكم destruction beating you guys ثاكيلتكم أمها تكم may your mother lose each and every one of you احيطوه siege him go around him وقتلوه kill him so what did they do they went around him رضي الله تعالى عنه حسين and they went around them with their horses and their swords and حسين whenever they came close to him whoever came close he would kill them but as we're all aware of the statement that the Arabs say ولكن الكثرة تغلبوا الشجاع but number will always overcome bravery courage is nothing in the eyes of number when nobody wants to go forward حسين يمنع علي no one wants to kill him شمر he screamed at the people he then said to them ماذا تنتظرون what are you guys waiting for اقدموا go forward فتقدموا they went forward to حسين فقتلوه they killed him the person who physically killed حسين it said one of two people some say سنان ابن آنسن نخعي killed him and the next one is شمر قبحهم الله may god destroy them و بعد أن قتل الحسين when حسين was killed his head was cut and it was taken to عبيد الله ابن زياد in كوفة to show him that حسين was killed they cut حسين's head and they took it and they brought it to who they brought it to عبيد الله ابن زياد he took a a twig and he started to place in the nose in the ears in the mouth of حسين's face and he was praying around with his head and as he was doing that انا سمر ماليكن came by and he stood over him and he said to him والله you're the evilest person I've ever seen لقد رأيته I saw رسول الله the message of Allah يقبل موضع عقاصيبك من فيه والله I saw the message of Allah kissing the places where you put I saw the prophet kiss him on his lip on his cheeks as a young kid I saw the prophet do that and today you're playing well around with it like that ابراهيم النخع يسد as إمام الطبراني narrated in his معجم الكبير بسند صحيح a chain of narration which is authentic ابراهيم النخع يسد if I was from those people who killed حسين لو كنت في من قتل الحسين if I was from those people who killed حسين or I was from those who fought against حسين ثم ادخلت الجنة and then I was put into جنة after that استحييتوا I would be shy and أمر على النبي for me to go buy the prophet فينظر في وجه and for him to look at my face I'll be shy with him if I told حسين and I went to جنة I would be shy for the messengers to see my face صلى الله عليه وسلم who was killed with حسين ابراهيم النخع رضي الله تعالى عنه from his family the people that were killed with حسين ابراهيم النخع were his brothers who were from his father's side عليه من أبي طالب his children جعفر was killed عباس was killed ابو بكر was killed محمد was killed عثمان were killed and these were all his brothers also his own children his son عبد الله was killed his older Ali he had two sons that were Ali the older Ali was killed but the Ali زين العبدين he was a killed and the children of حسين were killed الله قاسب أنا بو بكر were killed also the children of عقيل were killed the children of عقيل they were killed عقيل who was the brother of who the brother of عليه من أبي طالب جعفر was killed عبد الله was killed عبد رحمان was killed عبد الله من مسلم was killed ابراهيم عقيل مسلم العقيل was killed because he was killed before in كوفا the children of عبد الله عبد الله جعفر were killed عون and محمد were both killed how many were killed from the ألو البيت 18 men were killed from the ألو البيت all of them were killed in one battle in a battle that they were equal towards حسيني من عالي رضي الله تعالى عنه the man who killed him عبيد الله من زياد who ordered the killing of حسيني من عالي من أبي طالب he didn't remain for very long but he didn't stay for very long and he wanted المختار من أبي عبيد he done انتقام للحسين he retaliated and took revenge for حسيني من عالي حسيني من عالي من أبي طالب because مختار من أبي عبيد he was from those people who deceived مسلم العقيل he deceived who he deceived مسلم العقيل and he also deceived حسيني من عالي من أبي طالب he didn't support him he didn't help him rather the people of كوفا the only person who stood up for حسيني من عالي from the people of كوفا was الحر من يزيدة تميمي and those 30 that came with him those 30 were the only ones as for the remaining of other أهل كوفا خذالوه they deceived him and it's sad because you see a people who they deceived حسيني من عالي they're the ones who killed حسيني من عالي مع ذلك they claimed they claim the innocence and that they've lost حسيني من عالي من أبي طالب brothers and sisters that is a summary of what took place and an overview of the story that happened in كربلا إن شاء الله تعالى in my upcoming sits I'm going to go through إن شاء الله تعالى عقيدة of the ear شيء عن رافضة and how they came into place in more details إن شاء الله تعالى


Wagner Group uses social networks to hunt new recruits. The Wagner Group, the infamous Russian private military company, has been using Twitter and Facebook to recruit medics, drone operators and even psychologists to aid in combat operations, including those in Ukraine. Political media outlet wrote about it. Job ads for Wagner, which has mercenaries operating in several countries, have reached nearly 120,000 views on the two social media platforms over the last 10 months, according to Logically, a United Kingdom research group focused on busting disinformation and fake news. Posts are published in dozens of languages, including French, Vietnamese and Spanish. They also include contact phone numbers, telegram accounts and touted monthly salaries of 240,000 pupils or 2,800 euros, with benefits including healthcare. According to Kyle Walter, head of research for Logically, some of these posts are actual propaganda films combined with phone numbers. One post in French boasted that Wagner employees enjoy a number of opportunities such as paid time off, healthcare, well-paid jobs and the opportunity to work all over the world. The Meta company, which is the owner of Facebook, the popular social networking service, assures users that they are aware of the danger of the Wagner group and it cannot have a presence on their platforms. They added that the company removes this content when they become aware of it.