And so our next speaker will be Sebastian talking about, I guess, 3D in Gino. Okay, hi. So I'm Sebastian. I'm working at Gino Labs, and I have been doing this for the last, I have almost 10 years. So in the summer, like two years ago, we got a little project founded that we were able to look into 3D acceleration with a, like, a view on GPU virtualization. And so, since we are Gino Labs, we decided to have a deeper look in it and try to get something working on our, like, microkernel component-based system. Hence the very complicated title. I maybe should have called it 3D acceleration on microkernels. So, short outline, as usual, short motivation, then I try to explain what parts do we have on a GPU that needs to be multiplexed and shared by multiple clients, how we do it in user space or in Gino, especially, and then adjust the results. So it's going to be a pretty straightforward talk. Come in, please. So, like Tim Sweeney says, during the last 20 years or something, GPUs have become more and more important and way bigger. Like, they used to be very small, very slow, but in the future it may be, it may even look like this. So this would be an Intel Skylake core with integrated Intel graphics. If you just look at it, you see that's the GPU, and that's your quad core processor. So maybe these things are kind of important, and since your browser and everything now uses this thing, like multiple applications will use it, you probably should, like, take care of it. So another thing is GPUs have become really programmable. So in the past, you would say, okay, draw these polygons, draw these vertices and stuff like this, and now you have something called shader languages. So you can have this code. The driver will compile it at runtime into GPU, like machine instruction set code, and this can produce something like this. So this was taken from shader toy. And this is, like, executed by a browser, right? This browser application. You can write your shader, compile it, and it will be there. So this would be some disassembly of the code that's generated. So it kind of looks like x86-ish, like with vector stuff and this kind of thing. So one idea we had, maybe we should treat GPUs a little in a way like we do, let's say CPUs or virtual CPUs or something like this. So, okay, can you hear me? Actually, should I try to talk a little louder? So I have a very low voice, so that's very hard. Okay, so that was the motivation. And what we did next, we just had a look into, like, a modern graphics deck on Linux. So what does it look like? So some people might be familiar with it. On top, you have your 3D application, and then you need some sort of 3D library, let's say OpenGL. So Mesa supports OpenGL. And OpenGL consists of one or more layers, depending on what you want to do. On the left side, you see the embedded GL, that's the integration into the rendering system. There you say, okay, OpenGL, use this window, use this surface, and here's your frame buffer on things like this. You say, what's the bit depth and all these kinds of things. Then you have your natural OpenGL layer, the application programs against, and a back end called the direct rendering infrastructure. So, and here you have, for example, an Intel back end and a software rendering back end. And of course, the Intel back end will eventually talk to the kernel driver and render the graphics. But the first thing we did, we had to look at this whole thing, like to evaluate the complexity of this. So we looked it up and just tried to identify the parts, which are important if I wanted to support an Intel graphics course. And you can see like the whole Mesa stuff roughly consumes 300K lines of code, including a shader compiler, so it's highly complex. And you really don't know what they do. So you don't really know how they translate shader language and what instructions on a GPU because this stuff is not really documented that well. And then of course, we will talk about this later because that's the main point. The kernel driver, which actually schedules the general code onto the GPU. So, but first, we tried to get this Mesa thing working on our keynote system just as a proof of concept. So we said, how hard could this be? And we don't want to use the Intel stuff yet. We want to use the software when we were first. So we kind of put it over to keynote and how this usually goes is, you see a lot like pictures like this. So then you just work further and you see things like that. And eventually, you see your demo, standard-gear demo in software-random. So after this, we decided, okay, let's follow the actual GPU path, which requires enabling the Intel backend for the direct rendering phase. And then you have this library called the Direct Rendering Manager. And this library directly talks to the kernel. That's the kernel interface. And sends jobs to GPU. So basically, what this library does is it tells the kernel driver, giving you some memory from the GPU, giving you some, they call it buffer objects where you can store your image data in a vertex data. And this driver fills it with compiled code with image data with vertex data, sends it to the kernel and the kernel will execute it. So the big question arises, how does the kernel GPU driver part actually multiplex stuff? Because what you want to have is, if you have multiple clients in user end, you want to kind of separate them so they cannot cross talk or anything. We looked at the kernel driver and we saw a lot of outdated code, like for older Intel graphics devices. And features, we didn't really want to support. And this is why the 100,000 lines of code is like not fair comparison because we just wanted to do the new sort of GPU generations. And we just wanted to take it once of new features. So the question arose, is this feasible and how complex is this? So next I'm going to talk about the things you actually want to multiplex and the features we found that we want to use. So that's going to be a little bit more technical but you don't really have to understand it but I kind of will give you an idea. So this is how Linux does the multiplexing. Here are two clients and Linux multiplexes through the DRM library. So you have a DRM context in the kernel for each client and the client tries to identify the context and separate the coding. So let's see here. Okay, so in the GPU you have like one or more engines, they call it engines. The most important one is the render engine. This is the engine where you send your 3D operations and your general purpose GPU processing stuff. The next important one is the glitter engine. This is meant to copy like very fast copying pixel data from one buffer to another. And then you have some nice to have things like V2 decoding support or something. Since we wanted to constant render 3D we just looked at the render engine in our scenario. Okay, so now it really gets more technical. So basically here you have your GPU there you have the render engine and the only way for the CPU to actually access the graphics memory is through a so-called apparture. And apparture is some MMIO window, so memory in your memory and the CPU has to map this memory through the apparture. And this is done, I will come to this thing called the global GTT, the global graphics translation table. So of course, like in real CPUs, GPUs have page tables. There are all ones called the purpose page tables and we will come to this next. So you can see the render engine actually can access both page tables but mapping of graphics memory only goes through the apparture. Okay, here we go. So every graphics card has this global graphics translation table and there is just one and it's shared by all GPU clients. So if you want to map something through there, like through the apparture, it has been handled by the GPU driver in kernel. So here's to make sure that there's no cross-talking or whatever. On the other hand, you can say, okay, each client can have a per-pressives page table and this one is hierarchical, like we know it from x86 CPUs and can be associated to a client. That's the important thing. So the format of whatever doesn't really matter for us now. So then what you actually really have to share is the apparture because it's limited. So if you want to like map your image through the apparture, you have to make sure each client only sees his things in the page table. There's like almost minor thing. It's called fencing because the GPU doesn't store like image data linear in the memory. It just tiles it and stores it non-linear. So you can like detail and tile it through the fancy with registers to just say something like, okay, this region is x-tiled and the GPU will do the rest for you. These things are also limited. There are only 32 registers with multiple clients who somehow have to share or by a check like doing partitions or dynamically assigning it to clients. So these are the most important parts you actually have to share. And the other thing is the rendering has some sort of command buffer. It's a ring and there you can write GPU commands too. Like let's say something, a batch buffer is like a buffer containing GPU commands and you say, execute this buffer. And in the past, all the clients shared this hardware ring. So this has changed too. So in the path you had this hardware ring you would write your commands in there and there was a head and a tail pointer and it would say, okay GPU, new head pointer and GPU would execute it until it reached the tail pointer. So now in your GPUs they have this notion of exec lists and maybe the best comparison of the exec list is something like you're familiar with VTX. They have this VMCSE area where they store the whole machine state and this is kind of comparable. So you have per client, you have this exec list and in there is a page table pointer. There's like the ring with the commands the client wants to execute. There's a hardware context when the GPU stores like if you pre-empt the context the GPU stores this whole state in there and there's a hardware status page where they can say, okay what's the state of GPU? So you can have this per client and the only thing you do now is all right let's write our commands to the logical ring set the page table and then say to GPU, okay execute it and the GPU will switch the page table copy the logical ring on the real hardware execution buffer and you're basically done. So you have to make sure that there's an interrupt or something in there because what you still can do is like okay I have a shader with an endless loop so you cannot do anything about it the only thing you can do you can have a watchdog timer says if the GPU doesn't come back because of some malicious attack you have to reset it. So that's the only thing you can do at a time. Okay, so now this is kind of what I just explained and how we did it on Gino on Gino we have this notion of sessions so you have like your GPU multiplexer in user space so not on hardware but in kernel space and you have this session interface where a client can say okay I want to have a GPU session and the multiplexer will allocate all the necessary resources for you and you can like submit your command bubbles like you would do in DRM just in user land and through a different kind of session so here you have like N clients and this is the information every client has to you know be provided so and as I said you have to slice the fences and the aperture there's a page table per client and a hardware context okay so this would be like put in this in the Mesa context we had before so we still have our 3D application we say this whole Mesa thing is for us it's way too complex and way too large so we link it into the client application and we exchange the DRM library back end so it will talk our GPU session talk so and establish a GPU session to the GPU component and you basically have a user level component with the component driver which cannot crash the kernel at all okay we start the portal again and writing the multiplex again this time you could actually could have seen something and the gears were there for like a second then they were gone turned out we didn't do the synchronization well so the rendering wasn't done and we already painted the picture so most of the time you would get like portions of it or nothing this is what tiling looks like so if you don't detail anything you always get this nice lines there so then you know you did something wrong and the other thing that's really hard to do when writing something like this I mean they got pretty good documentation until like 10 volumes or something but you cannot do without the Linux driver because there's so much missing some little details so then have you flush I'm almost done so then have you flush to flush the cache or you know so this little there are croaks for some devices so this took us a long time to actually achieve so in the end we ended up with a demo scenario like this so this would be a G-node demonstration component system it's a very small one so you got your root task here it launches our GPU multiplexer slash 3D renderer this is not really necessary but just for demonstration then we have our frame buffer must come from somewhere right now like our real hardware frame buffer and our nitpicker GUI multiplexer so in the launcher this would be one 3D subsystem we'll start a 3D application and we have this nitframe buffer which connects to our GUI server it gets like a frame buffer session so the multiplexer in the session the renderer offline picture into the 3D application and the application will copy it to the nitframe buffer and the buffer will multiplex on screen so you can have multiple applications on one screen ok results already shortly so we are actually pretty proud that we could we are able to write this whole multiplexer in 10,000 lines of code but this is just broadwell so for the Skylake and Kabylake you will have to do some more stuff for Skylake for example you have to load microcode again it's the new trend at Intel microcode for anything even Wi-Fi so and the whole like trusted computing base really gets smaller so we are not able to add the Mesa library to the trusted computer base because it's too complex and way too large ok so coincidentally we will be supporting Mesa for broadwell CPUs in the next step we want to take advantage of the Blitter engine so we can copy pixel data very quickly we do it in software right now as I said Skylake Kabylake support would be nice and maybe we look into a GPU virtualization but we don't know because Intel has something called they call it GVTG Intel graphics virtualizations technology or something and there they have a device model internal which is kind of interesting so you don't need to implement your own graphics device model ok some references for if you want to look it up online and it's demo time I think ok alright so this is like real old demo here you can use it a lot so let's see what this does some standard Mesa demo stuff and to see that we can have two there would be our gears so these would be two separated clients at this time and we did some more sophisticated things we ported a shader from shader toy like one to one we'll see if it lives so this is one dot compiled at you have the shaders compiled in the machine code load on the GPU and execute it so that would be free so let's see if we can do four yes just because you can alright that's it ok the question is if we have some priority handling we haven't implemented that but it's you kind of schedule context you can implement something like this it's not a problem you can say what you actually can nest next in the driver so you have yeah you kind of can can you set the car and you say there is some notion of preemption but we haven't really found that yet but the question was if you can stop rendered work if it's rendering we wrote a backend for the EGL so did you try to did we have modified the Mesa library no because it's too we didn't have some changes upstream because it's too old by now so we ported 11 to and since then we are now at 17 or something they changed the versioning yeah we just have a new EGL backend and we changed the DRM backend but DRM is not Mesa so the question is if we could communicate to the Linux community yeah maybe maybe yeah but you know the Linux community is so big and Intel has how many people 150 in open source so they are doing things their things and until now we were the small guys so I don't know we could try is copying to the frame buffer the question is if the output of the graphics causes copy to the frame buffer right copying of the the image to the frame buffer is it possible to use dpd dpd for that too so he asked if it's possible to use gpu for pixel copying and yes you use the Blitter engine that's what it's there for and we want to use that in the future but it was out of scope of this kind of work and I mean you can copy 3 gigahertz per second on a model CPU so it's sufficient more questions project for this did you do this for showing that the geno can do this so do we have some project for this yes so we want to use geno in a daily basis we use it on a daily basis we want to have 3D acceleration so it's going to be integrated into the main geno our scenarios someday hopefully pretty soon and our thing is we want to look into this graphic cards virtualization thing and it would be cool to have like a virtual Linux or what which could take advantage of 3D somehow and all the approaches right now up until now were not really good yep so are we looking into our gpu course yeah we could look into AMD there is documentation there and we are of course in no go because you know Nevo and all this thing yeah but it's I mean this took like too many years to do so you don't do it like every day and you don't do it without finding it's a little hard for fun yeah if I can maybe briefly speak to the last point because you had a question and I actually worked for AMD on our graphics drivers so the mesa part I think you can just take one on one like you did for Intel and it's the right way to go for the kernel stuff, multiplexer stuff you really are biting off a lot of stuff and the drivers are huge trying to reimplement them there are lots of people working on those and if you can find any way of taking the code that's in the Linux kernel and maybe somehow adapting it that would probably be the way to go I know it's not the microkernel philosophy so the guy from AMD said it's hard to get the kernel part working he said the user part is fine and what we should do is like try to port the kernel part so guess what we did with the Intel driver first no we did it on this one we had Craig wheel yeah but yeah that's good okay I think that's it


That's right live to you Bring the gate out a little bit so we can have some Check check check bring the gate up a little bit so that this way you can hear me better on the mic One check one two, so you hear me nice and clear, right? Yes, nice and clear Everybody We don't care it's nice and clear Welcome everyone I am the host some feedback over in my bed 21 live James P. Madonna and I am with my co-hosts and very special guests for the second time the Performing artists and clothing designer extraordinaire. Mr. Paul Anthony Mantia, okay now. It's not muted Okay, no because I had I had to do my spiel.  All right, so here he is here. He is for the second time the new I don't hear you man. You don't hear me Hold on. Let's see what's going on here. I hear you the vast new and improved I Can't hear you James. I That's weird I don't hear nothing. I can't hear you at all Not at all. Gee. That's weird Go out and come back Go out and come back in Hope you didn't hope you didn't mute is his Speakers anyway Oh There's always technical difficulties Arising unfortunately I Sent the link out to the other Supposed guests for prospective guests and they come when they come anyway, I You could see all my shows On I've seen on the web. I'm on Facebook on tumblr on YouTube under progressive discussions and Megalife 21 live Can be seen on the progressive discussions a YouTube channel Okay, I've been doing progressive Internet talk Whether it be serious deep talk concerning politics current news current news topics science Religion Theology all kinds of deep subjects or whether it be very light Entertaining fun topics like topics involving comedy Cooking cooking advice recipes barbecuing Craft beer review. Oh, he's back. Let me see if it's worth. You hear me Unbelievable, I don't I don't understand it. You hear me now. Let me see something James. I'll talk now Did you mute? Did you mute your? I know what it is. Hold on keep on talking Don't stop right. Okay. I know you're live, but you know talking don't stop Well, we are waiting. We are waiting right now for Paul Anthony, Manthia to He cannot hear me, but I can hear him I Can hear him loud and clear I Have no idea what the technical difficulty is I'll tell you right now. I can hear him perfectly his acoustics are out. I don't know man. It's just weird. I can't hear you Well, yeah, I wish we weren't alive, but we have no joy Sorry, nobody helps me around here. So, you know, it's kind of like hard I'm gonna hire some people to help you out soon They're gonna start hiring people James. Yeah, definitely Yeah, I don't have no sound here on this end. I don't get it. Yeah, you can hear me though. I Can't hear you. You cannot hear me That is the right, I think I got you I got you. All right All right Okay, you know what that's why you're here let go because I have you on reverb Oh, that's why well, you know, you know what you do since I got a job since you hear me now You might as well play some music Well, first of all, what did you think of my club my clothing ideas? Okay, you're right, you're right. Yeah, what did you think of the idea? The idea for the baseball caps and all that and all that It's a great idea.  I don't want to really mention their names.  Yeah, pretty much I mean Anyone You're closest closest Hey, you're breaking up a little bit. Oh Now I'm breaking it. I'm breaking it. Yeah, how's my I sound like I can get more light in here So, oh, no, you're probably not you're perfect. Yeah, I like okay, so you can see me perfect I can see you perfectly And I can hear you and and you were cool. It's great. It's great All right, as long as it's the acoustic sounds good, you know, that's all important. So Everything is very very very important. I'm trying to get this camera to stand like that. So let me see We want inside what I've made a technical difficult difficult did not did not expect it, but now but now it's correct It's correct.  So it always happens, you know, we're dealing with the internet.       That's why yeah, but but Metro that Metro that Metro They're even worse.  I Knock on wood. I see very well yeah, I Hear you So that's because it got nice equipment, you know, and we can even make it sound even better, you know, it's like We have all the sound effects in the world You can play with It's like a little keyboard and new toy Yeah, I know This is a thing, you know, I mean I sound it sounds good in the studio when you're but I still have not set up You know the way I want it set up I'd like to have the like this camera. Let me show you this camera here. I would love to have this actually This one set up, you know This is high definition the cannon I'd like to have this setup where I can actually go online with it, but I don't know if that's possible You know, I'm not sure if that can be not sure don't like that. You mean if it could be I would have to bring a video guy media video guy in here, too You know The internet connection from that metro is going to control the Time you want to go Incredible Well, what do you eat my guitar now that you showed me? Oh, yeah, it's right in front of me the keyboard's right here actually, but I can't I only have one camera So pretty soon we'll have to be able to go, you know four ways That's a good album. We got to invest some money.  We'll start the whole show go, you know Go crazy on it. Yeah You have to bring one or two Bring the Polish What happened to that one man, I just don't know bro. I think she's pregnant.  I'll be right back. No problem. I probably sing a little bit later But you know, let me um, I gotta go and get something to drink I Oh Anthony Northern lights and Paul it sounds like you're breaking up games you sound like you're going in and out like Yep, it's not like a robotic It's not like a robot Technical difficulties here we go again. Hey, where are these other jabronis?  All right. You're back Yeah, I can hear you. I mean either way you look at it. We're gonna have streaming issues anyway, because you know Yeah, but So how do we figure they're gonna fill you out the audio on the show From coming from me and coming from you So the important thing is that the audio for the show is going Okay, that's good because that's what we need excellence excellence excellent excellent day What you got set up Really professional for Yeah, man, I spent Man tens of thousands of dollars, okay, but we're not gonna tell everybody that No, how did you do that?  I've done, you know I would tell the secrets, you know Well payday came, you know a couple of times ready for a few things I've done so I think your idea for a store location will remain secret I won't divulge it, but your idea the first store location Not too far from the Atlantic Ocean Set it's really smart that sounds like Yeah, yeah, we're definitely gonna work on that, you know, but I'm looking for a We're looking for a 10 to 15,000 square foot warehouse now in Brooklyn But you know In time a couple of months outside, you know Well the problem is the problem is we got a full team now, so I'll explain to you on the personal level So how we're gonna work that, you know We we already got the plan and we have the investor. So, you know, it's just there So, you know, I mean it's just a matter of time everything takes time Like over to just flew away all of a sudden.    What can they be creating what we don't know?   So So this is a track I put together today It's a bass the drums The guitars and some synths in the back. Well, we got a nice guitar solo for this Just freeze freestyle officers. I'll put some of the melody line in there.  There was a really no alcohol in the cans and really no one How familiar or anything because it's illegal on television now Hi You can buy it right at the corner store Like, you know, I we used to joke around I Hung out with Cheech and Chong at on Woodstock when they came in Back in the 90s and they came in the bus and we're hanging out and then talking about, you know How one day we're gonna walk into the store and you're gonna be able to order it Let me get a pack of marijuana goats Now they sell it in cans it looks like cat food cans I'm serious, man. I'll get one in a minute. Oh, yeah, I showed you, right?  All right. It's never been reported that this is told anybody You know, I think they were always afraid to give it out because now if you notice now that marijuana is legal There's a lot less crime going on People smoking it. They're more happy You know, but then you got the nuts out there, you know, you got the nuts like the you know Well Well, the good thing now they have Sativa and indica All right, so sativa is more of the energy blast Right and then it is more of the relaxation So you could break it up You know, you gotta switch up on the strings because there's so many different things out there now, you know But I kind of keep it organic. I Like the pot that we had back in the day, you know in the late in the early 90s, you know They had white white rhinos and purple haze.  It helps everything. I mean Depending on which direction you go and what you're smoking what you're eating.  Anyway, so what does it matter? You mean like chewing a backup? You know, they have on edibles they have they the oils are really good and I'm the Rick Simpson oils, but Good luck getting online and trying to find it because there's so many fake sites with Rick Simpson that He's the one that made the world that cures cancer, you know, but I got him as a personal friend But he's never on Facebook. So forget it. I was like gotta go.  All right, so listen, let's do this tune real quick Okay, let me get a sound check. So I'm gonna play it real quick. I just want you to listen Yeah You can hear the bass good to yes, yes, all right good because that's important that is clean.  No How is that now good? No, it's good. It's good. Okay. All right, so we're good.  I'm trying to get the guitar to see it. It's not easy to do that.    Are you there James? Yeah, yeah, so that sounded right the guitar what yeah, no, it's Is And see we can explain why why doesn't he get the high note sometimes? Because you know sometimes, you know, you have to say that throat and for the studio We always let the crowd just sink, you know here get that note for us, please, you know Why We should we should rehearse that It's like the Miami sound machine days of music Are you ready to hear that James? Hey listen I Listen to this the effect I created today It goes with the song the song I just played It sounds like more like a spiritual mantros, you know like every mantros like the Indian You know Exactly so where these are the two smuck smucks, man Put me on the show my Yeah He's based on guitar bro, but he is here online Oh Yes, and and you are watching We all died we now own more than like entertainment again, I bought that out again just had a snatched up Thank God one thing man, I tell you I notice like a lot of my friends that Bigger musicians out there They're getting couple tunnel man. Like I see a lot of people like I feel bad, you know Couple tunnels and playing, you know, thank God.  We have calluses for years But you know Calluses, you know after a while they start getting sore And I mean when you just you don't give it a break.  How fast you going? Well, you know Here's a practice though. So you play for all Oh, I've been playing since I'm a kid since I'm like, yeah five my father gave me the first plastic guitar You know northern lights It's good to have, you know, it's good to play every instrument to be, you know, I could play some piano too if you want I don't know it's the same keyboard I got I mean, it's just Now I got the same exact keyboard it's just I have different Just It's on plugins basically I got every every plug in you can ever think about I mean and imagine and I haven't ever had time myself to go through Have for the instruments I have in here, but you know in this studio everything's got to be played live because we cannot mess around Yeah, nice baby grand piano That's the one thing I had to say the grand piano the beautiful man.   I don't know. It's hard to do that. But you know the keyboards down here But I can't really, you know, I have the camera on me until we figure out how to do that, you know After time I don't even I might I have the studio here 24-7 so And after time I might even in it man, but I got so many projects. You know, it's just so much to do It gets hot man when you when I'm playing even with the air conditioning on it's like when you have all that energy coming out You know, yeah between the Actually, that's not true though electronics don't keep that much No, because I have I have a cooling fan a couple of cooling fans. I bought they go under the boards. I Think when you You It's hard to see everything but you know The other the other pianos and repair that's in the other room that it's a Kurtz well, so I need to um I Actually got that piano free man It's like a $6,000 Kurtz well and this lady was getting ready to throw it in the trash, right? I said, oh, let me take that off your hand I walked it a block up I took it home plugged in in USB it works The only thing is a couple of the keys are out So I'm gonna repair it. I'm gonna put a whole new keypad in and the back of it has a special kind of AC adapter But it works so she was throwing away.  You know, so it's like, alright, I got to send that in for repair People don't get you know, you're constantly putting things in for repair.   I basically um, I Put a To blow it was like pretty close to his Now I'm saying like from the soul nothing from nothing leave nothing.  She should sing sing I don't know what that is I forget that I know a lot of different songs that um Tonight my voice is not hitting or when I do no joe B It's not happening bro, but let me see what I have in the library. Yeah Let me see what I have up here. I'm looking at my screen Say what what is that shake that group thing that was like a Mega group Yeah, I know you're playing man. All right, I got you know exactly what that is I Got so much I Got a lot of stuff in here. So I'm just trying to like look at it. Yeah. Yeah, it's hot in here tonight Yeah, it's pretty hot Happened was she able to see the performance I did or should I The solo one You mean It would be over Take my my over shirt off because I'm dying of heat control in here But it's not that it's cool in the room.  I played out in the cold And I've literally got off the stage sweating because it's like all the feeling of what you're playing in that You're putting all your energy into that guitar. I mean into the notes that whatever I'm playing man creating in my mind while playing you know And I'm trained very well like you know when I'm in the studio and lighting Yeah, I'm hitting bad notes and I'm trying to find Certain notes that I want but when I'm live Never catch me hit the bed.  I mean it you better not make a mistake up there I mean, you got to be able to be able to follow Not even follow just be creative on your own, you know, and Like when I'm in a band and I'm playing in the bands in five quarter I can be playing in second quarter of the car, you know going off-time signatures And I'm saying that that's when I'll talk about feeling because people sitting there How do I learn how to play in seven quarters? It's not about learning how to play it It's learning how to feel it man. I mean if you can't feel it don't don't play it I wish you want to tell her That's what he meant Like a man She's dancing Her head So James here's the new news so I'm not gonna mention his name yet, but I have one of the guitar players that I'm actually I met him on the boardwalk here And he was playing like Flamingo style guitar acoustic. Oh, that's pretty I love that kind of style So I know he can play right off the bat by just hearing him solo You know because he's not trying to overkill it, but he's he was feeling it at the same time You know, it's a difference between playing speed notes, you know and doing like 16th to 32 You know, whatever you're playing on that guitar, you know It is also a point where you want to also play some slow You know where you say you bringing the blues flavor in a little bit a little bit on be a little bit of fusion a little bit Everything, you know, just like showing where the plane goes, you know, how far can I take my playing? You know, so there's no it's unlimited because you never stop learning anyway, but this guy I have We're gonna we're starting a band. He's from the band kicks. Remember kicks kix He's actually one of the guitar players one of the originals I spoke to him today and I said What are you doing man?   I can't walk there to play It's not even about money. It's just being out there playing, you know, and having a chance to be on the stage again, but So yeah, I'm gonna put something killer together man I have to find guys that are on my playing levels, you know, because if I haven't won the top player that's on that playing level That's he's been playing for 35 years, you know, so he's on the same all those on pretty much the same playing level You know, you see when you have two guys that have that experience of being he's a he teaches guitar as well You know, so you know that if he's teaching people how to read music That means he knows how to sit in those time keys and all that which means now I have myself somebody that is not an amateur So you have to separate the pros from the cons in music So now the next step is being able to find somebody that plays better than I do on piano, okay? Because to me finding somebody that can create their own style and everything You know, there's a band listen. I want you to check them out.   I'm just playing basically, you know feeling my song when I wrote but Well, you do the better sound the way you sound the studio on that stage That's all I have to say don't put no extra tweaks in your voice and auto-knit and you know change the vocal pitch They don't sing without a mic. They can't see without a mic.  Yeah Yeah It's it was it looked both of us We banged out pretty damn good show. I mean for a for a live stream You want me to take the show out with that solo or no? Uh, uh, are you uh as you voiced as you voiced? Oh, no, no, I'm not singing. I'm not gonna sing. No, I don't sing. I don't sing Oh, we we can we can we can you doing instrumental stuff stuff Yeah, yeah, you don't have I mean you're saying You're not feeling I'm feeling I'm feeling fine. Actually, I'm just saving my voice for I have things to do this week in the studio I have clients coming in and I'm okay. No, we're gonna be doing a lot of tracks on vocals this week. So all right No more Then the smoking doesn't help Well, the kick well, you've got that you've got that Yeah, that's true Yeah, because nobody is coming on and I mean and I think we should set it up with all of these guys I think I don't know these guys didn't show up tonight. So maybe maybe uh, mr.  You said you're gonna yeah, I'm working on it. I'm working on I want I also want to get somebody in here to um like uh Get the video working the way I want it, you know, where I want to be able to capture Every corner like getting the equipment maybe even hanging hanging the camera from up Capturing down because I got the green in front of me You know here, but yeah, the green that the green Oh, no, I painted the wall green. Oh, okay. The whole wall. This is behind the back. I'm not done yet You know, I'm still I'm still working on the apartment. I still have to do the ceiling white, you know, and there's a lot to do here It's just like Yeah Well, yeah, I think we should cut the show it shows a little shorter and do a whole bunch of them, you know Yeah Let's let's start bringing people on the show interviews. We can do that. I'll set it up if you want Well, this is part this part part two and then you know, we'll continue Okay, I'm thinking We can bring chag chag the drummer that played in that song We can bring him on You know, and we will have a lot of people watching Well, he he doesn't We will bring he'll bring a serious crowd He doesn't have to leave home Yeah, me too He can be on his end. I could be on my end and you know And as long as I can hear everybody, that's it. We'll be straight. I can have six I can have six Six people on video including me. Let me talk to chad and let me set it up But he's gonna want to go live on facebook too. So how do we do that? Can we? I don't I don't have that professional I have the freedom All right, I'm like, you know what talk to me on the other end. They won't I'll get the professional account That's true, that's true, but we go for that money And we're trying to we get that monetization on facebook and Everybody's getting paid You know, I mean the more hits you get on that video the more you get paid So we want to try to go to every single end in every avenue that we have on social media In linking, you know the stream the live streaming into every single sector of The media, I mean Because if you we get syncing it to tick tock into everything that would be great And I know there's a way of doing that and that's why You know, I'm going to pay for the premium membership. Be the way I'll get the unlimited It doesn't matter to me. I'll spend the money because it's important to me but You know, it's important to get my my my end in your end the sounds And the video clean With internet clean is how well the previous The most important thing is the internet provider The internet connection I mean the the professional Is You can have 10 People on your show show. That's it. It's only 25 a month. That's damn idea. So you're gonna tell me like 300 a month No, it's like 25 All right, so many information after the show You know, actually I'll tell you what let's cut the show Was what's ended with something we'll see the next time. Why don't you give me a call? Let's talk on the phone. See what we can set up. Yeah. Yeah All right What do you want to close out with?   You know how I work I'm gonna release it and I'm gonna say I don't know where that came from. Shit, man.  Who is that? But then on the other hand that that can cause people to go and completely insane and I don't want to be responsible for anything So you understand what I'm saying? I can share it amongst everybody though well, uh, mean uh, you know, like, uh, Portuguese Like Portuguese food So there's a lot of You know, I don't know I got Maybe you should take her out But she lives in she lives in uh rodai rodai That's not too far No, it's not She's in California Now Rhode Island is right over by Boston Uh, uh, right after I remember I lived in Rhode Island for a while and then I stayed in South Boston and Right, right after Right after uh, Yeah, I did mystic. Um that mystic uh, Connecticut. I did a big show there. I remember backing god I don't know. I don't remember the year but I did a big concert out there and some some parade and I played at the mystic pizza shop, but they had a podium upon the roof Where the stage set up for us I went out there and they had everything set up for me already in the sound system sound engineer I think I did about a half hour show and I was out of there Yeah, well by uh My friend South south east south east of mass massachusetts, uh, near almost near the cave I think So anyway What about your theme song Oh, you mean the car? Yeah, I'll tell you what we can do this. I mean No, I'll do the car Yeah, yeah, do the carnival thing tonight. All right. I mean the next show I'll set up some, you know some funny sound effects and shit We can have you know a couple people on the show, you know, let me slam the door or whatever All right, we'll have it all set up I can write jingles for the show if you like All right, here we go So while you're doing that, I'm gonna I'm gonna show a couple of things while you're doing that New stuff All right Hey, there there's There we go, and uh, let me give them a let me give them a nice White screen And there's just some Some of Of his Hundreds and hundreds of these designs You see the Brooklyn bridge in there Or the shark's mouth and his teeth, you know, you can make that whatever you want right there.  It looks like Yeah, man How on you it's whatever my mind thinks when i'm creating, you know You know what know what I bet I bet You can do What's that you can do Oh, yeah, I've done an abstract art on the on the on the design i'll bring that into the show next time though Okay, okay. Yeah Thank you. Thank you over and out All right. See you next time


Okay. Can you guys hear me? Perfect. So, as you said, yeah, I'm Michael Bauer for the last nine months or so. I worked at the GSI. You can find me on github at Bauer M97. Send me an email, www.bauerm.umich.edu if you want. You find this particularly interesting. So I'm here to talk a little bit about singularity. So singularity is a container solution for HPC environments. Here's our homepage, singularity.lbl.gov. You can also check us out on github. We've had a lot of contribution in the last six or seven months. Lots of activity. Feel free to come by and contribute. There's a list of our contributors. So the project head is Greg Kurtzer at Lawrence Berkeley National Lab. Myself, Brian, Krishna, and Vanessa are some of the core developers. And then we also have people who have made other significant contributions to our code base. So I first want to kind of pose a question. I'm assuming that at the container dev room, most everybody actually knows what a container is already. But it's important to kind of strictly define what we mean by a container. And to do that, I want to talk first about virtual machines. And Wikipedia talks, and they say, computing a virtual machine is an emulation of a computer system. Just at its most basic level.  Some pros, you can run different operating systems. So back when I was younger, it was really cool that I could run Windows on my dad's Mac OS laptop. It's awesome. You can save money, so we didn't have to buy, you know, a second laptop in order to run programs that only run on Windows. Namely games for me. And it's a little bit easier to maintain than hardware. You can download a new virtual machine if you want, right? Some downsides at the same time is performance. It's going to be a little bit slower, most likely. And you're going to have memory requirements and storage requirements that you wouldn't otherwise have. So to run a virtual machine, you know, maybe you need 10 gigabytes of space. And to talk about containers, then we can say that containers are very similar in their goal to a virtual machine. With the container you want to take some environment you want to encapsulate it and store it into some file or system of files to be distributed. With the container, however, we don't do any sort of kernel emulation. There's no architecture virtualization. It's just software packaged into one file. And what that means is we don't have to waste an extra 5% performance to emulate the kernel. We have a much smaller footprint on the disk. We have nearly instantaneous startup time. So when you guys, you know, when you do Docker run, you're running immediately rather than waiting two or three minutes for virtual box to start up its virtual machine. And you can run multiple instances of the same container using just one image on disk.  My internet died. Hold on. Is it possible to get full screen? It's fine. So kind of to talk about specifically running multiple instances of the same container just off of the same base image. This is a depiction of Docker's image structure. So you have multiple read only layers built on top of each other. And then at the top here, cut off, if I can scroll up just a little bit. You can see that you have a small thing, maybe a couple megabyte read write layer where any running application can do read write operations on.  That allows you to do some really kind of cool things with containers. So a lot of places actually using containers, you know, you can deploy on Amazon with AWS, Google Cloud Platform. You have companies that are now providing their services as a container that you use. Some big websites even are using containers such as Reddit to deploy all of their infrastructure. And so I want to talk about specifically containers for scientific computing. If I what? Yeah, I am and it doesn't align. It won't align. It's okay. It was worth the try. Anyway, so I want to talk about specifically containers and scientific computing. We see containers a lot in industry, but for instance at the place I've been working at for the last nine months, they were not using containers at all and they wanted to investigate containers for their HPC. And why would we want to do that? Well, you can first and foremost escape dependency, Helen. So, you know, they always had the issue that we were trying to run some version of code and it was depending on library A, we had library B and it didn't run and users get pissed off because of that and they yell at IT because of that. Secondly, you want your remote code to work the exact same way as your local code every single time. And with containers, you know, you can run the container on your local machine, send the container to the cluster and the cluster just runs the container and it only runs what's inside the container. And third of all, and maybe most important even for scientific computing, is that you can have one file that distributes your entire environment used to generate results. And that's really important for reproducibility of results. When we're doing an experiment and you can have everything that you used for that experiment inside one file and give that file to other people, you can promise them that they're going to be able to reproduce your results using the exact same environment that you did.  It works perfectly. You run code on the cluster. It works not perfectly. Okay, this is supposed to be animated but it's a PDF so I can't do that. I created a little bit of a checklist when I was working there when we were investigating container solutions to use for our HBC for what we actually need in order to have a container solution. So, one, can any user run containers without special privileges? Two, can we integrate it seamlessly into our infrastructure? Do we have to install something like Kubernetes or, you know, Docker Swarm and do we have to waste time and effort to get that configured properly? Third, is it portable between mini systems? Can we send it on older hosts? Can we send it across the ocean to somebody else to let them use our container? And fourth, can we let users bring containers onto our cluster without any sort of administrative oversight? You know, I don't want to have to scan my users' containers to make sure they don't have malicious content and I don't want to have to trust my users. And so, we first, we investigated Docker as most people do. It fills three out of four of the check boxes. It doesn't really integrate nicely into our architecture. We would have probably had to install something like Kubernetes on our server to get it to work properly. So, that kind of ruled Docker out for us. And another really, really important point for HBC is that your container software can't come anywhere close to root privileges ever. So, you have to ensure that if you give users an ability to run a container, they're not going to be able to do something that they would otherwise not be able to do on the host machine. And with Docker, we actually find out that we run into a slight bit of an issue where Docker actually runs a root-level process all the time as a daemon that spawns containers and kills containers in charge of scheduling containers. And so, when we had initially proposed to IT that we installed Docker on the cluster and try it out, they just told us no. They wouldn't let us install something that had a root process on every machine. And that really ruled out Docker for us. And so, that's kind of where the investigation came in, and then we stumbled upon singularity. And singularity is the solution that we chose, designed for high-performance computing. So, as you can see, we can go back to the checklist. We're filling up every single check mark this time. There's no root-level daemon, so we don't have to worry about the first point where with Docker, you kind of have to worry about that. It can be run essentially the same way as you run any application. You can directly execute a container as a file on your disk. It's portable, so singularity is built. It doesn't require any kernel features from most recent kernel. You can run it back. I think one of our users was running it on Red Hat with 2.6. So, you know, designed for maximum portability. And then, we don't actually have to worry about trusting our users. There's no issue with giving a user full control over what containers they're running on the cluster. We won't run into issues with that, and we'll talk a little more about that in a bit. Again, with singularity, any container can be run by any user at any time for any reason. So, when you run a container with singularity, the user is the same user ID inside the container as they are on the host. You don't have to, again, just to reiterate, you don't have to change your workflow at all to use singularity. So, you know, in our instance, it was just changing our batch submission script to just run singularity instead of running whatever other executable we had to run, or directly executing the image file itself. Thirdly, it's a single image file, and so that ties in really well, and that's kind of one of the big topics that we discuss in a paper that we submitted for publishing recently about singularity is that we have just one file, and so you can distribute that file and it contains everything necessary to run, everything that you wanted to run. And it's safe.  As you can see, there are a lot of places, actually, now that are trusting our container software to run on their cluster. So, place I work, GSI Homeworld Center, the running on our cluster, a couple other big ones, there's people running at MIT, on Stampede, at Texas. So, here I want to compare singularity to a couple other leading container solutions. So, a shifter, which you may or may not have heard of, is similar to singularity. It's containers designed for high performance computing. Charlie Cloud is also the same. It's a slightly less mature version, Docker, which I'm sure everybody's heard of, and then singularity. And there are three kind of main points that make this really important for high performance computing. One, native support for GPUs. Inside your container, it's really important, especially for scientific computing, that you can use your GPU inside the container without any extra integration. The second, native support for InfiniBand. Again, it's really important that we can run InfiniBand and use not just IP over InfiniBand but verbs or RDMA inside your container without having to do any extra integration. So, you can just simply specify you want to use InfiniBand and then inside the container you'll have access to it. And also, we have native support for OpenMPI. Other to HPC containers, they provide also similar native solutions. So, it's not such a big difference, but from Docker, it's a big difference. Right, so essentially what we're saying is we don't have to rely on any other software to build a singularity image. So, with essentially Shifter and Charlie Cloud, you can only use Docker images. And so, on those services, you have to rely on Docker in order to have an image. On singularity, it's possible to use something like Debootstrap to just build your image yourself kind of natively and really quickly and easily. Does that answer your question? Yeah. So, we actually, we submitted a paper. This is where that graphic is from. Talking about containers for scientific computing specifically. So, now I would like to cover a bit of a basic usage of singularity. Kind of just to get people into the workflow of singularity and how you would actually use it. So, I like to think of singularity being used kind of in three separate segments. You have a create process where you run pseudosingularity create, give it a name, and that puts the physical image file on the disk. You have a pseudosingularity bootstrap, and then you tell it an image and then a definition file, which I'll talk about in a bit. And all that does is it takes a set of rules inside your definition file, analogous to a Docker file, and it builds your image, either from scratch or you can also build it from, you know, a Docker image from upstream, hosted on Docker Hub. And then running your image. So, there are three main ways to run an image. You have singularity shell, which just as it, you know, as it may seem, it opens up an interactive shell inside the container for you to play around with. Singularity exec can just execute any sort of executable that you specify, and then singularity run. And also directly executing the image. These are kind of the same. And what that does is there's a script that you can specify inside the bootstrap file, called a run script, and it just executes that script when you run direct execution. So, now I have, and hopefully this is still cache because I don't have internet. It's not cooperating. So, I have an ASCII cinema little demonstration just for the basic bootstraping process. Why would that not? Okay, I don't know why it's not going full screen over here. But essentially, if we can maybe play it and zoom out a bit there. This will just show you kind of how simple it is to get started with Singularity, and it'll take you through the creation process, which we're doing right now, by specifying a size of 568 megabytes. Correct. So, the thing about the creation of the image is, we want users to create their image locally on their own computer. So, it's actually, because of the way Singularity works, we're doing set UID and we don't have user namespaces. It's kind of, for now, not feasible to be able to let a user bootstrap with just some arbitrary code on a remote environment. Because in order to do most bootstrapping things, you need to actually have sudo app get install or whatever. Right. And so, I bootstrap the image here with just a basic definition file. And then, all we do to run it is Singularity shell. And that's actually, as you can see, we don't need sudo there to use it. So, Singularity supports several different actually image formats. So, the most common, most basic would just be the image file, which isn't listed here. But it's just a .img file. It's a file system that's formatted by Singularity Create. And it's just one file, usually four to 500 megabytes, or maybe a little bit more. We also allow you to have just a directory, so you can directly, if you extract, for instance, a Docker, the archive generated by Docker, export. If you have that in a directory, you can then use that as a container, or if you have an archive itself, you can use that as a container as well. So, Singularity also, for users that are more accustomed to Docker, we have direct Docker integration. So, if you watch this, what I'm going to be able to do is I'm going to run, just as non-root user, I'm going to do a Singularity shell command, and I'm going to directly call on the Docker API, and we're going to run the 121404 version without any extra privileges. And this was done in my hotel room, and it was awful downloading 100 megabytes, so it takes two minutes. And if we go to about here, yeah, you can see it finishes downloading and extracting and puts us directly into a Docker container. And the next thing that I want to point out is I'm going to exit the container here, and then I'm going to rerun again Singularity shell and directly call on the Docker API, and you'll notice that it's just instant this time. And that's because we actually, we store these in cache and normally in the home directory, also optionally in the temp directory. So if you run 10 instances on some node somewhere on your cluster of the same Singularity container or Docker container, you're actually only going to have to download it once. So we support that as well. There's no native integration for any schedulers except for Slurm. So we integrate directly with the Slurm plugin manager and I was going to talk about that. It's actually on my slides online except those aren't behaving properly. Actually, they might be now. They're not. But essentially what you can do is if you're using Slurm, you can just add a line into an sbatch submission script that specifies what container and it'll start the container before the job is run and then run the job script inside the container. So the primary... Is there a question? The primary motivation for searching for Singularity at where I was working was we were running jobs for Alice. We're in Alice Tier 2 Center. And our cluster there is a Debian-based system whereas Alice and CERN, they're expecting scientific Linux. And what happened is that we were getting, you know, an exceptional amount of errors, something like 40% error rate which is not normal, not what they're expecting, not acceptable either. So for those of you who are unfamiliar, this is kind of what the Alice project is looking like or more in general WLCG. They generate data at the Tier 0 inside actually the Alice detector at CERN. The data is distributed to Tier 1 centers. There's one in each member country and then it's also distributed later for processing to Tier 2 centers. And the GSI is a Tier 2 center and that's where I work and that's our computer that we've been building the GreenCube. We'll have 300,000 cores. So again, as I said, we're trying to run 2,000 jobs at the same time except 40% of them are failing because we're running on the wrong operating system. And so our current solution for what they were doing is mounting some libraries in Luster in some weird directory. They have their slurred job submission script. They actually have to intercept the script, modify the script, hack in some LD library path fix. I think there's more. I don't know what they were doing. They couldn't explain it to me well enough for me to figure it out. The point is that it's a big, ugly mess that they're doing. And so we converted that to run on Singularity. So the Singularity solution to this problem is much simpler. So we package Scientific Linux 6 into a container. We modify our slurred submission script simply just to be executing the container instead of executing the Alice job environment. We no longer have to modify or to mount Luster to get access to the libraries that we were supposed to have on Scientific Linux. And we can test our containers locally for deployment. So here we have a small diagram kind of this is our Singularity build file. And essentially what it does is it will pull the Scientific Linux Docker container from upstream. CERN actually provides in the Alice repository a Docker file for building a Docker container. And since Singularity can just natively integrate with Docker, it's just a simple one line inside the build file and we're using Scientific Linux already. Then we package it up into a container. We store it on our file system and it's then accessible anywhere on the cluster. So how exactly do we do what we're doing? How can we have such secure containers? How can we ensure to system administrators that users aren't going to be able to escalate their privileges? So Singularity, it's actually very basic at its heart. It's a set UID binary and it's owned by root. And so when you call it, your real user ID is zero. What we do is we mount all the necessary sort of files. We bind mount the image. We do any sort of bind mounting that the user wants. Clean the environment to ensure that there are no leftover environment variables being passed through.  So the one that we always use, we always use the mount namespace on newer kernels. If you're interested in trying it, it's possible to now use user namespaces. It's also possible to hide behind a process, a PID namespace so that you can only see processes running in your container. So running with a set UID binary is kind of insecure by nature. I mean, you're asking a system administrator to trust something that immediately grants the user root privileges by calling it. So in order to securely do this and do it in a way that's auditable, we have two function calls in our code. We have a call called singularity privilege escalate, which changes the effective user ID inside the running process to zero. We then do one action at a time. So for instance, in order to bind mount, we have to have root privileges to do that. And then we drop it immediately after using singularity privilege drop. And we only ever use any of these function calls immediately followed by the other one.  And you can audit the code to see that we're actually doing what we say we're doing. So another thing that we get asked a lot is why don't we use user namespaces always and by default because, for instance, Docker always is running namespaces now I believe by default. And inside a namespace, what we have is we map a user ID on the host to a user ID inside of the container, inside of the user namespace. And so for instance, if your user ID is 1,000 on the host, you might have a mapping present that's 1,000 to zero, and then you'll be user ID zero inside the namespace. And it looks and feels for the most part besides a few system calls like root. You can do almost everything you want to do. The issue with this specifically for us is that it has the potential to break portability between environments. So if you actually go back and you inspect what you've done outside of the user namespace again and you look into the container's file system, you'll notice that things that you created with user ID zero inside the file system now have user ID 1,000 on the host. And so if you send that to another system where you don't have an account with ID 1,000 but 1,010, we would have to do extra work to make sure that your new user ID is mapped to your old user ID, which is mapped to root. So that provides a portability issue for us. As I've iterated earlier, this is really the real danger for us when developing this code. And this is what we try our best. And actually, I mean, right now we succeed. We make sure that no user code gets executed as user root ever. If it does, you've got to consider your system compromised because you can't trust that your users are one intelligent enough to know what they're doing or two non-malicious. So again, security and singularity. This is one really big thing that we focus on. We spend a lot of time looking through our code and ensuring that there are no edge cases that are going to cause security issues. And the first principle that we have is never let users run code as real root ever. Just doesn't happen. Two, we only use the effective UID of zero when absolutely 100% necessary to do one thing in our code. So what I mentioned earlier, mounting a bind mount, we have to escalate. Do it, and then drop it. And three, we drop permissions and capabilities when forking into a new thread. So when you fork into a new thread, the child process actually waits, and it stands idle until the parent process is able to drop permissions, confirms that the permissions have been dropped, and then tells the child process through a pipe that it's good to continue. And then you can continue on with the child processes code. And that's kind of to ensure that when we fork into the actual container process, the user can't force a race condition and try and beat us and take over the parent process with root. Another thing we have to look at is isolation. Kind of what do we isolate in order to provide the most secure environment for a user? One thing we do, we bind mount the image file just into the host file system at a specific location. You CH root to move into the mounted images location. We mount vices, we mount the host host file, also resolve conf we mount, and a few other files. And then we use namespaces when possible. I talked about username spaces earlier, PID namespaces. We can mount those, and we do it on request but not by default. The idea is to make it as close to the host's environment as possible without doing any extra necessary work. A big difference between Docker and Singularity specifically is the network isolation. So in Docker, you might be familiar with lib network where you have to actually go in and set up a network and kind of tell Docker how you want containers to interact with the network. On Singularity, we don't have that. So you simply are just on the host using the host's network. So whatever network stack you have on your host, your users can still make use of that. IP over in Finny Band, Ethernet, whatever, all works. And you can also use your host's physical devices inside the container. We make it a point to isolate as little as necessary while still maintaining security. And that allows you to do things like using your GPU. So just a little more to touch on some more specific things that we do to ensure that we're not letting users escalate permissions. One thing we do is only the... So there's two binaries installed by Singularity. There's a setUID binary and a non-setUID binary. And when you call the non-setUID binary, it in turn calls the setUID a binary. And only that non-setUID binary is allowed to call the setUID binary. No user can call it on their own. And that kind of ensures that you never have any extra arguments that would potentially corrupt our runtime. And it forces the setUID binary to be sure that it has the proper arguments to actually execute. And as we talked about earlier, we have the Singularity priv escalate and drop to escalate and drop privileges. And also when we're mounting files, we do it with the msNoSUID bit. And essentially that ensures us that there are no setUID files put into the container. So if you have a container that should have a setUID final file, you mount that container that will no longer look like a setUID binary. And we also, when we open files, we have closeOnExecFlag set so that the child process is unable to compromise the parent process. So that's about all I have in terms of really specific stuff. This is kind of a table that we have that displays all of our commands, how we can use things and what they do. So I think, yeah, we have some time for questions now. Go ahead. So I've been working just like about the setUID stuff, getting more and more satisfied this time. So yes, we can discuss all the security matters which happened, like to avoid something bad happening while we're in setUID. But ultimately, I'm kind of struggling to understand the defense in depth. Yes, we can execute the minimal amount of code while we're in setUID, but at the same time, security vulnerabilities in Linux, and then there's balls in the room, whatever, having all the time is just a fact of life. So I'm just wondering why you're using this very old setUID mechanism instead of much more modern capabilities, management, all the things like this. The point is specifically to use an older mechanism. So the question is why I'm using setUID in Singularity when we can use sort of newer methods, such as user namespaces or whatever. Correct? So we actually do interact with capabilities. So when we create, we have the no new privileges capability set inside of the child processes. And we do that when we can. And so there are certain things that we allow. So when you compile Singularity, it assembles a list of abilities that the kernel has. And so the MS no new privs is one of the things that we look at. And if we have that, then we say, OK, we're going to set this specifically because it's available. And so we do all that we can, but it's sort of as available by the kernel. And the idea is we want this to be useful on as old of a kernel as possible. And so we do have a user using like 2.6 kernel, I think. And I'm not sure what features are available. OK. And one more question. You said it's safe to run auto-free user code. Like how do you retain this without validating the comments first? But I don't really understand how that works in this industry. Sure. Even if we prevent them from getting through, then yes, they don't get actually through. But that's like a very old way of both. Like I care if they run a machine, but I care much more if they couldn't run a botnet on my HTC cluster or they can somehow do something malicious using my computer power. But I don't understand how this works. Right. So the second half of the question is not understanding why it's important, essentially, that why we care that we just want to limit user's ability to run as root. And so the answer to that is I think your users, if they're just running on your system normally, and they can submit some arbitrary code there, then they could probably do the same things as you want them to as you're afraid of them doing. And so the idea is that you have all the same measures on your cluster. They work inside the container the same. It's just that it's inside of a container now. Exactly. It's not that you care. Right. It's not that you can take this and put this on an insecure system and all of a sudden users can't do anything. It's just that whatever measures you have in place on your cluster, we're not going to destroy your efforts to do that. And we're going to ensure that they can't have root while doing that. And so if there is a kernel vulnerability, we can't prevent that. And we're not going to prevent that. That's not what I'm asking. You said you can run code without, you can run that container and not have to validate the container. But I don't understand how that's supposed to work. How that's supposed to be saved when they do the arbitrary code in the container that you don't isolate, say, the network or you don't. Even if you isolate this thing, people can use your code. I just don't understand how this is going to work. That's HBC. Your point is more secure. Your point is secure. That's what I don't know. I mean my point is just that there's no inherent vulnerability by running the container. Right? I mean users, if you're using an HBC, the users are running arbitrary code on your system just by definition, essentially. And that's why we're having singularities for users to run that arbitrary code. Go ahead. Go ahead. Yeah, yeah. Go ahead. So you started off by saying that you use containers a lot. And then when you're done, I would consider the other products containers. What does singularity actually provide to the quantity of each? Sorry. What was it? Can you repeat the question? I couldn't hear very well. So you seem to be tapping against a lot of what I would consider the features of containers. Specifically, modern features that support containers like I don't know if you mentioned secret. But what does singularity actually provide that you don't get from a true and a little bit of a change? Right? So the question is, I seem to be talking negatively about other features that maybe other container implementations are using that are features of containers. And why am I talking negatively of those? And why don't we implement them? And the answer is, you know. So what does the singularity provide that's not just a change root and just mounting an image? Right? So one, we do use isolation features when it's possible. So user namespaces, when it's possible, we do so. But we only do so at the request of the user. So the idea in the HPC environment is we want to make our containers kind of as similar to the host as possible so that we don't have to mess with what users and what the administrators are already used to. And so, for instance, network isolation. We don't want to have to have somebody defining new policies to create a new system of network. We just want it to appear like it's the host but allow users to bring their own environment. And singularity, specifically, what we do is we search for certain things that we can do to provide security and isolation for your container that you maybe otherwise can't do without root. And so, essentially, yes, what we're doing is mounting an image, change rooting into it. But in order to do certain aspects of that, you are going to need root. And so we do have a set UID binary to do that. And it's done in a sort of defined, controlled way. So the question is, can we not use essentially the set UID part of slurm to do whatever is necessary to mount the container? And the answer is we can and we do. And that's slurm integration, actually. We have a slurm plugin that we wrote. I was supposed to have a slide that showed you exactly how that worked. But, of course, my internet died. See if I can bring it over here. Anyway, we do have a plugin that was created for us by a user. And it does integrate exactly as you specified. It uses sort of the root part of s-batch, hooks into that. You can define it with just the pound sign s-batch and then minus minus singularity and then path to the container. And then it does all of that just using the singularity library that comes with it, not using the singularity executable. Yeah, go ahead. Is that the question? No. The question is, what happens to you around or the comments? Can you talk louder? Sorry, I can't hear. The question is, what happens to you around the same time? The same time? Yeah. The same thing. So those are read-only layers, right? And so they will all look at the same cache directory. So if you say, if you run just on your terminal, on your computer, you run singularity, run some Docker container, and you do that twice on two separate windows, it'll just look at the same cache directory since they're read-only. They'll both just read from it at the same time. And it works just as you would expect, like doing it simultaneously with Docker. So the question is, when I'm talking about portability, does it mean something else rather than Linux? And the answer is essentially no. It's just for Linux. I mean, all the HBC environments that I've personally been involved with have been all on Linux. And the goal is just to make it portability between different versions of Linux, different distributions, different libraries, different users, different sites. I think that was all the time we have. Yeah, we're out of time. So thank you very much. Thank you.