I suspect yours will be easier He's got an unfair advantage All right, any last questions Well, I'm sorry repeat again To change the shell Oh, oh doing that sort of thing. Yeah, so that that's uh another one of those things where the terminal would need to Provide the shell that you want. So like if you want fish instead of bash I will say our current images we're building do not have fish And they're probably not going to because some of that comes from that base os the root file system there That's as a snap, but you could absolutely have your own terminal app Packaged with fish included or Again back to the containers spin up a container App get install fish and you're done And when these containers are created like you can map in your access to your gpu your home Your home folder all those sorts of things can be mapped directly into your container. So it feels at home That's what I do today. All my containers are so eggy Okay, so any progress on running a container inside a container his reference case here is like what we do today would steam Right? So steam downloads a pressure vessel runtime and it basically Creates its own container and in our case we're running steam in a snap So it creates its own container inside of that. You have another use case. It's very similar You know as a generic kind of case that's complicated Like we've had to do some special stuff in snap d to make all the pressure vessel stuff work I would argue we probably could make the steam support interface a little bit more generic And make that work for what you're doing as well. So So Yeah, so precisely I think I mean maybe what we should do is actually look to make a more generic not not call it steam You know type of interface that allows all of that stuff that's necessary for pressure vessel And then it could be reused for snaps that are leveraging your framework. So let's let's talk about that a little bit I mean, I'm here today and tomorrow. Let's check I think it would be very compelling. So I'd love to hear more about that All right, I think we're out of time Thank you all. I appreciate it Hope you enjoy the rest of ubukan and enjoy your lunch Thank you very much for that. That was really good for me. I've been curious about this since Nathan first drew my attention to it. It was well done. We do have a one hour break Enjoy two o'clock back here for orchestrating your home devices with juju Check check. All right, sweet Hey there everyone. Welcome back from lunch. We're just going to give it a couple minutes here for more folks to kind of Pop in as they get back from lunch. So Excited to have you here for the afternoon session Thank you all for coming back. I hope you had a great and filling lunch We have some more talks for you now so you can fill your your brain So our next talk is charming your home network with juju and presenting that is alex slow and callan covax All right Thank you and uh, welcome and thank you everyone for coming to our talk Um, like you said, i'm callan covax. This is alex slow and we're here to talk to you about charming your home network with juju And a little bit about us before we get started alex and i are both software engineers at canonical on the starcraft team And as you can see by the wild card glob The starcraft team works on charm crafts rock crafts and snap crafts And these are all of course packaging tools used to package different things And just a quick overview of those snap crafts Which you're probably familiar with is our most popular tool for packaging iot and desktop applications Next rock craft is for packaging container images So this is similar to using docker to make a docker image except rock crafts makes oci images And most relevant to this presentation is charm craft Which is used for packaging server applications and the operations code for those applications And we'll get into more detail about that shortly But before we go any further, let's talk about you. Who are you? Perhaps you just want to run an application at home Like a media server and you're not particularly interested in the installation or the maintenance You just want it to work if this describes you then then juju and charms are not the right tech stack In this case just use the apps recommended installation methods because they work they have the most support and They're the easiest to use So we're not here encouraging you to move to juju if you have something that already works But perhaps you're someone who's heard about these technologies like juju and charms And you want to try these out at home and experiment with them But you're not sure if it's even possible Or what charms you would deploy at home because when you look online That's things like telco apps or grafana things that maybe don't make too much sense to do at home Well if you can relate with person two, uh, then you've come to the right talk We're here to show you how you can experiment with charms at home with apps that you would actually use at home In this presentation, we're going to demo juju and lxd to deploy some common home server apps So we'll show qubit torrent the torrent client Open vpn as a vpn client home assistant for home automation and jellyfin as a media server And we're also going to show you how you can charm other applications and how to contribute to the ecosystem Yeah, so a quick note on that Um juju has sort of Two types of charms that you can run you've got machine charms and kubernetes charms Which are roughly what what they sound like machine charms can be virtual machines machine like containers like With with xd they can even run on bare metal hardware just depending on on What kuban or juju backhand you're using Um kubernetes Trams are specifically for a kubernetes pod so a machine a machine charm you The charm runs as root in uh in this machine you uh, so You can install software you can add repositories and anything that you need The kubernetes charms have access to kubernetes. They can Uh, they can manage containers and And images and stuff like that. So it's it's a difference Between a vendor agnostic All of my cloud vm's as code and vendor agnostic all of my kubernetes stuff as code But juju can handle both of them And a few disclaimers as well. Um, first this is a proof of concept This is relatively new area of interest and in the charms we created for this demo. There's plenty of missing features And then second the tools like juju are certainly overkill for using at home And that's okay because that's not exactly the point And speaking of juju, let's talk more about it So juju is an open source tool to deploy integrate maintain cloud applications on any infrastructure And when I say any infrastructure I mean any infrastructure As you can see juju is designed to run on many different back ends and this isn't even a complete list And uh, it abstracts this so when you write a charm, there's no back end specific code to deal with The only exception to that rule is what alex just described is that there's two categories of charms the machine charms and kubernetes charms um For our deployments, we chose lexity as the back ends and which is a tool to run containers and virtual machines Some of the advantages of using this is that it's lightweight. It's easy to use Uh, but the downside is that it's integration with juju is primarily for local testing So some of the features available and other back ends aren't implemented for the alex d back end And now alex is going to show us how to bootstrap juju with alex d Yeah, so Top right corner in there. I'm going to be running h tops. Uh, so you can see What system resources it's running? I am going to do this inside of a brand new Ubuntu 22.0 for virtual machine So the first step is just Creating that virtual machine with four CPUs and and eight gigs of RAM And you'll see sometimes a 10x or whatever Flashing that's just because I don't want to bore you with my slow internet so We've got we've got this machine. The first thing we need we need to do is update to the latest lexity The ubuntu 22.04 comes with the lts version of lexity makes sense. So lts ubuntu comes with lts Lexity, but we do use some features That are only available in newer versions. So just Downloading that The second thing to do is To install juju and hey look, it's a it's a snap as well. So Pretty pretty standard in install so far. Um, I actually also have juju running on On this laptop, so so if if there are questions later on I can I can do some live demo there juju is a strictly confined snap So Comes with all of the the pros and cons of that So i'm initializing Lexity there and then literally just running juju bootstrap lexity This will create a This will create a machine inside of Your container inside of lexity, which actually runs the controller One of the cool things about juju is you don't need any additional Infrastructure to to manage these clouds it will manage itself from within Whatever cloud you're doing So That that was it we we have juju up and running on this blank machine Now what else is juju? It's an orchestration engine for software operators or charms and with that said let's talk about charms Imagine you want to deploy an application. There may be a lot of steps. You have to install it update configuration files and then enable and start services And then when it comes time to upgrade that application, uh, you'll have another set of commands to run stopping the service backing up configuration data and then Doing the upgrade and restarting And if you're doing those things often enough you're probably going to write a script for them Now what if there is a way for everyone to come together and collaborate on those same scripts to operate an application? The scripts would be configurable and extensible And what if there was a framework for those scripts to ensure that they were consistent and testable? Well, that's what a charm is a charm is just a package of these operation scripts and tests Uh, so in other words charms are packaged operations code for an application And this brings us to charm hub, which is the official repository for charms Uh, it's similar to docker hub, but for charms The source code for charms are hosted somewhere else like github or gitlab But the charms themselves are posted on charm hub and juju can automatically download charms From charm hub when you're going to deploy And finally we have charm craft charm craft is a tool to create build and publish your charms so charm craft can Create an initial template for your charm. It will build it in a clean and reproducible environment It will provide ways for you to test your charm and allows you to publish to charm hub And it has the same user experience and design language as snap craft and rock craft So if you're already familiar with one of those other tools, you can quickly learn how to use charm craft Let's look at what's inside a charm Um starting from scratch charm craft the knit just creates a simple charm from a template And first we have the charm craft dot yaml file And this is where you define metadata like the title and summary and description, but it's also where you define the build process charm craft organizes the build process into a series of parts for example, you could have a part that clones a git repository and Builds a go application from that and another part that builds a python application from code in the source directory here And then you could package both those together in your charm Um now even though you can do that you probably won't Charms are definitely the simplest artifact of all the crafts applications the star craft team manages So even though it provides this flexible and customizable way most charms don't do it They just will have a single part That uses the python code in the source directory here Uh next are the license and readme files and requirements which are self-explanatory enough. So I'll skip those Um, and then we have the source directory where the operation code lives And alex is going to tell us more about that in a moment. Um But finally we have the tests and when you first initialize a charm These are just stubbed out tests, but this emphasizes that charms are built With good coding practices from the start And coming soon charm craft will have a way to do more comprehensive end-to-end tests as well Yeah, so uh, this is a screenshot of my favorite text editor With the open vpn tron that i'll be demoing later It's It's just a pretty standard python class there the the only real magic is here in the in the init Where you use the operator operator framework that canonical provides to Tell it to observe certain events So juju is is for the event based in this case The events we're observing are the install event the config change event the start event and the stop event which is Feel like those are all self-explanatory and the and kind of the The minimum set of what you want to be able to do with the service Going into the the audience and solve Some might notice that that i'm literally using sub-process dot run to to do african update again this This charm is us basically a script that runs as root for in your machine So Open vpn is already in in the ubuntu repos for me I don't need to do anything fancy to to get open vpn this so Why not just build on top of uh of what's already there? Everything else is is just managing that's that system deservers and handling things When you pass a change configuration from Some client Well, some some administrators client machine Somewhere in the universe to the to this charm Everything else is just you know system control start system control control stop here I'm sure I'm sure everyone's familiar with everything that's happening here So we've got the we've got that charm written now. I'm going to Package and deploy it same same structure on the on the screen that I showed you before Yeah, come on network. I I think funny cats deserve priority. So I I totally get that If we try just going back Yeah I thought here I thought that uh connecting to the ethernet was a great idea, but You know what? Well, I was on the ethernet. I'm just gonna switch to wifi. There we go Okay, so I'm literally running charm craft pack in the in the directory This sets up a container that that has just a basic build environment So first time you do that it'll it'll take a little bit of time, but that container does get reused Talk about about a minute To do an initial build there for me. Then I add a model in juju For whatever whatever I want to call the model and deploy it I deployed it with with a I deployed cuba turned with with a bit of configuration This varies from charm to charm So you'll have to look at the charm docs, but that fired up a lexity container for for me installed cuba torrent and Set up the the web interface and everything I got to set my own username and password There and look i'm downloading ubuntu Um, oh, so so in that case I was downloading the ubuntu torrent. Sorry Um We do We support ubuntu lts CentOS 7 and almalinux 9 currently As the machines that that the charms can run on And but then It's it's per back end how how that gets gets downloaded So like lexity will download the ubuntu image once and and then we'll just use that to fire up new containers No, but we do But we will be happy to Except prs I I am the maintainer for for for charm craft. I'm definitely happy to to accept prs on that front so, um another example, let's say you want Now you've got your bit torrents client going you want to post some home videos for yourself and At and your family, you know Some some funny videos of your cats We've got jellyfin here already available So this is I made a model just each model is you know Things within a model interact things between models. There are additional layers to that to that interaction So so I made a model I deployed jellyfin from charm hub was three three commands and As soon as it In this case the charm actually Uses the official jellyfin After a app repository it adds that into the machine and then it installs jellyfin But but then in the end I've got I've got jellyfin there it defaults to using slash srv And we've also got some other charms online that that lets you Mount a network share to Slash srv so so that you can use your Home nest as the actual location for serving that Now one more thing is that that sometimes your isp rarely really hates linux isos And so they won't let you you use bit tarn because as we as we all know linux isos are the only known use case for bit tarned so then you might uh, so then you might need to Set up open vpn And put put a vpn client on it. Can you wind a little bit to? About 15 seconds in At post there. Yeah So in the in this case, I am Installing the open vpn charm And i'm telling it to deploy two machine zero which is the same container that That already contains My cubit torrent charm So when it's doing that it will fire up Open vpn just for just for that container you can Okay And It won't affect any other machines Are in juju or indeed the machine that's hosting it So so now I can once once again download Ubuntu, but Just checking that i'm using the tunnel interface And This will then Make sure the cubit torrent uses that tunnel interface that tunnel interface is being set up by my configured open vpn charm and so when I Download ubuntu again I'm very thankful to Whoever was seating that ubuntu arm 64 Torrent that particular day because I downloaded it about 10 times I I'm taking just a little moment to get a few peers there not revealing anyone's ip address, but um The this presentation and the videos will be available so you can check more thoroughly later on But all of this is going through through my vpn endpoint None none of that traffic is actually going directly to the stuff just To to prove that it's actually using the the vpn. That's kind of important All right, um and in this next demo, we're going to be showing home assistant The deployment is similar, but what we're going to show at this point is one of the limitations we ran into And With most back ends with juju if you want to expose your application To the network you configure the ports when you're building the charm And then you type juju expose when you're deploying it and that would expose those ports to the network Unfortunately that feature is not yet implemented for the lxt back end. Um, so we kind of have to do this manually And so here, um, I deploy it I immediately list my containers and I see the first one Which is where juju itself is running as a controller And then a moment later, uh, you can see it spawns up a new container, which is where it's going to deploy home assistant And All i'm doing is just doing a Back end specific command here. So i'm exposing a port For this container And then I'll just do juju status and wait for it to finish installing In this particular case, uh, kelhan used the home assistant snap Yes, and I will I'll talk about that too in a minute. Um, But yeah, uh, and then home assistant is right there. So I'll just brief through the setup and the log in and There you have it home assistant as a charm Um, and like alex just mentioned oops Home assistant is available as a snap. So, uh, one of the interesting things That you can do is if something is already snapped. It's really easy to charm Uh, because all the work in the details are already taken care of there in the snap itself So if you're interested in charming something I recommend you check if there's a snap for it already because You could have it deployed in truly minutes Yeah, easiest way to uh to make a charm is to charm a snap second easiest way is to charm something that's That's already in the repos Speaking of which We're starting a little organization on github. It's called charming cottage um If anyone's familiar with snap crafters, I'm sure most people here are familiar with snap crafters to some level um We're kind of trying to do the the same sort of thing but only for charms for your home For home stuff if you if you want to Charms something that you know Runs on big iron servers and whatever that's wonderful highly encourage it Charming cottage isn't the right place for it if you want to charm, I don't know pie hole And and run a charm version of uh of uh of pie hole for yourself One good luck. Uh, we both failed to uh to properly charm pie hole It does a lot of really confusing stuff But uh, but to Yes, this is this is the right place. We will be uh, we will be happy to uh to host that charm to uh to Provide help For you Any anything that you that you're wanting to do for the home use stuff Charming cottage is is there. We've got discussion Forums available and Yeah, if you if you want to play around with juju on your home network, this is This is probably the easiest way to do it and uh all the charms we demoed and more We already have on this organization So with that said, thank you Any questions? Yes Go ahead Thank you So just had a question about how uh someone coming from the enterprise Where we don't you know, unfortunately, we don't use a lot of You know fancy tools How would somebody integrate something like this? So I saw that you guys talked about azure so, um juju is It is basically a level above your uh your cloud if you if you will so Let's say you've got a bunch of stuff Both in azure kubernetes and there's a bunch of azure Azure virtual machines You can bootstrap a juju instance in your In your azure stuff and you can start migrating vms and kubernetes pods Into juju Which uh, which is a matter of making sure they're they're properly defined for uh for juju So that juju knows what's going on with them and then just telling juju and manage this By deploying the charm Easiest way to do that on it honestly is Get charms of of what you're doing Deploy a new and new machine with the charm Version of it that and and then just migrate over But that can all happen inside of that same cloud Yeah, and uh to build on that too. I think that One of the goals with this too is that it is hard to like bring something new to the workplace And so many of these like open source tools. It's things that developers play with at home And then they want to bring it to the workplace. Um, and Yeah, it can be very challenging. Yeah, especially with the bigger tools like this Yeah, the the point of charming cottage is to get you familiar with the juju tools and and stuff like that I in Honestly a low stakes environment because if if you Mess up your personal streaming server. Well done. You can't watch that at that home video again tonight, so I noticed when you were doing the the charm craft in it that I created a good amount of python files Is python the only language that is supported by that? so Uh, so, um, I'm going to say yes and no Python is the uh, it is the Most widely used way of doing it and like the operator framework is written in python for uh for python objects Stuff like that, but uh, but You know how an oci image is is just a It's just a fancy toggle with the different layers Uh, a charm is a fancy zip file With each of each of the events being any executable that can that can run on that on that system so we There are we have charms like that are still running within canonical that uh that are A set of bash scripts within a zip file and and then a couple of yaml files to give them at the metadata You could you can do that if you wanted to you could uh, you could write the actual event Software for your charm in go or rust or whatever and charm craft can pack all of that for you um It will be a little bit steeper steeper learning curve. I would Recommend that for one's uh first charms at least use use the operator framework and uh and uh What's provided but once once you're familiar um Please do weird stuff with charm craft. Please, uh, please break it in weird and wonderful ways and please Send me Many very detailed bug reports Yeah So yeah, I definitely will agree with your comment there where you can do like bash based scripts But basically, you know, you have like a dispatch script inside your charm and then you can write that in any language you want but um I do find the one benefit of like using like the operator framework and using python is that it's very easy for like other people Who are familiar with charms to contribute um because you basically have a common standard language for being able to kind of collaborate together So, you know start hooks behave the same way Configure hooks all that and even custom events as well. So One thing that's really interesting about charms is that they can basically share libraries. So similar to say like, uh Python package dependencies almost they can share libraries and then let's say like, you know You want to integrate like an identity server like you want to have I don't know Linux users for all your family members in your house You could then like take a library from like one of those ldap charms And then you could put it in like your home assistant or something like that And then they would be able to kind of speak a common language that allows them to be able to Configure in kind of mount the Yes, but I also did actually have a question But I wanted to know how long did it take you to put together all the videos and the charms for the home assistant and whatnot um The charms Took us a little while because we were experimenting with With a lot of stuff The videos that that that I did I did all of that in a weekend Yeah, we um We have tried quite a bit uh, alex spent a bit of time using open stack or trying to set up open stack for this as well um And I struggled to get open uh open stack running at at at home But then again, this is also someone who has managed to install gen 2 but not arch Or susan So I noticed you charm craft and it also creates some unit tests and some integration tests just some boiler plates for those Um, are those ran when you're creating the charm? Are those ran when you're actually deploying it? When are those ran and and what are the use cases for those really? So, uh, the full template for uh, that charm crafting and it gives you includes a Talks.ini file With that a pre preset to to run your unit and integration tests there. It also includes Static linting with rough and uh, my pie. Is it my pie or pie right? I think we use pie right on it. I'm not sure So so that's uh, so you would use talks to to run those We have an upcoming Feature that'll be a a charm craft test command Which will then run not just Those unit and integration tests, but uh developer defined end tests where you have the packed charm and uh, and you're running That inside of a juju environment Doing certain poking at it. It's uh, et cetera to uh, extend on what uh, what jason was uh, was talking about to, um I am currently trying to figure out how to Detach jellyfin from its database so that I can use the charmed version Of my sequel or postgres as the database for uh, for the jellyfin charm And I'll be used to be using them the the database integration Libraries for that so that I can you know run the run the database on a separate machine. Yeah Uh, the more detaching of such things that you do Aren't you running into the possibility of getting to situations that the original vendor had never in in their life tested for? I am um, this this was this is why we get uh, we gave that caveat up upfront None of the none of the projects that we're doing that we're doing this for Or Maybe one of them's aware of it. We certainly haven't Told any of them. Hey, we're trying to do this and we're not trying to make this the official way to do it I'm uh, I'm doing that for my own interest in And using that for uh for jellyfin not I've heard that home assistant people can can be rather belligerent against people who are trying to repackage their software Yeah, we uh, we've we've seen that and we're uh, what we are trying to work to work with on things that are as big and complex as uh, Home assistant and as pi whole is rather than rather than trying to repackage What what they do? Every time work work more on providing this charm that will use the their official packaging if at all possible so I've got a work in progress for for pi whole that actually Uses their install script inside of the charm Um The their install script is very much designed to to be user user interactive and charms are very much not So i'm doing some weird hacky work to to work around that but but The the the preferences is definitely Try to use as much of the official stuff as as possible And charms are a way to wrap that into Something that is fairly standardized Any other questions? All right. Well, oh, thank you very much everyone. Thanks folks Okay, so we have a bit of a break here and at three if you wanted to learn how to use uh Superpower computing to do science you will be in the right place. We hope to see you back again here at Three so get some water stretch your legs. I'll see you all soon Since there's a bit of an extended break if you want a swag We do have some more stickers and I think a couple mugs and journals Now's Now's the time Yeah, it's on. Okay. Hello. Hello. Can everyone hear me? Okay, sweet Okay, so our next talk is accelerate your time to science with the boon to and open on demand And this will be given by none other than our own jason Neutroni All right. Thank you. Nathan. Um, so why don't we uh get started in here. Um, first of all, thank you all for being with me this afternoon Um, so yes, the title of my talk is accelerate your time to science with the boon to and open on demand So first things first. I'm just a little introduction. Um, if you weren't here for the impromptu sort of talk that I gave this morning My name is jason c. Neutroni. Um, I am an hbc engineer at canonical And I am one of the not so ancient elders of ubuntu hbc Um, and so if you're kind of you know wondering what is you know ubuntu hbc You know, what does that acronym mean? Um, at least for the first two years You know, I was like, oh, everyone knows this but then I quickly got reminded that sometimes when you don't have the proper context acronyms are a bunch of nothing so Give a quick little definition here. Um, what is high performance computing and you know, why exactly should you care about it? It is a paradigm where you utilize, you know, supercomputers Computing clusters and grids to solve advanced scientific challenges. Um, and so, you know, you would think like, oh, you know What industry specifically leverages this technology who uses it?  So, um, some examples of possible industries where hbc is really popular It's like aerospace so they can use it for like modeling jets and whatnot Agriculture, um, they can kind of use it to predict weather trends.  Um, and then it's also used in finance as well So it can be used for like advanced prediction, um, fraud detection And then also just kind of modeling potential future financial trends based on kind of various compounding factors And then if you see that qr code there if you go ahead and scan that that will just kind of take you to A page on the ubuntu website. Um, that'll just kind of go over some more Use cases of where high-performance computing is, you know, really popular for use And so then the next thing that's probably like is what does ubuntu have to do with high-performance computing? And what exactly is ubuntu hbc? so here Ubuntu hbc is one of ubuntu's newer community teams We've been around for almost a year now or one of your birthday. I think will be in may If you read the text on the front there, um, basically if you go to our team page I think it's on both launchpad and on the ubuntu community website Um, it's just basically a definition of what we work on and so mostly our focus area is You know, how do we make ubuntu a better operating system for doing high-performance computing? Um, and to kind of give a little bit of context about how this community team, you know came into creation It actually started at the 2022 ubuntu summit Two organizations met obviously the first one was canonical And then the second one was this other organization that specializes in doing high-performance computing on ubuntu Called omnifactor And so at that point canonical, you know, at least me we were working on Doing hbc. So we were doing packaging writing some juju charms for from the previous talk also Once again packaging just lots of packaging And then, you know, we kind of got talking, you know, we kind of realized that we had a lot of common challenges And then at that point what you see on stage left here Is uh ubuntu hbc. That's kind of our community logo right there We decided to start a community started a community team that was kind of open for anybody To join and contribute So you might be asking yourself, you know, if you're thinking back to the title of the talk, you know What is open on demand have to do with um ubuntu or hbc? so You know When you think of a super computer you look at a picture here, um, and you can see what do you see you see a lot Of computers. Um, and so super computers are massive machines So you think your laptop will think that you have an entire building dedicated to just running a Single computer. Um, it draws many megawatts of power You actually have to build it in very specific places because not every power grid can support running an hbc system Um, and you have thousands of compute cores, you know, you have thousands of people who are all using resources on that system and so then Maybe some not so humble artwork here. Um, this was made by a friend by the way.  Um, and nobody can stop me But an often problem that I find um, at least you know, my early years of hbc when I was in consultant is that Most people they start out feeling like this and what ends up actually happening is this So yes, so why do most hbc users even though they're and don't get me wrong They're very bright and smart people, you know, they're like phds, you know leaders in their field They know how to do everything, you know, they're masters of You know biology they're masters of genomic analysis and yet we all kind of have this patrick accidentally nailing a board in said moment And you know, I'd like to maybe make the assertion about why they struggle to use supercomputers is because of this Or actually wait a minute. They got a little outside. Okay, so hbc is everywhere um and so You know yet a majority of individuals struggle to get started So why are some possible reasons why new users could, you know, struggle to use hbc systems? So I kind of alluded to this a little bit earlier Um, but what I wanted to specifically mention is, you know Some maybe naive reasons of thinking or at least, you know, easy Areas the point at um, so the first one could be is like, oh do people just struggle because it's a different tech sec so You know most maybe modern computer scientists are very familiar with say like the python programming language But in hbc, you know, language is just like c and for trainer much more prevalent and much more used Um, you know, you could also maybe, you know, make the challenge that there's a lack of adequate user New user training. I often find that most hbc institutions actually have very excellent documentation How to do at use cases, um, you could also maybe challenge is like Maybe they just don't understand the problem. They're trying to solve and they don't understand like You know how to, you know, correctly model that on the computer You can maybe say it's like the system too advanced for these noobs Or you could say like are they just lifelong windows or max users and they're just not familiar with the linux environment Or they're just not very smart And so what I kind of gave a little bit of a sneak peek to um in the Or the next slide, um, is that I'd like to make the assertion that what the problem really is is this the terminal And so you might look at the terminal and say why does you know Why is this guy up here on the stage think that the terminal is the problem? Well, maybe a rough show of hands You don't have to raise your hands if you want But how many people would get the idea that they have access to a super massive computer that is capable of solving The most advanced problem that you can think of if all you see in front of you is a black box That's telling you the type something Who thinks they could figure that out? Okay, so all right. Who's been using linux for like 10 years show of hands. Oh all the same people So Yeah, yeah, so you don't even know a computer you're talking to from that prompt and that's kind of a problem is that It's very hard for the user, you know Or a new hvc user who doesn't have 10 years of experience with linux like most of this room here They're just trying to get something done because either a you know the principal investigator or even their boss or even maybe they're just trying to Test something out try something new and they just you know, they get access to a super computer and they say like You know type these things, you know, click a couple buttons and like say putty or another ssh program And you just get given this black box And so that's kind of what the problem is is that you know, you can Look at it and say in the beginning There was the shell and so now here's the case against it And so why might the shell or the terminal not be The best entry point for new hvc users. Well first you can kind of think of affordances And so if you're not exactly familiar with what an affordances It's basically, you know a device, you know, maybe say like a phone or a laptop or even your car You know an affordance is like it telling you that it's capable of doing something So you think like a steering wheel, you know, you can steer the car with the gas pedals You know you can accelerate and so with an affordance, you know when you go into the terminal What it tells you is that you can type something but what it doesn't tell you is what exactly to type How to efficiently type it or how to do anything correctly And then the second thing is too is that, you know, the terminal can be kind of a learnability challenge And why can it be a learnability challenge? Well, if you don't even really know what the terminal is like, you know If you're maybe like six years old and just open like the terminal, you know Your mac laptop for the first time or something The shell and it's complementing scripting language doesn't necessarily Teach you anything right off the bat And so the idea is that if you want to learn something new you have to know The right places to go like you have to know how to open manual pages You have to know how to use the internet how to search, you know You need to be able to go on the stack overflow and you know Who's blowing smoke and who actually knows what they're talking about And so how do you learn what you don't know if you don't even know how to learn what you don't know And then the next thing you could also kind of argue against the terminal is consistency And so the issue with like, you know, the terminal being consistent Not necessarily say the user interface of it itself But if you commonly look at like a lot of command line applications You know, you find that depending on the framework that they were written in They kind of a very specific style guide So, you know, I guess kind of the low hanging fruit here that, you know I could kind of pick on about usability is that not every CLI application kind of has the same interface for just kind of getting the help information So, you know, some commands, they only support like the short option dash h You know, some only support the long option like double Help and then some, you know, they actually have a built-in help command that then offers extra, you know Information for the sub commands But it can be very frustrating when you're first starting out and you think like, oh, every application takes this dash dash help option And then some applications don't even actually offer a help dialogue And then the last thing is, you know, kind of visibility and, you know, what do I mean about Visibility in the terminal, I basically kind of Allude to it as it doesn't really show you what to do next So let's say that you go online, you get something correct like how to read a file Well, what do you do after that? How do you copy that, say, into a clipboard?  How do you further process that using like grep or something? Or how do you, you know, move data around from one directory to the next? It doesn't really tell you what to do And so now that I've kind of maybe brailed against the terminal a bit, you're likely asking yourself You know, so what exactly is open on demand and what does it do? Open on demand is a interactive portal for accessing HPC resources over the internet or through your web browser And so essentially what open on demand and if you scan the QR code there, it'll take you to their main website Is it is a web-based application that kind of moves beyond the terminal that I like to say And it kind of now focuses on providing a visual interface for taking advantage of your HPC resources So okay, I saw some people scanned it, but now Yeah, basically what it is is a web portal And so what I have here is kind of some example screens of what open on demand specifically is so for here The best one that I really like to show is that it's just basically an interactive Interface for figuring out what's going on at your cluster So here what this screen is showing is like listing jobs So typically in HPC systems, not everything is totally synchronous typically Will happen is that you submit up to say like a scheduler or something And at that point then that scheduler is responsible for deciding when to run your workload based on available resources and other constraints And so in that case what this screen is doing is effectively showing you where those jobs are but Now if you did it say in the terminal, it would just basically be like your job submitted And doesn't tell you anything else it doesn't tell you if it's running It doesn't tell you if it's queued up It doesn't even tell you if it's actually good enough, you know It could get totally rejected and then you're just kind of left wondering what happened And so, you know, there's there's there's quite some several benefits of using an application like open on demand for accessing your HPC resources So for example, the first thing that I really like about it personally is that it has support for interactive applications So it's not just open on demand that you're going ahead and running on your system So for example, you can get graphical interface to a lot of really popular data science Applications, so some of the examples I have listed here is like jupiter You can use open on demand to queue up jupiter sessions on your super computer You can also use it to queue up a vs code server So for example, if you want to develop code directly on the system, so If you're like someone here is really flexing their h100 gpu collection on me I saw him laugh you can actually go ahead there. That's how I know it landed But uh, you can launch your vs code server directly on that node and then take advantage of those resources right there And then there's some other popular applications as well. So for example, there's like pluto.jl Which is a similar application like jupiter, but it's dedicated specifically for the julia programming language There's also like shiny Which is like web apps through the r programming language Which is really popular in data science and then you could also like look at like r studio Which is like an ide for the language and then matlab, which is a popular application among Uh Scientists as well And so then moving over to the next category of benefits here You also get to take advantage of say like interactive desktops. So one thing that's really nice about Open on demand is that it gives you the ability to get graphical sessions through a vnc on directly on your compute nodes So if you're not really a terminal person if you're not interested in going You know through typing in a bunch of text and kind of you know being a bash scripting king Instead what you can do is you can actually launch a desktop session directly on your cluster And so in this case, um, some of the examples that I have here, um, it's like Default ubuntu so you can run default gnome. Um, and then it also has support for two other Desktop environments, which is like x fce, which I've also commonly heard referred to as x-force I don't know if anyone else calls it x-force, but that's what I've heard sometimes and then also mate. So Does anyone actually call it x-force or Nobody calls it that nobody calls it that if I just been wrong this whole time. All right, okay Yeah, and so then the the the last thing here the last benefit that I really like to say is a scheduler integration So you could maybe argue that schedulers is a bit more specific Hpc knowledge, but one thing is that everybody kind of has their preferred scheduler that they like to use so some of the Logos that I have up here the first one is for lfs. Um, that is a workload scheduler that is produced by ibm And then there's slurm, which is based off of the drink from episode 10 of season one in futurama I'm able to read that back because I get asked that a lot. Um, what is slurm? And it's like it's a simple linux utility resource manager But they prefer that they don't call it that because they want you to call it the slurm workload manager now And there's open pbs, which is a open portable batch system. It's similar to slurm, but it has some difference in tactic differences Um And lastly it also integrates with kubernetes So if you want to be able to run your jobs on your kubernetes cluster You just basically configure your scheduler there and it'll go ahead and launch a container up. That's pretty neat And so now that I've kind of you know, maybe gone through the benefits of using open on demand a little bit. Um Oh, jeez. I just noticed that typing error, but it's sorry It's like, uh, how is open on demand important to the bunch of hbc community? And so, you know to give a little bit of historical context here If you kind of look at this diagram that has been through a couple iterations here We as a community and ubuntu hbc are currently working on developing this Charmed application called charmed hbc. So charmed high performance computing And so exactly what that is is that we're looking to leverage, you know, say the work of Callahan and alex here using, you know, juju and charms and all that to be able to deploy a fully functional High performance computing system practically basically wherever you want it. So it could be on like azure cloud It could be on aws or even on a local lxt instance on your laptop and so The current this is kind of the current architecture of charmed hbc So as you can see here kind of on stage right The first part is traditional way of accessing it is via the ssh protocol You'll basically go into say like a login slash head in node or a head node Um And then that's where you'll see kind of that terminal experience where you're now on a host And then you have the challenge of knowing like, okay, what computer am I running on? How do I leverage those use resources and how do I even know that i'm leveraging the best way possible? And then after that, um, kind of from that login slash head node there, um, you're kind of then interacting with the job scheduler So in this case, um, we have chosen to use slurm as our job scheduler implementation And so from basically that login slash head node, um, you'll submit jobs up to slurm From then which at that point, um, typically what's slurm will then go ahead is, you know Take inventory of all the available compute nodes, which are on kind of stage left here Um, and then go run the jobs on those and so compute nodes can be typically composed of very different resources They can be um homogenous or heterogeneous depending on kind of the clusters needs slash architecture Um, so for example here, we have like this back package manager Which is a like user level application that can bring in all sorts of different packages Quite a great community. I love working with them And then it can also have some optimizations for like network and whatnot So you can choose to use like high speed ethernet or you can also choose to use something like incentive band and so then Now that we've kind of looked at what specifically slurm is, you know Now we have some other auxiliary applications that we have as well Um, so for using users, um, we use ldap. Um, that's just basically I forget the acronym off the top of my head now that i'm on the stage But that we use that for kind of providing the identity and access management So that's how we're able to Um, you know make sure that users are the same person across every node on the cluster Um, and then typically we'll also have some kind of storage implementation parallel storage And that's basically how we make sure that um files are available on all of the compute nodes So, you know, let's say that somebody goes into the login head node and they submit A job and it goes to compute a But then they also have you know another resource Or another job that they need to run that also needs to use something that their job that on compute a is using If it goes on compute be it'll still be accessible and all that And then you might see my sql up there kind of in the top there What my sql is mostly for is just kind of collecting um cluster data and usage data And then that way it's like if you're a site admin or something You're able to then just execute queries against that to see what your clusters overall usage is or find out If you're not charging your users enough And then lastly, um, we kind of have two components We have cause which is kind of an acronym for canonical observability stack And so that's just basically a suite of tools for Collecting metrics and kind of viewing metrics about your cluster. So for example We could collect health information about all of the compute nodes pipe that in the cause and then you go to like say Gravana dashboard and that'll tell you everything that you need to know about that compute node And then the last thing is that we have Moz which is like metal as a service And that's basically our backing cloud implementation that we could use We're able to slot pretty much anything in there, but in this case Moz what it does is it allows us to request like bare metal resources And so now With the sunglasses and finger guns emoji charmed hbc with open on demand And basically how this implementation is different is that instead of having that Traditional ssh login head node where you have that shell environment Instead now you kind of then have another entry point that's through the web using open on demand So basically you just have the firefox logo there You go into open on demand and then you're still able to kind of fully leverage your Systems resources so you can integrate with slurm, you know, you still have the same users You also have access to the storage that the cluster is using Yeah, so it's pretty nice The next question is is that now that I've done a little bit of talking about open on demand and understanding So now where does the work currently stand on integrating open on demand into Charmed hbc So this is kind of something that we've been working on as community is getting this application There's quite a lot of steps open on demand has pretty complex architecture So we've been making steady progress on it. Um, and so yeah, basically, you know, the short of it is that we're hard at work Still working on it yesterday when I was traveling here so A couple of things that we have done and some things in in-flight And then there's other things that we're still kind of hammering out and working on So the first thing is that we have a functional snap recipe So we're able to get a successful building snap package that has all of the services that open on demand requires Later in this presentation, I'll show it but it requires like three different web services It's a bit interesting how that works and we had to figure out how to cram all three of those inside a snap and then also We worked on developing some abstractions for configuration Specifically it has a lot of yaml configuration files I think you can do up to like 100 depending on how granular you want to get in like The config dot d directory So we developed those abstractions to make it really easy to do that using snap ctl So somebody wants to basically say like oh, you know, I want to be able to I want like jobs to have this lifetime or something you can directly configure that through snap Um, and then we also have a working apache htp integration for serving the web application itself open on demand Um, so open on demand uses apache as the primary entry point And so we have that working now so you're able to kind of start getting to some of the main interfaces And now we're kind of have the in-flight parts and some of these are a bit complicated So the first thing is that we have to now add a um open id connect integration So basically that'll be for kind of the 2fa slash login service So we have ldap working But we need to be able to kind of be able to authenticate our users as they access open on demand So working on that, um, you'll basically be able to use any open idc provider We'll be starting with dex, but then probably looking at like orihydra And then also, um, the other part that we're working on now is kind of the nginx Um integration for running like user confined web applications So for example, like when open on demand schedules, uh, and runs jupiter Um, it's using nginx to basically proxy that back to the user Um, and then it's confined as that user on the host. Um, so that They can't just like, you know, be like, oh, I'm root now and I'm going to delete all your data And then, um, two things that we kind of have, um, slotted off for the future then Is that we want to have initial support for a base set of interactive applications So there's a couple of defaults that they recommend, um, specifically like jupiter, um, vs code server And then they have some other default apps available as well as kind of reference point And then we also want to develop a juju operator, which is basically a charm And then that way it'll be really easy.  Well, you know, I just want to give a little bit of an overview of the architecture of open on demand So on the kind of Stage left side here, um, you'll basically be looking at the auxiliary services So these are things that are kind of outside the domain of open on demand But things that open on demand integrates with And then on stage right, um, those are components that go into open on demand So kind of going from top to bottom here If you look at the first part, um, you see the front end proxy And what the front end proxy is that basically the Apache service, um, that serves up the Ruby on Rails application that, you know, you kind of work through a user would work through to Request jobs, you know, log in, um, they can even get an interactive terminal if they wanted it And also just like view status of services And then kind of then beneath that, you have the per user aspect of open on demand So in this case, what you have then is like that back end proxy That's where the engine X service that we're currently working on comes into And then you have the application runner, which is a fusion passenger That's like a Ruby application for starting it And then the application itself, kind of all the way at the bottom And so per user, what that is, they commonly refer to it the open on demand upstream As per user engine X process Is it is responsible for basically queuing up user jobs, you know, starting Jupiter But then running that kind of under that user's namespace And then that way, you know, they can't accidentally access or, you know, maliciously Access someone else's data and that's the way, you know, everybody is who they're supposed to be On each of the compute hosts And then kind of then on the stage left side, you'll see the client The authentication, so site specific, basically that means it's like whatever Identity and access management platform you want to use So you could be like LDAP or you could be like Active Directory The HPC scheduler, as you saw kind of in an earlier slide that would be Slurm And then like the nodes themselves or the network file system For providing the parallel storage implementation And so now you might be wondering, okay, so like where is our work coming in here So kind of on the stage right side here This is the, those are the components that are going inside the snap So basically the front end proxy, the back end proxy, the application runner And the application, well not necessarily the application itself But parts of the application have to go into the snap So the first part that we have that's really working nicely Is the Apache front end service Or the front end proxy that's working quite nicely So currently now if you install the snap You can kind of be able to start navigating to some pages And then what we're working on is kind of that per user aspect Which is like the back end proxy and the application runner But we do have engine X and both passenger buildings successfully into that snap And then basically now the parts that are highlighted by kind of for Juju on stage left Those are currently, we have those available So we have a scheduler implementation available We also have authentication available We have the node hardware itself And then we also have the storage implementation So really it's about getting that kind of front end part working inside the snap And so there's been quite a few lessons learned during this process I've at least been working on it since, what was it? Since we released Mantic Minotaur So back in November So kind of the first title here You know, I guess a little bit playful maybe But you know, I ain't met no bug, I can't squash And so what we've learned along the way You know, maybe a bit bold there But so, you know, kind of the first major lesson that we learned Is that we needed to bundle a custom Apache with open IDC inside the snap So if you've ever used like the next cloud snap You'll kind of notice that they ship their own Apache instance And so we have to do the same thing And so the main challenge with that is Is that Apache from Archive does not stage correctly Using Snapcraft Which I don't think is actually Snapcraft's fault So don't worry about it But yes, so it doesn't stage correctly So there's some like, you know, at the alternative scripts That don't run right So we have to build it ourselves And then we also have to add the open IDC plugin as well So that open on demand is happy when users log in Another thing that we also found Is that we needed to build a custom passenger in Nginx And so the main reason for that Is that I believe the Nginx that we have currently in Archive Only has the Apache integrations And the other thing is as well is that Nginx itself Doesn't support dynamic modules So you have to compile them in statically So that was something that we had to do So basically in the snap recipe We have to basically pull down passenger And then we have to go inside passenger And then we have to pull down Nginx And basically then we compile Nginx And point to some paths that are provided by Passenger And then the other thing that we found too Is that Debian rules files can be your best friend For understanding how to properly stage and prime things With a snap So I find that usually if I'm ever snapping something That's also available in Archive I just basically unpack the Debian package And then just look at that rules file And then from there I kind of have a good idea of like Okay, what needs to be in a configuration script What needs to be in an install script And where do files need to be copied inside of the snap image And then also another thing is that Source code can be the best source of documentation I know a common problem is people like Oh, the documentation says one thing But the code actually does another thing And it's like what? Documentation is wrong Yeah, so in that case what I really found Is that I am very thankful that Ruby is so easy to read Oh my god So we had to take advantage To make open on-demand work inside the snap We had to use some undocumented environment variables So that might be interesting later down the line If we go to the upstream maintainers and say Hey, we're using these features And it's like, oh, you weren't supposed to do that And it's like, well too bad So it's all good, it's all good But we found that source code Can be the best source of documentation Really because you can quickly identify If like, oh, we need to author something ourselves Or if maybe there's potentially an opportunity For us to make meaningful contributions upstream Or even at that understanding like Oh, the full feature set of the application So we don't need to modify it in our own way And then the last thing is that Client libraries can really help reinventing the wheel And make it easier to manage various components So if you actually do scan the QR code here This is a small library that I offered That basically just kind of wraps Like a couple of core YAML file Configuration files that Open on-demand has And basically makes it really easy to Kind of be able to manipulate that information And so how it's being used inside the SNAP Is that basically if I want to say like Oh, I want to change like the server name For this instance or I want to do something else All I have to do is just basically do Snapset on-demand And then just the configuration options I want to pass And so now for a short demo Basically go really basic implementation Of what we have working now So I'm going to pop over here, exit out Can I escape?  Is that good? Okay, sweet Thank you So first you'll see here on my terminal It's just basically a simple biobu instance I hope I said that correctly But what it is is that I'm currently In the open-on-demand project So if I do like LS here You should see some files So, you know, typical Git project It has the license, it has the readme It has the helpers aspect Which is basically just the configuration And installation hooks for open-on-demand And then the snap directory that defines the build And then overlays is just some custom files That I basically copy in when I build the snap And so for the sake of not trusting conference internet I already have a pre-built copy of the snap I've been burned before where it's like Oh yeah, I hope you don't mind having a megabit a second Which ended up not being great So I already have a pre-built snap And so what I'm going to do here Is just do a little reverse search And push it into a example LXD snap that I have Now we're going to clear that out And then just I'm going to shell inside the snap real quick On-demand test There we go So now I'm in on-demand test If I do the simple ls I should see, oh yeah, it's available And so now what I'm going to do Is I am going to do a snap install So snap install And then should be able to tab autocomplete Nice And then one thing is that I'm going to do dangerous I'm not basically just signals that Oh, I want dangerous, not angerous Daggerous, not that either Okay, there we go So we have dangerous That just basically says that we're installing a local copy And not something from the store And then classic environment And so the reason we have classic confinement Is open on demand actually qualifies Due to the fact that it is an HBC workload Orchestrator So it manages workloads Not the machines themselves And then there's also some aspects as well Where it needs to be able to drop privileges To run as the confined user So basically when you start engine next stage Or you run another job It needs to be able to assign that process The user is effective like UID and GID And so that's why we have that So maybe Ken could prove me wrong But yeah, he did that So I guess I'm in the clear But we do the snap install So hopefully everything goes well Doesn't break You can see mount snap on demand I also pre-downloaded a lot of the core snaps That I needed as dependencies So the snap is built itself off of core 22 So it uses all the libraries from there And then also just like having snap de-available So now if I go ahead here I should be able to run a basic command That does open on demand Or on demand update portal So that's just basically the front entry point So you can see there It just basically quickly generates an Apache config If you actually want to look at that I should be able to change to the snap directory I'll clear that so that everyone can read it VAR, snap, on demand Common, here let's go And then uh-oh It's getting long here Uh-oo-oo-dee Ooh, not PPP, oo-oo-dee And then config To be able to look here And then you'll see Oh, what's the oo-oo-dee portal YAML file I should be able to last that Hopefully less is available There we go And it's just basically a YAML file That specifies a couple parameters here So for example You could say like Oh, where to like put the access log For when people log in You can also specify If you want to collect user analytics as well We all know how everyone feels about Not being able to opt in to telemetry And then we can have some user reg access And then just like a couple common like URIs And redirects as well And then if we change up here Real quick And then we just change into the Apache 2 I might have spelled Apache wrong Change directory Apache 2 There we go And then we go into comf.d Oh, I think I Oh, I did not Okay, and then we Read the portal file here This is the automatically generated Apache access file In this case, since I have not set up OIDC It does the rewrite engine And then basically it will navigate the user to a file That says, hey, you need to set up authentication So people can't just log into your cluster And indiscriminately do things So now if I exit out here And I control L and then I go back to home I can actually go ahead and start on demand here So I should be able to just do snap start on demand And I should say started And now if I grab the main IP of this server Which is 10.6.29 Let me pull over a browser window real quick I heard somebody laugh at me There we go And now I type in the IPv6.4 There, welcome to open on demand It started successfully It's signaling to you that you need to start And then basically at this point here The part that we're still kind of working on Is being able to get that authentication information inside the snap And then basically then get it started Where it will take you to like a login page So for example, just like username and password And then even like a 2FA So you could like go on your phone And be like, yes, this is my login attempt And so one thing that I do want to point out here is There is a bit of a spelling error there So I actually did make it pull request to fix that Currently it is not released in a bug fix So they're still working on the next release But once that's available I'll be able to rebase the snap on that new version And that spelling error will be fixed Which is pretty nice I'm a contributor Okay, so now going back to the slide show here Let's go now after the short demo And so now you might be saying after that I've showed you here what happens after this This is kind of a rough timeline So this is like the ubukan at scale logo that we did Which is like the penguin waving inside the circle of friends I quite liked it You can credit goes there and for that And then I have a picture of a numbat For noble numbat release And so you can kind of look at it And see like, oh, what are we specifically working on? So the next steps that we have is We're going to integrate nginx stage Which is basically that utility For starting the per user nginx process For the interactive applications Need to get that working The next part after that Is getting the open idc integration going You can actually use open on demand On authenticated but it's very clear That it's unsupported And not an intended way to use open on demand So I would not encourage it And then after we kind of get that core experience working Then we're going to publish the snap to the snap store And then that way people can start Kind of pulling down edge release We basically just don't want to publish it yet Until it's actually working Because the last thing we want is Everyone to pull down the snap And be like, your snap sucks, it barely works And it's like, oh yeah, it's not finished yet And then the last part here Is then creating a juju operator To make it easy for site admins And charmed hbc users to effectively start Open on demand and easily bring it into Their clusters with minimal configuration And so now if you actually want to see the source code Because you know it's open source You can go ahead and scan that qr code there I will apologize that the readme Isn't exactly my best work We'll update it later But yeah, if you want to kind of start Getting like a preview look at like the work That's going on in there You'll be able to scan it See the snap craft yaml file Some of the other Python scripts that I'm working on For like the configuration and installation And client libraries as well And you know, kind of maybe the last thing here Is a call for today So as this is a project of ubuntu hbc If you are interested in potentially contributing Using it or you're just kind of interested In tracking its development progress I encourage you to join ubuntu hbc If you scan the qr code It will take you to our community page On the ubuntu website And it'll just basically give some information How to get started as well as, you know, ask an FAQ Because the big question we get is like This is really interesting But I'm not in any way involved with hbc Can I still join? And the answer is yes So yeah, if you have any questions also Yeah, feel free to get in touch as well That's pretty much it for my presentation And thank you for listening Yes, we have a microphone there So I'm not sure about this Is this intended to be a user interface To existing data analysis programs Or are you hoping to take over Some of the data analysis yourself So I'd say it's more meant to be like user facing So it's basically a web portal That allows them to effectively manage Like the resources that they have available In their cluster So for example, somebody deploys The kind of termed hbc core components Like the slur and workload scheduler Effectively, if they were to deploy open on demand They would then have an interactive web interface That allows them to access those resources Rather than having to say Because I would doubt you'd want to take on Replacing those data analysis programs Many graduate students died For the sake of creating those programs Over the years And so if they have 20 years of understanding That this particular set of packages works well They're not going to easily move to something else As a user interface, you're probably okay Yeah, so for a user interface, it's definitely fine And so maybe one thing I didn't really touch on That I could have a little bit more Is that you can actually create your own applications Then integrate with this front end So for example, if they had, say, these codes That they've been working for 20 years Like, oh, they have a GUI program That they click through and allows them To kind of access maybe some libraries They wrote for data analysis They could write their own interactive application That plugs into open on demand And then at that point, through the web interface They can just basically go ahead and start it Clicking through a couple of buttons and launching a session Is it good for interacting with things like NumPy and SciPy Because those are some of the main things Yeah, yeah, so it can be really great for that If you're looking for a development environment To use like SciPy or NumPy You know, you could basically just install Inside a Python virtual environment And then access that virtual environment Through, like, the code server You could also install it into your IPy kernel for Jupyter But the nice thing about it too Is, like, if you actually develop through open on demand And then want to, like, submit your jobs Through, like, the traditional way Using, like, the batch scheduler And all that, you can still do that Because those libraries will be installed on the system Is that kind of okay? Yes I noticed that you had a use case where The on-demand user would essentially go straight to Of node Yes By passing the job scheduler And wondering, like, what kind of use cases That are you thinking about with that As, I mean, is there any interaction With the job scheduler To know that you're using these nodes Is it the idea that node's not in the cluster? Or... Yeah, so the one thing that's nice About saying, you know, potentially having, like, Direct access to the node I guess maybe let me walk it back a bit So the reason why, like, the job scheduler Kind of exists is that typically When you have a lot of these resources, you know A, users maybe don't effectively know How much, you know, resources they need So if they kind of go onto a node That's, like, super powerful Like, say, like, an H100 or, you know, Massive core system They're gonna be like, well, I'm gonna use all those cores Even if I need only one Because it's performance, right?  Why am I getting all this, like, incorrect results Or having all these issues And it could just be because, like, the node is oversubscribed So that's kind of the problem And so the one thing that's nice about Open On Demand Is that it integrates with the scheduler So, like, when you say, like, Request an interactive session Like an interactive desktop It'll actually make a request out to the scheduler And then it will wait to start your session You know, depends on the cluster utilization For how long it takes for that session to be granted But then once it is, the scheduler will actually go ahead And then it will execute a command Which is, like, that nginx And so then it'll start the nginx And then basically it gives a URL to the user That they can click And it'll take them directly to that session Running on that specific compute resource Yeah Any other questions? Why is it that the software needs to use Both Apache and nginx? Yeah, I get that question a lot Thank you So, to be honest, you know, it's not quite clear to me, myself Why specifically that's there For me, at least I can give a justification For why I'm supporting that It's mostly because that's how, like, upstream Has it? A lot of their tooling is kind of built around using Apache And I think when they kind of specifically look at nginx They kind of see it as more of a reverse proxy Than, say, like a web server And so the reason why I guess they have both Is that, like, Apache? You know, that's the main web server component And then they have, like, the nginx stage With, like, the custom plugins that they need built in And then that way, you know, they're not cross-conflicting With, like, oh, say, like, a normal nginx instance That's, like, well-optimized Or just being a web server And then say, like, their custom, you know, modified version That's, you know, specifically built for launching Both Node.js, Python, and Ruby apps So that's kind of why they have both And then they also have a lot of utilities That are built around supporting Apache Any other questions? Well, then maybe have a round of applause For Jason, thanks a lot Thank you, thank you Well, thank you for that We have a short break again And then we come back at the hour For the next talk is a good one also Also in this area Yes, high-performance open stack That's right, okay So we'll see you back at the top of the hour See you back in about 13 minutes We'll talk to people at places like Oak Ridge National Laboratory About things that they do and use this for As, you know, getting a connection between yourself And the end users is, I would think, very important Yes, yes, so we have talks quite a bit With some institutions So part of our HBC community We have a lot of folks in there So we have someone from AMD who hangs out with us He's, like, responsible for all their GPU Driver packaging for Debian and Ubuntu And we have a lot of other folks too Like our OmniVector They have a lot of customer use cases They bring to us, okay And so we're actively seeking those sponsorships And then I also do go to a few HBC conferences And I'm sure it's going to mean like the A lot of the high-performance computing work Goes on systems that have Yeah Like you'll have programs that were originally written 50 years ago Yes, yes, yeah They're still in use because A zillion papers have written Yeah, yeah And they do the right thing Yeah So they don't want to rewrite that over again Yeah, yeah So that's definitely something we've Talked about But I've never seen that Okay Yeah So, yeah And it's definitely like open on demand It's not like, oh, you're going to rewrite Your everything in like a common language The idea is like, oh, you know You can access that 50-year-old program Through the web rather than having to use the terminal Yeah Yeah, because the system that I know the most about Is what's called X-ray crystallography It's, for example, what you Was used to do things like figure out the structure of COVID-19 Yeah, yeah, yeah Things like that And you would be astonished as to how many steps There are in the process of doing that Like you have You start out with Biologists who are Graduate students spending three years Figuring out that crystallize one particular molecule Because you have to take the Crystallized form of the molecule and put it in an X-ray beam And produce a lot of patterns of X-ray spots And that's the part where I am I'm running the equipment that does that Yeah And then you hand it over to biochemists To now run it through like 15 different programs Yeah I saw a flow chart once of all these various programs on the screen And it's a huge amount of stuff to marshal And show you If you go into that community you may get resistance because of people who say that But we already have programs to know how to marshal all of it Yeah, yeah, yeah And so the idea is that like we're providing the I guess I would look at it as like we're helping provide the infrastructure For running those applications So it's not like we're trying to replace core components The idea is like we're trying to make it easier for a lot of these You know teams like these I guess multi-discipline teams Where if you have the biologist then you have like the software engineers who write the code and all that We want to make it easy for them to just have a platform that they can all work on Both those applications that they have But there are like places along the way where There are proprietary programs written on Closed source stuff that can be The people in the question can be very cantankerous to interact with Yes, yes It's sort of like if you hit them the wrong way on the wrong day That they may refuse to want to do anything with you Yeah Like there's this one particular one that I won't mention But uh at least a dozen other people have tried over the years to write the places code and failed Yeah, yeah, yeah, I think I mean that's always going to be a challenge Where it's like you have that self-determined bit for like some folks just don't want to work with you And it's like you got to you got to take your wins and losses when you can So it's like you know ideally like we try and have an upstream first policy With Ubuntu HPC and so like you know we've gone this back we've done some work with them to like Get their snaps and like they recently adopted an upstream We've also done some work around command they're very interested Doing that Ubuntu support because they've traditionally been like CentOS7 So that kind of helps so like yeah we do have to be talk to them like Not not everyone want to deal with us some folks are like well You know everything was great with scientific Linux and all that so Well uh also you also have to deal with the fact that there's large chunks of that community Who are now very wedded to red hat enterprise linux Yes, yes, they are that's that's the other issue So that's one market we're trying to penetrate So we're hoping that like as folks see that like oh, yeah, look like this is a nice platform and whatnot And also like with containers as well Yeah, well some and some of the partner vendors and stuff only want to handle maybe Enough you don't even need all the fingers at one hand to list with platforms or willing to support Yeah, yeah, yeah Yeah, so that's something to and that's also not a thing that we're working on Of course, it's like going to these proprietary software vendors.  And a lot of the time it's a showing command from users We want to use those applications on Ubuntu and then also sometimes just providing the technical support Some of these decisions are made by people in probably high locations in the hierarchy. Yeah.  Yeah So I wish you well, you know chosen a tall mountain decline. Yes, I have Thank you though. Thanks for the questions. I appreciate it. Okay. Yeah, well, he's getting set up for the talk I have a Yeah, what is it? So You don't use the devian rules file. No, here's the thing. So Well, I don't think it's directly how it worked is everything in the rules file the entire make file for the entire package that That was the base Recent years has been made a lot simpler by dead public Right, um, so dead public actually has a bunch of scripts that overwrite I don't know if it's a large part of the scan that it needs to be Ah, yeah, it's fine, it's fine. Really, you're looking at dead public, I think it holds Yeah, yeah, I know they Oh Okay, yep Okay, everyone. We're ready for our next talk called OpenStack for high performance workloads And our presenter is Felipe Perez Hello, everyone So OpenStack for high performance workloads So I'm Felipe, I'm software engineer at Canonical. I I've been charm OpenStack DL at the moment OpenStack contributor for many years by now I've contributed not just to charm OpenStack But some more upstream OpenStack projects like Nova and Magnum And The purpose of this presentation is to try to give you the idea that running Software in a private cloud doesn't have to be slow typically when you're running Programs in a in a shared environment You are going to be facing challenges of contention resources and typically that gives you a bad experience But OpenStack provides you a bunch of ways so you can mitigate those And eventually depending on if you have enough hardware resources, you're going to be having a pretty good experience So We're going to go through four four aspects. What is OpenStack and why it matters some of the components that compose on OpenStack clouds What high performance Characteristics have and what metrics we care about and we're going to not going to go too deep on this It's it's an representation on its own, but we're going to take a quick look at what metrics are relevant here And finally what configuration options OpenStack have to to deal with all these aspects So what is OpenStack? Set of components that provide common services for a cloud infrastructure That that sounds pretty big and if you if you take a look to the high overview This is this is what you're dealing with an api Rest api so you can get bare metal servers virtual machines or in OpenStack terms. They are called instances And containers you can configure us also to get Containers and then you have a short networking and storage And there are a bunch of techniques to mitigate All these aspects so you you don't have to compete for them Now if we if we open this this nice bug that looks pretty simple You will start getting into these like it can explode very easily and you start finding a bunch of things like The message queue that allows to have long running tasks and have things on the back end Now this is an incomplete picture. It's part of the installation guide It has some components that in the case of charm OpenStack We don't have like drove and and saara But also it's missing some other that are newer and very relevant for For OpenStack clouds like Octavia Octavia is that the one that provides you load balancer as a service And and it's pretty critical when you're deploying coordinates on top of OpenStack So now From all these very large picture when we're going to just speak three components And the ones that are more relevant for what we're dealing with here Now nova nova is the one that typically you're going to be interacting with is the one you are going to be requesting your The creation of your instances. It's the one you are going to be Interacting with other services on your behalf because when you are Requesting the creation of of an instance, you're going to have You're going to need networking so you can access it, but you're also going to need some storage And and nova is capable of doing all this on your behalf So it's important to to mention that nova itself itself is not a hypervisor It's just a control plane in the back and it's going to hand off all the specific Virtualization to the driver in in our case is going to be libvirtan kvm It's it's the one that we use by default and the one that it has most of the features we We have interest on Then you have cinder cinder is in taking care of creating the volumes In our case, we use f by default, but you can use other or stuff like lvm nfs There are a bunch of drivers so you can have a specific network Network appliances that you may have already in your center so you can take advantage of those In the case of charm openness that we also have we have support for pure storage nfs and on some others So you can take a look at the documentation, but Seth is the is the good one Then you have neutron And neutron is the one that is going to provide you all the overlay networking So you can have your own IP IP space you can have your own private network in top of your cloud And it's going to allow your services to talk to each other no matter which no they're running on And and again nova is the one that is going to be creating ports for you It's going to be attaching those ports to your bm's and you don't have you don't have to talk to Neutron directly But depending on how sophisticated the network topology that you can define you may want to create your own subnet Your own ports if you want Instances with multiple ports you you have to do all of that on your own Nova is going to give you like like the typical bm one one nick one volume to store your root file system And and that's it um Also with neutron you can model The gateways and and that way you you can configure how how your network is going to how your traffic is going to flow on your network So We have this this is the control plane and now we what we need is uh To understand what metrics we want to to monitor because when we're going to be optimizing We're going to be caring about the specific things not everyone cares about everything. It's just about pros and cons Now when it comes to something that you want to perform well Depending on who you are asking to he's going to be Thinking on memory someone may be thinking about IOPS input output transactions per second Some people are going to be thinking about throughput Some others are going to be cpu bound problems and some others about latency in case you have a service That you are expecting to serve those requests under a given threshold Latency is going to be crucial All of them all of these metrics get interrelated. So every time you are pushing To have cpu optimizations, you may be affecting some others So so you need to be make measuring those metrics and and understand how how they are being Managed now. This is a it's a very well known diagram Uh, this one is from Brandon Greg. He's a well known person having given multiple presentations around performance and observability tools and this one gives you an idea of Depending on what you want to take a look into in the in the system what tools you may have available So if it's an application if it's sockets tcp connections and so on So, um, I really recommend visiting Brandon Greg's website. It has there are many presentations He has a book on the topic and all of that now open stack configurations, so One of the the most easy to set up is host aggregates. So You may have a fleet of nodes And those those nodes may have different properties And in in this case, we're gonna we are creating a specific That is For vm for for nodes that have ssd So we want to create a flavor and the flavor is the one that is going to define What characteristics the the instances you're creating will have And so you create the the aggregate and and you will be You will be setting a property in this case ssd equals true and After that, you're gonna be start adding all the different nodes that Have this property. This is all on the on the administrator side not Not a regular user of an open stack cloud is gonna have access to those because this is very intimate on on your hardware Then when you create the flavor, you're gonna be hinting that this flavor Can only be used in When when this condition is true this ssd equals true Now this allows the The novice scheduler the one that is gonna look for an available hypervisor to decide where your instance should be created on And and this way someone who is using your cloud and is seeing oh, I have the The flavor ssd.large is gonna have 8 gigs of ram is gonna be a disc size of 80 gigs But also it's gonna have the guarantee that this disc is gonna be created on a on an ssd And and that that way you can manage your your different needs now All these you can do it in this case These are just flags so you can do the same for gpu's you can do for any any hardware aspects that you may care about Also, if you have different cpu architectures, you care about differentiating about amd apx or In contrast to xeon for for any reason that you may be It's relevant to your kind of workloads You can use any of these Now the problem with this is that anyone can use this So anyone who has access to create an instance based on this flavor is going to be taking advantage of of these features But there are many cases where you want to segregate An specific set of hypervisors for a specific project because The budget was given to that project. So they they purchase those those nodes And in those cases you can also create host aggregates, but in this case, you're filtering by project And and that will give you the guarantee that Every every instance that this project in this case, we're addressing it by id There are the instances are going to be landing on those that set of hypervisors That way you are no longer competing with with the resources by and for with another project And and if your project is something about research, so those those nodes are are really beefy beefy machines You are going to be Having a better experience your your instances are going to be in the end running much better so Then you're going to start Optimizing by cpu So open stack Has a series of configuration options and some of them are available on the command line over the rest api While some others are only available at the at the nova.com file now When you are defining your Your instances in the beginning we were saying, okay, this this instance has eight virtual cpu's and and that was it But then if your workload is more specific and you want to model a specific kind of kind of machines, you're going to start saying Okay, this the I want A vm that is going to have this number of sockets Each socket is going to have this number of cpu's and Sorry this number of socket this Each socket is going to have this number of cores And then each one of those cores is going to have this this number of threads Now this allows you to for more hardware aware programs To run in a specific way again This is very intimate on on what you are trying to achieve Because then how this maps back to the hardware to the hypervisor. It's going to be relevant because If you're if you're putting too many socket cpu sockets in the for virtual cpu's you may have effectively Reduction in your performance because that doesn't map really well to the hypervax or actual hardware So for instance here, we have a flavor that is going to be defining Two sockets each socket is going to have four cores and Each core is going to have two threads So if we launch a vm based on this flavor Effectively when we're sshing into it and we run a less cpu you're going to be seeing that this machine has 16 16 cpu's right and if you start looking into how this is laid out in the In the in the machine is it's going to have this cpu architect topology now That this is this is all fine. So we can define Many sockets many cores many threads as we want But we still have the problem that There is No way to guarantee how this will be running in the hypervisor The the scheduler is going to be moving all these different processes around and you may be having Still like a bad experience from the performance perspective So then you're going to start defining Stricter ways to how these different different virtual cpu's are going to be allowed to run so When you have the the cpu policy you can start saying Okay, the default is shared right? So the the cpu's processes can be can be floating across all the host cores now If you have A dedicated set You're going to have that each virtual cpu thread At the hypervisor is going to be allocated to a specific core in the in that hypervisor So you're going to have mapping one to one And and now it's when you are starting the benefits of not Overcommitting your cpu's now you have that those virtual cpu's are actually running on a specific core So they are going to be running More smoothly And the thread policy allows you to decide When those virtual cpu's are allowed to run in a in a cpu in a physical cpu that if it has A hyper threading enabled or not because when you have two threads running on the same core There are still some some memory shared there And depending on what you're trying to do that may not be A good idea. It can have privacy concerns for the workload or performance concerns Now Then you have you have numma and and you can also model these in the at the flavor level level Now numma is what allows It's a way where the memory is not really accessed equally depending on where your task in the kernel is Really running The way the way it looks is this way So you have that there really the the memory ram is associated to a given socket And so if your program is running here in this socket, you don't see makers So if if it's running in this core for instance, and it's trying to access memory pages that are located in this section The the timing the latency is going to be larger So again, depending on the on your workload and how it's manipulating all this it's going to be relevant so Open stack is going to allow you to define how many How many numma nodes your virtual main your virtual machines once you have By default, it's only one and it's going to be floating around But you're going to start making more specific things like so I want I want Two numma nodes and each numma node if it's going to it's going to be composed of of this number of cpu's And then each cpu a given cpu is going to have access to a portion of the memory So then we can create things like Let me find my cursor here So You can create A symmetric definitions so In this case we're defining two numma nodes and then we are saying that for the for the cpu zero We're gonna for the for the node zero of the of the numma We're gonna be assigning zero and one cpu's and we're gonna be assigning two jigs of memory Then in the other in the other numma numma node We're going to be assigning one Two three four and five and we're going to be assigning four jigs of memory That way a program that is aware of all this characteristic is going to be able to manage that memory And and take advantage of this setup And this is going to be being translated by the scheduler And when the vm is defined at the livery level it's going to be Giving you a good representation So the hardware can be taken advantage advantage of now This is this is all great We're we're all we're trying to map these definitions that are abstract that abstract in the flavor and how all those map into livery But we still have the problem Where the kernel at the hyper basal level is going to be trying to assign a bunch of other administrative tasks And in in in course that we may want to be really using for virtualization And in those cases the recommendation is to start using isle cpu's So this way you can in this case in this example here isle cpu's is a It's a kernel command line option. You are going to be adding it into your graph configuration So this way in in isle cpu's equals zero dash 31 You are saying from from the cpu zero until the 31 Don't scale scale anything there. So the kernel is not going to be considering those To to sign in any task. So anything running in the hyper basal It's going to be only running from the 32 and above And this gives you the guarantees that any vm's using from the zero and 31 Are really running Along there and they are not competing for resources They are not going to be being being post to run other Administrative things that the hyper basal may be trying to do all right, so That's all what I have for today Anyone has any questions Well, first of all, great talk really like the content. I'm quite relevant to what I do But I guess you know one question that I potentially had is like You know, I guess You know here what you showed is that it's like possible to you know, tune your open stack like your private cloud To get the you know best performance possible But I guess what I'm interested to know is like, how do you maybe like improve the messaging the documentation?  This is crap, but that's just because they you know Potentially didn't know how to configure it Yeah, that that's true. The documentation Makes a lot of assumptions like for instance if you go to the nova documentation and how to configure noma And how to define flamers flavors that take advantage of It's not going to go up you through on why someone may really want to do that There is Baking assumption that someone really knows what it's doing and And then you have that As I was saying at the beginning of the presentation Every time you're taking one of one of these decisions to push into a given direction to to take advantage of your hardware You are putting In a different position when it comes to launching bm So the reason why many of these are not defaults It's because every time you put an extra constraint in the flavor definition The scheduler is going to have to work harder to find you in a spot in the in your fleet of nodes And and depending the number of bm's you may have you you and how large your your set of hypervisor is You're going to have More times where the scheduler is really not finding a good place to put your bm is just failing And give you an error. So How to make that accessible? Probably try to at least give you pointers on when where Where to read more about certain things like numma how that affects to the programs that are running there, I think Some of those links could be made from the Nova documentation to leap it and chemo, which is the one that probably explains Much better and more depth how all this lays out back in the hypervisor All right, thank you I know what you're saying because a lot of times when you go there you just assume default works Well, I do default works Well for me, but I think providing use cases as examples Maybe that will help the users because if you have to go rate a lot of men on this and that There's so many options, right?  I have a time the sprint, you know This is not something you can spend a lot of time on but I think I like use cases if you can just list Well then everybody does something different, right? I'm just saying some of the common uses and give example what would be the The parameters, you know, you put it in there or whatnot and I think that will be helpful But that require work from the developers like you, right? Oh, I had to think about what would most people But that's what I would recommend because there's so many different options, right? How do you even You worked out for a long time, right? So, you know, what's the optimum thing to do? But most of us I don't know about most of it for People like me a lot of times I want something quick, right? I may not have the time to get into details But having seen some use cases that will help me to determine What should I use, right? So anyway, that's just my suggestion Or you can get really details on multiple use cases too, right? I I agree with the sentiment behind that idea The tricky part is that many times people take those examples If they were golden rules that they can just copy and paste without really understanding so I remember a good example here where someone was applying Was dropping the file system cache and they were running it in a cron job So they were trashing away the performance of the page catching And and they didn't really have a reason It was something that they just copy and paste from the internet because they said the internet said that this was To improve the performance, but it it was under very specific characteristics And if you don't you don't understand the characteristic of the thing you're running It's better to stay in the in the default values So real quick I saw you using like an m1 as an example up there If I were to like compare this to say like a like an m1x large or like any sort of aws instance Have you guys done a comparison of like? Okay, this is an m1. This is an m2. Sorry. This is like an m4 1x large m4 2x large or whatever aws or Your favorite cloud provider instance types And then said, okay, this is what they're doing. This is what we're doing Like get a comparison of performance or anything like that so I could start to understand like The like the scheduling algorithms you're using or the just the assignment algorithms the like how how things are happening under the hood And as a cross-comparison to other hypervisors and other ways of doing isolation And specifically We haven't ever done that because every every cloud is choosing every private cloud based on open stack is choosing different set of hardware Because the hardware has been changing over time Many times they they buy a set of nodes In one point in time and then when you're later, they're getting a different set of hardware because they want to expand So so it's really difficult to say If you apply this set of settings, you're going to be an equivalent experience to aws Flavor x Because in the end all this depends on the hardware you're running more than which setup you have Fair and I guess the other question I have is when ec2 first started It was a really big problem with noisy neighbors Like on the same instance Do you got how did how did you tackle the problem of noisy neighbors? In in what you're doing here This way this way Basically you start you you need to start Isolating cpu's you need to start Stop over committing the cpu in many times because that's what hurt you the most Even if you are not really Tuning all these aspects If you say that you're one to one in terms of virtual cpu's and actual cpu in physical cpu's By default you're going to get a much better experience But in reality many private clouds They want to overcome it because they don't have the budget and they want to get the idea. They have a large capacity any other question I'm sorry. What does noisy neighbor mean in this case? Do you see a decoration in performance or what in terms of you're talking about aws, right? I just don't understand that Right where everything is on print for my organization. So I don't know what that means noisy neighbors I see I get it now. All right Okay, thanks I was a good clarification because we all have questions like that Any other questions Here, thank you very much for that. Thank you. We have We have another short break again. So stretch your legs and we'll come back for our final talk of today on ai is yes So that'll be pretty great here at five All right. So for our last talk for today We are very very very lucky to have an expert here who uh is um Uh has a really really cool talk for us. It's called learning language learning models from zero to hero and um So it is my pleasure to introduce Andrea Thank you, and I hope you can hear me. Thank you everyone for staying until so late I know by now we're all thinking of beers and having dinner and all these things So I'll try to to make it interesting for everyone To start by introducing myself. I'm Andrea Montenu. I work with canonical. I'm an ai product manager I have a background in data science and machine learning, especially in retail and telcos But at some point because I was too frustrated to deal with all the tools I Changed places. I joined the product team in order to Build solutions that are actually easy for data scientists machine learning engineers This talk is going to be introductory and is going to talk about language models As well as how you can benefit from open source. I'm I'm a big believer in open source. I love communities that leverage open source Um, so let's not make it any longer. So I'll tell you how it all started It was three years ago tragedy popped up I was about to go on holidays and I was telling one of my co-workers I really want to go to the mildeeves. I love diving She said fine. Andrea. You can go just do it Okay, okay, that's gen ai, but that's how I got introduced to large language models And if you're just getting started in this industry, I think there is a lot of confusion between gen ai llms And that's all right But today we'll focus on large language models and for those who are not familiar with it They refer to models that use deep learning techniques to capture complex patterns To produce text they usually train on self supervised learning And they are beyond the scene of large transformers Well, let's look a bit more in depth at them because Large language models are large. They are trained on data sets with many parameters gpt3 Which was the one that I looked at when I prepared this stock A couple of months ago now had more than 100 billion parameters And let me tell you something gpt4 it's at least double the size and it has more parameters It has also been trained gpt3 just to make clear on 45 terabytes of text And that's why it's so Capable it has the language part because it mainly operates on human language And then of course it's it has the Models part because they are used to find models or make predictions within the data Look whereas we all look at llms nowadays or in the last two years, whatever We are not really at the beginning of this journey I remember when I started with machine learning It was 10 years ago and everyone was telling me Andrea you're doing unicorns But machine learning was also around for quite some time. I think it's the same with llms They became famous a couple of years ago two to three years, but the truth is that around the The in the Or towards the second part of the 900s people started looking at idea or exploring the idea of llms by building basic rules As you may imagine that didn't fly out very well because it was too manual. It was not adaptable. It was not scalable. So they dropped it In the 2020 it's the year that I think large language models Became very popular and nowadays you have plenty of options that we will talk about in a second But then who can use large language models? Do you have a use case? Just raise your hand if you know of a use case that could benefit two three four people Five don't be shy Jason has one as well So there are plenty of use cases. Some of them are very familiar We all use chad gpt right and child bots are familiar But then the applications go beyond it. You can use it for tax generation story writing marketing content You can use it for summarization. I don't know how many of you stay in long meetings I do I have six hours meetings a day So if someone can summarize my meeting notes, I'm very happy for that Translations as well. It's very important and it's useful, especially in environments Where English is not your native language or you struggle with it. English is not my native language And if I'm tired, I might switch to Romanian. I'm not just like that anymore. So we are fine And then last but not least classification for example in marketing again There are a lot of sentiment analysis that's being done and llms are important One thing that I want to say here is that whereas large-language models have applications across all industries from Having chatbots in retail who tell you what are the latest offers to sentiment or to To telcos companies who are looking at how you talk about the latest offering to see if they keep it or not You should always look at the use case. Don't use large-language models just for the sake of it That's not how you should do it. You should start with a problem to solve What does it bother you every day? Does it bother you that you have to go through 20 Pages of meeting notes? Okay, let's summarize them. That's great But if you just do it for for the sake of doing it It might be fun, but it might not give you results that you're very happy with That's where it comes it becomes interesting large-language models versus traditional machine learning I already told you I started with machine learning when it was not that nice I was looking and working especially on structured data And that's a question that I get often at conferences as well as with customers. I Through the through my role I get the chance to meet a lot of companies and Most of them say they want to use large-language models or they want to do gen AI, but they have no use case So I say why not traditional machine learning and the question is what's the difference? While lm's are specialized in nlp or natural language processing and they're designed To handle structured data, but however, they are not specialized in image analysis or ranking structure data tasks, for example And there are other machine learning algorithms and models such as linear regression that are more suitable so The main question is what do you choose lm's or traditional machine learning? It all depends on your use case. It's very easy However, there are some benefits that lm's might have and you have to bear them in mind On one hand, you get improved performance nowadays. They they they do get better results on nlp specific tasks And often they surpass the traditional approaches that have been on the market They accelerate the learning so often you can use them for your use case with Leader training or leader optimization to no optimization and last but not least they have multilingual capabilities in canonical I think it was six months ago We were playing with some of our colleagues and we managed to build a chat bot that was answering in african The reason why we tried african is because we're south african and Mark was curious about it. It answered quite correctly And I said, okay, I'll try it in romanian as well It answered in romanian as well to the point where I I started using characters that are not in the Normal english language in the same area. They are nowadays lm specialized in unusual languages such as Urdu, it's something that picked up very quickly It was it's an lm that was developed. It's open source. It was developed in UAE by Pakistani Startup and I know about them just because I'm based there How do you take This lm's to production or how do you actually Move away from just the idea of ah, it exists so I can use it it with ml ops. Are you familiar with the concept of ml ops? Let's see some people. Yes Okay, I I still define it as DevOps for machine learning and Those who have been in the ml ops phase for a long time will feel like killing me But that's the shortest definition. It's it's a new practice or it's not that new anymore and it aims to Take machine learning projects to production in a scalable and repeatable manner But the truth is that I get a lot of complaints Training is resource heavy. I need a lot of compute power. It's expensive Yes, I know you're right But then that's why you can use pre-trained models such as olama for example Or lma 2 they are trained. You need to optimize them, but that's much cheaper It requires less resources nowadays you also have the public cloud for example that can give you the access to more powerful compute machines And to to maybe put it in context when we train when we've built our own chatbot That will share a bit later The architecture it was costing us 50 cents for each half an hour So it's okay. If you just want to play with it. It's not very expensive At the same time you should look at capabilities such as or features such as gpu sharing tools such as volcano Which is open source are very great for that as well as The idea of distributed training and using frameworks such as paddle paddle Costs are high that's what company styles us. It's too expensive. We don't want to do it We don't want to train our own model Again, you don't have to train your model. You can just optimize your model However, there are also things that you can look at And the most important part is that you can use open source tooling it enables you to Get started at a lower cost and then as you scale Of course your cost or you can invest more in it And it's very important to mention here that unlike any other innovation that I've seen on the market AI is Fully open source or is naturally open source There are big communities around it and the latest tools latest innovations. They happen in open source communities and in an open source space Also, you should always think of your long-term strategy and think of hybrid clouds You start on a public cloud as I told you because you don't have enough resources But then once you want to scale you can always move on prem You can always consider hpc clusters that jason's talked about and there's nothing to To stop you from that You start on one cloud you move towards a hybrid cloud scenario or a multi cloud scenario depending on your needs where you store your data But that's not all It's not reproducible. You know, I have a team of 20 data scientists They all do the same thing They they get the result. They don't know how to get back to it. I think A lot of data science still happens in small Black boxes that people don't know about and that's when you should think of open source tools again I feel like i'm a broken machine saying open source open source A million times, but I think that's the answer in the ispace Tools such as skip flow or ml flow enable reproducibility on one hand you can do you can Have capability such as experiment tracking Which is very easy, but then you can also build pipelines automate your work in order to make it seamless for your teams That's what i've done at home. I stopped coding at some point, but I still enjoy doing Fun things And it was really after played with ched gpt. I said, can I build my own chatbot? It's easy When you when you look at it it all starts with a user who queries something That query goes into the embeddings model and then it goes into the open search Which is basically a vector database where all your prompts are stored But that it's not all from there the data or the search for the similar searches are retrieved And they going through the open source model you get an answer Of course not all the answers are good or depending on How niche your topic is if you if you ask about something that is not really available on the internet I'm not sure if that's possible Then the answer might not be accurate, but that's When you look at this That's when you look at the more real world example That's designed not just to be your pet projects, but something that people can use And there are a couple of more capabilities that you need to ensure on one hand that you Find tune or you optimize your open source or your LLM with data that you have assuming you want to build a chatbot about your Offerings in a telco company. Maybe it's not the best example But then you should use that data that you have about your offerings about what customers usually ask to optimize an existing model At the same time and then you should also do it in an automated reliable manner I don't know it updates once a month with the latest data, but then also You should ensure that similarly all your data Is stored in a vector database all your searches are and prompts are stored in a vector vector database And then when the user queries The answer gets similarly to to the other one so LLM's projects can be done with open source everything that I have here is open source I've done it on my own. I didn't pay much On one hand you have open source models and I think that's what's Interesting for those who are familiar with the ml word probably they are also familiar with hugging faces That's where you're going to find a lot of models but then Not all of them are lms, but then there are also open source tools that can be used to build your own models to find you an existing models And open source Sorry There we are Open source language models. I can give you easily some examples llama to Mistral these are the ones that I think are falcon as well Are the most used ones have some benefits. It's easy to get started. There is nothing to to stop you I had I said earlier in the day that olama is also snapped But I think we have some technical difficulties here I don't want to say it you can try it out, but you might run into some problems But it's also cost efficient. You don't need to train your model training a model on billions of Parameters can be expensive not everyone can afford it I couldn't and also I would want to invest my money in that even if I would have the money because I would rather go on a holiday than train models and buy resources It also gives you flexibility because I'm a hundred percent sure that there is more data available on the internet than you have in your own Machine and then it benefits from community support. I Was not an open source enthusiast before I got into the machine learning world But the truth is that in the ml space there is really an active community that you can ask that's very passionate And then there's also a matter of co-transparency You can always see what goes in what goes out at the same time there are concerns I'm not going to say that everything is pink and and great because often there are questions about privacy And especially highly regulated industries Have some concerns about it. I can easily think of healthcare, for example That's why they've been looking at confidential computing as an alternative to to further optimize their models But then there's also lack of support and lack of enterprise support Especially which can be challenging and then the learning curve is not that easy your first three months in the machine learning world are going to be Very very frustrating. I would say I don't want to point anyone. I don't want to scare anyone, but it's annoying It's just a completely new word It's a lot of data sometimes at least when I started it was also there was not so much compute power which I would start training something it dies halfway through. How do I do it? How do I debug it? It can be Annoying and then there are some security risks as well Which are really related to the vulnerabilities that the packages that you use can benefit There you go. I knew I had a slide with this It's also interesting to look at langchain. It's not a language model per se. It's a framework, but it's widely adopted in this space Tools, do we know open source tools that we are using in the machine learning world? Come on There is one that everyone uses I'm sure No Jupyter notebooks now Okay, yes, I think it all starts with linux and Ubuntu Then it goes towards Jupyter notebooks Python, PyTorch, there there are plenty of programming languages and tools And again, it's easy to get started Which I like it you I know it's easy to deploy Jupyter notebooks. Everyone can do it. I guess or If they don't do it again, they look on the community Forums and they will find the answers even if it fails. It's going to fail a couple of times, but in the end it works It enables customization. I'm not sure how much or if you've tried tools that are already made such as The public cloud solutions, but then they are very rigid They they give you a solution that works well But if you have different needs or if you want to try something new, especially if you're in an exploration phase They are not that flexible and then it also Is not something that in the machine learning world we talk that much But open source tools give you the ability to contribute And especially if you're just getting started or you You like contributing you you like making the world a better place You really can do it and projects such as kubeflow, which I'm active in Really appreciate contributors By contributions, I have to say I don't really think only of code I'm one of those that think that code is not the only way to contribute to a project You can contribute by writing documentation. You can contribute by providing feedback You can contribute by trying the product and building a nice tutorial There are plenty of ways to do it. Don't just think of code Because especially if you want to do data science, it's maybe unlikely that you will Want to contribute a lot to how the project is built, but you can provide a lot of feedback That is very valuable That's how I started in the kubeflow community. I tried to deploy it and I couldn't I failed dramatically So I went in the community chat and I said hey This documentation is outdated. I don't know why it doesn't work In the end, I think we've learned why and we also improved it There are of course some downsides Security lack of support similar documentation. I just told you what happens, right? Jason told you what happens with the documentation He thinks that the code is the best documentation, but not everyone knows how to read the code. So documentation is important But also bug fixing whereas you can suggest bug fixes it might take a bit longer I asked you about tools. Well, when it comes to language models and tools that can be used I'll take them on by one kubeflow is an end-to-end mlops platform. It was a project that was started by google Five six years ago. It's quite some time now. It's part of the cloud foundation cloud native foundation It's a suite of tools the way that it was designed it was to have to be a suite of Famous or leading open source tools that can enhance the machine learning project So it has jupyter notebooks integrated For the training part, especially it has katib for model optimization. It has case serve for model serving Um, and then it has kubeflow pipelines, which I think are the heart of kubeflow And they're really used as mlops pipelines to automate machine learning workloads Then you have ml flow which is used for experiment tracking especially and model registry it's I think the most Famous machine learning tool on the market It's super easy to use it. It's the thing that whenever you get started. It's easy to install it. It's very intuitive as well. So if you're just Looking into something jupyter notebooks plus ml flow It's the way to go then you have spark for data streaming and last but not least open search, which Historically is famous for as being a no sequel database, but it can be used as a vector database And any project that uses llm's needs a vector database for storing its prompts Then when it comes to canonical and what we do I I do love it We do have a solution that includes these tools and a couple of others And they are used really to fine tune or what you can do is fine tune models Benefit from the all these components being pre-integrated Upgrading easily easily and easier as well updating But I move here What do you do when you want to scale your project? First thing the most important one start with a use case That's how you get started. You have a problem to solve Don't think that you scale something that you're just doing for fun because It can work or it cannot work Think of your users whereas now you're likely to work on your own on your machine In two years from now, it's like that you want to have a team of data scientists and then you want them to be secured to to Have clear isolation between their pipelines, for example, and so on Think of how to optimize your resources There are some reports that around 70 percent of the GPUs are underutilized And that's sad. It's sad and unsustainable. You also have to bear in mind that it takes six months nowadays to get the GPU So what do you do? Well use and look for resource schedulers There are many open source options for that and then it really enables you to on one hand even get started With maybe resources that you have in place But then also to optimize it in the long term Look at the reproducibility of your work as well as portability If you start especially if you start in a public cloud and you might want to migrate being able to To enable this will help you Look at the monitoring I didn't talk much about it But you should have and you should enable model monitoring data monitoring as well Data drift monitoring because otherwise it might be risky. You might have some malicious friends who try to to play negatively with your models and last but not least Look at the security and compliance There are plenty of packages that are being used in an animation learning project from python to numpy to pandas to there are many of them and Many of them have vulnerabilities. There are more and more attacks There was last year. There was a famous one called shell torch or Yeah, I think it was shell torch called or it was it was against a Pie torch and it it really harmed a lot of organizations Whereas when you have fun, it's okay. Especially you want to implement it in your organization Look at the security and compliance requirements And all in all have a long-term vision for your machine learning project Either if it's with large language models Either if it uses gen ai or it's just a traditional machine learning project, which doesn't look that interesting nowadays I think it's still very useful So Just to recap you should get started with the use case using open source tools Try to be able or to enable reproducibility of your experiments and try to view solutions that run on Any environment whether it's public or private cloud and then you should always scale with integrated solutions And once you start scaling look also for enterprise support And that was me That's where you have my contacts tomorrow jason. That's our boot nu