Good morning, good afternoon, good evening. Wherever you're handling from, welcome to another edition of the Data Services Office Hour. If you notice, the name changed during the show. So we're gonna talk a little bit about that today. I'm Chris Short, executive producer of OpenShift TV. I'm joined by Chris Bloom and Gion Motier. I'm very happy to have both of you here. I'm very happy to have learned that pronunciation this morning as well. So thank you very much for the lesson and French pronunciation this morning, Gion. So Chris, Data Services, what's up? You haven't been online for a while? You had to go to Portugal? Like you got a lot going on in your life right now. It's great to be back. Yeah, so some of you, our true fans that always watch the Office Hour here, they probably remember me. I was gone for a little bit because I live in Berlin, Germany. And a lot of people have decided to work from home and that kind of overloaded the internet capabilities of Germany. And so I was limited to 0.4 megabits upstream. And so I said, we want to deliver- Which surprisingly just doesn't work for live streaming. Nope. So right now I spent three months with my aunt, very remote, don't have postal address anymore, but we do have fiber internet here. So 100 megabits synchronous. And so I'm able to come back and I hope you're not too sad not to see Michelle today. So- Michelle will be back from time to time though. Just- She will be back. Yes. She will be back. So that's not the only change that I'm back. We also changed the name. So we changed from the OCS Office Hour. We changed it to the Data Services Office Hour. And that's not just a name change. It's also a change in how we change our focus. Where previously we always talked about storage. Storage storage and a lot of people just think about OCS as the solution to provide their persistent storage to PVCs and just all that single use case. Just do the dumb stuff that a lot of other people can do and it was sometimes difficult in these conversations to really position the true values of what our product does. So with the name change of Data Services we're going a step further. We're not just talking about storage because that is a problem that has been solved over and over by multiple companies. We want to show you that we were thinking one step further, we're not just trying to provide some kind of storage capacity. We want to help you run your workloads for the storage. And the storage is a little bit smarter. It understands a little bit more about what you want to do and helps you in doing that. So that's in a nutshell. In a nutshell, yes. So, we basically have two parts of this. So we have the, let's call it the old part. So OCS is now renamed to ODF, so OpenShift Data Foundation. And the other part is what we have Guillaume here now is we want to talk a little bit about data science. So do smart things on the storage and that involves AIML workloads and we can do phone stuff with Chupiterha, just prototype something in Python, let it run. So Guillaume will talk a little bit about that later. Awesome. And that's the idea here, if I may. It's like changing the perspective about what we are proposing. It's not only about storage, which is the implementation, how you do it really, but much more about the business value. What can you do with the storage? Which in fact is, okay, I will work with my data. For data science or purely data for your applications or things like that. But of course, at the end of the day, it's the same thing that is running. You have to store your bits and pieces inside some storage. But now we want to lean a little bit more on this aspect on how can I use my data? How do I integrate my data inside my overall architecture and not only living storage at the end of the chain as something that you don't even consider until you're really needed. It has to be part of your architecture. Therefore, the slight shift of approach. Right, like you can't just dump everything in one place, right? Like we need to think about this in a, like you know how we used to think about how we partition disks. We put the boot volume in the very front, right? Like you have to think about all of your data in not the same light, but you got to think about it, right? Like what am I going to do with it? It's just going to sit here. I have to keep it for regulatory reasons. Can I do anything else with it while it's there, right? Like how do we manage all that? So the entire engineering effort behind the data foundations, I think is like, it takes the name and actually applies values to it, right? Like Chris, you mentioned values and like it's a very interesting proposition here, right? Like I like it a lot. Yeah, so the foundation is literally a foundation. So what we had before is now the foundation of what we can put on top of it. And what we want to talk about is use cases. You tell us more high level about what you want to achieve. And then we can talk about how the ODF or the data services can support you in doing this. And also one of the consequences is we just released a new version. Now ODF 4.7, and with that, we also thought a little bit more about the pricing. So the pricing will be a lot easier to calculate with this new version. And yeah, and we're starting in this data services approach where we add things onto our foundation that you can then use. Beautiful.  Is it working so far? Yeah. Asking the hard questions here, Ana. Yeah, that's important. Someone needs to ask the hot web questions. If you have any more hard questions, just write them in the chat. Yeah. So I don't think I'm up for it. So like I said, we just released it. I think yesterday was the GA 4.7. So in the last 12 hours, it has worked. In the last 12 hours, we've killed it. Yeah, awesome. But obviously before the GA, we had a lot of internal discussions about this. We wanted to actually understand, are we doing the right thing? Do people want this? Do people understand this? There was a lot of conversation about how we should position this, how we should do it. And in the conversations, when people really understood what we want to do, obviously there were a lot of people that were sad, that were leaving that term storage. A lot of people look at this term and say, well, now we're not storage anymore. What does that mean? But it's a new term, it needs getting used to. But once it sinks into people, that's what we've seen internally here is that they understand that we were limited before. We were limited to being a storage department that just cares a little bit about disks and how to partition those disks, how to make them available, how to be a fast storage or storage that only needs little resources. But now we can actually drive our conversations further. We can talk further about, hey customer, what do you actually want to do? And then we can deliver a full blown solution to the problem, not just a product that you install, you put in a CD and install it, and then you're done and you never think about it anymore. This is interesting, so go ahead. It's also a change of the people we want to talk to. Not only the season means the storage and means, but go a little bit broader with the architect, the solutions architect, the CTO, the CIO. If you speak to a CIO about storage, you will say, oh no, it's the thing for my ITN means, I don't really care about storage. Now if you're talking about what you can do with the storage, what you can do with the data, then you have the attention. And anyway, we have experienced this shift for the past 10 years in all the IT infrastructure components becoming more and more commodities, especially with the cloud. Okay, let me take you back 10 years ago, back when I was working at Level University, we were out for an RFP for new servers and things like that. We would spend, the architects team, we would spend hours looking at the bus architecture with the processors and how it's handled and everything. Fast forward 10 years, oh, just bring me a server. A data on HP or whatever, I just don't care because that's not relevant anymore. What has become relevant is the containers that you are able to reschedule automatically or your VMs or things like that, but not the infra itself. Not saying it's not important, but it has become so easy, you know? Oh, I can have a server from AWS for measure or even on-prem, now I have all my pipelines to deliver VMs on my internal cloud or things like that. It's not really the subject anymore. Now the subject is how can you deliver this to your dev, to your people, for them to be able to use it in one hour, because that's what you are competing with, with AWS, you know, 10 years from now for servers. Oh, you want a new server? Yeah, call us back in three weeks because we have to order it and then it will be delivered and then we have to rack it and connect it and things like that. And you know, it's obvious for servers, but the same thing has been happening for storage, storage becoming a commodity because, oh, you want object storage, yeah? Just go to AWS S3 and you have object storage. That means we have to deliver exactly the same experience. Therefore, leaning more on the data aspect, what do you do with the storage? Of course we will continue to speak and to work really closely with the storage people, the pure storage people, because at some point you have to do this, you know, for performance and scalability and things like that. But if we want to have more impact, we have to go a little bit up the ladder and talk directly about the usage with those different personas as we were used to work with before. Nice. So change the name, change the product name, change all these things. We're thinking about data holistically now, not just, hey, we're going to sell you the storage and you know, three different open source projects glued together. What has changed in the product as a result of this so far? Did you hear me? I thought like the biggest change is actually an area where GeoM is, right? Okay. So internally in Red Hat, we merged two teams, which was fully the storage people and then we got the data science people on board too. So the biggest change that you can see today is that the data science part has been added and GeoM can talk a lot more about this. The other thing is that now with the 4.7 release, we're starting and it's now deaf preview in 4.7. We started looking at the DR things. So these will gradually improve now and you get a little preview of it in 4.7, 4.8 will already be a lot better and then we're looking at the releases afterwards where you can release that really, yeah. So that is now available, but the biggest change is the data science part that has been added. Nice. Yeah. And it's not, you know, we changed them last week, something like the announcement was yesterday for the official new name. So it's not about the changes that we put in the product because this has been happening for a few months, adding more features towards this easiness of consuming storage or being able to deliver data services. So we've been doing that and we will of course continue to do that. So that means integrating more things into directly into the OpenShift console, into the OpenShift UI so that for people, it's easier to work with storage, especially as a deaf. You know, you don't want to, in fact, you don't even want to talk about storage. You just want to put your data somewhere. So we... Mike, tell me where to dump those. Give me the API call. Exactly, give me an API. I only want to talk about an API and an SDK. The rest, in fact, it's not my skill set as a developer and I don't want to learn more about this because of tons of other things to learn that are directly linked to what I do. So this storage, again, is a commodity from this point of view. So we have already and we will continue bringing more of those features, of those integrations inside the OpenShift as much as we can. Let me give you an example. In SAF, since last year, we have this feature called backup notification in object storage. That means whenever something is happening on your bucket, of course, you configure it. Say you have uploaded a new image or something, this bucket has the ability to send a message, to send a notification to an endpoint. It's such a simple message saying, hey, this file with this name has just been created inside this bucket. And we can send this message to different endpoints and HTTP, REST API, Kafka, MQ messaging. And then you are able to act upon this event. Okay, so that's the first illustration where you bring data as an intelligent thing within your architecture because now it's part of your event-driven architecture. It's not just something where you dump your data and you retrieve it when you need it. Now it can totally be part of the architecture. But this feature, well, it's not that difficult to configure bucket notifications. It's pretty standard and we reuse the same mechanisms and protocol as you have in AWS S3. So all these decays are here and everything. But still it may be difficult for some people. So there are work, there is work going on right now to bring this as a configuration part of a YAML definition. So purely native Kubernetes way of programming thing. Oh, I want to have a bucket storage and I want it to send events to this endpoint. Three lines of YAML, bam, you have your object bucket and you are able to work with it from your applications and then create this event-driven architecture. You don't even need to know how it's implemented. You know, behind the curtain, you don't even know if it runs on SAF or whatever else or how many nodes or replicas or things like that. No, normally your IT team is supposed to take care of that and provide you with these performant scalable storage that you need to work with. That's the kind of things that are happening. Nice, so if, go ahead. Sorry Chris, it appears to me like I'll talk, not me. Yeah. So in addition to what Gion said, one of our goals is to keep ODF with the similar goals that we already have with OCS where you have an interface that's very simple to use. It's very integrated into the OpenShift experience. A lot of the other products that you can have that give you storage on the OpenShift platforms, they might not even be written for OpenShift or written for Kubernetes and they sometimes work with OpenShift and then there's a new version and there's compatibility issues. ODF is developed primarily for OpenShift and works with OpenShift. It's deeply integrated, you get dashboards and even though we add more and more features, we have a ton of features that we have in staff in the backend that we can pour to ODF but we do still want to have that ease of use so that you don't need dedicated storage people that need to understand it. It's just that it's there, you can use it but you can have your regular people use these products easily. Regular people using the product easily. That's music to my ears, it really is.  Like the data science idiot, right? Like I've helped the data science people embrace containers. Now it's like, can the data science people help me embrace some of what they're doing? That'd be awesome. So yeah, like I like this. I like the direction we're heading here. And we've shown this in the first couple of shows if you want to go back to the archive where Chris himself, he installed OCS and it worked. It worked, man. It's one of those things where it's like, I'm patiently just waiting for the next cool thing to come out about it, right? Like it's not, it's maintaining itself that does what it does. It's an operator, it's going to handle things for me. Just waiting for cool features. Is there anything else I can do with this? Is always the thing that I think about like, okay, the clusters here, it's doing things for me. Can it do more things for me, right? And that's what most people do with their infrastructure too, right? Like we'd like to take on a new project. Can we do what we got or do we need to add something else? A lot of people sit there and think that. And it sounds like ODF is going to have some foundations and data science that will have folks thinking, yes, we can do more with what we have now, right? Like that's kind of the premise is what I'm hearing. You don't have to go out and be architect or do that RFP that takes, you know, months to figure out what you need. And no, it's often running from the get-go kind of thing, just install OCS or ODF and be on your way. I'm sorry, that one's going to take a little bit. I told you I was going to say it. So where do folks go right now to learn more about ODF data services, that whole gamut of things? I dropped one link in here that I found, OpenShift Data Foundation from the technologies section, but it's just a high level overview. I'm assuming the docs and everything have been updated, right? Like, what else should be fine? Exactly, so we just updated our access.redhead.com site. Let me just fetch the link for you. And we try to make that more obvious what we talk about, what is data services and that. And yes, you'll find us in the documentation. There, you'll learn how to do it. And there's going to be a lot more material out there in the next couple of days that we'll talk about data services, how it is positioned. All the things that I talked about earlier, why the name changed from storage to data services, what does it mean? And... Right, I want to see here. I'm looking forward to... So what are you most excited about? Like, looking forward. Now, folks, this is when we're talking about the future. There's no dates, there's no times being promised here. Like, just keep that in mind and future talk is happening right now. So not saying that this is promised in the next release or promised ever, right? Like, what are you most looking forward to as a part of this change? I know it's bringing people together, which is always good. It's changing people's perception of what they can do with data, but what else? Well, a lot about these talks about the use cases. We want to understand more about what you run on your data foundation and then help you leverage that. Very good example was what Jim already talked about with the object bucket notifications. So you have an application and the object storage can enable you by adding new features to actually do that. And now that we have the object bucket notifications, we can deliver that to all platforms no matter where you run. Previously, maybe I only have that available in AWS or not environmental or anywhere. Now with ODF follows you wherever you want to go. The other thing that I'm looking forward to a lot is the DR thing. I already talked about it a bit, but that will enable you to have multiple sides and have your persistent data available wherever you need them. Side fails, you can switch over. And that is something we're actively working on. The underlying technology is already in step, so it's nothing new. It's not like we go out and we say, okay, it's quite complicated to synchronize data across an internet link. And sometimes you want to do it synchronously, so it's updated immediately on both sides. Sometimes you want to do it as synchronously. And that difficult part is already handled and has been used by customers already as it's, even though it's a new ODF feature, it's not going to be like you have to be careful or afraid to use it, but we want to make it so that the user experience is great. We want to make it so that it fits to the goals of ODF so that it follows you on whatever platform you use, but it's also easy to use. You don't need that storage guy. Chris Short can do it. Yay! We need a stamp, like. Chris Short can do it, yeah, that would be awesome. So right now at that phase where Chris Bloom can do it and then we want to get it to the stage where Chris Short can do it because it's easy, it's in the UI, and we do have specific Kubernetes DR objects that we can use to describe how we want to do the synchronization and that's what I'm looking forward to. And that's also an area where talking about data services takes the next step because over there, we're not only transferring your data, we also need to think about workloads because it doesn't help you in any way if you have to PV over on the other side, but your application isn't just in that one side. We also need to take your application, what we call the metadata and we move it to the other side as well, even though that's not persistent data. So talking hybrid here, let's think hybridly since you've mentioned that, hey, cross-cloud maybe or on-premises to cloud, what are the advantages of putting ODF across a fleet of clusters where data scientists can access it easily and then have like, there's a team over here, there's a team over there, there's one big bucket of data that they use. What is that experience gonna be like for everybody using it? If I'm pulling up a Jupyter notebook as a data scientist, what is that experience gonna be like? Is it gonna be more simplistic? More, I need to learn a little bit or I need this one snippet, tell me more about that journey for a data scientist now. Well, for a data scientist, if you work inside your Jupyter environment, you're already one layer over, so you shouldn't be concerned about storage. And there are different things you can do. I guess the main interesting point brought by ODF is that it brings all the three different types of storage you will need to make data science or data engineering happen, okay? I will take first example, see the team in Ontario that I helped build a data science platform for their COVID-19 research, okay? It's a loose group of 300 researchers from different organizations, different ministries and things like that in Ontario and they grouped up together as a community to work on the data that was available for COVID-19. Short story, they were kind of fed up with the way the government was publishing the data, which was not really useful. Well, the data was useful, but not for researchers because no, it was not raw data, it was not updated in the right way. So they took upon themselves to, okay, we'll do this, we'll do this data aggregation, data scraping and recreate data sets that we can really work with. So I helped them set up this open data hub environment, so this data science platform environment and here they had these specifications that they wanted to be able to share notebooks and they wanted to be able to share data between each other. How do you achieve that? Normally a notebook, when you launch it with Jupyter Hub, then it's connected to your storage, but that's your storage, your kind of stock. But with ODF, oh no, we have also five system storage with CFFS. So that means we are able to have those RWX volumes and meaning volumes that you can connect to multiple pods at the same time. And from this, you can build a shared library, a shared library of notebooks or a shared library of data. That's the first step. Second step is, oh yeah, but we want to be able to access also all those data from many different points on interconnect all those things together. Then object storage is much more suited for this kind of thing. And generally we tend to see more and more people shifting to object storage for this exact reason. It's easy to work with. Now it's built into most of those scientific or data science libraries. But because of this disconnected mode, it's not just five system that you mount on your server. It's only an HTTP request that you can make from wherever you are in the world. So this connection in between your notebook or the container or the VM that is processing the data and the storage makes it really well suited for data science environments. And it's also brought by ODF because ODF has also object storage. And then at some point you will need a database. Oh, for database, I would use blocks because I need this block approach and an intensive workload approach. Well, it's still ODF. So you see, that's where this I find interesting because granted, there are many different storage vendors that have fantastic offers in block or in object storage. But usually they don't have this fully integrated approach all across the board of storage, which is what you get in ODF. As we said, even Eucris, you are able to deploy it in a few clicks and bam, you have five block and object. And then you are able to do mostly whatever you want depending on the use case. So because data science and data engineering is exactly about this, it's always trying to reinvent something because the context changes or you want to use new things or test new things. It's really different from a standard application that say I'm an insurance company and I want to do the architecture for my new application. Well, I will work a few months on my architecture. I will say that, okay, I need this type of storage. I will go out by it. And then I bring everything and it will stay the same for five, 10 years. That's okay. It's not true in data science. In data science, what you are implementing now did not exist six months ago and will be obsolete six months from now. So if you don't have this agility, being able to pick and choose the different types of storage that you need or recreate easily architectures by just using, again, PVCs, person volume claims or object bucket claims or things like that, if you don't have this agility, it begins to get really, really difficult to work with. So again, I think the best thing is that all the F being fully integrated into OpenShift that totally makes it the platform of choice. To set up those data science environment. Plus, you're totally agnostic of the real infrastructure that is underneath. Meaning, whatever you are creating in AWS or Azure as a test, you're trying your things just to learn more or maybe you have a subscription to Rhodes to begin to use OpenShift data science and you're okay, you see, it fits my need. But I want to be able to do something on-prem. Yeah, you can totally do the same thing on-prem because you're not tied to the specific storage that is brought by AWS or when Rhodes will be on Azure, you're not tied to the specific storage that will be brought by Azure. So again, it's about flexibility and I guess that's our main strength here. So the flexibility is the main strength. My understanding is you have some examples, use case demo type things that you could maybe bring to the table here. Yeah, I can show you some of the things I'm doing. Let me share my screen. Screen share dance. Here we go, folks. Uh-oh, what is it telling you? There we go. All right, very small print right now though. Yeah, I will zoom in a little bit for you. Thank you. Is that better? Okay, so for those who are not familiar with Jupyter, it's an environment where you are able to write notebooks. And for an example of a notebook, I'm gonna open this one. Basically, a notebook is a web interface that connects you to a kernel, being the engine that will run your code. Okay, so here we can see, I'm in my environment. So again, fully web environment and I can see that I'm connected to a Python tree kernel. So that means whatever I would be running inside my notebook would be run against this kernel. And this kernel doesn't run on my computer. It's running on the cluster, on the OpenShift cluster in the container that I've launched. That's the first advantage of setting up this data sense platform on top of OpenShift because that means you can bring to your users the full capabilities of a cluster. Okay, I could do this from my iPad. It will work exactly in the same way. But the code that I will run will run on this cluster. So maybe with eight CPUs and two GPUs, 32 gigabytes of RAM, whatever I don't have on my iPad, it will still run. And the way it works with notebooks, you enter your code into cells like this one. And this is a Python cell. This is a Python code, okay. And you are then able to run those cells independently. So I will run the first one, I click on run. And I have the results here. This is what you entered. Okay, hello world, you know, very basic, but it has run only this cell. Now I want to run the other one, perfect. And then it has run the same function. It has run the function that I had created on my first cell, but with a new text. Okay, so interactive way of developing your Python code. This is basic and well, of course, you can take notes. You have cells with code and cells with markdown. And you can create your environments. This is basic, but where it's used, it's for this, for example, this is a notebook that I have that I used to create a model that is able to recognize a risk of pneumonia into chest x-rays. Okay, that's the kind of things that you do in with AI ML tools. And here I have my notebook. So I have first a few small description about what it is, what it does. Then I have my imports and my code. But as you see, it's fully documented code that people are able to read directly, understand what's going on and replay all of those different cells one by one. When you are developing your algorithms and things like that, that's a really easy way to do this because of course you are always adjusting parameters and things like that. And you don't want to rerun everything from scratch. Normally in standard development, you would put a break point and then rerun everything, see what happens at the break point or trying to debug everything. Here you can go totally step by step for each of your function, here each of your parameter and things like that. So here- And because it runs centrally, you can also share this, right? So Geom develops this, has a problem and he calls me for example and says, hey, can you look at this? Why is this Python thing not working? I can just open up his notebook and look at it. I know. That's why it has become so popular with data scientists because, and it's called a notebook because that's exactly what you would do as a researcher doing experiments. You have your research notebook and you take notes. Okay, here I'm running experiment number one with these parameters. You run the things. I don't know what you do. You're in chemistry. You mix up different liquids and see what happens and you write the results. Here it's exactly the same thing. I'm writing, okay, this is what I'm gonna do here on this specific cell and then I have the result. So it's the same workflow but applied to code. Okay, and once you have this, once you have this model that has been deployed, you can put it into motion into a real application. And this is what I have here. Let me first switch back to this view to give some explanation. Okay, here it's working in this way. I have extra images that I am sending into a bucket. Okay, an object storage bucket. But because this bucket has been enabled with notification, every time I am sending a new image, it will send a notification to Kafka, to a Kafka topic. Nice, okay. And I have here in my OpenShift, everything runs in OpenShift. In my environment, I have this, which is a Knative eventing component. It's listening to this Kafka topic and whenever something, some message has been coming in, it will send this message to here. I have a Knative serving component with a serverless function in which I have built my model, my model that is about to recognize the risk of pneumonia. And in this container, I'm making this risk assessment, okay? And then I will save the results into a database. But here you see, we have this workflow where we go from, okay, I'm sending my data to my data repository, which is an object bucket, which is taking part of this overall architecture where it sends message to Kafka and then to my risk assessment bucket and container.  We have here, you know, I started a while ago, the generator. So I'm sending all those images into my object bucket. I have here this, this counter that will count the images coming in, then. I think you need to zoom in a little. Yeah, but it will mess up a little bit to the dashboard, unfortunately. Can you just open up the pipeline progress as a own view? That would be nice. No, unfortunately I cannot. I mean, it's legible, but I can't read the print inside the blue boxes first. Okay, but let me describe, you know, it's what I've described here. Okay, I'm putting everything into my object storage. It's sent to a Kafka bus. And here I have my container that is doing the risk assessment. I have this counter that is counting the number of images that have been recognized so far. And some of those images have also to be anonymized because the model is not able to recognize exactly if there is a risk or not. So for further processing, the images are first anonymized and then they are sent to another process. And here I have all those data about the last images that were recognized and so on with the images themselves. But it's just to illustrate how you go from this, oops, this one, which is your data science development environment. Okay, and here again, you are leveraging different things. You are leveraging the block storage because that's where my notebook is residing. You are leveraging object storage because that's where the data set with my 6,000 or something raw images are to trend the model. And that's a small data set. You know, sometimes the data set that you have to work with is 500 terabytes of data. Of course you don't put it on your USB key. That means you have to have those bigger environments. And that's where OpenShift plus F with its scalability coming to play because you are about to have those 500 terabytes of data residing with no problem to self. And then you can have hundreds of data scientists using this data set, the central data set in object storage. You see that that's where it works. And you have your data scientists working, working on the data set, working on the data, creating those models. And then on the same platform, the very same OpenShift platform, they can deploy the model and use it for real in the real implementation. So again, that's what I find really interesting with the business proposition that we are making here. It's the same OpenShift plus ODF platform that you can use both for your data science development, day-to-day usage, and also application production. You don't change your environment and it's totally portable. So it's, yeah. Nice. So I can put all... Do you have any other demos that you wanted to show off or anything? Or was that the... Oh, no, not at this point. Maybe in a later show. Yeah, yeah, yeah. So the data set was public, I'm assuming, and we're just using it. Of course. Okay, cool. I don't have to put any disclaimers out there. Yeah. Yeah. Like that's incredibly powerful, right? Like to train models and be like, okay, taking this a step further, right? Like this patient had COVID, this patient didn't. What's the difference, right? Like we're gonna have to get through this pandemic. There's gonna be some aftermath, right? Like that has to occur or something has to happen for these people that are dealing with the after effects of COVID, right? Research is being done there. Hell, my wife just told me the other day that some group in Europe developed from mRNA, just like the COVID vaccines, but it's like pandemic agnostic. It doesn't matter, right? So it's like, okay, great. Like how did you do that? Like what data did you consume to figure out that you could create a vaccine to fight any coronavirus? That is impressive. And it seems like we're giving people the tools to be able to do that, like all in one spot, as opposed to having to bring together like your entire IT department to figure this out kind of deal. Yeah, that's why data science has been under rise for the past few years because now we have the capabilities, the processing power, we have the techniques, we have everything to be able to train this model, to do real AIML. The mathematics part of this is really old. It is 30 years, 40 years old, but until mid 2010s, we didn't have the real mean to be able to leverage that, okay? That's not true anymore. Now we have this. And here it was an example with image recognition. This is now pretty much standard. Image recognition is easy, but there are many other things that we have tried for COVID-19. For example, someone trained a model, you just cough a little bit on the phone and it's able to detect if there is a risk or not. Here it's the same. It's about having those thousands of samples of people coughing and training a model to be able to detect what the human here cannot do, obviously. So it's those tools that we are bringing, that have been brought into the world that were reserved for the past few years to some specialists and it was really difficult to use, it was really difficult to implement. Now it's a little bit more mainstream and by bringing it on top of OpenShift, it's even more mainstream because it's the standard platform that you may already have in your enterprise and most customers I'm working with, they already have some OpenShift installation or some OpenShift knowledge and now they're interested into this data science thing Oh yeah, we have this data and maybe we think it will be useful. How can we do this? Well, you already have OpenShift. Let's deploy OpenData, five minutes from now, you have all those tools. Oh, perfect. And where do I start? Yeah, if you notice like I have a bigger data set than I expected, because of OpenShift, you can use a machine set, you scale it out with a different instance type that is bigger, you don't need to touch anything because the OpenShift thing is handling all the installation and once you're done, you can get rid of it again. Yeah, and I have customers I'm working with that are doing exactly that. They have those huge processing to do periodically every 24 hours. You know, it takes tens of machines to be able to run that. But of course, as it runs in the cloud, they don't want to keep it running 24 seven because of fortune. So now it's part of the workflow at the beginning of the process. They will just increase the machine set. It will spawn some new things. Then they will launch the process using those, you know, those data science tools, Spark and the rest. They will do the processing takes a few hours. And then when everything has been done, to just scale down the cluster and they save a lot of money. I mean, this is really reminding me of a time where I worked for a financial services like marketing company and the data science team, we were having so many problems with like infrastructure and all this other stuff, right?  And now I waste it all that time and money, right? So OpenShift, like it puts all the power in the people's hands is what it feels like, right? Like I don't have to worry about some other team or some other, you know, configuration touching my workloads. It's, I'm managing this now. I have a machine set that does what I need. It spins up, it spins down as I'm processing data and off I go kind of deal. It seems really powerful, right? Like... And it's, you know, it's really close. You know, we know, we've known for a few years now all the benefits that OpenShift can bring to development. Okay. In general, all these flexibility, agility and everything. It's about bringing the exact same advantages to data science. It's really well suited. Now that most data science tools would run and will run into containers, then it's, oh, fantastic. Now we can run them on top of OpenShift and we can use all the know-how, the skill set that we have developed around DevOps and around creating those infrastructures to data science. That's perfect. And when you add to the mix, SAF with ODF, then you bring the scalability and the performance that you need for data science because it's not only about, you know, storing a few data here for now more and more people, we're talking about petabytes of data and petabytes of data that have to be, that have to be processed in as a small time as possible. So meaning you have to have performance on the storage board. And that's where SAF shines, you know, especially with the predictability of performance, this perfectly straight line, the more capacity you add, the exact same performance you get, that's really important in data science. You don't want to be, okay, now that I'm reaching over one petabyte of storage of my specific stuff, the performance are totally dropping because the storage is not able to cope up with it. We don't have those kind of issues with SAF. So it's kind of bringing best of both words, storage and Kubernetes to data science. That's why I'm so excited to work with it. It's, yeah, perfect patch. That's amazing. Yeah, but in my daily life, I'm not actually handling a lot of fake data or I'm not wearing lab codes or anything. So one thing that I want to mention about Jupyter Hub is it's not just to do what you showed us. You can also do regular development in it. And maybe Chris, you can share in the chat a link that I just sent. There's like a list of all kinds of kernels that you can use. I'll mention it in the beginning. The kernel is the language that you write in your notebook. And there are kernels for pretty much anything. I like to see that there are Go kernels so you can write your Go applications in the Jupyter notebook in your browser, share it with anyone. Or one thing that's very popular and that's pretty cool is you have an Ansible kernel.  You want to test it, you want to document it. You can start writing your Ansible playbook in Jupyter notebook and test it in there. And then you can immediately see what it does, what the output is and all of that. That's pretty cool. Yeah, I'm about that life. You say Ansible and my years like that, obviously. But yeah, I see it right here, Ansible Jupyter kernel. That's awesome.  But when you come from a different background or you're not, you know, I'm not a full-blown developer, that's not what I do. I have the same approach as Chris. You know, taking the best thing depending on what you want to do. And for Ansible, I've never done this before, but I have tons of Ansible playbooks to rewrite to deploy those demos into our HPDS. But I totally see the point of, oh no, I want to test only this part of the playbook and not rerun everything or just comment the parts that I don't want to replay because I'm just working on this part. That's the interactive mode proposal from the notebooks that is interesting in there. So, yeah. And you can document it in full markdown. So, it's also great if you want to teach someone to learn a certain language or Ansible, whatever. There's also bash kernel. So, if you want to teach all those millennials what you can all do in bash, then you can write a notebook and make it fancy with the markdown, tell them exactly, hey, this is a forluse and that's how you do it. And they can run it and see it immediately what it does, what the output is. So, where do folks go learn more? Is it our normal learning places? Like learn.openshift.com, for example? There are things on learn.openshift.com related to roads, Red Hat OpenShift Data Science. And let me check. Yeah, I was going to pull it up too. We need some elevator music. Sorry, I do need some elevator music, Bobby. So, for folks that aren't aware, I have an intern this summer and I'm very happy about that because he gets to take notes and tell me what I'm doing wrong because he has production, like this kind of production experience in his background. So, or not this kind, but like movie production experience. So, I'm sure he's like flushing or whatever in the background. But yeah, I like talking about my intern. So, yeah, some mood music as I'm searching for data here. AIML? Yeah, in the meantime, I can read some more kernel. There's Redis, there's PowerShell, Wolfram. Wolfram, oh, okay. Yeah, you can write Wolfram, Mathematica stuff in it. I like them because I have this app on my phone when I was a student. I didn't always have everything in my kitchen. So, sometimes I only had a weight. So, I wanted to know, okay, how much does 100 grams of flour weight? Right, yeah. For 100 milliliters of milk? Kind of deal, yeah. How many whatever leaders or you had to do a conversion or something? Yeah, and obviously Wolfram would overdo it. It would tell me like, okay, you have 100 milliliters of milk. Is that like whole milk? It wants to know the consistency. Yes. Because it would differ by probably two grams. Right, exactly. All right, I mean, let's not belabor the fact. We are approaching the top of the hour. Is there anything else we wanna talk about before we sign off? We don't have any questions in chat. So, or at least I haven't seen any, I hope I haven't lost any by just not looking at YouTube and Twitch directly. Okay, no, I haven't. All right. So, yeah, like anything you wanna sign off with? I would reiterate that, you know, for the part I'm working on, which is data science and data engineering, the important thing you have to consider when building the thing is the platform. Okay, it's not the tools only by themselves. The tools are easy to figure out, but it's a platform. And here, running those kind of workloads, you know, AIML or statistical workloads or pure data analysis on top of OpenShift with everything that goes with it, you know, all the F, all the other components that we have several S and so on, that makes a great platform. So that's, that would be my takeaway from this. Beautiful. All right, so don't ignore all the other things that you see happening in OpenShift.  All right, so thank you, Giamme and Chris. I appreciate your time today. As always, later on the channels, 11 o'clock Eastern, 1500 UTC, we're gonna be talking about the value of get-ups. And we're gonna have some guests on, so please tune in for that. And until next data sign or data services, Office Hour, we will see you then. Stay safe out there, everybody, for real. See ya.


Hey, Ty. How you doing? Alright. Alright alright alright alright alright alright alright alright alright alright alright. Hey ladies. Hey ladies. Uh, ladies and gentlemen and everybody in between. Welcome to Church of the Chair where we celebrate all the things we do while seated. I'm your host E. And I have internet issues.  Hang on, lemme change something. It might drop out. But if it does I'll just restart. Alright. Howdy ho. Howdy ho. Howdy ho. what's up Derek been a bit of a day but I'm okay now I'm sorry you had a rough day I'm glad you're okay now so Derek what's up Zeely so Chad wants to play with me and I think that would be cool if we can figure it out Judas full shit okay it's true I think you're all something else I swear how's the stream guys is it better is it good I forgot to unplug the ethernet cable from my computer and I'm gonna get a new router here in a couple days I don't really have a choice I need something that splits that splits the whatchamacallit hang on let me think that I can plug in two ethernet cables and the one that I have now is just seems to be better okay yeah I was hoping that was the case and it I don't know it's it's got to be some kind of bandwidth or RAM issue something like that because if the PlayStation isn't plugged in with the ethernet cable it glitches on here and that doesn't make any sense to me anyways because what what does the ethernet hat ethernet cable have to do with the quality of the because I'm not streaming from the PlayStation I'm streaming from my computer through a capture car hey solvents how you doing oh Zeely let's get you a moderator badge hang on I got to go over to the actual stream we all know and trust Zeely why why is Zeely not a not a mod yet that's ridiculous is what I got to say about it all right you've been you've been upgraded Zeely let me upgrade you upgrade you can you guys hear the game at all like whatsoever yeah okay for some odd reason it's not showing up on my of course Jude J rod says hello to everybody as long as they're not a jerk J rod super sweet Zeely the only the only thing you got to worry about as a moderator is if you're in here and you see someone acting an ass put them time out if they come back and they continue to act like an ass ban them from the channel people get one shot around here that's it especially if you see anybody with a suspect username also if you see new people come in here the way we do things around here is you wait and your weight you wait for me to acknowledge them and if I acknowledge them then you can say hi and that talk amongst yourselves got it thank you you rock Zeely all right so it says I have two viewers in here and there are so many of you oh zoop I got to send you tracking your thing went out today tie I got to send you tracking to both your packages went out today I also got to send it to the two people who won last game night it says eight watching on your end well I mean I guess I can I can click refresh chat I love stream labs y'all I love it so much but I never actually know how many people are actually watching is it it's literally still says to two viewers but I know y'all in here I know y'all in here oh I hate this so much but I'm gonna have to start doing it y'all like the stream if you like the stream dislike the stream if you dislike the stream you mean give me them thumbs give me them thumbs also I'm gonna start pushing memberships so expect that bullshit thank you solvents hey it but honestly if if you dislike it why would you guys be here if you dislike it but if you dislike it hit the thumbs down what if I want to keep my thumbs hey you can keep your thumbs elsewhere man I ain't got nothing to do with where you keep your thumbs you can put your thumb wherever you want to love you Jude I gotta get my phone out in case mr. Lutz keys like yo dog where you at from the front to back no I don't hear that what's the face for solvents alright we're gonna get started here in about eight minutes yeah you can put I'm not I'm not I don't hey whatever you want to do with your thumbs I'm an equal opportunity employer when it comes to thumbs if you want to keep them thumbs you'll use them on the light subscribe and membership but I just start threatening people be like listen listen if you don't want to end up in a Chinese finger trap yeah that is genius promotion a Philip DeFranco threatens to punch people in the throat so I mean people like that shit I don't know why there's a dude on tiktok who who's hilarious and at the end he goes follow me you fucking idiots and that dude has a millions of fucking followers so once again I don't really care about the subscriber numbers it's when I get like a mass exodus that I'm like what the fish okay whatever I always I always have to double think myself was like what did I do now what did I do now what I do and then there's also the trolls who follow me to and then get ignored and then they're like then they'll delete all their accounts and that drops off but I'm not getting any new subs either so hello AJ how are you doing my page number stopped working or what oh your page number stopped working on the book you're reading okay um maybe have you tried closing it and opening it it happens to me every now and again what does it be just joining us we got another five minutes before we get started I'm gonna get started at 5 30 I just want to hang out and chat with everybody before we get started also it helps you know not get the stream demonetized automatically also I'm trying something new I just put the Texas chainsaw up in up in there because I think massacre is the word that's getting it demonetized yes I don't know what's up I don't I don't know either I'm not no I'm not no job AJ are you coyote no you're not coyote Smith are you why do you look familiar why does that I think I wait a second never mind if you're who I think you are don't say anything Texas chainsaw happy fun time hour who are you mating in the game Anna Anna is my chick that's a good point solvents I had a whole moment yeah I know I know I got you I got you I did I just didn't I mean anyways never mind I that's that the last minute that's when I realized who you were I was like yeah if you don't want to say you don't have to but anyways solvents I was reading your thing is like the mass I was like what are you about to say next I got scarred I got scarred there for a second I'm actually making my posts for my Sims gameplay yesterday and then going to turn it on some more yeah parallel play awesome I prefer the Dallas lawnmower killing spree myself I like the California weedies or debacle that's a good one or the main hedge clipper party I don't know I lost the thread I already lost the thread I can't beat it yeah it does that's what I was getting at solvents that's exactly what I was getting at King takes a big risk with that character took a big risk all right we got three more minutes we get started I don't think I have any points to spend so we're gonna jump right into it once once I get started I know you didn't meet it like that I know that I realized that you were trying to say massacre and without saying it yeah I got the hiccups solvents solvents over here actively putting their foot in their mouth I'm kidding it's all right it's all right I was talking to shell about this game is like I'm actually pretty damn good at this game like I was watching other people play it and they're just getting killed like either super quick or it's like dozens of death in a row without a win I'm like dude I'm doing pretty good my my KD is like relatively decent I played about 20 games and I've won five that is not terrible in these games very boring don't don't don't have kids y'all don't have kids they come in and troll you during your stream Brian let me see let me let me check one more thing before we get started we got a minute left to go but I want to check something all right yeah we good all right closing that out in three two one let's get in here all right we going victims always victim the servers for the oh no y'all we might have to play armored core or something else the servers for the Texas chainsaw massacre literally playing right now awesome bookable well check yo discord and text messages fool all right hang on that's if I can even get in here look at this look at that all right hang on let me let me check my internet stuff I've been waiting for half I'm sorry give me a second I'm having problems getting in here so let me check my internet settings on my whatchamadoogie I do like to be the victim ask anybody on Twitter I am totally just a professional victim I know what it is my network isn't no it's connected I don't know hang on y'all I'm testing the internet on the PlayStation real quick it might be me it might be everything I don't know yeah it might be it might be then armored core does slap yeah I'm good I'm good we're now running at a hundred ninety thirty point four megabytes per second so we're good no that's not what I wanted all right bring it back up nope okay so what are we playing and because Texas chainsaw said no Chad if you're in here the one time we get we probably could have played together are you kidding me don't see anything on Twitter about servers being down I don't know man met hang on let me I guess let me restart the PlayStation maybe that's it let's go with a refresh because it's not the internet and also if it just went down Dallas lawnmower killing spree and the California weed eater shenanigans I don't know what I said I understand you're not getting that error but I am you on them fancy Xboxes what is even on my screen is that like a little tumbler full of marbles what the hell is that try the family all right yeah someone with a PlayStation jump on to see if you can oh there's something wrong with what's up cross if we can get into this damn game the sky listen I'm not gonna have any back talk is not the game server itself well they got different servers for different stuff so oh by the way guys if you hear me talking to someone named cross that snake it's going pretty good dude it is going pretty good yeah we're doing it give me a second give me a second hang on please please give me a second let me see if I can even get this thing working oh there we go alright okay let me back out of here let me bring up discord and then I also need but Jersey all these cables and wires and shenanigans and ish I'm coming Chad I'm coming he's like bro pause time to get eaten by cannibal not if I have anything to say about it discordian let me call this fool real quick answer your phone fool you're on mute too I see how it is I'm not even a hundred percent sure can you hear me yeah I can hear you can you guys hear Chad Chad can you hear Chad Chad Chad Chad Chad yeah okay they can hear you yeah they can hear me alright we good good I don't know how this works with a like inviting me or whatever I think you got to go to play and then create lobby yeah send me code there you go it's on the screen everybody and they mama finna join probably shouldn't have done it that way well it's 5 y RT R7 hold on hold on damn cross what you got against Chad unfortunately we can five five Y Y RT yeah our seven in Valico five Y RT R7 five Y RT R7 yeah let me five way it's it doesn't have an option for huh well why is it saying envelope here send me one why are well hang on I didn't X that of it go go ahead and send me what you damn right Derek disrespective Chad will be met with great vengeance and furious anchor yeah it's cross-platform but only on modern consoles and PC like Xbox one can only play with Xbox one huh for hang on Z Z is in zebra M is in monster hang on for to Z M to E3 E3 I like that one T3 T3 hi show give me a second baby I'm coming out is that better it's it yeah now mine saying invalid code okay let me try I don't even know if I've created a yeah I get that I don't know if I did either private lobbies require a game party of four players please select party options what that yeah it says down here that my lobby is still created the five Y RT R7 you stepped away for a sec how's it working it's not working right now well it I mean the game's working but what's your coat my coat is five what's your coat again yeah it's five Y as in you are as in reach T is in tacos you're all right five five way R2 T T is in taco R7 yeah hang on oh wait a second is it because there we go you got it all right I got it all right I got you yeah all right so we're playing victims right or do you want to I don't know what I want to play I'm coming out for cannibals to eat by a meat so sweet I'm coming out for cannibals to eat yeah my meat is sweet all right don't don't let me all sing that in public in public victory tacos hey cross you got this game I still feel like I'm too loud my mic keeps peeking into the yellow if I love I don't see my level chatter cheese you're zero level eight you got zero yeah I don't want to hear about how sweet your meat is too bad you just heard it family house night yes all right let's see here what what are we wearing y'all hello those what are we wearing we need a vote I wore I wore the I wore the clothes that I thought would be more the most camouflage the most camouflage I'll probably get armor core I got you yet armor core is great because of all the raging going on it's a nearly perfect game I don't have a single complaint all right zoop was first we got yellow so last time I wore yellow I won your stream volume is really quiet got to turn my phone up really high it says I'm peeking all right let's let's go up a little bit higher no you're you're perfect in my head in my ears all right we'll go somebody keep track we'll go yet we're going yellow this time we're going blue next time then forest then beach or Basie or H sorry solvents I had to wear nothing they might be flattered and let you go no that only works when you play as Julia Julie let me hush she'll told me I can't talk about Julie no more she was like I'm gonna murder you in your sleep and I was like all right fine I got you who needs to blend in when you could be a shining yellow light in the shadows yet to turn your phone is it is it better should I should I go louder I really don't want it to like start peeking and breaking and it says right now that I'm talking in the yellow I got this much green and then about that much yellow good luck to you too thank you thank you I'm okay I'm a carry Chad if I have to so how you only four more levels higher than I am but I don't have but have you been playing as Leland okay then each character has their own individual level you think this sounds good now okay I've only played the game nine times hey that that's cross that's the whole reason why I don't play on PC because I don't want to fuck around with the hackers hi y'all here we go oh you're talking about this game I thought you know my armored core it's whatever I mean it would take I'm gonna start talking shit it would take two leather faces to take me down no I haven't there I will tomorrow though tomorrow's uh we're gonna do some armored core tomorrow but they'll get banned eventually it's it's not gonna be a thing I don't know if we should stick together Chad this game is not really one of those games but I mean if I get an opening maybe you I don't know no no never mind never mind I thought you were I thought you were following so if I get a door open I'll let you know problem is I don't know how to describe it same here but I don't know how to describe like over here yeah I got you me actually I don't think I'd be good at that either so what the hell am I even talking about hi down to pay attention hello mr.


Thank you. Thanks for attending my talk. I hope you had a great lunch and you are not too tired and You can have a nap, but just be quiet. Don't snore. I would love to have a nap, personally. So yeah, as Antoine said, I'm going to talk about PKG base, which is a big project in FreeBSD to use PKG to to distribute and update the base system. So first of all who I am. So my name is Emmanuel Vado. I'm Manuat FreeBSD.org. I'm a FreeBSD user since 2004. Source committers since 2016. In source, I mostly deal with arm and arm64 related stuff. I'm a FreeBSD port committer since 2018 and they are also deal with arm and arm64 related stuff, most most of all bootloaders. And I'm a freelance developer. So what is PKG base? It's using PKG for packaging and updating FreeBSD base, not really surprising from the name. PKG is a default package manager since 12.0, which means that every supported branch is using PKG. The old PKG tool have been deprecated. The goal is to split base into multiple packages. So that package manage and started a long time ago in 2015 by Baptiste Dawassin, Baptiste at FreeBSD. He first presented a talk at PSD Count 2015 and he started around January or February of this year. So it's been a long time. It was abandoned for a lot of time and I've picked up last March. So the goal are to provide binary upgrades for all the FreeBSD release, the FreeBSD stable branch, and the FreeBSD curable branch. It allows fine-grained installation. So if you want to have a system without send mail, without NFS, without toolchain because your appliance or your use case don't need it, it's better. So you can do that really easily. It allows PKG to deal with configuration file. PKG have a freeway merging mechanism where it's more or less you will never have any problem if you do some local modification on some files. So when you will update the base system and it will update a new default configuration file, it will do a freeway merge and most of the time you won't have any problem. And it allows developers to provide package for users to test. For example, if somebody said to me, okay, so I think that this driver is broken but I do not really know or want to compile a new kernel or some specific part of the OS, I can just send it to him one package and he will install it, test and provide feedback and then roll back to the official package. It also allows us to provide some, for example, some kernel package that enables some new flag, for example some new way of doing locking or whatever FreeBSD could provide a new kernel configuration, put some package and this will be easier for users to test those modifications and report problems. The goals are also that everything must live in the build system. So currently you build your package with the make packages targets. We don't want any external tool. So not using the port street, not using some shell scripts that live in a user branch or whatever, we want everything integrated in the build system. It needs also to be able to build and create packages as a user, no need for root accounts. And this is currently the case. It also needs to be able to create package for other architecture. It's really important for small embedded boards where if you want to build the base system or even the kernel, it could take ages. Sometimes you saw people on the FreeBSD Armani English that say, oh, yeah, I've just run a make build wall on my Raspberry Pi, which is running at maybe 400 megahertz on an SD card, a really slow SD card. And yeah, and two days later, I'm an age to do it or it broke. And you don't want to wait two days just to test an update or just to update your system. And so I wrote, I want people to create FreeBSD distros. What I mean by that is that I see a lot on the internet people creating Linux distros like based on Debian. For example, for our board, you have something called a pie hole, which is just a Debian based on a Raspian. So Debian distros for Raspberry Pi that are you to have a DNS hole and not have any ads on the internet on YouTube, etc. Having FreeBSD package will be way easier for people to create some FreeBSD distros that is done for a specific task. You could replace FreeBSD distros by Appliance too, but Appliance is like more professional way and distros is like, yeah, I just did that. I put that on YouTube and if you want to donate some money, so I continue, etc. These two Appliance is more professional. So how the package are generated? We first install a fake root during the make target called the wall stage and kernel stage. This is called automatically when you are doing the make package, package these targets. And so everything will be installed in your OBGD under the under directory called wall stage and kernel stage. And it uses two mechanisms. The first thing that it passes dash D no root, which will not call install with dash O option. So it will never set the correct ownership or set the correct rights. Instead, everything will be returned in a file called Metallog, which is just an M3 file. And so you will have the wall stage that are just a file just copied and associated with that you have a Metallog file where you find the correct right, the correct owner, etc. And TAR, more precisely Libre archive, which is used by TAR and by PKG, can read a Metallog file. And so when you will create the package, it will just read the correct right and correct owner from this file to be associated with the file. It also had some tags in the Metallog. So that's just M3 tags, which contains the name of the package and this will be used to choose a destination package for a specific file. By default, if someone, if some part of the build do not specify a package, everything will go to the free BSD dash utilities package. This is a package that I created last week or maybe two weeks ago. It used to be different one for the default package. It's it's done like that because it's just easier. I will talk about later on what's the plan to split the free BSD utility package. If you want more information on how this is done, you can look at the BSD.mk file in share the slash mk and you will see everything about the package creation. As I say, mk file can override the package by just putting package equal blah blah blah in the mk file and that's how you choose the target package. It uses UCL, so that's Universal Configuration Language. This is language used by PKG for the manifest file. It also used in base for different things. I think JAILs can use UCL file. What was the tool that you converted to C-Slog or new C-Slog? UCL is a bit used in free BSD. We would like more use of it and since package already used it, it makes sense to use UCL to define the package and all the UCL file that defines the package for PKG base are under the release slash packages directory and the playlist creation. To create a package, you have to provide a description of the package like who is the metna, what's the name, the version, etc. And you have to provide a list of file and this is automatically generated with the help of the metna log file. And what the make packages target does, it not only creates the packages, it also creates the repository. So at the end, you only have to R-sync or SCP, the directory and have it served either locally or by some HTTP and you can distribute package for everyone to use. So, pros and cons, it splits. First of all, we cannot please everyone if I think that some part of the base system should be in its own package, someone else will have some other arguments, not better, not worse, but another argument. So yeah, we cannot please anyone, everyone. So for now, since I'm the only one that's working on PKG base, well, I have my arguments and you have to deal with it. So unless you have a really better argument, we will go my way. Thank you. First of all, the current split is not final. In the past month, I've done a lot of commits that changes a lot of stuff and I'm still not done. I hope to Yeah, I hope to set up maybe in a month or two, but yeah, we'll see. So first of all, each kernel is in its own package based on the config file. What does that mean that you can have multiple kernel package? So for a release, think about distributing a generic package and a generic slash debug package so that would allow users to just install a new package to use the debug kernel, provide information to the user, etc. For current branch, the default is to build the debug kernel and have a dash, no debug. So the same thing, if you want to run current but don't have all the overhead of running debug kernel, well, you can choose and if you define your own your own kernel config, it will also be created and it will be in a package. We have, of course, FluBSD bootloader contains everything bootloader related, so that's all the file in slash boot except the kernel. So all the configuration file either Lua or Forth, etc. We have the SELIP package that contains the Serentime, so that's the runtime inical, libc, lib thread, etc. The reason is that they are in their own package is that so I'm not sure that I've understood everything correctly. There is a mail from Kibco-Sorten-Beliozov from April 29 on the PKG based mailing list. So you can have a look just to be sure that I've understood correctly. So all the external ABI are backrock compatible and will always work between, for example, an old binary is linked with an old libc, but you update your libc, it will work with a new libc. But all the internal between libc, lib thread, etc. are not stable and will never be stable. So you have to update them more or less at the same time. And if you put them in a really, really big package with a lot of other stuff, there is a lot of chance that libc will be updated first because alphabetically it's one of the first files and lib thread will be updated very, very long, very long time after, so you will most likely have a big breakage. And we'll have to boot from external media to recover your FreeBSD installation. We have the FreeBSD runtime package. This used to be the default to-go package. It was very big. It was, even at the time, it was not correctly named. So, and now it's even less correctly named. So I need to change the name. I still don't know how I will call it, but the goal is to have everything to be able to boot into a single user and repair on this installation. So this means is that if you install the FreeBSD kernel package, the bootloader, celebs, and runtime, you can boot onto a single user and have everything to repair UFS or ZFS disk. You have enough network tools to be able to configure the network and maybe grab some file, et cetera. So that's one of the goals. So maybe I will really need to FreeBSD core utils, but core utils is used in Linux. But yeah, I mean, the name makes sense to call it core utils, but I don't feel, so yeah, it will change. I still don't know what name I will choose. If you have any idea, please. What? Prepare? Oh, repair. Yeah, well, it's not only for repair. I mean, this is also tools that you need to have a full FreeBSD. Naming is hard. That's all. You have the FreeBSD-RC package which contains the RC sub-system. I've split those not so long ago for two reasons. There is some people working on OpenRC for FreeBSD, the IAC system guy. I don't know if we will ever see some port or whatever to be able to use that in FreeBSDvania, but at least know the RC file in their own package. And if you want to just experiment with another RC system, it will be easier. And also if you just take the FreeBSD runtime, the FreeBSD-C-LiP package, it's very handy to create a small MFS route that have everything to bootstrap to everything else. And most of the time when you do a small embedded MFS route, so that's a memory file system that is loaded by the kernel or integrated into the kernel, you will have a specific slash RTC RC file that just do what you want to do. So that's the reason it was splitted. And yeah, as I said, FreeBSD utilities, it's not the default package and it contains a lot of different things. Yeah, it's the to-go package, so there is still some stuff that I need to take out. Every package is also split with dash debug, dash development, dash profile package. That means that also debug file will end up into a dash debug, development file, so adders, etc., in their own package and same thing for the profile file. That's really useful for FreeBSD users because most of the time you don't need debug file, you don't need development file, you will just install your FreeBSD on your server, install some package and run it and if you have any problems, you can still install the dash debug file corresponding to the subsystem that you want to debug, do your debugging and remove the package. Same thing for development and profile. And this is done automatically based on the type of the file. On 64-bit ARG that have 32-bit support, so that's only MD64 for now. We also create some dash lib32 package. I don't know if anyone still use lib32 on MD64. I personally have no use for it. I'm pretty sure that there is a lot of use. I think that wine on MD64 at least used to use those, but I'm not sure anymore. Everything coming from Contrib, so some stuff that we pulled from another repository, are in their own package. The reason is that for security advisories or error attenuities, you are usually not, you could be not one in advance and since you need to roll out a package very very quick, it's easier to have this in their own package, so you will only have some part of the tree to roll distributes. FreeBASD tests contain all the test suites. Again, most of the time people don't care about tests. Users don't care about tests. Only developers should care about tests. And I'm not sure that a lot of developers do care about. I personally don't really, sorry. I'm bad for that. I know I'm bad. You can boo me. At least I'm honest. I know that other people will not say the same. So anyway, everything in the FreeBASD tests contain all the test suites. Maybe we should pull the QIA, some QIA binary, we have some QIA binary entry. Maybe we should just put everything in the test package. I don't know. And the other package are application or lib-specific. For example, I've created not so long ago FreeBASD does Bluetooth package. You don't need Bluetooth in a server, I hope. You don't need Bluetooth in a jail. Same thing for WPA, OSTAPD, etc. So I've created those packages. So if you don't want them, don't need them. Well, it's there for you if you want to install them. But I don't think that most of the people will use them. And yeah, I will continue to move things out of the utility package. Maybe I should move NFS out of it. Again, you don't need NFS in a jail, unless you have some really real jail setup. Same thing for Kerberos. I used to love Kerberos and use it a lot when I was at university. I don't think it's very, very used outside of the FreeBASD cluster right now. So maybe it makes sense to just put everything in a Kerberos package that will also possibly help if people want to install either MDAL or MIT Kerberos from Ports. You will not have conflict with the base Kerberos utilities. I don't know. So at the beginning of the package base, there was a lot of complain about the number of packages that we have. It used to be around 800 that was, if you count debug, development, profile, and lib32 package, apparently it matters to some people. For me, it only matters because if you have a really huge amount of package, the first time you install is very long. It's way longer than what we are currently doing, which is just extracting base.txt and kernel.txt. Instead of dealing with two TAR files, you're dealing with 800. So of course it's longer. I personally don't really care. Apart from these reasons, I don't care to have a huge number of package as long as the splitting is done in a logical way. At least one that could be explained. Right now, we have 300.92 package. If you don't count the debug, development, profile, it's only 118. And if you don't install lib32, it's only 80 package, which I think is good. It's not that long to install. As I said, I will continue to split up some stuff off the utility package, but I think we can have a target of around 100 package and it will still be okay performance-wise. So, yeah, that's what I'm planning to do. So, in Flubia's system, we have a with and without. In Assassin's Creed.conf, it's at control what will be built. And also, if we build with some library supports, with PKG base, you have two ways. Well, there is two things that will happen if you start to tweak. If, for example, you don't build without APM or without MD, those package will simply not be created. If you don't build, for example, without Capsicum, it will change the content of the package. For now, there is no resolution. Ports have a mechanism called Flavors. So, we can maybe add something like that that we have, for example, FreeBSD-utilities-noCapsicum. I don't really have a plan for now to work on that, but yeah, we need something. Another complaint about from user was, yeah, each time I'm rebuilding my system, I make the package, I need to reinstall and upgrade every package even if the content haven't changed. And this is true if you just do what I just said, so make packages and distribute the package. The thing is that there is a way to provide some good package just that it was never documented. I think that must slide as the only documentation, so I will try to put that on the FreeBSD Wiki, which could be useful for people. So, the first thing is to define with ReproducerBuild. ReproducerBuild will strip all the dates in compiled binaries, et cetera. So, that means if you use the same compiler to compile the same source at any point of time, you will have the same binaries, the same binaries produced. This is the default on release and current, and it's not the default on, sorry, release and stable, and it's not the default in FreeBSD current. You need to define source date epoch.  Yeah, I didn't, I tell you about it. No, you didn't. So, yeah. This is used to tweak LibarCive output. It will just set the timestamp to every file to the timestamp that you set in this variable. You need to pass ReproducerBuild to make to have the repository add some pass in your system. You make packages, and then you can distribute. So, this is how you bootstrap your package. For example, for 13.0. Then, when we will have 13.0-P1, P2, P3, meaning we have some security advisories or ERETA, what we will do is, again, define ReproducibleBuild. We use the same source date epoch as the one we used to bootstrap. You need to set package version to make, because by default, if you make the package, it will be called 13.0-P1, and it will be called that also inside the tar, inside the manifest of the package. So, you need to use the same value as a bootstrap run. Use temporary Reproducer. You compare the package with a bootstrap run. You can just use MD5, SHA or whatever. You regenerate the package with the correct source date epoch and the new package version. And all the packages that have changed, you delete them from the original repo. You copy the new one. You regenerate the repository. And then you will, for example, only the kernel have ERETA, you will end up with just one new package, which is freeBSD-Carnal-Generic-13.0-P1. There is nothing currently in the source tree that does that automatically. This is something that I'm working on and I hope to have everything ready soon. There is still one problem with package that if you, for example, I create a new package and people just pick a G-upgrade to upgrade the system, the new package will not be installed unless it's a new dependency of an existing package. So, there is a plan to use pick a G-group. This is a feature that I am currently developing. Baptiste designed pick a G-groups a long time ago, but only on the paper. And when I'm in paper, I'm in his head. So, this is basically meta package like we have on the source tree, but at the repository level. So, when you create your pick a G-group repository, you give it a directory with a lot of pick a G-groups, and it will create fake packages that only have dependencies. So, then the freeBSD installer could just have freeBSD-bassgroup-debug-slip32 and we just select which groups that you want to install. And, yeah, since you are updating a group now, a new package will be installed automatically on updates. For example, a few days ago, maybe it was yesterday, Matthew Siemens sent a mail to pick a G-base and said, yeah, I've updated my pick a G-base system and I was missing a lot of files, a lot of package. And, yeah, this is the problem and we need to solve it with pick a G-groups. One thing that I plan to do, maybe not for the first one of pick a G-groups, will be multiple candidates for one package. So, if the freeBSD release engineering team wants to create a dash-no-manual or dash-no-capsicum package, in the group, you will have as a candidate for freeBSD-utilities multiple candidates. So, that will be the first one that is without any src.com for tweaking. And, you could have another one which is with src.com for tweak. I think it will be good, but it will be probably on the pick a G-group version too. So, current work, I have bsd install support. That means that the installer is able to install a pick a G-base installation. It's a bit crappy but I don't plan to uncrap it because I want first to use pick a G-group. So, yeah, I just made the patch so I could test that everything was working. Same thing, I have a release image support, but there is no point of committing that if the installer don't have support and the installer needs a pick a G-group. But at least it's done and I know that it is working. I've made a script called canalsedex which will allow to... So, on the pick a G-base system, when you install the kernel, it will install in slash boot slash kernel dot canconf. And kernel select is just a tool that will auto-register the kernels that you install by registering a link at slash boot slash kernel so the bootloader could find it. And it will also remove itself and fall back to a default one if you remove a package. For example, if you have installed your freeBSD 13.0 with pick a G-base and you want to test the freeBSD debug kernel, if you pick a G-install freeBSD dash kernel that's generic debug, it will auto-select it for the next boot and if you remove it, it will remove the link and use the previous one or if the previous one is not installed anymore, it will just use the freeBSD dash generic kernel. And I need to do more package splits again. So, if you're to work, I really need to talk to release engineering so we can have official package even if we say to everyone, don't use it or use it only for tests. Not having right now a package in an official repo means that people cannot really experience correctly the pick a G-base project. Yeah, I think it's bad. I still need to do a freeBSD update, a utility. Right now freeBSD update is just pick a G update, pick a G upgrade and everything is done correctly. But yeah, I need to do that in a script shell that will take exactly the same argument as freeBSD update. So if people have Chrome job or, I don't know, anti-ball or salt job that does freeBSD update, it will just work for them. I might want to do pudria-image support. So pudria-image, pudria is a tool to build package and can also build an entire image of the freeBSD system with the package that you built. And yeah, we need to add some pudria-image support, but if someone else wants to beat me to that, I will be perfectly happy. So are we there yet? Not yet, but much closer that it used to be and at least we have someone working on it. If you want to help, there's still a few bugs in the MK file. I'm honestly very surprised and very scared when people tell me that they are using freeBSD base, pick a G-base for a long time because I honestly don't know how some stuff could have worked. Right now I'm only aware of one bug in the MK file and it's not really a bad bug, it's just that some profile and debug file end up in the wrong package. So you will still have them, it's just not the right package. And if you want to test, just create a minimal freeBSD base package and install, for example, freeBSD-iSQZD and try to use iSQZD just to see if the split was done correctly, if the package contained everything and if all the dependencies are brought in automatically. I will not have time to test every package independently, so yeah, we need some help on that. And there is a mailing list, pick a G-base at freeBSD.org. Just an email, I try to read, I usually don't respond often, but yeah, I will try to do that more. I want to thank Baptiste and Glenn Barber, who first of all reviewed all my work on pick a G-base and reviewed my slide also, so big thanks to them. And now if you have any questions, I will be happy to answer. Yeah, thank you for the talk. I was interested in, when you split packages like NFS, which have kernel support, does the kernel parts go into the kernel or the package with the feature? No, the kernel part will not be split as long as the generic image defines NFS support in the kernel, it will stay like that. But all the user-land utilities will be in their own package. Okay, thank you. And same thing for every other subsystem. What about things that are kernel modules? So when I started I said, okay, I just want to have one package for kernel, but the more I do work and the more things that we should have some kernel split in. Like, there is a lot of modules that I don't care about. And if I find an elegant and easy way to generate some kernel package that only contains the kernel and splits the module into different sub-packages, maybe that would be something that I would like, but it should be done... The easy one to start there might be just all the firmware stuff? Yeah, firmware file, for example, should be in a separate package, yeah. So I should just commit quickly while nobody is really working at the package base because I'm sure that if I do something like that, it will bike shed for months and months and years and years. So my question was more about... You talked about pudrare image support. Does it make sense to build it into pudrare image if it's just for building the packages or do we need like a pudrare base, kind of like bulk, but for building just the base repo? So pudrare image support, I think the best way would be that you will not compile your gel and it will just insert the official PKG so you don't have to compile anything. Right, but what if I want to compile just the repo for a base system that's packaged? Well, you could use PKG base, so I don't touch that either. You will bootstrap your stuff, give the USB key to someone and you could automatically update it. I don't know. But please open some merge requests on pudrare and we'll talk about that. If I may have a question. So one of the things which I really liked about FreeBSD and I actually use it daily is that if you download the tar balls from the distribution from the download to untar them all, you have the FreeBSD in that directory, like the fully functional thing. So with the PKG base, will that still be possible without any... And I do that sometimes on other operating systems, like on Linux for example, right? So will I have to have a binary of PKG to install stuff in a directory or I still will be able to just use tar to create a folder full of FreeBSD? I think that we have some plan to remove the base.txz and kernel.txz tar creation. It will not make sense. Okay, but after considering we are fully PKG base and forgot about ever having non-PKG base, will it still be... I don't see how it plays well together. Like having base.txz with the package of base at the same time. Well, some people would probably want to still use them because they have a lot of CI jobs using it, et cetera. I also plan to make a script that will convert your existing system into a PKG base so we just fix the record if you have the correct file. And so you could use, for example, you could just extract base.txz, run this script and you will have PKG base system without ever running PKG install. But you are talking something way in the future and we should have a lot of stuff to do before. Yeah, that would be cool if we have that feature. Thank you.  The plus manifest, yeah. Plus manifest and three other files, you can just extract all the packages into a C. And you will have exactly the same thing. My question was going to be about how do we convert an existing system, but your answer is... Yeah, so the idea of the script came up when I was talking to Baptiste and he said, yeah, but if people complain about the install time because this was when we had a high-toned road package, he said, yeah, we can just do exactly the same thing and do a script that will just create a fake PKG base system but that can be upgraded after. First of all, thank you very much for all your work on this. I think it's great that someone's picked it up and is charging it forward. So my next question is not meant to disenfranchise at all, but I just wanted to understand what your thoughts were about the true OS idea of having the package base out of source. And when I sort of saw that, I thought, that's an interesting idea. I mean, a lot of work had already been put in to getting package base in the source tree, but if I look at every other program you build, you don't put the instructions on how to make a package in the source tree for that program. You do that separately, like in the ports tree. So it kind of aligned well from that point of view of my mind. What are your thoughts on that and I guess now it's been dropped? Yeah, yeah, apparently. So, yeah, AIAC systems started a side project, a side PKG base project that used the ports tree to build the ZER package system. Split was not done the same way and I'm not going to talk about that, but they have maybe 12, 20 package. I think it's not enough. My main problem is you cannot use the port system efficiently to cross build. So, meaning that I cannot use right now the AIAC system method to cross build for MV7 or M64. That's a big showstopper for me. I have a lot of other minor concerns, but that's one of the primary ones. And I had one other question. I guess concerned with the large number of packages, which I don't agree with you. I don't care too much either, except when I download them each time I knew it starts to download the next package, there's a slight pause before the download then continues. And I started wondering how does FreeBSD update the port snap do that so quickly? And then I found that they have this thing called pipeline HTTP or PHP HTTP get, as it's called, and you can look up the man page for it. Do you know if there's any work to try and get package to use pipeline HTTP get? There is no work, but I've bugged BAPT about adding multiple download, multiple concurrency download into package and he's thinking of doing it. Each time I see him I try to convince even a little more that he should really work on that. Same thing for the installation. There is a lot of package that you could install in parallel because they do not conflict. And sometimes it makes sense to extract a lot of torque in the same place. Like if you have an NVMe drive, you can extract maybe 4, 5, 10 TGZ at the same time and it will be okay. So it's planned, I would say. But nothing is really happening. For the download side, the gentleman that implemented that I think was Colin Percival, who's next door doing another talk right now. So maybe getting BAPT and Colin to talk might be a good idea. Yeah, exactly. So what about versioning schemes and package releases? If you follow... What about upgrading and versioning scheme and patch releases? So upgrading patch releases will be... All the packages will be served on the same repository because it's still, for example, 13.0. And if you want that one-point upgrade to 13.1, when 13.1 is out, this is something that the fake FreeBSD updates script shell that I will make will handle. So it will just replace your pkg.conf, pkgbase.conf. It will replace the repository. And just update your kernel first. You fetch all the packages, reboot on the new kernel with the old base, install the new base, and there you're good to go. Okay, I was thinking more of the patch releases when it goes to P1, P2, when there's a security update, and maybe just one package is actually influenced by that. And what about what happens then? Is there anything you can talk about? Just unload the new package. Just that one package? Yeah, yeah. So if there's a bug in ZFS tools, for example, you will just get the new TPS. ZFS tools and that will have P1, and the rest won't. Yeah, exactly. So how does it comply to FreeBSD version? Will that be upgraded still then? Sorry about that? This is the FreeBSD versioning command. Okay, you have 12-1, P1. Yeah, I don't know if it's currently working with FreeBSD-Version, but I will make sure of it if it's not. And I don't see why it will not work. But I have to look. I have to look. Because really, FreeBSD-Version... The goal is to still have the FreeBSD-Version command. Yeah, but every package would actually then influence that one binary, right? Yeah, so maybe we will need to put everything related to FreeBSD-Version in its own package, and this package will be bumped at each dash P something. I don't know. That's actually a good question that I don't really thought about it. Because you have like 200 packages, you would actually have 200 versions. Yeah, yeah, yeah, yeah. Like in Linux. Yeah, I will look at that. Yeah, thank you. Thank you, Emmanuel. Thank you.