My name is Ian Kruppens. I'm a collaborator at 10.10. I studied, well, I'm working here now almost two years, so no in the game, let's say. I studied at the University in Ghent, where I also did my thesis a bit on the subject of circular economy and the reproductive city, starting from looking at production in the city, specifically for Brussels. And I also did a few projects of which M4H, in Rotterdam, dealing with flows, metabolism and circular economy. And circular city ports. Not to forget, still going on. Yes, and then I'm in Aria Casabela, a principal at 10.10. I'm an assistant professor at the ULB Faculty of Architecture at Cameroon. I started using, I heard it for the first time in a lecture. It was a lecture given at the university by, I think, Eric from Metabolic. He was then doing research on Rotterdam in the framework of the IABIR, Productive Cities. And that's how I kind of came into the subject and also later on wanted to do my thesis about this subject. We also had to do a kind of critical paper of this topic series. And what struck me when I dive into the lecture is that this concept has been there for a long time, like Metabolic Rifts, this kind of concept. But on the other hand it was very eye-opening. And for me it was a kind of, it showed a bit, it pinpointed a bit more, looked more at kind of making the subject of urbanism more relevant again where it started, I think. On the other hand, the study was place-based but not with actors. And I think hearing the process of the long year of thesis for me was very enriching in that fact to look at the kind of who is going to do it in the end. And I think that's also something that Tenten emphasizes in its practice to look at these actors who are busy with it in their daily life but also who are busy with it to plan it, to institutions that could carry out policies to look at what their role could be. So to design a bit the agency, the capability of acting upon these flows. So the first time I heard of Metabolism, I think it was in the Netherlands. I've been working there for a long time in different periods along my life, across my life. And I think it was through high-endness, the work of high-endness and the Xymons in particular, he was mentioning about this aura metabolism and this brought me to this Marxist theory of the metabolic rift, of course. And then it stayed a bit there, kind of in the background without gaining any kind of importance or being a guiding topic in the way I was working. Until we started at the Faculty of Architecture of La Cambre Urta, we started organizing different masterclasses and also refining the objectives of the new research centre of Louis. We were discussing what this research centre should be about and then in the beginning it was about infrastructures. Logically, so, because we had been working around not only railway infrastructures and I had also passed in fundamental research around transit oriented development but then at that moment I started to feel that maybe the introduction of the ecology in the research centre, the ecological part, was very important. That we were busy with the kind of realisation and programming of all these infrastructures but we did not pay attention to the environmental impact. Not from a technical point of view but from a holistic understanding of the environment where you are acting upon. And I thought, yeah, maybe it's an opportunity to start talking about these issues and then these brought us to certain people, not only you later on who was doing doctoral research, doctoral dissertation around urban metabolism but of course at the same time as we introduced this issue in the research centre of course Dubigno and all the kind of theory that had been built around urban ecology in Brussels became very important kind of a reference in everything we were thinking at that point. And then of course 2014 the Rotterdam Biennial was a very important leverage point I mean the work of Faric that was exhibited there displayed, demonstrated with a lot of consistency that it was possible to use that to kind of define a new way of doing urban design that it could again make your role or become the main character in your practice beyond being kind of a knowledge data that would inform the things you were doing that could become as I said the real protagonist of your work the centre point, not just informing but kind of manipulating trying to intervene upon the metabolic flows. So that was 2014 the Biennial was a very important shifting point for us. I mean of course we had already started working on different projects trying to introduce this understanding but rather as I said it was rather understanding about ecological dynamics so it was I mean from 2011 the master plan of a parking in Albert of course the question of water is very present there but also a question of wood which kind of species, how do they grow the question of actors because there were a lot of conflicts about how much space can I use and what for and all these you start seeing that there is a kind of a conflict field where water is always pushed on the background because the football field needs a lot of space and that the football field is not really busy with the kind of pollution they are kind of bringing or causing in the water courses of the park you start seeing that all these factors are very much interacting and that they result in a specific environment and that you as a designer can intervene upon all these things and try to create bridges but also in a way raise awareness about the situated or the embeddedness of the things you do and the impact they have on the organisation of the environment but again so I think 2014 I think it was really very important for us as again also confirmation that it made sense I think there are a few which we had starting from the economic point of view where well there are three projects of which one M4H also in the framework of IABIR starting from this kind of question about support economics as a kind of gateway being urbanised but yet searching for new economies and then the question was kind of very broad kind of pilot project what is this economy about and who are these people that are busy with it because the area which had been kind of not neglected but being kind of transforming and being kind of left behind by big economies moving to the main port leaving space behind space to test space for companies to start doing things and I think the first thing that we start was exploring talking to the people and this was kind of for everybody and also for the commissioner a kind of surprise on how and what they are busy with the companies and from there we detected three flows or like building materials agri-food and textile that for us kind of were an argument to look at to make coalitions to start thinking about certain collaborations because on the one hand on the bigger scale because these flows were passing by you have to be close to the city to gather and collect agricultural monostreams or agri-food monostreams for the textile we saw that actually the port was a kind of major knot in this more European but even global chain where there is an argument to start working also because the actors were there so I think this is the most clear starting from the point of view of economies how we could foster new economies that we started to document these flows in how they are being used already or how they could make new projects by and I think the main outcome on the one hand indeed is kind of no imaginaries of people that are busy with it actually it was a major kind of step forward to a project but also getting these people, different people, the companies but also authorities like sectoral organisations or more socio-economic associations around the table that they get to know each other so maybe there wasn't a clear kind of project in the end but there was a conversation and a kind of exchange that went different directions not only the public authority followed it up but also the private companies they started working with it but I think what is important is that we don't have a single way of working or using or there is no single way in which our metabolism forms or directs our work so I think what Ian is explaining is that for instance the project of Matterfield Heavens was probably together with the circular city ports the two projects that are most directly related to I mean we talk nowadays rather about circular economy because we look at urban metallis rather as a way to analyse to understand the flows and then we talk about circular economy in the sense of how we can intervene upon how we can guide them in order to rethink our industrial system so this is very much the case and then the question of this commission that was put on our plate was to circularise the Matterfield Heavens port area and then what we answered to this commissioner question was maybe it's not about circularise but it's asking the actors first of all what do they need in order to know whether they are circular already and in case they are not what they would need so circularisation is not something is not a normative framework you can post from the top and say and now we are all going to become circular but for the port and city authorities it was obviously an agenda point and they had to answer it in any way and then they organised this competition and we are selected because we said yeah okay but let's see what is there and let's understand better what the actors need in order to become more circular so this was a very interesting way of working because then we made interviews with companies and a lot of interviews with a lot of companies and then we were looking at the growth story so yeah how big they were and how fast their growth had happened the growth had happened but also why and then also the circular story and in the growth story one of the things we asked is what do you dream of I mean because you have to be aware companies they have their profit oriented we are talking about companies based in the port they are not there to make us happy they are there to become rich not to generate wealth that can be resistivated of course I mean it's not that they only they also look to have the ambition to generate jobs and to resistivate this wealth through the job creation but also the construction and the investment on new infrastructures but this was a very interesting first point because we understood better or we thought we understood better the kind of flows that were across the area so in the circular story of every company that was based on the port but also what they saw in circularity that was important for them and in which way they could incorporate it into their future plans and then at the same time it was very interesting because Metabolic was doing their Metabolic study for Rotterdam and they had been mapping all the flows of all the inputs and outputs of the city and by looking into their research of course we did not make these analyses at the scale of Rotterdam we could identify we can kind of compare some of the flows they identified and the analysis they made and we could compare it to what was happening locally and identify which one of these flows were relevant for this area to say okay then we came down to the question of after a lot of looking for the right flow of course we thought okay so building materials notably because there are a lot of companies busy with recycling but also re-manufacturing of building components there textile because it was a very important point of collection of textile used textile but also because we saw that there were in the surrounding neighborhoods a lot of ateliers not only women were working so it was the only possibility for immigrant women to find a job in a way were Naya ateliers so Siewin ateliers to repair, to sort out to recompose these textiles so we thought it's a very interesting mix between the big scale of these big companies linked to this global supply chain to a local economic dynamics workshops the same with the building materials by the way because there were in the surrounding neighborhoods there were also a lot of small workshops that were linked to the bigger companies based in the harbor where people were being formed in a different way of for instance dismantling or demolishing buildings or reusing booths they would kind of mine from the existing building patrimony and these people were obviously building a network to it with the companies based in the harbor so it was building companies textile and of course organic waste because also there was a very important a couple of very important companies Grun Collect and Sugu busy with these flow in different ways Sugu in a much more experimental way trying to develop new packaging forms textiles and new products based on organic waste and Grun Collect which is trying to kind of diversify the way of collecting organic waste in order to be able to separate already some flows that can be recuperated more easily so to keep the priority of the organic waste flow so this is what we did and it was I think very interesting also to look at the flow chart trying to make the flow chart based on the work that Metabolic had done and to identify already the companies that were busy in the area on the immediate surroundings in order to see what can we do with them in which way they could benefit from this organization and then at the end of course our question in terms of how we can answer all that as urban designers and architects is to say you need to build specific infrastructures for these things to happen you can discover all these very nice initiatives but they need an infrastructure in order to survive but it always is a question it doesn't matter if it's about mobility or organic waste it's always all these people they are very inventive and innovative but after a couple of years if you don't as public authority you don't help them to blossom they will die away so we cannot propose as it is three different infrastructures you could say pilot projects in order for these people to meet their expertise and to keep on growing so what we found through the interviews of what is your dream how do you see yourself growing in the area so that was an extremely interesting parkour for us so research why is that also meeting the people then we organize a series of workshops at the end but also in terms of identifying which kind of jobs are that I mean we think of circularity but we don't think about all these expertise that are needed in order for the circularity to exist and to be realized and this was very important in our work actually but then yeah of course here for instance the work of Innovative Estates Havens these I printed some things for instance for us in the beginning when we started the project all the other type of reflection so the fact that water contains a lot of nutrients but also sediments and that working on the A lunch area in Antwerp was a very kind of a strategic spot to think about all these flows but of course this was not taken back by the client the commissioner so it's also maybe we start from a specific point of view that might be metabolic in a way but if the commissioner does not show any interest we are also I mean we have a responsibility with them we have to provide an answer to the questions at the end so we are forced to always reorient our project so we were very interested in this kind of path but yeah this path was really not interested at all so we had even though there were a couple of companies also very closely working with companies and making interviews with them and also understanding the way they were operating and even though there were a couple of companies very much interested for instance in this sediment thing so the way they are treating the sediments and keeping the depth of the harbor today is totally responsible but also in terms of the pollutants and there was a company which had been developing alternative to that so there was a kind of a real possibility but yeah Hackebespa is not interested in that and they are really much more focusing on building materials and mobility you have to also kind of react to the way I mean to the questions at the end you are paying a service and providing a service but yeah for instance in these projects for the Kuslin in Austin and but also for instance for the strategic plan of Ville Borde which is online so I didn't print it we always started from issues like circle economy or water management or in this case coastal defense that already in the way we read the place play a very important role so for instance here the way we were reading the place and this is a scheme that for us was also very important to discover all the different families that exist in a sustainable approach to urban design so that you can be urban political ecology but you can be also an optimist technology optimist and all these things coexist at the same time but so what I wanted to show yeah for instance one of the we kind of identified for different cities for different cities based on for instance the flow of tourism we were talking about the city that accordion city that is designed in order to accommodate a big amount of tourists at certain moments throughout the year but at the same time has to live with this heritage of big infrastructure oversized infrastructures throughout the year even when all these people are not there so these kind of notions of thinking for instance all the mobility infrastructure is huge and it occupies an enormous amount of space and then thinking in a way is a metabolic thinking thinking about the flows but also in terms of the water both the fresh water in terms of the consumption demands that have peaks during the summer exactly at the moment when there is a period or moment of drought how do you live with that but also the cost of defence to think ok so this cost of defence is very much thought in the work with it in Austin what I was explaining is that this continuous flow of sun that is being brought artificially into the bridge so this replenishment of the beach with the sun coming from the offshore mine areas and then we thought yeah maybe it would make sense to start thinking about regeneration the species so that it come resilient against the the force of the sea the waves the storms so we thought we started exploring the possibilities and then we discovered that this has been done by Delta S for instance in the Netherlands to bring in nutrients to let parts of the bridge to keep the nutrients from the insects and some the composition of some of the plants so that you could create a kind of a green beach that would be more resilient against the force of the storms so that it would have the strength not to be washed out every time there is a storm and this would to a certain way to a certain extent keep the sun that is being continuously brought in and out because of the force of the sea so this is what the kind of things we were doing but also looking into the port area we discovered some sections of how the docks were built and then we saw that there are the rainwater used to be collected on the sides of the dock in order to be reused for washing the fish in the past and then we kind of wanted to reuse and restore these water conducts that are now in the harbor in order to collect the rainwater and reuse it in new industrial processes, doesn't matter so this thinking is there also for mobility water again for a lot of things and a lot of but it's not equally kind of visible or it does not inform or work to the same extent it depends very much because we are and since we are service providers the point is very much whether the commissioner takes it back or embraces it and says ok Coming back to what I started explaining that indeed this lecture was by Fabric of course Erik Fletters and what stroke me then and the thing I started developing from there or looking for is that at first it was more about the kind of looking what is there like quantitatively most of all where it's going spatial bound but also in volumes and it was eye opening but it stay also very under kind of abstract and at that time the discussion was also from the critical point of view yeah but what's next how are we gonna do this how are we gonna be doing this because then everybody claims can claim to use this flows and these kind of opportunities maybe to do their own projects so who then kind of gets a position to act upon this flow is very interesting for me to look at I think to make a kind of overview or analysis of different offices I'm not really sure I could make straight away I think in a way that's also what Nadia explained is kind of yeah you can read your commission you can read the city in that way with the lenses of urban metabolism which is helpful but at the end it's about making about the commission who is making the commission and who takes, who has the ability to take it further and from that point of view I think a lot has changed also after this M4H commission looked at circular city ports we researched it the past year and still going on with ABB architecture workroom and there you see actually that throughout all delta because we did a benchmarking of 11 city ports with the knowledge we had from Ostende from Innovatives Antwerp from Rotterdam we also looked at the French ports and even to Hamburg so the Lahavre Hamburg range that actually it's quite representative I think for what's going on is that every port was busy with this taking on certain ambitions to look at it flows from the point of view of circular economy to start acting upon it with some new initiatives but that there was also kind of this knowledge was very partial that they were all doing the same but on their own well at the same kind of ambitions or not to say that everybody has to do it the same way or to do it as the other ports but it was interesting to see that that they didn't really know what was going on in the other ports that they were also very interested to learn from each other and what we emphasized there was that like Nadia also says there is not one way to do things it comes very much on the situation of the context the size of the port or the actors that are there or the socio-economic dynamics the constraints yes you can be circular in many ways for a port to define a circular agenda they can be based on profit, volume recycling re-manufacturing there is a wide very long list of priorities you can define in your circular agenda and this is very much what we discovered actually but yeah that they were nevertheless despite the fact that all these initiatives are very kind of rooted in specific places and contexts and they depend also to a certain extent on the inertia that has been created in the past that they were nevertheless very interested to know what others are doing obviously I mean it's like what we are doing but in terms of practices I think that there are many ways of kind of integrating this metabolic understanding of cities in the way you work certainly we for instance you did not take part on that research but when we were working with Fabric on the Atelier Brussels then you see that of course they have an incredible incredible knowledge of how to gather the quantitative data and how to represent that but at the end what we found it was missing is a bit this kind of attention to the local capacities of the territory so that you cannot come always with one size fits all approach that they are not replicable solutions that you can adopt everywhere like it's a checklist and now we do that and that and that and this is what we found that maybe this is what we could contribute with and already I think the discussions about the methodology and the fact that it was not just because there was a lack of time a lack of budget to do a very all-encompassing metabolic analysis of Brussels but to decide to work on specific areas that was I think already something weird to them a different approach of looking at metabolism more from the local scale what was going to be found but also in terms and this is probably what was missing at that moment in terms of the actors you could engage and involve your because at the end it's not just about descriptive or analytical tool it's about a transformation tool so you need to count on people, specific people capabilities in order to go forward I mean for us is very much that we are not of course you can say you are very much paper architects or because we draw a lot of things and very little of what we draw is actually implemented but our ambition is to go for change I mean we don't want to keep on drawing things but nonetheless this the quantities and the knowledge the data is very important to make a project if you for example with master plans as well if you know the figures of how big the building is and how many water it can collect and how big then your landscape has to absorb or be a sponge then you have the do you have a kind of tool to design you have an argument to make a project so it's important this data is the basis but it shouldn't end there it should make a project and this project is for the future indeed for change specific people and places not all places and people can reach the same level of engagement transformation they don't have the same means so this is also you have to be careful with that so that you don't also for us that you don't create situations that are impossible for the people to embrace and to engage with but also for instance any of them in the southern part of Brussels that was very much the case we came with a section based on the productive landscape agricultural landscape and then after many interviews and many workshops you start realizing that I mean all these actors and especially our farmers who have been signing contracts with big companies they are completely captured trapped in a situation they cannot easily escape so it's not just about coming up with a nice future this is where you should go towards but it's also about providing an action plan of how can you get rid of the system you are trapped into so I think for us this is very important maybe the only quantitative part we do not spend much time honestly analyzing that we use some numbers also for mobility now some numbers related to flows in order to have a better idea of what we can do in order to change the situation so what is the extent of the problem but our approach is very much based around actors and institutions yes what can we do change takes action plan it takes years to to realize to realize to start somewhere where you want to be or re-evaluate so it needs embeddedness kind of actors that stay that keep on making them capable of doing it but on the other hand connecting this to a more overall institutions or programs or policies that also provide this local developers or area managers or farmers for that sake with the tools and the vision also the framework in which they can do it I think what is very important is this reality check so this attachment to reality to see what is there already who is operating there and what are their capabilities also for public authorities in all levels they have very much the tendency to close or propose in the best case scenario different way of organizing territories or flows but there is very little knowledge about the kind of initiatives or the kind of actors that are already busy with that and I think the first I think very much is this kind of local situated knowledge I think it's a responsibility for everyone who is trying to intervene on the organization of a space the organization of space and flows and actors you need to know that before doing anything so I think this and then of course the second step would be to identify actors or institutions or organizations that are already more or less consolidated in these territories that can act as area managers that can further this ambition to work with flows but also change the industrial system into a more circular one from linear to a more circular one so to identify these kind of key actors that become the magnets of transformation but of course I mean this is very much local bottom up local scale what can you do when you are an oban yes I think keep on doing a lot of things because on one hand you have this task of agenda setting and raising awareness and convincing your politicians of a different way of understanding what is at stake bringing in a benchmark benchmarking analysis of what happens elsewhere so I think this is this is very important so there is a kind of educating your politicians especially in Belgium in the Netherlands it's very different because the politicians do not have such a the last voice into the matter is actually the technical body the specialist who resides at the end and then you see also the differences of course with a lot of problems because there they have a very infrastructural and you can also have second thoughts about that but ok at least but ok in the case of Belgium you have to educate your politicians you have to indeed I think agree in a white paper in order to frame the questions that are at stake and the impact these questions or these challenges the impact they can have eventually on how you think about competitions small projects so a different scale from the public procurement so that you kind of bring in these questions from very early on and these very specific methodologies but also until the small execution project of public space that you have this kind of basic knowledge to know ok it's important to think about that so for instance when we make a very even if it's this stupid square here that we think about water I mean water management rain water management there is nothing we all know for years that we cannot keep on using the sewage for the rain water doesn't make any sense it's a waste so yeah that all these things are kind of that you create a kind of a body of knowledge that is easily accessible for very different levels but they know this is the way to go so but then you were also asking about the link to the academia and I think this is extremely important I mean to form through university because you give access to the students to a different way of understanding cities and a different way of reading and measuring all these flows I think it's crucial I mean if you don't form the next generation into a different way of understanding what is at stake I mean nothing is going to move they are they are the next I mean it's fundamental and then for me is still again amazing that within the faculty that you are being criticized continuously that you are found to be relevant too specific that you are kind of put aside of the architectural faculty because you are too specific for art days we should be doing talking about buildings and composition and that's it, that's it so this is I think problematic I think we cannot go on like that and then yeah I think yeah Ova can also have a big impact in the way new doctoral dissertations are being proposed the support they get they can have integration of these more fundamental research into applied projects I think this is fundamental that Ova supports this line of research at university and that the people who are busy with that yeah they do not get this courage but rather the opportunity of going to the city of Leuven and explain what you are doing and look for ways of applying it but also together with Ova do not forget this kind of multilevel so you need to continuously work at different levels but there is no single answer but yeah at least you have the technical knowledge that is terribly missing the politicians who are completely ignorant of what is at stake and then the future and the next generations you need to involve them


It's coming. I better catch this But you know all that spin Trips him up he gets embarrassed. He can't handle it. He throws reddit out of the game second technical boom. You're out Reddit is like what for throwing the ball to you That's not intentional at all either he said That is not intentional or he says that's not what I'm trying to do I don't know which one it is. He says I'm just spinning the ball to him I believe he did say spin in which is interesting now the official ejection at the NBA Said that he got ejected for passing the ball to the ref with force And here's the ref after the fact just reliving that nightmarish embarrassment can't catch a simple pass That's why I'm refereeing and not playing in this league.


I'm Shiva Merla. I'm from the cloud-native engineering team at NVIDIA. I've been focusing on GPU operator for the last few years and mainly working on GPU orchestration in Kubernetes. So we had a long journey in terms of using operator pattern to make it easy to consume GPUs in Kubernetes. So today, we're going to talk about how we went through this approach and the learnings we had through this journey. Introduce. My name's Kevin Cluz. I'm on the same team as Shiva at NVIDIA. And the way I always pitch the team that we have, what we do is we do everything that's necessary to enable GPU support in containers in Kubernetes. And then we build all the tooling around that to make using GPUs in this environment easier. And so yeah, the operator is our way to package all these things together and make it that much easier for you guys to take advantage of all of these things on Kubernetes. Thanks, Kevin. So this is the outline for the talk today. We're going to talk about why GPUs have become so ubiquitous and why GPUs in Kubernetes. And also, we'll walk through how the typical GPU software stack looks like. What are the main operational pain points we have when enabling GPUs, GPU software stack. And then we'll delve into operator pattern, how we have implemented GPU operator and some of the technical details of the GPU operator itself. Then we'll end with a demo and some of the lessons that we learned through this journey. So why GPUs are so popular, right? It's no surprise given the massive computational power and given the way we took a giant leap in terms of computational capacity over the last decade, they become so ubiquitous in Kubernetes to run AML jobs, deep learning, even in the scientific research. So it becomes so common everywhere. And recently, we have announced Blackwell, which took a giant leap in terms of computational capability. And Kubernetes, on the other hand, also over the last few years, have become a de facto standard to run AML jobs, be it in AML, deep learning, in the scientific fields, or data processing. So everywhere, Kubernetes have become a de facto standard because of its scalability, easy to scale your applications because of its resiliency, and also the way to manage these applications seamlessly. So it's become so common to use Kubernetes. So what do we need to enable GPUs in Kubernetes? So typically, we start with a device driver. Any kind of GPU, we have a device driver to be installed on the host. And we have some sort of hooks to enable with the container runtime. Because sometimes, we don't have the native support in the container runtime itself to enable the GPUs. So we need these custom runtime handlers to be installed on the host. We also have a Kubernetes standard device plugin. So we have a device plugin framework in Kubernetes to advertise GPUs, discover and advertise GPUs to the kubelet. And also, it's common to have a GPU monitoring software. We want to monitor GPU performance. So we want to get certain alerts in cases when they go faulty. So it's also common to have a GPU monitoring stack. So this is a typical GPU software stack with any kind of GPU. So what are the pain points that the operational teams are seeing these days when enabling the GPU stack? So we're going to talk about, at the high level, these pain points. We have a heterogeneous node software stack. So I'll talk about the challenges we're having in terms of managing this stack as we add newer GPUs into the cluster or as we release newer versions of the drivers over a period of time, how it becomes a pain point to manage. Also, the driver configuration itself. So most people install drivers on the OS itself. They have to configure certain things on the driver. So I will walk through how that is painful today. And the most common thing that we are also hearing in this conference is how to kind of efficiently use GPUs. It's most common to learn about how we can configure GPUs to share among multiple workloads. So again, the configuration is a per node and also per GPU. And also something that the operational teams have to manage over a period of time. And as I mentioned, we do apply certain container runtime configuration. We have certain hooks to be placed on the host based on the runtime type. So we have to configure. We have to reload the daemon. So there's certain things we need to know. And also monitoring GPU health. So it's also, there is no robust solution today to kind of monitor GPU health and take action. That's also a common pain point. Whenever some GPU becomes unhealthy, how do we monitor that and how do we kind of make sure that we don't schedule anything on that node? So to delve more into the heterogeneous node software stack, typically day zero, we install operating systems. We have a GPU software stack, which I talked about, and a standard Kubernetes stack. Everything works fine. Initial setup works fine. And then we keep adding more nodes with maybe like newer GPUs, newer driver versions. So these versions are constantly changing. We have maybe it might be because of the performance improvements or it is because of the CVEs. So you kind of have to keep updating the drivers over a period of time. So definitely we'll run into some sort of interoperative issues. So we'll run into issues in terms of the container runtime hooks that we have or the driver and the CUDA stack that you have on the node itself. So we'll run into a different sort of interop issues. Also there is a CPU versus GPU stack. So the operational teams have to maintain two different stacks depending on if it's a GPU node or a CPU node because they kind of build drivers into the OS image itself. So we end up using two different OS images. So when the cluster grows in size, so it becomes very common to see how challenging it is to manage it across different nodes and how hard it is to configure these things. When it comes to driver install, so we have various components in the driver. We have kernel mode components. We have user space components. On the user space, we have multiple services to be launched. And also there are the CLI utilities to configure parameters on the GPUs. Again, on a node by node basis, so depending on the version of the driver they have or depending on the type of GPUs they have, so the SRI teams might have configured a few things on these nodes. So again, it becomes challenging to kind of keep track of all these things in the cluster. Another common thing is a per node GPU configuration. So how we partition the GPUs in each of the nodes. So you need to have some sort of declarative mechanism to partition GPUs. This is when done with the standalone tools, so that means how to keep track of the config that is applied on each node, the different settings that are applied on each of the GPUs. So they'll have to kind of keep track of these things on each of the node. And sometimes these are not persistent. Like every time the node reboots, some of these are not persistent. So on every reboot, they'll have to have some sort of unit scripts or service demons to apply the same changes again. When it comes to runtime configuration, we do have, as I said, we have a custom runtime hooks in place. We are kind of standardizing towards CDI, but we are still not yet completely there. So we still have to install certain hooks on the host based on the runtime type. For example, container d, we have to modify the config file. We have to apply the runtime class configuration. We have to reload the demon. So same thing with Docker, a Docker demon. Same thing with CRIO. We also need to modify the default runtime itself. If users want to run mostly GPU jobs, so they can NVIDIA as a default runtime, in which case if they don't request a GPU, so we'll just fall back to the underlying runtime. And also, reloading a runtime demon is also a challenge, because whenever we apply these changes, we have to reload the demon. And with every new version, we might add some new configuration into the runtime class. So that means how to keep track of how these changes are made in the cluster. So this is something they have to keep monitoring. Another pain point is GPU health. So we have a DCGM exporter. We have a standalone Helm Chat to install DCGM exporter. Again, there is no lifecycle management, right? So the operational teams have to apply these changes. They'll have to set up service monitor, so make sure that they're configured right with Prometheus. So it's all manual process today. And also, there is no robust solution in terms of handling GPU errors. In case if there are GPU errors, how we need to propagate these to the Kubernetes itself? So how can we avoid scheduling jobs onto these nodes? So there is no robust solution that is built for this as well. We have a Kubernetes device plugin. But again, we have a very basic health checking in the device plugin just to make sure that we don't use those GPUs. But on the Kubernetes side, there is no easy way to orchestrate and say that you can't schedule any jobs on these nodes itself. So with that, let's get into Kubernetes operators. So we started looking into, initially, we had various ways to kind of deploy GPU stack. We had standalone Helm Charts. We had OS native packaging. It was everywhere. So a few years back, we started looking to kind of create like a unified solution to have a common API to configure these things in Kubernetes. So operators was an obvious choice. It gives a common controller pattern where you kind of declaratively define the config you need, and it goes and applies the desired state to make sure the actual state matches with the desired state. The control loop paradigm really worked great for us. We were able to bring all the software together and deploy this with the operator. Recently, with recent years, we have seen many cases where there were a lot of inter-op issues between these components. So it's become so easy by configuring with a single API, it becomes so easy to make sure that they stay consistent. Whenever we deploy these things, they stay consistent and don't break on upgrades. How are they built? So they're built using common tools like QBuilder. We have operator framework. Just with a few handful commands, we can easily build operators. These tools will help with generating the initial scaffolding. We can define an API, and also it automatically generate all the manifests that is required to deploy in Kubernetes. For OpenShift, there is again like a life cycle manager, operator life cycle manager. Using operator framework, we can also generate all the manifests that are required to deploy them in OpenShift as well. So why are they useful? So today we're gonna talk more into GP operator, how we have used this pattern and solve these issues. So to give an overview of GP operator, so we wanted to have a unified API to configure everything in Kubernetes. It gives a single pane of glass to kind of configure and manage the life cycle of all the components that we talked about. Starting with NVIDIA driver, container runtime, a device plugin, and the monitoring software. We also have some advanced components like mic manager deployed through GP operator. So it's given as a single API to kind of easily configure these things. Again, how we install this in Kubernetes? We have a standard tooling package manager called as Helm. With just with a single click install, we can deploy GP operator. And if you're using OLM, then we can deploy using operator SDK. We can easily spin up GP operator parts and we can create a custom resource which will enable deploying all the operands that we have. At the high level, this is the state machine. The soon after install, GP operator part comes up, right? And we also have a CRD called as a cluster policy. And also we have CRD called as NVIDIA driver API. So using these APIs, users can define what is the configuration that they need for the GPU stack. The operator comes up, so it kind of fetches the API config from cluster policy and NVIDIA driver. But nothing is deployed yet. So we have a dependency on a bootstrap operator called as NFD. So NFD is a node feature discovery operator which is open source. And mainly the functionality of NFD is to discover the hardware features on each of the node and kind of enable other applications to detect them and run suitable applications on them. So NFD enabled us to kind of seamlessly identify GPU nodes. So it has a CRD called as a node features. And based on the node features CRD, NFD parts will kind of label saying this node has a GPU. And this is the architecture of the node. This is the kernel version of the node. There are like multiple properties of the server itself that are labeled on the node that the applications can use to schedule onto those nodes. So once the NFD labeling is done, so we can start with the bootstrap. GP operator comes up and we deploy the containerized driver to start with. The driver installation takes around like three to five minutes, so we have, we currently support installing drivers through a run file which will dynamically compile and install. And also we have a pre-compiled packages supporting the driver containers as well. So depending on the type of installation we use, takes about like three to five minutes to bootstrap the driver on the node. And once the driver is installed, once the driver is loaded and all the libraries are installed, we bring up container toolkit, which is again our core service to kind of inject all the GPUs into application parts. So until the core services have come up, we can't really run anything else. So all the other parts will be waiting on these things to come up. It'll, so if you see, as soon as you install GP operator, if you see most of the parts stuck in its state, it is due to the reason that they're waiting for the driver installed to complete. So these things really order well among themselves. So once the driver installation is done, once the container toolkit setup is complete, that's when we bring up a rest of the stack that includes the device plugin, the GPU feature discovery, Mac manager, right, and also the rest of the monitoring stack. So all of the rest of the services will come up. And also we have validation built in into the operator itself. At every stage we perform like CUDA validation and the plugin validation to make sure that the stack is completely functional. And finally we mark the node as ready. So a bit more on NFD itself. So we have this dependency on node labeling because we didn't want to build another controller to kind of manage node labels. So it has some APIs defined like node features and node feature rules, where users can come in and define custom rules and say, if server have these properties, this is the label I want on the node. And also it has certain standard labels, for example, certain standard PCI labels. In this case it says, okay, it's NVIDIA GPU and it labels the node with NVIDIA vendor ID. So to talk more on the driver management itself. So the first thing we did is containerizing the driver. So we have had this from a long time, we were using Docker containers to just do like a testing, mostly in the testing environments. Later with the GP operator, we have built the whole lifecycle components into the driver itself. So we install everything through the container, we bind onto the host path, right? So the rest of the components can use them. And also we built the whole lifecycle aspect of the driver itself. So we, recently we built an upgrade controller, right? And we were also making progress in terms of having a common upgrade controller between multiple operators. We do have network operator which does a MoFed installs, MoFed driver installs, right? And we do GPU driver installs. We have like a synchronization between them to manage these things. And typically, so we install everything into a container, the container comes up and we can execute into the container and do all the commands that typically we do on the host with the drivers installed. So please refer to the talk done by my colleague yesterday about the, if you want to learn more about the driver installation and how the upgrade and all those things work. So just to give perspective of how we've been adding different features into the driver container itself over the past two to three years, we added significant features into these driver containers, making sure that we cover all the functionality that is supported by NVIDIA drivers. So initially in 2021, we had basic driver installs just compile and load the modules, bind monitor on the host, nothing else. But we added the lifecycle components. We added a NVIDIA VGPU driver support. And later in 2022, we added GPU direct RDMA support loading NVIDIA PRMAM, GDS drivers, GDR copy. So we added complete support for GPU direct technologies as well. So other notable ones are upgrade controller, advanced upgrade controller. You can listen to the talk again on this one. And recently the focus have been how we make it kind of easy to bootstrap the node. So we've been looking into pre-compiled drivers, how to use pre-compiled drivers across all the operating systems that we use. So Canonical builds pre-compiled packages for NVIDIA and publishes them. So at least for Ubuntu, we started publishing pre-compiled driver containers with NGC, our NVIDIA container registry. With certain kernels and driver versions, on a daily basis, we kind of build and publish these container images to NGC. So again, we are expanding pre-compiled drivers to other operating systems. We are looking at cloud-native, CSP-native operating systems. We are looking at REL or KOROS to have these pre-compiled packages. Again, like heterogeneous drivers, right? This is one common use case. I want to run different drivers in the same cluster. I want to run different kernels in the same cluster or operating system versions in the same cluster. So, or different drivers types in the same cluster. So we supported this as well. So the currently, as I mentioned, the focus have been having pre-compiled drivers everywhere, remove the dynamic dependencies that we have to install these drivers, right? We don't have to pull any packages or build them. So the focus have been mainly to kind of enable pre-compiled drivers everywhere. And we are also looking into CSP-native operating systems. With this, I'll hand over to Kevin to talk about how the GPU configuration is done on each node. Yeah, so in addition to all the great features that the GPU operator gives you in terms of driver management and so on, one of the big things that you're able to do with the GPU operator is actually configure the ways that you want the individual GPUs to be set up by the time your workloads go to run on them. And in particular, things you can do is set up the different sharing strategies that I talked about in the keynote on Wednesday. So being able to set up a set of big partitions on the GPU, being able to set up an MPF server to run and space partition your GPU in various ways, set up time slicing on these GPUs. And one thing to note that at least with the current GPU operator and the way that it currently works with the APIs that are available to it from Kubernetes, only this admin has the ability to do this. So you have to kind of a priori decide how do I want to divide my GPU up into different big devices? How do I want sharing set up by a time slicing an MPS? You have to drain the nodes. You have to apply these configuration settings. And once those are in place, a user workload can come along and consume them based on whatever the admin has decided is the right way to set this up on an individual node. In the future, once we have dynamic resource allocation support, it won't be admin driven anymore.  You can decide what configuration parameters you want on the GPU that you're going to be given access to. And so it kind of moves this ability to define what sharing settings you want on your GPU from the admin a priori to the just-in-time usage of the GPU when you get access to it on your workload. But at least in terms of how you use this today, the diagrams here on the right show how an admin can come along and configure a set of GPUs on a node with a set of big devices. So the one on the top shows how you can divide all of the GPUs into what's called a 1G, 5GB device. And you can get seven of these on a single GPU. And you can advertise them as nvidia.com slash GPU. And as you request that, you'll get one of these big devices rather than a full GPU. And this is what we call the single mode, because the GPU itself, or sort of the resource used to advertise this is the same as what you would have from a full GPU. So from an end user's perspective, he doesn't necessarily know or care that he's getting a big device or a GPU. You might want to advertise it this way, so that he doesn't have to change his pod spec when he's requesting these. On the flip side, you can set it up in something we call mixed mode, which allows you to change the name of the resource that you're actually trying to get access to, because you know that you exactly want one of these 1G, 5GB devices versus a 2G, 10GB device and so on. So there's different modes of operation that you, as the admin, can decide for how you want to share these GPUs and how you want to set these things up. And the APIs will be very similar once we get with DRA, but as I said, you'll be able to do that as an end user as you request access to these GPUs rather than it having to be set up a priori and you just grabbing a reference to what's already there. Next slide. And to enable all of this, as Shiva mentioned, we have this component in the system called the NVIDIA Container Toolkit. And just like the driver that you need to install on your host that runs in a containerized environment where you take the driver container, you run on the host, it's going to install a kernel module to represent the NVIDIA driver and as well install some, the user space libraries inside the container itself and you mount that back onto the host. Similar thing happens with our toolkit component. It's a containerized installer of the N-Toolkit. So you run this container and what it's going to do, when you see the three cylinders on the right of the box, it's going to go through and back on the host, install this binary called the NVIDIA Container Toolkit, which your container runtimes need to call out to in order to actually inject GPU support into a container. And that gets it installed back on the host. It's going to expose some of these socket files you see here and depending on the runtime that you actually have configured, it's also going to update the configuration files for those runtimes so that they know how to call out to this toolkit at the appropriate time. And the GPU operator automates this entire process. So if we go from top to bottom on that list that you see there on the right, the first thing it does is it determines whether you have your driver installed directly on the host or if you've used our GPU operator managed driver installation process to use a driver container to install it. It detects which mode of operation has been used to install your driver. It then optionally updates the default runtime that you have in place. So you don't necessarily have to go via the toolkit in order to run every single container that you have on your system, whether it uses GPUs or not. If you don't want that set up as the default runtime, then you can say that you want to use this runtime for containers that want access to GPUs. But you don't have to make that the default. And in order to enable this, it adds a runtime class spec so that you can reference specifically the runtime that's going to use GPUs versus not. It also, once it updates the configuration files for these runtimes, it will hook back into the system. And if you're using system D, it will restart system D. The system D unit that represents, say, container D as an example.  Thanks, Kevin. So let's also look at the diverse workloads that we support. So typically, container workloads injecting GPUs into containers. But recently, we have also seen use cases around multi-ten and use cases, and also how to securely run certain AI ML applications. So we've been looking at the virtualization solutions like Kubebert and Cata. Kubebert, it's been a few years now. We had support for Kubebert. We support virtual GPUs. We support pass-through GPUs. We have a very good solution out there for Kubebert VMs. Cata containers, currently, we have in tech preview. You can launch a Cata container with a pass-through GPU so that they will be also documented and published as a tech preview. So GPU operator makes it so easy to kind of determine, based on the workload type you want to run on each node. It automatically detects the software stack because each of these workloads need a different software stack, different kind of runtime, different kind of plugins for these ones. So just by labeling the node, saying if you want to run a container workload, VMVGPU or VM pass-through, we automatically bring up the necessary software stack. So again, refer to the talk by my colleague, the Nest Day, on the work we are doing with Cata containers and also confidential computing with this. So GPU monitoring, I'll not go much into this one. So we do deploy DCGM exporter. We have DCGM engine built-in. Or we also support launching DCGM as a separate container. We automate the lifecycle of the DCGM component itself. We also create automatically service monitor. And dynamically, you can also change the kind of metrics you want to collect using DCGM. So we also have a metrics built-in to the operator. We have operator level as well as operand level metrics. So you can easily, the SRA teams can easily monitor if every component is functioning correctly or if the upgrades are in process. You can see that the progress of upgrades as well. So we support a broader ecosystem, be it on-prem clusters or cloud providers. We support various cloud providers. We also have support for on-prem Kubernetes variants. Container runtimes, all the container runtimes. And operating systems, we are adding more. But currently, we are supporting mostly Ubuntu and CoroS and REL, and we are planning to add support for other operating systems soon. And please find the support metrics at this link. Troubleshooting, so we have a kind of script to kind of gather all the required logs to troubleshoot any issues. So if you see any issues, use this command to get all the logs. So let's go into the demo. So we're going to show in this demo how to deploy the operator. So we're going to show how the driver gets deployed, how the device plugin gets deployed. And also we're going to show, so currently, we are adding support for MPS. We are planning to release an RC version next week. So we're going to have MPS support as well, but I'm going to show in this demo. So in this demo, we're going to show how to install. Just with a simple Helm command, you can install GP operator. And once the GP operator install is complete, we can show how the different node labels are applied to bring up the rest of the stack and how to do validation to make sure the driver is installed successfully. So we can see, as soon after the GP operator install, the NFD pods come up. They do add certain node labels saying, OK, this node has a GPU. And based on that, the GP operator will bring up rest of the pods. So we can see the driver demon set coming up, the container toolkit coming up. And we can see some of them in the init state. So this is the ordering I was talking about. And the driver initializes, once the driver initializes, so while it's taking place, we can see the different node labels that the GP operator adds to control each of these. So once the device plugin comes up, we can see the allocatable GPUs on each of the node. So currently, it's still zero. The driver installation is complete. So we can see the allocatable GPU count increased. So each of them have one GPU. So one node has a 100 GPU and two nodes have L4 GPU. So now we'll go into applying different GPU sharing. Techniques, one is applying MIG. So just by applying MIG config on the node, saying 1G, 10GB, so I want all 1G, 10GB profiles, we created seven MIG instances on this node. The MIG manager created seven MIG instances on this node. And from the driver container, we can see all the MIG partitions on that GPU. And also we apply custom configuration for time slicing and also MPS. So we can define a custom configuration for device plugin to create MPS instances and also to create time slicing replicas. So what I'm doing is on the first node, I'm applying the MIG configuration. On the second node, I'm applying the time slicing configuration. On the third node, we are doing MPS configuration. So we can see different replicas created for each of these. So we can see the first node has seven instances, MIG instances, and the second node has three replicas, and the third node has one replica. And we are configuring MPS on the third node. So on the third node, we can see the MPS daemon coming up. MPS daemon comes up. And the device plugin and then the GFD will restart. So soon, we'll see how the allocatable GPUs increase from 0 to 10 replicas that we are applying for MPS. So we can see 10 replicas on each of those nodes. As you remember, so there are like one GPU in each of the node. And now with the time slicing and MPS, we have multiple replicas for each of this node. So I'll go into running some workloads. So with each of the time slicing, we run workloads. I have MIG workloads. We launched some time slicing workloads and also and some with MPS. So time slicing, we're launching a job with three replicas. So we can see three replicas coming up. And similarly, with MPS, we launched 10 replicas. And we can see 10 replicas coming up for MPS. OK, with that, we'll move on. So some of the lessons learned throughout this journey of developing operators is containerizing drivers is not easy. I mean, it's easy to get the driver installs done. But to manage the lifecycle, it's very challenging. So we learned some of these challenges and built advanced controllers to manage these drivers. And CRD management is another big issue. As the versions change, as the API change, you kind of end up managing multiple versions of CRD. And also with Helm, we also have challenges in terms of updating CRD. So we are learning through those. And we also had some features to kind of handle this in the operator. Memory consumption is another issue. We are a cluster level operator, so the client cache gets huge. We are looking into building early using watching selected resources to make sure that we don't consume too much memory. And also, we have a dependency on node labels. That is NFD. So again, we are looking into see how to make sure that NFD kind of don't bring down any of our components. In case if it loses access to the APS server. And here are the quickly some of the upcoming features. We're adding better health monitoring and reporting with the GP operator. We are adding support for Cata containers. I mentioned this is Tech Preview. This is going to be GA. And also, we are adding conventional container support that is going to be GA this year and more on the heterogeneous and pre-compiled drivers. And DRA, obviously, is again a big topic with this conference. So DRA integration is another main feature that we are looking to integrate with the GP operator. So these are the resources. So please, if you have any feedback, create an issue or reach out to the open source project so we can reach out or create an issue. And this is the documentation link. So yeah, please provide feedback and reach out to us. Questions? I have a question about your plans for the future operating systems. Because in Germany, SUSE Linux is used by a lot of companies. And it's also the preferred operating system for SAP. Thank you. Thanks for the question. I think, yeah, this comes up a lot. We are looking into SUSE Linux. I think the main challenge we had is supporting driver containers. So we didn't have the base images, write base images to kind of build and publish these. So what we did was we do support users to build their own container images. We have a process. We have steps documented. So users can build their own container images for driver and deploy it through the GP operator. So that part is users can still run them. But officially, we are working through to see how we can publish and manage through NGC. So how to publish through NGC. So something we are looking into, yeah, we can keep you posted. It's a process, an official process to apply it. So SUSE, to apply it for this build, for this that you built, or the built images, or make them generally available. I think our product managers are usually going through the requirements. Maybe you can create an issue, GitHub issue, and we can definitely prioritize this. If you have a good use case, a lot of customers are using SUSE. So we can definitely prioritize this. OK, thank you. Hey, hi. Thanks for the demo and the session. Very useful. You probably covered it, but I just wanted to understand a little bit more in terms of GPU management and for efficiency. You mentioned MPS, and you also demoed MPS, MIG, and time slaging. But what was not very clear for me, at least, was when to use what. Is there scenarios or this practices as to what would be used when? It depends a lot on your workload. And we have a blog post that tries to help walk through why you might use one versus the other. We didn't link the blog post in these slides, but if you go to the keynote talk that I gave on Wednesday, I have a link at the bottom of one of the slides that should help you make this decision. There's a giant matrix of what advantages and disadvantages of the different approaches are. And it also, the blog post itself talks about types of applications that could benefit from the different strategies. And if you still have questions, reach out to us afterwards, and we can try and help guide you based on your specific requirements. There's no one right answer, I guess, is the short answer. OK, we'll take a look. Thank you. Hi there. So over this side. If you're using a managed cloud provider for something like AKS and you're selecting NVIDIA node pools, do people like Azure already manually install these drivers? And then how will that work if I want to use the operator with some of the advantages that has? I just wonder if you could speak to that. Yeah, it's a great question. So we do support that kind of configuration. If you have typically, by default, if you're using Azure Linux, so they do have, sorry, Amazon Linux. They do have drivers pre-installed, container toolkit pre-installed. So we do support that kind of configuration. You can still take advantage of the mid-configuration or the mid-manager, the toolkit container that we have. You can still take care of all the advantage of those components. But what we are working with AKS or EKS specifically to add support for their native operating systems itself. In this case, Amazon Linux, we are looking to add support for Amazon Linux itself. So you can use the driver container instead of using a pre-installed driver management. So that is in the works. So it's actually possible if Azure already had those drivers on, I can still put the operator on and then kind of use some of the things where you were labeling the nodes to say, set this up to be make or still do that. Yeah, we have documented the process for that one. So we automatically detect that the driver is pre-installed and we disable the driver container on those nodes. So yeah, we support that. Thank you very much. Thank you. Thank you. So yes, my first question is actually kind of the same. So if pre-built images are already available in the cloud providers, apparently, yes. So the second one, you mentioned that it's a problem for monitoring the GPU health. From your experience, what would be the main issues that would happen? Because from what I've seen, if you have issues with your GPUs, your machine would not boot up or everything would freeze? Let me answer the first question. So in general, all of the operands that the GPU operator manages, you can decide if you want them to be managed by the operator or pre-installed on the system. So if you want to use the host manager, if you've installed your driver manually on the host and you don't want the operator to manage the driver lifecycle, you can turn off it using the driver container. If you've manually installed the NVIDIA Container Toolkit on your host, you can tell the operator you don't want it to install that because you've already done that step. Same thing with any of the other components. So there's options you can set when you deploy the Helm chart as to what you want on versus off when you deploy it. And then the second question, maybe you can answer. In terms of error monitoring, that is true. We can't recover most of the time. But currently, what is lacking today is properly propagating those errors at the Kubernetes level. When it's at the Kubernetes level, there is no indication at the node level saying if some GPU is unhealthy, and what is the issue with the GPU. So what is happening today is the device plugin will detect these errors and make sure that that's not allocatable anymore. That's all. So you'll see the number of allocatable GPUs going down. But there is no indication in terms of which GPU has gone bad or what is the error with the GPU. So that's where we are improving saying we'll propagate those as the node conditions and say that, OK, this GPU has gone bad and how to kind of recover those things, recover that node. It's also worth pointing out that we're internally right now trying to come up with a more comprehensive solution for not just GPU health, but node health in general. And the plan would be to eventually integrate this kind of node slash GPU health solution as an operand that the operator can deploy and people can make use of. We're still trying to figure out what that's going to look like, but that's the long-term plan. Thank you. Thank you. Hey, you mentioned for GPU driver, you have a containerized driver. So can you explain a little bit between the native GPU driver and the containerized driver? What's the difference? Yeah, so first off, there was a talk yesterday. He referenced it, Shiva referenced it in his slides here that goes into great detail what our host operator, sorry, our GPU operator managed driver looks like and how he managed the lifecycle of it. But the main idea is that it's kind of a misnomer. We call it a containerized driver, but it's really driver installer wrapped inside of a container. And so what it does is when you run this container, it will install the driver, the kernel module, into the host. It'll install the user space libraries into the container image that you have, but then it'll make that the root of that container's image or the container that's running, it'll make that the root of that file system available back on the host so that from the host's perspective, he has a path to using that driver from anything that's running on the host directly. So it looks different from what you installed directly on the host, only really by where the path to those user space libraries exist. OK, so does it mean NVIDIA will maintain this kind of two different GPU driver version and the user will have option to choose one of them? You can't have both installed at the same time. You either do a host install driver or you use the driver container. You can't do them both at the same time. So either way, but both will be co-exist. Both will be co-exist as the project. It's the same driver. So whether you're installing it directly on the host and then you get it at the root file system where all these files are or you install it in the container. Think of the container as just the DOS where you're installing this now. It's the exact same driver that you're installing though. It's just the method at which it's being made available to software on the host. I say, OK, thank you. I don't know if you mean, by the same project, if you mean same cluster, so then it's still possible to have some nodes pre-installed driver. I think what he meant was, are we maintaining different drivers, whether it's containerized versus not? And it's the same driver at the end of the day. Yeah, same driver. OK, thank you. Thanks for your good talk. So you mentioned cata container in your talk. So I want to enhance our isolation. So I want to use it. But as far as I know, the cata container is only support, don't support, doesn't support multi-GPU or a VGPU, right? So do you have a plan to improve the cata container to support it? Definitely it's in the plan. So this is the roadmap to add support for multiple GPUs and also VGPUs. So I think the current focus for us is to take a single GPU pass through as a GA, right? And then the confidential containers with a single GPU. So then eventually, or this or next year, we plan to add VGPU and multi-GPU support. OK, thanks. Thank you. All right, I think that's all the questions we have time for. Thanks, everyone. Thanks, everyone.


Once I think you get that aha moment and you figure out what presence truly is, it can be a very simple thing to master. But, you know, you have to start, obviously, the self-awareness is usually the hardest piece, recognizing how caught up you are in the future, you know, thinking ahead and jumping to conclusions, rather than just focusing on your breath, focusing on the moment and being like observant, basically. What's up everybody and welcome to the show today. We drop great content each and every week and we want to make sure that you guys get notified and in order to do that, you're gonna have to smash that subscribe button and hit that notification bell. And if you've gotten a lot of value out of this, make sure you give us a like and share our videos with your friends. Now, the second signal that you discussed was pressured speech. And this is another one that presents itself in those moments of tension. This need to fill that space with words and speed up the pace of the words that you're choosing, which, again, puts a lot of pressure on the other person. Pressurized speech, just thinking about it makes the other person uncomfortable. And when we're talking about a romantic setting, well, both parties want to be able to add to the conversation. They want to reveal their personality. And if you're filling all silence with your words and putting all of this extra energy into the conversation, again, very off-putting and unattractive. Yes, it's, you know, you have to be comfortable with those moments of silence. And I think that's where a lot of the magic happens, right? Like you can truly connect in a nonverbal way. And that's usually the more significant way that you connect with someone. And you can invite, you know, appropriate types of touch in those moments, too, or just, you know, just being together. But it's also true in friendships that you want to be mindful of that reciprocity that you have, that nice natural rhythm of sharing. So you're not dominating a situation. I think I did a video a long time ago about how oftentimes men will err on talking too much on a first date. And it's a it's a very typical error, but I think it's they want to make a good impression. They want to be interesting. So they start telling a whole bunch of stories. And I know you guys are very familiar with how to have better social skills and etiquette so that you can draw in the other person through your own curiosity and being interested in them. So it's the reverse really that creates better connection. Well, what's interesting about that is for a lot of our clients, they may have come to us in the beginning of with the idea of I had, I always run out of things to say and I want to be better in conversation. So of course they learn a ton from our podcasts and from our products of great conversational tools. And then they sling the pendulum the completely the other side. And now they're just their arm to the teeth and great conversation. But it doesn't stop. There's no there's no space for her. And and if you're not allowing her to share, well, that this conversation is uneven. It's weighted down by of all of his side. And she's not going to feel that it was very productive because he had left learning nothing about her. But yet she knows everything about him, which is the vulnerable position to be in, right? Right. I was just saying with that pressurized speech, we often have a tendency to overshare. We might end up sharing our insecurities and putting our foot in our mouth. Because again, we're filling that silence with words. And if you go on YouTube, you look at some of the best orators, you watch the creators that get tons of views. There are moments of silence. There are pauses to let the person who's listening, who's watching catch up to what you're saying. And if there's this incessant need to fill all of that space with more stories about yourself, more revealing who you are as a person, you're just loading up the other person with a lot more to think about. And you're not walking away knowing anything about them. And what's funny, again, in this video work segment, we'll ask at the end, well, what did you learn about her in that conversation? You talked for two, three minutes. What did you learn about her? And it's funny on day one, when they're first trying this video work, they'll often say, Oh, wow, that's right. I didn't learn anything about her. I felt the need to fill those two, three minutes with everything I could find to get her attention, get her interested. Or if they're thinking about what they're going to say next, they're not paying attention if she is sharing. So you definitely need to be present. And that is, once I think you get that aha moment and you figure out what presence truly is, it can be a very simple thing to master. But, you know, you have to start, obviously, the self awareness is usually the hardest piece, recognizing how caught up you are in the future, you know, thinking ahead and jumping to conclusions, rather than just focusing on your breath, focusing on the moment and being observant, basically. The rhythm of the conversation. And it's something that I've been focusing on as, as being on sales calls or being, being doing these interviews. And we had a few run-ins already where we had jumped on top of each other asking questions or whatnot, trying to establish what the rhythm of this interview is going to be. And you can't figure that out beforehand. You have to allow it to, to, to lay out and then you can see what it is from there. However, if you're constantly talking, trying to fill up that space or being focused inward with all your, with the anxiety going on, you're not going to be able to see that rhythm. And you use, something that you said in that observation that is going to be very important. You said the magic is in the, in those, in those silences. And if you're not picking up that rhythm, those silences are not going to be there. And you're missing out on allowing the magic to develop. And something I usually like to, to offer people when they're talking about these difficult exchanges is remembering if you're feeling uncomfortable, really tap into the empathic thinking, the emotional intelligence in the moment because you want to try to make the other person feel comfortable too. So maybe asking them a question or just putting yourself in their shoes so that you can technically like get outside of your own head and see things from their side of things. So empathy is so important too. So that's just something I wanted to mention. Well, in these moments in romance, we're looking to create sexual tension. And tension can be created one of two ways, silence, leaving the other person wondering what is going on, who is this person? Is there that spark? And then there's the inverse of sexual tension, which is loading the conversation with everything about you, leaving no room for mystery, no room for any wondering or thinking about after that date, who is this person, which often leads to the next signal, which is repeating yourself. And we see this in a lot of our coaching clients in our X Factor program. They'll ask a question of us and we're looking to help them work through and strategize and they'll end up repeating themselves and sharing the exact same thing a few times as if they're making a better point or they're making it more clearly. But in that repeating yourself, you actually don't sound self-assured. It doesn't sound like you know exactly what it is that you want to communicate. And that pattern can be one, again, that's very off-putting for the other person who's receiving the repeated message over and over again. Yeah, I mean, I think sexual tension is something else that's misunderstood or people think that it has to lead to that specific exclamation point at the end of the interaction and one way or the other could be a kiss or something, right?  So that can be tricky. Definitely. The over-explaining is, again, linked to this pressurized speech and repeating yourself this need to get into every last detail to fill the other person in with every possible thing that you're thinking and oftentimes can come across even worse like mansplaining and be completely off-putting. Yeah, hopefully I think people who are trying to mansplain a situation to death, I mean, hopefully the woman will kind of step up and like steer the conversation into a more relevant direction, but we don't want to leave it to them, right? Like you want to be able to catch yourself doing these things and at least redirect the conversation because you do have an opportunity to shift, right? If you see yourself making a mistake, it doesn't have to be like a disaster.  Like it's never too late. That's such an excellent point and I know many who are highly neurotic may also be perfectionists. So we'll often feel like, oh no, if you're listening to this right now and you're recognizing, okay, I do one or two of these things, all is not lost. Dating is imperfect and oftentimes some of these things will be imperceptible to her, but if it's repeated and if every interaction is filled with these signals, then of course you're pushing her away. You're removing that ability for her to be attracted to you.


People like stories. When things happen to us we like to construct an explanation, some narrative. We're very unwilling to admit it might just have been chance, just coincidence. Maybe we evolved to do this, but to me this suggests we really need mathematical help to try to distinguish signal from noise. Here's a map of London. Let's pretend that each grain of rice is the occurrence of some new disease. We've plotted them on the map, but we know in fact that I've scattered these grains at random. Well, I've tried to. And yet we've got clusters appearing. We could look at these disease occurrences and say, well, Willsden, Willsden, clustering Willsden, there must be something going on there. And they'll be deeply misleading. Events that occur completely at random do tend to cluster. And that's why we need rigorous statistical analysis to try to detect when a cluster is real or when it's just due to chance. Some great statistical papers have appeared in the philosophical transactions of the Royal Society. A real classic was Sir Ronald Fisher's 1922 paper when he essentially laid the foundation for modern statistical inference. One of Fisher's great contributions was to identify a measure of the compatibility between a set of data and a preformed hypothesis. And this is the p-value, which is the probability of observing such extreme data if this original hypothesis were true. Then if that p-value is small, we can say either something very surprising has occurred or our original hypothesis can be rejected. Take the Higgs boson. The scientists required very high levels of confidence before they concluded it was there. The p-value was less than 1 in 3 million. And that's both because it was a very important claim, but also because they searched so hard for it. We need to make sure that claim discoveries are not because of cherry-picking fortuitous extreme results. The published scientific literature might not tell the whole story. We need to know the trail by which the conclusions were reached. So I'm off to talk to Nicole Yance, who studies the reproducibility of research. So Nicole, what do you get your students to do? So my students take a published paper and try to reproduce the results. So they download the data, they do all the analysis, and they try to get exactly the same tables, exactly the same figures. Most of the students get very frustrated. They can hardly ever reproduce the same figures. So can we trust what we read in the scientific literature? Unfortunately not all of it. On the one extreme is fraud, where actually authors have manipulated data. There are honest mistakes as well, but just things where you feel like this hasn't been handled very well. You can do better. And then in the middle there's all kinds of things. For example, you get all kinds of results. And then you say, oh, I'm just going to cherry-pick all the significant ones. If you test multiple hypotheses, you should report all the results. If you report only selectively, I think it's definitely misconduct. And one of the problems is spotting this, of course, because how do you know what you're not being told? So how easy do you find it to get the full data from researchers? Well, that is a big of a problem. A lot of people are slightly afraid to give out their data because then someone will come and check. I think I'd be worried that somebody would find a mistake in my working as well. I would be worried too. Yes, of course. But I think, you know, if you work transparently, then at least I can always say I was open. And if you do find an error, yes, I'm human. That can happen. I wouldn't want to trust a study where I can't see the data. If you want to get published, usually you better have significant results that is backed up by all your figures and all your data.  And I thought, yes, exactly. That's my result. It's still something that we learned from. Of course we learn a lot from that, but it means that the story might not be newsworthy. Most often research might not be newsworthy in itself, but it helps science anyways. Journalists should really take up a stronger role in asking authors to upload their data and telling their reviewers that a null finding is not a reason to reject a paper. So we should work against publication bias. But I also think at the moment there's not a lot of courses in the UK that ask students to replicate published work. This is such a good way for students to learn stats, to do it better and to see when research is really reproducible and when it is really not. It's great that the analysis of individual studies is being checked and even better if independent teams repeat the entire study. But that's not the whole problem. We know that there's lots of studies that are done and never get published. And that's why we need open data. We need access to the totality of evidence, all the studies that are done, and not just the ones that receive prominent publication.


Hi, I'm Meg from Meg D on Art. I am here at Art Resin Headquarters today. I'm gonna be showing you how to make geode phone grips. They're a great way to make your phone unique and they're also super fun to make, so let's get into it. For this project, you're going to need some art resin powder pigments, but of course you can use resin dyes, alcohol inks, whatever you would like, and also some gold flakes. This is the mold I'm gonna be using. As you can see, the outside gives it that nice geode effect. We're gonna be painting that gold later. Of course, you're gonna need your phone grip base, some super glue to attach the top to the base, some gloss varnish to keep the metallic paint on, and your gold metallic paint. So now let's mix our resin. I'm going to be doing 50 milliliters, so 25 milliliters of resin and 25 milliliters of hardener. Okay, now let's set our stopwatch for three minutes and I'm just going to mix it. I try to mix at a pretty slow pace just to avoid getting a lot of bubbles and making sure you're scraping the sides and the bottom so that it all gets mixed together properly. First thing that I like to do is pour a little bit of clear resin in the center with some gold flakes. It gives it that geode effect. So I just pour it in the center. Even if you pour too much, it's not a big deal because we're gonna be adding some colors later, but I try to put about that much and then I will take these gold flakes and just put them in the middle and I kind of separate them into little pieces. And now I'm going to mix all the different colors with the resin. Pour a little bit of resin into each of these cups and now it's time to add the powder pigments. I add about that much and you just mix that in with the resin and you just repeat this process for all the other colors. So now we're going to pour the colors into the mold. I try to stay along the outside that way it doesn't seep into the clear resin that we've poured and then put your heat gun on a low setting to pop the bubbles. And now we're just gonna repeat that process for all of them. Now that we have poured these geode foam grip pieces, we're gonna let them cure for 24 hours. I like to cover mine just to make sure no dust or dirt gets into them. I will see you in 24 hours. All right, so I'm only at art resin for today so I have prepared some foam grip top pieces. I just have to paint the edges gold with this gold metallic paint, only paint in one direction because if you paint side to side it can take the paint off as you're doing it. I usually do about three to four layers of gold paint just to make sure that it's really solid and super pigmented. And the last step is to use some gloss varnish just to keep the paint on because we know we're all rough with our phones and I wanna make sure that this stays really solid and then I just let that dry. Final step is to super glue this onto a foam grip base, place the top piece on and I hold that down for about 30 seconds to make sure that it's really gonna stay. How to put it on your phone, you're just going to peel this piece back here and place it on. There we have it, a geode foam grip. I absolutely love making these. It's probably one of my favorite things that I make with resin. So I hope you had as much fun watching this video as I did making these. Thank you for being here with me today. Bye.