on the room and give Jonathan a few minutes to join us from wherever he's coming. So we're ready. Okay, for recording. Good morning, everyone. We apologize for whatever the glitch was that had us in two Zoom rooms, which is what the delay on the meeting start was. And we are trying to consolidate them. I'm Kathy Shane. I'm the chair of the committee and we'll pull up the agenda in a moment. But one of the first things I want to do is go around the room to make sure everyone can hear and be heard. We are conducting this meeting by Zoom and virtually and we are recording the meeting. So you will be able to see a virtual meeting later and we are posting these on our webpage. So I'm going to go around based on the people, faces that I can see on the screen, just to have people say, yes, they're here and they can be heard. So I'll start with Paul Bachman. I'm here. Steve Schreiber. I'm here. Phoebe. I'm here. Rupert. I am here. Ben. I am also here. Allison. Present. Anthony. Dwayne. Here. Mike. Present. And Diane. Present. And I think you said that Jonathan was in the other Zoom room, so I'm assuming that he will be joining us. And we were told that Ben, Sean Magnano is going to be joining us late. So we have a quorum of the full committee. So I think we will start now. And Anthony, if you could just pull up the agenda so that everyone can see what we are trying to go through this morning. We're going to be reporting out on the MBA meeting on February 11th, which I think you all know, because there have been a press release from the town. It was announced the school committee that we have officially been invited in to the feasibility study. And we can talk about what that means in terms of timeline. The big point of business for today's meeting is to discuss and review the draft for the request for services for the owner's project manager, the project manager that we will be contracting with to assist the committee, the building committee, as well as the school in designing and making key decisions on the school. We do have a draft that was done by the subcommittee. I'm anticipating that we may have some changes or suggested edits to this. So the committee will take notes on that to get to a final draft. Then the next after that is to discuss the proposed timeline for the OPM selection and key decision points and the roles of the subcommittee and the full committee in choosing the finalist, which will have to be done in a very telescope time if we want to make the June 7th board meeting for the MSBA, the panel that makes a decision on whether we're good to go with the selection of the OPM. And just for those in the subcommittee, when we were drafting it, on the 11th of February, we heard the timeline on the earliest we could be in was June 7th, because some of the advance drafts we have to give to MSBA before we can go out and post the request. So you'll see in the draft, we've adjusted some of those timelines. And then given that timeline, there's some suggested meeting dates for the full committee, but these are all in a suggested for discussion. So with that said, let me see whether Jonathan has joined us yet. Jonathan and Sean are both joined us. OK, so I think what we might want to do is just, Mike and or Paul, you were both at the MSBA meeting when we were invited in. If you want to just give a quick summary of what we heard at that meeting. And then I've asked Jonathan, who's the chair of the subcommittee, to walk us through some of the key parts of the draft, which are the goals that we're setting out for the all-owners project manager and the weights that we're giving. We have to say what we value in terms of the responses and the high scores will be invited for interviews and they become our finalists. And it influences the decision-making process. So Mike and or Paul, if you want to talk about what we heard on the 11th. I deferred to Mike. I can be very brief. I know the point of the meeting is much more at the OPM level, but it's a very pleasant meeting for a couple reasons. We were first. That always helps. But more seriously, MSBA noted that we'd completed all the tests required to be considered to enter the feasibility stage, which we'll talk very much about the rest of the meeting as a first step in that feasibility process around the OPM. I want to thank Matt from Desi. He's the Desi rep on the MSBA. He made a point to note that our buildings are in poor condition. He would definitely give us great feedback. He was very involved in the last project and just noted that he was really pleased to see us back again because he knows our staff and students really need a different learning and working experience than what they currently have. MSBA board voted unanimously to support the project to move to the next phase. Following that, there was a OPM module orientation that a number of us attended as well. But I'm sure Kathy and others may go over in terms of the steps and process there. But it was really nice. I know a couple of us, Paul and Kathy, for sure, stayed on and got to see some other schools who were at the next stage. They did all the feasibility entries, the rear four of us, and then they were able to see what it was like at the next phase where there's much more detailed design. And that was really, I found it really neat. I think there was a lot to learn from district, even if it was a secondary school, just what they shared and their process. So that was really, Paul, Kathy, Sean, I know you were there and others. Is there anything you want to add? That's kind of the quick summary of the meeting. It's like they vote and then right away you're whisked away to an orientation for the next phase. They're very efficient, which we appreciate. But Paul, Kathy, others, anything I'm missing? I think the only other thing is I did stay on for the module as did others. And when we got to the groups that were ahead in terms that they were starting to say their project and what they were doing, one thing of note is there was at least one net zero energy project in the pipeline for elementary schools. So we will have other places to look outside of Amherst for what those designs are, what they're thinking. And there is more than one elementary school in the pipeline as well. So it's not some of them were vocational schools and high schools, but there was one elementary school that is just it is basically just ahead of us, like six months, seven months. Not they were already putting a shovel in the ground. So I found that quite interesting. And then the thing that we had been warned or cautioned by Mike is that there is a schedule that MSBA has. And they work with us every step of the way. So even on the draft of the request for services that we're about to review, that we're drafting, they want to see that draft two weeks before it goes into a register to make sure that we've followed their template, that either that we're missing or that we did things that they have comments on. So that sets our timeline because then they want it to be out for a certain number of weeks for responses. And that means what and then they need our final selection at least a month before they take it to a panel. So it's each step of those things puts several weeks of time in it that's out of our control. I mean, it's in our control that we have to meet that deadline, but then we have to wait. So I think I'm going to turn at this point to Jonathan, who was chair of the subcommittee. And Anthony can pull the draft up, which we sent to everyone. And this is very much a draft. So we've had a couple readers send in some notes, but anyone should send us in some notes on if they see things that aren't quite right either in the background section or some wording they want clarified in later sections. So I'll do just the Anthony, maybe we can scroll through so people can get a sense. There's just a basic introduction that was more or less boilerplate on what we're doing. And then this background section, most of it was pulled from the original statement of interest that the school we submitted as a town. But it was updated what we've done since that statement of interest and gave a sense of where we're going. And including that we've done a space study in the middle school that determined there is enough room for the six to move up if we go in that direction. And so we do need to make sure all of these pieces are accurate. So all of this is the background for the bidders who are coming in and looking at this. And I'm now going to turn it over to Jonathan, whereas when we started filling in what the goals are we see for the project and what weight we've assigned to various characteristics of the bidders. So Jonathan. Good morning. So Anthony, why don't we start there. As Kathy said, a lot of this document is sort of pre-packaged boilerplate that comes from the MSBA. But there are parts that we can fill out and need to fill out. And one of the first and most important here, if Anthony scrolls up a little bit further, is the project objectives. Whoops. Sorry. We'll get to that. Yeah, there you go. You're seeing objectives, right? Oh, yes. I guess I was looking at the bullet point list because I'm assuming we don't want to try to read out every word during this process today. But some of these bullet points for project objectives, Kathy noted, were pulled from other documents. But we tweaked this to some degree and we tried to customize it for this project and for our community. And so certainly people should look at this and see if there are items that the subcommittee either missed or put too much emphasis on. I'll pause for a moment because this is kind of one of our big sort of instructions. But it's one of our opportunities to tell the OPM who will work for us what is important to us as a community. Can I ask a question? Yes. So dedicated space for a music computer and special education. Why were music and computer sort of separately identified as opposed to like R and P and other specials? Kathy, was that, did that bullet point get pulled from another document? Or maybe Anthony knows the answer to that one. Anthony may know because that was originally in Anthony's story during this. I think that particular bullet point came from the Fort River study, I think that phrasing. We talked about why we weren't calling out a gym and that was because a gym is part of the default MSVA setup. We did not discuss other art spaces. So maybe that needs an inclusion, I'm not sure. I see Mike's hand up. Yeah, Mike. Sorry, I can't. I guess I can see the hands, the little hand threw up on the screen, yeah. Oh, I'm sorry. I can do this thing too. So just a question or process question I may have missed a couple of years ago because I was having little Wi-Fi challenges. Is this an opportunity to offer feedback as we go through the sections now or from a process perspective, how would you like, because on that particular point, I actually do have some feedback. But if now is not the time, then I'm happy to offer it at a different time. I would say that now is the time to offer feedback on this. And then we haven't scheduled, but we will schedule another meeting of the subcommittee to go from what is very much a draft to more final. So it's both now people should raise issues. We will take notes and then share a revised draft with everyone, and people can send in any notes on this. Because this was done, as Jonathan said, using other documents, other towns, and then things that occurred to us. And it may even need some wordsmithing to make it clear what we're talking about. Sure. So I'll offer two pieces of feedback then. On the bulleted point that was raised, the second one down, I might suggest that we say dedicated space for special programs and special populations. For instance, computer is a tricky one. My recollection the last time is that MSBA wasn't really supporting building computer labs for elementary schools because it's part of the project. Most districts go into Chromebooks. And the idea of a lab instead of a maker space is a conversation we'd have to have. But I think keeping it more general, I think, talks about instrumental music without getting into the specifics that maybe misinterpreted. I think the other piece is going to be the least popular comment I make, perhaps. I just wonder under objectives if we also include a fiscal objective that I am aware, Sean, to be really pleased with me. I am aware that there's multiple projects being considered. And so I think we want to do all these things, but just being explicit because this is, again, trying to get an OPM, one of their jobs is to manage cost control and how many changes there are. I don't have all the language Jonathan, Rupert, others do. And Steve, but just something about that we want to we want these things. But we also want to be realistic about the cost of the town as well as the MSBA. So I don't know if that goes here or not, but it just said project objectives. And I think one of the project objectives, I think, has to be delivering all those, having all those deliverables happen and the town still being able to function financially. So I don't know if it's worth putting there. That's up to the group. But for me, it just seemed like it was missing. Well, I'll give you a reaction. I think it is definitely worth putting in. And we had an example from one other town that had cost effectiveness, doing what is needed, but being really paying attention to the cost. So we can, Anthony, you can put a placeholder there and we can wordsmith it. But it's in a cost effective way while controlling costs. Yeah. Kathy, I see another hand up. OK. Rupert. Yeah, I'm going to try to figure out how I can see hands. Rupert. Yes. I think I'm unmuted. One of my concerns, of course, going forward with the new building would be the maintainability of the building. And that may be reflected in the bullet point that says life cycle costs. Or maybe it needs to be pulled out specifically. I've seen a lot of especially high end, low energy buildings have extremely complex, difficult to maintain systems. And I think there needs to be some kind of a balance to keep the building maintainable. OK. I would say that I think that is the intent of that bullet point. But we can at the subcommittee look at that and figure out if there's a way to frame that a little better. Thank you. Kathy, I can't raise my hand because I'm a host. So in terms of smaller editorial comments, just put those in writing and send it to the committee. Is that your request? Yes, absolutely. Either write on the document or in an email to us. Either way, whichever way is easiest to you. And I think if we didn't, we can send you the Word document and you can plunk in a comment bubble if you don't have exactly how you want to word it. Got it. Thank you. Everyone should have the Word version, right? That was sent out. This question, I think, is more for Anthony, is the bullet point of compliance with the towns, wage stuff, bylaws. Is that meant to cover both of those, the two that we looked at, one that was more focused on sort of construction and then the other one? That was meant to be kind of all-encompassing, yeah. OK. I didn't really want to go into a long discussion here so much as this should prompt someone to look into what those bylaws are. Maybe have a link. Maybe we'll have a link to it. One thing people should just know on my screen, I've got the faces going down. If I try to put the hands up, I can't necessarily see them. So if anyone sees hands up, just please tell me. Kathy, if you open the participants window and you have another window open, you should be able to see the whole list of attendees. I can. And then I can't see, well, maybe make it smaller. I can't see the text, but that's fine. I've got it. Would it be too disorderly if people just spoke up like I just did? I don't mind if people just speak up. Because we don't want to move through this when people haven't had a chance to read. One of the questions I asked was auditorium space. So the same way people just called out special programs, if we call out an auditorium space, I think it says that we definitely want one in the school. And I'm certainly not against that. But I just think that there should be some agreement that we do. I grew up in a grade school that we use the gym and the gym. And it was what did you call it, Steve, a gym auditorium? Gym or cafeteria. We had something that was in two ways. But so it's things like that that I think we set these as goals and objectives. But we are somewhat same with the content of the building will be at this point too. This doesn't necessarily lock us into any particular configuration of the building. But it is a communication of the community's goals. If folks feel like they've given the comment they want to give, we could move on. But I don't want to rush things. Yeah, I just have one other comment, Jonathan. Is the creation of a net zero energy building, is the Northeast collaborative for high performance schools? Is that redundant? No, these are a series of different options that the MSBA requires. I believe at least my memory from the Fort River process was that you have to pick one of these to comply with, or the designer has to comply with as they're moving the project forward. And that net zero is really, that's a town requirement that's sort of on top of that. And so I think there's value to having all three items while they all get a greater interest in sustainability. It should be prudent. In fact, I think that point on the chips and the lead, that language I think is straight out of the boilerplate part of the document. Correct. OK, thank you. And there were some discussion about whether or not creation of a net zero energy building, so whether or not that should just state net zero ready or something like that, because you can't measure net zero until the building's operating. So you don't know if you've created it until after it's done. Yeah, I think we should probably talk a little bit at the subcommittee level about whether or not we have the right language. In the end, we're going to have to expound as a committee and a town significantly on what those goals are, and both the OPM and the designers will have to work with the bylaw as it's written. Right. I think the bylaw does have that test to see about performance. So thank you. Anthony, let's scroll down to four if we could. So the next section, as it says, is minimum requirements and evaluation criteria. Some of this is boilerplate, and there's other areas where we've edited it. And I will do my best to remember which sections are which and where we made edits. But I would freely welcome my subcommittee members pointing out stuff that I may miss as I'm going through here. Those first two bullet points are pretty much boilerplate. I don't think we edited the length of time for either or the experience levels for either of the two bullet points. They seemed reasonable to us. The next section that starts with the owner, that is the section we added. The standard document does not speak to minority business enterprises or women business enterprises. And while the town does not have a specific goal, we felt that it would be good for OPMs to work to discuss how they can be inclusive in what they bring to us in the way of services. Scrolling down a bit to evaluation criteria, I should probably back up and just talk briefly about if folks had a chance to look at the samples that Steve was able to find from other communities, most of them had a point category or a way to grade people on their responses. And so we have a list of evaluation criteria that we're proposing to judge the applicants against. And then we have a weighted maximum number of points that folks can achieve for each category. And so while we did use some of the documents that we looked at as guidance, we also made some of our own judgments on the points and therefore the weight that we thought and the importance as it were that we thought a certain thing held. And obviously we want to look both at the criteria but also the weighting. And again, folks can focus and tell us today in this format what they see, what they like or don't like, but we can also gladly take comment, as Kathy said, via email. So can I make a comment before we start here? So this is related to the previous stated goal of really encouraging women and minority participants. So which is the other direction. But this is going to be a comment regarding the whole thing. So we just had a speaker as part of the architecture lecture series who's from Minneapolis. And he made a comment that really has resonated with me that if you have the criteria in RFPs, so he used an example of performing arts like a theater in Minneapolis. And they issued an RFP. You must have done two other theaters. So he responded that no black architects in Minnesota have been able to do a theater. So therefore, by that very criterion, you have eliminated the possibility of a black-owned architecture firm responding to this RFP. So I hope that we'll look at all of these points that we have in all of these criteria that we have through that lens, that who are we eliminating the possibility of diversity by having criteria that or by having a point system that might have sort of a subtle bias like that in it. So I didn't mention this to the subcommittee because he literally just spoke at our school. But I thought that really resonated with me. And I just want to ask a question on this to show my ignorance of OPM versus the designer and the architect roles here. At the point where choosing a project manager, we haven't yet chosen the architect and the designer, although we're setting the stage for all of that. So do some of those concerns you've just raised, Steve, particularly come in when we are picking the group that's actually going to design the building for us? Absolutely.  And so we should just, I would suggest that we just be really careful as we look through all of these points to consider whether or not there's an implicit slanting of our requirements towards something. So the thing I hear from by looking at this is that we sort of rate value everything about the same. Everything gets five points. There are a couple that get 10 points. And it seems like that doesn't really establish what our highest priorities are other than the ongoing maintenance and knowledge of net zero because they get twice as many points as everything else. And I just wonder if we truly do value everything the same, that putting together a website is as important as knowledge of the state building code or whatever it is. I'm not sure which would have more importance to us, but it seems like just a lot of small things here that are there any things that we think are really hyper-important that we really need to give a lot of points to that we care a whole lot about. And I don't have an opinion on that, but just wondering if we should be communicating to the people responding what we're really concerned care about. Well, certainly at the subcommittee level, the ones that got the 10 points were ones we were valuing a little bit more than others. It doesn't necessarily mean we hit the right note, but kind of management, the ability of the OPM. So stepping back a second, what does the OPM do? The OPM represents us and works for us. And there, now I'm speaking for me personally, their ability to communicate for us and to us and with the designers and with the community, that's something that seems very important to me. Likewise, having the knowledge about it, how to get to a net or at least try to get to the net zero requirements seemed important. But others may value other items. And just one other thing I thought through, Jonathan and Steve had come with two possible ways of waiting this. One is just the straightforward you're seeing here. And one, another was extra weights, like if you were extraordinary on something, you got double the count. But if you look through this, the capacity and skill, the knowledge, and the team you're bringing to us, if you add those three together, it's a lot of weight. So that you've done this before, you're bringing a team we really like and they've got enough time to do it. They're dedicated time. And then that capacity and skill thing, we've given a lot of emphasis to what team are you proposing to us and what knowledge do they have and what evidence do we have that they've done a good job in the past. The other thing. Yeah, they're heavily weighted in the cluster of pieces. The other thing I would say is that, although in one sense items like knowledge of the building code and knowledge of procurement laws are extremely important. In another sense, they're also kind of basic expectations and rather low bars to clear. So even though items five through five and on might not matter if you don't have item two, it's also our expectation that everyone who inquires is gonna clear number two pretty easily. So if we're gonna differentiate them, having a bigger point spread somewhere else. So you could, or the other way you could say it, Anthony, is you could lower the points you're giving for two because everyone's gonna get them. And put it back down at one point, financial stability was only given a three. And then we realized we were two points short of 100. So we gave him two more points, just to be rounded back up, but we could look at those again and say, if we really think they're not gonna, if they're not a viable company, they're not gonna be make the cut either. So we could lower points and redistribute a few points to make more of a differentiation would be one possibility. If Mike Morris has his hand. Mike. Yep. So I think that's right. I agree with all the comments that I heard. And I think the thing to, just for people who haven't been through a process like this before, is this is really good to find the package by which we evaluate them, right? So most of the people on the call, I have to, some of the people on the call, we have a lot of architects on this call, aren't going to be able to answer and evaluate number two, right? So I think, you know, the point thing, point differential for me, I'm not gonna weigh in on what should or shouldn't be how many points. It's really gonna define where the potential applicants put their work. So if there's more points for familiar energy zero building projects, we can then expect that all the architects whether they've done what or not are gonna invest a lot of resources into describing how they would approach that. So I think, you know, the points piece, I would, in general, you're gonna probably find lots of qualified people to do this role. And so I think that the advantage of being more to Paul's point of being more clear on the points is that's really gonna define and let the potential applicants know they really care about net zero bylaw. We're gonna put a lot of time. We're gonna put our green person onto, you know, really doing a lot of work. And I think it's a signal that's important more than actually the points and how they get calibrated because likely there's gonna be people who are really solid, although people are really tiered points made or we're gonna be really solid on some of the core pieces. So I think that's the reason to be more clear and make sure they align. I just wanna say this for the general group is that it actually defines the presentation and the packet that each of these potential vendors creates. And I think if they don't have much experience, for instance, I'm picking on the net zero one because it's the one that's in the middle of my screen that says 10 points, then it's gonna stand out like a sore thumb, right? And so I think that's more than just how they all add up together. That's its own thing, but I think it's much more a signal. So the packets we get are consistent with the values that the committee has. I see Steve's hand is up, Steve, is that? Yeah, no, that's a... Yeah, so I'm gonna make a slightly counter. Well, I'm gonna make... And I know that I'm also on the subcommittee so I have a voice there. If I were in the zoo, I would move as many things up into minimum requirements as possible and then leave the evaluative criteria, the ones with points to those ones that we really think are important. So we've already talked about things that we think are baseline. Of course, you have to know the building code. Of course you have to, and quite frankly, you have to know A because that's in the template, right? So if you're working with schools and you have to know that already, but there are other things that I think that are immeasurable. In Amherst, the ability to work with a complex, with a very engaged community, that's kind of critical. In other words, and that's not, so for me, I would give many more points on a proven track record of working with a diverse community that wants to be very engaged in projects like this. That would be something that I, I've talked about diversity about how I think that's also very important for this, just to send a message through the people that we hire as consultants that we're committed to diversity, I would give more points to that. So that would be my suggestion is to put as much as we possibly can up into minimum requirements and then leave the points for things where we can really express our values. I don't think that's an option to us, unfortunately. Okay. To change the minimums. I don't, you know, they said there, but they said- Here's another thing that my most direct experience working with an OPM is on the design and construction of the design building at UMass. So at the time that that building was being considered, we did not know that that was gonna be a timber frame, mass timber building. So nothing about mass timber was written into the OPM criteria or into the designer selection criteria. So that all changed during design development where both the OPM and the architects had to quickly become experts in mass timber construction. What everyone knows about that building now is that it's made of mass timber. That was zero points on the OPM and zero points for the architect. What was critical in both the OPM and the architects was the ability to pivot. So they basically to have the flexibility to learn about a completely different construction system. So that's another criterion that I think is critical is this ability to learn, to know, to pivot. So even at zero, so I think that most OPMs and most architects can become well-versed in how one achieves net zero fairly quickly, whether or not they've actually had that direct experience. So, Anthony, can I just ask on the minimum, are you, I know we can't get rid of any of minimum, but what Steve was suggesting is adding some things to minimum. You think we can't add to minimum? I don't think so because we expanded some of these evaluation criteria, but most of them are already here in the evaluation criteria rather than minimum. So I don't think so. I mean, our instructions in that conference call were to not mess with anything that wasn't specifically highlighted as something we were supposed to fill in. So then, Steve- Like honestly, I'm not even sure about adding this here. Like we debated where it really belonged in the document. Oops, where it really belonged in the document and I'm not- Well, that is, they get this two-week period to review our draft, but then if we want to respond to what Steve's suggesting, it would be to lower the points in the evaluation and redistribute points as the way of strongly signaling this, correct? If we need to keep a particular category, even though we think it's more or less minimum, they're not gonna be applying if they don't have it. Yeah, okay. I can see that Sean has his hand up. Sean's hand is up. Can you hear me? Yes. This is just a more of a functional comment sort of based on the evaluation criteria. If you're allowed to, I would suggest, and maybe I missed it, maybe it's already in there, but I would suggest putting something later in the document that says the submissions that you get from the OPMs will clearly identify where this information is in their package. One of the things that's always been a challenge in evaluating OPMs and architects is trying to find, you're looking through the document and spending lots of time just trying to find how you can evaluate these criteria. And so if you can get them to submit it in a way where it's clearly marked, number one, here's our information. Number two, here's our information. It'll make the process that you guys have to do for evaluating it much cleaner and more straightforward. Yep, go ahead, Anthony. So everyone is going to submit the state form, the designer selection form, which is pretty restrictive and it lays out exactly how they're going to... Yeah, I'm talking about some of the ones like their approach to up above, there's things about how they're gonna, their approach to doing something or information on net zero. Those types of things won't be on this form or at least not their approach to how they're gonna do them. Yeah, there should be a section at the end of the state form. For architects, it's typically a maximum of six pages where you're supposed to address the evaluation, you're supposed to address specifically the evaluation criteria. I don't know if it's six pages in this one because I didn't look at that end of it, but I'm sure it's of some form of that is there, Anthony. Correct? Yeah, anything you can do to make it as straightforward as you can for comparing it will be good because I just know the last time around, it's a long time just kind of searching for things. Yeah, three double-sided letter size sheets. Is this what you're talking about, Jonathan? Yes, that is the spot. Okay, that's good. Yeah, I would expect that most of, we're not most, but much of our evaluation is gonna be based upon the information placed there because a lot of the rest of the response will be somewhat boilerplate. Just as much of this is boilerplate. Do others have additional criteria or additional comments on the evaluation criteria? I think we've had some good feedback and there'll be a good opportunity for us to revise it at the subcommittee level. Jonathan, this is Mike. I just wanted to note that Rupert and I both have a meeting at 9.30 when the meeting time was changed, it bumped into it. So I'm not pushing Rupert here already, shared some feedback, but just, we will be departing at 9.30 and just wanted to communicate it wasn't because we're not committed, but with the time change, we just couldn't change our 9.30 meeting. Well, you know, Paul with, Mike with that said, I think you should definitely then send comments, you know, and we will bring them into the committee and we will bring them in as a public document so that it's when you get a chance to focus on this and we will commit to, this is first draft, there will be second draft before it's final. So we'll figure out some way of, did we address concerns before it leaves the town? You know, goes into somebody else's mailbox. Yep, that'd be great. And I think maybe even with the minor edits when they get cleaned up, if we send it to the full committee, you know, I know Rupert and I will get back to you in a timely manner. So maybe Anthony just go next to the timeline on all of this. So this does not reflect the information we received at the last, at the meeting after our last subcommittee meeting. These are the old dates. Okay, so the one I think I sent to everyone, I went and changed those dates, but that's fine. You know, so these have the old dates in them, but now the target date is June 7th in terms of the panel date is June 7th. Sorry, that's the date. So working backwards from that, the earliest we would know for sure we're in is June 7th and they need to, a month before that is where they get who we've selected. And when you go back to the very beginning where it says it appears in the register, they need the draft two weeks before it goes into the register. So, you know, so that's what moves then this timeline and then it has to be out for at least two weeks. So that's the schema that we've been put into that's responding to them. So in between that, its respondents come back to us.  So the question here is when, what piece comes back to the full committee versus the subcommittee? So I will just- So just on this timeline, so we have to have this, the RFS finished by in two weeks. It has to, the RFS has to be, this timeline can't, they have to have our draft two weeks before it goes into the register. That's tomorrow, right? Right, so when we were listening in on February 11th and I verified this, I asked them a question, they said the earliest air mist is gonna come into a panel is June 7th. And it was because their timeline has them for the June 7th, the ad would go in no later than March 25th, but they had to have it two weeks before. And if you want it to be on the earlier, the May 11th, the May 3rd fall. Yeah, so- My question is, do we have to have the RFS ready by March 3rd? Is that my question? No, no, under the timing for June 7th, two weeks before March the 25th, so subtract 14 days from 25. So we have a little bit more time than that. I just changed them to the dates in your PDF, Kathy. Perfect, okay. Thank you, that's helpful. Okay, so that's what it actually comes out in the register. You have to do it a few days before, before them to post it. And then they want it two weeks before. Okay, and I said, if we can get it to them earlier, we can post it earlier. This is a deadline if we wanna get to June 7th. So if we're ready in a week, they will review it as soon as we're ready and give us a go ahead, to go ahead and post it, which would give us a little bit more time in the timeline to get the responses back and evaluate them. So we could look at our own, what's possible for us to do a selection process that makes sense. And that was, I think, Anthony, that was the news we got on the 11th on that they needed a full two weeks to review our draft. So we couldn't just, we could try to move really fast, but we couldn't move fast enough. And then the rest of that section is pretty much all boilerplate. Well, I don't think we have to make a decision right now, but when we look at the next time we would meet as a full committee, at what point in the, everyone can get a revised final draft before it goes out. And then it's a question of, in the selection process, when we're down to three on interviews, Mike is either waving goodbye or are you waving goodbye? I am, yeah, Rupert and I do need to get another call. Thanks everybody, appreciate it. But we'll follow up as indicated with the revised draft and offer any feedback we have. Okay, and future meetings will be at 7.30, we'll make the posting in time. Oh yeah, no, and I don't have a preference. It was just that I already, we both had already had another meeting scheduled, if it works better to be later for folks that's sort of arbitrary to me of just this week, this time I couldn't change a meeting that was already set. Okay, so maybe Anthony, I think you could take this down at this point. And so the scheduling meetings for the full committee and certainly if people want to be, we will schedule another meeting, at least one more meeting for the drafting committee, the full committee, the process would be, we get to a full graft, it goes in for review, then it gets posted, then we start getting responses back. We're sitting down and assigning evaluation criteria, doing background reference checks and interviews. So at what point are we doing which things at a full committee level was my question? We didn't have an answer to that in the subcommittee. Any suggestions or thoughts? I would suggest that it be at the subcommittee level, but be prepared to open it up to all committee members and post as a committee of the whole. That way if people who don't want to participate in that level don't have to, but if people do, they have the option. The document has written assumes that the OPM subcommittee will be the ones doing evaluations and reference checks and such, that was true. It was more at the final selection point when we're down to our top two, top three. So does that mean that the subcommittee will evaluate and do interviews, but then the decision will go to the full committee? Well, I'm just, I'm raising that as a question. I'm just raising it as a question. And Paul, I think was saying we could do everything at the subcommittee level, but invite everyone in to be part of those key decision points. Sean's hand is up. Yeah, sorry, I think, so I think it's a sort of a combination of what you both are saying. I think the subcommittee does it all and does the short list. And then at some point explanation of that short list of the full committee, why was this the number one would probably be helpful for the full committee, explaining what separated them from the other ones. And one of the things they said in our briefing on the module was when we submit our choice, we're supposed to do a full narrative of why we gave the points we gave, what the interview was like, what we heard in the interview, a narrative thing. So they want documentation that we went through these different pieces. So we will have to do that in any case. So that would be something we can be preparing both for the full committee, but it's something we have to prepare when we go and say, here's our selection. They want a full narrative report. And they said, they will turn it back if it doesn't have that. So they encourage us to take good notes. Any other thoughts on this? Because it does it, we can kind of think this through on when we would next meet as a full committee then. Everything will be much more active in terms of the full committee once an OPM is when our office, the owner's project manager is on board and there's going to be some waiting time once this is out. But I think we could be talking about what the steps in the process are going to be. So we can set a date for March and set another date for April. If people are ready to do that now or I can send out potential dates. You know, we would need to have a full draft in by the middle of March to make the deadline. So we could do a Wednesday in the middle of March date at 7th or in the morning is the next time the full committee meets would be my suggestion. And to be more specific, that would be the 17th of March. The 17th of March is the next full committee meeting? That would be next full committee. And by then I would expect we have a full draft and we may have already submitted it. Okay, so we wouldn't. So we're not looking for the full committee to approve the final draft then. We'll let the subcommittee do that. If that's all right with everyone. Can we, you know, with the open meeting law if we do a second draft and we submit it and get comments back from people and then discuss it at a full committee? I mean, if the comments just come in, I think we can get to a final draft that way unless people want to review a full draft as a full committee. I'm looking for a hands up any response to that. Yes, Paul. I think the full committee should sign off on the full draft. Okay, then we would need to meet on March 10th. Can I, I would love to do it even earlier. I would love to, I would love to be able to start our timeline earlier and have this out in the wild for longer. Then March 3rd. If we could, if we could, if we could do that, if we, if we approve the final draft on March 3rd and submitted it to MSBA that day, got it in, got it in the register, you know, March 24th or even, all right, March 22nd or. Then the question is, so the subcommittee will need to meet today's the 17th. We can't meet before Monday in terms of posting, but we would need to meet Monday to get to a final draft to have, or at least that week. So you're saying, so then a full committee meeting on the 3rd is where you're pushing for Anthony. I would, I would like, I would like that. Like we could, we can't, we can go later and we'd be okay to go later, but given the option, I'd love to have this, I'd love to have this posted with a response deadline of a month instead of two weeks. You know, give people. So how, how, everyone on subcommittee, are you okay for meeting next week and getting to final? And if we need to, we could post two meetings next week, you know, beginning of the week and end of the week. And if we only need one, then, then we're done. And that, that, does that work? And then March 3rd would be posted as a full committee meeting, is that all right with everyone? Yes. I don't see anyone saying no. I don't think anyone's, and no hands are up. Dwayne, does that work for you? Subcommittee, Dwayne's on our subcommittee too. Okay. Okay. I think I asked a clarifying question. No. So maybe Anthony, so what happens to walk me through the processes? People submit their proposals, you, you gather them. The subcommittee reviews them to see if they meet the, to do two things. One is to see if they meet the minimum requirements and then ranks them. Yes. Subcommittee interviews all applicants or selected highest ranking applicants. It's, it selects its shortlisted applicants, which must be at least three. And then those three are interviewed. Yes. Subcommittee ranks those interviews, comes to the full committee with saying, here's the person we think should be put for, here's the group that we think should be put forward. And MSBA also has a, has their own people in there. I'm not actually a hundred percent sure how the, the document itself lays some of this out. Yeah, I just read the document. I didn't, I didn't feel there was clarity on that piece of it enough. So I guess the question to the larger group here is, are we comfortable with all that work being done by the subcommittee or do this larger group want to have us say and be part of, and that's what I brought up on the interviews. Do you want to make that a committee of the whole or not? Or are we comfortable with the subcommittee carrying that load? It's a lot of, it's a lot of work. So laying it on the subcommittee is good by me, but just want to make sure we're, no one feels disenfranchised by the process. So if we meet on March 3rd, we can reopen that question again, Paul. So the other thought I had is having heard all of this for that next stage, we could make a slightly larger subcommittee or bring someone else on if we really, wanted to make sure someone was in the room. But there is, there's, there are key time points. So one of the piece would be, there's background reference checks where we'd be calling people who have worked with the OPM team in the past. So once we get to the short list, we would be doing those as well. I would be, you know, I would be happy to take all comers as far as who wants to participate in interviews and look at the, look at the shortlisted finalists, but my strong preference would be that anyone that wants to be involved at the end should probably be involved in the whole process, like receiving the initial applications and reading them and, and hearing the feedback from, from references, coming in, coming in just to do the interviews, I think would, would be, would be a difficult, I think it would be worse for that person than to, if they were involved at the start. Not in drafting the RFS, but at least in getting the responses. No, I agree with that. And so that's just, you know, whether Ben, Mike, Paul, any of you who aren't on the subcommittee want to be there at the beginning and seeing some of this, we can make that decision on March 3rd, but that is, that we will have the time to make that decision. And then we can certainly hold these as committees of the whole at key points, but not, not as Anthony said, reading all the documents. Is there anything else? I'm going to open it up for public comments if the committee doesn't have any other comments. I don't see any. So I think we are now open. We do have four people. Any of you who would like to make public comments, please raise your hand. And please know that you can also send us comments by email and the subcommittee when we're looking at the draft will be taking those into account as well. I'm not seeing any hands up. You'll give it another minute. No, I don't see anyone indicating public comments. So I think we're, I think we are finished with the agenda. We have, we will send out a meeting notice and we'll make sure we get it posted in time for March 3rd, 730 in the morning for a full committee meeting. And we will post our subcommittee meetings for the draft you've just seen and revisions that people have already remarked on and or some that have been sent to us will be discussing them as a subcommittee. I think if I don't see anything else, I think we have completed our business for today. I don't see any hands up. Move to adjourn. Then there's a move to adjourn. Is everyone in agreement? All right, just raise your hands. We're ready to go. Thank you all. Thank you all for joining us. Thank you, Kathy. Thank you, everyone. Bye. Enjoy for a moment. Thank you.


أرحامة الله وباركاته أهتم بأخكام SOS السلام عليكم كرماتكم ورحمة الله ورحمة الله كيف يجب عليكم ورحمة الله عشاءكم سلام عليكم بأخذت الخطأ كيف يجب عليكم إعادت المملكة على إلوانة السيدية لأيامها كيفية عمالاتها التي علينا أن نفهم. يجب علينا أن نوفرهم معهم ويكتب بها في حالة الإجتماعي حيث يجب علينا أن نساعدوا بفسطة أولا. إذا أردو لك in تجاوية فجنة فاقرة أن نجدهم؟ هل لن يفصل إلى حياته؟ إذا كنت مدين about كمينتنا تجلس أعادة المملكة على إلوانة؟ رحمان الرحيم الحمد لله رب العالمين وصل الله على محمد المواليه الطيبين الطاهرين سماحت السيد he mentions this messa'la and he says that it is mandatory it is وجب for the youth's leadership those who are responsible to lead a group of youth for example let's say in the community as a community leader or youth leader that they have to also have the concern about the health of the youths and especially nowadays and what's happening in this world that we live in in this day and age we see that depression anxiety mental health issue is rising and even within the Muslim community that's why they need support they need mental support they need psychological support they need face-to-face support in this case we have to facilitate that in somehow now in some countries the health system pays for these treatments and they take care of it but some country they don't so the Muslim community should encourage for those who are able to support to help for example the youth and as the said says to for example to offer the health insurance cover for the for the youth so if they fall into such issues they recover it is an organization who can pay for their treatment and of course they have access to the doctors professionals who can and experts who can talk to them and support them and of course religious support is again another way in which the one can seek and get help from by making them to remember Allah the hereafter the jinn the paradise the rewards for example and the importance of participating in the majalis for example and not to stay away from the majlis often the community for any reason for an excuse and so forth that can also help in this regard and of course the charity organizations can also help and support each individual separately with this issue of course the charities are paid by donators but people sometimes by the governments so they can seek help and advice from these organizations and centers شخنة you know the youth to their age I mean you know what they go out and they educate themselves and they can educate Islamically and also in other sciences but sometimes our youth are deviated how do we protect our youth from deviation of any sort in general generally speaking well again it is imperative for the youth leadership now be it's the parents or the community leader or whoever is in charge of the youths to try to protect the young girls and boys from sliding into the deviation and corruption and of course that is by teaching them the true beliefs of Allah and the day of judgment and the حساب and to teach them these values of Islam that we have a profound belief we have a strong عقيدة in the hereafter there's a حساب awaiting for us in the grave on the very first night of the grave there's a حساب the very first day in which the one is buried in the grave will be accountable for his deeds because to be honest no one can protect the youths from these deviations nor the the social services the government the police the security services nobody can protect they give overall advice and guidance but nobody can protect and educate them as much as the religious organizations and centers can provide and the and the حساب can provide such guidance and continuous as I said mentoring and educating them all the way otherwise things will be devastating and there are the results of deviation and corruption by the youth would be not only for them but even the society will will hurt and will see the consequences which we don't want to see that at all now it's funny you mentioned corruption because كاربشن is a big issue in our community you know Islamically and even outside Islam and in general and you know within the public there's issues of كاربشن how do we instill you know virtues and morality with our youth to you know stop them from being corrupted or being even involved in any sort of كاربشن at all well when we go back to the time of the Prophet صلى الله عليه واله when he established his Islamic government in مدينة المنورة we see that the Prophet صلى الله عليه واله he called for many virtues to be implemented in the Muslim society and community for example the brotherhood within the Muslims that the Muslim is the brother of the مسلم so they won't cheat they won't fight they won't hurt each other they are brothers the freedom the freedom in speech the freedom and activities the freedom of being able to be free in that geographic location without being chained with man-made laws and in which will have the the consequences on they use themselves and their society and of course other virtues such as for example known violence to be peaceful not only with your own community but with all other creeds and beliefs with all religions to be peaceful to respect all other let's say if you are a majority in your respect the minorities for example so these teachings in which were established in the time of the Prophet صلى الله عليه واله I should be also implemented today in our own communities and societies more and more and we try to implement the true عقيدة and implement in our hearts and in practice as well and of course to stick with the حكام الله صلى الله عليه واله in this case of course the one may be able to preserve these virtues and spread them to the next coming generations عقيدة شخنة what are because this is really important that we know we educate our youth because one thing is that you know they're Muslims we want to develop as you know virtuous pious people but at the same time when they reach the age of بلو when they reach adolescents a lot of things become واجب upon our youth now what sort of things what sort of I think the way we say it in the حوزة was تقليف is upon the youth once they reach the age of adolescents you see there are two main things that the say it mentions in his رسالة بوك with regard to educating and teaching the youth with regard to the شريع اللوز initially is to to fulfill the obligatory acts in which الله صمع تعالى has made them oblige the movement to implement them and practice them and they are for example daily prayers fasting paying زكات حاجة and so forth these احكام the minutes or the day in which that young boy or girl becomes بالغ and reach the age of puberty it becomes واجب in line with the adults they have reached now the age in which the احكام will be on them as it is واجب on their parents and their older sisters and brothers so initially they have to practice this احكام واجب of course we shouldn't be harsh on them gradually slowly slowly and that's again we have to start from the early age so we teach the kids the صلاة to stand next to us to pray let's say in the age of 304 just how can we say مك صلاة or practice the صلاة next to us you don't have to ask them to do the وضو as well because they're too young but when they reach the age for example of 6 7 8 gradually start to tell them to do وضو teach them how to do وضو go in the bathroom teach them yourself to see you how you do the وضو sequentially and then they do it themselves and eventually they start to learn the صورة حمد and the small صورة how to do the وضو and one step by step in this case you have fulfilled the first stage of learning and practicing the واجبات from such early age the second recommendation by the said is that to teach them to abstain from all حرام acts so you teach them what's حرام let's say you teach them that for example alcohol is حرام to drink but you can drink juice instead for example it's حرام to eat meat which was not slaughtered by the means of Islamic method by the قبلة by the Muslim and so forth it is حرام to eat pork meat for example it is حرام for example to eat fish with no scales for example except the puns as exception and so forth so we tried to teach them these حقام the حرام so they can have this immunity and protection from falling into the acts of حقام when they reach the age of puberty and youth because in that time it would be very difficult imagine if somebody let their kids to listen to music to songs when they reach the age of puberty and you can't turn them stop listening what were you for the past six seven years you let them off to listen to music you know day and night and and join music clubs for example dance clubs parties with their friends now you want to prevent them it's very difficult so you start from the early ages that listen this is حرام because of one two three you give them the reasons that they can understand you know because their mentality are different than the adults mentality and gradually gradually they build up with this notion and with this عقيدة and belief and strong understanding that well this is the right thing we are doing that we don't eat the حرام we don't do the حرام we don't join the حرام and so forth so when they are in the age of بلوغ and youth they are strong they know what and how to implement these حقام with the true belief and understanding and with the acceptance that's important that I internally accept and believe in these حقام and these rules to understand that I think it's very very important and you know what the things you highlighted especially that you know from from a young age that Muslims are rehearsing how to practice their religion abstaining from حرام so they don't develop any bad habits and in شاء الله and develop good habits and you know to pray on time and also how to do well and then to recite Quran and so forth thank you very much and thank you to all our viewers for joining us if you have a question that you'd like to propose to the Sheikh you can email us on حقام sos at Imam Hussein dot TV إن شاء الله we'll be able to address it join us the next time for another episode where we'll be discussing topic on youth إن شاء الله سلام عليكم رحمة الله وبركاته


Hello, everyone. Welcome to our session, Sharing is Caring, GPU Sharing and CDI and Device Plugins. My name is David Porter. I'm from Google. I work on the Google Kubernetes Engine team, where I work on Node and kind of our integration with accelerators. And this is Chris. Hey, I'm Chris Destiniotis. I'm a software engineer at NVIDIA. I work on our cloud native engineering team that works on enabling GPUs in the container runtime ecosystem and in Kubernetes. Cool. So to get started, let me just kind of set the landscape. So the landscape is everyone is well aware during this KubeCon. Devices are becoming increasingly important in Kubernetes, right? So we have all these new workloads, things like inference, training, fine-tuning, et cetera. And all of them are requiring devices. So it's important that devices and accelerators are kind of well integrated in Kubernetes, just like other resources are today. So what we're going to cover today, we're going to start by talking a little bit about how devices are integrated in Kubernetes today. Then we're going to kind of go one level lower at the container runtime level and take a look at how container runtimes integrate with devices and how CDI is a new technology that's going to help enable that. Then we're going to go up the stack a little bit and look at kind of resource management with some of these devices. And we're going to focus on GPUs and how GPU sharing can be enabled. And lastly, we're going to end with a little bit about the future of devices and device sharing in Kubernetes. So I think when we think about it kind of from the Kubernetes perspective, right? I think one of the big reasons Kubernetes is so successful and everybody really loves running Kubernetes is because it's really good at resource management. What Kubernetes allows you to do is take your whole fleet of nodes, right, with all different types of resources like CPU, memory, disk space, and really allow you to be able to manage them under one control plane with one API. And devices are now becoming increasingly important in that resource management space. And we want to enable you to manage them just as effectively as you do with CPU, memory, and disk today. So what we're trying to do from the community perspective is we're trying to build open standards here and common resource models, both to actually consume and interact with these devices and to share the resources. So we're going to be talking about what exists today. And what exists today is the device plugin framework in Kubernetes and CDI. Those are the open standards and frameworks that allow you to interact with devices today. In the future, and something that's been discussed during this KubeCon has been a new feature called DRA. And DRA is going to be a new API and new way to manage these resources and share them more natively. So let's talk a little bit about devices in Kubernetes. What is a device when we talk about devices here? So a device, we'd like to think about it as this abstract concept, an abstract resource that the user wants to use for some specific purpose. And it may be that it's a physical hardware device, but it may not be. And so usually, these devices, they don't just come in isolation. There's kind of a whole ecosystem around them. So usually, if it's a hardware device, you'll actually need some kind of kernel drivers. You'll need to access some device nodes to actually talk to that device. And probably, you'll need some libraries and utilities as well to actually be able to use that device in your application. So this whole bundle of things is usually what we call the device. And so in Kubernetes today, we have this API called the Extended Resources API. And Extended Resources API allows in Kubernetes to basically advertise resources under the node, just like CPU and memory. And we can advertise these devices and account. So the devices are accountable and have an integer account associated with them. The way that devices are integrated in Kubernetes is with something called the Device Plugin Framework. And the Device Plugin Framework is a plug-in framework that the vendor of the device usually writes. And so the Device Plugin Framework usually has three calls here. It has a register call, an allocate call, and kind of health checking. So usually, when the device starts up, it actually does some registration. And what this means is that the device will start up, and it'll actually talk to Kubelet, and it'll advertise a name. So for example, for GPUs, the name is invidia.com slash GPU, and it'll advertise account, how many of that resource we have. Then when a workload comes in, there's the allocation step. The allocation step is when we actually figure out what device to allocate for that workload and how to modify the container so that the workload can actually access the device. So things that are like mounting the device nodes, the libraries, et cetera. And lastly, the device plug-ins responsible for health checking. So if the device goes unhealthy, the device plug-in will react to that and update the node about that. So let's walk through kind of an example of how device plug-ins actually work in Kubernetes today. So let's start off here with a node. Let's imagine we provisioned a node either on-prem or in a cloud provider. You already have the GPUs attached to it. The first thing you would want to do is actually set up the kernel GPU drivers here. So on a cloud provider, this is managed for you. There's also the invidia GPU operator that installs these drivers for you. So you set that up. You install it, and then you install the device plug-in, which again can be managed by a call-up provider or the operator. And the device plug-in starts up, and the device plug-in is usually just a pod that's running there. So the first thing a device plug-in does is actually communicate with the GPU, communicate with the GPU drivers, and figure out how many of those GPUs there are. From then, the kubla is going to go ahead and basically see that there's a new device plug-in and ask to do the registration. So during the registration, the device plug-in is basically going to say, I have a resource. That resource name is invidia.com.gpu, like that's the name. And I have two of them. And that's going to be communicated back to kubla. And when kubla sees that, it's going to go actually up to the API server on the control plane. And already, the kubla is responsible to update the node object capacity field. And it'll update the capacity field. And just in addition to your CPU and memory, you'll have a new entry there, invidia.com.gpu, too. So at this point, all the components in the control plane can see that this node has this resource attached. So at some later time, you have a workload that comes in. And this is on the left side of the diagram here. You have a pod come in. And under the container request there, it'll request some number of that device. So in this example, it's requesting invidia.com.gpu1. It'll come in. And then it'll go up to the scheduler. And the scheduler is going to see, hey, this node has that resource available. It's going to go ahead and do that scheduling. And so after it does that scheduling, the kubla is going to see that, hey, it needs to start a new pod. At this point, kubla is going to communicate to the device plugin and actually do that allocation step I mentioned earlier. So it's going to figure out, hey, what GPU do I give it? And how do I make that GPU accessible in the container? What modifications need to be made to the container to access that GPU? Once that's done, kubla goes ahead and goes to the container runtime. So this is like container D or cryo and actually send that container spec there for it to start. Container D or cryo just pass it along to the low-level container runtime like Run C. Run C goes ahead and actually starts the container, does all the mounting and so forth. And at that point, the workload can start. And now the workload can actually access the device like the GPU. And at this point, the workload is directly talking to the GPU. It's not talking to any of these Kubernetes components like the device plugin when it's actually running. However, with some devices, the story is not so straightforward and so simple. And so Chris here is going to talk a little bit around some of the extra complexities and how CDI is going to help address them. Thanks, David. So yeah, so for more complex devices, there's actually more that comes into the picture than what was shown on the previous slide. So for NVIDIA GPUs, you not only need to install a driver and device plugin, but you also need something called NVIDIA Container Toolkit, which is a set of tools that ensure that GPU containers can be set up with all the right things they need to access to GPU. So this diagram on this slide is just like a sequence diagram from the point when Cubelet allocates a GPU, one or more GPUs for your container to all the way down to when your actual container is run. This is sort of what it looks like today. And historically, there's some NVIDIA specific components that under the hood make sure that your container has access to all of the right device nodes, driver libraries, and so forth, so that your CUDA application can run transparently. The major sort of component that's doing all the heavy lifting is step nine, this pre-start hook. So it's this library that's invoked as a pre-start hook before your container application. The main process starts. It is a lot of injecting of device nodes, mounting on the driver libraries and everything you need. And it's doing this sort of behind the knowledge of a container runtime like Run C. So there's some downsides here. One is this is not declarative. Like all of the edits to the container's environment are being happened under the hood, and they're not encapsulated in the container's spec. So the OCA runtime spec is the standard for that Run C uses to set up a container's environment. Run C has no idea that some of these things are actually being included in the container. So historically, that's led to some hard to debug issues and some inconsistencies when interacting with GPU containers. So ideally, we want to provide a standard to sort of overcome this and do things in a declarative way. And so there's a project called CDI which aims to solve these problems and standardize how we access GPUs and other accelerators and hardware in containers. So what is CDI? It's a container device interface. It's a CNCF-sponsored project under Tag Run Time. And like I said, previously, it aims to standardize how containers or how third-party devices are made available to containers. The way it does this is it uses a declarative specification. And it defines in that specification, you define as a vendor, you define what your device actually means, what access in the device means. That can be a list of device nodes, mounts, environment variables, and even container runtime or even container lifecycle hooks can be included. And each one of these maps to a set of modifications that need to be made to a container spec. So everything is sort of encapsulated declaratively, and it's encoded in the container spec. What happens is you request a CDI device and a container runtime which actually understands CDI can take that device, read the spec for it, and modify the container spec, the OCI runtime spec, so that your container has access to the device. So it's all done declaratively. The way that we name devices, there's a taxonomy that CDI uses for naming devices. It's a vendor slash class equals name. So for an example, for NVIDIA GPUs, it's a nvidia.com slash GPU equals and then some ID. We can use arbitrary naming. So like I said before, this is a declarative approach. And some of the main benefits are that low-level runtimes like RunC are doing the heavy lifting for us. They RunC is specialized in setting up the container's runtime environment. So we are leveraging it to actually provide access to accelerators and devices. And having support in container D in cryo means that we no longer need some of the vendor-specific tooling to make this all work under the surface. So here's an example of what a CDI spec looks like for an NVIDIA GPU. So this is a system where you have only a single GPU. So under the devices section, we just have one entry called GPU0. And we have a series of container edits. So these are edits that need to be made to your container spec so that you can get access to GPU0. This is device nodes, like the NVIDIA character device. Mounts. So like driver libraries, like libcuda, it specifies where in the host that library is and where it should be mounted in the container at what path. As well as container lifecycle hooks. So a simple one is updating the LD cache. So this is a standard hook that we like to use so that your main process automatically can discover the driver libraries when it runs. So that's a brief introduction to what CDI is and what the goals of the project are and some of the benefits. Where it can use it today. So it's actually supported in Kubernetes so that a CDI devices field was added to the device plugin API so that device plugins, when they provide an allocate response, instead of providing other information, they can provide CDI device names in that response field and that response struct. The CRI was also extended to contain a CDI device field and the container config message starting in v0.27.0. And like I mentioned, I think earlier, container dncryo already supports CDI so they can understand the specification and modify a container spec for us. So this has all been discussed in the context of Kubernetes device plugins, but CDI is actually a standard that's used outside and is meaningful outside of the device plugin use case. So for DRA, CDI is sort of the basis of defining and requesting resources. It's also useful outside of Kubernetes. So if you're launching containers interactively with Docker or Podman, you can use CDI there. And actually for NVIDIA GPUs with Podman, CDI is actually the recommended way today to access GPUs. For HPC containers, there's been some support for CDI in Singularity, which is a popular HPC runtime. And we can think of even other use cases. So even for networking devices, we can imagine a future of using CDI for that as well. So going back to my sequence diagram from a few slides, this is how I would look with CDI if our device plugin was leveraging CDI. You'll notice that the container runtime steps, there are no NVIDIA-specific components. The assumption here is that before running applications, we have generated a CDI spec for all of the GPUs on the system. So either some tooling is invoked or the device plugin itself can generate these specs that describe all the GPUs in the system and what it means to access those. So those are stored somewhere as a file on the system, a standard path that Runtimes know to look for. So that's done beforehand. But at runtime, what happens is container to your cryo, right, steps five through seven, it will read the CDI spec file. For all the requested GPUs for the container, it will update the container specs so that it has the right bit so that it can access that GPU. If for is that container spec to run C, run C creates your container with that specification and your container can run and has access to the devices that were requested. So this is all declarative. There's no vendor-specific, so actually this is a diagram that can be applicable for not just NVIDIA GPUs, but any other accelerator or device. Portable across container Runtimes and we're not using these vendor-specific hooks that have a lot of drawbacks. Okay, so we went into detail about device plugins and sort of the lower level details about container Runtimes and some standards that are being developed. I think we're gonna switch gears. I'm gonna go back to Dave and we're gonna talk about how can we leverage device plugins to more effectively share GPUs. Thanks, Chris. So let's jump gears a little bit more and go one level up. And so with these devices, it's becoming increasingly important to find ways to kind of maximize their utilization, right? So especially with GPUs, they're very expensive, they're limited and everybody right now is trying to figure out how we can squeeze kind of the most from them, right? So to do that, what we're really trying to do is to find ways that we can take these physical devices that we already have and be able to break them up and partition them so we can actually run multiple workloads simultaneously on them. And so we're gonna be looking specifically at GPU and the ways that we can share GPUs today. And so there's three main ways that you can do that. The first one is called time slicing. The second one is called multi-instance GPU MIG. And the third one, which is kind of a newer one, is called CUDA multi-process service MPS. So we'll go into each detail, into each one, how it works and kind of the trade-offs of when you might want to use which approach. So we'll start with the simplest one, which is time slicing. So normally, right, we just have a single workload that's running on the GPU, just a one-to-one mapping, really simple. With time slicing, what we can do is we can, first of all, define how many workloads we wanna actually be able to run on the GPU at a time. So you can see at the bottom of that command line, you can see there's a parameter that are max shared clients per GPU is 10. So in this case, we're basically saying we're allowing 10 workloads to run on the GPU at that time. And so with time slicing, how it works is we basically have context switching. So a single workload runs, then there's a context switch and then the second workload runs and then it switches back. And so that context switch is a little bit expensive, but it allows us to actually run multiple workloads at a time. And so the way it actually works from the device plugin perspective is instead of actually advertising before the physical number of devices, which was before what nvidia.com GPU represented, now we're kinda advertising the virtual number of devices, so the clients that can be used. The other kinda thing about time slicing that's worth to mention, each workload gets its own address space, but there's no memory limits enforced. So a workload can consume all the memory and that can disrupt other workloads. The other thing that worked to mention here is that in time slicing, each workload actually only runs at a single time, right? There's only one workload that's running. So there's concurrency here, but no parallelism. At a single time, there's only one workload that's actually running on the GPU. So that's available today and you can use time slicing. And then also with the nvidia GPU device plugin, the way to enable time slicing, it's pretty similar. You specify how many clients you wanna use the GPU. Here it's 10. And now under the nvidia.com slash GPU capacity that's reported instead of reporting one GPU and now reporting 10 GPUs. So kind of the meaning here has changed from the physical number of GPUs to the clients that can use the GPU. So when is time slicing useful? So one scenario where I think it can be very useful is kind of for interactive workloads, kind of more exploratory workloads. And so sometimes these workloads like Jupyter notebooks, if you're familiar, where you can kind of do data science experiments and kind of machine learning, kind of explorations are great candidates because they're kind of interactive. Sometimes they need a lot of resources, sometimes they don't. You probably, they're mostly for exploration so we don't have super strong kind of guarantees that we need to provide. So let's take a quick look at actually using time slicing in action. So what we're gonna do here is we're first gonna provision a cluster. So we provision a cluster here on GKE on 129. After that, I'm gonna set up a node pool, which is a set of nodes, and I'm gonna configure those nodes to be in time sharing mode. So I'm gonna specify the GPU time sharing strategy and I'm gonna say that I have three GPUs that I wanna do time sharing with. So I go ahead and create that node pool. It's gonna create the capacity in the node. I'm just creating one node here kind of for demonstration purposes. After that's there, I'm gonna use KubeCuttle and actually see like how many nodes I have. So I'm running KubeCuttle get nodes. I have two nodes available here. One of them is just a CPU node and then one is actually the one that we just provisioned with the GPUs. So I'm just gonna set an environment variable GPU node just so we can kind of play around with it. And the first thing I'm gonna do is I'm gonna look at the allocatable that's being reported from the API server. And you can see here, we have nvidia.comGP reported and we have three GPUs that are reported. So this node only has one GPU. I provisioned an A100 GPU on it. So it only has one physical GPU but due to the time sharing configuration, it's actually reporting that we have three GPUs here. So let's go ahead and we're gonna try running a workload here. So the first thing I'm gonna do is I'm gonna label the node with notebook node true and I'm gonna just use that as the node selector for the pods that I'm gonna deploy. So for the purposes of the demo, you can imagine here we're gonna deploy two Jupyter notebooks and you can imagine maybe there are like two different data scientists or researchers who are wanting to play around and use these notebooks. So I created two pods here, notebook pod one and in the node selector here, you can see I'm specifying I want an A100 GPU. I'm gonna use time sharing. This is the one where I set up the three clients and from the resource perspective, I'm just specifying I need one GPU here and then for the actual workload, I'm just specifying like the Jupyter notebook to start up. And so that's my first pod and then I have a second pod here which is basically the same thing, just a different pod that we can play around with the only thing I'm running it under different ports we can kind of explore it differently. So what we're gonna do, we're gonna deploy both of those pods here, notebook pod one and notebook pod two. Cool, and then we're gonna look at those pods started up, they're running, awesome. So now let's try to access those notebooks. So those notebooks, I'm gonna use kubectl port forward and I'm gonna port forward them on two different ports. And so I'm gonna start up the first, I'm gonna go to that port forward and look at the Jupyter notebook, you can see it started here on port eight eight eight and the other one I port forwarded on eight eight nine and you can see it's also running here. So awesome, they're both running but they're using time sharing but we're not actually using anything on the GPU yet. So let's actually run some GPU workload to try out the time sharing. So what I'm gonna do here, I'm gonna install some kind of libraries and then I'm gonna first of all use PyTorch and I'm gonna access the GPUs from PyTorch and the PyTorch is able to access the GPUs. I'm just printing the GPU kind of model here. So now let's try to actually run a workload. So I'm gonna use HuggingFace, so I just gonna log in with the HuggingFace credentials so I can download a model and I'm gonna actually try doing the Gemma kind of LLM model. So I'm using the open source Gemma model which is like the small two billion model here and I'm gonna try to do inference on it. So I'm gonna type in Kubernetes is the best and try to run inference on that model. So I'm gonna run it, now it's actually loading the model to the GPU and you can see here it's gonna load it here and we can see the LLM output at the bottom. Kubernetes is the best way to deploy and manage containerized applications. Cool, so it makes sense. So now you can imagine some later time like some other user wants to use that other notebook, right? So we're gonna switch over to the other notebook, different user and let's say they wanna run a different workload, maybe they wanna try a different model here. So very similar, I'm just gonna try the same thing but this user they're gonna try a different model. They're gonna try the larger Gemma seven billion parameter model. So same type of thing, different model though and we're gonna try a different input text, type in Kubernetes is great and see what the response is. It says Kubernetes is great and it's hard, it's hard to get started. So hopefully this talk helps a little bit with that. So there we go, we can see we're running both of those things. If we look at the metrics here that are reported, we can look at actually the metrics that are reported for those GPUs. We can see the VRAM usage here. You can see it spiked to around 20 gigs and then it jumped to 40. So when the first notebook user started playing around with the first model it allocated all that VRAM and then the second one started allocating more memory for the second model and it jumped up to 40. And just to verify that we can actually log into the node and I'm gonna just SSH into the node. I'm gonna run NVIDIA SMI and you can actually see here from NVIDIA SMI it shows like both Python processes running there and both allocated that amount of GPU memory which corresponds kind of to the metrics we saw earlier. So there you go, that's kind of like a demo of how you might use time slicing. So you can try that out today. So I'm gonna hand it back to Chris to talk a little bit about some of the other GPU sharing strategies. Cool. Yeah, thanks for the demo, David. That was really great. There we go. So there are two other sharing strategies we wanna cover in this talk. So the next one is CUDA MPS. This allows, this takes a next step above time slicing in the sense that it allows you to logically partition the GPU in terms of memory and compute. So it's all done in software, just like time slicing. So you can have multiple clients to the GPU using at the same time concurrently but also it sort of allows you to do some level of parallelism. So you can have clients A, B and C running GPU kernels on the same GPU at the same time and it's all sort of facilitated by MPS service or server process that's running. And so it sort of sits in between your actual clients and instructions running the GPU. So in software it can sort of enforce some sort of memory limits or enforce memory limits for each GPU client and compute resources in terms of active thread percentage. So it can lead to better utilization of your GPU in terms when compared with time slicing. And the one implementation detail is here is that we have one server that facilitates the GPU clients and running instructions in the GPU. So there's one shared CUDA context that's good in terms of context switching because it's not as much of an overhead between different clients running the same GPU as one compared with time slicing where you are doing a full context switch between two different CUDA contexts. But at the same time, in terms of like fault domains, like this is a shared fault domain. So if client A triggers some sort of fatal error on the GPU, it will affect clients B and C. In the NVIDIA GPU device plugin, you can configure MPS like this. It's almost very similar to the time slicing configuration. You just specify MPS instead of time slicing as a sharing strategy and you configure how many replicas. So how many concurrent processes you want to support running at the same time on the GPU. And so if we describe the node, we'll see similar to with time sharing, your one physical GPU is now advertised as 10, as being 10. In terms of support, so we have a new release of the device plugin coming out very soon, the 0.15.0 release, and that will have official support for MPS. We have some release candidates out already that people are trying out, trying MPS out with. The other sharing strategy is MIG. So multi-instance GPU, right? It's a hardware feature of some of the later generation NVIDIA GPUs that allows you to take a full GPU and some divide it into multiple, what we call GPU instances, right? And each of these has dedicated compute and memory resources. So you can have up to seven of these slices on your GPU and there's some standard like profile names that if you look at the MIG documentation, you can sort of partition your GPU into those fixed size slices. So on the bottom, we have an example of enabling MIG on GKE. So on your node pool with A100s, you can specify the partition size of your MIG instances, in this case, one G5GB, which is the smallest slice. So we can have seven of these, each with one sort of dedicated compute instance and five gigabytes of GPU memory. So how do we enable MIG in the device plugin? So the assumption here is that an admin or some automation has gone in and already enabled MIG on your GPU on the node and has already sliced the GPU into MIG instances. So that has to be done beforehand. So there is some static sort of configuration that has to happen. You have to sort of know what size profiles you want and configure the GPU in that way. The configuration of the device plugin is really simple. The device plugin will actually enumerate all of your MIG instances. And so if you have seven of the small instance types, then your node will have seven mba.com slash GPU resources allocatable. Instead of configuring MIG yourself, you can use a GPU operator. We have a component called a MIG manager that helps automate the enabling of MIG and configuring of MIG instances.  So now that you've kind of heard around those different resource sharing strategies, you might be asking yourself, when do I know which one to use and what are the trade-offs between them? So I guess to start off, the simplest one is the one we started with, which is time slicing. Time slicing, the big benefit of it, it's very dynamic. So you can add workloads on the fly, you can remove workloads on the fly, you can start out with just one workload and then it's identical to just not using resource sharing and as you add more workloads, you can kind of get more workloads to share that GPU. The negative with time slicing is you can start to introduce some kind of jitter, et cetera, because of that context switching overhead and you don't have strong kind of resource management guarantees because there are no memory limits that are available. So that's kind of what brings us to MPS. And MPS can be thought of as kind of an improvement over time slicing because you actually do get stronger guarantees there, right? You do get performance isolation, you can specify how much compute each application gets and you can specify memory limits so you can actually enforce memory limits per application. And then MIG is actually the one that gives you kind of the strongest guarantees because it actually partitions things at the hardware level, right? So that means that at the hardware level you have separate kind of memory that you can provide and you have separate kind of performance guarantees because it's actually split at hardware. The kind of the negative with MIG and kind of the downside is that you actually need to figure out how to partition your GPU kind of ahead of time, right? And it's not dynamic. So when there's no workloads running you have to kind of go in and understand, okay, depending on my workload size which MIG partition sizes make sense for my application. It works well, however, for example, for multi-tenant scenarios, right? Where you really do need strong kind of security and other types of isolation guarantees between workloads. So in summary, I think what we talked about today, right? Is we started at the Kubernetes device plugin framework and that's the existing framework that exists in the ecosystem to expose and expose devices like GPUs, TPUs, and FPGAs. And we also dove one level kind of lower at the container runtime level and gave you some information about CDI which is the new standard of how a device will be integrated at the container runtime level. Then we looked at GPU sharing and GPU sharing is a great way to improve utilization of your resources and we have different strategies there with their different trade-offs. And I think the big message we're trying to send is like as a community what we're trying to do is extend this resource model, make it really vendor agnostic, make it standard and make it kind of natively supported by Kubernetes with these devices. And so what we're doing in the future is we're trying to extend this model here, right? So this resource sharing, it's not natively supported and it's not very declarative because we had to kind of overload the meaning of what does the device mean depending on the resource sharing strategy. So we're trying to come up with new APIs to make it more declarative. And so in DRA, we're hoping that we can be able to express kind of some of these resource sharing mechanisms kind of more natively. And so it'd be really great to get your feedback around kind of some of what are kind of the challenges today with the device plugin and CDI and what you would like in the future in Kubernetes. Your feedback is really appreciated there. And so a couple of things I wanna shout out is there's some related talks both of this KubeCon that have already happened and in past KubeCons as well that go in more detail around devices in Kubernetes as well as GPU sharing which are some great resources there. So thank you so much. Any questions? Cool, I think there's Mike. Can we expect in some future that you provide the memory limit or so for time slicing? It's technically possible but it's not implemented in the driver. Yeah, I don't think that's currently planned. I think what some folks do is they actually enforce those memory limits at the application level. So a lot of applications, you can actually enforce memory limits at the application level but I don't think there's currently plans to actually support that natively with time slicing and the recommendation would be to use MPS. Yeah, thank you. Thank you for the talk. So my question would be can you combine MPS and time slicing either today or tomorrow? So saying that I wanna use MPS until the GPU is full basically and then start time slicing for additional workloads. I guess theoretically you could but in practice I don't think we've investigated or documented how to do that for our cloud native users. Okay, so it's theoretically possible today or? I think so. Thank you. You can also combine for example like MIG and time slicing as well, right? So you can do time slicing on MIG partitions as well for example. Yeah, you would need like multiple MPS server processes, right, and then they would time slice. Yeah, I think it's theoretically possible, yeah. So the example you gave, I guess it's now fractionalization is becoming a standard. Can I use those features in the new operator without DRA? I mean, this is enabling DRA but I can still do these things now, right? Yeah, so everything we showed is not using DRA. This is using the device plugin API. So all the sharing strategies we showed you are available and supported in the operator and the components that we deploy. And the new operator is about to be released with support for, I think you mentioned earlier that it's a new version of that operator. Yeah, so we're releasing a new version of the device plugin and then also the operator will pick that up next month sometime, hopefully, to support MPS. Okay, so if we all start learning to use this way of allocating fractions, then in a year's time or so the DRA will be the new way instead. So DRA will provide potentially like a new API to actually consume these in your workload so your pod spec may change but the underlying actually technology of those different research strategies that's not specific to DRA. DRA is just kind of the declarative API representation of how we wanna consume those. Okay, yeah, thanks. Thanks for the presentation. So my question is, is this possible to share something about how GKE supports time slicing? Is the mechanism, is this possible? How it's implemented or asked? How GKE supports time slicing? Yeah, so the way GKE supports time slicing is kind of the regular way, it's the same as with the NVIDIA GPU device plugin. So you just specify how many clients can use the GPU at a time, right? And then it'll basically just schedule those workloads there. And the device plugin will advertise the number of clients that you specify. So there's nothing kind of special done on GKE versus the NVIDIA device plugin. The underlying mechanism is kind of the same. Okay, thank you. Hi. Great talk. At the moment, we use time slicing for a combination of GL workloads and CUDA workloads. Is that what we can also expect from MPS to... I think it should work transparently, yeah. It's just a different way of sharing the GPU. I don't think the type of application you run necessarily matters. Okay, yeah, because you mentioned the contacts and that sounded very CUDA specific or... I actually don't know if there's implications there using the same CUDA contacts. I actually don't know. Okay, yeah. My understanding, the goal is for it to be compatible like for most workloads out of the box, right? So I think the expectation you shouldn't need to change your workloads significantly to pick up MPS from time slicing, yeah. And the GL workloads, will they remain to be supported? Because we're noticing a bit that the support for the container-based images, for example, for the GL has been lacking for the last few years. So can we expect that to remain supported by the cloud run times? Are you asking about the Docker? Like what specifically? Yeah, exactly, yeah, yeah. I'm not fully aware, honestly, around the support there. Yeah, me either, unfortunately, yeah. We'll hold you to check. Thanks. Hello, thank you very much for the great talk. One question would be for time slicing. Is there, let's say, a soft limit depending on the cluster size and how many compute nodes you have where it, like, tanks the performance in a way? Yeah, so for time slicing, it's all kind of based on a single node, right? So the number of nodes in your cluster and so forth, I don't think are actually important factors. It's the number of workloads that are running concurrently, right, on a single node. So I think the main factor is how many GPUs do you actually physically have on that node and how many clients do you have connected to it, right? The more clients are gonna have the less runtime each workload can have. So you can't use time slicing together with RDMA between, like, more nodes. RDMA connected nodes to, like, have a bigger pool of time slice GPUs, right? I mean, I don't see why you couldn't use time slicing with other technology, right? When the workload is actually running on the GPU, it's the only thing running, right? So you could use RDMA or any other techniques you're using today. Thank you very much. Yeah, thanks for the talk. I have one question for time slicing. It's currently the only solution that might have problems with workers fighting for GPU consumption. So it is more agile than all the others. Is there a way currently to natively achieve process prioritization? So you can say, okay, these processes here are coming first and then these are going to wait, or is this not currently possible? Yeah, I don't think that's possible today. I think you can customize, like, the time quota that's given for each application when it's time slice. So I think there's a configuration option in NVIDIA, somebody can say, like, short for the context window, but I don't think there's prioritization built in there. Because at a single moment, right, there's only one application that's running, and they all get the same share. Okay, thank you. All right, if there are no other questions, thank you, everyone. Yeah, thank you.