Today, I'm going to be talking about an overview of scheduling in the FreeBSD kernel. So we're going to look at the scheduler in the FreeBSD kernel. We're going to dive down into how it works, in particular some of the more recent changes that have been made in order to work with the ever larger number of core systems on which we're running. Scheduling is how we decide when and where and for how long to run all the threads that are in the system. There's a bunch of threads that are in the operating system. There's threads that are in the applications that are running. A particular process may have a single thread or it may have a whole set of threads that are running within it. So when it comes to scheduling, we are looking at all of the threads that I've just described. These threads get really divided into five different classes. We have the i thread, which is all the things running in the bottom half of the kernel. And for the most part, those are all the interrupts that are coming in, all the asynchronous activities. So disk interrupts, network interrupts, timer interrupts. Anything that is delivering interrupts and requires a thread to run in order to process those interrupts. We have the current threads, which are all of the threads that are running in the top half of the kernel. And the top half of the kernel is all the synchronous things that the kernel is doing. For the most part, it is being driven by system calls that are coming in from the applications. So you do a read or a write or an open or a close. That system call comes into the kernel. And while it's running in the top half of the kernel, it will get one of these current level priorities. And those tend to be higher than the priorities of processes that are running in user space. Because we want to get them through the kernel and out of the kernel so they're not holding locks that can potentially compete with other threads that are trying to get access to the data structures that they're looking at. Between these two, we have the real-time priorities. And the real-time priorities are for user processes that are real-time processes. And on the next slide, we're going to dig in a little deeper on how these priorities get set. User processes, for the most part, run in this timeshare range. And this is the area, the set of priorities that we mostly focus on when we're talking about schedulers. Because it's the set of priorities that the kernel is devising for each of the processes that's running in the system. And then finally, at the very lowest set of priorities here, we have what are called the idle priorities. And these are priorities that are for very much background tasks. They'd be like a screensaver or some other sort of activity that will not run if there's anything else that wants to go on within the system. So you can see the numeric priorities here run from 0 to 255. Higher values of priority imply lower levels of service. That is to say, 0 is the highest priority and 255 is the lowest priority. The i-thread and Kern classes are managed by the kernel. The real-time priorities and the idle priorities are managed by user processes. And so the user processes just set them. The kernel will then just work with whatever they're set to. And then finally, we have the time share class. And this is the management of the priorities that are shared between the kernel and user processes. So user processes can influence these. For example, you can raise or lower your nice value, which will bias them up or down. But the kernel is actually selecting what that base level priority ought to be. Let's look at each of these different types of priorities in turn. For the kernel priorities, the kernel is pretty much in charge of those. The system administrator can actually fiddle with some of those priorities if they want to. So typically, the interrupt threads are, the highest priority is given to things that need the lowest latency. And also that have high frequency. So things like network packets tend to get a high priority. Whereas things like terminals, if you have a serial line coming in, that tends to get a low priority. But for example, in the old days we used to use serial lines to actually do networking. We had a thing called serial line IP. And in that case, it was characters were coming in much faster than a user would typically type them. And so it was important that that artificially be given a higher priority so that you wouldn't lose the characters. Okay, but for the most part, the kernel priorities, we don't make any changes to. So that really gives us three sets of areas that we can work with. The first of these is real time. And here, the processes are setting the specific priority and the kernel doesn't second guess them. It doesn't like raise or lower them or change them. It just runs with that. And the effect of this is that if you set a real time priority and that process then goes into an infinite loop and never gives up the CPU, then the system will effectively look like it's locked up. Because the only thing that the kernel will be able to do is the interrupt threads. But none of the top half of the kernel will ever run. None of the user level will ever run. So if you have a shell and you're trying to fix something, you're out of luck. So you have to be very careful if you're doing real time priorities that you don't let yourself get into an infinite loop. That you do give up the processor periodically so that other things can happen. Then next to here are the schedulers that are available. The primary schedule that we use today is the interactive scheduler, the so-called ULE. And then we still have the traditional scheduler that's so called 4BSD scheduler. The 4BSD scheduler was actually written in 1978 by Bill Joy and myself as a stop-gap measure until we had time to do something right. And it actually lasted well into the OOs. ULE first came in FreeBSD version five. And it was originally supposed to be the scheduler that would be the default scheduler, but due to various problems that it had, it wasn't really until FreeBSD 8 that it could be made the default scheduler for the system. And 4BSD is still there. It's a very simple scheduler. It doesn't have a lot of the, as you'll see, the abilities that ULE has. But it's still useful for certain embedded applications, where you just have a small processor, a small number of processes running. And you don't need all of the bells and whistles of ULE. You just need something simple and fast. So this is still there. I'm going to not really talk much about it. In this lecture, I'm going to focus on ULE because that is a default scheduler and it's the one that has the capabilities that we need on modern multi-core systems. So ULE, as we will see, deals with things like processor affinity. And it actively calculates an interactivity score so that it can figure out which processes are batched, sort of long-running computing processes and which ones users are interacting with so that it can give better priority to those interactive processes, better response time. Finally, the idle scheduler, much like the real-time scheduler, the priorities are simply set by the system administrator. The kernel doesn't mouse around with them. It simply follows them. And but unlike real-time, if one of these goes into an infinite loop, it's not a problem because if there's anything else in the system that wants to run, they will preempt it. And so the idle priorities really are not going to get you into any trouble. Now, before we start diving down into the details of this, I want to talk about how scheduling is really divided into two different parts. There's what I call the low-level scheduler and the high-level scheduler. And the low-level scheduler is the scheduling that runs when we do a context switch. And on a modern processor that's busy, context switching is something that occurs tens of thousands or even hundreds of thousands of times a second. And so we need to have a very short time to figure out what we're going to run next. If the current process is blocked and says I don't need to run now, do something else, we do not want to spend a lot of time trying to figure out what the next process ought to be. And so the low-level scheduler, as we'll see, is just a set of priority queues. And the low-level scheduler just finds the highest priority thing to run and runs it. And then, as we'll see, that's a very simple process. It's like find a queue that's not empty, pick the first thing off the list, run it. The higher-level scheduler is the part that is figuring out where those different things should be, what their priorities ought to be. Now, of course, in the case of real time, those decisions are being made actually out in user application land. The user application is figuring out what the priorities of each of its threads ought to be. And that's presumably something that it reevaluates periodically and can change those priorities. For the ULE or for BSD, these decisions are being made inside the kernel itself. And as we will see, this is done much less frequently. So, for example, for BSD, once a second would run through all the processes in the system and recalculate what their priorities ought to be based on what they'd been doing recently. ULE goes a step beyond that. ULE wants to never have to look at every process to make decisions about priorities. So as you'll see, it tracks within each process the information that it needs to be able to make decisions about whether its priority should be raised or lowered, whether it should be considered interactive or batch. And thus figures out that priority in a way that doesn't require it looking at everything all at once. And again, the idle scheduler, much like the real-time scheduler, the high-level decisions are being made in the user-level application that's running where it can periodically decide if it wants to change the priorities that it's setting things to. The kernel is only dealing with it at the low level, and that is where the processes are in the queue or the threads are in the queue and where they ought to be run. This gets us to the low-level part of the scheduling. The four-BSD scheduler just has a single global set of run queues organized from highest to lowest priority. It was designed in a day where we didn't have multiprocessors, so just having a single queue for all of the processes made a certain amount of sense, even when you have a small amount of multiprocessing. You don't have too much contention for the lock to be able to go look at that queue. As you'll see in ULE, that's no longer true. So here, it just had the set of processes, so it would just scan the list by highest priority running. ULE scheduler actually uses three sets of run queues and there's a set of run queues for each CPU. So each core in the system has its own set of run queues. So when a processor stops running one thing and wants to select another one, it doesn't look across all the queues in the system. It just looks at its own queues. So in the case of ULE, each CPU has three queues from which to work. And first of all, we have the real-time queue, and it has on it all of the kernel, both the top and bottom half kernel, and all of the real-time processes. And those of the timeshare threads that are classified as interactive. All right, and this is organized as a set of priorities from 0 to 171. And so this queue is the first one that it will look at, as we'll see later. And as long as there's anything in this queue, that's what it will pick to run. When this one is empty, it will then move on to what's called the batch queue. And this has all of the timeshared threads that are classified as batch. And it's actually organized in a thing called a calendar queue. And I will have a slide a little later on that explains how calendar queues work. And then finally, there's an idle queue, and it has processes in this top idle set of priorities. So at a low level, we'll check this. If there's none there, we'll do this. If there's none there, then we will do that. And as I said, unlike the old one where we had a single global queue that we had to have a lock on in order to manipulate, here we just have the queues for each processor. So that processor doesn't have to lock the queue because it's the only CPU that's ever going to be looking at it. The one exception to that is that we may need to move processes from one CPU to another. This is called load balancing. And we'll talk about how load balancing works again in a later slide. Here's our priority based queues. And this priority based queue is the same as the old 4BSD one. Or as the first and third of the ULE ones. And the priority queues are just organized as an array of queue heads, link lists. And so what we need to do is starting at the lowest numbered one here. We scan through until we find one that has something on it. So initially we find that the priority 95 here has an emacs. So we're going to take that emacs. What we do is we take it off this list, we run it, and then if it goes to sleep, it's just gone until it touches time as it wakes up. If it uses up its time slice, then it will come back here and get put at the end of the Q95, which in this case would be the only thing there, so it would just essentially run until it went to sleep. There would then be nothing here, dot, dot, dot, dot, up until we get to 120. And now we're going to have rogue. And if it uses up its time slice, it'll go to the end. And so we would then run VI and rogue and VI and rogue and VI and rogue and VI until such time as both of them were sleeping. And then we'd get down here and start running XV and Firefox and so on and so forth. And if all of these were then sleeping, so there was nothing on this Q, then we would move on and go to the batch queue, in the case of ULE. Now, we do not want to have to scan through 10 or 20 or 30 or 40 Qs in order to find which one has something on it. And if you think about cache lines, there's a whole lot of cache lines we would have to hit to search through, since each one of these is at least eight bytes. So in order to speed finding where there's a non-empty Q, we actually have a bit array here, which is called the status array. And the bit is set if there's something in that Q and it's zero if there is nothing in that Q. So here we can just pick up this single word or two words, scan through to find the first bit that's set. And that gives us then the index where we're gonna find a Q that has something in it. So we can pick up this and then go directly to the Q that we need, pull the first thing off the Q, and run it. For ULE then, when we get to the batch level thing, we are going to switch from the interactive pure priority based to something called a calendar based Q. And the idea of a calendar Q is to be fairer than we are with the priority Qs. With the priority Qs, a high priority thing, it can essentially starve anything at a lower priority from ever getting to run. Now, if we've done our work right, we aren't gonna have anything on those higher priority Qs that's gonna run for very long. If it's interactive, it's supposed to do a little bit and then go to sleep. So if it starts misbehaving in the sense of using a lot of cycles, then what we will do is change it to being batch. And then it will end up down here with the other batch processes. Now, if it then goes back to behaving itself, then perhaps it will get to move back to the interactive Q. All right, so the idea of the calendar Q is we want to try and be fair and let everything run at least a bit. Now, things that have higher priority should get to run more than things that are at a lower priority. But we don't ever want to get to the point where something that's just running continuously at a higher priority just blocks all the lower things from going on. So a calendar Q works as a circular Q. And so the size of the Q here is NQ, the number of entries in the Q. And we have this pointer runQ here, which points to the current entry on which we are operating. And what's going to happen is that we are going to, once we get to runQ here, we are going to run everything that's on the list. So in this case, there's only one thing. It's the C compiler. It is going to run and it might go to sleep, but it'll probably just run until we decide that it's not supposed to run anymore. It's used up its time slice. And at that point, we are going to put it back onto the calendar Q. And where we put it back on the calendar Q is a function of its priority. So we have this int Q, and this is the point where we're inserting. And it's going to keep moving along as well, in a way I'll describe in a moment. But what we're going to do is when we've used up the time slice for a particular piece, we are going to insert it at the location of int Q. But we are going to then add in its priority. So the base priority here is the 171 or whatever the bottom priority is. So if it's running at sort of the maximum batch priority, then this priority minus that would be 0, and it would simply go in right here. But let's suppose its priority is 10 above that highest priority. And so then what would happen is it would be int Q plus 10, mod of course the size of the Q. So it's going to come much further down in this list. And that means a bunch of other things are going to get to run before it gets to run again, and then it's going to get put pretty far through the list.  So things that have a higher batch priority will get to run sort of each time slot as it comes along. And things with lower priority will only get to run every third time or every eighth time or every twelfth time. But everything's going to get some amount of opportunity. At least once, each time around the Q here, it's going to get to run. Okay, now in some cases we're going to get down here where we've got multiple things on the list. And so this one will run and then it'll get stuck ahead somewhere and then this one will run and it'll get stuck ahead somewhere as we described up here above. And this list might be at a higher priority, so it might just go one or two ahead of where it came off. Whereas T rough might be at a lower priority, so it gets stuck way up here somewhere. Okay, so in SQ, what we do is that we increment it every 10 milliseconds. So it's just sort of plotting along every 10 milliseconds. So if we get stuck with a whole bunch of things to do, then in SQ may actually streak out ahead of us. And the net effect of that is to sort of push the lower priority things even further into the future. Generally speaking though, if run Q gets, it's done everything here. This Q becomes empty, then it's going to increment up to the next Q. And after you've incremented run Q, if int Q is equal to run Q, then we also increment int Q. So int Q will always be at least one ahead of run Q. And it may be more. In most cases, it actually just walks along just one slot ahead of run Q. But if the system gets particularly busy, then it may get further up ahead of run Q. And the idea there again is to just push things further out into the future. Just summarizing what I said for ULE about the run priority. If that set of priority Qs, which includes the real time threads, any of the real time threads, then we're going to select the first thread in the highest priority, non-empty Q. We saw that couple of slides ago. If this is completely empty, then any of the batch Q threads run the calendar Q starting from the first thread at the current entry. And the time slice that we use there is typically on the order of five or ten milliseconds. But if there's a lot of entries in the Q, we will divide that slice by the number of entries in the Q. But we never let it get below two or three milliseconds, because otherwise we would end up just getting too much churn. And then finally, if there's no calendar Qs, then if there's anything in the idle Q, we will run that. And just for accounting purposes, we always have an idle thread, which is at the absolute highest priority. Which is just there so that we don't have to have a special case in the code for what if there's not anything in any of these. So you may notice when you do a PS you'll see each CPU will have some idle that will have accumulated some huge amount of time. And that's just soaking up any time that there was nothing for that particular core to be doing. So now let's drill down and look at the driver of this low level scheduling. How does this actually work? Well, first of all, how do we choose to do a context switch? And for the priority base Qs, we're going to run all the threads in the top Q. We give each one of them 10 milliseconds, but we want to make sure that every thread's going to run within 50 milliseconds. So a Q that's got 10 threads will use a five millisecond time slice. We do have a lower level on that. So that if we get down to having, if it would be less than five milliseconds, then we just give it five milliseconds. So that it can spread out a little bit. But generally speaking, we don't hit that limit. The idea is we don't want to just churn too quickly because there's a cost to doing the context switch of reloading caches and so on. For the calendar base Qs, we run all the threads in the current slot until it's empty. We give every one of them a time slice of the same duration as we used up here. So the same business of 50 milliseconds or 10 milliseconds, but 50 milliseconds max for any given Q, et cetera. And when a thread other than the currently running thread attains a better priority, then we will switch immediately. So if, for example, we're running something in the calendar Q, but something comes in, interrupt comes in on the top priority Q, then we will immediately switch over to that interrupt thread. Actually it's not quite immediate as we'll see on the next slide. But as soon as it's practical, we will switch over and run that. Scheduling a context switch, what actually causes these tens or hundreds of thousands of context switch per seconds to happen? The most common case is voluntarily going to sleep. So some process does some system call that can't proceed. So for example, you read from the keyboard and the user hasn't typed anything.  Or you're waiting for a packet to come down the network. And so you go to sleep waiting for the next packet to arrive. Or you have issued a disk I owe and you're waiting for the data to arrive from the disk. So this is the most common case and in this case, this is called a voluntary context switch. That is, you are saying I don't need the CPU anymore, run something else. Okay, another fairly common case is when a higher priority thread becomes runnable, and this is most commonly from interrupt level. So a network packet arrives or a timer goes off or a disk I owe completes. And this then causes the currently lower priority process to stop and the higher priority process to get to run. And then we have once per time slice. So this 10 millisecond timer that I talked about going off will say, okay, you've used up your time slice and now it's time for something else to get to run. Now, we said that, for example, when a higher priority process comes in, we want to switch to that process.  And so it can just be handed to the idle CPU and it can run and it doesn't need to interrupt the particular process that may have actually noticed that the interrupt happened. Because the interrupts generally are targeted at a particular CPU or a particular set of CPUs. But assuming that we want to stop running the lower priority thread on this machine so that the interrupt can be handled. For example, sometimes we will pin interrupts and say these interrupts have to run on this CPU. And so if there's a lower priority process, we want to clear it out of the way so that interrupt can run. We don't necessarily want to stop that other thread exactly where it is. It might be in the middle of a critical section. It certainly could be holding some longer term locks. And we don't want to put it to sleep while it's holding those locks, because that's going to just cause excess resource contention. So instead, what we do is we set a flag in its threads flag saying, I need to be rescheduled. And then that request will be processed after, if it's an interrupt thread, when the interrupt has returned or at the end of the current system call. So if we've done a system call as we're getting ready to return back out of the kernel, back to the user process, we notice that this reschedule flag has been set. And at that point, we know any locks that were held have been released, at least any short term locks. And so it is now safe to contact switch off to this higher priority process. Now, if the rescheduling is involuntary, that is the it's, we're not switching because the previous process went to sleep. But we're switching because it's one of these cases where it's a higher priority process that wants to run. In that case, the process that we're switching away from still does want to run. So we need to put it back onto the run queue. Because remember, what happens is we take it off the run queue, we run it. And then if we go to sleep, we're done. And we just go find the next one we want to run. But if it wants to continue running, it's used up its time slice or it's being preempted, then we need to put it back onto the appropriate run queue. So that some point in the future, it can continue running. How do we actually contact switch? Well, what we're doing is under kernel control, we're going to switch from one thread to another. And there's essentially three steps. We need to save the context of the current thread, which is the register's address space, etc. That's defined by the hardware, what you need to save. We are then going to choose the next thread that we want to run. And then we are going to, having chosen it, load its context. So whatever was the register's and address space needs to be reloaded here. Remember that I said this low level context which runs a lot. It's anywhere from typically one to as much as 5% of all the cycles on the machine are spent doing this. And so you really, really, really want this to be fast. Now, of the three parts, the context load and save are defined by the hardware. There's nothing you can do to make that any faster. So the only place you have any room to maneuver is this picking the next thread. And that's why we want the low level scheduler to be so simple. Find the bit of the queue you want, pull that thing off the queue, load it. Typically in FreeBSD this can be done in about 20 assembly language instructions. So that's why we split the kernel into high and low level schedulers. So that we can make that low level scheduler very, very quick. That allows us then to take much more elaborate steps in the higher level part of the scheduler because it's running much less frequently. So there it's okay for us to spend some time really sitting down and thinking about what we want to do. The next step that I want to look at then is ULE. What is it trying to do? And this now, we're talking about the high level part of the ULE scheduler. Our goals are we want to identify and give low latency to interactive threads. This will allow, that we want to allow for brief bursts of activity. In order to make this work, we have to differentiate the time where we are waiting for the CPU versus the time we are waiting for user input. If the system is really busy, you may not be running simply because you haven't been able to be scheduled yet. But we don't want to give you credit for that time waiting because you're in with everything else that's trying to run. So what you get credit for is the amount of time that you're actually not wanting to run. That is you've gone to sleep and you haven't taken some action that says now I want to run again. Another thing that we would like to do is in general to have a given thread to keep running on the same CPU over and over and over again. Rather than to have it bouncing around from one to another. This is called processor affinity. And the reason is because you have state that's on a processor. The processor has caches, memory caches, and VM caches and other things. And so to the extent that we can run you fairly soon on the same CPU, a lot of those memory cache, the L1, L2 cache, are going to still be there. And if we move you to a different CPU then you're going to have to get all that back into the L1, L2 cache of the other CPU that we've moved you to. So generally speaking we want to try and keep you on the same CPU. The fact that each CPU has its own set of scheduling queues sort of naturally gives us that. Because the only way you're going to switch from one to another is at a higher level we decide all right, that CPU is just too busy. We're going to move you from its queue to some other CPU. When we do decide to move you then we need to think about where we want to move you to and this is sometimes referred to as NUMA. So you have different, different CPUs have different characteristics relative to the one that you're on. So as I've said, the same CPU is sort of the ideal case. But if there's multiple cores that are on the same chip, they'll often share at least some of their caches. And so if we can move you from one CPU on the chip to another CPU on a chip, that is not going to be as bad as, for example, moving you to another chip that's on the same board. So if it's a dual socket board and we move you from a CPU on one of the sockets to the other socket, then there's no shared caching between the CPUs themselves. There's possibly some caching on the memory plane, that's the memory that's on that particular board versus have to, for example, go to the memory on a different board. But it's still further away than something that's within the same chip. And then of course, we may have multiple boards in the chassis and we may have multiple chassis. So again, staying on a CPU in our chassis is gonna likely be better than going somewhere else. The other goal of ULE is not to have anything where it starts with for every CPU or for every thread. We never have to look at every thread in order to make decisions. We occasionally do look at all or most of the CPUs. But again, that's done on the order of once a second. We amortize the fact that we do that over the fact that it doesn't happen very often. So let's start with how we differentiate interactive versus batch. So we've got a bunch of scheduling variables. The nice variable, which is the one that the user gets to set in a range of minus 20 to plus 20. The fault is zero. If you go up to positive, nice as that means, give me lower priorities. And if you go to negative, that says give me higher priorities. Typically, a user is only allowed to go to higher positive values of nice. And only the administrator is allowed to go into the negative range. The next two variables here, runtime and sleep tick, are tracking the recent CPU utilization and the recent voluntary sleep time.  And each time a tick comes in, whatever the currently running thread is on that CPU gets charged with a runtime tick. Similarly, when you're not running, you are accumulating sleep time. Now, we don't do that with a clock. We just mark what time you want to sleep and note what time you wake up. And that's the amount of time that gets put into your sleep tick. We then have a priority, which is the current priority that you're running at. And a user priority, which is the priority when you're running at user level. Normally, these two are the same value. But when you enter the top half of the kernel to do a system call, your priority may be temporarily boosted because we want to sort of urge you to get through the system call and back out into user space so that we are not going to be contending as much for kernel resources. So what's going to happen is the way that we get these things to be recent, CPU utilization and recent voluntary sleep time is by doing this decay. And so we decay the runtime and the sleep ticks whenever their sum exceeds five seconds. So when the two of them add up to a total of five seconds, we essentially shave off 20% of their value. So we take the runtime and we shave off 20% and we take the sleep time and we shave off 20%. So the sum of them will now be four seconds. And then we recompute the priority based on these values when the thread either accumulates a tick or is being awakened. And our decision on whether it is batch or interactive is if the sleep tick exceeds the runtime. So if we're sleeping more than we are running, then we're considered interactive. And if we're running more than we're sleeping, then we're considered batch. All right, so how do we decide which CPU we're going to run on? Remember, we get taken off the run queue when we go to sleep. And so now when we wake up, we need to decide where we're going to go. So we're going to follow that hierarchy pretty much that I already described to you. First of all, threads can have a hard affinity. Either the application or the kernel can just say, this thread must run on this CPU. Or in the case of the user interface, you can actually say it's got to run on one of this set of CPUs. So you might say it can run on any one of the CPUs, but only these four that are actually on this chip are permissible. If you have an affinity to a single CPU, then it's easy. We just put you on that queue and we're done. Interrupt threads that are being scheduled by the hardware interrupt handlers are scheduled on their current CPU. That is the one that handled the interrupt in the first place. But if their priority is high enough to be able to run immediately. If it's not high enough to run immediately on that one because there's something at a higher priority already running, then we will look first at the last CPU on which that thread ran. And then we just walk down that hierarchy. So we will first look for the one that last ran on, and otherwise one that's on the same chip, otherwise one that's on the same board, other than same chassis, et cetera. In the worst case, we will search the entire system for the least loaded CPU running a lower priority thread. So if the search ends up offering a better CPU choice than the last CPU on which it ran, we will switch it. And the longer the sleep time has been, the more willing we are to switch it because the longer you've been sleeping, the less stuff that's going to remain in the cache. Because if other things run, of course, they're going to load the cache up with their data. So if you've run on there in the last millisecond, then you probably have stuff that's still in the caches. After a number of milliseconds, five or 10 milliseconds, then the chances that there's much useful in the cache is pretty low. And so the effect of moving you to some other CPU to run, we're not throwing away nearly as much state because there probably isn't very much state left on the CPU on which you ran anyway. The last thing is that we're going to have to periodically rebalance threads between the CPUs. And when the CPU idles, first of all, if it's got nothing to do, it's going to look around for other CPUs to see if there's one from which it can still work. This can be a bad thing to do. You see some other CPU and you grab it. Well, if that was the only thing that CPU had to do, all you've done is pull it, and it's now lost, it's caching. And now that CPU is just going to have to find something to do. So generally speaking, you won't ever steal work from a CPU that only has one thing to do. You'll just leave that there. But if there are some other CPUs, then we will potentially take work from that. And if a job gets added to a CPU that has excessive loads, it will look for other CPUs to which it can push work. So for example, I might have a thread that has an affinity to only run on a particular CPU. And so it goes to run. And this CPU is already full of a whole lot of stuff. It will say, look, I realize I have to run this thread, but let me just shed this other one off my run queue and send it off somewhere else so that I won't be so busy and I will be able to better serve this particular thread that wants to run here. And then approximately once per second, the ULE load balancer runs. And it looks across all the CPUs and finds the busiest one. And assuming that there's more than one thing on that CPU, we'll take one of those and find the least busy CPU and put it there.  And the question is, why would you wait one second? Why not make it half a second or two seconds? And the answer is that if you get too frenetic about trying to move things around, you're really just stirring the pot to no great effect. On the other hand, if you wait too long, then you can get a particular CPU that is very busy and it just takes a long time for it to get offloaded to other CPUs. The upshot is that, empirically, what we've found is that doing this about once a second gives you the right balance between being overly moving things around but also being able to get things moved around when it makes sense to do so. So thank you very much. And I look forward to taking the questions that hopefully will come streaming in. Thanks, Kirk, for that talk. I think I'll unmute you, Kirk, so you can answer questions. Can you? Great, I am here, ready to answer questions. Actually, there's been a big stream of things in the chat box here about the difference between the 4BSD scheduler and the ULE scheduler. So let me just sort of give an overview. What's been in there has been correct. Jan and Jessica have been doing a good job on explaining it all. First of all, there was the question of can you switch back and forth between schedulers? And as was correctly denoted in the chat, it is necessary to recompile your kernel. You can't just dynamically switch between them. And the reason is because, first of all, there's overhead if you have to make it at every scheduling decision of what scheduler am I using? And as I've already said, this is happening tens or hundreds of thousands of times a second. And you definitely do not want to add that extra overhead. And the other thing is that you can't have some processes running on one scheduler and other ones running on a different scheduler. You really just have to say the system as a whole is doing this. And so we deliberately made it a compile time thing, so you have to decide upfront what you're doing. And the overhead of allowing the dynamic switching is just too excessive to put that in. It's not that it couldn't be. It just is not a reasonable thing to do. And then the other question that's been coming down is what exactly is the 4BSD scheduler? And again, it's correctly listed. There is one queue that is global. There's a single queue that has 256 priorities in it, 0 to 255. And when something wants to go to sleep, we just start it at the top of that queue, scan down, find the first thing, and that's what we run. And there's not a notion of real time per se, except that you artificially raise priorities on things. But as a practical matter, there is this one global queue. There is a global lock that controls access to it. So as is correctly pointed out, that's kind of works OK up to about four, maybe six or eight processes. But at that point, the contention for that lock to get access to the queue becomes a serious bottleneck. The benefit of the 4BSD scheduler is it's drop dead simple. It's like 100 lines of code and 40 of those are comments. It doesn't have corner cases. It's just straight down the road, find a thing with the highest priority, run it. And the other issue with it is that once a second, we have to look at all of the runnable processes and recompute their priorities. So again, if you start ending up with something where there's hundreds of runnable threads, that starts to become a serious overhead. And as is pointed out at the ULE, we never end up having to look at everything. And other than perhaps when we're thinking of moving things around, but even there, we don't have to look at everything in any reasonable amount of time there. It's just we kind of ploddingly work our way through that as time permits, unlike the 4BSD scheduler. So the 4BSD scheduler works very well for small embedded systems. Anything up to about four CPUs, you just don't need the complexity that comes from ULE. But as soon as you get to anything over about four CPUs, ULE is just the winning strategy. Yes, Jan just points out the 4BSD migrates threads like crazy between cores, because it has no affinity. It's just first come, first serve. And so if affinity is gonna be an issue for you, then that's 4BSD is not your friend. Okay, so other questions. I don't think people can unmute themselves. So I think you have to just put them into the chat. Okay, so Jan writes, how hard is it to run heavy computation in just a few real-time threads? Well, that's largely a function of your application. If you can get it down to a few real-time threads, then obviously you're gonna make this scheduling easier. And note that you have the ability to pin things to particular groups of CPUs or even to a particular CPU. So if you've got four CPUs and you've got four things that are needing to run, you can just pin them to those four CPUs and then essentially you're doing your own scheduling at that point. So spaVU, I don't know how that gets pronounced. Anyway, the question is so for a system like Raspberry Pi with four cores, 4BSD might be good. It's certainly worth trying. It's pretty evident that it's either gonna work well for you or it's not. For many, many years, people that were doing embedded systems, they wanted a small footprint in the kernel and certainly 4BSD is a much smaller footprint scheduler than ULE is. I actually, when I gave the version of this talk at BSDCan, I had an hour, so I actually discuss a paper that was written that appeared in a USNICS conference where they compared the schedulers of Linux versus FreeBSD and one of the data points there was, 4BSD scheduler is 100 lines of code. The ULE scheduler is 2,000 lines of code. The Linux scheduler is 20,000 lines of code. So right there you can sort of see the scaling of complexity. It was kind of interesting paper and I do recommend it or you can go back and look at this talk at BSDCan where I talk about this for the better part of 10 or 15 minutes. But the way they did the comparison is that they went into Linux and they removed the Linux scheduler and put ULE in its place. So they factored out the Linux versus FreeBSD in all the other ways. The only thing that they were running Linux, it was just the change of which scheduler they put in there. And as they pointed out, it was easier to put ULE into Linux than to put the Linux scheduler into FreeBSD because ULE is so much smaller and more compact. Anyway, the bottom line is that for the most part the two schedulers do about the same. There are a few cases where Linux does dramatically better. There's a few more cases where FreeBSD does dramatically better. And the real issue is that the Linux scheduler has gazillions of special cases in it to deal with certain workloads. And every now and then you've got a workload where the Linux scheduler misgages whether it's a special case. And so it flips into using a special case which is a distinct bad choice. So they end up having somewhere they're just really, really bad. Whereas ULE, it does have ones where it doesn't do as well as Linux, mostly because Linux has a special case code for that workload. But it doesn't have any that are just horrible. Okay, so let's see. So, yeah, yes. We're also coming up on the, I think it's the, at 1725 I think the room will restart in preparation for the next talk. So it's probably better to move on to Spatial.Chat for questions. Okay. Well, I will move over there if people have questions they should show up over there. Right, okay. Well, thank you very much for your talk. Correct? It was really interesting. So I'm hoping we can do something like this in person sometime next year. Right. So thank you very much everyone. Right, bye now.


Again, we'll just label the parts of this cell that we haven't already labeled. Although we need to remember that this prokaryote is missing stuff. There's stuff that isn't here that we did see in the other cells. All of our membrane-bound organelles are missing and a nucleus is missing. So let's label some parts. First of all, I'm just very happy to announce that we can see flagella, one flagellum, and cilia, multiple cilia. Cilia are like, again, both structures made out of cytoskeleton, but having different functions and different structures. So we have cilia covering the whole cell body. In the case of, remember the guy that we saw in the beginning who had the two stripes of cilia? So we can have different organizations of our cilia. We have ciliated cells in our respiratory tracts. We have this, it's called a mucociliary escalator, and we have these ciliated cells that actually move mucus up our respiratory tracts so that we go and swallow it down to our digestive tract. That's cool. With the intention of capturing prokaryotes and eating them before they infect our respiratory tract. Dude, we've got some pretty incredible strategies to protect ourselves from these guys.  Okay, but the flagellum and the cilia in a prokaryote or even in a protist are about movement. Remember, I told you that they do have DNA, but it's found in a general area in the cell and the general area isn't the nucleus because there's no membrane surrounding it. It's called the nucleoid, so it's just that general area. I don't know if you can see them, but I can, and I'm coloring them blue instead of orange for you. These are ribosomes, and what do you see in this weird, like what, how many layers of stuff are we looking at here? I'm just going with the inside layer. What do you think the innermost layer on this cell is? And you can point to the plant cell for inspiration. All cells have this. So cell membrane, prokaryotes have a cell membrane, and it's the innermost structure. What else do you see? Well, I know someone out there is going to know the answer to this. I don't teach microbiology, and so I'm sure that there is a name for each one of these structures. This is the Wendy Riggs name for the two structures that are outside of the cell membrane. Bacteria can have cell walls. I'm going to say just one cell wall because I'm going to put those two things together. I feel like one of them is like a capsid, or like they can also have this extra outer capsule that stains that you can see if you look through a microscope and you use a certain stain. But the thing I want you to hang on to, you'll learn all about that in micro, if you take a micro class because you're headed into some sort of healthcare thing, you'll learn all prokaryote anatomy. The thing I want you to hang on to is the fact that they have a cell wall, and that cell wall is totally protective for these guys. Makes it harder for us to break them apart or break them down. All right, I think that I don't see anything else. No endoplasmic reticulum, no mitochondrion, no chloroplast, oh, there's something I've forgotten on every single cell. So you get to know it in this guy. I have forgotten every single time to label, I don't even know how to label it. Let's just put it down here. Psytoplasm. Psytoplasm is the old juice, the stuff inside the cell membrane that everything floats in. That's all cytoplasm. Now being a physio, human physiology person, I don't tell anybody, but I think human physiology is my favorite class ever. I really love human anatomy. And when I teach this class, I really love general biology, so I don't know how I choose. But physio is really cool. And in physio, we look at the cytoplasm and we call it intracellular fluid. That makes sense, yes, and we abbreviate ICF. In a single celled critter, the intracellular fluid is what they maintain, it's their guts. In a multicellular critter, we have extracellular fluid that we maintain. Our blood plasma and we have interstitial fluid and we'll talk about the different fluid compartments in multicellular critters. But that cytoplasm is really important and it's different than the environment. If it wasn't different than the environment, then why even bother having a cell? Okay, so cytoplasm, we find it everywhere. On my list here, I have us talking about the endomembrane system next, but I'm not exactly sure why we need to do that again. So we'll do it anyway. I'll be right back.


And Johnny asked me to pick up Six cans of Sprite at the grocery store.  Thank you for all the good Today we have a new trailer that dropped from our dosty of Dost Rajkumar Rao. Hey And it's called bead Or bead. I've Be saw that bead so many times I have to skip by because the majority of our our Dosts and our nun bars and Bundles and we we we bondu. Yeah that too They post stuff. So I saw this on Rajkumar's But I had a just a quick like speed by because I knew it was gonna be info we'd probably be reacting to oh and Bhoomi Pedna car. Yeah, and this is the director. I do believe on a pop sina and this is I believe a True story of some sort, but let's let the trailer tell us before I read anything here. All right, and here we go Oh, no This shopping of all the state borders This for COVID Oh Hey She hits that Oh Rosh Kaur just gave me fucking chills bro, ah that looks like art, yep is what that looks like there's a lot to unpack there sure is um if that was a Hollywood film and I don't know how the film is but that would be coming out around December because that seems like a variant and it's super interesting that he's decided to do it in black and white yeah that's super interesting yeah very interesting that's a choice that is a that is a choice what do you think about the trailer I thought it was powerful I thought it was um it it I obviously not knowing the whole story and not knowing the screenplay it really feels like they gave you everything you needed to know without giving too much away and with this cast you know what you're gonna get you particularly know what you're gonna get if you are a fan of and if you're not you should be of Rosh Kaur Rao he's at a place in his career where he can make decisions on what he wants to do versus what a lot of actors have to do especially who are developing is is just be thankful for the work that they get and just take anything that's thrown their way he's in a position where people want him to be working on things and he can be very selective and the choices he consistently makes are always for Corbin and I the kinds of films as an actor you want to be making they have great script great character important messages that have some connection to society that holds up a mirror either to show the great things or to show the things that need to be changed and this just feels like it's just loaded with that and so I'm super excited because the director is the director of article 15 right between powerful great film and but I he also directed other films that I know are of that same ilk that we just haven't gotten to yet right there's that right which is with topsy there's mulk which is another one that I heard a lot about that people wanted us to watch yeah and I think that it's also topsy and reishi yeah and I think it's one of reishi's most powerful performances um and then there wasn't I thought there was another one that I a knock uh was that last year anyways he's known for doing these um I guess it would be considered as political but they're not really political it's just because if you tell us sorry in a certain way people think you're taking a stand in a certain way uh right you don't agree with the no there's a there's a difference between article 15 is considered political correct however there's a difference between films that are um not about politics right there's political films that are making it really clear by the creators here's where we stand politically and we are antagonizing the opposition with this particular movie um I consider um what was the movie I did not like that which one the decaprio film about climate change but it's a yeah don't look up I I never that's one of those films um it's very heavy-handed in terms of it's it's uh we are the liberal left and all of the conservative rights are assholes and morons and idiots and and and it's it's it's not so much about the issue as it is the combatants and we're on the right side of the story whereas when you're telling a story that does involve politics yeah but what you're looking for is the universal truth that irrespective of where you're coming from politically we should all be looking at the reality of just human need and human compassion and not the politics of it not the economics of it that that the most important thing in here and he seems to obviously like most of his films he is the conscience of the story and if it's a true story which is well it is I'm well aware of what happened in India during the um obviously when COVID immediate uh and obviously when the borders closed and then um and obviously the people that are always affected by everything are usually the poorest of course always because the the people in power and and the rich don't give two shits because they're not affected no because they have the money to leave they have the money to they're expendable whatever yeah the poor expendable to them so it's usually the poor and obviously this is a big thing I don't the ins and outs I don't know which is why I'd be excited to to learn some of the and ins and outs and I bet it's going to be done in a way that will show you not just we're right you're wrong I bet we're going to have moments where something's done that is at first blush totally inhumane yet the people doing it that maybe Rajkumar is against are going to give you some arguments that make you go oh wow crap they have a point yeah not that it's right but it makes it much more complicated to have an actor of Rajkumar and and obviously um oh Pankaj which is yes great to see him poor sorry well and uh bend the car as you it's boony yeah boomy she's always good but somebody of Rajkumar's caliber he showed it just in the trailer they give a little glimpse uh multiple times but especially there at the end of like a really heartfelt plea yeah of Rajkumar's character which gave me chills there at the end he's very much if I was going to make a comparison not stylistically in terms of his persona but stylistically in terms of the choices he makes and the characters he plays he's he's the Jimmy Stewart and if you don't know Jimmy Stewart that's a weird compare well if you don't know Jimmy Stewart's filmography better than Jimmy Stewart no no no I'm not talking about I again I wasn't talking about the style of his acting per se okay I'm talking about the stories he tells and who his characters are in those stories okay Jimmy Stewart always was the conscience of the film he played the attorney fighting to right a wrong okay he played the good guy who's getting shafted and shouldn't be and standing up for the little guy gotcha um that also that that's just that was very much what Jimmy Stewart's hallmark was he was always the frustrated moral compass who was fighting against the system that was injust and and he didn't have a he wasn't James Cagney he wasn't going to blow the doors down he was the guy who was trying to be compassionate and if he pushed him too far he'd lose it but he didn't want to lose it you know it's crazy and I agree I originally I didn't get a comparison because that's what I meant Roscoe Marshall was a thousand percent better for an actor than Jimmy Stewart I tell me Jimmy Stewart's quite a good actor yeah sure but the this comes out I think the 24th and then the next week I think that Ronnie film comes out about the versus oh yeah so that's two back to back hard to watch films I bet that's all right great art exactly and it's I'm I'm super interested to see the reasoning behind he chose black and white yeah I'm very interested in tonality um maybe because everything at the time looks so bleak maybe like um he didn't want he didn't maybe want to see people's different colors yeah it won't I don't know it won't surprise me if it's multifaceted I bet it has to do with the bleakness of it the sense of everybody's the same yep the lack of optimism it's a storm everybody's the same I think it also is going to go into a complex story about there is right and wrong however things do get gray and how do you overcome like on the one hand you need to be humane on the other hand what do you do there's so many times those kinds of decisions have to be made legally politically uh humanitarian I'm very very excited to watch me too very great uh anyways let us know what you thought about the trailer and also I know the directors of the film say I've been meaning they've been on my list for a long time sapped in mulk specifically yeah I remember those two I've been on my list for a while and I know they're divisive especially mulk because I think it's a specific political issue that certain people agree on one side right but right and it's very interesting I did see this director uh in early in his career directed raw one which is that shower con superhero movie and so I'm guessing that was like maybe one of his like things like here's here's the shower con movie prove yourself absolutely started doing what he wanted well it's like like we said he's at a position now where he can do the films he wants to do versus just doing anything that you so you can just be working exactly yeah absolutely anyways let us know what you thought about the trailer down below


Really? So yeah, today is the regular JSOC Special Interest Group meeting. We will sync up on the Google Summer of Code status and then just have some time to do corny with students whoever is interested in asking any questions. So I think I'll just share my screen. Okay, is it visible now? Yes, yes. Okay, so we have one of the things which we wanted to discuss is of course coding period because coding period starts next Monday. So this is our last office hours before we go to the coding and we ask all students to make sure that you actually went through the plan and that you perform all actions as a part of the community. So generally our expectation that if you get introduced to the community, you get enough knowledge transfers and that you know what to do as a part of the coding session. So yeah, just three expectations. Yeah, if something didn't happen so far, please think your mentors as soon as possible and please follow up on your action items as soon as possible. Actually my question is do students and mentors would be that do you have any help from orphanage? Okay, so I assume no questions right now. Okay, so yeah, one of the things which you also brought up the last meeting that there is a need to have a kind of dashboard or whatever mostly for this part so that you have a plan for the first coding phase. So currently we have a dashboard in Jinx Jira. So if you open Jira, there is a dashboard called the Google sun of port 2019. So yeah, I'll just show it. So yeah, here's our Jira. If you go there, you may see that there is a number of sprints. So what we started from that we have a sprint foolish phase. So community born according to phase one, according to phase two, according to phase three. And there is already a number of tickets. So in order to get an issue on the dashboard, you need to label it with a GSOC 2019 task prefix, I believe. Yeah, just a GSOC 2019. So GSOC 2019. And after that, the issue will be in the backlog. And after that, you can just drag and drop these issues. For example, I can take one from there and put it to calling phase one, just for an example. But yeah, it's pretty much a sample issue, which is already working in progress. So you need to fix it later. And you can do the same for your issues. And actually what is asked that before the coding phase starts, you play to piece the issues for the first coding phase and put them to the dashboard, you'll have all the filters. So you will be able to search issues in your projects. But yeah, it would be nice if you put these issues on the dashboard. Are there any questions about Jira dashboard and how it works? Because yeah, people may have a different experience with Jira. So if you have any questions, any feedback, let us know. When I'm on the dashboard, when I click on a particular issue, in the right side panel, it shows up the issue. So if you add a link to the particular issue, it doesn't show up in the right panel. When you just expand it, it shows up. You mean project links? Or what? No, issue link. There's a section for link in the issue when you create an issue. So yeah, unfortunately, this browser is not ideal. And unfortunately, we don't have all options to modify the layout. I'll check. And another thing about the pipeline authoring SAGC meeting. So you were saying that it would be happening today. No activity in the group. Yeah, so I believe that there is a coupon now. So I'm not sure whether Andrew is planning to have a meeting. Let's check. Okay, I just asked. Okay. And if you have such topics as a part of community bonding, don't hesitate to reach out to people directly. Because yeah, some projects are related to CX. So yeah, two projects for pipeline, two projects for cloud native C, one project for platform. And yeah, at least as a part of demos for the first coding phase, we will be using at least six platforms. But you can reach out to the special interest groups early in order to get some feedback. Okay. Or at least to get introduced. So yeah, should be something to consider. Okay. Okay. Any other questions? So yeah, we have some students who haven't yet created tickets in Jenkins Jira. Please do so. And if you haven't discussed what would be your plan for the first coding phase, also please do so with your mentors as soon as possible. Because yeah, the worst case scenario that coding phase starts and you don't know what exactly to code. In such case, yeah, firstly, it's just wasted time. And secondly, yeah, it's a planning issue. So we want to avoid that. If you need any help from open means in order to make things happen, please reach out to us through private channels. So we have a simple piece of open means and that if something is going wrong, we will help to facilitate the things. But yeah, by the time coding phase starts on Monday, we expect all projects to have at least implementation plan in Jenkins Jira, because it will be our main engine for communication during a few of the phases. So regarding the rest, yeah, get the keys to be able to work. We had key keys at least in several projects. Again, if you're missing something, if you're missing information, please think your mentors as soon as possible. So yeah, that's pretty much all I wanted to present today in terms of project organization. So we're gonna clarify some questions right now. So yeah, if you have any questions, or if you see that something is missing, please bring cover this topics. And yeah, we will be able to discuss that. So you think Natasha, you there? Is everything fine? Yeah, I was gonna say, I don't think I have any questions. We had a call yesterday and we haven't started doing like your tickets yet, but I will make sure to get those in the next couple of days. And yeah, I think at least in terms of my project, probably the first part is, I think going to be still kind of exploratory. So, but yeah, I will create tickets to reflect that. But other than that, I don't have any specific questions. Okay, yeah, that's fine. So if he says he is fine. Oleg, do we have any special interest group for our project? For all strategy, we don't have special interest group right now. So I was talking about having a security special interest group. So to have, so there is a security team in Jenkins, but the security team is highly focused on fixing security defects, etc. And they don't really have public meetings. So we have private meetings. I've wrote up a question about having something public, but yeah, I didn't get so much feedback about that. So yeah, one of the options that we press it without a special interest group for now, or we can just try to get it organized on the flight. So for example, sending a message to the security team. Well, yeah, probably it's not the best thing, but instead of that, we could reach out to potential users. So we can send a message to the user. The results are an opportunity to have a blog post earlier so that you reach out to the wider community and it can help to bring in some stakeholders. I sent the same email that I had sent to the developer mailing list to the user mailing list on the suggestion of, I think, Babweast. Yep. So yeah, there is Jenkins users mailing list. Yeah, actually it's something which, oh, I'm not changing my screen anymore, right? Okay. So I'm presenting again now. Yeah, here's Jenkins users mailing list. You may see that there is, the number of you is close to zero, but it's mostly because the fact that the most of people have subscriptions. So for example, I don't use a mailing list and many people do not really use a mailing list, but to interrupt here. But yeah, the fall projects, it might make sense to have some visibility here at some point. At least when you have something to share, it's perfectly fine to send it to the user's mailing list. And the same for the community channels. So like Jenkins and your blog, once you have something to share, then it makes sense to use these channels in order to promote your projects. And if you need more ideas, just talk to your mentors because there are ways to get things promoted. Okay. So yeah, whatever channel is something we could use. Okay, does it answer your question? Yes, it does. Yeah, regarding special interest group, yeah, let's try to see what we can do. But yeah, in the worst case, we just press it without special interest group and try to build coming into around the thing. Because the role strategy plugin has so many users that you probably do not really need any special interest group just to get some attention from users. Okay, let's see how to do that. Okay, thanks. What we could do, we could actually So, the role strategy plugin has a Mickey page. And we could post, for example, let's say we could have an announcement box right on the Mickey page if it ever opens. Okay, so we could probably just put a box here saying that, hey, there is an ongoing project for performance improvements. If you're interested to join our channels. It could also help. So for role strategy plugin, you can actually do it on your own because yeah, it's Viki. So you can edit this page. And maybe for other projects, it's something like that could be done. For example, for GitLab plugin, it definitely could be done. For external workspace manager, you can adjust maybe by GitHub to have links to your project. So yeah. So yeah, it's something you can use. Anything else? I have something I wanted to mention. And when we send pull requests, do we need to tag them with special keywords or special labels? Okay, yeah, it's something to discuss. So the problem with that in order to tag a pull request, you need to have the right permissions to the repository. So it means that it's not possible in all cases. What we usually recommend to do is, yeah, so we can just take a look at Jenkins CI Jenkins. Yeah, so what we did, for example, for a remote project, we created a team. So yeah, okay. What we did is that there is a teams. Yeah, so there is a GSOC remote team, which includes basically all mentors and the students right now and we can add more people if needed. And once this team is created, firstly, a student gets added to the organization. So he or she can use these teams for mentioned. And you also can request reviews from the team if needed. And you can see the team when you submit the pull request to request reviews. So my advice would be to actually when you create a pull request either to CC team or to CC your mentors directly because it's something helpful. Oh, like would it be good if when they send the pull request in that they actually add the mentor and the lead mentor as a reviewer in the pull request? Again, you need to write commissions to the project a little bit. Okay. Yeah, so for example, let's take example of this whole strategy pull request from our leader about microbenchmarks for Java. So for example, if you take a look at this pull request, now it doesn't mention the mentors, etc., because yeah, we have a chat for that. But what we could do, we could say that for example, we use from like, you know, so it's the most simple way to request the review. Once you do that, all mentors will get email notification. In order to do that, you don't need to have to be a mentor, sorry to be a member of the organization, you don't need to have to write permissions. In order to request reviews here, unfortunately, now you have to have write permissions to the repository. What it means that in order to be able to request reviews, you would have to be a contributor with merge permissions to the project. And for example, in this case, we could make a contributor. It's not a big deal. We could do that. But yeah, for some projects, for example, for Jenkins Core, if in some projects you need to update that, it wouldn't be possible. At least it would be really tricky to do that. So I would rather recommend to rely on mentions and then not to simplify mentions what we could do. We could actually just create new teams. So for example, we could create a team for GSOP to 7.19. So this team actually allows us to add you so this team will allow us to add you to the Jenkins organization so that you will be able to just comment everywhere, et cetera. And what you can do, you can just navigate to this URL. And then you can ask to become a member of this team. And later we can create specific teams for projects if you need it. Or it means create GSOP teams upon request. Hi, I'll say you're saying that instead of CCing people individually, like saying like CC at mentor, you're saying like if we join a team, we can just like add the whole team. Is that correct? So you can create a team for a project and then this team will be portable. So as long as a student or as a mentor or member of the Jenkins organization, and they will be able to join the team, they will be able to CC all the people without repeating this list every time. So in order to request such teams, you generally just go to Jenkins Jira. Here you create a new ticket. For example, yep, just a new ticket. So the project is in trust structure. Well, something like a task like this. And here in the company, you put GitHub and update a GitHub team. For example, for let's say, here, looking at all strategy projects. So something like that. Okay, once you do that, you just click create. Yeah, you can add some information, but these tickets, well, they get assigned to the new just assign them to me because you have permissions and they will be able to do it quickly. Yep, thanks Martin for doing that. Yeah, so if these teams help, you can just do something like that. You can also add them to GSOC to something like that. So they will appear in our dashboard also, or maybe not. If not, I will get it fixed. Okay, so yeah, basically, if you need that, just let us know. And if you need any other things from Jenkins infrastructure, it's also something we can help with. For example, if you need to host a new component, let us know we will help to deliver quickly. Or if you need any specific things, for example, Docker Hub, whatever you also can help it with. So speaking of that, does anybody need any infrastructure things to be done? Earlier, I needed access to enterprise version of GitLab. So GitLab has two versions like there is a community version and the ultimate or enterprise version. So as I don't have the link as of now, but Jenkins has the permission to access the enterprise version as a open source organization. So I'm not sure who has those credentials. Well, we don't have an enterprise account for GitLab. So if you need that, the best way is to start the thread in the information please. Maybe some time later, sorry, the audio breaks. GitLab lists some of the open source organization. Okay, can you hear me now? Yeah, you can. Hello? Yeah, now it's better. Okay, so maybe later I can share you the link. Okay, maybe later share the link like there, GitLab mentions a list of open source organizations who contribute towards GitLab. So maybe I am assuming during the development of GitLab plugin, it was the Jenkins org was a part of it and it lists Jenkins as one of the open source organization who has those access to the enterprise. Okay, so basically it's hard to say about that right now because I don't know what GitLab's conditions. I'm not sure whether they would be open to give access specifically to the Jenkins project because well, GitLab has GitLab CI. So it's not clear what would be the company's position about it. We can definitely discuss that but I don't have a definite answer right now. So what would be my advice is you discuss this topic with mentors and then once you have a decision whether you want to work on the enterprise version, reach out to the information piece so that you can get a sign-off and access for that because generally it's something to be approved in the governance meeting. Okay. Yeah, so there are ways to do that. I mean I believe that Rick knows how it's usually being done because yeah almost all such special requests go through the governance meeting. Okay, okay. Any other? Not from my side. Okay. What about others? I have a question. Sorry if this has been answered already. Is there one Scrum board per JSOC team or it's one Scrum board for all of them? One Scrum board for everyone but you can set up quick filters and I will set up quick filters for projects once EPICS are created because I need EPICS created by students so that they set up quick filters. If needed we could create separate boards and separate sprints for each project but yeah I'm not sure what would be the need in that. Okay, so students are expected to create the EPICS and put their the labels. The labels and it's one EPIC per sprint. Sorry, one EPIC per coding phase. Well EPICS up to you. You can have multiple EPICS if you want but right now we have one sprint per one coding phase. So it means that the sprints are approximately four weeks. If you want to have shorter sprints like two weeks, okay we could do that but yeah basically it's up to you. Well if it's common everybody has to follow the same sprint durations. If we use the sprint feature, if we just use it for filtering and ranking then then it doesn't matter. So that's why I didn't bother much to create it before community bonding started because last year we did sprint boards etc but effectively nobody used them. So here we created them after the discussion at the previous week but again if nobody uses them probably we don't need to spend time. If somebody wants to use them then yeah we can adjust upon the demand upon the request. So if you want to have two big sprints it's something we can help with. If you want to have specific filters separate boards separate sprints we can do that but yeah there should be such a request and yeah that's it. Does it make sense? So can you do you mind repeating like what the expectations for students are again? So students should create the epics and the tasks is that correct? Well we can even survive without epics. So epics are rather useful for you in order to define milestones and what needs to be done for them. So because for example you can set a milestone alpha release of a plugin management library and you can create an epics for that because a list of tasks you need to complete and then you have let's say 20 tasks and you discuss these tasks with mentors and you already did 15 tasks to be done during the first coding phase and then five tasks during the second coding phase. So you still have epic as a delivery but you put tasks to different spades. It's just one of the examples how you could use epics but yeah basically it's up to you how you operate with them. What we need you to do is to create tasks and to create and to set labels for them and yeah the rest is what we could do for you to know about the thing. Oleg do you mean tasks or do you mean JIRA stories? Well JIRA tasks, JIRA kikis, JIRA stories as you wish. Actually yeah it's easier okay so if yeah it's a terminology. Yeah it's a terminology but functionally JIRA stories are easier to manipulate inside sprint boards than tasks if we ever use the start sprint feature. So I would recommend stories it doesn't matter in the end but stories are easier to manipulate. Okay so let's talk a bit about the terminology because now I'm also confused. Okay so let's see what we have. In JIRA we have issues. So issues have multiple types task, bug, feature, improvement, whatever story. From a management standpoint it doesn't really there is no difference whether it's task, new story, new feature, bug, whatever. So whatever type you create you can easily manage it above. So maybe what you meant Martin is a sub task not task. Okay yeah so generally what is what I tried to say that yeah we expect you to create issues. These issues can have different types depending on your needs. So you can choose whatever approach you prefer and yeah that's it. Okay for the sake of simplicity personally I use epics and stories and the other ones I almost never use them because they don't behave well in the in the swim lanes when we start sprints. Maybe it's something specific to your JIRA. Yeah so in the case of Jenkins JIRA these all these steps have exactly the same workflow. So it doesn't really matter what you take except epics because epics are something for grouping but for the rest you can take whatever type. There is no practical difference between story and for example improvement or new feature in Jenkins JIRA. Okay so if they use the same workflow then it's going to get easier. To clarify epic, epic is is a grouping but it's a grouping or typically over a period of typically an epic is over a period of time which is longer than a sprint. Yeah it could be longer than sprint and it could be shorter than sprint so my personal approach for epics is that I use epics for deliverables. For example let's say I have a full bar plugin and I have epic full bar alpha release. So yeah alpha release of full bar I create this epic. Okay it's here and let's say it's yeah JSOC 2.7.19 and in this epic you can say that okay what do we need to create it. So we say that I want to have feature example and create a new let's say okay a new full bar page for Jenkins. Okay something like that. Okay I will definitely need to clean it up later. So yeah we put label and here you can create a few more of these tasks or you can just clone it. So here for example you say that post your plugin repository and here let's say write documentation. So clone write documentation. Okay so let's assume I'm a student. So as a student I have three epics or sorry I have an epic and I have three tickets in this epic. Then I go to the board and for example we have a meeting with these mentors and we agree that yeah we create a page which is a part of first coding phase. First coding phase then we create we host the plugin as a part of the first coding phase. What's going on with drop and drop. But regarding write documentation we can agree that we do it during the second coding phase and we can say that okay we have epic which sets the delivery but there are actually two tasks we want to perform during the first coding phase and the last task during the second coding phase. Or we can agree that it goes here so it's something like you may want and it's up to you and mentors to decide how you do that. But yeah epic is just a way to group the deliverables. So all of you have separate deliverables in your form but you can just decide how you do that. So for something like updating you know we have like our project pages and stuff like that. Would that go under an epic and then we'd have just like separate tasks for update after coding phase one, update after coding phase two and so on. Yeah okay. So we don't have a requirement to have dozens of Jira tickets or whatever. It's something you decide with mentors. But yeah what the history shows that if you have fun paying tickets then it's easier for you to manage the tasks. It's better for you to track the process. And yeah so our advice is generally to have a good regular rate of tickets. So write documentation, write tests is probably okay. Create new pages is probably okay. But if you have for example five features you want to deliver probably it's better to have separate tasks or separate stories inside the epic for that. Okay is it clear now? Or is it worse now? Okay well thanks Oleg for the explanation. I was disconnected from the network and I just rejoined so I'll watch the recording. Okay happens. Yeah so anything else to discuss today? Any questions? Any help needed? For example I was surprised that UFA didn't need a help with AWS accounts. So maybe it's something for mentors to clarify with the student. For other projects yeah not aware about any specific infrastructure requirements but if we are missing something just let us know. Any other questions, topic for the discussion etc. We have something like 10 minutes left. Okay so yeah this seems there is no other topics. So yeah if you need any help, if you need any clarification please use GitterChats. So we have the next meeting in a week or so when the coding starts. But if you need any time or clarification just use the GitterChats or all our commitments are present there, mentors are also present there. So yeah we will be able to help you without waiting for the next meeting. Then I think we can just stop the broadcast for now. Yeah by the way some organizational changes. So Marky will be joining the JSOC organizing team. Also Rick will be joining the organizing team. So we will have more capacity in terms of handling all things which was one of the feedback we got for community-born phase already. So we will try to have more bandwidth and yeah probably we'll be also doing some shifts in terms of lead mentors. So we have some background discussions about that. So please do not be surprised if such changes happen. Nothing really changes in terms of projects and goals. Please keep working with your mentors and we will provide you as much assistance as we can. Thank you. Thank you. I just forgot to tell about it in the beginning. For those that don't know me I've been in the Jenkins community for quite a while and I'm a maintainer on a few plugins and I've been a mentor. So I'm glad to be in the admin org and if you need anything please don't hesitate to reach out. Thank you. Okay I think we can close down the meeting. Have a good day. Good evening everybody. See you next. Bye everybody. The next project meeting. Thank you. Thank you.