All right, good morning everybody, right, and early today. So the goal today is to do kind of two things. So we'll finish up our look at virtualization with just discussing sort of how full virtualization works. Once we're done with that, we will segue into doing review for the exam. I have some slides for that that look at material that was covered last year and the second half of the class. So it's kind of semi-applicable just to give you an idea of some things that will be on the exam that weren't on the midterm. However, this year because we didn't get through as much VM, there will be more VM on the exam. So that's just a point of fact. The exam is going to cover, you know, it's designed to be cumulative, but the focus will be a little bit more, especially for some of the short answer questions, and I would say the multiple choice in the short answer will probably focus more on second half material. And then the medium and long answers will try to get you to integrate things from throughout the semester. All right. So yeah, so next Monday, we will, you know, like people will be in their seats with an exam at 8 a.m. And that's when we will start, right? I want to give everybody enough time to do the exam. So today was your practice, right? Some people were here at 8.10 or so, or 8.15, some people still aren't here, some people aren't going to come, which is fine. But you know, like any fine tuning that you might need to do in order to get yourself here at 8 o'clock, you know, to think about. Adigio this week is going to be covering the exam and doing exam review in recitation. So that's the plan for recitations this week. That's another chance for you guys to review. We posted the sample exam, which we can also, I'm happy to answer questions about today. The second half of class will hopefully be much more of a Q&A. So if you guys have questions about material we've covered or about the sample exam and things like that, I'll be happy to answer them. So far you guys are at 60% with the course feedback form. So good, but not up to that threshold.  So that is the number from 10 minutes ago, right? So unless there's been a mad rush of people, I mean some people have their laptops here. You could do the course feedback form while you're here. And I will call on you in the middle of doing it. Okay, so anyway, yeah, I'll keep monitoring that, but you know, you guys have a week. So you know, as soon as right now there is no exam yet. So there are no questions to release. But once we put the exam together today and tomorrow, then there'll be questions available. So you guys start hitting those thresholds and I'll throw our questions. All right, so let's go back to Friday, right? And we talked a little bit about why we would want to virtualize hardware. So we've talked all semester about these nice operating system abstractions that exist to protect processes from each other. But we came up with this idea that what we'd actually like to do in certain cases virtualize actual physical hardware. And the goal of virtualization, remember there were two approaches to virtualization. One was what we're going to talk about today, which is full virtualization, which means I can run an unmodified operating system inside my virtual machine. So the virtual machine monitors able to emulate a virtual machine so well that the operating system doesn't even have to know or have to be aware of the fact that it's running in a virtualized environment. There's a second approach called para-virtualization, which we're not going to cover that involves changes to the guest operating system designed to make it more virtualization compatible, right? And so what was hard about full virtualization? What was one of the challenges here in making this safe and effective? So first of all, in order to achieve good performance, what do we want to do in general, Jeremy? Okay, well I'll come back to you for that. So if we're going to get good performance with our virtual machine, what do we want to do as much as possible? Assume it, we want the performance, so to some degree, what's the best performance that a virtual machine can potentially achieve? Nick, which what hardware? Right, but what physical hardware? Like the performance, if I run a virtual machine, right? What bounds is performance, eventually? Main memory of what? Okay, the main memory of what, Tim? Yeah, the capabilities of the host machine, right? I can't have a virtual machine that runs faster, and this is not strictly true. I should be careful about saying this, because there are cases in which the optimizations that virtual machines can do might actually, but help me run faster. But the point is that in some sense, I'm bounded by the physical hardware, right? But what do I want to do to make the performance of the guest operating system and applications that are running inside my virtual machine be as close to, as possible, the performance of the underlying physical hardware? What's my goal, Tim? Yeah, I want to use hardware to run the instructions, right? I mean, there's going to be a stream of instructions that are going to be executed by the inside the virtual machine monitor, right? The virtual machine monitor is just running as an application on my host system. And the guest operating system and applications inside of the virtual machine are just going to be executing a series of instructions, right? As many of those instructions as possible, I want to use the real hardware to execute, right? Same thing I would do if I run a normal application, right? But why can't I do this in all cases, Frank? That's not quite what I'm getting, sugar. The question is, why can't I just take, so the best performance I can achieve is if I can take everything, all the instructions executed inside the virtual machine monitor and run them on physical hardware. But why can't I do that or what makes that hard, Jeremy? No, remember, I mean, to some degree, when the virtual machine monitor is running, it's like another application on the system, right? I'm just giving it control of the machine. Okay, so there's one problem here that we've identified, which is traps, right? But what's another potential problem, Spencer? Yeah, so if the guest OS runs on the hardware, how? What can I probably not allow the guest OS to do that it would normally be used to do it, John? Yeah, if I give full privileges to the guest operating system, there's a variety of different ways that it can pierce the virtual machine, right? Different ways that it can use the hardware to access parts of the machine that, whatever, somebody, it says, it's time to get up. So you can get to class. So if I give, and our example of this, a simple example of this was, if I allow it to control the page tables directly, it can load mappings for parts of memory that I haven't given the virtual machine monitor permission to use. So the virtual machine monitor, I say, here's a gigabyte of memory that you can use to create a virtual machine with a gigabyte of memory. But if I'm not careful and I allow the guest operating system to run with full privileges, it can load mappings that allow it to address any of the memory on the machine, right? And that's what it's used to be able to do, right? So yeah, so there are these two things, right? So the guest service is going to try to execute prodigious instructions. And in most cases, I don't want to allow it to execute them without inspecting it, right? I need, somebody needs to make sure first that what it's doing is safe, right? And then the second thing is handling these traps, right? So what's the bit of subtlety with traps or system calls that occur inside the virtual machine monitor? Why do I need to handle these a little bit more carefully, Bart? Right, so the hardware is hard-wired to do what when I trap, right? When a trap is, you know, when a system call, for example, is executed by an application, yeah. Yeah, it's hard-wired to start executing the host operating system, right? And so when the virtual machine monitor generates a trap, what are the two things that I might need to do? There's two, so normally if an application generates a trap, who handles it? The operating system, right? That's the interface between applications and the operating system. When I want the operating system to help me with something, I issue a system call and the operating system wakes up and does what I need, right? But when the virtual machine monitor is running, what are the two things that I might need to allow to handle a trap? What's one of them, Jeremy? So in certain cases, I might need the host operating system to handle it, right? But what's the other, so again, normally application is running, it executes a trap, host operating system handles the trap. With the virtualization, what other piece of code might need to handle a trap that's generated when the virtual machine monitor is running? Andrew, sure. AJ, anybody wanna? So if the trap is generated by the virtual machine monitor code, then we want it to be handled by the host operating system. But Anurag, what happens? What's another case where I need to be careful? Right, so if the trap is generated by an application running inside the virtual machine, then who should handle that trap? Peng. The guest operating system, right? So on some level, you can think about it as now I have multiple operating systems running on the machine, right? There's still someone who's in charge, right? So who's in charge here? Who do I always want to make sure that they maintain a charge? Yeah, Amit. The host OS, right? So I don't know, maybe you could think about this as sort of delegated management or something, right? The host OS is the big boss, right? Like he is in charge or she is in charge of making sure that everything on the system runs safely, right? But what the host OS is going to do is going to delegate in certain cases the ability to handle traps to the virtual machine monitor, which is going to use the guest operating system code to handle those traps, right? So some traps, again, when you guys are running Ubuntu on your system for this class, some traps that are being generated by your code are being handled by Ubuntu, some of them are being handled by Windows or Mac or some other flavor of Ubuntu that you're also running, right? So that's one way to think about it. The traps are now have to go into two different places, right? So we already talked about if we run the guest OS with kernel privileges, then everything works as I would expect, but this violates our safety property, right? Now, what happens if we run the guest OS with user privileges? What is going to happen? Alyssa, right, but what's actually going to happen, right? So I'm not going to tell, this is the trick, I'm not going to tell the guest operating system that it's not running with kernel privileges. I'm just going to let it run as it normally would and then at some point, what's going to happen? Lovely. Right, so remember, if I took an operating system and I ran it without kernel privileges, eventually we'd do something that needed kernel privileges and what would happen in that case? What would happen if an application tried to do it? Same. Right, it would generate an exception, right? Because the CPU is going to say, you're not allowed to do that. You don't have privileges necessary to execute that instruction and so it's going to jump into the host operating system. On some level, the safety feature becomes really nice for us, right, because this gives us a chance to take control and make sure that these things get handled properly, right? So these privilege instructions are going to trap, right? And then ideally, what will happen is it will give us a chance to handle them properly. So virtual machine need to modify. Yeah, so this came up yesterday. I don't think this is how this works, right? I think the exception handlers, so instead of allowing the virtual machine monitor to modify the exception handlers, I think what's done is that the kernel passes certain exceptions to the virtual machine, right? So the host OS will take control briefly, right? But then the host OS will say, OK, this exception was handled inside the virtual machine, right? So I'm going to actually pass that to the virtual machine model, right? I think there's a variety of reasons for this most simplest being that allowing the virtual machine monitor to overwrite the exception handlers will probably create a whole bunch of other safety problems, right? So this way, the host OS always gets control briefly, right? But instead of handling the exception itself, it says, OK, this virtual machine monitor was running, right? And so I need to send the exception to the virtual machine model handle. So essentially, this is what we want to happen, right? So the CPU is going to trap the instruction because I violated privilege level, right? I tried to run this instruction that needed kernel privileges, and I didn't have them, right? What I want is the trap to be handled by the virtual machine monitor. And this gives us a chance to check. So remember we said before, most of the time, these, so the kernel that's running, the host, sorry, the guest operating system kernel needs to be able to execute these instructions because that's what it does, right? It needs to be able to control the hardware that's been assigned to my virtual machine, right? This process gives the virtual machine monitor a chance to inspect what it's doing and make sure that what it's doing is OK. So go back to our example of modifying entries in the TOB. What would the virtual machine monitor be checking when the guest operating system modified a TOB entry? Well, what would I probably want to make sure of? Great. The question is, let's say, so we said before that we can't just allow the guest operating system to do something like modify a TOB entry, right? That would be unsafe. So now, assuming that the VMM has a chance to inspect its modification of the TOB, what would the VMM be checking for? Well, OK, so I'm going to be reading and writing to the TOB, but what am I checking about the content of those reads and writes? Jeremy? Well, not necessarily it's going to user space. Yeah, that whatever memory it's trying to point to, whatever physical memory it's trying to point to, is physical memory that I've allocated to the virtual machine. So the guest operating system isn't trying to, either because it's buggy or because it's malicious, see parts of the system that it shouldn't be able to see. This is essentially a version of the protection that the operating system already provides to applications. I don't want unintentional sharing between two applications. I don't want an application to be able to see into another application's address space. Normally, that's not a problem at all because the operating system handles all of the VM faults that are generated by the application. And so assuming the operating system is correct, I can have nice isolation between applications. But now I'm doing something weird, which is that I'm giving an application the ability to modify entries in the TOB. That's one way of thinking about it. And so somebody, in this case the virtual machine monitor needs to check to make sure those modifications are legitimate. Yeah, I don't, yeah, this gets, especially on x86, this gets really ugly and very odd. You actually, on x86, there's this concept of what are called shadow page tables that are used by the guest operating system. I don't want to get into that. I think that, again, that would be the x86. VM virtualization on x86 is worth several lectures. And it really gets to be mind blowing in terms of how many different levels of indirection get involved. So if you want to look into this, go Google shadow page tables, x86. And there'll be plenty of examples of people trying to explain it. But it requires a fair amount of thought to figure out what's actually happening. Yeah? Do a lot of these principles apply to each other? I mean, to some degree. I mean, emulating or simulating a system is a different process. In which, again, remember, our goal with virtualization is to, as much as possible, allow the instructions to run on bare mouth. So that's a good question. So what's one limit of virtualization that something like, then I would be able to get around in potential cases using something like simulation? Let's say I have my x86 machine that I've bought and I've gotten VMware and I'm really excited and I'm going to set up some virtual machines. What's going to be true about all those virtual machines? Yeah, Sam? And so what does that mean about them in this particular case? That's true, and that's what I'm getting at. But what can I not virtualize? Summon. They all should be x86. Yeah, they're all going to be x86. They're all going to have the x86 architecture. And so I can't change instruction sets, for example. I can't virtualize an ARM processor without doing a lot of extra stuff. So there I would have to run some kind of simulator. And then things would get quite slow. So that's one thing to keep in mind with virtualization. I can't, I'm limited by the hardware I have. And that's just simply because of speed. I want things to run fast. I want to run them on bare mouth. So we talked about how some of these instructions have this property where they're able, so I can essentially use the privilege mechanism of the CPU to trap these instructions, allow the virtual machine monitor to inspect them, and then allow them to complete assuming that they are valid. And this is called trapping, anyway. And if I can do this, essentially what I can do is now I can run my guest operating system code safely inside the virtual machine by simply deprivileging it. So taking away kernel privilege from it, which forces all instructions that would modify state that could be dangerous to flow through the host operating system and eventually through the virtual machine monitor. So this is what we can do. We talked a little bit about what has to happen. We just talked about this earlier. So if the trap is caused by an application, then the trap has to be handled by the guest operating system.  If the trap is caused by the guest operating system, then the trap is caused by the guest OS, then I want to adjust the state of the virtual machine. The trap might also be caused by the virtual machine monitor itself, in which case that has to be handled by the host operating system. So it's actually three different ways that I can handle traps. And this is the other thing to point out. So in order, when you install VirtualBox or VMware, you're asked to approve Windows pops up that dark dim screen of danger. And you have to say, this is OK, because essentially what it's doing is installing some drivers that run with kernel privileges, because this is not something applications normally get to do. This is something that I require privilege for. And again, so we can go through this in a little bit more detail. And this is, again, an example of what happens when an application inside a virtual machine makes a system call, right? When I make the system call, I jump into the host OS. The host OS passes that trap into the virtual machine monitor, which eventually passes it into the guest OS trap handling code. When that finishes, then we can return from exception works the same way. Yeah, so this is just another example, where I'll just flip through these. There's one thing I want to, well, actually, this is probably a good thing to walk through, right? OK, so a process inside the virtual machine creates a TOB fault, right? So the first thing I'm going to do is I need to trap into the host OS, right? That's where all the traps are going to go, right? The host OS is going to hand the trap to the virtual machine monitor, right? The virtual machine monitor will inspect the trap, see that it was generated by the application, and now it knows that the guest OS has to handle this trap, right? The guest OS, and now this is interesting, right? Because the guest OS will start to run, right? Now, keep in mind, the guest OS is running in user mode, right? The guest OS is not running with kernel privileges, right? It's running in user mode. It's going to try to load an entry to the TOB. What's going to happen? At this point, mukta. It's going to trap again, right? Because this is a privileged instruction, right? And it cannot be executed. But the host OS is going to hand the trap back to the VMM, and then the VMM will inspect the trap, see that it was generated by the guest OS and adjust the state of the virtual machine appropriately assuming that this modification of the TOB was valid, right? So you can say to some degree a little bit of the overhead that's caused by this process, right? So normally, if this happened on a machine running on bare metal, how many traps would be generated here in the process of a page fault, right? Or a TOB fault, in this particular case? Kabina. What would be the two faults that would be generated normally? An application generates a page fault that needs to be handled by the operating system. So if I was running on bare metal, not inside the virtual machine, right? I would trap, what would happen? I would trap into the operating system, operating system wouldn't handle the fault, and I would start to run again, right? So how many traps would be generated? Just one, right? Just trapping the guest into the operating system, right? Here now I have two separate traps, right? So, and remember, every time I do this, there's a considerable overhead in and out of the curl, so this becomes difficult, right? And this is kind of just an interesting point, right? So hardware itself has an interface, right? And that's essentially what we're virtualizing here. I won't go through this very carefully, the slides are online. Yeah, so you can compare this with virtual memory, right? So with virtual memory, we talked about the interface that we were virtualizing was the memory interface, right? Loads and stores, right? In order to make sure that things happen safely, we never allow processes to use hardware addresses, right? We create this virtual address space for them, and we never allow them to access physical memory directly. And the way that we get performance is by using all of these caches, right? We cache entries in the TOB and we use the hardware to help us make these translations fast, right? With hardware virtualization, the entire instruction set of the processor is what we're trying to virtualize, right? That's a way to think about it. We're trying to, you know, rather than allowing you to execute this directly on real hardware, we're trying to virtualize the whole interface, right? And for some architectures, that interface is pretty big, right? You know, like some of the older X86 machines had really big instruction sets, right? We ensure safety by making sure that we rewrite or intercept any instructions that could modify state that they're not supposed to, and the way that we get good performance is just trying to by running everything else as much as possible on the bare metal, right? Using the hardware directly. So this all sounds, you know, to some degree, trap and emulate sounds very nice, right? It sounds like a nice approach. Unfortunately, and this is sort of, you know, what gave rise to VMware and a bunch of other different approaches to virtualization, the X86 architecture was not, you know, what we would call classically virtualizable, right? All the unsafe instructions, we could not trap and emulate, right? There were two problems, right? Some instructions didn't trap correctly, right? So what do I need to be true for to allow me to trap and emulate, right? Remember the idea is I'm gonna take the GuestOS and run it without kernel privileges, right? But what do I need to happen? Spencer? So control needs to go, right? Essentially, that's the process of generating an exception, but what do I need to be true about the instruction set? So remember, there's some instructions that the GuestOS is gonna try to use that it's not allowed to use, right? But what do all those instructions need to do if I'm gonna be able to trap and emulate? If I'm gonna be able to trap and emulate, all those instructions have to what frame? Maybe this is too easy. They have to trap, right? You know? So what else could they, well, this is a good question, right? And maybe it's not, you know, not obvious until you think about it, right? So I take an instruction and in kernel mode, that instruction executes cleanly. Now I go to user mode and that instruction could trap, right? Some of these instructions do trap. That's what happens, but what else could it do? What else might happen if I run into user mode? Well, right, it could not trap, right? But then remember, the point is this is potentially an unsafe instruction. It can't complete. So what else could happen, Tim? Well, in order to do that, I would have to do what? Trap, right? So what else could happen? No, again, I have to, I remember I have to generate an exception in order to kill a process, yeah. It could fail, right? It just might not do what it was supposed to do, right? So exactly, like there are certain instructions on X86 that if you ran them in user mode, either they did something different, right? So they had different semantics in user and kernel mode, right? Or they just, I mean, essentially they didn't do what they were supposed to do in kernel mode and they didn't generate an exception, right? So in that case, we're in trouble, right? Because if I put the guest operating system down into user mode and run this instruction, I don't get a chance to fix it, right? I don't get a chance to modify the hardware of the virtual machine to reflect what that instruction should have done because I never see it because it ever traps, right? Another problem was that some of these, some of these X86 instructions had different side effects, right? When they ran a user or kernel mode, right? So an example of a side effect is I run an instruction and it like sets some bit in a register on like the coprocessor or something. There are instructions on MIPS that you guys are using to do this as well, right? And sometimes again, these instructions might still trap but they didn't generate the side effect properly, right? And there may have been no way to emulate the process of generating the side effect, right? So yeah, this was gross, right? And to some degree, the reason this was true was because when the X86 architecture was being designed, no one really thought about, hey, let's make it classically virtualizable, right? They were like, let's make it work. So there were some things in it that didn't fall into this category, right? And what VMware did to make this work, right? And we're not gonna go into this in detail, right? Was they developed the solution which essentially uses binary translation, right? So what they did is, as your code, remember, your code is running inside this virtual machine, right? The virtual machine monitor has full access to what your code is doing. As it's running, what it's doing is it's rewriting the binaries that are being used by, in particular, by the guest operating system, right? Because the guest operating system is gonna be the piece of code that's gonna be using these instructions that it needs to identify, right? So if I ran the guest operating system by itself, I would get some traps. There'd be some opportunities to do things properly but there's other things that wouldn't work, right? So what it would do is, on the fly, it would be looking at the code, finding these types of instructions that weren't classically virtualizable and rewriting them to series of instructions that would work right, right? You can imagine this is hard, right? Like gross, maybe, but they got it to work, right? This is pretty cool. And the other thing, of course, is once they've translated pieces of code, they don't re-translate every time, right? But there is overhead the first time you execute a new piece of code that VMware hasn't seen before because it has to scan through it in real time, locate all these instructions, rewrite them dynamically, right? Sometimes rewriting them means I have to change the addresses that things jump to and stuff like this. So there's all sorts of sort of subtlety here that makes this much harder than it might sound. It already sounds hard, right? It was, what, right? All right, so that's basically it for virtualization.  So there's a lot, and there's a lot of sort of interesting hardware, software, co-design, and sort of co-evolution that's taking place, right? Certainly haven't talked about privilege rings, you know, again, somebody brought up this issue of how do I handle page tables, especially on x86, right? Because x86 had a hardware managed TLB, right? So that got even weirder, right? But there are great resources, especially for primary and shadow page tables. That's something that's actually fun to look up at. And if you can get your mind around primary and shadow page tables for virtualized x86 architectures, then you really understand virtual memory at that point. That's a good sign that you really got it, right? Okay. So this point, does anybody have any questions about virtualization before we do some exam review? Maybe this is a good time for everybody to get up and stretch a little bit and go have more coffee and donuts and stuff like that. I know it's a marathon class today, so. What do you want to do? We can do a free form Q and A session. We can just go through a review of some things that we've covered in the second half of the class. We can look at the sample exam. How many people have looked at the sample exam? Does anyone want to look? Maybe we should look at the sample exam. Again, I have a long laundry list of topics here that we could talk about, but why don't I pull up the sample exam and we'll look at that together? Yeah, so I'll repeat this. The exam is cumulative, the multiple choice and short answer. So the format's very similar to the midterm with the addition of a couple of long answer questions. Format essentially, I think, is the midterm plus like one or two larger, longer questions. So it's the midterm and then there's two hour long, longer questions. And those longer questions are intended to be kind of, I don't know if this is actually a word, integrative. So they're intended to be kind of fun, but they're also intended to kind of force you to bring together some concepts from throughout the semester, things that we've talked about. So, all right, here we go. All right, here we go. Yeah, so this was last year's exam. It's 10 multiple choice questions. These are, again, from the second half. Six short answer questions. This is six, choose four. It's similar to the, again, very similar to midterm. So two medium answer questions. So you're supposed to answer one and that's worth 20 points. So that's essentially the first hour, you know, let's see here. I guess 100 minutes. How long is 100 minutes? How long is the exam time? Does anybody, it's three hours? Okay, so I guess I was just generous last year and gave 100 point exam in three hours. So, I mean, I want people to have time to do it. So this is the gist of it, right? So the three long answer questions are actually worth 25 points each. And those are, those integrate material from the entire semester, all right?  So this is an example of a short answer question, you know, describe how to move from one location on the disk to another. These are appropriate terminology. So, and this is, but let's look at some of the more fun ones, right? We can talk about these. I'm also promised that I wouldn't cover too many of these. All right, let's see here. Yeah, okay. So, so this is one of the medium answer questions from last year. So essentially the idea is describe how you would fork a virtual machine, right? So we've talked about fork with operating system processes, right? Fork is the process of making a new copy of an existing process, right? The idea being that the new copy is identical to the parent, but then sort of, you know, goes off in its own way and has its own fate, right? So you can apply the same conceptual idea to virtual machines, right? Which we just got done talking about. So in this case, I have, you know, one virtual machine when I start and then I call fork and I should essentially have two identical machines, right? So this, you know, this similarity now extends all the way down into the, you know, the operating system and everything. So it's almost like I took a snapshot of a machine and I rebooted that snapshot, right? But so, so right, so let's see here. So, you know, describe virtual machine fork at a high level, right? You know, why would we do this? And also like some of the ways of amortizing the overhead. So how many people have used virtual mint like VMware, like some of the VMware server software or things like this, right? So you can essentially almost, I think you can basically do this today, right? You can take a snapshot of a virtual machine and you can create clones of it, right? That are based on that, right? The trick with fork, right? Is that operating systems do some interesting things to try to optimize fork, right? But why would I want to do this anyway? Like describe it, maybe someone should just describe a use case for being able to fork a virtual machine. So what would be a case where I might want to do this? Okay, yeah, maybe I want to, I want to like try installing some new software, right? Or try, you know, implement some something else. Yeah, okay, I might want to migrate the machine. Yeah, and what else? Okay, getting warmer. And in particular, what might I want uses to do? Okay, but think about fork itself, right? What types of programs frequently will fork and want identical? So I could have like a fork and exec paradigm here, right? Where I fork a copy of my virtual machine and then what would exec be? What would be kind of the equivalent to exec after I do a fork of a virtual machine, right? What's that? Starting it maybe, but what does exec normally do? It's a good review, what's that? Well, it's not all it does, right? Yeah, exec would be kind of like completely re-imaging the machine, right? From top to bottom. And then the idea of doing fork would be kind of dumb, right? Like why didn't I just create the new virtual machine I wanted in the first place, right? You know, this is like, we already have this question when we talk about operating systems and fork. But now we're talking about potentially forking an entire virtual machine, right? So the overheads there are potentially pretty prohibitive, right? So I would say you wouldn't implement a fork exact design pattern in a virtual machine. It's just too dumb, right? Like it's a huge amount of extra overhead. And then like, so what's the other, again, this is good review, so what's the other, what's the other reason that we support fork and exec? Jeremy, let me come back to that answer. But the fork exact design pattern in operating systems, so exec is going to completely blow away the address space that is inherited from the parent and the child is gonna go off and do something else. But what's different on an operating system between fork and exec and just starting a new process from scratch? Yeah, but once I run exec, then I don't care, right? Exec is gonna blow away the address space anyway. So after exec, you know, until I run the exec, yeah, I might get some sharing benefits, right?  Well, what's left over, Spencer? Right, yeah, exactly. So I've created a relationship between two processes and some of the system calls you guys have implemented exploit that relationship, right? So wait, you know, frequently has semantics that are derived from the relationship between processes. Those processes share their file table afterwards so they can potentially communicate through shared file handles and things like this. So there's this little remnant, again, even if the child immediately does an exec, right? There's still this relationship between these two processes that's created by the fork exact combination, right? I don't know how you would map that down to fork exec on virtual machines, right? Maybe there's an analog, right? I don't know what it is, right? But in general, the overhead of doing fork and exec could be quite prohibitive on a virtual machine, right? So, but again, so let's come back to Jeremy's answer about fork, right? So, you know, what would be, you know, so do some sort of distributing computing, right, why would I want, why could I potentially want another copy of the same machine? That's not what I'm thinking of, yeah. Well, I've got the machine doing something, right? But then I'm saying, why would I suddenly want another copy of it? Yeah, Tim? Yeah, like for example, you know, the day April 11th, sorry, May 11th, right? We will probably create several more grading machines, right? Because there may be a little bit of a queue that is formed, right? Of assignments that need to be graded and I don't want them to take, I've already created one copy. So I have a machine sitting there up on our VMware server and it's great.  So, it's pretty cool, right? So that's one, you know, reason that you can do this, right? You might want to respond to load. So, let me do it, all right. Yeah, so, and then again, so explain, we've talked about this a little bit, right? So, you know, so one reason this might be a good idea would be that I might be able to respond to load effectively. What other thing might I be able to do if, you know, after a virtual machine fork? So somebody already brought this up already. It's an optimization that we do after fork on modern operating systems to try to reduce the amount of shared, the overhead of the shared state. I don't know if we talked about this in lecture or not, but it could appear on the exam. Does anybody remember what this was? Nick? Yeah, that's not what I'm thinking of. So, what is, so if I do fork an exact, right, then I have these two completely separate processes. But let's say I have something like a web server, you know, particularly in the pre, sort of pre-threading era that is going to, that's going to call fork to create several other web servers that are all going to work together to handle incoming traffic, right? I'm going to multiplex new connections or load across these multiple application servers, right? So, what's true about, you know, so that when the process starts up, it calls fork a few times and now there's four copies of it running. What's true about those processes? So, if I did fork in the most straightforward way, I take a process and when I'm, let's say the process has n pages in its address space, right? How many pages do I have after I do fork in the most naive way possible? Four? I have n pages in my parent and I create a child. How many pages does the system need to support now? Jeremy? Two and, right? Every page in the parent has to appear in the child's address space, right? But how many, so first of all, at the instant that I do fork, how many distinct pages do I have? Just n, right? Because every parent page is going to be copied into the child's address space, right? So, if I do fork naively, I have two n pages, but I really only have n pages of content, right? And let's continue this analogy. So, when the child goes forward and starts to run, right, now the child and parent are independent from each other and their address spaces are supposed to look different, right? Now, we had this example of the child says i equals two and the parent's is i equals three, then those two i should be different, right? But over time, how many, so at the moment I do fork, I have n distinct pages. Let's say I let the child run for a few minutes. How many distinct pages do you think I have then? The maximum is two n, right? But in reality, how many pages differ now between the parent and the child? Especially, let's say the child is not calling exec. If it calls exec, then it's a completely new address space, right? There's no point of continuing this analogy. Let's say it doesn't call exec. Let's say it just keeps on running, handling connections. How many pages will differ between it and the parent? This doesn't have to be like a super scientific answer. Yeah, I agree. Yeah, yeah, but in general, what do you think? What is your intuition? Yeah. So what's gonna change, right? My stack, right? The stacks of the thread or threads that are running and my child will change, right? The data section of that child will change. What won't change? What parts of the address space are not going to change, probably? The file tables might change, right? Especially if I have sockets, I'm opening and reopening things. The code sections, right? Code, any static read-only data sections that I have. In general, it's possible that those two address spaces still share a huge number of pages, right? All of the code section for like a big binary, all the libraries that they've loaded, right? Anything that's read-only, and in fact, even a lot of things that are read-write, right? Don't necessarily get written to again. And operating systems optimized for this by using a technique that's called copy on write. And this is something that was done partly to optimize for this fork exec design pattern, right? Because if I do fork exec, if I do fork and I copy all of the address spaces, now I've done all this work, right? And the child comes along and just destroys all that work I did and reloads a new binary, right? It's like if you're going to sell your house, right?  And just like start over from scratch, right? You'd be like, well, I wish they would have told me that, right? I would have done all that work. So in order to avoid this, what operating systems will do is they'll say, okay, actually I'm not gonna create any new pages for the child. I'm gonna create a child address space, but all of the virtual addresses, so in that case, how does this work, right? Let's say I want to do fork, but I am not going to create any, I'm not gonna copy any data from the parent to the child. Where is the child going to get its data from? I need to have a child virtual address space, right? That's required for the child to run. Where are the virtual address spaces and the child's address space going to point? Gina? Yeah, so remember, at the moment I do fork, the parent has an address space. It has some virtual addresses that are math that are in physical memory, some of them may be on disk or whatever.  I don't wanna copy a single page. I do not want to copy content from one page to another page. What can I do instead? Well remember, yeah, so the child's virtual address space has to be able to map the same addresses, right? And when I map them, the content should be the same. But what can I do to avoid all this copying, right? And what are the conditions associated with it? Yeah, so okay, so that's exactly right. So remember you said, I'm only going to make a copy of the page when I need to, right? So I'm gonna do this on demand. This is another one of these sort of procrastination is potentially useful sort of operating system design techniques. When do I need to make the copy? When do I have to make a copy of the page? When do I not have to make a copy of the page? That's a good question. Hey Jay, when do I not have to make a copy? Okay? Okay, if the parent process makes a change, what, who else do I need to trap changes by? Yeah, so what I'm gonna do is I'm, for every page in the parent's address space, that page is now going to be shared with the child. So, and again, you might think, wait, hold on. You know, we said we weren't gonna do that, right? The parent and child, they have separate lives now, right? The parent is not meddling in the child's life anymore. The child is off on its own. And we're still gonna make sure that's true, right? But initially what we're gonna do is that all the virtual addresses I create in the child are gonna point at pages that are in the parent letter, that are essentially owned by the parent, right? Now, when either the parent or the child makes a change, right, I'm going to create a new page, a new private page for the child, right? This is a technique that's called copy on write. When there's a write that's done, I'm going to make a copy of the page, right? How do I make sure that I can do this, right? How do I make sure that, you know, so what's the challenge here, right? This conceptually, this sounds nice, right? I create these shared pages. And the idea is that, especially when I do fork and exec, right? What's gonna happen is the child's gonna start to run. It's not gonna change very many pages. And then suddenly it's gonna call exec, and I could just blow away that whole address space. And now I've only had to copy one or two pages. Maybe like maybe one page of stack or whatever. Just the page it needed when it ran, just a little bit farther before it called exec, right? Jeremy. Well, so the write function, but what kind of writes am I talking about here? It's not writes that are done through a write function. These are writes that are done how? What's the interface that I'm using to do these writes? It's a command. They're stores, right? They're memory stores. So what do I have to do after copy on write to make sure that I can see any changes that are made by the child or the parent to any of these pages that are now shared, right? Again, I'm sharing them, but I'm not, they shouldn't actually be shared, right? Writes, modifications to those pages that are made by the parent or child should not be seen, right? But how do I make sure that I see modifications to all those pages? What do I need to be able to do? Truck. So let's say I had a page in the parent's address space that was read, write. That was a shared data page or something. Now I've done fork, and I'm using this copy on write technique, right? What do I need to make sure I see? Yeah, it has to be marked as read only in the TLB. I have to make sure that I see any changes. If I just load the same old TLB entry I've been using for that page that allows it to be modified, I'm never gonna see the modification, right? So I don't know if the parent changed or not, right? And I have to do this for both the parent and the child, right? So I take these pages, I mark them as read only in both address spaces, and the idea is that as soon as either the parent or child tries to write to that page, an exception will be triggered by the TLB. The kernel will run, the kernel will say, aha, this is a shared page that I created after copy and write. I need to make a new copy of the page and then I reload the TLB with the read write entry pointing to the new copy of the page, right? So the idea is the pages are shared until either the parent or the child writes to them and at that point I make copies, right? So again, I'm deferring as much of the work as possible in order to avoid having to do it at all, right? Yeah. Yeah, so you have to, yeah, this is a good question, right? So some of the pages in my address space may have been marked as read only when I forked, right? So I need a different notation in my PTEs, right? To indicate that the natural permissions of this page are read write but I marked it as read only so I can do copy and write, right? So yeah, I need to be careful about that. But this brings us to the other nice thing about copy and write. So again, so we talked about before how the fact that first of all, address space is usually contained a lot of content that is not written to, right? So let's say I have a couple of, like I'm using copy and write and again, go back to our web server example. My web server forks a couple of times, creates a few new copies of itself. A couple of minutes later, what do I expect to find when I look at that little family of processes? What will they be doing that will make me happy? Well, so they're allowed to do, they're allowed to be as, right, so because I forked, the children are allowed to be as different from their parents as possible, right? But this is like the little family of doctors or something, right? Or lawyers, okay, so what do I expect? Again, they're all web servers, right?  What's that? I think that's on the right track. So again, so this goes, you know, there's something that I will be happy that they will be doing in general probably, right? It involves read operations. So I'm gonna use copy and write, right? And what that means is that when they write to those pages, they'll get their own private copy, right? But remember, I could have created four N pages, right? For this one parent that created three children. I could have had four N pages that I would have to keep in potentially some portion of them in physical memory, right? But after they've been running for a while, what do I notice about them that makes me happy? I don't know, you were gonna ask me. I think maybe you guys are onto this, but I'll just say it. They're gonna be sharing a lot of pages, right? There's a lot of pages in their address space that I will never have to copy because no one will ever write to them, right? Maybe there are data pages that are actually marked as read write, but just nobody ever writes to anymore, right? It's possible, so a parent might set up some data structures that are, you know, might gather some information about the machine, write some variables, that's a data page that is potentially rewrite, but nobody ever writes to it. So after fork, there's no new copies of that page that I've had to create, and all the code and data, sorry, all the code and the libraries and stuff like that, all that stuff was read only to begin with, so I'll never have to copy that, right? So the nice thing about copy on write is that it optimizes for exact very well, right? Because I don't have to do the full copy, but even if processes go on and keep doing what their parent was doing, they end up sharing a lot of state, right? And so all those libraries are now only loaded once rather than end times, right? And then there's actually some other games that operating systems try to play to identify shared pages even across multiple processes that didn't start together before, right? So for example, you know, what's an example of the type of shared page that the operating system might be looking for, right? A type of page that would be probably loaded in almost every address space of any process that's running on the machine that I would be very eager to find redundant copies of and condense them down into one. What is probably every process on your machine using? Yeah, the C library. You know, like I'm guessing that, I don't know what the numbers are, but it wouldn't surprise me, it's like 80, 90% of applications are using the C library somewhere, right? You know, so, hey, libc is the same, right? And maybe there are different versions of it, right? But if I can collect versions of libc together, I can get rid of these shared pages because it's read only and so all the processes can now show, right? So anyway, let's go back to this virtual machine fork. So what's the kind of analog here of copy on write for VM fork? I'm ignoring you Jeremy. What would be something here that would potentially be an optimization that would be based on the fact that I've initiated this from within one of the virtual machines? Yeah, actually. Yeah, okay, but they can, so the idea here is hopefully maybe I can have some sharing, right, of memory. What memory can I share? You just have to bump it. You just take the copy on write concept and just bump it up one level, um, what's that? Sure, okay, so that's, yeah, that's true, but that's probably, that's probably not a huge contributor to the overall overhead. Yeah, what about the GuestOS? You know, same GuestOS running in every virtual machine that I create through fork, at least until maybe I install a new one, right? That's kind of like the equivalent of doing exec, right? But at least initially, a huge amount of the code pages, right? Operate systems are big programs, right?  So that's a nice thing, right? All right, any questions about this question? We'll go on and look at one of the long answer questions in a second. All right, let's look at one of the other ones, where am I? Ah, okay, I like this one. It's so sad, I really like these questions last year. I hope I can come up with good ones again. All right, all right, so here's, yeah, actually, sorry. Yeah, so this came up, so I was talking about somebody during the break about this. Virtual memory, so memory allocation between VMs is potentially quite problematic, right? And then it could be a good exam question, and there's variety of reasons for that. Sometimes, on certain VM approaches, they just give a chunk of memory to the virtual machine monitor, and they say that's yours, right? It's yours to manage, and I'm never gonna take pages away from you, right? In other cases, they're, so, okay, this is a good question. Well, I don't really wanna get into it. Talk to me after class, but it's an interesting question, but in general, there is, so the virtual, you know, the host operating system, or in some cases, you know, what's called the, you know, or in some cases, the virtual machine monitor itself, if it's supporting multiple virtual machines, has this challenge of how to allocate physical memory between those different virtual machines, and doing that well is a little bit of a challenge. All right, it's a good question. Any other questions before we go on to this? All right, so, okay, so who can describe demand paging? Who remembers what demand paging was, why we did it? Yeah, yeah. So on demand paging, right, when it's necessary, but as opposed to what, right? What would be the other thing I could do when a process starts to run that I distinguish from demand paging, okay? Yeah, so when exact actually runs, right, and when I call load elf, right, or when I process the elf binary, I could actually read every page out of that binary, and I could read all of the libraries, right, and I could carry copies of them and carry them all into your address space and put them out there all very nicely so that 100% of the code, at least the statically linked stuff that you've asked for is there. It's all there, right, and then I could start you off running and that would be nice, right? Why don't I do that? Yeah, because there's a lot of stuff in your address space we're not gonna use, right? And so what I'm gonna say is, hey, I don't know, maybe, I don't know, maybe I do, I don't know a lot of the pages that you're gonna use because a lot of it, I mean, why is that true, right? Is there just dead code lying around in the address space? What, this code that we're talking about that's not being used, what does it, what does it represent, Matt? Not sure, Sean. Yeah, so that's one, but what is it, what else is it? Summit. Yeah. So that's a canonical example, right? I mean, Word has at least six or eight ways to do everything, right? And I know one of them, or I probably don't actually, but I could, right, and you would know another one and if somebody asked us to do this thing, like I would do it my way and you would do it your way and neither of them would work perfectly, but the idea would be that if you looked at our code paths through the executable, they'd be totally different, right? And so the pages that I need and the pages that you need are quite different, right? Sharon, you have a question? Yeah, so the code exhibits locality, that's another good point that we can come back to here and actually we're gonna explain something even more interesting, right? So, okay, so anyway, but what's the downside, right? What's the downside to on-demand paging, right? On some level, the upside is there's a large amount of your address space that I never have to, I never have to put anything into, but what's the downside, right? Yeah, I generate a lot of page fault, right? And what else do I, so the fun thing about this question, I think, was that this doesn't interplay with disk scheduling, right? So let's say I wasn't going to do on-demand paging, I was just gonna take all the pages you need from all the executables and all the libraries, I was gonna go get them all at once, right? What is nice about this? You have to go back to thinking about disks. What is nice about this from the perspective of the disk? We kill. What's that? Yeah, I can do great disk scheduling. Remember, disk schedulers wanna know as much as possible about what you're about to do, right? They don't wanna do random IO because that makes the heads go all over the place, right? But if I told the disk, here's every block you have to get from the file system for this program, what is it gonna do? How is it gonna schedule those? Nick, what's that? We'll do them all, I hope, right? I told it to do it, but how will it do them all? What will the heads do? Yes, it will, but sequentially how? Jeremy. Well, they're gonna have to move from track to track, right? Because stuff that I need is gonna be scattered all over the disk, right? Unless I've done like a fantastic job of disk layout. But in general, the stuff I need to run, you know, Photoshop, right, is gonna be all over the place, right? But what will the disk do if I tell it everything all at once? Yeah, sugar. Well, it's just some cache, and I'm assuming there's too much here, right? Yeah, so someone said it doesn't in sequential order, but what ordering is it using? It's using its ordering, right? I mean, you can imagine it would go to one end of the disk, and it would do a single pass across the disk, right? Their heads would never jump around. The heads would just move in one direction, you know, maybe out in, probably from outside inside, right? Because the data density's higher on the outside, remember that? So it would just jump to the outside of the disk, start reading stuff, and basically just slowly bring the heads in until it got everything. And so my whole executable loans in one pass, right? Awesome, right? From the perspective of the disk, that's sweet, right? However, again, we still have this problem, which is that we've now loaded all this junk that we don't need, right? Wembley, do you have a? Yeah, in the best case, right? So in the best case, I would have everything all in like one little location, and that would be awesome, right? But in reality, what happens is libraries get put in different places, right? And, you know, so all the code for that specific executable might be in one place, but once I start actually having to load libraries and things like that, I still have to do some seeking. Yeah, correct. Oh, yeah. So fragmentation is more of a layout question, but yeah, whenever the, remember, what's the largest contributor to disk latency? Moving the heads, seek times between different tracks, right? So if I took all of my data, and I just did it in a random order, right? I'd be seeking all over the disk all the time, right? If I tell the disk everything I want to do, the disk will sort it in a way that's efficient and avoid seeking, right? So it'll essentially do kind of one slow, continuous seek across the disk, right? Essentially, the more data I give the disk at once, the better the disk can plan its own travel across the platter, right? Jeremy. Maybe, yeah, that could be true too, yeah, but yeah, the overhead over the bus at that point is not the, the communication overhead to get the, to tell the issue of the command is probably not the biggest contributor, yeah. Yeah, that is what it is doing, right? So yeah, there are layout issues here too, but let's, but let's factor out the layout issues. Let's just say, you know, I've got some stuff to get. It's not all in the same place. I'm gonna have to do some seeks. The fewest number of seeks I can do is if I know as much about what I need to get as possible. The best way to know everything the process could ever need all at once would be to just load everything when it starts up, right? But again, so now I have this problem, right? Which is that my on-demand paging, right? Gets me way more, sorry, so my on-demand paging does a great job of not wasting memory, but it potentially leads to very, very bad disk IO patterns. However, if I don't do on-demand paging, I can do this nice disk IO pattern, but I waste a lot of memory. So is there kind of a middle ground, right? Is there a way to try and get at least a little bit of the best of both worlds, right? And the way to answer this question is to ask yourself the following question. So when is an application's behavior, right? To some degree, remember the reason we didn't want to get all of the pages in the process's address space was that its behavior is unpredictable, right?  And so we don't want to get everything because we don't know exactly what's gonna happen in the future. But what is the period of time when the application's behavior and by extension its page access patterns are the most predictable? What is the, in general terms, if you have to look across all applications, what's the period of time during which they're the most predictable? I would say that's when they're the least predictable because then it's based on user behavior, right? When is the behavior of your application not based on user behavior? What's that? No, no, code execution again. The code paths that I start going through once the application is up and running are entirely determined by what the user is doing. Yeah, Nick. So start up, right?  Like there's not a lot of subtlety to it. I usually don't have to do any reads from the disk or any shutdown, right? It's just like, get off my operating system. But yeah, start up, right? You guys have started applications, right? How many people like started Photoshop, right? Photoshop's, like Photoshop sits there for a while doing stuff before it even allows you to interact with it, right? What is it doing? Photoshop starts, so again, let's say we're doing demand paging, right? I start Photoshop, what is Photoshop doing? Why can't I interact with it for a minute, right? Or 30 seconds or something. What is it doing? What's that? Yeah, it's like, it's starting up, right? It's initializing its environment, it's loading stuff in. And if I'm in my on-demand paging world, what is happening, right? If you could see the disk, what would the disk be doing? In purely on-demand paging, what would the disk head look like?  Oh, okay, now you faulted on this code page, right? It's just jerking all over the disk, right? Random IO over a big disk, right? Not good, okay? Slow and very, very poor performance. But what did we just say about this period of time? What's true about it? Right? It's predictable. Every time you start up Photoshop, it pretty much, you know, for a period of time, until you start to interact with it, it does the same thing every time, right? It's not taking any user input it's loading. Libraries, destroyed tooltips. It's, you know, checking for updates. Like, there's all this stuff that it's doing, right? So what can I do? How, who thinks that they have a way of exploiting this fact, right? The problem is that on-demand paging produces this very bad disk access pattern. The thing we've noticed is that when the disk starts up, sorry, when the application starts up, it's pretty predictable for a period of time, right? So how can I use that? Right, so what was one of our favorite? Or 21 mantra? Use the past to predict the future. Especially when the future is predictable if I know the past, right? That's the best, that's my best case, right? Otherwise, I'm just guessing, right? So when I find something that's predictable, then I want to be able to use that. So how do I do that in this particular case, right? What's the general approach, Robert? Yeah, and that's basically what it is, right? Everything that gets used every time the program runs during startup, I load it off the disk immediately when the program starts up, right? That's not everything, right? That is not the entire executable in all the libraries and everything, it's a subset of it, it's a potentially small subset, but it's the subsets that is completely predictable, right? When you double click on Photoshop, there's a set of pages that are going to be loaded. The probability that they are going to be loaded is like 100%, right? And so if I can identify what those pages are, then I can essentially tell the disk immediately I need all of these pages, right? And then what does the disk had look like during startup? One pass, right? Jeremy? How does the operating system remember this? Ah, how does the operating system remember this? Good, it's a very good question, right? And this is, I think, part of what the question was, right? So how does the operating system collect this information? Let's say, yeah, okay. Yeah, so when the application starts, and I don't know, I mean, there were a couple of iterations of Windows that did this, this may still be true actually, so it used to be you could go into your Windows laptop or machine, and if you ignore like the 18 warnings you get and you start poking around in the Windows systems folder, right?  Essentially what those files were are the output of essentially traces of these applications that were taken that were used to allow Windows to identify pages that were very, very likely to be used, right? So when the application runs, I start keeping track of the pages that get used, right? What's the problem here? What's a challenge with that approach, right? I ran the application once, right? I wrote down every page that, you know, every part, every piece, every block of the disk that it requested, right? But what's the problem? Right, like when does that predictable startup sequence end, right? And so there was some logic in the Windows pre-fretcher that used multiple traces, right? So you run the application once, maybe it has some heuristic that says okay, I only trace up to like a second, right? Or two seconds or something. Then you ran it again, it was like, oh, okay, well now I see that this is Photoshop and there's like 10, 20 seconds worth of predictable page access behavior, right? And as you ran it over and over again, it would continue to kind of refine that trace, right? And this is exactly what they did, right? So they, and once you have a trace that you're comfortable with, when the application starts you immediately go get all, all of those blocks from disk. And this improved the startup time of Windows applications tremendously, like a factor of two or three overhead, right? And this is significant when you're a user, right? I mean, two or three doesn't always sound like a super large improvement, but the difference between like 30 seconds and 10 seconds is pretty noticeable to people, right? Especially when they're sitting there twirling in their thumbs wondering why Photoshop still has its logo up and hasn't allowed me to start creating my beautiful picture. So what's another case where I can use this? Am I doing a question? No, that's exactly what it does, right? The Windows pre-fretcher created a per application. You could think of it as a profile, right? It's kind of a prefetch file, which said as soon as the application launches, I'm gonna take all of these pages and I'm essentially gonna tell the disk, go get all this stuff, right? Once that's loaded, then I do demand paging, right? Then as you start to interact with the application, different things happen, then I'm doing demand paging, but it's this nice mixture of non-demand paging, which would get everything, right? And non-demand paging, which causes this bad disk IO pattern. One of the reasons why, so we can connect this with another part of the class. What's one of the reasons why Circa, man, when was this? Circa 2000, 2001, engineers at Microsoft were particularly interested in these types of improvements. What sort of confluence of hardware trends was starting to take place then that would have caused them to worry about this type of thing, Frank? Yeah, you're, you okay? Man, emerged from the floor. There's a trap door in the room. Yeah, big, slow disks, right? At this point in time, I was interning at Microsoft and I was working in the desktop performance group and they were, one of their things they were terrified of was big, slow disks. People were getting these massive, 20 gigabyte hard drives, right? 40 gigabyte hard drives, they were big, they were slow, right? And because of that, the caching that the operating system was doing wasn't necessarily always keeping up because seek time was just to get stuff that started to die, right? So let me, let's just finish this question. So what's one other place where I could do this? What's one other place where I could apply this technique and it might give me a big win, yeah. Yeah, yeah, and they actually, they applied this to Windows Boot, same technique. Profile, send big batches to the disk and they got boot time down by a factor of two or three. It's pretty cool, right? All right, so I think that's it for me. There's just one other thing I wanted to put up before we leave, which is, I think we should, whoop, let's get this down. I think we should have a round of applause for the core staff. Only one of them is here, Edith, you'll want to stand up. They've all worked really hard this semester and they have a couple more weeks of helping you with assignments. So anyway, I will see you a week from today at 8 a.m. for the exam. Good luck finishing up the assignments and I'll see you on Monday.