All right, welcome back to 353. Thank you for joining me today. So today we get to start unraveling the mystery of virtual memory and how that works, which is one of the most important topics in this course. So if we think about it, how should we implement virtual memory mapping? So what are some ideas we have your ideas for actually implementing this? So remember, memory byte addressable, and I also want to be able to map multiple processes, virtual memory. So the same address and different process should be able to map to the same or a different physical memory, depending on whether or not I want to share that memory or not. Assuming everything's independent, each virtual address in a process should point to a different physical address. So how would I implement such a thing? Any ideas? Yep. Yeah, that one option, we could just block off some chunk of physical memory for each process and then do the mapping and maybe implement the mapping. Most people say like a hash table or something like that. Hash tables are slow, so you could essentially just like have an array of lookups. So you could like say, hey, this block of memory is associated with this physical memory. This block of virtual memory is associated with this block of physical memory. And that's basically how we do it. And we'll spend all of today unraveling that and all of next lecture as well. So remember our checklist for virtual memory. So the list of goals we want are we want multiple processes to be able to coexist at the same time and be completely independent. So even though they access the same virtual address, they should not share physical memory unless they kind of opt into it. And that's another way to do inter-process communication. But we'll assume we want things to be completely independent right now. Also, processes should not be aware of their sharing physical memory. So aside from maybe running out of memory, you shouldn't be able to read another processes memory or anything like that. That would obviously be some pretty bad security concerns. If I could just have any random rogue process, just start reading memory from, I don't know, my banking application or whatever. Probably a bad idea. So that's the third point. Can't access each other's memory unless you're explicitly allowed to. And that is, like I said, another way to do IPC and you can actually share memory, but too complicated for now. We'll just make sure every process is actually independent. The fourth point is we want performance close to using physical memory. So there's many ways to implement this. So like most people say hash table, but we want really good performance. Hash tables have to do a lookup and a hash function and all that stuff. You don't wanna do that for every memory access. It's gonna be really, really slow. And for some reason, people think hash tables are like some magical thing. A hash table is just an array. Hash table is basically just a compressed array. That's it. The only magic is that hash function. Otherwise, it's just an array. Also, we want to limit the amount of fragmentation so we don't want to waste memory. So if we have that block idea of just giving, each process a separate block of memory, we probably don't wanna give them each a gigabyte of memory whenever they're running, whether they use it or not. Otherwise, if my process only uses a few megabytes or kilobyte or whatever, I'm wasting all the rest of that memory. So I want to limit the amount of wasted memory I have. So remember, low level details from C and just computer architecture in general. Remember, memory is byte addressable. So an address is basically just saying, hey, what byte number is this? What byte do I access? And we can only access memory one byte as a time as like the smallest unit we have. We can't access individual bits without just reading an entire byte and then messing around with the bits and writing them back out. So memory one byte at a time should remember that at all times in any computing course. So just read and write one byte at a time at minimum each address. You can think of memory as just a gigantic byte array and the address is basically the index into that array. So, all right, it's what? Week four, we're walking in the wrong class. That's not good. All right. So each, yeah, so each address, it's like an index into a big byte array. Memory's basically just a big byte array, right? Any concerns about memory? We're all on the same thing. We all understand kind of what memory is because well, we have to implement virtual memory now. So good to understand physical. So back in the olden times, there was a way to implement it and it was called segmentation or yeah, just segmentations and they're not used anymore. I'll explain it just for the history lesson because anyone ever access invalid memory and you got a sag fault? Yeah, sag short for segmentation. Even though, funnily enough, we don't use segmentations anymore, but with computing, as soon as you expose a name to a user like a structure or anything like that, well, you can't change it. So any invalid memory access is called the segmentation fault just from now until the end of time because history. So how segments worked back in the day is they divided virtual addresses into like segments for specific uses. So like one segment for code, all of the instructions, another segment for data. So all of your global variables and everything like that. One for the stack. So any local variables you have in your function and then the last segment for your heap which is where your dynamic memory allocations happen. So this kind of looks like an elf file if we actually started exploring other elf files for whatever reason, they kind of look like this. They tell you where in that file is the code, where's the data, where's how much stack space to make, things like that. So each of these segments can be a variable size and you can dynamically resize them if you want. So they can be large and if you have to grow something, it's really, really costly to relocate it because well, you actually have to find a space in physical memory that's contiguous. Maybe you have to move all the memory there and it's really, really bad. So it leads to a bunch of fragmentation because well, if I have a gap in between two blocks of memory, well, maybe this gap isn't big enough to actually fit a new process running. So it's essentially going to be useless memory that I can't use for any purposes. So going to be fragmentation, fragmentation is just another word for wasted memory. So this is going to no longer be used in modern operating systems, but we get some things about it. So how it worked is each segment kind of looks like an array. So to describe where a segment is in physical memory, you tell it like the base address. So what address does this start at? What's its limit or size? Like how many bytes can I possibly are valid in this? And then some permissions. So like, can I read, write this memory? Can I execute it? Some memory permissions. So how is that? If you haven't seen the word offset before, it's just a generic term in computing just saying how many bytes from the beginning is something. So like, if I had, I don't know, an array of ints, well the first index zero would start at byte zero and an offset is always the byte number. So in that int array at index one, well that would be like the second element in that array, right? But it would begin since ints are four bytes, it would begin at byte four off. So byte zero through three would be the first element and then four through seven would be the next one, so on and so forth. So the MMU is the memory management unit, which we'll go into details later when we talk about real virtual memory, but that's basically going to check that this offset that you are selecting the segment is within that limit. So it's like a valid byte to actually get and then to get the physical address, it will just do the base address of that segment plus the offset you requested, do the permission checks and if it fails, you get a segmentation fault, otherwise you get a valid physical address. So for example, this would be your memory address, it'd be like segment one and then colon, I want byte ff is 255, so in your memory management unit that the operating system would control, it might have information like, oh, segment one, well the base physical address for that is 2000 in hex and it is one ff limit, so if we were to translate this, well ff is less than one ff, so it's a valid byte, so to get the physical address, it's just the base address plus whatever byte you requested, which in this case was ff, so our final physical address is gonna be 20 ff, yep. Oh, so the question is, why is the base two bytes in this one and a half bytes?  Yeah, the limit's just a hex number, it might be like the limit would be the size of that, would be dependent on like the CPU architecture you have. So like 32 bit memory, it would need to be up to 32 bits, 64 bit need to be 64. So the maximum limit is like, how much memory is allocated for that section? So like the address is 2000, all the way up to 2001 ff, would be the valid memory for segment one. Yeah, yeah, so Linux, if you look at the code because x86, it doesn't get rid of segments, so if you have Intel or AMD laptop, segments still exist in hardware, but the Linux kernel, whenever it boots up, the first thing it's gonna do is ignore them. So it sets every base to zero, just in case you decide to use this, because you could if you really wanted to, and then sets the limit equal to the maximum amount of memory you have.  So, yep. Why are they still there? So why are they still there? It's because x86, Intel, like everything has to be backwards compatible. So once it's there, you can't get rid of it, which Windows subscribes to x86 does. If you wanna get rid of it, you have to use an ARM processor. So our first insight we had was to divide memory into fixed size chunks. So I could just like have a virtual block of memory that's of some size, and I could just map that to a physical block of memory to figure out where that is in physical memory. And then even though some process might be using the same virtual block number, while the operating system could map those to different physical blocks, and then each process would be completely separate from each other. So our MMU or a memory management unit, that is the hardware on your machine that is responsible for actually doing the translation for you. So it will map a virtual address to a physical address, and also do all of the permission checks, like reading and writing, depending on what operation you want to do, because you can make memory that you're not allowed to write to. So one technique we can do is dividing memory into fixed size pages, and here the fixed size blocks that we had in the previous sides, while we call them pages when it comes to virtual memory, and the page size we've been using since the 70s is that magical number I used before, 4,096. So they decided that number was the perfect Goldilocks number to set the block of memory for, so we got good performance, and also we didn't have too much fragmentation. So some terminology you might see come up is a page, which is that fixed size block of memory for this course is probably typically always 4k or 4,096. A page in virtual memory, we just simply call that a page, you might see it referred to as a virtual page, and then that same page in physical memory, sometimes people call it a page, sometimes it's called a frame, whoops, my search and replace went bad, it's not called a slide, it's called a frame. Wow, that's a bad error on this. So a page in physical memory is called a frame, and I will fix that. So typically 64 bits for memory, that's a lot of accessible memory, so that's like what, like exabytes or something like that. So typically you don't really need all 64 bits, so you might have like different levels of virtual addresses you can use, but the implementation details are the same. So as we'll see as we go through the course, the more virtual memory you can actually map, the more expensive it is to actually keep track of that mapping, kind of makes sense, at least you'd need a bigger array. So we might wanna keep it small as useably, like small as possible so that it's still usable. So for most systems nowadays, we use a 39 bit virtual address space that's used by risk five, x86, ARM by default, and that allows every single process to address 512 gigabytes of memory. So if it was only a 32 bit machine, 32, so two to the 32, and that's how many bytes you could address, and if each, yeah, since each one thing you can address as a byte, that's four gigabytes you could address with a 32 bit machine, which is why we moved on from them because four gigs of memory isn't terribly special nowadays, and especially if you're using Chrome, four gigs is nothing. So had to move on to that. So right now we're at 39, and that system's called SV39 specifically for risk five, everyone else calls it a different thing, but this allows each process to address up to 512 gigs of memory, which should keep Chrome happy for like two more years. So how this is implemented is with a page table, and that page table is basically just a giant lookup array. So it's going to be indexed by a virtual page number, and what is stored in that array is the physical page number. So what that looks like visually is we'll have a virtual address here, that virtual address, while we could use up to 64 bits because we have a 64 bit machine, but since we're only supporting 39 bit virtual addresses, we're just going to ignore the top whatever bits. So we're going to split up that 39 into two sections. So the first section is called the offset, so that is where you are in a page. So this is directly related to the size of a page. So remember, our pages are 4,096 bytes, so in order to be able to address every single byte in that page, we need 12 bits, right? Because two to the 12 is 4,096. So the offset is directly related to our page size because it's telling you what byte you want to actually access on the page. So the other bits, so we just take 39 minus 12, so the other 27 bits are the index bits. So that's basically how many virtual pages we have to be able to store a mapping for. So in this case, we would have 27 bits, so in this page table, well, it would have two to the 27 entries, right? Because, well, powers of two. So this page table would have to have two to the 27 entries, so it would have to have a lot of entries, and each thing in this page table, its contents is that physical page number, and the size of that doesn't really matter. So it turns out the kind of only store 44 bits to keep track of the physical page number in this table. So whenever we need to translate an address, so if we have any virtual address, we treat these bits as the index, we would look it up in this page table, and then in this page table, we would get a physical page number, and then that would be the physical page number that the memory is actually located at, and then the actual physical address would just be the physical page number followed by the offset, because we don't need to translate the offset, it's where you are in a page, doesn't matter if it's virtual or physical, yep. So the question is how does it get to 27 bits to 44 bits? So the 27 bits here are just to use for an index into this table, and then this table, each element can be any size we want. So like, you can think of it, it could be like an array of ints, so then each one would be four bytes. In this case, an int would be too small, so it's basically an array of like eight byte entries. So each of them would be 64 bits, and turns out, I'll show you in the next slide why, but turns out we don't use all 64 bits, we only use 44 bits. So this is just a big array. So the page table is just a big array, and the operating system would be the thing that manages that, and the MMU just uses that to do the translations, yep. Yeah, so that's a good question. Do each process share the same page table, or do they have their own? So that each process would have their own page table. So that's how we can take the same mapping, because if you had the same virtual address and two processes, well, it would be the same entry in the page table, right? But the only way to make them independent is, I make each entry point to a different page, yep. So the unused part of this is just, right now it's just undefined, it's for future use if they decide to extend anything, right now it's just ignored on most machines. So there's some other space, because while it's a 64-bit machine, so this is what, 12 plus 44, that's 56. So there's six bits left over that aren't used, and they just say, we'll just leave it for now and figure it out what to use them for later if we come up with any other bright ideas. But that's just a implementation thing, yep. So do we, yeah, so the question, do we use binary or hexadecimal? They, I use what they both mean the same thing, like it eventually goes to ones and zeros. I use hex wherever it's convenient, and then whenever I'm forced to use binary, I use binary, yep, yep. But so the amount of 512 gigabytes of battery's memory that you mentioned, is that essentially the size of the index or the size of the offset that determines that? So how much memory you can access is just all of these bits together. So we take the address, so your virtual address would be 39 bits, right? And then the 39 bits get divided up like this, because while this is like which byte in the page and then this is what page, but there'd be in total two to the 39 bits we can address or bytes we can address here, which is everything we can address, yep. If, why are we using odd numbers of bits? So yeah, the question is, if everything's byte addressable on a computer, why are we using bits? So we're using bits here. Because you need to store that in a certain amount of bytes, right? So the bits here, the thing that actually operates on these bits is the MMU, which is hardware, so we don't actually have to operate on it. So the MMU does this translation. What we manage is this page table. So we manage this in software. And this is a nice size. It's also an odd number though. It's an odd number, but we'll see why in a second. We'll make it a nice number. All right, yeah. Also, by the way, that's like huge, right? That's a lot of entries to the 26. If anyone has a calculator and give me that number, that's a lot of entries. Probably not a good idea. So each element of that array in the page table, it's called a page table entry. So it stores a bit more than that physical page number. So what it actually stores is at least on this system, they're powers of two, they're nice. So each entry is eight bytes, which is 64 bits. So it turns out this is how it divides that entry up. So those 64 bits. So for this course, it has a bunch of permission checks. So there's has one bit here that says whether this entry is valid or not. So if this bit is a zero, it's not valid. If it's a one, it is valid. And then there's like some other permission checks, readable, writable. So is this page readable or writable? And is it executable? And that's probably all the flags we really have to be concerned about in this course. In fact, in this course, the only real flag we need to be aware of is this valid bit. It definitely has to be there. There's also some other bits on this architecture that make sure that, oh, this is only accessed in user mode. This is a global one. This is a dirty bit, which yet, don't worry about that term, like weird terms here. So we'll get up about what this dirty bit is later, maybe whether or not it's accessed. Then there's some bits here, which are reserved for the supervisor use. So there's two bits that the kernel gets to play with. And guess what, when you do lab three, and you actually kind of manage these entries, you get to use those bits for whatever you want as well. Then we have 44 bits, which is that physical page number. And then they decided to just reserve 10 bits for future use. If someone decides to come up with another flag or something later, then they can go ahead and reuse that without having to make this thing any bigger. But this is what the page table entry currently looks like. So the kernel, it handles like setting up the translations, setting up the page table so that the MMU can actually translate the virtual addresses. So the kernel would manage something like this page table, and this page table would be just a giant array. So I'll just write out the index in this left column, just to make it a bit easier to read. And then each entry, I'll just say the PPN number and assume it's valid. So I'll say that in the right column. So if I have this, well, if I do the translation between a virtual address to a physical address, say this is my virtual address on the left, turns out because if we go back, this offset, we don't have to translate it, right? It stays the same. It's just what byte on the page we want to access. Turns out it's a very nice number. So 12 bits, each hex digit is four bits. So essentially the last three hex digits, we don't have to translate if this is our page size. So in this case, the last three hex digits of this virtual address are A, B, zero. So we don't have to translate them. And the bits, everything before that is essentially that virtual page number or what index I should use in this array. So if I was trying to translate this address, well, I would figure out what the virtual page number is. So it'd be zero. And then I would look at the page table, which would be unique to a process and say, hey, at virtual page number one, that's PPN one, or sorry, at virtual page number zero, it's PPN one. So my physical address would be one. So I just essentially replace the virtual page number by the PPN I looked up. And then I keep the offset the same. So it's A, B, zero. Same thing for all the other translations. So for one, F, A, zero, well, F, A, zero, that is the offset. So I don't have to translate this part. And one is our virtual page number. So if I look it up, oh, at virtual page number one, it's physical page number four. So the resulting physical address is just four F, A, zero. So questions about that? Yep? Smallest units of memory that can be allocated? Yeah, so that offset the 12 bits. So in terms of memory allocation in the kernel, that's the smallest allocation it'll do, just one page at a time. And also that's why I use the magic number and a bunch of programs, because I know that's the smallest unit I get anyways. I may as well not ask for like a hundred bytes because I know I'm probably gonna get a page. So why bother wasting space? All right, so here is a typical question that you have to generalize any page translation. And these are fun ones to put on exams. Everyone loves these ones. So this question just comes up with a whole new virtual memory system. So it will probably tell you the page size, like the size of the virtual address, the physical address, and probably the page table entry size for other questions. And sometimes one may be left out and you have to figure out the other one. So let's go through this question. So that is not what I want. This is what I want. So in this question it says we have an eight bit virtual address and we have a 10 bit physical address. And it also says we have a 64 byte page size. So it goes through a bunch of questions. Like first question was how many, so how many virtual pages are there? Well, at least with this scheme where we just have a single page table, well we should divide up that virtual address and split it up into how many bits are the offset and then how many bits are the virtual page number. So remember the offset, how many bits we need for the offset is directly related to the page size. So for these, knowing your powers of two is pretty good. So anyone quick, what's 64 in powers of two? Six. Four, six. So I can only like multiply by two and divide by two like up to two times. So if you want to just remember like one or two good numbers to the power of five is 32. Just remember that and then you can figure it out from there and then two to the 10, you should also know that one, that's 1024. So knowing those, you can pretty much figure it out from there. Two to the five is 32. If I multiply that by two, I get 64, yay. So I'll put this back down here. So in this, my offset would be six bits. So how many bits do I have left over for this virtual page number? Two. So this asks how many virtual pages are there? So how many different pages could I index or how many different entries would be in that table? Four, right? So this would be to the power of two. We can keep it at that or in other words, four. All right, we are steamrolling. All right, other question. How many physical pages? Oops. So same idea, but now we would have like offset and then PPN for our actual physical pages. So our offset would hopefully stay the same at six bits because that's our page size. So how many bits do I have left over for the PPN? Four, because I said right here, I have a 10-bit physical address size, so it's just 10 minus six. So I have four bits left over for the PPN. And then to get the physical pages, same idea. I can actually have up to two to the four physical pages. So that would be two to the four. Or if you do the shortcut, hey, two to the five is 32 and then one less, I just have to divide by two once, that's 16. So I didn't have to remember that. All right, other one. Number three was how many entries in the page. So how many entries are there in the page table? In this page table for this question. Yeah. Four. Four, right? Kind of a trick question. Four? Yeah, four. Because it's just, I have to be able to have, for each virtual page, I need to be able to map that to some physical page. So it's just always going to be equal to whatever this is. So four. Yeah. Why the 4096 is the page size? So why 4096 is the page size is just because it was like a Goldilocks number that they discovered in the 70s, where it was fast enough if they used that number and didn't waste that much space. For no other reason than someone picked that in the 70s. Yeah. No, so the kernel only deals with memory and pages. It only cares about pages. Oh, okay. Yeah, so your program can ask for a byte, but the kernel is going to map, give it a page and it'll have a bunch of other memory it can use. Yeah. Yeah, so the question is, would there ever be the case where it's the opposite where I have like more virtual pages and physical and this is just a system I made up. So you could, it would probably be a bit silly because having a larger virtual address means that table gets bigger and it's more expensive to implement. And doesn't really make sense if it's bigger than the physical address size because then like can't access stuff, but turns out some systems use virtual addresses to map other things like map hardware and other things. So it might be able to actually use more virtual addresses if it wants. So you could design a system like that, but it'd be probably pretty weird, but you could. Yeah. So is that number of physical pages that we could have? Yeah, so the question is in the page table, like how big does each entry need to be essentially? So yeah, that's a good question of in the page table, how big does each entry need to be? So remember like the important things that entry stores. So like the page table entry, what does it store? It stores like the PPN and then like a bunch of permission bits for the purposes of this course. We can assume it always has to have like at least a valid bit. Oh, my voice is cracking. Nice. So if the page table entry contains the PPN and like a valid bit, it needs to be able at least that big, right? So in this case, our PPN is four bits and then the valid bit would be one bit. So in this case, yeah, so your PTE always has to at least hold the PPN in a valid bit. So my PPN in this case would have to be at least five bits. And because we store it in like a big array, it should probably be like bytes and powers of two. So this, we could just say probably just one, one byte is sufficient for this, right? But if my PPN was bigger, wow, that almost sounded really weird. All right, I should stop talking. So if it was like 10 bits or something like that, then if this was, I don't know what the YouTube rules are. So hopefully not. All right, so if my PPN was actually 10 bits, then it would be like 10 bits plus one. So it need to be at least 11. So next like power of two would probably be like two bytes.  So in this case, I just wrote out an array for the page table. So in this case, my page table was like two, five. I just simplified it a bit where the page table just contains physical page numbers and not the valid bit, but a real one would have a valid bit as part of it. So this would be index zero, index one, index two, index three. So if I want to translate the address, what address? So I want to translate the address F1, and this would be like the virtual address. And then question, do PPNs get mapped to non-volatile memory if the RAM isn't sufficient for some large entries? So we'll get into that later, but essentially, yeah, you can actually fake having more memory than there actually is because you could map virtual page numbers to whatever the hell you want. So you could map them to like hard drive space if you really wanted to. And that's one way you can actually use more virtual memory than you have physical memory on your machine, but we get into that later. All right, so back to this. So I want to translate the address, the virtual address, F1. So in this case, I have to take the virtual address and then split it off into like VPN and the offset. In this case, I have to do binary because like hexes don't really work for this because I have two bits and then six bits, so it doesn't really work that well. So if by some unfortunate happenstance, you have to actually translate hex to binary, again, there's short little ones you should know. So the only real important one that I remember is A. A is equal to 1010 or 10. A's supposed to be 10, in binary it's 1010. That's how I remember it because I'm dumb. So if you know that one, you can pretty much figure out the rest of them and F is like the last one. So F is always 1111. So at least in this one, I was a bit nice. So F is 1111 and then one in hex. Well, each hex is supposed to be four bits. So this would be 0001. So F1 would actually be this. So the last six bits here, this is my offset and then these two bits are my VPN. So what is 11, I don't know, in decimal three, right? So this is three. So that's basically the index in this page table I'm supposed to use. So I'm supposed to use this one, right? So in order to translate this address, well, I keep the offset the same, so it's 110001 and then in here, this is where I put the physical page number and eight in binary, sorry yeah, eight in binary is 10000. So if I want to write this back out in hex, I start grouping by fours. So this is the last hex digit, this is the next one and this is the other one. So if I want to write that a bit neater, this is 10, this is 0011 and this is 0001. So what is 10 in hex two? So if I write out the physical address in hex, it's gonna be two, what's this in hex? Three and then what's this in hex? One. So with this page table, if this process uses address F1, it gets translated to 231. Yeah, so here there's no valid bit in table, I'm just assuming everything's valid, just to save space. But on real ones, I would probably tell you if it's valid or not, or just put another call in there or something. All right, questions about that? All right, so yeah, preempted me a long time ago. Each process gets its own page table. So this also lets us explain what happens in fork and why we actually, it's fairly fast. So when you fork a process, it copies that page table exactly as it is in the parent and how it is fast is an implementation called copy on write. So basically what it will do is it will turn off write permission. So if that new process tried to access the same virtual address, which would be the same physical address, then it would get an error. And then only at that time when it's trying to actually change something, would we actually make a copy of that page and then otherwise they share memory and it's really, really fast. So all we have to do is copy the page table. So the problem though is you might realize that that page table, at least for that 39 bit virtual address, so going back to the real implementation, it had two to the 27 entries and each entry itself is eight bytes or two to the three. So if we figure out how much space that takes, well, that's two to the 30, yeah, that's two to the 30 or one gigabyte. So probably a pretty stupid idea that each process actually just requires one gigabyte by itself for a page table. So I don't know how much memory your laptop has. Mine has 16 gigs of RAM. So if this was true, as soon as I had 16 processes running, I'd be out of memory just from the page tables. So obviously that's not a thing, but if you actually read all the hardware documentation and everything, it actually magically does that. We do have 39 bit virtual addresses and in this case, they map to 56 bit physical addresses. So that 10 bits to spare, so for future use. And our page size is actually the 4096, which is the size of the offset field. So it seems like a lot of work. So also in the sub process lecture, we did a fork followed by an exec. So why the hell do we even need to copy the page tables anyways, if we're essentially just resetting them when we ever we exec something. So there's actually a way to control this. There's a separate system call called vFork. And what vFork will do is not bother to manage to copy the page tables. So that means if you do a vFork by default, you share all the memory with the parent and you can screw up your parent's memory, like okay, not any real life analogy there. Technically, if you read the C documentation, it says if the child starts changing around memory, it's undefined behavior, good luck, you're on your own, you're probably screwed. So the reason they implemented that was only through like very performant sensitive programs that just did fork exec and that's it. So I don't really know why I went on this tangent, but there's some control you can do over for what happens during a fork. Yep, mm-hmm. When we did the fork bomb, we did not run out of memory, no. So how is that possible? So the page table turned out wasn't actually that big. So we'll get into that later. And a question is the child process marginally slower than its parent if it's not using vFork. So it'll take some time to set up the page tables, but that'll be done by the fork. And it will be actually the first process to modify the memory will be slightly slower. Yep, did you have another question? Okay, we're gonna be on virtual memory for the next lecture too, so don't worry about. All right, so what we did today, use pages for memory translation divided that memory into blocks and we only have to translate once per block. We use page tables, which was basically a giant array of page table entries to access the PPN and flags. We should at least know there's a valid bit and we have a new problem we get to solve in the next lecture, which is these page tables are stupidly huge. There's no way this actually is how it works, right? So with that, just remember, I'm pulling for you, we're all in this together.


Hello people, I'm Ginny Metherill and this is my channel all about everyday easy witchcraft for everybody to do. I'm a fourth generation witch, I let my craft from my predecessors and as a result I don't tend to use expensive or way out ingredients and in fact part of being a traditional witch which is what I like to call myself is knowing how to use everyday simple objects to provide you the best that you can in your craft. So in today's video we're going to look at home protection using these. So today's video is going to use an essential oil burner and essential oils, some moon water which I created at the last full moon, I have some bay and I'm going to combine it with some other herbs from my garden and finally some incense for my incense burner. We're going to look at five quick techniques for protecting your home which is a really good idea to do this at the moment because as we come out of lockdown we're going to have I hope a lot more visitors crossing our threshold and what this will do with these protective charms is they will knock off the negativity from those people as you enter or any bad jujus that are hanging around won't be able to cross that threshold. So this is my video about home protection. You don't have to use all of them by any stretch of the imagination, you can use as many or as few as you like but at least obviously one because you've got to have one to give you some form of protection.  I've got to wait for them to go. So the first one we're going to do is we're going to use some moon water and I make moon water pretty much every single month because I get through quite a lot of it. This is the simplest and easiest one to do. We're going to use the moon water to draw a sigil upon the door and this sigil will help protect and keep away any negativity from your home. So we're going to use the pentacle which is one way coffee cup here which is a pentagram which is this five-pointed star within the circle. I love my coffee cup. One of my children gave it to me for Christmas and I never get to drink out of it because everyone else likes it too and nicks it so all I'm going to do is use my finger. Five-pointed star surrounded by a circle and as that dries that'll give them pretty good connection. However if you didn't want to use a five-pointed star you could use the eye of Horus. A friend of mine in fact has got an eye of Horus tattooed on his right arm. This is actually rather beautiful but he cannot therefore use that arm on a Ouija board. Now Ouija boards as you probably already know I think are completely and utterly dreadful because they only ever access bad Juju's demonic entity's negative spirit and it defines it very interesting that he who has the eye of Horus tattooed on his arm cannot use the Ouija board. Likewise the pentacle is one that has been used for generations upon generations. The circle enclosing the five-pointed star which we're going to use the correct way up rather than the upside down version but actually to be fair it doesn't really matter because it's your intention of drawing the circle and putting the pentacle in it that makes that pentacle protective. The moon is a guardian spirit and so adds protection automatically to all moon water so it's a great thing to use and also as the moon strikes your door if it ever reaches it it will recharge that symbol. The next thing we need to talk about is the herb so I'm just using bay and rosemary because that's what I've got in my garden and what I've got in my shelves but next I'm going to use some rosemary it's the only thing that's growing in my house at the moment just a couple of sprigs are fine rosemary's a great herb for protection especially when combined with other protective herbs it makes a really strong charm for the home. So I've got my bay and my rosemary I'm simply going to tie them up with a ribbon and hang them over the door bay and rosemary together make a really great charm and I recommend just doing this anyway because it smells so divine the rosemary and the bay. Yeah there we go isn't that charming? My next charm is to place an essential oil burner right by your door opening and let the candle burn down fully releasing the essential oil all over the front of the door. Most essential oils have some inbuilt form of protection from the actual oil that the essential oils are distilled into so you could use pretty much whatever you like I'm going to use some rose geranium and as anybody who's watched all of my videos will know I adore the scent of rose. Next up is incense and I've got a small incense burner here and a charcoal disc which I'm going to light and place it within the incense burner. On top of the charcoal disc I'm going to add some of my frankincense resin I love frankincense it reminds me of church and childhood so it's a beautiful smell there look at that smudging well again let it burn down and once the smoke has finished the spell is cast and finally for all of those who are able to do so you can use your wand and just cast this protective circle I start up the north that's how I did it's a bit difficult to explain I'm just using my intention to set the circle and what I want to happen within that circle it's sort of senior witchcraft that bit I would love to know which one you're going to use are you going to draw your sigils of moon water are you going to put a posey of protective herbs over your door are you going to burn some essential oils smudge or cast a circle let me know in the comments below I hope you enjoyed this video if you did please do give me a like and a subscribe and finally don't forget that if you want to join our cover meeting it's happening on Wednesday which is tomorrow 8th March go to patreon.com www.swordslash.com for details I'll see you in my next video


Hi, everyone. My name is Patrick Fuentes, and I work on the Chrome OS Developer Relations team. Today, we're going to cover three topics, new developments for Android on Chrome OS, how to ensure your apps have a high-quality experience on Chromebooks, and how to optimize the performance and experience of your games for Chrome OS. A lot has changed since Google Iow in 2019. We want to tell you about what's new and some things you may have missed. In that time period, usage of Android apps on Chrome OS has more than tripled. With all of this platform growth, there's an opportunity for you to engage more users than ever. We're here to talk about how you can create the best possible user experience for your Android app or game on Chrome OS. We'll also share some of the work that we've done to help make that easier. Most Chromebooks with Android are currently running Android 9, but we're happy to share that Chrome OS is in the process of moving devices to Android 11, bringing you the latest APIs and features. Another big change that's happening with the move to Android 11 is that Android runtime is migrating from a container to a virtual machine. The move to a virtual machine will improve security for users, and it'll make the Android environment more maintainable. This should enable Chrome OS to reduce its divergence from mainline Android, which will increase stability and the velocity of future Android updates. This is all happening under the hood, so there shouldn't be any changes for you as developers or for your users. We're also improving the Play Store on Chrome OS to serve as high-quality apps and games on the platform. This means that we'll be putting apps and games that do the work to create great user experiences in front of Chromebook users. So let's talk about what you can do as a developer to make your app look great and improve the user experience on Chromebooks. There are three main areas to look at. A thoughtful, large-screen user interface, support for a variety of hardware and software inputs, and desktop-like functionality. When designing your app, remember that your users will be engaging with it on a variety of form factors and screen sizes. This could mean laptops, tablets, convertibles, foldables, and more. If an app layout designed for a portrait phone is stretched to a 4K monitor, many kinds of problems can arise. UI elements may get stretched out, there could be wasted space, and the app may be difficult to use and navigate. Let's look at an example. Imagine you have a phone app with a bottom navigation bar. This allows users to quickly navigate with their thumbs and it may be a great user experience on handsets. But what happens if that same layout is shown on a wider landscape screen? The icons are difficult to see and click and they can be even more difficult to understand. Not being able to easily navigate through your app can disrupt the entire user experience. Transitioning to a side navigation bar on large screens can make your app easier to understand and interact with. Of course, there are more opportunities with large screens beyond navigation. For example, a simple list and detail view paradigm works well on handsets, but as you can see in this example, you can make better use of space on larger screens by also showing an image preview to the side. With more space available on larger screens, you can do much more with your layouts. You can display more information at the same time, allow for fluid multitasking with other apps, or immerse users more deeply in your experience. Fluid design approaches allow your app to take advantage of available screen space to provide the best app experience for the user. Different apps require different approaches, but you can find practical tips in two other talks at this year's Google I.O. Five things you can do right now to prepare your app for large screens with material design and what's new in foldables, tablets, and large screens. Be sure to check out the links in the description below. If you're looking for more design inspiration, check out the Rally demo app from the Material Design Studies. This is a great example of the way space can be used on different screen sizes and orientations to maximize functionality and visibility. Another important subject to consider when optimizing your app for large screens is configuration changes. Handling rotation changes in apps isn't anything new for many Android developers, but the increasing variety of form factors has made configuration changes more common. Changes like folding and unfolding or window resizing on the desktop environment like Chrome OS can be common parts of your user's workflows. Not handling this correctly can lead to the app restarting or getting into an unexpected state. You don't want to lose user data or the state of the UI, so be sure to maintain all UI state in a view model and or by using on saved instance state to ensure a smooth experience. Another key consideration across form factors is supporting different input methods. Users will be using your app with keyboards, mice, trackpads, game controllers, styluses, and more. We have an in-depth talk on just this subject this year, so be sure to check out Emily's talk, Input Matters for Chrome OS. It's linked in the description below.  Thank you, Patrick. I'm going to talk about gaming news and gaming best practices. Let's start with the best practices. As you know, Android phones are mostly based on ARM CPUs, and for that reason, chances are that your game might be supporting ARM binaries only. However, on the Chromebook world, although we have a lot of ARM devices, we have a lot of x86 ones too. Rest assured, all your ARM-only games will run perfectly fine on our x86 Chromebooks. But to get an extra boost in performance, we encourage you to support both ARM and x86 binaries. To do so from Android Studio, it is just adding a few lines of code in the build.gradle file. As you can see in the snippet code, you should include both ARM and x86, both for 32 and 64 bits. That's four ABIs in total. But it is worth mentioning that adding support for a new ABI does not increase the binary size delivered to the user. Up Bundle takes care of that. When the user downloads the game from Google Play, it will receive the right binary for their Chromebook, meaning that an ARM Chromebook will only receive ARM binaries and an x86 Chromebook will only receive x86 binaries. And if you are using Unity, we are happy to announce that with only a few clicks, you can add x86 support to your Unity game. Stay tuned on ChromeOS.dev for more details. If your game requires internet access, consider using the connectivity manager to sense whether internet is available. Don't just check whether Wi-Fi is enabled. Although all Chromebooks support Wi-Fi, some users might have ethernet as their main interface, and Wi-Fi might or might not be enabled. Let's take a look. Here is the code that checks whether the Chromebook is online. It gets the connectivity manager, grabs the first active interface and returns whether or not it is connected. One of the most requested features from developers was to support APK-side loading without having to go to developer mode. Not only it is now possible to install a game by double-clicking an APK file, but also the entire ADB suite is available. As an example, you can do ADB install, ADB log cut, or ADB damses without putting your Chromebook in developer mode. On Iona 18, we talked about the AR graphics tracing tool that is built-in ChromeOS. Since then, we kept improving our tools. We developed a new tool to measure FPS, render quality, power consumption, among other things. It is built-in ChromeOS, and you can try it by navigating to the ArcOverview tracing URL. Another useful tracing tool is Parfetto. It is similar to SysTrace with some extra benefits, like in addition to tracing Android events, you can also trace Linux kernel events. You can include log cut messages in the trace. This is very useful when you want to have more context of what's happening. It has a query API that can be used from Python. This means, among other things, that you can automate some of your performance tests. Parfetto is a tool that is available on devices running Android 10 or later, including Chromebooks with ArcR. Regarding gamepads, in the previous I.O., we announced that Arc was receiving raw gamepad events. This was a big improvement in gamepad compatibility. Since then, we kept improving the compatibility by updating the key layout files, the one that convert raw events into Android events. We backported the key layout files from the latest Android version into Arc. And we are also contributing back with our own key layout files. Please watch the talk, input matters from Chrome OS for further info. Regarding graphics, our graphics stack is complex and contains many subcomponents. And this image is an oversimplification of how it works. We fixed bugs, added improvements, and updated drivers in all these subcomponents. The changes were many, but just to mention two of them, we reduced the latency when handling graphics buffers, and we did so by implementing fences. And we improved Vulkan rendering. In particular, we fixed glitches that were affecting some games.  Thanks, Ricardo. We hope you found a few tips and tricks that will help make your app or game even better on Chrome OS. We're continuing to invest in improving your developer experience when building for Chromebooks because we want Chrome OS to be a great place for users to enjoy your Android apps and games. So what should you do to take advantage of this progress? If you'd like to learn more, please visit ChromeOS.dev. We've published a lot more guidance for large screen layouts, handling inputs, performance, and more. So be sure to take a look there for helpful tips. Most importantly, try running your app or game on a Chromebook and see what it's like for yourself. Try the tips we've shared and ask yourself what your users want. We look forward to seeing users enjoying your apps and games on Chrome OS devices and every other device that supports Android experiences.


The children in the town knew better than to cross the old bridge in the park especially during the evening hours on foggy nights Even the grown-up stayed clear Knowing that those who dared venture out were doomed never to be seen again And everyone knows grown-ups are usually pretty naive when it comes to such matters They had even made an effort to stop people from crossing the bridge by making a gate that closed every day an hour before Sundown to block access to the park The grown-up seemed so proud of themselves for protecting the little ones And so the children didn't have the heart to tell them that it was a waste of money But her spent buying birthday and Christmas presents Everyone knew better than to cross the bridge. So there was no reason for a gate at all Unfortunately Eleanor was nobody and even if she was somebody she was still a very stupid little girl Born and raised in the next town over little Eleanor had only recently moved when her parents divorced and her father and new Stepmother got full custody of her Though she fancied the idea that she was just too clever to get along with the other children the opposite was in fact true She was nearly as ignorant as a grown-up Constantly questioning things and demanding evidence for every trivial truth Some of the harsher rumors claim that she didn't believe in ghosts Aliens or even the completely Inarguable fact that the fourth-grade homeroom teacher mr. Mason was a werewolf demon hybrid from heck who worked for the devil himself Thus Eleanor had rightfully replaced past eating patty as the village idiot Even so Eleanor was very prideful No matter how much the other children chastised her for saying silly little things like There's no such thing as baby biting sewer clowns She persisted It wasn't long before everyone started avoiding her Fearing that her blatant disrespect of the supernatural made her dangerous and unlucky At first Eleanor didn't seem to mind too much As her vanity deluded her into thinking that she was better off alone anyway Since apparently she was just so much more intelligent than all the other children anyway Still even nobody's get lonely And so when she heard the other children discussing the matter of the old bridge in the park She couldn't help but butt in Oh, come on.  It's just a bunch of stones The children had heard her say many strange things But now she had gone too far Even the grown-ups knew how dangerous the old bridge was A few of the kinder more patient ones gave Eleanor a pitiful glance Reminding themselves that she didn't know any better But while things like mr. Mason and the sewer clowns were survivable The old bridge in the park promised at least a bazillion percent chance of never being seen again Sure the children hated Eleanor But they wouldn't wish such a fate upon even the worst of little girls They begged and pleaded with her to drop the subject as she demanded proof credible sources of information Some were valiant enough to promise to invite Eleanor to their birthday parties if she'd only promise never to cross the bridge But stubborn little Eleanor wouldn't listen to reason or bribery She swore to them that she'd prove that the old bridge was nothing to fear using a hand-me-down video camera She'd received on her ninth birthday Tonight i'm going to cross the old bridge. She said and then you'll see how foolish you all are There was no way to stop her from killing herself without putting some effort into trying Distraught the sweet and generous children decided they try and be nice to Eleanor on her last day alive All except patty who wasn't too keen on being the stupidest child in school again Eleanor didn't appreciate their efforts That night Eleanor snuck out of the safety of her home, biked to the park and struggled over the gate Tonight was particularly foggy And it took her longer than expected to get to the old bridge Apparently the gate wasn't the only precaution the grown-ups took They had also changed the paths around so that the only way to get to the old bridge was by following The winding deer paths through the marshy woods It was eerily quiet No owls hooded and no crickets chirped Any reasonable person would have recognized the silence as an omen of horrible tragedy to come But not a girl who denied even the existence of moon vampires Armed only with her reckless self-confidence and a flashlight Eleanor walked to her own demise By the time she found the bridge It was nearly a quarter past ten widely known in the town to be the most sinister of times Obviously this was true Why else would the grown-ups rush them to their rooms and demand they sleep until the sunrise promised their safety Eleanor either willingly ignored the time or just didn't know its significance as she turned on the camera Eleanor gave a cheery smile and said her hello's Almost passing as a normal little girl until going into a smug little rant about how stupid she thought everyone was And how she couldn't wait to see the looks on their faces when she proved to them that she was right She was so self-satisfied that god himself probably considered striking her down with a lightning bolt for being so prideful But then decided he was better off now wasting his precious time She lifted her foot to step onto the bridge But hesitated As if just for a second the ignorance clouding her mind had cleared It wasn't too late to turn back If she chickened out now the other children would surely only make fun of her cowardice for a few weeks And as a bonus she'd not suffer what was most likely a horrible death After an agonizing 12 seconds of thought Eleanor put her foot down on the stone of the bridge When she wasn't immediately exploded into confetti or elsewhere injured She let out a whoop of celebration and started bragging incessantly to her camera Meanwhile, god was probably reconsidering his leniency As she walked on she prattled onto the camera and bragged about how clever she was All in all it took nearly five minutes of self-righteous babbling before Eleanor realized something was amiss This is a very long bridge, isn't it? Thinking she had misjudged the whiff of the river Eleanor peeked over the railing only to see there was nothing below her And it's so foggy I can't see a thing Maybe people have gone missing because they fell off Despite her efforts to see the river The fog made it impossible to find it But it had to be there right And so she continued on Every step making her more and more anxious At this rate the camera was going to run out of batteries Any rational person would have turned tail and run screaming like a banshee that had stepped on a lego But again Eleanor hesitated She checked the time on the watch she had borrowed from her stepmother as if it would tick out Nope, everything is perfectly normal Do carry on in Morse code This must be broken Surely have not been walking for a whole 30 minutes Her panic only rose when she checked to see how long the camera had been recording Eleanor hadn't pressed the button and all this time she'd been talking to no one like a complete maniac Before she had time to swear aloud her flashlight had the nerve to flicker Not only had she forgot to record this experience She'd left her spare batteries at home Eleanor was in trouble Though she didn't yet realize how much trouble she was in Even now she thought the worst that would happen is she'd get lost on the way home without any light to guide her way Her shoulder slumped and with a pout she turned around to head back before her batteries died completely She felt almost as done as she was Wandering into the woods so unprepared Cockiness killed the cat Eleanor supposed Though by killed she actually meant non-lethally inconvenienced Maybe she'd try again tomorrow night. It felt like three hours passed, but in reality it was closer to two and a half Eleanor patron saint of not believing anything without proof Ignore the very real evidence of her stepmother's watch telling her it was nearly one in the morning Unable to control herself anymore. She burst into a sprint and ran as fast as her little legs could carry her Desperate to see the bank of the river Her flashlight gave out and so did her confidence She screamed as loudly as she could and fell into a heap onto the cool stone of the bridge What was she supposed to do now? Turning back clearly wasn't any good and seeing as the park was closed There was no one around to hear her pitiful cries for help And so Eleanor gave up Instead of running around screaming. She opted to just sit there and wait for the sun to come up Surely it'd be a waste of energy to do anything more Hours passed by slower than maple syrup right out of a too cold fridge Eleanor did anything she could think of to pass the time practice multiplication in her head Figure out what she'd tell her father when he found out she'd been out all night wandering the woods Anything but it knowledge that maybe she never get off this bridge alive Though instead of fearing monsters she was still afraid of earthly things like dying of wolves or boredom To prevent at least one of those things Eleanor put on her camera for real this time And began recording her goodbyes At the very least she thought she could delete it if everything turned out all right As if on cue a light in her peripheral vision caught her attention She quickly turned the camera off and got to her feet Was it the sun already? No, it wasn't the right color And it was too small could it be? Like a moth drawn to a bug zapper Eleanor scrambled to her feet and darted for the light Help me Hello The light Or more accurately the ancient old lady holding the light responded Are you lost little girl? Eleanor slowed to a quick jog Tears of relief swelling in her eyes No child in the entire world had ever been more happy to see a weird stranger on a bridge Yes. Oh, thank you so much. I was getting worried that I'd never see anyone again She couldn't yet see that the old woman had no eyes Or that her right arm appeared to be made of solid gold She didn't see the many rows of bristle like tea Or the fingers that seem to have twice as many bones as they should have If she had Maybe she'd have stopped herself before running straight into the monster's arms and hugging her close Please will you take me home? The monster was taken aback She'd eaten many children But this was the first to so quickly give itself up to her This was very fortunate for Eleanor If the monster were more focused She'd have unhinged her jaw and swallowed her whole Before Eleanor had the chance to look up and realize she was in mortal danger With a scream she threw herself off the monster and back the way she came without even looking back to confirm what she saw was real Skeptic or not, she was more prepared to face the lonely bridge before facing what seemed to be an actual monster The old woman regained her wits and let out a cackle before chasing after Eleanor Don't you want my help anymore?  I can't keep up Eleanor never listened, but for once it did her some good Her lungs burned as she took in just enough air to propel her into safety Though she didn't know it the other children's carefulness was saving her The monster was weak enough from hunger that she actually stood a chance of surviving She was panting too hard to hear that the river was babbling below Her eyes were too clouded with tears For her to see that the sun was rising and the fog was fading away The moment the light of morning reached down to the bridge The monster let out a horrible screech and quickly scurried down into the river But Eleanor was still running until she was off the bridge and nearly out of the woods Finally she slowed to a walk to let her lungs rest She done it. She survived the old bridge With a start Eleanor looked at her camera and nearly sobbed with joy The whole thing had been recorded. There was concrete proof of the horrible monster and her escaping Heck, she'd only really screamed the one time so she probably looked pretty brave The other children would be so impressed with her Her tearful rejoicing settled into a worried frown If this was true then what else? Was Mr. Mason really a demon? Did she really have to worry about sewer clowns and moon vampires and ghosts? Heck, it didn't matter. All that mattered is that she survived And if she could survive this she could survive any crazy old thing Her confidence rose again No longer was she blind to the world around her Now she knew better and she'd use her knowledge for good Where did that bridge monster come from? Clearly it was weak to sunlight and could only live in darkness Maybe she could destroy it But she would need help The other children knew things but they were too afraid to fight them It was time for that to change Things in this town were going to be completely different from now on Together they would rid this town of whatever monsters may plague it Eleanor sprinted off the bridge and through the woods Breaking out into the road leading back home The truck had seemingly come out of nowhere Flattening Eleanor before she knew what hit her Literally And thus the first child ever to survive crossing the old bridge died Not because of the monster that haunted it But because she was too stupid To look both ways before crossing the street


going to be talking at 3.15 about really diving into the interactive component and building sort of bokeh apps, bokeh dashboards and stuff. So it'll be a really great segue this afternoon. So data at the heart of bokeh is the model, the column data source. And it's very simple. It's just a table of data just like you might expect. And so that's not a good sign. So we take a data dictionary in Python, a normal dictionary, keys, array. And we also, how many people here have worked with pandas? OK, almost everybody. If you haven't worked with pandas, it's fine. It's a lot like Excel, but way better in Python. And it's not a lot like Excel. It'll change your life. But the core of it is you can pass it a dictionary and you get out a table of data. And there's lots of other ways you can read from Excel, from JSON, from CSV, and so on. If you haven't used pandas before, I would totally recommend Brandon Rhodes' pandas tutorial from PyCon 2015. It really boils down like how to really just understand it, but keeps it at a nice, simple level. And so you can make your column data source from either a dictionary or from a pandas data frame. And your data ends up represented like that. And you'll notice the one difference between when I pass in my dictionary and when I pass in my data frame is that I've got this extra key value here. The index has come from the pandas data frame. It's this left index here that it's named index for us. So that is the column data source. The next thing that's at the heart of Bokeh is the plot. And I'm skirting over some details here, because I don't think they're that important. There are these models, plotting, charts, these three different ways to use Bokeh. But they have a lot in common. And you should find it pretty easy to jump between the three levels. But I think it can be a bit confusing knowing exactly what to do from the get-go, which is why I'm trying to show you all three. So at the lowest level, we have the plot object. The next level up, we have the figure. And finally, we have the charts. They're all different things. They come from different modules within Bokeh. But you're going to see they all have the same attributes attached to them, like a toolbar location, background fill, and so on. So once you're working with one, you can reasonably assume that the other one's going to behave similar. And they all have three really important methods on them. Add tools, add layout, and add glyph. And if you can remember those three methods, you're going to get a really long way into building all of your plots. Glyphs, I hear you say. No one said glyphs. OK, glyphs, to me, the word means nothing. I was so confused for a long time. It just means shapes. And I don't know why it's called glyphs. I'm sure there's some very, very logical reason. But glyphs are just shapes. And so this is actually from the examples. And these are all of the different kinds of shapes that you can spell within Bokeh. And in some combination of all of these, there's a whole other set here. In some kind of combination of all of these things, you can make pretty much anything that you could think of. If you couldn't, I'm sure we would add another one, but we really have got most things covered here. So glyphs are just shapes. So we're building a dashboard. We've talked about data. Let's talk about charts. But I keep going on about this models plotting charts thing. And I want to just lay it out a little bit more. So there's these three modules, Bokeh.models, Bokeh.plotting, and Bokeh.charts. Bokeh.models is the lowest level of Bokeh, but it doesn't mean it's a level you should be afraid of. It's an awesome level. I love to live in the models level. Everything above it is built on top of models. So it offers you the most control, but it doesn't mean you have to do all the work yourselves. On the complete opposite end, you have Bokeh.charts.  It just takes your data frame and gives you something awesome. So it's very, very quick to use, but it sort of does all the magic for you. Bokeh.plotting kind of lives in the middle and you have to sort of organize your data for it, but it then tries to pick sensible defaults, so you don't have to add the axes and add the grid and so on. So it tries to sort of give you something you want, but it gives you a bit more control. Where you want to work is really a matter of personal taste. Personally, coming from a web world, I started with charts, and charts for me were a gateway to models, but I think a lot of people coming from the data science, the data science world find plotting sort of more intuitive. But you'll see them all, and you can choose what you want. So let's start with charts. So this is one little pocket of the dashboard that I was demoing, and this bar chart here has come out of the charts interface. So one of the things about the charts interface is that it kind of does some magic. It just takes your data and spits out a chart. So you kind of need to know what it wants the data to look like, otherwise it can't do its magic. So often when I'm getting started with the chart, especially if I haven't used it before, I'll just set up some dummy data to make sure it's kind of roughly doing what I expected to do before I put in my real data, because then who knows whether it's okay or my data that's being weird. So we just have a simple thing here. We've spent some time with ponies and unicorns and other magic things. And now we get to importing our bar chart. And so we've imported it from the charts function. We import bar and now we just call it with our data frame and we show the bar chart one line. And here we have our chart. We have zoom, we have reset, we can save it, we can do that, we can resize it. And this is a web object. You could share this HTML. It's all gonna work beautifully for you. So that's working how we expected to. So now we're gonna try with some of our real data. All of the code that gets all the real data is in that repo that I shared with you earlier. I'm not gonna cover how I made it today. This is the raw data. And then this is the sort of process data that I did. So once again, I spit out my data. That's, I spit out my bar chart, which is almost how I want it. So now I'm gonna add a couple more options. I'm gonna stack my bar chart and I'm gonna set a palette. And now it's starting to look a lot like what I showed you earlier. Styling is later in the talk, so we'll get to that in a bit. So there we go. One line, bar chart, it's interactive. You can add hover, you have zoom, and there you are. What I didn't, what I forgot to put in was a list of all of the different kinds of charts that you can import. But at the moment, it's all the ones you might expect, line and time series and horizon and scatter and so on. So next up is IO. What you might have seen me do at a couple of the top of those notebooks is from bokeh.io import output notebook and show. And when I call output notebook in an IPython notebook, I see this bokeh.js successfully loaded. So that's how we work with bokeh in a notebook. And then there's also this output file, which is the sort of equivalent. And so then at the top of my notebook, I would call output file and I would give it an HTML file name. And then when I hit show, instead of showing in the notebook, it opens a new browser with my static HTML has been saved and I could share that with somebody. I like to use this extra argument mode equal CDN, which instead of dumping all of the JavaScript and CSS that power bokeh inline into the HTML, it just puts in a reference to the bokeh CDN, which takes your HTML from sort of 1.6 megabytes down to about 6K for a small plot. So it's a decent bandwidth saver. So yeah, so that's how you get started with spitting your stuff out. So plotting, we've done the charts interface and now we wanna look at the plotting interface. This is a plot that I couldn't build with the charts interface. There isn't a chart that's sort of pre-baked for me that does this. I have this categorical axis on the left side here, splitting up the different kinds of activities that I get up to. I have a time series axis on the bottom here and then I have these rectangles that I've plotted across. So once again, I'm gonna try with some test data first. This time, I'm gonna build my own column data source. You notice I didn't have to do that with charts. I just threw in a data frame and it just spat out a chart. But this time, I'm gonna have to build my own column data source and I'm gonna do it by passing in a dictionary. And then I'm going to instantiate my figure. Now, figure doesn't do anything on its own. In fact, if I just do show P here, what I get is a very helpful error message. It's just come into Bokeh saying, you have no glyph renderers. I'm not gonna be able to plot anything interesting. So we're starting to this sort of validation and error helping is new. And so it's a bit spotty. So there are some bits that do it and some bits that don't. So if you ever see a place where you're like, oh, why didn't it give me an error message? Just throw up an issue on GitHub because we keep adding these in. So once we have our sort of empty figure, now we have a whole bunch of methods on it. Remember all of those different glyphs that I showed you over and over and over again? For every one of those glyphs, there's a method. So there's P dot quad to plot a quad. There's P dot rect to plot a rect. There's P dot circle to plot a circle and so on and so forth. And so for our one, we're gonna use the quad, what do you call it, method and spit out a plot. This was just dummy data. So I just have linear axes on the left and bottom here. And so now let's think about how we might do this for our plot. So here's our data. We have the start and the end in time of each of our blocks of time and then we have our categorical name for our label for each of those time blocks. So we're gonna try plotting this. And it's not looking great. It hasn't given us anything. And if I open the console, what I'm gonna see is, although I tried to pass it some data and I really did try, if I open the console, it's gonna say could not set initial ranges. And if you're thinking opening the console really, this is where these requests for more validation can come in. Like this is totally something that we could have put one of those error messages in for we haven't done it yet. So we should do that and I'm sure it will come. But going to the console is always a good place to look if things didn't work how you expected to. So what happened was, instead of if I had linear data, like I had had up here just a series of numbers, when I instantiated my figure, it was clever enough to figure out that, okay, I can figure out the ranges, this is no big deal. But when I passed it, my data with time and categorical axes, it's like, I don't know what you want me to do here. Like you've got to give me some help. So now when we're instantiating our figure, we are going to specify the range, the time series, and we're gonna specify the categories on the left side. When we do this, we get out something that looks pretty good. But not quite right. What you can see is our little blocks of time are like teeny tiny. I don't know if you can even see that in the back, but they're there, but they're just really small. And so what happened is, how do you specify height on a categorical axis? This is a bit of a tangent, but it's a good one to know. So when you have time or naught to 10 or whatever, it's pretty obvious how you specify height or width. When you have a categorical axis, what does that mean? And so the way we do it in bokeh is from the left to the right of your categorical axis, or the bottom to the top, is naught to one. And so you append a colon to your label and you add like 0.1 for the bottom to 0.9 for the top or the left or the right or whatever. And so once we've done that, if we try again, now it's come the height we expected to. And so if I change this to be 0.4 to 0.6, they'd be much narrower and so on. So you can play with that as you wish. And one of the things you're seeing here is think about that original data frame. That original data frame, the chart that I showed you was essentially a switched around version of this. The chart function did all of that magic for you. It magically figured out how wide I'm gonna make things. It processed all the data, set that all up for you so that you didn't have to think about it, which is awesome until it doesn't quite do what you need it to do and then you have to do it yourself. And actually, to be honest, one of the ways I learned Bokeh was I started using the charts and then I went and read the code for the charts to figure out how it was doing it all and then I was able to build my own. So there we go. Okay, so that was plotting, a little bit more verbose, but a lot more control. You can do lots of different things there.  And so this is where we use the lowest level of Bokeh to build up our charts. In the docs, I'm pretty sure I read somewhere that it's like most people wouldn't need to do this, but I like living down here, but each to their own. So once again, I've pre-processed all my data and I've made a dictionary of data sources that I want to plot, which looks something like this, which is nice. So this should look somewhat familiar. This is our first attempt at making a plot and this should look a lot like what we just saw in plotting. We are specifying the X range and the Y range because if we don't Bokeh or FreeCal, now instead of we're not calling a method just yet, we're actually specifying our line here. So when we specify our line, we've imported line from Bokeh.models and we're saying I want the X to be powered, if you like, by this column called timestamp. I want the Y to be powered by the column called cumesum hours. And then what I'm gonna do is I'm gonna loop through and I'm gonna add lines for each of my data sources that I want to add. So it's allowing me to add lots of different lines. And so to do that, I use that addGliff method that I talked about before and I pass in a column data source and I pass in the specification for my line. So when I do that, this is what I get, which is what I asked for, but it's not beautiful. And that's because down at the models level, we really do have to spell everything out. So the first thing we might wanna spell out is some color and Bokeh has a whole bunch of built-in palettes for you that I would totally recommend using as defaults but you can of course spell your own palettes and palettes are just a list of either hex or IGBA two-pulls or a whole different bunch of things and you can see the reference at the palettes documentation. So let's try again with a little bit more style. Again, we just have our ranges specified. Now we're gonna try a bit of background fill, a bit of border fill and sort of mess around with this and now we're gonna add some layout. So this is where we're adding our linear axis on the left and our date time axis on the bottom. And then now our line, the specification has got a little bit more complicated. We're not only specifying the X and the Y, we're specifying a color, we're specifying some line width and so on and then once again, we're just adding, using this addGliff method, use the source. It's how they use the force, use the source and use the source and add the line. And once we do that, things start to look pretty promising. And so we can really start messing with this. So if I just turn this into a method so I can make it nice and repeatable. And now what I've done is I wanted to add, so this has no tools on it, right? The charts and the plotting came with all those tools just poof out the box. Now we have to manually add the tools ourselves to do this, it's very easy. You just use addTools and you specify the tool you want to add and now we have pan added and that's awesome except I don't really want it to pan up and down because that doesn't make any sense for my data. So I'm gonna try constraining it. So constrain equals width. Oh, that didn't work, that's weird. What happened? Oh, thank you, Bo K. It has told me that constrain is an unexpected attribute and the things that it's expecting are dimensions named plot-sessional tags and so this part of the error reporting of Bo K I find incredibly useful. If you, if I can't remember what it wants, I'll often just put in garbage so that it tells me what it's expecting which is really handy. So it said it wanted dimensions. So, oh, I Python fail. Oh, I think I just deleted my cell, hang on. No, where did it go? Oh, it's there, thank you. No, I changed it to markdown. There we go. So it said it wanted dimensions. I think it said that. It did say that but now it's saying it got an unexpected error and it's looking for a list and the list has to either contain width, height, X or Y but it just got width, it didn't get a list. So again, there's the properties reporting and then there's this value reporting which is part of what we use to help us be able to serialize from Python to JavaScript but it's also a very helpful for debugging your code as you go. And now, has our, it pans left and right and I can't go up and down however much I might wanna try. So we can keep going and going with that but I will spare you for now. So, quick mention of the examples in the Bokeh repo. So there's the gallery on the website but if you actually download the repo and find the examples directory there are just a bajillion examples in there and in particular for getting started with models they're incredibly helpful so I would totally recommend just downloading them and running them. So, styling. We started to see a little bit of that just now on the models but styling is possible for every single level for plotting charts, models, it just depends on how you want to do it really so I'm gonna show you for charts and models and take a pic. So, in the interest of time I'm gonna skip the little prototyping but on, this is us making a bar chart just like we did earlier and once we've got that we can then just start to set all of the attributes the toolbar location, the background fill, the outline and so on. We go and we grab the glyphs and we can specify their fill color, the line color and we set the attributes and we set the attributes and we set the attributes and then finally we show it and it looks gorgeous and so when you're in the IPython notebook the one of the nice things about it is the auto complete and the fact that you can really use IPython notebook to help you see what the attributes that you might have available to you are and so that's one of the ways that especially early on I used to play around with Bokeh a lot. Now with models, we spell it all first, right? We don't create something and then fix it we spell it all as we go and so what I like to do is and the other thing we like to do if we're making lots of plots and we all want them to look the same is we can start to build these dictionaries of properties like our line colors and so on and we can just splat them into all the axes or we can splat properties into all of the plots so I typically build up this style sheet and the top of the style sheet I'll have my colors and padding and things like that that will sort of mirror my CSS if I'm building a website and so that I can sort of have those common colors and things like that across and then I have this plot properties dictionary, access properties dictionary and so on set up in just a file called constants and then when I come to build my plot in models it actually starts to look quite clean I just make a plot and I put in the plot properties I make my two axes they show all these common stylings again I make my two rectangles they show all these common stylings and so this is why I kind of like working with models is because I'm doing a lot of heavy customization typically and I find this fairly clean as opposed to that big sort of quite verbose version that you end up doing where you fix all the attributes on a chart but each to their own anything is possible So, layouts there's... if you're just working within Bokeh you're not interested in putting stuff on the web there's plenty that's sort of in the box to help you make something nice from the get go in particular there's Hplot and Vplot that are in Bokeh.io and what they do is they just kind of make rows and columns and it's pretty self-explanatory so we've got here a Vplot and we've nested inside it an Hplot and two charts so our Hplot across here we've sort of lined up our three things this is not my dashboard this is just sort of giving you a demo of what you know what you could do and then I've stacked it on top of some other plots and so you could spit that into an HTML using the output file method and you're going to get something that's pretty presentable and all of these plots can interact with each other once they're in the same document together if you spell it all in a nested thing like this all of those things will be able to interact together because I typically sit in the web I use... if that's the very high level I use the very low level embed function and again to use this I use the components method and I take a dictionary of all of my plots and what the components method gives me back is my HTML script and a dictionary of all of the divs that I have to... the div tags that I have to stick into my HTML so I'll take a template this is a very simple one where I've just imported bokeh and maybe some CSS at the top and then I've started to template in the divs that bokeh has given me back and I template in very important my script and using that, rendering that out with the context of the script and the divs which if you're in... this works in Django, JINJA whatever your templating language is it's all the same bokeh comes with JINJA built in so that's kind of convenient you can start to build... this is an HTML page right ahead H1 building a dashboard my time selector and this is the actual HTML page that I could either be rendering through a web application or as a static HTML so that's layout and I'm pretty sure Fabio is going to talk a bit more about embedding and stuff in more detail later so if that was too fast they'll be more and so with all of that under your belt I promise that adding to it the pandas data frame method to HTML which turns your data frame into a nice HTML table you can make everything that you see here these tables here are pandas tables this is our plotting plot these are our bar charts these are our line charts and we've got it all with a nice, you know and what's different about it is I've used a I've used a nice template I've used some nice CSS and it all comes out very nicely so last but not least interactive so there's two parts of interactive the first part has been built into bokeh for quite a while we have our tools like hover select zoom pan which we've been talking about a little bit and the shared selection and panning so I'm going to quickly run through how to do that so just getting started I have a simple set of data and the first thing I'm going to do is I'm going to share some ranges and so what I've done here is I've got a figure I've got three figures, sorry that I've sat side by side and in the second figure I've used P1's X range and in the third figure I've used the P1's X range and its Y range and so when I do that it means that if I pan left to right it's going to pan out on all of the plots when I pan up and down because I only shared the Y range with one other plot only one is changing so this lets you sort of there's lots of different times when you might be wanting to see different dimensions and how they compare so you can plot different dimensions on different plots but start to move around in sync which is very handy the other thing that we've done here is we've shared a source across them so instead of making different sources for different plots I use one source with all of my data in it and I shared it across my three my three different plots but I plotted different things so my left plot has X and Y naught my middle plot and so on and so once I've shared a source when I start using selections they can when I start using selections they'll share across they'll share across all the different selections this is I'm running a bit low on time I'm very sorry about that this is the spelling for how to do hovers which I'm not going to talk about I'm going to whip through interactions to show you so this is but this is a quick example of something very nice I did with linking selections so when I click on a country this number here update the name of the country updates the value updates when I click over on this other tab I've also shared it across tabs and so Mali has updated here when I click on Ghana if I go back to my first tab Ghana will still be selected so that's a bit more sophisticated example of what you can do with that linking and last but not least I have four minutes and 14 seconds to whip you through callbacks and then Fabio is going to really take this and run with it this afternoon so callbacks have just been added to Bokeh and they're super awesome they allow you to write a tiny piece of JavaScript or a giant piece of JavaScript if you want to but you write some JavaScript in your Python code and you can start to do really sophisticated interactions and they're currently available on the sort of weird and somewhat ad hoc subset of all of the components of Bokeh and that's the list there and as a web developer I really love this because what I really want is that that really rich kind of experience nothing crazy fancy not doing sort of computation on the server or whatever but giving people the chance to really play with their infographic a little bit and this has really allowed me to do this so this is one I built recently they've got this slider at the bottom this is a Bokeh slider and what I'm doing it's very simple whenever I slide the value of this number at the back here changes and the data source that powers the circle sort of switches out so this is something that is a completely standalone piece of HTML all done in Bokeh using this new callback mechanism if you hate JavaScript more than life itself there are we're starting to in the same way that we made charts as these sort of canned one-liners that will sort of do all the magic for you we're starting to build up these actions so a common thing might be you tap on it you select a glyph and a URL opens so the first action we've made is open URL but there'll be more sort of pre-built things so you never have to write JavaScript coming here is an example of how to use of how to use the callback so excuse that typo where it just says code equals blank there so let's say so that we call we make a callback and we have some code like change the selected value to a new thing and when now instead of just adding a tool hover tool we add it with our callback what's exciting here and is a little hard to see but when you get used to it you realize how magic it is is I have passed I've passed in these arguments and what I've done is I've given a name my column data source and I've passed in the Python object so now when I in my JavaScript code I can use the word I can use my column data source as a as a JavaScript object and I can start playing with it and it's like I was playing with my Python object so you don't have to do any magic you can sort of be playing with your Python objects in in JavaScript and if you use that components method where you just have them all in one components method and all in one script that's how you can start interacting across all of your different plots and Fabio is going to to show you all about that later so with one minute left are there any questions? yeah you can shout I can repeat it yes you mean like a Google map or an open street map or whatever there oh sorry the question was it can you can you embed a Google street map a Google map or an open street map or something like that there is a there is actually a G map plot like bar chart or something like that like a plot that you can instantiate which works okay and but we wanted to make it better and one of the things that I meant to be doing this summer is actually working on our mapping support in general which is not as hot as we would like it but it's really high on my to-do list because it's something I want to see done so yeah Sara told us in the afternoon we have the possibility to learn a little bit more about bokeh so I want to thank you for your talk we have now


Experimental science is really the heart of our society. It's an attitude of mine that rejects authority and tries to find things out for itself. And I think this is really important in the modern world because it's full of complicated things that we all need to understand. We're inclined to think that there are these people called experts who will tell us the answer, but we need to be aware that they're finding things out themselves. There aren't always just right answers. And this is a message that we'd like to convey to children. People often think that science is only about things that happen in the lab, but we're trying to point out through the experiments that it's just a way of understanding the world. When you're actually with kids, they're so curious and they're so good at finding things out, but that's when it gets really rewarding. But we know it's something hugely innovative and new that no-one's ever heard of before. It's quite hard to un-push it and you get really messy as well, that's fun. We did the one where you had to stretch the plastic. We had to make a kind of rainbow collar and at the end we got to see it. I'm liking the lab in the right, because we've done really enough fun experiments about light and we've got to know more about it. There's a lot of stuff that people take for granted. They think that there's nothing else to discover in the world with iPods and wireless internet access and things like that. But there's still things to be discovered and we need scientists and we need to get the next generation scientists engaged. I think the lab in the lorry is one of the most spectacular things that the Institute of Physics has ever done in outreach terms and I think that the volunteers and the kids who use it, they really will get connected to science. I volunteered to be on lab in the lorry because I wanted to show people that science and especially physics wasn't just sitting in classrooms deriving equations, it was actually things you can do out in the real world. The students have been able to interact with the scientists that are there. They've seen the scientists as role models. These lorries are just so huge and so beautifully designed. They're just right for the job. It's just marvellous to see this all coming together. Everybody in the path they've taken life has always been one or two people who have been really influential. It could be your aunt who's taken you out with a telescope because a comet is going by or an uncle who takes apart a clock and shows you how it works. And that's what we're hoping that the lab in the lorry will be that uncle, that aunt, that friend of the family who just turned on a light bulb. I thought the lab in the lorry was really good because we got to try out these experiments that we'd never done before and they looked really cool. The volunteers were friendly because they explained it well. I think we can all remember just one incident at school that made a big difference to us and I hope that visiting the lorry will be that for the awful lot of children.